Descriptive Statistics and Exploratory Data Analysis

Introduction and Context

Business analytics begins with a deceptively simple question: what does the data show? Before forecasting demand, optimizing operations, or building a predictive model, an analyst must understand the structure, quality, and patterns of the available data. Descriptive statistics and exploratory data analysis (EDA) provide the first systematic examination of a dataset.

Descriptive statistics condense a large collection of observations into informative numerical summaries, tables, and visualizations. EDA goes further by combining these summaries with graphical and investigative techniques to identify distributions, relationships, unusual observations, data-quality problems, and possible explanations. Neither approach proves causation. Instead, both help analysts formulate sound questions and avoid premature conclusions.

For example, a retailer may know the revenue generated by 10,000 transactions but still need to determine whether the typical transaction is small or large, whether a few unusually large purchases distort the average, which stores vary most in performance, and whether discounting is associated with higher sales. These are descriptive and exploratory questions.

Learning Objectives

By the end of this chapter, you should be able to:

1. Distinguish populations, samples, variables, observations, and measurement scales.
2. Calculate and interpret measures of central tendency and position.
3. Calculate and interpret measures of dispersion and relative variability.
4. Recognize skewness, distributional shape, and the effect of outliers.
5. Explore association between quantitative and categorical variables.
6. Use an appropriate sequence of tables, graphs, and statistics for EDA.
7. Communicate descriptive findings without overstating what the data establishes.
8. Apply these techniques to a practical business problem.

1. The Data Description Problem

A dataset is usually organized as a rectangular table. Each row represents an observation, such as a customer, order, employee, or day. Each column represents a variable, such as revenue, age, region, or delivery time.

The population is the complete set of units relevant to a question. A sample is a subset used to learn about that population. A parameter describes a population, whereas a statistic describes a sample. For example, the true mean annual spending of all customers is a population parameter; the mean spending of 500 sampled customers is a sample statistic.

Variables should also be classified by their measurement scale:

A variable may be quantitative or categorical. Quantitative variables may be discrete, such as number of complaints, or continuous, such as delivery time. Correct classification matters because a statistic appropriate for one variable may be meaningless for another. For instance, calculating the mean of customer identification numbers has no interpretive value.

2. Organizing Data Before Analysis

EDA begins with data preparation rather than calculation. The analyst should inspect:

• variable names, definitions, and units;
• data types and allowable values;
• missing, duplicated, or inconsistent records;
• impossible values, such as negative quantities sold;
• the time period and population represented;
• whether observations are independent;
• whether a variable has been recorded in pounds, kilograms, dollars, or another unit.

A frequency table lists the number of observations in each category or numerical interval. A relative frequency is the proportion in a category:

A percentage is the relative frequency multiplied by 100. Frequency tables are particularly useful for categorical variables and for detecting rare or unexpected categories.

For numerical data, grouped intervals can be useful, but interval boundaries must be defined clearly. A chart should not suggest false precision or hide important observations through excessively wide intervals.

3. Measures of Central Tendency

Measures of central tendency describe a typical or central value. The most common are the mean, median, and mode.

3.1 Arithmetic Mean

For a population of N values, the population mean is:

For a sample of n values, the sample mean is:

Here, xᵢ is observation i, N is the population size, n is the sample size, μ is the population mean, and x̄ is the sample mean. The mean has the same units as the original variable and is appropriate for quantitative data when arithmetic averaging is meaningful.

The mean uses every observation and is therefore sensitive to extreme values. A mean revenue of $84 does not necessarily mean that a typical transaction was $84 if a few corporate purchases were exceptionally large.

3.2 Median

The median is the middle value after observations are arranged in ascending order. If n is odd, the median is the observation at position (n + 1) ÷ 2. If n is even, it is the average of the two central observations.

The median is resistant to extreme observations and is often preferred for income, house prices, waiting times, and transaction values with right-skewed distributions. It can also be used for ordinal data when the ordering is meaningful.

3.3 Mode

The mode is the most frequently occurring value or category. A dataset may have one mode, multiple modes, or no repeated value. The mode is particularly useful for nominal variables, for which a mean and median are inappropriate.

3.4 Weighted Mean

Some business data require observations to contribute unequally. The weighted mean is:

where xᵢ is a value and wᵢ is its non-negative weight. Weights may represent quantities, importance, or group sizes. For example, an overall customer satisfaction score should usually weight branch averages by the number of respondents, not simply average branch averages.

Worked Example 1: Central Tendency

Suppose five delivery times, in minutes, are 18, 20, 21, 22, and 49.

The mean is:

The median is 21 minutes, the third ordered observation. There is no mode because no value repeats. The mean is much higher than the median because the 49-minute delivery is unusually long. Reporting both values provides more information than reporting the mean alone.

4. Measures of Position and Spread

A central value is incomplete without information about variability. Two businesses may have the same average delivery time but very different reliability.

4.1 Range

The range is:

It is easy to calculate but depends only on two observations and is highly sensitive to outliers.

4.2 Percentiles and Quartiles

The pth percentile is a value below which approximately p percent of observations lie. The 50th percentile is the median. The first quartile, Q₁, is the 25th percentile; the second quartile, Q₂, is the median; and the third quartile, Q₃, is the 75th percentile.

Different software packages use slightly different percentile interpolation conventions. Analysts should document the convention when exact reproducibility matters. The interpretation remains more important than a small computational difference.

The interquartile range is:

The IQR measures the spread of the middle 50 percent of observations and is resistant to extreme values.

4.3 Variance and Standard Deviation

For a population, variance is:

For a sample, the usual unbiased estimator of population variance is:

The population standard deviation and sample standard deviation are the positive square roots:

The symbols μ and x̄ are the relevant population and sample means. The denominator n − 1 is used for the conventional sample variance because estimating the mean from the same sample uses one degree of freedom. Variance is expressed in squared units, while standard deviation returns to the original units.

Standard deviation is most informative when the mean is a suitable centre and the distribution is not extremely skewed. It is not a direct statement that every observation lies within one standard deviation of the mean.

Worked Example 2: Sample Spread

Consider weekly complaints from four stores: 2, 4, 4, and 6. The sample mean is 4. Deviations from the mean are −2, 0, 0, and 2. Squared deviations are 4, 0, 0, and 4, giving a sum of 8.

The standard deviation indicates the typical distance from the sample mean in the measurement unit, although “typical” should not be interpreted as an exact distance for every observation.

4.4 Coefficient of Variation

The coefficient of variation is:

Often it is reported as a percentage by multiplying by 100. The CV is dimensionless and allows relative variability to be compared across variables with different scales, provided the mean is positive and meaningfully away from zero. It is unsuitable when the mean is zero or close to zero, and it can be misleading for variables that may take negative values.

5. Distributional Shape

A distribution describes how observations are arranged across possible values. Important features include centre, spread, skewness, modality, and tails.

A roughly symmetric distribution has similar patterns on either side of its centre. In a right-skewed distribution, a long tail extends toward larger values; commonly the mean exceeds the median. In a left-skewed distribution, the tail extends toward smaller values; commonly the mean is below the median. These relationships are tendencies, not universal rules.

A distribution can be unimodal, bimodal, or multimodal. Multiple modes may indicate different customer segments, regions, products, or operating processes. Analysts should investigate such patterns rather than automatically treating them as errors.

The empirical rule is a useful approximation for a bell-shaped, approximately normal distribution: roughly 68% of observations lie within one standard deviation of the mean, 95% within two, and 99.7% within three. This rule should not be applied automatically to skewed or multimodal business data.

A standardized value, or z-score, is:

for a sample, or (xᵢ − μ) ÷ σ when population parameters are known. A z-score states how many standard deviations an observation is above or below the relevant mean. It is meaningful only when the mean and standard deviation are suitable summaries and the measurement process is comparable.

6. Outliers and Unusual Observations

An outlier is an observation that is unusually distant from the rest of the data. An outlier may result from:

• a data-entry or measurement error;
• a legitimate but rare event;
• a different population or business process;
• fraud or system manipulation;
• a change in operating conditions.

A common exploratory rule identifies potential outliers below:

and above:

This is a screening rule, not a law of nature. A value outside a fence should be investigated, not automatically deleted. Extreme values may contain the most valuable business information, such as a major service failure or an unusually profitable customer.

A box plot summarizes the median, quartiles, IQR, and potential outliers. A histogram shows the distribution of a quantitative variable. A scatter plot shows the relationship between two quantitative variables. A bar chart is generally appropriate for categories, while a line chart is often appropriate for ordered time observations. Using a bar chart to imply continuity in a time series, or using a pie chart for many similar categories, can reduce clarity.

7. Association Between Variables

Association means that variables show a systematic relationship. It does not establish that one variable causes another.

7.1 Covariance and Correlation

Sample covariance is:

where xᵢ and yᵢ are paired observations, and x̄ and ȳ are their sample means. Covariance depends on the units of measurement, so its magnitude is difficult to compare across datasets.

Pearson’s sample correlation is:

where sₓ and sᵧ are the sample standard deviations of X and Y. Equivalently, it is the standardized covariance. Correlation has no units and lies between −1 and +1. A positive value indicates that larger values of one variable tend to occur with larger values of the other; a negative value indicates the reverse. A value near zero indicates no linear association, but nonlinear association may still exist.

Correlation can be distorted by outliers and may be inappropriate for strongly nonlinear or ordinal relationships. It also cannot distinguish causation from confounding. For example, advertising expenditure and sales may be positively correlated because both increase during peak seasons.

7.2 Categorical Association

For categorical variables, analysts can use cross-tabulations, row or column percentages, and visualizations such as grouped or stacked bar charts. Percentages must be labelled clearly: the percentage of customers in each region who renewed is different from the percentage of all renewals contributed by each region.

For example, a cross-tabulation of subscription status by customer segment may show that renewal rates differ by segment. This is a descriptive pattern requiring further investigation; it is not proof that segment membership causes renewal.

7.3 Association Is Not Causation

A causal claim requires more than a descriptive relationship. Analysts should consider time order, alternative explanations, selection bias, measurement quality, and experimental or quasi-experimental evidence. EDA is valuable because it reveals patterns that can guide later causal or predictive analysis, but it does not replace those methods.

8. A Practical EDA Workflow

A disciplined EDA workflow may follow these stages:

1. Define the business question. Clarify the decision, unit of analysis, time period, and target population.
2. Audit the data. Check types, units, missingness, duplicates, invalid values, and unusual categories.
3. Describe individual variables. Use counts and percentages for categories; use centre, spread, percentiles, and graphs for quantitative variables.
4. Compare relevant groups. Examine differences in medians, means, IQRs, and distributions across stores, products, or customer segments.
5. Explore relationships. Use cross-tabulations, grouped graphs, scatter plots, and correlations where appropriate.
6. Investigate anomalies. Trace unusual records to source systems and business events.
7. Document decisions. Record exclusions, transformations, definitions, and software settings.
8. Communicate findings. State what is observed, how certain the description is, and what further analysis is required.

Good EDA is iterative. A histogram may reveal a second group, prompting a comparison by region. A box plot may reveal extreme delivery times, prompting investigation of a particular courier. The process is analytical rather than mechanical.

9. Detailed Illustrative Case: Online Retail Delivery Performance

Background

The operations team of an illustrative online retailer wants to understand delivery performance for 12 recent orders. Delivery time is measured from dispatch to customer receipt, in days:

1, 2, 2, 2, 3, 3, 3, 4, 4, 5, 6, 14.

The company’s informal target is delivery within three days.

Problem

Management needs to know the typical delivery time, the extent of variation, whether the target is commonly met, and whether the unusually long delivery should be treated as an error or an operational signal.

Analysis

The ordered data have 12 observations. The sum is 49, so:

The median is the average of the sixth and seventh observations, both 3, so the median is 3 days. The mode is 2 days and 3 days, making the distribution bimodal by frequency, although the modes are adjacent.

Using the median-of-halves method for this small illustrative dataset, the lower half is 1, 2, 2, 2, 3, 3, giving Q₁ = 2. The upper half is 3, 4, 4, 5, 6, 14, giving Q₃ = (4 + 5) ÷ 2 = 4.5. Therefore:

The upper fence is:

Thus, 14 days is a potential high outlier. It should be investigated rather than removed automatically.

Nine of the 12 orders took three days or less, so the target attainment rate is:

The mean of approximately 4.08 days is substantially higher than the median of 3 days because the 14-day order pulls the mean upward. If that order was caused by a documented warehouse shutdown, the team might report routine performance separately from disruption performance—but it should disclose the rule and retain the exceptional event for risk analysis.

Approach and Possible Solution

The analyst should:

• verify the 14-day record against dispatch and receipt timestamps;
• check whether it relates to a particular warehouse, courier, product, or destination;
• compare delivery distributions by courier and region;
• report both median and mean;
• report the 75% target attainment rate;
• examine whether the target is appropriate for all product categories;
• monitor the metric over a longer period before making a policy decision.

A dashboard might display median delivery time, IQR, target attainment, the distribution by courier, and a list of investigated exceptions. A single average would not adequately represent performance.

Outcome

The initial analysis suggests that a typical order arrives in three days, but occasional severe delays create operational risk. Management should focus not only on reducing average time but also on preventing or managing extreme delays.

Learning Points

1. Mean and median can tell different stories.
2. Outlier rules identify records for investigation; they do not determine business treatment.
3. Percentages require a clearly defined denominator.
4. A business target should be evaluated alongside distributional variation.
5. Descriptive analysis can reveal questions for operational and causal investigation.

10. Tools, Technology, and Reproducibility

Spreadsheets are suitable for small datasets and basic functions such as AVERAGE, MEDIAN, QUARTILE, COUNTIF, and STDEV.S. SQL is useful for grouping, filtering, counting, and summarizing data stored in databases. Python, R, and business-intelligence platforms support repeatable EDA, interactive visualizations, and larger datasets.

Technology does not remove the need for judgement. Automated dashboards can reproduce an incorrect metric if the definition, denominator, time zone, or missing-value treatment is wrong. AI-assisted analytics may suggest patterns or generate code, but analysts must validate calculations, inspect source data, protect confidential information, and document how outputs were produced.

Reproducibility requires preserving the source-data version, transformation steps, variable definitions, code or spreadsheet formulas, and decisions about missing or extreme values. Version control and peer review are valuable professional practices.

11. Limitations, Ethics, and Governance

Descriptive statistics may conceal important differences when data are aggregated. An overall renewal rate can hide poor performance in a particular region. This is one reason analysts should examine relevant subgroups while avoiding unjustified or invasive profiling.

Important governance issues include:

• Privacy: use only data necessary for the business purpose and restrict access to personal information.
• Fairness: check whether missingness, sampling, or group definitions produce misleading comparisons.
• Transparency: define metrics such as active customer, completed order, and late delivery.
• Data quality: distinguish a genuine outlier from a recording error and preserve an audit trail.
• Confidentiality: avoid exposing individual-level results when aggregated reporting is sufficient.
• Responsible communication: do not imply causation, certainty, or generalizability beyond the data.

A polished chart can create false confidence. Every result should be interpreted in relation to the sampling process, measurement method, and business context.

Best Practices

• Begin with the decision and unit of analysis, not with a favourite chart.
• Report centre and spread together.
• Use the median and IQR when distributions are skewed or contain influential extremes.
• Label units, time periods, denominators, and missing-data treatments.
• Inspect both numerical summaries and graphs.
• Compare subgroups only when the comparison is substantively justified.
• Investigate outliers before deleting or winsorizing them.
• Treat correlation as association, not causation.
• Keep an auditable record of transformations and exclusions.
• Communicate limitations as part of the finding, not as an afterthought.

Chapter Summary

Descriptive statistics summarize data through measures of centre, position, spread, and association. The mean uses all observations but is sensitive to extreme values; the median is resistant and often useful for skewed data; and the mode is especially suitable for categorical variables. Range, IQR, variance, standard deviation, and coefficient of variation describe variability under different conditions.

EDA combines numerical summaries, tables, graphs, subgroup comparisons, and anomaly investigation. Distributional shape, skewness, multimodality, and outliers can substantially affect interpretation. Pearson correlation describes linear association between quantitative variables but does not establish causation. Cross-tabulations and percentages support the exploration of categorical relationships.

A sound EDA workflow defines the business question, audits data quality, describes individual variables, explores relevant relationships, investigates unusual observations, documents decisions, and communicates limitations. In professional analytics, accurate calculation must be accompanied by contextual judgement, ethical governance, and reproducibility.

Key Terms

Population: The complete set of units relevant to a study.

Sample: A subset of a population used for analysis.

Parameter: A numerical characteristic of a population.

Statistic: A numerical characteristic calculated from a sample.

Variable: A recorded characteristic that can take different values.

Observation: One recorded unit or case in a dataset.

Mean: The arithmetic sum of values divided by their number.

Median: The middle ordered value, or average of the two middle values.

Mode: The most frequently occurring value or category.

Percentile: A value below which approximately a specified percentage of observations lies.

Quartile: A division point representing a quarter of an ordered distribution.

Interquartile range: The difference between the third and first quartiles.

Variance: The average squared deviation from the mean, with the appropriate population or sample denominator.

Standard deviation: The positive square root of variance.

Coefficient of variation: Standard deviation divided by the mean.

Outlier: An observation unusually distant from the rest of the data.

Correlation: A standardized measure of linear association between two quantitative variables.

Exploratory data analysis: The systematic use of summaries and visualizations to investigate data patterns and quality.

Skewness: Asymmetry in a distribution, commonly expressed through a longer tail on one side.

Cross-tabulation: A table showing the joint counts or percentages of categorical variables.

Review Questions

Short Answer Questions

1. Distinguish a population parameter from a sample statistic.
2. Why is the median often preferred to the mean for highly right-skewed transaction values?
3. What does the interquartile range measure?
4. What is the purpose of a frequency table?
5. Why does correlation not establish causation?

Long Answer Questions

1. Explain the differences among nominal, ordinal, interval, and ratio measurement scales, giving one business example of each.
2. Compare the mean, median, and mode, including their strengths, limitations, and appropriate uses.
3. Explain population variance and sample variance, including why the sample formula uses n − 1.
4. Describe a complete EDA workflow for evaluating customer churn data.
5. Discuss how outliers can be errors, legitimate events, or signals of a different process, and explain how an analyst should respond.

Analytical / Application Questions

1. A store records weekly sales of 80, 82, 84, 85, 86, and 160 units. Calculate the mean and median, describe the distribution, and recommend which measure should be emphasized in a report.
2. A manager reports a correlation of 0.78 between advertising expenditure and sales. Identify at least three reasons why this result should not automatically be interpreted as advertising causing sales.
3. A service team has Q₁ = 4 minutes and Q₃ = 10 minutes for waiting time. Calculate the IQR and upper outlier fence, then explain what action should be taken if an observation is 22 minutes.

Multiple Choice Questions with Answers

Question 1

Which measure of centre is generally most resistant to an extreme value?

A. Mean

B. Median

C. Variance

D. Range

Correct Answer: B. Median

Question 2

A dataset contains the values 4, 6, 8, and 12. What is the sample mean?

A. 6.5

B. 7.0

C. 7.5

D. 8.0

Correct Answer: C. 7.5

Question 3

If Q₁ = 10 and Q₃ = 18, what is the interquartile range?

A. 8

B. 14

C. 28

D. 36

Correct Answer: A. 8

Question 4

Which graph is usually most appropriate for examining the relationship between two quantitative variables?

A. Pie chart

B. Scatter plot

C. Bar chart

D. Frequency table

Correct Answer: B. Scatter plot

Question 5

An observation above Q₃ + 1.5 × IQR is commonly classified in exploratory analysis as what?

A. A guaranteed error

B. A population parameter

C. A potential outlier

D. A median

Correct Answer: C. A potential outlier

Question 6

A correlation of −0.85 indicates what type of relationship?

A. Strong negative linear association

B. Weak positive linear association

C. Proof of negative causation

D. No possible relationship

Correct Answer: A. Strong negative linear association

Question 7

Which statistic is especially appropriate for a nominal variable such as product category?

A. Mean

B. Standard deviation

C. Mode

D. Coefficient of variation

Correct Answer: C. Mode

Question 8

For the sample values 2, 4, 4, and 6, the sample variance is approximately what?

A. 1.33

B. 2.00

C. 2.67

D. 4.00

Correct Answer: C. 2.67

Question 9

Why should a potential outlier not be deleted automatically?

A. Outliers always improve the mean

B. It may represent a legitimate and important business event

C. The IQR cannot identify unusual values

D. Deletion always increases sample size

Correct Answer: B. It may represent a legitimate and important business event

Question 10

Which statement best describes exploratory data analysis?

A. It proves causal relationships from descriptive patterns

B. It replaces data-quality checks

C. It uses summaries and visualizations to investigate patterns and anomalies

D. It requires every variable to be normally distributed

Correct Answer: C. It uses summaries and visualizations to investigate patterns and anomalies