Descriptive Statistics and Exploratory Data Analysis

Introduction and context

Descriptive statistics converts raw observations into understandable evidence. In business analytics, a manager rarely needs to inspect every transaction individually; instead, the manager needs to know what is typical, how much values vary, where unusual observations occur, whether variables move together, and what patterns deserve further investigation. Descriptive statistics provides numerical summaries, while exploratory data analysis (EDA) combines those summaries with visual and contextual examination.

EDA is not merely a preliminary formatting exercise. It is a disciplined process for discovering structure, identifying data-quality problems, challenging assumptions, and generating questions for predictive or inferential analysis. It does not, by itself, prove causation. A strong analyst therefore distinguishes between a pattern observed in the data and a conclusion justified by a research design.

Learning objectives

By the end of this chapter, you should be able to:

• distinguish population and sample statistics;
• calculate and interpret measures of central tendency, dispersion, position, shape, and association;
• select summaries appropriate to the measurement scale and distribution;
• identify outliers, missing values, skewness, and other data-quality issues;
• construct and interpret effective statistical visualisations and dashboards;
• apply a structured EDA workflow to a business problem; and
• communicate findings without overstating what descriptive evidence can establish.

1. Data structures and analytical foundations

A population is the complete set of units of interest, such as every customer who purchased during a year. A sample is a subset observed to learn about that population. A numerical summary of a population is a parameter; a numerical summary calculated from a sample is a statistic. This distinction matters because sample statistics are subject to sampling variation.

Variables may be:

• categorical, such as region or payment method;
• ordinal, where categories have order but not necessarily equal distances, such as satisfaction ratings;
• discrete quantitative, such as number of complaints; or
• continuous quantitative, such as delivery time or revenue.

The measurement scale affects the operations that are defensible. The mode is meaningful for nominal categories; a mean of product codes is not. Medians are appropriate for ordered data and skewed quantitative data, while standard deviations require meaningful numerical distances.

Before calculating statistics, the analyst should establish the unit of analysis. A dataset containing one row per order is not equivalent to one containing one row per customer. Mixing levels can create misleading summaries, such as treating repeated orders as independent customers.

2. Measures of central tendency

Central tendency describes a representative or central value.

Arithmetic mean

For observations x₁, x₂, …, xₙ, the sample mean is:

Here, n is the number of observed values and xᵢ is observation i. The formula assumes numerical values for which addition and division are meaningful. The mean uses every observation, but it is sensitive to extreme values.

A weighted mean is appropriate when observations contribute unequally:

where wᵢ is a non-negative weight. The denominator must not be zero. Weights may represent quantities, sampling probabilities after adjustment, or revenue shares, but the analyst must document their meaning.

Median

The median is the middle ordered observation. For an odd number of observations it is the central value. For an even number, it is commonly the arithmetic mean of the two central values. It is resistant to extreme observations and is often preferable for income, transaction value, and response-time data.

Mode

The mode is the most frequently occurring value or category. A dataset can have more than one mode or no unique mode. It is particularly useful for categorical variables, such as the most common delivery option.

Geometric mean

For n positive values, the geometric mean is:

It is suitable for multiplicative growth factors and strictly positive ratios, not ordinary data containing zero or negative values. For growth rates, it is usually applied to factors such as 1.10 and 0.95 rather than directly averaging percentages.

Worked example 1: centre

Suppose five delivery times, in days, are 2, 3, 3, 4, and 13. The mean is (2 + 3 + 3 + 4 + 13) ÷ 5 = 5 days. The median is 3 days and the mode is 3 days. The mean is pulled upward by the 13-day delivery; the median better represents the typical customer experience. Reporting both is more informative than selecting one automatically.

3. Measures of dispersion

Two datasets can have the same mean but very different operational risk. Dispersion describes the spread of observations.

Range and interquartile range

The range is:

It is simple but depends entirely on two observations. The interquartile range (IQR) is:

where Q₁ is the first quartile and Q₃ is the third quartile. The IQR covers the middle 50% of observations and is resistant to extremes. Quartile calculation conventions can differ for small datasets, so the chosen software or method should be documented.

Variance and standard deviation

The sample variance is:

The sample standard deviation is:

The n − 1 denominator is used when estimating population variance from a sample under standard random-sampling assumptions. For a complete population, the descriptive population variance uses N in the denominator and population mean μ:

Variance is expressed in squared units, whereas standard deviation has the original units. Standard deviation is most interpretable when the distribution is reasonably symmetric and outliers are not dominating the result.

Coefficient of variation

For a positive mean, the coefficient of variation is:

It is often reported as 100 × CV%. It compares relative variability across measures with different units or scales. It is not meaningful when the mean is zero, close to zero, or negative, and comparisons require conceptually comparable variables.

Worked example 2: spread

For delivery times 2, 3, 3, 4, and 13, the mean is 5. Squared deviations are 9, 4, 4, 1, and 64, summing to 82. The sample variance is 82 ÷ 4 = 20.5 days², and the sample standard deviation is √20.5 ≈ 4.53 days. The range is 11 days. The large standard deviation relative to the median signals substantial inconsistency, largely caused by the 13-day case.

4. Measures of position

Position measures locate an observation within an ordered dataset.

A percentile is the value below which a specified percentage of observations falls. The 90th percentile of delivery time, for example, is a service-level measure: approximately 90% of observed deliveries are no slower than that value, subject to the percentile definition used.

Quartiles divide ordered data into four parts: Q₁, the 25th percentile; Q₂, the median; and Q₃, the 75th percentile. A five-number summary consists of the minimum, Q₁, median, Q₃, and maximum. It is the basis of a box plot.

A z-score expresses distance from the mean in standard-deviation units:

For a population standardisation, μ and σ replace x̄ and s. A z-score is dimensionless. It is useful for comparing relative positions across variables, but interpretation depends on the distribution and on a reliable estimate of spread.

Under an approximately normal distribution, the empirical rule suggests that about 68% of observations lie within one standard deviation of the mean, about 95% within two, and about 99.7% within three. These are approximations, not universal laws; skewed or heavy-tailed data do not satisfy them well.

5. Shape: skewness, tails, and modality

Skewness describes asymmetry. Right skewness has a long or heavy upper tail and often produces mean > median. Left skewness has a longer lower tail and often produces mean < median. These relationships are indicative rather than absolute.

Kurtosis concerns tail weight and concentration relative to a reference distribution. Software may report either kurtosis or excess kurtosis, so the definition must be checked before comparison. Heavy tails increase the probability of extreme observations and can make means and standard deviations unstable.

A distribution may be unimodal, bimodal, or multimodal. A bimodal sales distribution may indicate two customer segments, weekday and weekend processes, or a mixture of regions. Aggregation can conceal these structures; analysts should inspect relevant subgroups.

Histograms, density plots, box plots, empirical cumulative distribution plots, and Q–Q plots provide complementary evidence. Bin width affects a histogram’s appearance, and smoothing affects a density plot; conclusions should not depend on one arbitrary setting.

6. Association between variables

Association describes how variables vary together. It does not establish that one causes the other.

Covariance

Sample covariance is:

Positive covariance means that above-average values of one variable tend to occur with above-average values of the other. Its magnitude depends on measurement units, limiting direct comparison.

Pearson correlation

Pearson’s sample correlation is:

where sₓ and sᵧ are sample standard deviations. Provided both standard deviations are non-zero, r lies between −1 and +1. Values near +1 indicate strong linear association, values near −1 strong negative linear association, and values near 0 little linear association. A correlation near zero can coexist with a strong nonlinear relationship.

Pearson correlation is sensitive to outliers and should be examined with a scatter plot. For ordinal data, monotonic but non-linear relationships, or data containing influential extremes, Spearman’s rank correlation may be more appropriate. Correlation can also be distorted by range restriction, aggregation, confounding, and mixing distinct groups.

For a simple linear regression with an intercept, r² is the proportion of sample variation in the response linearly associated with the fitted predictor, but it is not proof of causation or a complete measure of predictive usefulness.

7. Detecting outliers and data-quality problems

An outlier is an observation unusually distant from the general pattern. It may be a recording error, a rare but valid event, a fraud signal, a process failure, or evidence of a separate population. It should never be removed merely because it is inconvenient.

A common box-plot rule flags values below or above:

This is a screening rule, not a hypothesis test or automatic deletion rule. A value beyond a fence is conventionally called a potential outlier; it may be perfectly legitimate.

A practical investigation sequence is:

1. verify the record, unit, timestamp, and data-entry process;
2. determine whether the observation is possible in the business context;
3. compare it with relevant segments and time periods;
4. assess its effect on mean, standard deviation, correlation, and models;
5. document any correction, exclusion, winsorisation, or transformation; and
6. report sensitivity analyses when the decision is consequential.

Missingness also requires investigation. Missing values may be missing completely at random, related to observed variables, or related to the unobserved value itself. A missing-value percentage by variable and subgroup is a useful first diagnostic. Imputation is an analytical decision, not a substitute for understanding why data are absent.

8. Visualisation and dashboard design

A visualisation should answer a defined question. Choose the chart according to the analytical task:

Effective dashboards use a visual hierarchy: a small number of decision-relevant key performance indicators (KPIs), trend context, segmentation, and an ability to drill into exceptions. Titles should state the message or question, axes should include units, colour should have a defined meaning, and filters should display their current state. Decorative 3-D effects, excessive colours, truncated axes, and dual axes without clear explanation can mislead.

A dashboard should distinguish descriptive metrics from targets and forecasts. For example, “median delivery time” is descriptive, “target ≤ 3 days” is a managerial benchmark, and “forecast next month” is predictive. Combining them without labels encourages false certainty.

9. A structured EDA workflow

A robust workflow is iterative:

1. Define the decision. Clarify the business question, population, time window, and unit of analysis.
2. Audit the data. Check types, ranges, duplicates, missingness, units, keys, and time stamps.
3. Describe univariate behaviour. Use counts, proportions, central tendency, spread, percentiles, and distributions.
4. Compare meaningful groups. Examine region, product, customer segment, channel, and time period without indiscriminate subgroup searching.
5. Explore associations. Use cross-tabulations, scatter plots, correlations, and carefully selected transformations.
6. Investigate anomalies. Trace unusual records to source systems and domain events.
7. Test robustness. Compare conclusions using median versus mean, with and without influential observations, and across reasonable definitions.
8. Communicate and document. Record data preparation, calculation conventions, exclusions, uncertainty, and limitations.

Code and dashboards should be reproducible. Versioned datasets, documented transformations, controlled metric definitions, and a data dictionary reduce the risk that two teams report different figures for the same KPI.

Applied case: illustrative online retailer delivery analysis

Background and problem

An illustrative online retailer receives complaints that delivery performance is deteriorating. Operations leaders ask whether the problem is widespread and which service requires attention. The analyst has 50,000 order records containing order date, promised date, actual delivery date, region, carrier, and customer segment.

Analysis

The analyst first defines one order as the unit of analysis and calculates delivery days consistently as the difference between actual and order dates. Duplicate order IDs are investigated rather than silently dropped. Missing actual-delivery dates are reported separately because they may represent open orders, not random missingness.

The overall mean is 4.8 days, while the median is 3.1 days and the 90th percentile is 9.6 days. A histogram and box plot show strong right skew. Regional box plots reveal that the mean is highest in the remote region, but the IQR is also wider there. A scatter plot of order volume against delivery time shows no simple linear relationship. Correlations are calculated only for variables where a linear association is meaningful.

Potential outliers beyond the 1.5-IQR fences are reviewed. Some are legitimate weather disruptions; others contain incorrect date entries. The analyst corrects verified data-entry errors using the source record and retains legitimate disruptions, marking their cause.

Approach and outcome

The dashboard presents median delivery time, 90th percentile, late-order rate, and order volume by region and carrier. It separates completed orders from open orders and displays the reporting period and definitions. Management discovers that the overall median has changed little, but the 90th percentile and late-order rate have increased sharply for one carrier in the remote region. A carrier review and contingency routing plan are initiated.

Learning points

The mean alone would have understated the customer experience of the slowest orders. Segmenting prevented aggregation from hiding a localised problem. Outlier investigation distinguished operational events from data errors. Finally, descriptive analysis identified where to act but did not prove that the carrier caused every delay; a subsequent operational investigation is required.

Limitations, ethics, and governance

Descriptive statistics depend on data quality, coverage, definitions, and sampling. A dashboard can create an appearance of precision when the underlying process is unstable or the sample is biased. Small subgroup counts can reveal individuals, especially when combined with other attributes. Access controls, aggregation, anonymisation, retention limits, and privacy-by-design are therefore essential.

Analysts should avoid discriminatory proxy interpretations, such as treating geographic differences as inherent customer characteristics when they may reflect infrastructure or service allocation. Automated anomaly detection and AI-assisted visualisation can accelerate EDA, but generated summaries must be checked against source data, definitions, subgroup effects, and disclosure risks. AI does not remove the analyst’s responsibility for validation, explainability, and governance.

Contemporary practice increasingly combines interactive dashboards with statistical notebooks, data-quality observability, semantic metric layers, and automated drift monitoring. These tools are valuable only when metric definitions and accountability are explicit.

Best-practice checklist

• Start with the business decision and unit of analysis.
• Report centre and spread together.
• Use robust summaries for skewed data.
• Show distributions, not only averages.
• Investigate rather than automatically delete outliers.
• Separate data errors from valid rare events.
• Label population, sample, period, units, and missingness.
• Treat correlation as association, not causation.
• Use accessible, non-deceptive visual encodings.
• Document transformations, conventions, and revisions.
• Protect privacy and review subgroup effects.
• Validate automated and AI-generated outputs.

Chapter Summary

Descriptive statistics summarises data through measures of central tendency, dispersion, position, shape, and association. The mean is efficient but sensitive to extremes; the median and IQR are robust alternatives for skewed data. Variance and standard deviation quantify spread, while percentiles and z-scores describe relative position. Skewness, modality, and tail behaviour should be examined visually as well as numerically.

Exploratory data analysis is an iterative process of auditing, summarising, visualising, comparing, investigating anomalies, and communicating evidence. Outlier rules such as the 1.5-IQR fences support investigation but do not justify automatic removal. Correlation measures linear association and cannot establish causality. Effective dashboards connect clearly defined metrics to decisions, disclose assumptions, and avoid misleading design. Professional EDA also requires reproducibility, privacy protection, ethical judgement, and critical evaluation of automated tools.

Key Terms

Population: The complete set of units about which conclusions are sought.

Sample: A subset of a population that is observed and analysed.

Parameter: A numerical summary of a population.

Statistic: A numerical summary calculated from a sample.

Mean: The arithmetic average of numerical observations.

Median: The middle value in an ordered dataset.

Mode: The most frequently occurring value or category.

Variance: The average squared deviation, with the denominator depending on population or sample context.

Standard deviation: The square root of variance, expressed in the original units.

Interquartile range: Q₃ − Q₁, representing the middle 50% of observations.

Percentile: A value indicating a position below which a specified percentage of observations falls.

Z-score: A standardised distance from a mean measured in standard-deviation units.

Skewness: Asymmetry in a distribution.

Outlier: An observation unusually distant from the prevailing pattern.

Covariance: A unit-dependent measure of joint variation between two variables.

Correlation: A standardised measure of linear association.

Exploratory data analysis: Iterative investigation using numerical summaries, visualisations, and contextual reasoning.

Dashboard: An organised visual interface for monitoring measures and supporting decisions.

Unit of analysis: The entity represented by one observation or row.

Review Questions

Short Answer Questions

1. Distinguish a population parameter from a sample statistic.
2. Why can the median be preferable to the mean for delivery-time data?
3. What does the IQR measure?
4. State one reason why a correlation near zero may be misleading.
5. Why should an observation beyond a box-plot fence not be deleted automatically?

Long Answer Questions

1. Explain how mean, median, mode, standard deviation, and IQR complement one another in a business report.
2. Describe a complete EDA workflow for a customer-transaction dataset.
3. Explain the assumptions, interpretation, and limitations of Pearson correlation.
4. Compare histograms, box plots, scatter plots, and line charts, including when each is appropriate.
5. Discuss ethical and governance requirements for a management dashboard containing customer data.

Analytical / Application Questions

1. A retailer reports an average basket value of $86, a median of $42, and a few very large corporate orders. Which measure should headline a consumer-shopping dashboard, and what additional evidence should be shown?
2. A carrier’s mean delivery time is unchanged, but its 90th percentile rises from 7 to 12 days. What distributional change might this indicate, and which visualisations would you use to investigate it?
3. Two regions have the same average complaint rate, but one has a much larger IQR and several extreme observations. Propose an analysis and operational response that does not rely on the mean alone.

Multiple Choice Questions with Answers

Question 1

Which measure is generally most resistant to an extreme value?

A. Mean

B. Median

C. Variance

D. Range

Correct Answer: B. Median

Question 2

For the values 2, 3, 3, 4, and 13, what is the median?

A. 2

B. 3

C. 4

D. 5

Correct Answer: B. 3

Question 3

What does the interquartile range represent?

A. The total distance between minimum and maximum

B. The average squared deviation from the mean

C. The spread of the middle 50% of observations

D. The percentage of observations above the mean

Correct Answer: C. The spread of the middle 50% of observations

Question 4

Which formula is the sample standard deviation based on?

A. Squared deviations divided by n

B. Squared deviations divided by n − 1, followed by a square root

C. Absolute deviations divided by n + 1

D. The range divided by the median

Correct Answer: B. Squared deviations divided by n − 1, followed by a square root

Question 5

A correlation of −0.85 most directly indicates:

A. Strong negative linear association

B. Strong positive causation

C. No relationship of any kind

D. That both variables have negative values

Correct Answer: A. Strong negative linear association

Question 6

Under the common box-plot rule, the upper potential-outlier fence is:

A. Q₁ + 1.5(IQR)

B. Q₃ + 1.5(IQR)

C. Mean + 2(IQR)

D. Maximum + 1.5(IQR)

Correct Answer: B. Q₃ + 1.5(IQR)

Question 7

Which display is most directly suited to examining the relationship between two quantitative variables?

A. Pie chart

B. Scatter plot

C. Bar chart of one category

D. Single KPI card

Correct Answer: B. Scatter plot

Question 8

For the delivery times 2, 3, 3, 4, and 13, what is the arithmetic mean?

A. 3

B. 4

C. 5

D. 13

Correct Answer: C. 5

Question 9

Why can a mean and standard deviation be problematic for a heavily skewed dataset?

A. They cannot be calculated for numerical data

B. They are sensitive to extreme observations

C. They always equal the median and IQR

D. They describe only categorical variables

Correct Answer: B. They are sensitive to extreme observations

Question 10

Which statement about correlation is correct?

A. Correlation alone proves causation

B. A zero correlation rules out nonlinear association

C. Correlation measures association and should be interpreted with context and plots

D. Correlation is unaffected by outliers

Correct Answer: C. Correlation measures association and should be interpreted with context and plots