Descriptive Statistics and Data Summarisation

Introduction

Descriptive statistics provides the first systematic description of a dataset. Before a machine-learning model is trained or a business decision is made, analysts need to know what the data contain, how values are distributed, whether unusual observations are present, and whether variables move together. Descriptive statistics does not, by itself, prove causation or generalise from a sample to a population. Its purpose is to organise, summarise and communicate observed data accurately.

In artificial intelligence and machine learning, descriptive analysis supports data cleaning, feature selection, exploratory data analysis, model diagnostics and communication with stakeholders. A single average can conceal important differences; a graph can reveal skewness that a table does not; and a correlation can be mistaken for a causal relationship. Sound analysis therefore combines numerical summaries, visualisation, contextual knowledge and critical judgement.

Learning Objectives

By the end of this chapter, you should be able to:

• construct and interpret frequency distributions;
• calculate and interpret measures of central tendency, dispersion and position;
• describe skewness, outliers and the shape of distributions;
• calculate covariance and correlation and explain their limitations;
• select suitable summaries for numerical and categorical variables;
• recognise misleading aggregates, inappropriate comparisons and aggregation bias;
• apply descriptive statistics to an AI or business analytics problem.

1. Types of Data and the Purpose of Summarisation

A variable is a characteristic recorded for each observation. An observation may be a customer, transaction, patient, image, machine or web session. Variables are commonly classified as follows:

The measurement scale matters. Averaging department names is meaningless, while calculating the mean of temperature may be meaningful. A five-point satisfaction scale is ordered, but the difference between ratings 1 and 2 may not represent exactly the same amount of satisfaction as the difference between 4 and 5. Consequently, medians and proportions are often safer summaries for ordinal data.

A population contains all units of interest, whereas a sample is a subset. Population quantities are often denoted by μ for the mean and σ² for the variance. Sample quantities are commonly denoted by x̄ for the mean and s² for the variance. The distinction is important because sample variance uses a correction for estimating population variability from incomplete data.

2. Frequency Distributions

A frequency distribution records how often each value or category occurs. The frequency of a value is its count. Relative frequency is its proportion of the total:

where fᵢ is the frequency of category or value i and n is the total number of observations. Percentage frequency is the relative frequency multiplied by 100.

For numerical data with many distinct values, observations can be grouped into class intervals. A useful frequency table includes the interval, frequency, relative frequency and cumulative frequency. Class intervals should be mutually exclusive and collectively exhaustive. For continuous measurements, boundaries must be specified so that an observation cannot belong to two classes.

Worked Example 1: Frequency Distribution

Suppose the numbers of support tickets received over 12 days are:

4, 7, 5, 4, 9, 6, 4, 8, 5, 7, 6, 4.

The frequency table is:

The mode is 4 tickets because it occurs most often. The cumulative frequency shows, for example, that 8 of the 12 days had no more than 6 tickets. A bar chart is appropriate for these discrete values; a histogram is generally used for continuous measurements grouped into adjacent intervals.

For a grouped table, the approximate mean is calculated using each class midpoint mᵢ:

This is an approximation because all observations in a class are represented by its midpoint. The approximation becomes less reliable when intervals are wide or observations are unevenly distributed within intervals.

3. Measures of Central Tendency

Central tendency describes a typical or central value.

Arithmetic Mean

For n observations x₁, x₂, …, xₙ, the sample arithmetic mean is:

The mean uses every observation and is appropriate for numerical data when values are measured on a meaningful numerical scale. It is sensitive to extreme values. A weighted mean is used when observations have different weights:

where wᵢ is the weight assigned to xᵢ. The denominator must not be omitted.

Median

The median is the middle ordered observation. For an odd number of observations, it is the central value. For an even number, it is usually the arithmetic mean of the two central values. The median is resistant to extreme observations and is particularly useful for skewed variables such as income, waiting time and house price.

Mode

The mode is the most frequently occurring value or category. A dataset may have one mode, several modes or no repeated value. The mode is the only central-tendency measure directly suitable for nominal categories.

Geometric Mean

For positive values, the geometric mean is:

It is useful for multiplicative growth rates and ratios, not ordinary additive measurements. It requires positive observations and should not be used casually when zeros or negative values occur.

Worked Example 2: Mean and Median

Consider response times, in minutes: 3, 4, 4, 5, 6, 8, 30.

The mean is:

The median is the fourth ordered value, 5 minutes. The mean is much larger because the 30-minute observation is influential. Reporting only the mean would give a misleading impression of a typical response time; reporting the median together with a spread measure gives a fuller description.

4. Measures of Dispersion

Two datasets can have the same mean but very different variability. Dispersion measures quantify how widely observations are spread.

Range and Interquartile Range

The range is:

It is simple but depends entirely on two observations. The first quartile Q₁ is the 25th percentile and the third quartile Q₃ is the 75th percentile. The interquartile range is:

The IQR contains the middle 50% of observations and is resistant to outliers. Different software may use slightly different interpolation rules for percentiles, so the method should be documented when precision matters.

Variance and Standard Deviation

For a population, variance is:

For a sample used to estimate population variability, the usual unbiased sample variance is:

The corresponding standard deviation is:

Here μ and σ² describe a population, N is the population size, x̄ and s² describe a sample, and n is the sample size. Squared deviations have squared units, whereas standard deviation has the same units as the original variable. The n − 1 denominator, often called Bessel’s correction, is appropriate when the sample mean has been estimated from the same sample and the goal is to estimate population variance. If the dataset is the complete population of interest, division by N is appropriate.

Worked Example 3: Sample Variance

For the sample 2, 4, 6, the mean is x̄ = 4. Deviations are −2, 0 and 2; squared deviations are 4, 0 and 4. Thus:

Therefore s = √4 = 2 units. The population variance for these three values, if they constitute the whole population, would instead be 8 ÷ 3 ≈ 2.67 units².

Coefficient of Variation

For a positive variable with a meaningful zero, the coefficient of variation is:

It expresses standard deviation relative to the mean and can help compare relative variability across variables with different units. It is not appropriate when the mean is zero or near zero, or when zero does not represent an absence of the quantity.

5. Measures of Position

Position measures locate an observation within an ordered distribution. Percentiles divide data into 100 parts; quartiles divide them into four parts; deciles divide them into ten parts. A percentile is not necessarily the percentage of observations below a value under every calculation convention, especially in small samples, so the percentile definition or software method should be stated.

A z-score standardises a value relative to a mean and standard deviation:

When sample estimates are used, x̄ and s replace μ and σ. The z-score is dimensionless. A value of z = 2 means the observation is two standard deviations above the reference mean, assuming the selected mean and standard deviation are meaningful. A z-score is not automatically an outlier rule; unusualness depends on distributional shape and context.

The five-number summary consists of the minimum, Q₁, median, Q₃ and maximum. It is the basis of a box plot. A common exploratory rule flags values below Q₁ − 1.5 × IQR or above Q₃ + 1.5 × IQR as potential outliers. This is a screening convention, not proof that a value is erroneous. Legitimate rare events can be operationally important.

6. Shape, Skewness and Outliers

A symmetric distribution has comparable tails on both sides of its centre. In a positively skewed distribution, the right tail is longer and unusually large values often pull the mean above the median. In a negatively skewed distribution, the left tail is longer and the mean may fall below the median. The relationship between mean and median is a useful clue, not a universal theorem.

Skewness can be described graphically with histograms, density plots or box plots. A sample moment coefficient is sometimes written as:

Definitions and bias corrections differ across software, so numerical skewness values should not be compared without checking the convention. For diploma-level analysis, visual inspection and the mean–median relationship are often more interpretable than a single coefficient.

Outliers may arise from data-entry errors, sensor faults, changes in measurement systems, fraud, unusual but valid cases or genuine rare events. A responsible procedure is to verify the record, inspect the source and context, document any correction, and perform sensitivity analysis with and without the observation. Deleting an inconvenient value merely to improve a summary is poor practice.

7. Covariance and Correlation

Covariance describes the joint direction of two numerical variables. The sample covariance is:

Positive covariance means that above-average values of one variable tend to occur with above-average values of the other. Negative covariance indicates an opposite tendency. Its units are the product of the two variables, so its magnitude is difficult to compare across datasets.

Pearson’s sample correlation standardises covariance:

where sₓ and sᵧ are sample standard deviations. Equivalently, when both standard deviations are non-zero:

The value of r lies between −1 and +1. Its sign indicates linear direction and its magnitude indicates linear association. Correlation is unit-free, but it is not resistant to outliers and does not detect every nonlinear relationship. A curved relationship may have r close to zero. Correlation also does not establish causation: a third variable, selection bias or reverse causality may explain an association.

For ordinal variables or heavily skewed data, Spearman’s rank correlation may be more suitable because it operates on ranks. It still measures association, not causation.

8. Selecting Appropriate Summaries

Summary choice should follow the variable type, distribution, measurement scale and decision purpose.

Percentages must include denominators. “A 40% increase” is incomplete without the baseline: an increase from 1 to 1.4 and from 10,000 to 14,000 have the same percentage change but very different practical implications.

9. Misleading Aggregates and Aggregation Bias

An aggregate combines observations into one summary. Aggregation is useful but can conceal subgroup differences. Simpson’s paradox occurs when an association observed within several groups reverses or disappears after the groups are combined. Possible causes include unequal group sizes, confounding variables and different exposure levels.

For example, an overall model accuracy of 92% may hide 97% accuracy for one demographic group and 72% for another. If group sizes are unequal, the larger group dominates the overall figure. An analyst should therefore report relevant subgroup counts, rates and uncertainty where appropriate. In operational AI, fairness assessment requires disaggregated performance measures rather than one global average.

Other common problems include averaging percentages with unequal denominators, comparing means from groups with different compositions, ignoring missing data, mixing time periods and using a mean for a heavily skewed variable. Weighted averages can be appropriate, but weights must represent a justified target quantity rather than being chosen to obtain a preferred result.

10. A Practical Descriptive-Analysis Workflow

A reliable workflow is:

1. Define the unit of analysis. Decide whether each row represents a person, transaction, day or event.
2. Inspect data types and scales. Identify categorical, ordinal, discrete and continuous variables.
3. Check data quality. Look for missing values, duplicates, impossible values, inconsistent units and coding errors.
4. Summarise categories. Use counts and percentages with explicit denominators.
5. Visualise numerical distributions. Use histograms, box plots, dot plots or empirical percentiles.
6. Calculate robust and conventional summaries. Compare mean with median and standard deviation with IQR.
7. Investigate relationships. Use scatter plots and correlation only after checking scale, shape and outliers.
8. Disaggregate important results. Compare relevant groups, time periods and operational conditions.
9. Document decisions. Record definitions, missing-data treatment, percentile convention and outlier handling.
10. Communicate limitations. State what the summaries show and what they cannot establish.

Software such as Python, R, spreadsheets and database query tools can calculate summaries efficiently. Automation does not remove the need to inspect definitions and plots. A technically correct calculation can still answer the wrong question.

Applied Case: Support-Ticket Response Analysis

Background

This illustrative case concerns a help-desk team analysing first-response times for 20 randomly selected tickets during one week. The team wants to set a realistic service target and identify whether the average is representative.

Data and Analysis

Response times in minutes are:

3, 4, 4, 5, 5, 6, 6, 7, 8, 8, 9, 10, 10, 11, 12, 14, 15, 18, 25, 60.

The total is 240 minutes, so the mean is 12 minutes. The ordered middle observations are the 10th and 11th values, 8 and 9, giving a median of 8.5 minutes. The 60-minute ticket pulls the mean upward. The range is 57 minutes, and the upper tail is much longer than the lower tail, indicating positive skewness.

A box-plot rule would require Q₁ and Q₃ to identify potential outliers precisely. Regardless of the rule, 60 minutes should be investigated rather than automatically deleted. It may represent a valid escalation, an incident or a recording error.

Approach and Recommendation

The team should report the median response time of 8.5 minutes, the mean of 12 minutes, an appropriate upper percentile and the number of tickets. It should separately examine ticket priority, because high-priority incidents may legitimately take longer. A service promise based only on the mean could misrepresent the usual customer experience, while a promise based only on the median could ignore important delays. The team should also report the proportion meeting a defined threshold, such as 10 minutes, with the denominator clearly stated.

Learning Points

The case demonstrates that central tendency and dispersion answer different questions. The mean is useful for total workload calculations, while the median better describes a typical ticket in this skewed sample. Contextual investigation is essential before classifying an extreme value as an error. Grouping by priority may reveal that an apparently poor overall result is caused by a particular operational category.

Challenges, Ethics and Governance

Descriptive statistics can affect resource allocation, employee evaluation, customer treatment and model deployment. Analysts should protect personal information, minimise access to sensitive fields and avoid publishing small-cell results that enable re-identification. Data provenance and reproducibility require records of source systems, transformations and calculation rules.

Fairness requires checking whether summaries differ systematically across relevant groups. Privacy-preserving aggregation may require suppressing or combining very small groups, but suppression should not be used to conceal poor outcomes. Analysts should distinguish missingness from zero: a missing income is not an income of zero, and an unrecorded event is not evidence that the event did not occur.

Contemporary analytics increasingly uses automated dashboards and streaming data. Real-time summaries can be valuable, but provisional data, changing denominators and delayed records can make current values unstable. Dashboards should display timestamps, definitions, sample sizes and warnings when data quality is incomplete.

Best Practices

• Use both numerical and graphical summaries.
• Match the summary to the measurement scale and distribution.
• Report denominators, units and time periods.
• Use median and IQR when skewness or outliers make mean-based summaries misleading.
• Investigate unusual observations before deleting or winsorising them.
• Do not interpret correlation as causation.
• Compare relevant subgroups rather than relying on a single aggregate.
• Document formulas, software conventions and missing-data decisions.
• Preserve the distinction between descriptive evidence and inferential claims.

Chapter Summary

Descriptive statistics organises data into frequency distributions and summarises its centre, spread, position and shape. Counts and percentages are appropriate for categorical variables, while numerical variables may require means, medians, standard deviations, IQRs and percentiles. The mean is sensitive to extreme values; the median and IQR are more robust for skewed data. Variance uses squared units, whereas standard deviation returns to the original units. Covariance indicates joint direction but depends on measurement units; correlation standardises linear association and ranges from −1 to +1. Neither establishes causation. Sound analysis checks outliers, missingness, measurement scales, subgroup differences and denominators. In AI and business applications, descriptive summaries support responsible data preparation and decision-making, but they must be interpreted in context and documented transparently.

Key Terms

Variable: A characteristic recorded for each observation.

Observation: A unit or case represented in a dataset.

Frequency: The number of occurrences of a value or category.

Relative frequency: A frequency divided by the total number of observations.

Mean: The arithmetic average of numerical observations.

Median: The middle value after observations are ordered.

Mode: The most frequently occurring value or category.

Range: The maximum value minus the minimum value.

Quartile: A value marking a specified quarter of an ordered distribution.

Interquartile range: The difference Q₃ − Q₁, containing the middle 50% of observations.

Variance: The average squared deviation from a mean, with a population or sample definition.

Standard deviation: The square root of variance, expressed in the original units.

Percentile: A positional value used to divide an ordered distribution into 100 parts.

Z-score: A standardised distance from a reference mean measured in standard deviations.

Skewness: Asymmetry in the shape of a distribution.

Outlier: An observation unusually distant from the main pattern of data.

Covariance: A measure of the joint directional variation of two numerical variables.

Correlation: A unit-free measure of linear association between two numerical variables.

Aggregation bias: Distortion caused when combined summaries conceal subgroup patterns.

Simpson’s paradox: A reversal or disappearance of an association after data are aggregated across groups.

Review Questions

Short Answer Questions

1. What is the difference between frequency and relative frequency?
2. Why is the median often preferred to the mean for highly skewed data?
3. Distinguish population variance from sample variance.
4. What does a negative correlation indicate?
5. Why should percentages be reported with denominators?

Long Answer Questions

1. Explain how measurement scale and distribution shape influence the choice of descriptive summaries.
2. Describe the calculation and interpretation of the mean, median, mode, range, IQR and standard deviation.
3. Explain how outliers can affect the mean, standard deviation and Pearson correlation.
4. Compare covariance and correlation, including their units, ranges and limitations.
5. Discuss how aggregation can produce misleading conclusions, using Simpson’s paradox as a framework.

Analytical / Application Questions

1. A dataset of delivery times has a mean of 42 minutes, a median of 25 minutes and several very large values. Which summaries and visualisations should be reported, and why?
2. A classification system has 95% overall accuracy, but accuracy is 98% for Group A and 70% for Group B. Explain why the overall value may be misleading and propose a better reporting approach.
3. Two variables have a Pearson correlation close to zero, but a scatter plot forms a clear U-shape. What does this reveal about the limitations of correlation, and what additional analysis would you recommend?

Multiple Choice Questions with Answers

Question 1

Which summary is most appropriate for a nominal categorical variable?

A. Arithmetic mean

B. Standard deviation

C. Frequency and percentage

D. Interquartile range

Correct Answer: C. Frequency and percentage

Question 2

For the values 2, 4, 6, 8, and 10, what is the arithmetic mean?

A. 5

B. 6

C. 7

D. 8

Correct Answer: B. 6

Question 3

Which measure is generally most resistant to an extreme outlier?

A. Mean

B. Range

C. Median

D. Variance

Correct Answer: C. Median

Question 4

If Q₁ = 12 and Q₃ = 20, what is the interquartile range?

A. 8

B. 16

C. 32

D. 0.6

Correct Answer: A. 8

Question 5

What is the main reason sample variance commonly uses n − 1 rather than n in its denominator?

A. To convert units into percentages

B. To correct for estimating the population mean from the sample

C. To remove all outliers

D. To ensure the variance is always zero

Correct Answer: B. To correct for estimating the population mean from the sample

Question 6

A positive covariance between two variables means that they tend to:

A. Have identical units

B. Move in opposite directions

C. Be unrelated in every possible way

D. Be above or below their means together

Correct Answer: D. Be above or below their means together

Question 7

Which statement about Pearson correlation is correct?

A. It proves that one variable causes another

B. It measures linear association and is bounded between −1 and +1

C. It is unaffected by extreme observations

D. It can only take positive values

Correct Answer: B. It measures linear association and is bounded between −1 and +1

Question 8

A distribution with a long right tail is commonly described as:

A. Positively skewed

B. Negatively skewed

C. Uniformly coded

D. Perfectly symmetric

Correct Answer: A. Positively skewed

Question 9

For the sample 2, 4, 6, the sample variance is:

A. 2.00

B. 2.67

C. 4.00

D. 8.00

Correct Answer: C. 4.00

Question 10

Why can an overall model accuracy be misleading?

A. Accuracy never uses numerical data

B. It may conceal substantially different results for subgroups

C. It is always lower than every subgroup accuracy

D. It cannot be expressed as a percentage

Correct Answer: B. It may conceal substantially different results for subgroups