Exploratory Data Analysis

Introduction

Exploratory Data Analysis (EDA) is the systematic investigation of a dataset before formal statistical modelling or machine learning. It combines numerical summaries, visual displays, data-quality checks, segmentation, and domain reasoning to discover structure in data. EDA does not prove causal relationships; rather, it helps analysts understand what the data contain, identify unusual observations, assess assumptions, and formulate questions that can later be tested.

EDA is especially important in data science because real-world datasets are rarely clean, symmetric, complete, or measured under ideal conditions. A model trained on incorrectly typed variables, duplicated records, unrecognised outliers, or biased samples may produce apparently precise but misleading results. EDA therefore serves as a bridge between data preparation and inferential or predictive analysis.

A useful EDA process is iterative:

1. clarify the analytical question and unit of analysis;
2. inspect structure, types, ranges, and missingness;
3. summarise individual variables;
4. examine distributions and relationships;
5. segment observations into meaningful groups;
6. investigate anomalies and assumptions;
7. formulate and document testable questions.

The process should be guided by both statistical reasoning and subject-matter knowledge. A value is not anomalous merely because it is unusual: it may represent a genuine event, a measurement error, or a different population.

Learning Objectives

By the end of this chapter, you should be able to:

• distinguish univariate, bivariate, and multivariate EDA;
• calculate and interpret common summary measures;
• select suitable graphical displays for different variable types;
• describe distributions using centre, spread, shape, and unusual values;
• examine relationships without confusing association with causation;
• use segmentation to reveal heterogeneity;
• identify data-quality problems and assess modelling assumptions;
• communicate EDA findings responsibly and convert them into testable questions.

1. The Analytical View of a Dataset

A dataset consists of observations (rows) and variables (columns). The unit of analysis may be a customer, transaction, patient, product, web session, or day. Defining this unit is essential. If one row represents a transaction but the analyst interprets it as a customer, counts, averages, and relationships may be distorted.

Variables are commonly classified as follows:

Measurement scale affects valid operations. It is meaningful to calculate an average response time, but not an average browser type. An ordinal rating can be ordered, although treating the difference between “satisfied” and “very satisfied” as exactly equal to the difference between lower categories may require justification.

A data dictionary should record variable names, definitions, units, permissible values, coding schemes, and sources. Without this documentation, EDA can produce technically correct calculations with incorrect interpretations.

2. Data Quality as the First Stage of EDA

Before searching for patterns, inspect the data-generating and collection process. Important checks include:

• Completeness: Which variables or records are missing?
• Validity: Are values within permitted ranges?
• Consistency: Do related fields agree, such as an end date following a start date?
• Uniqueness: Are records duplicated, and should they be?
• Accuracy: Do values plausibly represent the phenomenon?
• Timeliness: Are dates and refresh frequencies appropriate?

Missingness should be described, not silently deleted. For variable X, the missing proportion is:

where m is the number of missing observations and n is the total number of observations. This proportion is dimensionless. It should be calculated for relevant subgroups as well as for the whole dataset, because missingness may be systematic.

Three useful conceptual mechanisms are:

• MCAR (Missing Completely At Random): missingness is unrelated to observed and unobserved values;
• MAR (Missing At Random): missingness depends on observed variables;
• MNAR (Missing Not At Random): missingness depends on the unobserved value itself or an unmeasured cause.

These mechanisms cannot generally be established from a table of missing values alone. For example, high missingness among users on older devices may indicate a data-capture problem rather than random absence.

Do not automatically convert special codes such as −99, 999, or “unknown” into ordinary numerical values. They may represent missingness, refusal, or a valid category and must be interpreted from metadata.

3. Summary Measures

Summary measures compress a dataset, but every summary omits information. They should therefore be used together with plots and contextual knowledge.

3.1 Measures of centre

For numerical observations x₁, x₂, …, xₙ, the arithmetic mean is:

Here x̄ is the sample mean, n is the number of non-missing observations, and xᵢ is observation i. All xᵢ must be measured on a numerical scale for which averaging is meaningful. The mean is sensitive to extreme values.

The median is the middle observation after sorting. For an odd n it is the central value; for an even n it is commonly the arithmetic mean of the two central values. The median is robust to extreme observations and is often preferable for skewed variables such as income or transaction value.

The mode is the most frequent value or category. It is particularly useful for categorical variables, although a dataset may have more than one mode or no unique mode.

3.2 Measures of spread

The range is:

It uses only two observations and is highly sensitive to extremes. The interquartile range (IQR) is:

where Q₁ is the first quartile (25th percentile) and Q₃ is the third quartile (75th percentile). The IQR describes the spread of the middle half of the data. Percentile definitions can vary slightly between software systems, so the chosen convention should be documented.

The sample variance and sample standard deviation are:

The denominator n − 1 is used when estimating population variability from a sample and assumes observations are appropriately sampled for that purpose. Standard deviation has the same units as x; variance has squared units. Standard deviation is sensitive to outliers and is most naturally interpreted when the distribution is not extremely skewed.

The coefficient of variation is:

It is dimensionless and can compare relative variability across variables with different scales, but it is unsuitable when the mean is zero, close to zero, or not meaningful.

3.3 Worked example

Suppose five delivery times, in minutes, are 18, 20, 21, 22, and 39. The mean is:

The median is 21 minutes. The mean is higher because the 39-minute delivery pulls it upward. Using the sample formula, the deviations from the mean are −6, −4, −3, −2, and 15. Their squared deviations sum to 36 + 16 + 9 + 4 + 225 = 290, so:

and:

The difference between mean and median suggests right-skew or an unusual long delivery. A dot plot or box plot should be used before deciding whether 39 minutes is an error or a genuine service event.

4. Distributions and Visual Exploration

A distribution describes how values are arranged across possible values. Important features are centre, spread, shape, modality, gaps, and tails.

4.1 Graphical displays

• Bar chart: compares frequencies or proportions of categories. Bars are separated because categories are distinct.
• Histogram: displays numerical values in intervals (bins). Adjacent bars represent continuity; bin width can materially affect the apparent shape.
• Density plot: a smoothed representation of a numerical distribution. Smoothing can hide small groups, so it should not replace raw counts.
• Box plot: shows Q₁, median, Q₃, and often whiskers. A common convention labels values below Q₁ − 1.5IQR or above Q₃ + 1.5IQR as potential outliers. This is a screening rule, not proof of an error.
• Dot plot or strip plot: displays individual observations and is effective for small datasets.
• Time-series plot: places observations in temporal order and can reveal trend, seasonality, cycles, and sudden changes.

A histogram should be interpreted with sensitivity analysis: changing the bin width can reveal or conceal structure. For skewed data, a logarithmic transformation may improve visual readability, but transformed values must be clearly labelled and interpreted appropriately.

4.2 Shape and transformations

A symmetric distribution has broadly balanced tails. In a right-skewed distribution, a long upper tail often causes the mean to exceed the median; in a left-skewed distribution, the reverse may occur. A bimodal distribution may indicate two underlying populations, different operating conditions, or a mixture of processes.

Transformations can support exploration. For positive values, a logarithmic transformation such as z = log(x) compresses large values and can make multiplicative differences easier to inspect. It requires x > 0 and changes the scale of interpretation. A transformation should not be used merely to make an inconvenient pattern disappear.

5. Relationships Among Variables

Bivariate and multivariate EDA examine whether variables move together and whether patterns differ across groups.

5.1 Numerical–numerical relationships

A scatter plot is usually the first display for two numerical variables. It can reveal linearity, curvature, clusters, changing spread, and influential observations.

The Pearson correlation coefficient is:

where x̄ and ȳ are sample means. Provided both variables have non-zero variation, r lies between −1 and +1. It measures the strength and direction of linear association, not general dependence and not causation. It can be strongly affected by outliers and nonlinear patterns.

A correlation near zero does not imply no relationship: a U-shaped relationship may have little linear correlation. Spearman’s rank correlation can be useful for monotonic relationships and ordinal data, but it also does not establish causation.

5.2 Categorical–numerical relationships

Use grouped box plots, violin plots, strip plots, or summary tables to compare a numerical outcome across categories. Report group sizes, because an impressive difference based on a very small group may be unstable. Compare both centre and spread; two groups may have equal means but very different distributions.

5.3 Categorical–categorical relationships

Contingency tables show counts or conditional proportions. The direction of conditioning matters. “The proportion of churners using plan A” is different from “the proportion of plan-A users who churn.” A mosaic plot or heat map can reveal concentration, but raw counts should remain available.

5.4 Time and multivariate structure

Time ordering is analytically important. Randomly shuffling time-based observations before EDA can conceal drift or leakage. Examine values by day, week, month, or operational period, while accounting for unequal exposure and seasonality.

For more than two variables, use colour, shape, small multiples, faceting, or interactive filtering carefully. Too many encodings create visual overload. Dimensionality-reduction methods such as principal component analysis can support exploration, but their results depend on scaling, missing-value treatment, and linear assumptions. They are not substitutes for understanding the original variables.

6. Segmentation and Heterogeneity

Overall summaries can conceal important subgroup behaviour. Segmentation divides observations according to meaningful attributes such as region, customer tenure, device type, product category, or risk band.

A responsible segmentation process is:

1. define the business or research purpose;
2. select variables available before the outcome, avoiding leakage;
3. check segment size and representativeness;
4. compare distributions and outcomes within segments;
5. assess whether differences are practically meaningful;
6. validate segments on new or later data.

Segmentation may be rule-based, such as age bands, or algorithmic, such as clustering. Algorithmic clusters are patterns in selected variables, not naturally existing human categories. Choices of distance measure, scaling, number of clusters, and missing-value treatment affect the result.

A classic EDA warning is aggregation bias. An overall relationship may weaken, disappear, or reverse within subgroups. This phenomenon is related to Simpson’s paradox. Analysts should therefore inspect plausible confounders and report both aggregate and stratified results.

7. Outliers, Influence, and Assumptions

An outlier is an observation that is unusually distant from the rest under a specified definition. It may be:

• a data-entry or sensor error;
• a legitimate but rare event;
• an observation from another population;
• evidence that the assumed model is inadequate.

Never delete an outlier solely because it changes a result. First verify its source, unit, timestamp, and related fields. Compare analyses with and without the observation only when the procedure is documented and justified.

EDA also assesses assumptions relevant to later methods:

• Independence: observations are not improperly repeated or clustered;
• Linearity: a linear model is a reasonable approximation;
• Constant variance: residual spread is reasonably stable across fitted values;
• Distributional assumptions: normality may matter for some inferential procedures, but not all predictive methods;
• No severe multicollinearity: predictors are not redundant in a way that destabilises interpretation;
• Stationarity: relationships and distributions do not change unaccountably over time.

Diagnostic plots should be interpreted in context. A normal Q–Q plot can assess approximate normality, while residual-versus-fitted plots can reveal curvature or unequal spread. Assumption checking is not a ritual: each assumption should be connected to the method and the consequence of violation.

8. A Worked Applied Case: E-Commerce Delivery Performance

The following is an illustrative case. An online retailer wants to understand late deliveries and decide whether a predictive model is warranted. The dataset contains order ID, region, carrier, promised date, delivered date, order value, item count, and customer type.

Background and problem

Management reports a rising complaint rate but does not know whether delays are widespread or concentrated in particular routes. The analytical outcome is delivery delay in days, computed from delivered date minus promised date. The unit of analysis is one order, not one customer; repeat customers may therefore contribute multiple rows.

Initial analysis

The analyst first checks duplicate order IDs, invalid dates, negative item counts, missing carrier values, and whether cancelled orders are incorrectly included. Missing carrier values are more frequent in one fulfilment centre, suggesting a process problem rather than random missingness.

A histogram of delay shows a mass near zero and a long positive tail. The median delay is 0 days, while the mean is 0.8 days. This difference indicates that a minority of severely delayed orders increases the mean. A box plot flags several large delays, which are then checked against tracking records rather than removed automatically.

Relationships and segmentation

A bar chart of the proportion of late orders by carrier suggests that Carrier B has the highest late rate. However, Carrier B handles a larger share of remote-region orders. Grouped plots by both carrier and region show that the carrier difference is small within several regions, while remote routes are consistently slower. This demonstrates why an aggregate comparison can be misleading.

A scatter plot of order value and delay shows little overall association, but a table by item count suggests that very large orders have greater delay variability. A time-series plot reveals a sharp increase during a holiday week. Treating that week as ordinary would distort routine performance estimates.

Approach and outcome

The team reports median delay, mean delay, IQR, late-order proportion, segment sizes, and confidence intervals in a later inferential stage. It recommends improving carrier metadata at the fulfilment centre, analysing remote routes separately, and adding holiday indicators to any predictive model. It does not claim that carrier choice causes delay because route assignment is not random and operational factors are confounded.

Learning points

1. The unit of analysis determines the meaning of summaries.
2. Median and mean can tell different but complementary stories.
3. Outliers require investigation, not automatic deletion.
4. Segmenting by region exposed structure hidden in aggregate results.
5. Time and operational context are essential to interpretation.
6. EDA produces testable questions, such as whether route type predicts delay after controlling for season and order size.

9. Communication, Ethics, and Reproducibility

An EDA report should state the data source, extraction period, unit of analysis, exclusions, missing-value rules, transformations, software versions, and plotting conventions. Every figure needs a descriptive title, labelled axes, units, and an explanation of the relevant pattern. Avoid truncated axes when they exaggerate differences, and do not use colour schemes that obscure accessibility.

Privacy must be considered even during exploration. Remove direct identifiers when they are unnecessary, restrict access to sensitive attributes, and avoid publishing small-cell results that could re-identify individuals. Sensitive variables such as disability, ethnicity, or location may reveal inequities in data collection or service outcomes. Analysts should inspect subgroup performance, document limitations, and avoid treating historical disparities as neutral facts.

AI-assisted tools can accelerate chart generation, code completion, and anomaly screening, but they may misinterpret variable meanings, invent explanations, or reproduce bias. Human review, reproducible code, data lineage, and validation against source records remain necessary. Automated anomaly detection should be treated as a prioritisation mechanism, not an adjudicator.

10. Best-Practice Workflow

A defensible EDA workflow includes the following controls:

• preserve the raw dataset and create a versioned analysis copy;
• define the question, population, time window, and unit of analysis;
• inspect metadata before calculating statistics;
• use both robust and non-robust summaries where appropriate;
• plot individual observations when sample size permits;
• compare aggregate patterns with meaningful subgroup patterns;
• investigate missingness, duplicates, outliers, and data leakage;
• distinguish exploratory findings from confirmatory claims;
• record decisions and alternative analyses;
• formulate hypotheses for later testing on appropriate data.

EDA is successful when it improves the quality of subsequent reasoning, not when it produces the largest number of charts. A visually attractive graph with an undefined population or misleading denominator is less useful than a modest, carefully documented analysis.

Chapter Summary

Exploratory Data Analysis is an iterative process for understanding data before formal modelling. It begins with the unit of analysis, metadata, and data-quality checks. Measures of centre and spread summarise numerical variables, but their interpretation depends on skewness, outliers, scale, and measurement type. Histograms, box plots, scatter plots, contingency tables, and time-series displays reveal distributional and relational structure that summaries alone can hide.

EDA examines association without claiming causation, uses segmentation to identify heterogeneity, and treats outliers as observations requiring investigation. It also evaluates assumptions such as independence, linearity, constant variance, and temporal stability. Ethical practice requires privacy protection, transparent documentation, careful subgroup analysis, and human oversight of AI-assisted tools. The ultimate product of EDA is a defensible understanding of the dataset and a set of precise, testable questions.

Key Terms

Exploratory Data Analysis: Systematic investigation of data using summaries, visualisations, and contextual reasoning.

Unit of analysis: The entity represented by one observation or row.

Data dictionary: Documentation describing variables, units, codes, and permissible values.

Mean: Arithmetic average of numerical observations.

Median: Middle value after observations are ordered.

Mode: Most frequently occurring value or category.

Variance: Average squared deviation used to quantify sample variability.

Standard deviation: Square root of variance, expressed in the original units.

Quartile: A percentile dividing ordered data into four parts.

Interquartile range: Difference between the third and first quartiles.

Skewness: Asymmetry in a distribution’s shape.

Outlier: An unusually distant observation under a specified criterion.

Correlation: A measure of association, commonly linear association for Pearson’s r.

Segmentation: Division of observations into meaningful subgroups.

Confounding: Distortion of an apparent relationship by another variable.

Missingness: The absence of a recorded value.

Data leakage: Use of information unavailable at the time a prediction would be made.

Multicollinearity: Strong dependence among explanatory variables.

Reproducibility: Ability to obtain an analysis again using documented data and procedures.

Review Questions

Short Answer Questions

1. Why must the unit of analysis be defined before EDA?
2. State one advantage of the median over the mean.
3. What information does the IQR provide?
4. Why does correlation not establish causation?
5. Name two possible explanations for an outlier.

Long Answer Questions

1. Explain how summary measures and visualisations complement one another in EDA.
2. Describe a complete procedure for investigating missing values in a dataset.
3. Compare histograms, box plots, and time-series plots, including appropriate uses and limitations.
4. Explain how segmentation can reveal patterns hidden by aggregate statistics.
5. Discuss the ethical and reproducibility requirements of an EDA report.

Analytical / Application Questions

1. A dataset has a mean income far above its median. Propose an EDA plan to determine whether the difference reflects skewness, data errors, or population mixing.
2. A carrier appears to have the highest late-delivery rate overall, but it serves mostly remote regions. Design a segmented analysis and explain how you would avoid an unjustified causal conclusion.
3. A predictive model assumes a linear relationship and constant residual variance. Identify plots and numerical checks you would use during EDA, and explain what actions would follow from violations.

Multiple Choice Questions with Answers

Question 1

Which measure is generally least affected by a single extremely large observation?

A. Mean

B. Median

C. Range

D. Variance

Correct Answer: B. Median

Question 2

For five delivery times of 18, 20, 21, 22, and 39 minutes, what is the arithmetic mean?

A. 21 minutes

B. 22 minutes

C. 24 minutes

D. 39 minutes

Correct Answer: C. 24 minutes

Question 3

What does the interquartile range measure?

A. The total distance between minimum and maximum

B. The spread of the middle 50% of observations

C. The average squared deviation from the mean

D. The most frequent category

Correct Answer: B. The spread of the middle 50% of observations

Question 4

Which graph is most directly suited to examining the relationship between two numerical variables?

A. Scatter plot

B. Bar chart

C. Pie chart

D. Contingency table

Correct Answer: A. Scatter plot

Question 5

A Pearson correlation close to zero can occur when two variables have which type of relationship?

A. A perfect positive linear relationship

B. A perfect negative linear relationship

C. A nonlinear relationship such as a U-shape

D. Identical numerical values for every observation

Correct Answer: C. A nonlinear relationship such as a U-shape

Question 6

Which statement about a box-plot outlier rule using 1.5IQR is correct?

A. It proves that the observation is erroneous

B. It identifies observations for possible investigation

C. It applies only to categorical variables

D. It guarantees a normal distribution

Correct Answer: B. It identifies observations for possible investigation

Question 7

Why should missingness be examined by subgroup as well as overall?

A. Overall missingness is always zero

B. Subgroup patterns can indicate systematic data-collection problems

C. Subgroup analysis removes the need for metadata

D. Missing values automatically become outliers

Correct Answer: B. Subgroup patterns can indicate systematic data-collection problems

Question 8

What is the principal risk of comparing carriers only with an aggregate late-delivery rate when carriers serve different regions?

A. The chart will contain too many colours

B. The comparison may be confounded by region

C. The median cannot be calculated

D. The data become nominal

Correct Answer: B. The comparison may be confounded by region

Question 9

Which condition is required for the transformation z = log(x) as described in the chapter?

A. x must be negative

B. x must equal zero

C. x must be positive

D. x must be categorical

Correct Answer: C. x must be positive

Question 10

Which practice best supports reproducible EDA?

A. Delete all unusual observations without recording them

B. Report only the most attractive chart

C. Document data sources, exclusions, transformations, and software procedures

D. Replace metadata with analyst assumptions

Correct Answer: C. Document data sources, exclusions, transformations, and software procedures