Exploratory Data Analysis

Introduction and Context

Exploratory Data Analysis (EDA) is the systematic examination of data before formal modelling, hypothesis testing, or operational decision-making. It combines numerical summaries, visualizations, grouping, and anomaly investigation to discover structure, assess data quality, identify plausible relationships, and formulate useful questions.

EDA is not simply “making charts.” A good analysis connects each analytical action to a decision. For example, a retailer may ask why weekly sales declined; a hospital may investigate whether waiting times differ by department; and a bank may examine whether unusual transactions indicate fraud. In each case, EDA helps transform a vague concern into measurable, decision-relevant questions.

EDA is usually iterative. An analyst may begin with a distribution, discover extreme values, inspect records associated with them, segment the data by region or customer type, and then revise the original question. The process is exploratory rather than confirmatory: patterns generated through EDA should generally be tested with appropriate statistical methods or validated on new data before strong causal claims are made.

Learning Objectives

By the end of this chapter, you should be able to:

1. Distinguish between univariate, bivariate, and multivariate exploration.
2. Select and interpret appropriate distribution plots and summary statistics.
3. Construct and interpret cross-tabulations and grouped summaries.
4. Measure and explain correlation without confusing it with causation.
5. Segment data to compare meaningful populations.
6. Detect, investigate, and appropriately handle anomalies.
7. Use EDA findings to formulate decision-relevant questions.
8. Communicate exploratory findings responsibly, including uncertainty and limitations.

1. The EDA Framework

A practical EDA workflow has six connected stages:

1. Understand the question and data-generating process. Identify the unit of observation, time period, population, variables, measurement units, and collection method.
2. Audit data quality. Examine missing values, duplicates, invalid categories, impossible values, and inconsistent formats.
3. Explore individual variables. Study distributions, frequencies, central tendency, spread, and unusual observations.
4. Explore relationships. Compare variables using cross-tabulations, grouped summaries, plots, and correlations.
5. Investigate segments and anomalies. Determine whether patterns differ across relevant groups and whether unusual records are errors, rare legitimate cases, or important signals.
6. Generate and prioritize questions. Convert observations into testable questions linked to decisions.

The unit of observation is especially important. A dataset may contain one row per customer, transaction, visit, product, or customer-month. If transactions are treated as customers, frequent customers may be overrepresented and conclusions may be misleading.

EDA should also distinguish descriptive, diagnostic, and predictive purposes. Descriptive EDA asks what happened. Diagnostic EDA asks where and under what conditions it happened. Predictive exploration asks which variables may help anticipate future outcomes. None of these automatically establishes causation.

2. Variable Types and Analytical Choices

Variable type determines which summaries and visualizations are appropriate.

A categorical code such as 1 = basic, 2 = standard, 3 = premium is not automatically a quantitative measurement. Arithmetic operations on it may be inappropriate unless the coding represents a meaningful scale and the analysis is designed accordingly.

3. Univariate Exploration: Understanding Distributions

A distribution describes how observations are allocated across possible values. Its key features include:

• Centre: a typical or central value.
• Spread: the degree of variation.
• Shape: symmetry, skewness, modality, and tails.
• Gaps: ranges containing few or no observations.
• Outliers: observations unusually distant from the main body.

3.1 Frequency and relative frequency

For category i, the relative frequency is:

Here, n_i is the number of observations in category i and n is the total number of non-missing observations. The value is dimensionless and ranges from 0 to 1. Percentages are obtained by multiplying by 100. Frequencies should be based on a clearly defined denominator; percentages can be misleading when group sizes differ.

3.2 Measures of centre

The arithmetic mean for n observations is:

The mean uses every observation and has the same units as the variable. It is sensitive to extreme values and is most informative when the distribution is reasonably balanced.

The median is the middle ordered value, or the average of the two middle values when n is even. It is resistant to extreme observations and is often preferable for income, transaction value, waiting time, and other right-skewed variables.

The mode is the most frequent value or category. It is particularly useful for categorical data.

3.3 Measures of spread

The range is:

It is easy to understand but depends only on two observations. The interquartile range is:

where Q₁ is the 25th percentile and Q₃ is the 75th percentile. The IQR describes the spread of the middle 50% of observations and is relatively resistant to outliers.

For a population, variance is:

For a sample used to estimate a population variance, the commonly used unbiased estimator is:

Here, μ and σ² are the population mean and variance; x̄ and s² are sample estimates. The denominator n − 1 reflects degrees-of-freedom adjustment. Variance is expressed in squared units, while standard deviation is:

Standard deviation has the same units as the original variable. These formulas assume numeric observations and a meaningful scale. They are descriptive summaries, not automatic evidence that a distribution is normal.

3.4 Distribution visualizations

A histogram divides a quantitative scale into intervals called bins and displays the count or proportion in each bin. Conclusions can change with bin width, so analysts should avoid interpreting a single arbitrary binning as definitive.

A box plot displays the median, quartiles, and selected extreme values. A common convention marks values below Q₁ − 1.5IQR or above Q₃ + 1.5IQR as potential outliers. This is a screening rule, not a universal definition of an error.

A density plot provides a smoothed estimate of distribution shape. Smoothing choices influence the appearance and should be interpreted cautiously. For categorical variables, bar charts are usually more appropriate than pie charts because category lengths are easier to compare than angles.

Worked Example 1: Choosing summaries

Suppose five delivery times, in minutes, are 18, 20, 21, 22, and 79. The mean is:

The median is 21 minutes. The mean is pulled upward by the 79-minute delivery, whereas the median better represents a typical delivery. The 79-minute observation should not be removed merely because it is inconvenient. The analyst should check whether it reflects a data-entry error, an exceptional event, or a genuine service failure.

4. Bivariate and Multivariate Exploration

Bivariate EDA examines two variables; multivariate EDA examines several variables simultaneously. The objective is to identify association, differences, interactions, and possible confounding factors.

4.1 Numerical versus numerical variables

A scatter plot displays paired observations. It can reveal positive or negative association, non-linear patterns, clusters, changing spread, and influential observations. A line of best fit may summarize a linear tendency, but it should not replace inspection of the points.

Covariance describes whether two variables tend to move together, but its magnitude depends on measurement units. Pearson’s correlation standardizes covariance:

Here, x and y are paired quantitative variables, and r lies between −1 and +1 when both variables have non-zero variation. Positive values indicate that larger x values tend to accompany larger y values; negative values indicate the opposite linear tendency. Correlation is unit-free.

Pearson correlation is most useful for a linear relationship and can be strongly affected by outliers. It does not prove causation, does not capture all non-linear relationships, and may be distorted by restricted ranges or mixtures of subgroups. Spearman’s rank correlation can be useful for monotonic relationships or ordinal data because it uses ranks, although it also measures association rather than causation.

4.2 Categorical versus numerical variables

Compare distributions rather than only group means. Useful displays include side-by-side box plots, violin plots, dot plots, and grouped summary tables. Report group sizes because a mean based on five observations should not receive the same confidence as one based on 5,000 observations.

4.3 Categorical versus categorical variables

A cross-tabulation, or contingency table, counts combinations of categories.

The email purchase rate is 80/200 = 40%, while the social-media purchase rate is 60/200 = 30%. The appropriate comparison is conditional on channel, not the raw count alone. Row percentages, column percentages, and overall percentages answer different questions and must be labelled clearly.

For an observed cell count O and an expected count E under independence, the chi-square statistic is:

Expected counts are calculated as:

The chi-square procedure assumes independent observations and sufficiently large expected counts for the chosen approximation. It tests whether categorical variables are statistically associated, not whether one causes the other. For small samples or sparse tables, an exact test or category consolidation may be more appropriate.

5. Segmentation and Comparative Analysis

Segmentation divides observations into meaningful groups for comparison. Possible bases include geography, customer tenure, product category, age band, service channel, or risk level. Segmentation is useful because an overall average can conceal important differences.

A responsible segmentation process follows these steps:

1. Define the decision that segmentation should support.
2. Select variables with a substantive rationale rather than trying every possible split.
3. Check segment size, completeness, and stability over time.
4. Compare distributions, rates, and uncertainty—not just averages.
5. Investigate whether observed differences remain after considering confounding variables.
6. Avoid labels that stigmatize people or imply unsupported explanations.

For example, an overall customer-retention rate of 82% might conceal rates of 95% for annual subscribers and 65% for monthly subscribers. The appropriate response is not automatically to target monthly subscribers with a campaign. Analysts should examine tenure, price, service usage, complaints, and acquisition channel before recommending action.

Simpson’s paradox and aggregation

An aggregate relationship can differ from relationships within subgroups. This may occur when groups have different sizes or when a third variable affects both variables being compared. Therefore, analysts should compare an overall table with relevant stratified tables and ask whether the segmentation variable is a confounder, mediator, or merely a descriptive grouping.

6. Anomaly Detection

An anomaly is an observation that differs substantially from an expected pattern. Anomalies may be:

• data errors, such as an impossible date;
• process problems, such as a delayed shipment;
• rare but legitimate events, such as a very large order;
• fraud or security incidents;
• changes in the data-generating process.

Anomaly detection is therefore an investigative activity, not a deletion procedure.

6.1 Rule-based checks

Domain rules are often the strongest first line of defence. Examples include quantity ≥ 0, end date after start date, valid currency codes, and age within a plausible operational range. Rules should distinguish impossible values from merely unusual values.

6.2 Standardized scores

A z-score is:

It expresses the distance from the sample mean in sample-standard-deviation units. It is most interpretable when the distribution is approximately symmetric and not heavily contaminated by extreme values. A threshold such as |z| > 3 is a screening convention, not a law and not proof of an error.

6.3 IQR fences

The common box-plot screening rule identifies lower and upper fences:

This method is useful for skewed data but may flag many legitimate observations in a long-tailed population. For time series, anomalies should also be assessed against seasonality, trend, and comparable time periods. A sales value that is unusual on an ordinary Tuesday may be normal during a holiday promotion.

6.4 Robust and multivariate methods

The median and IQR are more robust than the mean and standard deviation. For multiple variables, an observation may be ordinary on every individual variable but unusual in combination. Methods such as clustering, isolation forests, or distance-based approaches can assist, but they require careful scaling, parameter choices, validation, and domain review. Automated scores should support human investigation rather than make irreversible decisions without oversight.

7. From Patterns to Decision-Relevant Questions

A pattern becomes useful when it leads to a precise question. The following transformation is effective:

Observation → qualification → possible explanation → decision question → evidence required.

Example:

• Observation: customers using mobile checkout have a higher average order value.
• Qualification: the mobile group is smaller and contains more loyalty-program members.
• Possible explanations: device convenience, customer mix, or loyalty status.
• Decision question: does mobile checkout increase order value after accounting for customer type and product category?
• Evidence required: customer-level longitudinal data, an appropriate comparison design, and possibly an experiment.

Good questions specify a population, outcome, comparison, time frame, and possible action. Weak question: “Why are sales different?” Stronger question: “Among comparable first-time customers during the last quarter, is the conversion rate lower for mobile users on the payment page, and which error types explain the difference?”

EDA should record decisions about filters, transformations, exclusions, and visual scales. This creates an analytical audit trail and reduces the risk of unconsciously selecting only favourable findings.

8. Applied Case: Investigating Hospital Outpatient Waiting Times

This is an illustrative case based on a hypothetical outpatient clinic.

Background

A clinic’s monthly report shows that average waiting time increased from 34 to 46 minutes. Managers must decide whether to add staff, redesign appointment scheduling, or address a data-quality problem.

Problem definition

The analysis defines one observation as a completed outpatient visit. Variables include department, appointment type, scheduled time, arrival time, consultation start time, day, and recorded waiting time. The team first confirms that waiting time means consultation start minus patient arrival, rather than scheduled time minus arrival.

Analysis

The histogram is strongly right-skewed: most patients wait less than 40 minutes, but a small number wait several hours. The median is 29 minutes, while the mean is 46 minutes. This difference indicates that the mean is influenced by long waits.

A cross-tabulation by department shows the following illustrative results:

The specialist clinic has both a higher typical wait and a much longer upper tail. A time-of-day chart shows that delays accumulate after late-morning appointments. A further segmentation by appointment type reveals that urgent add-on visits are concentrated in the specialist clinic and are more likely to create schedule disruption.

Anomaly checks identify 18 records with negative waiting times. Investigation shows that these records resulted from clocks being synchronized at different times, not from unusually fast service. They are corrected using the central timestamp system and documented rather than silently deleted.

Approach and solution

The clinic does not immediately hire staff. It first tests three actions: a separate capacity block for urgent add-ons, automatic alerts when a clinic is projected to exceed capacity, and standardized timestamp capture. A dashboard reports median wait, 90th percentile wait, visit volume, and the proportion exceeding the service target by department.

Outcome

After implementation, the median changes little, but the 90th percentile falls substantially in the specialist clinic. This shows why monitoring only the average would have hidden the operational improvement. The clinic continues to review whether changes affect access for urgent patients and whether the new timestamp process remains reliable.

Learning points

1. The mean and median answer different operational questions.
2. Tail measures can be more relevant than averages for service-level management.
3. Segmentation identified where the problem was concentrated.
4. Anomalies required investigation and correction, not automatic removal.
5. EDA generated an intervention that could later be evaluated formally.

9. Tools, Reproducibility, and Communication

Spreadsheet software is suitable for small datasets and initial pivot tables. SQL is valuable for filtering, joining, grouping, and producing reproducible summaries from databases. Python or R supports programmable workflows, visualization, statistical analysis, and automated quality checks. Regardless of tool, the analytical logic should be reproducible: preserve source data, record transformations, version code, document assumptions, and separate exploratory outputs from production reports.

A professional EDA report should state:

• the data source, population, time period, and unit of observation;
• definitions of variables and calculated measures;
• missing-data treatment and exclusions;
• relevant subgroup sizes;
• the chart or statistic used and why;
• important uncertainty and limitations;
• the decision or next question supported by the evidence.

Charts should have informative titles, labelled axes, units, clear legends, and honest scales. Avoid truncated axes when they exaggerate differences, excessive colours, three-dimensional effects, and displays that hide the denominator. Accessibility requires sufficient contrast, readable text, and alternatives to colour-only distinctions.

10. Ethics, Governance, and Limitations

EDA can expose sensitive information even when the initial purpose appears harmless. Analysts should apply data minimization, access controls, de-identification where appropriate, retention rules, and applicable privacy law. Small segments may permit re-identification when combined with external information.

Fairness must be examined when segmentation or anomaly detection affects people. A model or rule trained on historical decisions may reproduce unequal treatment. Analysts should compare error rates across relevant groups, investigate proxy variables, and ensure that automated flags receive appropriate human review.

Common analytical limitations include missing-not-at-random data, measurement error, selection bias, duplicate records, changing definitions, multiple comparisons, and temporal confounding. Repeatedly searching for attractive patterns increases the risk of false discoveries. Exploratory findings should therefore be labelled as exploratory and, when important, confirmed using a pre-specified analysis, holdout data, an experiment, or independent replication.

11. Contemporary Developments

Modern analytics platforms increasingly provide automated profiling, natural-language chart generation, anomaly alerts, and AI-assisted explanations. These tools can accelerate routine exploration, but they do not understand the business meaning of every field or guarantee valid conclusions. AI-generated summaries should be checked against the underlying data, particularly for denominators, missingness, subgroup sizes, and causal wording.

Interactive dashboards enable users to filter and segment data, but interactivity can also encourage uncontrolled “dashboard wandering.” Good governance defines approved metrics, data refresh schedules, ownership, and safeguards against misleading comparisons. Responsible EDA combines computational speed with human domain knowledge and transparent documentation.

Best-Practice Checklist

Before presenting an exploratory finding, ask:

1. What is the unit of observation?
2. What population and period do the data represent?
3. Are the variables measured consistently and in appropriate units?
4. Have missing, duplicate, and impossible records been assessed?
5. Is the chosen statistic resistant to skew and outliers where necessary?
6. Are denominators and subgroup sizes visible?
7. Could aggregation or confounding explain the pattern?
8. Is the result association rather than causation?
9. Have anomalies been investigated before removal?
10. What decision-relevant question follows from the evidence?

Chapter Summary

Exploratory Data Analysis is an iterative approach for understanding data, assessing quality, identifying patterns, and generating questions for action or formal investigation. Distributions reveal centre, spread, shape, and unusual values; summary statistics must be selected with attention to skewness and measurement scale. Cross-tabulations compare categorical groups, while scatter plots and correlation describe relationships between quantitative variables. Correlation does not establish causation.

Segmentation can reveal differences hidden by aggregate summaries, but analysts must consider group size, confounding, and aggregation effects. Anomalies may be errors, rare legitimate cases, process failures, or valuable signals; they should be investigated rather than automatically deleted. Effective EDA connects observations to precise, decision-relevant questions and communicates methods, denominators, uncertainty, and limitations. Reproducibility, privacy, fairness, and human oversight are essential when EDA informs professional decisions.

Key Terms

Anomaly: An observation that differs substantially from an expected pattern.

Bivariate analysis: Examination of the relationship between two variables.

Box plot: A display summarizing quartiles, median, and potential extreme observations.

Correlation: A standardized measure of association between two quantitative variables.

Cross-tabulation: A table showing counts or proportions for combinations of categorical variables.

Distribution: The allocation of observations across possible values.

EDA: Exploratory Data Analysis; systematic examination of data to discover structure and formulate questions.

IQR: Interquartile range, calculated as Q₃ − Q₁.

Mean: The arithmetic average of numerical observations.

Median: The middle value in ordered data.

Mode: The most frequent value or category.

Outlier: An observation unusually distant from the main body of data.

Percentile: A value below which a specified percentage of observations lies.

Segmentation: Division of observations into meaningful groups for comparison.

Skewness: Asymmetry in a distribution.

Standard deviation: A measure of spread expressed in the original units of a variable.

Univariate analysis: Examination of one variable at a time.

Z-score: A standardized distance from a mean, measured in standard-deviation units.

Review Questions

Short Answer Questions

1. What is the difference between a mean and a median, and when might the median be preferable?
2. What information does a box plot provide?
3. Why must percentages in a cross-tabulation have clearly stated denominators?
4. Why does correlation not establish causation?
5. Identify two possible explanations for an anomalous observation.

Long Answer Questions

1. Explain a complete EDA workflow from data understanding through generation of decision-relevant questions.
2. Compare the uses and limitations of histograms, box plots, bar charts, and scatter plots.
3. Explain how segmentation can reveal patterns hidden by aggregate statistics, including the risk of confounding.
4. Discuss the assumptions and interpretation of Pearson correlation and the chi-square statistic.
5. Describe how privacy, fairness, reproducibility, and communication quality affect professional EDA.

Analytical / Application Questions

1. A dataset contains transaction values of 12, 15, 16, 18, 20, and 140 currency units. Which measures of centre and spread would you report, and how would you investigate the largest value?
2. A campaign has a 12% overall response rate. Segment A has 10% and Segment B has 20%, but Segment B contains only 25 customers while Segment A contains 2,000. What additional summaries or analyses are needed before recommending action?
3. A hospital dashboard shows that average waiting time improved, but the 90th percentile worsened. Propose an EDA plan to determine what happened and identify a decision-relevant next question.

Multiple Choice Questions with Answers

Question 1

Which measure of centre is generally most resistant to an extreme value?

A. Mean

B. Median

C. Variance

D. Range

Correct Answer: B. Median

Question 2

If Q₁ = 10 and Q₃ = 22, what is the interquartile range?

A. 12

B. 16

C. 32

D. 220

Correct Answer: A. 12

Question 3

What does Pearson’s correlation primarily measure?

A. Causal impact of one variable on another

B. Linear association between two quantitative variables

C. Difference between two categorical frequencies

D. Number of missing observations

Correct Answer: B. Linear association between two quantitative variables

Question 4

In a cross-tabulation, why are conditional percentages useful?

A. They remove all sampling error

B. They allow rates to be compared within groups

C. They prove that categories are independent

D. They replace the need for group sizes

Correct Answer: B. They allow rates to be compared within groups

Question 5

Which rule is commonly used to flag potential box-plot outliers?

A. Values above the mean only

B. Values outside Q₁ − 1.5IQR and Q₃ + 1.5IQR

C. All values below the median

D. Values with an even-numbered rank

Correct Answer: B. Values outside Q₁ − 1.5IQR and Q₃ + 1.5IQR

Question 6

A correlation of −0.85 indicates that the variables have:

A. A strong negative linear association

B. No relationship of any kind

C. A strong positive causal relationship

D. Equal means

Correct Answer: A. A strong negative linear association

Question 7

Which is the best first response to an apparently impossible negative waiting time?

A. Delete it without documentation

B. Replace it with the mean

C. Investigate the measurement and timestamp process

D. Treat it as proof of exceptional performance

Correct Answer: C. Investigate the measurement and timestamp process

Question 8

What is the main purpose of segmentation in EDA?

A. To conceal variation in the whole dataset

B. To compare meaningful subgroups and reveal hidden patterns

C. To guarantee statistically significant results

D. To convert categorical variables into dates

Correct Answer: B. To compare meaningful subgroups and reveal hidden patterns

Question 9

For a sample standard deviation, the usual denominator in the variance calculation is:

A. n

B. n + 1

C. n − 1

D. 2n

Correct Answer: C. n − 1

Question 10

Which statement is most appropriate when reporting an exploratory pattern?

A. The pattern proves its cause

B. The pattern is exploratory and may require confirmation

C. The pattern is valid regardless of missing data

D. The pattern requires no knowledge of the data-generating process

Correct Answer: B. The pattern is exploratory and may require confirmation