Exploratory Data Analysis

Introduction

Exploratory Data Analysis (EDA) is the systematic examination of data before formal modelling, hypothesis testing, or managerial decision-making. It combines numerical summaries, visual displays, domain knowledge, and careful questioning to reveal structure in a dataset. EDA helps analysts identify distributions, trends, segments, relationships, missing values, data-quality problems, and unusual observations.

EDA is not a substitute for statistical inference. A visible pattern may be produced by sampling variation, measurement error, confounding, or biased data collection. Consequently, the central discipline of EDA is not merely finding patterns, but judging which patterns are credible, useful, and sufficiently supported by evidence.

In business analytics, EDA may be used to understand customer churn, investigate sales performance, compare operational units, detect unusual transactions, evaluate marketing campaigns, or prepare data for predictive modelling.

Learning Objectives

By the end of this chapter, you should be able to:

• distinguish univariate, bivariate, and multivariate analysis;
• select suitable statistical summaries and visualisations for different variable types;
• interpret centre, spread, shape, association, segments, trends, and anomalies;
• calculate and explain common descriptive statistics;
• recognise the effects of skewness, outliers, missing data, and aggregation;
• conduct a reproducible EDA workflow;
• avoid unsupported causal or generalised conclusions; and
• communicate exploratory findings to technical and managerial audiences.

1. The Role and Logic of EDA

A useful EDA process is iterative:

1. clarify the business question and unit of analysis;
2. inspect data structure, variable definitions, and measurement scales;
3. assess quality, missingness, duplicates, and impossible values;
4. examine variables individually;
5. examine pairs and groups of variables;
6. investigate multivariate structure and unusual observations;
7. test whether findings are robust to reasonable analytical choices; and
8. document findings, uncertainties, and proposed next steps.

The unit of analysis must be explicit. A row may represent a customer, order, product, account-month, or store-day. Mixing units can produce invalid conclusions. For example, analysing transaction rows while interpreting the result as a customer-level relationship gives heavy buyers disproportionate influence.

Measurement and variable types

A numerical code does not automatically make a variable quantitative. Coding “low,” “medium,” and “high” as 1, 2, and 3 does not justify calculating a meaningful mean unless the scale has an appropriate interpretation.

2. Univariate Analysis

Univariate analysis studies one variable at a time. Its purposes include describing the typical observation, measuring variability, understanding distributional shape, and detecting potential data errors.

Frequency and proportion

For a category or value occurring f times in a dataset of n observations, its sample proportion is:

Here, p̂ is the observed proportion, f is the frequency, and n is the number of non-missing observations used. The proportion has no physical unit and lies between 0 and 1. Percentages are obtained by multiplying by 100. The denominator should be stated because percentages based on all records differ from percentages based only on valid responses.

Measures of centre

The arithmetic sample mean is:

where xᵢ is observation i, and n is the number of observations. The mean is appropriate for quantitative data when values are meaningfully averaged, but it is sensitive to extreme observations.

The median is the middle ordered observation, or the average of the two middle observations when n is even. It is resistant to extreme values and is often preferable for income, transaction value, waiting time, and other right-skewed variables.

The mode is the most frequently occurring value or category. It is especially useful for categorical variables, although a dataset may have more than one mode or no uniquely informative mode.

Measures of spread

The range is the maximum minus the minimum. It is easy to understand but depends entirely on two observations.

The sample variance is:

The sample standard deviation is:

The denominator n−1 is used when s² estimates the variance of a broader population from a sample, under standard random-sampling assumptions. The standard deviation has the same units as the original variable; variance has squared units.

The interquartile range is:

where Q₁ is the 25th percentile and Q₃ is the 75th percentile. The IQR describes the spread of the central half of observations and is less affected by extremes than the range or standard deviation.

A commonly used exploratory rule flags observations below Q₁ − 1.5IQR or above Q₃ + 1.5IQR as potential outliers. This is a screening convention, not proof that an observation is erroneous or should be removed.

Worked example: delivery times

Suppose five delivery times, in minutes, are 20, 22, 25, 27, and 56. The mean is:

The median is 25 minutes. The mean exceeds the median because the 56-minute delivery pulls the mean upward. If the 56-minute observation represents a genuine service failure, removing it would conceal an operational issue. A better analysis reports both the typical delivery time and the exceptional event, then investigates its cause.

Distributional shape

A distribution may be symmetric, left-skewed, right-skewed, unimodal, multimodal, heavy-tailed, or concentrated at a boundary such as zero. Histograms, density plots, box plots, and empirical cumulative distribution plots help reveal shape.

A histogram depends on bin width and starting points. Excessively wide bins hide structure; excessively narrow bins exaggerate random fluctuation. Density plots can be useful for comparison but may obscure sample size and depend on smoothing choices. Box plots compactly show the median, quartiles, IQR, and potential extreme observations, but they do not display every feature of a distribution.

A useful standardised value is the z-score:

It expresses an observation’s distance from the sample mean in standard-deviation units. It is most interpretable for distributions that are approximately symmetric and not excessively heavy-tailed. A large z-score is a screening signal, not a universal definition of an anomaly.

3. Bivariate Analysis

Bivariate analysis examines two variables to investigate differences, association, and possible predictive relationships. The appropriate method depends on whether variables are categorical, quantitative, or time-based.

Two categorical variables

A contingency table cross-classifies observations. For example, a retailer may compare membership status with renewal outcome. Row percentages answer questions such as “within each membership group, what proportion renewed?” Column percentages answer a different question. Percentages must therefore be labelled clearly.

A grouped or stacked bar chart can show the same information visually. However, stacked bars make precise comparisons difficult when segments do not share a baseline. Grouped bars are often better for comparing category proportions across groups.

One categorical and one quantitative variable

Compare distributions rather than only group means. Useful displays include side-by-side box plots, violin plots, strip plots, and summary tables containing sample size, median, mean, and spread.

A difference in means can be expressed as:

where x̄₁ and x̄₂ are the sample means of groups 1 and 2. The difference has the same units as the outcome. It should be accompanied by group sizes, variability, and a clear definition of the comparison. A difference may reflect confounding, selection, or unequal group composition rather than a treatment effect.

Two quantitative variables

A scatterplot is the principal exploratory display. It can reveal direction, form, strength, clusters, changing variability, and influential observations. Use transparency, jitter, or aggregation when many points overlap, while ensuring that aggregation does not hide important variation.

The sample covariance is:

where ȳ is the sample mean of y. Covariance retains measurement units and is therefore difficult to compare across variable pairs.

The Pearson correlation is:

where sₓ and sᵧ are sample standard deviations. The value lies between −1 and +1 and measures linear association. It is not a measure of causation, and a value near zero does not rule out a nonlinear relationship.

Correlation can be distorted by outliers, restricted ranges, nonlinearity, and mixtures of subgroups. Always inspect a scatterplot alongside the numerical coefficient. Rank-based measures such as Spearman’s correlation may be more appropriate for monotonic relationships involving ordinal data or non-normal distributions, but they also do not establish causality.

Time-based bivariate analysis

A time series should generally be plotted in chronological order. Look for level, trend, seasonality, cycles, breaks, and changing variability. A moving average can smooth short-term fluctuations:

where k is the window length and yₜ is the observation at time t. Smoothing is descriptive; it can conceal turning points and should not be interpreted as a causal model.

4. Multivariate Analysis

Multivariate EDA examines several variables jointly. It is essential because relationships observed in aggregate can change after conditioning on another variable.

Conditioning and confounding

Suppose a company observes that customers receiving a discount have lower profit. This may be because discounts are targeted to difficult-to-serve customers. Customer difficulty is a potential confounder: it is associated with discount assignment and profit. Stratified analysis, colour-coded scatterplots, faceting, or regression models can help reveal such structure, although observational EDA alone cannot prove causality.

Simpson’s paradox occurs when an association in aggregated data reverses or changes direction within relevant subgroups. Analysts should therefore ask whether comparisons are fair, whether group composition differs, and whether a key variable has been omitted.

Segmentation

Segmentation groups observations with similar characteristics. Simple business segmentation may use rules, quantiles, or domain-defined categories. More formal approaches include k-means clustering, hierarchical clustering, and density-based methods.

For k-means clustering, observations are assigned to one of K clusters by minimising within-cluster squared Euclidean distance:

Here, Cⱼ is cluster j, xᵢ is an observation represented by selected numerical features, and μⱼ is the cluster centroid. The method assumes that numerical variables are appropriately scaled and that distance-based clusters are meaningful. Results depend on feature selection, scaling, initialisation, and the chosen value of K. Clusters are not automatically natural market segments; they require business interpretation and validation.

A heat map, pair plot, parallel-coordinate plot, or low-dimensional projection can support multivariate exploration. Dimensionality-reduction methods such as principal component analysis may help visualise high-dimensional structure, but the transformed axes must be interpreted carefully. A visually separated projection does not by itself establish meaningful or stable groups.

Correlation matrices and multiple comparisons

A correlation matrix summarises pairwise linear associations among quantitative variables. Strong correlations may indicate redundancy, a common underlying driver, or a data-processing artefact. Examining many variable pairs increases the probability of finding apparently interesting relationships by chance. Exploratory findings should therefore be labelled as exploratory and assessed on new data or through a pre-specified confirmatory analysis.

5. Anomalies, Missingness, and Data Quality

An anomaly is an observation that differs substantially from expected behaviour. It may be a data-entry error, fraud, system failure, rare but legitimate event, or a change in the underlying process.

A responsible anomaly workflow is:

• verify the record and its provenance;
• check units, timestamps, duplicates, and joins;
• compare with relevant peer groups and historical values;
• consult domain experts;
• preserve the original value;
• document any correction, exclusion, or transformation; and
• assess whether conclusions change under alternative treatments.

Missingness should be described by variable, time, subgroup, and reason where known. Three conceptual mechanisms are commonly distinguished: missing completely at random, missing at random conditional on observed variables, and missing not at random. These are assumptions about the data-generating process, not labels that can usually be proven from the observed dataset alone.

A missing-value indicator can itself be informative. For example, income may be missing more often among applicants who decline to disclose it. Automatically replacing all missing values with zero is usually invalid because “missing” and “zero” have different meanings.

6. A Reproducible EDA Workflow

A professional workflow begins with a data dictionary specifying names, definitions, units, valid ranges, source systems, and privacy restrictions. Analysts should retain a raw-data layer and create documented transformed layers rather than overwriting source data.

A practical sequence is:

1. Frame the question. Identify the decision, population, time period, and unit of analysis.
2. Audit structure. Inspect dimensions, types, unique values, duplicates, keys, and joins.
3. Validate quality. Check ranges, impossible dates, inconsistent categories, missingness, and measurement units.
4. Summarise univariately. Use appropriate numerical and graphical summaries.
5. Compare relevant groups. Examine distributions, not only averages.
6. Explore relationships. Use scatterplots, contingency tables, time plots, and stratification.
7. Investigate anomalies. Verify rather than automatically delete.
8. Assess robustness. Repeat important summaries using reasonable alternatives.
9. Communicate. State evidence, uncertainty, limitations, and recommended action.

Reproducibility requires versioned data extracts, documented code or workflow steps, fixed definitions, recorded filters, and a clear distinction between exploratory and confirmatory analyses.

7. Applied Case: Exploring Customer Churn

The following is an illustrative case. A subscription-based software company wants to understand why monthly customer churn increased from 4.2% to 5.6% over two quarters. The dataset contains customer identifier, plan type, tenure, monthly fee, support contacts, product usage, region, payment method, cancellation status, and month.

Background and problem

Management initially suspects that a price increase caused the rise in churn. The analytical problem is to explore whether churn differs by tenure, plan, usage, region, and price exposure without claiming that any observed association proves causation.

Analysis

The analyst first checks the unit of analysis. There are multiple monthly records per customer, so the analysis distinguishes customer-month churn from customer-level summaries. Duplicate customer-month keys are investigated. The data dictionary reveals that “support contacts” includes both human-agent and automated interactions, so the variable is split before interpretation.

Univariate analysis shows that monthly fees are right-skewed, tenure is concentrated among newer customers, and usage has a substantial mass at zero. A histogram of fees and a box plot by plan show that the mean fee is pulled upward by enterprise accounts.

Bivariate analysis compares churn proportions by plan and tenure band. Churn is highest among customers with less than three months’ tenure. A line chart shows that the increase began shortly after a billing-system migration, but this temporal proximity is not evidence that the migration caused the increase. A scatterplot indicates a nonlinear relationship between product usage and churn: both very low usage and unusually high usage appear in different customer groups.

Multivariate exploration stratifies the tenure comparison by plan. The apparent price–churn relationship weakens after plan and tenure are considered together, suggesting that the original aggregate association was partly due to newer customers being concentrated in plans with different fees. A heat map of missingness reveals that usage data are more frequently absent for customers onboarded through one partner channel.

Potential anomalies include a set of monthly fees equal to zero after the migration. Investigation shows that these are temporary promotional credits, not missing values. They are retained but flagged with a promotion indicator.

Approach and proposed solution

The analyst recommends a dashboard containing churn by cohort, plan, tenure, and region; a data-quality monitor for billing and usage fields; and a subsequent confirmatory model designed before further testing. The exploratory report explicitly separates observations from hypotheses:

• Observation: early-tenure customers have higher observed churn.
• Hypothesis: onboarding quality or early product value may influence retention.
• Not established: that price increases caused the overall churn increase.

Outcome and learning points

The company postpones a broad price rollback and instead investigates onboarding and billing-system issues. It also establishes cohort-based monitoring because aggregate churn concealed changes in customer composition.

The case illustrates five principles: define the unit of analysis, inspect distributions before averages, condition on relevant groups, investigate anomalies rather than delete them, and distinguish temporal association from causal evidence.

8. Visualisation and Communication Best Practices

A visualisation should have a clear analytical purpose. Use position and length for precise comparisons, colour sparingly, and direct labels where possible. Bar charts should generally begin at zero when bar length represents magnitude. Line charts require an ordered horizontal axis. Avoid three-dimensional effects, unnecessary decoration, and scales that exaggerate differences.

Every chart should identify the population, time period, unit, denominator, and relevant filters. Report sample sizes and missing-value treatment. A technically accurate graph can still mislead if it uses an inappropriate denominator or omits an important subgroup.

A strong analytical statement has three parts: evidence, interpretation, and limitation. For example: “Customers with fewer than three months’ tenure had the highest observed churn in the available records. This association is consistent with an onboarding problem, but plan mix and acquisition channel may contribute; a follow-up analysis is required.”

9. Ethical, Governance, and Professional Issues

EDA can expose sensitive information even when the original purpose appears routine. Analysts should apply data minimisation, access controls, appropriate retention, and de-identification. Small groups may be re-identifiable through combinations of attributes.

Fairness analysis should examine whether data quality, missingness, or sampling differs across protected or vulnerable groups. A model or dashboard based on incomplete data may systematically disadvantage some customers. Suppressing a group because it is small can also conceal unequal outcomes; privacy-preserving aggregation and careful disclosure control are preferable.

Analysts must not manipulate chart scales, selectively report favourable segments, or remove inconvenient observations without justification. AI-assisted analytics can accelerate coding and visualisation, but generated code and interpretations require human review. Sensitive data should not be entered into unauthorised tools, and all automated transformations should be auditable.

10. Limitations and Contemporary Developments

EDA is vulnerable to researcher degrees of freedom: many variables, filters, transformations, and charts increase opportunities for accidental findings. Exploratory patterns may fail to replicate. Large datasets can make trivial differences appear visually or statistically impressive, while small datasets may hide meaningful effects.

Modern tools support interactive dashboards, automated profiling, anomaly detection, and natural-language querying. These tools improve speed but do not remove the need for domain knowledge, sound denominators, causal caution, and reproducibility. Automated systems may select misleading charts, infer incorrect variable meanings, or amplify historical bias.

The best practice is to use automation for mechanical inspection while retaining human responsibility for definitions, assumptions, ethical judgement, and communication.

Chapter Summary

Exploratory Data Analysis is an iterative process for understanding data before formal modelling or decision-making. Univariate analysis describes individual variables through frequencies, measures of centre, spread, shape, and unusual values. Bivariate analysis examines group differences, contingency tables, time patterns, scatterplots, covariance, and correlation. Multivariate analysis conditions on subgroups, investigates confounding, supports segmentation, and examines high-dimensional structure.

EDA depends on correct units of analysis, suitable visual encodings, explicit denominators, and careful treatment of missing values and anomalies. The mean, median, standard deviation, IQR, correlation, and moving average are useful summaries only when their assumptions and limitations are understood. Patterns are not automatically causal, representative, stable, or actionable. Reproducible workflows, ethical governance, domain validation, and clear communication transform exploratory findings into responsible analytical recommendations.

Key Terms

Unit of analysis: The entity represented by one observation or record.

Univariate analysis: Examination of one variable at a time.

Bivariate analysis: Examination of two variables and their relationship.

Multivariate analysis: Examination of several variables jointly.

Mean: Arithmetic average of quantitative observations.

Median: Middle value after observations are ordered.

Mode: Most frequently occurring value or category.

Variance: Average squared deviation measure, usually using a sample correction.

Standard deviation: Square root of variance, expressed in original units.

Interquartile range: Difference between the third and first quartiles.

Outlier: An observation unusually distant from other observations under a defined criterion.

Skewness: Asymmetry in a distribution.

Correlation: Standardised measure of linear association between two quantitative variables.

Confounding: A situation in which another variable helps explain an observed association.

Segmentation: Division of observations into interpretable groups.

Missingness: The condition in which a value is unavailable or unrecorded.

Robustness: Stability of a finding under reasonable analytical alternatives.

Cohort: A group sharing a defining characteristic or starting period.

Reproducibility: Ability to recreate an analysis from documented data and procedures.

Exploratory finding: A pattern discovered for further investigation rather than established as confirmed evidence.

Review Questions

Short Answer Questions

1. Why should the unit of analysis be specified before beginning EDA?
2. When is the median preferable to the mean?
3. What information does the IQR provide?
4. Why does correlation not establish causation?
5. Name two possible explanations for an apparent outlier.

Long Answer Questions

1. Explain a complete EDA workflow for a customer-retention dataset, including quality checks and communication requirements.
2. Compare histograms, box plots, density plots, and empirical cumulative distribution plots. Discuss the strengths and limitations of each.
3. Explain how confounding and Simpson’s paradox can mislead business analysis.
4. Discuss the assumptions, interpretation, and limitations of Pearson correlation.
5. Explain how missing data and data governance affect the credibility of exploratory findings.

Analytical / Application Questions

1. A retailer finds that customers receiving discounts have lower profit. Design an EDA plan to determine whether customer type, product category, or order size may explain the relationship without claiming a causal effect.
2. A dataset contains delivery times of 18, 20, 21, 22, 23, and 96 minutes. Calculate the mean and median, describe the effect of the extreme value, and recommend an appropriate next investigation.
3. A churn dashboard shows that churn is higher in Region A than Region B. Propose at least three stratified analyses and explain how each could change the interpretation of the regional difference.

Multiple Choice Questions with Answers

Question 1

Which statistic is generally most resistant to an extreme observation?

A. Mean

B. Median

C. Variance

D. Range

Correct Answer: B. Median

Question 2

What does the interquartile range measure?

A. The distance between the minimum and maximum

B. The average distance from the mean

C. The spread of the central 50% of observations

D. The proportion of missing observations

Correct Answer: C. The spread of the central 50% of observations

Question 3

Which display is most directly appropriate for examining the relationship between two quantitative variables?

A. Scatterplot

B. Pie chart

C. Single bar chart

D. Frequency table of one variable

Correct Answer: A. Scatterplot

Question 4

A correlation of zero necessarily implies which conclusion?

A. There is no relationship of any kind

B. There is no linear association in the sample, although nonlinear association may exist

C. One variable causes the other

D. The variables have identical means

Correct Answer: B. There is no linear association in the sample, although nonlinear association may exist

Question 5

In the sample variance formula, why is n − 1 commonly used?

A. To change the units of variance

B. To remove all outliers

C. To provide the usual sample correction when estimating population variance

D. To guarantee a normal distribution

Correct Answer: C. To provide the usual sample correction when estimating population variance

Question 6

Which is the most responsible first action when an apparent outlier is found?

A. Delete it immediately

B. Replace it with the mean

C. Verify its provenance, units, and validity

D. Hide it from the visualisation

Correct Answer: C. Verify its provenance, units, and validity

Question 7

What is a major risk of interpreting an aggregate association without subgroup analysis?

A. The chart will always be unreadable

B. Confounding or Simpson’s paradox may distort the conclusion

C. The sample size becomes exactly zero

D. All variables become categorical

Correct Answer: B. Confounding or Simpson’s paradox may distort the conclusion

Question 8

For the observations 2, 4, 6, 8, and 10, what is the sample mean?

A. 5

B. 6

C. 7

D. 8

Correct Answer: B. 6

Question 9

Which statement about k-means clustering is correct?

A. It requires no choices about scaling or number of clusters

B. It assigns observations while minimising within-cluster squared distance

C. It proves that discovered clusters are natural market segments

D. It is designed only for nominal categories

Correct Answer: B. It assigns observations while minimising within-cluster squared distance

Question 10

Why should percentages in a contingency table identify their denominator?

A. Different denominators answer different comparison questions

B. Percentages cannot be used with categorical variables

C. The denominator determines whether data are quantitative

D. Percentages are always causal measures

Correct Answer: A. Different denominators answer different comparison questions