Exploratory Data Analysis

Introduction

Exploratory Data Analysis (EDA) is the systematic process of examining, summarising and visualising data before formal modelling or hypothesis testing. Its purpose is not merely to produce attractive charts. EDA helps an analyst understand how a dataset is structured, identify errors and unusual observations, discover potentially useful relationships, compare groups and formulate questions for subsequent investigation.

EDA is especially important in artificial intelligence and machine learning. A model learns from the data supplied to it; therefore, missing values, inconsistent categories, measurement errors, duplicated records, outliers and sampling bias can affect model performance more seriously than the choice of algorithm. EDA provides an evidence-based bridge between raw data and responsible modelling.

Exploration must be distinguished from confirmation. An exploratory finding is a useful pattern or hypothesis suggested by the data. It is not automatically proof of causation, generalisability or statistical significance. Confirmatory analysis uses pre-specified questions, appropriate statistical tests, confidence intervals or validation procedures. The same dataset should not be repeatedly searched until a desirable result is found and then presented as if it had been predicted in advance.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain the purpose and workflow of EDA.
2. Conduct univariate, bivariate and multivariate analysis.
3. Use summary statistics, grouping and cross-tabulation appropriately.
4. Select visualisations that match variable types and analytical questions.
5. Detect skewness, outliers, missingness, dependence and possible data-quality problems.
6. Separate exploratory observations from confirmatory claims.
7. Communicate findings with appropriate uncertainty, ethical awareness and reproducibility.

1. The EDA Mindset and Workflow

A useful EDA workflow is iterative rather than strictly linear:

1. Clarify the data-generating context. Identify the population, unit of observation, time period, collection method and intended use.
2. Inspect structure. Determine the number of rows and columns, data types, identifiers, targets and candidate predictors.
3. Audit quality. Check missing values, duplicates, invalid ranges, inconsistent labels and implausible measurements.
4. Explore individual variables. Examine distributions, frequencies and summaries.
5. Explore relationships. Compare variables and groups using statistics and graphics.
6. Investigate multivariate structure. Examine interactions, segments and dependence among several variables.
7. Record hypotheses and decisions. Document transformations, exclusions and findings.
8. Validate promising patterns. Use held-out data, formal tests or a new sample where appropriate.

The unit of observation must be established early. A row may represent a customer, transaction, sensor reading, patient visit or daily aggregate. Treating repeated observations from one person as independent customers, for example, can create misleading conclusions.

EDA should also distinguish between variables by role:

An identifier may appear numeric but should not normally be averaged. Similarly, an ordinal variable has order, but the distance between categories may not be equal.

2. Univariate Exploration

Univariate analysis examines one variable at a time. It describes the variable’s centre, spread, shape and data-quality characteristics.

2.1 Numerical summaries

For observations x₁, x₂, …, xₙ, the arithmetic mean is:

Here, x̄ is the sample mean, xᵢ is observation i and n is the number of non-missing observations. The mean is appropriate for quantitative data when addition and averaging are meaningful. It is sensitive to extreme values.

The sample variance and sample standard deviation are:

The denominator n − 1 is used when s² estimates the variance of a larger population from a sample under standard random-sampling assumptions. Standard deviation has the same units as x; variance has squared units.

The median is the middle ordered value, and the mode is the most frequent value. For skewed data, the median is often more representative of a typical observation than the mean. Quartiles divide ordered data into four parts. The interquartile range is:

where Q₁ is the first quartile and Q₃ is the third quartile. The IQR is resistant to extreme values.

A common exploratory rule identifies possible outliers below Q₁ − 1.5IQR or above Q₃ + 1.5IQR. This is a screening rule, not proof that observations are erroneous. A high salary, transaction or temperature may be genuine.

For a dataset containing 2, 4, 5, 7 and 12, the mean is 6.0 and the median is 5. The value 12 raises the mean above the median, suggesting right skew. The range is 8, while Q₁, Q₃ and IQR depend on the quartile convention used by the software. Analysts should document that convention when exact reproducibility matters.

2.2 Shape and distribution

A histogram displays numerical values in intervals called bins. Bin width influences the apparent pattern: very wide bins can hide structure, while very narrow bins can exaggerate random variation. Density plots provide a smoothed representation, but the smoothing bandwidth also affects appearance. A box plot displays the median, quartiles and potential extreme observations compactly.

Important distributional features include:

• Symmetry: left and right sides have similar shape.
• Skewness: one tail is longer or heavier than the other.
• Modality: the number of prominent peaks.
• Tails: the frequency of unusually small or large values.
• Gaps: intervals with few or no observations.

A bimodal distribution may indicate two populations combined in one dataset, such as two customer segments or two operating regimes. It should prompt investigation rather than immediate clustering.

2.3 Categorical and temporal variables

For categorical variables, use frequency counts and proportions. If category j occurs nⱼ times among n valid observations, its proportion is:

The proportions across mutually exclusive and exhaustive categories sum to 1. Percentages can be misleading when the denominator is omitted, especially when comparing groups of different sizes.

For dates and times, examine coverage, granularity, missing periods, seasonality and changes in recording practice. A line chart can reveal trends, but time ordering must be preserved. Randomly shuffling time observations before exploration can conceal operational events or leakage.

3. Bivariate Exploration

Bivariate analysis examines a relationship between two variables. The appropriate method depends on whether the variables are numerical, categorical or mixed.

3.1 Numerical–numerical relationships

A scatter plot displays paired numerical observations and is often the first diagnostic for association. Look for direction, form, strength, clusters, influential observations and changing variability.

The Pearson correlation coefficient is:

Here, xᵢ and yᵢ are paired observations, x̄ and ȳ are sample means, and r lies between −1 and 1 when both variables have non-zero variation. Pearson correlation measures linear association. It can be strongly affected by outliers and does not detect every nonlinear relationship.

Spearman’s rank correlation applies Pearson correlation to ranks and is useful for monotonic relationships or ordinal data. Neither correlation implies causation. A third variable, selection bias or common time trend may explain an observed association.

For example, a company may observe that advertising expenditure and sales are positively correlated. This does not establish that advertising alone caused the increase: seasonal demand, product launches or economic conditions may affect both.

3.2 Numerical–categorical relationships

To compare a numerical variable across categories, use grouped box plots, violin plots, strip plots, summary tables or group means and medians. A group summary should include sample size because a median based on five observations should not be interpreted like one based on 50,000.

For group g, the group mean is:

where n_g is the number of valid observations in group g. Differences between group means may reflect confounding. For instance, average delivery time may differ by shipping method because regions, order sizes or peak periods differ across methods.

3.3 Categorical–categorical relationships

A cross-tabulation, or contingency table, counts combinations of two categorical variables. Row percentages answer questions conditional on the row category; column percentages answer questions conditional on the column category. These are not interchangeable.

Suppose a service dataset contains the following illustrative counts:

The renewal rate is 72% for Basic and 90% for Premium. The difference is 18 percentage points. This is an exploratory group comparison, not proof that the plan caused renewal. Customer tenure, price, usage and marketing exposure may differ between plans.

For larger tables, a heat map can show counts or percentages. Always label whether cell colour represents a count, row percentage, column percentage or another quantity.

4. Multivariate Exploration

Multivariate analysis examines three or more variables simultaneously. It is essential because a bivariate pattern can disappear, reverse or become clearer after considering a third variable.

4.1 Conditioning and stratification

A simple approach is to condition a plot on a third variable. For example, plot income against spending separately for each region, or display monthly sales with separate lines for product category. Faceting reduces overplotting and exposes heterogeneity.

If an association is positive in each subgroup but negative after aggregation, the pattern may illustrate aggregation bias or Simpson’s paradox. This is a warning that group composition matters.

4.2 Colour, shape and size

Additional variables can be encoded by colour, marker shape or size. These encodings should be used sparingly. Too many visual dimensions reduce readability and may cause viewers to infer precision that the chart does not support. Colour should be accessible to people with colour-vision differences and should not be the only way to distinguish categories.

4.3 Correlation matrices and dimensional views

A correlation matrix summarises pairwise numerical associations. It is useful for screening redundancy, but it does not replace scatter plots because nonlinear forms, clusters and outliers can produce the same correlation value.

For many variables, pair plots can reveal relationships and distributions, although they become impractical as dimensionality grows. Principal Component Analysis (PCA) can create lower-dimensional representations for exploration. PCA is sensitive to scaling, so standardisation may be necessary when variables have different units. A two-dimensional projection is a view of the data, not a complete representation of its structure.

4.4 Grouping and aggregation

Grouping operations calculate summaries by one or more categories, such as average revenue by region and quarter. Aggregation can reveal operational segments but can also conceal within-group variation. Every grouped result should retain group size and, where useful, measures of spread.

A weighted average may be necessary when groups differ in size or importance. If group means x̄_g are combined using group sizes n_g, the overall mean is:

This formula assumes that the group means refer to the same measurement definition and that groups do not overlap. Averaging group means without weights is only appropriate when equal weighting of groups is explicitly intended.

5. Visual Diagnostics and Chart Selection

The question and variable type should determine the visualisation.

Bar charts should generally begin at zero when comparing lengths, because truncated axes can exaggerate differences. Line charts imply an ordered or continuous horizontal dimension; connecting unrelated categories with a line is misleading. Pie charts become difficult to compare with many categories, and three-dimensional effects usually reduce accuracy.

A good chart includes a descriptive title, labelled axes, units, a clear legend, an indication of missing or excluded values where relevant, and an appropriate scale. Visualisations should expose data rather than conceal it. Overplotting can be addressed with transparency, jitter, binning or aggregation, but each choice changes the display and should be documented.

6. Missing Data, Outliers and Data Quality

Missingness should be measured and visualised by variable and, where relevant, by time or subgroup. The mechanism matters:

• MCAR (Missing Completely At Random): missingness is unrelated to observed and unobserved values.
• MAR (Missing At Random): missingness depends on observed variables after conditioning.
• MNAR (Missing Not At Random): missingness depends on unobserved or omitted information.

These mechanisms cannot always be established from the dataset alone. Replacing missing values with zero is usually inappropriate unless zero genuinely means “none.” Mean imputation reduces variation and can distort relationships. More advanced imputation should be fitted using training data only when building predictive models, to avoid leakage.

Outliers may arise from data-entry errors, sensor malfunction, fraud, rare but valid events or a different population. Do not delete them automatically. Investigate their origin, compare with domain limits and assess how conclusions change with and without them. Robust summaries such as medians and IQRs can complement means and standard deviations.

Quality checks should include:

• duplicate identifiers and duplicate rows;
• impossible dates or negative quantities;
• inconsistent units and currency;
• spelling variants in categories;
• values outside documented ranges;
• unexpected changes in measurement procedures;
• target values recorded after the prediction time.

7. Separating Exploration from Confirmation

EDA generates hypotheses, such as “customers with long response times may be more likely to leave.” Confirmation requires a clearly defined outcome, comparison, analysis method and uncertainty assessment. A p-value, confidence interval or model score does not repair a biased sample or an improperly defined question.

Multiple exploration creates a multiple-comparisons problem: if enough relationships are searched, some will appear unusual by chance. Remedies include pre-registering confirmatory questions, using a holdout dataset, correcting for multiple testing where appropriate, replicating findings and reporting the number and nature of searches performed.

A responsible report distinguishes:

• Observation: “The premium group had a higher renewal rate in this sample.”
• Interpretation: “The difference may be associated with plan membership or with characteristics of customers selecting that plan.”
• Claim: “The plan causes higher renewal.”

The third statement requires a causal design or strong causal assumptions, not merely an exploratory chart.

8. Illustrative Case: Exploring Customer Churn

Background

A fictional subscription company has 12 months of customer records. Each row represents a customer-month, and variables include customer ID, region, plan, monthly usage, support contacts, account age and a churn indicator for the following month.

Problem

The company wants to identify patterns associated with churn before developing a predictive model. Management is interested in whether support contacts and plan type are useful signals.

Analysis

First, the analyst checks the unit of observation. Because customers appear in several months, treating all rows as independent customers would overstate the amount of independent evidence. The analyst then checks whether the churn indicator is defined after the observed predictors; otherwise, target leakage could occur.

Univariate analysis shows that monthly support contacts are strongly right-skewed: most customer-months have zero or one contact, while a small number have many. The median is therefore more informative than the mean for describing a typical month. A missingness audit finds that usage is absent more often during a system migration period.

A grouped table shows churn rates by plan. The premium plan has a lower observed churn rate, but premium customers also have longer account age and higher usage. A cross-tabulation by plan and account-age band shows that the apparent plan difference is smaller within comparable age bands. This demonstrates why subgroup analysis matters.

A scatter plot of support contacts and usage, coloured by churn status, reveals that high contact frequency is concentrated among low-usage customers. However, the plot contains repeated observations for the same customers, so it is descriptive rather than a test of independent effects.

Approach and solution

The analyst documents all definitions, excludes records created after the prediction time, retains a missingness indicator for usage, and proposes a time-based train–validation split for modelling. EDA findings are treated as candidate features and hypotheses. A later model is evaluated on future months rather than on randomly mixed records.

Outcome

The analysis identifies three practical possibilities: customers with sudden usage decline may warrant attention; repeated support contacts may indicate dissatisfaction; and plan comparisons should be adjusted for tenure and customer composition. It does not establish that any variable causes churn.

Learning points

1. The unit of analysis determines what constitutes independent evidence.
2. Group comparisons require denominators and relevant confounders.
3. Temporal ordering is essential in predictive analytics.
4. EDA can improve data quality and feature design without becoming a causal conclusion.

9. Tools, Reproducibility and Professional Practice

Common tools include spreadsheet software for small datasets, SQL for filtering and aggregation, and Python or R for reproducible analysis. A professional workflow stores data dictionaries, code, package versions, chart specifications and decisions about exclusions or transformations. Random seeds should be recorded for procedures involving randomness.

EDA code should be modular and rerunnable. Analysts should avoid manually editing source data, preserve raw files, and create derived datasets through documented transformations. Automated data-quality tests can check ranges, uniqueness, missingness thresholds and schema changes.

AI-assisted analytics can suggest charts, summaries or anomalies, but generated code and interpretations require human verification. Models may reproduce bias present in historical data, and visual summaries can expose sensitive information. Access controls, data minimisation, anonymisation where suitable and disclosure of limitations are part of responsible practice.

10. Challenges and Limitations

EDA is powerful but not infallible. Observational data may reflect selection bias, survivorship bias, measurement error or nonresponse. Aggregation can hide variation, and visual patterns can be influenced by scale, binning and colour choices. Correlation does not establish causation, and an apparent anomaly may be a genuine rare event.

Privacy is also relevant. Combining several seemingly harmless variables can re-identify individuals, especially in small groups. Analysts should suppress or aggregate sensitive categories when necessary and follow applicable institutional and legal requirements.

The best practice is not to seek a single “correct” chart or statistic. Instead, use multiple complementary views, test the stability of findings, consult domain experts and clearly state what the data can and cannot support.

Chapter Summary

Exploratory Data Analysis is a disciplined investigation of data structure, quality, distributions, relationships and unusual observations. Univariate analysis describes individual variables through frequencies, centre, spread and shape. Bivariate analysis uses scatter plots, grouped summaries and cross-tabulations to examine associations between pairs of variables. Multivariate exploration uses stratification, faceting, correlation views and aggregation to reveal interactions and subgroup differences.

Effective EDA requires careful attention to the unit of observation, denominators, missingness, outliers, temporal order and visual encoding. Grouped differences and correlations are exploratory evidence, not automatic causal conclusions. Reproducible code, documented decisions, validation on new data and ethical safeguards turn EDA into a reliable foundation for machine learning and professional decision-making.

Key Terms

Exploratory Data Analysis: Systematic examination of data to discover structure, quality issues and potential patterns.

Univariate analysis: Analysis of one variable at a time.

Bivariate analysis: Analysis of the relationship between two variables.

Multivariate analysis: Analysis involving three or more variables.

Mean: Arithmetic average of numerical observations.

Median: Middle value after observations are ordered.

Variance: Average squared deviation measure, commonly estimated with denominator n − 1 in a sample.

Standard deviation: Square root of variance, expressed in the original units.

Quartile: A value dividing ordered data into parts of approximately equal frequency.

Interquartile range: Difference between the third and first quartiles.

Outlier: An observation unusually distant from the main pattern or distribution.

Skewness: Asymmetry in a distribution.

Cross-tabulation: Table showing counts or percentages for combinations of categorical variables.

Correlation: Numerical measure of association, often specifically linear association.

Stratification: Examining results separately within defined subgroups.

Aggregation: Combining observations to calculate group-level summaries.

Missingness: The condition in which a variable value is unavailable.

Data leakage: Use of information unavailable at the time a prediction would be made.

Exploratory claim: A pattern or hypothesis suggested by analysis but not yet confirmed.

Confirmatory analysis: Pre-specified analysis used to evaluate a defined claim.

Review Questions

Short Answer Questions

1. What is the primary purpose of Exploratory Data Analysis?
2. Why can the median be preferable to the mean for a highly skewed variable?
3. What information does a cross-tabulation provide?
4. Why does correlation not establish causation?
5. What is the difference between an identifier and a numerical measurement?

Long Answer Questions

1. Explain a complete EDA workflow from understanding the data-generating context to validating exploratory findings.
2. Compare univariate, bivariate and multivariate analysis, giving suitable examples and visualisations for each.
3. Explain how missing values and outliers should be investigated rather than automatically removed.
4. Discuss how visual design choices such as bin width, axis scale and colour can affect interpretation.
5. Explain why temporal ordering and the unit of observation are important in machine-learning EDA.

Analytical / Application Questions

1. A dataset shows that customers using Product A have lower churn than customers using Product B. Design a grouped and multivariate exploration that investigates whether tenure, region and usage could explain the difference.
2. A histogram of transaction values has a long right tail and several very large observations. Recommend numerical summaries, visualisations and data-quality checks, and explain how you would decide whether the large values are errors.
3. An analyst discovers a strong relationship between two variables after testing hundreds of variable pairs. Explain why this result should be treated cautiously and propose a confirmation strategy.

Multiple Choice Questions with Answers

Question 1

Which statistic is generally most resistant to extreme numerical observations?

A. Mean

B. Median

C. Variance

D. Range

Correct Answer: B. Median

Question 2

What does a cross-tabulation primarily summarise?

A. The relationship between categorical variables

B. The distribution of one continuous variable only

C. The coefficients of a regression model

D. The order of observations in a time series

Correct Answer: A. The relationship between categorical variables

Question 3

If Q₁ = 10 and Q₃ = 18, what is the interquartile range?

A. 8

B. 14

C. 28

D. 80

Correct Answer: A. 8

Question 4

Which visualisation is most suitable as a first diagnostic for the relationship between two numerical variables?

A. Pie chart

B. Scatter plot

C. Single bar chart

D. Frequency table of one category

Correct Answer: B. Scatter plot

Question 5

Why should a grouped mean normally be reported with the group size?

A. The group size shows the chart colour

B. The group size indicates how much evidence supports the summary

C. The group size converts a mean into a median

D. The group size proves causation

Correct Answer: B. The group size indicates how much evidence supports the summary

Question 6

Which statement best describes an exploratory finding?

A. It is automatically a causal conclusion

B. It is a pattern that may generate a hypothesis for later validation

C. It is valid only when the p-value is zero

D. It cannot be affected by data quality

Correct Answer: B. It is a pattern that may generate a hypothesis for later validation

Question 7

What is a major risk of using information recorded after the prediction time?

A. Underplotting

B. Data leakage

C. Loss of categorical labels

D. Reduced screen resolution

Correct Answer: B. Data leakage

Question 8

For a highly skewed distribution, which pair is usually more robust for describing centre and spread?

A. Mean and standard deviation

B. Median and interquartile range

C. Mode and variance only

D. Maximum and minimum only

Correct Answer: B. Median and interquartile range

Question 9

What does Pearson correlation primarily measure?

A. Linear association between paired numerical variables

B. Causation between two categorical variables

C. The number of missing values

D. The proportion of each category in a table

Correct Answer: A. Linear association between paired numerical variables

Question 10

Why can comparing overall group rates be misleading?

A. Groups may differ in composition or be affected by confounding variables

B. Percentages cannot be calculated from counts

C. All group comparisons are invalid

D. A larger sample always creates bias

Correct Answer: A. Groups may differ in composition or be affected by confounding variables