Exploratory Data Analysis and Visualization

Introduction

Exploratory Data Analysis (EDA) is the systematic examination of data before formal modelling or decision-making. It combines numerical summaries, graphical displays, domain knowledge and critical reasoning to identify structure, anomalies, data-quality problems and plausible relationships. EDA is not merely the production of attractive charts. It is an investigative process through which analysts understand what the data can support, what assumptions may be reasonable, and what questions require further study.

In artificial intelligence and machine learning, EDA is particularly important because model performance depends on the quality, representativeness and measurement properties of the data. A model may learn missing-value patterns, sampling bias, leakage or arbitrary identifiers unless these features are discovered in advance. Visualization also provides a communication bridge between technical specialists, managers, domain experts and affected stakeholders.

Learning Objectives

By the end of this chapter, learners should be able to:

1. Distinguish descriptive, distributional, bivariate and multivariate analysis.
2. Calculate and interpret appropriate summary statistics.
3. Diagnose skewness, outliers, missingness and data-quality problems.
4. Select visual encodings that accurately communicate analytical findings.
5. Use stratification, transformations and robust statistics responsibly.
6. Design dashboards and narrative visualizations for different audiences.
7. Apply EDA to a realistic data-science problem while recognizing ethical and methodological limitations.

1. The EDA Workflow

A defensible EDA workflow is iterative rather than strictly linear:

1. Frame the question. Define the decision, population, unit of analysis and time period.
2. Audit the data. Inspect schema, types, keys, duplicates, missing values, ranges and provenance.
3. Examine individual variables. Study frequencies, distributions and unusual values.
4. Explore relationships. Compare variables through grouped summaries and suitable plots.
5. Investigate subgroups. Stratify by relevant demographic, geographic, operational or temporal variables.
6. Test data assumptions. Consider independence, measurement validity, linearity, constant variance and selection effects.
7. Communicate and document. Record transformations, exclusions, uncertainty and decisions.

EDA should be separated conceptually from confirmatory inference. A pattern found by searching many variables and plots is useful for generating hypotheses, but it is not automatically evidence of a causal or statistically significant relationship. When formal tests are later used, the exploratory search process and multiple comparisons should be acknowledged.

2. Data Structure and Measurement

The first analytical decision is identifying the observational unit: a customer, transaction, patient visit, sensor reading or time interval. Combining records with different units can produce invalid summaries. For example, averaging transaction-level revenue is not equivalent to calculating revenue per customer.

Variables may be numerical, categorical, ordinal, temporal, spatial or textual. Measurement scales matter:

Identifiers should not automatically be treated as numerical predictors. A customer ID may contain digits but has no quantitative distance. Dates should generally be decomposed into analytically meaningful features, such as weekday, month, elapsed time or season, while avoiding leakage from future information.

3. Summary Statistics

For observations x₁, x₂, …, xₙ, the arithmetic mean is:

Here, x̄ is the sample mean, n is the number of non-missing observations and xᵢ is observation i. The mean is appropriate when values are quantitatively measured and extreme observations do not dominate interpretation. It has the same units as x.

The sample variance and sample standard deviation are:

The denominator n − 1 is used when s² estimates the variance of a larger population from a sample under standard random-sampling assumptions. Population variance, when the complete finite population is being described, uses n instead. Variance has squared units; standard deviation returns to the original units.

The median is the middle ordered value, or the average of the two middle values when n is even. The first and third quartiles, Q₁ and Q₃, divide the ordered data into approximate quarters. The interquartile range is:

The IQR is robust to extreme values. A commonly used exploratory rule labels observations below Q₁ − 1.5IQR or above Q₃ + 1.5IQR as potential outliers. This is a screening convention, not proof that an observation is erroneous.

The coefficient of variation is:

It is dimensionless and can compare relative dispersion when the mean is positive, meaningfully non-zero and the measurement scale has a meaningful zero. It is unsuitable when the mean is near zero or negative.

Worked Example: Robust and Non-robust Summaries

Suppose delivery times in minutes are 12, 14, 15, 16 and 53. The mean is 22 minutes, while the median is 15 minutes. The large value 53 increases the mean substantially. The ordered middle value is 15, and Q₁ and Q₃ are approximately 13 and 34 under a common quartile convention, giving IQR ≈ 21. The exact quartiles can vary slightly by software convention, so the convention should be documented. The difference between mean and median signals right skew or a possible exceptional delivery.

4. Distributional Analysis

A distribution describes how values occur across their possible range. Analysts should inspect centre, spread, shape, tails, gaps and multiple modes.

• Histogram: displays counts or densities over numerical bins. Bin width affects the apparent shape; several reasonable widths should be considered.
• Density plot: provides a smoothed view but can conceal local structure and depends on bandwidth.
• Box plot: shows median, quartiles and potential outliers compactly, especially across groups.
• ECDF: plots the proportion of observations less than or equal to each value and avoids arbitrary binning.
• Bar chart: compares counts or proportions of categories; bars should not imply continuous numerical distance.

Skewness describes asymmetry. Right-skewed data often have a long upper tail and a mean above the median; left-skewed data often have the reverse pattern. These are tendencies, not universal rules. A formal sample skewness measure may be reported, but graphical inspection and domain context remain essential.

Logarithmic transformations can make multiplicative patterns easier to examine. For positive x, analysts may use log(x). If zero values occur, log(1 + x) may be used, but its interpretation differs and should be explained. Transformations must not be used merely to hide inconvenient values.

Normality is frequently overemphasized. Many machine-learning algorithms do not require normally distributed inputs, although some statistical procedures and residual analyses do. A Q–Q plot compares sample quantiles with theoretical normal quantiles; systematic curvature indicates departures from normality. A normality test should not replace graphical and substantive reasoning, especially with large samples where trivial departures become statistically detectable.

5. Missing Data, Duplicates and Outliers

Missingness may be:

• MCAR: missing completely at random;
• MAR: missingness related to observed variables;
• MNAR: missingness related to unobserved or missing values themselves.

These are assumptions about the missingness mechanism, not labels that can be established from a single plot. Missingness should be summarized by variable and subgroup. A missing-value indicator can itself be predictive, but using it may encode process or institutional differences that require ethical review.

Possible treatments include deletion, constant or group-based imputation, model-based imputation and explicit missing categories for suitable categorical variables. Imputation must be performed within training data during model development; otherwise information from validation or test data leaks into the model. The imputation strategy and its uncertainty should be documented.

Duplicates may represent repeated legitimate events or erroneous replication. Investigate keys, timestamps and source-system rules before deleting them. Outliers may be errors, rare but valid cases, influential observations or evidence of a different population. Winsorization, removal and robust modelling are different decisions and must not be conflated.

6. Bivariate and Multivariate Exploration

For two numerical variables, a scatter plot reveals form, direction, clusters, changing spread and influential points. Pearson correlation is:

Here xᵢ and yᵢ are paired observations, and x̄ and ȳ are their sample means. The value lies between −1 and 1. Pearson correlation measures linear association, not causation, and can be distorted by outliers, nonlinearity and restricted range. Spearman correlation uses ranks and is useful for monotonic relationships, but it also does not establish causality.

For a numerical response and categorical predictor, compare group distributions with box plots, violin plots or grouped ECDFs. Always show group sizes; a visually impressive difference based on very few observations is weak evidence. For two categorical variables, contingency tables, proportions and mosaic-like displays can reveal association, but unequal denominators must be considered.

A correlation matrix is useful for screening redundant numerical features, but it can obscure nonlinear relationships and subgroup reversals. Pair plots may become unreadable as dimensionality grows. Alternatives include:

• small multiples by a meaningful subgroup;
• faceted plots with consistent scales;
• dimensionality-reduction views such as PCA, used as exploratory projections;
• clustering diagnostics;
• parallel-coordinate plots for carefully selected variables;
• interactive filtering with transparent definitions.

A crucial danger is aggregation bias. A relationship observed across all observations may disappear or reverse within groups, a phenomenon often associated with Simpson’s paradox. Stratification is therefore not optional when subgroup composition plausibly affects the result.

Figure: Subgroup trends can be obscured or misinterpreted when observations are analyzed only in aggregate.

7. Principles of Visual Design

A visualization is an argument constructed through visual encodings. Position along a common scale is generally more accurately judged than angle, area or colour intensity. Consequently, a sorted dot plot or bar chart is often preferable to a pie chart for comparing categories.

Effective visual design follows several principles:

1. Match chart to question. Use lines for ordered time, bars for category comparison, scatter plots for paired numerical measurements and maps only when geography is analytically relevant.
2. Reduce cognitive load. Remove unnecessary decoration, excessive gridlines and three-dimensional effects.
3. Make scales honest. Truncated axes can exaggerate differences in bar charts. If truncation is necessary, disclose it clearly.
4. Use colour semantically. Sequential palettes suit ordered magnitude; diverging palettes suit values around a meaningful midpoint; categorical palettes should remain distinguishable.
5. Annotate the message. Titles should state the finding or question, while subtitles define population, units and period.
6. Show uncertainty. Confidence intervals, prediction intervals, quantile bands or data-density cues prevent false precision.
7. Design accessibly. Use adequate contrast, readable type, colour-blind-safe palettes and redundant encodings such as labels or symbols.

A dashboard should have a defined audience, decision purpose and refresh process. It should establish a hierarchy: headline indicators, diagnostic trends, explanatory breakdowns and detailed records. Filters must not silently change denominators or definitions. Every metric should have a documented numerator, denominator, time window, source and owner.

8. Narrative and Interactive Visualization

A narrative visualization combines evidence with an explanatory sequence. A strong narrative identifies the question, establishes context, presents the main pattern, acknowledges exceptions and states the decision implication. Selective emphasis is legitimate when it does not conceal relevant counterevidence.

Interactivity is useful when users need to explore plausible subgroups, time periods or scenarios. It is harmful when controls create an illusion of analysis without definitions or when important findings are hidden behind defaults. Tooltips, filters and drill-downs should preserve context and provide units, sample sizes and data provenance.

Typical technologies include Python libraries such as pandas, matplotlib, seaborn, Plotly and Altair; R packages such as ggplot2 and Shiny; and business-intelligence platforms such as Tableau and Power BI. Tool choice should follow reproducibility, governance, audience and deployment requirements rather than fashion. Code-based analysis supports version control and repeatability; dashboard platforms support distribution and operational use, but both require testing.

9. Illustrative Case: Predicting Subscription Churn

Background

A fictional streaming company wants to understand why customers cancel subscriptions and improve retention without targeting customers unfairly.

Problem

The data contain customer tenure, monthly fee, plan type, support contacts, viewing hours, device count, cancellation status and region. Initial management reports claim that high support-contact customers are likely to churn.

Analysis

The analyst first defines the unit as one customer at the prediction snapshot and excludes records created after cancellation to prevent leakage. A data audit finds duplicate customer IDs, missing plan types and a small group with implausibly high device counts. Missingness is more frequent in one region, suggesting that it may reflect operational processes rather than random absence.

A histogram shows that support contacts are right-skewed, so the median and IQR are reported in addition to the mean and standard deviation. A bar chart reveals that the largest plan category also has the largest customer count; raw cancellation counts therefore exaggerate its apparent risk. Cancellation rates, with denominators shown, are compared by plan. A time plot reveals a temporary cancellation spike after a price change.

A grouped analysis shows that support contacts and churn are associated mainly among customers exposed to the price change. Within each price-change subgroup, the relationship is weaker. This demonstrates why an aggregate association should not be interpreted as a simple causal effect of support contacts.

Approach and Outcome

The team builds a leakage-controlled modelling dataset, uses a time-based validation split, and compares a transparent logistic model with a tree-based model. EDA informs feature transformations, missingness handling and fairness checks across regions. The dashboard reports cancellation rate, confidence intervals, customer counts and the date of the latest update. It does not display individual risk scores to customer-service staff until governance review is completed.

The result is a revised retention intervention focused on explaining price changes and improving support quality, rather than automatically penalizing customers who contact support frequently. A post-deployment evaluation is planned to determine whether interventions reduce cancellations without increasing inequitable treatment.

Learning Points

The case illustrates that EDA can identify leakage, denominator errors, subgroup effects, temporal changes and ethical risks before modelling. It also shows that a predictive association is not evidence that changing the associated variable will cause the desired outcome.

10. Limitations, Ethics and Governance

EDA is vulnerable to researcher degrees of freedom: analysts may try many transformations and charts until a compelling pattern appears. Maintain an analysis log, distinguish exploratory from confirmatory claims, and use held-out data or preregistered hypotheses where appropriate.

Privacy risks arise when dashboards expose small cells, precise locations or sensitive attributes. Apply data minimization, role-based access, aggregation thresholds and disclosure review. Fairness analysis should examine missingness, measurement error, subgroup performance and the consequences of decisions—not merely compare one average metric. Protected characteristics may be necessary for auditing even when they are excluded from prediction.

Charts can mislead through selective time windows, unequal axes, hidden denominators, inappropriate averages or causal wording. Governance should require data lineage, versioned code, metric definitions, review of transformations and reproducible exports. Analysts should communicate uncertainty and limitations rather than presenting EDA as objective or complete.

11. Best-Practice Checklist

• Define the observational unit and target population.
• Record data sources, extraction dates, transformations and exclusions.
• Inspect both raw values and derived variables.
• Report counts and denominators with percentages.
• Compare mean, median and robust spread when distributions are skewed.
• Investigate missingness and outliers by subgroup and time.
• Use stratification to examine confounding and aggregation effects.
• Avoid causal language for descriptive associations.
• Use accessible, honest and appropriately scaled graphics.
• Preserve reproducibility through code, metadata and version control.
• Validate dashboards with domain experts and representative users.
• Review privacy, fairness and potential downstream harms.

Chapter Summary

Exploratory Data Analysis is an iterative process for understanding data, checking assumptions and generating defensible questions before modelling. Summary statistics describe centre and variation, while medians, quartiles and IQR provide robustness against skew and extreme values. Distributional graphics reveal shape, tails and gaps; missingness, duplicates and outliers require investigation rather than automatic deletion. Bivariate and multivariate analysis must account for nonlinearity, subgroup structure, aggregation bias and the difference between association and causation. Effective visualization matches graphical form to analytical purpose, communicates uncertainty, uses honest scales and supports accessibility. Dashboards and interactive narratives should define metrics, preserve denominators and provide provenance. In professional AI and ML practice, EDA also includes leakage prevention, reproducibility, privacy, fairness and governance.

Key Terms

Exploratory Data Analysis: Systematic investigation of data using summaries, graphics and contextual reasoning.

Observational unit: Entity or event represented by one record.

Mean: Arithmetic average of numerical observations.

Median: Middle ordered value.

Variance: Average squared dispersion, with a specified sample or population denominator.

Standard deviation: Square root of variance, expressed in the original units.

Quartile: Value dividing ordered data into approximate portions.

Interquartile range: Difference between the third and first quartiles.

Outlier: Observation unusually distant from the main pattern under a stated criterion.

Skewness: Asymmetry of a distribution.

Missingness mechanism: Assumption about how and why values are absent.

Correlation: Standardized measure of association, not necessarily causation.

Aggregation bias: Distortion caused by combining observations from different groups.

Simpson’s paradox: Reversal or disappearance of an association after stratification.

Data leakage: Use of information unavailable at prediction time.

Small multiple: Series of comparable plots divided by subgroup or condition.

Dashboard: Interactive display of selected measures for monitoring or decision-making.

Uncertainty interval: Range communicating estimation or prediction uncertainty.

Data provenance: Record of a dataset’s origin, transformations and lineage.

Visual encoding: Mapping of data values to position, length, colour, shape or other visual properties.

Review Questions

Short Answer Questions

1. Why should the observational unit be defined before calculating summary statistics?
2. Distinguish sample variance from population variance.
3. What information does an IQR provide?
4. Why does correlation not establish causation?
5. State two requirements for an effective dashboard metric.

Long Answer Questions

1. Explain a complete EDA workflow from data audit to communication, including the role of domain knowledge.
2. Compare histograms, box plots, ECDFs and density plots, including one limitation of each.
3. Discuss MCAR, MAR and MNAR missingness and their implications for imputation.
4. Explain how visual design choices can improve or distort analytical communication.
5. Discuss why subgroup analysis is essential in machine-learning data exploration.

Analytical / Application Questions

1. A dataset has a mean delivery time of 31 minutes and a median of 18 minutes. Propose an investigation plan and identify at least three plausible explanations for the difference.
2. A dashboard shows that Plan A has more cancellations than Plan B. Describe the additional summaries and visualizations required before concluding that Plan A has higher churn risk.
3. Design an EDA and governance plan for a healthcare readmission dataset, addressing leakage, missingness, privacy, fairness and uncertainty.

Multiple Choice Questions with Answers

Question 1

Which statistic is generally most robust to a single extremely large observation?

A. Mean

B. Median

C. Variance

D. Standard deviation

Correct Answer: B. Median

Question 2

For a sample used to estimate a population variance, why is the denominator n − 1 commonly used?

A. It removes all outliers

B. It makes the units dimensionless

C. It provides the usual unbiased variance estimate under standard assumptions

D. It guarantees normality

Correct Answer: C. It provides the usual unbiased variance estimate under standard assumptions

Question 3

What does the interquartile range equal?

A. Q₁ + Q₃

B. Q₃ − Q₁

C. Mean − median

D. Maximum − minimum

Correct Answer: B. Q₃ − Q₁

Question 4

Which plot is most directly suited to examining the relationship between two numerical variables?

A. Scatter plot

B. Pie chart

C. Single bar chart

D. Frequency table of one variable

Correct Answer: A. Scatter plot

Question 5

A strong correlation between advertising expenditure and sales proves that advertising caused all observed sales growth.

A. True because correlation ranges from −1 to 1

B. True when the sample is large

C. False because correlation alone does not establish causation

D. False only when the variables are categorical

Correct Answer: C. False because correlation alone does not establish causation

Question 6

Which practice most directly helps prevent model leakage during imputation?

A. Impute the complete dataset before splitting it

B. Use the test-set mean for missing training values

C. Fit the imputation procedure using training data within the modelling pipeline

D. Delete every row containing a missing value

Correct Answer: C. Fit the imputation procedure using training data within the modelling pipeline

Question 7

Why should cancellation counts be accompanied by denominators when comparing subscription plans?

A. Counts always have squared units

B. Plans may have different numbers of customers

C. Denominators eliminate all bias

D. Percentages cannot be computed from counts

Correct Answer: B. Plans may have different numbers of customers

Question 8

Which colour scheme is generally appropriate for values diverging around a meaningful midpoint?

A. Diverging palette

B. Random categorical palette

C. Single hue with no ordering

D. Transparent palette only

Correct Answer: A. Diverging palette

Question 9

What is a major risk of analyzing only aggregate data?

A. It always increases sample size incorrectly

B. It can conceal subgroup reversals or confounding

C. It prevents calculation of a mean

D. It guarantees Simpson’s paradox

Correct Answer: B. It can conceal subgroup reversals or confounding

Question 10

Which statement about a box-plot outlier rule using 1.5IQR is correct?

A. Every flagged observation is a data error

B. It is a screening convention for potential unusual observations

C. It proves the distribution is non-normal

D. It applies only to categorical variables

Correct Answer: B. It is a screening convention for potential unusual observations