Exploratory Data Analysis and Visualization

Introduction

Exploratory Data Analysis (EDA) is the systematic process of examining, summarising, visualising and questioning data before formal modelling or hypothesis testing. It is not merely the production of charts. EDA combines numerical summaries, distributional reasoning, comparisons across groups, association analysis and visual diagnostics to understand how a dataset was generated and what analytical claims it can support.

For postgraduate data scientists, EDA has two complementary purposes. First, it is a discovery activity: it reveals structure, unusual observations, missingness, heterogeneity and potentially useful relationships. Second, it is a quality-control activity: it exposes errors, inconsistencies, data leakage, measurement problems and assumptions that could invalidate later analysis. A sophisticated model cannot compensate for a poorly understood dataset.

EDA should be conducted with domain knowledge and a clear record of decisions. Analysts must distinguish between an observed pattern, a plausible explanation and a causal conclusion. A correlation may suggest a useful predictor, but it does not by itself establish that changing one variable will change another.

Learning Objectives

By the end of this chapter, you should be able to:

1. Characterise variables using appropriate descriptive statistics.
2. Analyse distributions, skewness, tails, outliers and missing-data patterns.
3. Construct and interpret cross-tabulations and conditional summaries.
4. Calculate and critically interpret covariance and correlation.
5. Select visualisations appropriate to data type, analytical purpose and audience.
6. Use visual diagnostics to identify anomalies, non-linearity and model-assumption violations.
7. Recognise misleading graphical practices and statistical misinterpretations.
8. Design accessible, reproducible and ethically responsible EDA workflows.

1. The EDA Mindset and Analytical Workflow

A useful EDA workflow is iterative rather than strictly linear:

1. Clarify the analytical question. Determine the population, unit of analysis, outcome, explanatory variables and decision context.
2. Audit the data structure. Inspect row and column counts, data types, identifiers, duplicate records, ranges, units and coding conventions.
3. Assess data quality. Examine missingness, impossible values, inconsistent categories, suspicious timestamps and potential measurement error.
4. Summarise individual variables. Use appropriate statistics and distribution plots.
5. Compare variables and groups. Use cross-tabulations, conditional summaries and bivariate plots.
6. Investigate associations. Examine correlation, dependence, interactions and possible confounding.
7. Diagnose unusual observations. Determine whether an outlier is an error, a valid extreme case or evidence of a different subgroup.
8. Document findings and limitations. Record transformations, exclusions, visual choices and unresolved uncertainty.

EDA should generally be performed on training data separately from validation and test data when predictive modelling is planned. Computing target-dependent summaries using the full dataset can create information leakage. Similarly, transformations based on means, medians or quantiles should be estimated on the training set and then applied to later datasets.

2. Variable Types and Measurement Scales

The measurement scale determines which summaries and visualisations are legitimate.

An integer-coded category is not automatically quantitative. For example, education levels coded 1, 2 and 3 should not be averaged as if the difference between levels were necessarily equal. Conversely, a binary variable may be represented numerically for modelling, while remaining conceptually categorical during EDA.

The unit of analysis must also be explicit. A dataset containing several transactions per customer is not equivalent to a dataset containing one row per customer. Treating transaction rows as independent customers can produce distorted averages and artificially narrow uncertainty estimates.

3. Descriptive Statistics

3.1 Measures of centre

For observations x₁, x₂, …, xₙ, the arithmetic mean is:

Here, x̄ is the sample mean, xᵢ is observation i, and n is the number of non-missing observations. The mean is expressed in the same units as the variable. It is sensitive to extreme values and is most informative when the distribution is reasonably symmetric and the measurement scale supports arithmetic operations.

The median is the middle observation after sorting the data. With an odd number of observations it is the central value; with an even number it is commonly the average of the two central values. The median is resistant to extreme observations and is often preferable for income, response time and transaction-value data.

The mode is the most frequent value or category. It is particularly useful for categorical variables, although a distribution may have more than one mode.

3.2 Measures of spread

The range is the maximum minus the minimum. It is easy to interpret but depends entirely on two observations. The interquartile range (IQR) is:

where Q₁ is the first quartile and Q₃ is the third quartile. The IQR contains the middle 50% of observations and is robust to extreme values.

The sample variance is:

The sample standard deviation is s = √s². The denominator n − 1 is used when estimating population variability from a sample and reflects the loss of one degree of freedom after estimating the mean. If the complete population is being described rather than a sample-based estimate, the population variance uses denominator N, the population size.

Standard deviation is in the original units, whereas variance is in squared units. Standard deviation is therefore usually easier to communicate. It is sensitive to skewness and outliers, so it should often be reported alongside the median and IQR.

The coefficient of variation is:

It is dimensionless and can compare relative variability across positive variables measured on different scales. It is inappropriate when the mean is zero, close to zero, or not meaningful; it is also problematic when values can be negative.

3.3 A worked example

Suppose five delivery times, in minutes, are 8, 10, 10, 12 and 25. The mean is 13 minutes. The median is 10 minutes, showing that the mean is pulled upward by the 25-minute delivery. Using the ordered data, Q₁ is commonly estimated as 9 and Q₃ as 18.5 under one standard percentile convention, giving IQR = 9.5 minutes. Quartile algorithms differ slightly, especially in small samples; the chosen convention should be documented.

The practical conclusion is not simply that “the average delivery time is 13 minutes.” A fuller statement is that the typical delivery is about 10 minutes, while a long-delay subgroup increases the mean. The analyst should investigate whether the 25-minute observation reflects traffic, a data-entry error, a special route or a process failure.

4. Distribution Analysis

A distribution describes how values are positioned across their possible range. Important features include centre, spread, shape, modality, skewness, tail behaviour, gaps and unusual observations.

4.1 Histograms and density plots

A histogram divides a quantitative axis into bins and displays the number or proportion of observations in each bin. The bin width affects interpretation: very narrow bins may show noise, while very wide bins can conceal meaningful structure. A density plot provides a smoothed estimate of distribution shape, but its appearance depends on the bandwidth. Density estimates should not be treated as direct counts.

For heavily skewed positive variables, a logarithmic transformation may reveal multiplicative structure and make groups easier to compare. However, transformed values must be clearly labelled, and interpretations must be returned to the original scale when communicating practical effects.

4.2 Skewness and heavy tails

Right skewness occurs when a relatively small number of large values create a long right tail. In such a distribution, the mean is often greater than the median. Left skewness has a long lower tail. Heavy-tailed data contain more extreme observations than would be expected under a light-tailed reference distribution.

Skewness is not automatically a data problem. It may be an important feature of the phenomenon. For example, healthcare expenditure and insurance losses are naturally right-skewed. Replacing all extreme values with a cap may erase important risk information.

4.3 Quantiles and empirical cumulative distributions

The pth quantile is a value below which approximately p × 100% of observations fall, where 0 ≤ p ≤ 1. Quantiles support statements such as “90% of cases were completed within 48 hours.” An empirical cumulative distribution function (ECDF) displays the proportion of observations less than or equal to each value and is useful for comparing distributions without choosing histogram bins.

4.4 Outliers

A common exploratory rule identifies values below Q₁ − 1.5 × IQR or above Q₃ + 1.5 × IQR as potential outliers. This is a screening convention, not a universal definition of error. In a normal distribution, approximately 0.7% of observations would fall beyond these fences, but the percentage differs for skewed or heavy-tailed distributions.

Outlier handling should follow investigation:

• Correct a value if a verified recording or unit error exists.
• Retain a valid extreme case when it represents the target population.
• Use robust summaries or transformations when appropriate.
• Conduct sensitivity analysis with and without influential observations.
• Avoid deleting observations solely because they disagree with a preferred narrative.

5. Missingness and Data Quality as EDA Findings

Missing values should be analysed as data, not merely removed. Compute missing counts and percentages by variable and, where relevant, by time, location, group or outcome. A missingness indicator may reveal operational processes: for example, a laboratory test may be missing because clinicians did not consider it necessary.

Three conceptual mechanisms are commonly distinguished:

• MCAR (Missing Completely At Random): missingness is unrelated to observed and unobserved values.
• MAR (Missing At Random): conditional on observed variables, missingness does not depend on the missing value itself.
• MNAR (Missing Not At Random): missingness depends on the unobserved value even after accounting for observed variables.

These are assumptions about a data-generating process, not properties that can usually be proven from the observed dataset alone. Missingness maps, grouped missingness rates and pattern tables can provide evidence relevant to the assumptions. Imputation, complete-case analysis or modelling missingness should be selected in relation to the analytical objective and the plausibility of these mechanisms.

6. Cross-Tabulation and Group Comparison

A cross-tabulation summarises the joint frequencies of two categorical variables. Let nᵢⱼ be the count in row category i and column category j. A row percentage is:

A column percentage is:

The denominator must be stated because row and column percentages answer different questions. For example, row percentages may show the composition of outcomes within each treatment group, whereas column percentages may show the composition of treatment among outcome groups.

For a categorical outcome, grouped bar charts, stacked bars and mosaic plots can complement a cross-tabulation. Stacked bars are difficult to compare when internal segments do not share a baseline; grouped bars or dot plots may be more readable.

For numerical outcomes, report group-specific counts, centre and spread. A mean difference can be informative, but distributions should also be visualised. Two groups may have equal means but radically different variability, skewness or multimodality. Small groups require cautious interpretation because estimates are unstable.

A cross-tabulation can also be affected by confounding. An overall association may reverse or disappear after stratifying by a third variable. This is a form of aggregation bias often illustrated by Simpson’s paradox. Stratification does not automatically solve confounding, but it prompts the analyst to ask whether group composition differs in a substantively important way.

7. Correlation and Association

Covariance measures whether two quantitative variables tend to vary together:

The sample Pearson correlation is:

where sₓ and sᵧ are sample standard deviations. Pearson’s r is unitless and lies between −1 and +1. Positive values indicate an increasing linear association; negative values indicate a decreasing linear association; values near zero indicate weak linear association, not necessarily independence.

Pearson correlation is most interpretable when observations are appropriately paired, variables are quantitative, the relationship is approximately linear and extreme influential points are investigated. It can be misleading for curved relationships, mixtures of subgroups, restricted ranges and strong outliers.

Spearman’s rank correlation is the Pearson correlation calculated on ranks. It assesses monotonic association and is less dependent on the original measurement scale. It is not a general solution to outliers or non-independence, and ties require appropriate rank handling.

Correlation does not imply causation. A relationship may arise from reverse causality, confounding, selection effects, common time trends or chance. A strong correlation can also coexist with poor predictive usefulness if the relationship is unstable or if the variables are measured after the outcome.

A correlation matrix is useful for screening quantitative relationships, but it should not replace pair plots, domain reasoning or investigation of subgroup structure. For many variables, a visually striking matrix can encourage multiple-comparison fishing. Exploratory findings should be confirmed with appropriate validation or pre-specified analysis.

8. Visualisation for Analysis and Communication

A chart should have an analytical purpose. Common choices include:

A scatter plot should normally show the raw observations, meaningful axes, units and an adequate sample description. A fitted line may summarise an association, but it should not imply a causal mechanism. For dense data, transparency, jittering, hexagonal binning or aggregation may improve visibility without inventing precision.

Box plots show the median, quartiles and selected whisker convention; they do not show the full distribution. Violin plots show a smoothed density and can imply detail unsupported by small samples. Combining a box plot with jittered observations is often useful.

Time-series charts must respect temporal order. Analysts should investigate seasonality, calendar effects, structural breaks, changing measurement procedures and autocorrelation. Randomly shuffling time observations before visual inspection can conceal important operational patterns.

9. Visual Diagnostics and Model Readiness

EDA supports, but does not replace, formal diagnostics. Before fitting a regression model, an analyst may examine:

• outcome and predictor distributions;
• scatter plots for linearity;
• residual-versus-fitted plots for non-linearity and unequal variance;
• Q–Q plots for residual-tail behaviour;
• leverage and influence measures for observations that disproportionately affect estimates;
• group and time plots for dependence and structural change.

A normal Q–Q plot compares ordered observations with theoretical normal quantiles. Systematic curvature suggests departure from normality. In regression, normally distributed residuals are not required for unbiased coefficient estimates under all conditions, but normality can matter for small-sample interval and test procedures. Independence, correct functional form and appropriate variance assumptions are often more consequential.

The phrase “the data are normal” is imprecise. A variable can be non-normal while a sampling distribution or residual distribution is approximately normal. Analysts should identify exactly which quantity is being assessed and for what purpose.

10. Accessible and Honest Visual Design

Accessibility is part of analytical quality. Use sufficient contrast, readable text, direct labels and patterns or shapes in addition to colour. Red–green-only encodings exclude many colour-vision-deficient readers. A chart should remain interpretable when printed in grayscale or viewed on a small screen.

Axis choices must be transparent. Truncating a bar chart’s zero baseline can exaggerate differences because bar length is interpreted proportionally. For point estimates, a non-zero axis may be reasonable if clearly labelled, but the visual effect should be considered. Dual axes can create false visual relationships and should generally be avoided unless scales and encodings are exceptionally clear.

Avoid three-dimensional bars, decorative effects, excessive gridlines, unexplained abbreviations and unnecessary decimal places. Report denominators, uncertainty and sample size where these affect interpretation. Percentages without counts can conceal that a seemingly large difference is based on very few observations.

Uncertainty may be communicated through confidence intervals, prediction intervals, bootstrap intervals or credible intervals, depending on the inferential framework. An interval is not automatically a probability statement about a fixed parameter; its interpretation depends on the method and assumptions used.

11. Reproducibility, Ethics and Governance

A professional EDA process should preserve the raw data, record cleaning rules, version code and software environments, and generate figures reproducibly. Every exclusion, recoding, transformation and aggregation should be documented. Random operations should use a recorded seed where reproducibility is required.

Privacy risks arise when visualisations expose small cells, rare combinations or geographic locations. Suppression, aggregation, perturbation or access controls may be necessary. Analysts should avoid presenting sensitive attributes in ways that encourage stereotyping or discriminatory decisions. A chart can be technically accurate yet ethically harmful if it frames a vulnerable group as a problem without context.

Fairness analysis should examine whether missingness, measurement error, sampling and subgroup representation differ across protected or operationally important groups. Aggregate performance or descriptive statistics can conceal disparate experiences. Analysts must distinguish a group difference from evidence of individual deficiency and should involve domain experts and affected stakeholders when consequences are material.

12. Contemporary Tools and AI-Assisted EDA

Notebook environments, SQL, spreadsheet tools, statistical packages and dashboard platforms can support EDA. SQL is valuable for validating counts and aggregations close to the data source; programming languages support reproducible transformations and automated tests; dashboards support monitored, repeated exploration.

AI-assisted tools can propose charts, generate code or identify unusual patterns. Their outputs require verification because automated systems may choose inappropriate aggregation levels, misinterpret data types, leak sensitive information or produce plausible but incorrect explanations. A responsible workflow treats AI suggestions as hypotheses, checks calculations independently, inspects the underlying records and documents tool use.

Automated anomaly detection should not be confused with anomaly interpretation. A rare observation may be a system error, a legitimate event or a previously unseen subgroup. Human and domain review remains essential.

13. Applied Case: Exploring Online Retail Returns

Background

The following is an illustrative case. An online retailer wants to reduce product returns without worsening customer satisfaction. The dataset contains order identifier, customer segment, product category, price, delivery time, rating, return indicator, purchase date and fulfilment centre.

Problem

Management initially asks for “the variables most correlated with returns.” This request is too narrow. Returns are binary, product categories differ in price and expectations, and multiple orders may belong to the same customer. The analytical team first defines the unit of analysis as an order and identifies the observation period and eligible orders.

Analysis

The audit finds duplicate order identifiers, missing ratings concentrated among newly introduced products, and delivery time recorded in both hours and days by different fulfilment centres. The team standardises units, retains a raw copy, and flags unresolved records.

A bar chart shows that the return proportion is highest in apparel and lowest in household supplies. However, the category counts differ substantially, so the chart includes counts and percentages. A cross-tabulation by category and customer segment shows that premium customers purchase more apparel and have a different return rate. This indicates that the aggregate category comparison may partly reflect customer mix.

Histograms reveal a right-skewed price distribution and a small group of very long deliveries. A box plot by fulfilment centre identifies one centre with an unusual upper tail. Investigation finds that a system migration recorded weekend delivery intervals incorrectly. These records are corrected where traceable and excluded from a sensitivity analysis where correction is impossible.

A scatter plot of delivery time and price, coloured by return status, shows that long delivery times are associated with returns, but the relationship is not linear and is concentrated in apparel. Pearson correlation is therefore not treated as a sufficient summary. The team uses stratified plots and later considers an interaction between category and delivery performance in a predictive model.

Approach and outcome

The team produces a reproducible EDA report containing data-quality tables, missingness patterns, distribution plots, stratified return rates and documented decisions. It recommends improving delivery-event capture and collecting ratings through a mechanism that distinguishes “not requested” from “customer declined.” A pilot intervention targets delayed apparel deliveries rather than applying a universal policy.

The analysis does not claim that faster delivery causes every return to disappear. It identifies a plausible operational target, clarifies data limitations and proposes a testable intervention. Follow-up evaluation uses a time-based holdout and monitors return rate, customer satisfaction and subgroup effects.

Learning points

1. Data auditing can change the analytical question.
2. Aggregated patterns may conceal subgroup composition effects.
3. Correlation is not a substitute for distributional and causal reasoning.
4. Outliers often reveal process or measurement problems.
5. EDA should lead to decisions that can be tested, not unsupported certainty.

14. Common Failure Modes and Best Practices

Failure: treating every numeric column as continuous. Best practice: inspect meaning, coding and measurement scale.

Failure: reporting only the mean. Best practice: pair centre with spread, distributional displays and sample size.

Failure: deleting outliers automatically. Best practice: verify, classify and perform sensitivity analysis.

Failure: using percentages without denominators. Best practice: show counts and define the denominator.

Failure: interpreting correlation causally. Best practice: consider design, confounding, temporal order and alternative explanations.

Failure: allowing software defaults to determine the story. Best practice: review bin widths, scales, missing-value handling and aggregation.

Failure: producing attractive but inaccessible charts. Best practice: use clear labels, contrast, non-colour cues and direct annotations.

Failure: exploring the test set repeatedly. Best practice: separate exploration, model selection and final evaluation.

Chapter Summary

Exploratory Data Analysis is an iterative discipline for understanding data quality, structure, distributions, associations and anomalies before formal modelling. Descriptive statistics should be selected according to variable type and distribution; means and standard deviations are sensitive to extremes, whereas medians and IQRs are more robust. Histograms, ECDFs, box plots and density plots reveal shape, skewness, tails and potential outliers.

Cross-tabulations require explicit denominators and should often be stratified to investigate composition and confounding. Covariance and Pearson correlation describe linear association, while rank correlation addresses monotonic association; neither establishes causality or guarantees independence. Visual diagnostics can expose non-linearity, unequal variance, dependence and influential observations.

Effective visualisation is accurate, accessible, purposeful and transparent about uncertainty. Ethical EDA protects privacy, considers subgroup effects and documents decisions. Reproducible workflows, domain review and cautious use of AI-assisted tools help transform exploratory patterns into defensible analytical decisions.

Key Terms

Exploratory Data Analysis: Iterative examination of data using summaries, visualisations and questions.

Unit of analysis: Entity represented by one observation or row.

Mean: Arithmetic average of quantitative observations.

Median: Middle value of ordered observations.

Variance: Average squared deviation, using an appropriate population or sample denominator.

Standard deviation: Square root of variance, expressed in original units.

Interquartile range: Difference between the third and first quartiles.

Quantile: Value dividing ordered data at a specified cumulative proportion.

Skewness: Asymmetry in a distribution.

Outlier: Observation unusually distant from the main pattern; not necessarily an error.

Histogram: Binned display of a quantitative distribution.

ECDF: Empirical cumulative distribution function showing cumulative proportions.

Cross-tabulation: Joint frequency or percentage table for categorical variables.

Covariance: Unstandardised measure of joint variation between two variables.

Correlation: Standardised measure of association, commonly bounded between −1 and +1.

Confounding: Distortion of an association by another variable related to the variables studied.

Missingness mechanism: Process governing why values are absent, such as MCAR, MAR or MNAR.

Overplotting: Visual obstruction caused by many observations occupying similar positions.

Data leakage: Use of information unavailable at prediction time or improperly shared across evaluation stages.

Visual diagnostic: Plot used to assess structure, assumptions, anomalies or model behaviour.

Review Questions

Short Answer Questions

1. Why should the median and IQR sometimes be preferred to the mean and standard deviation?
2. What is the difference between a row percentage and a column percentage in a cross-tabulation?
3. Why does correlation not establish causation?
4. What is the purpose of a missingness map or missingness summary?
5. Identify two design practices that improve visual accessibility.

Long Answer Questions

1. Explain a complete EDA workflow for a dataset intended for predictive modelling, including safeguards against data leakage.
2. Compare histograms, density plots, ECDFs and box plots, including the strengths and limitations of each.
3. Discuss how outliers should be investigated and handled in a professional analysis.
4. Explain Pearson and Spearman correlation, their assumptions, and situations in which each may be misleading.
5. Discuss how aggregation, confounding and Simpson’s paradox can affect interpretation of group comparisons.

Analytical / Application Questions

1. A customer-spending variable has a mean of 420, a median of 180 and a small number of very large observations. Which summaries and visualisations would you use, and what interpretation would you offer?
2. A hospital reports that readmission is 12% for one ward and 8% for another. What additional counts, stratifications and visual diagnostics are needed before concluding that ward performance differs?
3. A scatter plot shows almost zero Pearson correlation between advertising expenditure and sales, but the plot appears U-shaped. Explain how you would investigate the pattern and why the correlation is inadequate.

Multiple Choice Questions with Answers

Question 1

Which statistic is generally most resistant to extreme observations?

A. Mean

B. Median

C. Variance

D. Range

Correct Answer: B. Median

Question 2

If Q₁ = 12 and Q₃ = 30, what is the interquartile range?

A. 18

B. 21

C. 42

D. 360

Correct Answer: A. 18

Question 3

A row percentage in a cross-tabulation uses which denominator?

A. The grand total only

B. The total for the relevant row

C. The total for the relevant column

D. The sample standard deviation

Correct Answer: B. The total for the relevant row

Question 4

Pearson correlation primarily measures:

A. Any causal relationship

B. Linear association between quantitative variables

C. Difference between two medians

D. The number of missing values

Correct Answer: B. Linear association between quantitative variables

Question 5

Which plot is especially useful for displaying cumulative proportions without selecting histogram bins?

A. ECDF

B. Pie chart

C. Bar chart

D. Radar chart

Correct Answer: A. ECDF

Question 6

An observation beyond a 1.5 × IQR fence should be interpreted as:

A. Automatically erroneous

B. Automatically deleted

C. A potential outlier requiring investigation

D. Proof that the variable is normally distributed

Correct Answer: C. A potential outlier requiring investigation

Question 7

Which practice most directly reduces misleading colour interpretation for colour-vision-deficient readers?

A. Use red and green as the only distinctions

B. Add labels, shapes or patterns in addition to colour

C. Remove all axis labels

D. Use three-dimensional bars

Correct Answer: B. Add labels, shapes or patterns in addition to colour

Question 8

A correlation close to zero can occur when two variables have:

A. A strong curved relationship

B. Exactly equal means only

C. No observations

D. Identical units

Correct Answer: A. A strong curved relationship

Question 9

Which missingness mechanism assumes missingness may depend on observed variables but not on the missing value after conditioning on them?

A. MCAR

B. MAR

C. MNAR

D. 完全 random

Correct Answer: B. MAR

Question 10

Why should exploratory transformations such as estimating a scaling mean be fitted on training data only in predictive modelling?

A. To increase the number of test observations

B. To prevent information leakage from evaluation data

C. To guarantee normality

D. To eliminate all outliers

Correct Answer: B. To prevent information leakage from evaluation data