8. Exploratory Data Analysis and Visualisation

Introduction and Context

Exploratory Data Analysis (EDA) is the disciplined process of examining, summarising and visualising data before formal modelling or decision-making. It is not merely the production of attractive charts. EDA combines statistical reasoning, domain knowledge and visual perception to reveal structure, identify data-quality problems, detect unusual observations, formulate hypotheses and determine which analytical methods are appropriate.

In data science, the quality of a model depends heavily on the quality of the understanding that precedes it. A predictive algorithm may produce impressive accuracy while learning from leakage, measurement errors, unrepresentative samples or hidden subgroups. EDA helps expose such problems. It also prevents analysts from treating summary statistics as complete descriptions of a dataset. Two datasets can have the same mean and variance while having very different distributions, relationships and practical implications.

Visualisation extends EDA by converting numerical patterns into perceptible forms. Effective visual design makes comparisons, trends, variability and exceptions easier to recognise. Poor design, however, can exaggerate differences, conceal uncertainty or encourage unsupported conclusions. Consequently, EDA and visualisation should be understood as analytical activities governed by statistical and professional judgement.

Learning Objectives

After studying this chapter, students should be able to:

1. Explain the purpose, scope and workflow of exploratory data analysis.

2. Select and interpret suitable descriptive statistics.

3. Analyse distributions, skewness, outliers and missingness.

4. Measure and interpret association without confusing it with causation.

5. Choose visualisations appropriate to data types and analytical questions.

6. Use EDA to generate defensible hypotheses and guide modelling.

7. Recognise common analytical, visual and ethical errors.

8. Apply EDA in professional and organisational settings.

The Logic of Exploration

EDA is usually iterative rather than linear. An analyst begins with a question and a data inventory, examines data quality, summarises individual variables, studies relationships, investigates unexpected patterns and revises the questions as evidence develops. Domain expertise is essential because an observation that is statistically unusual may be operationally normal, while a small numerical difference may be highly important in a particular context.

A useful EDA workflow contains the following activities:

• clarify the decision or research question;
• identify units of analysis, variables, time periods and sources;
• inspect data types, missing values, duplicates and coding conventions;
• summarise individual variables;
• examine relationships among variables and across groups;
• investigate anomalies and assess their plausibility;
• document findings, assumptions and unresolved uncertainty; and
• translate observations into hypotheses for confirmatory analysis or modelling.

EDA does not prove hypotheses. Its role is to make the data-generating process more intelligible and to support appropriate subsequent analysis.

Data Types and Measurement Scales

The type of variable determines which summaries and graphics are meaningful. Nominal variables represent categories without an inherent order, such as payment method or department. Ordinal variables represent ordered categories, such as satisfaction levels, although distances between categories may not be equal. Interval variables have meaningful differences but no meaningful zero, as in temperature measured in Celsius. Ratio variables possess a meaningful zero, as in revenue, duration or physical weight.

Variables may also be classified as numerical or categorical, discrete or continuous, and cross-sectional or temporal. A date is often stored technically as text but should be treated analytically as a temporal variable. Correct classification prevents errors such as calculating an average of customer-identification codes or displaying a nominal category on a misleading numerical scale.

Descriptive Statistics: Compressing Data Without Losing Meaning

A descriptive statistic is a numerical summary of observed data. The arithmetic mean is calculated as

mean = (sum of all observations) / (number of observations).

It is useful for approximately symmetric distributions but can be strongly affected by extreme values. The median is the middle ordered observation, or the average of the two middle observations when the sample size is even. It is more resistant to skewness and outliers. The mode is the most frequent value and is particularly useful for categorical data.

Measures of spread describe variability. The range is the difference between the maximum and minimum, but it is unstable because it depends on two observations. The interquartile range (IQR) is Q3 − Q1, where Q1 and Q3 are the first and third quartiles. It represents the central 50 per cent of observations and is relatively robust. Variance is the average squared deviation from the mean, while standard deviation is its square root and is expressed in the original units. For a sample, variance is commonly estimated using n − 1 in the denominator.

The coefficient of variation, standard deviation divided by the mean, can compare relative variability across measurements with different scales, provided the mean is meaningfully above zero. Percentiles are often more informative than a single average. For example, a service manager may care more about the 95th percentile of waiting time than the mean because it reflects the experience of slower customers.

A summary should normally include the number of valid observations, missing observations, an appropriate measure of centre, a measure of spread and relevant quantiles. A mean without a measure of variability is rarely sufficient for responsible interpretation.

Understanding Distributions

A distribution describes how values are arranged. Analysts should consider centre, spread, shape, modality and tails. A symmetric unimodal distribution has roughly balanced tails around one central peak. A right-skewed distribution has a long upper tail, commonly seen in income, transaction value and response time. In such cases the mean often exceeds the median. A left-skewed distribution has a longer lower tail.

A distribution may be bimodal or multimodal, indicating mixtures of populations, different operating conditions or latent groups. For example, delivery times may have one mode for local deliveries and another for remote deliveries. Aggregating the groups can conceal this structure.

Histograms divide numerical values into intervals. Their interpretation depends on bin width: very wide bins hide structure, while very narrow bins create visual noise. Density plots provide a smoothed representation but may obscure sample size and depend on a smoothing parameter. Box plots show the median, quartiles and potential extreme observations. The conventional whisker rule flags values below Q1 − 1.5(IQR) or above Q3 + 1.5(IQR), but this is a screening convention rather than proof of error.

A useful transformation can make patterns easier to interpret. Logarithmic transformations are often used for positive, strongly right-skewed variables. Standardisation converts values to comparable units, commonly through z-scores: z = (x − mean) / standard deviation. Transformations should be documented because they change interpretation and may affect subsequent models.

Missing Data and Anomalies

Missingness is not simply an inconvenience. It may occur because a measurement was unavailable, a respondent refused to answer, a system failed or a value was not applicable. Analysts should distinguish structural missingness from accidental missingness and determine whether missingness is related to observed or unobserved characteristics.

Three useful conceptual categories are missing completely at random, missing at random conditional on observed information and missing not at random. These assumptions influence whether deletion, imputation or a missingness indicator is defensible. A missingness heat map, variable-level missingness table and comparison of records with and without missing values can reveal patterns.

An anomaly is an observation that conflicts with an expected pattern. It may be a data-entry error, a sensor failure, fraud, a rare but genuine event or a change in the process. Removing anomalies automatically is poor practice. First inspect provenance, timestamps, units and related variables. Retain valid extreme cases, correct demonstrable errors, and record any exclusion with a reason.

Association and Dependence

Association describes how variables vary together. For two numerical variables, covariance indicates whether deviations tend to have the same or opposite signs, but its magnitude depends on measurement units. Pearson correlation standardises covariance and ranges from −1 to +1. It measures linear association and can be distorted by outliers or nonlinear relationships.

Spearman’s rank correlation measures monotonic association using ranks. It is often more appropriate for ordinal variables or relationships that are monotonic but not linear. A correlation near zero does not establish independence: a strong U-shaped relationship, for example, may have little linear correlation.

For categorical variables, a contingency table displays joint frequencies. The chi-square test can assess whether observed frequencies differ from those expected under independence, while Cramér’s V provides a standardised measure of association. For a categorical and numerical variable, group comparisons, box plots and effect sizes are useful. Analysts should report group sizes because an apparent difference based on a very small group may be unstable.

Association is not causation. A relationship may result from confounding, reverse direction, selection bias, common time trends or measurement processes. For example, ice-cream sales and emergency visits may rise together because both are affected by temperature. EDA can identify candidate explanations, but causal claims require an appropriate design, such as randomisation, natural experiments or carefully justified observational methods.

Visualisation as Analytical Reasoning

A visualisation should be selected according to the question, not according to the software’s default menu. For one numerical variable, use a histogram, density plot, box plot or empirical cumulative distribution. For two numerical variables, use a scatter plot, possibly with a fitted trend and transparency when points overlap. For a numerical variable across categories, use dot plots, box plots or violin plots with sample sizes where useful. For time series, use a line chart with an appropriate time scale. For categorical counts, use ordered bars rather than decorative three-dimensional graphics.

A good chart has a clear analytical purpose, an informative title, labelled axes, units, readable scales and a legend only when necessary. Colour should encode meaningful distinctions rather than decoration. Sequential palettes suit ordered magnitude; diverging palettes suit deviations around a meaningful midpoint; qualitative palettes suit unordered categories. Colour should not be the sole carrier of information because some readers have colour-vision deficiencies and charts may be printed in grayscale.

Axes deserve particular attention. Truncating a bar chart’s zero baseline can exaggerate differences because bar length implies magnitude. In line charts, a non-zero baseline may sometimes be acceptable when the focus is change over time, but the scale must not mislead. Dual axes can imply relationships that are merely coincidental and should be used sparingly.

Overplotting can be addressed through transparency, jittering, aggregation or small multiples. Faceting a chart by region or customer segment can reveal heterogeneity that an overall chart conceals. Annotation should direct attention to relevant events without replacing the underlying evidence.

From EDA to Hypotheses and Models

EDA supports model selection and feature engineering. A heavily skewed predictor may require transformation; a categorical variable may need encoding; correlated predictors may raise concerns about redundancy; a time trend may require temporal validation rather than random splitting. Grouped patterns may suggest interaction terms, while nonlinear shapes may motivate tree-based models, splines or transformations.

However, exploratory findings can become overconfident when analysts repeatedly inspect the same data until a pattern appears. This is a form of implicit multiple testing. Important patterns should be tested on new data, through cross-validation, holdout samples or a subsequent confirmatory study. Analysts should distinguish clearly between descriptive findings, exploratory hypotheses and validated conclusions.

A Practical Case: Investigating Customer Delivery Delays

Background

Consider an illustrative case involving a fictional online retailer, Northstar Market. Management reports that the average delivery time has increased and asks the analytics team to identify the cause.

Situation and Initial Inspection

The dataset contains order identifier, order date, promised date, actual delivery date, warehouse, carrier, destination region, package weight, shipping method and customer rating. Initial inspection identifies duplicate order identifiers, inconsistent date formats and a small number of negative delivery durations. Negative durations are traced to a date-parsing error rather than genuine early delivery.

The team defines delivery time as actual delivery date minus order date and separately calculates delay relative to the promised date. This distinction matters: a delivery can be slow but still meet a generous promise, or fast but late relative to a short promise.

Analysis

The overall mean delay is positive, but the median is close to zero and the distribution is strongly right-skewed. A box plot shows that a small number of very late orders affect the mean. Histograms and time-series plots reveal a second mode during a period of warehouse relocation. Faceted plots show that the apparent company-wide problem is concentrated in two warehouses and one carrier.

A contingency table of late versus on-time orders by carrier indicates an association, but carrier assignment is not random: the problematic carrier handles more remote destinations. After stratifying by destination region, the carrier difference becomes smaller, suggesting confounding. A scatter plot also shows that heavy packages have longer delays, although the relationship is nonlinear.

Approach and Solution

The team recommends separate operational analyses for warehouse processing time, carrier transit time and destination effects. It creates a dashboard containing median delay, 90th and 95th percentiles, late-delivery rate and order volume by warehouse and region. Rather than deleting extreme delays, the team verifies them against shipment records and retains genuine events. A future predictive model will use a time-based validation split to avoid learning from future logistics conditions.

Outcome

Management learns that the mean alone overstated the typical customer experience but understated a serious tail-risk problem. Corrective action is directed toward the affected warehouses and remote-route capacity instead of imposing a uniform change on all carriers.

Learning Points

The case demonstrates the importance of defining measures precisely, examining distributions rather than relying on averages, segmenting aggregated results, investigating anomalies, distinguishing association from causation and reporting tail behaviour. It also shows how EDA can improve managerial decisions before any sophisticated machine-learning model is deployed.

Applications in Organisations and Industry

EDA is used in customer analytics to examine churn, purchase frequency, satisfaction and campaign response. In finance, it supports fraud screening, portfolio analysis and risk monitoring, where extreme values and changing distributions are especially important. In manufacturing, control charts, sensor distributions and multivariate plots help detect process drift. In healthcare, exploratory analysis can reveal subgroup differences, missingness patterns and possible measurement bias, although privacy and clinical validation are essential. In human resources, visualisations may expose disparities in recruitment or promotion, but analysts must consider lawful purpose, fairness and the risk of reproducing historical discrimination.

Modern tools such as Python libraries, R packages, SQL, notebook environments and business-intelligence platforms make it possible to automate profiling, interactive filtering and dashboard production. Automated EDA can rapidly identify duplicated fields, type errors, missingness and unusual distributions. It cannot replace domain interpretation. AI-assisted analytics may suggest charts or narratives, but every generated claim requires verification against the data, definitions and sampling process.

Challenges, Risks and Professional Judgement

Common mistakes include treating missing values as zero, comparing groups without considering sample size, using inappropriate averages, removing outliers without investigation, interpreting correlation causally, ignoring selection bias and creating charts with distorted scales. Another risk is Simpson’s paradox, in which an aggregate association reverses or disappears after data are divided into meaningful groups.

Data leakage is especially damaging in predictive projects. If EDA uses information that would not be available at prediction time, it can lead to unrealistic expectations. Repeated exploratory analysis can also produce chance findings. Analysts should maintain a data dictionary, record transformations, preserve a reproducible workflow and separate exploratory from confirmatory datasets when feasible.

Ethically, visualisation must respect privacy and avoid exposing individuals through small-cell reporting or re-identification. Sensitive attributes should be handled according to applicable law and organisational policy. Fairness assessment should examine performance and outcomes across relevant groups, while recognising that group definitions and measurements may themselves be imperfect. A defensible analysis communicates uncertainty, limitations and alternative explanations rather than presenting a visually persuasive story as fact.

Current Developments and Best Practices

Contemporary EDA increasingly involves high-dimensional data, streaming data, geospatial information and interactive dashboards. Dimensionality-reduction methods such as principal component analysis and nonlinear embeddings can support exploration, but their visual distances should not be interpreted as definitive evidence of natural clusters. For streaming systems, distribution monitoring and drift detection are used to identify changes in data-generating processes. Reproducible notebooks, versioned datasets and automated data-quality tests connect EDA with production analytics.

Best practice is to begin with the decision context, inspect data provenance, combine robust numerical summaries with suitable graphics, examine subgroup variation, investigate unusual observations, document every transformation and communicate uncertainty. A strong EDA report should enable another analyst to reproduce the findings and understand what the data cannot establish.

Chapter Summary

Exploratory Data Analysis is an iterative method for understanding data before formal inference or modelling. It combines data-quality inspection, descriptive statistics, distribution analysis, association measures and visual reasoning. Means, medians, quantiles and measures of spread must be chosen according to distributional shape and decision context. Missingness and anomalies require investigation rather than automatic deletion. Correlation and other association measures identify relationships but do not prove causation. Effective visualisations match chart type to analytical purpose, use honest scales and communicate uncertainty clearly. EDA supports feature engineering, model selection, operational diagnosis and hypothesis generation, while reproducibility, privacy, fairness and validation protect against misleading conclusions.

Key Terms

Descriptive statistics: Numerical summaries that describe observed data.

Exploratory Data Analysis: Iterative examination of data to discover structure, errors, relationships and hypotheses.

Mean: Arithmetic average of numerical observations.

Median: Middle value of an ordered dataset.

Quantile: A value below which a specified proportion of observations falls.

Interquartile range: Difference between the third and first quartiles.

Standard deviation: Square root of variance, expressing typical dispersion in original units.

Skewness: Asymmetry in the shape of a distribution.

Outlier: An observation unusually distant from the pattern of the remaining data.

Missingness: The condition in which a variable value is unavailable or unrecorded.

Histogram: A chart showing frequencies across numerical intervals.

Box plot: A graphic summarising quartiles, median and potential extreme values.

Correlation: A standardised measure of association, commonly for numerical variables.

Confounding: A situation in which another variable influences an observed association.

Contingency table: Table displaying joint frequencies of categorical variables.

Overplotting: Visual overlap that makes individual observations difficult to distinguish.

Data leakage: Use of information unavailable at the time a prediction would be made.

Small multiples: Repeated comparable charts divided by groups or conditions.

Hypothesis generation: Development of propositions for later testing based on exploratory evidence.

Reproducibility: Ability to recreate an analysis from documented data and procedures.

Review Questions

Short Answer Questions

1. Why is the median often preferable to the mean for a strongly right-skewed variable?

2. What information does the interquartile range provide?

3. Distinguish an anomaly from a confirmed data error.

4. Why does a correlation near zero not necessarily imply independence?

5. State two principles of honest visual design.

Long Answer Questions

1. Explain a complete EDA workflow from defining the analytical question to producing hypotheses for modelling.

2. Compare histograms, density plots and box plots, including their strengths and limitations.

3. Discuss how missing-data mechanisms influence analytical decisions.

4. Explain why association cannot be interpreted automatically as causation, using an example.

5. Describe how EDA can guide feature engineering, model selection and validation.

Analytical / Application Questions

1. A company reports an increase in average response time, but the median is unchanged and the 95th percentile has risen sharply. What analyses would you conduct, and what operational interpretation might follow?

2. A scatter plot shows a strong relationship between advertising expenditure and sales. Identify at least three alternative explanations and propose an EDA strategy to investigate them.

3. Design a visual and statistical analysis for comparing late-delivery rates across carriers while accounting for region, package weight and warehouse.

22. Multiple Choice Questions with Answers

Question 1

A delivery-time variable has a long right tail caused by a small number of very late orders. Which summary is generally most resistant to the influence of those orders?

A. Mean

B. Median

C. Range

D. Variance

Correct Answer: B. Median

Question 2

A correlation close to zero is observed between two variables, but a scatter plot shows a clear U-shaped pattern. What is the best interpretation?

A. The variables are independent

B. There is no relationship of any kind

C. The linear association is weak, but a nonlinear relationship exists

D. The correlation calculation must be positive

Correct Answer: C. The linear association is weak, but a nonlinear relationship exists

Question 3

Which visualisation is most suitable for examining the distribution of a numerical variable across several categorical groups?

A. A box plot grouped by category

B. A pie chart of the numerical values

C. A three-dimensional surface chart

D. A word cloud

Correct Answer: A. A box plot grouped by category

Question 4

An extreme observation in a dataset should normally be removed when it is:

A. Larger than the mean

B. Outside the central 50 per cent

C. A genuine rare event

D. Verified as a data-entry or measurement error

Correct Answer: D. Verified as a data-entry or measurement error

Question 5

A carrier appears associated with late deliveries, but it handles mainly remote destinations. Which issue should the analyst investigate first?

A. Confounding by destination region

B. Whether the chart uses a colour palette

C. Whether the median equals the mode

D. Whether all variables are normally distributed

Correct Answer: A. Confounding by destination region

Question 6

Which chart design is most likely to exaggerate differences in bar heights?

A. Ordering bars from largest to smallest

B. Using a labelled horizontal axis

C. Truncating the value axis above zero

D. Displaying category names clearly

Correct Answer: C. Truncating the value axis above zero

Question 7

What is the principal purpose of exploratory data analysis before predictive modelling?

A. To guarantee causal conclusions

B. To discover structure, quality problems and modelling implications

C. To replace the need for validation

D. To maximise the number of visualisations

Correct Answer: B. To discover structure, quality problems and modelling implications

Question 8

If missing values occur more frequently for a particular customer segment, treating them as completely random may:

A. Remove all sampling bias automatically

B. Create a more representative dataset by definition

C. Hide systematic information about the data-collection process

D. Guarantee unbiased estimates

Correct Answer: C. Hide systematic information about the data-collection process

Question 9

Which measure is especially appropriate for assessing association between two ordinal variables?

A. Spearman’s rank correlation

B. The raw range

C. The arithmetic mean of category labels

D. The maximum value

Correct Answer: A. Spearman’s rank correlation

Question 10

A data scientist repeatedly searches a dataset until finding a statistically interesting pattern and reports it without new-data validation. What is the main risk?

A. Underplotting

B. Chance discovery and overconfident inference

C. Excessive standardisation

D. Loss of all descriptive information

Correct Answer: B. Chance discovery and overconfident inference