11. Visualisation, Statistics and Exploratory Analytics

Introduction and Context

Artificial intelligence and machine learning projects rarely fail because a model cannot be trained. They often fail because the data was misunderstood, the target variable was poorly defined, an important subgroup was overlooked, or a result was communicated without sufficient evidence. Visualisation, descriptive statistics, and exploratory analytics provide the first disciplined opportunity to identify such problems.

Exploratory data analysis (EDA) is an investigative process. It combines numerical summaries, graphical techniques, sampling, and domain reasoning to understand the structure and quality of data before formal modelling. In Python, this work is supported by NumPy, pandas, Matplotlib, Seaborn, SciPy, and interactive tools such as Plotly. These libraries do not replace statistical reasoning. They make reasoning faster, more transparent, and more reproducible.

For postgraduate AI and ML practitioners, visualisation has two audiences. The first is the analyst, who needs to discover distributions, anomalies, dependencies, and uncertainty. The second is the stakeholder, who needs an accurate and relevant explanation of what the data suggests and what it does not establish. A technically attractive chart can still be misleading if its scales, aggregation, sample, or uncertainty are poorly chosen.

Learning Objectives

After completing this chapter, students should be able to:

• distinguish descriptive statistics, exploratory analysis, and inferential claims;
• select appropriate Matplotlib, Seaborn, and interactive visualisations;
• analyse distributions, relationships, missingness, outliers, and subgroup differences;
• use sampling and bootstrap methods to communicate uncertainty;
• formulate and investigate hypothesis-oriented questions without confusing association with causation;
• design reproducible, accessible, and stakeholder-focused analytical reports;
• identify statistical, ethical, and communication risks in exploratory analytics.

A Statistical View of Data Exploration

A dataset can be viewed as a collection of observations measured on variables. Variables may be numerical, categorical, ordinal, temporal, spatial, or textual. Their measurement scale affects the operations and visualisations that are meaningful. For example, calculating an average of income is usually interpretable, whereas averaging postal codes is not.

Descriptive statistics summarise observed data. For a numerical variable, location may be described by the mean or median, while spread may be described by the range, variance, standard deviation, interquartile range (IQR), or median absolute deviation. Shape is reflected in skewness, tails, multimodality, and gaps. For categorical data, counts and proportions are generally more useful than numerical moments.

The arithmetic mean is

x̄ = (1/n) Σ xi

and the sample variance is

s² = [1/(n − 1)] Σ(xi − x̄)².

The median is less sensitive to extreme values than the mean. The IQR, defined as Q3 − Q1, describes the central half of observations and is therefore useful for skewed data. These measures are complementary rather than competing: a large difference between mean and median may indicate skewness, unusual observations, or a mixture of populations.

A distribution should not be reduced to a single number. A histogram displays binned frequencies, while a density plot provides a smoothed approximation of a distribution. Histograms depend on bin width, and density plots depend on a smoothing parameter; neither should be interpreted as an exact representation of an underlying population. Box plots show median, quartiles, and selected potential outliers. Violin plots combine density and summary information, but their shape can suggest more precision than the sample supports, especially for small groups.

A useful initial statistical audit includes:

• number of rows and columns;
• data types and measurement units;
• missing-value counts and patterns;
• duplicate records;
• unique category counts;
• minimum, maximum, quartiles, and plausible ranges;
• class balance for a supervised learning target;
• time coverage and possible changes in data collection.

A summary is meaningful only when its denominator and population are clear. “Average customer spend was 42” is incomplete without knowing the currency, period, customer definition, treatment of refunds, and whether the average is weighted.

A Reproducible EDA Workflow

EDA is not a rigid sequence, but a defensible workflow usually moves from data familiarisation to focused investigation. First, establish provenance: identify the source, collection process, observation unit, time period, and known limitations. Next, inspect schema and quality. Then examine individual variables, relationships between variables, and differences among important subgroups. Finally, formulate questions for confirmatory analysis or modelling.

The following compact example uses a public-style customer dataset represented by a pandas DataFrame named `df`:

```python

import pandas as pd

import matplotlib.pyplot as plt

import seaborn as sns

sns.set_theme(style="whitegrid", context="talk")

print(df.shape)

print(df.dtypes)

print(df.isna().mean().sort_values(ascending=False).head())

print(df.describe(include="all").T)

fig, ax = plt.subplots(figsize=(8, 5))

sns.histplot(data=df, x="annual_spend", bins=30, kde=True, ax=ax)

ax.set(title="Distribution of annual spend", xlabel="Annual spend", ylabel="Customers")

plt.tight_layout()

plt.show()

```

A reproducible analysis fixes random seeds where sampling or simulation is used, records software versions, retains the code that generated each figure, and uses meaningful labels. Exploratory notebooks are useful for discovery, but important results should eventually be converted into scripts, reports, or pipelines that can be rerun.

Univariate, Bivariate, and Multivariate Analysis

Univariate analysis examines one variable. For a continuous measure, combine a numerical summary with a distribution plot. For a categorical variable, use counts or proportions, preferably ordered by frequency or by a meaningful domain order. A bar chart is usually preferable to a pie chart because comparisons of lengths are easier than comparisons of angles.

Bivariate analysis examines a pair of variables. A scatter plot is appropriate for two numerical variables; transparency (`alpha`) and a sampling strategy can help when points overlap. A box plot or violin plot can compare a numerical variable across categories. A contingency table and a normalised heatmap can examine two categorical variables. For time-indexed data, line charts should preserve chronological order and make the time interval explicit.

Multivariate analysis adds conditioning variables, colour, shape, facets, or small multiples. These devices can expose patterns that disappear in aggregate data. However, encoding too many variables in one chart increases cognitive load. Several simple, consistently scaled panels are often more informative than one overloaded graphic.

Correlation measures linear association, not general dependence and certainly not causation. Pearson correlation is sensitive to outliers and is appropriate for approximately linear relationships. Spearman correlation uses ranks and can detect monotonic relationships that are not linear. A correlation matrix is useful for screening, but it may hide subgroups, nonlinear structure, confounding, and temporal dependence.

```python

fig, axes = plt.subplots(1, 2, figsize=(14, 5))

sns.scatterplot(data=df, x="age", y="annual_spend", hue="membership", alpha=.6, ax=axes[0])

sns.boxplot(data=df, x="membership", y="annual_spend", ax=axes[1])

axes[0].set_title("Spend and age by membership")

axes[1].set_title("Group-wise spend distribution")

plt.tight_layout()

```

Visual Grammar and Tool Selection

Matplotlib provides fine-grained control over figures, axes, annotations, scales, legends, and output formats. It is especially valuable when a figure must meet publication, reporting, or branding requirements. Seaborn builds statistical graphics on Matplotlib and offers concise functions for distributions, categorical comparisons, regression displays, and relational plots. Its semantic arguments—such as `hue`, `style`, and `size`—make group structure explicit.

Plotly and related interactive libraries support hover information, zooming, filtering, and linked exploration. Interactivity is particularly valuable when many observations overlap or when a stakeholder needs to inspect individual records. It should not be used to hide an unclear message. A static summary remains important for documents, accessibility, archiving, and reproducibility.

Chart selection should follow the analytical question:

• distribution: histogram, empirical cumulative distribution function, box plot, or violin plot;
• comparison: ordered bar chart, dot plot, box plot, or interval plot;
• relationship: scatter plot, hexbin plot, or line chart;
• composition: stacked bar chart or area chart when categories and totals are meaningful;
• time: line chart, calendar view, or rolling summary;
• geographic pattern: map only when spatial position is analytically relevant.

Axes must begin at zero when the visual comparison depends on bar lengths, although other charts may use a meaningful nonzero scale if clearly labelled. Truncated axes can exaggerate small differences. Three-dimensional charts, decorative effects, excessive colours, and dual axes should be used rarely because they often impair accurate comparison.

Colour requires special care. A sequential palette represents ordered magnitude, a diverging palette represents deviation around a meaningful midpoint, and a qualitative palette distinguishes categories. Colour should not be the only channel used to encode an important distinction; redundant labels, shapes, or line styles improve accessibility. Figures should remain interpretable for readers with colour-vision deficiencies and should include descriptive titles, axis units, and alt text where appropriate.

Sampling and Uncertainty

A dataset may be a census of a defined process, a sample from a larger population, or a convenience collection produced by an application system. The distinction determines what can reasonably be generalised. Random sampling gives each eligible unit a known selection mechanism; stratified sampling ensures representation of important groups; cluster sampling selects naturally occurring groups; convenience sampling is easy but vulnerable to selection bias.

Sampling error is variation caused by observing a sample rather than the entire target population. Bias is systematic distortion arising from selection, measurement, nonresponse, coverage, or processing. Increasing sample size can reduce random error but does not automatically correct bias. A very large dataset collected from the wrong population may still support poor conclusions.

A confidence interval describes uncertainty under a specified procedure and model. It is not a probability statement that a fixed parameter lies in a particular realised interval. In exploratory work, interval estimates are often more informative than isolated point estimates. Error bars should be labelled: they might represent standard deviation, standard error, a confidence interval, or a percentile interval, which are not interchangeable.

The bootstrap estimates sampling variability by repeatedly resampling observations with replacement. For a statistic such as the median, a basic implementation is:

```python

import numpy as np

rng = np.random.default_rng(7)

values = df["annual_spend"].dropna().to_numpy()

boot_medians = np.array([

np.median(rng.choice(values, size=len(values), replace=True))

for _ in range(5000)

])

interval = np.quantile(boot_medians, [0.025, 0.975])

print(np.median(values), interval)

```

Bootstrap results are not automatically valid. Dependence between observations, clustered data, time series, severe missingness, and a biased sample require appropriate resampling designs or modelling assumptions. In a time series, randomly shuffling individual observations can destroy temporal structure; block bootstrap or time-aware validation may be more suitable.

Hypothesis-Oriented Exploration

Exploration often begins with a question such as: “Do customers receiving a reminder renew at a higher rate?” This is more useful than searching for any statistically significant pattern. A hypothesis-oriented workflow identifies the unit of analysis, outcome, comparison groups, plausible confounders, and practical effect size before examining results.

A null hypothesis typically represents no difference or no association under a specified model. A p-value measures how compatible the observed data are with the null model and assumptions; it does not measure the probability that the null hypothesis is true, nor does it measure practical importance. Statistical significance can result from a tiny effect in a very large sample. Conversely, a meaningful effect may be uncertain in a small sample.

When comparing two groups, report the group estimates, their difference, an interval estimate, sample sizes, and an appropriate effect measure. For binary outcomes, risk difference, relative risk, and odds ratio answer different questions. For numerical outcomes, a mean difference may be unsuitable when distributions are strongly skewed; a median difference or transformed analysis may be more interpretable.

Multiple comparisons create a major exploratory risk. If enough hypotheses are tested, some will appear unusual by chance. Analysts should distinguish hypotheses generated by exploration from hypotheses specified in advance, control false discovery where appropriate, and replicate important findings on new data. Visual inspection should precede mechanical testing, and assumptions should be checked rather than silently accepted.

A relationship observed in aggregated data can reverse or disappear after disaggregation, a phenomenon related to Simpson’s paradox. Confounding occurs when a third variable influences both the apparent predictor and outcome. Temporal precedence is necessary, but not sufficient, for a causal interpretation. EDA can reveal candidate explanations; it generally cannot establish causality without an appropriate design or causal framework.

Missing Data, Outliers, and Data Quality

Missingness is not merely an inconvenience. Missing completely at random, missing at random conditional on observed variables, and missing not at random imply different risks. A missing-value map, grouped missingness rates, and a comparison of records with and without values can reveal operational patterns. Imputing a median may be adequate for a baseline model in some settings, but it can distort variance and relationships. Imputation must be performed within the training data during model development to avoid leakage.

An outlier is an observation unusual relative to a chosen reference distribution, not automatically an error. It may represent a data-entry problem, a rare but valid case, fraud, a new operating condition, or a separate population. Analysts should investigate provenance, units, timestamps, and domain limits before deleting or winsorising values. Robust statistics and transformations may be preferable to arbitrary removal.

Data leakage occurs when information unavailable at prediction time enters exploration, preprocessing, feature construction, or evaluation. For example, analysing a post-outcome field as a predictor can create a highly persuasive but unusable relationship. EDA should therefore document the prediction point and distinguish historical information from future information.

Stakeholder-Focused Reporting

A stakeholder report is not a catalogue of every chart produced during exploration. It should connect a decision to evidence. A useful report states the question, defines the population and measures, presents the principal finding, communicates uncertainty, identifies limitations, and recommends an action or next investigation.

Technical audiences may need code, diagnostics, assumptions, and sensitivity analyses. Executives may need a concise trend, business impact, uncertainty range, and decision implication. Operational teams may need thresholds, alerts, and record-level drill-down. The underlying analysis should remain consistent, but the presentation should be adapted to the audience.

A strong figure has a message-oriented title, an honest scale, units, an informative subtitle or annotation, and a source note. Instead of “Monthly Sales,” a title such as “Renewal volume fell after the January pricing change” communicates the analytical claim, provided the evidence supports it. Annotations should clarify important events without implying causality that has not been established.

Interactive dashboards require governance. Filters should show the current population and denominator; default views should not conceal missing data or small groups; downloaded values should be traceable to a defined query; and access controls should protect personal information. A dashboard that allows unrestricted slicing may encourage unreliable subgroup conclusions, particularly when counts are small.

Illustrative Case Study: Exploring Service Cancellation

Background

A fictional subscription company, Northstar Learning, observes an increase in monthly cancellations. The analytics team receives customer-level records containing subscription age, plan, monthly fee, support contacts, region, and cancellation status. The leadership team asks whether a new premium plan is causing higher cancellation.

Situation and Initial Analysis

The analysts first define the unit of analysis as an active subscription at the start of the observation month. They remove duplicate subscription identifiers, verify that the cancellation label is measured after the observation point, and inspect missingness. Monthly fee is right-skewed, and support contacts contain a small number of unusually high values. Rather than deleting these records, the team checks whether they represent legitimate enterprise accounts.

A bar chart shows that the premium plan has a higher raw cancellation rate than the standard plan. A box plot indicates that premium customers are older subscriptions and have more support contacts. A segmented chart by subscription age reveals that the difference is much smaller among customers with comparable tenure. Region also differs in plan composition.

Analysis and Approach

The team formulates the exploratory question: “After accounting descriptively for tenure and region, is cancellation consistently higher among premium subscribers?” They report cancellation rates with denominators for each plan-tenure group and use bootstrap intervals for rate differences. They avoid presenting the result as causal because customers self-select plans and the premium plan may attract complex accounts.

For modelling preparation, the team creates a time-based training and validation split, excludes fields created after cancellation, and documents the feature-generation date. A logistic regression and a tree-based model are considered later, but the EDA report does not treat predictive importance as causal influence. An interactive Plotly view allows managers to filter by plan and tenure while displaying group counts and uncertainty warnings for small cells.

Outcome

The initial aggregate comparison is revised. Premium membership remains associated with cancellation in some segments, but support burden and tenure explain part of the apparent difference. Management decides to run a controlled retention intervention for high-support customers rather than withdraw the premium plan. The next study is designed prospectively with a clear treatment definition and outcome window.

Learning Points

The case demonstrates why denominators, subgroup analysis, temporal definitions, and uncertainty matter. It also illustrates the difference between an exploratory association and a causal claim. A visually compelling aggregate chart could have led to an expensive and incorrect decision.

Limitations, Risks, and Professional Responsibilities

EDA is vulnerable to confirmation bias, p-hacking, selective reporting, overplotting, inappropriate aggregation, and the “garden of forking paths,” in which many analytical choices are made after seeing results. Analysts should preserve exploratory outputs, document decisions, and label preliminary findings. Reproducibility is a professional safeguard, not merely a convenience.

Privacy is relevant even when names are absent. Combinations of quasi-identifiers can re-identify individuals, and interactive dashboards may expose sensitive subgroup information. Apply data minimisation, role-based access, aggregation thresholds, and retention policies. Consider fairness: a chart can conceal poor performance for a minority group when averages are dominated by a majority. Report relevant subgroup results, examine measurement quality across groups, and avoid using protected attributes casually as if they were simple predictive variables.

Statistical graphics can also create organisational harm through exaggerated certainty. Analysts should state what was measured, what assumptions were made, what was not observed, and what decision the evidence can support. Where analysis informs high-impact decisions such as credit, employment, health, or education, governance should include documentation, review, contestability, and monitoring for disparate impact.

Emerging Directions and Best Practices

Modern EDA increasingly combines notebooks, automated profiling, interactive dashboards, experiment tracking, and data-quality monitoring. Automated tools can identify missingness, cardinality, correlations, and distribution shifts, but their outputs require human review. Large language models may assist with code generation or narrative drafting, yet analysts must verify calculations, privacy implications, and unsupported interpretations.

For professional practice, the following principles are especially valuable:

1. Begin with a decision or question, not with a favourite chart.

2. Inspect raw records, metadata, units, time boundaries, and missingness before interpreting patterns.

3. Combine robust numerical summaries with suitable visualisations.

4. Show denominators, sample sizes, uncertainty, and relevant subgroups.

5. Treat outliers and missing values as investigation topics rather than automatic cleaning targets.

6. Separate exploratory findings from confirmatory claims and causal conclusions.

7. Use time-aware splits and prevent leakage when analysis supports prediction.

8. Design accessible, labelled, reproducible figures and dashboards.

9. Document data provenance, transformations, assumptions, and analytical decisions.

10. Communicate limitations with the same clarity as findings.

Chapter Summary

Visualisation and exploratory analytics provide a foundation for reliable AI and ML work. Descriptive statistics summarise location, spread, and shape, but distributions and subgroup structure require graphical inspection. Matplotlib supports precise figure construction, Seaborn simplifies statistical graphics, and interactive tools enable controlled investigation of complex datasets. Sampling design, uncertainty intervals, bootstrap methods, and effect sizes help distinguish stable evidence from random variation. Hypothesis-oriented exploration must account for multiple comparisons, confounding, aggregation, and the difference between association and causation. Missing data, outliers, leakage, privacy, fairness, and accessibility are central analytical concerns. A professional report connects a clearly defined decision to transparent evidence, uncertainty, limitations, and an appropriate next action.

Key Terms

Exploratory Data Analysis: The iterative investigation of data using summaries, visualisations, and domain reasoning.

Descriptive Statistics: Numerical measures that summarise observed data.

Distribution: The pattern of values, frequencies, and shape for a variable.

Mean: The arithmetic average of numerical observations.

Median: The middle ordered value, relatively robust to extreme observations.

Interquartile Range: The difference between the third and first quartiles.

Skewness: Asymmetry in a variable’s distribution.

Outlier: An observation unusually distant from a reference pattern or distribution.

Correlation: A measure of association between variables, commonly linear or rank-based.

Sampling Bias: Systematic distortion caused by how observations enter a dataset.

Sampling Error: Random variation arising because a sample rather than a population is observed.

Bootstrap: A resampling method used to estimate variability of a statistic.

Confidence Interval: An interval produced by a procedure intended to capture a parameter with a stated long-run frequency.

Null Hypothesis: A model or statement representing no specified effect or association.

P-value: A measure of compatibility between data and a null model under its assumptions.

Confounding: Distortion of an apparent relationship by a third variable.

Data Leakage: Use of information in analysis or modelling that would not be available at the relevant prediction time.

Small Multiples: A set of similarly designed panels used to compare subgroups.

Dashboard: An interactive display of selected measures, filters, and visual evidence.

Reproducibility: The ability to regenerate analytical results from documented data and procedures.

Review Questions

Short Answer Questions

1. Why should a histogram generally be accompanied by numerical summaries?

2. Distinguish sampling error from sampling bias.

3. What information does an interquartile range provide?

4. Why is correlation insufficient for establishing causation?

5. State two requirements of an accessible statistical chart.

Long Answer Questions

1. Explain a reproducible EDA workflow for a newly received machine-learning dataset.

2. Compare Matplotlib, Seaborn, and interactive tools with respect to control, efficiency, and communication.

3. Explain how missing values and outliers should be investigated before preprocessing.

4. Discuss confidence intervals, bootstrap intervals, and the risks of ambiguous error bars.

5. Describe how a stakeholder-focused analytical report differs from an exploratory notebook.

Analytical / Application Questions

1. A premium customer segment has a higher cancellation rate, but also longer tenure and more support contacts. Design an analysis that investigates whether the raw difference is confounded.

2. An analyst tests 100 subgroup relationships and reports the three smallest p-values. Identify the inferential risks and propose a more defensible workflow.

3. Design a dashboard for monitoring model performance across demographic groups. Specify the measures, uncertainty information, privacy safeguards, and visual conventions you would include.

22. Multiple Choice Questions with Answers

Question 1

Which visualisation is most appropriate for examining the distribution of a single continuous variable and its possible skewness?

A. Histogram with an appropriate binning choice

B. Pie chart

C. Categorical bar chart of unrelated groups

D. Three-dimensional surface plot

Correct Answer: A. Histogram with an appropriate binning choice

Question 2

A large difference between the mean and median of annual income most directly suggests that the analyst should investigate:

A. Whether the variable is categorical

B. Skewness or influential extreme observations

C. Whether a pie chart is required

D. Whether correlation proves causation

Correct Answer: B. Skewness or influential extreme observations

Question 3

What is the principal limitation of increasing sample size to address a convenience sample?

A. It always increases measurement error

B. It prevents the use of confidence intervals

C. It does not automatically remove systematic sampling bias

D. It makes descriptive statistics impossible

Correct Answer: C. It does not automatically remove systematic sampling bias

Question 4

In the Northstar Learning case, why was the raw premium-plan cancellation rate potentially misleading?

A. Cancellation is always a categorical variable

B. Premium customers differed in tenure and support burden

C. Box plots cannot show numerical variables

D. Interactive charts cannot represent rates

Correct Answer: B. Premium customers differed in tenure and support burden

Question 5

What does a bootstrap procedure primarily estimate?

A. The causal effect of an intervention without assumptions

B. The variability of a statistic through repeated resampling

C. The number of missing values in a dataset

D. The exact population value with certainty

Correct Answer: B. The variability of a statistic through repeated resampling

Question 6

Which statement about a p-value is correct?

A. It is the probability that the null hypothesis is true

B. It measures practical importance directly

C. It describes compatibility of the data with a null model under assumptions

D. It proves that one variable causes another

Correct Answer: C. It describes compatibility of the data with a null model under assumptions

Question 7

An analyst uses a field recorded after a customer cancels to predict cancellation. This is an example of:

A. Stratified sampling

B. Data leakage

C. Robust estimation

D. Small multiples

Correct Answer: B. Data leakage

Question 8

Which chart is generally most suitable for comparing a numerical outcome across several categorical groups while showing distributional spread?

A. Box plot

B. Pie chart

C. Unlabelled icon array

D. Network diagram

Correct Answer: A. Box plot

Question 9

Why should a dashboard display denominators and warn about very small groups?

A. To make all charts use identical colours

B. To prevent users from interpreting unstable subgroup rates as reliable evidence

C. To eliminate all sampling bias

D. To convert association into causation

Correct Answer: B. To prevent users from interpreting unstable subgroup rates as reliable evidence

Question 10

Which practice best supports an ethical and reproducible exploratory analysis?

A. Delete all outliers before inspecting their provenance

B. Report only statistically significant subgroup results

C. Document transformations, assumptions, data provenance, and limitations

D. Use interactivity to conceal sensitive observations

Correct Answer: C. Document transformations, assumptions, data provenance, and limitations