Exploratory Data Analysis and Visualization

Introduction

Exploratory Data Analysis (EDA) is the systematic process of examining, summarising, visualising, and questioning a dataset before formal statistical modelling or machine-learning development. It occupies a critical position between data acquisition and predictive analysis. EDA helps analysts understand what the data contains, whether it is trustworthy, which variables may be useful, and what assumptions are reasonable.

EDA is not simply the production of attractive charts. It is an investigative activity. An analyst uses numerical summaries and visual representations to discover distributions, relationships, missingness, anomalies, measurement errors, subgroup differences, and possible data-generation processes. In artificial intelligence and machine learning, EDA can reveal target leakage, class imbalance, biased sampling, duplicated records, and features that will not generalise beyond the training sample.

A sound EDA process combines three levels of analysis:

• Univariate analysis: examination of one variable at a time.
• Bivariate analysis: examination of association between two variables.
• Multivariate analysis: examination of several variables jointly.

Python provides a powerful environment for this work. Pandas supports data manipulation, Matplotlib provides low-level plotting control, Seaborn supplies statistically oriented graphics, and tools such as Plotly and Streamlit support interactive exploration and dashboards.

Learning Objectives

By the end of this chapter, a learner should be able to:

1. Explain the purpose, scope, and limitations of EDA.
2. Conduct reliable univariate, bivariate, and multivariate analysis.
3. Select appropriate plots for variable types and analytical questions.
4. Use Pandas, Matplotlib, and Seaborn to produce reproducible visualisations.
5. Identify outliers, anomalies, missing-data patterns, and possible bias.
6. Distinguish association from causation and describe uncertainty responsibly.
7. Design a concise dashboard or analytical report for a non-specialist audience.
8. Apply EDA findings to feature engineering, modelling, and decision-making.

1. The Role and Logic of EDA

A typical analytical workflow is:

1. Define the business, scientific, or operational question.
2. Acquire and document the data.
3. Inspect structure, data types, units, and quality.
4. Explore distributions and relationships.
5. Clean or transform data with recorded justification.
6. Form hypotheses and select suitable methods.
7. Build and validate models or make decisions.
8. Communicate findings and limitations.

EDA is iterative rather than strictly linear. A histogram may reveal impossible values, which leads to data-quality investigation. A scatter plot may show that a relationship differs by customer segment, leading to stratified analysis. A model may expose residual patterns that require further exploration.

The central principle is question-driven exploration. Every table or graphic should answer, or help formulate, a meaningful question. Producing many plots without interpretation increases the risk of false discoveries and selective reporting.

1.1 Data types and measurement scales

The choice of summary and chart depends on the nature of a variable.

Measurement units must be retained. A variable called weight is not interpretable unless the unit—kilograms, pounds, or another measure—is known. Mixing units can create plausible but incorrect results.

2. Data Audit Before Visualisation

Before analysing patterns, inspect the data structure. A basic audit should answer:

• How many rows and columns are present?
• What does one row represent?
• Which columns are identifiers, measurements, targets, or metadata?
• Are data types correctly assigned?
• Are values missing, duplicated, or impossible?
• Are observations independent, or are there repeated measurements?
• What time period and population do the data represent?

Illustrative Python inspection:

import pandas as pd

 df = pd.read_csv("sales.csv")
print(df.shape)
print(df.head())
print(df.dtypes)
print(df.isna().sum().sort_values(ascending=False))
print(df.duplicated().sum())
print(df.describe(include="all").T)

The leading space before df in the example should not be included in an executable script; it is important that code blocks preserve valid Python indentation.

A data dictionary should document variable names, meanings, types, units, allowed values, missing-value codes, and collection methods. Values such as -99, unknown, or 9999 may represent missingness rather than genuine measurements. They should not automatically be treated as numerical observations.

2.1 Numerical summaries

For observations x₁, x₂, ..., x_n, the arithmetic mean is:

Here, n is the number of observations and xᵢ is observation i. The mean is measured in the same units as the variable, but it can be strongly affected by extreme values.

The sample variance is:

The sample standard deviation is s = √s². The denominator n − 1 is used when estimating population variability from a sample under conventional independent-sampling assumptions. If the complete population is being described rather than a sample estimate, a denominator of n may be appropriate.

The median is the middle ordered value, or the average of the two middle values when n is even. Quartiles divide ordered data into approximate sections. The interquartile range is:

where Q₁ and Q₃ are the first and third quartiles. The IQR is resistant to extreme observations and is particularly useful for skewed data.

For a category with count n_j in a dataset of size n, its sample proportion is:

Percentages should normally be accompanied by denominators, especially when groups differ greatly in size.

3. Univariate Analysis

Univariate analysis examines each variable independently. Its purpose is to understand scale, centre, spread, shape, frequency, and data quality.

3.1 Categorical variables

For categorical variables, calculate frequencies and proportions. A bar chart is generally preferable to a pie chart because lengths are easier to compare than angles. Categories should be ordered logically—for example, from highest to lowest frequency or from low to high satisfaction—rather than alphabetically when that would obscure interpretation.

import seaborn as sns
import matplotlib.pyplot as plt

order = df["region"].value_counts().index
sns.countplot(data=df, y="region", order=order)
plt.xlabel("Number of observations")
plt.ylabel("Region")
plt.title("Observations by region")
plt.tight_layout()
plt.show()

Rare categories deserve attention. They may represent important minority groups, data-entry errors, or categories that should be combined only with substantive justification.

3.2 Numerical variables

A histogram displays observations across intervals called bins. Bin width affects the visual impression: overly narrow bins exaggerate noise, whereas overly wide bins hide structure. A density plot provides a smoothed estimate of distribution shape, but smoothing can conceal local features and should not replace inspection of the raw data.

fig, axes = plt.subplots(1, 2, figsize=(11, 4))
sns.histplot(data=df, x="income", bins=30, kde=True, ax=axes[0])
sns.boxplot(data=df, x="income", ax=axes[1])
axes[0].set_title("Income distribution")
axes[1].set_title("Income box plot")
plt.tight_layout()

A box plot conventionally shows the median, quartiles, and a rule for flagging observations beyond a distance of 1.5 IQR from the quartiles. Such points are potential outliers, not automatically errors. A high income, unusually long hospital stay, or exceptional transaction may be valid and substantively important.

Skewness describes asymmetry. Right-skewed variables, such as transaction values, may have a mean greater than the median. Logarithmic transformation can sometimes make a positive variable more suitable for modelling:

This requires x > 0 and changes the scale of interpretation. If zero values occur, an explicitly justified transformation such as log(1 + x) may be considered, but it should not be applied mechanically.

3.3 Time variables

Time series should generally be plotted in chronological order. A line chart can reveal trend, seasonality, structural breaks, missing periods, and changing variability. Aggregation may simplify a chart but can also hide important within-period variation. Analysts should record the time zone, frequency, and whether observations are complete.

4. Bivariate Analysis

Bivariate analysis examines how two variables behave together. The appropriate chart depends on whether variables are numerical or categorical.

4.1 Association between numerical variables

The Pearson correlation coefficient is:

Here, xᵢ and yᵢ are paired observations, and x̄ and ȳ are sample means. When both variables have non-zero variance, −1 ≤ r ≤ 1. Pearson correlation measures linear association; it does not detect every nonlinear relationship and does not establish causation. It can also be distorted by outliers.

Spearman’s rank correlation is often more suitable when variables are ordinal, strongly skewed, or related monotonically but not linearly. Both measures should be considered descriptive rather than causal evidence.

sns.scatterplot(data=df, x="advertising_spend", y="sales", alpha=0.6)
sns.regplot(data=df, x="advertising_spend", y="sales", scatter=False,
            line_kws={"color": "red"})
plt.title("Advertising spend and sales")

A fitted line is a summary, not proof that changing advertising spend causes sales to change. Confounding variables, reverse causality, or common time trends may explain the association.

4.2 Numerical variable by category

Group comparisons should show distributions rather than only group means. Box plots, violin plots, and jittered points reveal differences in centre, spread, sample size, and unusual observations. Confidence intervals or bootstrap intervals can communicate uncertainty around estimated group summaries, but an interval is not a guarantee that the true value lies inside it.

4.3 Two categorical variables

A contingency table records joint counts. Conditional proportions are often more informative than raw counts because they account for unequal group sizes. For example, the proportion of customers who churned within each subscription type is more useful than the total number of churners in each type.

A heatmap can encode proportions or counts, but the legend and denominator must be explicit. A heatmap of counts may misleadingly suggest stronger association in a large group.

5. Multivariate Exploration

Multivariate analysis studies interactions among three or more variables. It is essential because a pattern visible in aggregate data may disappear or reverse after stratification. This phenomenon is commonly associated with Simpson’s paradox.

Useful methods include:

• Colour, shape, or facet panels in scatter plots.
• Pair plots for a small number of variables.
• Correlation heatmaps for numerical screening.
• Parallel-coordinate plots for moderate-dimensional data.
• Dimensionality-reduction displays such as PCA or t-SNE, used cautiously.
• Grouped time-series graphics.

features = ["age", "income", "balance", "tenure", "churn"]
sns.pairplot(df[features], hue="churn", corner=True)
plt.show()

Pair plots become unreadable with many variables and may encourage unplanned hypothesis generation. A correlation heatmap is similarly a screening device, not a complete account of dependence.

The Pearson correlation matrix contains pairwise correlations. If several features are highly correlated, a model may experience multicollinearity, unstable coefficients, or redundant information. Correlation does not identify nonlinear dependence, conditional dependence, or causal structure.

Principal Component Analysis (PCA) transforms centred numerical variables into orthogonal directions of decreasing variance. If the data matrix is X and centred matrix is X_c, PCA commonly uses the covariance matrix:

Eigenvectors of S define principal directions and eigenvalues quantify variance along those directions. Standardisation is usually advisable when variables have different units or scales; otherwise, variables with large numerical units may dominate. PCA is useful for visualisation and compression, but its components may be difficult to interpret and do not necessarily represent meaningful causes.

6. Visualisation Design and Statistical Communication

An effective visualisation has a clear analytical purpose, an honest scale, and sufficient annotation for independent interpretation.

6.1 Choosing an encoding

Position along a common scale is generally easier to compare than area, volume, or colour intensity. Bar charts should normally begin at zero when bar length represents magnitude. Truncated axes can be appropriate for line charts when clearly labelled, but they may exaggerate small differences. Three-dimensional charts often introduce perspective distortion and should be avoided unless there is a compelling reason.

Use colour deliberately:

• Sequential palettes represent low-to-high magnitude.
• Diverging palettes represent deviation around a meaningful midpoint.
• Qualitative palettes distinguish categories without implying order.

Colour should not be the only channel for important distinctions because some readers have colour-vision deficiencies. Direct labels, line styles, symbols, or annotations improve accessibility.

6.2 Uncertainty

Uncertainty can arise from sampling variation, measurement error, missing data, model assumptions, and changing populations. It should be communicated using confidence intervals, prediction intervals, error bars, distributions, ranges, or sensitivity analyses where appropriate.

A standard error describes uncertainty in an estimated statistic under a specified sampling model. It is not the same as the standard deviation of individual observations. Error bars must state whether they represent standard deviation, standard error, confidence intervals, or another quantity.

For a sample mean under independent observations with finite variance, a commonly used standard error is:

where s is the sample standard deviation and n is sample size. The formula does not account for clustering, serial dependence, or complex sampling unless modified accordingly.

A confidence interval should not be described as the probability that a fixed parameter is inside this particular interval. Under repeated sampling and the method’s assumptions, a 95% confidence procedure captures the parameter in approximately 95% of constructed intervals.

6.3 Reproducible graphics

Use explicit figure dimensions, labels, units, legends, titles, source notes, and a consistent theme. Set random seeds for stochastic procedures and preserve the code and data version used to create each figure. Avoid reporting only the most attractive or statistically significant plot. A reproducible report should make transformations and exclusions visible.

7. Interactive Exploration and Dashboards

Static graphics are appropriate for reports and publications. Interactive tools are useful when users need to filter by date, region, product, or customer segment. Plotly supports interactive charts, while Streamlit can turn Python analysis into a lightweight application.

A professional dashboard should have:

1. A defined audience and decision purpose.
2. A small number of high-value metrics.
3. Clearly labelled filters and current selections.
4. Consistent denominators and time definitions.
5. Explanatory notes for unusual values and missing data.
6. Accessible colour and keyboard-friendly interaction where possible.
7. Protection against disclosure of sensitive records.

Dashboards can create false confidence if users see precise values without uncertainty or context. A filter that silently changes the denominator is especially dangerous. Every key metric should specify its population, time interval, aggregation rule, and refresh date.

8. An Illustrative Case: Exploring Customer Churn

Background

A subscription company has 50,000 customer records containing tenure, monthly charge, contract type, service usage, support contacts, region, and a binary churn indicator. Management wants to understand churn patterns and identify priorities for a retention model.

Problem

The initial request—“find the variables that cause churn”—is too strong for observational data. The immediate EDA objective is to identify reliable associations, data-quality problems, possible leakage, subgroup differences, and variables suitable for later modelling.

Analysis

The analyst first checks whether each row represents one customer, verifies that churn is measured after the observation window, and removes no records without documenting a reason. Duplicate customer identifiers reveal 620 repeated records. Investigation shows that these are monthly snapshots rather than duplicate customers; the data must therefore be treated as longitudinal, not as 50,000 independent customers.

A frequency table shows that churn is 18%, but the rate differs substantially by contract type. A bar chart of counts is misleading because contract groups have different sizes, so the analyst plots the conditional churn rate with denominators in annotations. Box plots show that churned customers have shorter observed tenure, but the distributions overlap considerably.

A scatter plot of monthly charges against tenure, coloured by churn status, reveals that high charges are concentrated among newer customers in one contract group. A simple overall correlation between charges and churn would conceal this interaction. Faceted plots by contract type show that the relationship is not uniform.

Missingness analysis finds that service-usage values are absent primarily for customers who joined during a system migration. Missingness is therefore potentially informative and may reflect a process change rather than random omission. The analyst records a missingness indicator and consults the data owner before choosing an imputation strategy.

A target-leakage check identifies a field called cancellation_reason, recorded after cancellation. It is excluded from predictive features because it would not be available at the time a retention decision is made.

Approach and solution

The final exploratory report includes:

• A data-quality table showing duplicates, missingness, ranges, and invalid codes.
• Churn rates by contract type with group denominators.
• Tenure distributions by churn status.
• A faceted charge–tenure scatter plot.
• A missingness heatmap and a timeline of the migration period.
• A documented list of excluded leakage variables.

The modelling team then performs a time-based train–validation split rather than a random split, because customer behaviour changes over time. EDA findings are incorporated into feature engineering, but the team avoids treating exploratory associations as causal claims.

Outcome

The analysis does not prove that short tenure or high charges cause churn. It does establish that churn is heterogeneous, that data collection changed during migration, and that contract type modifies observed patterns. The resulting model-development plan is more credible because its population, time ordering, leakage controls, and missing-data assumptions are explicit.

Learning points

1. The meaning of a row determines which summaries and models are valid.
2. Rates require denominators; raw counts can mislead.
3. A global association may conceal subgroup structure.
4. Missingness can itself contain information about data collection.
5. Features recorded after the outcome create target leakage.
6. EDA improves modelling decisions but does not by itself establish causation.

9. Limitations, Ethics, and Governance

EDA is vulnerable to confirmation bias, multiple comparisons, poor sampling, measurement error, and overinterpretation. Analysts may unconsciously search until an attractive pattern appears. Exploratory findings should therefore be distinguished from confirmatory tests, and important findings should be validated on new data or through a pre-specified study.

Privacy is relevant even when names are removed. Combinations of age, location, timestamp, and rare behaviour can re-identify individuals. Analysts should minimise data access, apply appropriate aggregation, avoid displaying sensitive individual records, and follow institutional policy and applicable data-protection law.

Fairness analysis should examine performance and data quality across relevant groups. A dashboard can encode institutional bias if it presents historically unequal outcomes as neutral facts. Analysts should ask who is represented, who is missing, how categories were defined, and whether a variable is a proxy for protected characteristics.

Automated anomaly detection also requires governance. An anomaly may be fraud, a genuine rare event, a sensor failure, or a legitimate minority pattern. Decisions affecting people should not rely on an unexplained threshold or an automatically discarded record. Maintain an audit trail of transformations, exclusions, code versions, and approval decisions.

10. Best-Practice Workflow

A professional EDA workflow can be expressed as follows:

1. State the decision or research question.
2. Define the unit of analysis, population, and observation period.
3. Create a data dictionary and inspect data types and units.
4. Quantify missingness, duplication, invalid values, and sampling coverage.
5. Perform univariate analysis for every important field.
6. Use question-driven bivariate and multivariate graphics.
7. Investigate anomalies before deleting or winsorising observations.
8. Check subgroup patterns, time ordering, and possible leakage.
9. Communicate uncertainty, denominators, and limitations.
10. Record code, assumptions, transformations, and decisions.
11. Validate important findings on a holdout sample or new period.
12. Translate findings into modelling, operational, or research actions.

Chapter Summary

Exploratory Data Analysis is a disciplined process for understanding data before modelling or decision-making. Univariate analysis describes individual distributions and data quality; bivariate analysis examines associations between pairs of variables; and multivariate analysis investigates interactions, subgroup differences, and structure across several variables. Numerical summaries such as the mean, standard deviation, quartiles, IQR, proportions, and correlation are useful only when their assumptions and limitations are understood.

Matplotlib and Seaborn support reproducible static visualisation, while Plotly and Streamlit can support carefully designed interactive dashboards. Effective graphics use appropriate encodings, honest scales, clear labels, accessible colour, and explicit uncertainty. EDA must distinguish association from causation, treat outliers and missingness as investigative issues, and protect privacy and fairness. In machine-learning projects, EDA is especially important for detecting leakage, temporal structure, class imbalance, and distribution shift. A strong EDA report connects evidence to decisions while documenting uncertainty and limitations.

Key Terms

EDA: Exploratory Data Analysis; systematic examination of data before formal modelling.

Univariate analysis: Analysis of one variable at a time.

Bivariate analysis: Analysis of the relationship between two variables.

Multivariate analysis: Analysis involving several variables jointly.

Data dictionary: Documentation describing fields, meanings, types, units, and valid values.

Histogram: A chart showing numerical observations across intervals or bins.

Box plot: A distribution plot showing quartiles, median, and potential extreme values.

Interquartile range: The difference between the third and first quartiles.

Outlier: An observation unusually distant from other observations; it is not necessarily an error.

Skewness: Asymmetry in a probability or empirical distribution.

Correlation: A numerical measure of association, often linear association for Pearson correlation.

Confounding: A situation in which another variable helps explain an observed association.

Target leakage: Use of information unavailable at prediction time or recorded after the target outcome.

Missingness: The condition in which a value is not observed or recorded.

Conditional proportion: A proportion calculated within a specified subgroup or denominator.

Dashboard: An interactive display of selected metrics and visualisations for decision support.

Confidence interval: An interval produced by a method designed to capture a population parameter at a stated long-run frequency.

PCA: Principal Component Analysis, a transformation into orthogonal directions of decreasing variance.

Data storytelling: The structured communication of evidence, interpretation, uncertainty, and implications.

Review Questions

Short Answer Questions

1. What is the difference between univariate, bivariate, and multivariate analysis?
2. Why can a bar chart of raw counts mislead when group sizes differ?
3. Define target leakage and give one example.
4. What information does the IQR provide?
5. Why does correlation not establish causation?

Long Answer Questions

1. Explain a complete EDA workflow from data audit to communication of findings.
2. Compare histograms, box plots, density plots, and violin plots, including situations in which each is useful.
3. Discuss how missing data, duplicates, and incorrect units can affect machine-learning analysis.
4. Explain the principles of uncertainty communication in dashboards and analytical reports.
5. Discuss ethical and governance issues that arise when EDA is applied to customer or personal data.

Analytical / Application Questions

1. A hospital dataset shows that average waiting time is lower for one department, but that department treats mostly low-severity cases. Design an EDA plan to investigate whether department is associated with waiting time after accounting for severity.
2. A fraud-detection dataset contains a highly predictive variable that is updated after an investigation is completed. Explain why this variable is problematic and propose a leakage-resistant analysis strategy.
3. Design a dashboard for monitoring monthly subscription churn. Specify at least four metrics, suitable visualisations, denominators, filters, uncertainty information, and privacy safeguards.

Multiple Choice Questions with Answers

Question 1

Which chart is generally most appropriate for comparing the frequencies of nominal categories?

A. Bar chart

B. Scatter plot

C. Line chart

D. Box plot

Correct Answer: A. Bar chart

Question 2

The interquartile range is calculated as:

A. Mean minus median

B. Q₃ minus Q₁

C. Maximum minus minimum only

D. Variance divided by sample size

Correct Answer: B. Q₃ minus Q₁

Question 3

What is a key limitation of Pearson correlation?

A. It can only be used with categorical variables

B. It proves that one variable causes another

C. It primarily measures linear association and can be affected by outliers

D. It always detects nonlinear relationships

Correct Answer: C. It primarily measures linear association and can be affected by outliers

Question 4

Why can conditional proportions be preferable to raw counts in a group comparison?

A. They remove the need for a denominator

B. They account for differences in group size

C. They guarantee causal interpretation

D. They eliminate missing data

Correct Answer: B. They account for differences in group size

Question 5

Which situation is an example of target leakage?

A. Using age measured before prediction

B. Using a customer’s region at registration

C. Using cancellation reason recorded after cancellation to predict churn

D. Using historical monthly usage available before prediction

Correct Answer: C. Using cancellation reason recorded after cancellation to predict churn

Question 6

For a sample mean, under independent-observation assumptions, the conventional standard error is:

A. s × n

B. s ÷ √n

C. n ÷ s

D. s² ÷ n

Correct Answer: B. s ÷ √n

Question 7

Which visualisation is especially useful for examining the relationship between two numerical variables?

A. Scatter plot

B. Pie chart

C. Single-category bar chart

D. Frequency table only

Correct Answer: A. Scatter plot

Question 8

What does PCA primarily do?

A. Converts all variables into categorical labels

B. Creates orthogonal directions ordered by explained variance

C. Proves causal relationships

D. Automatically removes all outliers

Correct Answer: B. Creates orthogonal directions ordered by explained variance

Question 9

Which practice improves dashboard integrity?

A. Changing denominators silently when filters are applied

B. Using three-dimensional effects for all charts

C. Clearly stating populations, time periods, and aggregation rules

D. Showing only statistically attractive results

Correct Answer: C. Clearly stating populations, time periods, and aggregation rules

Question 10

Why should a potential outlier not automatically be deleted?

A. All outliers are required for every model

B. It may be a valid and important observation rather than an error

C. Deletion always increases bias

D. Outliers cannot affect summary statistics

Correct Answer: B. It may be a valid and important observation rather than an error