Exploratory Data Analysis and Statistical Reasoning

Introduction

Exploratory data analysis (EDA) is the disciplined process of examining, summarising and visualising data before formal modelling or decision-making. It combines computational tools with statistical reasoning. In Python for Data Science, EDA commonly uses pandas, NumPy, Matplotlib, Seaborn and SciPy, but software does not replace judgement. A plot can reveal a pattern, while statistical reasoning determines whether that pattern is credible, practically important and relevant to the question being investigated.

EDA has three connected purposes:

1. Understand structure: identify variables, data types, distributions, missingness, duplicates and possible errors.
2. Generate and assess explanations: examine associations, differences and plausible mechanisms.
3. Quantify uncertainty: distinguish signal from random variation and avoid overconfident conclusions.

A central principle is that descriptive analysis and inferential analysis answer different questions. Descriptive statistics summarise the observed data. Inferential statistics use a sample to reason about a wider population, subject to assumptions and uncertainty.

Learning Objectives

By the end of this chapter, you should be able to:

• distinguish populations, samples, parameters and statistics;
• calculate and interpret common measures of centre and spread;
• use probability concepts and distributions to reason about variation;
• explain sampling error, bias and the central limit theorem;
• investigate data with suitable tables, plots and Python workflows;
• interpret covariance, correlation and confounding cautiously;
• formulate hypotheses and perform basic tests with confidence intervals and p-values;
• recognise multiple testing, practical significance, uncertainty and misleading conclusions;
• communicate an evidence-based analytical recommendation.

1. The Analytical Starting Point: Question, Data and Context

A sound EDA begins before code is written. The analyst should specify the unit of observation, target population, time period, measurement process and decision to be supported. For example, “average delivery time” is incomplete unless one knows whether the observations are orders, customers or delivery routes, and whether cancelled orders are excluded.

A useful data audit asks:

In Python, a first inspection might be:

import pandas as pd

 df = pd.read_csv("orders.csv")
print(df.shape)
print(df.head())
print(df.dtypes)
print(df.isna().mean().sort_values(ascending=False))
print(df.describe(include="all"))

These commands provide an initial inventory, not a complete analysis. describe() may conceal skewness, outliers or meaningful subgroups. Dates should be parsed explicitly, categorical codes should not automatically be treated as numerical measurements, and identifiers should not be summarised as if they were quantities.

2. Descriptive Statistics

2.1 Centre

For observations x₁, x₂, …, xₙ, the arithmetic mean is:

Here, x̄ is the sample mean, n is the sample size and xᵢ is observation i. The mean is appropriate for quantitative data when addition and averaging are meaningful. It is sensitive to extreme values.

The median is the middle ordered observation, or the average of the two middle observations when n is even. It is resistant to extreme values and is often preferable for income, waiting time and other right-skewed variables. The mode is the most frequent value and is useful for categorical variables.

A weighted mean is:

where wᵢ is a non-negative weight. Weights must represent a defensible sampling or measurement rationale; arbitrary weighting can create distortion.

2.2 Spread

The range is max(xᵢ) − min(xᵢ), but it depends only on two observations. The sample variance is:

The sample standard deviation is s = √s², in the same units as x. The denominator n − 1, called Bessel’s correction, provides an unbiased estimator of population variance under random sampling and standard assumptions. It does not make biased data unbiased.

The interquartile range is IQR = Q₃ − Q₁, where Q₁ and Q₃ are the 25th and 75th percentiles. A common exploratory rule identifies values below Q₁ − 1.5IQR or above Q₃ + 1.5IQR as potential outliers. This is a screening convention, not proof that an observation is erroneous.

The coefficient of variation is:

It is useful for comparing relative variability when the mean is positive and meaningfully above zero. It is inappropriate when the mean is near zero or when the measurement scale has no meaningful zero.

Worked Example: Resistant Summaries

Suppose delivery times are 8, 9, 10, 11 and 42 minutes. The mean is 16 minutes, while the median is 10 minutes. The extreme observation raises the mean by 6 minutes relative to the median. A box plot and histogram would prompt investigation: 42 minutes may be a genuine exceptional delivery, a recording error or evidence of a separate process. Deleting it solely because it is inconvenient is not valid.

2.3 Shape and Visualisation

A histogram shows the distribution of a numerical variable, while a density plot provides a smoothed approximation. A box plot displays the median, quartiles and potential outliers. A bar chart is generally appropriate for categorical counts; a pie chart is usually less precise for comparisons. Scatter plots show relationships between two quantitative variables. Line charts are suitable when the horizontal axis has an ordered time or sequence meaning.

Visual choices should reflect the question. Truncated axes can exaggerate differences in bar charts. Unequal bin widths can mislead histograms unless density is represented correctly. Overplotting can hide points, so transparency, sampling or hexbin plots may be useful.

3. Probability and Distributions

Probability quantifies uncertainty about outcomes. For an event A, 0 ≤ P(A) ≤ 1. The complement rule is P(Aᶜ) = 1 − P(A). For events A and B:

The subtraction avoids counting the intersection twice. Conditional probability is:

Independence means P(A ∩ B) = P(A)P(B), equivalently P(A | B) = P(A) when P(B) > 0. Independence should not be assumed merely because two variables appear unrelated.

Bayes’ theorem reverses conditional probabilities:

This is crucial in diagnostic testing. A positive test does not necessarily imply that a person has a rare condition; the base rate matters.

A random variable assigns numerical values to outcomes. Its expected value is the probability-weighted average. For a discrete variable X:

A variance is Var(X) = E[(X − μ)²], where μ = E(X). The standard deviation is the square root of variance.

Important Distributions

• Bernoulli: one trial with success probability p; X is 1 or 0.
• Binomial: number of successes in n independent Bernoulli trials with constant p. Its mean is np and variance is np(1 − p).
• Normal: a continuous, symmetric distribution described by mean μ and standard deviation σ. It is a model, not a claim that all data are normal.
• Poisson: counts of events in a fixed interval under assumptions of a constant rate and independent events; its mean and variance are both λ.
• Exponential: waiting times between Poisson events under a constant-rate model.

Distributional assumptions should be examined through plots, domain knowledge and residual diagnostics, not by relying on a single normality test.

4. Sampling, Bias and the Central Limit Theorem

A population is the complete set of units of interest; a sample is the observed subset. A population parameter, such as μ, is usually unknown. A sample statistic, such as x̄, is computed from data and used to estimate the parameter.

Sampling variability is variation between samples drawn from the same population. Sampling bias is systematic deviation caused by the selection or measurement process. Common sources include:

• coverage bias: some population members cannot be sampled;
• non-response bias: selected units do not participate;
• voluntary-response bias: participation is related to opinion;
• measurement bias: instruments or questions systematically distort values;
• survivorship bias: analysis includes only units that remain observable.

Random sampling can reduce selection bias, but it does not remove non-response, poor measurement or confounding. Stratified sampling ensures representation of specified groups; cluster sampling samples groups and may increase dependence; convenience sampling is easy but weak for generalisation.

The standard error of the sample mean, under independent observations with finite variance, is:

Since σ is usually unknown, it is estimated by s/√n. The formula shows that quadrupling sample size halves standard error, not that it doubles precision in every practical sense.

The central limit theorem states, broadly, that for independent observations with a finite mean and variance, the sampling distribution of x̄ approaches a normal distribution as n increases, under regularity conditions. It concerns the distribution of a statistic across repeated samples; it does not assert that the original observations are normally distributed.

5. Correlation, Association and Causation

Covariance measures whether two variables vary together:

Its units depend on both variables. Pearson’s sample correlation is:

where sₓ and sᵧ are sample standard deviations. Thus −1 ≤ r ≤ 1. Pearson correlation measures linear association and is sensitive to outliers. A correlation near zero does not rule out a nonlinear relationship. Spearman’s rank correlation uses ranks and can be more suitable for monotonic relationships or ordinal measurements, although it also does not establish causation.

A third variable can create confounding when it influences both variables being compared. For example, ice-cream sales and swimming incidents may rise together because temperature affects both. Stratification, experimental design and causal modelling can help, but adjustment alone cannot guarantee causal interpretation.

A strong analytical report should state: the population studied, time period, variables, missing-data treatment, visual evidence, effect size, uncertainty and plausible alternative explanations.

6. Hypothesis Testing and Confidence Intervals

Hypothesis testing formalises a comparison between a null hypothesis H₀ and an alternative hypothesis H₁. For example, H₀: μ = μ₀ versus H₁: μ ≠ μ₀. The null specifies a reference model; it is not automatically true or substantively important.

A p-value is the probability, assuming H₀ and the test assumptions are correct, of obtaining a result at least as extreme as the observed result. It is not the probability that H₀ is true, nor the probability that the result occurred by chance in an unrestricted sense.

For a one-sample mean with unknown population standard deviation, the t statistic is:

Under independent observations and an appropriate model, t follows a t distribution with n − 1 degrees of freedom when H₀ is true. For a two-sided 95% confidence interval for μ:

Here t₀.₉₇₅,ₙ₋₁ is the 97.5th percentile of the t distribution. A 95% confidence procedure produces intervals that contain the fixed population parameter in approximately 95% of repeated samples under the model. It is not technically correct to say that there is a 95% probability that a particular fixed parameter lies in the calculated interval under the frequentist interpretation.

Practical and Statistical Significance

A small p-value can result from a trivial effect in a large sample. Conversely, an important effect may be estimated imprecisely in a small sample. Report effect size and confidence interval, not p-value alone. A mean difference of 0.2 minutes may be statistically detectable but operationally irrelevant; a 5-minute reduction may matter even if the sample is too small for precise inference.

Errors and Power

A Type I error rejects a true null hypothesis; its probability is controlled by the significance level α under the test procedure. A Type II error fails to reject a false null hypothesis. Statistical power is 1 − β, where β is the Type II error probability. Power depends on sample size, noise, effect size and α. “Fail to reject” does not mean “prove no effect.” Equivalence or non-inferiority designs are more appropriate when demonstrating practical similarity.

Multiple Testing

Testing many hypotheses increases the chance of at least one false positive. If m independent tests each use α = 0.05, the probability of at least one false positive is approximately 1 − (1 − 0.05)ᵐ. With 20 tests this is approximately 0.64. The independence assumption is not generally exact, but the inflation principle remains. Pre-specification, replication and corrections such as Bonferroni or false discovery rate procedures reduce misleading discoveries.

7. A Reproducible Python EDA Workflow

A defensible workflow is iterative:

1. define the question and estimand;
2. inspect schema, units, duplicates and missingness;
3. validate ranges and categorical levels;
4. describe distributions overall and by meaningful groups;
5. visualise relationships and time patterns;
6. assess assumptions and influential observations;
7. quantify effects with intervals or suitable tests;
8. document decisions and sensitivity analyses;
9. communicate limitations and recommendations.

Example code:

import seaborn as sns
import matplotlib.pyplot as plt
from scipy import stats

sns.histplot(data=df, x="delivery_minutes", kde=True)
plt.show()

sns.boxplot(data=df, x="region", y="delivery_minutes")
plt.xticks(rotation=45)
plt.show()

stat, p_value = stats.ttest_ind(
    df.loc[df.region == "East", "delivery_minutes"].dropna(),
    df.loc[df.region == "West", "delivery_minutes"].dropna(),
    equal_var=False
)
print(stat, p_value)

Welch’s t-test does not assume equal population variances, but independence and reasonable measurement conditions remain important. Code should be accompanied by a data dictionary, environment information, random seeds where relevant and version-controlled notebooks or scripts.

Applied Case: Delivery-Time Improvement

Background. This illustrative case concerns a fictional courier company that introduces route recommendations in one city zone. Management wants to know whether average delivery time falls.

Problem. A dashboard reports that treated deliveries average 31 minutes compared with 34 minutes in other zones. A simple comparison may be misleading because the treated zone has shorter routes and fewer evening deliveries.

Analysis. The analyst first defines the unit as a completed delivery and excludes no observations without investigating them. Delivery time is plotted by zone, hour and route distance. The treated zone has a different distance distribution, indicating confounding. Missing GPS durations are reported separately rather than silently imputed. A pre-intervention period is compared with a post-intervention period, and a difference-in-differences model is considered:

The first difference is the change in the treated group; the second is the change in the comparison group. This design requires a credible parallel-trends assumption: absent the intervention, both groups would have followed similar trends. It also requires attention to staggered adoption, seasonal effects, clustering and concurrent operational changes.

Outcome. Suppose the raw post-period difference is −3 minutes, but the difference-in-differences estimate is −1.2 minutes with a 95% confidence interval from −2.4 to 0.0 minutes. The evidence suggests a possible improvement, but includes no effect at the 5% two-sided level. Management should avoid claiming a definitive causal benefit. A pilot extension, pre-specified outcome and measurement of fuel cost and customer satisfaction would improve the decision.

Learning points. Group comparisons require comparable groups; uncertainty must accompany estimates; statistical significance is not the same as operational value; and a transparent design is more credible than a selectively reported dashboard number.

8. Misleading Conclusions and Responsible Practice

Frequent analytical failures include:

• treating correlation as causation;
• ignoring the sampling frame;
• deleting outliers without a documented reason;
• replacing missing values with zero automatically;
• using a mean for a strongly skewed distribution without context;
• performing many unreported tests until one is significant;
• interpreting a p-value without an effect size;
• using a bar chart with a truncated axis to exaggerate change;
• assuming independent observations when repeated measurements come from the same person;
• reporting only a successful analysis.

Ethical EDA includes privacy protection, data minimisation, secure handling of personal data, disclosure of uncertainty and avoidance of discriminatory interpretations. Analysts should consider whether missingness is concentrated in protected groups, whether a proxy variable encodes sensitive characteristics and whether automated decisions could produce disparate impact. Reproducibility does not require publishing confidential records; synthetic or de-identified data, documented methods and controlled access may be appropriate.

Contemporary practice increasingly combines automated profiling, anomaly detection and AI-assisted code generation with human review. Such tools can accelerate exploration but may propagate leakage, incorrect assumptions, fabricated interpretations or privacy risks. Every automated finding should be validated against the data-generating process and domain knowledge.

Chapter Summary

EDA is an evidence-building process rather than a collection of plots. Descriptive statistics summarise centre, spread and shape, while probability and sampling theory explain variation and support cautious generalisation. Means and standard deviations are sensitive to extremes; medians and IQRs can be more robust. Sampling bias cannot be repaired simply by increasing sample size. The central limit theorem concerns sampling distributions, not automatic normality of raw data.

Correlation describes association, not causation. Hypothesis tests require explicit hypotheses, assumptions and careful interpretation of p-values. Confidence intervals express estimation precision, while effect sizes and operational context determine practical importance. Multiple testing, confounding, dependence, missing data and selective reporting can produce misleading conclusions. A professional Python workflow combines data auditing, visual exploration, appropriate statistical methods, reproducibility and ethical communication.

Key Terms

Population: Complete set of units about which inference is intended.

Sample: Observed subset of a population.

Parameter: Numerical characteristic of a population.

Statistic: Numerical quantity calculated from a sample.

Mean: Arithmetic average of quantitative observations.

Median: Middle value after ordering observations.

Variance: Average squared deviation from a mean, with a specified population or sample convention.

Standard deviation: Square root of variance, in the original units.

IQR: Difference between the third and first quartiles.

Outlier: Observation unusually distant from the rest under a stated rule or model.

Probability: Numerical representation of uncertainty about events.

Conditional probability: Probability of one event given another event.

Sampling bias: Systematic distortion caused by how observations are selected or measured.

Standard error: Estimated variability of a statistic across repeated samples.

Correlation: Standardised measure of linear association between two variables.

Confounding: Distortion of an association by a third variable.

Null hypothesis: Reference claim evaluated by a statistical procedure.

p-value: Probability, under the null and assumptions, of a result at least as extreme as observed.

Confidence interval: Interval produced by a procedure designed to cover a parameter at a stated long-run rate.

Statistical power: Probability of detecting a specified effect under a stated design.

Review Questions

5 Short Answer Questions

1. Why can the median be preferable to the mean for highly skewed data?
2. Distinguish a parameter from a statistic.
3. What does a p-value measure, and what does it not measure?
4. Why does increasing sample size not necessarily remove sampling bias?
5. State one reason why correlation does not establish causation.

5 Long Answer Questions

1. Explain a complete Python-based EDA workflow from data audit to communication of results.
2. Compare mean, median, standard deviation and IQR, including their sensitivity to outliers.
3. Explain conditional probability and Bayes’ theorem using a diagnostic-testing example.
4. Discuss Type I error, Type II error, statistical power and practical significance.
5. Explain how missingness, confounding and dependence can invalidate a group comparison.

3 Analytical / Application Questions

1. A company reports a significant correlation between advertising expenditure and sales. Design an analysis that investigates whether the relationship could be confounded by seasonality or promotional discounts.
2. A survey of customers is completed voluntarily through a company website. Evaluate the likely sources of bias and propose a more defensible sampling strategy.
3. An experiment tests 30 outcomes and reports the smallest p-value of 0.03. Explain why this result requires caution and recommend an appropriate reporting and replication plan.

Multiple Choice Questions with Answers

Question 1

Which measure is generally most resistant to an extreme observation?

A. Mean

B. Median

C. Variance

D. Range

Correct Answer: B. Median

Question 2

If two events A and B are independent, which relationship is correct?

A. P(A ∩ B) = P(A) + P(B)

B. P(A | B) = 0

C. P(A ∩ B) = P(A)P(B)

D. P(A ∪ B) = P(A)P(B)

Correct Answer: C. P(A ∩ B) = P(A)P(B)

Question 3

What does the standard error of a sample mean primarily describe?

A. The spread of individual observations

B. The variability of the mean across repeated samples

C. The probability that the null hypothesis is true

D. The largest observed value

Correct Answer: B. The variability of the mean across repeated samples

Question 4

A correlation of zero necessarily means that two variables have no relationship of any kind.

A. True because correlation measures all relationships

B. True when sample size exceeds 30

C. False because a nonlinear relationship may exist

D. False only when both variables are categorical

Correct Answer: C. False because a nonlinear relationship may exist

Question 5

For a two-sided one-sample mean test with unknown population standard deviation, which statistic is appropriate under the stated assumptions?

A. t = (x̄ − μ₀)/(s/√n)

B. t = (x̄ + μ₀)/(s√n)

C. t = (s − μ₀)/(x̄/√n)

D. t = (x̄ − μ₀)/(n/s)

Correct Answer: A. t = (x̄ − μ₀)/(s/√n)

Question 6

Which statement about a p-value is correct?

A. It is the probability that the null hypothesis is true

B. It is the probability that the observed result was caused by chance alone

C. It is calculated under the null hypothesis and measures extremeness of the result

D. It measures the practical importance of an effect

Correct Answer: C. It is calculated under the null hypothesis and measures extremeness of the result

Question 7

If a test is repeated across many independent outcomes at α = 0.05, what generally happens to the chance of at least one false positive?

A. It decreases to zero

B. It increases

C. It remains exactly 0.05

D. It becomes equal to statistical power

Correct Answer: B. It increases

Question 8

Which issue occurs when a third variable influences both variables in an observed association?

A. Confounding

B. Randomisation

C. Stratified sampling

D. Measurement precision

Correct Answer: A. Confounding

Question 9

What is the sample variance formula’s denominator under the usual unbiased-estimation convention?

A. n + 1

B. n

C. n − 1

D. √n

Correct Answer: C. n − 1

Question 10

A 95% confidence interval is best interpreted, in the frequentist framework, as:

A. A 95% probability that the fixed parameter lies in this particular interval

B. An interval-generating procedure that covers the parameter in about 95% of repeated samples

C. Proof that the null hypothesis is false

D. A guarantee that 95% of observations lie inside the interval

Correct Answer: B. An interval-generating procedure that covers the parameter in about 95% of repeated samples