Exploratory Data Analysis and Business Statistics

Introduction

Business analytics begins with a deceptively simple question: what is happening in the data? Exploratory data analysis (EDA) provides a disciplined way to answer that question before formal modelling or decision-making. It combines numerical summaries, visual inspection, distributional reasoning, sampling, segmentation, correlation and hypothesis testing.

EDA is not merely the production of charts. It is an investigative process used to discover structure, detect errors, identify unusual observations, evaluate assumptions and generate business questions. In Python, libraries such as pandas, NumPy, Matplotlib, Seaborn and SciPy support this process, but software does not replace statistical judgement. A statistically impressive result can still be operationally irrelevant, biased or misleading.

This chapter develops a practical foundation for analysing business data. Examples use a fictional retail dataset containing customer transactions, order values, customer segments and marketing exposure.

Learning objectives

By the end of this chapter, you should be able to:

• distinguish populations, samples, parameters and statistics;
• calculate and interpret common measures of centre and variation;
• describe distributions and identify outliers and data-quality problems;
• use sampling concepts to evaluate representativeness;
• segment observations using meaningful business variables;
• interpret covariance, correlation and association cautiously;
• formulate and conduct basic hypothesis tests;
• implement an EDA workflow in Python; and
• communicate statistical findings with appropriate limitations.

1. The EDA workflow

A reliable EDA process normally follows these stages:

1. Define the business question. For example, “Did the redesigned email campaign increase average order value?” is more useful than “Explore the data.”
2. Understand the data-generating process. Identify how, when and why records were created.
3. Inspect structure and quality. Check data types, missing values, duplicates, impossible values and inconsistent categories.
4. Summarise individual variables. Examine counts, centre, spread and distributions.
5. Explore relationships. Compare groups, plot associations and investigate time patterns.
6. Formulate testable explanations. A pattern observed in EDA may motivate a hypothesis test.
7. Communicate evidence and uncertainty. Separate what the data show from what they merely suggest.

A typical Python start is:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

orders = pd.read_csv("orders.csv")
print(orders.head())
print(orders.info())
print(orders.isna().sum())
print(orders.describe(include="all"))

describe() is a useful first inspection, not a complete analysis. It may conceal skewness, multiple subgroups, data-entry errors or changes over time.

2. Populations, samples and measurement

A population is the complete set of units relevant to a question, such as all customers who purchased during a financial year. A sample is a subset observed or selected for analysis. A parameter describes a population; a statistic describes a sample.

Common measurement scales influence the analyses that are appropriate:

The distinction between a transaction and a customer is especially important. If one customer makes ten purchases, treating orders as independent customers can overrepresent that customer and understate uncertainty.

3. Summary statistics

3.1 Measures of centre

For observations x₁, x₂, ..., x_n, the arithmetic mean is:

Here, n is the number of observations and x̄ is the sample mean. It is appropriate for quantitative data when extreme values do not dominate interpretation. Its units are the same as the original variable.

The median is the middle ordered observation, or the average of the two middle observations when n is even. It is resistant to extreme values and is often preferable for income, order value and waiting time, which are commonly right-skewed.

The mode is the most frequent value or category. It is especially useful for categorical variables, although a dataset may have more than one mode.

For positive observations, the geometric mean is:

It is applicable when all x_i > 0 and is useful for multiplicative growth rates. It should not be used as a routine replacement for the arithmetic mean.

3.2 Measures of variation

The range is max(x_i) - min(x_i). It is simple but highly sensitive to extremes.

The sample variance is:

The sample standard deviation is s = √s². The denominator n − 1 provides the usual unbiased estimator of population variance under random sampling. Standard deviation has the same units as the observations, whereas variance has squared units.

The interquartile range is:

where Q₁ and Q₃ are the 25th and 75th percentiles. A common exploratory rule flags values below Q₁ − 1.5IQR or above Q₃ + 1.5IQR as potential outliers. This is a screening rule, not proof that an observation is erroneous.

The coefficient of variation is:

It is meaningful mainly for ratio-scale variables with a positive, non-zero mean. Multiplying by 100 expresses it as a percentage. It supports comparison of relative variability across measures with different units.

Worked example: order values

Suppose five order values are £20, £25, £25, £30 and £100. The mean is:

x̄ = (20 + 25 + 25 + 30 + 100)/5 = 40

The median is £25. The mean is substantially higher because the £100 order is influential. The ordered data have Q₁ = 22.5 and Q₃ = 65, so IQR = 42.5. The upper exploratory fence is 65 + 1.5(42.5) = 128.75, meaning £100 is not flagged by this particular rule, even though it strongly affects the mean. This illustrates why analysts should use several summaries rather than an automatic outlier rule.

4. Distributions and visual analysis

A distribution describes how values are allocated across possible outcomes. Important features include:

• Centre: a typical or central value;
• Spread: the degree of variability;
• Shape: symmetry, skewness, modality and tails; and
• Unusual observations: gaps, clusters and potential outliers.

A histogram displays frequencies across numeric intervals. Bin width affects its appearance, so conclusions should not depend on one arbitrary choice. A box plot displays the median, quartiles, interquartile range and potential outliers. A density plot gives a smoothed estimate and should be interpreted cautiously, especially for small samples.

A right-skewed distribution has a longer tail toward larger values; typically the mean exceeds the median. A left-skewed distribution has a longer lower tail. A bimodal distribution may indicate two customer populations, such as retail and wholesale buyers, and should not automatically be summarised by one average.

For a normally distributed variable with mean μ and standard deviation σ, the empirical rule states approximately 68%, 95% and 99.7% of observations lie within one, two and three standard deviations of the mean, respectively. This rule applies only approximately to distributions that are reasonably normal; it is not a universal law.

Python examples:

sns.histplot(data=orders, x="order_value", bins=30, kde=True)
plt.xlabel("Order value (£)")
plt.show()

sns.boxplot(data=orders, x="customer_segment", y="order_value")
plt.xticks(rotation=30)
plt.show()

Charts should include units, meaningful labels, an appropriate scale and a clear business interpretation. Truncated axes, excessive decimal places and three-dimensional effects can distort perception.

5. Sampling and uncertainty

Sampling is necessary when measuring every unit is expensive, slow or impossible. A simple random sample gives each population unit a known and equal selection chance. Stratified sampling divides the population into relevant strata, such as region, and samples within each. Cluster sampling selects groups, such as stores, and observes units within selected groups. Convenience sampling is easy but often biased because inclusion depends on accessibility.

A sample statistic varies from sample to sample. This variation is sampling error, not necessarily an error in data collection. A confidence interval expresses uncertainty about a population quantity under a specified sampling model.

For a sample mean, when observations are independent and the population standard deviation is unknown, a common interval is:

where t* is a critical value from the t distribution with n − 1 degrees of freedom, s is sample standard deviation and n is sample size. The formula is most defensible when the sample is random and the mean’s sampling distribution is approximately normal. Large samples can support approximation through the central limit theorem, but dependence and severe data problems remain concerns.

A 95% confidence interval does not mean there is a 95% probability that a fixed parameter lies in this particular interval. In repeated samples, a procedure constructed this way would capture the parameter approximately 95% of the time under its assumptions.

Sampling bias can arise from undercoverage, non-response, voluntary participation, survivorship and measurement design. Increasing sample size does not remove systematic bias.

6. Segmentation and group comparison

Segmentation divides observations into meaningful, relatively homogeneous groups. Business segmentation may use:

• demographics or firmographics;
• behaviour, such as purchase frequency;
• value, such as customer lifetime value;
• geography or channel; and
• needs, attitudes or engagement.

A useful segment is actionable, measurable, sufficiently large and connected to a decision. Segments should be defined using variables available before the decision being studied where possible. Using future revenue to define “high-value” customers and then claiming that a campaign predicted high value creates leakage.

A simple RFM framework uses recency, frequency and monetary value. For example, customers can be scored from 1 to 5 within each dimension and combined into interpretable profiles. Quantile-based scores are relative: a “5” means high compared with the analysis population, not necessarily high in absolute business terms.

rfm = customers.assign(
    r_score=pd.qcut(customers["recency_days"], 5, labels=[5, 4, 3, 2, 1]),
    f_score=pd.qcut(customers["orders_count"].rank(method="first"), 5,
                    labels=[1, 2, 3, 4, 5]),
    m_score=pd.qcut(customers["revenue"].rank(method="first"), 5,
                    labels=[1, 2, 3, 4, 5])
)

The direction of recency is reversed because fewer days since purchase indicates greater recency. Analysts should validate segments using stability checks, business knowledge and outcomes such as retention—not only algorithmic separation.

7. Correlation and association

Covariance describes whether two quantitative variables tend to move together, but its magnitude depends on measurement units. Pearson’s sample correlation is:

Here, r ranges from −1 to 1 when both variables have non-zero variation. Positive values indicate a linear tendency to increase together; negative values indicate an inverse linear tendency. A value near zero means little linear association, not necessarily no relationship.

Pearson correlation is sensitive to outliers and can miss nonlinear relationships. Spearman’s rank correlation assesses monotonic association using ranks and is often more robust to scale and outliers, although it also does not establish causation.

Correlation does not imply causation. A relationship may result from confounding, reverse causality, common time trends, selection bias or coincidence. For example, advertising spend and sales may correlate because both rise during seasonal peaks.

sns.scatterplot(data=orders, x="discount", y="order_value")
print(orders[["discount", "order_value"]].corr(method="pearson"))

A scatter plot should precede interpretation of a correlation coefficient. Analysts should inspect group structure, time ordering and influential points.

8. Hypothesis testing

Hypothesis testing provides a formal framework for evaluating evidence against a null hypothesis. The null hypothesis, H₀, represents a reference claim; the alternative, H₁, represents the effect or difference of interest.

A p-value is the probability, assuming H₀ and the test assumptions are correct, of obtaining a result at least as extreme as the observed result in the specified direction. It is not the probability that H₀ is true, nor the probability that the finding will be commercially important.

A conventional significance level is α = 0.05. If the p-value is less than or equal to α, the analyst rejects H₀; otherwise, the analyst fails to reject it. “Fail to reject” is not the same as proving no effect.

Comparing two means

For two independent groups, a Welch two-sample t-test is often preferred because it does not require equal population variances. Its statistic is:

where x̄₁, x̄₂ are sample means, s₁², s₂² are sample variances and n₁, n₂ are sample sizes. The test assumes independent observations and a sufficiently appropriate sampling distribution; severe skewness and dependence require caution.

For paired observations, such as the same stores before and after a policy, analyse within-pair differences rather than treating observations as independent. For categorical outcomes, a chi-square test can assess association between categorical variables when expected counts are adequate. For multiple group means, analysis of variance (ANOVA) tests whether at least one population mean differs, but significant ANOVA does not identify which groups differ without suitable follow-up comparisons.

Effect size and confidence intervals should accompany p-values. A small p-value with a trivial effect may not justify action, while a practically important effect may be estimated imprecisely in a small sample. Multiple testing increases the chance of false positives; pre-specification, correction procedures and transparent reporting are important.

Python example:

from scipy import stats

treatment = orders.loc[orders["campaign"] == "new", "order_value"]
control = orders.loc[orders["campaign"] == "old", "order_value"]
result = stats.ttest_ind(treatment, control, equal_var=False)
print(result.statistic, result.pvalue)

A test result should be reported with group sizes, means, a confidence interval or effect estimate, the test used, assumptions and the business interpretation.

9. Illustrative case: evaluating a retail email campaign

Background

A fictional online retailer introduced a personalised email campaign. Management wants to know whether the campaign increased order value and whether its effect differed by customer segment.

Problem

The analytics team has 12,000 orders over eight weeks, with fields for order value, campaign assignment, customer ID, segment, device, region and order date. Assignment was not random: marketing first targeted customers who had recently purchased.

Analysis

The team first checks duplicates, missing campaign labels, currency consistency and whether one customer contributes many orders. A histogram shows strongly right-skewed order values, while box plots reveal that wholesale customers have higher values and greater variation than retail customers. Comparing raw campaign means would therefore mix campaign exposure with segment composition.

The team aggregates order-level outcomes to the customer level for the campaign period, because the decision concerns customer response. It reports medians and means, displays distributions, and compares campaign groups within segment and region. A Welch test is used as a descriptive comparison, but the team does not describe it as causal evidence because assignment was non-random.

A regression or matched analysis could adjust for pre-campaign order history, but adjustment cannot guarantee that all confounding has been removed. The strongest future design is a randomised holdout test, subject to privacy, consent and customer-experience constraints.

Outcome

The campaign group has a higher mean order value, but the difference is concentrated among recent high-value customers. The median difference is small, and the confidence interval is wide for smaller segments. Management decides to conduct a stratified randomised experiment rather than roll out the campaign universally.

Learning points

1. The unit of analysis must match the business question.
2. Mean, median and distribution plots can lead to different conclusions.
3. Segmentation can reveal effect heterogeneity hidden in an overall average.
4. Statistical significance cannot repair selection bias.
5. A well-designed experiment is generally stronger evidence for causation than an observational comparison.

10. Practical, ethical and governance considerations

Data preparation decisions can materially affect conclusions. Missing values should be investigated rather than silently replaced. Imputation must use information available at the relevant time and should be documented. Removing outliers is justified only when they are demonstrably erroneous or outside the defined target population; deleting inconvenient valid customers is unethical.

Analysts should protect personal data through data minimisation, access controls, retention limits and pseudonymisation where appropriate. Customer segmentation can create discriminatory outcomes if protected characteristics or strong proxies are used without governance. Automated targeting should be explainable enough for responsible review, and analysts should assess disparate impact where decisions affect access, price or service.

AI-assisted analysis can generate code, summaries and visualisations, but generated output must be checked for incorrect formulas, leakage, fabricated interpretations and inappropriate statistical tests. Reproducible notebooks, versioned data definitions and audit trails improve accountability.

Best-practice checklist

• Begin with a precise decision question.
• Confirm the unit of analysis and time window.
• Profile data types, missingness, duplicates and invalid values.
• Use both numerical summaries and suitable plots.
• Compare mean and median for skewed variables.
• Investigate, rather than automatically delete, outliers.
• Examine segments before interpreting aggregate results.
• Distinguish association from causation.
• Report effect size, uncertainty and assumptions alongside p-values.
• Document every transformation and preserve reproducibility.
• Consider privacy, fairness and the consequences of deployment.

Chapter Summary

Exploratory data analysis is a structured investigation of data quality, distributions, relationships and possible explanations. Summary statistics describe centre and variation, but their interpretation depends on scale, skewness and outliers. Histograms, box plots and scatter plots expose patterns that a single number may conceal. Sampling determines how confidently results can generalise, while confidence intervals communicate estimation uncertainty. Segmentation supports targeted decisions when groups are meaningful and actionable. Correlation measures association, not causation. Hypothesis testing evaluates evidence under explicit assumptions, but p-values must be interpreted with effect sizes, confidence intervals and business relevance. Responsible analytics also requires attention to sampling bias, privacy, fairness, reproducibility and governance.

Key Terms

Population: The complete set of units relevant to a study.

Sample: A subset of a population observed for analysis.

Parameter: A numerical characteristic of a population.

Statistic: A numerical characteristic calculated from a sample.

Mean: The arithmetic average of quantitative observations.

Median: The middle value of ordered observations.

Standard deviation: A measure of typical distance from the sample mean.

Variance: The squared measure of dispersion.

Interquartile range: The difference between the 75th and 25th percentiles.

Outlier: An unusually distant observation requiring investigation.

Distribution: The pattern of values and their frequencies or probabilities.

Sampling bias: Systematic distortion caused by how observations are selected.

Confidence interval: An interval produced by a procedure designed to capture a parameter at a stated long-run rate.

Segmentation: Division of observations into meaningful groups.

Correlation: A standardised measure of linear association between two quantitative variables.

Null hypothesis: A reference claim evaluated by a statistical test.

p-value: Evidence measure calculated under the null hypothesis and test assumptions.

Effect size: A measure of the magnitude or practical importance of a difference or relationship.

Confounding: Distortion of an apparent relationship by another variable.

Data leakage: Use of information unavailable at the time a prediction or decision would be made.

Review Questions

Short Answer Questions

1. Distinguish a population parameter from a sample statistic.
2. Why can the median be more informative than the mean for order values?
3. What does the interquartile range measure?
4. Why does correlation not establish causation?
5. What does “fail to reject the null hypothesis” mean?

Long Answer Questions

1. Explain a complete EDA workflow for a customer-retention dataset, including data-quality checks and visual analysis.
2. Compare simple random, stratified, cluster and convenience sampling, including one business use and one limitation for each.
3. Explain how skewness, outliers and multimodality affect the choice of summary statistics.
4. Describe how RFM segmentation can support marketing decisions and identify risks in its construction.
5. Explain the assumptions, interpretation and limitations of a Welch two-sample t-test.

Analytical / Application Questions

1. A campaign dataset shows a large increase in mean order value but almost no change in median order value. Propose an analysis to determine what caused the difference and what decision should follow.
2. A correlation of 0.72 is found between advertising spend and sales. Identify at least three alternative explanations and design a stronger evaluation.
3. Design an ethical and statistically defensible experiment to test personalised pricing, specifying the unit of randomisation, outcome measure, segmentation checks and governance safeguards.

Multiple Choice Questions with Answers

Question 1

Which measure is generally most resistant to an extreme order value?

A. Mean

B. Median

C. Variance

D. Range

Correct Answer: B. Median

Question 2

For a sample of five values 20, 25, 25, 30 and 100, what is the arithmetic mean?

A. 25

B. 30

C. 40

D. 100

Correct Answer: C. 40

Question 3

What does a 95% confidence interval describe under repeated use of the same valid procedure?

A. A 95% probability that the fixed parameter changes

B. The long-run capture rate of the procedure

C. A 95% probability that the null hypothesis is true

D. The probability that every observation is correct

Correct Answer: B. The long-run capture rate of the procedure

Question 4

Which sampling method divides a population into groups such as regions and samples within each group?

A. Convenience sampling

B. Stratified sampling

C. Voluntary-response sampling

D. Systematic deletion

Correct Answer: B. Stratified sampling

Question 5

A correlation close to zero indicates:

A. No possible relationship of any kind

B. No linear association detected

C. That one variable causes the other

D. That both variables have equal means

Correct Answer: B. No linear association detected

Question 6

In hypothesis testing, a p-value is calculated:

A. Assuming the null hypothesis and test assumptions are correct

B. As the probability that the alternative is true

C. As the practical value of a business intervention

D. Without reference to a sampling distribution

Correct Answer: A. Assuming the null hypothesis and test assumptions are correct

Question 7

Which test is often appropriate for comparing two independent means without assuming equal variances?

A. Welch two-sample t-test

B. Paired t-test only

C. Chi-square goodness-of-fit test

D. Pearson correlation test

Correct Answer: A. Welch two-sample t-test

Question 8

Which issue is a form of data leakage?

A. Using a training feature available before prediction

B. Using future revenue to define a segment used to predict that revenue

C. Checking for missing values

D. Reporting a confidence interval

Correct Answer: B. Using future revenue to define a segment used to predict that revenue

Question 9

If a distribution is strongly right-skewed, which relationship is commonly observed?

A. Mean is often greater than median

B. Mean is always equal to median

C. Median must be greater than the maximum

D. There can be no outliers

Correct Answer: A. Mean is often greater than median

Question 10

Which statement about a statistically significant result is correct?

A. It proves the effect is commercially important

B. It proves the study is unbiased

C. It indicates evidence against the null under the stated assumptions

D. It establishes causation in an observational study

Correct Answer: C. It indicates evidence against the null under the stated assumptions