Exploratory Data Analysis and Visualization

Introduction

Exploratory Data Analysis (EDA) is the systematic process of examining, summarising and visualising data before formal modelling. Its purpose is not merely to produce attractive charts. EDA helps a data scientist understand the structure and quality of a dataset, identify unusual observations, discover relationships, test whether assumptions are reasonable and formulate useful questions for further analysis.

In practice, EDA connects raw data with informed decision-making. A business analyst may use it to investigate declining sales; a public-health analyst may examine disease patterns; and a machine-learning practitioner may use it to detect missing values, class imbalance and data leakage. EDA is therefore both a technical activity and a reasoning process. The analyst must decide what to calculate, which visual representation is appropriate, and whether an observed pattern is meaningful or an artefact of poor measurement or biased sampling.

A useful EDA workflow is:

1. Clarify the question and understand the data-generating context.
2. Inspect data types, dimensions, labels and metadata.
3. Assess missing values, duplicates, invalid values and measurement errors.
4. Calculate suitable summary measures.
5. Examine individual-variable distributions.
6. Explore relationships between variables using tables and graphics.
7. Investigate anomalies and possible explanations.
8. Communicate findings honestly, including uncertainty and limitations.

EDA is exploratory rather than confirmatory. An apparent relationship found during exploration should normally be tested with appropriate statistical methods or validated using new data before it is treated as evidence of a general effect.

Learning Objectives

By the end of this chapter, you should be able to:

• distinguish categorical, ordinal, discrete and continuous variables;
• calculate and interpret common measures of centre, spread and position;
• describe distributions using tables, histograms, box plots and density plots;
• construct and interpret frequency tables and cross-tabulations;
• select effective visualisations for different analytical questions;
• identify outliers, anomalies, skewness and possible data-quality problems;
• explain why association does not establish causation;
• recognise misleading visual presentation and improve chart integrity;
• conduct an EDA workflow using reproducible and ethically responsible practices.

1. Understanding Data Before Analysis

1.1 Units of observation and variables

The unit of observation is the entity represented by one row, such as a customer, transaction, patient or day. A variable is a measured characteristic of that entity. For example, a customer dataset might contain customer ID, region, age, annual income, number of purchases and subscription status.

The meaning of a variable depends on its measurement scale:

Data may also be classified as discrete, where values are countable (for example, number of complaints), or continuous, where values can theoretically take any value in an interval (for example, delivery time). These classifications guide the choice of summary and chart.

1.2 Data quality inspection

Before calculating statistics, inspect the dataset’s shape, data types and permitted values. Typical checks include:

• Are dates stored as dates rather than text?
• Are monetary values expressed in the same currency and time period?
• Are category labels consistent, such as “North” and “north”?
• Are there duplicate records?
• Are impossible values present, such as a negative age or a percentage above 100%?
• Does a missing value mean “not recorded”, “not applicable” or “zero”?

Replacing every missing value with zero is particularly dangerous because zero usually has a substantive meaning. Data cleaning decisions should be documented because they can change analytical conclusions.

2. Summary Measures

Summary measures compress a dataset into interpretable numerical descriptions. They should be used together with plots because a single statistic can hide important structure.

2.1 Measures of centre

For observations x₁, x₂, ..., x_n, the arithmetic mean is:

Here, n is the number of observed values and x_i is observation i. The mean uses every value and is appropriate for quantitative data when extreme values do not dominate interpretation. It is sensitive to outliers.

The median is the middle ordered value. If n is odd, it is the value at position (n+1)/2. If n is even, it is commonly the mean of the two central values. The median is less affected by extreme observations and is often preferable for income, waiting time and house prices.

The mode is the most frequently occurring value or category. It is especially useful for nominal variables, for which a mean is not meaningful.

2.2 Measures of spread

The range is:

It is easy to understand but depends only on two observations. The interquartile range (IQR) measures the spread of the middle half of the data:

where Q₁ is the first quartile and Q₃ is the third quartile. Exact quartile algorithms can differ slightly, especially for small samples, so software conventions should be recorded.

For a sample, the sample variance is:

The sample standard deviation is:

The denominator n−1 is used when estimating population variability from a sample and is called Bessel’s correction. If the analyst has the complete population rather than a sample, population variance uses n in the denominator. Variance is expressed in squared units, whereas standard deviation has the same units as the original variable.

The coefficient of variation is:

Often it is reported as CV × 100%. It compares relative variability, but it is inappropriate when the mean is zero or close to zero and is difficult to interpret for variables that can be negative.

2.3 Position and standardisation

Percentiles divide ordered data into relative positions. The 90th percentile is a value at or below which approximately 90% of observations fall. Quartiles are the 25th, 50th and 75th percentiles.

A z-score expresses the distance from the mean in standard-deviation units:

For a sample, x̄ and s are the sample mean and standard deviation. A positive z-score indicates a value above the mean. Standardisation is useful for comparing variables measured in different units, but it does not make a skewed distribution normal.

Worked Example: Delivery Times

Suppose five delivery times, in minutes, are 20, 22, 25, 25 and 48. The mean is:

x̄ = (20 + 22 + 25 + 25 + 48)/5 = 140/5 = 28 minutes.

The median is 25 minutes, the mode is 25 minutes and the range is 28 minutes. The mean is higher than the median because the 48-minute delivery pulls it upward. Reporting only the mean would conceal the fact that a typical delivery took about 25 minutes while at least one delivery was unusually slow.

3. Distributions and Unusual Observations

A distribution describes how values are arranged. Important features include centre, spread, shape, modality and tails.

• A symmetric distribution has similar shapes on both sides of its centre.
• A right-skewed distribution has a long upper tail; income and waiting time often show this pattern.
• A left-skewed distribution has a long lower tail.
• A unimodal distribution has one prominent peak.
• A bimodal distribution has two prominent peaks, which may indicate two subgroups.
• Heavy tails indicate more extreme observations than would be expected under a light-tailed distribution.

An outlier is an observation unusually distant from the rest of the data. An outlier may be a recording error, a valid rare event, a member of a different population or evidence of a process failure. It should not be removed simply because it is inconvenient.

A common exploratory rule identifies observations below Q₁ − 1.5 × IQR or above Q₃ + 1.5 × IQR as potential outliers. This is a screening rule, not proof that an observation is erroneous. The 1.5 multiplier is conventional and should not be treated as a universal scientific law.

4. Tables and Cross-Tabulations

A frequency table lists categories or intervals and their counts. Relative frequency is:

It may be multiplied by 100 to obtain a percentage. For quantitative data, intervals should be clearly defined and should not overlap. For example, age groups might be 18–24, 25–34 and 35–44.

A cross-tabulation, or contingency table, summarises two categorical variables jointly. For example, a retailer might compare subscription status by region.

The North subscription proportion is 180/300 = 0.60, or 60%. The South proportion is 140/300 ≈ 0.467, or approximately 46.7%. Comparing row percentages is appropriate because the row totals are equal. When group sizes differ, percentages rather than raw counts should normally be compared.

Cross-tabulations can be extended with conditional percentages and visualised with grouped or stacked bar charts. They show association, not causation. A difference between regions may be explained by age, product mix, marketing exposure or another confounding variable.

5. Visualising Data Effectively

A chart should answer a particular question. The best chart is not necessarily the most elaborate one; it is the one that makes the relevant comparison accurate and easy.

5.1 Bar charts and histograms

A bar chart is used for categorical values. Bars are separated because categories are distinct. A histogram is used for quantitative values grouped into numerical intervals, so bars normally touch. Confusing these charts can misrepresent the data.

Histogram conclusions depend on bin width and bin boundaries. Very narrow bins may create distracting noise; very wide bins may hide meaningful structure. Analysts should inspect more than one reasonable binning choice and explain the choice when it affects interpretation.

5.2 Box plots

A box plot commonly displays the median, first and third quartiles, the IQR and potential outliers. It is efficient for comparing distributions across groups but does not show every feature, such as multiple modes or sample size. Box plots should therefore be supplemented with points or distribution plots when group sizes are small.

5.3 Scatter plots and correlation

A scatter plot displays paired quantitative observations. It can reveal positive, negative, nonlinear or absent association, clusters and influential points. A line of best fit may summarise a linear trend, but it should not obscure curvature or group structure.

Pearson’s correlation coefficient is:

Here, x_i and y_i are paired observations, and x̄ and ȳ are their sample means. The value lies between −1 and +1 when both variables have nonzero variation. Pearson correlation measures linear association; a value near zero does not rule out a strong nonlinear relationship. It is sensitive to outliers and should be interpreted alongside the scatter plot.

Correlation does not establish causation. For example, ice-cream sales and swimming incidents may both rise during hot weather. Temperature is a possible confounding variable.

5.4 Time-series charts

Time should generally be placed on the horizontal axis in chronological order. A line chart can reveal trend, seasonality, cycles and sudden changes. Analysts should distinguish a genuine change from a change in data collection, such as a new recording system or altered definition of a transaction.

6. Avoiding Misleading Presentation

Ethical visualisation requires truthful visual encoding and adequate context. Common problems include:

• truncating a bar-chart axis so small differences appear enormous;
• using three-dimensional effects that distort lengths or areas;
• comparing counts when group sizes differ substantially;
• using inconsistent scales across panels;
• selecting only a convenient time period;
• hiding missing data or excluding inconvenient observations without explanation;
• using colour combinations inaccessible to people with colour-vision deficiencies;
• displaying excessive decimal places that imply unjustified precision;
• presenting a correlation as proof of causation;
• using a dual-axis chart that encourages unrelated trends to appear connected.

Bar charts comparing magnitudes should normally begin at zero because bar length encodes quantity. For line charts, a nonzero baseline may sometimes be acceptable when the purpose is to show small changes, but the scale and context must be clear. Labels should include units, time period, population and, where relevant, sample size.

Colour should have a meaningful role. Sequential palettes are appropriate for ordered magnitude; diverging palettes are useful when values depart from a meaningful midpoint; categorical palettes should distinguish discrete groups without implying an order. Accessibility should be tested through contrast and non-colour cues such as labels or patterns.

7. A Practical EDA Workflow

A professional workflow is iterative rather than strictly linear.

Step 1: Define the analytical question

Clarify whether the task concerns comparison, distribution, relationship, composition, change over time or anomaly detection. A vague question produces unfocused charts.

Step 2: Establish data provenance

Record the source, collection method, date range, population, sampling process, definitions and known limitations. Provenance is essential when findings may affect people or organisational decisions.

Step 3: Profile the data

Inspect row and column counts, variable types, unique categories, missingness, duplicates and summary measures. Review both automated reports and actual records.

Step 4: Visualise one variable at a time

Use frequency tables for categories and histograms or box plots for numerical variables. Examine skewness, gaps, unusual values and possible subgroups.

Step 5: Explore relationships

Use cross-tabulations for categorical variables, grouped summaries for numerical variables by category, scatter plots for paired numerical variables and line charts for time-indexed data.

Step 6: Investigate anomalies

Trace unusual records back to source systems. Determine whether they represent errors, rare valid cases or process changes. Document any correction, exclusion or transformation.

Step 7: Validate interpretations

Check whether patterns persist under reasonable choices of scale, bin width, subgroup definition and missing-data treatment. Compare results with domain knowledge.

Step 8: Communicate and reproduce

Save code, data versions, chart settings and decisions. A reproducible analysis allows another person to obtain the same result and assess the reasoning.

Applied Case: Investigating Customer Churn

The following is an illustrative case. A subscription company observes that monthly churn appears higher among customers using its mobile application than among customers using the website.

Background and problem

The company has 12,000 customer records containing acquisition channel, application use, contract type, tenure, monthly fee and churn status. Management asks whether mobile-app use is associated with churn and whether a retention campaign should target app users.

Analysis

The analyst first checks definitions. “Churn” means cancellation within the 30-day reporting period, and each row represents one active customer at the start of that period. Duplicate customer IDs are removed after investigation. Missing contract types are not recoded as “monthly”.

A frequency table shows that 3,600 customers use the application and 8,400 do not. The churn rates are:

The crude comparison suggests higher churn among app users. However, cross-tabulation by contract type reveals that app users are more likely to have flexible monthly contracts, while non-users are more likely to have annual contracts. Box plots also show that app users have shorter median tenure. A scatter plot of tenure and monthly fee is inspected for unusual values, and a line chart confirms that a recent application redesign coincided with a temporary increase in support complaints.

Approach and solution

The analyst does not conclude that application use causes churn. Instead, the company segments the analysis by contract type and tenure bands, investigates redesign-related complaints and conducts usability interviews. A controlled retention experiment is proposed: eligible customers are randomly assigned to receive either an improved onboarding message or the existing message, with churn measured over a defined follow-up period.

Outcome and learning points

EDA changed the managerial question from “Should app users be targeted?” to “Which combination of contract, tenure and app experience is associated with churn?” The analysis demonstrated the value of base rates, cross-tabulations, distributions and contextual investigation. It also showed why an observed group difference is not automatically a causal effect.

Limitations, Ethics and Governance

EDA can reveal patterns, but it cannot by itself establish causal mechanisms, eliminate sampling bias or guarantee predictive performance. Small samples may produce unstable summaries, and selective missingness can distort every chart. Aggregation may conceal important subgroup differences, a problem known as ecological fallacy.

Privacy must be protected when exploring personal data. Analysts should minimise data access, remove direct identifiers where possible, restrict exports and follow applicable privacy law and organisational policy. Sensitive variables such as health, ethnicity or financial status require special care. A chart that exposes a small identifiable group can create privacy risk even if names are absent.

Fairness should be assessed when EDA informs decisions about credit, employment, education or access to services. Differences between groups may reflect historical disadvantage, unequal measurement or unequal opportunity rather than individual characteristics. Analysts should document variable definitions, exclusions, transformations and the intended use of findings.

Modern analytical tools and AI systems can automatically generate profiles, charts and anomaly flags. These tools improve speed but do not replace domain reasoning. Automated systems may choose unsuitable scales, infer incorrect data types, amplify biased historical patterns or produce confident explanations unsupported by evidence. Human review, provenance tracking and reproducible code remain necessary.

Best-Practice Checklist

• Start with the decision or question, not with a preferred chart.
• Inspect raw records and metadata before trusting summaries.
• Match the statistic and visualisation to the measurement scale.
• Report both counts and percentages when group sizes differ.
• Use robust summaries such as the median and IQR for skewed data.
• Investigate, rather than automatically delete, outliers.
• Label axes, units, populations, dates and sample sizes.
• Use accessible colour and avoid unnecessary decoration.
• Distinguish association from causation.
• Record all cleaning and visualisation decisions.
• Test whether conclusions change under reasonable analytical choices.
• Protect privacy and consider fairness throughout the workflow.

Chapter Summary

Exploratory Data Analysis is an iterative process for understanding data quality, distributions, relationships and anomalies before formal modelling. Effective EDA begins with the unit of observation, variable type, provenance and measurement context. Means, medians, modes, ranges, quartiles, IQRs, variances and standard deviations provide complementary descriptions of centre and spread. Distributional shape, skewness and outliers must be examined visually as well as numerically.

Frequency tables and cross-tabulations are useful for categorical data, while histograms, box plots, scatter plots and line charts support different analytical purposes. Chart selection should follow the question and measurement scale. Honest visualisation requires accurate axes, meaningful labels, appropriate aggregation, accessible colour and explicit context. Correlation and group differences indicate association but do not prove causation. Finally, reproducibility, privacy, fairness and careful documentation are essential professional responsibilities in EDA.

Key Terms

Exploratory Data Analysis: Systematic examination of data to discover structure, quality issues, patterns and relationships.

Unit of observation: The entity represented by one record or row.

Variable: A measured characteristic recorded for each observation.

Categorical variable: A variable whose values represent groups or labels.

Continuous variable: A quantitative variable that can take values across an interval.

Mean: The arithmetic average of numerical observations.

Median: The middle value of ordered observations.

Mode: The most frequently occurring value or category.

Variance: The average squared deviation from the mean, with a sample version using denominator n−1.

Standard deviation: The square root of variance, expressed in the original units.

Quartile: A value dividing ordered data into portions, commonly at 25%, 50% or 75%.

Interquartile range: The difference between the third and first quartiles.

Outlier: An observation unusually distant from other observations.

Frequency table: A table showing counts or proportions for values or categories.

Cross-tabulation: A table jointly summarising two categorical variables.

Histogram: A plot showing the distribution of quantitative values across intervals.

Box plot: A compact display of quartiles, median, spread and potential outliers.

Scatter plot: A graph of paired quantitative observations.

Correlation: A numerical measure of association, commonly linear association for Pearson’s r.

Confounding variable: A variable related to both an apparent explanatory variable and an outcome that can distort their association.

Review Questions

Short Answer Questions

1. What is the difference between a bar chart and a histogram?
2. Why is the median often preferred to the mean for highly skewed data?
3. What information does the interquartile range provide?
4. Why should percentages be used when comparing groups of different sizes?
5. Why does correlation not prove causation?

Long Answer Questions

1. Explain a complete EDA workflow from data inspection to communication of results.
2. Compare nominal, ordinal, interval and ratio measurement scales and discuss suitable summaries for each.
3. Explain how histograms, box plots and scatter plots reveal different features of data.
4. Discuss common forms of misleading visualisation and recommend practices for preventing them.
5. Explain how missing data, outliers and data provenance can affect EDA conclusions.

Analytical / Application Questions

1. A dataset contains household incomes with a few extremely large values. Which measures and charts would you use to describe the data, and why?
2. A hospital reports that one ward has twice as many incidents as another, but the wards have very different patient populations. How would you use cross-tabulation and visualisation to analyse the claim?
3. A scatter plot shows a strong relationship between advertising expenditure and sales. Design an EDA plan to investigate whether the relationship may be confounded or misleading.

Multiple Choice Questions with Answers

Question 1

Which measure is least affected by an extreme outlier?

A. Mean

B. Median

C. Range

D. Variance

Correct Answer: B. Median

Question 2

What does the interquartile range measure?

A. The total number of observations

B. The difference between the maximum and minimum

C. The spread of the middle 50% of observations

D. The average distance from zero

Correct Answer: C. The spread of the middle 50% of observations

Question 3

Which chart is most appropriate for examining the relationship between two quantitative variables?

A. Scatter plot

B. Pie chart

C. Bar chart

D. Frequency table

Correct Answer: A. Scatter plot

Question 4

A histogram differs from a bar chart because a histogram generally:

A. Uses only categorical variables

B. Has touching bars for numerical intervals

C. Must begin at zero

D. Shows two variables simultaneously

Correct Answer: B. Has touching bars for numerical intervals

Question 5

For five delivery times of 20, 22, 25, 25 and 48 minutes, what is the median?

A. 22 minutes

B. 25 minutes

C. 28 minutes

D. 48 minutes

Correct Answer: B. 25 minutes

Question 6

Which statement about Pearson correlation is correct?

A. It proves that one variable causes another

B. It measures only categorical association

C. It measures the strength and direction of linear association

D. It is unaffected by outliers

Correct Answer: C. It measures the strength and direction of linear association

Question 7

In a cross-tabulation, why are row percentages useful?

A. They remove all sampling bias

B. They allow comparison of proportions within each row group

C. They convert categories into continuous variables

D. They guarantee causation

Correct Answer: B. They allow comparison of proportions within each row group

Question 8

Which practice is most likely to mislead viewers of a bar chart?

A. Including units on the axis

B. Starting the magnitude axis far above zero without explanation

C. Labelling each category

D. Reporting the observation period

Correct Answer: B. Starting the magnitude axis far above zero without explanation

Question 9

An observation below Q₁ − 1.5 × IQR is commonly treated as:

A. A guaranteed data-entry error

B. A potential outlier requiring investigation

C. The median

D. A missing value

Correct Answer: B. A potential outlier requiring investigation

Question 10

Which is an appropriate ethical practice during EDA?

A. Delete all unusual records automatically

B. Hide missing values from the audience

C. Document cleaning decisions and protect personal data

D. Use colour alone to distinguish groups

Correct Answer: C. Document cleaning decisions and protect personal data