Exploratory Data Analysis and Visualization

Introduction

Exploratory Data Analysis (EDA) is the systematic investigation of a dataset before formal modelling or decision-making. It combines summary statistics, visual methods, data-quality checks and contextual reasoning to reveal distributions, relationships, unusual observations and possible errors. EDA does not prove causal relationships; rather, it helps analysts formulate useful questions, select appropriate methods and communicate evidence clearly.

In professional data science, EDA is both an analytical activity and a communication activity. A data scientist may use detailed plots and diagnostic statistics to understand a dataset, while a manager may need a concise dashboard showing trends, risks and opportunities. Effective work therefore requires technical accuracy, visual literacy and awareness of the audience.

Learning Objectives

By the end of this chapter, you should be able to:

• distinguish univariate, bivariate and multivariate EDA;
• calculate and interpret important summary statistics;
• select appropriate charts for distributions, comparisons, relationships and time series;
• identify skewness, outliers, missingness and possible data errors;
• use correlation carefully and distinguish association from causation;
• design accurate, accessible dashboards and data stories;
• communicate analytical findings to technical and managerial audiences;
• evaluate ethical, operational and governance risks in visualization.

1. The Role and Workflow of EDA

A useful EDA workflow is iterative rather than strictly linear:

1. Understand the context. Identify the decision, population, unit of observation, time period and definitions of variables.
2. Inspect structure. Examine data types, unique values, duplicate records, missing values and measurement units.
3. Assess quality. Check validity, completeness, consistency, timeliness and possible selection bias.
4. Summarize variables. Use appropriate numerical and graphical summaries.
5. Investigate relationships. Compare groups, examine associations and analyse temporal or spatial patterns.
6. Question anomalies. Determine whether unusual values are errors, rare valid cases or evidence of an important process.
7. Document findings. Record transformations, assumptions, exclusions and interpretations.
8. Communicate appropriately. Build a visual explanation suited to the audience and decision.

EDA should be performed on the data actually used for analysis. If records are filtered, transformed or aggregated, the analyst should state how and why. Repeatedly examining data until a preferred result appears can produce misleading conclusions; exploratory findings should be confirmed with suitable statistical or experimental methods where necessary.

2. Data Types and Measurement Scales

The variable type determines which summaries and charts are valid.

Measurement scales also matter. Ratios such as revenue and distance have a meaningful zero, whereas temperature in Celsius does not. Averages and differences must therefore be interpreted in context. A categorical code such as 1 = bronze, 2 = silver and 3 = gold is not automatically a numerical measurement; calculating its mean may be inappropriate.

3. Summary Statistics

3.1 Counts, proportions and rates

For category i, its proportion is:

where n_i is the count in category i and n is the total number of observations. The proportion is unitless and lies between 0 and 1. A percentage is 100 × p_i percent. Rates require a clearly defined denominator and often a time or population unit, such as defects per 1,000 units.

3.2 Measures of central tendency

For observations x₁, x₂, ..., x_n, the arithmetic mean is:

The mean is appropriate when values are numerical and the distribution is not excessively skewed or dominated by extreme values. It has the same units as the observations.

The median is the middle ordered value. For an even number of observations, it is commonly the average of the two middle values. The median is more resistant to extreme observations than the mean.

The mode is the most frequent value or category. It is particularly useful for categorical data, although a dataset may have more than one mode or no unique mode.

The weighted mean is:

where w_i is a non-negative weight. The weights must represent a justified basis such as sampling design, exposure or quantity. An unjustified weighted mean can distort results.

3.3 Measures of spread

The range is the maximum minus the minimum. It is simple but strongly affected by extremes.

The sample variance is:

The sample standard deviation is:

Here n is the sample size and x̄ is the sample mean. The denominator n − 1 is used when estimating population variability from a sample under standard statistical assumptions. Variance has squared units, while standard deviation has the original units.

Quantiles divide ordered data into cumulative proportions. The first quartile, Q₁, is the 25th percentile; the median is Q₂; and the third quartile, Q₃, is the 75th percentile. The interquartile range is:

A commonly used exploratory rule flags values below Q₁ − 1.5 × IQR or above Q₃ + 1.5 × IQR as potential outliers. This is a screening convention, not proof that a value is erroneous.

The coefficient of variation is:

It is dimensionless and is meaningful mainly when the mean is positive and reasonably far from zero. It should not be used indiscriminately for variables with negative, zero or arbitrary-origin values.

3.4 Worked example: delivery times

Suppose five delivery times, in minutes, are 20, 22, 23, 25 and 60. The mean is:

The median is 23 minutes. The mean is much larger because of the 60-minute delivery. The ordered data have Q₁ = 22 and Q₃ = 25 under a common percentile convention, so IQR = 3 minutes. The upper screening boundary is 25 + 1.5 × 3 = 29.5 minutes; therefore 60 minutes is a potential outlier. The analyst should investigate whether it represents a service failure, a special route or a recording error rather than deleting it automatically.

4. Visual Exploration

4.1 Univariate displays

A bar chart compares counts or percentages across discrete categories. Bars should generally be separated, categories should be labelled clearly, and ordering should support comparison.

A histogram displays the distribution of numerical values using contiguous bins. Bin width affects interpretation: very narrow bins can show noise, while very wide bins can conceal structure. The horizontal axis must identify the measurement and units.

A density plot provides a smoothed estimate of a distribution. Smoothing can conceal local features, so it should usually be supported by raw observations or a histogram.

A box plot displays the median, quartiles and potential extreme observations. It is useful for comparing distributions across groups but hides the detailed shape and sample size.

A dot plot or jittered strip plot displays individual observations and is often preferable for small datasets because it exposes sample size and repeated values.

4.2 Bivariate displays

A scatter plot is appropriate for two numerical variables. It can reveal direction, form, clusters, changing variability and influential observations. Use transparency or sampling when many points overlap.

A grouped bar chart compares categories across a second categorical variable. For many groups, a dot plot or small multiples may be clearer.

A line chart is suitable when the horizontal axis has a meaningful order, especially time. Connecting unrelated categories with lines can imply a false continuity.

A heat map represents values through colour across a matrix, such as hour by day or product by region. Labels, scale and colour meaning must be explicit.

4.3 Multivariate displays

Small multiples repeat a consistent chart for several groups, allowing comparison without excessive visual clutter. Faceting, colour, shape and size can encode additional variables, but excessive encodings increase cognitive load. A chart should prioritize the variables relevant to the decision.

5. Distribution, Relationships and Anomalies

Analysts should examine shape, centre, spread and unusual observations. Distributions may be symmetric, right-skewed, left-skewed, heavy-tailed, multimodal or truncated. A right-skewed income distribution, for example, may be better summarized by the median and quartiles than by the mean alone.

Relationships should be assessed through several questions:

• Is the relationship positive, negative or absent?
• Is it approximately linear or curved?
• Are there clusters or subpopulations?
• Does variability change across the range?
• Are a few influential observations driving the apparent pattern?
• Could time, selection or another variable explain the association?

Pearson correlation is:

where x_i and y_i are paired observations, and x̄ and ȳ are their sample means. The value lies between −1 and 1 and measures linear association. It is sensitive to outliers and does not establish causation. A nonlinear relationship can have a correlation near zero, and a high correlation may result from a third variable or common time trend.

Spearman’s rank correlation is often useful for monotonic relationships, ordinal variables or data affected by non-normality. It measures association between ranks, not necessarily a linear relationship.

Anomalies can arise from data-entry mistakes, unit mismatches, sensor faults, fraud, rare events or genuine process changes. A responsible investigation checks source records, timestamps, units, related variables and domain context. Imputation, correction, exclusion or retention should be documented and justified.

6. Missing Data and Data Quality

Missing values are not automatically random. They may be missing completely at random, related to observed variables, or related to the unobserved value itself. For example, high-income respondents may be less likely to report income. A missingness indicator and a plot of missing values by group or time can reveal structure.

Common responses include:

• reporting the extent and pattern of missingness;
• retaining missingness as a meaningful category where appropriate;
• using justified imputation;
• performing complete-case analysis when defensible;
• conducting sensitivity analysis under alternative assumptions.

Replacing all missing values with zero is generally unsafe because zero usually means a measured absence, not an unknown value. Analysts should preserve the original data and create documented derived fields.

7. Dashboard and Data-Story Design

A dashboard is an operational interface for monitoring or exploring information. A data story is a guided communication that establishes context, presents evidence and supports a conclusion or action. They are related but not identical.

7.1 Principles of accurate visualization

• Begin with the decision and audience, not the chart type.
• Use a truthful scale; bar charts should normally begin at zero when bar length encodes magnitude.
• Label values, units, dates, definitions and data sources.
• Use consistent colours and ordering across related charts.
• Avoid 3-D effects, unnecessary decoration and misleading dual axes.
• Show denominators, sample sizes and uncertainty where relevant.
• Distinguish actual values from forecasts, targets and benchmarks.
• Use colour palettes accessible to people with colour-vision deficiencies.
• Provide text alternatives and readable contrast.
• Make filters visible and indicate when a view is filtered.

A dashboard should have a clear hierarchy: headline measures, diagnostic trends, explanatory breakdowns and details for investigation. It should not simply display every available metric. A key performance indicator requires a definition, target, time window, owner and action threshold; otherwise it may encourage superficial optimization.

7.2 Technical and managerial audiences

Technical audiences may require transformation rules, confidence intervals, model diagnostics, code reproducibility and data lineage. Managerial audiences usually need the business meaning, magnitude, trend, risk, recommendation and decision deadline. These are different levels of detail, not different standards of truth. Both audiences need accurate definitions and limitations.

A strong data story often follows this structure:

1. Context: What decision or problem matters?
2. Observation: What pattern is visible?
3. Evidence: How large, reliable and consistent is it?
4. Explanation: What plausible factors may account for it?
5. Action: What should be investigated or changed?
6. Qualification: What is uncertain or outside the analysis?

8. Applied Case: Retail Delivery Performance

The following is an illustrative case.

Background

A retailer receives complaints about late deliveries. Management believes that one distribution centre is responsible, but the data team is asked to investigate before resources are reallocated.

Problem and data

The dataset contains order identifier, promised delivery date, actual delivery date, region, carrier, product category and delivery cost for six months. The unit of observation is an order. Duplicate order identifiers, cancelled orders and missing actual dates are assessed before analysis.

Analysis

A line chart of weekly on-time percentage shows deterioration during two holiday weeks. A bar chart by carrier appears to show Carrier B performing worst. However, a grouped chart by carrier and region reveals that Carrier B handles most remote-region deliveries. Median delivery time is more stable than the mean, which is inflated by a small number of extreme delays. A box plot identifies a cluster of delays associated with one warehouse and a particular weather period.

The team calculates on-time percentage as:

Cancelled orders are excluded because they were not delivered, while orders with missing actual dates are reported separately rather than silently treated as late. The denominator and exclusions are documented on the dashboard.

Approach and outcome

Instead of penalizing Carrier B immediately, management compares carriers within region and controls for route distance in a subsequent model. The operational response is to review warehouse scheduling and weather contingency procedures, while monitoring carrier performance using region-adjusted benchmarks. Complaints fall in the following month, although the case does not prove that the dashboard intervention alone caused the improvement.

Learning points

The case demonstrates why aggregation can conceal confounding, why denominators matter, why missing outcomes require explicit treatment and why an outlier may identify an operational cause. It also shows that EDA supports decisions but cannot substitute for causal evaluation.

9. Tools, Reproducibility and Contemporary Practice

Spreadsheet software is useful for small, transparent inspections, but it is vulnerable to accidental overwriting and undocumented transformations. Programmatic tools such as Python or R support repeatable pipelines, statistical computation and automated tests. Business-intelligence platforms support interactive dashboards, access control and scheduled refreshes.

Good practice includes version-controlled code, data dictionaries, reproducible transformations, validation checks, documented chart definitions and separate development and production dashboards. Automated monitoring can detect changes in missingness, category frequencies, distributional drift or data freshness.

AI-assisted visualization and natural-language analytics can accelerate chart generation and explanation. However, generated charts may use incorrect aggregations, omit denominators, infer inappropriate causal language or expose sensitive data. Human review remains necessary. Sensitive attributes should be protected through access controls, minimization, appropriate aggregation and disclosure review.

10. Limitations, Ethics and Governance

EDA can be distorted by biased samples, measurement error, survivorship bias, ecological fallacy, selective reporting and multiple exploratory comparisons. Aggregated data can hide inequities between groups. Suppressing small groups may protect privacy but can also conceal harm; the appropriate balance depends on risk and governance requirements.

Analysts should consider fairness, privacy, consent, purpose limitation, retention and auditability. A dashboard that ranks employees, customers or communities can influence real opportunities, even when presented as descriptive. Definitions, transformations and limitations should be accessible to affected stakeholders. Visual clarity must never be used to give weak evidence unwarranted authority.

Best-Practice Checklist

Before publishing an EDA report or dashboard, ask:

• Is the population, unit of analysis and time period clear?
• Are variables correctly typed and units consistent?
• Have duplicates, missing values and invalid records been assessed?
• Are mean, median and spread appropriate for the distribution?
• Are potential outliers investigated rather than automatically removed?
• Do charts use honest scales, labels and denominators?
• Are association and causation clearly distinguished?
• Can users identify filters, refresh dates and data sources?
• Is the presentation accessible and appropriate for its audience?
• Are assumptions, exclusions, privacy risks and limitations documented?

Chapter Summary

Exploratory Data Analysis is an iterative process for understanding data quality, distributions, relationships and anomalies before formal modelling or decision-making. Numerical summaries such as proportions, means, medians, variance, standard deviation, quantiles and the interquartile range must be selected according to variable type and distribution. Visual methods—including bar charts, histograms, box plots, scatter plots, line charts, heat maps and small multiples—make patterns visible but can also mislead when scales, denominators or encodings are poorly chosen.

Correlation describes association, not causation. Outliers and missing values require contextual investigation rather than automatic deletion or replacement. Dashboards should support a defined decision, provide transparent definitions and present information accurately and accessibly. Reproducible workflows, data lineage, validation and human review are essential, especially when AI tools assist analysis. Ethical EDA considers bias, privacy, fairness and the consequences of presenting descriptive evidence to decision-makers.

Key Terms

Exploratory Data Analysis: Systematic investigation of data using summaries, visualizations and contextual reasoning.

Univariate analysis: Analysis of one variable at a time.

Bivariate analysis: Analysis of the relationship between two variables.

Multivariate analysis: Analysis involving several variables simultaneously.

Mean: Arithmetic average of numerical observations.

Median: Middle value of ordered observations.

Quantile: Value dividing ordered data at a specified cumulative proportion.

Variance: Average squared deviation used to describe numerical dispersion.

Standard deviation: Square root of variance, expressed in the original units.

Interquartile range: Difference between the third and first quartiles.

Outlier: An observation unusually distant from the rest of the data or from an expected pattern.

Skewness: Asymmetry in a distribution.

Correlation: Numerical measure of association, commonly linear association for Pearson’s correlation.

Confounding: Distortion of an apparent relationship by another variable.

Histogram: Graph of numerical observations grouped into contiguous intervals.

Box plot: Display based on quartiles, median and potential extreme values.

Dashboard: Interactive or static display designed to monitor or explore measures.

Data story: Structured communication linking context, evidence, interpretation and action.

Data lineage: Record of data sources, transformations and movement.

Missingness: The pattern and mechanism by which values are absent.

Review Questions

Short Answer Questions

1. What is the main purpose of Exploratory Data Analysis?
2. Why might the median be preferred to the mean for a highly skewed variable?
3. What information does the interquartile range provide?
4. Why does correlation not establish causation?
5. State two principles for designing an accurate dashboard.

Long Answer Questions

1. Explain an appropriate EDA workflow from receiving a dataset to communicating findings.
2. Compare histograms, box plots and dot plots, including the situations in which each is useful.
3. Explain how missing data can bias analysis and describe responsible responses.
4. Discuss how a data story should differ for a technical audience and a managerial audience.
5. Explain the ethical and governance risks associated with dashboards used to rank people or groups.

Analytical / Application Questions

1. A dataset contains customer incomes of 28, 31, 33, 35, 36 and 220 thousand currency units. Which measures and plots would you use, and how would you investigate the extreme value?
2. A scatter plot shows a strong positive relationship between ice-cream sales and hospital visits over several months. Identify at least two possible explanations and propose additional analyses before making a causal claim.
3. A delivery dashboard reports 95% on-time performance, but it excludes orders with missing delivery dates. Explain why this may be misleading and design a more transparent set of metrics.

Multiple Choice Questions with Answers

Question 1

Which statistic is generally most resistant to a single extreme observation?

A. Mean

B. Median

C. Variance

D. Range

Correct Answer: B. Median

Question 2

What does the interquartile range equal?

A. Maximum minus minimum

B. Mean divided by standard deviation

C. Third quartile minus first quartile

D. Median minus mean

Correct Answer: C. Third quartile minus first quartile

Question 3

Which chart is most appropriate for examining the relationship between two numerical variables?

A. Scatter plot

B. Pie chart

C. Bar chart of one category

D. Frequency table only

Correct Answer: A. Scatter plot

Question 4

A Pearson correlation close to zero can occur when two variables have:

A. A perfect linear relationship

B. No possible relationship of any kind

C. A nonlinear relationship

D. Identical units

Correct Answer: C. A nonlinear relationship

Question 5

For a bar chart where bar length represents magnitude, which practice is usually recommended?

A. Use a truncated axis to exaggerate differences

B. Begin the value axis at zero

C. Remove category labels

D. Use 3-D perspective

Correct Answer: B. Begin the value axis at zero

Question 6

If 40 of 200 eligible orders were late, what was the on-time percentage?

A. 20%

B. 40%

C. 80%

D. 160%

Correct Answer: C. 80%

Question 7

Which action is generally unsafe when handling missing numerical values?

A. Report their extent

B. Investigate their pattern

C. Replace every missing value with zero automatically

D. Perform sensitivity analysis

Correct Answer: C. Replace every missing value with zero automatically

Question 8

What is a key danger of comparing an aggregated carrier performance rate without considering region?

A. The chart may contain too many colours

B. Regional mix may confound the comparison

C. The median cannot be calculated

D. Time cannot be displayed

Correct Answer: B. Regional mix may confound the comparison

Question 9

Which display is especially useful for comparing distributions across several groups?

A. Box plot by group

B. Unlabelled pie chart

C. Decorative 3-D chart

D. Single headline number

Correct Answer: A. Box plot by group

Question 10

Which statement about an exploratory outlier rule using 1.5 × IQR is correct?

A. It proves that flagged observations are errors

B. It is a screening convention requiring contextual investigation

C. It applies only to categorical variables

D. It removes the need to inspect source records

Correct Answer: B. It is a screening convention requiring contextual investigation