Exploratory Data Analysis and Visualization

Introduction and Context

Exploratory Data Analysis (EDA) is the systematic process of examining, summarising, visualising, and questioning data before formal modelling. It occupies a critical position in the data-science workflow because statistical models and machine-learning algorithms are only as reliable as the data, assumptions, and problem formulation on which they depend. EDA helps analysts understand data structure, discover relationships, identify anomalies, evaluate data quality, generate hypotheses, and communicate evidence to technical and non-technical audiences.

EDA is not merely the production of attractive charts. It is an iterative reasoning process that combines statistical summaries, visual perception, domain knowledge, computational tools, and critical judgement. In R, the tidyverse provides an integrated environment for EDA, with `dplyr` for transformation, `tidyr` for reshaping, `ggplot2` for visualisation, and packages such as `skimr`, `janitor`, and `visdat` for data-quality inspection.

A useful distinction is between exploratory and confirmatory analysis. Exploratory analysis is used to discover possible patterns and formulate hypotheses. Confirmatory analysis evaluates pre-specified hypotheses using formal statistical tests, confidence intervals, or predictive validation. The same dataset may support both activities, but conclusions discovered through exploration should ideally be tested on independent data or through suitable validation procedures.

Learning Objectives

After completing this chapter, students should be able to:

1. Explain the purpose, scope, and principles of EDA.

2. Perform univariate, bivariate, and multivariate exploration in R.

3. Select suitable graphical and numerical summaries for different data types.

4. Create effective, reproducible graphics using `ggplot2`.

5. Detect missing values, outliers, anomalies, and potential data-quality problems.

6. Interpret associations without confusing correlation with causation.

7. Design dashboards, reports, and evidence-based data stories.

8. Evaluate ethical, perceptual, and methodological risks in data visualisation.

1. Foundations of Exploratory Data Analysis

1.1 The analytical questions behind EDA

EDA should be guided by questions rather than by a random collection of plots. Typical questions include:

• What are the observational units: customers, transactions, machines, patients, or time periods?
• What variables are available, and what are their measurement scales?
• Are values missing, duplicated, impossible, or inconsistent?
• How are numerical variables distributed?
• Which groups differ in level or variability?
• Which variables appear to move together?
• Are relationships linear, nonlinear, conditional, or time-dependent?
• Are unusual observations errors, rare but valid cases, or evidence of a new phenomenon?

A professional EDA begins with a data dictionary and a statement of the decision context. Without this context, analysts may identify statistically interesting patterns that have no operational value.

1.2 Data types and measurement scales

The choice of summary and visualisation depends on the variable type.

• Nominal categorical variables represent labels without order, such as region or payment method.
• Ordinal variables have an order but not necessarily equal intervals, such as satisfaction levels from poor to excellent.
• Discrete numerical variables are countable, such as number of support tickets.
• Continuous numerical variables can take values in an interval, such as temperature or response time.
• Date-time variables encode temporal structure and often require special treatment.
• Geospatial variables contain locations or coordinates and may require maps.

A variable stored as numeric may actually be categorical. For example, codes 1, 2, and 3 may represent Bronze, Silver, and Gold membership. Treating these codes as measurements would produce misleading means and correlations.

1.3 A reproducible EDA workflow

A robust workflow is iterative:

1. Define the analytical and decision context.

2. Import data and preserve the raw version.

3. Inspect structure, dimensions, classes, and metadata.

4. Audit missingness, duplicates, ranges, and coding consistency.

5. Transform data transparently and document every transformation.

6. Explore variables individually.

7. Examine pairwise and multivariable relationships.

8. Investigate anomalies and competing explanations.

9. Communicate findings with appropriately designed graphics.

10. Validate important findings using statistical analysis, domain review, or new data.

The workflow is not strictly linear. A plot may reveal a coding error, which requires returning to data preparation. Similarly, domain experts may explain an apparent outlier as a legitimate event.

2. Numerical and Graphical Summaries

2.1 Measures of centre and spread

For observations $x_1, x_2, \ldots, x_n$, the arithmetic mean is

$$\bar{x}=\frac{1}{n}\sum_{i=1}^{n}x_i.$$

The mean uses every observation and is useful when the distribution is reasonably symmetric and not dominated by extreme values. The median is the middle ordered observation, or the average of the two middle observations when $n$ is even. The median is more resistant to extreme values and is often preferable for income, transaction value, or response-time data.

The range is the difference between the maximum and minimum. The variance is

$$s^2=\frac{1}{n-1}\sum_{i=1}^{n}(x_i-\bar{x})^2,$$

where $n>1$ and the denominator $n-1$ gives the usual unbiased estimator of population variance under independent random sampling. The standard deviation is $s=\sqrt{s^2}$ and is expressed in the original units. The interquartile range is $IQR=Q_3-Q_1$, where $Q_1$ and $Q_3$ are the first and third quartiles. The IQR is robust and forms the basis of a common exploratory rule for potential outliers: values below $Q_1-1.5IQR$ or above $Q_3+1.5IQR$. This is a screening convention, not proof that an observation is erroneous.

The coefficient of variation, $CV=s/\bar{x}$, is useful for comparing relative variability only when the mean is positive, meaningfully away from zero, and the ratio is substantively interpretable. It is not appropriate for variables measured on an arbitrary zero scale or for means near zero.

2.2 Distributional shape

A distribution may be symmetric, right-skewed, left-skewed, heavy-tailed, multimodal, or bounded. Shape influences the choice of summary and model. Histograms show frequencies across bins, but conclusions can change with bin width. Density plots provide a smoothed representation, but the bandwidth controls the degree of smoothing. Boxplots compactly show the median, quartiles, and potential extreme observations; they are useful for group comparisons but conceal multimodality and sample-size differences.

A useful practice is to combine a histogram or density plot with numerical summaries and, when appropriate, a logarithmic transformation. Transformation should be motivated by measurement characteristics or modelling needs, not used merely to make a plot look more regular.

2.3 Basic R inspection

```r

library(tidyverse)

glimpse(sales)

summary(sales)

sales |> count(region, sort = TRUE)

sales |> summarise(

n = n(),

missing_revenue = sum(is.na(revenue)),

median_revenue = median(revenue, na.rm = TRUE),

iqr_revenue = IQR(revenue, na.rm = TRUE)

)

```

`na.rm = TRUE` excludes missing values from the calculation; it does not repair missingness. Every analysis should report how missing values were handled and how many observations remained.

3. Univariate Exploration

Univariate EDA studies one variable at a time. For categorical variables, frequency tables, proportions, and ordered bar charts are usually suitable. For numerical variables, use a combination of centre, spread, shape, and an appropriate plot.

```r

ggplot(sales, aes(x = region)) +

geom_bar() +

labs(x = "Region", y = "Number of transactions") +

theme_minimal()

ggplot(sales, aes(x = revenue)) +

geom_histogram(bins = 30, colour = "white") +

labs(x = "Revenue", y = "Count") +

theme_minimal()

```

Bar charts should generally begin at zero because bar length encodes magnitude. A histogram is not a bar chart: its bins represent intervals of a continuous variable, and gaps between bins may be misleading if the variable is continuous and adjacent intervals are intended to touch.

For long-tailed variables, a plot on a transformed scale may reveal structure:

```r

ggplot(sales, aes(x = revenue)) +

geom_histogram(bins = 30) +

scale_x_log10() +

labs(x = "Revenue (log10 scale)")

```

The transformed axis must be labelled clearly. Zeros and negative values require special consideration because the ordinary logarithm is undefined for them.

4. Bivariate Exploration

Bivariate EDA examines relationships between two variables. The display depends on their types.

4.1 Numerical–numerical relationships

Scatterplots are the primary tool. They reveal direction, form, strength, clusters, unequal spread, and influential observations.

```r

ggplot(sales, aes(x = advertising_spend, y = revenue)) +

geom_point(alpha = 0.45) +

geom_smooth(method = "lm", se = TRUE, colour = "steelblue") +

theme_minimal()

```

The fitted line summarises a linear association; it does not establish causation. If observations are repeated for the same customer, store, or month, independence assumptions may fail. A trend line can also be distorted by outliers or confounding variables.

Pearson correlation is

$$r=\frac{\sum_{i=1}^{n}(x_i-\bar{x})(y_i-\bar{y})}{\sqrt{\sum_{i=1}^{n}(x_i-\bar{x})^2}\sqrt{\sum_{i=1}^{n}(y_i-\bar{y})^2}}.$$

It measures the strength and direction of linear association and lies between -1 and 1. It is sensitive to outliers and does not detect all nonlinear relationships. Spearman correlation uses ranks and can be more appropriate for monotonic relationships or ordinal variables, but it also does not imply causation.

4.2 Categorical–numerical relationships

Use boxplots, violin plots, jittered points, or summary intervals. Showing raw observations alongside summaries often improves transparency.

```r

ggplot(sales, aes(x = region, y = revenue)) +

geom_boxplot(outlier.shape = NA) +

geom_jitter(width = 0.15, alpha = 0.25) +

coord_cartesian(ylim = quantile(sales$revenue,

c(.01, .99), na.rm = TRUE)) +

theme_minimal()

```

`coord_cartesian()` zooms the display without deleting observations, unlike filtering the data. Group comparisons should consider sample size, uncertainty, exposure time, and possible confounding.

4.3 Categorical–categorical relationships

Contingency tables and proportional bar charts help assess association. Counts can be misleading when group sizes differ, so within-group percentages are often more informative.

```r

sales |>

count(region, channel) |>

group_by(region) |>

mutate(prop = n / sum(n)) |>

ggplot(aes(x = region, y = prop, fill = channel)) +

geom_col() +

scale_y_continuous(labels = scales::percent) +

theme_minimal()

```

The denominator must be stated: percentages may be calculated within region, within channel, or across the entire dataset.

5. Multivariate Exploration

Multivariate EDA examines three or more variables and is essential because many relationships are conditional. An overall association may disappear, reverse, or become stronger after stratification. This phenomenon is related to Simpson’s paradox.

Faceting provides small multiples with consistent scales:

```r

ggplot(sales, aes(advertising_spend, revenue, colour = region)) +

geom_point(alpha = 0.5) +

facet_wrap(~ channel) +

theme_minimal()

```

Colour, shape, size, and faceting should be used sparingly. Encoding too many variables in one figure creates cognitive overload. For high-dimensional numerical data, correlation heat maps, pair plots, parallel-coordinate plots, principal component projections, and interactive linked views may be useful. Each reduces information in some way and therefore requires careful interpretation.

A correlation heat map is descriptive, not causal. Correlations may result from a common cause, time trends, selection bias, or measurement artefacts. Standardising variables before a distance-based method is important when variables have different units; otherwise large-scale variables dominate the distance calculation.

5.1 Grouping and aggregation

Aggregation can reveal strategic patterns but may conceal individual variation. For example, monthly average response time may improve while a subgroup of high-priority customers experiences deterioration. Analysts should compare aggregate and disaggregated views and document the aggregation level.

5.2 Time-oriented EDA

Time-series exploration should preserve chronological order. Line charts are suitable for regularly ordered observations, while rolling summaries can reveal trends without replacing the raw series.

```r

ggplot(monthly_sales, aes(month, revenue)) +

geom_line() +

geom_point() +

scale_x_date(date_breaks = "3 months", date_labels = "%b %Y") +

theme_minimal()

```

Seasonality, structural breaks, missing periods, changing definitions, and delayed reporting must be investigated. Randomly shuffling time data for validation can cause leakage because future information may enter the training or exploratory process.

6. Effective Graphics with ggplot2

`ggplot2` implements a grammar of graphics. A graphic is constructed from:

• data;
• aesthetic mappings, such as x-position, y-position, colour, and fill;
• geometric layers, such as points, bars, and lines;
• statistical transformations;
• scales;
• coordinate systems;
• facets;
• themes and annotations.

The distinction between mapping and setting is important:

```r

Colour mapped to a variable

geom_point(aes(colour = region))

One fixed colour for all points

geom_point(colour = "navy")

```

A polished chart should have an informative title, clear axis labels, units, an appropriate legend, and a caption identifying data source or definitions. Avoid unnecessary decoration, excessive colours, truncated axes, and three-dimensional effects. Use colour palettes that remain interpretable for colour-vision deficiencies and distinguish meaning from mere emphasis.

A chart should answer a specific question. Before choosing a geometry, identify whether the goal is comparison, distribution, relationship, composition, change over time, or geographic pattern. For example, a slope chart may communicate change between two periods, while a line chart is better for a longer time sequence.

7. Missingness, Outliers, and Anomalies

Missing data may be missing completely at random, related to observed variables, or related to the missing value itself. These mechanisms cannot usually be established from a missingness plot alone. Missingness may reflect system failure, non-response, business rules, or deliberate suppression.

```r

sales |>

summarise(across(everything(), ~ sum(is.na(.)))) |>

pivot_longer(everything(), names_to = "variable", values_to = "missing")

```

Do not automatically replace missing values with zero. Zero and unknown are different states. Imputation should be justified by the analytical purpose, data-generating process, and uncertainty requirements.

An outlier is an observation unusual relative to a chosen reference distribution or group. An anomaly is unusual in a way that may indicate operational failure, fraud, a new regime, or data error. Detection methods include robust rules based on the IQR, standardised residuals, isolation-based methods, clustering, and time-series monitoring. Detection is not classification: domain investigation is required.

Useful diagnostic questions are:

1. Is the value physically or logically possible?

2. Is the unit or decimal placement correct?

3. Does the record duplicate another record?

4. Does the observation correspond to a known event?

5. Is the observation unusual only because the group has a different baseline?

6. Would removing it change the substantive conclusion?

Data should not be deleted merely because it weakens a preferred result. Maintain an audit trail of exclusions, corrections, and sensitivity analyses.

8. From Analysis to Dashboards and Data Stories

A report supports deeper explanation, while a dashboard supports repeated monitoring and rapid decision-making. A dashboard should have a defined audience, decision, refresh schedule, metric definitions, and escalation rules. It should not be a collection of every available chart.

A practical dashboard architecture has three layers:

1. Overview: a small set of key indicators and current status.

2. Diagnosis: trends, segments, and comparisons that explain movement.

3. Action: filters, alerts, ownership, and recommended next steps.

Effective data storytelling connects context, evidence, interpretation, and action. A typical narrative is: establish the baseline, identify the important change, show where it occurs, test plausible explanations, state uncertainty, and recommend a proportionate response. The analyst should distinguish observed facts from interpretations and recommendations.

Interactive tools such as Shiny can support filtering and exploration, but interactivity does not eliminate the need for clear defaults, accessible labels, reproducible calculations, and performance testing. A dashboard that changes silently because of an undocumented data refresh is not a reliable analytical product.

9. Illustrative Case: Reducing Customer Support Delays

Background

A software company receives support requests through email, chat, and a web portal. Management observes that average resolution time has increased and asks the analytics team to identify causes and recommend action. The dataset contains ticket identifier, creation date, channel, customer segment, priority, first-response time, resolution time, agent team, and satisfaction score.

Problem and initial inspection

The team first checks duplicated ticket identifiers, impossible negative durations, missing priority values, and whether time zones are consistent. They discover that resolution time is strongly right-skewed: most tickets close within two days, while a small number remain open for several weeks. Consequently, the median and upper quantiles are more informative than the mean alone.

A time plot shows an apparent deterioration after a new routing system was introduced. However, the number of high-priority enterprise tickets also increased. A simple comparison of overall averages could therefore confound routing-system effects with changes in ticket mix.

Analysis

The analysts create distributions by priority and customer segment, then facet time trends by channel. Boxplots and jittered points reveal that chat tickets are usually resolved quickly, while portal tickets have greater variability. A scatterplot of backlog size and resolution time suggests a relationship, but the team avoids claiming that backlog alone causes delay because staffing and incident severity may affect both.

Missing priority values are concentrated in portal tickets. Rather than assigning an arbitrary category, the team consults operations staff and discovers that a form field was optional. The field is made mandatory, and historical records are flagged as unknown. This prevents false precision.

The team compares median resolution time and the 90th percentile before and after the routing change, separately for priority and channel. They also perform a sensitivity analysis excluding tickets associated with a known major outage. The apparent overall deterioration is smaller after stratification, but high-priority portal tickets still show a meaningful increase in the upper tail.

Approach and solution

The proposed intervention is targeted rather than universal: improve portal triage for high-priority tickets, add an escalation alert at 24 hours, and monitor the 90th percentile by priority. A dashboard displays ticket volume, median resolution time, 90th percentile, backlog, and missing-field rates. Definitions and filters are documented, and access is restricted because ticket records may contain personal or confidential information.

Outcome and learning points

After implementation, the dashboard shows a reduction in the upper tail of high-priority portal resolution times, while median performance changes only modestly. This illustrates why a single average can conceal operational risk. The case also demonstrates the value of data-quality auditing, stratification, temporal context, robust summaries, and action-oriented visualisation.

10. Limitations, Ethics, and Governance

EDA can generate false discoveries because analysts may try many variables, transformations, and subgroup definitions. Exploratory patterns should be labelled as hypotheses until independently validated. Selection bias, survivorship bias, measurement error, and aggregation bias can produce persuasive but misleading graphics.

Visualisations also have ethical consequences. Truncated axes, inappropriate denominators, selective time windows, hidden missingness, and colour choices can distort interpretation. Privacy risks arise when small groups, locations, or combinations of attributes permit re-identification. Dashboards should apply access controls, minimisation, aggregation, and appropriate suppression rules.

Governance requires version-controlled code, documented data lineage, reproducible environments, clear metric definitions, review by domain experts, and an audit trail. When AI-assisted tools generate code, summaries, or charts, analysts remain responsible for checking calculations, data leakage, fabricated interpretations, and inappropriate disclosure. AI can accelerate exploration, but it cannot replace contextual judgement or accountability.

11. Contemporary Developments and Best Practices

Modern EDA increasingly includes automated data profiling, interactive notebooks, semantic layers, streaming anomaly detection, and augmented analytics. These tools can identify candidate patterns quickly, but automation may reproduce biased definitions or overwhelm users with alerts. Human review, threshold governance, and validation remain essential.

Recommended practices include:

• begin with a decision question and data dictionary;
• preserve raw data and make transformations reproducible;
• use numerical summaries and graphics together;
• inspect distributions before applying averages or models;
• stratify by meaningful groups and time periods;
• show uncertainty and sample sizes where relevant;
• investigate rather than automatically remove outliers;
• label transformed axes and define denominators;
• use accessible, restrained visual design;
• separate exploration from confirmatory claims;
• document missing-data treatment and exclusions;
• validate important findings with domain experts and new data.

Chapter Summary

Exploratory Data Analysis is an iterative process for understanding data quality, structure, distributions, relationships, and unusual observations before formal modelling or decision-making. Univariate analysis uses suitable summaries and plots for individual variables; bivariate analysis examines pairwise relationships according to variable types; and multivariate analysis reveals conditional patterns, confounding, and subgroup differences. `ggplot2` provides a grammar for constructing layered, reproducible graphics. Effective visualisation depends on accurate mappings, suitable scales, clear labels, accessible design, and a defined analytical purpose. Missing values, outliers, and anomalies require investigation rather than automatic deletion. Dashboards and data stories should connect evidence to decisions while communicating uncertainty and protecting privacy. At postgraduate level, EDA must be understood not only as a technical activity but also as a disciplined process of reasoning, validation, governance, and responsible communication.

Key Terms

Exploratory Data Analysis: Iterative examination of data to discover structure, quality issues, and possible relationships.

Univariate analysis: Analysis of one variable at a time.

Bivariate analysis: Analysis of the relationship between two variables.

Multivariate analysis: Analysis involving three or more variables.

Tidy data: A data organisation in which each variable is a column, each observation is a row, and each value is a cell.

Mean: Arithmetic average of numerical observations.

Median: Middle value of ordered observations.

Interquartile range: Difference between the third and first quartiles.

Standard deviation: Square root of variance, expressed in the original units.

Correlation: Numerical measure of association, commonly linear association for Pearson correlation.

Outlier: Observation unusually distant from the rest under a specified criterion.

Anomaly: Observation or pattern that is unusual and potentially operationally significant.

Missingness: Absence of an observed value in a dataset.

Grammar of graphics: Framework representing a graphic through data, mappings, geometries, scales, and other components.

Aesthetic mapping: Linking a data variable to a visual property such as position, colour, or size.

Faceting: Splitting a graphic into panels according to one or more categorical variables.

Data story: Structured communication connecting context, evidence, interpretation, and action.

Dashboard: Interactive or static display designed to monitor indicators and support decisions.

Data leakage: Use of information in analysis or modelling that would not be available at the relevant decision time.

Simpson’s paradox: Reversal or alteration of an association after data are divided into relevant subgroups.

Review Questions

Short Answer Questions

1. What is the main difference between exploratory and confirmatory analysis?

2. Why is the median often preferred to the mean for strongly right-skewed data?

3. What is the purpose of faceting in `ggplot2`?

4. Why should missing values not automatically be replaced with zero?

5. State two questions that should be asked when investigating an outlier.

Long Answer Questions

1. Explain a complete, reproducible EDA workflow from data ingestion to communication of findings.

2. Compare appropriate visualisations for numerical–numerical, categorical–numerical, and categorical–categorical relationships.

3. Explain how correlation can be misinterpreted and discuss the roles of confounding, outliers, and nonlinear relationships.

4. Discuss principles for designing an effective operational dashboard for managerial decision-making.

5. Critically evaluate the ethical and governance risks associated with automated and interactive data visualisation.

Analytical / Application Questions

1. A company reports that average delivery time increased from 3.2 to 3.8 days after a policy change. Design an EDA plan that determines whether the increase is associated with changes in region, product type, season, or order priority.

2. A dataset contains customer income, age, subscription type, monthly usage, and churn status. Propose a sequence of univariate, bivariate, and multivariate analyses, including suitable plots and cautions about interpretation.

3. A monitoring dashboard shows a sudden increase in machine-failure alerts. Explain how you would determine whether the pattern represents a real process change, a sensor problem, a change in alert thresholds, or a shift in production mix.

Multiple Choice Questions with Answers

Question 1

Which measure is generally most resistant to extreme observations?

A. Arithmetic mean

B. Median

C. Variance

D. Range

Correct Answer: B. Median

Question 2

What does the IQR equal?

A. Maximum minus minimum

B. Mean divided by standard deviation

C. Third quartile minus first quartile

D. Variance squared

Correct Answer: C. Third quartile minus first quartile

Question 3

In ggplot2, what is the effect of placing colour = region inside aes()?

A. It assigns one fixed colour to every point

B. It maps colour to the region variable

C. It removes the colour scale

D. It converts region into a numerical variable

Correct Answer: B. It maps colour to the region variable

Question 4

Which plot is most appropriate for examining the relationship between two numerical variables?

A. Scatterplot

B. Pie chart

C. Single bar chart

D. Frequency table only

Correct Answer: A. Scatterplot

Question 5

Why can overall group averages be misleading in a multivariate analysis?

A. Averages cannot be calculated for numerical data

B. Aggregation may conceal subgroup differences or confounding

C. Group averages always exaggerate outliers

D. Multivariate analysis uses only categorical variables

Correct Answer: B. Aggregation may conceal subgroup differences or confounding

Question 6

What does Pearson correlation primarily measure?

A. Causal impact

B. Linear association

C. Difference between medians

D. Missing-data proportion

Correct Answer: B. Linear association

Question 7

Which statement about the 1.5 IQR rule is correct?

A. It proves that flagged values are errors

B. It is a screening convention for potential outliers

C. It can be used only for categorical variables

D. It replaces the need for domain investigation

Correct Answer: B. It is a screening convention for potential outliers

Question 8

Why should percentages in a stacked bar chart have a clearly stated denominator?

A. The denominator determines the interpretation of the proportions

B. Percentages cannot be used with categorical variables

C. The denominator changes the data type to numerical

D. A denominator is required only for histograms

Correct Answer: A. The denominator determines the interpretation of the proportions

Question 9

Which practice helps prevent temporal data leakage?

A. Randomly mixing future and past observations

B. Using future information to define past outcomes

C. Preserving chronological order during validation

D. Deleting all date variables

Correct Answer: C. Preserving chronological order during validation

Question 10

Which is an appropriate dashboard design principle?

A. Include every available metric on the first screen

B. Hide metric definitions to reduce visual clutter

C. Define the audience, decision, refresh schedule, and indicators

D. Use three-dimensional effects for emphasis

Correct Answer: C. Define the audience, decision, refresh schedule, and indicators