1. Statistical Thinking, Data and Exploratory Analysis

Introduction and Context

Statistics for data science begins before a model is fitted. It begins with deciding what is being measured, whose data are being studied, how observations were obtained, and whether the resulting evidence is sufficiently trustworthy for the intended decision. Statistical thinking is therefore more than calculating averages or producing charts. It is a disciplined way of reasoning under variability and uncertainty.

A data scientist may be asked to estimate customer churn, compare medical treatments, detect fraud, or forecast demand. In each case, the observed dataset is only a partial and imperfect representation of a broader phenomenon. Statistical thinking helps the analyst distinguish signal from noise, describe what has been observed, identify possible sources of bias, and avoid conclusions that the data cannot support.

Exploratory data analysis (EDA) occupies a central position in this process. It uses summaries, visualizations, and investigative questions to understand the structure and quality of data before formal statistical modelling. EDA is not a substitute for confirmatory analysis; rather, it reveals patterns, anomalies, assumptions, and questions that should guide subsequent analysis.

Learning Objectives

After studying this chapter, the learner should be able to:

• distinguish populations, samples, parameters, and statistics;
• identify variables and classify them by measurement scale and data type;
• explain sampling methods and recognize sampling and non-sampling errors;
• perform systematic data-quality checks;
• select and interpret appropriate descriptive statistics and visualizations;
• explain the value of robust summaries and exploratory data analysis;
• evaluate an applied dataset while recognizing ethical, technical, and interpretive limitations.

Statistical Thinking: From Questions to Evidence

Statistical reasoning generally follows a chain: a substantive question is translated into measurable variables; observations are collected; data quality is assessed; patterns are described; and conclusions are communicated with appropriate uncertainty. Errors at an early stage cannot usually be repaired by sophisticated algorithms later. For example, a highly accurate classifier trained on a sample that excludes an important population group may produce systematically unfair decisions.

The analyst must continually ask four questions. What is the target population? What process generated the observations? Which features of the observations are stable and which may be random? What claims are justified by the design and quality of the data? These questions shift attention from computation alone to the data-generating process.

Population, Sample, Parameter and Statistic

A population is the complete set of units or outcomes relevant to a question. It might be all registered voters in a region, all transactions processed by a bank during a year, or all patients eligible for a clinical service. A sample is the subset actually observed. Studying an entire population is sometimes possible, but even a census may contain missing, inaccurate, or inconsistently recorded values.

A numerical characteristic of a population is a parameter, whereas a numerical characteristic calculated from a sample is a statistic. The population mean is commonly denoted by μ and the sample mean by x̄. Parameters are often unknown; statistics are used to estimate them. Sampling variability means that different samples from the same population may yield different statistics.

The distinction between a target population and an accessible population is important. A company may wish to understand all customers but analyse only users of its mobile application. The accessible population is narrower, and conclusions may not generalize unless the difference is understood and addressed.

Units, Observations and Variables

A statistical unit is the entity on which measurements are made: a person, household, product, web session, loan, or time interval. An observation is the collection of recorded values for one unit. A variable is a characteristic that can take different values across units, such as age, region, income, device type, or purchase amount.

Variables may be categorical or quantitative. Categorical variables assign observations to groups. Quantitative variables represent numerical amounts for which arithmetic comparisons are meaningful. A count, such as number of complaints, is discrete; a measurement such as response time is often treated as continuous, even when recorded to a finite precision. A date or text field may be technically encoded as numbers but should not automatically be analysed as a quantitative measurement.

The distinction between explanatory and response variables is contextual rather than intrinsic. In a study of advertising and sales, advertising expenditure may be treated as an explanatory variable and sales as a response. This terminology does not by itself establish causation.

Measurement Scales and Meaningful Operations

Measurement scale determines which comparisons and transformations are defensible.

Nominal measurement assigns labels without inherent order. Examples include blood group, browser type, and department. Frequencies, proportions, and the mode are appropriate summaries; arithmetic means of nominal codes are meaningless.

Ordinal measurement provides an order, but differences between adjacent categories are not necessarily equal. Satisfaction ratings such as poor, fair, good, and excellent are ordinal. Medians, percentiles, and rank-based comparisons are often appropriate, while treating category codes as precise measurements requires justification.

Interval measurement has ordered values with meaningful equal differences but no meaningful absolute zero. Temperature in Celsius is a standard example: 20°C is not twice as warm as 10°C. Addition and subtraction are meaningful, whereas ratio statements require care.

Ratio measurement has equal intervals and a meaningful zero, allowing statements such as “twice as much.” Duration, mass, distance, and revenue are common examples. The scale classification depends on the measurement system and the interpretation intended, not merely on the storage format.

Measurement validity concerns whether a variable adequately represents the concept of interest. Reliability concerns consistency of measurement. A survey may reliably record responses while still failing to measure the intended construct, such as using click frequency as an incomplete proxy for customer satisfaction.

Sampling: How Observations Enter the Dataset

Sampling is the process of selecting units from a population. Probability sampling gives each unit a known, non-zero probability of selection and supports principled generalization. In a simple random sample, units are selected randomly from a sampling frame. A systematic sample selects every kth unit after a random starting point. Stratified sampling divides the population into relevant subgroups and samples within each, often improving representation and precision. Cluster sampling selects groups such as schools or branches and observes units within selected groups; it may reduce cost but can increase sampling variability when cluster members are similar.

Non-probability methods include convenience sampling, purposive sampling, and voluntary-response sampling. They may be useful for exploratory work or hard-to-reach populations, but inclusion probabilities are unknown and selection bias may be substantial. A large convenience sample is not automatically representative.

Sampling error is the random difference between a sample result and the corresponding population value caused by observing only part of the population. It generally decreases as effective sample size increases, although not necessarily at a linear rate. Sampling bias is systematic distortion caused by the selection process, such as excluding people without internet access.

Non-Sampling Errors

Non-sampling errors arise from processes other than random sampling. They include:

• coverage error, when the sampling frame omits or duplicates population units;
• nonresponse error, when selected units do not respond and respondents differ systematically;
• measurement error, caused by ambiguous questions, faulty instruments, or inaccurate reporting;
• processing error, introduced during coding, data entry, merging, or transformation;
• survivorship bias, when analysis includes only units that remain observable;
• response or social-desirability bias, when participants alter answers because of pressure or perceived expectations.

Increasing sample size does not automatically solve these problems. A very large biased dataset can provide highly precise estimates of the wrong quantity.

Data Quality Before Data Analysis

Data quality is multidimensional. Accuracy concerns closeness to the true value; completeness concerns missing required values; consistency concerns agreement across fields and sources; validity concerns adherence to allowed formats or rules; uniqueness concerns duplicate records; and timeliness concerns whether data remain suitable for the decision.

A practical quality assessment begins with provenance: where did the data originate, what definitions were used, and what transformations have already occurred? The analyst should inspect the data dictionary, units, collection dates, key fields, coding conventions, and linkage procedures before computing summaries.

Typical checks include:

1. confirming row and column counts and identifying the observational unit;

2. checking data types, permitted ranges, and category labels;

3. quantifying missing values overall and by important subgroups;

4. searching for duplicate keys and impossible combinations;

5. inspecting distributions for extreme values, spikes, gaps, and suspicious rounding;

6. comparing related fields, such as order date and delivery date;

7. documenting every correction, exclusion, imputation, and transformation.

Missingness should not be treated as a purely technical inconvenience. A missing income value may reflect privacy concerns, and missing clinical measurements may occur because a patient is especially ill. Analysts should consider whether data are missing completely at random, missing at random conditional on observed information, or missing not at random. The appropriate response may involve investigation, sensitivity analysis, or a carefully justified imputation method rather than automatic deletion.

Outliers are observations that are unusually distant from the rest of a distribution. They may be errors, rare but valid events, or evidence of a different subgroup. An outlier should not be removed simply because it changes the mean. The analyst should verify its source, assess its substantive plausibility, and compare conclusions with and without it.

Describing Distributions

A distribution describes how values occur across a variable. Its centre, spread, shape, and unusual features should be considered together.

The arithmetic mean is the sum of values divided by the number of observations. It uses every value and is useful for approximately symmetric distributions without influential extremes. The median is the middle ordered value, or the average of the two middle values when the sample size is even. It is resistant to extreme observations and often better represents a skewed distribution. The mode is the most frequent value and is particularly useful for categorical data.

Measures of spread include the range, interquartile range (IQR), variance, and standard deviation. The range is simple but depends only on two observations. The IQR, Q3 − Q1, describes the central half of the data and is robust to extremes. Variance is the average squared deviation from the mean, with a sample estimate commonly using denominator n − 1. Standard deviation is the square root of variance and is expressed in the original units.

Percentiles locate values within an ordered distribution. The 90th percentile is a value at or below which approximately 90 percent of observations fall, subject to the chosen percentile convention. Quantiles support service-level monitoring, wage comparisons, and risk analysis more effectively than a mean alone in many settings.

Skewness describes asymmetry. In a right-skewed distribution, a relatively small number of large values pull the mean above the median. Income, transaction amounts, and waiting times often display this pattern. A histogram, density plot, box plot, or empirical cumulative distribution can reveal features that a single summary conceals.

Robust Summaries

A statistic is robust when it is relatively insensitive to deviations from assumptions or to a small number of extreme observations. The median and IQR are robust location and spread summaries. A trimmed mean removes a specified proportion of the smallest and largest observations before calculating the mean. A winsorized mean replaces extremes with less extreme boundary values. Robust methods are valuable when distributions are heavy-tailed, contaminated, or uncertain, but they do not make unusual observations irrelevant. They change the question from average total magnitude toward typical central behaviour.

Group summaries must be interpreted carefully. An overall mean can conceal major differences between regions, customer segments, or time periods. Conversely, subgroup comparisons may be unstable when groups are small. Aggregation can also produce misleading conclusions, a problem associated with ecological inference and, in some settings, Simpson’s paradox. Analysts should examine both overall and stratified results.

Exploratory Data Analysis as an Investigative Process

EDA is a cycle rather than a single chart. The analyst begins with the substantive question, inspects data structure and quality, examines one variable at a time, studies relationships, investigates anomalies, and records hypotheses for later testing.

For one categorical variable, frequency tables and bar charts are suitable. For one quantitative variable, histograms, density plots, box plots, and quantile summaries are useful. To compare a quantitative variable across categories, side-by-side box plots, violin plots, strip plots, or group-specific summaries may be used. For two quantitative variables, scatter plots reveal direction, form, clusters, and unusual points. Time-series plots reveal trend, seasonality, level shifts, and missing intervals. A correlation coefficient can summarize linear association but cannot detect every nonlinear relationship and does not establish causation.

Visualization is an analytical instrument, not decoration. Axes should be labelled, units stated, scales chosen honestly, and transformations disclosed. Truncated axes may exaggerate differences in bar charts; excessive aggregation may hide variation; and colour choices may be inaccessible to people with colour-vision deficiency. Interactive dashboards should preserve definitions and allow users to inspect denominators and filters.

A useful EDA record includes the question asked, data version, transformations, charts, notable findings, alternative explanations, and unresolved limitations. Reproducible scripts or notebooks are preferable to undocumented manual manipulation. Exploratory findings should be distinguished from confirmatory claims because repeatedly searching data for a pattern increases the chance of false discovery.

A Practical Workflow for Analysts

A defensible workflow can be organized as follows:

1. Frame the problem. Define the decision, target population, unit of analysis, time period, and estimand or quantity of interest.

2. Acquire and document data. Record sources, permissions, definitions, collection procedures, and known limitations.

3. Audit structure and quality. Check types, keys, missingness, duplicates, ranges, and consistency.

4. Prepare cautiously. Standardize formats, resolve or flag errors, and preserve raw data separately from analysis-ready data.

5. Describe marginal distributions. Use appropriate numerical summaries and visualizations for each variable.

6. Explore relationships. Compare relevant subgroups, examine time and geography, and investigate possible confounding.

7. Challenge initial impressions. Test sensitivity to outliers, definitions, exclusions, and alternative aggregations.

8. Communicate findings. State what is supported, what remains uncertain, and what further analysis is required.

This workflow is iterative. A plot may reveal that the unit of analysis was misunderstood; a missingness pattern may require revising the research question; or a suspected outlier may expose a data-entry rule.

Illustrative Case: Exploring Customer-Service Response Times

Background

A fictional online retailer, Northstar Commerce, wants to determine whether customers in different regions receive comparable support. The operations team provides a dataset containing ticket identifier, region, issue type, creation time, first-response time, resolution time, customer channel, and satisfaction rating for one quarter.

Situation and Problem

The initial report states that the average first-response time is 8.2 hours. Managers are concerned because complaints from one region appear to be increasing. The data science team must assess whether the average is informative and whether operational differences are real.

Analysis

The team first defines the observational unit as a support ticket and checks uniqueness of ticket identifiers. Twelve percent of tickets have no first-response time because they were closed automatically; these are not treated as zero-hour responses. Timestamps reveal that some records use local time and others use UTC, producing impossible negative durations after merging. The team corrects the time-zone conversion and documents the change.

A histogram and box plot show a strongly right-skewed response-time distribution. The mean of 8.2 hours is substantially higher than the median of 2.6 hours because a small number of tickets waited several days. The 90th percentile is 21.4 hours, which is more relevant to a service-level concern than the mean alone. Region-level box plots show that the apparently problematic region has a higher proportion of complex technical tickets and a larger share of weekend submissions.

The team compares regions within issue type and separates weekday from weekend tickets. The regional difference becomes smaller after these stratifications, suggesting that workload composition and timing contribute to the original gap. However, one region still has a higher upper tail, indicating a possible staffing problem for complex cases.

Approach and Solution

The team reports median, IQR, 90th percentile, and the proportion responded to within four hours, rather than relying only on the mean. It recommends a dashboard with definitions, denominators, time-zone rules, and filters for issue type and submission period. Operations schedules additional weekend coverage and investigates the long-delay tickets individually. Automatic closures are reported separately rather than merged with human-response times.

Outcome

The organization avoids an unjustified conclusion that one regional team is generally inefficient. It identifies a narrower operational issue: delayed responses among complex weekend tickets. The revised metrics support a more targeted staffing decision and create a reproducible basis for later service-level modelling.

Learning Points

The case demonstrates that data quality, distributional shape, robust summaries, subgroup analysis, and clear definitions are inseparable. A single average can conceal operationally important tail behaviour, while an unexamined data-processing error can create a false regional comparison.

Risks, Limitations and Common Mistakes

Several recurring mistakes weaken exploratory analysis. Analysts may confuse correlation with causation, interpret a sample as representative without examining selection, or report a mean for a heavily skewed distribution. They may delete outliers without investigation, code missing values as zero, compare groups with different denominators, or create charts before defining the unit of analysis.

Another risk is p-hacking or “HARKing”—presenting an exploratory pattern as if it had been predicted in advance. EDA can generate valuable hypotheses, but confirmatory analyses should disclose how hypotheses and variables were selected. Multiple comparisons, researcher degrees of freedom, and unstable small-group estimates should be acknowledged.

Data leakage is a technical and statistical problem in predictive work. A variable recorded after the outcome or influenced by the decision being predicted may produce impressive but unusable performance. Temporal ordering and the intended deployment context must therefore be considered during data preparation.

Ethical, Governance and Professional Considerations

Statistical analysis carries responsibilities beyond numerical correctness. Analysts should protect personal information through data minimization, access controls, de-identification where appropriate, and compliance with applicable privacy and sector regulations. Re-identification risk can remain even after direct identifiers are removed when multiple quasi-identifiers are combined.

Definitions and quality limitations should be communicated to decision-makers, especially when metrics affect employment, credit, healthcare, education, or public services. Fairness assessment may require subgroup-specific quality checks, missingness analysis, and examination of whether measurement processes disadvantage particular groups. Transparency does not mean publishing confidential records; it means making the data provenance, assumptions, transformations, and limitations understandable to authorized stakeholders.

Digital Tools and Emerging Practice

Modern data-science platforms support profiling, automated validation, visualization, notebook-based analysis, and data-lineage tracking. Great Expectations-style validation frameworks, SQL quality checks, version control, and reproducible pipelines can detect schema changes, duplicate records, and invalid ranges before analysis reaches production. Automated anomaly detection is useful for triage but cannot determine whether an unusual value is erroneous without contextual judgment.

AI-assisted coding and visualization can accelerate EDA, but generated transformations and interpretations require verification. An automated system may silently infer incorrect types, remove unusual observations, or produce a persuasive chart with a misleading denominator. Current practice is moving toward data contracts, observability, documented lineage, and continuous monitoring of data distributions. These developments strengthen, but do not replace, statistical reasoning.

Best Practices for Exploratory Analysis

A professional analyst should:

• begin with the population, unit, decision, and measurement definitions;
• preserve raw data and maintain a documented transformation trail;
• inspect missingness, duplicates, ranges, distributions, and subgroup coverage;
• use summaries suited to the scale and shape of each variable;
• report robust and conventional summaries when extremes are plausible;
• visualize denominators, uncertainty, and relevant subgroups;
• investigate rather than automatically delete unusual observations;
• separate exploratory findings from confirmatory conclusions;
• assess whether the sample and measurement process support generalization;
• make analyses reproducible and communicate limitations clearly.

Chapter Summary

Statistical thinking treats data as evidence generated by a measurement and sampling process. Populations, samples, parameters, statistics, variables, and measurement scales provide the conceptual vocabulary for describing that evidence. Sampling can introduce random variation and systematic selection bias, while non-sampling errors arise from coverage, nonresponse, measurement, processing, and survivorship processes.

Sound analysis begins with data-quality assessment and proceeds through descriptive summaries and exploratory visualizations. Means and standard deviations are useful in suitable distributions, while medians, IQRs, percentiles, trimmed means, and other robust summaries are valuable when data are skewed or contain influential extremes. EDA is an iterative investigation of distributions, relationships, anomalies, and subgroup differences. Its results guide later modelling but should not be presented as automatically confirmatory.

Professional practice requires reproducibility, transparent definitions, privacy protection, fairness awareness, and careful communication of uncertainty. The central lesson is that sophisticated computation cannot compensate for unclear questions, poor measurement, biased sampling, or unexamined data quality.

Key Terms

Population: The complete set of units relevant to a statistical question.

Sample: A subset of a population that is observed or analysed.

Parameter: A numerical characteristic of a population.

Statistic: A numerical characteristic calculated from a sample.

Variable: A characteristic that can take different values across observational units.

Observational unit: The entity on which measurements are recorded.

Nominal scale: A measurement scale consisting of unordered categories.

Ordinal scale: A scale with ordered categories but not necessarily equal intervals.

Interval scale: A scale with equal differences but no meaningful absolute zero.

Ratio scale: A scale with equal differences and a meaningful zero.

Sampling error: Random discrepancy caused by observing a sample rather than a whole population.

Sampling bias: Systematic distortion caused by the selection process.

Nonresponse error: Bias arising when selected units do not respond and differ from respondents.

Measurement error: Difference between a recorded value and the intended true or construct value.

Missingness: The absence of a value that was expected or potentially relevant.

Outlier: An observation unusually distant from the main pattern of a distribution.

Robust statistic: A statistic relatively resistant to extreme values or assumption violations.

Interquartile range: The difference between the third and first quartiles.

Exploratory data analysis: Systematic use of summaries and visualizations to investigate data structure and patterns.

Data provenance: Documentation of a dataset’s origin, collection, transformation, and use.

Review Questions

Short Answer Questions

1. Distinguish a population parameter from a sample statistic.

2. Why is the arithmetic mean inappropriate for some nominal variables?

3. Define sampling bias and give one example.

4. What information does the interquartile range provide?

5. Why should missing values not automatically be recoded as zero?

Long Answer Questions

1. Explain the four measurement scales and discuss the statistical operations appropriate for each.

2. Compare simple random, stratified, cluster, and convenience sampling, including their practical trade-offs.

3. Describe a systematic data-quality audit for a newly received dataset.

4. Explain how skewness and outliers influence the choice between mean, median, standard deviation, and IQR.

5. Discuss EDA as an iterative process and explain why exploratory findings should be distinguished from confirmatory conclusions.

Analytical / Application Questions

1. A delivery dataset has a mean delay of 14 minutes, a median of 5 minutes, and a few delays above 300 minutes. Which summaries and visualizations would you report, and why?

2. A survey of employee wellbeing is completed only by volunteers using an internal website. Identify likely errors and propose ways to improve the design or qualify the conclusions.

3. A regional response-time difference disappears after separating weekend from weekday tickets and simple from complex issues. Explain how this changes the interpretation and what further analysis is warranted.

22. Multiple Choice Questions with Answers

Question 1

A researcher wants to estimate the average income of all households in a city but surveys 1,000 selected households. The average income computed from those households is a:

A. Population parameter

B. Sample statistic

C. Sampling frame

D. Measurement scale

Correct Answer: B. Sample statistic

Question 2

Which measurement scale permits meaningful statements that one value is twice another?

A. Nominal

B. Ordinal

C. Interval

D. Ratio

Correct Answer: D. Ratio

Question 3

A large online survey is open to anyone who chooses to respond. Its main inferential risk is:

A. Guaranteed low sampling error

B. Voluntary-response selection bias

C. Automatic measurement validity

D. Excessive stratification

Correct Answer: B. Voluntary-response selection bias

Question 4

For a strongly right-skewed distribution with several very large observations, the most robust typical-value summary is usually the:

A. Median

B. Range

C. Arithmetic mean

D. Variance

Correct Answer: A. Median

Question 5

Which action is most appropriate when an extreme transaction value is discovered?

A. Delete it immediately

B. Replace it with the mean without documentation

C. Verify its source and assess whether it is valid

D. Convert it to a missing value automatically

Correct Answer: C. Verify its source and assess whether it is valid

Question 6

A dataset records missing income values mainly among respondents who decline sensitive questions. This pattern suggests that missingness:

A. Can safely be treated as zero

B. May be related to the unobserved value or response behaviour

C. Is always completely random

D. Is a form of ratio measurement

Correct Answer: B. May be related to the unobserved value or response behaviour

Question 7

Which visualization is most directly suited to examining the relationship between two quantitative variables?

A. Pie chart

B. Scatter plot

C. Single frequency table

D. Bar chart of one category

Correct Answer: B. Scatter plot

Question 8

In the Northstar Commerce case, why was the mean response time insufficient on its own?

A. The variable was nominal

B. All tickets had identical response times

C. The distribution was right-skewed and the upper tail mattered operationally

D. The sample contained no timestamps

Correct Answer: C. The distribution was right-skewed and the upper tail mattered operationally

Question 9

Which statement best describes sampling error?

A. It is always caused by inaccurate instruments

B. It is random discrepancy arising because only part of a population is observed

C. It disappears whenever a sample is large

D. It is identical to processing error

Correct Answer: B. It is random discrepancy arising because only part of a population is observed

Question 10

An analyst reports a regional difference in service time without accounting for the fact that regions receive different proportions of complex tickets. The principal concern is:

A. Confounding or compositional differences

B. Nominal coding of the ticket identifier

C. Use of a meaningful zero

D. Excessive data completeness

Correct Answer: A. Confounding or compositional differences