Statistics, Inference and Data Analytics

Introduction

Computer systems generate measurements continuously: response times, failure counts, model predictions, user interactions, network traffic and energy consumption. Statistics converts such observations into defensible conclusions while explicitly representing uncertainty. It also helps distinguish genuine improvements from random variation, association from causation and reproducible evidence from accidental findings.

A statistically sophisticated engineer must ask not only whether an algorithm produced a numerical result, but also whether the data were representative, assumptions were justified, uncertainty was quantified and conclusions can be reproduced.

Learning Objectives

After studying this chapter, students should be able to:

• summarize numerical and categorical data appropriately;
• design probability and non-probability samples and identify sampling bias;
• construct and interpret confidence intervals;
• conduct one-sample, two-sample, paired and categorical-data tests;
• distinguish pooled-variance and Welch two-sample t tests;
• fit and evaluate linear regression models;
• design randomized experiments and analyze causal claims;
• recognize confounding, selection bias, measurement error and multiple testing;
• evaluate reproducibility and statistical validity in data-driven systems.

1. Data, Variables and Measurement

A population is the full set of units relevant to a question. A sample is the subset actually observed. A population quantity, such as its mean μ, is a parameter; a quantity computed from a sample, such as \bar{x}, is a statistic.

Variables may be classified as follows:

Measurement scales affect valid operations. Averaging latency is meaningful; averaging arbitrary numerical codes assigned to operating systems is not. Data quality also depends on construct validity: a convenient metric may not represent the intended concept. For example, click-through rate is not necessarily a valid measure of user satisfaction.

2. Descriptive Analysis

2.1 Measures of location

For observations x₁,x₂,\ldots,x_n, the sample mean is

where n is the sample size. The mean has the same unit as the observations but is sensitive to extreme values.

The median is the middle ordered value, or the average of the two middle values when n is even. It is more resistant to outliers. The mode is the most frequent value and is applicable to categorical as well as numerical data.

2.2 Measures of dispersion

The sample variance and sample standard deviation are

Here s² has squared measurement units, while s has the original unit. The denominator n-1 makes s² an unbiased estimator of population variance under independent, identically distributed sampling with finite variance.

The interquartile range is

It measures the spread of the central 50% of observations and is resistant to extreme values. A commonly used exploratory rule flags values below Q₁-1.5IQR or above Q₃+1.5IQR. Such values require investigation; they are not automatically errors.

For nonnegative measurements with a meaningful positive mean, the coefficient of variation is CV=s/\bar{x}. It is dimensionless but is inappropriate when the mean is zero or close to zero.

Worked Example: Service latency

Five response times are 92, 96, 99, 103 and 110 ms. Their mean is

The squared deviations sum to 64+16+1+9+100=190 ms². Therefore,

A complete descriptive report should also use a histogram, box plot or empirical distribution because identical means and variances can conceal very different distributional shapes.

3. Sampling and Sampling Distributions

3.1 Sampling designs

A simple random sample gives each eligible unit a known and equal selection probability. Stratified sampling divides the population into meaningful strata and samples within each, often improving precision and representation. Cluster sampling selects natural groups, such as data centers, and is operationally efficient but may produce correlated observations. Systematic sampling selects every kth unit after a random start.

Convenience and volunteer samples have unknown selection probabilities. Large size does not remove their selection bias. Ten million unrepresentative records can provide a more precise estimate of the wrong population quantity.

3.2 Sampling distribution of the mean

If independent observations have population mean μ and finite variance σ², then

The standard error describes the variation of an estimator across hypothetical repeated samples; it is not the standard deviation of individual observations.

The central limit theorem states that, under standard independence and finite-variance conditions, the standardized sample mean approaches a standard normal distribution as n increases:

No universal sample size guarantees a good approximation. Strong skewness, heavy tails or dependence may require much larger samples or alternative methods.

3.3 Dependence and effective sample size

Repeated requests from the same user, time-series measurements and observations from the same server are often correlated. Treating them as independent understates uncertainty. Remedies include cluster-robust standard errors, mixed models, time-series methods, block bootstrap procedures or sampling one independent unit per cluster.

4. Estimation and Confidence Intervals

A point estimate is a single estimate of a parameter. An interval estimate quantifies uncertainty.

For an approximately normal population with unknown variance—or a sufficiently large independent sample—the confidence interval for μ is

Here t_{1-\alpha/2,n-1} is a Student t critical value, and 1-\alpha is the confidence level.

A 95% confidence procedure has 95% long-run coverage under its assumptions. After an interval is observed, the frequentist interpretation is not that the fixed parameter has a 95% probability of being inside it.

Worked Example: Mean processing time

Suppose n=25, \bar{x}=48.0 ms and s=5.0 ms. With 24 degrees of freedom, t_0.975,24\approx2.064. Thus,

and the 95% confidence interval is

Increasing sample size reduces standard error in proportion to 1/\sqrt{n}, not 1/n. Approximately four times as many observations are required to halve the standard error.

Bootstrap confidence intervals estimate sampling uncertainty by repeatedly resampling observed units with replacement. Resampling must respect the data structure: users, clusters or time blocks should be resampled when individual records are dependent.

5. Hypothesis Testing

A hypothesis test compares a null hypothesis H₀ with an alternative H₁. A test statistic measures incompatibility between the data and H₀. The p-value is the probability, assuming the null hypothesis and model assumptions are true, of obtaining a result at least as extreme as the observed one.

It is not the probability that H₀ is true. Statistical significance also does not imply practical importance.

A sound test specifies hypotheses, significance level, test statistic and decision rule before examining outcomes when possible.

5.1 One-sample t test

To test H₀:μ=μ₀, use

with n-1 degrees of freedom. It assumes independent observations and a normal population for exact small-sample inference; moderate departures become less important for sufficiently large, well-behaved samples.

5.2 Welch two-sample t test

For independent groups without assuming equal population variances, test H₀:μ₁-μ₂=Δ₀ using

The Welch–Satterthwaite degrees of freedom are

This test requires independent samples and is generally preferable when group variances or sample sizes differ.

Worked Example: Welch test

Suppose two deployment configurations produce:

Throughput is measured in requests per second. For H₀:μ₁-μ₂=0,

and

The degrees of freedom are

The two-sided p-value is approximately 0.027, so the difference is statistically significant at α=0.05. The estimated effect is 9 requests per second; its engineering importance must be assessed separately.

5.3 Pooled-variance two-sample t test

If the two populations can reasonably be assumed to have the same variance σ₁²=σ₂², the common variance is estimated by

The test statistic is

with n₁+n₂-2 degrees of freedom. This is not a general replacement for Welch’s test: it requires independent samples, approximately normal populations for small samples and a defensible equal-variance assumption.

For the preceding data,

so s_p=9.129, SE=3.909, t=2.302 and the degrees of freedom are 20. Because the observed variances differ substantially, Welch’s result is more defensible here.

5.4 Paired t test

When each unit is measured twice—such as the same server before and after optimization—analyze differences d_i. The statistic is

with n-1 degrees of freedom. Applying an independent-samples test to paired data discards the matching and can produce misleading uncertainty.

5.5 Tests for categorical data

For a contingency table, Pearson’s chi-square statistic is

where O_ij and E_ij are observed and null-expected counts. For an r\times c table, the degrees of freedom are (r-1)(c-1). Observations should be independent and expected counts sufficiently large; Fisher’s exact test is suitable for small 2 × 2 tables.

5.6 Effect size, power and multiple testing

Confidence intervals and effect sizes should accompany p-values. For two groups, a standardized mean difference may be reported as d=(\bar{x}₁-\bar{x}₂)/s_p, provided its assumptions and interpretation are suitable.

Power increases with effect size, sample size and significance level, and decreases with measurement noise. Sample-size planning should use the smallest practically meaningful effect rather than an optimistically large expected effect.

If many hypotheses are tested, false positives accumulate. The Bonferroni method controls family-wise error by testing each of m hypotheses at α/m. False-discovery-rate procedures, such as Benjamini–Hochberg, are often more powerful when controlling the expected proportion of false discoveries is appropriate.

6. Correlation and Regression

6.1 Correlation

Pearson correlation is

It measures linear association and lies between −1 and 1 when both variables vary. It is sensitive to outliers and does not establish causality.

6.2 Simple linear regression

The population model is

where \beta₀ is the intercept, \beta₁ is the expected change in response per unit increase in X, and \varepsilon_i is an error term.

Ordinary least squares minimizes \sum e_i², producing

For classical inference, the model requires linearity, independent errors, constant conditional error variance and normally distributed errors for exact small-sample tests. Normality of the predictor is not required. Residual plots should be used to assess model form, unequal variance, outliers and dependence.

The sums of squares satisfy

for an ordinary least-squares model containing an intercept, where

and

Worked Example: Regression fit

Suppose a latency regression has SST=6{,}875 ms² and SSE=27.5 ms². Then

Thus 99.6% of the observed sample variation in latency is explained by the fitted model. This does not demonstrate causality, guarantee accurate extrapolation or prove that the model is correctly specified.

A confidence interval for the mean response at a predictor value is narrower than a prediction interval for a new observation because the prediction interval includes both uncertainty in the fitted mean and irreducible observation-level variation.

6.3 Multiple regression and prediction

Multiple regression extends the model to

Each coefficient is interpreted conditionally on the other included predictors. Important concerns include multicollinearity, interactions, nonlinear relationships, omitted variables and data leakage.

Prediction accuracy must be evaluated on data not used for model fitting. Cross-validation estimates out-of-sample performance, but preprocessing, feature selection and tuning must occur within each training fold. Otherwise, information leakage produces optimistic estimates.

7. Experimental Design and Causal Reasoning

Causal questions ask what would happen under an intervention. For unit i, let Y_i(1) and Y_i(0) denote potential outcomes under treatment and control. The individual causal effect Y_i(1)-Y_i(0) cannot normally be observed because only one potential outcome is realized. Random assignment makes treatment groups comparable in expectation, supporting estimation of average causal effects.

Figure: Random assignment supports causal comparison, while results inform future experimental design.

Core design principles are:

• Randomization: balances known and unknown confounders in expectation.
• Control: provides a counterfactual comparison.
• Replication: estimates variation and improves precision.
• Blocking: randomizes within homogeneous groups such as device type or region.
• Blinding: reduces behavior and assessment changes caused by treatment knowledge.
• Factorial design: studies multiple factors and interactions efficiently.

An A/B test must define its experimental unit. Randomizing page views while outcomes depend on users can cause interference and invalid standard errors. SUTVA, the stable unit treatment value assumption, requires a well-defined treatment and no relevant interference between units.

Observational adjustment can reduce measured confounding through regression, matching, stratification or weighting. It cannot guarantee removal of unmeasured confounding. Temporal precedence and correlation alone are insufficient for causality.

8. Applied Case: Evaluating a Caching Algorithm

Background

The following is an illustrative case. A cloud platform proposes a new caching algorithm intended to reduce request latency without increasing error rates.

Problem

Historical comparisons are unreliable because the new algorithm was initially deployed in faster data centers. Data-center capacity therefore confounds algorithm choice and latency.

Approach

Engineers define users, rather than requests, as experimental units to avoid exposing one user to both versions. Eligible users are stratified by region and device class and randomly assigned within each stratum. The primary outcome is median user-level latency over seven days. Error rate is a safety outcome, and infrastructure cost is a secondary outcome.

A minimum practically important reduction of 5 ms is specified before data collection. The analysis plan fixes sample size, exclusion rules and a two-sided significance level of 0.05. Automated bots are removed using a rule defined without reference to treatment outcomes.

Analysis

The treatment effect is estimated as the difference in group means of user-level latency, with a confidence interval. Because latency is right-skewed, engineers also report medians and percentile effects. Region-adjusted regression improves precision without replacing randomization. Error rate is analyzed as a proportion, while cost is reported with uncertainty rather than selected only when significant.

The team checks allocation balance, missingness, sample-ratio mismatch and treatment contamination. It does not repeatedly inspect p-values and stop as soon as significance appears, because such optional stopping inflates false-positive risk unless a valid sequential design is used.

Outcome

Suppose the estimated mean reduction is 7.2 ms with a 95% confidence interval from 4.1 to 10.3 ms. The result supports a genuine average improvement, although the lower confidence limit falls below the predeclared 5 ms practical target. Error rates show no material deterioration, and subgroup estimates reveal weaker benefits on low-memory devices.

Learning Points

Randomization removed the historical capacity confounding. Predefined metrics limited selective reporting. User-level assignment aligned randomization with dependence in the data. Confidence intervals communicated the range of effects compatible with the observations, while subgroup findings were treated cautiously because multiple comparisons reduce reliability.

9. Bias, Validity and Reproducibility

Internal validity concerns whether a reported effect is credible for the studied units. External validity concerns whether it generalizes to other populations, environments or times.

Common threats include:

Missing data are commonly classified as missing completely at random, missing at random conditional on observed variables, or missing not at random. Complete-case analysis is not automatically valid. Multiple imputation and sensitivity analysis may be required.

Reproducibility requires more than sharing source code. A defensible analysis records data provenance, software versions, random seeds, preprocessing decisions, model configurations and the distinction between exploratory and confirmatory analyses. Containerized environments and executable workflows improve computational repeatability, but they do not correct biased data or invalid assumptions.

10. Ethics, Governance and Contemporary Analytics

Statistical systems can produce disparate outcomes even without explicitly using protected attributes. Evaluation should therefore examine performance and error rates across relevant groups while respecting privacy and legal constraints.

Data minimization, access controls, retention policies and informed consent are governance concerns, not merely implementation details. Differential privacy can limit disclosure by injecting calibrated randomness, but it creates a privacy–accuracy trade-off that must be reported.

Modern machine learning emphasizes prediction, yet inferential principles remain essential. Adaptive experimentation, online learning and automated model selection can invalidate conventional p-values when the selection process is ignored. Valid approaches include held-out confirmation, selective-inference methods, sequential testing procedures and transparent preregistration.

Best practice is to connect the question, estimand, design, data-generating process, model, uncertainty measure and decision. Sophisticated algorithms cannot compensate for an ill-defined question or biased measurement process.

Chapter Summary

Statistics supports the responsible transformation of data into evidence. Descriptive measures summarize distributions but should be accompanied by graphical and contextual analysis. Sampling design determines whether results represent the intended population, while sampling distributions and standard errors quantify estimator variability.

Confidence intervals communicate uncertainty, and hypothesis tests assess compatibility with a null model. Welch’s test accommodates unequal variances; the pooled two-sample t test requires a defensible equal-variance assumption. Effect sizes, power and multiple-testing control are necessary for meaningful interpretation.

Regression models quantify conditional relationships and support prediction, but high explanatory power does not establish causality. Randomized experiments, blocking, replication and well-defined experimental units strengthen causal conclusions. Valid data analytics also requires attention to dependence, confounding, bias, leakage, ethics, reproducibility and practical significance.

Key Terms

Population: The complete collection of units relevant to a statistical question.

Sample: A subset of population units that is observed.

Parameter: A numerical characteristic of a population.

Statistic: A numerical quantity calculated from sample data.

Standard error: The standard deviation of an estimator’s sampling distribution.

Sampling bias: Systematic error caused by nonrepresentative selection.

Confidence interval: An interval produced by a procedure with specified long-run coverage.

Null hypothesis: A formal claim evaluated by a statistical test.

P-value: The probability, under the null model, of a result at least as extreme as that observed.

Type I error: Incorrect rejection of a true null hypothesis.

Power: The probability of rejecting the null hypothesis when a specified alternative is true.

Welch t test: A two-sample mean test that does not assume equal population variances.

Pooled-variance t test: A two-sample test based on a common-variance assumption.

Effect size: A quantitative measure of the magnitude of a difference or relationship.

Regression: A model relating a response variable to one or more predictors.

Residual: The difference between an observed and fitted response.

Confounding: Mixing of an exposure effect with the influence of another variable.

Randomization: Assignment by chance to create comparable treatment groups in expectation.

Reproducibility: The ability to obtain consistent computational results from documented data and procedures.

External validity: The extent to which findings generalize beyond the studied setting.

Review Questions

Short Answer Questions

1. Distinguish a population parameter from a sample statistic.
2. Why is the standard error different from the sample standard deviation?
3. State the correct frequentist interpretation of a 95% confidence interval.
4. What assumptions justify using Pearson’s chi-square test?
5. Why does a high value of R² not establish causality?

Long Answer Questions

1. Explain probability sampling methods and compare simple random, stratified and cluster sampling.
2. Derive and interpret the one-sample t statistic, including its assumptions.
3. Explain the Welch and pooled-variance two-sample t tests. Compare their standard errors, degrees of freedom and applicability.
4. Describe the assumptions of ordinary least-squares regression and explain how residual diagnostics assess them.
5. Develop an experimental design for comparing two compiler optimization strategies, including units, randomization, outcomes, power and threats to validity.

Analytical / Application Questions

1. Two independent systems have n₁=15, \bar{x}₁=82, s₁=9, n₂=12, \bar{x}₂=75 and s₂=5. Conduct a two-sided Welch test and construct a 95% confidence interval for the mean difference.
2. A regression model reports SST=4{,}000 and SSE=600. Calculate and interpret R², then identify two reasons the model may still predict poorly after deployment.
3. An online platform tests 40 interface changes independently at the 5% significance level. Analyze the multiple-testing risk and recommend an appropriate error-control strategy.

Multiple Choice Questions with Answers

Question 1

For an independent random sample, why does the usual sample variance use n-1 in the denominator?

A. To make the sample standard deviation exactly normal

B. To make the sample variance an unbiased estimator of population variance

C. To eliminate all effects of outliers

D. To make variance have the same unit as the data

Correct Answer: B. To make the sample variance an unbiased estimator of population variance

Question 2

A sample has n=25 and s=10 ms. What is the estimated standard error of the sample mean?

A. 0.4 ms

B. 2 ms

C. 5 ms

D. 10 ms

Correct Answer: B. 2 ms

Question 3

What is a Type I error?

A. Failing to reject a false null hypothesis

B. Rejecting a true null hypothesis

C. Accepting a model with low predictive accuracy

D. Using a sample that is too small

Correct Answer: B. Rejecting a true null hypothesis

Question 4

For a Welch test, n₁=12, n₂=10, s₁²=118.8 and s₂²=40.0. What is the Welch–Satterthwaite degrees of freedom, approximately?

A. 9.00

B. 18.08

C. 20.00

D. 22.08

Correct Answer: B. 18.08

Question 5

When is the pooled-variance two-sample t test appropriate?

A. When the samples are paired observations

B. When the response variable is categorical

C. When independent populations can reasonably be assumed to have equal variances

D. Whenever the sample means differ

Correct Answer: C. When independent populations can reasonably be assumed to have equal variances

Question 6

A regression has SST=6{,}875 ms² and SSE=27.5 ms². What is R²?

A. 0.004

B. 0.040

C. 0.960

D. 0.996

Correct Answer: D. 0.996

Question 7

What is the primary inferential purpose of random assignment in an experiment?

A. To guarantee that every measured outcome is normally distributed

B. To balance potential confounders between groups in expectation

C. To ensure that the sample represents every external population

D. To remove the need for uncertainty estimates

Correct Answer: B. To balance potential confounders between groups in expectation

Question 8

Why is a prediction interval for a new regression observation generally wider than a confidence interval for the mean response?

A. It includes both fitted-mean uncertainty and individual observation variation

B. It always uses fewer observations

C. It assumes the residual variance is zero

D. It excludes uncertainty in the fitted coefficients

Correct Answer: A. It includes both fitted-mean uncertainty and individual observation variation

Question 9

Which statement correctly interprets a p-value?

A. It is the probability that the null hypothesis is true

B. It is the probability that the study will be replicated

C. It is the probability, under the null model, of a result at least as extreme as the observed result

D. It is the magnitude of the practical effect

Correct Answer: C. It is the probability, under the null model, of a result at least as extreme as the observed result

Question 10

Which practice most directly reduces data leakage in predictive modeling?

A. Performing feature selection on the full dataset before cross-validation

B. Using the test set repeatedly during model tuning

C. Running preprocessing and feature selection separately inside each training fold

D. Selecting the model with the largest training-set accuracy

Correct Answer: C. Running preprocessing and feature selection separately inside each training fold