Statistics and Data Analytics

Introduction

Statistics provides methods for collecting, organizing, summarizing, analyzing, and communicating data. Data analytics applies these methods, often with computing tools, to support explanation, prediction, and decision-making. For computer applications professionals, statistical thinking is essential in software testing, user-experience research, business intelligence, cybersecurity, database analysis, artificial intelligence, and performance monitoring.

A statistical conclusion is never merely a calculation. It depends on how data were defined, measured, sampled, cleaned, analyzed, and interpreted. A technically correct average can still mislead if the sample is biased, the measurement is inappropriate, or uncertainty is ignored. This chapter therefore treats statistics as both a mathematical discipline and a disciplined process of reasoning under uncertainty.

Learning Objectives

After studying this chapter, you should be able to:

• distinguish populations, samples, parameters, and statistics;
• summarize quantitative and categorical data appropriately;
• calculate and interpret measures of center and variation;
• explain sampling error, bias, and sampling methods;
• estimate population parameters and construct confidence intervals;
• formulate and interpret hypothesis tests;
• analyze correlation and simple linear regression;
• identify confounding, misleading graphs, and inappropriate conclusions;
• discuss privacy, fairness, reproducibility, and responsible data use.

1. Statistical Thinking and Data Types

A population is the complete set of units about which a study seeks knowledge. A sample is the subset actually observed. A parameter describes a population, whereas a statistic describes a sample. For example, the average response time of every transaction in a year is a parameter if the entire year is examined; the average response time of 1,000 selected transactions is a statistic.

An observation is a recorded value for one unit. Variables may be:

• Categorical, such as operating system or payment method;
• Numerical, such as age, revenue, or response time;
• Discrete, taking countable values, such as number of defects;
• Continuous, taking values on an interval, such as execution time;
• Binary, having two categories, such as pass/fail.

Measurement scales also matter. Nominal data identify categories without order. Ordinal data have order but not necessarily equal intervals. Interval data have meaningful differences but no meaningful zero, while ratio data have both meaningful differences and a true zero. Arithmetic operations must respect the scale: calculating a mean of product codes is meaningless, while calculating a mean of response times is generally meaningful.

A typical analytical workflow is:

1. define the question and population;
2. identify variables and measurement procedures;
3. collect or obtain data ethically;
4. inspect, clean, and document the data;
5. summarize and visualize;
6. model uncertainty and test relevant claims;
7. communicate conclusions, limitations, and possible actions.

2. Descriptive Statistics

Descriptive statistics summarize observed data; they do not automatically generalize beyond those data.

2.1 Measures of Center

For observations x₁, x₂, ..., x_n, the arithmetic mean is

where n is the number of observations and x̄ is the sample mean. The mean uses every value and is sensitive to extreme observations.

The median is the middle ordered value. If n is odd, it is the observation at position (n+1)/2. If n is even, it is the average of the two middle observations. The median is less affected by outliers and is often preferable for skewed variables such as income or website latency.

The mode is the most frequent value or category. It is especially useful for categorical data.

2.2 Measures of Variation

The range is the maximum minus the minimum. The interquartile range is

where Q₁ and Q₃ are the first and third quartiles. The IQR describes the spread of the middle 50% of observations.

For a sample, the sample variance is

and the sample standard deviation is

The denominator n−1 is used when estimating population variability from a sample; it provides a correction for estimating the mean. If the complete population is being described, population variance uses denominator N, where N is the population size. Standard deviation has the same units as the original variable, whereas variance has squared units.

The coefficient of variation is

usually expressed as a percentage by multiplying by 100. It is meaningful only when the mean is positive and the zero point is meaningful. It compares relative variability across measurements with different scales.

Worked Example 1: Summarizing Response Times

Suppose five response times, in milliseconds, are 120, 130, 130, 140, and 280. The mean is

The median is 130 ms. The unusually large value of 280 ms raises the mean substantially, so reporting only the mean would hide the typical user experience. The range is 160 ms. In practice, the median, IQR, and a high percentile such as the 95th percentile may communicate system performance more effectively than the mean alone.

2.3 Tables and Graphs

Frequency tables show counts or proportions. A relative frequency is

A bar chart is suitable for categories; a histogram is suitable for numerical measurements grouped into intervals. A box plot displays the median, quartiles, and potential extreme observations. A time-series plot preserves temporal order and can reveal trends, seasonality, or sudden failures.

Graphs can mislead when axes are truncated, intervals are unequal, three-dimensional effects distort areas, or percentages lack denominators. Every graph should identify the population or sample, units, time period, and source.

3. Sampling and Data Quality

A probability sample gives each population unit a known, non-zero chance of selection. In a simple random sample, every sample of a specified size has equal probability. Stratified sampling divides the population into relevant groups and samples within each group. Cluster sampling selects naturally occurring groups, such as branches or classrooms. Systematic sampling selects every kth unit after a random starting point.

Non-probability methods include convenience sampling, volunteer sampling, and quota sampling. They can be useful for exploratory work but do not generally support unbiased population estimates.

Sampling error is random variation caused by observing a sample rather than the whole population. Bias is systematic distortion. Important forms include:

• coverage bias, when some population members cannot be selected;
• nonresponse bias, when respondents differ from nonrespondents;
• measurement bias, caused by flawed questions or instruments;
• survivorship bias, when only successful or surviving cases are observed;
• selection bias, when inclusion depends on variables related to the outcome.

Increasing sample size reduces random sampling error under appropriate sampling, but it does not remove systematic bias. A very large biased dataset can produce a very precise wrong answer.

Data quality also concerns missing values, duplicate records, inconsistent units, invalid timestamps, and changes in data-collection procedures. Cleaning rules should be documented because deleting observations can itself introduce bias.

4. Probability, Sampling Distributions, and the Central Limit Idea

A probability model assigns numerical probabilities to possible outcomes. For an event A, its probability satisfies 0 ≤ P(A) ≤ 1. For mutually exclusive events, probabilities add.

If independent events A and B are considered, then

Independence is an assumption, not a consequence of merely observing two variables.

A sampling distribution is the probability distribution of a statistic over repeated samples from the same population. If observations are independent with population mean μ and finite population standard deviation σ, then the sample mean has

and, for sampling with replacement or a sufficiently small sampling fraction,

In practice, σ is usually unknown and is estimated by s, giving s/√n. The central limit theorem states, broadly, that the sampling distribution of the mean becomes approximately normal as n grows under suitable independence and finite-variance conditions. It does not say that the original data become normally distributed.

5. Estimation and Confidence Intervals

A point estimate is a single value, such as x̄ estimating μ. An interval estimate gives a range of plausible parameter values and reflects sampling uncertainty.

For a population mean with unknown standard deviation, a common confidence interval is

where t* is a critical value from the t distribution with n−1 degrees of freedom. The formula assumes an independent random sample and either an approximately normal population or a sufficiently large sample without severe outliers. The margin of error is t* s/√n.

For a population proportion, with sample proportion p̂ and sufficiently large expected counts, an approximate interval is

where z* is the standard-normal critical value. Approximation conditions should be checked; for small counts, exact or alternative methods may be preferable.

A 95% confidence procedure means that, over repeated samples and application of the same method, about 95% of intervals would contain the fixed population parameter. It does not mean there is a 95% probability that a particular fixed parameter lies inside the already calculated interval.

Worked Example 2: Mean Estimation

A random sample of 25 support calls has mean duration 8.4 minutes and standard deviation 2.0 minutes. Using the approximate 95% critical value t* ≈ 2.064 for 24 degrees of freedom:

Thus the interval is approximately 8.4 ± 0.826, or 7.57 to 9.23 minutes. This interval describes uncertainty due to sampling, not measurement error, nonresponse, or systematic bias.

6. Hypothesis Testing

Hypothesis testing evaluates whether observed data are sufficiently inconsistent with a specified null model. The null hypothesis, H₀, usually represents no difference, no association, or a reference value. The alternative hypothesis, H₁ or Hₐ, represents the research claim.

A test statistic measures how far the sample result is from the null value relative to its standard error. For a one-sample mean with known population standard deviation, a z statistic is

where μ₀ is the null mean. When σ is unknown, a one-sample t statistic is

The p-value is the probability, assuming H₀ is true, of obtaining a result at least as extreme as the observed result in the direction specified by the alternative. A small p-value indicates inconsistency with H₀; it does not measure the probability that H₀ is true, nor does it measure practical importance.

A significance level α, commonly 0.05, is selected before analysis. If the p-value is less than or equal to α, the result is described as statistically significant and H₀ is rejected. Otherwise, the correct conclusion is to fail to reject H₀, not to prove it true.

A Type I error rejects a true null hypothesis. A Type II error fails to reject a false null hypothesis. Statistical power is the probability of detecting a specified effect when it exists. Power increases with larger samples, larger true effects, lower variability, and a larger significance level, although increasing α also increases Type I error risk.

Statistical significance should be accompanied by an effect size and confidence interval. Multiple testing increases the chance of false positives; corrections or a pre-specified analysis plan may be required.

7. Correlation and Regression

Covariance indicates whether two numerical variables tend to vary together. The Pearson correlation coefficient is

where x̄ and ȳ are sample means. It satisfies −1 ≤ r ≤ 1. Values near 1 indicate strong positive linear association; values near −1 indicate strong negative linear association; values near 0 indicate little linear association. Correlation can be distorted by outliers and may miss nonlinear relationships.

Correlation does not establish causation. A third variable may confound the relationship, the direction may be reversed, or the association may be coincidental.

Simple linear regression models a response variable y using a predictor x:

Here ŷ is the predicted response, b₀ is the intercept, and b₁ is the estimated change in predicted y for a one-unit increase in x. The least-squares estimates minimize the sum of squared residuals ∑(yᵢ − ŷᵢ)². The coefficient of determination is

For a model with an intercept, R² is the proportion of sample variation in y explained by the fitted linear relationship. It is not proof of causation and can be high even for a poorly designed predictive use.

Regression assumptions commonly include linearity, independent errors, constant error variance, and suitable treatment of unusual observations. Predictions outside the observed range are extrapolations and may be unreliable.

8. Applied Case: Monitoring an Online Service

Background

This illustrative case concerns a university learning platform. The operations team wants to determine whether a software release changed page-response time and whether a proposed caching configuration improves performance.

Problem

The team compares 400 randomly selected transactions before the release with 400 after it. The logs contain response time, time of day, page type, browser, and server region. Some requests are duplicated because of retries.

Analysis

Analysts first remove documented duplicate log entries without deleting legitimate retries. They report medians and IQRs because response times are right-skewed, and display distributions by page type. They also stratify by server region because the post-release period has a different regional mix.

A difference in means alone could be misleading. For example, if the later period contains more lightweight pages, the apparent improvement might be caused by workload composition rather than the release. A regression model or stratified comparison can adjust for relevant measured variables, although adjustment cannot eliminate unmeasured confounding.

The team constructs confidence intervals for the difference in average response time and tests the pre-specified null hypothesis of no change. It also examines practical importance: a statistically significant reduction of 2 milliseconds may not justify deployment effort, whereas a reduction of 200 milliseconds might.

Outcome

The analysis finds a reduction in median response time, but the 95th percentile worsens for one region. The release is therefore not labeled simply “faster.” Engineers investigate regional capacity and revise the deployment plan. The dashboard is changed to show median, upper percentile, sample size, and data-quality warnings.

Learning Points

This case illustrates that appropriate summaries depend on distribution shape, sampling and data-cleaning decisions affect conclusions, subgroup analysis can reveal inequity or operational risk, and statistical significance must be combined with engineering relevance.

9. Ethics, Privacy, Fairness, and Governance

Data analytics may expose personal information even when names are removed. Quasi-identifiers such as location, time, device, and rare behavior can enable re-identification. Responsible practice includes purpose limitation, data minimization, access control, retention limits, encryption, audit trails, and documented lawful or ethical basis for processing. Aggregation and anonymization reduce risk but do not guarantee anonymity.

Fairness requires examining how data and models affect different groups. A model can have different error rates across groups because of unequal data quality, historical disadvantage, measurement differences, or choice of outcome. “Fairness” is not a single mathematical property; criteria can conflict. Analysts should define the relevant harm, evaluate subgroup performance, involve affected stakeholders, and provide routes for correction or appeal.

AI systems often produce statistical predictions rather than certainty. Training data may encode bias, and high predictive accuracy can coexist with unfair allocation. Automated decisions should be monitored after deployment, with human oversight proportionate to risk.

Reproducibility requires versioned data-processing code, documented definitions, a record of exclusions, preserved analysis plans where feasible, and clear distinction between exploratory and confirmatory findings. Analysts must avoid p-hacking, selective reporting, unjustified causal language, and presenting many subgroup results without accounting for multiple comparisons.

10. Best Practices and Common Misleading Conclusions

• Report sample size, sampling method, units, missingness, and time period.
• Use median and percentiles for skewed operational data.
• Display data before fitting a model.
• Report effect sizes and confidence intervals, not p-values alone.
• Distinguish association from causation.
• Avoid extrapolation beyond observed conditions.
• Investigate outliers rather than automatically deleting them.
• Separate training, validation, and test data in predictive projects.
• Examine subgroup performance and potential disparate impact.
• Protect personal data throughout collection, analysis, storage, and publication.

Common invalid statements include “the p-value is the probability the null is true,” “a non-significant result proves no effect,” “correlation proves causation,” and “a larger dataset guarantees an unbiased conclusion.” Each confuses a statistical procedure with a stronger claim than the evidence supports.

Contemporary Tools and Developments

Spreadsheets, SQL, Python, R, and visualization platforms support statistical work, but software does not replace study design or judgment. Automated dashboards can encourage metric fixation if definitions change without notice. Reproducible notebooks and data pipelines improve auditability. Privacy-preserving analytics, differential privacy, federated learning, causal inference, and responsible machine learning are growing areas, but each introduces assumptions and trade-offs that must be made explicit.

Chapter Summary

Statistics supports reasoning from data while acknowledging variation and uncertainty. Descriptive statistics summarize center, spread, shape, and composition. Sampling design determines whether results can generalize, and bias cannot be repaired merely by increasing sample size. Sampling distributions explain standard errors and provide the foundation for confidence intervals and hypothesis tests. A confidence interval describes the long-run performance of a method, while a p-value measures compatibility with a null model; neither alone establishes practical importance or causation. Correlation and regression describe and predict relationships but require careful assumptions and do not prove cause and effect. Responsible analytics also requires attention to data quality, privacy, fairness, reproducibility, and transparent communication.

Key Terms

Population: Complete set of units of interest.

Sample: Observed subset of a population.

Parameter: Numerical characteristic of a population.

Statistic: Numerical characteristic calculated from a sample.

Mean: Arithmetic average of numerical observations.

Median: Middle ordered observation.

Standard deviation: Measure of spread in the original units.

Variance: Mean squared deviation, with a sample correction when appropriate.

Interquartile range: Difference between the third and first quartiles.

Sampling error: Random difference between a sample result and population value.

Bias: Systematic distortion in collection, measurement, or analysis.

Standard error: Typical sampling variation of a statistic.

Confidence interval: Interval produced by a procedure designed to capture a parameter at a stated long-run rate.

Null hypothesis: Reference claim evaluated by a statistical test.

p-value: Probability, under the null model, of a result at least as extreme as observed.

Type I error: Rejection of a true null hypothesis.

Correlation: Standardized measure of linear association between two numerical variables.

Regression: Model relating a response to one or more predictors.

Confounding: Distortion caused by a third variable related to both variables of interest.

Data minimization: Collecting only data necessary for a defined purpose.

Review Questions

5 Short Answer Questions

1. Distinguish a parameter from a statistic.
2. Why can the median be preferable to the mean for highly skewed data?
3. What is the difference between sampling error and bias?
4. Interpret a p-value in the context of a null hypothesis.
5. Why does correlation not establish causation?

5 Long Answer Questions

1. Explain the statistical workflow from defining a population to communicating conclusions, including sources of uncertainty.
2. Compare simple random, stratified, cluster, systematic, and convenience sampling.
3. Derive and interpret the components of a confidence interval for a population mean.
4. Discuss Type I error, Type II error, statistical power, effect size, and practical significance.
5. Explain the assumptions and limitations of simple linear regression.

3 Analytical / Application Questions

1. A website’s mean response time decreases after a release, but the median is unchanged and the 95th percentile increases. What analyses and operational questions should be considered before declaring improvement?
2. Design a sampling strategy to estimate student satisfaction across multiple departments. Identify possible sources of bias and methods for reducing them.
3. A predictive model has high overall accuracy but substantially higher false-negative rates for one demographic group. Analyze possible causes and propose technical and governance responses.

Multiple Choice Questions with Answers

Question 1

Which measure is generally least affected by an extreme outlier?

A. Mean

B. Median

C. Variance

D. Range

Correct Answer: B. Median

Question 2

A sample has values 2, 4, 6, 8, and 10. What is its arithmetic mean?

A. 5

B. 6

C. 7

D. 8

Correct Answer: B. 6

Question 3

What does a small p-value indicate, assuming the null hypothesis is true?

A. The null hypothesis has a high probability of being true

B. The observed result is relatively unusual under the null model

C. The effect is necessarily practically important

D. The sample is free from bias

Correct Answer: B. The observed result is relatively unusual under the null model

Question 4

Which issue is systematic distortion rather than random sampling variation?

A. Sampling error

B. Rounding error

C. Selection bias

D. Standard error

Correct Answer: C. Selection bias

Question 5

In a 95% confidence procedure, approximately what happens over many repeated samples?

A. 95% of the data values fall in every interval

B. 95% of the calculated intervals contain the population parameter

C. The parameter has a 95% probability of changing

D. 5% of observations are automatically removed

Correct Answer: B. 95% of the calculated intervals contain the population parameter

Question 6

If the correlation coefficient is close to −1, the variables have what relationship?

A. Strong negative linear association

B. Strong positive linear association

C. No possible relationship

D. Identical means

Correct Answer: A. Strong negative linear association

Question 7

In the regression equation ŷ = b₀ + b₁x, what does b₁ represent?

A. The predicted value when x is zero only

B. The sample size

C. The predicted change in y for a one-unit increase in x

D. The residual standard deviation

Correct Answer: C. The predicted change in y for a one-unit increase in x

Question 8

Which practice most directly reduces the risk of collecting unnecessary personal information?

A. Data minimization

B. Increasing the sample size

C. Using a three-dimensional graph

D. Removing all confidence intervals

Correct Answer: A. Data minimization

Question 9

A large dataset is collected only from users who volunteer feedback. Which concern remains especially serious?

A. The dataset must have zero variance

B. Volunteer or self-selection bias

C. The mean cannot be calculated

D. A histogram cannot be created

Correct Answer: B. Volunteer or self-selection bias

Question 10

What is a key danger of extrapolating a regression model beyond the observed predictor range?

A. Predictions may be unreliable because the model is used outside its data-supported region

B. The correlation must become exactly zero

C. The sample automatically becomes random

D. The intercept is necessarily eliminated

Correct Answer: A. Predictions may be unreliable because the model is used outside its data-supported region