6. Statistical Estimation and Uncertainty

Introduction and context

Data science rarely has access to an entire population. Instead, analysts observe a sample and use it to learn about unknown features of the population, such as an average transaction value, a failure probability, a regression coefficient, or the proportion of customers likely to churn. Statistical estimation provides the formal language and methods for making such inferences.

An estimate is not the parameter itself. It is evidence-based information about the parameter, produced under assumptions about how the data were generated. Consequently, a responsible data scientist must communicate both an estimate and the uncertainty surrounding it. A sample mean of 72.4 may be useful, but its interpretation changes substantially if the associated interval is (71.9, 72.9) rather than (55, 90).

This chapter develops point and interval estimation, with particular attention to the method of moments and maximum likelihood. It also examines likelihood, sufficiency, consistency, efficiency, confidence intervals, bootstrap methods, and practical communication of uncertainty. These ideas connect probability models with statistical decision-making and form a foundation for inference, experimentation, predictive modelling, and risk analysis.

Learning objectives

By the end of this chapter, you should be able to:

1. Distinguish parameters, statistics, estimators, estimates, and pivotal quantities.

2. Construct estimators using the method of moments and maximum likelihood.

3. Interpret likelihood functions without confusing them with probability distributions for parameters.

4. Assess estimators using bias, variance, mean squared error, consistency, efficiency, and sufficiency.

5. Construct and interpret confidence intervals, including bootstrap intervals.

6. Explain how sample size, variability, and confidence level affect interval width.

7. Recognise assumptions, sources of uncertainty, and common inferential errors.

8. Communicate statistical uncertainty clearly in professional and managerial settings.

From a sample to an unknown parameter

Let X1, X2, ..., Xn denote observations from a population described by a model with unknown parameter θ. A statistic T(X1, ..., Xn) is any function of the observed data that does not depend on unknown quantities. When a statistic is designed to provide information about θ, it is called an estimator and is often written as θ-hat. Once data have been observed, the numerical value of the estimator is called an estimate.

For example, if customer spending is modelled with population mean μ, the sample mean X-bar is an estimator of μ. Before observing data, X-bar is a random variable and has a sampling distribution. After observing the sample, X-bar may equal 72.4; this is the estimate.

A parameter is fixed but unknown within a specified model, whereas an estimator varies from sample to sample. This distinction is essential. An estimate can be close to the parameter in one sample and less accurate in another, even when the same method is used.

Sampling distributions

The sampling distribution of an estimator describes its behaviour over repeated samples from the same population. It is the basis for measuring uncertainty. If observations are independent with mean μ and variance σ², then the sample mean has

E(X-bar) = μ and Var(X-bar) = σ²/n.

Thus, increasing the sample size reduces the variance of the sample mean, although it does not automatically remove bias caused by poor sampling or model misspecification. Under suitable regularity conditions, the central limit theorem implies that a suitably standardised sample mean is approximately normal for sufficiently large n. The phrase “sufficiently large” depends on the population distribution, skewness, dependence, and the purpose of the analysis.

Constructing estimators

Method of moments

The method of moments matches sample moments with corresponding population moments. The first population moment is E(X), the second is E(X²), and so forth. If a model has one unknown parameter, the first sample moment is usually equated to the first population moment. For p unknown parameters, p equations are formed using the first p moments.

Suppose X follows a Poisson distribution with parameter λ. Since E(X) = λ, the method-of-moments estimator is

λ-hat = X-bar.

As another example, suppose a uniform distribution on (0, θ) is assumed. Its mean is θ/2. Equating the sample mean to θ/2 gives

θ-hat-MM = 2X-bar.

The method is computationally simple and often provides a useful starting value for more sophisticated estimation. However, moment estimators may be inefficient, may not respect parameter constraints in unusual settings, and need not have desirable small-sample properties.

Maximum likelihood estimation

Maximum likelihood selects the parameter value under which the observed data are most plausible. For observations x1, ..., xn with joint density or probability mass function f(x1, ..., xn; θ), the likelihood is

L(θ; x) = f(x1, ..., xn; θ),

viewed as a function of θ with the data held fixed. The maximum likelihood estimator is the value θ-hat that maximises L(θ; x). In practice, analysts usually maximise the log-likelihood

l(θ) = log L(θ; x),

because products become sums and numerical calculations become more stable.

Consider independent Bernoulli observations Xi with success probability p. The likelihood is

L(p) = p raised to the number of successes times (1-p) raised to the number of failures.

The log-likelihood is maximised at p-hat = X-bar, the observed proportion of successes. Thus, the sample proportion is the maximum likelihood estimator of a Bernoulli probability.

For independent observations from a normal distribution with unknown mean μ and known variance σ², maximising the likelihood yields μ-hat = X-bar. If both μ and σ² are unknown, the maximum likelihood estimator of μ remains X-bar, while the maximum likelihood estimator of variance is

σ²-hat-MLE = (1/n) sum (Xi - X-bar)².

This differs from the unbiased sample variance, which divides by n-1. The MLE is biased downward for finite n, but the bias tends to zero as n increases.

Likelihood and log-likelihood in practice

Likelihood is relative evidence about parameter values given the observed data. It is not a probability distribution over θ unless a prior distribution is introduced in a Bayesian model. Comparing L(θ1) and L(θ2) indicates how much more compatible the data are with one parameter value than another under the assumed model.

The score function is the derivative of the log-likelihood, U(θ) = d l(θ)/dθ. A solution to U(θ) = 0 is a candidate maximum, but it must be checked against boundaries and the curvature of the likelihood. The observed information is the negative second derivative of the log-likelihood. Greater curvature near the maximum generally indicates more precise estimation.

In data science, likelihood optimisation underlies logistic regression, Poisson regression, survival models, mixture models, and many machine-learning procedures. Numerical methods such as Newton–Raphson, quasi-Newton optimisation, and expectation-maximisation may be required when a closed-form solution is unavailable.

How estimators are evaluated

No single property completely characterises an estimator. Evaluation requires considering both statistical performance and the practical consequences of errors.

Bias and variance

The bias of an estimator T for θ is

Bias(T) = E(T) - θ.

An unbiased estimator has expected value equal to θ. Unbiasedness is attractive, but it is not always essential. A biased estimator with much smaller variance may have lower overall error.

The mean squared error is

MSE(T) = E[(T - θ)²] = Var(T) + Bias(T)².

This decomposition expresses the bias–variance trade-off. In predictive analytics, reducing variance through regularisation may introduce a small bias while substantially improving generalisation. In estimation, the appropriate criterion depends on the cost of different errors and the intended use of the estimate.

Consistency

An estimator Tn is consistent for θ if it converges in probability to θ as sample size increases. Informally, repeated estimation with increasingly large samples becomes increasingly concentrated around the true parameter. Consistency is an asymptotic property; it does not guarantee good performance at a particular sample size.

For example, under independent sampling with finite variance, X-bar is consistent for μ because its variance decreases as n increases. An estimator can be biased in finite samples and still be consistent if its bias disappears and its variability shrinks with n.

Efficiency

When comparing unbiased estimators of the same parameter, an estimator with smaller variance is generally considered more efficient. The Cramér–Rao lower bound gives a theoretical lower bound on the variance of unbiased estimators under regularity conditions:

Var(T) is at least 1 divided by the Fisher information.

The Fisher information measures how much information the sample contains about θ. For independent observations, information commonly adds across observations. Maximum likelihood estimators are often asymptotically efficient: for large samples, their variance approaches the lowest achievable variance under the model.

Efficiency should not be interpreted mechanically. A highly efficient estimator under a badly specified model can be misleading. Robustness, interpretability, computational cost, and sensitivity to outliers may matter more than a small theoretical variance advantage.

Sufficiency

A statistic S(X) is sufficient for θ if, after observing S(X), the remaining details in the sample provide no additional information about θ. The factorisation theorem provides a practical criterion: if the joint likelihood can be written as

f(x; θ) = g(S(x), θ)h(x),

where h does not depend on θ, then S is sufficient.

For independent Bernoulli observations, the total number of successes is sufficient for p. For a normal sample with known variance, the sample mean is sufficient for μ. Sufficiency supports data reduction, though it does not imply that a statistic is unbiased, efficient in every sense, or appropriate when the model is wrong.

Interval estimation and confidence

A point estimate alone hides sampling variability. An interval estimator gives a range of plausible values produced by a stated procedure. A 95% confidence interval is a procedure that, under repeated sampling and valid assumptions, contains the fixed parameter in approximately 95% of constructed intervals.

The frequentist interpretation is often misunderstood. After an interval has been calculated, the parameter is not assigned a 95% probability of lying inside that particular interval in the frequentist framework. The 95% refers to the long-run coverage of the method. Statements such as “we are 95% confident” are acceptable practical shorthand if the procedure and assumptions are understood.

Normal-based intervals

If the population standard deviation σ is known and the population is normal, a confidence interval for μ is

X-bar plus or minus z_(α/2) σ divided by square root of n,

where 1 − α is the confidence level. When σ is unknown and the population is normal, the t distribution is used:

X-bar plus or minus t_(α/2,n−1) s divided by square root of n.

The t distribution has heavier tails than the standard normal distribution, particularly for small samples, reflecting additional uncertainty from estimating σ. As n increases, the t distribution approaches the standard normal distribution.

For a proportion p, the simple Wald interval p-hat plus or minus z times the estimated standard error can perform poorly with small samples or probabilities near zero or one. Wilson or exact methods are often preferable. A data scientist should select an interval method based on the sampling design and model, not merely because it is familiar.

Bootstrap confidence intervals

When an analytic sampling distribution is difficult to derive, the bootstrap approximates it by repeatedly resampling observations with replacement from the observed sample. For each resample, the statistic is recalculated. The resulting empirical distribution approximates the sampling distribution of the statistic.

A percentile bootstrap interval takes selected quantiles of the bootstrap estimates. Other methods, such as bias-corrected and accelerated intervals, adjust for skewness and bias. Bootstrap methods are powerful but not magical: they can fail when the sample is unrepresentative, observations are dependent, the statistic is unstable, or the sample is too small to reflect the population.

Width, precision, and sample size

The width of an interval is influenced by variability, sample size, and confidence level. For a mean, the standard error is proportional to 1 divided by the square root of n. Therefore, reducing the standard error by half generally requires approximately four times as many independent observations. A higher confidence level produces a wider interval, while lower variability produces a narrower interval.

Precision is not the same as accuracy. A narrow interval can be precisely wrong when systematic bias, selection bias, measurement error, dependence, or model misspecification is ignored. Uncertainty communication must therefore distinguish sampling uncertainty from other sources of uncertainty.

A practical workflow for estimation

A defensible estimation analysis usually follows these stages:

1. Define the target parameter and the population to which the conclusion will apply.

2. Clarify the sampling process, unit of analysis, measurement scale, and dependence structure.

3. Choose a probability model or a robust alternative, documenting assumptions.

4. Construct a point estimator using moments, likelihood, or another justified method.

5. Evaluate bias, variance, consistency, efficiency, and sensitivity where relevant.

6. Quantify uncertainty using an appropriate analytical, bootstrap, or simulation-based interval.

7. Diagnose outliers, influential observations, missingness, and model misfit.

8. Report the estimate, interval, sample size, method, assumptions, and practical meaning.

This workflow prevents the common mistake of treating estimation as a purely computational task. The validity of an interval depends at least as much on how data were collected and modelled as on the formula used to calculate it.

Illustrative case: estimating service reliability

Background

A cloud-service provider monitors 400 randomly selected service requests to estimate the probability p that a request completes without a critical error. Of the requests, 376 are successful.

Situation and analysis

The natural point estimate is p-hat = 376/400 = 0.94. Under an independent Bernoulli model, the estimated standard error is approximately the square root of p-hat(1 − p-hat)/n, which is about 0.0119. A 95% normal approximation interval is therefore approximately 0.94 plus or minus 1.96(0.0119), or (0.917, 0.963).

However, the analyst investigates the data-generating process before publishing this result. Requests were sampled across time, but several requests came from the same customer session. This clustering violates the simple independence assumption and may make the interval too narrow. The analyst also discovers that automated retries were counted as successful completions, although the service-level agreement defines success as completion on the first attempt.

Approach and solution

The team revises the definition of success, removes duplicate retry records according to a pre-specified rule, and uses a cluster-robust or session-level analysis. A bootstrap that resamples sessions rather than individual requests is used as a sensitivity analysis. The report presents the revised estimate and interval, explains the sampling unit, and separately reports first-attempt and eventual-success rates.

Outcome

The original estimate appeared highly precise but answered an ambiguous question and understated uncertainty. The revised analysis produced a more defensible measure for operational decisions, even though its interval was wider.

Learning points

The case illustrates that an estimator is inseparable from the estimand, sampling design, and measurement definition. A narrower interval is not automatically better. Transparent reporting of uncertainty can lead to better resource planning, more credible service commitments, and fewer disputes between technical and managerial teams.

Common errors, limitations, and professional responsibilities

Several errors recur in applied estimation. Analysts may confuse an estimate with a parameter, interpret a confidence level as a posterior probability, use a normal approximation with a very small or extreme sample, or report excessive decimal places. They may also ignore dependence, treat nonresponse as random without evidence, or calculate a highly precise interval from biased data.

Model selection creates another risk. Maximum likelihood is conditional on the assumed likelihood. If the distribution, independence assumption, or link function is inappropriate, optimisation can produce a precise estimate of the wrong quantity. Sensitivity analysis using alternative models, robust standard errors, resampling schemes, or simulation is therefore valuable.

Ethically, uncertainty should not be hidden when estimates affect individuals or groups. Credit limits, fraud alerts, medical prioritisation, and employee evaluations may rely on estimated probabilities. Decision-makers should know the uncertainty, relevant subgroup performance, data limitations, and consequences of false positives and false negatives. Privacy-preserving aggregation may also be necessary when releasing estimates from sensitive data. Governance should include reproducible code, versioned datasets, documented assumptions, and an audit trail for changes to estimands or models.

Technology and emerging practice

Statistical software makes estimation accessible through optimisation libraries, resampling frameworks, probabilistic programming, and automated reporting. Automatic differentiation enables efficient likelihood optimisation in complex models. Bayesian computation, including Markov chain Monte Carlo and variational inference, extends uncertainty analysis by representing uncertainty about parameters with probability distributions; however, these methods require prior specification and diagnostics.

Modern machine-learning systems increasingly use conformal prediction, calibrated probabilities, and distribution-free or weak-assumption intervals to quantify predictive uncertainty. These tools address uncertainty about future observations rather than only uncertainty about a population parameter. Their guarantees depend on assumptions such as exchangeability and may weaken under distribution shift. The distinction between parameter uncertainty, sampling uncertainty, measurement uncertainty, and predictive uncertainty should remain explicit.

Best practices for communicating uncertainty

A professional report should state what was estimated, for which population, from what sample, using which method. It should present the point estimate and interval together, use units and meaningful rounding, and explain the practical implication. Charts should show interval bars or bands where appropriate rather than displaying estimates alone.

Avoid claims such as “there is a 95% chance that the true value is in this interval” unless using a Bayesian interpretation with a clearly specified posterior. Avoid describing a non-significant result as proof of no effect; a wide interval may simply indicate insufficient information. Finally, distinguish statistical significance from practical importance. An estimate can be statistically distinguishable from zero yet operationally trivial, while a practically important effect may be estimated too imprecisely to reach a conventional threshold.

Chapter Summary

Statistical estimation uses sample data to learn about unknown population parameters. Point estimators such as the sample mean, method-of-moments estimators, and maximum likelihood estimators provide numerical summaries, while confidence intervals and bootstrap procedures quantify sampling uncertainty. Estimators should be assessed through bias, variance, mean squared error, consistency, efficiency, and sufficiency rather than by a single criterion. Likelihood measures relative compatibility of parameter values with observed data, and log-likelihood simplifies computation. Interval width depends on variability, sample size, and confidence level, but precision does not guarantee accuracy when data or models are biased. Sound professional practice requires explicit estimands, appropriate sampling assumptions, sensitivity analysis, transparent uncertainty communication, and ethical attention to the consequences of statistical decisions.

Key Terms

Parameter: A fixed but unknown numerical characteristic of a population or probability model.

Statistic: A function of observed data that does not depend on unknown parameters.

Estimator: A statistic intended to estimate an unknown parameter.

Estimate: The numerical value obtained when an estimator is applied to observed data.

Sampling distribution: The probability distribution of a statistic over repeated samples.

Bias: The difference between an estimator’s expected value and the parameter it estimates.

Variance: A measure of the dispersion of an estimator across repeated samples.

Mean squared error: The expected squared difference between an estimator and its target, equal to variance plus squared bias.

Consistency: The property of converging in probability to the target parameter as sample size increases.

Efficiency: Relative precision of an estimator, commonly assessed through variance under comparable conditions.

Sufficiency: The property whereby a statistic retains all sample information relevant to a parameter under a model.

Likelihood: A function expressing the relative compatibility of parameter values with observed data.

Log-likelihood: The logarithm of the likelihood, usually maximised for computational convenience.

Maximum likelihood estimator: The parameter value that maximises the likelihood of the observed data.

Method of moments: An estimation method that equates sample moments with population moments.

Confidence interval: An interval produced by a procedure designed to achieve stated long-run coverage.

Standard error: The standard deviation of an estimator’s sampling distribution, or its estimate.

Bootstrap: A resampling method that approximates sampling variability using repeated samples from observed data.

Fisher information: A measure of the information a sample contains about an unknown parameter.

Estimand: The precisely defined population quantity that an analysis seeks to estimate.

Review Questions

Short Answer Questions

1. Distinguish an estimator from an estimate using an example.

2. Define consistency and explain why it is an asymptotic property.

3. What is the difference between likelihood and probability?

4. State the bias–variance decomposition of mean squared error.

5. Why is the t distribution used when estimating a normal mean with unknown variance?

Long Answer Questions

1. Explain the method of moments and maximum likelihood, including their advantages and limitations.

2. Discuss bias, variance, mean squared error, consistency, efficiency, and sufficiency as criteria for evaluating estimators.

3. Explain the frequentist interpretation of a 95% confidence interval and contrast it with a common incorrect interpretation.

4. Describe bootstrap confidence intervals, including situations in which ordinary bootstrap resampling may be inappropriate.

5. Develop a practical workflow for estimating a population parameter and communicating its uncertainty in a data-science project.

Analytical / Application Questions

1. A sample of 100 Bernoulli trials contains 82 successes. Explain which point estimator you would use, identify assumptions behind a simple confidence interval, and describe why an alternative interval method might be preferred.

2. An analyst reports a very narrow confidence interval based on 10,000 records, but the records come from only 50 repeated customers. Analyse why the interval may be misleading and propose a better approach.

3. Two estimators are available: one is unbiased with high variance, and the other has small bias but much lower variance. Explain how you would compare them using mean squared error and what additional practical considerations should influence the decision.

22. Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes an estimator from an estimate?

A. An estimator is fixed, whereas an estimate varies across samples.

B. An estimator is a rule or statistic; an estimate is its observed numerical value.

C. An estimator is a population parameter; an estimate is always a confidence interval.

D. There is no distinction between the two terms.

Correct Answer: B. An estimator is a rule or statistic; an estimate is its observed numerical value.

Question 2

For a Poisson model with parameter λ, which method-of-moments estimator follows directly from E(X) = λ?

A. The sample variance divided by the sample mean

B. The reciprocal of the sample mean

C. The sample mean

D. The sample maximum

Correct Answer: C. The sample mean

Question 3

Why is log-likelihood commonly maximised instead of likelihood?

A. It changes a parameter into a random variable.

B. It converts products into sums and is usually numerically more stable.

C. It guarantees that the estimator is unbiased.

D. It removes all model assumptions.

Correct Answer: B. It converts products into sums and is usually numerically more stable.

Question 4

An estimator has bias 2 and variance 9. What is its mean squared error?

A. 7

B. 9

C. 11

D. 13

Correct Answer: D. 13

Question 5

Which property means that an estimator approaches the true parameter in probability as sample size increases?

A. Sufficiency

B. Consistency

C. Unbiasedness

D. Ancillarity

Correct Answer: B. Consistency

Question 6

A 95% frequentist confidence interval is correctly interpreted as a method that:

A. Assigns a 95% probability to the fixed parameter after the interval is observed.

B. Contains 95% of the sample observations.

C. Captures the fixed parameter in approximately 95% of repeated applications under valid assumptions.

D. Guarantees that the model is correctly specified.

Correct Answer: C. Captures the fixed parameter in approximately 95% of repeated applications under valid assumptions.

Question 7

Why can a narrow confidence interval still be misleading?

A. Narrow intervals always indicate excessive sample size.

B. Sampling precision does not eliminate bias from poor sampling or model misspecification.

C. Confidence intervals cannot be calculated for large samples.

D. Narrow intervals imply that the estimator has infinite variance.

Correct Answer: B. Sampling precision does not eliminate bias from poor sampling or model misspecification.

Question 8

For a normal population mean with unknown variance and a small sample, which distribution is generally used for the standardised mean?

A. Uniform distribution

B. Poisson distribution

C. Student’s t distribution

D. Bernoulli distribution

Correct Answer: C. Student’s t distribution

Question 9

In the service-reliability case, why was resampling sessions rather than individual requests more appropriate?

A. Sessions contain no information about reliability.

B. Requests within a session may be dependent, so the session is the relevant sampling unit.

C. Session-level resampling always produces narrower intervals.

D. It converts a frequentist analysis into a Bayesian analysis.

Correct Answer: B. Requests within a session may be dependent, so the session is the relevant sampling unit.

Question 10

Which statement about maximum likelihood estimation is most accurate?

A. It is valid regardless of whether the assumed likelihood represents the data-generating process.

B. It chooses the parameter value that maximises compatibility of the observed data under the specified model.

C. It always produces an unbiased estimator in finite samples.

D. It cannot be used in regression models.

Correct Answer: B. It chooses the parameter value that maximises compatibility of the observed data under the specified model.