Probability and Statistical Inference

Introduction and Context

Probability and statistical inference provide the mathematical language for uncertainty, variability, and evidence. In electronics and communication engineering, uncertainty arises from thermal noise, fading, component tolerances, packet errors, measurement error, manufacturing variation, and incomplete knowledge of a system. Probability models these uncertainties; statistics uses observed data to estimate unknown quantities, test hypotheses, and support engineering decisions.

The distinction is fundamental. Probability begins with a model and predicts possible observations. Statistical inference begins with observations and draws conclusions about an underlying population or stochastic mechanism. Modern communication theory, signal detection, coding, reliability engineering, quality control, and machine learning all depend on this relationship.

Learning Objectives

After completing this chapter, the learner should be able to:

1. Define sample spaces, events, probability measures, and conditional probability.
2. Apply Bayes’ theorem and the laws of total probability.
3. Distinguish discrete, continuous, and mixed random variables.
4. Calculate expectation, variance, covariance, and correlation.
5. Use important probability distributions in engineering models.
6. Explain laws of large numbers and the central limit theorem.
7. Construct estimators and evaluate their properties.
8. Formulate confidence intervals and hypothesis tests.
9. Interpret regression, likelihood, and statistical evidence critically.
10. Recognize assumptions, limitations, and ethical issues in statistical analysis.

1. Probability Foundations

1.1 Experiments, sample spaces, and events

A random experiment has a well-defined set of possible outcomes, but the particular outcome cannot be predicted with certainty. The sample space, denoted by Ω, is the set of all outcomes. An event A is a subset of Ω. For example, in transmitting one binary symbol, Ω = {0, 1}; the event A = {1} represents reception of symbol 1.

A probability measure P assigns a number to each event and satisfies:

For mutually exclusive events A₁, A₂, …,

The complement of A is Aᶜ, and

For two events,

The subtraction avoids counting the intersection twice.

1.2 Conditional probability and independence

If P(B) > 0, the conditional probability of A given B is

Events A and B are independent when knowledge of one does not change the probability of the other:

Independence is stronger than mutual exclusivity. Two mutually exclusive events with positive probabilities cannot be independent.

The multiplication rule is

If B₁, …, Bₙ form a mutually exclusive and exhaustive partition of Ω, then the law of total probability is

Bayes’ theorem reverses conditioning:

Here P(Bⱼ) is a prior probability, P(A | Bⱼ) is a likelihood, and P(Bⱼ | A) is a posterior probability.

Worked Example 1: Diagnostic decision

Suppose 2% of manufactured sensors contain a defect. A test detects a defective sensor with probability 0.95 and incorrectly flags a good sensor with probability 0.04. Let D denote defect and + denote a positive test.

The numerator is 0.019 and the denominator is 0.0582. Therefore,

Thus only about 32.65% of positive tests indicate an actual defect. The result illustrates the base-rate effect: a highly sensitive test can still have a modest positive predictive value when the defect is rare.

2. Random Variables and Probability Models

A random variable X is a numerical function defined on the outcomes of a random experiment. It may be discrete, continuous, or mixed.

For every random variable, the cumulative distribution function (CDF) is

The CDF is nondecreasing, right-continuous, and approaches 0 as x approaches −∞ and 1 as x approaches +∞.

2.1 Discrete random variables

A discrete random variable has a probability mass function (PMF)

with pₓ(x) ≥ 0 and ∑ₓ pₓ(x) = 1. For any set S,

2.2 Continuous random variables

A continuous random variable is described by a probability density function (PDF) fₓ(x), where

The density itself is not a probability; probabilities are areas under the density. The conditions are fₓ(x) ≥ 0 and ∫₋∞∞ fₓ(x) dx = 1. For a continuous variable, P(X = x) = 0.

2.3 Expectation and variance

The expectation or mean represents a probability-weighted average. For a discrete variable,

For a continuous variable,

For a function g(X),

Variance measures dispersion about the mean μ = E[X]:

The standard deviation is σ = √Var(X), expressed in the same units as X. For constants a and b,

2.4 Joint distributions and covariance

For random variables X and Y, the covariance is

The correlation coefficient is

provided σₓ and σᵧ are nonzero. It satisfies −1 ≤ ρₓᵧ ≤ 1 and is dimensionless. Zero covariance implies uncorrelatedness; it does not generally imply independence. For jointly Gaussian variables, however, zero covariance does imply independence.

For a random vector X, the covariance matrix is

It is symmetric and positive semidefinite. Diagonal entries are variances, and off-diagonal entries are covariances.

3. Important Probability Distributions

The binomial PMF is

where n is the number of independent trials, p is the common success probability, and C(n,k) = n!/[k!(n−k)!].

The Poisson PMF is

where λ is the expected number of events in the specified interval. The Poisson model assumes events occur independently at a constant average rate.

The normal PDF is

where μ is the mean and σ > 0 is the standard deviation. Standardization gives Z = (X−μ)/σ, which has the standard normal distribution.

The exponential PDF is fₓ(x) = λe⁻λˣ for x ≥ 0. It has the memoryless property P(X > s+t | X > s) = P(X > t), which is useful but often unrealistic for ageing components.

Worked Example 2: Gaussian noise threshold

Let receiver noise N be normally distributed with mean 0 V and standard deviation 2 mV. The probability that |N| exceeds 4 mV is

Using Φ(2) ≈ 0.97725,

Approximately 4.55% of noise samples exceed the threshold in magnitude.

4. Limit Theorems and Sampling Distributions

The law of large numbers states, under suitable independence and finite-mean conditions, that the sample mean converges to the population mean as sample size increases:

The central limit theorem (CLT) states that if X₁,…,Xₙ are independent and identically distributed with mean μ and finite variance σ², then approximately for sufficiently large n,

Consequently, X̄ₙ is approximately normal with mean μ and variance σ²/n, even when the individual observations are not normal. The approximation may be poor for heavy-tailed distributions, strong dependence, or small samples.

If the population is normal and σ is unknown, the standardized statistic uses Student’s t distribution:

with n−1 degrees of freedom, where S is the sample standard deviation.

5. Statistical Inference

A statistical model specifies a family of distributions indexed by an unknown parameter θ. A sample X₁,…,Xₙ provides evidence about θ. Inference includes estimation, confidence intervals, hypothesis testing, and prediction.

5.1 Point estimation

An estimator θ̂ is a function of the data. Important criteria include:

• Unbiasedness: E[θ̂] = θ.
• Bias: Bias(θ̂) = E[θ̂] − θ.
• Variance: Var(θ̂), describing repeated-sample variability.
• Mean squared error:

• Consistency: θ̂ converges in probability to θ as n increases.
• Efficiency: an estimator has comparatively small variance among estimators satisfying the same conditions.

The sample mean X̄ is unbiased for μ. The unbiased sample variance is

The denominator n−1, rather than n, corrects the degrees-of-freedom loss caused by estimating μ with X̄.

5.2 Maximum likelihood estimation

Given observations x₁,…,xₙ and parameter θ, the likelihood is

The maximum likelihood estimator (MLE) is the value θ̂ that maximizes L(θ). It is usually easier to maximize the log-likelihood:

For independent normal observations with known variance σ², maximizing the likelihood with respect to μ gives μ̂ = X̄. MLEs often have desirable large-sample properties, including consistency and approximate normality, but may be biased in finite samples.

5.3 Bayesian inference

Bayesian analysis combines a prior distribution π(θ) with a likelihood:

The posterior distribution π(θ | x) represents updated uncertainty. Bayesian methods are valuable when prior engineering knowledge exists, data are limited, or sequential updating is required. Results depend on the prior, which should be justified and subjected to sensitivity analysis.

6. Confidence Intervals and Hypothesis Tests

A 100(1−α)% confidence interval is a procedure that covers the true parameter in 100(1−α)% of repeated samples under the model assumptions. It is not technically correct to say that a fixed interval has a 95% probability of containing a fixed parameter in frequentist inference.

For a normal population with known σ, a confidence interval for μ is

where z₁₋α/₂ is the standard-normal quantile. If σ is unknown and the population is normal, use

For a population proportion p, a large-sample interval may use

provided the normal approximation is adequate. For small samples or proportions near 0 or 1, Wilson or exact methods are generally preferable.

A hypothesis test specifies a null hypothesis H₀ and an alternative H₁. The test statistic is compared with a rejection region or used to calculate a p-value. A p-value is the probability, assuming H₀ is true, of observing a result at least as incompatible with H₀ as the observed result. It is not the probability that H₀ is true.

Two errors are possible:

The power of a test is 1−β. Sample size, effect size, variability, significance level, and test design all influence power. Statistical significance does not necessarily imply engineering significance; an extremely small effect can become significant with a very large sample.

Worked Example 3: Mean response time

A communication device is tested on n = 25 packets. The mean response time is 102 ms and the sample standard deviation is 10 ms. Assuming approximate normality, a 95% confidence interval for the mean uses t₀.975,24 ≈ 2.064:

Thus the interval is approximately (97.87 ms, 106.13 ms). The interpretation concerns the long-run performance of the interval procedure, not a probability assigned to the already fixed mean.

7. Regression and Analysis of Dependence

Simple linear regression models a response Y as

where β₀ is the intercept, β₁ is the mean change in Y per unit increase in x, and εᵢ is random error. Under ordinary least squares, β₀ and β₁ are chosen to minimize

The fitted slope and intercept are

The coefficient of determination is R² = 1 − SSE/SST, where SST = ∑(yᵢ−ȳ)². R² measures the fraction of sample variation explained by the fitted linear model; it does not establish causation or guarantee good prediction outside the observed range.

Regression assumptions should be checked: linearity, independent errors, constant variance, and appropriate treatment of unusual observations. In engineering experiments, randomization, replication, calibration, and blocking can substantially improve inference.

8. Reliability and Communication Applications

If component lifetimes are independent and exponentially distributed with rate λ, system reliability at time t is

For a series system that operates only when every component operates, independent component reliabilities Rᵢ(t) give

For independent parallel components, where the system operates if at least one operates,

In digital communication, a binary symmetric channel with crossover probability p has error probability p for each transmitted bit. If bit errors are independent, the probability of exactly k errors in n bits is binomial. Independence may fail under burst noise, interference, or fading; in such cases, Markov, hidden-state, or burst-error models may be more appropriate.

Detection can be formulated using likelihoods. For hypotheses H₀ and H₁ and observation x, a likelihood-ratio test compares

with a threshold selected according to false-alarm and detection requirements. The threshold should reflect the costs of the two decision errors, not merely a conventional probability cutoff.

9. Illustrative Case: Estimating Packet Error Performance

Background

An engineering team evaluates a wireless receiver under a fixed modulation, bandwidth, and signal-to-noise ratio. The receiver records whether each packet is correct or erroneous.

Problem

In 10,000 transmitted packets, 83 are erroneous. The team must estimate the packet-error rate (PER), quantify uncertainty, and decide whether the receiver meets a requirement PER ≤ 1%.

Analysis

Let Xᵢ = 1 for an erroneous packet and 0 otherwise. Under an independent Bernoulli model, the MLE of p is

The estimated standard error is approximately

A simple 95% Wald interval is p̂ ± 1.96SE, approximately (0.00652, 0.01008). Because the upper endpoint slightly exceeds 1%, the evidence is not sufficient to claim compliance using this interval. A Wilson interval is preferable for a proportion and should be reported in a formal study.

A one-sided test of H₀: p = 0.01 against H₁: p < 0.01 can be conducted using an exact binomial or an appropriate large-sample method. The decision must state the chosen significance level before examining the outcome.

Approach and outcome

The team repeats measurements at several temperatures and signal conditions, verifies packet independence, records confidence intervals, and reports both PER and the number of packets tested. If errors cluster in bursts, the Bernoulli model is inadequate and an error-burst model is required. The initial result suggests good performance but does not establish universal compliance across all operating conditions.

Learning points

1. A point estimate alone is incomplete evidence.
2. Rare-event estimation requires adequate sample size.
3. Model assumptions must be tested rather than silently accepted.
4. Engineering acceptance should consider effect size, risk, operating conditions, and costs.
5. Repeatedly testing until a desired result appears inflates false-positive risk.

10. Limitations, Governance, and Contemporary Practice

Probability models are abstractions. Dependence, nonstationarity, sensor drift, selection bias, missing data, measurement uncertainty, and model misspecification can invalidate apparently precise conclusions. Simulation is useful for sensitivity analysis, but simulation output is only as credible as its inputs and randomization procedure.

Multiple hypothesis testing requires control of false discoveries. Common approaches include family-wise error control, Bonferroni adjustment, and false-discovery-rate procedures. Researchers should distinguish exploratory analysis from confirmatory analysis and document all exclusions, transformations, and stopping rules.

AI and machine-learning systems use probability for classification, prediction, uncertainty quantification, and decision-making. Calibration is essential: among cases assigned probability 0.8, approximately 80% should occur in the predicted class over comparable data. Accuracy alone is insufficient when class imbalance, asymmetric costs, or safety risks exist. Data leakage, biased sampling, privacy violations, and opaque automated decisions are governance concerns. Engineering practice should preserve traceability, reproducibility, human oversight, and appropriate validation under distribution shift.

Best practice includes defining the estimand before collecting data, calibrating instruments, randomizing where possible, using independent validation data, reporting uncertainty, checking residuals and assumptions, preserving raw data and analysis code, and communicating limitations clearly.

Chapter Summary

Probability formalizes uncertainty through sample spaces, events, conditional probability, independence, and Bayes’ theorem. Random variables are described by PMFs, PDFs, and CDFs; their means, variances, covariances, and correlations summarize stochastic behavior. Bernoulli, binomial, Poisson, exponential, uniform, and normal distributions provide useful engineering models, but each depends on assumptions.

The law of large numbers and central limit theorem explain why sample averages become stable and approximately normal. Statistical inference uses data to estimate parameters, construct confidence intervals, test hypotheses, and model relationships through regression and likelihood. Estimator quality involves bias, variance, consistency, efficiency, and mean squared error. Engineering decisions must distinguish statistical significance from practical significance and must address dependence, rare events, multiple testing, uncertainty, and ethical governance.

Key Terms

Probability: Numerical measure of uncertainty for an event.

Sample space: Set of all possible outcomes.

Event: Subset of a sample space.

Conditional probability: Probability of an event given another event.

Independence: Property whereby joint probability equals the product of marginal probabilities.

Bayes’ theorem: Rule for updating probabilities using observed evidence.

Random variable: Numerical function of a random outcome.

PMF: Function giving probabilities of values of a discrete variable.

PDF: Density whose integral over an interval gives a continuous probability.

CDF: Function Fₓ(x) = P(X ≤ x).

Expectation: Probability-weighted mean of a random variable.

Variance: Mean squared deviation from the mean.

Covariance: Measure of joint linear variation between variables.

Estimator: Data-based rule for estimating an unknown parameter.

Bias: Difference between an estimator’s expected value and the parameter.

Likelihood: Probability model viewed as a function of its parameter.

Confidence interval: Repeated-sampling procedure for parameter coverage.

p-value: Probability, under H₀, of evidence at least as extreme as observed.

Power: Probability of rejecting a false null hypothesis.

Model calibration: Agreement between predicted probabilities and observed frequencies.

Review Questions

Short Answer Questions

1. Distinguish probability from statistical inference.
2. State the multiplication rule for two events.
3. What is the difference between a PMF and a PDF?
4. Define covariance and correlation.
5. What does a p-value represent?

Long Answer Questions

1. Explain Bayes’ theorem, including priors, likelihoods, evidence, and posterior probability.
2. Derive the relationship Var(X) = E[X²] − [E(X)]².
3. Compare the binomial, Poisson, exponential, and normal distributions and state their engineering applications.
4. Explain the central limit theorem and its practical limitations.
5. Discuss bias, variance, consistency, efficiency, and mean squared error of estimators.

Analytical / Application Questions

1. A communication channel has independent bit-error probability 0.002. For a 1000-bit frame, calculate the probability of exactly two errors and discuss whether a binomial model is justified.
2. A sensor’s measured output has sample mean 5.2 V, sample standard deviation 0.4 V, and n = 16. Construct and interpret a 95% confidence interval for the mean assuming normality.
3. A receiver produces clustered rather than independent packet errors. Explain why a Bernoulli model may fail and propose an improved statistical investigation.

Multiple Choice Questions with Answers

Question 1

If P(A)=0.4, P(B)=0.5, and A and B are independent, what is P(A ∩ B)?

A. 0.10

B. 0.20

C. 0.45

D. 0.90

Correct Answer: B. 0.20

Question 2

In the diagnostic example, approximately what is P(D | +)?

A. 0.0200

B. 0.0400

C. 0.3265

D. 0.9500

Correct Answer: C. 0.3265

Question 3

For a continuous random variable X, which statement is correct?

A. P(X=x)=1 for every x

B. P(X=x)=0 for every individual x

C. Its PDF values must sum to one

D. Its CDF can decrease

Correct Answer: B. P(X=x)=0 for every individual x

Question 4

If X has mean μ and variance σ², what is Var(3X+5)?

A. σ²+5

B. 3σ²

C. 9σ²

D. 9σ²+25

Correct Answer: C. 9σ²

Question 5

Which distribution is commonly used for the number of rare independent events in a fixed interval?

A. Poisson

B. Uniform

C. Normal

D. Exponential

Correct Answer: A. Poisson

Question 6

For independent normal observations with unknown variance, which distribution is used for the standardized sample mean?

A. Poisson

B. Student’s t

C. Binomial

D. Uniform

Correct Answer: B. Student’s t

Question 7

What does a 95% frequentist confidence interval mean?

A. There is a 95% probability the fixed parameter lies in this interval

B. 95% of individual observations lie in this interval

C. The procedure captures the true parameter in 95% of repeated samples

D. The null hypothesis is true with probability 0.95

Correct Answer: C. The procedure captures the true parameter in 95% of repeated samples

Question 8

The MLE of the Bernoulli success probability based on observed data is:

A. The sample mean of the 0–1 observations

B. Always 0.5

C. The sample variance

D. The population median

Correct Answer: A. The sample mean of the 0–1 observations

Question 9

If a test has significance level α, α is the probability of:

A. A Type I error

B. A Type II error

C. Correctly accepting a false null hypothesis

D. Correctly rejecting a true null hypothesis

Correct Answer: A. A Type I error

Question 10

Which statement about correlation is correct?

A. Zero correlation always implies independence

B. Correlation has physical units of volts

C. Correlation lies between −1 and 1

D. Correlation measures only nonlinear dependence

Correct Answer: C. Correlation lies between −1 and 1