2. Foundations of Probability

Introduction and Context

Data science is concerned with learning from incomplete, variable, and uncertain information. A predictive model may estimate whether a customer will default, whether a transaction is fraudulent, or whether a machine will fail, but such predictions are rarely certain. Probability provides the formal language for representing uncertainty, combining evidence, and making rational decisions under incomplete knowledge.

Probability is not merely a collection of formulas. It is a mathematical framework built from sets, logical operations, counting principles, and rules for updating beliefs when new information becomes available. These foundations support statistical inference, machine learning, simulation, risk analysis, experimental design, and Bayesian modelling. This chapter develops those foundations and illustrates how they operate in practical data-science settings.

Learning Objectives

By the end of this chapter, students should be able to:

1. Describe random experiments, sample spaces, events, and probability measures.

2. Apply the axioms of probability and the principal laws derived from them.

3. Use set operations and Venn-diagram reasoning to analyse events.

4. Solve counting problems using permutations, combinations, and the multiplication principle.

5. Compute conditional probabilities and distinguish dependence from independence.

6. Apply the law of total probability and Bayes’ theorem.

7. Interpret probabilistic results in classification, risk, quality-control, and decision-making contexts.

8. Identify common errors, modelling assumptions, and ethical issues in probabilistic analysis.

1. The Probabilistic View of Data

A random experiment is a process with a set of possible outcomes, although the particular outcome is not known in advance. Examples include selecting a record from a database, observing whether a payment is fraudulent, or measuring the lifetime of a component. The set of all possible outcomes is the sample space, denoted by \(S\) or \(\Omega\).

An event is a subset of the sample space. If a die is rolled, \(S=\{1,2,3,4,5,6\}\), and the event \(A=\{2,4,6\}\) represents obtaining an even number. Events can be combined using set operations. The union \(A\cup B\) means that at least one of the events occurs; the intersection \(A\cap B\) means that both occur; and the complement \(A^c\) means that \(A\) does not occur. Two events are mutually exclusive if \(A\cap B=\varnothing\).

The sample space must be defined carefully. For a fraud-detection problem, the outcomes might be simply {fraud, legitimate}, or they might include transaction amount, location, device type, and time. The appropriate sample space depends on the question being asked. Poor definition of the outcome space can produce logically incorrect probabilities even when the calculations appear correct.

2. Axioms and Laws of Probability

A probability function assigns a number \(P(A)\) to every event \(A\), satisfying three Kolmogorov axioms:

1. Non-negativity: \(P(A)\geq 0\).

2. Normalization: \(P(S)=1\).

3. Countable additivity: For mutually exclusive events \(A_1,A_2,\ldots\),

\[P\left(\bigcup_i A_i\right)=\sum_iP(A_i).\]

These axioms imply several useful results. For any event \(A\),

\[P(A^c)=1-P(A).\]

If \(A\) and \(B\) are mutually exclusive, then \(P(A\cup B)=P(A)+P(B)\). For arbitrary events,

\[P(A\cup B)=P(A)+P(B)-P(A\cap B).\]

The subtraction is necessary because the intersection is counted twice when the two probabilities are added.

If \(A\subseteq B\), then \(P(A)\leq P(B)\). The probability of an event always lies between zero and one. In a finite sample space with equally likely outcomes,

\[P(A)=\frac{|A|}{|S|},\]

where \(|A|\) denotes the number of outcomes in \(A\). Equal likelihood, however, should not be assumed automatically. In real data, outcomes may have different frequencies or unequal exposure.

Set reasoning in an example

Suppose 60% of users open an email, 25% click a link, and 15% do both. The probability that a user opens or clicks is

\[

P(O\cup C)=0.60+0.25-0.15=0.70.

\]

The probability of neither action is \(1-0.70=0.30\). The result demonstrates why union probabilities cannot generally be obtained by simple addition.

3. Counting Uncertainty: Combinatorics

Counting methods are useful when probabilities are based on equally likely arrangements. The multiplication principle states that if one task can be completed in \(m\) ways and a second in \(n\) ways, the combined task can be completed in \(mn\) ways.

A permutation counts ordered arrangements. The number of ways to arrange \(r\) objects selected from \(n\) distinct objects is

\[

P(n,r)=\frac{n!}{(n-r)!}.

\]

A combination counts selections where order does not matter:

\[

\binom{n}{r}=\frac{n!}{r!(n-r)!}.

\]

For example, selecting three analysts from ten for a committee gives \(\binom{10}{3}=120\) possible committees. Assigning those three analysts to the roles of chair, secretary, and reviewer gives \(P(10,3)=720\) possible ordered assignments.

Combinatorial reasoning is often used in sampling, feature selection, security analysis, and randomised algorithms. It must be applied with care when sampling is with replacement, when objects are indistinguishable, or when the outcomes are not equally likely.

4. Conditional Probability and the Updating of Uncertainty

The probability of an event may change when additional information is known. The conditional probability of \(A\) given \(B\), where \(P(B)>0\), is

\[

P(A\mid B)=\frac{P(A\cap B)}{P(B)}.

\]

This formula restricts attention to the cases in which \(B\) occurs and measures the proportion of those cases that also belong to \(A\). Rearranging gives the multiplication rule:

\[

P(A\cap B)=P(A\mid B)P(B)=P(B\mid A)P(A).

\]

Consider a population in which 4% of transactions are fraudulent. A screening system flags 90% of fraudulent transactions and incorrectly flags 5% of legitimate transactions. The probability of a flag is

\[

P(F)=P(F\mid D)P(D)+P(F\mid D^c)P(D^c)

\]

\[=(0.90)(0.04)+(0.05)(0.96)=0.084.

\]

Although the detector is effective, the probability that a flagged transaction is actually fraudulent is

\[

P(D\mid F)=\frac{(0.90)(0.04)}{0.084}\approx0.429.

\]

Thus, only about 42.9% of flagged transactions are fraudulent. This result illustrates the base-rate effect: the prevalence of an event substantially influences the meaning of a positive signal.

5. Independence and Dependence

Events \(A\) and \(B\) are independent if learning that one occurred does not change the probability of the other. Formally,

\[

P(A\cap B)=P(A)P(B).

\]

Equivalently, if the relevant probabilities exist, \(P(A\mid B)=P(A)\) and \(P(B\mid A)=P(B)\).

Independence is a mathematical assumption, not a synonym for “unrelated” in ordinary language. Two events can be associated through a common cause even when they appear separately recorded. For example, ice-cream sales and swimming incidents may both increase during hot weather; they are not necessarily causally related to each other, but they are statistically dependent through temperature.

Independence should also be distinguished from mutual exclusivity. If two events with positive probability are mutually exclusive, they cannot be independent, because their intersection has probability zero while the product of their probabilities is positive. In data science, assuming independence can simplify computation, as in the naive Bayes classifier, but the assumption may be inaccurate. The model can still perform well in some classification tasks, yet its probability estimates may require calibration.

6. Total Probability and Bayes’ Theorem

Suppose \(B_1,B_2,\ldots,B_k\) form a partition of the sample space: they are mutually exclusive and their union is the entire space. The law of total probability states that

\[

P(A)=\sum_{i=1}^{k}P(A\mid B_i)P(B_i).

\]

It decomposes an overall probability into contributions from distinct groups or causes.

Bayes’ theorem follows from the multiplication rule:

\[

P(B_j\mid A)=\frac{P(A\mid B_j)P(B_j)}{\sum_{i=1}^{k}P(A\mid B_i)P(B_i)}.

\]

In applied work, \(P(B_j)\) is the prior probability, \(P(A\mid B_j)\) is the likelihood of the evidence, and \(P(B_j\mid A)\) is the posterior probability after observing the evidence.

Bayesian reasoning is central to diagnosis, recommendation systems, anomaly detection, document classification, and adaptive experimentation. Its value lies not in replacing evidence with belief, but in providing a transparent rule for combining prior information with observed data. The prior must be justified, the likelihood model must be appropriate, and the sensitivity of conclusions to alternative assumptions should be examined.

7. A Structured Approach to Probability Problems

A reliable solution usually follows these stages:

1. Define the observational unit and sample space.

2. State the events using precise notation or plain language.

3. Determine whether the events are mutually exclusive, exhaustive, dependent, or independent.

4. Select the appropriate rule: complement, union, multiplication, total probability, or Bayes’ theorem.

5. Substitute values and retain sufficient precision.

6. Check that the result lies between zero and one and makes contextual sense.

7. Interpret the result in the language of the application.

Probability trees are particularly useful when events occur in stages. The probabilities along a branch are multiplied, while probabilities of alternative branches are added. Contingency tables provide an equivalent representation for observed categorical data and are useful for calculating conditional probabilities, sensitivity, specificity, and predictive values.

Applied Case: Prioritising Alerts in a Hospital Monitoring System

Background

An illustrative hospital analytics team uses a predictive system to identify patients at risk of a complication. The complication occurs in 2% of monitored patients. The system raises an alert for 95% of patients who develop the complication and for 8% of patients who do not.

Situation and analysis

Let \(C\) denote the complication and \(A\) an alert. Then

\[

P(A)=P(A\mid C)P(C)+P(A\mid C^c)P(C^c)

\]

\[=(0.95)(0.02)+(0.08)(0.98)=0.0974.

\]

The probability that an alerted patient truly develops the complication is

\[

P(C\mid A)=\frac{(0.95)(0.02)}{0.0974}\approx0.195.

\]

Only about 19.5% of alerts correspond to actual complications, despite high sensitivity. The low base rate and false-positive rate create a substantial workload.

Approach and decision

The team should not evaluate the system using sensitivity alone. It should also examine specificity, positive predictive value, alert volume, the cost of missed cases, and the cost of unnecessary interventions. A risk-tiering strategy could combine the alert with clinical history and laboratory measurements. Thresholds should be selected according to the consequences of false negatives and false positives, not solely according to statistical accuracy.

Outcome and learning points

The case shows how total probability estimates the total alert rate, while Bayes’ theorem estimates the probability that an alert is meaningful. It also demonstrates why model performance depends on prevalence. If the system is deployed in a population with a different complication rate, its predictive value will change even if sensitivity and specificity remain constant.

Applications in Data Science and Professional Practice

Probability supports binary and multiclass classification, risk scoring, Monte Carlo simulation, reliability analysis, A/B testing, inventory planning, credit assessment, and fraud detection. In machine learning, predicted probabilities allow organisations to rank cases, choose decision thresholds, quantify uncertainty, and allocate limited resources.

For managers, probability clarifies the difference between a model score and a decision rule. A probability of default of 0.18 does not itself determine whether credit should be denied; the decision also depends on loss severity, regulatory requirements, customer impact, and organisational risk tolerance. For technical teams, probability provides a basis for evaluating calibration, sampling variation, missing data, and distribution shift.

Digital systems increasingly automate probabilistic decisions. Such automation should include monitoring for changes in base rates, documentation of assumptions, reproducible data pipelines, and human review for high-impact cases. A probability estimate should not be presented as certainty, especially when data are sparse or the model has not been validated in the target population.

Common Errors, Limitations, and Ethical Considerations

A frequent error is confusing \(P(A\mid B)\) with \(P(B\mid A)\). Another is ignoring the base rate, adding overlapping probabilities without subtracting their intersection, or treating a sample proportion as an exact population probability. Independence should not be assumed merely because variables are recorded separately. Selection bias can also distort probabilities when the observed sample does not represent the population to which conclusions are applied.

Probability models are abstractions. They depend on a defined population, measurement process, time period, and set of assumptions. A model based on historical data may reproduce discriminatory patterns or perform unevenly across demographic groups. In high-impact applications, analysts should document data provenance, assess subgroup performance, protect privacy, explain uncertainty, and provide mechanisms for review or appeal. Ethical practice requires distinguishing statistical association from causal evidence and avoiding unjustified certainty in communication.

Best Practices

• Define events and populations before calculating probabilities.
• Use partitions and probability trees to make assumptions visible.
• Report both conditional probabilities and their denominators.
• Check base rates whenever interpreting tests, alerts, or classifiers.
• Validate independence assumptions rather than accepting them automatically.
• Compare model probabilities with observed frequencies through calibration analysis.
• Conduct sensitivity analysis for uncertain priors and changing prevalence.
• Align thresholds with consequences, costs, and professional standards.
• Communicate uncertainty clearly to technical and non-technical stakeholders.

Current Developments

Modern probabilistic data science increasingly combines classical probability with probabilistic programming, Bayesian computation, uncertainty quantification, and simulation-based inference. Machine-learning systems are being evaluated not only for predictive accuracy but also for calibration, robustness under distribution shift, and reliability of uncertainty estimates. These developments make foundational probability especially important: sophisticated software can calculate a result quickly, but it cannot determine whether the sample space, assumptions, or interpretation are appropriate.

Chapter Summary

Probability formalises uncertainty through sample spaces, events, and probability measures. The axioms lead to complement, union, and inclusion–exclusion rules. Counting methods provide probabilities for equally likely finite outcomes, while conditional probability describes how information changes uncertainty. The multiplication rule supports sequential analysis; independence simplifies joint probabilities but must be justified. The law of total probability combines probabilities across groups, and Bayes’ theorem reverses conditional relationships to update prior beliefs using evidence. In data science, these tools underpin classification, diagnosis, risk scoring, alert management, simulation, and evidence-based decisions. Correct interpretation requires attention to base rates, assumptions, data quality, calibration, fairness, and the consequences of error.

Key Terms

Sample space: The set of all possible outcomes of a random experiment.

Outcome: A single possible result of a random experiment.

Event: A subset of the sample space representing a condition of interest.

Probability: A number from 0 to 1 quantifying the likelihood of an event.

Complement: The event consisting of all outcomes not in a specified event.

Union: The event that at least one of two or more events occurs.

Intersection: The event that two or more events occur together.

Mutually exclusive events: Events that cannot occur simultaneously.

Conditional probability: The probability of an event given that another event has occurred.

Independence: A relationship in which occurrence of one event does not alter the probability of another.

Permutation: An ordered arrangement of selected objects.

Combination: An unordered selection of objects.

Partition: A collection of mutually exclusive events that exhausts the sample space.

Law of total probability: A rule expressing an event probability as a weighted sum across a partition.

Bayes’ theorem: A rule for computing a conditional probability in the reverse direction.

Prior probability: The probability assigned before considering new evidence.

Likelihood: The probability of observed evidence under a specified condition or hypothesis.

Posterior probability: The updated probability after incorporating evidence.

Base rate: The prevalence or prior frequency of an event in a population.

Positive predictive value: The probability that a positive result corresponds to the condition of interest.

Review Questions

Short Answer Questions

1. What is the difference between a sample space and an event?

2. State the three axioms of probability.

3. When are two events mutually exclusive?

4. Define conditional probability and state its formula.

5. What does it mean for two events to be independent?

Long Answer Questions

1. Explain the union rule for probability and illustrate why the intersection must be subtracted.

2. Compare permutations and combinations, including an example of each.

3. Derive the multiplication rule from the definition of conditional probability.

4. Explain the law of total probability and Bayes’ theorem using a diagnostic or classification example.

5. Discuss the role of base rates, calibration, and uncertainty in automated decision systems.

Analytical / Application Questions

1. A disease affects 3% of a population. A test has sensitivity 92% and a false-positive rate of 6%. Calculate the probability of a positive test and the probability that a person with a positive test has the disease.

2. In a dataset, 45% of customers use a mobile application, 30% purchase a subscription, and 18% do both. Find the probability that a randomly selected customer uses the application or purchases a subscription, and the probability of neither.

3. A company receives applications from three regions representing 50%, 30%, and 20% of applicants. Their acceptance rates are 4%, 7%, and 10%, respectively. Calculate the overall acceptance probability and the probability that an accepted applicant came from the second region.

22. Multiple Choice Questions with Answers

Question 1

A user opens an email with probability 0.60, clicks its link with probability 0.25, and does both with probability 0.15. What is the probability that the user opens or clicks the email?

A. 0.40

B. 0.70

C. 0.85

D. 1.00

Correct Answer: B. 0.70

Question 2

Which condition defines independence of events A and B?

A. P(A∩B)=0

B. P(A∪B)=P(A)+P(B)

C. P(A∩B)=P(A)P(B)

D. P(A|B)=P(B|A)

Correct Answer: C. P(A∩B)=P(A)P(B)

Question 3

A rare condition has prevalence 2%. A test detects 95% of affected cases and has an 8% false-positive rate. Approximately what is the probability that a positive result indicates the condition?

A. 19.5%

B. 50.0%

C. 92.0%

D. 95.0%

Correct Answer: A. 19.5%

Question 4

Which expression correctly represents the law of total probability for a partition B1,...,Bk?

A. P(A)=ΣP(Bi|A)

B. P(A)=ΣP(A|Bi)P(Bi)

C. P(A)=P(A|B1)P(A|B2)

D. P(A)=P(Bi|A)P(A)

Correct Answer: B. P(A)=ΣP(A|Bi)P(Bi)

Question 5

If two events with positive probabilities are mutually exclusive, which statement is correct?

A. They must be independent

B. Their intersection has probability one

C. They cannot be independent

D. Their union has probability zero

Correct Answer: C. They cannot be independent

Question 6

How many unordered committees of three can be selected from ten distinct analysts?

A. 30

B. 120

C. 720

D. 1,000

Correct Answer: B. 120

Question 7

Which probability is the posterior probability in Bayes’ theorem?

A. P(B), assigned before evidence

B. P(A|B), evidence under a condition

C. P(B|A), condition after observing evidence A

D. P(A), the marginal evidence probability

Correct Answer: C. P(B|A), condition after observing evidence A

Question 8

A probability model reports P(default)=0.18 for a customer. What is the most accurate interpretation?

A. The customer will definitely default

B. The model estimates an 18% probability of default under its assumptions

C. Exactly 18% of all customers will default

D. The customer has an 18% chance of being approved

Correct Answer: B. The model estimates an 18% probability of default under its assumptions

Question 9

If P(A)=0.4 and P(B|A)=0.5, what is P(A∩B)?

A. 0.08

B. 0.20

C. 0.45

D. 0.90

Correct Answer: B. 0.20

Question 10

Why can a highly sensitive fraud detector still produce many false alarms?

A. Sensitivity always equals specificity

B. The sample space has no complement

C. A low fraud base rate and false-positive rate affect the predictive value

D. Bayes’ theorem applies only to medical data

Correct Answer: C. A low fraud base rate and false-positive rate affect the predictive value