Foundations of Probability

Introduction and Context

Probability is the mathematical language used to represent uncertainty. In artificial intelligence and machine learning, uncertainty arises because data may be incomplete, observations may be noisy, and future outcomes cannot be known with certainty. A medical classifier may assign a probability that a patient has a disease; a robotics system may estimate the probability that an obstacle is present; and a recommendation system may estimate the probability that a user will click an item.

Probability provides a rigorous framework for describing possible outcomes, combining evidence, updating beliefs, and making decisions. This chapter develops the foundations required for later study of random variables, probability distributions, statistical inference, Bayesian learning, and probabilistic machine learning.

Learning Objectives

By the end of this chapter, you should be able to:

1. Define sample spaces, outcomes, events, and probability models.

2. Apply the axioms of probability and basic set operations.

3. Use counting principles to calculate probabilities in finite experiments.

4. Distinguish conditional probability from joint and marginal probability.

5. Determine whether events are independent.

6. Apply the law of total probability and Bayes’ theorem.

7. Interpret odds, likelihood, and posterior probability correctly.

8. Model uncertainty in practical AI and ML applications.

9. Identify limitations, assumptions, and ethical risks in probabilistic reasoning.

1. Experiments, Outcomes, and Sample Spaces

A random experiment is a process whose exact outcome cannot be predicted with certainty, even when the procedure is repeated under apparently similar conditions. Examples include tossing a coin, selecting a record from a database, measuring sensor noise, or observing whether a machine-learning prediction is correct.

An outcome is one possible result of an experiment. The sample space, denoted by \(\Omega\), is the set of all possible outcomes. An event is a subset of the sample space.

For a single toss of a fair coin,

\[

\Omega=\{H,T\}.

\]

If two coins are tossed and order is recorded,

\[

\Omega=\{HH,HT,TH,TT\}.

\]

The event “exactly one head” is

\[

A=\{HT,TH\}.

\]

The sample space must be defined carefully. For example, when two coins are tossed, \(HT\) and \(TH\) are distinct outcomes if the first and second tosses are distinguishable. Failing to define the sample space can produce incorrect probabilities.

A sample space may be:

• Finite, such as the faces of a die.
• Countably infinite, such as the number of attempts until a success.
• Continuous, such as the exact temperature or time of arrival.

For continuous sample spaces, individual outcomes generally have probability zero, while intervals or regions may have positive probability. Continuous probability is developed more fully through probability density functions.

2. Events and Set Operations

Because events are sets, set theory provides the basic language for probability.

For events \(A\) and \(B\):

• The union, \(A\cup B\), means that \(A\) occurs, or \(B\) occurs, or both occur.
• The intersection, \(A\cap B\), means that both events occur.
• The complement, \(A^c\), means that \(A\) does not occur.
• The empty event, \(\varnothing\), is impossible.
• The certain event is \(\Omega\).

Two events are mutually exclusive or disjoint if

\[

A\cap B=\varnothing.

\]

The event “a die shows an even number” is \(A=\{2,4,6\}\), and the event “a die shows a number less than 4” is \(B=\{1,2,3\}\). Their intersection is \(\{2\}\), so they are not mutually exclusive.

The complement rule is

\[

P(A^c)=1-P(A).

\]

For any two events, the addition rule is

\[

P(A\cup B)=P(A)+P(B)-P(A\cap B).

\]

The subtraction term prevents the shared outcomes from being counted twice. If \(A\) and \(B\) are mutually exclusive, then \(P(A\cap B)=0\), and therefore

\[

P(A\cup B)=P(A)+P(B).

\]

For three events, inclusion–exclusion gives

\[

P(A\cup B\cup C)=P(A)+P(B)+P(C)

-P(A\cap B)-P(A\cap C)-P(B\cap C)

+P(A\cap B\cap C).

\]

3. Axioms and Properties of Probability

A probability measure \(P\) assigns a number to events and must satisfy three Kolmogorov axioms:

1. Non-negativity:

\[

P(A)\geq 0.

\]

2. Normalization:

\[

P(\Omega)=1.

\]

3. Countable additivity: If \(A_1,A_2,\ldots\) are mutually exclusive, then

\[

P\left(\bigcup_i A_i\right)=\sum_iP(A_i).

\]

Several useful results follow from these axioms:

\[

0\leq P(A)\leq 1,

\]

\[

P(\varnothing)=0,

\]

and

\[

P(A^c)=1-P(A).

\]

If \(A\subseteq B\), then

\[

P(A)\leq P(B).

\]

The axioms do not require all outcomes to be equally likely. A fair die has equal probabilities for its six faces, but a biased die does not. Equally likely outcomes are a modeling assumption, not a consequence of the probability axioms.

4. Finite Probability Models and Counting

When all outcomes in a finite sample space are equally likely,

\[

P(A)=\frac{|A|}{|\Omega|},

\]

where \(|A|\) is the number of outcomes in event \(A\), and \(|\Omega|\) is the total number of outcomes.

Multiplication Principle

If a process consists of \(k\) stages, with \(n_1,n_2,\ldots,n_k\) choices at each stage, the total number of possible results is

\[

n_1n_2\cdots n_k.

\]

For example, a password with three letters followed by two digits has \(26^3\times10^2\) possible strings if repetition is allowed.

Permutations

The number of ordered arrangements of \(r\) objects selected from \(n\) distinct objects is

\[

{}_nP_r=\frac{n!}{(n-r)!}.

\]

Combinations

The number of unordered selections of \(r\) objects from \(n\) distinct objects is

\[

{}_nC_r=\binom nr=\frac{n!}{r!(n-r)!}.

\]

The distinction is whether order matters. Selecting a president and secretary is an ordered assignment; selecting a two-person committee is not.

Worked Example: Card Selection

A standard deck contains 52 cards, including 13 hearts. If one card is selected uniformly at random, the probability of selecting a heart is

\[

P(\text{heart})=\frac{13}{52}=\frac14=0.25.

\]

The probability of selecting a heart or a king is not \(13/52+4/52\) without adjustment, because the king of hearts belongs to both groups. Thus,

\[

P(\text{heart or king})=\frac{13+4-1}{52}=\frac{16}{52}=\frac4{13}.

\]

5. Conditional Probability

Conditional probability measures the probability of one event given that another event is known to have occurred. For \(P(B)>0\),

\[

P(A\mid B)=\frac{P(A\cap B)}{P(B)}.

\]

Here, \(P(A\mid B)\) is read as “the probability of \(A\) given \(B\).” The condition restricts attention to the outcomes in \(B\).

Rearranging gives the multiplication rule:

\[

P(A\cap B)=P(A\mid B)P(B).

\]

Equivalently,

\[

P(A\cap B)=P(B\mid A)P(A).

\]

These expressions are equal because the intersection is the same event regardless of the order in which it is described.

Worked Example: Quality Inspection

Suppose 2% of manufactured components are defective. A screening test detects a defective component with probability 0.95. Let \(D\) denote defectiveness and \(S\) denote a positive screening result. Then

\[

P(D)=0.02,

\qquad P(S\mid D)=0.95.

\]

The joint probability of a defective component and a positive result is

\[

P(D\cap S)=P(S\mid D)P(D)=0.95(0.02)=0.019.

\]

This is not the same as \(P(D\mid S)\), which requires consideration of nondefective components that may also test positive.

6. Independence

Events \(A\) and \(B\) are independent if the occurrence of one does not change the probability of the other. Formally, provided the relevant probabilities are defined,

\[

P(A\cap B)=P(A)P(B).

\]

Equivalent forms include

\[

P(A\mid B)=P(A)

\]

when \(P(B)>0\), and

\[

P(B\mid A)=P(B)

\]

when \(P(A)>0\).

Independence is different from mutual exclusivity. If two events are mutually exclusive and both have positive probability, they cannot be independent, because their intersection has probability zero while the product of their probabilities is positive.

For example, when a coin is tossed twice, the event that the first toss is heads and the event that the second toss is heads are independent. However, the event “first toss is heads” and “exactly one head occurs” are not independent.

For several events, pairwise independence does not always imply mutual independence. Mutual independence requires the product rule to hold for every finite combination of the events.

7. Law of Total Probability

Suppose \(B_1,B_2,\ldots,B_k\) form a partition of \(\Omega\). This means that the events are mutually exclusive and exhaustive:

\[

B_i\cap B_j=\varnothing \quad(i\ne j),

\qquad \bigcup_{i=1}^{k}B_i=\Omega.

\]

For any event \(A\), the law of total probability states that

\[

P(A)=\sum_{i=1}^{k}P(A\mid B_i)P(B_i),

\]

assuming the conditional probabilities are defined.

This law is useful when an event can arise through several alternative causes.

Example: Multiple Suppliers

Suppose Supplier 1 provides 60% of components and has a defect rate of 1%. Supplier 2 provides 40% and has a defect rate of 3%. If \(D\) denotes a defect, then

\[

P(D)=P(D\mid S_1)P(S_1)+P(D\mid S_2)P(S_2)

\]

\[

=(0.01)(0.60)+(0.03)(0.40)=0.006+0.012=0.018.

\]

Thus, the overall defect probability is 1.8%, not the simple average of 2%, because the suppliers contribute different proportions.

8. Bayes’ Theorem

Bayes’ theorem follows from the multiplication rule. For \(P(B)>0\),

\[

P(A\mid B)=\frac{P(B\mid A)P(A)}{P(B)}.

\]

If \(A_1,\ldots,A_k\) form a partition, then

\[

P(A_j\mid B)=

\frac{P(B\mid A_j)P(A_j)}

{\sum_{i=1}^{k}P(B\mid A_i)P(A_i)}.

\]

The components have useful interpretations:

• \(P(A)\): prior probability.
• \(P(B\mid A)\): likelihood of the evidence under the hypothesis.
• \(P(B)\): marginal or evidence probability.
• \(P(A\mid B)\): posterior probability after observing evidence.

Worked Example: Medical Screening

Assume a disease affects 2% of a population. A test has sensitivity 95%, so \(P(+\mid D)=0.95\). Its false-positive rate is 5%, so \(P(+\mid D^c)=0.05\). The probability of a positive test is

\[

P(+)=P(+\mid D)P(D)+P(+\mid D^c)P(D^c)

\]

\[

=(0.95)(0.02)+(0.05)(0.98)=0.019+0.049=0.068.

\]

Therefore,

\[

P(D\mid +)=\frac{(0.95)(0.02)}{0.068}

=\frac{0.019}{0.068}\approx0.2794.

\]

The probability that a person actually has the disease after a positive result is approximately 27.94%, not 95%. The difference results from the low base rate and false positives among the much larger non-diseased population.

9. Odds and Likelihood Ratios

For an event \(A\) with \(0<P(A)<1\), the odds in favor of \(A\) are

\[

\operatorname{Odds}(A)=\frac{P(A)}{1-P(A)}.

\]

If \(P(A)=0.25\), the odds are \(0.25/0.75=1/3\), or one to three.

Bayes’ theorem can be expressed in odds form:

\[

\operatorname{Odds}(A\mid B)

=

\operatorname{Odds}(A)

\times

\frac{P(B\mid A)}{P(B\mid A^c)}.

\]

The ratio

\[

\frac{P(B\mid A)}{P(B\mid A^c)}

\]

is the likelihood ratio for evidence \(B\). A likelihood ratio greater than 1 supports \(A\); a value less than 1 supports \(A^c\); and a value of 1 provides no discrimination.

Odds are useful in diagnostic testing, Bayesian updating, and logistic regression. However, odds are not probabilities and must not be interpreted as percentages without conversion.

10. Probability, Likelihood, and Evidence

Probability and likelihood are related but conceptually distinct. Probability describes uncertainty about an outcome when a model or parameter is fixed. Likelihood evaluates how compatible observed data are with different parameter values or hypotheses.

For observed data \(x\) and parameter \(\theta\), a probability model may be written as

\[

P(X=x\mid\theta).

\]

When \(x\) is fixed and \(\theta\) varies, the same expression is treated as a likelihood function:

\[

L(\theta\mid x)=P(X=x\mid\theta)

\]

for a discrete model, or the corresponding density for continuous data. A likelihood need not sum or integrate to one over parameter values. It is therefore not itself a probability distribution over \(\theta\).

In machine learning, maximum likelihood estimation chooses parameter values that maximize the likelihood of observed training data. Bayesian inference combines the likelihood with a prior to obtain a posterior distribution.

11. A Practical AI and ML Application

Illustrative Case: Email Spam Classification

Background. An email service wants to classify incoming messages as spam or legitimate. The classifier observes features such as the presence of suspicious words, unusual links, sender reputation, and message structure.

Problem. A message contains the word “prize.” The system must estimate the probability that the message is spam given this evidence.

Let \(S\) denote spam and \(W\) denote the presence of the word “prize.” Suppose historical data suggest:

\[

P(S)=0.20,

\qquad P(W\mid S)=0.60,

\qquad P(W\mid S^c)=0.05.

\]

Using total probability,

\[

P(W)=P(W\mid S)P(S)+P(W\mid S^c)P(S^c)

\]

\[

=(0.60)(0.20)+(0.05)(0.80)=0.12+0.04=0.16.

\]

Bayes’ theorem gives

\[

P(S\mid W)=\frac{(0.60)(0.20)}{0.16}=0.75.

\]

Thus, under these assumptions, the message has a 75% probability of being spam.

Analysis. The word “prize” is informative because its likelihood ratio is

\[

\frac{P(W\mid S)}{P(W\mid S^c)}=\frac{0.60}{0.05}=12.

\]

The evidence is twelve times more likely in spam than in legitimate mail. However, a real classifier would use many features. A naive Bayes classifier may assume conditional independence of features given the class. This assumption simplifies computation but may be unrealistic because suspicious words and links can be correlated.

Outcome. The system may place messages with posterior probability above a chosen threshold into a spam folder. The threshold should reflect the relative costs of false positives and false negatives. Deleting a legitimate business message may be more harmful than allowing one spam message into an inbox.

Learning points.

1. Base rates influence posterior probabilities.

2. Evidence must be evaluated relative to both hypotheses.

3. Conditional independence is a modeling assumption, not a fact.

4. Classification thresholds should incorporate consequences, not only accuracy.

5. Probabilities should be calibrated so that predicted values correspond to observed frequencies.

12. Assumptions, Limitations, and Common Errors

Confusing \(P(A\mid B)\) with \(P(B\mid A)\)

These quantities are generally different. A symptom may be common among patients with a disease, while the disease may still be uncommon among people with the symptom.

Ignoring Base Rates

A highly accurate test can produce many false positives when the condition being detected is rare. Base-rate neglect is particularly dangerous in medical, security, and fraud-detection systems.

Assuming Independence Without Justification

Features, observations, or events may be correlated. Independence should be supported by the data-generating process, domain knowledge, or diagnostic analysis.

Treating Model Probabilities as Certainty

A probability is a model-based assessment, not a guarantee. Poor data, distribution shift, sampling bias, and misspecified assumptions can make numerical probabilities misleading.

Equating Association with Causation

A conditional probability or correlation does not establish that one event causes another. Causal conclusions require an appropriate causal design or model.

Sampling and Data Bias

Training data may underrepresent groups or operating conditions. A probability estimate learned from biased data may perform unfairly or inaccurately for populations not represented in the sample.

13. Ethical and Governance Considerations

Probabilistic systems influence decisions about credit, employment, education, healthcare, insurance, and public safety. Important governance practices include:

• documenting data sources and assumptions;
• testing calibration and performance across relevant groups;
• distinguishing uncertainty from confidence in a system’s correctness;
• monitoring changes in population distributions;
• communicating probabilities in understandable terms;
• preserving privacy when probabilities are estimated from personal data;
• providing human review for high-impact decisions;
• auditing false-positive and false-negative consequences.

A mathematically valid probability model can still produce unjust outcomes if its data or deployment context is inappropriate. Technical quality and ethical responsibility must therefore be considered together.

14. Best Practices for Probability Modeling

1. Define the experiment, outcome, and sample space explicitly.

2. State whether outcomes are assumed equally likely.

3. Separate joint, marginal, conditional, and posterior probabilities.

4. Use partitions when alternative causes or groups are involved.

5. Check whether independence is justified before applying product rules.

6. Use complements when they simplify calculations.

7. Verify that every probability lies between 0 and 1.

8. Check that probabilities across an exhaustive partition sum to 1.

9. Report assumptions and distinguish model results from observed facts.

10. In AI applications, evaluate calibration, subgroup performance, and the costs of errors.

Chapter Summary

Probability is a formal framework for representing uncertainty. A sample space contains all possible outcomes, while events are subsets of that space. The axioms of non-negativity, normalization, and countable additivity support rules such as the complement and addition rules. Counting principles provide probabilities for equally likely finite outcomes.

Conditional probability describes uncertainty after information is observed and leads to the multiplication rule. Independence means that one event does not alter the probability of another; it is distinct from mutual exclusivity. The law of total probability combines probabilities across a partition of possible causes. Bayes’ theorem updates prior beliefs using evidence and produces posterior probabilities. Odds and likelihood ratios provide an alternative form of Bayesian updating, while likelihood must be distinguished from probability.

These concepts are central to AI and ML systems, including classification, diagnosis, anomaly detection, and decision support. Reliable application requires attention to base rates, dependence, sampling bias, calibration, uncertainty communication, and ethical consequences.

Key Terms

Sample space: Set of all possible outcomes of a random experiment.

Outcome: A single possible result of an experiment.

Event: A subset of a sample space.

Probability: A number representing the uncertainty assigned to an event.

Complement: Event that an event does not occur.

Union: Event that at least one of two or more events occurs.

Intersection: Event that two or more events occur together.

Mutually exclusive events: Events that cannot occur simultaneously.

Conditional probability: Probability of an event given that another event is known.

Joint probability: Probability that multiple events occur together.

Marginal probability: Probability of an event without conditioning on another event.

Independence: Condition in which occurrence of one event does not change the probability of another.

Partition: Mutually exclusive and exhaustive collection of events.

Law of total probability: Rule for combining conditional probabilities over a partition.

Bayes’ theorem: Formula for updating a probability using evidence.

Prior probability: Belief assigned before observing new evidence.

Posterior probability: Updated belief after observing evidence.

Likelihood: Compatibility of observed data with a hypothesis or parameter value.

Odds: Ratio of the probability of an event to its complement probability.

Likelihood ratio: Ratio comparing the probability of evidence under two hypotheses.

Review Questions

Short Answer Questions

1. What is the difference between a sample space and an event?

2. State the three axioms of probability.

3. When are two events mutually exclusive?

4. Define conditional probability.

5. What is the difference between a prior and a posterior probability?

Long Answer Questions

1. Explain the addition rule for two events and why the intersection term must be subtracted.

2. Compare permutations and combinations, giving one application of each.

3. Derive Bayes’ theorem from the multiplication rule.

4. Explain why independence and mutual exclusivity are different concepts.

5. Discuss how base rates affect the interpretation of diagnostic test results.

Analytical / Application Questions

1. A factory obtains 70% of components from Supplier A, whose defect rate is 2%, and 30% from Supplier B, whose defect rate is 5%. Calculate the overall defect probability.

2. A fraud detector flags 8% of legitimate transactions and 80% of fraudulent transactions. If 1% of transactions are fraudulent, calculate the probability that a flagged transaction is actually fraudulent.

3. A classifier gives a 0.7 probability of spam based on one feature. Explain what additional information is needed to determine whether this probability is well calibrated and suitable for automatically deleting messages.

Multiple Choice Questions with Answers

Question 1

Which expression represents the complement rule?

A. P(Aᶜ)=P(A)

B. P(Aᶜ)=1−P(A)

C. P(Aᶜ)=P(A)+1

D. P(Aᶜ)=P(A∩A)

Correct Answer: B. P(Aᶜ)=1−P(A)

Question 2

If A and B are mutually exclusive, which statement is true?

A. P(A∩B)=1

B. P(A∩B)=P(A)P(B) always

C. P(A∩B)=0

D. P(A)=P(B)

Correct Answer: C. P(A∩B)=0

Question 3

A fair die is rolled once. What is the probability of obtaining an even number?

A. 1/6

B. 1/3

C. 1/2

D. 2/3

Correct Answer: C. 1/2

Question 4

Which formula defines conditional probability P(A|B), assuming P(B)>0?

A. P(A)/P(B)

B. P(A∩B)/P(B)

C. P(A∪B)/P(B)

D. P(B)/P(A∩B)

Correct Answer: B. P(A∩B)/P(B)

Question 5

If P(A)=0.4 and P(B)=0.5 and A and B are independent, what is P(A∩B)?

A. 0.1

B. 0.2

C. 0.4

D. 0.9

Correct Answer: B. 0.2

Question 6

Which quantity is the posterior probability in Bayes’ theorem?

A. P(A)

B. P(B|A)

C. P(B)

D. P(A|B)

Correct Answer: D. P(A|B)

Question 7

A disease has prevalence 2%, and a test has sensitivity 95% and false-positive rate 5%. What is approximately P(disease|positive)?

A. 2%

B. 5%

C. 27.9%

D. 95%

Correct Answer: C. 27.9%

Question 8

What is the likelihood ratio for evidence B when comparing hypothesis A with its complement?

A. P(A)/P(B)

B. P(B|A)/P(B|Aᶜ)

C. P(A|B)/P(A)

D. P(B)−P(A)

Correct Answer: B. P(B|A)/P(B|Aᶜ)

Question 9

Which statement about likelihood is correct?

A. Likelihood is always a probability distribution over parameters

B. Likelihood evaluates data compatibility across parameter values

C. Likelihood must equal the prior probability

D. Likelihood is defined only for mutually exclusive events

Correct Answer: B. Likelihood evaluates data compatibility across parameter values

Question 10

A supplier provides 60% of components with a 1% defect rate, and another provides 40% with a 3% defect rate. What is the overall defect probability?

A. 1.0%

B. 1.8%

C. 2.0%

D. 3.0%

Correct Answer: B. 1.8%