Probability and Uncertainty

Introduction

Information technology systems rarely operate with complete information. A network may fail, a transaction may be fraudulent, a diagnostic classifier may be wrong, or a sensor may report an imprecise value. Probability provides a formal language for representing uncertainty and for making rational decisions when outcomes are not known in advance.

Probability is not merely the study of gambling. It supports reliability engineering, cybersecurity, database sampling, software testing, artificial intelligence, risk analysis, forecasting, and performance modelling. This chapter develops the rules needed to calculate probabilities, update beliefs using evidence, model uncertain quantities with random variables, and interpret common probability distributions.

A probability is a number between 0 and 1. Probability 0 represents impossibility, probability 1 represents certainty, and intermediate values express degrees of likelihood. Probabilities may also be expressed as percentages: a probability of 0.8 is 80%.

Learning Objectives

By the end of this chapter, you should be able to:

1. Define sample spaces, events, probability, independence, and conditional probability.
2. Apply complement, addition, multiplication, total-probability, and Bayes’ rules.
3. Distinguish mutually exclusive events from independent events.
4. Use discrete and continuous random variables and interpret expectation and variance.
5. Apply the binomial, Poisson, uniform, and normal distributions in IT contexts.
6. Assess reliability, risk, diagnostic evidence, and probabilistic AI outputs.
7. Recognize modelling assumptions, sources of bias, and limitations of probabilistic conclusions.

1. Foundations of Probability

Experiments, outcomes, and events

A random experiment is a process whose outcome cannot be predicted with certainty before it occurs. Examples include selecting a packet from a network trace, observing whether a server request fails, or classifying an email as spam.

The sample space, denoted by S, is the set of all possible outcomes. An event is a subset of S. For example, if a bit is transmitted, S = {0, 1}; the event “the bit is 1” is {1}.

For a finite sample space with equally likely outcomes,

where P(A) is the probability of event A, |A| is the number of outcomes in A, and |S| is the number of outcomes in the sample space. This formula applies only when the outcomes are equally likely.

The axioms of probability are:

• P(A) ≥ 0 for every event A.
• P(S) = 1.
• If A and B cannot occur together, then P(A ∪ B) = P(A) + P(B).

Here, A ∪ B means “A or B,” while A ∩ B means “A and B.” The complement of A, written Aᶜ, means that A does not occur.

Basic rules

The complement rule is

The general addition rule is

The intersection is subtracted because it is counted twice in P(A) + P(B). If A and B are mutually exclusive, then A ∩ B is empty and the rule simplifies to P(A ∪ B) = P(A) + P(B).

Two events are independent when knowledge that one occurred does not change the probability of the other. For independent events,

Mutual exclusivity and independence are different. If two events have positive probability and are mutually exclusive, they cannot be independent, because occurrence of one makes the other impossible.

Worked example: server alerts

Suppose a monitoring system reports a CPU alert with probability 0.20 and a memory alert with probability 0.15. In a particular operating model, the alerts are independent. The probability of both alerts is

P(CPU ∩ memory) = 0.20 × 0.15 = 0.03.

The probability of at least one alert is

P(CPU ∪ memory) = 0.20 + 0.15 − 0.03 = 0.32.

Thus, the probability of at least one alert is 32%, not 35%, because simultaneous alerts were initially counted twice.

2. Conditional Probability and Bayes’ Theorem

Conditional probability

Conditional probability measures the probability of A when B is known to have occurred:

This formula requires P(B) > 0. The notation P(A | B) is read “the probability of A given B.” Rearranging gives the multiplication rule:

More generally, P(A ∩ B) = P(B | A)P(A). These two expressions are equal, but the conditional probabilities need not be equal. Confusing P(A | B) with P(B | A) is a common error known as the inverse-probability fallacy.

Events A and B are independent exactly when P(A | B) = P(A), provided P(B) > 0. This is equivalent to the product rule stated earlier.

Total probability

Suppose B₁, B₂, …, Bₖ form a partition of the sample space: they are mutually exclusive and together cover all possibilities. Then

This rule is useful when an event can arise through several different sources, such as failures from different server regions.

Bayes’ theorem

Bayes’ theorem reverses conditional probabilities:

The prior probability P(Bⱼ) represents belief before observing A. The likelihood P(A | Bⱼ) measures how compatible the evidence is with hypothesis Bⱼ. The posterior probability P(Bⱼ | A) is the updated belief after observing A. The denominator is the evidence probability P(A).

Bayesian reasoning is central to intrusion detection, medical diagnosis, spam filtering, fault diagnosis, and machine learning.

Worked example: intrusion alert

Assume that 1% of login attempts are genuinely malicious. An intrusion detector flags 95% of malicious attempts and falsely flags 4% of legitimate attempts. Let M denote “malicious” and F denote “flagged.” Thus:

P(M) = 0.01, P(F | M) = 0.95, and P(F | Mᶜ) = 0.04.

First calculate the overall flag probability:

P(F) = P(F | M)P(M) + P(F | Mᶜ)P(Mᶜ)

= (0.95)(0.01) + (0.04)(0.99)

= 0.0095 + 0.0396 = 0.0491.

Bayes’ theorem gives

P(M | F) = (0.95 × 0.01) / 0.0491 ≈ 0.1935.

Only about 19.35% of flagged attempts are malicious. The detector has high sensitivity, but the low base rate and false-positive rate mean that most alerts are still false alarms. This is why security teams should not treat an alert as proof of an attack.

3. Counting and Probability Models

Counting methods are useful when outcomes are discrete and structured. The number of ordered arrangements of n distinct objects is n!. The number of ways to choose r objects from n without regard to order is

where 0 ≤ r ≤ n. This is also written “n choose r.”

For example, the probability of exactly two defective devices in a sample of five can be calculated with combinations when each device has the same independent defect probability. Counting alone is insufficient if outcomes are not equally likely; probabilities must then be weighted by their likelihoods.

A probability model should identify:

• the possible outcomes;
• the assumptions governing their probabilities;
• the random quantity being measured; and
• the decision or inference for which the model will be used.

4. Random Variables and Numerical Summaries

A random variable assigns a numerical value to each outcome. A discrete random variable takes countable values, such as the number of failed requests. A continuous random variable can take values on an interval, such as response time.

For a discrete random variable X with possible values x and probability mass function p(x) = P(X = x),

The expected value E[X] is the long-run average under repeated comparable trials. It does not necessarily represent an outcome that can actually occur.

Variance measures spread around the mean μ = E[X]:

The standard deviation is σ = √Var(X), expressed in the same units as X. Variance is in squared units, so standard deviation is usually easier to interpret.

For a continuous random variable, probabilities are computed from a probability density function f(x). The probability of an interval is

A continuous variable has P(X = a) = 0 for any single exact value; probability is assigned to intervals. The density itself is not generally a probability and may exceed 1, provided its integral over the entire range is 1.

5. Common Distributions in IT

Bernoulli distribution

A Bernoulli random variable represents one trial with success probability p and failure probability 1 − p. If X = 1 denotes success and X = 0 failure,

Examples include whether a request succeeds, whether a packet is corrupted, or whether a classifier is correct.

Binomial distribution

The binomial distribution counts successes in n independent Bernoulli trials with the same success probability p:

for k = 0, 1, …, n. Its mean and variance are np and np(1 − p), respectively. The model requires a fixed number of trials, two outcomes per trial, constant p, and independence. If requests influence one another or p changes over time, the binomial model may be inappropriate.

Example: If five independent software tests each pass with probability 0.8, the probability that exactly four pass is

C(5,4)(0.8)⁴(0.2) = 5 × 0.4096 × 0.2 = 0.4096.

Poisson distribution

The Poisson distribution models the number of events in a fixed interval when events occur independently at a stable average rate λ:

where k = 0, 1, 2, … and λ is the expected number of events in the interval. Both mean and variance equal λ. It can model incoming requests, fault reports, or support tickets when the constant-rate assumptions are reasonable.

If a service receives an average of 3 alerts per hour, the probability of exactly 2 alerts in one hour is

P(X = 2) = e⁻³ × 3² / 2! ≈ 0.224.

Uniform distribution

A continuous variable uniformly distributed on [a,b] assigns equal density throughout the interval:

Its mean is (a + b)/2 and its variance is (b − a)²/12. It is useful for representing bounded uncertainty when no value within an interval is preferred, although that assumption must be justified rather than used automatically.

Normal distribution

The normal distribution has mean μ and standard deviation σ > 0, with density

It is symmetric and bell-shaped. The standardized value, or z-score, is

A z-score states how many standard deviations x lies above or below the mean. The normal model is often useful for measurement noise and aggregated effects, but it is not suitable for every dataset. Response times, for example, are often nonnegative and right-skewed.

The empirical 68–95–99.7 guideline states that approximately 68%, 95%, and 99.7% of a normal population lie within 1, 2, and 3 standard deviations of μ, respectively. These are approximations, not universal laws.

6. Reliability and Risk

Reliability is the probability that a system performs its required function for a specified time under stated conditions. Risk combines uncertain events with consequences. A simple expected-loss model is

This value supports comparison of controls, but it does not capture every concern. Low-probability catastrophic events may require protection even when their expected loss is moderate.

For independent components connected in series, where the system works only if every component works,

where Rᵢ is the reliability of component i. For two independent parallel components, where the system works if at least one works,

These formulas require independence and clearly defined operating periods. Common-cause failures, such as a shared power supply or software defect, violate the independence assumption.

Applied case: cloud service reliability

Background. An illustrative online retailer uses two independent application servers behind a load balancer. Each server has monthly reliability 0.98. The load balancer has reliability 0.99.

Problem. Estimate the monthly reliability of the service, assuming the service operates when the load balancer and at least one application server operate.

Analysis and approach. The application tier reliability is

R_app = 1 − (1 − 0.98)² = 1 − 0.0004 = 0.9996.

The total service reliability is

R_service = 0.99 × 0.9996 = 0.989604,

or approximately 98.96%. The corresponding monthly unavailability probability is 1 − 0.989604 = 0.010396, about 1.04%.

Interpretation. Adding a second server greatly improves the application tier, but the load balancer remains a single point of failure. A redundant load-balancing design may therefore produce more benefit than adding another application server.

Learning points. Reliability analysis must define the system success condition, identify dependencies, and examine bottlenecks. A numerical estimate is only as credible as its assumptions and data.

7. Probability in Artificial Intelligence and Data Systems

A classifier may output P(class | features), such as the probability that a message is spam given its words. This output is useful only when it is interpreted as a conditional probability under a defined data-generating context.

Two important evaluation ideas are discrimination and calibration. A discriminating model ranks positive cases above negative cases. A calibrated model produces probabilities that match observed frequencies: among cases assigned probability 0.7, approximately 70% should be positive over a sufficiently large comparable group.

Accuracy alone is inadequate for imbalanced problems. Precision and recall are defined as

where TP is true positives, FP false positives, and FN false negatives. Precision answers “of the positive predictions, how many were correct?” Recall answers “of the actual positives, how many were detected?” The appropriate measure depends on the cost of errors.

Probability estimates can be distorted by sampling bias, data drift, label errors, proxy variables, and feedback loops. A fraud detector trained on historical investigations may learn investigators’ past priorities rather than fraud itself. Systems that affect employment, credit, healthcare, or access should therefore include validation, monitoring, documentation, human oversight, and procedures for contesting decisions.

8. Uncertainty, Assumptions, and Responsible Use

Probability does not eliminate uncertainty; it quantifies it under a model. Important distinctions include:

• Aleatory uncertainty: inherent variation, such as random packet loss.
• Epistemic uncertainty: incomplete knowledge, such as uncertain failure rates.
• Model uncertainty: uncertainty about whether the chosen distribution or causal structure is appropriate.

Best practices include checking data quality, stating the population and time period, reporting confidence or credible intervals when estimating quantities, testing sensitivity to assumptions, and distinguishing correlation from causation. Probabilities should not be presented with unjustified precision. A value of 0.731 may falsely suggest greater knowledge than a value of approximately 0.73.

In governance-sensitive applications, probability-based decisions should be auditable and proportionate. Privacy must be protected when estimating risks from personal data, and fairness should be examined across relevant groups. A model’s probability output is evidence for a decision, not automatically the decision itself.

Contemporary Developments

Modern IT increasingly uses probabilistic graphical models, Bayesian optimization, probabilistic programming, ensemble prediction, and uncertainty-aware deep learning. These methods attempt to represent uncertainty in parameters, predictions, or data. However, sophisticated algorithms do not remove the need for representative data, appropriate assumptions, calibration, and domain review. In production systems, monitoring probability calibration and detecting distribution shift are as important as measuring initial accuracy.

Chapter Summary

Probability represents uncertainty numerically through a sample space, events, and axioms. Complement and addition rules calculate probabilities of non-occurrence and combined events. Conditional probability describes uncertainty after evidence is observed, while multiplication, total-probability, and Bayes’ rules connect joint, conditional, prior, and posterior probabilities. Independence is a specific mathematical condition and must not be confused with mutual exclusivity.

Random variables convert uncertain outcomes into numerical quantities. Expected value describes a long-run average; variance and standard deviation describe spread. Bernoulli and binomial models describe binary trials, Poisson models count events at a stable rate, the uniform model represents bounded equal-density uncertainty, and the normal model describes many symmetric measurement processes. Each distribution depends on assumptions.

Reliability and risk analysis use probability to evaluate system operation and expected loss. In AI, probability outputs must be assessed for discrimination, calibration, bias, and decision consequences. Responsible probabilistic reasoning requires transparent assumptions, suitable data, sensitivity analysis, monitoring, and ethical governance.

Key Terms

Probability: A numerical measure between 0 and 1 expressing uncertainty about an event.

Sample space: The set of all possible outcomes of an experiment.

Event: A subset of the sample space.

Complement: The event that a specified event does not occur.

Mutually exclusive: Events that cannot occur together.

Independent events: Events for which occurrence of one does not change the probability of the other.

Conditional probability: The probability of an event given that another event has occurred.

Bayes’ theorem: A rule for updating a probability using evidence.

Prior probability: Belief about a hypothesis before new evidence.

Posterior probability: Updated belief after evidence.

Random variable: A numerical function of random outcomes.

Expected value: The probability-weighted long-run average of a random variable.

Variance: The probability-weighted squared spread around the mean.

Probability mass function: Probabilities assigned to values of a discrete random variable.

Probability density function: A function whose integral over an interval gives a continuous probability.

Binomial distribution: A model for successes in a fixed number of independent binary trials.

Poisson distribution: A model for event counts over a fixed interval at an average rate.

Reliability: The probability that a system performs its required function under stated conditions.

Calibration: Agreement between predicted probabilities and observed frequencies.

Expected loss: The probability-weighted consequence of uncertain events.

Review Questions

Short Answer Questions

1. What is the difference between a sample space and an event?
2. State the general addition rule for two events and explain why an intersection term is subtracted.
3. What condition is required for the conditional probability P(A | B) to be defined by its standard formula?
4. Distinguish mutual exclusivity from independence.
5. What does a calibrated probability model mean?

Long Answer Questions

1. Explain Bayes’ theorem, including the roles of prior probability, likelihood, evidence, and posterior probability. Illustrate the explanation with an IT example.
2. Compare the binomial, Poisson, and normal distributions, including their variables, assumptions, and suitable IT applications.
3. Explain expected value, variance, and standard deviation and discuss why each is useful in uncertainty analysis.
4. Derive the reliability of a two-component parallel system under independence and explain how common-cause failures affect the result.
5. Discuss why accuracy alone is insufficient for evaluating a probabilistic cybersecurity or AI system.

Analytical / Application Questions

1. A detector flags 8% of login attempts. Among flagged attempts, 25% are later confirmed malicious. Explain what additional information is needed to calculate the detector’s false-positive rate and how Bayes’ theorem relates the quantities.
2. A service has two independent servers, each with reliability 0.97, and a load balancer with reliability 0.995. Calculate the service reliability when the service requires the load balancer and at least one server. State the assumptions.
3. A support platform receives an average of 4 independent tickets per hour. Use an appropriate distribution to calculate the probability of exactly 2 tickets in an hour, and evaluate whether the model would be questionable if ticket arrivals were strongly clustered after outages.

Multiple Choice Questions with Answers

Question 1

If P(A) = 0.3, what is P(Aᶜ)?

A. 0.3

B. 0.7

C. 1.3

D. −0.3

Correct Answer: B. 0.7

Question 2

For independent events A and B, which relationship is correct?

A. P(A ∩ B) = P(A) + P(B)

B. P(A | B) = 0

C. P(A ∩ B) = P(A)P(B)

D. P(A ∪ B) = P(A)P(B)

Correct Answer: C. P(A ∩ B) = P(A)P(B)

Question 3

A test has P(M) = 0.01, P(F | M) = 0.95, and P(F | Mᶜ) = 0.04. Approximately what is P(M | F)?

A. 0.95

B. 0.04

C. 0.50

D. 0.1935

Correct Answer: D. 0.1935

Question 4

Which distribution is most directly suited to the number of events in a fixed interval with a stable average rate?

A. Poisson

B. Uniform

C. Bernoulli

D. Normal

Correct Answer: A. Poisson

Question 5

If five independent tests each pass with probability 0.8, what is the probability that exactly four pass?

A. 0.32768

B. 0.4096

C. 0.512

D. 0.8192

Correct Answer: B. 0.4096

Question 6

For a Poisson random variable with λ = 3, what are its mean and variance?

A. Mean 3 and variance 9

B. Mean 9 and variance 3

C. Mean 3 and variance 3

D. Mean 0 and variance 3

Correct Answer: C. Mean 3 and variance 3

Question 7

A system works if at least one of two independent components works. If their reliabilities are 0.9 and 0.8, what is system reliability?

A. 0.72

B. 0.80

C. 0.90

D. 0.98

Correct Answer: D. 0.98

Question 8

What does a z-score of 2 indicate?

A. The value is two standard deviations above the mean.

B. The value is twice the mean.

C. The probability is exactly 2.

D. The variance is two units.

Correct Answer: A. The value is two standard deviations above the mean.

Question 9

Which measure answers: “Of the positive predictions, what proportion were correct?”

A. Recall

B. Precision

C. Variance

D. Reliability

Correct Answer: B. Precision

Question 10

A model is calibrated if, among cases assigned probability 0.7, approximately:

A. 70% are positive in a comparable large group

B. 70% are always classified correctly

C. 30% are positive

D. All cases have identical features

Correct Answer: A. 70% are positive in a comparable large group