11. Information Theory and Probabilistic Losses

Introduction and Context

Machine learning systems make decisions under uncertainty. A classifier does not merely assign a label; it often estimates a probability distribution over possible labels. A language model does not simply predict the next token; it assigns probabilities to an enormous set of alternatives. A representation-learning system attempts to preserve information useful for a task while discarding irrelevant variation. Information theory provides a mathematical language for describing these activities.

The central ideas in this chapter are entropy, cross-entropy, relative entropy, mutual information, likelihood, and coding. These concepts connect probability models with learning objectives. Cross-entropy becomes the standard loss for probabilistic classification, relative entropy measures the mismatch between distributions, and likelihood explains why many losses are statistically justified. Mutual information supports reasoning about feature selection, representation learning, and dependence. Coding interpretations make these quantities intuitive: a good probabilistic model assigns short codes to events that occur frequently and long codes to events that are surprising.

The chapter develops these ideas from their mathematical foundations and then relates them to classification, model evaluation, feature selection, and modern AI systems.

Learning Objectives

After studying this chapter, students should be able to:

1. Define self-information, entropy, cross-entropy, relative entropy, and mutual information.

2. Derive the relationship between likelihood and probabilistic loss functions.

3. Explain why cross-entropy is appropriate for classification.

4. Distinguish entropy, cross-entropy, and Kullback–Leibler divergence.

5. Apply mutual information to feature selection and representation learning.

6. Interpret probabilistic losses from both statistical and coding perspectives.

7. Evaluate models using log loss, calibration, perplexity, and related measures.

8. Identify limitations, implementation risks, and ethical concerns associated with information-based objectives.

Information as Uncertainty Reduction

Let a discrete random variable X take values in a finite set with probabilities p(x). The information, or surprisal, associated with observing x is

I(x) = -log p(x).

The logarithm may be taken in base 2, producing units called bits, or in base e, producing nats. A rare event has high surprisal, whereas a highly probable event has low surprisal. The negative sign ensures that information is non-negative when 0 < p(x) ≤ 1.

The entropy of X is the expected surprisal:

H(X) = -Σx p(x) log p(x).

Entropy measures the average uncertainty before observing X. For a fair binary variable, entropy is one bit. For a binary variable with probabilities 0.99 and 0.01, entropy is much lower because the outcome is more predictable. For a categorical variable with k equally likely outcomes, entropy is log k, the maximum possible entropy for that support.

Entropy is not the same as variability in every informal sense. It is a property of a probability distribution and depends on the uncertainty relevant to the variable being considered. A dataset can contain many distinct values while still having low entropy if one value dominates.

Joint, Conditional, and Chain-Rule Relationships

For two random variables X and Y, joint entropy is

H(X,Y) = -Σx,y p(x,y) log p(x,y).

Conditional entropy measures the remaining uncertainty in X after Y is known:

H(X|Y) = -Σx,y p(x,y) log p(x|y).

The chain rule states

H(X,Y) = H(Y) + H(X|Y) = H(X) + H(Y|X).

This identity is important in machine learning because prediction can be understood as reducing conditional uncertainty. If observing a feature vector Z substantially lowers H(Y|Z), then Z contains information useful for predicting Y. However, predictive usefulness is not identical to causal importance: a feature may be highly informative because it acts as a proxy or because it records leakage from the target.

Cross-Entropy and Relative Entropy

Suppose the true distribution is p and a model uses q. The cross-entropy from p to q is

H(p,q) = -Σx p(x) log q(x).

It is the expected code length when outcomes generated by p are encoded using a code optimized for q. If q assigns very small probability to events that occur under p, the cross-entropy becomes large.

The Kullback–Leibler divergence, or relative entropy, is

DKL(p||q) = Σx p(x) log[p(x)/q(x)].

It satisfies

H(p,q) = H(p) + DKL(p||q).

Since entropy H(p) does not depend on the model q, minimizing cross-entropy with respect to q is equivalent to minimizing DKL(p||q). Gibbs’ inequality guarantees DKL(p||q) ≥ 0, with equality if and only if p and q are equal almost everywhere. KL divergence is not generally symmetric: DKL(p||q) and DKL(q||p) need not be equal. It is therefore a divergence rather than a distance.

A critical practical issue is support mismatch. If p(x) > 0 but q(x) = 0, then DKL(p||q) and cross-entropy are infinite. In numerical implementations, probabilities are commonly clipped or computed through stable log-softmax operations, but such techniques do not remove the conceptual problem of assigning zero probability to plausible events.

Mutual Information and Dependence

Mutual information measures the reduction in uncertainty about one variable obtained by observing another:

I(X;Y) = H(X) - H(X|Y).

Equivalent forms are

I(X;Y) = H(Y) - H(Y|X) = DKL(p(x,y)||p(x)p(y)).

Thus, mutual information is zero exactly when X and Y are independent. Unlike correlation, it can capture nonlinear as well as linear dependence. It is symmetric and non-negative.

For feature selection, I(Xj;Y) can rank individual features according to their association with a target. However, selecting features independently may produce redundant variables. Two features may each have high mutual information with Y while conveying nearly identical information. More advanced selection criteria consider conditional mutual information, such as I(Xj;Y|S), where S is the set of already selected features.

Mutual information also has a representation-learning interpretation. Let Z be a learned representation and Y a target. Maximizing I(Z;Y) encourages task-relevant information to be retained. In contrast, minimizing I(Z;X) can encourage compression of the input. These goals may conflict: a representation cannot preserve every detail, discard all irrelevant information, and remain perfectly informative about every possible future task. The desired balance depends on the application.

Likelihood as a Learning Principle

Let a dataset D = {(xi, yi)} from a probabilistic model pθ(y|x). The conditional likelihood is

L(θ) = Πi pθ(yi|xi).

Because products can be numerically unstable, learning normally maximizes the log-likelihood:

log L(θ) = Σi log pθ(yi|xi).

Equivalently, it minimizes the negative log-likelihood (NLL):

NLL(θ) = -Σi log pθ(yi|xi).

Dividing by the number of observations produces average log loss. Under an appropriate model and sampling assumption, maximum likelihood estimation selects parameters that make the observed data probable. It is also a form of empirical cross-entropy minimization. For one-hot classification labels, the empirical cross-entropy is

-1/n Σi Σk yik log qθ(k|xi),

which reduces to -1/n Σi log qθ(yi|xi).

This connection explains why cross-entropy is not an arbitrary engineering convention. It is the negative log-likelihood of a categorical model and a proper scoring rule: in expectation, a forecaster minimizes its score by reporting the true distribution.

Probabilistic Losses in Classification

Binary classification

For yi ∈ {0,1} and predicted probability pi = P(Y=1|xi), binary cross-entropy is

L = -[yi log pi + (1-yi)log(1-pi)].

The loss strongly penalizes confident incorrect predictions. Predicting 0.99 for a negative case incurs a much larger penalty than predicting 0.60. This property is desirable when probabilities are used for ranking, decision thresholds, resource allocation, or risk estimation.

The logistic model represents pi through the sigmoid function σ(z) = 1/(1+e^-z). Its NLL is convex in the linear score under standard conditions, although neural-network versions are generally non-convex in their parameters.

Multiclass classification

For K classes, logits zk are converted to probabilities by softmax:

qk = exp(zk)/Σj exp(zj).

With one-hot target y, multiclass cross-entropy is

L = -Σk yk log qk.

If the correct class receives probability 0.8, its contribution is -log(0.8). If it receives 0.01, the contribution is much larger. The loss therefore evaluates the entire probabilistic prediction, not merely whether the top-ranked class is correct.

Label smoothing and class imbalance

When labels may contain noise, one-hot targets can encourage excessive confidence. Label smoothing replaces a target probability of one with a value slightly below one and distributes a small mass among other classes. This can improve generalization and calibration, but it changes the target distribution and may be harmful when exact probabilities are important.

In imbalanced classification, unweighted cross-entropy may be dominated by the majority class. Class weights, balanced sampling, focal loss, or threshold adjustment may be used. These methods optimize different objectives. A weighted loss can improve minority-class performance while making its raw probabilities poorly calibrated unless additional calibration is performed.

Coding, Compression, and Perplexity

The coding interpretation gives a unified view of probabilistic loss. An event assigned probability q(x) has ideal code length approximately -log q(x). A model that assigns high probability to observed events compresses the data efficiently. Average negative log-likelihood is therefore average codelength under the model.

For a language model, the exponentiated average log loss is perplexity:

PPL = exp[-1/n Σi log q(xi|x<i)]

when natural logarithms are used. Perplexity can be interpreted as the effective number of equally likely choices predicted at each step. Lower perplexity indicates better predictive coding under the evaluation distribution, but it is not automatically equivalent to better factuality, fairness, usefulness, or downstream task performance.

The coding perspective also clarifies why a model that is accurate in top-1 classification may still have poor log loss. Accuracy only checks the largest predicted probability. Log loss evaluates how probability mass is distributed across all alternatives.

Evaluation: Accuracy Is Not Enough

A probabilistic classifier should be assessed using metrics suited to its intended use. Log loss evaluates probability quality. The Brier score for binary prediction is

B = 1/n Σi(pi-yi)^2.

Unlike log loss, the Brier score penalizes errors quadratically and is bounded. Calibration examines whether predictions assigned probability p are correct approximately p proportion of the time. Reliability diagrams, expected calibration error, and calibration curves are common diagnostic tools, though estimated calibration error depends on binning and sample size.

Discrimination and calibration are distinct. A model may rank positive cases well, yielding a high area under the ROC curve, while producing systematically overconfident probabilities. Conversely, calibration can be good while discrimination is weak. Evaluation should reflect the decision problem, class prevalence, costs of errors, and whether probabilities will be consumed by humans or downstream systems.

Train-test cross-entropy comparisons are meaningful only when data distributions, label definitions, preprocessing, and evaluation protocols are comparable. A lower loss on a shifted or leaked test set is not evidence of general improvement.

Applications in Feature Selection and Representation Learning

Information-based feature selection is useful when the number of candidate variables is large. Mutual information can detect nonlinear relationships that correlation misses. Nevertheless, estimates from finite samples can be biased, especially for continuous or high-cardinality variables. Discretization choices, nearest-neighbor estimators, and regularization affect results.

In representation learning, autoencoders minimize reconstruction losses, while supervised encoders often minimize cross-entropy or a related task loss. Contrastive methods frequently seek representations in which related examples are close and unrelated examples are separated. Some methods use mutual-information-inspired objectives, although estimating mutual information in high dimensions is difficult. Variational bounds replace an intractable quantity with an optimizable surrogate, introducing a trade-off between statistical tightness and computational stability.

The information bottleneck perspective considers an encoder p(z|x) and seeks a representation that retains information about Y while limiting information about X. A typical objective has the form

I(Z;X) - β I(Z;Y),

or an equivalent sign convention depending on whether the objective is minimized or maximized. The parameter β controls the compression–prediction trade-off. This framework is conceptually valuable, but empirical claims about deep networks and information compression must be interpreted carefully because deterministic continuous representations can have technically problematic mutual-information quantities.

An Illustrative Case: Probabilities for Medical Triage

Background

Consider an illustrative hospital triage system that predicts whether a patient is likely to require intensive monitoring within 24 hours. The model uses symptoms, vital signs, laboratory results, and historical information. Its output probability influences bed allocation and the timing of clinical review.

Situation and problem

Two candidate models have similar accuracy. Model A has a slightly higher area under the ROC curve, while Model B has lower validation cross-entropy and better calibration. The deployment team must decide which model is safer and how it should be monitored.

Analysis

Because the output is used as a risk estimate rather than merely a label, probability quality matters. A model that predicts 0.90 should correspond approximately to a 90% event rate among comparable cases. Overconfident probabilities can cause unnecessary escalation and resource use; underconfident probabilities can delay care. Log loss is especially sensitive to highly confident errors, making it useful for identifying dangerous probability assignments. The team also examines subgroup calibration, missing-data patterns, prevalence changes, and whether historical treatment decisions have introduced label bias.

Approach and outcome

The team selects Model B provisionally, applies probability calibration on a separate validation set, and defines operating thresholds with clinicians rather than treating 0.5 as automatically optimal. Monitoring includes log loss, calibration curves, sensitivity at clinically important thresholds, and subgroup performance. A human review process remains in place for cases with unusual combinations of symptoms or missing measurements.

Learning points

The case illustrates that classification accuracy and ranking performance do not fully characterize a probabilistic system. The appropriate loss depends on how predictions are used. Calibration, distribution shift, subgroup analysis, and human governance are part of responsible deployment, not optional additions.

Limitations, Risks, and Common Mistakes

A frequent mistake is to treat cross-entropy as a universal measure of model quality. It is appropriate when the predictive distribution itself matters, but a cost-sensitive decision may require a different utility function. Another mistake is to compare losses across datasets with different label noise, class prevalence, or tokenization schemes.

KL divergence is directional, so its direction must be interpreted. DKL(p||q) asks how costly it is to use q when p generates the data; reversing the arguments answers a different question. Mutual information estimated from small samples may be severely biased upward, particularly when variables have many categories. Feature selection performed before cross-validation can also cause leakage.

Numerical errors arise when probabilities are represented directly and logarithms of zero are computed. Stable implementations use logits, log-sum-exp identities, and appropriate masking. In sequence models, padding tokens must not contribute to the loss. For imbalanced tasks, changing class weights changes the effective training distribution and may require recalibration.

Information objectives can reproduce social and institutional biases present in data. A feature with high mutual information may encode protected status or a proxy for it. A compression objective may remove information needed to serve minority groups. Privacy is also relevant: highly informative representations can retain sensitive attributes, while model outputs and gradients may reveal information about training examples.

Governance and Professional Practice

Probabilistic systems should document the target variable, sampling population, loss function, calibration method, and intended decision. Teams should distinguish statistical uncertainty from operational uncertainty and communicate that a predicted probability is conditional on the data and model assumptions. High-stakes applications require audit trails, access controls, monitoring for drift, and procedures for contesting or reviewing automated decisions.

Model evaluation should include subgroup-specific analyses where legally and ethically appropriate. However, subgroup calibration can involve trade-offs, particularly when base rates differ. Such choices should be made transparently with domain experts and affected stakeholders rather than hidden in optimization code.

Current Developments

Recent machine-learning practice increasingly combines probabilistic prediction with uncertainty estimation, conformal prediction, distribution-shift detection, and selective prediction, in which a model may abstain when confidence is inadequate. Large language models are evaluated with token-level cross-entropy and perplexity, but current research also emphasizes calibration, factual reliability, preference alignment, and task-specific utility. Generative models use likelihood-related objectives, variational bounds, score matching, or diffusion-based criteria, each reflecting different approximations to probability modeling.

The continuing challenge is to connect mathematically coherent objectives with socially meaningful outcomes. A lower information-theoretic loss is valuable evidence, but it is not a complete definition of intelligence, safety, fairness, or usefulness.

Best Practices

1. Use negative log-likelihood or cross-entropy when calibrated probabilistic predictions matter.

2. Evaluate calibration separately from discrimination and accuracy.

3. Use stable logit-based implementations and mask irrelevant observations.

4. Select the direction of KL divergence according to the modeling question.

5. Estimate mutual information with attention to sample size, bias, and redundancy.

6. Perform feature selection inside the training folds to prevent leakage.

7. Recalibrate probabilities after class weighting, resampling, or deployment shift.

8. Define decision thresholds using domain costs rather than defaulting to 0.5.

9. Examine subgroup performance, privacy risks, and proxy variables.

10. Report the data distribution, target definition, evaluation protocol, and limitations.

Chapter Summary

Information theory supplies a rigorous framework for uncertainty, prediction, and representation. Entropy measures expected uncertainty, cross-entropy measures the coding cost of using one distribution for data generated by another, and KL divergence isolates the excess cost caused by distributional mismatch. Mutual information quantifies dependence and supports feature selection and representation-learning analysis. Maximum likelihood leads directly to negative log-likelihood and classification cross-entropy, explaining their widespread use.

The coding view interprets log loss as average description length, while evaluation practice distinguishes probability quality from accuracy and ranking. Applications require attention to calibration, class imbalance, leakage, numerical stability, distribution shift, privacy, and fairness. Information-based objectives are powerful but should be selected and interpreted in relation to the decisions a system supports.

Key Terms

Entropy: Expected surprisal of a random variable.

Self-information: The quantity -log p(x) associated with an individual event.

Cross-entropy: Expected negative log probability assigned by q to outcomes generated by p.

Kullback–Leibler divergence: Directional measure of excess coding cost between two distributions.

Mutual information: Reduction in uncertainty about one variable gained from another.

Conditional entropy: Uncertainty in one variable given knowledge of another.

Likelihood: Probability of observed data viewed as a function of model parameters.

Log-likelihood: Logarithm of the likelihood, usually optimized for numerical stability.

Negative log-likelihood: The negative log-likelihood used as a probabilistic loss.

Proper scoring rule: A scoring rule whose expected optimum is truthful probability reporting.

Surprisal: Information associated with an event, equal to its negative log probability.

Calibration: Agreement between predicted probabilities and observed frequencies.

Brier score: Mean squared error between predicted probabilities and binary outcomes.

Perplexity: Exponentiated average negative log-likelihood, often used for sequence models.

Conditional mutual information: Information shared by two variables given a third.

Label smoothing: Replacement of hard targets with softened probability distributions.

Information bottleneck: Framework balancing retained target information against input compression.

Distribution shift: Change between training, validation, or deployment data distributions.

Review Questions

Short Answer Questions

1. Why is entropy interpreted as expected surprisal?

2. State the relationship between cross-entropy and KL divergence.

3. What does mutual information equal when two variables are independent?

4. Why is log-likelihood preferred to likelihood in practical optimization?

5. Distinguish calibration from discrimination.

Long Answer Questions

1. Derive the multiclass cross-entropy loss from the likelihood of a categorical model.

2. Explain the coding interpretation of entropy, cross-entropy, and KL divergence.

3. Discuss the use of mutual information in feature selection, including redundancy and estimation bias.

4. Compare accuracy, log loss, Brier score, and calibration as classification-evaluation tools.

5. Explain the information bottleneck perspective and its limitations for continuous deep representations.

Analytical / Application Questions

1. A classifier assigns probabilities 0.99, 0.60, and 0.01 to three positive examples. Explain which example contributes most to binary log loss and why.

2. A model has excellent ROC-AUC but poor calibration after class-weighted training. Propose an evaluation and remediation procedure.

3. Two candidate features have high mutual information with a target but are nearly duplicates. Explain how a conditional-information strategy could select between them.

22. Multiple Choice Questions with Answers

Question 1

For a true distribution p and model distribution q, which identity is correct?

A. H(p,q)=H(p)-DKL(p||q)

B. H(p,q)=H(p)+DKL(p||q)

C. H(p,q)=DKL(q||p)-H(p)

D. H(p,q)=H(q)+DKL(q||p)

Correct Answer: B. H(p,q)=H(p)+DKL(p||q)

Question 2

A binary classifier assigns probability 0.01 to the class that actually occurs. Why is its log loss large?

A. The prediction has low entropy only

B. The model has assigned very little probability to the observed event

C. Accuracy is necessarily zero on the whole dataset

D. The Brier score must be infinite

Correct Answer: B. The model has assigned very little probability to the observed event

Question 3

Which quantity is zero exactly when X and Y are independent?

A. H(X)

B. H(X|Y)

C. I(X;Y)

D. H(X,Y)

Correct Answer: C. I(X;Y)

Question 4

Maximum likelihood for a categorical classifier is equivalent to minimizing which objective?

A. Average negative log-likelihood

B. Mean absolute error only

C. Entropy of the input features

D. ROC-AUC loss exclusively

Correct Answer: A. Average negative log-likelihood

Question 5

A model has high ROC-AUC but systematically predicts probabilities that are too large. What property is deficient?

A. Discrimination

B. Calibration

C. Support size

D. Feature dimensionality

Correct Answer: B. Calibration

Question 6

Why can independent mutual-information feature ranking select a poor feature subset?

A. Mutual information cannot detect any dependence

B. It always selects only continuous features

C. It may select redundant features carrying similar information

D. It is identical to accuracy

Correct Answer: C. It may select redundant features carrying similar information

Question 7

What does perplexity represent when computed from a language model's average negative log-likelihood?

A. The number of model parameters

B. The effective number of equally likely choices per prediction

C. The fraction of correctly classified tokens

D. The entropy of the training labels only

Correct Answer: B. The effective number of equally likely choices per prediction

Question 8

Which implementation practice helps prevent numerical problems in multiclass cross-entropy?

A. Taking logs of raw probabilities without safeguards

B. Using logits with a stable log-sum-exp computation

C. Replacing every probability by zero

D. Removing all low-frequency classes

Correct Answer: B. Using logits with a stable log-sum-exp computation

Question 9

After class-weighted training, what may be necessary if predicted probabilities are to be interpreted as risks?

A. Calibration on suitable validation data

B. Removing the validation set

C. Maximizing entropy of predictions

D. Using accuracy as a probability estimate

Correct Answer: A. Calibration on suitable validation data

Question 10

In an information-bottleneck objective, the compression term generally encourages the representation to do what?

A. Retain every detail of the input

B. Discard input information judged less relevant to the target

C. Guarantee causal features

D. Make all classes equally probable

Correct Answer: B. Discard input information judged less relevant to the target