Model Evaluation, Interpretation and Error Analysis

Introduction

A deep-learning or natural-language-processing model is not useful merely because it produces predictions. It must be evaluated against an appropriate objective, interpreted responsibly, and examined for systematic errors. Evaluation answers questions such as: How often is the model correct? Which types of examples does it misunderstand? Does it perform consistently across groups and domains? Can its output be trusted in practice?

Model evaluation is therefore broader than calculating one score on a test set. It includes experimental design, metric selection, statistical comparison, qualitative inspection, explainability, robustness testing, and communication of limitations. In NLP, evaluation is particularly challenging because language has many valid expressions of the same meaning. A translation, summary, or generated answer may be useful even when it does not exactly match a reference sentence.

Learning Objectives

By the end of this chapter, you should be able to:

• design suitable training, validation, and test procedures;
• construct and interpret confusion matrices;
• calculate and explain accuracy, precision, recall, F1, ROC-AUC, perplexity, BLEU, and ROUGE;
• distinguish threshold-dependent from threshold-independent evaluation;
• perform qualitative error analysis and identify failure patterns;
• use explainability methods critically rather than treating them as proof of correctness;
• test robustness, fairness, and distribution shift;
• communicate evaluation results in a technically responsible manner.

1. Evaluation Design: From Data Splitting to Evidence

1.1 Training, validation, and test data

The training set is used to estimate model parameters. The validation set is used during development to select hyperparameters, architectures, preprocessing decisions, thresholds, and stopping points. The test set is reserved for final evaluation after development decisions have been completed.

A common split is 70% training, 15% validation, and 15% testing, but there is no universal ratio. The appropriate design depends on dataset size, task variability, computational cost, and the need for subgroup analysis. For small datasets, k-fold cross-validation may provide a more stable estimate. In k-fold cross-validation, the data are divided into k non-overlapping folds; each fold is used once for validation while the remaining folds are used for training.

The test set must not influence model selection. Repeatedly checking test performance and changing the model creates test-set leakage: the test set gradually becomes part of the development process, and the reported score becomes optimistic.

1.2 Preventing leakage

Data leakage occurs when information unavailable at prediction time enters the training process. Examples include:

• normalising all records before splitting, when summary statistics contain information from the test set;
• placing nearly identical documents in both training and test sets;
• using future events to predict a past event;
• allowing multiple messages from the same user to appear in different partitions;
• selecting features using labels from the complete dataset.

Splitting should normally occur before preprocessing, with preprocessing parameters fitted only on the training data and then applied unchanged to validation and test data.

1.3 Stratification and temporal evaluation

For imbalanced classification, stratified sampling preserves approximately similar class proportions in each partition. For time-dependent applications, random splitting may be inappropriate because it allows future patterns to predict the past. A temporal split trains on earlier observations and tests on later observations. This better represents deployment, although it may produce a harder and more realistic test.

1.4 Repeated runs and uncertainty

Deep-learning training is stochastic. Random initialisation, minibatch order, data augmentation, and hardware operations can produce different results. A professional report should record the random seeds and, where feasible, report the mean and standard deviation across multiple runs. A confidence interval can also be reported, but its construction must match the sampling design and should not be interpreted as a guarantee that the true performance lies within it.

2. Classification Metrics

Consider a binary classifier. A true positive (TP) is a positive example correctly identified; a true negative (TN) is a negative example correctly rejected; a false positive (FP) is a negative example incorrectly labelled positive; and a false negative (FN) is a positive example missed by the model.

2.1 Accuracy

Accuracy is the proportion of all predictions that are correct:

The denominator is the total number of evaluated examples. Accuracy is appropriate when classes and error costs are reasonably balanced. It can be misleading for rare-event detection. If only 1% of transactions are fraudulent, a classifier that always predicts “not fraud” achieves 99% accuracy but detects no fraud.

2.2 Precision and recall

Precision measures the reliability of positive predictions:

Recall, also called sensitivity or true-positive rate, measures the proportion of actual positives detected:

Precision is important when false alarms are costly, such as automatically blocking legitimate accounts. Recall is important when missing a positive case is costly, such as failing to identify a dangerous medical condition. Both formulas require a non-zero denominator.

Specificity measures the proportion of actual negatives correctly rejected:

2.3 F1 score

The F1 score is the harmonic mean of precision and recall:

It is defined when precision plus recall is non-zero. The harmonic mean is strongly reduced when either component is low. F1 is useful when positive-class performance matters and precision and recall have similar importance. It does not include true negatives and therefore is not a complete description of performance.

Weighted or macro-averaged scores are often used for multiclass tasks. Macro-F1 calculates F1 separately for each class and averages the class scores, giving each class equal importance. Weighted-F1 weights each class by its number of examples. Micro-averaging aggregates decisions before calculating the metric and is influenced strongly by common classes.

Worked example

Suppose a classifier produces TP = 72, FP = 18, FN = 8, and TN = 102. There are 200 observations. Accuracy is (72 + 102) ÷ 200 = 0.87, or 87%. Precision is 72 ÷ 90 = 0.80. Recall is 72 ÷ 80 = 0.90. Therefore:

F1 = 2 × 0.80 × 0.90 ÷ 1.70 ≈ 0.847.

The model detects most positives, but one in five positive predictions is incorrect. Whether this is acceptable depends on application costs.

3. Thresholds, ROC-AUC, and Precision–Recall Analysis

Many classifiers produce a probability or score rather than a final class. A threshold converts the score into a decision. Lowering the threshold usually increases recall and decreases precision; raising it often has the opposite effect.

The receiver operating characteristic (ROC) curve plots true-positive rate against false-positive rate over all classification thresholds. The false-positive rate is:

The area under the ROC curve, ROC-AUC, measures ranking ability across thresholds. A value near 1 indicates strong separation, 0.5 indicates performance similar to random ranking, and values below 0.5 indicate reversed ranking that may be corrected by reversing the score direction. ROC-AUC does not specify the operating threshold and can appear impressive in highly imbalanced data.

For rare positive classes, the precision–recall curve is often more informative because it focuses on positive predictions. The area under this curve should be compared with an appropriate baseline, which depends on positive prevalence. ROC-AUC and PR-AUC are complementary, not interchangeable.

Threshold selection should reflect business, safety, or operational costs. A model report should state the chosen threshold, validation procedure, and expected consequences of false positives and false negatives.

4. NLP Evaluation Metrics

4.1 Perplexity

Perplexity evaluates a probabilistic language model. For a token sequence of length N with conditional probabilities p(wᵢ | w₁,…,wᵢ₋₁), perplexity is:

Here, log is usually the natural logarithm, N is the number of evaluated tokens, and probabilities must be computed using the same tokenisation and vocabulary convention. Perplexity is the exponential of average negative log-likelihood. Lower perplexity means that the model assigns greater probability to the observed sequence.

Perplexity is useful for comparing language models under identical evaluation conditions. It is not a direct measure of factuality, helpfulness, grammatical quality, or human preference. Scores from different tokenisers or datasets should not be compared casually. For a uniform random model over V tokens, perplexity is V; this interpretation assumes a fixed vocabulary and uniform probabilities.

4.2 BLEU

BLEU was designed primarily for machine translation. It measures modified n-gram precision between a candidate translation and one or more reference translations, combined with a brevity penalty:

Here, pₙ is modified n-gram precision, wₙ are non-negative weights that sum to 1, and N is the maximum n-gram order. The brevity penalty is:

where c is candidate length and r is the effective reference length. BLEU penalises very short outputs and limits repeated matching n-grams through modified precision. Smoothing may be needed for sentence-level BLEU when an n-gram precision is zero.

BLEU is useful for comparing systems on the same benchmark, but it can undervalue valid paraphrases and does not guarantee semantic adequacy. Corpus-level BLEU is generally more stable than averaging sentence-level scores.

4.3 ROUGE

ROUGE is commonly used for summarisation. ROUGE-N measures n-gram overlap, while ROUGE-L uses the longest common subsequence (LCS). For an overlap-based measure, recall, precision, and F-measure can be calculated as:

ROUGE-Recall is often emphasised in summarisation because it measures how much reference content is captured. ROUGE-L can reward preservation of sequence order without requiring exact contiguous phrases. Like BLEU, ROUGE is an automatic similarity measure rather than a complete assessment of truth, coherence, safety, or usefulness. Human evaluation or task-specific checks should supplement it.

5. Qualitative Error Analysis

A single metric hides structure. Error analysis examines individual predictions and groups them into meaningful categories. A practical workflow is:

1. collect correct and incorrect examples;
2. retain inputs, labels, predictions, confidence scores, and relevant metadata;
3. sample errors randomly as well as by confidence and subgroup;
4. assign error categories using a documented coding scheme;
5. count categories and inspect representative examples;
6. connect patterns to possible data, model, or deployment causes;
7. implement a remedy and evaluate again.

Typical classification error categories include ambiguous labels, negation, sarcasm, spelling variation, long-range context, rare terminology, domain shift, entity confusion, and annotation inconsistency. NLP generation errors may include omission, repetition, unsupported claims, contradiction, poor style, incorrect entities, and copying from the prompt.

Confidence is diagnostically useful. High-confidence errors may indicate systematic bias, leakage, spurious correlations, or calibration problems. Low-confidence correct examples may reveal genuinely ambiguous cases. An error taxonomy should distinguish label problems from model problems; changing the model cannot solve a fundamentally inconsistent annotation policy.

6. Interpretation and Explainability

Interpretability asks how a model reaches or supports an output. Common tools include:

• feature or token attribution, which assigns importance scores to input elements;
• attention visualisation, which displays attention weights;
• gradient-based saliency, which uses changes in output with respect to inputs;
• perturbation methods, which mask or alter features and observe output changes;
• surrogate models, which approximate a complex model locally;
• counterfactual explanations, which identify changes that would alter the prediction.

These methods answer different questions. An attribution can suggest which tokens influenced a decision, but it does not necessarily establish a causal explanation. Attention weights should not automatically be interpreted as explanations. Explanations must be checked for stability: small irrelevant changes should not produce radically different explanations. They should also be evaluated for faithfulness, meaning that removing highly important features should meaningfully affect the prediction, and for plausibility to domain experts.

Interpretation is especially important in high-impact applications. Explanations should not expose sensitive information unnecessarily, imply certainty that the model does not possess, or encourage users to override professional judgement without evidence. A model card or evaluation report should document intended use, limitations, data sources, metrics, subgroup results, and known failure modes.

7. Robustness, Fairness, and Reliability Testing

A model that performs well on an independent test set may fail after deployment. Robustness testing deliberately changes conditions while preserving the intended task. Useful tests include spelling errors, punctuation changes, paraphrases, dialect variation, noise, missing fields, longer inputs, adversarial prompts, domain changes, and out-of-distribution examples.

A paired perturbation test compares predictions before and after a controlled change. If the change should not alter meaning, a stable model should usually preserve its prediction. For a classification task, one may report the proportion of pairs with unchanged predictions and separately inspect harmful changes.

Fairness analysis should report performance across relevant groups, subject to lawful and ethical data use. Compare accuracy, recall, precision, false-positive rate, and false-negative rate by group where appropriate. These measures need not be equal simultaneously, especially when group prevalences or score distributions differ. Fairness is a sociotechnical issue: metric parity alone does not determine whether a system is just or appropriate.

Calibration concerns whether predicted probabilities correspond to observed frequencies. A group of cases assigned probability 0.8 should be positive approximately 80% of the time if the model is well calibrated in that range. Calibration plots and proper scoring rules such as log loss can supplement discrimination metrics. High ROC-AUC does not imply good calibration.

8. A Practical Evaluation Protocol

A professional evaluation plan can follow this sequence:

1. Define the task, deployment population, decision threshold, and costs of errors.
2. Establish a leakage-resistant data split that reflects deployment.
3. Select metrics linked to the task, class balance, and stakeholder risks.
4. Fit models and preprocessing only on training data.
5. Use validation data for tuning and threshold selection.
6. Freeze the system before accessing the test set.
7. Report primary and secondary metrics with sample sizes and uncertainty.
8. Conduct subgroup, calibration, robustness, and qualitative analyses.
9. Compare against simple baselines, such as majority class, keyword rules, or a smaller model.
10. Document limitations, monitoring signals, and conditions for retraining.

No metric should be reported without its evaluation context. For example, “F1 = 0.84” is incomplete unless the positive class, averaging method, threshold, dataset, and confidence procedure are specified.

9. Illustrative Case Study: Customer-Support Intent Classification

Background

A fictional telecommunications company develops an NLP classifier that routes customer messages to billing, technical support, cancellation, or account-security teams. Security-related messages are rare but high risk.

Problem

An initial random split reports 96% accuracy. However, support agents observe that messages from new customers and messages containing spelling errors are often routed incorrectly. The team suspects that repeated templates from the same customers appear in both training and test data.

Analysis

The team creates a customer-level split so that messages from one customer occur in only one partition. It also creates a time-based test set containing messages from the most recent month. Performance falls to 89% accuracy. Security recall is 0.71, while its precision is 0.93. The high precision shows that security alerts are usually legitimate, but recall indicates that 29% of security messages are missed.

Qualitative analysis identifies three major error types: indirect security language, code-switched messages, and confusion between cancellation and billing. A robustness test with realistic spelling errors reduces macro-F1 substantially, while a paraphrase test reveals that the model relies too heavily on a small set of template phrases.

Approach and solution

The team removes duplicated templates, adds representative historical data, introduces character-level features for spelling variation, and uses class-weighted training. The threshold for security routing is lowered using the validation set because missed security cases are more costly than additional reviews. Human agents review low-confidence security predictions.

Outcome

On the untouched time-based test set, security recall increases to 0.88, security precision decreases to 0.86, and macro-F1 improves. The company accepts the precision reduction because false alarms receive human review. Monitoring is established for monthly recall, subgroup performance, distribution drift, and the proportion of low-confidence cases.

Learning points

The original accuracy was inflated by leakage and class imbalance. A realistic split, class-specific metrics, qualitative analysis, threshold selection, and operational safeguards produced a more useful evaluation. The best model was not selected by one score; it was selected by matching measured behaviour to deployment risk.

10. Limitations and Contemporary Practice

Automatic metrics are approximations. BLEU and ROUGE may reward lexical overlap without measuring meaning; perplexity may favour fluent but false text; accuracy may hide minority-class failure; and explanations may be unstable or misleading. Human evaluation remains important for open-ended generation, but human ratings require clear criteria, representative sampling, trained evaluators, and checks for inter-rater agreement.

Modern evaluation increasingly combines benchmark tests with task-based assessment, adversarial evaluation, red teaming, retrieval-grounded factuality checks, calibration analysis, and continuous post-deployment monitoring. Large language models require additional checks for hallucination, prompt sensitivity, instruction following, privacy leakage, toxicity, and over-reliance on demographic or cultural stereotypes.

Evaluation data must be governed carefully. Personal data should be minimised and protected; sensitive attributes used for fairness analysis require lawful handling; test examples and prompts may contain confidential information; and models should not be evaluated or deployed beyond their documented scope without review. Responsible evaluation is therefore both a technical and organisational responsibility.

Chapter Summary

Reliable deep-learning and NLP evaluation begins with a leakage-resistant train-validation-test design that reflects deployment conditions. Classification metrics answer different questions: accuracy measures overall correctness, precision measures positive-prediction reliability, recall measures detection, F1 balances precision and recall, and ROC-AUC measures ranking across thresholds. Perplexity evaluates probabilistic language modelling, while BLEU and ROUGE measure particular forms of reference overlap and must not be treated as complete measures of language quality.

Qualitative error analysis reveals patterns hidden by aggregate scores. Explainability techniques can support diagnosis, but explanations require faithfulness and stability checks. Robustness, calibration, subgroup analysis, and monitoring are essential because test performance does not guarantee safe deployment. A professional evaluation report connects metrics and observed errors to operational costs, stakeholder needs, ethical responsibilities, and model limitations.

Key Terms

Train set: Data used to estimate model parameters.

Validation set: Data used for model selection and tuning.

Test set: Held-out data used for final evaluation.

Data leakage: Unintended use of information unavailable at prediction time.

True positive: A positive case correctly identified.

False positive: A negative case incorrectly identified as positive.

Precision: The proportion of positive predictions that are correct.

Recall: The proportion of actual positives detected.

F1 score: The harmonic mean of precision and recall.

Specificity: The proportion of actual negatives correctly rejected.

ROC-AUC: Area under the receiver operating characteristic curve.

Perplexity: Exponential average negative log-likelihood for a language model.

BLEU: An n-gram precision-based metric commonly used for translation.

ROUGE: A reference-overlap metric commonly used for summarisation.

Calibration: Agreement between predicted probabilities and observed frequencies.

Robustness: Ability to maintain suitable behaviour under relevant changes.

Error analysis: Systematic examination and categorisation of incorrect outputs.

Data shift: Change between development data and deployment conditions.

Explainability: Methods used to describe or investigate model behaviour.

Macro-averaging: Averaging class metrics while giving each class equal weight.

Review Questions

Short Answer Questions

1. Why must the test set be kept separate from model-selection decisions?
2. Distinguish precision from recall.
3. Why can accuracy be misleading for an imbalanced dataset?
4. What does perplexity measure in a language model?
5. State one limitation of BLEU or ROUGE.

Long Answer Questions

1. Explain how to design a leakage-resistant train-validation-test procedure for an NLP classifier.
2. Compare accuracy, precision, recall, F1, ROC-AUC, and PR analysis, including suitable use cases.
3. Explain the assumptions, interpretation, and limitations of perplexity, BLEU, and ROUGE.
4. Describe a systematic qualitative error-analysis workflow and illustrate possible NLP error categories.
5. Discuss why explainability, calibration, robustness, fairness, and monitoring should supplement aggregate test metrics.

Analytical / Application Questions

1. A fraud model has TP = 45, FP = 15, FN = 5, and TN = 935. Calculate accuracy, precision, recall, and F1. Which metric would you prioritise if missed fraud is much more costly than investigation workload?
2. A summarisation system obtains high ROUGE but frequently invents dates and names. Design an evaluation plan that would identify and reduce this failure.
3. A classifier performs well on random test data but poorly on messages received six months later. Diagnose likely causes and propose a revised evaluation and monitoring strategy.

Multiple Choice Questions with Answers

Question 1

Which dataset should normally be used to select a classification threshold?

A. Training set only

B. Validation set

C. Final test set repeatedly

D. An unrelated unlabelled set

Correct Answer: B. Validation set

Question 2

A model has TP = 80 and FP = 20. What is its precision?

A. 0.20

B. 0.50

C. 0.80

D. 0.90

Correct Answer: C. 0.80

Question 3

Which metric is most directly concerned with the proportion of actual positive cases detected?

A. Precision

B. Recall

C. Specificity

D. Accuracy

Correct Answer: B. Recall

Question 4

What is a major limitation of accuracy on highly imbalanced data?

A. It cannot be calculated for binary tasks

B. It ignores all true negatives

C. It may be high when the minority class is never detected

D. It requires probability calibration

Correct Answer: C. It may be high when the minority class is never detected

Question 5

What does ROC-AUC primarily measure?

A. Ranking separation across thresholds

B. Exact sequence overlap

C. Average generated-sequence length

D. Calibration at one threshold

Correct Answer: A. Ranking separation across thresholds

Question 6

Lower perplexity generally means that a language model:

A. Assigns higher probability to the observed sequences

B. Has higher classification recall

C. Produces more reference n-gram matches

D. Uses a larger vocabulary

Correct Answer: A. Assigns higher probability to the observed sequences

Question 7

BLEU primarily combines:

A. Token-level recall only

B. Modified n-gram precision and a brevity penalty

C. Human ratings and calibration error

D. Classification precision and recall

Correct Answer: B. Modified n-gram precision and a brevity penalty

Question 8

Why should attention visualisations not automatically be treated as causal explanations?

A. Attention is never computed in neural networks

B. Attention weights do not necessarily prove which features caused the output

C. Attention can only be used for images

D. Attention always produces random values

Correct Answer: B. Attention weights do not necessarily prove which features caused the output

Question 9

Which practice best tests robustness to harmless wording changes?

A. Removing the validation set

B. Using only the majority-class baseline

C. Comparing predictions before and after controlled paraphrases

D. Increasing the test-set leakage

Correct Answer: C. Comparing predictions before and after controlled paraphrases

Question 10

A customer-level split is especially important when:

A. All observations are independent by design

B. The same customer may contribute related or repeated messages

C. The task has no labels

D. Only image data are available

Correct Answer: B. The same customer may contribute related or repeated messages