10. Evaluation, Experimentation, and LLM Analytics

Introduction and Context

A large language model (LLM) can produce fluent, persuasive, and apparently knowledgeable text while still being inaccurate, unsafe, expensive, or unsuitable for a particular user. Consequently, evaluation is not a final inspection performed after deployment. It is a continuous engineering, research, governance, and product-management activity. Evaluation determines whether a model solves the intended problem, under which conditions it fails, and whether its benefits justify its operational and social costs.

For conventional software, correctness is often assessed against deterministic outputs. LLM systems are different. They generate multiple acceptable answers, depend on prompts and context, and may fail unpredictably when inputs are rephrased, extended, or adversarially designed. A useful evaluation programme must therefore combine automated metrics, curated benchmarks, expert and user judgments, safety testing, statistical experimentation, and production analytics.

This chapter develops a framework for evaluating LLM applications rather than treating a single benchmark score as a complete account of quality. It considers accuracy, relevance, robustness, calibration, hallucination, toxicity, fairness, latency, cost, and user outcomes. It also examines LLM-as-judge methods, red teaming, controlled experiments, observability, and organizational decision-making. These ideas connect model development with the broader lifecycle of prompt design, retrieval-augmented generation, agents, and deployment.

Learning Objectives

By the end of this chapter, students should be able to:

1. Distinguish model, system, safety, operational, and outcome-level evaluation.

2. Select suitable metrics for accuracy, relevance, robustness, calibration, hallucination, toxicity, fairness, latency, and cost.

3. Design benchmark datasets, human-evaluation protocols, and LLM-as-judge procedures.

4. Explain red teaming, failure analysis, controlled experiments, and online monitoring.

5. Interpret evaluation results without confusing correlation, benchmark performance, and real-world impact.

6. Recommend governance and improvement actions for an LLM product or organizational deployment.

Evaluation as a Multidimensional Construct

The quality of an LLM application is multidimensional. A chatbot may answer questions accurately but expose confidential information. A summarization system may be factually faithful but too slow for a call-centre workflow. A coding assistant may improve productivity for experienced developers while increasing insecure code among novices. Evaluation must therefore begin with an explicit statement of purpose, users, risks, and acceptable trade-offs.

A practical evaluation stack has five layers. Capability evaluation measures what the underlying model can do, such as reasoning, translation, coding, or summarization. System evaluation assesses the complete application, including prompts, retrieval, tools, memory, filters, and interfaces. Safety evaluation examines harmful content, privacy leakage, bias, security weaknesses, and misuse. Operational evaluation measures latency, availability, throughput, token consumption, and cost. Outcome evaluation asks whether users achieve better decisions, faster work, higher satisfaction, or improved business or public-service results.

The distinction between a model and a system is essential. A retrieval-augmented application may fail because the retriever selected irrelevant documents, because the prompt did not require citations, or because the generator ignored the evidence. Reporting only the base model’s benchmark score conceals these system-level causes.

Important Terminology

An evaluation instance is a single input, context, expected response, and associated metadata. A benchmark is a curated collection of instances and a scoring procedure. A gold answer is a reference answer or set of acceptable properties, not necessarily a single exact string. A metric maps observed outputs to numerical or categorical evidence. A slice is a meaningful subgroup of data, such as language, domain, difficulty, user type, or demographic context. Validity concerns whether an evaluation measures the intended construct; reliability concerns whether it produces stable results. Test leakage occurs when evaluation information enters training or prompt construction, inflating apparent performance.

Evaluation results should include uncertainty. A score calculated on a small sample may change substantially when new examples are added. Confidence intervals, bootstrap estimates, repeated runs, and significance tests can help, although statistical significance does not automatically imply practical importance.

Designing a Sound Evaluation Programme

Evaluation begins with a specification. The specification should identify the task, users, success criteria, failure severity, data sources, deployment environment, and decision thresholds. For example, a legal-document assistant might define success as answering only from approved documents, citing supporting passages, refusing unsupported legal conclusions, and returning within three seconds for 95 percent of requests.

A representative test set should reflect the distribution of expected use while deliberately oversampling rare but severe failures. It commonly contains:

• routine production-like examples;
• difficult, ambiguous, or incomplete inputs;
• adversarial and security-oriented prompts;
• multilingual and accessibility-related cases;
• examples from historically underserved groups;
• regression cases corresponding to earlier failures.

The data should be separated into development, validation, and held-out test sets. If human annotators or prompt designers repeatedly inspect the test set and modify the system accordingly, the test set becomes a de facto development set. Versioning datasets, prompts, models, retrieval indexes, evaluation scripts, and annotation guidelines is therefore as important as versioning source code.

A useful error taxonomy classifies failures by cause and consequence. Causes may include retrieval failure, instruction conflict, unsupported inference, tool error, ambiguity, bias, prompt injection, or formatting failure. Consequences may be minor inconvenience, financial loss, privacy exposure, physical danger, or unlawful discrimination. This taxonomy supports targeted remediation instead of the vague conclusion that the model is “not accurate enough.”

Measuring Accuracy, Relevance, and Faithfulness

Accuracy is task-dependent. For classification, accuracy, precision, recall, F1 score, and area under the receiver operating characteristic curve may be suitable. Precision measures the proportion of positive predictions that are correct, whereas recall measures the proportion of actual positives that are found. In high-risk screening, recall may be prioritized; in automated customer notifications, precision may be more important.

For open-ended generation, exact string matching is usually inadequate because multiple responses can be correct. Token-level overlap measures such as ROUGE or BLEU can provide limited evidence for summarization or translation, but they may penalize valid paraphrases and reward superficial lexical similarity. Semantic similarity metrics compare meanings using embeddings, yet they can miss subtle factual contradictions.

Human or model-assisted criteria are often more informative. Evaluators may score correctness, completeness, relevance, clarity, style, citation quality, and instruction adherence on a defined scale. Rubrics must describe observable standards. “Good answer” is too vague; “identifies all three documented causes and does not introduce an unsupported cause” is more reliable.

In retrieval-augmented generation, retrieval precision asks how much retrieved material is relevant, while retrieval recall asks whether necessary evidence was retrieved. Context precision and context recall evaluate the usefulness and completeness of supplied context. Answer faithfulness measures whether claims are supported by that context. Answer relevance measures whether the response addresses the user’s question. These dimensions should be separated: a response may be relevant but unsupported, or well-supported but irrelevant.

Citation evaluation should verify both citation correctness and citation completeness. A citation is correct when it genuinely supports the associated claim. Citation completeness concerns whether important claims that require evidence are cited. Merely counting citation markers can therefore produce a misleadingly positive result.

Hallucination and Factual Reliability

A hallucination is a generated claim that is unsupported, false, or inconsistent with the available evidence. The term covers several phenomena: invented facts, fabricated sources, incorrect calculations, temporal errors, and statements that exceed the supplied context. Hallucination is not identical to uncertainty. A model can express uncertainty appropriately, while a confident answer can still be false.

Evaluation should define the reference universe. In a closed-book question-answering system, the relevant universe may be a document collection. In an open-domain assistant, factual verification may require external sources and time-sensitive checking. Useful measures include the proportion of responses containing unsupported claims, claim-level factual precision, contradiction rate, and abstention quality. Abstention quality evaluates whether the system refuses or qualifies answers when evidence is insufficient, without refusing questions it could answer.

Claim-level analysis is more informative than judging an entire response as simply correct or incorrect. A response can contain nine supported statements and one dangerous fabrication. Weighting claims by severity may be appropriate in regulated or high-risk applications. Automated fact-checking can accelerate screening, but human review remains important for ambiguous, specialized, or consequential claims.

Robustness, Calibration, and Reliability Under Change

Robustness is the ability to maintain acceptable performance when conditions vary. Evaluation should include paraphrases, spelling errors, dialects, long contexts, irrelevant material, conflicting instructions, prompt injection, distribution shifts, and tool or retrieval failures. A robustness report should show both average performance and worst-slice performance; a high mean can conceal catastrophic failure in a small but important group.

Calibration concerns the relationship between a system’s confidence and the probability that its answer is correct. A calibrated system that assigns 0.8 confidence to a set of answers should be correct approximately 80 percent of the time in comparable conditions. Common measures include the Brier score, expected calibration error, reliability diagrams, and selective risk. Selective risk evaluates accuracy when the system answers only cases above a confidence threshold. A useful assistant may improve safety by abstaining on uncertain cases, but confidence expressed in natural language is not automatically a calibrated probability.

Calibration can be assessed through explicit probability requests, verbal confidence categories, agreement across sampling runs, or external uncertainty estimators. These methods have limitations. Models may produce overconfident language because of stylistic conventions, and self-reported confidence may not correspond to epistemic uncertainty. Consequently, confidence should be validated against observed outcomes rather than accepted at face value.

Safety, Toxicity, Fairness, and Security Evaluation

Toxicity evaluation tests whether outputs contain abusive, hateful, sexually explicit, violent, or otherwise harmful language. Classifiers and toxicity lexicons are useful screening instruments but can misclassify reclaimed language, quotations, educational discussion, or minority dialects. Evaluation should include contextual human review and should distinguish generation of harmful content from transformation or detection of such content.

Fairness is not represented by one universal metric. Depending on the task, evaluators may examine error-rate parity, demographic parity, equal opportunity, subgroup calibration, representation, stereotype association, or quality differences across languages and identities. These criteria can conflict. A fair evaluation therefore makes the relevant protected attributes, decision context, harm model, and trade-offs explicit. Testing only explicit demographic terms is insufficient; indirect references, names, dialects, and intersectional identities should also be considered.

Red teaming is a structured attempt to discover failures by adopting the perspective of an attacker, vulnerable user, or determined adversary. Red-team prompts may target jailbreaks, prompt injection, data exfiltration, harmful instructions, misinformation, impersonation, model manipulation, and tool misuse. Effective red teaming is iterative: testers create attacks, record conditions and impact, develop mitigations, and retest after changes. A successful refusal on a known prompt does not establish general security.

Safety evaluations should consider both false negatives and false positives. Overblocking can prevent legitimate educational, medical, or political discussion and can disproportionately affect particular users. Governance should define escalation paths, incident severity, logging rules, privacy protections, and human review for high-risk outputs.

Human Evaluation and LLM-as-Judge

Human evaluation remains important when quality is subjective, context-dependent, or consequential. A sound protocol specifies the task, rubric, sampling method, evaluator qualifications, blinding procedure, number of judgments, tie handling, and agreement analysis. Pairwise comparison—asking which of two responses is better—can be easier than assigning absolute scores, although it may encourage superficial preferences. Inter-rater agreement statistics reveal ambiguity in the rubric but should not be interpreted as proof of correctness.

Evaluators should be shown only the information necessary for the judgment and, where feasible, should not know which system produced an answer. Position randomization reduces order bias. Calibration exercises and adjudication procedures help resolve disagreements. Payment, workload, emotional burden, and exposure to harmful material are ethical considerations in human annotation.

An LLM-as-judge uses a language model to score or compare outputs. It is attractive because it is fast, scalable, and capable of assessing open-ended qualities. However, judges may exhibit verbosity bias, position bias, preference for their own style, sensitivity to formatting, weak performance on specialized facts, and correlated errors with the evaluated model. A judge should therefore receive a precise rubric, relevant reference evidence, and structured output requirements. Pairwise judgments should be randomized, and a sample should be audited by experts.

LLM-as-judge is best treated as an instrument requiring validation, not as an unquestionable authority. Agreement with human judgments, stability across judge models and prompts, sensitivity to known quality differences, and resistance to superficial changes should be examined. For safety-critical decisions, automated judging should support rather than replace accountable human review.

Experimentation and Causal Learning

Evaluation describes performance; experimentation helps determine what causes improvement. A controlled experiment changes one factor while holding other conditions as constant as possible. In LLM systems, factors may include model version, system prompt, retrieval strategy, temperature, context length, tool policy, interface design, or safety filter.

Offline A/B evaluation compares system variants on a fixed test set. Online A/B testing randomly assigns users or sessions to variants and measures predefined outcomes. Randomization supports causal inference, but interference, learning effects, repeated users, and unequal traffic can complicate interpretation. Guardrail metrics—such as harmful-output rate, latency, complaint rate, and cost—should accompany primary success metrics.

A/B tests require a hypothesis, target population, unit of randomization, sample-size rationale, duration, primary and secondary metrics, stopping rule, and analysis plan. Multiple unplanned comparisons increase the chance of false discoveries. Sequential monitoring and early stopping should be statistically justified. In high-risk settings, staged rollouts, canary deployments, and human approval are safer than immediate full-scale release.

Product outcomes may include task completion, time to resolution, successful first response, retention, conversion, learning gain, or decision quality. User satisfaction is useful but insufficient: users may prefer confident answers that are less accurate. Conversely, a system may improve objective outcomes while initially receiving lower satisfaction because it appropriately refuses risky requests. Mixed-method evaluation combines telemetry with interviews, usability studies, and qualitative error analysis.

LLM Analytics in Production

Production analytics converts operational traces into evidence about system behavior. Important signals include request volume, prompt and completion tokens, model and prompt versions, retrieval results, tool calls, latency percentiles, retries, refusals, user feedback, escalation rates, and safety incidents. Logs must be designed with privacy in mind; sensitive prompts should be minimized, access-controlled, redacted, or securely retained according to policy.

Latency should be reported by percentiles rather than only averages. The 50th percentile describes typical performance, whereas the 95th or 99th percentile exposes slow experiences that affect a substantial minority. Cost analysis should include input and output tokens, embedding and retrieval operations, tool calls, storage, human review, retries, and engineering or infrastructure overhead. A cheaper model may increase cost if it produces longer answers, requires more retries, or causes more human escalation.

Dashboards should distinguish leading indicators, such as retrieval failure or rising refusal rates, from lagging indicators, such as complaints or business loss. Monitoring must support segmentation by model version, geography, language, customer tier, task type, and risk category. Drift detection compares current input and outcome distributions with a reference period. Changes may reflect user behavior, new documents, seasonal demand, policy changes, or attacks rather than model degradation alone.

A mature analytics workflow closes the loop: detect an anomaly, inspect representative cases, classify the failure, assign ownership, implement a change, rerun offline tests, conduct a controlled rollout, and monitor post-release effects. Without this loop, dashboards become descriptive rather than operationally useful.

Illustrative Case Study: Evaluating an Internal Policy Assistant

Background

A fictional organization, Northbridge Services, deploys an assistant that answers employee questions from internal human-resources and information-security policies. The system uses retrieval-augmented generation and is expected to provide citations, avoid inventing policy, and direct sensitive cases to HR specialists.

Situation and Problem

After launch, employees report that answers are fluent but occasionally cite obsolete policies. The product team’s initial evaluation shows 92 percent “helpfulness,” but it contains only 100 development examples and no held-out test set. Average latency is acceptable, yet some complex questions take more than ten seconds. Managers also observe that employees in two regional offices receive more refusal messages because their policy documents use different terminology.

Analysis

The team defines separate objectives: evidence retrieval, answer faithfulness, policy completeness, appropriate abstention, regional parity, latency, and cost. It constructs a versioned dataset of realistic questions, ambiguous cases, obsolete-policy traps, regional variants, and deliberately unanswerable questions. HR experts annotate required policy points and escalation conditions. Security specialists add prompt-injection and confidential-information tests.

Error analysis reveals three distinct causes. First, the retriever ranks an obsolete document above the current policy. Second, the generator treats a related policy as direct evidence. Third, the refusal classifier is more sensitive to the terminology used in one region. The original helpfulness score had combined these cases and therefore concealed the problem.

Approach

The team introduces document version metadata and filters obsolete policies before generation. The prompt requires claim-level citations and an explicit “insufficient evidence” response when no current source supports an answer. It evaluates retrieval recall, citation correctness, faithfulness, abstention precision, and subgroup performance separately. Two independent HR reviewers assess a stratified sample, while an LLM judge performs larger-scale preliminary scoring; disagreements and high-risk cases receive expert review.

A staged experiment compares the old and revised systems. Primary metrics are policy-faithful answer rate and successful task completion. Guardrails include unsupported-claim rate, regional disparity, 95th-percentile latency, cost per resolved question, and escalation rate. The rollout begins with a small internal group and includes a feedback button that records the task type and failure category rather than unrestricted sensitive text whenever possible.

Outcome

The revised system improves evidence use and reduces unsupported claims, but its abstention rate rises. Qualitative review shows that some abstentions are appropriate, while others result from overly strict document matching. The team therefore improves terminology normalization rather than simply lowering the safety threshold. It also retains the old system’s faster response path for simple questions while routing complex cases through deeper retrieval.

Learning Points

The case demonstrates that evaluation must reflect the full system and its risks. A single helpfulness score was insufficient. Versioned evidence, claim-level analysis, subgroup slices, operational metrics, controlled rollout, and human review produced a more defensible decision. It also shows that safety and user experience should be optimized together: indiscriminate refusal is not a successful solution to hallucination.

Limitations, Risks, and Common Mistakes

Several evaluation practices produce false confidence. Benchmark overfitting occurs when teams repeatedly optimize for a public test set. Metric gaming occurs when a system learns to produce answers that satisfy a scorer without improving the underlying task. Small or unrepresentative datasets conceal rare failures. Aggregate averages hide subgroup disparities and high-severity outliers. Automated judges may reward verbosity or stylistic similarity. Human raters may disagree because the rubric is ambiguous, not because the model is genuinely borderline.

Another mistake is evaluating the model in isolation when the product includes retrieval, tools, memory, and user-interface constraints. Testing only benign prompts ignores adversarial behavior. Testing only average latency ignores tail latency. Tracking token cost without measuring successful outcomes encourages false economies. Finally, collecting detailed logs without a retention and access policy creates privacy and security risks.

Evaluation itself can cause harm if sensitive user data is copied into annotation systems, if red-team prompts expose workers to disturbing content without safeguards, or if demographic labels are inferred irresponsibly. Organizations should apply data minimization, access control, secure storage, informed annotation practices, and documented retention limits.

Governance and Professional Practice

Evaluation results should feed release decisions and accountability structures. A model card or system card can document intended use, data and evaluation scope, known limitations, subgroup performance, safety tests, and prohibited uses. An evaluation report should record the model and prompt versions, dataset provenance, sampling method, metrics, uncertainty, failures, mitigations, and unresolved risks.

High-impact applications may require domain experts, legal and compliance review, impact assessment, incident reporting, audit trails, and a human appeal mechanism. Organizations should define thresholds that trigger additional review rather than presenting a score as an automatic authorization for deployment. Governance is strongest when product, engineering, security, legal, domain, and user representatives share responsibility.

Emerging Directions and Best Practices

Current work is moving toward agent evaluation, where success depends on multi-step planning, tool selection, state management, and recovery from errors. Long-context evaluation examines whether models can locate and correctly use information across large documents. Multimodal evaluation introduces image, audio, and video grounding questions. Continuous evaluation uses automatically collected failure cases, synthetic perturbations, and live feedback while guarding against privacy leakage and feedback loops.

Another important direction is outcome-based evaluation. Rather than asking only whether an answer resembles a reference, researchers measure whether users make better decisions, learn more, solve tasks faster, or avoid harmful actions. This is more meaningful but also more expensive and difficult to attribute solely to the model.

A practical set of best practices is as follows:

• define success and unacceptable failure before testing;
• evaluate the complete application, not only the base model;
• use representative, adversarial, multilingual, and high-severity slices;
• combine automated metrics, expert review, user evidence, and operational telemetry;
• report uncertainty, subgroup results, and tail behavior;
• validate LLM judges against human judgments;
• maintain a regression suite from real failures;
• use controlled rollouts with guardrail metrics;
• protect evaluation data and annotation workers;
• treat evaluation as a continuous feedback and governance process.

Chapter Summary

LLM evaluation is a multidimensional discipline covering capability, system behavior, safety, operations, and user outcomes. Accuracy and relevance must be adapted to the task; open-ended systems require rubrics, evidence checks, and human or validated model-assisted judgment. Hallucination evaluation benefits from claim-level faithfulness and appropriate abstention measures. Robustness testing explores distribution shifts, adversarial prompts, and retrieval or tool failures, while calibration relates confidence to observed correctness. Toxicity, fairness, privacy, and security require contextual and subgroup-sensitive analysis.

Reliable evaluation combines held-out benchmarks, human evaluation, validated LLM-as-judge methods, red teaming, error taxonomies, controlled experiments, and production analytics. A/B tests and staged releases support causal learning, while latency, cost, drift, and user outcomes reveal whether a system is practically valuable. Governance requires transparent documentation, privacy safeguards, domain expertise, incident processes, and explicit thresholds for deployment. The central principle is that no single score can establish the quality or trustworthiness of an LLM application.

Key Terms

Benchmark: A curated evaluation dataset and procedure used to compare systems.

Calibration: The degree to which stated confidence corresponds to observed correctness.

Claim-level faithfulness: The extent to which individual generated claims are supported by available evidence.

Confidence interval: A statistical range expressing uncertainty around an estimated quantity.

Drift: A change in input, output, or outcome distributions over time.

Evaluation slice: A defined subgroup or condition used for disaggregated analysis.

Hallucination: An unsupported, false, or inconsistent generated claim.

Human evaluation: Systematic assessment of outputs by qualified people using a specified protocol.

LLM-as-judge: Use of a language model to score or compare generated outputs.

Metric: A defined numerical or categorical measure of performance.

Operational evaluation: Assessment of latency, availability, throughput, resource use, and cost.

Precision: The proportion of predicted positive cases that are correct.

Recall: The proportion of actual positive cases that are identified.

Red teaming: Adversarial testing intended to discover safety, security, or reliability failures.

Robustness: The ability to maintain acceptable performance under changed or difficult conditions.

Selective risk: Error among cases answered after the system abstains from uncertain cases.

Test leakage: Unintended use of evaluation information during development or tuning.

Toxicity: Harmful, abusive, hateful, violent, or otherwise inappropriate language or content.

Validity: The extent to which an evaluation measures the intended construct.

Review Questions

Short Answer Questions

1. Why is evaluating an LLM application different from evaluating a deterministic software program?

2. Distinguish retrieval precision from answer faithfulness in a retrieval-augmented system.

3. What does calibration mean in the context of LLM confidence?

4. Why should latency be reported using percentiles as well as averages?

5. State two limitations of LLM-as-judge evaluation.

Long Answer Questions

1. Explain a multidimensional evaluation stack for LLM applications, including capability, system, safety, operational, and outcome evaluation.

2. Describe how to construct a reliable benchmark and error taxonomy for an enterprise LLM application.

3. Discuss the methods and limitations of evaluating hallucination, citation quality, and abstention.

4. Explain how human evaluation and LLM-as-judge methods can be combined responsibly.

5. Design a controlled A/B experiment for comparing two versions of an LLM assistant, including primary metrics and guardrails.

Analytical / Application Questions

1. An assistant has high average helpfulness but produces unsupported medical claims for a small language group. Which evaluation slices, metrics, and governance actions should be used before deployment?

2. A new prompt reduces hallucination but doubles refusal rates and increases user complaints. Analyze the trade-off and propose an experimental and monitoring plan.

3. A retrieval system shows high answer relevance but low citation correctness. Diagnose possible causes and recommend technical and evaluation interventions.

22. Multiple Choice Questions with Answers

Question 1

Which evaluation layer examines whether users complete tasks faster or make better decisions?

A. Capability evaluation

B. Outcome evaluation

C. Operational evaluation

D. Syntax evaluation

Correct Answer: B. Outcome evaluation

Question 2

In a retrieval-augmented generation system, answer faithfulness primarily measures whether the answer

A. contains enough words

B. matches the user’s preferred style

C. is supported by the retrieved evidence

D. has low token cost

Correct Answer: C. is supported by the retrieved evidence

Question 3

A calibrated system assigning 0.8 confidence to many predictions should be correct approximately

A. 20 percent of the time

B. 50 percent of the time

C. 80 percent of the time

D. 100 percent of the time

Correct Answer: C. 80 percent of the time

Question 4

Which practice best reduces position bias in pairwise LLM-as-judge evaluation?

A. Always showing the preferred system first

B. Randomizing response order

C. Removing the evaluation rubric

D. Using only longer responses

Correct Answer: B. Randomizing response order

Question 5

Why are latency percentiles important in production analytics?

A. They reveal slow-tail user experiences hidden by averages

B. They eliminate the need to measure cost

C. They measure factual accuracy directly

D. They guarantee model calibration

Correct Answer: A. They reveal slow-tail user experiences hidden by averages

Question 6

Which is an example of test leakage?

A. Holding out a test set

B. Adding real failures to a regression suite

C. Repeatedly tuning the system on a public test set

D. Reporting confidence intervals

Correct Answer: C. Repeatedly tuning the system on a public test set

Question 7

A red-team exercise is primarily intended to

A. maximize response length

B. discover adversarial safety and security failures

C. replace all human evaluation

D. reduce the number of benchmark examples

Correct Answer: B. discover adversarial safety and security failures

Question 8

Which metric is most directly concerned with the proportion of actual positive cases identified?

A. Recall

B. Precision

C. Brier score

D. Latency

Correct Answer: A. Recall

Question 9

In the Northbridge case, obsolete policy documents caused which principal failure?

A. Retrieval and evidence-selection failure

B. Excessive GPU utilization only

C. Position bias in human judging

D. A failure of demographic parity alone

Correct Answer: A. Retrieval and evidence-selection failure

Question 10

Why should an LLM-as-judge be validated against human judgments?

A. Because model judges are guaranteed to be unbiased

B. Because model judges can show style, position, and correlated-model biases

C. Because human judgments never disagree

D. Because automated metrics cannot be computed

Correct Answer: B. Because model judges can show style, position, and correlated-model biases