Evaluation, Experimentation and AI Analytics

Introduction and context

A natural language processing (NLP) or generative-AI system is not successful merely because it produces fluent text. A production system must be useful for a defined task, factually dependable, safe, fair, sufficiently fast, affordable, and valuable to its stakeholders. Evaluation is therefore an engineering, scientific, and governance activity rather than a final inspection step.

Evaluation answers questions such as:

• Does the system perform the intended task accurately?
• Does it remain reliable when prompts, users, topics, or data distributions change?
• Are its responses relevant, grounded, and appropriately detailed?
• Does it produce harmful, biased, or confidential content?
• Can it meet latency and cost requirements at realistic traffic levels?
• Does improved model quality produce a measurable organisational benefit?

This chapter presents a layered framework combining intrinsic metrics, task-based evaluation, human judgment, controlled experimentation, robustness testing, and AI analytics. It also explains how to design benchmarks that support responsible decisions rather than impressive but misleading scores.

Learning objectives

By the end of this chapter, a learner should be able to:

1. Distinguish intrinsic, task-based, human, operational, and business evaluation.
2. Select metrics appropriate to generation, classification, retrieval, and conversational tasks.
3. Design reliable test sets, baselines, ablations, and controlled experiments.
4. Assess factuality, relevance, robustness, bias, latency, cost, and safety.
5. Interpret confidence intervals, error analyses, and significance results responsibly.
6. Construct an evaluation dashboard and connect model outcomes to business impact.
7. Recognise methodological, ethical, and governance limitations in AI measurement.

1. The evaluation problem: one system, several meanings of quality

Quality is multidimensional. A chatbot may have high linguistic fluency but hallucinate product policies. A classifier may achieve high accuracy because the majority class dominates, while failing the minority class that matters most. A retrieval-augmented generation (RAG) system may cite relevant documents but still draw an unsupported conclusion. Evaluation must therefore begin with a specification of the task, users, risks, and constraints.

A useful evaluation statement identifies:

• Unit of analysis: message, document, conversation, user session, or business case.
• Target behaviour: answer, classify, summarise, retrieve, translate, extract, or recommend.
• Reference standard: expert label, verified answer, policy rule, source document, or user outcome.
• Failure severity: inconvenience, financial loss, discrimination, privacy breach, or physical risk.
• Operating conditions: language, domain, prompt format, traffic, tool availability, and time period.
• Decision threshold: the minimum acceptable performance and maximum acceptable risk.

A metric is not inherently good or bad. It is appropriate only relative to a decision. For example, recall may be prioritised in safety incident detection, whereas precision may be more important when every flagged case triggers an expensive human investigation.

A layered evaluation model

Evaluation can be organised into five layers:

1. Intrinsic evaluation measures properties of outputs or representations against references, such as perplexity, BLEU, ROUGE, or embedding similarity.
2. Task-based evaluation measures performance on the intended application, such as intent-classification F1, answer exact match, retrieval recall, or successful task completion.
3. Human evaluation examines qualities that automated metrics cannot fully capture, including usefulness, coherence, factual support, and appropriateness.
4. Operational evaluation measures latency, throughput, availability, token usage, error rates, and resource consumption.
5. Business and societal evaluation examines cost savings, conversion, resolution time, accessibility, fairness, safety, and compliance.

These layers should be interpreted together. A model should not be promoted because it wins one metric while violating a safety or service-level requirement.

2. Preparing an evaluation programme

2.1 Define the evaluation population

A test set should represent the conditions in which the system will operate, not merely the data that are easy to collect. Stratify examples by language, user expertise, topic, difficulty, document length, ambiguity, and risk. For a customer-support assistant, separate billing, cancellation, complaints, accessibility, and regulated-product queries.

Separate development, validation, and final test data. The final test set must remain isolated from prompt tuning, model selection, and repeated manual inspection. Otherwise, the test score becomes partly a measure of memorisation or optimisation against the benchmark.

Data contamination is especially important for foundation models. A public benchmark may have appeared in training data, in model-development data, or in online prompt repositories. Private, newly authored, or temporally held-out tests can reduce this risk.

2.2 Establish baselines

A baseline makes improvement interpretable. Suitable baselines include:

• majority-class or random classifiers;
• keyword or regular-expression systems;
• an existing production model;
• a smaller or cheaper language model;
• retrieval without generation;
• the same model without a proposed feature.

An experiment should report absolute performance and change from baseline. A small numerical improvement may not justify a large increase in cost or latency.

2.3 Define an evaluation matrix

An evaluation matrix maps requirements to evidence.

No single score should conceal a critical failure. In practice, teams often use a quality gate: all essential constraints must be satisfied, after which trade-offs among desirable dimensions can be considered.

3. Intrinsic and task-based metrics

3.1 Classification metrics

For binary classification, define true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN). Accuracy is:

Precision and recall are:

Precision answers: among predicted positives, how many are correct? Recall answers: among actual positives, how many were detected? These formulas assume the denominators are non-zero.

The F1 score is the harmonic mean of precision and recall:

It is appropriate when both precision and recall matter and neither should be traded away completely. For imbalanced multiclass data, report macro-F1, which gives each class equal weight, and micro-F1, which aggregates decisions before calculating the score. Weighted-F1 weights each class by its support. The choice should reflect the application rather than be selected after seeing the results.

A confusion matrix is often more informative than accuracy because it shows which classes are confused. For probabilistic classifiers, calibration is also important: a group of predictions assigned probability 0.8 should be correct approximately 80% of the time if the probabilities are well calibrated. Calibration can be examined using reliability diagrams or measures such as expected calibration error, but calibration estimates depend on binning and sample size.

3.2 Information retrieval metrics

For a query, precision at k is the proportion of the first k retrieved items that are relevant. Recall at k is the proportion of all relevant items found in the first k. Mean reciprocal rank (MRR) focuses on the first relevant result:

Here, Q is the set of queries and rank_q is the rank of the first relevant result for query q. MRR is useful when users usually need one answer. Normalised discounted cumulative gain (nDCG) supports graded relevance and gives greater weight to high-ranked results; it is useful when several results may be useful with different degrees of relevance.

Retrieval evaluation should distinguish retrieval recall from answer quality. A generator cannot reliably cite a fact that retrieval failed to supply. Conversely, retrieved material may be relevant but the generated response may misinterpret it.

3.3 Text generation metrics

For machine translation, BLEU compares n-gram overlap, usually with a brevity penalty. ROUGE measures overlap oriented toward recall and is widely used for summarisation. METEOR incorporates alignment and some linguistic variation. These metrics are inexpensive and reproducible, but they may penalise valid paraphrases and reward copied wording that is irrelevant or factually wrong.

Perplexity measures how well a language model predicts a sequence. For a token sequence of length N with conditional probabilities p(w_t | w_<t),

Lower perplexity indicates better predictive likelihood on the evaluated distribution. It is meaningful for comparing models under comparable tokenisation and data conditions, but it does not directly measure helpfulness, factuality, instruction following, or user satisfaction.

Semantic similarity metrics based on embeddings can recognise paraphrases better than surface overlap. However, high semantic similarity does not guarantee logical entailment. A statement can be topically similar to a source while reversing its meaning. For this reason, factuality evaluation should use entailment checks, source-grounded verification, expert review, or structured claims analysis.

3.4 Generative-AI quality dimensions

A response may be evaluated along separate dimensions:

• Factuality: whether claims are true and supported by reliable evidence.
• Faithfulness or groundedness: whether the response follows the supplied source rather than inventing unsupported content.
• Relevance: whether it addresses the user’s actual request.
• Completeness: whether important requested elements are present.
• Coherence: whether the response is logically organised and understandable.
• Instruction following: whether constraints such as format, audience, and length are respected.
• Safety: whether it avoids prohibited, dangerous, discriminatory, or privacy-violating content.

A practical factuality protocol decomposes an answer into atomic claims, verifies each against authoritative sources, and records whether the claim is supported, contradicted, unverifiable, or irrelevant. Claim-level analysis is more diagnostic than asking whether an entire paragraph is “good.”

Automated judge models can assist at scale, but they should not be treated as unquestionable authorities. They may prefer longer answers, reproduce their own biases, be sensitive to position and wording, or agree with fluent but false statements. Use calibrated rubrics, blinded comparisons, judge agreement checks, and human audits.

4. Human evaluation

Human evaluation is indispensable when quality depends on context, pragmatics, or social impact. A rubric should define criteria, scale points, examples, and an “insufficient evidence” or “not applicable” option where appropriate.

Common designs include:

• Absolute rating: each response receives a score on a defined scale.
• Pairwise preference: a judge chooses between two anonymised systems, optionally allowing a tie.
• Best–worst scaling: judges identify the strongest and weakest item among a set.
• Error annotation: judges label specific failures such as unsupported claim, missed instruction, toxicity, or privacy leakage.

Randomise response order and blind judges to system identity. Include duplicate items to estimate within-rater consistency. Use more than one judge for high-impact decisions. Agreement can be summarised with measures such as Cohen’s κ for two categorical raters or Krippendorff’s α for multiple raters and missing labels, but agreement is not the same as validity: judges may consistently apply a flawed rubric.

A strong rubric distinguishes observable criteria. For example, “factual support” might be scored as 0 = materially false, 1 = partly supported or contains a minor unsupported claim, and 2 = all material claims supported. Training and calibration sessions improve consistency. Sampling should deliberately include difficult, ambiguous, and high-risk cases rather than only average cases.

5. Robustness, bias, and safety evaluation

5.1 Robustness

Robustness is the ability to maintain acceptable behaviour under legitimate variation or deliberate stress. Test paraphrases, spelling errors, dialects, long contexts, multilingual input, prompt injection, irrelevant distractors, missing information, and distribution shifts over time.

Useful analyses include:

• performance by subgroup and task slice;
• degradation from clean to perturbed input;
• sensitivity to prompt wording and decoding parameters;
• out-of-domain and temporal hold-out performance;
• recovery after tool failure or unavailable evidence;
• abstention or escalation when confidence is inadequate.

A robustness test must specify the perturbation and its plausibility. Random character noise may be mathematically easy but operationally irrelevant, while a realistic policy update or adversarial instruction may be more valuable.

5.2 Bias and fairness

Bias evaluation begins by defining the protected or vulnerable groups, the harm of concern, and the decision context. Compare error rates, selection rates, response quality, toxicity, refusal behaviour, and stereotype associations across relevant groups. Group comparisons require adequate sample sizes and careful control of confounding variables.

For a binary decision, demographic parity compares selection rates across groups, while equal opportunity compares true-positive rates. These criteria can conflict, especially when base rates differ. No fairness metric resolves the underlying social or legal question automatically. Qualitative review and consultation with affected communities are necessary.

Avoid using demographic labels as simplistic proxies for identity. Protect annotator and user privacy, document the provenance and limitations of subgroup data, and do not report unstable percentages from tiny samples. A model may show similar aggregate accuracy across groups while producing different kinds of harmful errors.

5.3 Safety and red teaming

Safety evaluation combines policy-based test cases, historical incidents, adversarial red teaming, and monitoring of real interactions. Tests should cover self-harm, violence, hate, sexual content, fraud, cyber abuse, privacy, regulated advice, copyright-sensitive requests, and prompt injection where applicable.

Record both unsafe generations and unsafe refusals. Excessive refusal can deny legitimate assistance, while inadequate refusal can create harm. A mature system provides safe alternatives, uncertainty statements, human escalation, and audit logs.

6. Operational and economic analytics

Quality that arrives too slowly or at unsustainable cost is not production quality.

6.1 Latency and reliability

Measure latency from the user’s perspective and report percentiles rather than only an average. p50 describes the median experience; p95 and p99 expose the slow tail that affects a substantial minority of requests. Separate time to first token (TTFT), generation time, tool-call time, queueing time, and total completion time for streaming systems.

Throughput is often measured in requests per second or tokens per second. Load tests should vary concurrency and payload length, because long prompts and generated outputs can change both latency and cost. Also record timeout rate, tool failure rate, availability, and fallback frequency.

6.2 Cost measurement

If input and output prices differ, an approximate model-serving cost for n requests is:

Here, r_in and r_out are average input and output tokens per request, p_in and p_out are prices per token in consistent currency units, and C_fixed represents fixed costs such as infrastructure, monitoring, evaluation, and engineering. This is a planning equation: actual invoices may include caching, batching, minimum charges, or hardware utilisation effects.

Cost per successful task is often more informative than cost per request:

N_success is the number of requests meeting the predefined success criterion. A cheaper model with lower success may have a higher cost per successful task.

6.3 Business impact and return on investment

Business metrics must be linked to a credible causal design. Examples include first-contact resolution, average handling time, qualified leads, conversion, defect rate, employee productivity, complaint rate, and customer retention. A simple ROI expression is:

Benefits and costs must be measured over the same period and expressed in comparable monetary units. If benefits are uncertain, report a range or scenario analysis rather than a falsely precise point estimate.

Offline benchmark gains do not prove business value. A response-quality improvement may reduce handling time, but it may also increase review work if staff must verify every answer. Instrument the full workflow.

7. Experimentation and causal inference

7.1 Controlled experiments

An A/B test randomly assigns eligible units—users, sessions, tickets, or documents—to control and treatment conditions. Randomisation helps balance observed and unobserved factors, but interference can occur when users share information or when a model changes system load for everyone.

Define the primary metric before running the experiment. Predefine eligibility, sample-size logic, stopping rules, guardrail metrics, and the analysis window. Guardrails may include complaint rate, unsafe-output rate, latency, and cost. Do not repeatedly inspect results and stop at a convenient moment without accounting for sequential testing.

For a binary outcome, an estimated treatment effect may be expressed as the difference in rates:

For a continuous metric, use the difference in means or a relative change when a ratio is meaningful. Report uncertainty, such as a confidence interval, and explain practical significance. Statistical significance alone does not establish that the effect matters operationally.

7.2 Offline experiments and ablations

When live testing is unsafe or expensive, use offline replay, shadow deployment, or human-in-the-loop simulation. An ablation removes one component—such as retrieval, reranking, safety filtering, or conversation memory—to estimate its contribution. Keep data, prompts, decoding settings, and evaluation procedures fixed except for the factor being studied.

Factorial designs can study interactions, for example whether retrieval quality and model size reinforce one another. Beware of tuning a system on the same evaluation set used to claim improvement. Maintain a locked final test set and, where possible, replicate important findings on a second dataset or time period.

7.3 Statistical interpretation

Use paired analysis when the same prompts are evaluated by two systems; pairing reduces noise and is often more powerful than treating observations as independent. Bootstrap resampling can estimate uncertainty for complex metrics such as ROUGE, nDCG, or human preference rates. Clustered data require appropriate clustering because many observations from one user are not independent.

A confidence interval describes uncertainty under specified assumptions; it does not mean that a fixed percentage of future results will fall inside the interval. Report sample size, missing data, exclusions, metric definitions, and multiple-comparison procedures.

8. Error analysis and AI analytics architecture

A score identifies that a problem exists; error analysis helps explain why. Build an error taxonomy such as:

• retrieval miss;
• source conflict or outdated source;
• unsupported generation;
• instruction failure;
• ambiguity or missing context;
• language or dialect issue;
• unsafe response or over-refusal;
• tool or infrastructure failure.

Sample errors proportionally and by severity. Review both common errors and rare catastrophic errors. Link each category to an intervention, such as better chunking, updated knowledge sources, prompt revision, fine-tuning, routing, policy changes, or human escalation.

An AI analytics system commonly contains:

1. Event capture: prompt and response identifiers, model version, retrieval IDs, latency, token counts, safety decisions, and user feedback.
2. Privacy controls: redaction, access control, retention limits, and separation of sensitive content.
3. Metric computation: batch evaluation, online monitoring, slice analysis, and drift detection.
4. Experiment registry: hypotheses, versions, cohorts, dates, and analysis plans.
5. Decision dashboards: quality, safety, operational, and business indicators with thresholds and owners.
6. Incident workflow: alert, triage, rollback, remediation, and post-incident learning.

Do not log sensitive prompts indiscriminately. Use pseudonymisation, encryption, least-privilege access, retention schedules, and documented legal bases for processing. Metrics should be traceable to data and code versions so that results are reproducible.

9. Illustrative case: a retrieval-augmented support assistant

Background

A fictional telecommunications provider deploys an assistant to answer broadband-support questions using a knowledge base of service policies and troubleshooting articles. The assistant must reduce agent workload without giving customers incorrect advice.

Problem

An offline benchmark shows that the new model produces more fluent answers than the production system. However, early review finds occasional claims about fees and contract eligibility that are not supported by current policy documents. Long responses also increase latency and token cost.

Analysis

The team defines success as an answer that is relevant, supported by the current knowledge base, complete for the question, and safe to send without mandatory agent correction. It creates a time-split test set containing recent policy updates, difficult wording, multilingual queries, and high-risk billing questions.

The evaluation matrix includes retrieval recall at 5, claim-level groundedness, expert pairwise preference, unsafe or materially incorrect response rate, p95 latency, cost per successful answer, and agent correction time. The existing system is the baseline. Human reviewers are blinded to system identity and use a two-point groundedness rubric.

Approach and solution

An ablation compares: (A) generation alone, (B) retrieval plus generation, (C) retrieval plus generation with mandatory citation and abstention when evidence is absent. A controlled pilot randomly assigns support sessions to the baseline or C, while monitoring resolution rate, escalation rate, complaints, latency, and cost.

Error analysis shows that many failures are retrieval misses caused by outdated document aliases. The provider improves document versioning and retrieval metadata, adds a policy-date filter, shortens the answer template, and routes high-risk billing questions to human agents. The assistant is instructed to state when the knowledge base does not contain enough evidence rather than guessing.

Outcome

The pilot is judged successful only if resolution improves without exceeding safety, latency, or cost guardrails. Suppose the treatment increases first-contact resolution but also raises escalation slightly. The decision then depends on the monetary value of resolved contacts, the cost of escalations, and the severity of residual errors—not on language quality alone. The provider continues monitoring because policy changes and user behaviour can invalidate a previously successful evaluation.

Learning points

1. Fluent generation is not evidence of factual support.
2. Retrieval and generation require separate diagnostics.
3. A time-split test exposes outdated-knowledge failures.
4. Abstention and escalation can be signs of quality, not failure.
5. Business decisions require live, causal evidence and guardrails.

10. Best practices and limitations

Use a portfolio of metrics, maintain a locked test set, publish baselines, stratify results, and inspect severe errors. Prefer task-specific and source-grounded tests over a single generic benchmark. Report uncertainty and practical effect size. Re-evaluate after model, prompt, data, tool, policy, or traffic changes.

Important limitations remain. Human judgments are expensive and culturally situated. Automated metrics inherit reference-set errors. Benchmark optimisation can produce Goodhart’s-law effects: once a metric becomes a target, systems may improve the score without improving the underlying objective. Fairness criteria may conflict. Online experiments can expose users to risk. Business outcomes can be affected by seasonality, training, pricing, and other simultaneous changes.

Governance should therefore require documented evaluation plans, model and data versioning, risk-based approval, auditability, incident reporting, accessibility review, and a clear owner for every metric. Evaluation is continuous assurance, not a certificate that remains valid forever.

Contemporary developments

Current practice is moving toward evaluation harnesses that combine deterministic tests, model-based judges, retrieval verification, red teaming, and production telemetry. Synthetic data can expand coverage, but synthetic examples require validation because they may omit real language variation. LLM-as-a-judge systems can reduce annotation cost, especially for pairwise comparisons, but should be calibrated against human judgments and tested for positional, verbosity, and identity bias.

Other important developments include agent evaluation over multi-step trajectories, multimodal groundedness, evaluation under tool failure, energy and carbon reporting, privacy-preserving analytics, and continuous monitoring for distribution drift. These developments reinforce a central principle: evaluation must match the actual system, environment, and consequences of use.

Chapter Summary

Evaluation of NLP and generative-AI systems is multidimensional. Intrinsic metrics such as perplexity and n-gram overlap are useful diagnostic tools, but task-based metrics, human judgment, operational measures, and business outcomes are needed for deployment decisions. Classification requires careful interpretation of precision, recall, F1, calibration, and class imbalance; retrieval requires measures such as recall at k, MRR, and nDCG; generation requires explicit assessment of factuality, groundedness, relevance, completeness, and safety.

Reliable evaluation depends on representative and uncontaminated data, clear baselines, locked test sets, stratified error analysis, and reproducible procedures. Robustness testing examines perturbations and distribution shifts, while fairness evaluation compares relevant outcomes across groups without reducing social questions to one statistic. Latency, cost, reliability, and cost per successful task connect technical quality to production feasibility.

Controlled experiments and causal reasoning are necessary for claims about user or business impact. A/B tests, offline replay, ablations, confidence intervals, and guardrails support disciplined decisions. AI analytics should capture versioned events, protect privacy, expose failures, and connect dashboards to incident response. No metric is sufficient alone; responsible evaluation combines quantitative evidence, human expertise, domain knowledge, and continuous governance.

Key Terms

A/B test: A randomised comparison between control and treatment conditions.

Ablation: An experiment that removes one system component to estimate its contribution.

Calibration: Agreement between predicted probabilities and observed frequencies.

Cost per successful task: Total operating cost divided by the number of outputs meeting the success criterion.

F1 score: Harmonic mean of precision and recall.

Factuality: Truth of claims in an output.

Groundedness: Support for an output in supplied or authoritative evidence.

Human evaluation: Assessment of system outputs by people using a defined rubric.

Intrinsic metric: A metric comparing an output or representation with a reference or internal property.

Latency: Time taken to produce a response or complete a system operation.

Macro-F1: Average of class-level F1 scores, weighting classes equally.

MRR: Mean reciprocal rank, emphasising the rank of the first relevant retrieval.

nDCG: Normalised discounted cumulative gain for ranked results with graded relevance.

Perplexity: Exponentiated average negative log-likelihood of a sequence.

Precision: Proportion of predicted positive cases that are correct.

Recall: Proportion of actual positive cases that are detected.

Robustness: Ability to maintain acceptable performance under variation or stress.

Safety guardrail: A constraint or monitored metric designed to prevent unacceptable harm.

Task-based evaluation: Measurement of performance on the intended application task.

Test contamination: Unwanted overlap between evaluation data and training or tuning data.

Review Questions

Short Answer Questions

1. Why is a single fluency score insufficient for evaluating a generative-AI assistant?
2. Distinguish precision from recall in classification.
3. What is the purpose of a locked final test set?
4. Name two operational metrics relevant to production language systems.
5. Why should claim-level factuality analysis be used instead of only whole-response judgments?

Long Answer Questions

1. Explain the five layers of evaluation and show how they complement one another.
2. Compare BLEU, ROUGE, perplexity, semantic similarity, and human evaluation, including their principal limitations.
3. Describe a rigorous procedure for evaluating robustness, bias, and safety in a customer-facing assistant.
4. Explain how an A/B experiment for an AI system should be designed, analysed, and governed.
5. Discuss how an AI analytics architecture can support reproducibility, privacy, monitoring, and incident response.

Analytical / Application Questions

1. A classifier has TP = 80, TN = 850, FP = 50, and FN = 20. Calculate accuracy, precision, recall, and F1, and explain which metric would change most if false negatives became more costly.
2. Design an evaluation matrix for a RAG assistant used by hospital administrators. Include factuality, retrieval, safety, latency, cost, human review, and business or workflow outcomes.
3. A new model improves offline relevance but increases p95 latency and cost per successful task. Propose an experiment and decision rule that determines whether it should replace the current production model.

Multiple Choice Questions with Answers

Question 1

Which metric is the harmonic mean of precision and recall?

A. Accuracy

B. F1 score

C. Mean reciprocal rank

D. Perplexity

Correct Answer: B. F1 score

Question 2

What does recall measure in binary classification?

A. The proportion of predicted positives that are correct

B. The proportion of all cases that are classified correctly

C. The proportion of actual positive cases that are detected

D. The proportion of negative cases that are rejected

Correct Answer: C. The proportion of actual positive cases that are detected

Question 3

Why is a time-split test set useful for a policy-question answering system?

A. It guarantees zero annotation cost

B. It exposes failures caused by outdated knowledge and temporal shift

C. It eliminates the need for human evaluation

D. It always increases model accuracy

Correct Answer: B. It exposes failures caused by outdated knowledge and temporal shift

Question 4

A retrieval system returns the first relevant document at rank 4 for one query. What is that query's reciprocal-rank contribution to MRR?

A. 0.25

B. 0.40

C. 4.00

D. 0.04

Correct Answer: A. 0.25

Question 5

Which metric is most directly concerned with the slow tail of user response times?

A. p99 latency

B. Macro-F1

C. BLEU

D. Selection rate

Correct Answer: A. p99 latency

Question 6

What is the main purpose of an ablation study?

A. To hide system components from evaluators

B. To estimate the contribution of a component by removing it

C. To replace a final test set with training data

D. To guarantee statistical significance

Correct Answer: B. To estimate the contribution of a component by removing it

Question 7

If a response is topically similar to a source but contradicts its conclusion, what limitation is illustrated?

A. Precision is always zero

B. Semantic similarity does not guarantee factual entailment

C. Latency has been incorrectly measured

D. Randomisation has failed

Correct Answer: B. Semantic similarity does not guarantee factual entailment

Question 8

Which practice best reduces identity and position bias in human or model-based judging?

A. Reveal the system identity to judges

B. Use only the longest answer

C. Blind system identity and randomise response order

D. Evaluate only examples selected by the model

Correct Answer: C. Blind system identity and randomise response order

Question 9

A system costs 0.02 currency units per request and completes 1,000 requests, with no fixed cost. What is the total variable cost?

A. 0.20

B. 2.00

C. 20.00

D. 200.00

Correct Answer: B. 2.00

Question 10

Which statement best describes a business-impact claim from an offline benchmark improvement?

A. It proves that revenue will increase

B. It is causal evidence without a live experiment

C. It is suggestive evidence that requires workflow or controlled outcome measurement

D. It makes safety evaluation unnecessary

Correct Answer: C. It is suggestive evidence that requires workflow or controlled outcome measurement