Generative AI, Question Answering, and RAG

Introduction and Context

Information retrieval traditionally returns documents, passages, or ranked records in response to a query. Generative artificial intelligence changes the user experience by producing a natural-language answer rather than merely displaying retrieved material. This creates a powerful but consequential combination: a retrieval system supplies evidence, while a large language model (LLM) interprets and verbalizes that evidence.

Retrieval-augmented generation (RAG) is the principal architecture for combining these capabilities. Instead of requiring a language model to contain all relevant knowledge in its parameters, RAG retrieves current, domain-specific, or access-controlled information at query time and places it in the model’s context. A well-designed RAG system therefore addresses both information retrieval and language generation. It must retrieve relevant evidence, assemble it within context limits, produce an answer faithful to that evidence, and communicate uncertainty when evidence is insufficient.

RAG is used in enterprise search, legal research, biomedical question answering, technical support, education, financial analysis, and conversational assistants. Its value is not simply that it produces fluent text. Its value lies in connecting an answer to identifiable evidence and allowing knowledge to be updated without retraining the entire model.

Learning Objectives

By the end of this chapter, a learner should be able to:

1. Explain the architecture and rationale of retrieval-augmented generation.
2. Design document-ingestion, chunking, indexing, retrieval, reranking, and context-assembly pipelines.
3. Distinguish sparse, dense, hybrid, and agentic retrieval strategies.
4. Engineer question-answering systems with citations, grounding, and abstention.
5. Evaluate retrieval quality, answer quality, faithfulness, latency, cost, and safety.
6. Diagnose hallucinations and apply practical mitigation techniques.
7. Assess governance, privacy, security, and professional implications of RAG deployments.

1. Question Answering as an Information Retrieval Task

A question-answering (QA) system maps a question to an answer. In an extractive system, the answer is a span copied from a source. In an abstractive system, the answer is generated, possibly by synthesizing multiple sources. Open-domain QA searches a broad collection, whereas closed-domain QA operates over a defined corpus such as company policies or medical guidelines.

A modern QA pipeline commonly contains five stages:

1. Question analysis: identify intent, entities, time constraints, language, and required answer type.
2. Candidate retrieval: find potentially relevant documents or passages.
3. Evidence selection: rank, filter, and sometimes diversify candidates.
4. Answer generation: formulate a response from selected evidence.
5. Verification and presentation: attach citations, check support, and communicate uncertainty.

The distinction between retrieval and generation is fundamental. A fluent answer is not necessarily a correct answer, and a relevant document is not necessarily sufficient evidence for every claim in the answer.

Important terminology

• Corpus: the collection of indexed documents.
• Passage or chunk: a retrievable segment of a document.
• Query: the user’s information need expressed as text or another modality.
• Retriever: a component that selects candidate passages.
• Reranker: a model that reorders candidates using a more expensive relevance calculation.
• Generator: generally an LLM that produces the final response.
• Grounding: constraining or relating an answer to supplied evidence.
• Citation: a reference that allows a reader to locate supporting evidence.
• Hallucination: an unsupported, fabricated, or contradicted claim.
• Abstention: declining to answer, or explicitly stating that evidence is insufficient.

2. RAG Architecture and Information Flow

A RAG system has an offline indexing path and an online query path. During indexing, documents are acquired, cleaned, segmented, enriched with metadata, embedded or term-indexed, and stored. During a query, the system interprets the question, retrieves candidates, reranks them, assembles context, generates an answer, and applies validation and presentation policies.

A useful abstract formulation is:

where Q is the user question, C is the assembled evidence context, I is the system instruction or policy, G is the generator, and A is the answer. The notation is conceptual rather than a claim that generation is deterministic. In practice, the output may also depend on decoding parameters, conversation history, and model state.

The context should not be viewed as an undifferentiated text dump. It is a deliberately selected evidence set. Each passage should ideally retain document identity, section title, page or paragraph location, timestamp, access permissions, and other provenance metadata.

A production architecture usually includes:

• document connectors and parsers;
• normalization and deduplication;
• chunking and metadata extraction;
• sparse and/or vector indexes;
• query rewriting and retrieval;
• reranking and context compression;
• an LLM gateway;
• citation and claim-verification services;
• observability, evaluation, access control, and audit logging.

3. Document Preparation and Chunking

Chunking strongly affects both recall and answer quality. A chunk that is too short may omit definitions, conditions, or exceptions. A chunk that is too long may contain irrelevant material, consume the context window, and reduce the number of independently useful passages.

3.1 Chunking strategies

Fixed-length chunking divides text by a character or token budget, often with overlap. It is simple and robust but may split tables, lists, or arguments at inappropriate points.

Sliding-window chunking uses overlapping segments. If a chunk has length L tokens and adjacent chunks overlap by O tokens, the stride is L − O. Overlap can preserve boundary information, but excessive overlap increases index size and may cause redundant evidence.

Structure-aware chunking respects headings, paragraphs, clauses, list items, tables, and code blocks. It is usually preferable for manuals, regulations, and scholarly documents.

Semantic chunking detects topical shifts and groups text with similar meaning. It can improve coherence but introduces model dependence and computational cost.

Hierarchical chunking stores summaries or parent sections alongside smaller child passages. A system may retrieve a precise child chunk and then expand to its parent section for interpretation.

There is no universally optimal chunk size. The correct choice depends on document structure, query distribution, language, embedding model, context budget, and evaluation results. Chunking should be treated as an experimentally tunable retrieval parameter rather than a fixed preprocessing convention.

3.2 Metadata and special content

Metadata should include source, title, author, publication date, version, jurisdiction, department, security label, and location. Tables require special care: row and column headings must remain associated with values. PDFs may require layout-aware extraction because visual reading order differs from textual extraction. Scanned documents require OCR, whose errors should be recorded and evaluated.

Documents should be versioned. A policy answer based on an obsolete version can be more dangerous than an explicitly incomplete answer. Deletion and permission changes must propagate to all indexes and caches.

4. Retrieval Methods

4.1 Sparse retrieval

Sparse retrieval represents documents using terms. BM25 remains a practical baseline because it rewards term overlap while accounting for document length and term rarity. A common BM25 score is:

Here, f(t,d) is the frequency of term t in document d, |d| is document length, avgdl is average document length, k₁ controls term-frequency saturation, b controls length normalization, and IDF(t) is an inverse-document-frequency weight. Typical parameter values are tuned empirically; they are not universal constants. BM25 is especially useful for names, identifiers, exact phrases, rare terms, and legal or technical terminology.

4.2 Dense retrieval

Dense retrieval maps a query and passage to vectors and ranks them by similarity. Cosine similarity is frequently used:

Here, q and d are embedding vectors, q · d is their dot product, and ||·||₂ is the Euclidean norm. Cosine similarity is applicable when vector direction is intended to represent semantic proximity and zero vectors are excluded. If embeddings are normalized to unit length, cosine similarity equals the dot product.

Dense retrieval can match paraphrases and related concepts even when words differ. However, it may miss exact identifiers, rare product codes, negation, or newly introduced terminology. Embeddings also inherit biases and language limitations from their training data.

4.3 Hybrid retrieval

Hybrid retrieval combines sparse and dense candidates. A simple normalized score is:

The component scores must be placed on comparable scales before combination. The parameter α is selected using validation data. An alternative, often more robust when score scales differ, is reciprocal rank fusion:

Here, rank_r(d) is the rank of document d in retrieval system r, and k is a positive smoothing constant. RRF uses ranks rather than raw scores and is therefore useful for combining heterogeneous systems.

4.4 Query transformation

A query may be rewritten, expanded with synonyms, decomposed into subquestions, or translated into a structured filter. Rewriting can improve recall but can also change the user’s intent. The original query should therefore be retained, and transformations should be logged. For multi-hop questions, decomposition can retrieve evidence for separate subproblems before synthesis.

5. Reranking and Context Assembly

Initial retrieval prioritizes speed and recall. A reranker examines each query–passage pair more deeply, often using a cross-encoder or LLM-based relevance model. A cross-encoder jointly processes the query and passage, allowing detailed interaction, but it is more expensive than independently embedded vectors.

Reranking should consider more than topical similarity:

• direct answerability;
• authority and source quality;
• recency and version;
• coverage of required subquestions;
• contradiction and redundancy;
• user authorization;
• geographic or regulatory scope.

Context assembly is an optimization problem under a token budget. A good assembly strategy may select a small set of passages that maximizes relevance and coverage while minimizing duplication. It should preserve headings, source labels, and passage boundaries. Chronological or logical ordering may be preferable to ranking order when the question concerns a procedure or historical development.

Context compression can remove irrelevant sentences or produce passage summaries. Compression improves capacity but risks deleting qualifications. For high-stakes use, retaining original excerpts alongside compressed representations is safer.

A prompt should clearly distinguish instructions from evidence. Retrieved text must be treated as untrusted data because a document may contain prompt-injection instructions such as “ignore previous directions.” The model should be told to use evidence for answering, not to execute instructions found inside evidence.

6. Grounded Generation, Citations, and Abstention

Grounded generation requires the answer to be supported by the retrieved context. A practical answer format separates claims, citations, and uncertainty. For example, each factual paragraph may cite a source passage, while interpretation is explicitly labeled as interpretation.

Citations are meaningful only when they are accurate and useful. A citation should identify the source and a stable location such as page, section, paragraph, URL, or document version. “Citation presence” alone is insufficient: a system can attach citations that do not support its claims.

A useful verification process is:

1. Split the draft into atomic claims.
2. Retrieve or identify evidence for each claim.
3. classify each claim as supported, contradicted, or unsupported.
4. revise, remove, or qualify unsupported claims.
5. attach precise citations.

Abstention is a capability, not a failure. The system should abstain or ask a clarifying question when evidence is absent, sources conflict materially, the question is ambiguous, or the user lacks authorization. Confidence should not be inferred solely from the LLM’s wording. It should be calibrated against evidence and evaluation data.

7. Hallucination and Its Mitigation

Hallucination has multiple causes: missing evidence, retrieval failure, ambiguous prompts, model priors, incorrect synthesis, overconfident decoding, contaminated context, and pressure to answer every question. It may be an invented fact, an incorrect number, an unsupported causal claim, a distorted quotation, or a citation mismatch.

Mitigation should operate across the pipeline:

• improve parsing, chunking, and metadata;
• use hybrid retrieval and query expansion where appropriate;
• rerank for answerability and authority;
• provide explicit evidence boundaries;
• require citations for factual claims;
• use structured outputs for claims and evidence identifiers;
• apply entailment or claim-verification models;
• lower generation freedom for factual tasks;
• permit abstention and clarification;
• use human review for high-impact decisions;
• monitor failures continuously.

No single technique eliminates hallucination. Retrieval can itself be wrong, and a model can misinterpret correct evidence. Grounding is therefore a system property requiring testing at retrieval, generation, and interface levels.

8. Evaluation Framework

Evaluation should distinguish retrieval quality from answer quality. Let the relevant set for query q be Rel(q), and let Retrieved_k(q) be the first k retrieved items. Recall at k is:

It measures how much of the known relevant set was retrieved. Precision at k is:

Recall is important when missing evidence is costly; precision is important when context capacity and reviewer attention are limited. These metrics require judged relevance and are sensitive to the definition of relevance.

Mean reciprocal rank (MRR) emphasizes the first relevant result:

Here, Q is the evaluation-query set and rank_q is the rank of the first relevant result for query q. MRR is appropriate when one highly relevant result is usually sufficient, but it is less suitable for multi-document evidence requirements.

Answer evaluation should include:

• Correctness: does the answer reach the right conclusion?
• Faithfulness: are claims supported by the supplied evidence?
• Completeness: are important aspects omitted?
• Citation precision: does each citation support its claim?
• Citation recall: are important claims cited?
• Relevance and clarity: does the response address the question appropriately?
• Safety: does it avoid harmful or unauthorized advice?

Automatic LLM-as-judge evaluation can scale assessment but may share biases with the generator. It should be calibrated against expert judgments and supplemented by exact tests, human review, adversarial tests, and error analysis.

Operational metrics include latency, cost per query, cache-hit rate, throughput, index freshness, access-control violations, and abstention rate. Evaluation datasets should contain ordinary, ambiguous, adversarial, temporal, multilingual, and unanswerable questions. Test sets must be versioned to prevent unnoticed benchmark leakage.

9. Agentic Retrieval

Agentic retrieval allows an LLM or controller to choose tools and retrieval steps dynamically. An agent may decide to search a vector index, query a relational database, inspect a document, call an API, or decompose a question. This is useful for multi-hop questions and heterogeneous enterprise data.

A controlled agentic loop is:

1. classify the task and identify constraints;
2. select an authorized tool;
3. issue a bounded query;
4. inspect results and record provenance;
5. decide whether evidence is sufficient;
6. repeat only within a step and time budget;
7. synthesize and verify the answer.

Agentic retrieval should not be confused with unrestricted autonomy. Tool permissions, query budgets, timeouts, sandboxing, and human approval are essential. The agent must distinguish retrieved content from executable instructions. It should also expose an audit trail showing which tools were used and which evidence supported the result.

10. Illustrative Case: University Policy Assistant

Background. A university wants an assistant that answers questions about assessment, admissions, academic integrity, and accessibility policies. The corpus contains web pages, PDFs, committee decisions, and policy versions.

Problem. A basic chatbot gives fluent answers but occasionally cites obsolete policies, merges undergraduate and postgraduate rules, and fails to recognize when a question depends on a student’s programme or jurisdiction.

Analysis. The principal risks are document versioning, access and scope metadata, long policy sections, and exceptions embedded in footnotes. Exact terms such as policy codes require sparse retrieval, while paraphrased student questions benefit from dense retrieval.

Approach. The university creates structure-aware chunks with headings, clauses, page locations, effective dates, programme level, and jurisdiction. BM25 and dense retrieval are combined, followed by a cross-encoder reranker. A metadata filter removes expired policies before ranking. Context assembly selects the governing clause and relevant exception, preserving the policy version. The generator must produce an answer, citations, an effective-date note, and an abstention when no current policy applies. A claim checker flags unsupported statements for review.

Outcome. In evaluation, retrieval recall improves because policy codes and paraphrases are both covered. Citation audits reveal that some answers cite a whole policy even when only one clause supports the claim, leading to more precise clause-level citations. The assistant is deployed first for low-risk navigation, while appeals and disciplinary decisions remain human-led.

Learning points. Domain metadata is as important as embeddings; currentness must be explicitly modelled; citations require support checking; and deployment scope should reflect consequences, not merely model accuracy.

11. Governance, Security, and Professional Practice

RAG systems may expose confidential documents through retrieval, prompt context, logs, or citations. Access control must be enforced before context assembly, not merely in the user interface. Tenants, departments, and security labels require isolation. Logs should minimize sensitive content and follow retention policies.

Prompt injection can occur in user queries or retrieved documents. Defenses include treating retrieved text as data, tool allow-lists, output validation, instruction hierarchy, content scanning, and adversarial testing. Data poisoning is another risk: an attacker may insert misleading documents that are highly retrievable. Provenance, source approval, version control, and anomaly detection help reduce this risk.

Privacy obligations may govern personal data, health information, educational records, and employment information. Organizations should establish lawful purpose, retention limits, deletion propagation, access review, incident response, and human appeal mechanisms. Copyright and licensing also matter when documents are indexed, transformed, or reproduced in answers.

Professional users should know the system’s scope and limitations. An answer should not silently replace legal, medical, financial, or academic decision-making authority. Documentation should describe corpus coverage, update frequency, known failure modes, evaluation results, and escalation routes.

12. Best-Practice Design Pattern

A defensible RAG implementation follows these principles:

1. Define answerability, risk, and source-authority requirements before selecting a model.
2. Build a representative evaluation set, including unanswerable and adversarial questions.
3. Preserve document structure, provenance, permissions, and versions.
4. Establish sparse retrieval as a baseline and test dense or hybrid alternatives.
5. Tune chunking and retrieval using task-specific measurements.
6. Rerank candidates for relevance, authority, currentness, and coverage.
7. Assemble compact context with visible source boundaries.
8. Instruct the generator to distinguish evidence, inference, and uncertainty.
9. Verify claims and citations rather than measuring citation count alone.
10. Provide abstention, clarification, and human escalation pathways.
11. Monitor latency, cost, drift, security, and real-world errors.
12. Re-evaluate whenever the corpus, embedding model, generator, or policy changes.

Chapter Summary

RAG integrates information retrieval with generative language modelling. Its quality depends on the entire pipeline: document preparation, structure-aware chunking, metadata, sparse and dense retrieval, reranking, context assembly, grounded generation, citations, verification, and abstention. BM25 supports lexical matching, embeddings support semantic matching, and hybrid retrieval often provides stronger coverage. Evaluation must separate retrieval metrics such as Recall@k and MRR from answer correctness, faithfulness, completeness, citation quality, safety, latency, and cost. Agentic retrieval can support multi-step and heterogeneous tasks, but it requires bounded tools, permissions, provenance, and auditability. Hallucination mitigation is not achieved by prompting alone; it requires evidence-aware design, verification, governance, and human oversight.

Key Terms

Corpus: The collection of documents available for retrieval.

Chunk: A retrievable segment of a document.

Sparse retrieval: Term-based retrieval using representations containing mostly zero values.

Dense retrieval: Semantic retrieval using continuous embedding vectors.

Hybrid retrieval: Combination of sparse and dense retrieval.

BM25: A probabilistic term-ranking function using term frequency and length normalization.

Embedding: A numerical vector representing semantic or other properties of content.

Reranker: A component that reorders initially retrieved candidates.

Context assembly: Selection and organization of evidence supplied to a generator.

Grounding: Relating an answer to explicit supporting evidence.

Citation: A reference identifying the source of a claim.

Faithfulness: The degree to which an answer is supported by supplied evidence.

Hallucination: An unsupported, fabricated, or contradicted generated claim.

Abstention: Declining to answer when evidence or authorization is insufficient.

Query rewriting: Transforming a user question to improve retrieval.

Agentic retrieval: Dynamic, tool-using retrieval controlled by an agent or planner.

Prompt injection: An attempt to manipulate system behaviour through instructions in input or retrieved content.

Provenance: Information about the origin, version, and location of evidence.

RAG: Retrieval-augmented generation, in which retrieved information supports generated responses.

Review Questions

Short Answer Questions

1. What is the difference between a retriever and a generator in a RAG system?
2. Why can structure-aware chunking be preferable to fixed-length chunking?
3. State one strength and one limitation of dense retrieval.
4. What does answer faithfulness measure?
5. Why is abstention important in high-stakes question answering?

Long Answer Questions

1. Explain the complete offline and online architecture of a production RAG system.
2. Compare sparse, dense, and hybrid retrieval, including situations in which each is useful.
3. Discuss how chunking, reranking, and context assembly jointly affect answer quality.
4. Explain a systematic framework for detecting and mitigating hallucinations.
5. Design an evaluation plan for an enterprise policy assistant, including retrieval, answer, safety, and operational metrics.

Analytical / Application Questions

1. A technical-support corpus contains product codes, synonyms, diagrams, and frequently updated release notes. Design a retrieval and versioning strategy and justify each component.
2. A RAG assistant retrieves a correct paragraph but generates an answer that adds an unsupported exception. Diagnose the failure and propose pipeline, prompt, verification, and governance controls.
3. Design a bounded agentic workflow for answering a question that requires combining a policy document, a student record, and a database query. Address authorization, provenance, failure handling, and human escalation.

Multiple Choice Questions with Answers

Question 1

Which component primarily selects candidate passages for a user query?

A. Retriever

B. Generator

C. Citation renderer

D. Tokenizer

Correct Answer: A. Retriever

Question 2

What is a major strength of BM25 retrieval?

A. It generates fluent summaries

B. It handles exact terms and rare identifiers well

C. It guarantees semantic understanding

D. It removes the need for metadata

Correct Answer: B. It handles exact terms and rare identifiers well

Question 3

If normalized embedding vectors are used, cosine similarity is equal to which quantity?

A. The Euclidean distance

B. The vector dimension

C. The dot product

D. The document length

Correct Answer: C. The dot product

Question 4

What is the principal purpose of reranking?

A. To deeply reorder an initial candidate set

B. To delete all metadata

C. To replace document parsing

D. To increase the model’s vocabulary

Correct Answer: A. To deeply reorder an initial candidate set

Question 5

Which metric measures the fraction of known relevant items found in the first k results?

A. Precision@k

B. Recall@k

C. MRR

D. Latency@k

Correct Answer: B. Recall@k

Question 6

Why should retrieved documents be treated as untrusted data?

A. They always contain incorrect facts

B. They may contain prompt-injection instructions

C. They cannot be embedded

D. They prevent sparse retrieval

Correct Answer: B. They may contain prompt-injection instructions

Question 7

Which practice most directly improves citation reliability?

A. Increasing response length

B. Attaching one citation to every answer regardless of support

C. Checking each atomic claim against evidence

D. Removing document version information

Correct Answer: C. Checking each atomic claim against evidence

Question 8

What does abstention mean in a question-answering system?

A. Returning the longest possible answer

B. Declining or qualifying an answer when evidence is insufficient

C. Using only dense retrieval

D. Reranking every document in the corpus

Correct Answer: B. Declining or qualifying an answer when evidence is insufficient

Question 9

What is a key additional control for agentic retrieval?

A. Unlimited tool access

B. Bounded tools, permissions, and audit trails

C. Removing provenance records

D. Ignoring intermediate results

Correct Answer: B. Bounded tools, permissions, and audit trails

Question 10

Which issue is especially important when answering from changing policies?

A. Using obsolete document versions

B. Increasing embedding dimension automatically

C. Avoiding all metadata

D. Disabling access control

Correct Answer: A. Using obsolete document versions