Information Extraction and Information Retrieval

Introduction

Natural Language Processing (NLP) systems increasingly operate on large collections of unstructured text: news archives, scientific papers, customer messages, legal records, clinical notes, and web pages. Two capabilities are central to making such collections useful. Information extraction (IE) converts text into structured representations such as entities, relations, and events. Information retrieval (IR) finds and ranks documents or passages relevant to a user's information need. Modern question-answering systems combine both capabilities with statistical ranking, neural representation learning, and generative language models.

IE and IR are related but not identical. IE asks, “What structured facts are expressed in this text?” IR asks, “Which text units best satisfy this query?” An enterprise search system may retrieve documents using an inverted index, identify entities and relations within them, and then answer a question using evidence selected from the retrieved passages.

Learning Objectives

By the end of this chapter, you should be able to:

1. Design pipelines for named-entity, relation, and event extraction.
2. Explain sequence labelling, span classification, and document-level extraction.
3. Construct and query an inverted index.
4. Calculate and interpret TF–IDF and BM25 scores.
5. Compare sparse, dense, and hybrid retrieval.
6. Design retrieval-augmented question-answering pipelines.
7. Evaluate extraction, retrieval, and question-answering systems.
8. Identify reliability, privacy, bias, and governance risks.

1. Information Extraction as Text-to-Structure Conversion

Information extraction maps linguistic content to a structured schema. A schema specifies entity types, relation types, event types, attributes, and temporal or provenance information. For example, from “NVIDIA acquired Mellanox for $6.9 billion in 2020,” an IE system might produce:

• Entity: NVIDIA, type ORGANIZATION
• Entity: Mellanox, type ORGANIZATION
• Entity: $6.9 billion, type MONEY
• Entity: 2020, type DATE
• Relation: acquired(NVIDIA, Mellanox)
• Event: acquisition, buyer=NVIDIA, target=Mellanox, price=$6.9 billion, time=2020

A useful extraction record should preserve the source span, document identifier, confidence, and possibly the extraction model version. Provenance is essential when extracted facts are used in regulated or high-impact settings.

1.1 Named-Entity Recognition

Named-entity recognition (NER) identifies spans referring to predefined categories such as PERSON, ORGANIZATION, LOCATION, PRODUCT, DATE, MONEY, or DISEASE. A common representation is BIO tagging:

For “University of Delhi opened a campus in Dubai,” the sequence may be B-ORG I-ORG I-ORG O O O B-LOC.

NER can be implemented using conditional random fields, recurrent models, transformer encoders, or span-based classifiers. Token classification is efficient for conventional schemas, whereas span classification can represent nested entities more naturally. In “Bank of America,” the complete organization is one span; in some domains, “America” may also be annotated as a location. The annotation policy must therefore be defined before training.

Important design decisions include tokenization, treatment of abbreviations, nested entities, discontinuous entities, domain vocabulary, and label consistency. General-purpose models often perform poorly on biomedical, financial, and legal text because entity boundaries and meanings differ from ordinary news data.

1.2 Relation Extraction

Relation extraction identifies semantic links between entities. Relations may be directional, such as employs(person, organization), symmetric, such as located_near(place₁, place₂), or n-ary, such as a transaction involving buyer, seller, asset, amount, and date.

A pipeline may first detect entities and then classify every plausible entity pair. A relation classifier can use the sentence, entity markers, dependency paths, and contextual embeddings. However, pairwise extraction becomes difficult in long documents because arguments may occur in different sentences. Document-level relation extraction therefore uses coreference resolution, discourse context, entity graphs, and cross-sentence attention.

Relation extraction must distinguish assertion from speculation and negation. “The report denied that Company A acquired Company B” should not normally yield an asserted acquisition relation. Systems should record attributes such as polarity, modality, source, and confidence rather than treating every linguistic mention as fact.

1.3 Event Extraction

An event is an occurrence with a trigger and one or more arguments. In “The regulator approved the merger on Monday,” approved is a trigger, the regulator is an approver, the merger is the object, and Monday is the time.

Event extraction commonly proceeds through four stages:

1. Detect event triggers.
2. Classify the event type.
3. Identify argument spans.
4. Assign argument roles and normalize time, quantity, or location.

Events can be represented as graphs rather than isolated triples. This is important for incidents, clinical episodes, financial transactions, and supply-chain disruptions. Event schemas should define optional and mandatory roles, temporal relations, and whether repeated mentions refer to the same real-world occurrence.

1.4 Coreference and Entity Linking

Coreference resolution connects expressions that refer to the same entity: “Microsoft announced its results. The company reported higher revenue.” Entity linking maps a mention to a canonical knowledge-base identifier. Linking “Apple” requires contextual disambiguation among the technology company, the fruit, and other organizations.

A production IE pipeline commonly follows this flow:

Figure: A production information-extraction pipeline.

2. Extraction Design and Evaluation

A robust pipeline begins with a task specification: entity definitions, relation direction, event roles, annotation rules, allowable evidence, and treatment of uncertainty. Training data should reflect the intended deployment distribution. Data splits must prevent leakage, such as placing documents from the same organization or near-duplicate articles in both training and test sets.

For binary extraction decisions, precision and recall are defined as:

Here TP is the number of correct positive predictions, FP is incorrect positive predictions, and FN is missed positives. The harmonic mean is the F1 score:

These metrics require a matching policy. Strict entity evaluation usually requires exact span and type agreement; relaxed evaluation may allow overlapping spans. Relation evaluation should generally require correct entity arguments and relation type. Micro-averaging aggregates counts across all instances, while macro-averaging computes scores per class and then averages them. Macro scores reveal poor performance on rare classes.

Extraction errors arise from boundary mistakes, ambiguity, domain shift, long-distance dependencies, annotation disagreement, negation, temporal language, and hallucinated relations. Human adjudication and error slices by source, genre, language, and entity type are often more informative than one overall score.

3. Information Retrieval Foundations

Information retrieval seeks documents, passages, or records relevant to a query. A retrieval system normally includes document ingestion, normalization, indexing, candidate retrieval, ranking, filtering, and presentation. The unit of retrieval may be a whole document, paragraph, sentence, product, or database row.

3.1 Inverted Indexes

An inverted index maps each term to a postings list containing documents in which it occurs. A postings entry may include term frequency, positions, fields, and payloads. For example:

Index construction includes parsing, Unicode normalization, tokenization, case handling, optional stemming or lemmatization, stop-word decisions, and storage of positions. Phrase queries require positions. Field-aware retrieval can give title terms greater importance than body terms.

Boolean retrieval uses operators such as AND, OR, and NOT. It is transparent and useful for filtering, but pure Boolean matching does not naturally rank partially matching documents. Ranked retrieval assigns a score and returns the highest-scoring candidates.

3.2 TF–IDF and BM25

Term frequency–inverse document frequency represents the intuition that a term is important when it occurs often in a document but is uncommon across the collection. A common inverse document frequency definition is:

Here N is the number of documents and df(t) is the number containing term t. One simple document weight is tf(t,d) × idf(t), although practical systems use normalization and field weighting.

BM25 improves upon raw TF–IDF through term-frequency saturation and document-length normalization:

The sum is over query terms t. f(t,D) is the frequency of t in document D; |D| is document length; avgdl is average document length; k₁ > 0 controls frequency saturation; b, usually between 0 and 1, controls length normalization. A common IDF variant is ln((N − df(t) + 0.5) ÷ (df(t) + 0.5)); implementations should document which variant they use. BM25 assumes that term occurrence statistics provide useful evidence of relevance and is especially strong for exact terminology, rare names, and identifiers.

Worked example. Suppose a query has one term, N = 1,000, df = 100, k₁ = 1.2, b = 0.75, average document length 100. For a document with f = 2 and length 80, using the common BM25 IDF:

idf ≈ ln(900.5 ÷ 100.5) ≈ 2.19.

The length-normalization denominator is 1.2 × [0.25 + 0.75 × 80 ÷ 100] = 1.02. The term-frequency factor is (2 × 2.2) ÷ (2 + 1.02) ≈ 1.46. Thus the contribution is approximately 2.19 × 1.46 = 3.20. This score is meaningful only relative to scores computed with the same index, parameters, and query-processing rules.

4. Dense and Hybrid Retrieval

Sparse retrieval represents text using explicit vocabulary dimensions. Dense retrieval maps queries and documents to vectors in a lower-dimensional continuous space. A similarity function such as cosine similarity is:

q and d are vectors, q · d is their dot product, and ||q|| and ||d|| are Euclidean norms. Cosine similarity is defined for nonzero vectors and ranges from −1 to 1. If vectors are normalized to unit length, cosine similarity equals the dot product.

Dense encoders can retrieve paraphrases and conceptually related passages even when query and document vocabulary differs. Approximate nearest-neighbour indexes, such as graph- or quantization-based structures, reduce latency at the cost of possible recall loss and additional memory. Dense retrieval may underperform for rare identifiers, exact codes, spelling variants, and newly introduced terms.

Hybrid retrieval combines sparse and dense evidence. A simple normalized fusion is:

Because the two scores may have different scales, normalization or rank-based fusion is necessary. Reciprocal rank fusion, for example, combines rankings using contributions proportional to 1 ÷ (k + rank), where k is a stabilizing constant. Hybrid systems are often effective because lexical matching supplies precision while dense matching supplies semantic recall.

A practical architecture uses BM25 to retrieve a broad candidate set, vector search to retrieve another set, union and deduplicate candidates, and then apply a cross-encoder reranker. A cross-encoder reads the query and candidate together and usually improves ranking quality, but it is more expensive and is therefore applied only to a limited candidate set.

5. Question Answering and Retrieval-Augmented Generation

Open-domain question answering (QA) must locate evidence before producing an answer. A retrieval-augmented generation (RAG) pipeline typically performs query analysis, candidate retrieval, reranking, context construction, answer generation, and citation or evidence validation.

The distinction between extractive and generative QA is important. Extractive QA selects a span from the evidence and is easier to audit. Generative QA composes an answer and can synthesize information across passages, but it can also introduce unsupported claims. Retrieval quality places an upper bound on evidence-grounded answer quality: if the necessary passage is not retrieved, a downstream generator cannot reliably cite it.

Important design choices include chunk size, overlap, metadata filters, query rewriting, multi-query retrieval, citation boundaries, and handling of tables. Chunks that are too small lose context; chunks that are too large dilute relevance and exceed the model context window. Answers should be constrained to retrieved evidence when factual grounding is required, and the system should abstain when evidence is insufficient.

QA evaluation should separate retrieval from generation. Retrieval metrics include Recall@k, Precision@k, mean reciprocal rank, and normalized discounted cumulative gain. If the first relevant result occurs at rank r, reciprocal rank is 1 ÷ r; mean reciprocal rank averages this value over queries. Generation can be assessed for exact match, token-level F1, factual correctness, citation entailment, completeness, and harmful hallucination rate. Human evaluation remains necessary for ambiguous or multi-hop questions.

6. Illustrative Case: Research-Policy Intelligence System

Background. A university wishes to monitor research-policy announcements, funding calls, and regulatory changes across government websites and scholarly news.

Problem. Users ask questions such as, “Which agencies introduced artificial-intelligence funding programmes in 2024, and what were the eligibility requirements?” Keyword search returns duplicate pages, while a language model without retrieval may confuse dates and agencies.

Analysis. The information need contains entities (agencies, programmes, institutions), relations (agency introduced programme), events (announcement and funding call), temporal constraints, and document-level evidence. Exact programme names and dates favour sparse retrieval; paraphrased descriptions favour dense retrieval.

Approach. The system crawls approved sources, stores document versions and publication dates, removes boilerplate, and splits pages into semantically coherent passages. An inverted index supports BM25 retrieval over title, headings, body, and metadata. A domain-adapted encoder supplies dense vectors. Candidate union is reranked by a cross-encoder. NER identifies agencies, dates, funding amounts, and eligibility expressions. Event extraction records announcement trigger, programme, agency, deadline, and source span. A QA component generates answers only from reranked passages and attaches citations. A validator checks that every date and amount in the answer is supported by cited text.

Outcome. The hybrid system improves recall for paraphrased policy language while retaining precise matching for programme codes. Versioned documents prevent current pages from erasing historical evidence. Human reviewers inspect low-confidence extractions and conflicting sources.

Learning points. The case demonstrates that retrieval, extraction, and generation should be evaluated separately; provenance is a first-class data field; metadata filters are as important as semantic similarity; and abstention is preferable to an unsupported answer.

7. Engineering, Governance, and Professional Practice

Production systems require latency, freshness, cost, and reliability targets. Indexes should support incremental updates, deletion requests, access-control filtering, and rollback. Security must address prompt injection in retrieved documents, malicious content, data poisoning, and leakage of confidential passages. Retrieval filters must be applied before generation so that a model cannot expose documents a user is not authorized to access.

Privacy is especially important when extracting people, health information, or financial records. Data minimization, retention limits, encryption, audit logs, and role-based access should be designed into the pipeline. Automated extraction may reproduce demographic or linguistic bias because training data and source collections are uneven. Evaluation should therefore include subgroup and language performance, not only aggregate scores.

A system card or model documentation should state intended use, known limitations, training and evaluation data, confidence interpretation, and escalation procedures. Confidence scores are not automatically probabilities; calibration should be measured if scores drive decisions. Human oversight is required when errors can affect employment, credit, healthcare, legal status, or public safety.

8. Contemporary Directions and Limitations

Current research explores instruction-tuned extractors, structured decoding, multilingual and multimodal extraction, long-context retrieval, graph-enhanced retrieval, learned sparse representations, late-interaction models, and agentic search. Large language models can perform zero-shot extraction and query rewriting, but reproducibility, schema adherence, cost, and hallucination remain concerns.

A key research issue is evaluation realism. Static benchmarks often fail to represent changing terminology, adversarial queries, evolving web content, or enterprise access controls. Future systems will need temporally aware indexes, continual learning with safeguards, uncertainty-aware answers, and evaluations that measure evidence quality and user outcomes rather than surface similarity alone.

Best-Practice Checklist

• Define entity, relation, and event schemas before annotation.
• Preserve source spans, document versions, timestamps, and model versions.
• Evaluate strict and relaxed matching where boundary ambiguity exists.
• Use BM25 for exact terminology and dense retrieval for semantic variation.
• Normalize or calibrate scores before hybrid fusion.
• Rerank a manageable candidate set rather than the entire corpus.
• Apply access-control and metadata filters before answer generation.
• Require citations and support for factual answers.
• Measure recall, ranking quality, factuality, latency, cost, and fairness.
• Provide abstention and human escalation for uncertain or high-impact cases.

Chapter Summary

Information extraction converts text into entities, relations, events, and linked knowledge. NER identifies typed spans; relation extraction connects arguments; event extraction models triggers and roles; coreference and entity linking improve document-level coherence. Precision, recall, and F1 provide basic extraction evaluation, but matching policies and error analysis determine whether scores are meaningful.

Information retrieval uses inverted indexes for efficient lexical matching. TF–IDF motivates term weighting, while BM25 adds frequency saturation and document-length normalization. Dense retrieval represents meaning in vector space and can recover paraphrases, whereas sparse retrieval remains strong for rare and exact terms. Hybrid retrieval, reranking, and metadata filtering combine these strengths.

Question-answering systems should retrieve evidence, construct context, generate or extract an answer, and validate citations. RAG improves grounding but does not eliminate hallucination or access-control risks. Effective professional systems preserve provenance, evaluate retrieval separately from generation, support abstention, and address privacy, bias, security, freshness, and governance.

Key Terms

Information extraction: Converting unstructured text into structured facts.

Named-entity recognition: Identifying and typing entity spans in text.

BIO tagging: A token-label scheme marking beginning, inside, and outside positions.

Relation extraction: Identifying semantic relations between entities.

Event extraction: Detecting events, triggers, arguments, and roles.

Coreference resolution: Linking expressions that refer to the same entity or event.

Entity linking: Mapping a mention to a canonical knowledge-base identifier.

Inverted index: A term-to-postings data structure for efficient retrieval.

Postings list: The documents and associated statistics stored for a term.

Term frequency: The number of times a term occurs in a document.

Inverse document frequency: A measure that gives greater weight to rarer terms.

BM25: A probabilistic lexical ranking function with saturation and length normalization.

Dense retrieval: Retrieval using learned continuous vector representations.

Sparse retrieval: Retrieval using explicit vocabulary features and postings.

Hybrid retrieval: Combining sparse and dense retrieval evidence.

Reranker: A model that scores an initial candidate set more precisely.

Retrieval-augmented generation: Answer generation conditioned on retrieved evidence.

Provenance: Information recording the origin and support of an extracted fact.

Calibration: Agreement between confidence scores and observed correctness.

Abstention: Declining to answer when evidence or confidence is insufficient.

Review Questions

Short Answer Questions

1. What is the difference between named-entity recognition and entity linking?
2. Why are source spans and provenance important in information extraction?
3. What purpose does an inverted index serve?
4. Explain the role of document length normalization in BM25.
5. State one strength and one limitation of dense retrieval.

Long Answer Questions

1. Describe a complete pipeline for extracting entities, relations, and events from a collection of news articles.
2. Explain BM25, defining each variable and discussing its assumptions and limitations.
3. Compare Boolean, BM25, dense, and hybrid retrieval in terms of representation, strengths, and deployment costs.
4. Design an evaluation framework for a retrieval-augmented question-answering system.
5. Discuss privacy, bias, access control, and provenance requirements for an enterprise IE and IR platform.

Analytical / Application Questions

1. A search engine performs poorly on both product codes and paraphrased user questions. Propose a hybrid retrieval and reranking design, and justify each stage.
2. An event extractor reports that a company acquired another company, but the source sentence says the acquisition was denied. Explain the likely error and design a solution using polarity, modality, or assertion attributes.
3. Design a policy-document QA system that must answer historical questions accurately. Specify indexing, versioning, extraction, retrieval, citation, evaluation, and abstention mechanisms.

Multiple Choice Questions with Answers

Question 1

Which representation marks the beginning of an organization entity span in BIO tagging?

A. B-ORG

B. I-ORG

C. O

D. B-LOC

Correct Answer: A. B-ORG

Question 2

Which metric is the harmonic mean of precision and recall?

A. Accuracy

B. F₁ score

C. Mean reciprocal rank

D. Recall@k

Correct Answer: B. F₁ score

Question 3

What does an inverted index map?

A. Documents to language models

B. Queries to generated answers

C. Terms to postings lists

D. Entities to tokenizers

Correct Answer: C. Terms to postings lists

Question 4

In BM25, what is the principal effect of parameter k₁?

A. It controls term-frequency saturation

B. It removes stop words

C. It sets the number of documents

D. It determines vector dimensionality

Correct Answer: A. It controls term-frequency saturation

Question 5

Which retrieval approach is generally strongest for exact rare identifiers?

A. Dense retrieval alone

B. Lexical sparse retrieval

C. Random retrieval

D. Generative decoding without retrieval

Correct Answer: B. Lexical sparse retrieval

Question 6

If vectors are normalized to unit length, cosine similarity is equal to which quantity?

A. The vector sum

B. The dot product

C. The document frequency

D. The reciprocal rank

Correct Answer: B. The dot product

Question 7

Why is score normalization often needed in hybrid retrieval?

A. Sparse and dense scores can have different scales

B. Dense vectors cannot be stored

C. BM25 requires image input

D. Normalization deletes duplicate documents

Correct Answer: A. Sparse and dense scores can have different scales

Question 8

What is the main purpose of a cross-encoder reranker?

A. To tokenize the entire corpus once

B. To classify documents before indexing

C. To score query-candidate pairs more precisely

D. To replace access control

Correct Answer: C. To score query-candidate pairs more precisely

Question 9

Which practice best reduces unsupported claims in retrieval-augmented QA?

A. Answering without retrieved context

B. Requiring evidence-linked citations and validation

C. Increasing temperature during generation

D. Removing document metadata

Correct Answer: B. Requiring evidence-linked citations and validation

Question 10

What should a system do when retrieved evidence is insufficient for a high-stakes answer?

A. Invent the most probable answer

B. Ignore access restrictions

C. Abstain or escalate for human review

D. Delete the query

Correct Answer: C. Abstain or escalate for human review