Text Processing and Document Representation

Introduction

Information retrieval (IR) systems operate on language, but computers do not retrieve “meaning” directly from raw language. They process documents by transforming text into structured representations that can be indexed, compared, ranked, and presented to users. This transformation is commonly called the text-processing pipeline.

A document may contain words, punctuation, tables, hyperlinks, dates, author information, headings, images, and machine-generated metadata. The appropriate representation depends on the retrieval task. A legal search system may need exact phrases and citations; a multilingual news system may require language identification and morphology-aware processing; a semantic search system may represent documents as dense vectors produced by a language model.

Text processing is therefore not merely a sequence of cleaning operations. Each decision changes the evidence available to the retrieval model. Removing a negation, conflating unrelated word forms, or discarding a product code can reduce retrieval quality. Good preprocessing is task-dependent, reproducible, language-aware, and evaluated empirically.

Learning Objectives

By the end of this chapter, you should be able to:

• explain the role of tokenization, normalization, stemming, lemmatization, and stop-word handling;
• distinguish words, subwords, phrases, fields, metadata, and entities as retrieval units;
• design phrase-detection and metadata-extraction procedures;
• calculate and interpret term frequency, inverse document frequency, and TF–IDF weighting;
• compare sparse representations with dense neural representations;
• select preprocessing strategies for different languages and domains;
• identify failure modes, ethical risks, and governance requirements in text processing.

1. The Text-Processing Pipeline

A typical pipeline is:

acquisition → decoding → language identification → structural parsing → tokenization → normalization → linguistic processing → feature extraction → weighting → representation → indexing.

The stages are conceptually distinct, although production systems may combine them. A web document, for example, can be decoded from UTF-8, separated into title and body fields, tokenized, normalized, enriched with named entities, and represented both as a sparse lexical vector and a dense embedding.

1.1 Acquisition and decoding

The system must preserve the original document and record its source, timestamp, version, and encoding. Unicode normalization is important because visually identical strings can have different underlying code-point sequences. A system may apply NFC normalization to combine characters into canonical forms. NFKC may additionally convert compatibility characters, but it can be unsafe when distinctions matter, such as mathematical notation, typography, or identifiers.

Encoding errors should be logged rather than silently repaired. Reproducibility requires retaining the raw input, preprocessing version, language, and configuration.

1.2 Language and domain identification

Language identification determines whether the system should apply English stemming, Arabic segmentation, Chinese word discovery, or another procedure. Documents may be multilingual, and a single language label can be inadequate. Domain identification is also useful: biomedical, legal, social-media, and source-code collections have different tokenization and vocabulary requirements.

1.3 Structural parsing

HTML tags, XML fields, PDF layout, email headers, and document sections can provide valuable signals. Text in a title or heading often deserves more weight than text in boilerplate. Structural parsing should distinguish visible content from navigation menus, advertisements, scripts, and duplicated templates.

2. Tokenization

Tokenization divides a character sequence into units used by later processing. In the sentence “The model’s accuracy was 92.5%,” plausible tokenizations include:

• word tokens: The | model’s | accuracy | was | 92.5%;
• word-plus-punctuation tokens: The | model | ’s | accuracy | was | 92.5 | %;
• subword tokens used by neural models: model | ##’s, or another vocabulary-specific segmentation.

There is no universally correct tokenization. The choice should reflect retrieval needs.

2.1 Word and punctuation tokenization

Whitespace splitting is simple but fails for punctuation, contractions, URLs, email addresses, hyphenated terms, and scripts without spaces. Rule-based tokenizers can preserve URLs, dates, version numbers, and product identifiers. For example, “C++,” “COVID-19,” “BERT-based,” and “[email protected]” may need specialized treatment.

Punctuation can be informative. In programming-code retrieval, “C#” and “C++” are meaningful terms. In sentiment analysis, emoticons and repeated exclamation marks may matter. In ordinary prose, commas and full stops often contribute little to topical matching.

2.2 Subword tokenization

Subword methods, including byte-pair encoding and unigram language-model tokenization, divide unknown or rare words into reusable fragments. They are useful for neural language models, morphologically rich languages, spelling variation, and unseen words. However, subword boundaries are not necessarily linguistic morphemes. A subword representation can also make exact phrase interpretation more difficult.

2.3 Language-specific issues

Chinese, Japanese, and Thai do not consistently use spaces between words. Word segmentation is therefore a modeling problem. Arabic and Hebrew involve clitics and rich morphology. German compounds may be written as one word, while Turkish can encode several grammatical functions in suffixes. Multilingual systems should not assume that English whitespace rules generalize across languages.

3. Normalization and Canonicalization

Normalization reduces irrelevant variation while preserving distinctions needed by the task. Common operations include:

• case folding, such as converting “Retrieval” to “retrieval”;
• Unicode normalization;
• punctuation and whitespace standardization;
• spelling-variant mapping, such as “colour” and “color” where appropriate;
• date, number, currency, and unit normalization;
• expansion or preservation of abbreviations;
• URL and email canonicalization.

Case folding improves matching between “Database” and “database,” but it may harm retrieval when case distinguishes “US” from “us,” chemical symbols, or programming identifiers. A robust system may retain both the original form and a normalized form.

Normalization must be conservative. Converting “C++” to “c” or removing the hyphen from “well-being” can destroy important meaning. In sensitive domains, normalization rules should be documented and reviewed by subject experts.

4. Stemming and Lemmatization

Morphological processing attempts to connect related surface forms.

4.1 Stemming

A stemmer applies heuristic rules to remove affixes. The Porter stemmer, for example, may map “connect,” “connected,” and “connection” to related stems, although the resulting stem may not be an English word. Stemming is fast and language-specific, but it can over-stem, conflating unrelated terms, or under-stem, failing to connect variants.

4.2 Lemmatization

Lemmatization maps a word to a dictionary form, or lemma, using vocabulary and often grammatical information. “Cars” may become “car,” and “was” may become “be.” Part-of-speech information can be essential because the correct lemma of “saw” differs when it is a noun or a verb.

Lemmatization generally requires more linguistic resources and computation than stemming. It may be valuable in morphologically rich languages, but a lemma alone can remove distinctions that matter for exact retrieval.

A practical system can index both surface and normalized forms. This supports exact matching while improving recall through morphological equivalence.

5. Stop Words and Function Words

Stop words are frequent words that are sometimes excluded from indexing, such as “the,” “of,” and “is.” Removing them reduces index size and may improve efficiency. However, frequency alone is not a sufficient reason for removal.

Function words can be crucial in phrases (“to be or not to be”), legal expressions (“without prejudice”), questions, and negation. Search engines often retain stop words for phrase queries or use them in proximity information. A field-specific policy is preferable to a universal list.

The decision should consider:

1. Task: known-item search and legal retrieval may need exact function words.
2. Language: stop-word lists are language-specific.
3. Query behavior: users may submit short, phrase-like queries.
4. Index design: positional indexes can preserve phrase evidence even if ordinary term scoring ignores it.
5. Evaluation: compare retrieval effectiveness before and after removal.

6. Phrases, Collocations, and Proximity

Individual terms do not always represent meaning adequately. “New York,” “machine learning,” and “information retrieval” are more informative as expressions than as unrelated words.

A phrase is an ordered sequence of tokens. A collocation is a sequence that occurs together more often than expected by chance. Phrase detection can be rule-based, dictionary-based, statistically learned, or generated by a language model.

A positional inverted index stores token positions in each document. A phrase query can then verify that terms occur consecutively and in the correct order. A proximity query allows a limited distance between terms. Phrase indexing improves precision, but aggressive phrase extraction can reduce recall when users paraphrase the concept.

Statistical association measures include pointwise mutual information (PMI). For terms x and y, under a specified corpus and event definition:

Here, P(x) and P(y) are the probabilities that x and y occur in the selected units, and P(x,y) is their joint probability. PMI is high when the terms co-occur more often than independence predicts. It is unreliable for very rare events, so minimum-frequency thresholds and alternative measures such as likelihood ratio or t-score are often used.

7. Metadata and Fielded Documents

Metadata describes a document rather than constituting its main body. Examples include title, author, publication date, source, language, geographic location, document type, access rights, and hyperlinks. Some metadata is explicit; other metadata is extracted.

Named-entity recognition can identify persons, organizations, places, products, and events. Date and quantity extraction can normalize “12 March 2025,” “2025-03-12,” and related forms for filtering. Metadata extraction should retain provenance and confidence because automated extraction can be wrong.

Fielded retrieval treats a document as a collection of fields. A title match may receive greater importance than a body match. A simple field-weighted score can be written as:

where d is a document, q is a query, Sᵢ is the retrieval score for field i, and wᵢ is its nonnegative weight. The weights are design or learned parameters and should be validated on representative relevance judgments. Scores from different fields must be calibrated before combining them; otherwise, a field with a larger numerical scale can dominate.

Metadata also supports access control, freshness filtering, deduplication, faceted navigation, and collection analysis. Incorrect metadata can create systematic exclusion, such as misclassifying content by language or geographic origin.

8. Sparse Vector Representations

A vocabulary of V terms defines a vector space. A document can be represented as a vector whose coordinates correspond to terms. Most documents contain only a small fraction of the vocabulary, so these vectors are sparse.

8.1 Term frequency

Let tf(t,d) be the number of occurrences of term t in document d. Raw counts favor long documents. A common alternative is logarithmic scaling:

The formula reduces the effect of repeated occurrences. It assumes that additional repetitions provide diminishing evidence of topical relevance.

8.2 Inverse document frequency

Let N be the number of documents and df(t) the number of documents containing t. A basic inverse document frequency is:

This is defined for 1 ≤ df(t) ≤ N. Rare terms receive larger values than common terms. In practical systems, smoothed variants are used to avoid extreme values and to control the treatment of terms appearing in every document.

8.3 TF–IDF

The standard product is:

A document vector contains w(t,d) for each vocabulary term. Query and document vectors can be compared using cosine similarity:

Here, q and d are nonzero query and document vectors, q · d is their dot product, and ||·||₂ is the Euclidean norm. Cosine similarity measures orientation rather than raw vector magnitude and is applicable when term dimensions are aligned.

Worked example. Suppose N = 1,000, term t occurs 10 times in document d, and t appears in df(t) = 100 documents. Using logarithmic term frequency:

tf_log(t,d) = 1 + log₁₀(10) = 2

and:

idf(t) = log₁₀(1000/100) = 1.

Therefore:

w(t,d) = 2 × 1 = 2.

The result is dimensionless because both components are counts transformed by logarithms. This score is not a probability and should not be interpreted as one.

8.4 Normalization and weighting alternatives

Vector length normalization reduces the advantage of long documents. BM25, a probabilistic ranking function, uses term frequency saturation and document-length normalization. Its exact implementation varies, but the important design principle is that the first few occurrences of a term usually provide more evidence than later repetitions.

Sparse representations are interpretable, efficient for exact term matching, and well suited to inverted indexes. Their limitations include vocabulary mismatch, weak handling of synonymy, and difficulty representing broader context.

9. Dense Representations and Neural Retrieval

A dense representation maps a document or passage to a vector with relatively few zero coordinates, usually learned from large text collections. Transformer encoders can generate contextual representations in which a token’s vector depends on surrounding text. Dual-encoder systems independently encode queries and documents, then compare their vectors, often with a dot product or cosine similarity.

Dense retrieval can match “automobile repair” with “car maintenance” even when the exact terms differ. It can also capture contextual and multilingual relationships. However, dense vectors may be less interpretable, can encode social bias, require substantial computation, and may miss exact identifiers, numbers, or rare names. Approximate nearest-neighbor indexes improve speed but introduce search-accuracy trade-offs.

A strong contemporary architecture is hybrid retrieval: sparse lexical search retrieves exact and rare matches, while dense retrieval supplies semantic recall. A re-ranking model can then evaluate a smaller candidate set using richer query-document interaction. Hybrid systems must avoid double-counting evidence and should be evaluated separately for exact-match, semantic, multilingual, and out-of-domain queries.

10. Choosing Representations Across Languages and Domains

There is no universally best representation. Selection should follow the collection and task.

Cross-lingual retrieval may translate queries, documents, or both, or use multilingual representations. Translation can introduce terminology errors, while multilingual encoders may have uneven performance across languages. Evaluation must therefore report results by language, domain, and query type rather than only an overall average.

11. Illustrative Case: A Multilingual Legal-Information Portal

Background. A public legal-information portal contains court decisions in English and Hindi. Users search by case name, statute, legal concept, date, and location. Documents contain long passages, citations, headings, and scanned PDFs.

Problem. An initial keyword system produces poor results because OCR introduces spelling errors, case names occur in titles and body text, and users search for concepts that do not exactly match statutory language. Removing all stop words also damages phrase queries such as “right to privacy.”

Analysis. Error analysis reveals four distinct needs: exact citation retrieval, phrase and proximity search, multilingual morphology, and semantic matching. OCR confidence is lower in scanned documents, and metadata fields are inconsistently populated.

Approach. The redesigned pipeline preserves the raw PDF and OCR output, records OCR confidence, applies Unicode normalization without deleting legal symbols, detects language at paragraph level, and indexes title, citation, statute, date, and body as separate fields. It retains positional information and uses a curated list of legal phrases. Lemmatization is applied selectively, while surface forms remain searchable. Sparse BM25 retrieval handles citations and names; multilingual dense retrieval supplies conceptually related candidates. A cross-encoder re-ranks the top candidates. Human assessors judge relevance separately for English, Hindi, exact citations, and conceptual queries.

Outcome. The system becomes more robust because no single representation is expected to solve every problem. The most important improvement comes not from one algorithm but from preserving structure, measuring errors by query category, and combining complementary evidence.

Learning points. Preprocessing is part of retrieval design. Metadata quality, OCR quality, positional indexing, language handling, and evaluation methodology can matter as much as the ranking model.

12. Limitations, Ethics, and Governance

Text processing can reproduce historical biases in training data and metadata. Entity extraction may misidentify people, especially across languages or naming conventions. Aggressive normalization can erase cultural or dialectal distinctions. Stop-word lists and language identifiers can systematically disadvantage minority languages.

Privacy is a major concern when documents contain names, health information, locations, or communications. Organizations should apply data minimization, access controls, retention policies, and, where appropriate, de-identification. De-identification is imperfect: combinations of supposedly harmless attributes may re-identify individuals.

Governance should include versioned preprocessing configurations, audit logs, provenance for extracted metadata, human review for high-impact use cases, and evaluation across demographic and linguistic groups. Retrieval systems used in legal, medical, employment, or public-sector contexts should not present relevance scores as judgments of truth or personal worth. Users need clear information about source, date, uncertainty, and automated processing.

13. Best-Practice Workflow

1. Preserve raw documents and record provenance.
2. Identify languages, scripts, domains, and document structure.
3. Define tokenization and normalization rules from the retrieval task.
4. Preserve original fields alongside processed fields.
5. Use stemming or lemmatization only after measuring its effect.
6. Retain positions when phrase and proximity queries matter.
7. Extract metadata with confidence and provenance.
8. Build a transparent sparse baseline before adding dense models.
9. Evaluate hybrid systems by query type, language, and domain.
10. Version every preprocessing component and test for regressions.
11. Monitor bias, privacy risk, and failures after deployment.

Chapter Summary

Text processing transforms heterogeneous language data into searchable evidence. Tokenization determines retrieval units; normalization reduces irrelevant variation; stemming and lemmatization address morphology but can create false equivalences. Stop-word handling must be task- and language-sensitive. Phrase detection and positional indexing preserve compositional meaning, while metadata extraction supplies fields for filtering, boosting, and faceted search.

Sparse representations such as TF–IDF and BM25 are efficient and interpretable for lexical matching. Dense representations support semantic and multilingual matching but may be opaque, computationally demanding, and weak on rare identifiers. Hybrid retrieval combines their strengths. Effective postgraduate practice requires empirical evaluation, language and domain awareness, reproducibility, and governance of bias and privacy.

Key Terms

Tokenization: Division of text into processing units.

Normalization: Transformation that reduces irrelevant variation in text.

Unicode normalization: Canonical or compatibility standardization of Unicode sequences.

Case folding: Converting text to a common case for matching.

Stemming: Heuristic removal of affixes from words.

Lemmatization: Mapping an inflected word to a dictionary lemma.

Stop word: A frequent function word that may be excluded or down-weighted.

Phrase: An ordered sequence of tokens treated as a unit.

Collocation: Terms that occur together more strongly than expected by chance.

Positional index: An index that records term locations within documents.

Metadata: Information describing a document or its context.

Named-entity recognition: Identification and classification of entities in text.

Sparse vector: A vector in which most coordinates are zero.

Dense vector: A representation with relatively few zero coordinates.

Term frequency: The occurrence count or transformed frequency of a term in a document.

Inverse document frequency: A measure that increases as a term occurs in fewer documents.

TF–IDF: A weighting scheme combining term frequency and inverse document frequency.

Cosine similarity: Similarity based on the angle between nonzero vectors.

Hybrid retrieval: Combination of lexical sparse retrieval and semantic dense retrieval.

Provenance: Information recording the origin and processing history of data.

Review Questions

5 Short Answer Questions

1. Why is tokenization language- and domain-dependent?
2. Distinguish stemming from lemmatization.
3. What information does a positional index preserve?
4. Why can stop-word removal harm phrase retrieval?
5. State one strength and one limitation of dense representations.

5 Long Answer Questions

1. Explain the major stages of a production text-processing pipeline and show how errors at an early stage affect retrieval.
2. Compare surface forms, stems, lemmas, and subwords as indexing units.
3. Derive and interpret TF–IDF weighting, including the role of document frequency.
4. Compare sparse, dense, and hybrid retrieval for a multilingual legal collection.
5. Discuss ethical, privacy, and governance issues in metadata extraction and automated retrieval.

3 Analytical / Application Questions

1. Design a preprocessing pipeline for a social-media search system containing hashtags, emojis, code-switching, URLs, and spelling variation. Justify which information you would preserve.
2. A biomedical search engine retrieves too many documents containing common terms but misses documents using synonyms. Propose an evaluation-driven combination of normalization, metadata, sparse weighting, and dense retrieval.
3. A court-search system removes all stop words and applies aggressive stemming. Analyze the likely failure cases and propose a safer revised design.

Multiple Choice Questions with Answers

Question 1

Which operation divides a character sequence into units used by later processing?

A. Tokenization

B. Lemmatization

C. Re-ranking

D. Deduplication

Correct Answer: A. Tokenization

Question 2

What is the main distinction between stemming and lemmatization?

A. Stemming uses document fields, while lemmatization uses metadata

B. Stemming heuristically removes affixes, while lemmatization maps words to dictionary forms

C. Stemming creates dense vectors, while lemmatization creates sparse vectors

D. Stemming detects phrases, while lemmatization detects languages

Correct Answer: B. Stemming heuristically removes affixes, while lemmatization maps words to dictionary forms

Question 3

Why are term positions stored in an inverted index?

A. To estimate document encoding

B. To verify phrase and proximity relationships

C. To replace inverse document frequency

D. To identify the document author automatically

Correct Answer: B. To verify phrase and proximity relationships

Question 4

If N = 1,000 and df(t) = 100, what is idf(t) using idf(t)=log₁₀(N/df(t))?

A. 0.1

B. 1

C. 10

D. 100

Correct Answer: B. 1

Question 5

Which representation is generally most suitable for exact matching of rare product identifiers?

A. Only a dense document embedding

B. A sparse lexical representation preserving surface forms

C. Only a stop-word list

D. A representation containing document length alone

Correct Answer: B. A sparse lexical representation preserving surface forms

Question 6

What does cosine similarity compare?

A. The alphabetical order of two documents

B. The angle between two nonzero vectors

C. The number of metadata fields

D. The encoding formats of two files

Correct Answer: B. The angle between two nonzero vectors

Question 7

Which statement about stop-word removal is most accurate?

A. It is always necessary for high-quality retrieval

B. It should be evaluated because function words can matter in phrases and negation

C. It applies identically across all languages

D. It eliminates the need for positional indexes

Correct Answer: B. It should be evaluated because function words can matter in phrases and negation

Question 8

What is a principal advantage of dense retrieval over purely lexical retrieval?

A. It always provides interpretable term matches

B. It can match semantically related expressions with little lexical overlap

C. It requires no training data or computation

D. It preserves every exact identifier automatically

Correct Answer: B. It can match semantically related expressions with little lexical overlap

Question 9

In a field-weighted score, what does wᵢ represent?

A. The number of documents in the collection

B. The weight assigned to field i

C. The token position in field i

D. The Unicode encoding of field i

Correct Answer: B. The weight assigned to field i

Question 10

Which practice best supports reproducible text processing?

A. Overwriting raw documents after normalization

B. Changing preprocessing rules without recording versions

C. Preserving raw input and versioning preprocessing configurations

D. Using one English stop-word list for every language

Correct Answer: C. Preserving raw input and versioning preprocessing configurations