Statistical Language Models and Text Representation

Introduction

Natural language processing (NLP) systems require a numerical account of language. A computer does not directly manipulate meanings, words, or grammatical intuitions; it processes symbols, vectors, probabilities, and learned parameters. Statistical language models provide one of the earliest and most influential ways to represent linguistic regularities. Text-representation methods such as bag-of-words, TF–IDF, and distributional vectors transform documents into forms suitable for classification, retrieval, clustering, generation, and downstream machine learning.

This chapter develops two complementary perspectives. First, a language model estimates the probability of a sequence and can therefore rank, predict, or generate text. Second, a document representation maps text to a feature space in which documents can be compared. These perspectives are related but not identical: language models focus on sequential dependence, whereas bag-of-words and TF–IDF generally discard word order in exchange for computational simplicity.

Learning Objectives

By the end of this chapter, you should be able to:

• formulate n-gram language models and explain the Markov assumption;
• estimate probabilities using maximum likelihood and explain sparsity;
• apply smoothing methods, including additive smoothing and backoff concepts;
• calculate and interpret cross-entropy and perplexity;
• construct bag-of-words and TF–IDF representations;
• select and engineer lexical, syntactic, and domain-specific features;
• calculate cosine, Jaccard, and Euclidean similarity;
• explain distributional semantics and word embeddings;
• evaluate the curse of dimensionality and identify appropriate representations;
• analyse practical, ethical, and governance issues in language modelling.

1. Probability as a Model of Language

Let a tokenized sentence be represented by a sequence w₁, w₂, ..., w_T, where T is the number of tokens. A language model assigns a probability to the sequence:

Here, the chain rule of probability is used; no independence assumption is required. The conditional probability at position t represents the likelihood of the next token given its entire history. In practice, estimating a separate probability for every possible history is infeasible because the number of histories grows rapidly and most sequences are unseen.

A language model can support several tasks:

• Prediction: estimate the next likely word or token.
• Generation: repeatedly sample or select tokens.
• Scoring: compare the plausibility of alternative sentences.
• Recognition: assist speech recognition, OCR, or spelling correction.
• Evaluation: measure how well a model predicts held-out text.

The vocabulary V contains the tokens that the model recognizes. Unknown or rare words are commonly mapped to a special <UNK> token. Sentence boundary symbols such as <s> and </s> allow a model to represent beginning and ending contexts.

2. N-Gram Language Models

An n-gram model approximates the complete history by retaining only the preceding n-1 tokens:

Thus, a unigram model uses no context, a bigram uses one preceding token, and a trigram uses two. The sequence probability becomes:

The initial positions are handled using start-of-sentence markers. The approximation is computationally useful but linguistically limited. A trigram model cannot directly distinguish contexts that differ more than two tokens earlier, even when that earlier information is essential for agreement or meaning.

2.1 Maximum-likelihood estimation

For a bigram model, the maximum-likelihood estimate (MLE) is:

C(w_i-1, w_i) is the count of the bigram, and C(w_i-1) is the count of the preceding word as a context. More generally:

where h is an n-gram history. The estimate is appropriate when counts are reliable and the denominator is non-zero. For each fixed history, MLE probabilities sum to one over the observed continuation vocabulary.

Worked example: a bigram model

Consider the illustrative corpus:

• <s> cats chase mice </s>
• <s> cats eat fish </s>
• <s> dogs chase cats </s>

The context cats occurs three times: before chase, eat, and </s>. Therefore:

• P(chase | cats) = 1 ÷ 3 ≈ 0.333
• P(eat | cats) = 1 ÷ 3 ≈ 0.333
• P(</s> | cats) = 1 ÷ 3 ≈ 0.333

The probability assigned by the bigram model to <s> cats chase mice </s> is:

The relevant counts are all one except that cats follows <s> three times, so the value is (3 ÷ 3) × (1 ÷ 3) × (2 ÷ 2) × (1 ÷ 1) = 1 ÷ 3. This calculation illustrates both the chain-product structure and the dependence on corpus counts.

3. Sparsity, Smoothing, and the Curse of Dimensionality

A central problem in n-gram modelling is sparsity. Even a large corpus contains only a small fraction of all possible n-grams. If the vocabulary has |V| types, the number of possible bigrams is |V|², and the number of possible trigrams is |V|³. Increasing n gives more context but produces many unseen combinations.

Under unsmoothed MLE, one unseen n-gram receives probability zero. Since sentence probabilities multiply component probabilities, one zero makes the entire sentence probability zero. This is usually too harsh: an unseen sequence is not necessarily impossible; it may simply be absent from the training corpus.

3.1 Additive smoothing

Additive, or Laplace, smoothing adds a positive constant α to every possible continuation:

Here, α > 0, C(h,w) is the n-gram count, C(h) is the history count, and |V| is the vocabulary size. The denominator preserves normalization because there are |V| possible continuations. Laplace smoothing corresponds to α = 1. Smaller values, often called Lidstone smoothing, reduce the distortion applied to frequent events.

Worked smoothing calculation

Suppose a context occurs 10 times, a particular continuation occurs twice, the vocabulary contains 5 tokens, and α = 1. Then:

The unsmoothed estimate would be 0.20 in this particular case. An unseen continuation, however, receives 1 ÷ 15 ≈ 0.067, rather than zero. The example also shows that additive smoothing can substantially reduce high-count probabilities when the vocabulary is large.

3.2 Other smoothing and discounting methods

More effective n-gram systems commonly use:

• Good–Turing estimation: reallocates probability according to the number of events observed once, twice, and so forth.
• Backoff: uses a lower-order model when a higher-order n-gram is unseen.
• Interpolation: combines multiple orders even when the higher-order estimate is available.
• Kneser–Ney smoothing: discounts counts and uses continuation diversity, making it particularly effective for language modelling.

A generic interpolated model can be written as:

The weights satisfy λ_i ≥ 0 and ∑λ_i = 1. The history h' is shorter than h. The exact estimation of the weights is a modelling choice and should be validated on held-out data.

3.3 The curse of dimensionality

The curse of dimensionality refers to the rapid growth of possible configurations as the number of variables or features increases. In NLP it appears in at least three forms:

1. Combinatorial growth: possible n-grams increase exponentially with n.
2. Sparse document vectors: most documents contain only a tiny fraction of the vocabulary.
3. Statistical inefficiency: data become insufficient to estimate reliable parameters for rare combinations.

Increasing model order is therefore not automatically beneficial. A larger context can reduce bias but increase variance, memory use, and sensitivity to domain mismatch. Smoothing, parameter sharing, subword modelling, and neural representations address different aspects of this problem.

4. Evaluating Language Models with Perplexity

For a test sequence of T tokens, the average negative log-likelihood in base 2 is the cross-entropy:

Perplexity is the exponential transform of cross-entropy:

The two expressions are equivalent when the logarithm base matches the exponential base. Perplexity can also be expressed as the inverse geometric mean probability:

Lower perplexity indicates that the model assigns higher probability to the held-out sequence. Perplexity is meaningful only when models use comparable tokenization, vocabulary, handling of unknown tokens, and test data. It is not a direct measure of usefulness, factuality, fairness, or human linguistic quality.

Worked perplexity example

Suppose a model assigns probabilities 0.5, 0.25, 0.5, and 1 to four successive tokens. Their product is 0.5×0.25×0.5×1 = 0.0625. Thus:

The model has a perplexity of 2 on this sequence. Informally, its average uncertainty is equivalent to choosing between two equally likely alternatives at each position.

5. Text Representation: From Tokens to Features

Text representation is the process of converting linguistic input into measurable variables. The representation determines what information a model can learn. A representation that discards word order may be excellent for topic classification but weak for negation or syntactic interpretation.

A typical preprocessing pipeline includes:

1. document collection and quality checking;
2. sentence segmentation and tokenization;
3. normalization of case, punctuation, spelling, and Unicode;
4. optional stop-word handling and stemming or lemmatization;
5. vocabulary construction;
6. feature extraction and scaling;
7. train–validation–test splitting without leakage.

Preprocessing should be task-dependent. Removing stop words may reduce noise in topic retrieval but can damage sentiment signals such as “not”. Stemming reduces vocabulary size but may create non-words; lemmatization uses morphological analysis and is more linguistically informed, though computationally more expensive.

6. Bag-of-Words Representation

The bag-of-words (BoW) model represents a document by counts of vocabulary items and ignores word order. Let the vocabulary be V = {v₁, ..., v_m}. A document d is represented by:

The vector has dimension m. It is usually sparse because most documents contain few vocabulary terms. BoW is transparent and effective for linear classifiers, information retrieval, and baseline experiments. However, “dog bites man” and “man bites dog” receive the same representation.

N-gram features as an extension

Word unigrams can be supplemented with bigrams or trigrams. Features such as “not good” preserve some local compositional information that individual-word counts lose. The cost is a larger and sparser feature space. Character n-grams can improve robustness to spelling variation, morphology, and noisy user-generated text.

7. TF–IDF Weighting

Raw counts overemphasize terms that occur frequently in a document collection. TF–IDF gives high weight to terms frequent in a document but relatively rare across documents.

A common term-frequency definition is normalized frequency:

where f_t,d is the count of term t in document d. If N is the number of documents and df_t is the number containing t, inverse document frequency is commonly:

The TF–IDF weight is:

If a smoothed version is desired, one may use ln((N+1) ÷ (df_t+1)) + 1. The chosen convention should be documented because implementations differ. Terms appearing in every document have IDF zero under the unsmoothed definition, reflecting their weak discriminative value.

Example

Assume N = 100 documents and a term appears in df_t = 10. If its normalized frequency in document d is 0.05, then:

and:

TF–IDF does not understand synonyms or word order, but it often provides a strong, interpretable baseline.

8. Similarity and Distance Measures

Once texts are vectors, similarity can be calculated geometrically.

8.1 Cosine similarity

For non-zero vectors x and y:

where x·y is the dot product and ||x||₂ = √(∑_i x_i²) is the Euclidean norm. Cosine similarity measures orientation rather than magnitude and is widely used with TF–IDF. It is undefined for a zero vector.

8.2 Euclidean distance

Euclidean distance is sensitive to vector magnitude and is often less suitable for raw document counts unless vectors are normalized.

8.3 Jaccard similarity

For sets of terms A and B:

It ignores frequency and is useful when presence or absence matters, such as comparing keyword sets. Weighted Jaccard variants can incorporate term weights, but their definition must be specified.

Worked cosine example

Let x = (1,2) and y = (2,1). Their dot product is 4, and each norm is √5. Therefore:

The vectors are strongly aligned, although not identical.

9. Feature Engineering for NLP

Feature engineering should reflect the decision problem, data-generating process, and likely sources of error. Useful feature families include:

• Lexical: word and character n-grams, vocabulary richness, word length, punctuation counts.
• Syntactic: part-of-speech patterns, dependency relations, parse-tree fragments.
• Semantic: named entities, topics, embeddings, synonym indicators.
• Discourse: paragraph position, coreference links, cohesion, dialogue turns.
• Metadata: author, timestamp, source, language, or domain—used cautiously to avoid leakage.
• Task-specific: negation scope for sentiment, dosage units for clinical text, or legal citations for case retrieval.

Feature selection can reduce computation and overfitting. Frequency thresholds, mutual information, chi-square statistics, regularization, and dimensionality reduction are common approaches. Feature selection must be performed using training data only; selecting features using the full dataset leaks information from the test set.

10. Distributional Semantics

Distributional semantics is based on the principle that words occurring in similar contexts tend to have related meanings. A word can be represented by a vector derived from its contexts. Unlike BoW document vectors, distributional word representations aim to capture semantic regularities across a corpus.

10.1 Count-based representations

A word–context matrix records how often a target word occurs near a context word. Raw counts may be transformed using positive pointwise mutual information (PPMI):

PPMI is:

PMI is positive when co-occurrence is more likely than would be expected under independence. It can overemphasize rare events, so minimum-count thresholds and alternative weighting are important.

10.2 Predictive embeddings

Methods such as skip-gram and continuous bag-of-words learn dense vectors by predicting contexts or target words. Dense embeddings provide parameter sharing and lower-dimensional representations, often improving generalization over sparse counts. Similarity is commonly measured with cosine similarity.

Embeddings are not automatically meanings. They reflect the distribution, genre, time period, and social biases of their training data. Static embeddings assign one vector to a word, so “bank” has one representation despite multiple senses. Contextual models produce representations dependent on surrounding text and are better suited to polysemy, but they introduce greater computational cost and more complex evaluation.

11. Case Study: Classifying Customer Support Requests

This is an illustrative case. A telecommunications company wants to route incoming customer messages to billing, technical support, cancellation, or account-access teams.

Background and problem

The company receives 200,000 short messages annually. Manual routing is slow, and incorrect routing increases resolution time. Messages contain spelling errors, abbreviations, product names, and personally identifying information.

Analysis

A baseline BoW model with unigram TF–IDF features is attractive because it is fast and interpretable. However, phrases such as “cannot log in” and “do not want to cancel” require local word combinations and negation awareness. Character n-grams may help with misspellings such as “passwrd”. Metadata such as customer identity should not be used as a predictive shortcut.

Approach

The team creates a stratified training, validation, and test split. Personally identifying information is masked before modelling. Three systems are compared:

1. unigram TF–IDF with logistic regression;
2. word and character TF–IDF with a linear classifier;
3. a contextual language model fine-tuned on the labelled messages.

Evaluation includes macro-F1, because minority categories matter, as well as a confusion matrix and calibration analysis. A human-review threshold is introduced for low-confidence predictions.

Outcome

The character-plus-word baseline substantially improves robustness to spelling variation while remaining inexpensive and auditable. The contextual model provides higher validation performance but requires stronger monitoring for drift and higher infrastructure cost. The final deployment uses the robust baseline for automatic routing and sends uncertain or high-impact cases to humans.

Learning points

• A simple representation can outperform a complex model when data and governance constraints matter.
• Accuracy alone can conceal poor performance on minority classes.
• Data leakage, especially through customer or agent identifiers, can produce misleading results.
• Human escalation is an operational safeguard, not merely a fallback for technical failure.

12. Limitations, Governance, and Best Practices

Statistical language models learn correlations rather than guaranteed meanings or intentions. They may reproduce spelling, demographic, political, or cultural biases in training data. A low perplexity model may generate fluent but factually false text. Similarity systems can also amplify historical stereotypes by treating biased associations as semantic evidence.

Professional practice should include:

• documenting corpus sources, dates, licenses, and demographic coverage;
• separating training, validation, and test data by source or time when appropriate;
• reporting tokenization and vocabulary policies with perplexity;
• testing subgroup performance and failure modes;
• masking personal and confidential information;
• preserving audit logs and model-version records;
• using human review for high-impact decisions;
• monitoring distribution shift after deployment;
• avoiding the use of proxies for protected attributes without justification;
• evaluating both utility and social risk.

Data protection laws and sector-specific regulations may apply when text contains health, financial, employment, educational, or identity information. Consent, purpose limitation, retention controls, access governance, and explainability should be considered at system-design time rather than added after deployment.

13. Contemporary Developments

N-gram models remain useful for constrained systems, baselines, decoding components, and settings with limited computational resources. Neural language models extend statistical modelling through distributed representations and learned long-range dependencies. Subword tokenization reduces unknown-word problems and handles morphology and rare names. Retrieval-augmented systems combine language generation with external evidence, but retrieval quality, citation correctness, and privacy remain open concerns.

For postgraduate practice, the important methodological lesson is not that one representation replaces all others. Sparse representations offer transparency and efficient linear learning; dense representations offer parameter sharing and semantic generalization; contextual models offer dependence on usage context. A sound system selects among them according to task requirements, data conditions, computational budget, evaluation evidence, and governance obligations.

Chapter Summary

A statistical language model assigns probabilities to token sequences and predicts the next token using observed linguistic regularities. The chain rule is exact, but n-gram models approximate it by restricting context to the previous n−1 tokens. Maximum-likelihood estimation is simple but assigns zero probability to unseen events, making smoothing essential. Additive smoothing, backoff, interpolation, and Kneser–Ney methods address sparsity in different ways.

Cross-entropy and perplexity evaluate predictive performance on held-out data, but comparisons require consistent tokenization and vocabulary policies. Bag-of-words provides a transparent sparse representation, while TF–IDF emphasizes terms that distinguish documents. Cosine similarity, Euclidean distance, and Jaccard similarity quantify different notions of closeness. Feature engineering can incorporate lexical, character, syntactic, semantic, discourse, and domain-specific information.

Distributional semantics represents words through their contexts. Count-based methods such as PMI and PPMI and predictive embeddings capture useful semantic regularities, although all representations inherit properties and biases from their data. The curse of dimensionality explains why increasing vocabulary and context creates sparsity and estimation difficulty. Effective NLP practice therefore combines careful representation choice, valid evaluation, leakage prevention, privacy protection, and continuous monitoring.

Key Terms

Language model: A model that assigns probabilities to sequences or predicts tokens from context.

Token: A unit produced by tokenization, such as a word, punctuation mark, or subword.

Vocabulary: The set of token types recognized by a model.

N-gram: A contiguous sequence of n tokens.

History: The contextual tokens conditioned on when predicting a token.

Maximum-likelihood estimation: Estimating probabilities from relative observed frequencies.

Sparsity: The condition in which most possible events or vector entries are absent or zero.

Smoothing: Adjusting probability estimates to assign reasonable mass to unseen or rare events.

Backoff: Using a lower-order model when a higher-order event is unavailable.

Interpolation: Combining probability estimates from multiple model orders.

Cross-entropy: Average negative log probability assigned to held-out tokens.

Perplexity: The exponential transform of cross-entropy, representing predictive uncertainty.

Bag-of-words: A document representation based on term counts while ignoring order.

TF–IDF: A weighting scheme combining within-document frequency and corpus rarity.

Cosine similarity: The dot-product similarity of normalized vectors.

Jaccard similarity: Intersection size divided by union size for two sets.

Distributional semantics: The study of meaning through patterns of contextual use.

PMI: Pointwise mutual information measuring association beyond independent occurrence.

Embedding: A learned dense vector representation of a linguistic unit.

Data leakage: Use of information during training or feature selection that would not be available at deployment.

Review Questions

Short Answer Questions

1. What independence or limited-context assumption does a trigram model make?
2. Why does an unseen n-gram cause a problem for unsmoothed maximum-likelihood estimation?
3. Define perplexity and state what a lower value generally indicates.
4. Why does TF–IDF reduce the importance of terms appearing in many documents?
5. What information is lost by a standard bag-of-words representation?

Long Answer Questions

1. Derive the bigram maximum-likelihood estimate and explain the role of sentence-boundary markers.
2. Compare additive smoothing, backoff, and interpolation, including their practical trade-offs.
3. Explain the curse of dimensionality in n-gram models and sparse document representations.
4. Compare count-based distributional semantics with predictive word embeddings.
5. Design an evaluation plan for a text-classification system that reports both predictive performance and governance risks.

Analytical / Application Questions

1. A document collection contains 1,000 documents. A term occurs 20 times in one document and appears in 50 documents overall. Using normalized term frequency with document length 200 and unsmoothed natural-log IDF, calculate its TF–IDF weight.
2. Two support messages are represented by x=(1,0,2) and y=(2,1,1). Calculate their cosine similarity and explain whether cosine or Euclidean distance is more appropriate if message length varies substantially.
3. An n-gram classifier performs well overall but fails on rare cancellation requests and on misspelled product names. Propose a feature and evaluation strategy that addresses both problems without introducing data leakage.

Multiple Choice Questions with Answers

Question 1

Which expression is the maximum-likelihood estimate for a bigram probability?

A. C(w_i) ÷ C(w_i−1)

B. C(w_i−1,w_i) ÷ C(w_i−1)

C. C(w_i−1) ÷ C(w_i−1,w_i)

D. C(w_i) ÷ |V|

Correct Answer: B. C(w_i−1,w_i) ÷ C(w_i−1)

Question 2

What is the main purpose of smoothing in an n-gram language model?

A. To remove all rare words from the vocabulary

B. To force every word to have equal probability

C. To assign non-zero probability to plausible unseen events

D. To increase the order of the model automatically

Correct Answer: C. To assign non-zero probability to plausible unseen events

Question 3

If a test sequence has cross-entropy H = 3 bits per token, what is its perplexity?

A. 3

B. 6

C. 8

D. 16

Correct Answer: C. 8

Question 4

Which statement best describes a bag-of-words representation?

A. It preserves complete syntactic structure

B. It represents documents using term information while generally ignoring word order

C. It assigns one contextual vector to every word occurrence

D. It can represent only documents with equal lengths

Correct Answer: B. It represents documents using term information while generally ignoring word order

Question 5

Under the unsmoothed IDF definition ln(N ÷ df_t), what is the IDF of a term appearing in every document?

A. 0

B. 1

C. N

D. Undefined in all cases

Correct Answer: A. 0

Question 6

What does cosine similarity primarily compare?

A. The lengths of two vectors only

B. The orientation of two non-zero vectors

C. The number of documents in a corpus

D. The order of tokens in two sentences

Correct Answer: B. The orientation of two non-zero vectors

Question 7

Which representation is most directly designed to capture spelling variation through local character patterns?

A. Character n-grams

B. Document length alone

C. Unsmoothed unigram probabilities only

D. A document identifier

Correct Answer: A. Character n-grams

Question 8

Why can increasing n in an n-gram model worsen estimation reliability?

A. It eliminates all contextual information

B. It makes the vocabulary empty

C. It creates many possible contexts with insufficient observed counts

D. It guarantees every probability becomes one

Correct Answer: C. It creates many possible contexts with insufficient observed counts

Question 9

What is the central idea of distributional semantics?

A. Words have meaning independently of any context

B. Words occurring in similar contexts tend to have related meanings

C. Only word frequency determines meaning

D. Document order is always more important than context

Correct Answer: B. Words occurring in similar contexts tend to have related meanings

Question 10

Which practice helps prevent feature-selection leakage?

A. Selecting features using the complete dataset before splitting

B. Selecting features using training data only and applying the selection to held-out data

C. Selecting features from the test labels

D. Using customer identifiers as features

Correct Answer: B. Selecting features using training data only and applying the selection to held-out data