2. Language Modeling and NLP Foundations

Introduction and Context

Generative artificial intelligence systems that produce text are built on a long sequence of developments in natural language processing (NLP). Before a model can generate a coherent answer, it must represent language computationally, identify regularities in sequences, estimate relationships among words, and select likely continuations. These operations are collectively described by the term language modeling.

A language model assigns probabilities to linguistic sequences. Given a sequence of tokens, it estimates what token is likely to occur next, or it evaluates how probable the entire sequence is. Modern large language models (LLMs) extend this basic idea through massive datasets, neural networks with billions of parameters, attention mechanisms, and large-scale pretraining. Nevertheless, their operation remains connected to foundational NLP concepts: preprocessing, tokenization, embeddings, sequence modeling, context, objective functions, and decoding.

This chapter establishes the conceptual vocabulary required for studying LLMs. It also explains why design decisions made at apparently basic stages—such as tokenization or context selection—have substantial effects on model cost, accuracy, fairness, and usability.

Learning Objectives

After completing this chapter, students should be able to:

1. Explain the purpose and stages of text preprocessing.

2. Compare word-level, character-level, and subword tokenization.

3. Distinguish token IDs, embeddings, contextual representations, and positional information.

4. Describe n-gram, recurrent, convolutional, and Transformer-based sequence models.

5. Explain self-attention and its role in modern language models.

6. Compare masked-language-modeling and causal-language-modeling objectives.

7. Explain context windows, perplexity, sampling, temperature, top-k, and top-p decoding.

8. Analyze practical, technical, ethical, and organizational consequences of language-model design choices.

From Text to Computable Sequences

Human language is continuous, variable, and highly dependent on context. A single concept may be expressed through different words, spellings, grammatical structures, or languages. Computers, however, require discrete symbols and numerical representations. NLP therefore includes a transformation pipeline that converts raw text into structured numerical input.

Text preprocessing

Preprocessing prepares text for analysis or model training. Typical operations include Unicode normalization, handling unusual whitespace, identifying document boundaries, removing or preserving markup, and filtering corrupted or duplicated records. Earlier NLP systems often applied stemming, which reduces words to approximate roots, or lemmatization, which maps words to dictionary forms. For example, “studies,” “studied,” and “studying” may be related to the lemma “study.”

Large language models do not always benefit from aggressive normalization. Capitalization, punctuation, formatting, code syntax, and spelling variation may carry meaning. Consequently, preprocessing must be aligned with the task. Removing punctuation may help a narrow statistical analysis but damage a code-generation system or a sentiment classifier. Data cleaning is therefore not simply deletion; it is a modeling decision involving information preservation, quality control, and risk management.

Tokenization

Tokenization divides text into units called tokens. A token may be a word, character, punctuation mark, or subword fragment. The tokenizer maps each token to an integer ID from a fixed vocabulary. For example, a simplified tokenizer might represent “language models learn” as [421, 88, 135]. The integers themselves have no semantic meaning; they are addresses used to retrieve learned vectors.

Word-level tokenization is easy to interpret but produces very large vocabularies and struggles with rare words, spelling variations, and new terms. Character-level tokenization can represent any string and handles novel words well, but sequences become long and learning long-range linguistic structure becomes more difficult.

Subword tokenization offers a compromise. Frequent words may be represented as single tokens, whereas rare or morphologically complex words are split into reusable fragments. A word such as “unhappiness” might be divided into “un,” “happy,” and “ness,” depending on the vocabulary and algorithm. Common methods include Byte Pair Encoding (BPE), WordPiece, and Unigram language-model tokenization. Modern systems may use byte-level approaches, which ensure that arbitrary text can be represented even when it includes unusual symbols.

Tokenization affects computational cost because attention operates over token positions rather than necessarily over words. A language with a less efficient tokenization scheme may require more tokens to express the same content. This has consequences for context-window usage, inference latency, pricing, and multilingual performance.

Representing Meaning: IDs, Embeddings, and Context

A token ID is a discrete index, not a meaningful numerical measurement. Neural models require dense vectors, so each token ID is mapped to an embedding vector. An embedding is a point in a high-dimensional space learned so that linguistic regularities can be represented geometrically. Tokens used in related contexts often acquire related representations, although the exact interpretation of dimensions is not usually transparent.

It is important to distinguish three related ideas. A static embedding assigns one vector to a token type regardless of its sentence. A contextual representation changes according to surrounding tokens. The word “bank,” for example, should receive different internal representations in “river bank” and “commercial bank.” A token embedding is therefore an initial representation, while the hidden states produced by successive neural layers become increasingly contextual.

Position also matters. The sequences “dog bites man” and “man bites dog” contain the same words but express different propositions. Since attention mechanisms do not inherently know order, models incorporate positional information. This may be added to token embeddings through learned position vectors or fixed mathematical functions, or represented through mechanisms such as rotary positional embeddings. Position encoding allows the model to distinguish the first, second, and later locations in a sequence.

Sequence Modeling Before the Transformer

A language model can be defined probabilistically through the chain rule:

P(x₁, x₂, ..., xₙ) = P(x₁)P(x₂|x₁)P(x₃|x₁,x₂)...P(xₙ|x₁,...,xₙ₋₁).

This decomposition means that a model can estimate a sequence by predicting one token at a time. An n-gram model simplifies the problem by conditioning on only the previous n−1 tokens. A trigram model, for instance, estimates P(xₜ|xₜ₋₂,xₜ₋₁). N-gram models are relatively interpretable and computationally simple, but they suffer from sparsity and limited context. Many valid sequences are absent from the training data, and increasing n rapidly increases the number of possible combinations.

Neural sequence models address sparsity by learning distributed representations. Recurrent neural networks (RNNs) process tokens sequentially and maintain a hidden state intended to summarize previous information. Long short-term memory (LSTM) and gated recurrent unit (GRU) architectures use gates to regulate information flow and reduce, though not eliminate, vanishing-gradient problems.

RNNs have two important limitations. First, sequential processing restricts training parallelism. Second, information from distant positions must pass through repeated state updates, making long-range dependencies difficult to preserve. Convolutional sequence models can process local patterns in parallel, but they also require multiple layers or special mechanisms to connect distant tokens.

These limitations motivated the Transformer architecture, whose central operation is attention.

Attention as Context Selection

Self-attention allows each token position to construct a representation by selectively relating itself to other positions in the same sequence. In the standard formulation, each input produces a query, key, and value vector. Attention weights are computed from the compatibility between a query and keys, commonly using scaled dot products:

Attention(Q,K,V) = softmax(QKᵀ/√dₖ)V.

The query represents what a position is seeking, the key represents what another position offers for matching, and the value contains information to be aggregated. The softmax converts compatibility scores into weights that sum to one. A token can therefore assign greater importance to contextually relevant positions.

For example, in “The report that the analysts prepared was published,” the representation of “was” may benefit from attending to “report” rather than the intervening plural noun “analysts.” Attention does not guarantee grammatical reasoning, but it provides a flexible mechanism for modeling such relationships.

Transformers use multi-head attention, in which several attention operations learn different relational patterns. One head may emphasize local syntax, another long-distance agreement, and another recurring semantic associations, although such interpretations must be established empirically rather than assumed. Transformer blocks also include feed-forward networks, residual connections, normalization, and, in many implementations, causal masks.

A causal mask prevents a position from attending to future tokens during autoregressive training. Without this restriction, the model could simply inspect the answer it is supposed to predict. Encoder-style Transformers may instead allow bidirectional attention because their objective is often to reconstruct masked input tokens.

Pretraining Objectives and Model Families

A pretraining objective determines what statistical task the model repeatedly solves before fine-tuning or deployment. The objective strongly influences the model’s representations and generation behavior.

Causal language modeling

In causal language modeling, the model predicts the next token from preceding tokens:

L = −Σₜ log P(xₜ | x₁,...,xₜ₋₁).

This objective is naturally suited to text generation. During training, the correct previous tokens are generally supplied at every position, a procedure called teacher forcing. During generation, however, the model must use its own previously generated tokens. This difference is known as exposure bias and can contribute to error accumulation.

Masked language modeling

Masked language modeling hides selected input tokens and asks the model to predict them using both left and right context. It encourages bidirectional representations and is useful for language understanding tasks. However, because the model is not trained directly to generate an entire sequence from left to right, additional adaptation may be needed for open-ended generation.

Sequence-to-sequence objectives

Encoder-decoder models encode an input sequence and generate a target sequence. They are particularly useful for translation, summarization, and instruction transformation. The encoder can examine the source bidirectionally, while the decoder generates the target causally and may attend to the encoded source through cross-attention.

Training usually minimizes cross-entropy, the negative log probability assigned to the correct target tokens. Optimization reduces average predictive error, but it does not guarantee factuality, safety, or usefulness. A model may achieve a low loss by learning frequent patterns while retaining serious weaknesses on rare, ambiguous, or socially sensitive cases.

Context Windows and Computational Boundaries

A context window is the maximum number of tokens a model can process as one input sequence, including prompts, retrieved documents, conversation history, and sometimes the generated response. It is not equivalent to memory in the human sense. Information outside the window is unavailable unless it is summarized, retrieved, or stored through an external system.

Longer context windows can support larger documents and extended conversations, but they increase computational and memory demands. They also do not ensure that every included detail receives equal attention. Information may be overlooked, especially when relevant evidence is buried among distracting material. Effective systems therefore combine context-window capacity with retrieval, ranking, summarization, chunking, and explicit citation practices.

Decoding: Turning Probabilities into Text

At each generation step, a language model produces a probability distribution over the vocabulary. Decoding converts that distribution into an actual token sequence.

Greedy decoding selects the highest-probability token at every step. It is simple and reproducible but may produce repetitive or locally optimal text. Beam search maintains several candidate sequences and is useful in some structured generation tasks, although it can also favor generic outputs.

Temperature rescales logits before the softmax. Lower temperature makes the distribution sharper and more deterministic; higher temperature increases variation. Temperature does not create knowledge absent from the model. Top-k sampling restricts selection to the k most probable tokens. Top-p sampling, or nucleus sampling, selects from the smallest set whose cumulative probability reaches p. Top-p adapts the candidate set to the uncertainty of each step.

Decoding is a product and governance decision, not merely a technical parameter. Creative writing may tolerate higher variation, while legal drafting, database queries, or regulated communication generally require constrained generation, validation, and human review.

Practical and Professional Applications

Language-model foundations inform many organizational systems. Search and question-answering applications use tokenization, embeddings, retrieval, and context construction to locate relevant evidence. Translation systems rely on sequence modeling and cross-attention. Coding assistants model programming tokens, where punctuation and indentation may be semantically significant. Classification and sentiment systems use contextual representations rather than relying only on word counts.

In enterprise deployment, technical teams must decide how documents are chunked, how many tokens are placed in prompts, whether retrieval results are trustworthy, and how outputs are validated. Product managers must balance response quality against latency and inference cost. Risk and compliance teams must consider data leakage, copyright, privacy, bias, auditability, and the consequences of incorrect generation.

Illustrative Case Study: A University Policy Assistant

Background

A university develops an assistant that answers questions about examination rules, attendance, assessment deadlines, and academic appeals. The system is intended for students and administrative staff.

Problem

An initial prototype uses a general language model with a large prompt containing selected policy documents. It produces fluent answers but occasionally cites an outdated regulation, merges rules from different faculties, or answers confidently when the policy is silent.

Analysis

The problem is not solved merely by increasing model size. Document versions were not tracked, the context contained irrelevant material, and the generation process did not require evidence. In addition, token limits forced some documents to be truncated. The system treated linguistic plausibility as if it were institutional authority.

Approach

The university introduces a retrieval-augmented workflow. Official documents are versioned and divided into semantically coherent chunks. Queries are embedded and matched against the document collection. Retrieved passages are ranked, inserted into a bounded context, and accompanied by metadata such as faculty, effective date, and policy owner. The model is instructed to answer only from retrieved evidence, state when evidence is insufficient, and provide document references. A rule-based layer handles dates and eligibility calculations, while complex or disputed cases are escalated to staff.

Outcome

The revised system is less likely to provide unsupported answers and makes its sources visible. It is not treated as an autonomous decision-maker; staff retain responsibility for formal interpretations. Evaluation includes retrieval accuracy, citation correctness, refusal quality, latency, and performance across faculties.

Learning Points

The case demonstrates that tokenization, context construction, objective design, and decoding are connected to governance. A fluent output is not necessarily a reliable output. High-quality language applications require authoritative data, explicit uncertainty handling, evaluation, and human accountability.

Challenges, Risks, and Common Mistakes

A frequent mistake is assuming that preprocessing is universally beneficial. Removing case, punctuation, or formatting can destroy useful information. Another is treating embeddings as direct explanations of meaning; geometric similarity is useful but does not establish causation or truth.

Language models also inherit biases and omissions from data. Tokenizers may represent some languages, scripts, dialects, or names less efficiently than others. Rare words may be split into many tokens, increasing cost and reducing effective context. Training data may contain personal information, copyrighted material, malicious instructions, or duplicated text.

Evaluation based only on loss or benchmark accuracy is inadequate. Practical evaluation should include robustness, calibration, factuality, safety, subgroup performance, latency, cost, and human usefulness. Prompt injection can manipulate systems that retrieve external content, while overly long prompts may introduce distraction or conflicting instructions.

Ethical and Governance Considerations

Language systems should be designed with data minimization, lawful data handling, access control, and retention policies. Sensitive applications require clear responsibility for decisions, mechanisms for correction, and records sufficient for audit. Users should know when they are interacting with an automated system and should not be misled about its certainty or sources.

Fairness requires testing across languages, demographic groups, dialects, and accessibility needs. Copyright and licensing obligations affect both training data and retrieved documents. In regulated settings, human oversight must be substantive rather than a nominal approval step. Governance should address the entire pipeline—from collection and tokenization to deployment, monitoring, and retirement.

Current Developments

Current research is extending context windows, improving multilingual and multimodal tokenization, and developing more efficient attention mechanisms. Retrieval-augmented generation connects models to changing external knowledge. Parameter-efficient adaptation allows organizations to specialize models without updating every parameter. Tool use and structured decoding constrain outputs to APIs, schemas, or executable formats.

Other important directions include evaluation of long-context reliability, interpretability of attention and hidden states, uncertainty estimation, synthetic-data quality, and energy-efficient inference. These developments do not eliminate foundational issues; they make understanding those foundations more important because system behavior still depends on representations, objectives, context, and decoding.

Best Practices for Language-Model Systems

1. Define the task and acceptable error before selecting a model.

2. Preserve meaningful formatting and document provenance during preprocessing.

3. Measure tokenization efficiency across languages and user populations.

4. Distinguish token IDs, embeddings, and contextual representations in system design.

5. Match the pretraining or adaptation objective to the intended use.

6. Treat context construction as an information-retrieval problem, not merely prompt writing.

7. Select decoding controls according to risk and required variability.

8. Evaluate factuality, robustness, bias, safety, cost, and latency together.

9. Use retrieval, validation, tools, and human review where model knowledge is insufficient.

10. Monitor deployed systems and revise data, policies, and evaluation as conditions change.

Chapter Summary

Language modeling represents language as probability distributions over token sequences. Text must be cleaned and tokenized before it can be processed, but preprocessing should preserve information relevant to the task. Token IDs are mapped to embeddings, enriched with positional information, and transformed into contextual representations. N-gram and recurrent models established important principles, while Transformers enabled highly parallel sequence modeling through self-attention.

Causal, masked, and sequence-to-sequence objectives create different capabilities and use cases. Context windows define the information available at inference, while decoding methods determine how probability distributions become outputs. In professional systems, these technical choices affect cost, reliability, multilingual performance, privacy, fairness, and accountability. Effective language-model applications therefore combine model knowledge with retrieval, validation, governance, and human judgment.

Key Terms

Tokenization: The process of dividing text into computational units called tokens.

Vocabulary: The finite set of tokens recognized by a tokenizer.

Token ID: An integer index assigned to a token in a vocabulary.

Embedding: A learned dense vector representing a token or other object.

Contextual representation: A representation whose value depends on surrounding tokens.

Positional encoding: Information that enables a model to represent token order.

Language model: A model that assigns probabilities to sequences or predicts tokens.

N-gram model: A model that predicts a token using a limited preceding history.

RNN: A recurrent neural network that processes sequential inputs through hidden states.

LSTM: A gated recurrent architecture designed to preserve information across sequences.

Transformer: A neural architecture based primarily on attention and feed-forward layers.

Self-attention: An operation allowing positions in a sequence to weigh information from other positions.

Causal mask: A restriction preventing a position from attending to future tokens.

Causal language modeling: Predicting each token from preceding tokens.

Masked language modeling: Predicting hidden tokens using their surrounding context.

Context window: The maximum token sequence processed in one model interaction.

Perplexity: An exponential transformation of average negative log-likelihood used to assess predictive uncertainty.

Temperature: A decoding parameter controlling the sharpness of the output probability distribution.

Top-p sampling: Sampling from the smallest token set whose cumulative probability reaches a chosen threshold.

Retrieval-augmented generation: A workflow that supplies retrieved external information to a generative model.

Review Questions

Short Answer Questions

1. Why is a token ID not itself a semantic representation?

2. What problem does positional encoding solve in a Transformer?

3. How does causal language modeling differ from masked language modeling?

4. Define a context window and explain one practical consequence of its limit.

5. What is the purpose of temperature in decoding?

Long Answer Questions

1. Explain the progression from n-gram models and RNNs to Transformers, emphasizing the limitations addressed at each stage.

2. Describe the complete path from raw text to contextual representations in a modern language model.

3. Explain self-attention using queries, keys, values, scaling, and softmax weighting.

4. Compare greedy decoding, beam search, top-k sampling, and top-p sampling, including suitable applications.

5. Discuss how tokenization, context limits, and pretraining objectives influence multilingual and enterprise language applications.

Analytical / Application Questions

1. A legal assistant frequently produces fluent but unsupported answers from outdated documents. Diagnose the technical and governance causes and propose a revised architecture.

2. A model performs well on English but consumes far more tokens and performs poorly on a morphologically rich language. Analyze possible tokenization causes and recommend evaluation measures.

3. Design a decoding and validation strategy for a system that generates database queries, explaining how your choices would differ from those for creative writing.

22. Multiple Choice Questions with Answers

Question 1

Why are subword tokenizers commonly used in modern language models?

A. They guarantee that every token corresponds to a complete word

B. They balance vocabulary size with the ability to represent rare and novel words

C. They eliminate the need for embeddings

D. They ensure that all languages use the same number of tokens

Correct Answer: B. They balance vocabulary size with the ability to represent rare and novel words

Question 2

What is the primary role of positional information in a Transformer?

A. To convert token IDs into probabilities

B. To prevent the model from using embeddings

C. To represent the order of tokens in a sequence

D. To select the highest-probability output token

Correct Answer: C. To represent the order of tokens in a sequence

Question 3

In causal language modeling, a token at position t is trained to be predicted from:

A. Only future tokens

B. The entire sequence, including the target token

C. The preceding tokens, subject to a causal restriction

D. A separately retrieved database only

Correct Answer: C. The preceding tokens, subject to a causal restriction

Question 4

Which problem is self-attention particularly designed to address compared with strictly sequential recurrent processing?

A. The inability to represent any vocabulary

B. Limited parallelism and difficulty connecting distant positions

C. The need to assign integer IDs to tokens

D. The absence of output probability distributions

Correct Answer: B. Limited parallelism and difficulty connecting distant positions

Question 5

A system must answer from current university regulations and cite the relevant source. Which design is most appropriate?

A. Use greedy decoding without supplying documents

B. Use retrieval-augmented generation with versioned documents and citations

C. Remove policy metadata to reduce context size

D. Increase temperature so the model explores more regulations

Correct Answer: B. Use retrieval-augmented generation with versioned documents and citations

Question 6

What happens when temperature is lowered during decoding?

A. The output distribution generally becomes sharper and more deterministic

B. The model receives a larger context window

C. Future tokens become visible during causal training

D. The vocabulary is automatically expanded

Correct Answer: A. The output distribution generally becomes sharper and more deterministic

Question 7

Which statement best distinguishes an embedding from a contextual representation?

A. An embedding is always a probability distribution, while a contextual representation is an integer

B. An embedding is an initial vector, whereas a contextual representation reflects surrounding tokens

C. An embedding contains document citations, whereas a contextual representation does not

D. There is no conceptual distinction between them

Correct Answer: B. An embedding is an initial vector, whereas a contextual representation reflects surrounding tokens

Question 8

Why can a longer context window fail to guarantee better answers?

A. Models cannot process more than one token at a time

B. Additional content may be irrelevant, distracting, or overlooked

C. Longer contexts remove all positional information

D. Longer contexts prevent retrieval from being used

Correct Answer: B. Additional content may be irrelevant, distracting, or overlooked

Question 9

Which objective is most directly aligned with predicting missing tokens using both left and right context?

A. Masked language modeling

B. Causal language modeling

C. Greedy decoding

D. Top-k sampling

Correct Answer: A. Masked language modeling

Question 10

For a database-query generator, which approach is generally most appropriate?

A. Use unrestricted high-temperature sampling and no validation

B. Use controlled decoding, structured output constraints, and query validation

C. Remove punctuation during preprocessing

D. Select outputs solely by their length

Correct Answer: B. Use controlled decoding, structured output constraints, and query validation