Sequence Models for Language

Introduction

Language is sequential: the interpretation of a word depends partly on the words that precede and follow it. In “the bank approved the loan,” bank is likely a financial institution; in “the boat reached the bank,” it refers to a river edge. A language model therefore needs mechanisms for representing order, context, variable-length input, and dependencies that may extend across many words.

Sequence models are neural architectures designed for ordered data. In natural language processing (NLP), they support text classification, language modelling, named-entity recognition, speech recognition, machine translation, summarisation, and dialogue systems. This chapter develops recurrent neural networks (RNNs), long short-term memory networks (LSTMs), gated recurrent units (GRUs), bidirectional models, encoder–decoder systems, attention, and sequence-to-sequence learning.

Learning Objectives

By the end of this chapter, you should be able to:

• explain how recurrent networks process variable-length sequences;
• identify vanishing- and exploding-gradient problems;
• implement and compare RNN, LSTM, and GRU layers;
• explain when bidirectional processing is appropriate;
• design encoder–decoder and sequence-to-sequence systems;
• describe attention and calculate attention weights;
• evaluate sequence models using appropriate metrics and responsible deployment practices.

1. Representing Language as a Sequence

A sentence is commonly tokenised into a sequence x₁, x₂, …, x_T, where T is the number of tokens. Each token is mapped to an integer and then to an embedding vector e_t ∈ ℝ^d. The embedding dimension d is selected during model design. Padding allows sentences in the same mini-batch to have a common length; a mask prevents padding symbols from contributing to the loss or hidden-state computation.

A vocabulary of size V can be represented using one-hot vectors, but one-hot vectors are sparse and do not express similarity. An embedding matrix E ∈ ℝ^V×d maps token index i to row E_i. Embeddings learned jointly with a task often capture useful semantic and syntactic regularities, although they can also reproduce biases in training data.

A language model estimates the probability of a token sequence using the chain rule:

The model predicts the next token from preceding context. During training, the target at position t is normally the next token, so input and target sequences are shifted by one position. A special beginning-of-sequence token and end-of-sequence token are useful for generation.

2. The Recurrent Neural Network

An RNN maintains a hidden state that summarises previous inputs. At time t, the standard or “vanilla” RNN is:

Here x_t is the input vector, h_t is the hidden state, h₀ is an initial state (often zero), W_xh, W_hh, and W_ho are learned weight matrices, b_h and b_o are biases, and o_t is an output logit vector. For classification over C classes, probabilities are obtained with softmax:

The softmax equation applies when the output classes are mutually exclusive. The cross-entropy loss for a one-hot target y_t is:

For a sequence prediction task, the loss is usually summed or averaged over valid, non-padding positions. Teacher forcing supplies the correct previous token during training, rather than the model’s own sampled token.

The same parameters are reused at every time step. This makes RNNs compact and able to process arbitrary sequence lengths. However, unrolling an RNN reveals a long chain of derivatives. If the recurrent Jacobian has a dominant magnitude below one, gradients tend to shrink; if it is above one, gradients may grow. These are the vanishing-gradient and exploding-gradient problems. Vanishing gradients make long-range learning difficult, while exploding gradients can make optimisation unstable. Gradient clipping limits the norm of a gradient, for example:

where g is the gradient, ||g|| its chosen norm, and τ a positive clipping threshold. Clipping does not solve vanishing gradients, but it can improve stability.

3. LSTM Networks

The long short-term memory (LSTM) architecture introduces a cell state and gates. Gates use the sigmoid function, whose values lie between zero and one, to regulate information flow:

The input gate i_t controls candidate information ĝ_t, the forget gate f_t controls retained cell state, and the output gate o_t controls exposure of the cell state as hidden state. The symbol ⊙ denotes element-wise multiplication. All gates and states have the same hidden dimension. Biases, including a frequently positive forget-gate bias, are learned parameters.

The additive cell-state update creates a relatively direct path for information and gradients. LSTMs therefore often learn longer dependencies than vanilla RNNs, at the cost of more parameters and computation.

4. GRU Networks

The gated recurrent unit (GRU) combines gating functions with a simpler state structure:

The update gate z_t controls replacement of the previous state, while the reset gate r_t controls how much previous state contributes to the candidate. Some software libraries use the complementary convention h_t = z_t ⊙ h_t-1 + (1−z_t) ⊙ ĝ_t; the equations must therefore be checked against the implementation.

GRUs have fewer gates and parameters than LSTMs. They can train faster and may perform similarly, but there is no universally best choice. Sequence length, dataset size, compute budget, and validation performance should guide selection.

5. Bidirectional Recurrent Models

A unidirectional model uses only past context. A bidirectional model combines a forward recurrence with a backward recurrence:

where →h_t processes tokens from left to right, ←h_t processes them from right to left, and semicolon denotes concatenation. The output dimension is therefore usually twice the individual direction’s hidden dimension.

Bidirectional models are valuable when the complete input is available before prediction, as in document classification, offline named-entity recognition, and part-of-speech tagging. They are unsuitable for strictly causal real-time prediction because the backward direction uses future tokens. A bidirectional encoder can, however, be used in a translation system whose entire source sentence is available before decoding begins.

6. Encoder–Decoder and Sequence-to-Sequence Learning

A sequence-to-sequence (seq2seq) system maps an input sequence of length T_x to an output sequence of length T_y, where the lengths may differ. The encoder reads the source and produces representations. The decoder generates the target one token at a time.

In the simplest architecture, the final encoder state is passed to the decoder as its initial state. At decoder step t, the conditional probability is:

where s_t is the decoder state and y_<t denotes previously generated target tokens. The sequence likelihood factorises as:

Training commonly uses teacher forcing: the true previous target token is provided to predict the next token. At inference, the model must use its own previous prediction. This mismatch is called exposure bias and may cause errors to accumulate.

Greedy decoding selects the highest-probability token at each step. Beam search keeps a fixed number B of partial hypotheses and expands them at every step. Beam search can improve sequence likelihood, but a larger beam does not guarantee better human quality and increases computation. Decoding must stop at an end-of-sequence token or a maximum length.

7. Attention Mechanisms

A fixed final encoder state can create an information bottleneck, particularly for long inputs. Attention allows the decoder to access all encoder states. Let encoder states be h₁, …, h_Tx. At decoder step t, an alignment score compares decoder state s_t-1 with each encoder state:

A common additive score is:

Attention weights are a softmax over source positions:

Thus, for every decoder step, α_t,i ≥ 0 and ∑_i α_t,i = 1. The context vector is the weighted sum:

The decoder uses c_t together with its recurrent state to predict the next token. Dot-product or multiplicative attention uses a score such as e_t,i = s_t-1^T W h_i. When dimensions match, scaled dot-product attention divides by √d_k to reduce excessively large logits:

This equation is the basis of Transformer attention, although Transformers do not rely on recurrence. Attention weights indicate where the model allocated computational focus, but they should not automatically be interpreted as faithful explanations of the model’s reasoning.

Worked Attention Example

Suppose a decoder produces three alignment scores for one step: 2, 1, 0. Their exponentials are approximately 7.389, 2.718, 1. The sum is 11.107, giving attention weights approximately 0.665, 0.245, 0.090. The first encoder position receives most weight, but all positions remain available. If the corresponding scalar encoder values are 4, 2, and 1, the context value is:

The result is a weighted average because the weights sum to one. In an actual network, encoder states are vectors and the same calculation is performed component by component.

8. Implementation Workflow

A practical workflow for an RNN-based NLP task is:

1. Define the task and split data. Separate training, validation, and test data before fitting the vocabulary. For temporal or user-generated data, avoid leakage across splits.
2. Tokenise and normalise carefully. Preserve information such as case, punctuation, and spelling when it is relevant.
3. Build vocabulary and numericalise. Include unknown, padding, beginning, and end tokens where appropriate.
4. Batch variable lengths. Pad sequences and supply masks or packed sequences.
5. Choose the output structure. Use one output for sentence classification or one output per token for tagging; use a decoder for variable-length generation.
6. Train with an appropriate loss. Ignore padding in token-level cross-entropy. Clip gradients when exploding gradients occur.
7. Validate by task metrics. Accuracy may be inadequate for imbalanced classification or generation.
8. Inspect errors. Examine long sentences, rare words, negation, names, dialects, and out-of-domain examples.
9. Save reproducible artefacts. Store the tokenizer, vocabulary, preprocessing rules, model configuration, and version information.

A minimal PyTorch-style design for classification is conceptually:

embedding = Embedding(vocabulary_size, embedding_dimension, padding_idx)
recurrent = LSTM(embedding_dimension, hidden_dimension,
                  bidirectional=True, batch_first=True)
classifier = Linear(2 * hidden_dimension, number_of_classes)

x = embedding(token_ids)
h, state = recurrent(x)
representation = masked_last_valid_state(h, mask)
logits = classifier(representation)
loss = cross_entropy(logits, labels)

The exact API varies by framework. In a bidirectional model, the factor of two reflects concatenated directions. For packed sequences or masks, “last valid state” must refer to each example’s actual length rather than the final padded position.

9. Evaluation

For classification, accuracy is:

Precision, recall, and F1 are useful when classes are imbalanced. For token tagging, report entity-level scores where appropriate rather than only token accuracy.

Language models are often evaluated with cross-entropy or perplexity. If average negative log-likelihood per token is L, perplexity is:

This definition assumes natural logarithms and a clearly specified tokenisation and evaluation set. Lower perplexity indicates better predictive likelihood under the same evaluation conditions, but it does not guarantee factuality, helpfulness, or grammatical quality.

Machine translation and summarisation may use BLEU, ROUGE, chrF, or learned metrics, alongside human evaluation. Automatic metrics can reward surface overlap and miss factual errors. Evaluation should include robustness, subgroup performance, toxicity, privacy leakage, and calibration when the application is consequential.

10. Illustrative Case: Customer-Support Intent and Routing

Background. A fictional online retailer receives short customer messages such as “My parcel arrived damaged” and “I was charged twice.” The company wants to route each message to the correct support team.

Problem. Messages vary in length and contain spelling errors, product names, and ambiguous words. A bag-of-words classifier ignores word order, so “refund not received” and “received refund” can look similar.

Analysis. The team tokenises messages, builds a training vocabulary, pads batches, and creates an embedding layer. It compares a vanilla RNN, GRU, and bidirectional LSTM. Since the complete message is available before routing, bidirectional processing is permissible. The evaluation set is held out by customer and time period to reduce leakage.

Approach. The bidirectional LSTM produces a contextual representation from the final valid forward and backward states. A linear layer maps that representation to intent logits. Weighted cross-entropy is considered because some intents are rare. Validation includes macro-F1, confusion matrices, and error review for negation and unseen product names. A confidence threshold sends uncertain messages to human agents.

Outcome. The model reduces routing time in the illustrative scenario, but it is not approved for autonomous decisions until performance is checked across languages, writing styles, and accessibility-related communication. The system logs model version, input-processing version, confidence, and human overrides.

Learning points. Architecture must match information availability; padding and leakage can invalidate results; macro-F1 can reveal poor minority-class performance; and human escalation is an important safety control.

11. Limitations, Ethics, and Governance

Recurrent models may struggle with very long contexts, rare words, and parallel training because tokens are processed sequentially. LSTMs and GRUs reduce but do not eliminate long-range difficulties. Attention improves access to source information but can be computationally expensive and is not automatically an explanation. Modern Transformer models often outperform recurrent systems on large-scale language tasks through parallel self-attention and large pretraining, although recurrent models remain useful for compact, streaming, low-latency, and resource-constrained applications.

Language data can contain personal information, copyrighted material, stereotypes, and discriminatory associations. Teams should establish a lawful data basis, minimise collection, protect identifiers, control access, document dataset provenance, and define retention periods. Outputs should be tested for memorisation, harmful content, demographic disparities, and security attacks such as prompt or data poisoning where relevant. Human review, audit logs, model cards, incident procedures, and clear user disclosure support accountable deployment.

Best practices include comparing against simple baselines, fixing random seeds where possible, tuning on validation data only, reporting confidence intervals or repeated-run variation when feasible, monitoring drift, and testing on realistic out-of-distribution examples. A model should not be deployed merely because its aggregate score is high.

Chapter Summary

Sequence models represent language as ordered token inputs and estimate conditional probabilities over sequences. Vanilla RNNs reuse parameters through time but can suffer from vanishing and exploding gradients. LSTMs use cell states and input, forget, and output gates; GRUs provide a simpler gated alternative. Bidirectional models combine past and future context and are appropriate only when future input is available. Encoder–decoder architectures map variable-length source sequences to variable-length targets, while teacher forcing makes training efficient but creates exposure bias. Attention computes alignment scores, normalised weights, and context vectors, reducing the fixed-vector bottleneck. Practical success depends on correct masking, leakage-free data splits, suitable metrics, careful decoding, error analysis, and responsible governance.

Key Terms

Token: A unit of text used as a model input.

Embedding: A learned dense vector representation of a token.

Hidden state: A recurrent representation passed between time steps.

Cell state: The LSTM memory pathway that carries information across steps.

RNN: A neural network that reuses a recurrent state over an ordered sequence.

Vanishing gradient: A gradient that becomes very small during backpropagation through time.

Exploding gradient: A gradient whose magnitude becomes excessively large.

Gradient clipping: Rescaling gradients above a selected norm threshold.

LSTM: A gated recurrent architecture with a separate cell state.

GRU: A gated recurrent architecture using reset and update gates.

Bidirectional model: A model that processes a sequence in both directions.

Encoder: The component that converts an input sequence into representations.

Decoder: The component that generates an output sequence.

Teacher forcing: Training with the true previous target token as decoder input.

Attention: A mechanism that weights encoder states for a current prediction.

Context vector: The attention-weighted combination of encoder states.

Beam search: A decoding method that retains several high-scoring partial sequences.

Perplexity: The exponential of average negative log-likelihood per token.

Exposure bias: The difference between training with true history and inference with generated history.

Data leakage: Unintended use of information from evaluation data during training.

Review Questions

Short Answer Questions

1. Why are embeddings generally preferred to one-hot vectors for language models?
2. What is the purpose of the forget gate in an LSTM?
3. Why can a bidirectional model be unsuitable for real-time next-token prediction?
4. Define teacher forcing and state one advantage and one limitation.
5. Why should padding positions be masked during sequence training?

Long Answer Questions

1. Explain the recurrence of a vanilla RNN and discuss vanishing and exploding gradients.
2. Compare LSTM and GRU architectures, including their gates, state mechanisms, and practical trade-offs.
3. Describe a complete encoder–decoder translation system from tokenisation through decoding.
4. Derive the attention-weight and context-vector calculations and explain their role in seq2seq learning.
5. Discuss evaluation, ethical risks, and governance requirements for a language-routing system.

Analytical / Application Questions

1. A classifier receives complete customer messages before prediction. Would you choose a unidirectional or bidirectional recurrent encoder? Justify the choice and identify one situation in which your choice would be inappropriate.
2. An RNN’s training loss becomes unstable and gradients have very large norms. Propose a debugging and improvement plan, distinguishing gradient clipping from solutions to long-range memory problems.
3. Design an evaluation plan for a summarisation model whose ROUGE score improves but whose human reviewers report more factual errors. Which additional tests and deployment controls would you introduce?

Multiple Choice Questions with Answers

Question 1

Which mechanism in an LSTM directly controls how much previous cell state is retained?

A. Input gate

B. Forget gate

C. Output projection

D. Reset gate

Correct Answer: B. Forget gate

Question 2

What is the main reason for masking padding tokens during sequence loss calculation?

A. To increase vocabulary size

B. To make all sequences have equal semantic meaning

C. To prevent artificial padding positions affecting learning

D. To replace the softmax function

Correct Answer: C. To prevent artificial padding positions affecting learning

Question 3

A bidirectional recurrent encoder is generally appropriate when:

A. Only past tokens may be accessed during prediction

B. The complete input sequence is available before prediction

C. The output must always have one token

D. The model cannot use embeddings

Correct Answer: B. The complete input sequence is available before prediction

Question 4

In attention, what do the softmax-normalised alignment scores represent?

A. The vocabulary size

B. Weights assigned to encoder states for a decoder step

C. The number of decoder layers

D. The gradient norm

Correct Answer: B. Weights assigned to encoder states for a decoder step

Question 5

If the average negative log-likelihood is L using natural logarithms, perplexity is:

A. L²

B. log(L)

C. exp(L)

D. 1/L

Correct Answer: C. exp(L)

Question 6

Which statement best describes teacher forcing?

A. The decoder is never given any previous token

B. The true previous target token is supplied during training

C. The encoder processes tokens backward only

D. Gradients are clipped at every time step

Correct Answer: B. The true previous target token is supplied during training

Question 7

What is a principal advantage of a GRU over an LSTM?

A. It always produces more accurate results

B. It uses a simpler gating structure and often fewer parameters

C. It cannot suffer from exploding gradients

D. It requires no hidden state

Correct Answer: B. It uses a simpler gating structure and often fewer parameters

Question 8

In the attention context equation, attention weights normally sum to:

A. 0

B. The sequence length

C. 1

D. The vocabulary size

Correct Answer: C. 1

Question 9

Why is gradient clipping used in recurrent-network training?

A. To control excessively large gradient updates

B. To create bidirectional context

C. To convert tokens into embeddings

D. To guarantee the absence of bias

Correct Answer: A. To control excessively large gradient updates

Question 10

Which metric is especially useful for revealing performance problems on minority classes?

A. Macro-F1

B. Sequence length

C. Vocabulary count

D. Beam width

Correct Answer: A. Macro-F1