Sequence Models and Natural Language Processing

Introduction

Many forms of information are sequential. A sentence is an ordered sequence of words, a stock price is a sequence of observations over time, and a speech signal is a sequence of acoustic measurements. In each case, the meaning of the current input may depend on earlier inputs. Ordinary feed-forward neural networks do not naturally represent this dependence because they process each input independently. Sequence models address this limitation by maintaining information about previous elements and using it to interpret or predict later elements.

Natural language processing (NLP) applies computational methods to human language. Modern NLP systems commonly use dense word representations, recurrent neural networks (RNNs), gated recurrent units (GRUs), long short-term memory networks (LSTMs), and sequence-to-sequence architectures. Although transformer models now dominate many large-scale NLP applications, recurrent models remain important for learning the foundations of sequential computation, for resource-constrained systems, and for many time-series applications.

Learning Objectives

By the end of this chapter, the learner should be able to:

1. Explain why sequential data requires specialised models.
2. Describe tokenisation, vocabulary construction and embeddings.
3. Explain the operation of an RNN and the causes of vanishing and exploding gradients.
4. Compare the structure and behaviour of LSTMs and GRUs.
5. Describe sequence-to-sequence learning and attention at an introductory level.
6. Apply sequence models to text, time-series and other ordered data.
7. Evaluate sequence models using suitable metrics and recognise their limitations and ethical risks.

1. Representing Sequential Data

A sequence can be written as x₁, x₂, …, x_T, where T is the sequence length and x_t is the observation at time step t. In language, x_t may be a token; in a sensor system, it may be a vector of measurements.

A sequence model estimates relationships such as:

• next-step prediction: predict x_t+1 from earlier observations;
• sequence classification: assign one label to the whole sequence;
• sequence labelling: assign a label to every element;
• sequence generation: produce a new sequence;
• sequence transformation: convert one sequence into another.

The ordering of observations is essential. Shuffling the words in “the child opened the door” changes its interpretation. Similarly, changing the order of temperature measurements can invalidate a forecasting model.

Text preprocessing

A typical text pipeline includes:

1. Normalisation: handling case, punctuation, spelling variants and Unicode.
2. Tokenisation: dividing text into words, subwords or characters.
3. Vocabulary construction: assigning an integer identifier to each permitted token.
4. Special tokens: using symbols such as <PAD> for padding, <UNK> for unknown tokens, <BOS> for beginning of sequence and <EOS> for end of sequence.
5. Batch preparation: padding sequences to a common length or using masking.

Word-level tokenisation is simple but produces a large vocabulary and struggles with rare words. Character-level tokenisation handles unknown words but creates longer sequences. Subword tokenisation provides a practical compromise by representing frequent words as units while decomposing rare words into smaller parts.

Padding makes sequences in a batch equal in length. A mask is then used so that padding does not contribute to the loss or hidden-state interpretation. Without masking, the model may learn patterns from artificial padding symbols.

2. Embeddings

A token identifier is categorical, not numerical in a meaningful geometric sense. If “cat” is assigned 14 and “dog” is assigned 15, the numbers do not imply that the words are nearly identical. An embedding maps each token to a dense vector in a continuous space.

Let the vocabulary contain V tokens and let the embedding dimension be d. The embedding matrix is:

If token i is represented by a one-hot vector o_i, its embedding is:

Here, e_i is a vector of length d. In implementation, an embedding lookup directly selects row i of E rather than performing an inefficient multiplication by a one-hot vector.

During supervised training, the embedding values are adjusted through backpropagation. Tokens appearing in similar contexts may develop similar vectors. Similarity is often measured with cosine similarity:

The dot denotes the dot product and ||e|| is the Euclidean norm. The value lies between −1 and 1 for non-zero vectors. Cosine similarity compares direction and is less affected by vector magnitude.

Embeddings may be:

• Randomly initialised and task-trained: learned specifically for the application.
• Pre-trained: learned from a large text collection and then reused.
• Context-independent: one vector represents a token in every context.
• Contextual: the representation changes according to surrounding words.

A limitation of traditional word embeddings is that a word such as “bank” receives one vector whether it refers to a financial institution or a river bank. Contextual models address this ambiguity.

3. The Recurrent Neural Network

An RNN processes one element at a time and carries a hidden state forward. The hidden state is a compact, learned summary of preceding information. A basic RNN is defined by:

Here, x_t is the input vector, h_t is the hidden state, y_t is the output, W_xh, W_hh and W_hy are learned weight matrices, b_h and b_y are bias vectors, and tanh is the hyperbolic tangent activation function. The same weights are reused at every time step. This parameter sharing allows one model to process sequences of different lengths.

For classification, the final hidden state may be passed to a softmax layer. For sequence labelling, an output can be produced at every time step. For regression, the output layer may be linear. For a classification problem with class probabilities p_k, the cross-entropy loss for a true class c is:

For a batch, the usual loss is the mean of the individual losses. The logarithm is natural unless another base is specified.

Backpropagation through time

Training an RNN involves unfolding it across time and applying backpropagation through this unfolded computation. The gradient with respect to an early hidden state contains repeated products involving W_hh and activation derivatives. If these products become very small, the gradient vanishes; if they become very large, it explodes.

Vanishing gradients make it difficult to learn long-range dependencies. Exploding gradients can produce unstable updates and numerical overflow. Common responses include:

• gradient clipping, such as limiting the gradient norm;
• careful parameter initialisation;
• gated architectures such as LSTM and GRU;
• normalisation and suitable learning rates;
• truncated backpropagation through time for very long streams.

If the gradient vector is g and its norm exceeds a threshold τ, global norm clipping replaces it with:

If ||g|| ≤ τ, the gradient is unchanged. The threshold is a hyperparameter and has the same units as the gradient norm.

4. Long Short-Term Memory Networks

An LSTM separates short-term output information from a cell state designed to carry information over longer periods. It uses gates whose values are usually between 0 and 1 because they use the sigmoid activation function.

Here, i_t is the input gate, f_t the forget gate, o_t the output gate, ĉ_t a candidate cell update, c_t the cell state, h_t the hidden state, σ the sigmoid function, and ⊙ element-wise multiplication. The matrices and bias vectors are learned. All vectors in each equation must have compatible dimensions.

The cell update allows information to pass along an approximately additive path. The forget gate can preserve useful information, while the input gate controls new information and the output gate controls exposure to the next layer. LSTMs do not eliminate all training difficulties, but they generally learn long dependencies more effectively than a basic RNN.

5. Gated Recurrent Units

A GRU combines some LSTM functions and uses fewer gates and parameters. A common formulation is:

The update gate z_t controls the balance between the previous state and candidate state. The reset gate r_t controls how strongly the previous state contributes to the candidate. Some software libraries use an equivalent but differently named convention, so the implementation equations should always be checked.

GRUs have no separate cell state and usually require fewer parameters than LSTMs of the same hidden size. They may train faster and perform similarly, but there is no universally best choice. Architecture selection should be based on validation performance, latency, memory and data characteristics.

6. Sequence-to-Sequence Learning

A sequence-to-sequence (seq2seq) model maps an input sequence to an output sequence, possibly of a different length. Applications include machine translation, summarisation, speech processing and conversational response generation.

A conventional architecture contains:

1. Encoder: reads the input and produces hidden representations.
2. Decoder: generates output tokens one at a time.
3. Output layer: converts decoder states into a probability distribution over the target vocabulary.

At training time, the decoder often uses teacher forcing: the correct previous target token is supplied when predicting the next token. The objective for a target sequence y₁,…,y_U is commonly the negative log-likelihood:

Here, x is the source sequence and y_<u denotes target tokens before position u. At inference time, the true previous token is unavailable, so the decoder feeds back its own prediction. This difference can create exposure bias.

A single fixed-size encoder vector can become a bottleneck for long inputs. Attention reduces this problem by allowing the decoder to construct a context vector from all encoder states. If h_s is the encoder state at source position s and α_u,s is an attention weight, then:

The weights are non-negative and normally sum to 1 across source positions for each decoder step. Attention does not necessarily provide a complete causal explanation of a model’s decision; it is a learned information-selection mechanism.

7. Applications Beyond Text

Time-series forecasting

For a univariate series, a sliding window may use the preceding k observations to predict the next value:

For multivariate data, each x_t is a vector containing measurements such as temperature, pressure and flow rate. Important practices include chronological train–validation–test splitting, scaling using training data only, and avoiding future information in features. Randomly splitting time-series observations can cause leakage because near-identical future patterns may appear in training and testing sets.

Common regression metrics include mean absolute error:

and root mean squared error:

Here, y_i is the observed value, ŷ_i the prediction and n the number of test observations. Both metrics have the same units as the target; RMSE penalises large errors more strongly.

Sequential classification

A wearable device may classify an activity from accelerometer windows. The sequence can be represented by the final hidden state, pooled hidden states, or an attention-weighted representation. Window length affects both context and latency.

Event and anomaly detection

An RNN can model normal sequences and identify observations with unusually high prediction error or low likelihood. However, rare legitimate events may also appear anomalous. Thresholds should therefore be selected using operational costs, not only a convenient percentile.

8. Illustrative Case: Predictive Maintenance for an Industrial Pump

Background. An engineering company monitors vibration, temperature and pressure from pumps every minute. Failures are expensive and sometimes hazardous.

Problem. The company wants to predict whether a pump will require maintenance within the next 24 hours. Sensor values have different units, missing readings and occasional outliers.

Analysis. A sequence of the previous 120 minutes is used for each decision. The data is divided chronologically: earlier months for training, a later month for validation and the most recent month for testing. Scaling parameters are calculated only from the training period. Maintenance records are checked to ensure that labels are based on information available at prediction time.

Approach. Each time step contains the scaled sensor vector and a missing-value indicator. A two-layer GRU produces hidden representations, followed by dropout and a sigmoid output for binary classification. The binary cross-entropy loss for one labelled example is:

Here, y is 0 or 1 and p is the predicted probability of maintenance. Class weighting is considered because failures are rare. The decision threshold is selected using the relative cost of missed failures, unnecessary inspections and emergency shutdowns.

Outcome. Suppose the model achieves higher recall than a simple threshold rule while maintaining an acceptable false-alarm rate. The engineering team does not deploy it as an autonomous shutdown controller. Instead, it produces ranked maintenance alerts, displays the relevant time window and requires human review.

Learning points. The model’s value depends on data quality, leakage prevention, threshold selection and integration with maintenance procedures. A high accuracy score could be misleading if failures are rare. Monitoring must continue after deployment because sensor calibration, operating conditions and failure patterns can change.

9. Evaluation, Deployment and Best Practices

For text classification, accuracy is useful when classes are balanced. Precision, recall and F1-score are often more informative for imbalanced tasks. If TP, FP and FN denote true positives, false positives and false negatives:

The denominators must be non-zero. For language generation, perplexity is related to average negative log-likelihood. For a sequence of N tokens with probabilities p_i,

Lower perplexity indicates better predictive likelihood on the evaluation data, but it does not guarantee factuality, usefulness or fairness.

Professional practice includes:

• establish a non-neural baseline;
• keep test data isolated until final evaluation;
• record tokenisation, vocabulary and preprocessing versions;
• use masks for padded sequences;
• inspect errors by class, language variety and input length;
• monitor latency, memory, drift and failure rates;
• calibrate probabilities when decisions depend on risk;
• preserve reproducibility through seeds, configuration files and experiment tracking.

10. Limitations, Ethics and Contemporary Developments

Sequence models can reproduce social stereotypes present in training data. NLP systems may perform worse for minority dialects, low-resource languages or users with atypical communication patterns. Applications involving employment, education, healthcare or policing require special care because errors can cause unequal harm.

Text data may contain personal information, copyrighted material or confidential business records. Organisations should define lawful data collection, retention, access controls, deletion procedures and audit responsibilities. Sensitive predictions should not be treated as objective facts merely because they are generated by a neural network.

Recurrent models process tokens sequentially, which limits parallelism during training and may make very long contexts difficult. Transformer architectures use self-attention to relate positions more directly and are generally preferred for large-scale language modelling. Nevertheless, RNNs, LSTMs and GRUs remain valuable where streams arrive continuously, memory is limited, interpretability of state transitions is useful, or low-latency online processing is required.

Chapter Summary

Sequence models represent ordered observations and maintain information across time. Tokenisation converts language into manageable units, while embeddings map categorical tokens into learned dense vectors. A basic RNN repeatedly updates a hidden state but can suffer from vanishing and exploding gradients. LSTMs introduce a cell state and input, forget and output gates; GRUs use a simpler gated design. Seq2seq systems combine encoders and decoders to transform sequences, while attention gives the decoder access to different encoder states. These methods support text classification, translation, generation, time-series forecasting, anomaly detection and predictive maintenance. Reliable application requires leakage-free evaluation, suitable metrics, attention to class imbalance, monitoring and responsible handling of privacy, bias and human consequences.

Key Terms

Embedding: A learned dense vector representation of a token or categorical item.

Tokenisation: Dividing text into words, subwords or characters.

Vocabulary: The set of token types recognised by a model.

Hidden state: A recurrent model’s learned representation of preceding information.

Recurrent neural network: A neural network that reuses parameters while processing sequence elements.

Backpropagation through time: Training an RNN by unfolding it across time and propagating gradients through the unfolded graph.

Vanishing gradient: A gradient that becomes very small during propagation.

Exploding gradient: A gradient whose magnitude becomes excessively large.

Gradient clipping: Restricting gradient magnitude to improve training stability.

LSTM: A recurrent architecture with a cell state and three principal gates.

GRU: A gated recurrent architecture using update and reset gates.

Cell state: The long-term information pathway in an LSTM.

Sequence-to-sequence learning: Mapping one sequence to another, often with an encoder and decoder.

Teacher forcing: Supplying the correct previous target token during decoder training.

Attention: A mechanism that weights information from different sequence positions.

Padding: Adding special tokens so sequences in a batch have equal length.

Mask: An indicator that prevents padding or invalid positions from affecting computation.

Exposure bias: The difference between training with true previous outputs and inference with model-generated outputs.

Data leakage: Unintended use of information unavailable at prediction time.

Perplexity: An exponential measure derived from average negative log-likelihood.

Review Questions

Short Answer Questions

1. Why is a token identifier not an adequate semantic representation of a word?
2. What information is carried by the hidden state of an RNN?
3. State the roles of the input, forget and output gates in an LSTM.
4. Why must padded positions be masked during sequence processing?
5. Distinguish precision from recall.

Long Answer Questions

1. Explain the operation of a basic RNN and discuss vanishing and exploding gradients.
2. Compare LSTMs and GRUs in terms of state representation, gates, complexity and use cases.
3. Describe an encoder–decoder sequence-to-sequence system, including teacher forcing and attention.
4. Design a leakage-free evaluation procedure for an RNN time-series forecasting problem.
5. Discuss ethical, privacy and governance issues in an NLP system trained on user-generated text.

Analytical / Application Questions

1. A maintenance dataset contains 2% failure cases. Explain why accuracy alone is inadequate and propose suitable metrics and threshold-selection considerations.
2. Design a text-classification pipeline for customer complaints, specifying tokenisation, embeddings, model architecture, evaluation and monitoring.
3. A GRU performs similarly to an LSTM but has lower memory use and latency. Recommend a deployment choice and identify additional evidence needed before implementation.

Multiple Choice Questions with Answers

Question 1

What is the primary purpose of an embedding in an NLP model?

A. To convert tokens into dense vectors that can encode learned relationships

B. To remove all unknown words from a document

C. To guarantee that every sentence has the same meaning

D. To replace the loss function during training

Correct Answer: A. To convert tokens into dense vectors that can encode learned relationships

Question 2

In a basic RNN, which quantity carries information from earlier time steps?

A. The vocabulary size

B. The hidden state

C. The batch index

D. The padding mask only

Correct Answer: B. The hidden state

Question 3

Which LSTM component controls how much previous cell-state information is retained?

A. Input gate

B. Output gate

C. Forget gate

D. Embedding layer

Correct Answer: C. Forget gate

Question 4

What is a common effect of exploding gradients?

A. Training updates may become numerically unstable

B. The vocabulary automatically becomes smaller

C. All sequence lengths become equal

D. The model stops using activation functions

Correct Answer: A. Training updates may become numerically unstable

Question 5

Compared with an LSTM, a standard GRU generally has:

A. No recurrent connections

B. A separate cell state and four memory states

C. Fewer gates and no separate cell state

D. Only convolutional layers

Correct Answer: C. Fewer gates and no separate cell state

Question 6

What is teacher forcing in sequence-to-sequence training?

A. Using the correct previous target token as decoder input

B. Removing the encoder from the architecture

C. Clipping every gradient to zero

D. Randomly shuffling target sequences

Correct Answer: A. Using the correct previous target token as decoder input

Question 7

Why is chronological splitting important for time-series evaluation?

A. It ensures that future information does not improperly enter training

B. It makes every class equally frequent

C. It eliminates the need for scaling

D. It guarantees zero prediction error

Correct Answer: A. It ensures that future information does not improperly enter training

Question 8

If TP = 80 and FP = 20, what is precision?

A. 0.20

B. 0. fifty

C. 0.80

D. 0.90

Correct Answer: C. 0.80

Question 9

What does attention provide in a sequence-to-sequence model?

A. A weighted combination of encoder representations for a decoder step

B. A replacement for all training data

C. A guarantee of factual output

D. A method that prevents every possible bias

Correct Answer: A. A weighted combination of encoder representations for a decoder step

Question 10

Which metric is especially appropriate for measuring average absolute error in the original target units?

A. MAE

B. Cosine similarity

C. Recall

D. Perplexity

Correct Answer: A. MAE