Sequence Models and Temporal Learning

Introduction and Context

Many important data sources are ordered: words in a sentence, acoustic frames in speech, sensor readings in an industrial plant, financial prices, medical observations, and actions taken by an autonomous agent. In such data, the meaning of an observation depends partly on what occurred before it. A model that treats each observation independently may therefore lose temporal structure.

Sequence models address this problem by learning a mapping from an input sequence to an output sequence, a single output, or a future prediction. Deep sequence learning combines this temporal representation with nonlinear neural networks. This chapter develops recurrent neural networks (RNNs), long short-term memory networks (LSTMs), gated recurrent units (GRUs), sequence-to-sequence models, attention, and practical forecasting methods. It also explains why long-term dependencies are difficult and how modern systems manage them.

Learning Objectives

By the end of this chapter, learners should be able to:

1. Explain temporal dependence, hidden state, and autoregressive prediction.
2. Derive and interpret the basic RNN, LSTM, and GRU update equations.
3. Explain vanishing and exploding gradients and methods for controlling them.
4. Select suitable architectures for language, speech, time-series, and sequential decision tasks.
5. Design training, validation, and evaluation procedures that respect temporal order.
6. Analyse practical, ethical, and operational limitations of sequence models.

1. Sequential Data and Temporal Dependence

A sequence is an ordered collection x₁, x₂, …, x_T, where T is the sequence length and x_t is the observation at time step t. Observations may be vectors, tokens, images, or sensor measurements. A target may also be sequential, such as a translation, or may be a single value, such as next-day demand.

The conditional distribution of a sequence can be factorised as:

Here, the product is over time and the first term is conditioned on an empty history. This factorisation is general, but a practical model must represent the history compactly. Recurrent models use a hidden state h_t as a learned summary:

where f_θ is a parameterised nonlinear function, h₀ is an initial state, and θ denotes trainable parameters. The state is not necessarily a complete or lossless record of the past; its usefulness depends on the task and model capacity.

Important sequence-learning settings

A key distinction is between causal prediction, which uses only past and present information, and offline sequence analysis, which may use future observations. A bidirectional model is appropriate for transcription or sentence tagging when the complete sequence is available, but it is invalid for real-time forecasting if it accesses future values.

2. The Vanilla Recurrent Neural Network

A common RNN cell is:

W_xh, W_hh, and W_hy are weight matrices; b_h and b_y are biases; y_t is the output; and g may be softmax, sigmoid, or a linear function depending on the task. The hyperbolic tangent produces values between −1 and 1. For a classification problem, softmax gives class probabilities:

where z is the vector of logits and i indexes a class. The probabilities sum to one.

The same matrices are reused at every time step. This parameter sharing makes an RNN more efficient than assigning separate parameters to each possible sequence position and allows it to process sequences of variable length.

Training through time

For a target sequence, a typical loss is the sum or mean of per-step losses. For categorical targets with one-hot labels y_t, cross-entropy is:

where K is the number of classes and ŷ_t,k is the predicted probability for class k. Mean loss divides by the number of valid prediction positions. Padding positions must be masked so that they do not contribute to the loss.

Backpropagation through time (BPTT) unrolls the recurrence across time and applies the chain rule. Truncated BPTT processes a limited window of steps, reducing memory and computation but potentially weakening learning across longer intervals.

3. Why Long-Term Dependencies Are Difficult

Suppose the recurrent transition is simplified to h_t = tanh(Wh_t-1). The influence of an earlier state on a later state involves repeated Jacobian products:

If the typical norm of these Jacobians is below one, the product tends to zero: this is the vanishing-gradient problem. If it is above one, the product may grow rapidly: this is the exploding-gradient problem. Saturating nonlinearities such as tanh can worsen vanishing gradients because their derivative is small when the activation is near −1 or 1.

Consequences include difficulty learning relationships such as “the subject introduced many words ago determines the verb,” or “a machine vibration pattern predicts failure several hours later.” Common controls include:

• gated cells such as LSTM and GRU;
• careful parameter initialisation, including orthogonal recurrent matrices;
• gradient clipping;
• normalisation and suitable learning rates;
• residual or skip connections;
• attention mechanisms that provide shorter information paths;
• transformer architectures, which replace recurrence with attention.

Gradient clipping does not correct the underlying representation problem, but it limits destructive updates. For global-norm clipping, if gradient norm g exceeds threshold c, the gradient is replaced by c g / ||g||.

4. Long Short-Term Memory Networks

An LSTM separates short-term output state from a memory cell. Its gates regulate information flow:

Here, i_t, f_t, and o_t are input, forget, and output gates; g_t is a candidate memory; c_t is the cell state; h_t is the hidden state; σ is the sigmoid function; and ⊙ denotes element-wise multiplication. All gate vectors have the same dimensionality as the cell state.

The cell update has an additive path. When a forget-gate value is near one, information and gradients can pass through many steps with less attenuation than in a basic RNN. The gates are learned, not manually programmed. A bias initialisation that makes forget gates initially more open is sometimes useful, but it is a heuristic rather than a guarantee of long-term learning.

Illustrative scalar calculation

Assume one-dimensional values: c_t-1=0.80, f_t=0.90, i_t=0.20, and g_t=0.50. Then:

c_t = 0.90×0.80 + 0.20×0.50 = 0.72 + 0.10 = 0.82.

The new cell content is therefore 0.82. If o_t=0.70, the hidden state is approximately h_t = 0.70×tanh(0.82) ≈ 0.70×0.675 = 0.473. This example is illustrative; actual LSTM vectors contain many dimensions and gate values are produced by learned affine transformations.

5. Gated Recurrent Units

A GRU uses fewer gates and does not maintain a separate cell state in the standard formulation:

The update gate z_t controls interpolation between the previous state and candidate state. The reset gate r_t controls how much previous state contributes to the candidate. Some software uses the opposite convention for the update gate, so the equations must be checked before interpreting learned values.

GRUs often have fewer parameters and may train faster than LSTMs. Neither architecture is universally superior. Performance depends on sequence length, data volume, hidden dimension, regularisation, optimisation, and task structure.

6. Sequence-to-Sequence Learning and Attention

A sequence-to-sequence model contains an encoder that maps an input sequence to representations and a decoder that generates an output sequence. In early encoder–decoder systems, the decoder received a single final context vector. This created a bottleneck for long inputs.

Attention allows the decoder to construct a context vector from encoder states. With query q_t and encoder key k_s, an additive attention score can be written:

Attention weights are:

and the context is:

where v_s is an encoder value. The weights are nonnegative and sum to one over source positions. Attention improves access to relevant earlier states, but attention weights should not automatically be interpreted as definitive explanations of model reasoning.

Transformers use self-attention and positional information rather than recurrence. They are highly parallel during training and often perform well on language and long sequences, but may require substantial memory, data, and careful handling of long contexts. RNNs remain useful for streaming, low-latency, and resource-constrained applications.

7. Applications

Natural language and speech

RNNs can model token probabilities, classify documents, label words, and generate text. Speech systems process acoustic feature frames, often using recurrent or attention-based encoders. Practical systems use tokenisation, padding masks, teacher forcing during training, and constrained decoding. Teacher forcing supplies the true previous target during training; at inference, the model must use its own previous prediction. This difference can cause exposure bias.

For next-token prediction, the model estimates P(w_t | w₁,…,w_t-1). During generation, greedy decoding selects the highest-probability token at each step, whereas beam search retains several partial hypotheses. Beam search may improve sequence likelihood but can reduce diversity and does not guarantee the best human-perceived output.

Time-series forecasting

For a univariate series y_t, a one-step forecast is ŷ_t+1=f(y_t,y_t-1,…,y_t-p+1), where p is the look-back window. For multivariate forecasting, inputs may include exogenous variables such as weather, promotions, or machine settings.

Forecasting must preserve chronology. Randomly splitting overlapping windows can leak future information into training. A sound procedure uses earlier periods for training, a later validation period for model selection, and the latest untouched period for testing. Scaling parameters must be fitted on training data only.

For actual values y_t and forecasts ŷ_t, common metrics include:

MAE has the same units as the target and is less sensitive to large errors than RMSE. RMSE penalises large errors more strongly. MAPE, (100/n)∑| (y_t−ŷ_t)/y_t |, is undefined when y_t=0 and unstable near zero; it should not be used uncritically.

Sequential decision problems

In reinforcement learning, an agent observes states, chooses actions, and receives rewards. A standard objective is the discounted return:

where r_t+k+1 is a future reward and 0≤γ<1 is the discount factor. A recurrent policy can use hidden state to infer unobserved aspects of a partially observed environment. However, recurrence does not by itself guarantee safe exploration or correct credit assignment. Simulation, constraints, monitoring, and human override are essential in high-risk applications.

8. A Practical Development Workflow

1. Define the prediction boundary. Specify exactly what information is available at decision time.
2. Inspect temporal structure. Examine trends, seasonality, missingness, irregular intervals, regime changes, and outliers.
3. Construct sequences carefully. Choose window length based on domain latency and expected dependency length.
4. Establish baselines. Compare against persistence, seasonal averages, linear models, or classical methods such as ARIMA where appropriate.
5. Start with a small model. Use a single-layer RNN, GRU, or LSTM before increasing depth.
6. Train with masks and stable optimisation. Use appropriate padding masks, validation monitoring, learning-rate schedules, and gradient clipping when needed.
7. Evaluate by time and subgroup. Report overall performance, horizon-specific errors, worst cases, and relevant population groups.
8. Test deployment behaviour. Measure latency, memory, missing-input behaviour, drift, and retraining requirements.

Teacher forcing, scheduled sampling, dropout, recurrent dropout, weight decay, and early stopping are possible regularisation tools. Dropout should be applied consistently with the framework’s recurrent implementation; careless independent masking can alter temporal dynamics.

9. Illustrative Case: Predictive Maintenance for Industrial Pumps

Background. A water utility records vibration, temperature, pressure, and flow every minute from pumps. Failures are expensive and safety-sensitive.

Problem. Engineers want a warning at least 30 minutes before a failure, while avoiding excessive false alarms.

Analysis. The data contain daily operating cycles, maintenance-induced distribution changes, missing sensor intervals, and strong correlation between adjacent observations. A random row split would leak information because neighbouring windows from the same failure event would appear in both training and testing.

Approach. The team creates 60-minute windows and predicts whether a failure occurs in the next 30 minutes. Windows are split by calendar period and pump, with the latest period reserved for testing. Sensor normalisation uses training-period statistics only. A persistence baseline, logistic regression on engineered features, GRU, and LSTM are compared. Class-weighted binary cross-entropy addresses imbalance, but the operational threshold is selected using the cost of missed failures and false alarms rather than accuracy alone.

A binary cross-entropy objective for label y_t∈{0,1} and predicted probability p_t is:

Outcome. In this illustrative case, the LSTM does not automatically dominate. The GRU achieves similar detection with lower latency, while the engineered baseline is easier to audit. The chosen system combines the GRU with rule-based sensor-quality checks and sends alerts only when risk remains above threshold across multiple consecutive windows.

Learning points. Temporal validation, cost-sensitive thresholds, sensor-quality handling, and operational integration matter as much as cell architecture. A model’s probability is not a maintenance decision until calibrated, thresholded, monitored, and assigned to a responsible workflow.

10. Limitations, Ethics, and Governance

Sequence models can encode historical bias. A language model may reproduce harmful associations; a clinical model may perform poorly for underrepresented groups; a predictive-maintenance system may shift risk toward facilities with less reliable sensors. Performance should therefore be reported by relevant subgroups and operating conditions.

Temporal leakage is a major governance concern. Using revised records, future measurements, post-event labels, or future-derived normalisation can produce impressive but invalid results. Data provenance and feature-availability timestamps should be documented.

Other issues include privacy in speech and location sequences, re-identification from apparently anonymous trajectories, uncertain predictions, and model drift. Systems affecting employment, credit, healthcare, safety, or public services require human oversight, audit trails, access controls, incident procedures, and compliance with applicable data-protection and sector-specific laws. Explainability should combine model diagnostics with domain evidence; attention visualisations alone are insufficient.

11. Contemporary Developments

Current sequence modelling increasingly combines recurrent or convolutional local processing with attention, state-space models, and efficient long-context mechanisms. Streaming speech and edge devices motivate compact recurrent models, quantisation, pruning, and knowledge distillation. Foundation models use large-scale pretraining and transfer learning, but their outputs remain sensitive to data quality, prompt context, and distribution shift.

Research directions include continuous-time neural models for irregular observations, probabilistic forecasting that represents uncertainty, retrieval-augmented sequence generation, causal temporal modelling, and methods for reliable long-context reasoning. A useful professional principle is to choose the simplest model that meets accuracy, latency, interpretability, and safety requirements.

Chapter Summary

Sequence models represent ordered observations and learn how current outputs depend on previous information. Vanilla RNNs reuse parameters across time but suffer from vanishing and exploding gradients. LSTMs introduce input, forget, and output gates plus a cell state; GRUs provide a simpler gated alternative. BPTT trains recurrent models, while truncation reduces cost at the expense of long-range credit assignment. Encoder–decoder models and attention support variable-length transformations, and transformers offer highly parallel alternatives to recurrence. Applications include language, speech, forecasting, and partially observed decision problems. Reliable practice requires causal data design, temporal validation, suitable baselines, metric selection, monitoring, privacy protection, subgroup analysis, and operational governance.

Key Terms

Sequence: An ordered set of observations.

Temporal dependence: The influence of earlier observations on later observations.

Hidden state: A learned internal representation of sequence history.

Recurrent neural network: A neural model that updates a state repeatedly over time.

Backpropagation through time: Gradient computation through an unrolled recurrent network.

Vanishing gradient: Gradients becoming very small across repeated transformations.

Exploding gradient: Gradients becoming excessively large during training.

LSTM: A gated recurrent architecture with a separate memory cell.

GRU: A gated recurrent architecture using update and reset gates.

Cell state: The LSTM memory pathway.

Gate: A learned element-wise control on information flow.

Teacher forcing: Training a decoder using the true previous target.

Attention: A weighted mechanism for selecting relevant representations.

Encoder–decoder: A model that transforms an input sequence into an output sequence.

Autoregressive model: A model that predicts using previous observations or outputs.

Temporal leakage: Unintended use of information unavailable at prediction time.

Gradient clipping: Limiting gradient magnitude to stabilise training.

Concept drift: Change in the data-generating process over time.

Calibration: Agreement between predicted probabilities and observed frequencies.

Discounted return: A weighted sum of future rewards in sequential decision learning.

Review Questions

Short Answer Questions

1. What is the role of the hidden state in an RNN?
2. Distinguish vanishing gradients from exploding gradients.
3. What functions do the forget and input gates perform in an LSTM?
4. Why can random train–test splitting be invalid for time-series data?
5. State one advantage and one limitation of GRUs compared with LSTMs.

Long Answer Questions

1. Derive and explain the principal equations of a vanilla RNN and discuss how BPTT trains it.
2. Compare vanilla RNNs, LSTMs, and GRUs in terms of structure, computation, and long-term dependency learning.
3. Explain encoder–decoder learning and show how attention reduces the fixed-vector bottleneck.
4. Design a rigorous workflow for forecasting electricity demand with a recurrent model.
5. Discuss ethical, privacy, and governance issues in speech or sequential healthcare modelling.

Analytical / Application Questions

1. A forecasting team randomly splits overlapping windows from a two-year sensor record. Identify the leakage risks and propose a corrected evaluation design.
2. For an LSTM with c_t-1=0.6, f_t=0.8, i_t=0.25, and g_t=0.4, calculate c_t and explain the contribution of retained and new information.
3. A safety system has a high cost for missed failures but limited staff for false alarms. Explain how you would select a decision threshold, evaluate calibration, and monitor the system after deployment.

Multiple Choice Questions with Answers

Question 1

Which component of an LSTM primarily controls how much of the previous cell state is retained?

A. Input gate

B. Forget gate

C. Output projection

D. Softmax layer

Correct Answer: B. Forget gate

Question 2

What is the main cause of vanishing gradients in long recurrent chains?

A. Repeated Jacobian products with typical magnitude below one

B. Too many output classes

C. Using a validation set

D. Padding every sequence

Correct Answer: A. Repeated Jacobian products with typical magnitude below one

Question 3

Which split is most appropriate for evaluating a causal forecasting model?

A. Randomly shuffle all observations

B. Use future observations for training and past observations for testing

C. Train on earlier periods and test on later periods

D. Duplicate the test windows in training

Correct Answer: C. Train on earlier periods and test on later periods

Question 4

In a GRU, the update gate controls:

A. The vocabulary size

B. Interpolation between the previous and candidate hidden states

C. The number of training examples

D. The sampling frequency of sensors

Correct Answer: B. Interpolation between the previous and candidate hidden states

Question 5

Which metric has the same units as the forecast target?

A. MAE

B. MAPE expressed as a percentage

C. A classification accuracy

D. A softmax probability

Correct Answer: A. MAE

Question 6

What problem does attention primarily help address in an encoder–decoder model?

A. The need to store labels

B. The fixed-length context bottleneck

C. The definition of a sigmoid

D. The removal of all computation

Correct Answer: B. The fixed-length context bottleneck

Question 7

Why must scaling parameters for a time-series model be fitted only on training data?

A. To prevent future information from entering the model indirectly

B. To increase the number of classes

C. To guarantee zero forecasting error

D. To make all values integers

Correct Answer: A. To prevent future information from entering the model indirectly

Question 8

For c_t-1=0.6, f_t=0.8, i_t=0.25, and g_t=0.4, what is c_t?

A. 0.48

B. 0.52

C. 0.58

D. 0.70

Correct Answer: C. 0.58

Question 9

Which statement about MAPE is correct?

A. It is always defined when the target is zero

B. It can be unstable when actual values are near zero

C. It has the same units as the target

D. It is identical to RMSE

Correct Answer: B. It can be unstable when actual values are near zero

Question 10

What is the purpose of gradient clipping?

A. To limit excessively large parameter updates

B. To remove the need for labels

C. To guarantee causal data splits

D. To convert a GRU into an LSTM

Correct Answer: A. To limit excessively large parameter updates