Neural NLP and Sequence Modeling

Introduction

Natural language is sequential, context-dependent, and highly variable. The meaning of a word may depend on preceding words, following words, sentence structure, discourse context, and world knowledge. Neural NLP models address these challenges by learning distributed representations and functions that transform sequences of tokens into predictions, classifications, or generated text.

This chapter develops the principal neural architectures used before and alongside modern transformer systems: feed-forward networks, convolutional neural networks (CNNs), recurrent neural networks (RNNs), long short-term memory networks (LSTMs), gated recurrent units (GRUs), attention mechanisms, and encoder–decoder models. These architectures remain important because they explain the foundations of sequence modeling, support efficient solutions for constrained applications, and provide conceptual insight into contemporary generative AI.

A neural language system normally begins with tokenization. A sequence of tokens is mapped to integer identifiers and then to vectors through an embedding matrix. If the vocabulary contains |V| tokens and the embedding dimension is d, the embedding matrix is E ∈ ℝ^{|V| × d}. Token i is represented by the corresponding row E_i. The embedding is learned jointly with the task model or obtained from pre-training.

Learning objectives

By the end of this chapter, learners should be able to:

1. Explain how neural networks represent words, phrases, and sequences.
2. Distinguish feed-forward, convolutional, recurrent, gated, attention-based, and encoder–decoder architectures.
3. Derive and interpret the principal equations of RNNs, LSTMs, GRUs, and attention.
4. Implement small sequence models using a deep-learning framework.
5. Select an architecture for language understanding or generation.
6. Evaluate sequence models while recognising their computational, ethical, and operational limitations.

1. From word vectors to sequence models

A feed-forward neural language model predicts a target token from a fixed-size context. Suppose the context contains the previous k tokens. Their embeddings can be concatenated:

Here, e_i ∈ ℝ^d is an embedding, [;] denotes concatenation, W_c and b_c are learned parameters, and φ is an activation function such as tanh or ReLU. The output distribution is obtained with softmax:

where o = W_o h + b_o and j indexes the vocabulary. The probabilities are non-negative and sum to one. During training, the usual loss is categorical cross-entropy:

For a single target token, the sum contains one term. The formula assumes a one-hot target and natural logarithms; averaging over tokens is commonly used in implementation.

The major weakness of a feed-forward model is its fixed context window. It cannot directly condition on an arbitrarily long history. Increasing k increases the input size and parameter count, while relevant information outside the window is ignored. Nevertheless, feed-forward networks are useful baselines, classifiers, and components in larger systems.

Embeddings and representation learning

An embedding maps discrete symbols into a continuous vector space. Similar words may acquire similar vectors when they occur in similar contexts, although similarity is task- and corpus-dependent rather than guaranteed to represent true semantic equivalence. Subword embeddings help handle rare words, spelling variation, and productive morphology.

A practical pipeline is:

1. Normalize and tokenize text.
2. Add special symbols such as <PAD>, <UNK>, <BOS>, and <EOS>.
3. Convert tokens to integer IDs.
4. Pad or pack sequences for batching.
5. Map IDs to embeddings.
6. Apply a sequence model and output layer.

Padding is not linguistic content. Loss functions and attention mechanisms should therefore use masks so that padding positions do not influence learning.

2. Convolutional neural networks for text

A one-dimensional CNN applies a learned filter across neighbouring token representations. If X ∈ ℝ^{T × d} is a sequence of length T and embedding width d, a filter with width w produces features such as:

W has a shape compatible with w consecutive vectors, b is a bias, and f is an activation function. Applying max pooling over positions selects the strongest detected feature. Different filter widths can detect patterns resembling short phrases, negation, or local syntactic constructions.

CNNs offer parallel computation and are effective for sentence classification, intent detection, spam filtering, and character-level modeling. Their receptive field can be enlarged by stacking layers or using dilation. However, a plain CNN does not naturally preserve an explicit state across arbitrary distances. It may therefore be less suitable than recurrent or attention-based models when long-range relationships are central.

For classification, a typical architecture is embedding → several parallel convolutions → nonlinear activation → global max pooling → concatenation → dense layer → softmax or sigmoid output. A sigmoid output is used for independent multi-label decisions; softmax is used for mutually exclusive classes.

3. Recurrent neural networks

An RNN processes one time step at a time and maintains a hidden state. For input x_t at time t:

The output distribution may be:

W_x, W_h, W_y and the bias vectors are shared across all time steps. This parameter sharing allows the same model to process sequences of different lengths. The initial state h_0 is commonly a zero vector or a learned vector.

For sequence classification, the final hidden state can represent the complete input, although this creates a bottleneck for long sequences. For sequence labeling, every h_t can produce a tag such as a named-entity label. For language modeling, x_t is commonly the embedding of the previous token and y_t predicts the next token.

Training through time

RNNs are trained using backpropagation through time (BPTT). The computational graph is unrolled over the sequence, gradients are calculated through each time step, and shared parameters receive contributions from all steps. Long products of recurrent Jacobian matrices can cause gradients to vanish or explode. Gradient clipping limits excessive updates, for example by rescaling a gradient vector g when its norm exceeds a threshold τ:

This controls update magnitude but does not solve the underlying difficulty of preserving information over long intervals.

4. LSTM networks

Long short-term memory networks introduce a cell state and gates. A standard LSTM is defined by:

Here, i_t is the input gate, f_t the forget gate, o_t the output gate, ĝ_t the candidate cell content, c_t the cell state, h_t the hidden state, σ the logistic sigmoid, and ⊙ element-wise multiplication. All gates have values between 0 and 1. The cell update is additive, which creates a comparatively direct path for information and gradients.

The LSTM does not guarantee perfect long-term memory. It still depends on data, optimization, sequence length, and parameter capacity. It is also more computationally expensive than a simple RNN because it uses several gates.

5. GRU networks

The gated recurrent unit simplifies the LSTM by combining some functions. One common formulation is:

z_t is the update gate and r_t is the reset gate. Some libraries use an algebraically different convention for z_t, so equations and implementation documentation must be checked together. GRUs often train faster and use fewer parameters than LSTMs, although neither architecture is universally superior.

Comparing recurrent architectures

6. Attention mechanisms

A recurrent encoder may compress a source sequence into one vector. Attention reduces this bottleneck by allowing a decoder to consult all encoder states. Let query q, keys k_i, and values v_i be vectors. The general attention operation is:

where weights are calculated by:

The weights are non-negative and sum to one. A common dot-product score is score(q,k_i) = qᵀk_i. Scaled dot-product attention divides the score by √d_k to reduce excessively large values when the key dimension d_k is large:

Q, K, and V are matrices of queries, keys, and values. In self-attention, they are derived from the same sequence; in cross-attention, queries come from one sequence and keys and values from another. A causal mask prevents a language model from attending to future tokens during next-token prediction.

Multi-head attention projects inputs into several subspaces, computes attention independently, concatenates the results, and applies a final projection. This can allow different heads to model different relationships, although attention weights should not automatically be interpreted as complete explanations of model decisions.

Attention has a computational cost that is commonly quadratic in sequence length for full self-attention: with n tokens, the attention score matrix contains n² entries. Memory-efficient kernels, sparse attention, recurrence, retrieval, and state-space approaches seek to reduce this cost.

Figure: An encoder–decoder system in which attention connects decoder queries to all encoder states.

7. Encoder–decoder models for language generation

An encoder transforms an input sequence into contextual representations. A decoder generates an output sequence one token at a time. For source sequence x and target sequence y, the conditional probability factorizes as:

where y_{<t} denotes previously generated target tokens. Training commonly uses teacher forcing: the true previous token is supplied to predict the next token. The negative log-likelihood loss is:

At inference, the model does not know the true previous target. It feeds back its own generated token. This difference is called exposure bias and can produce error accumulation.

Greedy decoding selects the highest-probability token at every step. Beam search retains a specified number of partial hypotheses and can improve sequence-level probability, but it may produce generic or repetitive text. Sampling methods introduce controlled variation. Temperature τ modifies logits z before softmax:

For τ > 1, the distribution is flatter; for 0 < τ < 1, it is sharper. Temperature does not ensure factuality or quality and should be evaluated empirically.

Encoder–decoder models support machine translation, summarization, question answering, dialogue, and structured generation. In a traditional recurrent system, the encoder and decoder may be LSTMs or GRUs. In modern systems, transformer encoders and decoders are common, while decoder-only models generate by predicting the next token from preceding context.

8. Implementation blueprint

The following pseudocode describes a sequence classifier. It is framework-neutral and omits library-specific syntax:

vocabulary = build_vocabulary(training_text)
for batch in data:
    ids, mask, labels = prepare_batch(batch, vocabulary)
    embeddings = embedding_layer(ids)
    states = sequence_model(embeddings, mask)
    representation = select_final_state(states, mask)
    logits = classifier(representation)
    loss = cross_entropy(logits, labels)
    optimizer.zero_grad()
    loss.backward()
    clip_gradients(model, max_norm=1.0)
    optimizer.step()

For generation, the final classifier is replaced by a vocabulary projection. During training, shifted target sequences are used: the input begins with <BOS> and the labels end with <EOS>. Padding positions are excluded from the loss.

Important engineering choices include:

• Sequence length: Truncation improves speed but may remove decisive evidence.
• Batching: Padding, bucketing by length, or packed sequences improves efficiency.
• Regularization: Dropout, weight decay, early stopping, and vocabulary control can reduce overfitting.
• Optimization: Adam-family optimizers are common; learning-rate schedules and warm-up may stabilize training.
• Reproducibility: Fix seeds where possible, record data versions, save configurations, and monitor hardware-dependent variation.
• Evaluation: Separate training, validation, and test data by document or user when leakage is possible.

For language modeling, perplexity is often reported:

N is the number of evaluated tokens. Lower perplexity indicates better average predictive likelihood under the same tokenization and evaluation conditions. It does not directly measure factuality, helpfulness, fairness, or human preference.

9. Illustrative case: multilingual customer-support routing

Background

A telecommunications provider receives customer messages in English, Spanish, and code-mixed language. Messages must be routed to billing, network failure, account access, or cancellation teams.

Problem

A keyword system performs poorly when customers use spelling variation, negation, or mixed languages. The provider needs a model that is accurate, fast, auditable, and tolerant of variable message length.

Analysis

A feed-forward model with a fixed window is inexpensive but may lose information in long messages. A CNN can detect local phrases such as “cannot log in” or “charged twice” and offers low-latency parallel computation. A bidirectional GRU can model sequence order and longer dependencies, while an attention layer can identify which tokens contribute to the decision. If the output is one of four mutually exclusive routes, softmax cross-entropy is appropriate.

Approach and solution

The team constructs a shared subword vocabulary and adds a language-identification feature. It compares three systems: a feed-forward baseline, a CNN, and a bidirectional GRU with attention. Class weights are used if cancellation requests are rare. The dataset is split by customer rather than randomly by message to prevent the same customer’s writing patterns from appearing in both training and test data.

Evaluation includes macro-F1, per-class recall, latency, and calibration. Macro-F1 is important because a high-frequency billing class could otherwise dominate accuracy. Human reviewers examine false routes, particularly cases involving vulnerable customers or urgent network outages.

Outcome

The GRU-attention model provides the strongest recall on long and code-mixed messages, while the CNN provides the best latency. The organization deploys the GRU model for complex messages and the CNN for low-risk, high-volume traffic. Low-confidence cases are sent to human agents, and feedback is periodically reviewed for new terminology and demographic disparities.

Learning points

Architecture choice depends on the operational objective, not only on benchmark accuracy. Data splitting, class imbalance, latency, calibration, and human oversight are part of the NLP system. Attention may assist inspection, but it is not by itself a proof of causal explanation.

10. Limitations, ethics, and governance

Neural sequence models learn statistical regularities from data. They can reproduce stereotypes, confidential information, toxic language, and historical inequities. Generative models may hallucinate plausible but unsupported claims. Risks increase when users treat fluent output as verified knowledge.

Professional practice requires:

• documented data provenance and lawful use;
• removal or protection of personal and sensitive information;
• testing across languages, dialects, demographic groups, and writing styles;
• explicit handling of uncertainty and abstention;
• access controls, audit logs, and incident response;
• human review in high-impact decisions;
• copyright and licensing review for training and generated content;
• monitoring for distribution shift and adversarial input.

A model should not be deployed solely because its average score is high. The relevant question is whether its errors, costs, and benefits are acceptable for the intended context. Regulation may impose additional requirements for transparency, risk management, privacy, and accountability.

11. Contemporary developments

Transformers have become dominant because self-attention permits extensive parallelism during training and provides direct interactions between tokens. Pre-training on large corpora followed by fine-tuning or prompting has reduced the need for task-specific labeled data. Retrieval-augmented generation can supply external documents, but retrieval quality, source authority, prompt injection, and citation verification remain concerns.

Efficient sequence modeling is an active research area. Techniques include grouped or multi-query attention, sparse attention, quantization, distillation, parameter-efficient fine-tuning, recurrent memory, and state-space models. These methods address the cost of long contexts and deployment on limited hardware. Nevertheless, recurrent and convolutional models remain valuable for streaming, edge devices, low-latency services, and educational understanding of sequence computation.

Best-practice checklist

1. Establish a clear task definition and error-cost analysis.
2. Select tokenization and sequence length using the application’s language and domain.
3. Begin with a transparent baseline before adopting a complex architecture.
4. Mask padding and future tokens correctly.
5. Use validation data for model selection and reserve test data for final assessment.
6. Report macro and per-class metrics where imbalance exists.
7. Inspect qualitative failures, not only aggregate scores.
8. Monitor drift, latency, memory, and energy consumption.
9. Protect personal data and document governance decisions.
10. Require human escalation when uncertainty or potential harm is high.

Chapter Summary

Neural NLP represents tokens with learned vectors and transforms them into predictions through structured architectures. Feed-forward networks use fixed contexts; CNNs detect local patterns efficiently; RNNs maintain sequential hidden states; LSTMs and GRUs use gates to preserve or discard information; and attention provides content-based access to multiple representations. Encoder–decoder models factorize conditional generation into next-token decisions and support translation, summarization, dialogue, and other sequence-to-sequence tasks. Training requires careful treatment of padding, teacher forcing, optimization, and evaluation. Practical deployment additionally requires attention to latency, data leakage, fairness, privacy, hallucination, monitoring, and human oversight.

Key Terms

Embedding: A learned continuous vector representation of a discrete token.

Feed-forward network: A network whose computation moves from input to output without recurrent state.

Convolution: A sliding local operation that detects patterns across neighbouring positions.

Receptive field: The portion of an input that can influence a unit’s output.

RNN: A recurrent neural network that updates a hidden state sequentially.

Hidden state: The recurrent representation carried from one time step to the next.

BPTT: Backpropagation through time, used to train recurrent networks.

LSTM: A gated recurrent architecture with input, forget, output gates, and a cell state.

GRU: A gated recurrent unit using update and reset gates.

Cell state: The persistent memory pathway in an LSTM.

Attention: A weighted aggregation of value vectors based on query–key compatibility.

Self-attention: Attention in which queries, keys, and values originate from the same sequence.

Cross-attention: Attention connecting representations from different sequences or modules.

Encoder: A network that converts an input sequence into contextual representations.

Decoder: A network that generates or predicts an output sequence.

Teacher forcing: Training a decoder with the true previous target token.

Exposure bias: The difference between training with true history and inference with generated history.

Perplexity: The exponentiated average negative log-likelihood of a token sequence.

Causal mask: A mask preventing a position from using information from future positions.

Gradient clipping: Rescaling gradients to limit excessively large updates.

Review Questions

Short Answer Questions

1. Why is a fixed context window a limitation of a feed-forward language model?
2. What roles do the forget and input gates play in an LSTM?
3. Distinguish self-attention from cross-attention.
4. Why are padding masks required during batched sequence processing?
5. What does teacher forcing mean in encoder–decoder training?

Long Answer Questions

1. Explain the evolution from feed-forward networks to gated recurrent networks, including the limitations each architecture addresses.
2. Derive and interpret the scaled dot-product attention equation and explain the role of the scaling factor.
3. Compare CNN, LSTM, GRU, and attention-based models for a real-time text-classification service.
4. Explain how an encoder–decoder model is trained and decoded, including exposure bias and beam search.
5. Design an evaluation and governance plan for a multilingual customer-support NLP system.

Analytical / Application Questions

1. A dataset contains long customer messages in which decisive evidence may occur at the beginning or end. Which architecture would you test first, and what experiments would distinguish a genuine long-range advantage from a data or preprocessing problem?
2. A model has high accuracy but very low recall for a rare emergency class. Which metrics, data strategies, threshold decisions, and human-review mechanisms should be investigated?
3. A generated summary is fluent but contains unsupported claims. Analyse at least three possible causes and propose an evaluation and mitigation plan involving retrieval, decoding, verification, or human oversight.

Multiple Choice Questions with Answers

Question 1

Which architecture uses a fixed-size context window rather than an explicit recurrent state?

A. Feed-forward neural network

B. LSTM

C. GRU

D. Recurrent neural network

Correct Answer: A. Feed-forward neural network

Question 2

What is the primary purpose of the LSTM forget gate?

A. To convert logits into probabilities

B. To control which previous cell-state information is retained

C. To identify convolutional features

D. To prevent all gradient updates

Correct Answer: B. To control which previous cell-state information is retained

Question 3

In scaled dot-product attention, why are scores divided by √d_k?

A. To remove the vocabulary

B. To make every attention weight equal

C. To reduce excessively large dot products as key dimension grows

D. To force causal decoding

Correct Answer: C. To reduce excessively large dot products as key dimension grows

Question 4

Which mechanism prevents a decoder from attending to future target tokens during autoregressive training?

A. Dropout

B. A causal mask

C. Max pooling

D. Gradient clipping

Correct Answer: B. A causal mask

Question 5

For four mutually exclusive routing classes, which output and loss pairing is generally appropriate?

A. Softmax with categorical cross-entropy

B. Sigmoid with mean squared error only

C. Linear output with no loss

D. Softmax with cosine distance only

Correct Answer: A. Softmax with categorical cross-entropy

Question 6

What is exposure bias?

A. The cost of storing embeddings

B. The difference between training with true previous tokens and inference with generated tokens

C. The use of too many convolution filters

D. The inability to tokenize punctuation

Correct Answer: B. The difference between training with true previous tokens and inference with generated tokens

Question 7

If temperature is increased above 1 in temperature-scaled softmax, what generally happens?

A. The distribution becomes flatter

B. The vocabulary becomes smaller

C. All probabilities become zero

D. Future tokens become visible during causal decoding

Correct Answer: A. The distribution becomes flatter

Question 8

Which evaluation measure is especially useful when class frequencies are highly imbalanced?

A. Macro-F1

B. Sequence length alone

C. Embedding dimension

D. Batch size

Correct Answer: A. Macro-F1

Question 9

What does gradient clipping primarily control in recurrent-network training?

A. Vocabulary coverage

B. The magnitude of excessively large gradients

C. The number of attention heads

D. The number of output classes

Correct Answer: B. The magnitude of excessively large gradients

Question 10

In an encoder–decoder model, what does the encoder usually produce?

A. Only a scalar loss

B. Contextual representations of the input sequence

C. A random target sequence

D. The final human evaluation score

Correct Answer: B. Contextual representations of the input sequence