Neural NLP and Representation Learning

Introduction

Natural language processing (NLP) systems must convert language into representations that support prediction, classification, retrieval, generation, and reasoning. Earlier statistical systems commonly represented text with one-hot vectors, counts, n-grams, or manually designed features. These representations were useful but sparse, high-dimensional, and largely insensitive to semantic similarity. Neural NLP changed this situation by learning dense representations and composing them through differentiable functions.

A neural NLP model typically performs four operations: it represents tokens numerically, transforms those representations through one or more neural layers, produces a task-specific output, and learns parameters by minimizing a loss function. The same general architecture can support sentiment analysis, named-entity recognition, language modelling, machine translation, summarization, and question answering.

The central idea is representation learning: rather than specifying every useful linguistic feature, the system learns representations from data and task objectives. However, learned representations are not automatically meaningful, fair, or robust. Their quality depends on data, objective functions, optimization, architecture, and evaluation design.

Learning Objectives

After studying this chapter, the reader should be able to:

• explain distributed word and subword representations;
• compare word2vec-style embeddings, contextual representations, and subword models;
• describe the operation and applications of CNNs, RNNs, and LSTMs for NLP;
• formulate encoder–decoder models and explain attention;
• diagnose overfitting, underfitting, vanishing gradients, exposure bias, and data leakage;
• design and evaluate a neural NLP pipeline responsibly;
• assess limitations involving bias, interpretability, robustness, privacy, and computational cost.

1. From Sparse Features to Distributed Representations

A vocabulary of size |V| can represent a token as a one-hot vector in ℝ^{|V|}. Exactly one component is 1 and all others are 0. Such vectors distinguish tokens but do not encode similarity: the Euclidean distance between any two distinct one-hot vectors is identical. Moreover, a vocabulary of millions of words creates large sparse inputs.

A distributed representation assigns each token an embedding vector in ℝ^d, where d is usually much smaller than |V|. If E ∈ ℝ^{|V|×d} is an embedding matrix and e_i is a one-hot vector, the token representation is:

Here x_i ∈ ℝ^d. The matrix E is learned, and each row is a dense vector. Similar contexts may lead to similar vectors, although similarity is an empirical property rather than a guarantee.

1.1 Distributional semantics

The distributional hypothesis states that words occurring in similar contexts tend to have similar meanings. Word2vec operationalized this principle with two influential objectives:

• Continuous bag of words (CBOW): predict a target word from surrounding words.
• Skip-gram: predict surrounding words from a target word.

For skip-gram, given centre word w_t and context words c, a simplified objective is:

The conditional probability is commonly defined with softmax:

where u_w is the input vector for w, v_c is the output vector for context word c, and V is the vocabulary. Full softmax can be expensive for a large vocabulary, so negative sampling or hierarchical softmax is often used. Negative sampling trains the model to distinguish observed word–context pairs from randomly sampled noise pairs.

These embeddings are static: a word has one vector regardless of context. Thus, “bank” receives one representation in “river bank” and “bank loan,” limiting polysemy modelling.

1.2 Subword representations

Word-level vocabularies create an out-of-vocabulary problem. Subword methods represent a word through character n-grams, morphemes, byte-pair units, or unigram pieces. A word vector may be formed by combining vectors for its subunits:

Here z_w is an optional whole-word vector, G(w) is the set of subword units for word w, and z_g is the vector for subunit g. The expression is a conceptual representation; implementations may use averaging, summation, concatenation, or another composition function.

Subwords improve handling of rare words, spelling variations, inflections, and productive morphology. They do not, however, solve all semantic problems. A misleading segmentation or insufficient training data can still produce poor representations.

1.3 Contextual representations

Contextual models compute a token representation as a function of the entire sequence:

where x_1,…,x_T are token representations, h_t is the representation at position t, T is sequence length, and θ denotes learned parameters. The representation of “bank” can therefore differ across contexts. Transformer-based language models later made contextual representation learning dominant, although recurrent and convolutional models remain conceptually important and useful in constrained applications.

2. Neural Language Modelling

A language model estimates the probability of a token sequence using the chain rule:

The model predicts the next token from preceding context. Training usually minimizes negative log-likelihood, equivalently cross-entropy for a one-hot target:

where w_{<t} denotes preceding tokens and θ denotes model parameters. Per-token perplexity is:

when L is the total natural-log loss over N predicted tokens. Lower perplexity indicates better predictive fit on the same tokenization and evaluation distribution, but it does not necessarily imply better factuality, safety, or task performance.

A neural language model commonly consists of an embedding layer, a sequence-processing network, a projection layer, and a softmax output. Weight tying may use the same matrix for input embeddings and output projections, reducing parameters and sometimes improving generalization.

3. Convolutional Neural Networks for Text

A one-dimensional convolution applies a filter to local sequences. For an input sequence X and filter W of width k, a feature at position t can be written as:

where X_{t:t+k−1} is a local window, b is a bias, W is the filter, and φ is an activation function such as ReLU. Multiple filters detect different patterns. Max pooling selects the strongest activation over positions:

CNNs are effective for sentiment classification, spam detection, and local phrase recognition. They parallelize well and can be computationally efficient. Their receptive field is initially local; deeper layers or dilation are needed to capture long-range relationships. Pooling can also discard word-order detail and precise location information.

4. Recurrent Neural Networks

An RNN processes a sequence iteratively:

Here x_t is the input at time t, h_t is the hidden state, W_x and W_h are parameter matrices, b is a bias vector, and φ is an activation function. The hidden state carries information from earlier tokens, while an output layer may produce a prediction at every time step or only at the end.

RNNs naturally model variable-length sequences, but repeated multiplication by W_h can cause gradients to vanish or explode during backpropagation through time. If the largest relevant singular value of the recurrent transformation is below 1, repeated products tend to shrink; if it is above 1, they may grow. This is an intuition rather than a complete theorem because nonlinearities and gating also affect gradient behaviour.

Practical responses include gradient clipping, careful initialization, normalization, truncated backpropagation, and gated architectures.

5. LSTMs and Gated Sequence Models

The long short-term memory (LSTM) introduces a cell state and gates. A standard LSTM is:

The input gate i_t controls new information, the forget gate f_t controls retained information, and the output gate o_t controls exposure of the cell state. σ is the sigmoid function, tanh is the hyperbolic tangent, and ⊙ denotes element-wise multiplication. Gate values lie between 0 and 1, allowing soft control rather than a fixed memory rule.

GRUs provide a simpler alternative with update and reset gates and no separate cell state. LSTMs and GRUs often outperform plain RNNs on long dependencies, although they process tokens sequentially and can be slower to train than highly parallel architectures.

Bidirectional RNNs process a sequence from left to right and right to left. They are useful when the entire input is available, such as in named-entity recognition, but are unsuitable for strictly causal next-token prediction unless only the forward direction is used.

6. Encoder–Decoder Models

An encoder–decoder model maps an input sequence x_1,…,x_T to an output sequence y_1,…,y_S. The encoder produces hidden states and the decoder estimates:

The encoder may be an RNN, CNN, or Transformer. A basic recurrent decoder uses its previous state, the previous output token, and a context vector. During training, teacher forcing supplies the true previous token. During inference, the decoder must use its own previous prediction. This mismatch is called exposure bias and can cause error accumulation.

Decoding strategies include greedy decoding, which selects the highest-probability token at each step, and beam search, which maintains several partial hypotheses. Beam search can improve sequence likelihood but may produce bland, repetitive, or overly short outputs. Length normalization, coverage penalties, constrained decoding, and diverse decoding objectives may be useful depending on the task.

7. Attention Mechanisms

A fixed-size encoder vector can become a bottleneck for long inputs. Attention allows the decoder to access all encoder states. Given decoder query q_s and encoder states h_t, additive attention may compute:

The attention weights are:

and the context vector is:

The weights satisfy α_{s,t} ≥ 0 and ∑_{t=1}^{T} α_{s,t} = 1. Dot-product attention uses a similarity such as q_s^T h_t; scaled dot-product attention divides by √d_k to control the magnitude of dot products:

where Q, K, and V are query, key, and value matrices, and d_k is the key dimension. Multi-head attention uses several learned projections, allowing different heads to model different relationships. Attention improves access to distant information and supports parallel computation in Transformer architectures, but its weights should not automatically be interpreted as faithful explanations of model decisions.

8. Training and Generalization

Neural NLP training usually involves mini-batch stochastic gradient descent or an adaptive optimizer. For parameters θ and learning rate η, a basic update is:

A sound experimental procedure separates training, validation, and test data. The test set should be used for final evaluation, not repeated model selection. For sequence data, splitting must prevent near-duplicate documents, author overlap, temporal leakage, or shared conversations from crossing partitions.

Diagnosing common problems

Dropout randomly masks activations during training and rescales them so that expected activation magnitude is preserved. Weight decay discourages large parameter values. Early stopping selects the checkpoint with the best validation behaviour, but repeated inspection of the validation set can itself lead to overfitting.

Evaluation should match the task. Classification may use accuracy, precision, recall, and macro-F1. For a class, precision is TP/(TP+FP), while recall is TP/(TP+FN), where TP, FP, and FN denote true positives, false positives, and false negatives. Generation requires multiple perspectives: token-level likelihood, BLEU or ROUGE where appropriate, factuality, semantic adequacy, fluency, toxicity, human judgments, and task-specific utility. No single metric is sufficient.

9. Worked Example: A Small Attention Calculation

Suppose a decoder query attends to three encoder states. Let the compatibility scores be e = [1, 0, −1]. The attention weights are the softmax values:

Using exp(1) ≈ 2.718, exp(0) = 1, and exp(−1) ≈ 0.368, the denominator is approximately 4.086. Therefore, the weights are approximately [0.665, 0.245, 0.090]. They sum to approximately 1. If the corresponding scalar encoder values are [2, 4, 6], the context value is:

The first encoder state contributes most because it received the largest compatibility score. In a real model, states and contexts are vectors, and the same weighted-sum principle applies component by component.

10. Applied Case: Illustrative Customer-Support Intent System

Background

A telecommunications provider receives millions of short messages such as “my data stopped working,” “I was charged twice,” and “how do I change my plan.” The organisation wants an intent classifier that routes messages to specialised teams.

Problem

A bag-of-words baseline performs well on frequent intents but fails on spelling variation, short messages, and new product names. The company also needs reliable performance across languages and customer groups.

Analysis

The data team first defines an intent taxonomy and audits label consistency. It removes duplicate messages and creates a customer-level and time-aware split, preventing messages from the same interaction from appearing in both training and test sets. Baselines include a linear classifier over TF–IDF features and a majority-class classifier. Metrics include macro-F1, per-intent recall, calibration, and escalation rate.

Approach

The team trains a subword CNN and a bidirectional LSTM with attention. Subwords reduce failures on product codes and misspellings. The LSTM captures sequence order, while attention provides a diagnostic view of influential positions, though the team does not treat attention as a definitive explanation. Class weights are used cautiously for rare intents, and uncertain predictions are routed to human agents. Hyperparameters are selected on validation data only.

Outcome

The neural systems improve macro-F1 over the linear baseline, especially for rare spelling variants. However, performance falls on a newly launched product because of distribution shift. The production design therefore includes drift monitoring, periodic evaluation on recent labelled data, a rejection threshold, and a human escalation path.

Learning points

The case demonstrates that architecture alone does not guarantee success. Taxonomy quality, leakage-resistant evaluation, subword handling, calibration, monitoring, and operational fallback are equally important. A small increase in aggregate accuracy may not justify deployment if it increases harmful routing errors or conceals poor performance for minority language groups.

11. Limitations, Ethics, and Governance

Neural representations encode regularities from their data, including stereotypes and social prejudice. Association in an embedding space is not proof of a social fact, and debiasing methods can remove useful information or provide only superficial improvements. Evaluation should include subgroup performance, intersectional analysis where feasible, and qualitative review of harmful outputs.

Privacy is another concern. Training data may contain personal information, confidential conversations, or memorised text. Data minimisation, access controls, retention limits, redaction, privacy testing, and documented provenance are essential. In high-stakes settings, organisations should provide human oversight, contestability, audit logs, and clear accountability.

Models can also be vulnerable to adversarial spelling, prompt manipulation, poisoned data, and distribution shift. Robustness testing should include misspellings, code-switching, dialect variation, long inputs, incomplete inputs, and out-of-domain examples. Governance should document intended use, prohibited use, data sources, known limitations, evaluation populations, and rollback procedures.

12. Contemporary Directions and Best Practice

Modern NLP commonly uses large pretrained language models, subword or byte-level tokenization, self-supervised objectives, parameter-efficient adaptation, retrieval augmentation, and instruction or preference tuning. These methods improve transfer but increase concerns about compute, data provenance, hallucination, and evaluation contamination.

A professional development workflow should:

1. define the task, users, harms, and success criteria;
2. establish a simple, reproducible baseline;
3. inspect and document data and labels;
4. choose tokenization and architecture according to latency, context, and resource constraints;
5. track experiments, random seeds, versions, and preprocessing;
6. use leakage-resistant splits and confidence intervals where possible;
7. evaluate aggregate, subgroup, robustness, calibration, and operational metrics;
8. deploy with monitoring, human escalation, and rollback controls.

The best architecture is therefore not necessarily the largest one. A smaller subword model may be preferable when latency, privacy, energy consumption, interpretability, or on-device inference matters.

Chapter Summary

Neural NLP replaces sparse symbolic features with learned distributed representations. Word embeddings capture distributional regularities, while subword models improve rare-word and morphological handling. CNNs model local patterns efficiently; RNNs model sequential structure; LSTMs and GRUs mitigate, but do not eliminate, long-range gradient problems. Encoder–decoder systems generate conditional sequences, and attention provides dynamic access to encoder states. Training requires appropriate objectives, optimization, regularization, leakage-resistant data splits, and task-relevant evaluation. Deployment additionally requires robustness testing, monitoring, human oversight, privacy protection, and fairness analysis. Representation learning is powerful, but its outcomes reflect both the architecture and the data, objectives, and institutions surrounding it.

Key Terms

Embedding: A learned dense vector representation of a token or other linguistic unit.

Distributional hypothesis: The principle that linguistic units with similar contexts tend to have related meanings.

Static embedding: A representation assigning one vector to a word regardless of context.

Contextual representation: A representation whose value depends on surrounding tokens.

Subword representation: A representation constructed from character, morpheme, byte, or token fragments.

Skip-gram: An objective that predicts surrounding words from a centre word.

CBOW: An objective that predicts a centre word from surrounding words.

CNN: A convolutional neural network that detects local patterns through learned filters.

RNN: A recurrent neural network that updates a hidden state sequentially.

LSTM: A recurrent architecture using gates and a cell state to regulate information flow.

GRU: A gated recurrent unit with a simpler gated design than an LSTM.

Encoder–decoder: A conditional sequence model with an input encoder and output decoder.

Teacher forcing: Training a decoder with the true previous output token.

Exposure bias: The train–test mismatch caused by teacher forcing.

Attention: A weighted mechanism for selecting information from a set of representations.

Beam search: A decoding method that retains several high-scoring partial sequences.

Perplexity: The exponential of average negative log-likelihood under a specified tokenization.

Gradient clipping: Restricting gradient magnitude to reduce unstable updates.

Data leakage: Unintended use of information from evaluation data during training or selection.

Calibration: Agreement between predicted confidence and empirical correctness.

Review Questions

Short Answer Questions

1. Why are one-hot word representations inadequate for expressing semantic similarity?
2. Distinguish static embeddings from contextual representations.
3. What problem do subword representations address?
4. Why can plain RNNs suffer from vanishing gradients?
5. What is exposure bias in encoder–decoder training?

Long Answer Questions

1. Explain skip-gram training, including the role of negative sampling and its computational motivation.
2. Compare CNNs, RNNs, and LSTMs for text classification and sequence generation.
3. Derive the attention-weight and context-vector calculations and explain their purpose in encoder–decoder models.
4. Discuss a rigorous procedure for diagnosing overfitting and distribution shift in a neural NLP system.
5. Evaluate the ethical, privacy, and governance concerns involved in deploying learned language representations.

Analytical / Application Questions

1. Design a leakage-resistant evaluation plan for a customer-support intent classifier with repeated messages from the same users.
2. A model has low training loss but poor macro-F1 for rare intents. Diagnose at least three possible causes and propose an intervention for each.
3. Choose between a subword CNN, bidirectional LSTM, and large pretrained model for an on-device multilingual application. Justify the choice using accuracy, latency, memory, robustness, and governance considerations.

Multiple Choice Questions with Answers

Question 1

What is the principal limitation of a static word embedding?

A. It cannot represent any token numerically

B. It assigns the same vector to a word in every context

C. It always has a larger dimension than a one-hot vector

D. It requires labels for every training sentence

Correct Answer: B. It assigns the same vector to a word in every context

Question 2

In skip-gram training, the model primarily learns to:

A. Predict surrounding words from a centre word

B. Predict a centre word from surrounding words

C. Translate a source sentence into a target sentence

D. Detect convolutional features in an image

Correct Answer: A. Predict surrounding words from a centre word

Question 3

What is the main advantage of subword representations?

A. They eliminate the need for training data

B. They guarantee perfect semantic disambiguation

C. They improve handling of rare and unseen word forms

D. They prevent all demographic bias

Correct Answer: C. They improve handling of rare and unseen word forms

Question 4

Which component of an LSTM controls how much previous cell-state information is retained?

A. Input gate

B. Forget gate

C. Output projection

D. Softmax layer

Correct Answer: B. Forget gate

Question 5

Why is gradient clipping used in recurrent neural-network training?

A. To increase vocabulary size

B. To prevent all underfitting

C. To reduce the effect of excessively large gradients

D. To convert words into subwords

Correct Answer: C. To reduce the effect of excessively large gradients

Question 6

In attention, what is the result of applying softmax to compatibility scores?

A. A probability-like set of nonnegative weights summing to one

B. A binary mask with exactly one active position

C. A new vocabulary

D. A recurrent hidden state with no input

Correct Answer: A. A probability-like set of nonnegative weights summing to one

Question 7

What does teacher forcing do during decoder training?

A. Replaces the encoder with a CNN

B. Supplies the true previous output token to the decoder

C. Forces every attention weight to be equal

D. Removes the need for a loss function

Correct Answer: B. Supplies the true previous output token to the decoder

Question 8

A model has low training loss and substantially higher validation loss. Which diagnosis is most directly supported?

A. Overfitting

B. Guaranteed data sufficiency

C. Successful calibration

D. Absence of model capacity

Correct Answer: A. Overfitting

Question 9

If attention scores are [1, 0, −1], which position receives the greatest attention weight?

A. The first position

B. The second position

C. The third position

D. All positions receive exactly the same weight

Correct Answer: A. The first position

Question 10

Why should attention weights not automatically be treated as faithful explanations?

A. Attention weights are always negative

B. They may show information allocation without fully determining the model’s decision

C. They cannot be computed for sequences

D. They are unrelated to encoder states

Correct Answer: B. They may show information allocation without fully determining the model’s decision