Deep Learning and Representation Learning

Introduction and Context

Deep learning is a family of representation-learning methods in which multiple computational layers transform raw observations into increasingly abstract features. Unlike conventional machine-learning pipelines, where domain experts often design features explicitly, deep networks learn representations jointly with the predictive task. This capability has made deep learning central to computer vision, speech recognition, natural-language processing, recommender systems, scientific computing, and autonomous systems.

At postgraduate level, deep learning should be understood not merely as the use of large neural networks, but as an interaction among function approximation, optimization, statistical generalization, data representation, computational architecture, and systems engineering. A successful model depends on architecture, initialization, normalization, regularization, optimization, data quality, evaluation design, and deployment constraints.

Learning Objectives

By the end of this chapter, students should be able to:

1. Explain representation learning and the operation of multilayer neural networks.

2. Derive and apply backpropagation for common loss functions and activation functions.

3. Select initialization, normalization, regularization, and optimization strategies.

4. Analyze convolutional neural networks, attention mechanisms, transformers, and autoencoders.

5. Compare supervised, self-supervised, and generative representation-learning approaches.

6. Design a deep-learning workflow while recognizing computational, ethical, and governance risks.

1. Neural Networks as Representation Learners

A feed-forward neural network maps an input vector \(x\) to an output \(\hat y\) through a sequence of layers. For layer \(l\),

\[

z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)},\qquad a^{(l)}=\phi^{(l)}(z^{(l)}),

\]

where \(W^{(l)}\) is a weight matrix, \(b^{(l)}\) is a bias vector, \(a^{(l-1)}\) is the previous layer's activation, and \(\phi^{(l)}\) is an activation function. The input is \(a^{(0)}=x\). Hidden layers learn intermediate representations; the final layer converts the last representation into predictions.

Common activations include the rectified linear unit (ReLU), \(\phi(z)=\max(0,z)\), and the smooth GELU activation used in many transformer models. ReLU is computationally simple and reduces saturation for positive inputs, although units can become permanently inactive when their inputs remain negative. Sigmoid and hyperbolic tangent are useful in selected output or recurrent settings but can produce small derivatives at extreme values.

For binary classification, a sigmoid output is

\[

\hat y=\sigma(z)=\frac{1}{1+e^{-z}},

\]

with binary cross-entropy loss

\[

\mathcal L=-[y\log \hat y+(1-y)\log(1-\hat y)].

\]

For a mutually exclusive \(K\)-class problem, softmax produces

\[

p_k=\frac{e^{z_k}}{\sum_{j=1}^{K}e^{z_j}},

\]

and the cross-entropy for class label \(y\) is \(\mathcal L=-\log p_y\). In practice, logits should be passed directly to numerically stable loss implementations rather than exponentiated manually.

Representation learning can be supervised, where representations are optimized for labelled prediction; unsupervised, where structure is learned without labels; or self-supervised, where labels are generated from the data itself. A representation is useful when it preserves information relevant to the intended task while discarding irrelevant variation.

2. Backpropagation and Gradient-Based Learning

Training minimizes an empirical objective, commonly

\[

J(\theta)=\frac{1}{n}\sum_{i=1}^{n}\mathcal L(f_\theta(x_i),y_i)+\lambda\Omega(\theta),

\]

where \(\theta\) denotes all trainable parameters, \(n\) is the number of training examples, \(\Omega\) is a regularization term, and \(\lambda\geq 0\) controls its strength. The objective assumes that training examples are sampled from a distribution for which minimizing average loss is a meaningful proxy for future performance.

Backpropagation applies the chain rule efficiently from the output toward the input. For a scalar loss and a layer with \(z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)}\), define \(\delta^{(l)}=\partial \mathcal L/\partial z^{(l)}\). Then

\[

\delta^{(l)}=\left((W^{(l+1)})^T\delta^{(l+1)}\right)\odot \phi'(z^{(l)}),

\]

where \(\odot\) denotes elementwise multiplication. Gradients are

\[

\frac{\partial\mathcal L}{\partial W^{(l)}}=\delta^{(l)}(a^{(l-1)})^T,

\qquad

\frac{\partial\mathcal L}{\partial b^{(l)}}=\delta^{(l)}.

\]

For softmax combined with cross-entropy, the derivative with respect to the logits is especially simple: \(\partial\mathcal L/\partial z_k=p_k-\mathbb 1[k=y]\). This result assumes one-hot or equivalent single-label classification and explains why the pairing is computationally stable.

A parameter update using mini-batch stochastic gradient descent is

\[

\theta_{t+1}=\theta_t-\eta g_t,

\]

where \(g_t\) is the gradient estimated from a mini-batch and \(\eta\) is the learning rate. Mini-batches provide a compromise between noisy inexpensive updates and costly exact-gradient updates.

Worked Example: One Neuron

Consider a neuron \(z=wx+b\) with sigmoid output and one observation \(x=2\), \(y=1\), \(w=0.4\), and \(b=-0.2\). Then \(z=0.6\), and \(\hat y=\sigma(0.6)\approx0.646\). With sigmoid cross-entropy, \(\partial\mathcal L/\partial z=\hat y-y\approx-0.354\). Therefore,

\[

\frac{\partial\mathcal L}{\partial w}=(\hat y-y)x\approx-0.708,

\qquad

\frac{\partial\mathcal L}{\partial b}\approx-0.354.

\]

With learning rate \(0.1\), the updated parameters are approximately \(w=0.4708\) and \(b=-0.1646\). The positive increase in both parameters raises the logit for this positive example.

3. Initialization, Normalization, and Regularization

Initialization

Poor initialization can cause exploding or vanishing activations and gradients. If a layer has \(n_{in}\) inputs and \(n_{out}\) outputs, Xavier/Glorot initialization commonly uses variance approximately \(2/(n_{in}+n_{out})\), especially for approximately symmetric activations such as tanh. He initialization uses variance approximately \(2/n_{in}\), and is designed for ReLU-like activations. These are variance-preserving heuristics, not guarantees; their success also depends on architecture, data, and optimization.

Weights should generally not be initialized identically, because symmetry would cause equivalent units to receive identical gradients. Biases may often be initialized to zero, although specialized architectures use other choices.

Normalization

Batch normalization standardizes activations using mini-batch statistics during training and running estimates during inference. For a feature channel, a simplified form is

\[

\operatorname{BN}(x)=\gamma\frac{x-\mu_B}{\sqrt{\sigma_B^2+\epsilon}}+\beta,

\]

where \(\mu_B\) and \(\sigma_B^2\) are batch mean and variance, \(\epsilon\) prevents division by zero, and \(\gamma,\beta\) are learned scale and shift parameters. Batch normalization can improve conditioning and permit useful learning rates, but its statistics become unreliable with very small or highly variable batches.

Layer normalization computes statistics across features within each individual example. It is therefore independent of batch size and is widely used in recurrent and transformer architectures. Group and instance normalization are useful alternatives in vision, style-transfer, and small-batch settings. Normalization is not a substitute for sound data preprocessing or regularization.

Regularization

L2 weight decay penalizes large parameter values. In a simple objective, \(\Omega(\theta)=\frac12\|\theta\|_2^2\), producing an additional gradient proportional to \(\theta\). Dropout randomly removes activations during training and rescales them so that the expected activation is approximately preserved. It discourages co-adaptation but may be unnecessary or harmful in some modern architectures.

Other methods include early stopping, data augmentation, label smoothing, stochastic depth, and transfer learning. Regularization should be selected according to the source of overfitting: augmentation addresses input variation, weight decay controls parameter magnitude, and early stopping limits fitting to training noise. Validation data must not be repeatedly exploited without control, since it then becomes part of the effective training process.

4. Optimization and Training Diagnostics

Momentum maintains a moving direction of past gradients, often accelerating progress along consistent directions. Adaptive methods such as Adam maintain first- and second-moment estimates and scale updates coordinate-wise. AdamW separates decoupled weight decay from the adaptive gradient update, which generally makes the regularization interpretation clearer than simply adding an L2 term to Adam's gradient.

Learning-rate schedules include warm-up, cosine decay, step decay, and one-cycle policies. A large model can fail because the learning rate is too high, too low, or changed at an inappropriate time. Training loss, validation loss, gradient norms, activation distributions, calibration, and task-specific metrics should be monitored together.

Overfitting is indicated by improving training performance while validation performance degrades. Underfitting may result from insufficient capacity, poor optimization, noisy features, or excessive regularization. Data leakage, class imbalance, distribution shift, and an inappropriate validation split can mimic or conceal either condition.

5. Convolutional Neural Networks

A convolutional layer applies local filters across a spatial or temporal structure. For a one-dimensional signal, a cross-correlation form is

\[

y[i]=b+\sum_{c=1}^{C_{in}}\sum_{r=0}^{K-1}w[c,r]x[c,i+r],

\]

where \(C_{in}\) is the number of input channels and \(K\) is kernel width. Deep-learning libraries generally implement cross-correlation while referring to it as convolution. Stride controls displacement, padding controls boundary treatment, and dilation spaces kernel elements.

For a two-dimensional layer with input height \(H\), kernel size \(K ext{ by }K\), padding \(P\), and stride \(S ext{,}

\[

H_{out}=\left\lfloor\frac{H+2P-K}{S}\right\rfloor+1,

\]

assuming symmetric padding and the standard discrete output convention. The width has an analogous expression. A convolution shares parameters across positions, creating translation-aware inductive bias and fewer parameters than a fully connected layer.

Pooling, strided convolution, or learned downsampling reduces spatial resolution and increases receptive field. Residual connections use \(a^{(l+1)}=F(a^{(l)})+a^{(l)}\), when dimensions match, making optimization of deep networks easier by providing a direct information path. Modern CNN design also uses depthwise separable convolution, feature pyramids, and attention-like channel recalibration.

CNNs remain effective where locality, spatial structure, and computational efficiency matter. Their limitations include sensitivity to distribution shifts, potentially weak long-range interaction without sufficient depth, and dependence on representative training data.

6. Attention and Transformer Architectures

Attention allows a representation to selectively combine information from other positions. Given query matrix \(Q\), key matrix \(K\), and value matrix \(V\), scaled dot-product attention is

\[

\operatorname{Attention}(Q,K,V)=\operatorname{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V,

\]

where \(d_k\) is the key dimension. Scaling reduces the tendency of large dot products to push softmax into saturated regions. A mask may set disallowed logits to a very negative value, such as in causal language modeling where a position cannot attend to future tokens.

Multi-head attention projects inputs into several query-key-value subspaces, computes attention in parallel, concatenates the results, and applies an output projection. A transformer block typically contains attention, a position-wise feed-forward network, residual connections, and layer normalization. Positional information is necessary because standard self-attention alone is permutation-equivariant; it does not inherently know token order. Positional encodings may be fixed, learned, or represented through relative-position mechanisms.

Self-attention provides flexible long-range interaction but standard attention has quadratic time and memory complexity in sequence length, because it forms an \(n\times n\) attention matrix. Efficient attention, sparse attention, linear approximations, recurrent memory, and retrieval mechanisms address this limitation. Transformers are powerful but computationally demanding and can reproduce biases and confidential information present in training data.

7. Autoencoders and Generative Representation Learning

An autoencoder consists of an encoder \(z=f_\theta(x)\) and decoder \(\hat x=g_\phi(z)\). It minimizes a reconstruction objective such as

\[

\mathcal L_{AE}=\frac{1}{n}\sum_i\|x_i-g_\phi(f_\theta(x_i))\|_2^2,

\]

for continuous data under a squared-error assumption. A bottleneck or other constraint prevents the model from simply copying the input. Denoising autoencoders reconstruct clean inputs from corrupted ones and can learn representations robust to perturbations.

A variational autoencoder (VAE) defines a probabilistic latent variable model. Its objective is an evidence lower bound,

\[

\mathcal L_{VAE}=\mathbb E_{q_\theta(z|x)}[-\log p_\phi(x|z)] + D_{KL}(q_\theta(z|x)\|p(z)),

\]

written here as a loss to minimize. The first term is reconstruction negative log-likelihood; the second encourages the approximate posterior to remain near a prior, commonly a standard normal distribution. The expectation is estimated by sampling, usually through the reparameterization \(z=\mu+\sigma\odot\epsilon\), with \(\epsilon\sim\mathcal N(0,I)\). VAEs provide structured probabilistic latents but may generate blurrier samples than adversarial or diffusion approaches.

Contrastive self-supervised learning trains representations so that related views have similar embeddings and unrelated examples are separated. The definition of positive pairs, augmentation policy, batch composition, and temperature can strongly affect the learned representation. Masked prediction, in which portions of input are hidden and reconstructed or predicted, is another influential self-supervised paradigm.

8. Scalable Representation Learning and Transfer

Large-scale systems often learn general representations through pretraining and adapt them to downstream tasks. Full fine-tuning updates all parameters; feature extraction freezes the backbone; parameter-efficient methods update adapters, low-rank modules, prompts, or selected layers. The choice depends on compute, data size, privacy, latency, and the degree of domain shift.

Distributed training uses data parallelism, model parallelism, pipeline parallelism, or combinations. Mixed-precision arithmetic reduces memory and can increase throughput, but loss scaling and numerical monitoring may be required. Checkpointing, deterministic data pipelines, experiment tracking, and versioned datasets are essential for reproducibility.

Representation quality should not be judged solely by training accuracy. Useful evaluations include downstream transfer, linear probing, robustness under perturbation, subgroup performance, calibration, retrieval quality, and nearest-neighbor structure. A representation can perform well on an average metric while encoding spurious correlations or underrepresenting minority populations.

9. Illustrative Case: Defect Detection in Semiconductor Manufacturing

Background and Problem

A manufacturer inspects wafer images for microscopic defects. Defects are rare, labels are expensive, and production equipment changes over time. A conventional classifier trained on one tool's images performs well in the laboratory but degrades after deployment.

Analysis

The principal risks are severe class imbalance, domain shift between tools, leakage caused by placing images from the same wafer in both training and validation sets, and false negatives that are more costly than false positives. Images also contain nuisance information such as tool identifiers and lighting patterns.

Approach

An illustrative solution uses a CNN backbone initialized from a large visual dataset, followed by supervised fine-tuning on wafer images. Splits are made by wafer and production period rather than by individual image. Augmentation models plausible changes in illumination and small geometric variation, while avoiding transformations that would erase defect orientation or morphology. Weighted or focal loss is considered, but the decision threshold is selected using production costs rather than default probability 0.5.

A self-supervised pretraining stage on unlabelled factory images learns general visual features. Batch normalization statistics are checked under the new domain, and a small, independently collected temporal test set measures drift. Grad-CAM-like explanations are used as an audit aid, not as proof of causal reasoning. Performance is reported with precision-recall curves, recall at an acceptable inspection workload, calibration, and subgroup or tool-level analysis.

Outcome and Learning Points

The expected benefit is better label efficiency and improved transfer to new equipment, not guaranteed elimination of false alarms. Human inspectors remain in the decision loop for uncertain cases. The case demonstrates that architecture alone cannot solve data-splitting, shift, cost-sensitive evaluation, or governance problems.

10. Limitations, Ethics, and Governance

Deep models can be opaque, computationally expensive, brittle under distribution shift, and vulnerable to adversarial or accidental perturbations. Memorization can create privacy risks, especially when training data contain personal or proprietary information. Generative models may reproduce copyrighted material, sensitive attributes, or training-set records.

Responsible deployment requires data provenance, lawful processing, access controls, retention limits, documentation, security testing, bias assessment, model cards or equivalent records, and monitoring after deployment. High-impact applications require human oversight, contestability, traceable decisions, and clear accountability. Explainability methods should be validated for faithfulness rather than treated as definitive causal explanations.

Environmental costs include energy and hardware use. Engineers should consider smaller architectures, transfer learning, distillation, quantization, sparse computation, and efficient inference when they preserve required performance. Model selection is therefore a multi-objective engineering decision involving accuracy, latency, cost, safety, privacy, and maintainability.

11. Best-Practice Design Workflow

A defensible deep-learning project should begin with a clearly defined task, prediction target, decision context, and failure cost. Data should be audited for representativeness, leakage, missingness, duplicates, and label quality. Establishing a simple baseline provides an essential comparison.

Next, select an architecture whose inductive bias matches the data: CNNs for strong local spatial structure, transformers when flexible long-range interaction is valuable, and autoencoders or self-supervised objectives when labels are scarce. Use validated initialization, suitable normalization, explicit train-validation-test separation, and a controlled hyperparameter budget.

During training, log configurations, random seeds, data versions, metrics, resource use, and checkpoints. Diagnose gradients and learning curves rather than tuning blindly. Evaluate on temporally or operationally realistic data, report uncertainty where possible, and test subgroup and stress performance. Before deployment, define rollback criteria, monitoring signals, retraining triggers, and human escalation procedures.

Chapter Summary

Deep learning learns hierarchical representations through differentiable compositions of layers. Backpropagation applies the chain rule to obtain efficient gradients, while optimization methods update parameters using mini-batch estimates. Successful training depends on suitable initialization, normalization, regularization, learning-rate control, and careful diagnostics.

CNNs exploit locality and parameter sharing; attention enables content-dependent interaction; transformers combine attention with feed-forward, positional, residual, and normalization mechanisms. Autoencoders learn compressed or denoising representations, while VAEs and contrastive or masked self-supervised methods extend representation learning beyond ordinary supervised labels. Transfer learning and scalable distributed training make these methods practical, but computational cost, domain shift, fairness, privacy, interpretability, and governance remain central concerns. A reliable system is therefore defined by its complete data, model, evaluation, deployment, and monitoring lifecycle.

Key Terms

Representation learning: Learning features directly from data for prediction or generation.

Activation function: A nonlinear function applied to a layer's pre-activation.

Backpropagation: Reverse-mode application of the chain rule for computing gradients.

Logit: An unnormalized output score before sigmoid or softmax conversion.

Cross-entropy: A loss measuring discrepancy between target distributions and predicted probabilities.

Initialization: The rule used to set parameters before training.

Batch normalization: Normalization using mini-batch statistics during training.

Layer normalization: Normalization across features within an individual example.

Weight decay: A regularization mechanism that discourages large weights.

Dropout: Stochastic removal of activations during training.

Convolution: A locally connected, parameter-sharing operation over structured data.

Receptive field: The input region that can influence a unit's activation.

Residual connection: A shortcut that adds a block input to its transformed output.

Attention: A weighted combination of values determined by query-key compatibility.

Transformer: An architecture built principally from attention and position-wise transformations.

Autoencoder: An encoder-decoder trained to reconstruct input data.

Variational autoencoder: A probabilistic autoencoder with a regularized latent distribution.

Self-supervised learning: Learning from targets generated from the data itself.

Transfer learning: Reusing a pretrained model or representation for another task.

Distribution shift: A change between training and deployment data distributions.

Review Questions

Short Answer Questions

1. What role does the chain rule play in backpropagation?

2. Why is He initialization commonly used with ReLU activations?

3. Distinguish batch normalization from layer normalization.

4. What inductive bias does a convolutional layer provide?

5. Why does standard self-attention require positional information for ordered sequences?

Long Answer Questions

1. Derive the gradient updates for a single sigmoid neuron trained with binary cross-entropy.

2. Compare SGD with momentum, Adam, and AdamW, including situations in which each may be appropriate.

3. Explain how residual connections, normalization, and initialization support training of deep networks.

4. Compare CNNs and transformers for image or sequence representation learning.

5. Explain the objectives and limitations of ordinary autoencoders, denoising autoencoders, and VAEs.

Analytical / Application Questions

1. Design an evaluation protocol for a defect-detection model when images from the same physical object must not cross train and test sets.

2. A transformer is accurate but exceeds the deployment latency budget. Propose and justify at least three methods for reducing inference cost.

3. A model has high average accuracy but substantially lower recall for a minority subgroup. Analyze possible causes and propose technical and governance responses.

Multiple Choice Questions with Answers

Question 1

For softmax combined with single-label cross-entropy, what is the derivative of the loss with respect to logit z_k?

A. p_k

B. p_k minus the indicator that k equals y

C. y minus p_k squared

D. 1 minus p_k

Correct Answer: B. p_k minus the indicator that k equals y

Question 2

Which initialization is particularly designed for ReLU-like activations?

A. He initialization

B. Zero initialization for every weight

C. A fixed value of one for every weight

D. A distribution with variance proportional only to batch size

Correct Answer: A. He initialization

Question 3

What is the main inductive bias of a standard convolutional layer?

A. Global connectivity with no parameter sharing

B. Local connectivity and parameter sharing across positions

C. Random connectivity that changes at every inference step

D. Exclusive dependence on class labels

Correct Answer: B. Local connectivity and parameter sharing across positions

Question 4

In scaled dot-product attention, why is QK transpose divided by the square root of d_k?

A. To remove positional information

B. To make all values exactly binary

C. To reduce excessively large dot products and softmax saturation

D. To eliminate the value matrix

Correct Answer: C. To reduce excessively large dot products and softmax saturation

Question 5

If an input has height 32, kernel size 3, padding 1, and stride 2, what is the output height under the standard convolution formula?

A. 15

B. 16

C. 17

D. 32

Correct Answer: B. 16

Question 6

Which normalization computes statistics across features within each individual example?

A. Batch normalization

B. Layer normalization

C. Dataset-level min-max scaling only

D. Dropout

Correct Answer: B. Layer normalization

Question 7

What prevents an underconstrained autoencoder from simply learning an identity mapping?

A. Using no decoder

B. A bottleneck or another learning constraint

C. Increasing the input dimensionality without limit

D. Removing the reconstruction objective

Correct Answer: B. A bottleneck or another learning constraint

Question 8

What is a principal computational limitation of standard self-attention for sequence length n?

A. It cannot use matrix multiplication

B. Its attention matrix has quadratic size in n

C. It always uses constant memory

D. It cannot represent relationships between positions

Correct Answer: B. Its attention matrix has quadratic size in n

Question 9

Which split is most appropriate for assessing wafer-defect generalization when images from one wafer are correlated?

A. Randomly split individual pixels

B. Split individual images without regard to wafer identity

C. Split by wafer, with a realistic temporal test set

D. Use only training accuracy

Correct Answer: C. Split by wafer, with a realistic temporal test set

Question 10

Which statement best describes self-supervised learning?

A. It requires manually labelled targets for every example

B. It creates training signals from structure or transformations in the data

C. It cannot learn representations

D. It is identical to random initialization

Correct Answer: B. It creates training signals from structure or transformations in the data