Module 3 · Foundations

Neural Networks

Here the pieces from Modules 1–2 assemble into an actual trainable network: neurons, layers, the forward pass, backpropagation, the Adam optimizer, regularization, and the training curves your examiners will put in front of you.

▶

Audio recap

A ~2-minute spoken summary of this module — great for revision on the go.

3.1 From one neuron to a layer

A single artificial neuron takes an input vector x, computes a dot product with its own weight vector w, adds a bias b, and passes the result through a nonlinear activation σ (e.g. ReLU): output = σ(w·x + b). One neuron, one number out.

A layer is just many neurons looking at the same input. Stack their weight vectors as rows of a matrix W and their biases into a vector b, and the whole layer is Module 1's favorite equation with a nonlinearity wrapped around it:

y = σ(W x + b)

Layers between the input and the output are called hidden layers, and a plain stack of such fully-connected layers is a multi-layer perceptron (MLP) — the simplest deep network.

In the papers Paper 2 uses a small MLP as its lightweight classification head: the heavy lifting is done by the ViT encoder, and a modest MLP on top maps the learned tactile representation to the 32 material classes. Paper 1's dense layers after the BiLSTM play the same role.

3.2 The forward pass

The forward pass is simply running the composition of Module 1.3: the input enters layer 1, its output feeds layer 2, and so on until the final layer. For a K-class classifier, the last layer outputs K raw scores — the logits — and softmax (Module 1.6) converts them into a probability distribution over the classes.

input x→ layer 1→ layer 2→ …→ logits→ softmax→ p(class | x)

During training, the forward pass ends with one more step: the predicted distribution and the one-hot label go into the cross-entropy loss, producing the single badness number the whole update revolves around.

3.3 Backpropagation

To do gradient descent we need ∂L/∂w for every weight in the network — possibly millions of them. Backpropagation is the chain rule (Module 1.4) organized efficiently: starting from the loss, it sweeps backward through the layers, reusing intermediate results so that one forward pass plus one backward pass yields the gradient of the loss with respect to every single weight.

You will never derive these gradients by hand. Modern frameworks record the forward computation and apply the chain rule automatically ("autodiff") — PyTorch, which Paper 2 uses, does this with a single call. What you must be able to say in the exam: backprop computes gradients, it is exact (not an approximation), and its cost is roughly the same order as the forward pass.

One training step, end to end Forward pass (batch → predictions → loss) → backward pass (backprop gives ∇L for all weights) → optimizer update (w ← w − η∇L, or Adam's smarter version below). Repeat for the next batch. That loop, millions of times, is all of deep learning training.

3.4 Optimizers: from SGD to Adam

Plain SGD takes the raw (noisy) mini-batch gradient and steps. Two upgrades dominate practice:

Momentum: keep a running average of recent gradients and step along that instead. Intuition: a ball rolling downhill doesn't jerk to a new direction at every pebble — accumulated velocity smooths out mini-batch noise and speeds travel along consistent slopes.
Adaptive learning rates: give each parameter its own effective step size, scaled by the history of its gradient magnitudes — parameters with consistently large gradients get gentler steps, rarely-updated ones get bolder steps.

Adam combines both: it tracks a momentum-style running mean of the gradient (controlled by β₁) and a running mean of its squared magnitude (β₂), and divides the step by the square root of the latter (with a tiny ε to avoid dividing by zero).

In the papers Paper 1 trains with Adam, initial learning rate 1×10⁻⁴, β₁ = 0.9, β₂ = 0.999, ε = 1×10⁻⁸ (the standard defaults), justifying the choice as "adaptive learning rates per parameter, stable convergence." In the exam, be ready to SAY why Adam over plain SGD: per-parameter adaptive step sizes plus momentum give fast, stable convergence with little manual learning-rate tuning — valuable when different parts of the network (embedding, conv filters, LSTM gates) need very different step sizes.

3.5 Regularization

Regularization is any technique that trades a little training-set fit for better generalization (the overfitting cure family from Module 2.6). The two you must know cold:

Dropout: during training, randomly zero out each unit in a layer with some probability (e.g. 0.5), a fresh random mask every batch; at test time, use all units. Because any unit might vanish at any moment, no unit can rely on ("co-adapt with") specific others — each must carry independently useful signal. The effect resembles training a huge ensemble of thinned networks that share weights.
Weight decay (L2 regularization) in one line: add a penalty proportional to the size of the weights to the loss, gently shrinking them toward zero so the model prefers simpler functions.

Data augmentation — creating extra training examples by transforming real ones — is regularization by another route: it enlarges and diversifies the training set instead of constraining the model. Both papers rely on it; the details live in Module 4.

3.6 Early stopping and checkpoints

Early stopping operationalizes the overfitting diagnostic from Module 2.6: after every epoch, measure the validation loss. If it hasn't improved for a set number of epochs — the patience — stop training. Then restore the checkpoint (saved copy of the weights) from the epoch with the best validation loss, not the last epoch's weights: those final epochs may already have drifted into overfitting.

In the papers Paper 1: trains for a maximum of 50 epochs with early stopping at patience 7 on validation loss, restoring the best-validation-loss weights. In practice training typically converged in about 12 epochs — the 50 is a ceiling, not a target. Know all three numbers; "why didn't you train for all 50 epochs?" is a classic probe.

Why it counts as regularization Early stopping halts the descent before the model has time to fit training-set noise. It's the cheapest regularizer there is: one extra validation pass per epoch and a saved file.

3.7 Training curves you must be able to read

Examiners love handing you a plot of loss (or accuracy) vs. epochs, with one curve for training and one for validation, and asking "what's happening here?" The three patterns to recognize instantly:

Pattern	What you see	Diagnosis / action
Healthy convergence	Both curves fall together and flatten out, with a small, stable gap (validation slightly above training)	All is well — stop when the validation curve flattens (early stopping does this for you)
Overfitting	Training loss keeps falling; validation loss bottoms out and climbs — the gap widens	Regularize: dropout, weight decay, augmentation, more data; stop at the validation minimum
Learning rate too high	Loss oscillates wildly, plateaus at a high value, or explodes upward — often from the very first epochs, on both curves	Lower η; the optimizer is overshooting (Module 2.4's divergence)

Two more cues worth naming aloud: both curves flat and high from the start suggests underfitting (model too small, or η too small to move); training accuracy near 100% with much lower validation accuracy is the accuracy-view of overfitting. Paper 1's reported behavior — validation loss tracking training loss down and flattening by ~epoch 12 — is the healthy pattern.

Exam warm-up — say it out loud

Write (or say) the equation for one fully-connected layer and name every symbol in it.
In one breath: what does backpropagation compute, and how is it related to the chain rule?
Why did Paper 1 choose Adam over plain SGD? Include the words "per-parameter" and "momentum."
Explain Paper 1's early-stopping setup (50, 7, ~12) and why the best-validation checkpoint is restored rather than the final weights.