Module 1 · Foundations

Math Foundations

Everything in both papers reduces to a handful of mathematical ideas: vectors, matrices, functions, derivatives, probabilities, and logarithms. Master these short sections and no equation in the papers will scare you.

▶

Audio recap

A ~2-minute spoken summary of this module — great for revision on the go.

1.1 Vectors: lists of numbers

A vector is just an ordered list of numbers, e.g. v = (2, −1, 3). We say this vector lives in ℝ³ ("3-dimensional real space") because it has 3 entries. In deep learning, everything becomes a vector: a DNA letter, a word, a tactile image — all get converted to lists of numbers so a model can compute with them.

Key operations:

Addition: element-wise. (1, 2) + (3, 4) = (4, 6).
Scaling: 2 · (1, 2) = (2, 4).
Dot product: multiply matching entries and sum: (1, 2)·(3, 4) = 1·3 + 2·4 = 11. The dot product measures similarity: aligned vectors give large positive values, opposed vectors give negative values, unrelated (perpendicular) vectors give 0.

Why you care Paper 2's whole method is "make the tactile vector point the same way as the language vector." Similarity of vectors — dot products and the distributions built from them — is the beating heart of representation learning, attention, and distillation.

1.2 Matrices: grids of numbers, machines that transform vectors

A matrix is a rectangular grid of numbers, e.g. a 2×3 matrix has 2 rows and 3 columns. Multiplying a matrix by a vector produces a new vector: each output entry is a dot product between one matrix row and the input. So a matrix is a machine that transforms vectors — rotating, stretching, mixing their entries.

y = W x + b

This is the single most important equation in the course: a linear layer. x is the input vector, W ("weights") is a matrix, b ("bias") is a vector, and y is the output. Training a neural network means finding good numbers to put inside W and b. Nothing more mysterious than that.

Example W = [[1, 0], [0, 2]], b = (0, 1), x = (3, 4). Then Wx = (1·3+0·4, 0·3+2·4) = (3, 8), and y = Wx + b = (3, 9).

1.3 Functions and composition

A function maps inputs to outputs: f(x) = x². Functions can be composed — fed into each other: g(f(x)). A deep neural network is nothing but a long composition:

output = f_L( … f₂( f₁(input) ) … )

where each f_i is a simple layer (a matrix multiply followed by a simple nonlinear function). "Deep" literally means "many functions composed."

One special function family you must know: nonlinear activations, like ReLU(x) = max(0, x), which outputs x if positive and 0 otherwise. Without nonlinearities between them, stacked linear layers collapse into one linear layer — the network could never learn curves, only straight lines. Paper 1's Conv1D layers use ReLU (the σ in its Equation 1).

1.4 Derivatives and gradients: which way is downhill?

The derivative of f at a point tells you how much f changes if you nudge the input slightly: the slope. If f(x) = x², then f′(x) = 2x; at x = 3 the slope is 6, so increasing x increases f. To decrease f, move x in the opposite direction of the slope.

With many inputs, we collect the slope with respect to each input into a vector called the gradient, written ∇f. The gradient points in the direction of steepest increase; its negative points steepest downhill.

Why you care Training = defining a "badness score" (the loss) as a function of all the weights, then repeatedly nudging every weight a small step downhill: w ← w − η ∇L. That step size η is the learning rate — Paper 1 sets it to 1×10⁻⁴, Paper 2 tunes it to 2×10⁻⁵. You'll meet this as gradient descent in Module 2.

For compositions there is the chain rule: the derivative of g(f(x)) is g′(f(x)) · f′(x) — multiply the slopes of each stage. Backpropagation (Module 3) is just the chain rule applied systematically through every layer of a network.

1.5 Probability in five minutes

A probability distribution over K classes is a list of K non-negative numbers that sum to 1 — e.g. over 3 materials: (0.7, 0.2, 0.1) means "70% wood, 20% steel, 10% fabric."
A model that classifies into K classes outputs exactly such a distribution, and we typically predict the class with the highest probability (the argmax).
Conditional probability p(y | x) reads "probability of label y given input x" — e.g. p(cystic fibrosis | this DNA window).
Expectation 𝔼[·] is a probability-weighted average. When papers write 𝔼_(x,y)[Loss], they mean "the average loss over the data."

1.6 Exponentials, logarithms, and softmax

exp(x) = e^x turns any real number into a positive one, exaggerating differences. log is its inverse: log(1) = 0; log of a small probability like 0.01 is a large negative number (≈ −4.6). Two properties you'll use constantly: log(ab) = log a + log b, and log is monotonic (bigger input, bigger output).

Networks internally produce raw unbounded scores called logits, one per class. The softmax function converts logits u = (u₁, …, u_K) into a valid probability distribution:

softmax(u)_c = exp(u_c)Σ_k exp(u_k)

Every entry becomes positive, and the entries sum to 1. The biggest logit gets the biggest probability.

Worked example Logits (2, 1, 0) → exp gives (7.39, 2.72, 1.00), sum ≈ 11.11 → softmax ≈ (0.67, 0.24, 0.09). A clean probability distribution favoring class 1.

Where the papers use this Paper 1's classifier ends in a softmax over 21 classes, trained with cross-entropy loss (its Eq. 5). Paper 2 goes further: it applies softmax with a temperature T (dividing logits by T before softmax) to soften the teacher's and student's feature distributions before comparing them — you'll unpack that in Module 8.

1.7 Comparing distributions: a taste of cross-entropy and KL

Once model outputs are probability distributions, we need to measure "how far" a predicted distribution p is from a target distribution q. Two related measures dominate deep learning:

Cross-entropy: −Σ_c q_c log p_c. When the target q puts all its mass on the true class (a "one-hot" target), this reduces to −log p_true: the loss is small when the model gives the true class high probability, and explodes when it gives it probability near 0.
KL divergence: KL(q ∥ p) = Σ_c q_c log(q_c/p_c) — how many "extra bits" you pay for believing p when the truth is q. It is zero exactly when p = q, and always ≥ 0. It is not symmetric: KL(q ∥ p) ≠ KL(p ∥ q) in general.

The one-sentence summary Cross-entropy trains a model to match hard labels (Paper 1's disease classes, Paper 2's material classes); KL divergence trains a student to match a teacher's whole soft distribution (Paper 2's distillation loss, its Eq. 7). Same family, different targets.

Exam warm-up — say it out loud

What does a dot product measure, geometrically?
Why do neural networks need nonlinear activations at all?
What does the gradient of a loss tell you, and what do you do with it?
Walk through softmax on logits (1, 1, 3) qualitatively — which class wins and why is the output a valid distribution?