Module 8 · Core Architectures

Knowledge Distillation & Transfer

The last block of general machinery: how knowledge moves between models (teacher → student), between tasks (pretraining → fine-tuning), and between data regimes (big data → few-shot). Paper 2 is built entirely out of these ideas — here you learn each one in general form; Module 10 assembles them into the full paper.

▶

Audio recap

A ~2-minute spoken summary of this module — great for revision on the go.

8.1 Transfer learning: don't start from zero

Transfer learning means reusing knowledge acquired on one task or domain as the starting point for another. Instead of initializing a network with random weights and hoping your small dataset can teach it everything, you start from a model pretrained on a large, related corpus — ImageNet for vision, web text for language — and adapt it.

Why training from scratch is wasteful: the early layers of any perception model spend most of their capacity learning generic structure (edges, textures, word meanings) that is the same across tasks. If your target dataset is small, relearning all of that from random weights either fails outright or overfits. A pretrained model has already paid that cost on someone else's data.

Why you care Both papers lean on this. Paper 2 uses two pretrained models: a pretrained ViT student and a pretrained, frozen BART teacher whose language knowledge — acquired from massive text corpora — is the whole supervisory signal. The paper never trains a language model; it borrows one.

8.2 Fine-tuning vs freezing

Once you have a pretrained model, adaptation comes in two flavors:

Full fine-tuning: unfreeze everything and let all weights move on the new task. Maximum flexibility — but with little data, the model can overwrite (and destroy) the general structure it arrived with, a failure often called catastrophic forgetting, and it can overfit the small dataset.
Frozen encoder + small head: keep the pretrained encoder's weights fixed, and train only a small new classifier (often a one- or two-layer MLP) on top of its features. Cheap, fast, preserves the learned representation, and dramatically reduces the number of parameters your small dataset must pin down — so overfitting risk plummets.

	Full fine-tuning	Frozen encoder + head
Trainable parameters	All (millions)	Tiny head only
Data needed	A lot	Very little
Risk	Overfitting, forgetting pretrained structure	Underfitting if the features don't suit the task
Cost per new task/sensor	Full retrain	Train one small MLP

Where Paper 2 uses this After distillation, Paper 2 freezes the tactile encoder (θ* = θ) and trains only a lightweight MLP classifier h_φ per downstream dataset — just 0.017% of all parameters are trainable. Freezing is not just economy: it protects the language-aligned semantic structure the distillation built, so each new sensor or dataset adapts without eroding it.

8.3 Knowledge distillation: the classic recipe

Knowledge distillation (Hinton et al., 2015) trains a student network to imitate a teacher network. The trick is what the student imitates: not the teacher's hard predictions, but its full soft probability distribution over classes.

Those soft targets carry what Hinton called "dark knowledge." A hard label says only "this is a truck." The teacher's full distribution says "truck 0.85, car 0.12, carrot 0.0001" — encoding that a truck resembles a car far more than a carrot. That relational structure between classes is information a one-hot label simply does not contain, and a student trained on it generalizes better.

Temperature: turning up the dark knowledge

A confident teacher's softmax output is nearly one-hot — the interesting small probabilities are squashed toward zero. The fix: divide the logits u by a temperature T > 1 before the softmax:

p_c^(T) = exp(u_c / T)Σ_k exp(u_k / T)

Larger T flattens (softens) the distribution, magnifying the relative similarities between non-winning classes; T = 1 recovers the ordinary softmax; T → ∞ approaches uniform. The student then minimizes the KL divergence (Module 1.7) from the teacher's softened distribution to its own softened distribution — "believe what the teacher believes, including the doubts."

Worked intuition Teacher logits (8, 4, 1). At T = 1: softmax ≈ (0.98, 0.018, 0.001) — nearly one-hot, similarities invisible. At T = 3.5: logits become (2.29, 1.14, 0.29) → softmax ≈ (0.63, 0.20, 0.09)... suddenly the student can see that class 2 is much closer to class 1 than class 3 is. That visibility is the entire point of temperature.

8.4 Cross-modal distillation: language teaches touch

Nothing in the recipe requires teacher and student to share a modality. The teacher can read text while the student reads images — as long as their outputs are compared in a common space. This is cross-modal distillation: transferring the semantic structure of one modality's representation into another's.

Paper 2's twist: instead of distilling class logits, it distills the d-dimensional feature vectors themselves. The frozen BART teacher produces z_text = g_BART(ℓ); the trainable ViT student produces z_tactile = f_θ(x). Each feature vector is treated as if it were a vector of logits: softened by temperature, turned into a distribution, and matched:

p_t^(T) = Softmax(z_text / T), p_s^(T) = Softmax(z_tactile / T)

(Paper 2, Eq. 7)L_KD-feat = KL( p_t^(T) ∥ p_s^(T) )

Minimizing this pulls the tactile embedding toward the language embedding — the student learns to "point the same way" as the teacher in the shared d-dimensional space, dimension by dimension, with temperature exposing the relative structure across dimensions.

Why features, not logits, across modalities Classic distillation matches class-probability outputs. But across modalities the class-logit spaces don't align well — the teacher was never trained on the student's classes, and logits compress away most of the representation. The intermediate features are where the semantic structure lives. Paper 2's ablation AS-2 confirms it empirically: feature-level KD scores 82.66 vs 58.64 for cosine-similarity matching and 34.02 for DKD (a logit-based method) — a +20 to +50 point gap. Aligning representations beats matching predictions when the teacher speaks a different modality.

8.5 Combining losses: the α blend

The student has two masters: the ground-truth labels (via cross-entropy, L_student, Paper 2's Eq. 8) and the teacher (via L_KD). A weighting hyperparameter blends them:

(Paper 2, Eq. 9)L = α · L_student + (1 − α) · L_KD

Read the extremes — examiners love them:

α = 1: pure supervised training. No language guidance at all — you've deleted the paper's contribution and are back to an ordinary tactile classifier.
α = 0: pure teacher-following. No label signal — the student mimics language embeddings but is never told what the material classes actually are.
α = 0.25 (the paper's best): the teacher term dominates at 75% weight, but the 25% of label supervision anchors the representation to the task. Ablation AS-3: α = 0.25 scores 83.29 vs 80.68 at α = 0.8. Both purely supervised and heavily teacher-dominated regimes underperform the moderate blend.

The general lesson Any multi-objective loss needs a balancing hyperparameter, and the balance is an empirical question — you tune it, you ablate it, and you should be able to say what each extreme means. The same pattern appears everywhere: regularization weights, auxiliary losses, GAN objectives. Paper 2 also tunes the temperature the same way: T = 3.5 (83.46) beats T = 3.0, 4.5, and 5.0.

8.6 Few-shot learning: new classes from K examples

Few-shot learning asks a model to handle new classes given only K labeled examples per class ("K-shot"; K = 0 is zero-shot). The standard protocol matches Section 8.2 exactly: freeze the representation, train a tiny classifier on the K samples per class, and measure accuracy.

Why can this work at all? Because if the representation is good, examples of each class already form tight, linearly separable clusters in embedding space — so even a trivial classifier fitted on a handful of points can draw the boundaries. Few-shot performance is therefore the sharpest test of representation quality: it measures what the encoder knows, not what the classifier can memorize.

Where Paper 2 uses this Paper 2 evaluates K ∈ {0, 10, 100, 1000}. The language-guided model shows its largest gains over the baseline in the low-data regimes: even 10 examples per class yield a substantial jump, and at 100 shots it reaches about 95% accuracy — approaching its own 1000-shot performance. That data-efficiency is the practical payoff of language-guided representations: a new material (or a new sensor, Section 8.7) costs a handful of labeled touches, not a data-collection campaign.

8.7 Domain shift & cross-sensor transfer

Domain shift is the general disease: the test distribution differs from the training distribution. A diagnostic model trained in one hospital meets another hospital's scanner; a speech model trained on adults meets children; a tactile model trained on one sensor meets different optics, gel, and lighting. Models that latched onto domain-specific artifacts collapse.

The general cure is a domain-invariant representation: features that encode the underlying semantics (the material, the pathology) while discarding the nuisance factors (the sensor, the scanner). With such features, moving to a new domain needs only light adaptation — a small head, a few shots — instead of full retraining.

Domain A data→ Invariant encoder→ Shared semantic space← Same encoder← Domain B data

Both papers face a version of this Paper 2: the domains are sensors. Its DIGIT-trained encoder, anchored to sensor-agnostic language, transfers to unseen GelSight Hex sensors with a frozen encoder and a few-shot head — that's cross-sensor transfer, the paper's headline claim (numbers in Module 10). Paper 1: the analogous worry is that its negatives and augmented training data are partly synthetic, while deployment means real clinical variants — a train/test distribution gap its authors acknowledge and defer to future real-world validation. Same disease, different clothes: expect the examiner to ask you to connect them.

Exam warm-up — say it out loud

Explain "dark knowledge" in one sentence, with the truck/car/carrot example.
Why divide logits by a temperature T > 1 before the softmax? What happens as T → 1 and T → ∞?
What would setting α = 1 in Paper 2's combined loss mean? And α = 0?
Why does freezing the encoder help in low-data regimes rather than hurt?
Why does Paper 2 distill features rather than class logits across modalities?