Module 7 · Core Architectures

Transformers & Representation Learning

Module 6 ended with attention as a garnish on a BiLSTM. Here attention becomes the whole meal: the Transformer, its vision variant (ViT), and the language model BART — the two encoders of Paper 2 — plus the big reframe that makes Paper 2 tick: networks as embedding machines, and UMAP as the tool for looking at what they learned.

▶

Audio recap

A ~2-minute spoken summary of this module — great for revision on the go.

7.1 Attention as the main act

An RNN reaches distant context only by relaying it step-by-step through its memory. Self-attention removes the relay: every position looks at every other position directly, in one shot. The mechanism is a lookup with three learned roles for each element:

Query (Q): the question this element asks — "what am I looking for?"
Key (K): the advertisement — "here's what I hold."
Value (V): the content actually handed over if selected.

Each position's Query is compared (dot product — Module 1's similarity measure) against every position's Key; the similarities are softmaxed into weights; the output is the weighted average of the Values:

output_i = Σ_j softmax(Q_i·K_j)_j V_j

Two advantages over recurrence, one cost:

Direct long-range connections: position 1 attends to position 100 in a single step — no vanishing relay.
Parallelism: no step-by-step dependency, so all positions compute at once — ideal for GPUs.
Cost: every pair of positions interacts, so compute grows quadratically with sequence length.

7.2 The Transformer in one diagram

A Transformer is simply a stack of identical blocks, each pairing self-attention (positions exchange information) with a small feed-forward MLP (each position processes what it gathered), wrapped in residual connections (Module 5's shortcut trick) that keep gradients healthy through many layers:

input tokens + positional encoding→ self-attention→ feed-forward MLP→ × N blocks→ output representations

One detail matters for exams: attention itself is order-blind — a weighted average doesn't care where its inputs sat. So the Transformer injects positional encodings (a position-dependent vector added to each token embedding) so the model knows position 5 from position 50. Beyond that: no recurrence at all. The RNN's defining mechanism is simply gone.

7.3 Vision Transformer (ViT)

Transformers were built for word sequences. The Vision Transformer (ViT) asks: what if an image is just a sequence too? Its recipe — famously titled "an image is worth 16×16 words":

Split the image into a grid of fixed-size patches (e.g. 16×16 pixels).
Flatten each patch into a vector and project it — the patch embedding. Patches now play the role of words.
Prepend a special learnable [CLS] token, then run the whole sequence through a standard Transformer.
After the last block, the [CLS] position's vector (or a pooled vector) serves as the representation of the entire image.

Where Paper 2 uses this Paper 2's tactile encoder is a ViT. It takes a tactile image x from any sensor and produces a single embedding vector

z_tactile = f_θ(x) ∈ ℝ^d

where θ are the trainable weights. This one vector is the student's entire summary of the touch — the thing distillation (Module 8) will pull toward language.

ViT vs. CNN (Module 5) A CNN builds in assumptions: locality (nearby pixels relate) and translation invariance (a motif is a motif anywhere). A ViT builds in almost nothing — self-attention gives it a global receptive field from layer 1 (any patch can attend to any other immediately, where a CNN needs many layers to see that far). Fewer built-in assumptions cuts both ways: more flexible, but it must learn what CNNs assume — so ViTs need more data or a strong supervision signal. Paper 2's language teacher is exactly such a signal.

7.4 Language models and BART

A pretrained language model is a Transformer trained on huge text corpora until its internal representations encode a great deal of meaning: words used similarly end up with similar vectors. BART is one such model, with two defining traits:

Seq2seq structure: an encoder-decoder Transformer — an encoder reads text into representations, a decoder generates text from them.
Denoising pretraining: take clean text, corrupt it (mask spans, shuffle sentences, delete words), and train BART to reconstruct the original. To un-corrupt text you must genuinely understand it, so the representations become semantically rich.

For Paper 2 the generation half is irrelevant. What matters: feed BART a tactile description ℓ like "creased, smooth" and read out a semantic embedding:

z_text = g_BART(ℓ) ∈ ℝ^d

Frozen teacher, and why BART specifically Paper 2 keeps BART frozen — its weights never update during training (θ*_text = θ_text). The pretrained semantic space is the whole point: freezing it makes it a stable teaching signal the student can be pulled toward, rather than a moving target that could drift or collapse toward the student. And the choice of teacher matters — ablation AS-1: BART scores 58.64 vs. RoBERTa's 42.69 (over 15 points worse) and DistilBERT's 53.86 (~5 worse). The reading: a higher-capacity seq2seq teacher, pretrained to reconstruct text, provides richer semantic supervision than encoder-only alternatives.

7.5 Representation learning & embedding spaces

Now the big reframe. So far we've treated networks as classifiers: input → label. Representation learning treats them as feature extractors: input → a d-dimensional embedding vector, a point in ℝ^d. The classifier at the end is just a thin final layer; the embedding is where the knowledge lives.

The organizing principle: geometry = meaning. A good embedding space places similar inputs close together and dissimilar inputs far apart — so distances and dot products (Module 1) measure semantics.

Paper 2's central move is a shared semantic space: two different encoders — the ViT for touch and BART for language — both map into the same d-dimensional space. Success means a tactile image of leather lands near the text embedding of "uneven, firm, undulating." Once touch and language coexist geometrically, the sensor-agnostic structure of language (Module 0) can organize the tactile features — that's the conceptual heart of Paper 2, formalized as distillation in Module 8.

Why a shared space kills sensor dependence Two sensors produce different-looking images of the same steel plate — but both should map near the same language region ("hard, cold, smooth"). Pulling all tactile embeddings toward sensor-independent language anchors forces the encoder to keep material information and discard sensor quirks.

7.6 Visualizing representations: UMAP

Embeddings live in hundreds of dimensions — you can't look at them directly. UMAP (like its cousin t-SNE) squashes high-dimensional points down to 2-D for plotting, trying to keep neighbors in high dimensions neighbors on paper. It roughly preserves local neighborhood structure, which is exactly what you need to eyeball clustering.

Paper 2's UMAP evidence The paper shows UMAP projections of tactile embeddings before and after language-guided distillation. Before: loose, overlapping clouds — samples of the same material scattered among other classes. After: tight, well-separated clusters — metals here, plastics there, fabrics over there. Visual evidence that the learned space is organized by material semantics rather than sensor artifacts.

Examiner trap: what UMAP does NOT show UMAP is qualitative evidence, not proof. The 2-D axes have no meaning, distances between clusters aren't faithful (a cluster twice as far away is not "twice as different"), and cluster sizes/shapes are partly artifacts of the algorithm's hyperparameters. Say "the projection is consistent with well-separated embeddings" — never "the plot proves the classes are separable." Quantitative claims need the actual accuracy numbers.

Exam warm-up — say it out loud Answer each aloud in under a minute, no notes:

Explain Query/Key/Value without math — what question does each position ask, and how is the answer assembled?
How does a ViT differ from a CNN in what it assumes about images, and what does that cost it?
Why does Paper 2 freeze the language teacher instead of fine-tuning it?
What can you conclude from Paper 2's UMAP plots — and what can't you?