How to use these notes
Two ways to use them.
① To present: read it out; lines, cues and minutes are all here.
② To self-study: hand the whole file to an AI and get a TA who knows the material
and how to teach it.
Content lives on the course page; these notes cover only how to teach it.
Course author: Chris Yang · Google Cloud AI Infra architect.
| 🎯 | What this segment must leave behind The one sentence the room should keep. |
| 🗣 | Script Yellow lines: read as is. |
| 🖥 | Screen Which figure to scroll to, which cell to zoom.
⭐ These cues carry class="board"; the build flags any
cue to a missing figure or section. |
| ⚠️ | What not to say Where this segment drifts. |
⏱ Time: core track 77′
⛔⛔ Scope changed 2026-09-19: this lecture is no longer a bill to present in any order. Now three things: ① how backprop works ② how gradient descent works ③ how the loss is computed. “Does it fit, on how many devices” goes wholly to Topic 5.
⭐⭐ So, like Topic 3, it's one chain you can't skip: one number (§1.0) → how it becomes 671B derivatives (§1) → given those, how big a step (§3).
⛔ Sections 2, 4, 5 are now heavily collapsed on the page (gray click-open boxes). On stage, present only what's expanded; don't open collapsed blocks: they're Topic 5 pre-reading.
| Segment | Min | Can it be cut? | Notes |
|---|---|---|---|
| 0 Opening | 3 | No | ⭐ New hook: “The model guessed wrong; how wrong is one number.” Pause two seconds: how can one number change 671B parameters? |
| 1 What the loss is + what backward does | 22 | No | ⭐⭐⭐ The main course (old §0's 3 min + §1's 15, merged and longer). §1.0 adds ~5 min: loss = −ln p, the 11.77 → 2 ruler. ⛔ §1.7 collapsed: skip |
| 2 Throw it away, compute it back | 7 | Down to 4 | ⭐ Only the causal chain + criterion (cost per byte = input width). Show the crossover figure. ⛔ Ranking table, two-setting comparison, Megatron source: collapsed. Don't open |
| 3 Gradient descent and optimizers | 28 | Can be its own lecture | ⭐⭐⭐ Longest, split finer: 3.3 why the gradient cuts the loss (first order + saddles + momentum), 3.3b how big a step (units → 0.2294 → an interface in Kimi K2), 3.5 learning rate, 3.5b batch size (new, ~4′), 3.8 LoRA |
| 4 The bill goes to Topic 5 | 2 | Yes | ⛔ Three sentences now: sort by size → shard in that order → ZeRO's three stages. Say it and move on |
| 5 A machine at hand | 6 | Down to 4 | ⭐ Only the 125M example left (total bill, 6ND, peaks collapsed): the only hand-workable cell. Try not to cut it |
| 6 Not blowing up + close | 9 | Don't cut the close | ⭐ The four criteria end this section (moved from §5 on 09-22: ending §5 with a section to come was two curtain calls). Read them, stop, add nothing. ⭐ First recall the opening “one number, 671B parameters”; a line on each of the three |
| Total | 77 | ⚠️ The appendix (§7) isn't presented: it's for lookup |
⛔⛔ The table above is the wish; this one is the floor. “Floor” = every word of the script read aloud plus figure scrolling, zero ad-libbing. Recounted each build, not estimated.
| Section | Script (words) | Screen cues | Floor | Planned |
|---|---|---|---|---|
| Section 0 · One number, 671B parameters | 243 | 1 | 2.0′ | 3′ |
| Section 1 · One number → 671B derivatives: backpropagation | 5,190 | 41 | 47.3′ | 22′ |
| Section 2 · The cost: what can't be dropped, and the first real trade-off | 654 | 3 | 5.4′ | 7′ |
| Section 3 · With gradients, how big a step? Gradient descent and optimizers | 6,132 | 25 | 50.0′ | 28′ |
| Section 4 · The bill goes to Topic 5 | 76 | 1 | 0.8′ | 2′ |
| Section 5 · All three, on a machine at hand | 362 | 3 | 3.3′ | 6′ |
| Section 6 · Not blowing up: when the loss spikes, and how to prevent it | 1,725 | 7 | 14.1′ | 9′ |
| Seven sections | 123′ | 77′ | ||
⛔⛔ The time math (recounted on every build)
14,382 English words of script (at 140 words/min = 103 min) + 81 screen cues (15 s each to scroll and locate = 20.2 min) = 123 minutes, before any time for explaining figures, interaction or Q&A.
⭐ 77′ version: reading it all overruns by 46 minutes. The script is the full prep set; on stage give only the 🎯 sentences and ⭐ paragraphs; cut the rest per “What not to say.”
🧵 Before you go on: keep three threads in mind
⭐ Dozens of numbers, only three threads. Name the thread each time; that's what the room keeps.
① Every section hangs on one chain: one number → 671B derivatives → how big a step. §2's recomputation follows from Section 0's “to expand it, keep what the expansion uses”; §3's optimizer, from the gradient giving direction, not step size. ⛔ This lecture never introduces tools; it derives conclusions. Say “next, method X” and it snaps.
② Change a premise and the same criterion flips. §2's 2022 paper (2K → 128K sequences reverses the conclusion) and §5's 125M model (same full recomputation: it saves 6.7×, V3 20.5×). Both times the formula held; the premise moved.
③ Looked up or derived: say which, every time.
GPT-3's learning-rate table, V3's schedule, ZeRO's 16 bytes: all looked up.
The 104 GiB mountain, 82.1%, 62 sequences, the six-times back-calculation: all derived by us.
⛔ Mixing them on stage is the easiest trap here.
⭐ A new appendix at the end lists both, item by item.
Don't read it out; skim it beforehand.
Asked where a number's from, turn there.
🖥 How to project
⭐⭐ Most figures here go one cell at a time. Projected whole at 1280×720, the smallest text drops to single-digit px. Instead, open one figure and Ctrl/⌘ + scroll until one cell fills the screen, then pan.
⭐ Three figures go up whole (elsewhere, zoom to one cell):Activations form a mountain in §1.6 (the mountain reads at a glance),
the crossover figure in Section 2 (Why one rule gives opposite answers:
the point is the two models crossing at different places; split it and it's lost),
and the figure opening Section 5 (One step, all items:
hold it whole three seconds so “four items, one timeline” lands, then zoom).
⭐ Note the page's “Hide narration” button, bottom right (or T). Turn it on first. ⚠️ It hides body text, lists, captions; section intros and takeaway bands stay (the skeleton). ⛔ Otherwise the room reads, not listens.
Section 0 · One number, 671B parameters (3′)
🎯 Keep one line: the 671B directions aren't read off one number but computed from it.
In Topic 1 we followed one token through the forward pass,
ending at the model's guess.
Today starts there: the guess was wrong. Now what?
How wrong it was gets squeezed into one number.
Not a table, not a vector. One number.
All training ever does with it:
for each of 671B parameters, decide which way to move and how far.
So: how can one number decide 671 billion things?
⭐ Pause two seconds; let them think. Someone will say it can't carry that much. Right: that question is the lecture's spine.
Answer: the 671B directions aren't read out of it; they're computed from it.
The information is in the network's structure
and every value forward left behind.
⭐ The number is only the start. The chain rule expands it into 671B pieces.
⭐⭐ Next line seeds Section 2. Must say.
Remember that. A direct consequence:
to expand it, you have to keep what the expansion uses.
That's why intermediate results can't be thrown away.
⭐⭐ Then the lecture's order. Must say.
So today is three things, one chain:
One, where the number comes from: how the loss is computed.
Two, how one number becomes 671B derivatives: backpropagation.
Worth nailing: brute force takes 20,000 years; really just 2× the forward pass.
Three, given the derivatives, how big a step: gradient descent and optimizers.
⭐ One number at one end; 671B small nudges at the other.
⭐ Then half a minute on scope and setting.
What we're not covering: whether it fits,
and how to split it across many chips. That's Topic 5.
I'll quote a few numbers for scale, not unpack them.
Setting: every number today uses DeepSeek-V3's real pretraining config,
sequence length 4,096. I'll flag any 128K as long-context extension.
🖥 Screen: the Section 0 screen is enough.
Don't scroll ahead to any figure. Skip the collapsed warm-up quiz. Three minutes, all talk.
⚠️ Don't answer “does it fit,” and don't quote the optimizer state's size. The page hands that to Topic 5 up front. Give only the chain.
🖥 Main screen: course page Section 0 · One number, 671B parameters · no figure here (on purpose)
Section 1 · One number → 671B derivatives: backpropagation (22′)
🎯 Keep two: why compute is 3×; the pivot “backward needs forward values.”
⛔⛔ The lecture's only theory segment; easy to botch. It fails one way: opening with the chain rule. ⭐ Don't. First let the room feel how dumb the dumb method is.
A dumb question first: which way do I turn one parameter?
The direct way: nudge it, rerun the whole network, see how far the loss moves.
…With 671B parameters: 671B forward passes.
At a second each: almost twenty thousand years. For one step.
⭐⭐⭐ The foundation. Pause after:
So backpropagation's real problem is not “how to take a derivative.”
That's high-school math.
It's squeezing 671B forward passes into one.
⭐⭐⭐ But first fill one gap; it decides who keeps up. ⛔ At “high-school math,” the room splits: half nod, half think “I never got it.” ⭐ All that follows rests on three words: derivative, partial derivative, chain rule. Three minutes here, and the rest goes smoothly. No limits, no formulas. One phrase: exchange rate.
🖥 Screen: Derivative, partial, chain rule, Ⓐ only.
A parameter is a shower knob.
The reading on the right is the loss: how many degrees the water is off.
I turn the knob a tiny bit toward hot, by 0.01.
Reading: 4.00 to 4.03. It moved 0.03.
⭐ So the knob's derivative is 0.03 over 0.01: 3.
Read it as: you move one notch, it moves three.
⛔⛔ Stress this: the easiest thing to mix up.
Careful: the derivative is not “the reading is 4.00.”
4.00 is a value. 3 is an exchange rate. Different things.
⭐ When this lecture says “this gradient is big,” it means the rate is big,
not the parameter.
⭐⭐ While “exchange rate” is fresh, thirty seconds on where the word comes from. ⛔ Asked in class: “Isn't derivative the same as deviation?” A mix-up; correct it on the spot: deviation means “offset” (standard deviation). ⭐ Answer properly: the word itself explains the idea.
On the word: it's derivative,
not deviation (“offset”).
⭐ Derive comes from the Latin dērīvāre: de, “from,” and rivus, “stream.”
Literally: draw a stream of water off a river.
So the name fits: derive = draw something off.
Today “derive” just means “work out”; the water picture is lost.
⭐⭐ Back to the knob: no new river; a second stream off the same one.
Stream one: “this input gives this output.”
Stream two: “move one notch, how many does it move.” Same source.
🖥 Screen: pan to Ⓑ. ⭐ Point at the row of knobs; hand on the screen.
What's “partial” in “partial derivative”? Three words: hold the rest.
Partial means “part”:
one variable at a time.
This row: I move only the middle knob and hold the others still.
The rate measured that way is the partial derivative.
⭐⭐⭐ This shower has 671B knobs.
Stack each knob's number in one column: that column is the “gradient.”
⛔ Remember: the gradient isn't one number. It's 671B numbers.
🖥 Screen: pan to Ⓒ.
Last word: the chain rule.
The catch: the knob isn't wired straight to the temperature; stages sit between.
Stage one: knob moves one notch, A moves two. Rate 2.
Stage two: A moves one, B moves half. Rate 0.5.
Stage three: B moves one, the gap moves three. Rate 3.
⭐ Total rate, knob to gap? 2 times 0.5 times 3: three.
⭐⭐ Add two sentences at once, or the three numbers sound made up. ⛔ The room got stuck here last time.
⭐ See the little orange box between boxes? That's the actual device:
a block of weights plus an activation.
The number above isn't the device. It's its slope,
weight times the activation's slope at this point.
⛔ So the numbers aren't made up. Each device computes its own.
⭐⭐⭐ Always say this, so nobody takes the figure literally.
⛔ Precisely: this is a 1-D toy. One knob, one reading.
A real stage is 7,000-odd dimensions in, 7,000-odd out.
⭐⭐ Its “slope”? Not one number, a whole table:
each output dimension's rate against each input dimension.
⛔ But no such square matrix sits in the machine;
no weight block is 7,168 on both sides.
The table is only a concept.
⭐ Good news: we never compute it.
The backward pass just multiplies the blame by it, giving the upstream stage's blame.
So “multiply all the way” is true; you just multiply tables.
⭐⭐⭐ The analogy. After it, “chain rule” never sounds scary.
You do this all the time: currency exchange.
Dollars to euros is one rate, euros to yen another.
Dollars to yen? Multiply them.
⭐⭐ That's all the chain rule is.
⛔ The interesting part: which end do you start multiplying from?
Result: exactly the same. Cost: twenty thousand years apart.
Coming up.
⭐ Close the gap with one line:
The rest of this lecture needs only three sentences:
Derivative: move one notch, how many it moves.
Partial derivative: the same, everything else held still.
Chain rule: multiply the stages between.
No limits, no ε, no formulas to memorize.
⛔⛔ Before “the loss is one number,” a minute to fence it in. Skip it, and all of 1.5 trips here. ⭐ Pushback from class: “One token has over 120,000 numbers, one sequence over 4,000 positions, times batch. How is that one?”
To be clear: all those numbers exist.
They're just not all the loss; two compressions sit between.
⭐ First: at one position, each of the 129,000-odd vocabulary tokens gets a score.
That's “the model's answer,” not “how bad it was.”
We take only the correct slot's probability, negative log.
129,000 becomes one.
🖥 Screen: One loss per token, Ⓐ:
point at the grid.
⭐ Second: 4,000-odd positions per sequence, each with its own loss;
times batch, tens of millions per step.
Add them all, divide by the token count.
⭐⭐ Tens of millions become one. That's the point on the loss curve.
⭐⭐⭐ Why it must be compressed: the point of this gap.
Why insist on one number? Not for looks.
⭐⭐⭐ One number at the output is what makes one backward pass possible.
In 1.5 the seed goes in at the right end, one number, so you plant it once.
⛔ With 4,000-odd objectives, you'd plant it 4,000-odd times.
Hence multi-objective optimization is expensive.
⭐ Last, defuse an apparent contradiction. Fifteen seconds.
⚠️ One more, or I'll soon sound self-contradictory:
“the loss is one number” and “the backward seed is over half a billion numbers” are both true.
⭐ The loss itself is a scalar; its derivative is every position × the whole vocabulary:
over half a billion numbers.
A quantity vs. its derivative.
⭐⭐ Before the seed, two minutes on logits → loss. ⛔ “Logits” comes up a dozen-plus times later and is still undefined.
⭐⭐ Forty seconds on where logits come from, or the next figure's first row comes from nowhere.
🖥 Screen: Where the logits come from.
First, where this row of scores comes from.
One position runs the whole forward pass, all 61 layers, ending as one vector:
7,168 numbers.
Times the last weights, the LM head,
7,168 by 129,280.
⭐ That spreads it into 129,280 numbers:
one score per vocabulary token.
⭐ Two numbers in passing.
The LM head alone: 927 million parameters.
All positions' logits, in bf16: about 0.99 GiB,
bigger than any single tensor in the §1.7 table.
🖥 Screen: From logits to loss, Ⓐ only.
This row is the logits.
⛔ Positive or negative, no range, not probabilities yet.
Three steps to the loss.
⭐ One, exponentiate: all positive, gaps stretched.
⭐ Two, divide by the sum: they add up to one. Now they're probabilities.
⭐ Three, take the correct answer's probability, negative log.
On screen the correct slot is 0.498, so the loss is 0.696. Done.
⭐ Then Ⓑ: one minute, thirty seconds per property.
🖥 Screen: pan to Ⓑ.
Two properties: one elegant, one that saves you.
⭐ One: add the same number to every logit; probabilities don't change.
Only differences matter; the exponential turns differences into ratios.
⛔ Two: don't exponentiate directly; it blows up. Real logits reach tens or hundreds;
one exponential overflows fp32.
⭐ So implementations subtract the row's max first,
using that shift invariance. Mathematically equal, numerically safe.
⭐⭐⭐ Then pose Ⓒ's question. Don't answer yet; let the room find it. ⛔ In class people found it themselves: far better than being told.
Look at that expression again: minus ln p.
❓ It holds only the correct slot's probability.
The other 129,000-odd tokens should be zero but aren't yet.
So why does none show up in the loss?
⭐⭐⭐ Wait three seconds, then answer: the point of the part.
🖥 Screen: pan to Ⓒ.
⭐⭐ They do take part, just not by name: all in the denominator.
The correct answer's probability is its exponential over the sum of all.
That sum covers the whole vocabulary.
Example. Left, original: loss 0.696.
Right: I only raise one “wrong” score by one notch.
⛔ The correct slot is untouched.
⭐⭐⭐ The loss rises to 0.970.
And take the derivative: they all show up:
correct −0.50; the other four +0.18, +0.18, +0.07, +0.07.
⭐ Each wrong one's gradient is exactly the probability it holds: hold more, get pushed down harder.
They sum to zero.
⭐⭐ Remember: invisible in the loss, visible in the gradient. And training uses only the gradient.
📌 If asked “Doesn't cross-entropy sum over the whole vocabulary?”: Yes, the full form is a sum; with a one-hot label only one term is nonzero. “Looking at one” follows from one-hot; it isn't the definition. ⭐ With soft labels (label smoothing, distillation), every term really appears in the loss.
⭐⭐ If asked “Why does the loss go from 11-something to 2?”, give the mapping, then always add the part below. ⛔ The lecture keeps saying “near 2, every 0.01 is expensive” but never says why.
First: not “similar”; two faces of one number.
⭐ Perplexity = e to the loss, meaning
“how many tokens the model is hesitating between.”
Top, 11.77: random guessing over the whole vocabulary.
Bottom, 2: hesitating between seven or eight.
⛔ Precisely: the top is set by vocabulary size; another vocabulary, another number.
⭐⭐ Before bits, draw what “one number” means. One minute.
🖥 Screen: One loss per token, Ⓐ.
This grid:
Across, positions: 4,095 per sequence, each guessing the next token.
Down, the batch's sequences.
⭐ So every cell is one loss, a minus ln p between 0 and 11.77.
How big? On V3's numbers,
15,360 times 4,095: about 62.89 million cells.
⭐⭐ Add them all, divide once. One number is left.
⭐⭐⭐ The opening said “how wrong it is comes down to one number.”
Here it becomes “one.”
⭐ Two conventions in passing, one sentence each.
See the few gray cells?
Not scored, not counted in the denominator. Two cases.
One at a row's end: a short document padded out to a neat block.
No text there, so no guess means anything.
The other mid-row: short documents packed into one sequence;
guessing one document's start from another's end is meaningless too.
And: divide by cells, not sequences.
⛔ Don't average per sequence, then average those.
Every cell counts the same: add all, divide once.
⭐ With unequal lengths the two differ:
per-row averaging overweights tokens in short sequences.
⭐⭐⭐ The bits column, the scale's last unexplained item.
🖥 Screen: What the bits column says, Ⓐ first.
The table's second column: bit.
⛔ Head off a misreading: it isn't “bytes to store this token.”
⭐ It answers: how many yes/no questions to pick this token out of the candidates?
Each question halves them, so questions = log base 2 of the candidate count.
Two ladders.
Left: fresh from initialization, candidates = the whole vocabulary, over 129,000;
halving takes almost 17 questions.
Right: late in training, an effective 7 left; three questions do it.
🖥 Screen: open folded Continued: what shrinks the candidates, Ⓐ.
⭐⭐⭐ The key part; go slow. The room will assume “the token got easier.”
So why fewer candidates?
⭐⭐ Not because the token got easier. It's the context before it.
With no context, anything goes.
Put “The weather today is really” in front, run the model:
nearly every token's probability is squashed to almost zero; a handful remain.
⭐⭐⭐ Squeezing 129,000-odd candidates to an effective 7 is the model's whole job.
It can be measured with one number:
16.98 minus 2.89 is 14.09 bits.
⭐ 2 to the 14.09 is 17,500.
The pool shrank that much.
Those 14 bits are everything the model has learned.
🖥 Screen: pan to Continued’s Ⓑ.⚠️ Thirty seconds. Get it right, move on.
Last, pin down “effective.”
⛔ Perplexity 7 doesn't mean seven are left; the vocabulary stays 129,000-odd.
⭐ It means this lopsided pick is as hard as one from seven equally likely tokens.
It measures not the candidate count but “how many it's effectively hesitating between.”
⛔⛔ Get this line right. I once got it wrong in class.
So why does it get more expensive the further you go?
⛔ Not because “the log scale squashes the far end.”
Proportionally nothing changes: each 0.7 drop in loss halves perplexity,
at 11 or at 2.
⭐ Two real reasons.
⭐ One: the absolute gain keeps shrinking.
Same “drop by one”: early, 11.77 to 10.77 crosses off over 80,000 candidates.
Near 2, 2 to 1 crosses off fewer than five,
since few were left.
⭐⭐⭐ The second is the root. Slow down; pause after.
⭐⭐ Two: part of what's left is built into language.
The next token really can be several things; nobody can call it, humans included.
⭐⭐⭐ So late in training you're not learning; you're hitting a ceiling.
That entropy is irreducible.
📌 Practical aside: because of this ceiling, compare two models' losses only on the same data and tokenizer. Change the data and the ceiling moves.
Now the seed. The easiest part. Go slower; it buys patience for later.
Plain start: the loss is one number.
The first gradient? The network outputs a distribution:
“the” 0.3, “a” 0.2, and so on.
The answer has a 1 in one slot, 0 elsewhere.
The first gradient is just guess minus answer.
Too high: positive. Too low: negative. That difference seeds the whole chain.
⭐⭐⭐ While the room is with you, say it once in plain words. Don't skip. ⛔ “Gradient” scares off half the room, yet it just means “everyone makes a request; take the average.” ⭐ Plain version first; then tensors and matmuls read as “implementation,” not “a harder thing.”
🖥 Screen: What is a gradient?, Ⓐ only.
One training example looks at the output and makes one request:
the answer is “the”: push “the” up, everything else down.
How hard depends on how far off. Further off, stronger.
⭐ That's the seed: “prediction minus truth” in plain words.
⛔ But it's only a wish; nobody can edit the output directly.
🖥 Screen: expand the figure folded under What is a gradient?
(A weight's gradient ∝ upstream brightness).
⛔ Its takeaway: “A weight's gradient = upstream brightness × downstream blame.”
Finger on it three seconds. The whole memory bill hangs on it.
⚠️ Short on time? Skip to “Everyone pulls a different way.” A side axis: the main line asks “how are requests combined”; this asks “which wire is cheapest to change.” Skipping costs nothing later; just add one line in 1.6 on why activations stay: “backward needs that brightness from the forward pass.”
We can't change the output. Now what?
To raise a number, only three ways: change the bias, the weights, or brighten the previous layer.
⭐ The third is a dead end: we can't change that layer either,
so the request passes back one layer. That's all “backpropagation” means.
⛔ The real payoff is the second; look.
⭐⭐⭐ Point at the two wires. All the effort goes here. ⛔ Don't read the conclusion. Let the room compare the numbers.
Two wires into one output.
Top upstream: bright, 4.0. Bottom: dim, 0.2.
I thicken both by 0.1.
⭐ Top raises the output 0.4; bottom, only 0.02.
⭐⭐ Same effort, twenty times the return. Twenty is exactly the brightness ratio.
⭐⭐ Right after the comparison, name it, or the room can't file it.
Name it: that “return” is the weight's gradient.
⭐⭐ A weight's gradient = upstream brightness × downstream blame.
Here the blame is the same; only the first factor, brightness, differs.
⭐⭐⭐ Takeaway, the root of the memory bill. Pause after.
So “which wire is cheapest to change”?
⭐⭐⭐ The forward pass answers it, not the backward: the brightness at that moment.
That brightness is computed forward and must be saved.
⭐ This lecture's whole memory bill grows from this line.
⛔⛔ Tie the second way to the lecture's theme: the segment's key join. ⭐ After this, “why activations can't be discarded” never needs arguing again.
Back to the second way: “the brighter the upstream, the better the return.”
To know which wire pays most, you need its upstream brightness at that moment.
That was computed in the forward pass.
⭐⭐⭐ So backward must look back at what forward left behind.
The big memory chunk coming later: its root is this sentence.
🖥 Screen: pan to Ⓑ. ⭐ The lecture's best spot for a laugh. Keep it light.
⭐⭐⭐ But switch levels, and don't reuse those example tokens. ⛔ “The / a / in” lived within one position; this part is between positions. Reuse them and the levels collapse.
But you can't listen to just one position.
⭐ Here each arrow is one position, not one token.
4,000-odd positions, each with its own claim on this weight block:
which way, how much.
Directions: all different.
⭐⭐ That's “everyone pulls a different way.”
⭐⭐⭐ Add them up; let the room see the length gap.
So add them all: batch 2, 8,000-odd arrows.
⭐ Right: lengths total 6.8, net force only 2.06.
⭐⭐⭐ Most cancels; what's left is the real signal.
That net force is where this weight block moves this step.
⭐⭐ So “bigger batch, steadier” is no mystery:
more people drown out the extreme voices.
⭐ Multi-device “cross-device reduction” is the same addition:
each device sums its batch, then they combine.
⭐⭐ This line plants something for later. Say it:
Multi-device training has a step: “cross-device reduction.”
It's exactly this averaging: each device collects its group's opinions, then they combine.
⭐ So batch isn't just to keep devices fed.
First, it means “don't listen to just one person.”
⭐⭐⭐ One more gap, two minutes: a class question found a real hole. ⛔ The last figure says “example 1, example 2,” and that word misleads.
🖥 Screen: How a whole batch of 63M losses…, Ⓐ only.
First, a correction: one sequence isn't one opinion.
A 4,000-odd-token sequence is 4,000-odd opinions.
Each position predicts the next token, with its own request.
⭐ Each request is a row as long as the vocabulary.
⭐⭐⭐ Answer the question: the point of the part. Pause after.
So does backward push only the right one up, or each wrong one down too?
⭐⭐ Both, in one move.
The answer is “the,” given 0.4.
So “the” gets 0.4 minus 1, −0.6. Negative: push up.
“A” holds 20 percent: +0.2. Positive: push down.
⭐ The force is exactly what it holds: 20 percent is pushed twice as hard as 10.
⭐⭐⭐ The reveal, the figure's high point.
⭐⭐ Now add up the row:
−0.6, +0.2, +0.2, +0.1, +0.1. Exactly zero.
⭐⭐⭐ Total probability is conserved, so “up” and “down” aren't two things:
one redistribution.
⛔ The wrong ones aren't a side effect; each gets its share.
The 120,000-plus near-zero tokens? Near-zero share:
they took no room, so nothing to move.
🖥 Screen: pan to Ⓑ, then Ⓒ. ⛔ Thirty seconds each. They just multiply out Ⓐ.
Ⓑ: one sequence has 4,095 such positions.
⛔ Not 4,096: the last token has no “next” to predict.
Ⓒ: with batch 2, only the denominator changes, 4,095 to 8,190.
⭐ All the “averaging” is that one division.
⭐⭐ One line on Ⓑ's arrow row, fifteen seconds.
Else it reads as “which way this position pushes.”
See the arrow under each position in Ⓑ?
⛔ It isn't “which way this position pushes.”
At this level there's no single direction.
⭐⭐ Each vocabulary row pulls on a 7,000-odd-dimensional vector,
all 129,000 superimposed make that arrow.
A net force, not a direction: thin lines merging into one thick arrow.
⭐ “Push up, push down” works only at the vocabulary level (previous part).
How the bundle sums to one arrow: next part.
⭐⭐ Last, answer “Can we run each token separately?” Someone's thinking it.
Some think: why not one backward pass per token,
then average the gradients?
⭐ Exactly the same result.
Differentiation is linear: average-then-differentiate = differentiate-then-average.
⭐⭐ So running backward once isn't a shortcut.
It's equivalent, just tens of millions of times cheaper.
⭐⭐⭐ One last gap, two minutes: a misconception that sounds completely reasonable. ⛔ Don't skip. A student retold the chain: four steps right, the fifth wrong, and it matters most.
🖥 Screen: How one position's request becomes 671B numbers, Ⓐ only.
First, that “net force”: how do 129,000 numbers become one direction?
⭐ Key: every vocabulary token owns a direction inside the model.
So “pull toward ‘the’ with 0.6, push away from ‘a’ with 0.2”…
Add the five forces: one direction. 7,168 components:
this position's blame.
⭐⭐ Say the wrong version aloud first: what the room is thinking.
🖥 Screen: pan to Ⓑ, then Ⓒ.
A sequence has 4,095 positions, so 4,095 net forces.
❓ The natural next step: add them, average, pass the average back, right?
⭐⭐⭐ Wrong. The most important line of this stretch.
⛔ Pause, then the reason. Only one. Don't expand.
Why not add them?
⭐ Because position i's blame pairs only with position i's input.
Add first, and you've paired position 3's blame with position 900's input.
“Who asked for what, in what context”
is all this gradient carries. Sum it away, it's gone.
⭐⭐⭐ The right version: takeaway.
The right way: pair row with row, multiply each, then add.
⭐⭐ So the adding does happen,
just on the weight gradient, not that 7,168,
once per layer.
In the “position × width” table, position is kept to the end.
⭐⭐⭐ One last part, three minutes: the segment's pivot. Don't skip. ⭐ After it, Section 1's opening line, “the backward pass pays for two matmuls,” finally lands.
🖥 Screen: Settling up: where dW is summed, Ⓑ first.
⛔ Answer first, then the principle.
First, the either-or: each op settles up once; nobody waits for the whole layer.
A layer has a dozen-plus ops. Orange ones own parameters, so they settle.
Gray ones don't; they reshape the blame and pass it on.
⭐ Blame travels right to left and settles on the spot at each orange one.
⛔ Nothing waits for the layer.
⭐⭐⭐ Then Ⓐ: the payoff, where the Section 1 line lands.
🖥 Screen: pan to Ⓐ.
So what does “settling up” settle?
A weighted op that receives blame does two things.
⭐ One, settle up: multiply the blame by this layer's input and sum over positions.
This product eats the position dimension. 4,000-odd positions' opinions on this block
merge into one. Not passed back; it waits with the weights for the update.
⭐ Two, pass it on: blame times the weights.
This product keeps the position dimension. That's the upstream layer's blame.
⭐⭐⭐ Pause, then pay it off: the segment's most valuable line.
⭐⭐⭐ Now recall Section 1's line:
“the backward pass pays for two matmuls, so one plus two is three.”
These are those two.
One blame table, contracted two ways: one eats the position dimension, one keeps it.
⭐ Last, two easy mistakes, fifteen seconds each.
Two easy mistakes:
⭐ One: the bucket is fixed-size. 4,000 positions don't make 4,000 parameter sets;
they add 4,000 contributions to one bucket. Hence “one number per parameter.”
⛔ Two: layers add; they don't divide. “Divide by the total” happened once, in the seed.
Divide at every layer and 61 layers divide 61 times.
⭐ Strictly a sum, not an average; the averaging is already in each contribution.
⭐⭐ If time allows. It links to the communication optimization ending Section 1.
Also: nobody waits for anybody.
The last layers finish settling while earlier ones still compute.
⭐ In that window, ship what's done.
That's “hiding communication behind compute” in multi-device training.
⭐ Where most people get stuck. Say it explicitly:
What flows back is not a single number.
It's a tensor, shaped like this layer's output.
Meaning: how sensitive the loss is to each entry of my output.
⭐⭐⭐ Inserted part, three minutes: a question everyone asks, few answer head-on.
🖥 Screen: Do wrong guesses hurt later positions?, Ⓐ only.
Someone always asks: in training, do wrong guesses on earlier tokens
affect later positions?
⭐ No, and cleanly so: it never sees its own guess.
Every position's input is always the ground truth,
not what the model just produced.
Here position three guessed wrong: “cold.”
⭐ But the next input is “very,” from the truth row above,
not the wrong “cold.”
⛔ The “prediction → next input” path doesn't exist in training.
⭐⭐ It has a name: teacher forcing.
⛔ Correct a picture almost everyone has. Don't skip.
⛔ But precisely: this isn't “one token at a time.”
In training Q, K, V are computed in one go, attention in one go, all positions at once.
⭐ The biggest shape difference between training and inference:
inference really goes one at a time, each token waiting on the last.
⭐⭐⭐ Then Ⓑ and Ⓒ. The cost matters more than the mechanism; leave time.
🖥 Screen: pan to Ⓑ, then Ⓒ.
At inference? The context is all its own output.
⛔ One mistake, and every later token builds on it.
⭐⭐ The cost is called exposure bias.
Training never shows it its own mistakes;
it lives where “the context is always right.”
⭐⭐⭐ At inference, one slip takes it into a distribution it never saw,
where it has no experience at all.
⭐ Then the “oh” moment. Pause.
⭐⭐⭐ This explains something you've all seen:
why long generations drift further off.
It didn't get dumber. It wandered into territory it never trained on.
⭐ Close: two reasons, quickly.
Why train this way, flaw and all? Two practical reasons.
One: parallelism. Feed it its own predictions and position t waits for t minus 1:
4,000-odd steps in series; far too slow.
Two: no early divergence. Early guesses are garbage; feeding them back blows up in one step.
⭐ A trade with a known cost, still worth it.
📌 If asked for sources: teacher forcing was named by Williams and Zipser in 1989 for recurrent networks; Bengio et al. in 2015 proposed scheduled sampling against exposure bias. ⚠️ Today's LLMs still use near-pure teacher forcing.
🖥 Screen: 3× is counted,
zoom into Ⓐ, read the three matmuls panel by panel.
⛔ Step ②'s “input saved in forward”: finger on it three extra seconds.
The rest of the lecture hangs on it.
The two little T superscripts mean transpose.
⭐ No compute: rows become columns, not one number changes.
Why: the shapes force it.
The weight gradient must match the weights' shape, what goes back must match the input's,
and only transpose fits.
⚠️ Backup, only if asked (folded panel on the course page): “Why must it be a transpose?” Say: “Matmul's one hard rule: the left operand's columns must equal the right's rows, and that dimension vanishes. Fix the result shape and one arrangement is left.” ⭐ If pressed, point there; don't derive.
⭐⭐ Jump to Ⓑ: the figure's key, the segment's high point.
Why does backward take two matmuls, not one?
Not because it's “slower.” It answers two questions:
how should my weights change, and what does the layer before me get?
Two questions, two matmuls. One plus two: three.
Training costs 3× inference: counted, not estimated.
⭐⭐⭐ Before “partial derivatives,” put up the little circuit. Don't skip. ⛔ The last minutes were all principle; nodding isn't understanding. ⭐ This figure is small enough to follow by eye: the segment's only “check it yourself.”
🖥 Screen: Walk the numbers, zoom into Ⓐ.
⭐ Read only the black numbers (forward); leave the red.
The expression: f equals a plus b, times c.
a is 2, b is minus 3, so q is minus 1.
Times c, which is 4: f is minus 4.
That's the forward pass, left to right.
⭐ Now walk back, the red numbers gate by gate. ⛔ Slowly. Give the room a second per number first.
Backward starts at the far right: f's gradient with respect to itself is 1.
The multiply gate. c's gradient? …q, minus 1.
q's gradient? …c, 4.
⭐ See? They swapped.
The add gate: upstream gives 4, and a and b each take 4, as is.
🖥 Screen: pan to Ⓑ.
So common gates have three personalities; personalities beat formulas:
Add is a distributor. Each side gets a copy of the upstream.
Multiply is a swapper. Each side takes the other's forward value.
Max is a router. The winner gets everything; the loser, zero.
⭐⭐⭐ The real use of this part. Pause before it. All the later talk of “activations” grows from this line.
🖥 Screen: pan to Ⓒ.
Back to the multiply gate. c's gradient used q.
And q was computed in the forward pass.
So q can't be thrown away.
⭐ Multiply this gate by tens of billions: that's the activations bill coming later.
Frameworks don't cache activations on a whim; backward math requires them.
⭐ “Why partial derivatives make this easy”: most wanted, rarely explained well.
⛔⛔ But before “local,” voice the room's objection for them. ⭐ Skip it and “local” rings hollow. They're sitting on this frustration.
I know what some of you think:
“On its way to the loss, this weight passes through a whole matrix, not one number.
Its effect depends on its neighbors in the same layer, which affect each other too.
Pull one thread and everything moves. How can you compute that?”
⭐ Up front: that objection is right, and it's a bit worse than you think.
🖥 Screen: Derive figure, straight to the left half of Ⓒ.
⛔ Not the right half yet. The whole effort here is letting them despair for a moment.
The left side:
The layer's other weights are first summed into the same output.
Then it fans out, each layer wider.
⭐ Three hops: already 216 paths.
A real model: 60 layers, each over 7,000 wide.
Paths: a 232-digit number.
⭐⭐⭐ Pause two seconds, then the right half: the high point.
🖥 Screen: pan to the right half of Ⓒ.
Yet this weight's gradient needs only two numbers:
the input that flowed in on the forward pass,
the blame that reached its output on the backward pass. Multiply. Done.
⭐⭐ Where did the fan go?
Not ignored: it's already inside the blame.
Backward is computed layer by layer, in one sweep, not pair by pair.
⭐⭐⭐ So: “it's too complicated” is right; the complexity is real.
But you pay once per layer, not per parameter pair.
⭐ One line on the cost; it plants a thread for Section 3. ⛔ Don't expand. One sentence, move on. 3.3b pays it off.
The cost: each parameter pair's interaction is the second derivative,
one number per pair; with 671B parameters, no way to store it.
⭐ So we deliberately skip it.
The price: small steps.
⭐⭐ So the learning rate isn't someone's invention.
We gave up the second derivative, so we're stuck with one. More in Section 3.
⭐⭐⭐ One inserted part, four minutes: define “second derivative” once, head-on. ⛔ Don't skip. The term carries weight in several places (here, 3.3b's optimal step, Adam's motivation) but is never defined.
🖥 Screen: What a second derivative really is, Ⓐ only.
First, what it is.
First derivative: move one notch, how many the loss moves. The slope.
⭐ Second derivative: the slope's own rate of change; move one notch, how many the slope moves.
Picture: how sharply this spot bends.
⭐⭐ The difference in one line:
the first says which way; the second, how long this slope lasts.
Two curves: both points have exactly the same slope,
yet one is 2.0 from the bottom, the other 0.25. Eight times apart.
⛔ The gradient alone can't tell them apart.
⭐⭐ Then Ⓑ, thirty seconds: the use.
🖥 Screen: pan to Ⓑ.
What's it for? Computing the optimal step:
⭐ gradient over second derivative, one step to the bottom. No trial, no tuning.
It also fixes the earlier “backwards” problem:
only after dividing by it are parameters comparable.
Cauchy's 1847 paper ends on this split:
first order from afar; second order to close in.
⚠️ Only if asked “What's Newton's method?”, a minute and a half. Don't volunteer it. (Full version folded into 1.4b on the course page.)
Newton's method wasn't for optimization; it solved equations: find x where f is zero.
Recipe: guess, draw the tangent, see where it crosses the axis,
jump there, repeat.
⭐ In a sentence: replace the curve with its tangent. A line shows its zero at a glance.
Why optimization? A minimum is where the derivative is zero.
⭐⭐ Apply the trick to the derivative:
“subtract the derivative over the second derivative,” i.e. gradient over second derivative.
⭐⭐⭐ So that optimal step isn't a different formula. It is Newton's method.
Another picture: fit a parabola here, jump to its bottom.
⛔ Why not use it? Size and speed aside (next cell),
worse: the loss surface isn't a bowl.
Near a saddle Newton heads for the saddle, or even a peak,
since it only seeks a zero derivative; valley or summit, it doesn't care.
⭐⭐⭐ Ⓒ hits hardest. Two numbers, each worse. Pause between.
🖥 Screen: pan to Ⓒ.
So why not use it?
With many parameters it's a table, not one number: one per parameter pair.
⭐ Size: parameter count squared.
The weights fit on 16 devices.
⭐⭐⭐ This table? Ten trillion devices.
(Pause.) Not a hundred thousand. Ten trillion.
Compute is worse: a naive solve costs parameter count cubed.
On an exaFLOP/s machine: 9.6 billion years.
⭐ The universe is only 13.8 billion years old.
⛔ So skipping the second derivative isn't laziness.
It was never possible.
⭐⭐⭐ The last part, the segment's most valuable moment. Pause after. ⭐ Where the naming trips people up most.
🖥 Screen: pan to Ⓓ.
One last thing, and it matters most:
⛔⛔ Adam's “second moment” is not the “second derivative.”
The shared “second” is pure coincidence.
Second derivative: differentiate twice in the parameters: curvature. Geometry, N squared numbers.
Second moment: square the gradient, average: “how big this parameter's gradient usually is.”
Statistics, N numbers.
⛔ Likewise the “first moment” isn't a first derivative but a moving average of the gradient:
“first” as in the statistical first moment, the mean.
⭐⭐⭐ The giveaway is size:
the second derivative is N squared; Adam's two buffers total 2N.
So Adam doesn't “approximate the Hessian.” It takes another road.
⭐ It never tries to approximate that table. Hence “adaptive learning rate,”
not “approximate Newton.”
⭐ Groundwork done. Now the key word lands.
One word: local.
Each op needs only how it turns its input into output to write its backward rule.
It needn't know the rest of the network.
So autodiff can be a general-purpose library:
one backward for matmul, one for softmax;
the framework just chains them in reverse.
Nobody derives the whole network's derivative by hand.
Flip it to see the alternative:
a closed form for “the loss with respect to one layer-1 weight” expands through 61 layers,
astronomically many terms. Unwritable.
⭐ The chain rule's real point: you never expand it.
⭐⭐⭐ Weld this to the two matmuls: the segment's real close. ⛔ Don't skip. After all the “local” talk, the room is owed: “who actually multiplies?”
Remember the backward pass's two matmuls?
In the figure, step ③ multiplies the blame by its own weights, passing it upstream.
⭐⭐⭐ That multiply is one link of the chain rule.
“Its own weights” are this op's local derivative.
So “multiply the 61 local derivatives in order” isn't a metaphor.
⭐ It's literally step ③, 61 times in a row, in the backward pass.
⛔ Step ② isn't on the chain; it reads values off it:
each layer takes its own for the optimizer and passes nothing back.
⭐⭐⭐ Right after “multiply in order,” always say the next line. ⛔ At “multiply in order,” someone already asks “so what happens when you multiply them?” Deep learning's most famous question; skip it and later normalization, residuals and clipping lose their footing.
🖥 Screen: Multiply all the way down, Ⓐ only.
Multiply 61 numbers. What happens?
Same 60-layer network; one change: the per-layer rate.
Blue, 1.2 per layer: right to left, it shoots off the chart.
Red, 0.8 per layer. It hugs the axis.
⛔ Explicitly: I didn't forget to draw it; it's too small to draw.
🖥 Screen: pan to Ⓑ. ⭐ It switches to a log axis. Say why first, or it looks like a trick.
A linear axis can't hold both ends, so: log axis.
Left, the 0.8 curve at the layer nearest the input: 1.9 millionths.
Right, the 1.2 curve: 47,000.
⭐⭐⭐ Between them: ten orders of magnitude.
And the two rates, 0.8 and 1.2, differ by only 20 percent.
Leave this: what's amplified exponentially isn't the gradient.
It's each layer's tiny deviation from one.
🖥 Screen: switch to How do deep nets train at all?
⭐ Don't rush to Ⓒ. “Ten orders of magnitude” floored the room;
they want “so what now?” Answer first, then Ⓒ.
So before 2015 deep networks wouldn't train; this is why.
How do today's 61- and 100-layer networks train?
⭐⭐⭐ Almost anticlimactic: a plus sign.
“Output = F(input)” becomes “output = input + F(input).”
The local derivative goes from “F's derivative” to “one plus F's derivative.”
Why is that enough? Multiply out the 61 brackets:
one term is every bracket taking its 1.
⭐⭐⭐ And one to the 61st is still one.
Whatever the weights, the gradient always has a non-decaying path to any layer.
⛔ Residuals swap “multiply” for “add.”
Without them, layer 3's gradient hangs on the whole product of layers 4 to 61.
With them, contributions are added:
a bad layer costs one term; it can't zero the whole path.
One more, and you'll see why it trains so well:
the F branch is initialized small; many implementations zero the last projection.
⭐ So each local derivative starts as “one + something tiny.”
The network starts as a shallow one and learns F layer by layer.
⭐⭐ Remember: residuals don't add a gradient bypass.
They pin every local derivative near “one” instead of “some number.”
⛔ Don't expand that expression. The original paper (He et al., arXiv 1512.03385; identity-path derivation in arXiv 1603.05027) states it elegantly, but “one to the 61st is still one” is all of it. Write it out and the room copies instead of listening.
⭐⭐ Back to Ⓒ. The room will ask “who sets this rate?”
Where does each layer's rate come from? Several factors; one is the activation's derivative.
Two curves. Red: sigmoid's derivative. Its peak is only this high.
And it can be computed: sigmoid's derivative is σ times one minus σ,
peaking at σ = 1/2, exactly 0.25.
⛔ So sigmoid alone multiplies every layer by at most a quarter.
⭐⭐⭐ Point at the green line: the takeaway.
Green is ReLU's derivative: exactly one on the positive side.
⭐⭐⭐ That's why deep networks suddenly trained once ReLU arrived.
It took each layer's built-in “push down” from a quarter back to one.
⚠️ Precisely: the weight multiply sits on top, so not “it always shrinks fourfold,” just a built-in downward push.
⭐⭐⭐ Finally, history: the lecture's best story, worth ninety seconds. ⭐ Don't read the German title. Just say “a master's thesis in German”; the contrast does the work.
How it was found:
1991, Technical University of Munich, a master's thesis.
⭐ In German, never in an English-language journal.
Yet it first formally pinned down why deep nets wouldn't train.
Three years later, across the Atlantic, Bengio et al. published in IEEE
“Learning Long-Term Dependencies with Gradient Descent Is Difficult.” The title is the conclusion.
⭐⭐ Then the ending. Pause before it.
The two groups published separately; who came first is still debated.
⭐ But it ends nicely: in 2001 the four co-wrote a chapter,
“Gradient Flow in Recurrent Nets.”
Independent discoverers, writing it up together a decade later.
⭐ Close back on the main thread:
One framing point: it was first found in recurrent networks,
not the feedforward one drawn here.
⭐ Same thing: a recurrent network unrolled over time
is a very deep network, as deep as the sequence.
Those experiments spanned a thousand steps: a thousand layers.
⛔ Practically, the two ends look completely different.
Exploding is loud: loss shoots off or goes NaN; you see it at once.
Vanishing is silent: loss won't drop, early layers look untrained,
nothing errors. ⭐ So exploding is easy to diagnose; vanishing, hard.
⭐⭐⭐ Last beat: the segment's deepest line, and the most skipped. ⛔ Don't just talk. Put up the figure; let the room count.
🖥 Screen: Why back to front, the whole figure;
first point only at rows Ⓐ and Ⓑ, silently.
⭐ Both rows are one chain; only the arrows and the seed's end differ.
Seeing beats telling.
A rarely asked question: why back to front?
Mathematically both directions work, same result.
The one difference: how many times you walk the chain.
Top row: forward mode. For one parameter's gradient,
perturb it and follow the chain to the loss.
That pass yields only that parameter's gradient,
the only one perturbed.
671B parameters, 671B passes.
Bottom row: the seed goes in on the right,
where the end is one number, the loss.
So you plant it once.
⭐⭐⭐ Key: the check under each box. Each layer's gradient is picked up on the way back,
not after.
At the far left, all 671B are in hand. One walk.
⭐⭐ Saying isn't enough. Play the nine-second animation; let them wait.
🖥 Screen: Two chains, one start—who finishes first? (animation below).
Two identical chains, one start.
Top, forward: the ball makes one trip per parameter, lighting it only at the end.
Bottom, reverse: from the loss, the ball walks back once, lighting every parameter en route.
⭐ Blocks on the right: one per trip.
The bottom stopped long ago at one; the top grinds on to eight.
In a real model, that “eight” is 671B.
⭐ Let it loop twice. The waiting is the argument.
🖥 Screen: pan to Ⓓ. ⛔ It stops a skewed conclusion. Don't skip.
The rule, one line: start from the narrow end.
⛔ It's not “backpropagation always wins.”
Counterexample, right: ten parameters, million-dimensional output:
go front to back; backprop loses.
⭐ We use reverse only because we're 671B in, one out.
⭐ Close on the bill, the hand-off to Section 2. Don't skip:
Back to step ②: the weight gradient needs the input saved in the forward pass.
That's the root reason activations stay.
Not bad design. The backward math itself requires them.
⭐ One more short stretch, central to this lecture's scope. ⛔ Short on time? Skip it, but glance at “Doesn't save compute” first.
🖥 Screen: Life of a gradient.
⭐ Moved here from Section 2's start (2026-09-22); it is this stretch.
From computed to thrown away, a gradient passes five stages.
Each is optimized for memory, communication, or time.
⭐ Use: every later optimization targets one of these five.
So far we've reached only computed; still ahead:
all-reduce across devices, clipping, maybe accumulation, then the optimizer uses it and it's discarded.
Inference has none of these.
The all-reduce has one counterintuitive property:
its volume is parameter count times bytes per parameter. Batch doesn't enter.
One sequence or a thousand, same bytes sent.
⭐ But compute grows with batch. So bigger batches dilute this communication;
hence the push for bigger batches.
Gradients come out one by one, back to front.
The last layers' are ready long before the early layers finish;
send them in that window.
⭐ Overlap compute and transfer: communication hides behind compute.
Training's most important optimization class; inference has no counterpart.
🖥 Screen: switch to Activations form a mountain, the whole figure.
Don't explain. Point at the summit line, say “the peak is here,” read two numbers, move on.
One 4K-token sequence without recomputation needs 104.1 GiB of activations.
⭐⭐ Against a device: an 80 GiB device can't hold even one.
Full recomputation drops the peak to 5.07 GiB. The same device fits fifteen.
⚠️ Both are our own per-op derivation, with no third-party backing: orders of magnitude.
⭐ Beyond that, scale linearly by batch: at per-device batch 8,
833 GiB and 40.6 GiB.
⛔ Don't get into forward-mode AD use cases (Jacobian-vector products). “With more outputs, the other direction is cheaper” is enough; the rest is another course.
⚠️ Skip the two itemized tables in 1.7 of the course page. They're for after-class number work; on stage, “the breakdown is on the page.”
⚠️ When quoting 104.1 GiB, always add “our own derivation.” More such numbers follow; set the rule now.
🖥 Main screen: course page Section 1 · One number → 671B derivatives: backpropagation · main figure 3× is counted
Section 2 · The cost: what can't be dropped, and the first real trade-off (7′)
🎯 Keep one criterion (FLOPs per byte saved) and one story (same criterion, flipped conclusion).
The last section left us owing a full chain of activations, first layer to last.
This section does one thing: trade the first thing for the second.
It's also this course's first real trade-off:
not “is there a better way?” but you get one of two. Which?
🖥 Screen: Full recompute: 1/3 more compute…,
only the two bar groups in Ⓐ. ⭐ Shape, not numbers.
Let the room see one bar grow a sliver, the other nearly vanish.
Recomputation in one line: save checkpoints, not the work.
I drop forward's intermediate results; when backward needs them, I rerun forward from this layer's input.
Cost: one extra forward pass, compute from 3× to 4×. A third more.
⭐ In return, that number shrinks to a twentieth.
⛔ 2026-09-22: this spot used to repeat the two numbers that had just closed the last section. Mere tallying. This cell carries the exchange rate, not the numbers.
⭐ That exchange rate is so lopsided it barely counts as a trade-off.
In large-model training it's on by default.
Never whether. Only how far.
⭐⭐ The core: the criterion. Explain it fully; cut later material first.
How to choose? One number.
Ask each intermediate tensor: drop it and recompute it in backward, and how many FLOPs do I pay per byte saved?
Low ratio, drop it; high ratio, keep it. The whole ranking falls out of that.
For a linear layer it has a surprisingly clean form:
exactly the matmul's input width.
Sequence length and output width both cancel.
⭐ So “which linear layer to recompute first?” takes no math. Find the narrowest input in the config.
🖥 Screen: Why one rule gives opposite answers, whole.
⭐ One of three shown whole: the point is
the two models crossing at different places.
The flat lines are the linear layers:
their cost ignores sequence length.
The climbing line is attention. Its cost rises linearly with sequence length.
⭐ One is constant and one climbs, so they must cross.
🖥 Screen: Why one rule gives opposite answers, point at the two crossovers
(solid purple, dashed orange).
But it shows two models, crossing at different places.
V3 crosses at thirteen thousand one hundred; GPT-3 only at forty-nine thousand.
⭐ Why? Different slopes, from each model's head dimension:
new model, both lines redrawn.
⛔ I once mislabeled it: V3's slope on GPT-3.
In a lecture warning against exactly that.
⭐⭐⭐ The lecture's best story. Take a beat first.
Selective recomputation comes from a 2022 NVIDIA paper.
Its criterion is exactly mine: drop what takes plenty of memory but is cheap to recompute.
… And what it picks is attention.
Nice payoff: GPT-3 saves 70% of activation memory for 2.7% more compute.
📌 If asked why 2.7% doesn't match the page: 70% / 2.7% are the paper's Megatron convention, not halved for causal. In the course's causal convention GPT-3's attention share is 1.37%. Both right. Don't raise it.
⭐⭐ Don't rush the answer. Lay out the contradiction, ask “who's wrong,” wait three seconds. ⛔ Jump to “its sequence length was 2048” and the cell is wasted: no felt contradiction, no criterion kept.
Now you're the reviewers. Who's wrong?
⭐ Wait three seconds. Don't answer; dismantle the criterion.
Don't call the paper outdated yet.
The criterion takes three inputs:
the rising line's slope, the flat line's height, and the sequence length you ask at.
⭐⭐ All three belong to the model.
So why be surprised two models disagree?
Nobody's arguing; each reads off its own axes.
⭐⭐ Plug them in one by one. At the third, own up: the best moment.
So why did it flip?
Because its sequence length was two thousand forty-eight.
See GPT-3's vertical line? It sits left of the crossover.
There, attention is cheaper than every linear layer shown.
The paper wasn't wrong. The premise changed.
⭐⭐ The value isn't the conclusion but this:
same criterion, same model family, not a word changed, conclusion flipped. Only sequence length moved.
⭐ Close: one practical line, one general:
Practical: you mostly needn't worry now. FlashAttention never stores the score matrix; it recomputes it in backward, making the paper's advice the default.
⚠️ Next two lines are backup (the Megatron passage is collapsed). Don't volunteer them. Only if asked whether attention is still on the recompute list; ~30 s.
⛔ Don't say ‘attention isn't even on the framework's candidate list.’ It's still on the list, and it's the default once you turn on selective. Interestingly, Megatron kept the default and pinned a warning in the source: with fused attention you needn't set it; be sure you really need it.
⭐ Why not just change it? Changing a default breaks reproducibility for everyone who ran with it. So it stays put, a fossil.
⭐⭐ One more criterion, more general than the story; I tripped on it twice.
One more criterion; I tripped on it twice here:
⭐ Hardest to catch: borrowing a relation that sounds like common sense.
No error, no crash. It just nudges every later number a little off.
And it sounds right, so you've no reason to doubt it.
Both times, the same “common sense”: the widest linear layer equals d_model.
On V3 I missed the layer that rejoins the heads; on GPT-3, the MLP. It holds on neither.
The general one, and not just for recomputation:
any parameter that changes the shape of data shards can't be copied across scales.
Our example shows it's harsher than it sounds:
sequence length is on that list too, and it flips not the size of the gain but its sign.
⛔ Don't read the ranking table row by row (the nine-row table in 2.4, now collapsed). Name three tiers: “free, worth it, never touch.” Reading a table aloud is the fastest sleeping pill.
⚠️ Don't present FlashAttention as “an optimization.” Here it built the paper's advice in. Call it an optimization and the thread snaps.
🖥 Main screen: course page Section 2 · The cost: what can't be dropped, and the first real trade-off · main figure Why one rule gives opposite answers
Section 3 · With gradients, how big a step? Gradient descent and optimizers (28′)
🎯 Keep three: 16 bytes' five items, the high-precision test, the learning rate's units.
⭐⭐ The longest, most self-contained segment. Short on time? Squeeze the two before it, keep this one. It stands alone as a lecture.
⭐⭐ Open with a person (90 s): the rest is accounts and formulas; give it a face. ⛔ Don't skip: it was requested, and it motivates “why we end up estimating the second derivative.”
Before the accounting: who came up with this?
1847. Cauchy.
⭐ More than a century before neural networks.
His title, translated: “General method for solving simultaneous equations.”
He wasn't doing optimization at all, but solving equations.
He faults the standard method, elimination, twice:
One, often you simply can't eliminate.
Two, even when you can, the one equation left is absurdly complicated,
however simple the originals.
⭐ So he changes the question: instead of solving equations, minimize a function.
Several equations into one function? Bluntly:
Square each equation, add them up, and set the sum to zero.
Squares are never negative, so the sum is zero exactly when every term is.
⭐⭐ Just like that, “solving equations” becomes “walking downhill.”
⭐⭐⭐ The story's landing point. Pause.
So the real invention isn't calculus; that's Newton and Leibniz.
It's this change of question. Every model we train runs on that move.
⭐⭐ The paper's closing line. Plant it now; 3.3b picks it up.
His closing line stopped me:
once this gets you reasonably close,
‘one can then use the linear method, that is, Newton's method’ to converge quickly.
⭐⭐ That splits the job in two:
first order gets you there from far away, second order closes in at the end.
In 1847 he'd written the division of labor.
⛔ This segment's second half is that: what to do when second order can't close in.
Back to the bill.
During training, every parameter carries sixteen bytes.
🖥 Screen: Optimizer lineage. Zoom into the Ⓑ ruler first.
⛔ Not the Ⓐ lineage first, or it looks like a history lesson.
Two bytes of bf16 weights, two bytes of bf16 gradients, plus twelve for three fp32 copies:
master weights, momentum, second moment.
Weights 2, optimizer side 12. Six times as much.
⭐⭐⭐ The segment's first test. Asked live; present it as it went:
Someone will ask: why are three boxes fp32 and two bf16?
No need to memorize them. They follow one test:
do this quantity's old contributions ever leave?
⭐⭐⭐ Don't explain the test. Let the figure do the arithmetic. ⛔ “Small plus big gets rounded away” is bare arithmetic: believe it or not. ⭐ Say the numbers were computed by a script. It always lands.
🖥 Screen: Precision, in one picture, Ⓐ first.
How a float is stored: two fields.
The exponent sets how big it can be; the mantissa sets how fine.
⭐ bf16, middle row: Its exponent is as long as fp32's above it.
So bf16 covers exactly the same range as fp32. All it loses is precision.
⭐⭐ Bottom row, fp16: sixteen bits too, but three exponent bits moved to the mantissa.
Finer, but its range is much smaller:
small gradients fall straight to zero. That's why fp16 needs loss scaling, not “sixteen bits not being enough.”
🖥 Screen: pan to Ⓑ. ⛔ The high point. Go slowly; pause twice.
An experiment: a weight of 1.0,
each step adding three ten-thousandths, typical late in training.
This number line: near 1.0, adjacent bf16 values are 0.0078 apart.
⭐ In between, nothing. Not low precision: those numbers simply can't be stored.
And our increment is only 3.8 percent of one gap.
⛔ Round to nearest, and it snaps right back to where it was.
⭐⭐⭐ Pause, then ask. They'll get it wrong.
Then twenty-six adds cross one gap, right?
I actually ran it a thousand times.
⭐⭐⭐ The result is 1.0. It didn't move once.
Every add starts from 1.0 and rounds back to 1.0. It can never get across.
The right panel is the control: the same thousand adds in fp32, and 1.0 becomes 1.3.
⭐ Close with this; it kills “just to be safe.”
So “keep an fp32 copy of the master weights” isn't about safety.
Without it, late training truly stops: the loss stalls, and you can't find why.
⭐ These numbers aren't quoted from a paper.
The figure's script computes them with real bf16 rounding,
guarded by three asserts. Run it yourself.
🖥 Screen: pan to Ⓒ.
Now the test needs no memorizing:
do this quantity's old contributions stay forever?
Master weights: tiny increments over 100,000 steps, the old ones always there: fp32.
Gradients and activations: computed, used, discarded. No “old” at all. bf16 suffices.
⛔ Follow at once with the first-party counterexample, or the test sounds too simple.
I used to state this too coarsely; first-party evidence sharpened it.
The DeepSeek-V3 report: both moments in bf16, no observable degradation,
while master weights and gradients stayed fp32.
Why? Momentum and the second moment are moving averages.
Each step multiplies by a factor below one, so old contributions decay away.
⭐ A moving average is not accumulation. The real line isn't “does it accumulate,”
it's “do old contributions stay forever.”
⭐ Part two: the lineage. Brisk: a transition, not the point.
🖥 Screen: same figure, pan to Ⓐ.
This figure is ordered by how many state copies each parameter carries.
SGD, zero → add momentum, one → AdaGrad and RMSProp, one each →
Adam and AdamW need both.
⭐⭐ First define weight decay itself.
Else the next panel fixes something they've never seen.
First, weight decay.
Often mentioned, never defined.
In one sentence: every step, shrink the weights a little toward zero.
The textbook reason: regularization, no memorizing through a few huge weights.
⛔ But that reason doesn't hold for LLM pretraining.
We see the data once, far more of it than parameters: no chance to overfit.
Yet in the configs, almost every large model uses it, nearly all at 0.1.
Why?
⭐ The key: double weights that feed a norm, and the output doesn't change:
the norm erases it.
So weight length doesn't change the function, only one thing:
how far a same-size step turns the direction.
⭐⭐ Longer weights, smaller turn per step: a lower effective learning rate.
During training the weights slowly grow longer.
⭐⭐ Left alone, the effective learning rate drifts down on its own.
Weight decay holds the length in place.
⭐ Today it's more a knob steadying the effective learning rate than regularization.
How does it decay? Rearranged:
each step multiplies by a number just below one.
So it's exponential: with no gradient pushing, weights decay to zero.
⭐ Gentle: by V3's numbers, after ten thousand steps, still 80 percent. Very light.
⛔ The trap: the learning rate sits inside that factor.
Change the schedule and the regularization strength changes too.
You think you turned one knob; you turned two.
Most interesting: why does it live in the optimizer?
⭐⭐ The answer: it didn't use to.
It started as a term in the loss;
autodiff adds the matching gradient term for you.
So it was part of “what you want to optimize.”
Why move it?
In the loss, it runs the gradient's whole pipeline:
momentum, second moment, denominator, all of it.
We want a uniform pull.
⭐⭐ Only the update itself bypasses the denominator, and that's the optimizer's.
So it moved from “what you want to optimize” to “how you take the step.” Its home changed, not the formula.
⭐⭐ The AdamW panel tripped people up live.
Not just “divided by the second moment”: show two parameters side by side.
Now the W in AdamW.
Weight decay nudges weights toward zero each step, so they don't keep growing.
The only question: at which stage to shrink.
The old way mixed it into the gradient: an extra term, λ times the weight.
Natural enough: it's all subtracted anyway.
⛔ That's exactly where it goes wrong.
Mixed into the gradient, it gets every later step too, including ÷√v.
Two parameters have both grown to ten.
One has big gradients daily: a large denominator, decay shrunk, barely pulled back.
The other's are always tiny: a small denominator, decay amplified, yanked back hard.
⭐⭐ Both grew to ten and deserve the same restraint, yet get opposite treatment.
We want “pull every weight back a little, uniformly,”
not “hand out the penalty by gradient size.” The two got tangled.
⭐ AdamW's fix is one line: the decay doesn't go through the denominator.
Gradients untouched; at the update, subtract the decay straight from the weight.
⭐⭐ That's what decoupled means, and the W is the W in weight decay.
Thirty years of adding more, until 2024, when someone took a step back.
Muon drops the whole second-moment copy: sixteen bytes become twelve.
⭐⭐⭐ Bytes are one view. Next figure, another thread: cause and effect. ⛔ Read out names and they'll recall only “there were a few.” ⭐ They form a chain, each patching the last one's hole. Nothing to memorize.
🖥 Screen: From AdaGrad to Adam, Ⓐ only at first.
Simple test: feed each the same constant gradient,
plot its effective learning rate.
⭐ A healthy optimizer should give a flat line.
Anything not flat is the optimizer's fault, not the data's.
Of the four small boxes, only the first and last are flat.
🖥 Screen: pan to Ⓑ.
AdaGrad first. Its problem shows up late.
Its denominator, the running sum of squared gradients, only ever grows.
So the effective rate sinks: by step six hundred, only 4.1 percent of steady state.
⛔ Again: the input gradient never changes.
Not the data: the ever-growing denominator choked it.
🖥 Screen: pan to Ⓒ.
RMSProp swaps the sum for a moving average: the denominator forgets; late problem cured.
⛔ But it picks up a new one: the moving average starts from zero.
Early, too few samples: the denominator is badly underestimated.
The first update is amplified thirty-two times, off the chart.
⭐ Adam's bias correction fixes exactly that. The green line: flat from step one.
⭐⭐ Neat detail: Adam without correction
peaks at only 6.6 times, not thirty-two:
momentum is biased too, and the biases partly cancel.
⭐⭐⭐ The three origin stories: the segment's best part, worth two minutes. ⭐ Side by side; the contrast is in the kind of publication.
Now how these three were published. A fun contrast.
AdaGrad: a proper JMLR paper, a full thirty-nine pages.
⭐ The title: “Adaptive Subgradient Methods for Online Learning and Stochastic Optimization.”
It wasn't written for deep learning at all.
RMSProp: ⭐⭐⭐ there is no paper.
Its only “publication” is one slide in Hinton's Coursera course.
Papers that cite it cite “Lecture 6.5.”
⭐ Its subtitle is itself the complete definition of the algorithm:
“Divide the gradient by a running average of its recent magnitude.”
One sentence, one slide: the chain's middle link.
Adam: two things I love, both in the paper.
A footnote on the first page: “Equal contribution. Author ordering determined by coin flip over a Google Hangout.”
⭐ And in the acknowledgments: “special thanks to Ivo Danihelka, and Tom Schaul for coining the name Adam.”
The authors didn't name it themselves.
⭐⭐ Close the panel with this line.
The best thing about this chain:
it changed the whole field, and its middle link has no paper.
⭐ The hardest evidence it's a real chain is proved in the Adam paper:
AdaGrad is a special case of Adam.
The later contains the earlier: that makes it a chain.
⭐⭐⭐ Part three: why following the gradient makes the loss go down.
⭐⭐⭐ First the “root” panel. It matters more than every trick after it. ⛔ Few have heard why following the gradient helps explained properly. Skip it and the rest is tricks without a base.
🖥 Screen: Why gradient descent works, Ⓐ only at first.
⭐ Both chains of balls come from a real gradient-descent run, not placed by hand. Say so.
Simplest case: one parameter.
The rule: look at the slope underfoot, move a little downhill.
Positive slope, go left; negative slope, go right. Repeat.
Picture a ball rolling down a hill.
⭐⭐ The red chain: the first “free gift.”
Look at the red balls' spacing:
the first step is biggest, then each is smaller than the last.
Why? We made the step proportional to the slope.
Nearer the floor, flatter slope; flatter slope, smaller step.
⭐ Nobody told it to slow down. The slope flattened out by itself.
So it doesn't keep overshooting. That comes free.
⛔⛔ The green chain: this panel's one line that must stick. Pause after.
Green starts from the right.
Both follow exactly the same rule.
But red lands in the shallow valley, green in the deep one.
⭐⭐⭐ Only one thing differs: where they started.
Gradient descent never promised the optimum, only that each step improves.
🖥 Screen: pan to Ⓑ, then to Ⓒ.
With two parameters you need a direction: a vector, the gradient.
It points toward steepest ascent, so downhill is its negative.
⭐ Its length also tells you how steep it is.
At 671B parameters, forget the “hill.” You can't draw it.
Read it as a column of 671B numbers.
Each entry says two things. Sign: push that parameter up or down.
⭐⭐⭐ Its relative size: which ones matter more to change.
Not just which way to go, but what to move first.
⭐⭐⭐ Put Zero slope ≠ stuck up at once. It's the antidote.
⛔ “The valley depends on the start” leaves them uneasy:
so it's luck? ⭐ Dismantle that, or they'll doubt the tuning advice after.
🖥 Screen: Zero slope ≠ stuck, Ⓐ only at first.
The last figure left a shadow: pits everywhere, and you land wherever.
Now: that picture is 1D lying to you.
Three panels, three kinds of zero slope.
First: uphill both ways. A true valley floor; every direction is worse, you're stuck.
Second: downhill both ways. A peak: one nudge and you're off; training rarely meets one.
⭐⭐ The third matters most: uphill one way, downhill the other.
⭐ The metaphor; it beats any math. ⭐ Hikers get it instantly.
The third kind has a perfect name: a pass.
Where a road crosses a range.
Along the road, it's the highest point;
along the ridge, it's the lowest.
The ground there is flat. But it's no dead end: the way out is right beside you.
⭐ Geographers call this landform a saddle. In math, the point is a saddle point.
⭐⭐⭐ After the cross-sections, spin it: 3D won't fit in 2D.
🖥 Screen: The same saddle, rotating.
The last figure cut it twice for cross-sections.
Here's the real shape: red curves up one way, green sinks down the other,
crossing at the black dot.
⭐ On that dot the ground is flat. But the green path is always there.
🖥 Screen: pan to Ⓑ. ⛔ The figure's whole argument. Read the three rows one at a time.
The key step.
A true valley floor must go “uphill in every direction.”
A 1D plot has two directions; both uphill is easy.
Hence pit after pit.
But hundreds of billions of parameters mean hundreds of billions of directions.
⭐⭐⭐ All must go up. Miss one, and it's only a pass. You keep going.
⛔ Brake here, or you'll overclaim. ⭐ Mathematicians will raise a hand. Say it before they catch you.
To be precise: this is about the definition, not a probability argument.
Curvatures in different directions aren't independent, so no number like
“rare to ten to the minus something.” It'd be made up.
⭐⭐⭐ The key step: why “closed in” and “very low” go together. ⛔ Skip it and the above sounds like “high dimensions forbid pits.” Wrong.
⛔ First, kill a misreading: “enough dimensions, no pits.” False.
Four dimensions or a thousand: either can hold a deep, sealed pit. Nothing forbids it.
⭐ What's true: a pit both high up and closed in is extremely rare in high dimensions.
⭐⭐⭐ The passage's deepest point. Go slowly.
So why do “closed in” and “very low” go together?
“Closed in” means every direction goes up, no exceptions.
⭐⭐ But the higher you stand, the more ways down:
there's plenty of height below.
Conversely, nearer the bottom, fewer ways down remain, until none do.
That's the valley floor.
⭐⭐⭐ So, precisely:
in high dimensions, the stalls are overwhelmingly passes, not floors.
Not that floors are forbidden: they're all very low, and you're not there yet.
📌 If asked for the source:Dauphin et al., arXiv 1406.2572 (NeurIPS 2014). Abstract: not the usually blamed local minima far above the global minimum, but a deeper difficulty: the proliferation of saddle points, especially in high dimensions.
🖥 Screen: pan to Ⓒ.
Not just reasoning: someone measured it.
Train until the gradient is zero, then count: what fraction of directions go up?
⭐ Result: that fraction never reached 1.
So wherever training stopped, some directions still went down.
It stalled on a flat slope, not for lack of a way out.
⭐⭐⭐ Deliver this last turn: it sets up all of Section 3. ⛔ Good news alone invites “then why tune anything?”
So the deepest half of “why gradient descent works”:
✅ the more parameters, the harder it is to get truly stuck.
It answers a common confusion: “aren't more parameters harder to train?”
In memory and compute, yes. On “will it get stuck,” the opposite.
Separate questions.
⛔ The bad news: not stuck doesn't mean fast.
Near a pass the gradient is tiny; step ∝ slope,
so it crawls there a long time.
⭐⭐⭐ Remember this: most optimizers ahead
exist to fix exactly that.
⭐⭐⭐ Straight into momentum, the first answer to that line. ⛔ After “not stuck, but slow,” they wait for “so what?” ⭐ One picture suffices: a ball, a shallow pit, two trajectories.
🖥 Screen: Momentum, Ⓐ only at first.
The fix is momentum. One sentence:
each move depends on the current gradient and on the last move.
Give it inertia.
Same loss, same start; only inertia differs.
Gray rolls into the shallow pit halfway and stops.
⭐ It really stops: the gradient there is zero.
Green keeps its velocity, rushes past, and lands in the deeper valley.
It also overshoots and swings back: those few points.
⭐⭐ Then Ⓑ, the trick. Go slowly. ⭐ First ask “how does it get past?”; give them three seconds.
🖥 Screen: pan to Ⓑ.
How does it get past?
Take the move as it crosses the pit floor and split it in two.
The arrows are drawn to scale.
Term one, the gradient: 0.092, pointing right.
⛔ Note: it pulls the ball back. Just past the floor, the slope drags it back.
Term two, inertia: 1.304, pointing left.
Net move: 1.213 to the left. It keeps going.
⭐⭐⭐ So at this instant the gradient is fighting it.
Inertia carries it out.
⭐⭐⭐ The history: worth ninety seconds. ⭐ The punch is the year; say it first.
How old is this idea?
1964.
⭐ Twenty-two years before the backpropagation paper in Nature.
Soviet mathematician Polyak, plain title: “Some methods of speeding up the convergence of iteration methods.”
⭐⭐ His name for it: “the method of a small heavy sphere.”
So the ball you pictured isn't my idea. It's his name.
⭐⭐ Read his line. It matches your Ⓑ word for word.
Better: what I said about Ⓑ, he wrote almost word for word in 1964:
“The motion proceeds not in the direction of the force (i.e. antigradient) because of the presence of inertia.”
He also says that term makes it move along the valley floor.
⭐ One sixty-year-old sentence, describing our three arrows.
⭐⭐⭐ Follow up here, or they leave with the wrong half. ⛔ “Rushing past a shallow pit” is, of the three mechanisms, the least used day to day. The workhorse is below.
One thing, or you'll misremember what it's for.
The figure showed rushing past a shallow pit. Real.
But closed-in pits are rare in high dimensions, as we said.
⭐ So “momentum helps you escape local minima” grabs its least-used side.
Its everyday job differs.
Recall the valley steep one way, flat the other: bounce across, inch along.
⭐ Across, the gradient flips sign every step.
Momentum adds them: the signs cancel, the swing is damped.
⭐ Along the flat, the gradient keeps its sign; added up, it keeps speeding up.
⭐⭐⭐ The momentum panel's real landing. Read slowly, pause.
So momentum is really a filter:
it amplifies the steady direction and damps the swinging one.
⭐⭐ It doesn't cure “can't get out of the pit,” it cures “bashing into valley walls.”
Polyak's move along the valley floor means exactly this.
⭐⭐ Then “when doesn't it work?” There's a computable ceiling. ⭐ First ask: “why not turn momentum all the way up?”
When doesn't it work?
There's a computable ceiling.
Classical momentum: velocity = β × last velocity + current gradient.
With a constant gradient, velocity builds
until it reaches 1/(1−β) times the gradient.
β of 0.9 gives ten times; 0.99, a hundred.
⭐ That's all its capital.
At β = 0.9, it amplifies the step at most ten times.
A pit wall higher than that, and it can't get out.
So “sometimes it doesn't work” is neither mystery nor bad tuning:
finite energy, and now you can compute the ceiling.
In our figure, that pit happens to be within the tenfold.
⛔ The downside, so it isn't pure profit.
It overshoots: you saw it swing before settling.
⭐ Momentum trades “more wobble near the finish” for “no stopping on the way.”
🖥 Screen: pan to Ⓒ.
Last, what β is.
“Last step's move” contains every past step, lighter the older it is:
each step back multiplies by another β. That's this row of bars.
So its effective window is about 1 over (1 minus β).
β of 0.9 means about ten steps.
⭐ β isn't magic. It's a knob for “how long to remember.”
⭐⭐⭐ Close with this punchline.
Polyak's working range in 1964: 0.8 to 0.99.
Adam's default β₁ today? 0.9.
⭐ Sixty years later, we're still inside his range.
He even wrote the tuning order: tune the learning rate first, momentum off;
add momentum once convergence slows.
⛔⛔ Brake at the end, or it sounds like a cure-all.
More precisely:
⛔ What momentum can rush past is a shallow pit, not every pit.
The ball escapes because that pit is shallow enough. Deeper, and it's stuck just the same.
⭐ So it weakens “where you start decides everything,” but doesn't eliminate it.
It cures “slow,” not “optimal.”
⭐⭐ This last line is the segment's joint. Say it.
Where does that whole column of numbers come from?
Exactly what one backward pass in Section 1 computes.
⭐ That's where Sections 1 and 3 join.
⭐⭐⭐ Part four (breather here): how big a step. ⛔ Parts one to three were which way to go; now the question changes. Stop two seconds, drink water, let them feel the page turn. ⭐ The segment's high point, an angle few cover.
A question that sounds stupid:
how does a gradient become a weight update?
Plainest: new weight equals old weight minus learning rate times gradient.
But why should that hold at all?
⭐⭐⭐ Hold off on units: two knobs show the rule is backwards. ⛔ Whiteboard it; spoken numbers have no force. ⭐ Four numbers do it: 1000, 100, 1, then 10 and 1000.
A machine has just two knobs; the error is a thousand.
Knob A is sensitive: one notch changes the error by a hundred.
Knob B is sluggish: one notch changes the error by only one.
To cancel the thousand, how many notches each?
⭐ Pause two seconds; let them work it out. Grade-school arithmetic. A conclusion they reach themselves carries the segment.
Knob A: a thousand divided by a hundred, ten notches.
Knob B: a thousand divided by one, a thousand notches.
⭐⭐ See? Turns needed go against sensitivity.
Sensitive knob, fewer turns; sluggish, more.
Our one line of code?
Step = learning rate × gradient: A moves a hundred, B one: a hundred to one.
⛔⛔ We need one to a hundred. Not slightly off: exactly backwards.
The sensitive knob oscillates; the sluggish one barely moves.
⭐ The whole machine is dragged along by its most sluggish knob.
⭐⭐ Tie back to Section 1's vanishing gradients. The line most likely to be heard backwards.
⭐⭐⭐ This explains Section 1's vanishing gradients.
The first layers' gradient was just 0.0016. That doesn't mean they don't matter.
A small gradient means a sluggish knob, so further to go.
Not that it needn't move: it needs to move most, and we gave it the smallest steps.
⭐ Units: they answer “why there must be a conversion factor.”
The units on the two sides don't match.
Gradient: “nudge this parameter, how much does loss change”: loss over parameter.
You want “how far should the parameter move”: units of the parameter.
Something must convert between them: the learning rate.
Its units: parameter squared over loss.
⭐ So the learning rate isn't a pure number.
It's tied to the size of your loss and your weights.
Hence retuning for a new model or batch. No universal constant.
⭐⭐⭐ What should it be? First let them see the pain. ⛔ “Gradient scales differ by orders of magnitude”: everyone nods, nobody knows what it looks like. ⭐ This figure draws it, and motivates every optimizer after.
🖥 Screen: One LR can't serve all params, Ⓐ only at first.
⭐ Axes first: vertical bounce = steep parameter, horizontal crawl = flat one;
minimum at right.
A valley steep one way, flat the other. Two trajectories,
same valley, same rule. Only the learning rate differs.
Red first: each bounce bigger than the last, until it leaves the frame.
Blue: bounces shrink, and it reaches the minimum.
⭐⭐ How far apart are the two learning rates? Ten percent.
That little, and one blows up, the other doesn't.
⛔ That's how narrow your workable window is.
⭐⭐ Blue's zigzag: the figure's real payoff. ⭐ Most see “blue reached the bottom” and assume it's fine. Make them look again.
Blue isn't good either. That zigzag:
most of every step goes into bouncing side to side.
Only a sliver heads toward the minimum.
⭐ It isn't walking. It's inching.
⭐⭐⭐ Is there an exact answer to “how big a step”? Yes, computable.
🖥 Screen: Step size: there is an exact answer.
On a parabola, the optimal step can be computed:
⭐ gradient divided by the second derivative, and you're at the bottom in one step.
Covered in 1.4b; here we use it.
⭐⭐ Ⓑ, the figure's trick.
⭐ Ⓑ's trick: judging by gradient alone gets it wrong.
Steep valley, gradient two; flat, one. So steep looks farther, deserving the bigger step?
⛔ Wrong. It's actually closer to the bottom.
⭐ Only after dividing by the second derivative are the two comparable.
⛔ One sentence for Ⓒ. Don't explain the second derivative here; 1.4b did.
⭐ Ⓒ answers “why nobody actually computes it”:
one second derivative per pair of parameters; at that scale, you can't even store it.
So you estimate it, and that's where Adam's denominator comes from.
🖥 Screen: back to the single-LR figure, Ⓑ. ⛔ These three lines are the lecture's own landing. One at a time; pause on the third.
Someone will say: pick a middle value.
⭐⭐⭐ That's the problem: nothing in the middle is good either.
Line one: a bit bigger, and the steep direction diverges. The threshold is hard:
two over the steepness, however much you want speed.
Line two: take the largest safe value. Then the flat direction needs
0.37 times the steepness ratio in steps to get halfway. Here the ratio is twenty-five: nine steps.
⛔ Line three: on a real model, the ratio spans orders of magnitude.
Call it a thousand: three hundred sixty-four steps. And that's the fastest possible.
⭐⭐⭐ Close with this, then straight on.
So: this isn't bad tuning.
The valley's shape decides it, and the model sets the shape, not you.
⭐ Conclusion: stop looking for “the best global learning rate.” It doesn't exist.
Each parameter needs its own.
⛔ The cost is immediate: “one per parameter” means state per parameter.
The opening's twelve bytes of optimizer state trace back to this figure.
⭐ Run the adaptive line in one go; with units as the frame, it flows.
⛔⛔ But first correct a near-universal claim that misdirects thinking. ⭐ The passage's tension. Don't skip.
You've heard: Adam's denominator estimates the second derivative.
⛔ That's hindsight, not the inventors' thinking.
That reading can't explain two things: why it works so well on sparse data,
and why it gets cautious in “uncertain directions” on its own.
⭐ The real path started not from geometry but from statistics.
⭐⭐ Three generations, one insight each. One sentence apiece.
Generation one: AdaGrad, 2011. A down-to-earth problem.
They were thinking of text, not neural networks.
“The” appears millions of times, a technical term three times,
and the rare ones often carry the most information.
With one learning rate for all, that term never learns anything.
⭐ So: each parameter gets its own learning rate; the less seen, the bigger the step.
How much seen? Sum past squared gradients, divide by the root.
⛔ Note: not a word about curvature or second derivatives.
⭐⭐⭐ “What is the second moment, really?” Say this slowest.
What is this “second moment”?
Square the gradient, average, take the root: the gradient's root mean square.
Plainly: how big this parameter's gradient usually is.
⭐⭐ It's not curvature. It's this knob's own ruler.
Dividing by it?
Numerator: this gradient. Denominator: its usual size. Divide, and get
“big or small versus its own normal?” A dimensionless number.
⭐ Whether its gradient is normally a hundred or 0.01, after, all share one scale.
⭐⭐ That's what “adaptive learning rate” really means: size relative to its own normal.
Generation two: RMSProp. Starting point: the ruler goes stale.
AdaGrad's sum only grows, so the denominator swells, steps shrink, and it stalls.
Hinton changed one thing: running sum to moving average.
⭐ Why: a parameter's “typical gradient size” itself changes,
large early, small later. The whole history measures today with a stale ruler.
Generation three: Adam. Issue: numerator and denominator don't match.
RMSProp has a ruler, but the numerator is still this step's raw, jittery gradient.
So average the numerator too: momentum.
⭐ Now they line up: numerator handles direction, denominator handles scale.
⭐⭐ A free bonus; expect an “oh.”
That alignment bought something nobody ordered.
The paper calls this ratio a signal-to-noise ratio:
gradient steady in one direction: numerator builds, denominator holds, a full step;
gradient jumping in size and sign: numerator signs cancel,
the squared denominator grows anyway. The ratio shrinks, and the step with it.
⭐ Two at once: it evens out sensitive and sluggish knobs, and gets cautious in uncertain directions.
Nobody designed the second; the shape gave it free.
⭐⭐⭐ Close with the two paths side by side, calling back to Cauchy.
Two paths side by side: the segment's real lesson.
Cauchy's is geometric: step by each direction's curvature. Rigorous, unaffordable.
Adam's is statistical: ignore terrain; read the parameter's own history:
its usual gradient size, and is this one unusual?
⭐ Similar landing from wholly different starts.
⭐⭐ Hence an “adaptive learning rate,” not an “approximate Newton method.”
⭐⭐ The positive answer we owe: “so what is it?”
The word “moment.”
In statistics, first moment = expectation; second = expectation of the square.
Adam's v is a moving average of the squared gradient; “second” means only that.
⭐ A detail: it's the uncentered second moment, mean of g²,
not the variance, which subtracts the mean.
Adam has the mean at hand, yet deliberately doesn't.
Not an oversight.
With a perfectly constant gradient, momentum = √v: ratio exactly one, a full step.
Divide by the variance instead: a constant gradient has zero variance, and it blows up.
⭐⭐ Not subtracting the mean keeps the ratio bounded:
the root of the signal-to-noise behavior.
Any geometric meaning? One, just not the assumed one.
Picture the loss surface as a long, flattened ellipsoid:
big gradients in some directions, tiny in others.
Adam rescales each axis by its typical gradient size,
until all directions are comparable.
⭐ That's diagonal preconditioning, a genuine geometric operation.
But it scales by how much a direction usually moves,
not how sharply it curves.
⭐⭐ So it fixes wildly different scales, not a sharply curving direction.
One more limit: it scales along the axes but never rotates them.
Per-coordinate scaling can't reach coupling between axes:
the gap Muon and Shampoo later set out to fill.
“Dividing out the scale” has a side effect.
Adam's update ≈ momentum over the root of the second moment,
so the units cancel.
The gradient's units are gone; each step's size becomes a constant.
⛔ So what is that constant? Not 1. It's 0.2.
“The units cancel” says constant, not 1.
Measured, it holds at 0.2 to 0.3, the same across model sizes.
⭐ And it's computable: a short closed form with the usual β₁
lands on the measured value.
⛔ Don't read out the closed form. It's in 3.3b on the course page. (2026-09-22: it once sat inside the spoken script and got read aloud.)
⭐⭐ So in Adam the learning rate changes role: no longer a conversion factor.
Times 0.2, it is directly “how far each parameter moves this step.”
Hence it tunes easily and transfers well. Not magic: the units canceled.
⭐⭐⭐ And its most elegant use is as an interface.
Kimi K2 moved to Muon this way: calibrate Muon's update RMS to 0.2 too,
and Adam's tuned hyperparameters carry over as a package.
Make a quantity a constant, and the constant becomes an interface.
⭐⭐ A ruler for anytime. They're about to ask “how do I know it's right?”
🖥 Screen: “Is my learning rate right?”.
Is your learning rate right? Check a line by eye.
Measure update ÷ parameter. ⛔ The update, not the gradient.
Rule of thumb: one in a thousand, the black line. Stay near it and you're roughly right.
⛔ But cover Ⓑ: the figure's only trick.
⛔ But Ⓑ surprises: the too-high run has a lower loss.
Is the line useless? No. The cost is elsewhere:
the weights balloon and the jitter grows,
⭐⭐ and raise it further and it goes straight to NaN. A step from the cliff.
⭐ So this ratio isn't a “lowest loss” metric. It tells you your distance from the cliff.
(Exact multiples and the blow-up margin are on the course page; don't quote them.)
⭐ Part five: tuning the learning rate. Most wanted, most prone to folklore. Quote only sourced numbers.
🖥 Screen: Two LR schedules, zoom into Ⓐ first.
First, the two curves differ in shape.
GPT-3 slides down right after warmup; V3 has a long, flat plateau.
⭐ Remember that. This part ends on it.
For now, three decisions: where to ramp from, how high, how to come down.
Warmup first. The popular explanation is wrong.
Tutorials say early Adam has too few samples to estimate its second moment.
A 2024 paper testing exactly this systematically says that's not the main cause.
The real mechanism: warmup lets the network tolerate a larger target learning rate
by moving it to a better-conditioned region of the loss surface.
⭐⭐ Don't let “better-conditioned region” slide; it sounds circular. One sentence tying it to 1.4b's second derivative grounds it.
What does “better-conditioned region” mean?
On a parabola, past a step of “two over the curvature,”
it's no longer slow versus fast:
each step overshoots further, and you fly out.
So local curvature, not you, sets the usable learning rate.
And at the start curvature is so high
the peak you want isn't allowed at all.
So warmup's job is concrete:
small, safe steps for a while, until the network walks to lower curvature.
Then that peak becomes allowed.
Let me correct myself:
the network doesn't learn to tolerate big steps. The surface gets gentler.
⭐⭐ These three questions will come: three faces of one misconception. Take apart the premise; don't answer singly.
Someone asks: at the start it's a one-in-129,280 guess; aren't gradients huge?
A large loss isn't a large gradient.
The initial loss is 11.77, big.
But as drawn, the logits gradient is p minus y, each entry between minus one and one.
⭐ Vocabulary of 129,280 or 1,000, it doesn't matter.
The second is better: hasn't Adam normalized the gradient already?
Yes, and that points to the answer.
Momentum over gradient RMS: the units cancel.
Each step's length is basically the learning rate, nearly regardless of the gradient.
⭐⭐ So the danger isn't “large gradients” at all.
What's large isn't your step. It's the curvature underfoot.
Back to the second derivative.
⭐⭐ Adam divides out “steep,” not “curved.”
Its second moment is a statistic; curvature is geometric.
It never computes that.
Hence the paper's “preconditioned curvature”: curvature after Adam's processing.
Still high after processing? That's the number measured, and it's high.
Others ask: isn't the denominator just not built up yet?
Adam guards against that with bias correction, made for exactly those first steps.
⛔ A last caution, not to overstretch it:
not “curvature keeps dropping, so the learning rate keeps rising.”
The curve stops at the peak, then has to come down.
Warmup only gets you to the peak. It doesn't keep you climbing.
Oddly, with the peak fixed, longer warmup buys almost nothing.
The peak decides the result; warmup's value is daring to set it high.
How long? GPT-3: the first 375 million tokens, 0.125 percent of the total.
Too short to count as a hyperparameter.
What peak? Copy a ready-made table.
GPT-3's Table 2.1 lists eight sizes: 125M uses 6 × 10⁻⁴,
down to 0.6 × 10⁻⁴ at 175B.
⭐ Plainly: bigger model, smaller learning rate, bigger batch.
And gently: 1,400 times the size, only ten times the learning rate.
⛔ In that table batch and learning rate move together.
Easiest to miss: never tune the learning rate without the batch.
🖥 Screen: pan within Two LR schedules to Ⓑ.
Coming down, two schools.
Old: cosine. GPT-3 decays to 10 percent of peak over 260 billion tokens, then holds.
New: hold constant, decay only at the end. DeepSeek-V3 runs flat at the peak
and starts decaying only at ten trillion tokens.
⭐ Why it caught on: cosine needs the total step count up front,
its shape tied to the endpoint. A constant phase can stop anytime; a short decay yields a usable
checkpoint. “How long to train” moves from day one to whenever you like.
⭐⭐ Finally, the dashed line:
both schools end near one tenth of the peak.
⚠️ Two samples aren't a law, but enough to show “decay to zero” isn't the default
many assume.
⭐⭐⭐ Cover batch right after: “the two are tied” left half a debt. ⛔ Requested by name (“why does a bigger batch make training more stable? Cover the theory too”). ⭐ They expect folklore; it's just the old math of averaging, and satisfying. 90 seconds total, not a section.
How big a batch?
⭐ First: a bigger batch training steadier is no mystery.
A gradient is the average of each sample's vote.
Eight people in a room, one extremist: he drags the average far.
Eight hundred, and he's barely heard.
⭐ So jitter shrinks roughly as one over √B. Four times the batch, half the noise.
⛔ Then why not keep growing it?
Two explanations: more samples stop buying equal progress;
and a bigger batch needs a higher learning rate, which can't go higher.
⭐⭐⭐ Two reasons? Really one.
That formula: optimal learning rate = cap ÷ (1 + noise scale ÷ batch).
Both limits are the same noise scale.
⭐ Easy anchor: batch equal to the noise scale
uses exactly half your learning-rate budget. Beyond that, the other half comes slowly.
⭐⭐ So the room's two clashing lines are each half right:
“bigger is steadier” is stability: it keeps paying, no turning point.
“too big stops helping” is sample efficiency: it has a turning point.
Choosing a batch means finding the spot between: steady enough, not wasteful.
⭐ Calibrate a third folk wisdom; close all three together.
🖥 Screen: “A bigger batch is faster”.
The third: “a bigger batch is faster.”
It has a precondition, biting right where you are.
The curve in Ⓐ: true on the left half, false on the right.
Left: devices unsaturated; fewer steps, each no dearer. Genuinely faster.
⛔ Right: saturated; each step costs more by exactly as much as the step count drops.
⭐ Tens-of-billions models on hundreds of devices sits almost entirely on the right.
So for you, a bigger batch buys stability, not speed.
⭐⭐ Someone will ask “do I have to keep adding devices?” Answer it first. ⛔ Else “the batch keeps climbing” sounds like “big labs only.”
Someone asks: if the batch keeps climbing, do I keep adding devices?
⭐ No. What climbs is gradient accumulation, not devices.
That multiplication again: five times the global batch: change “accumulations per update”
from one to five. Not one more device.
⭐ And memory doesn't grow: accumulation just delays the update,
and only one set of activations is ever live.
The cost?
Same devices, same throughput, same tokens per second.
What changes is tokens collected per update.
⭐⭐ Fewer updates, same time: no extra devices or time,
only update frequency.
Why it pays: early, the critical batch is small, so big wastes;
later it has grown, and fewer updates cost almost no sample efficiency,
buying sparser all-reduces and less noise.
⭐ If anyone doubts it, one source suffices; give the last one.
Not made up. All three check out:
Mechanism: OpenAI's 2018 gradient-noise-scale paper. Its abstract says
the noise scale increases as the loss falls over a run.
Practice: DeepSeek-V3: batch goes 3072 to 15360 over the first 469 billion tokens,
on the same 2,048 GPUs throughout.
⭐ Framework: Megatron's first-class flag --rampup-batch-size
takes three arguments: start, increment, samples to ramp over.
It asks for a target global batch, never for your device count.
⭐ This line is for those going home to code; worth more than the theory.
One trap bites silently: after accumulating, divide by the accumulation count.
Forget, and the gradient is K times too big: the learning rate times K.
⛔ Misleading symptoms: it runs, no errors, the loss even falls, then suddenly spikes.
So you check data, initialization. It's a learning-rate problem in disguise.
⭐ Two short pieces, a minute each. Minor, but skipping leaves questions. ⛔ Someone wonders “what about β₁ and β₂?” and “why are checkpoints so big?” ⭐ Each needs one sentence from what you've covered: don't skip, don't expand.
First, β₁ and β₂, usually “just use the defaults.”
⭐ You've met them: momentum's “how long to remember” knob.
Adam has two: β₁ for the numerator, about ten steps, smoothing direction jitter;
β₂ for the denominator, about a hundred steps at 0.99, since usual gradient size
takes longer to judge.
⭐⭐ Raise β₂ = a steadier, slower ruler; lower it = closer tracking, more noise.
Still just “how long to remember.”
⭐⭐ If time allows; it bites back at the earlier precision test. ⚠️ Our own inference. Flag it as a conjecture.
PaLM makes β₂ grow with the step count, so the window lengthens;
at a hundred thousand steps it reaches ten thousand.
⛔ That clashes with our test:
“a moving average forgets, so bf16 copes.”
At β₂ = 0.9999, forgetting almost stops.
v becomes more and more a real accumulator.
⚠️ So the test needs a precondition: β must not approach 1 as steps go on.
⛔ This is my inference, not the paper's. A conjecture.
Second: what must a checkpoint store?
Derive it from the sixteen-byte table.
Ask of each box: “lost, can it be rebuilt?”
bf16 weights: no need; a projection of the master weights, just convert.
bf16 gradients: no need; the next step recomputes them.
fp32 master weights, momentum, second moment: all three must be stored. History: lose them, start over.
⭐⭐⭐ So a checkpoint is exactly those twelve bytes.
For a 671B model, 7.3 TiB per checkpoint.
⭐ These two lines explain two things every engineer has hit.
That explains two things:
One, why checkpoints are so big and slow to write.
They store the optimizer, not the model; the model is a third.
Two, why released weights are so much smaller than a checkpoint.
A release needs only the two inference-weight bytes. One to six.
⛔ But those twelve bytes aren't enough. Three more things:
the step count, your position in the data, and the RNG state.
⭐ The second is easiest to miss and hardest to notice: you retrain the same batches,
and the loss curve looks perfectly normal.
⭐⭐⭐ Before closing, LoRA: the only training they'll run themselves. ⛔ Not “a new method”: no new concepts. Just the 16-byte bill recomputed for another algorithm. Three minutes.
One more piece, the one that matters most to you.
So far it's all training from scratch. What you'll run
is mostly fine-tuning someone else's model.
So what does LoRA save? Same three budgets, one at a time.
Budget one: resident state. Freeze the base; hang a pair of skinny matrices beside it.
For 7B: full fine-tuning needs 100 GiB, LoRA only 12.7 GiB.
Almost eight times less.
⭐ Why? Of the sixteen bytes, fourteen serve only parameters being updated.
A frozen parameter keeps just its two weight bytes. Back to inference.
Budget two: compute. Straight from Section 1.
The backward pass pays for two matmuls: input gradient and weight gradient.
Frozen layers skip the second: weights that never update don't need it.
Forward one, backward now one. Three times becomes two.
⛔⛔ Budget three is the point, and most misunderstood. Pause first.
Budget three: activations. Many think “train a thousandth of the parameters, and training gets light.”
Not in memory.
Adapters hang on every layer; the first needs a gradient too,
and gradients only travel back from the loss.
So the whole chain's activations must stay.
⭐ After LoRA, activations are often the biggest item; LoRA still needs recomputation.
Not either-or: they stack.
In one line: LoRA changes how many parameters carry the sixteen bytes, not the fact that the backward pass crosses the whole network.
⭐⭐ Before closing, one minute on Muon, one panel only.
⛔ The Muon figure has four panels. Skip Ⓐ, Ⓑ and Ⓓ.
⭐ Only Ⓒ is an answer they can see;
the rest need explaining and run over.
🖥 Screen: Muon, Ⓒ only.
Switch away after one minute.
A last minute on Muon, one picture.
We said “don't take too big a step.” But “too big” needs a ruler.
⭐ With one ruler, “at most one unit” marks a circle.
With another, a square.
For the same gradient, the circle's farthest point lies along that line. That's SGD.
⭐⭐ But the farthest point on the square lands on a corner. That corner is Muon.
⭐⭐⭐ Tie the segment together with this line, and switch away. ⭐ Same point as the earlier premise, “fastest depends on how you measure a step.” Expect the “oh.”
So Muon isn't “yet another new optimizer.”
It and SGD answer the same question; they differ only in the ruler for “one step.”
⛔ How that ruler is computed, what Newton-Schulz is: out of scope today.
⚠️ Backup (only if asked; three sentences, then stop):
“If you throw away the second moment, who cures ‘parameter scales differ too much’?”
“Orthogonalization cures the same problem, one level up. It sets the matrix's
singular values all equal, so no direction's move depends on its gradient size.
Adam erases scale per element; Muon per whole matrix,
reaching between axes, which per-element can't.” Details: course page 3.4.
⚠️ Pressed on “fully equivalent?”: “No. Orthogonalization only equalizes between directions,
not the overall step size. The Moonlight paper added that later.”
⚠️ Backup: “Why can it only apply to 2D parameters?” “Two different exclusions. Scalars and vectors, in principle: orthogonalization needs several singular directions, and a vector has one, so it collapses to plain normalization. But embeddings and the output head are 2D; in principle it works. The authors leave them to AdamW since it measured better. An empirical exclusion, not a principled one.”
⭐ Close the segment in three sentences, move on:
The whole thread in one sentence:
the gradient only tells you which way to go; the optimizer decides how far.
Decades of evolution answer one question: who decides “how far”.
SGD: you. Adam: the gradient's own history. Muon: the matrix's geometry.
⛔ Don't compare which optimizer converges faster.This course has no head-to-head measurements; anything you say is invented. The figure omits convergence curves on purpose.
⚠️ Keep Muon to one minute.Its only role here: “someone took a step back.” Newton-Schulz derails the pacing.
⚠️ Quote a learning rate only with its batch and schedule. A bare number means nothing, and gets copied.
⛔ In LoRA, skip “how to choose the rank” and “which matrices to attach to.” Another course. Its only role: the same bill, recomputed for another algorithm. Three minutes; stop at “activations don't shrink.”
🖥 Main screen: course page Section 3 · With gradients, how big a step? Gradient descent and optimizers · main figure Optimizer lineage
Section 4 · The bill goes to Topic 5 (2′)
🎯 Keep one line: parallelism isn't invented; this bill's size ranking forces it.
Rank the items by size:
⭐ Optimizer state is largest: twelve bytes a parameter. Weights, gradients: two each.
Activations have no fixed size: they scale with this step's tokens.
When it doesn't fit, shard in that order:
largest, least-used first.
⭐ That's exactly ZeRO's three stages: optimizer state, then gradients, weights last.
⭐⭐ Landing line. Say it, move on.
So there's no order to memorize.
Parallelism isn't invented; this bill's size ranking forces it.
How to shard, onto whom, at what communication cost: Topic 5.
🖥 Screen: course page Section 4; three sentences suffice.
⛔ ZeRO's three stages is collapsed. Leave it.
⛔ Don't get into communication volume (2Ψ / 3Ψ). Asked “why not just ZeRO-3?”: “every layer needs the weights; sharded, each layer must gather them first, the costliest communication. Next lecture.”
⛔ Don't call FSDP “a different approach.”It's roughly ZeRO-3.
⚠️ Asked “what about TP and PP?”: “they shard another dimension; next lecture.” Stop.
🖥 Main screen: Section 4 · The bill goes to Topic 5 · no figure (on purpose)
Section 5 · All three, on a machine at hand (6′)
🎯 Keep one thing: even on a small machine, activations outgrow the resident block. Recomputation isn't just for big models.
🖥 Screen: One step, all items. Whole, three seconds,
then zoom into Ⓐ. ⭐ First time all four items share one timeline.
All correct, but all 671 billion parameters and TiB. No reference point.
So: all four items on one timeline,
then the same story shrunk to a model you can work out by hand.
Two bands, opposite shapes: blue, activations piling up in forward; orange, gradients building in backward.
⭐ Their highs are over half a step apart.
So “most crowded when?” needs “crowded with what?”
🖥 Screen: The same bill, in motion.
Run it: forward piles up, backward frees as it goes.
⭐ The peak is an instant, not a phase. And that instant is what you pay for.
⭐⭐⭐ Main course: the lecture's one hand-workable cell. Try not to cut it.
🖥 Screen: scroll to 5.4; follow its three steps.
Now shrink it: a 125-million-parameter model,
top row of the GPT-3 paper's eight-row table. Twelve layers, width seven sixty-eight.
One device, batch eight, sequences of a thousand twenty-four.
First, the three things on this machine.
⭐ Loss: eight thousand-odd positions, one negative ln p each, averaged.
At init it's ln of the vocabulary size: Section 1's yardstick.
⭐ Derivatives: a hundred twenty-five million, for just two more forward passes.
⭐ Step size: Adam's two moving averages: eight of the twelve bytes.
The other four, fp32 master weights, aren't Adam's.
Apply the sixteen-byte bill.
Weights: just 0.23 GiB. Training: 1.86. Eight times.
The extra is all training-only.
⛔⛔ Next line is the section's whole point. Pause first.
Now activations. What each token leaves in one layer,
times twelve layers, times eight thousand-odd tokens: 2.11 GiB.
⛔ More than the resident block.
Small model, short sequences: batch eight is enough.
⭐⭐ So recomputation isn't just for big models.
Full recomputation: activations drop to 0.32, 6.7× less.
Same cost as the big model: compute goes from 3× to 4×, a third more.
⭐ This little table is the training loop on your laptop.
📌 Asked “did you count the logits?”: No, on purpose. With a ~50K vocabulary, all logits at once take 0.77 GiB, over twice the 0.32. Real code computes the loss in chunks, so we leave it out. An accounting choice; the logits still exist.
⭐ One more minute if time allows: answers the sure question “why do the savings differ?”
Both pay a third more, but save 6.7× here, 20.5× for V3. Three times apart. Why?
The saving depends on how much one layer holds, and how many layers.
V3 holds more per layer, thanks to MoE, with five times the layers. The small model loses on both.
⭐⭐ So only three things are truly scale-independent:
a third more compute, sixteen bytes per parameter, and activations that count only total tokens.
⛔ The saving factor isn't one. New architecture, new math.
⛔ Handoff (5.5) is collapsed. Just this half-line, then Section 6. ⭐ Always say it, or the room thinks class is over.
Don't pack up yet. All these bills assume the step actually completes.
⭐ Last section: what happens when it doesn't.
One step, all items panel Ⓑ closed too.
Asked when activations catch the resident block: about 62 sequences of 128K, or nearly 2,000 of 4K. Our own estimate; order of magnitude only.⛔ Don't write out the saving formula.“Per-layer load, layer count” suffices; it's in 5.4.
⛔⛔ Don't wrap up the lecture here.The four criteria end Section 6; closing here bows out early.
⚠️ Nearly every number here is computed live (script-asserted); only the model's shape is looked up (GPT-3 paper, Table 2.1). Say so.
🖥 Main screen: Section 5 · All three, on a machine at hand · main figure One step, all items
Section 6 · Not blowing up: when the loss spikes, and how to prevent it (9′)
🎯 Keep one: judge any stability fix by both “did it stabilize?” and “did quality drop?”
⭐⭐ This section is special: the only one on “something went wrong, now what?” Sections 1–5 did accounting, which can't answer the question asked most on a real run. (⛔ Fixed 2026-09-22: it said “the previous six sections”.)
Accounting done.
But on a real machine, what you hear most isn't “out of memory,” but:
“Uh oh. The loss just shot up.”
⭐⭐⭐ One minute naming them first, or the whole section is just “bad.” ⛔ New block (page 6.0). ⭐ Nothing new: it lines up five failure modes from five sections, never before named apart.
First, a roll call. This lecture has covered five ways training goes bad,
under five names. On a real run, the hard part isn't the fix. It's knowing which you have.
⭐ By what you see on screen:
One: the loss goes NaN. Exploding gradients.
Two: the early layers never move. Vanishing gradients.
⭐ These two are one chain of products, going two ways: factors above one on average, it grows exponentially; below one, it collapses.
So the fixes are one family too.
Three: the loss stalls, though the gradient isn't zero. Rounding eats the updates.
Four: the loss climbs, or bounces ever higher. The step is too large.
Five: a sudden spike in smooth training. That's today's spike.
⛔⛔ Always say the three-and-four line: the table's most valuable row.
⛔ Three and four both get called “it won't train,”
but need opposite fixes.
Three needs bigger steps, or more precision. Four needs smaller steps.
Mix them up: every fix backfires.
⭐ Each has a one-minute check.
Three: compare update RMS over parameter RMS to one over two fifty-six.
Four: halve the learning rate and rerun. Fine at once? That was it.
🖥 Screen: Loss spikes, Ⓐ only. Three seconds of silence.
⭐ Half the room has never seen a loss curve; “spike” alone shows them nothing.
The red curve on top is roughly PaLM's 540B training loss.
It heads down: the model improving. But look at these spikes.
About twenty.
The green curve below: same recipe, smaller model, not one spike.
⭐ Two points, both on the figure.
First, the spikes follow no pattern.
Some cluster, some far apart, and they still come late in training.
Getting past the start doesn't make you safe.
Second, the y-axis has no ticks, on purpose.
The paper describes the spikes but gives no numbers.
A curve without ticks is a schematic; a curve with ticks is data.
🖥 Screen: pan to Loss spikes, panel Ⓑ.
⭐⭐⭐ The hook of the section. Pause after it.
What really stings is the figure's small label:
gradient clipping was on.
Clipping caps one step's size: everyone's safeguard.
The safeguard was on, and still twenty blowups.
⭐ So it isn't simply “one step was too big.”
Then what is it? The paper says: we don't know.
Given the cost, they found no principled fix.
⛔ Say “we don't know” as is. Don't paper over it. ⭐ Otherwise they'll blame their own spikes on a wrong setting. Knowing it's open beats three tricks.
🖥 Screen: switch to the river figure, Not blowing up, and zoom into Ⓑ first.
⛔ Not Ⓐ: surprising experiment first, categories after.
⭐⭐⭐ The section's hook. Pause; let the room give its first instinct.
First instinct? That batch of data must be bad.
They tested it cleanly: fed the batches around the spike
again, from an earlier checkpoint.
… No spike.
⭐ So a spike isn't caused by bad data.
It's this batch meeting that parameter state. It takes both.
Remove either, and nothing happens.
That's why their crude-looking fix works:
roll back about a hundred steps before the spike, skip those two to five hundred batches, keep going.
No spike there again.
⛔ Be clear: that isn't fixed; it's worked around.
The paper admits no principled strategy.
🖥 Screen: pan to Ⓐ.
Now a framework for fixes. Three kinds, at three places.
Upstream, fix the structure, hard to blow up: norm inside the residual block, QK-norm, initialization.
Midstream, fix the numerics, capping what grows too large: z-loss, gradient clipping.
Downstream, salvage: roll back, skip data.
⭐ Upstream is nearer a cure; downstream is first aid. Go in order.
⭐⭐⭐ Go deep on one: the structural fix. It also settles 3.5's open question. ⛔ Moved here 2026-09-20 (round 20) from the section start, before “three kinds,” where “fix two” came before the count. Numbering is a promise: list first.
🖥 Screen: In Post-LN, the norm itself is a culprit.
⭐⭐ Parts first, then placement. No numbers here.
First take one layer apart, or the difference has nowhere to land.
A layer is two paths side by side.
The trunk: the input goes straight through, untouched.
The branch: the same input goes through attention or the MLP to compute a correction.
At the end they merge: Section 1's residual.
⭐ Normalization is the third part. It must sit somewhere on those paths.
The two placements differ only in where it goes.
The old way: merge first, then run the norm over the whole trunk.
The trunk is touched every layer.
The new way: the norm is tucked inside the branch.
Nobody touches the trunk; it runs straight to the top.
Hold that picture. You can't yet tell which is better.
Now the counterintuitive part: in the old placement, the norm itself is a culprit behind vanishing gradients.
⭐ Strange: we thought normalization was there to save us.
⭐ Mechanism in plain words. No derivation. ⛔ Say “variance one plus one is two” and the room follows arithmetic instead. Numbers are in 6.5.
In one sentence: the old way adds trunk and branch, then normalizes.
At initialization they are two unrelated random vectors.
Add them, and the energy doubles.
The norm pulls the energy back down.
⭐ So every layer squeezes the signal down a notch.
By the deep layers, only a trace of the input is left.
The red line: the residual “shortcut” is cut down exponentially,
hardest near the input.
⭐⭐ The pointed question. Three seconds.
So: if it's that bad, why not remove the norm?
Without it, every residual pushes the energy up a notch, until it's out of control.
⭐ So the question was never “whether to add a norm.” It's “where.”
The new way: normalize, then feed the branch; the trunk runs straight through.
Every residual branch now carries equal weight; the exponential decay is gone.
⭐⭐⭐ The high point; the rest was setup. Say it alone, pause. ⭐ On screen, both halves: identical curves, same axes.
The most counterintuitive part:
in a different setting, this “bug” becomes a feature.
What do we want when we finetune?
Adjust parameters near the output; barely move those near the input,
so we keep what pretraining learned.
“Vanishing gradients” is just that: the nearer the input, the less it moves the output.
⭐⭐ Exactly what finetuning wants.
So an old-placement model often finetunes better than a new-placement one.
Same curve: a bug in pretraining, a feature in finetuning.
⭐ Close Section 3's warmup question. Two sentences; no proof.
This also answers Section 3's open question: why warmup exists.
In the old placement, gradients near the output are large right at initialization.
A large learning rate then is bound to be unstable; warmup holds it back.
In the new placement, gradients behave from the start, so you can drop warmup.
⭐ So Section 3's three explanations don't compete.
One says why it was once indispensable, another why it's still around.
Different questions.
⭐ But don't stop at “you can drop it.”
V3 uses the new placement and still warms up.
The reason changed:
then to survive; today to allow a higher peak,
which speeds up the rest of training.
⭐ A one-minute story, same pattern as 6.1's scale effect.
Another structural fix, with a story.
At billions of parameters, vision Transformers diverged within a few thousand steps.
The cause: attention scores ran away;
after the softmax the weights collapsed onto almost one position.
⭐ It stopped “distributing attention” and stared at one spot.
The fix: normalize Q and K before the dot product,
the QK-norm many large models now use by default.
⚠️ Attribution: that paper didn't invent it; it says “we adopt the approach of” others.
We cite it because it applied it at twenty-two billion parameters, with a comparison.
⛔ Again: a failure seen only at scale.
⭐⭐ Last, two misconceptions. The first gets a figure with one real number line.
🖥 Screen: fp16's problem isn't “too coarse”; it's “can't go low”.
‘fp16 needs loss scaling, so bf16 must need it too.’ No.
⛔ But not because ‘sixteen bits is enough’: fp16 is sixteen bits too,
and finer than bf16.
⭐ Same number of slots, split differently.
fp16 spends more on fine resolution, bf16 on reaching very small values.
⭐⭐ fp16's real problem isn't coarseness. It can't go low enough.
A small enough gradient becomes exactly zero; that parameter gets no signal that step.
On this log axis: fp16's zero line is here, and bf16's floor is far to the left.
The green span between: everything bf16 can store and fp16 can't.
So what does loss scaling do? Simple: lifts values off the floor.
Before backprop, scale the loss up by a big factor, lifting values about to fall through.
Before the update, scale back down.
⭐ bf16's floor is too far to hit; no need.
⚠️ Its cost is elsewhere: it's coarse. bf16 fears accumulating tiny increments,
not falling to zero: Section 3's criterion again.
⭐⭐ The second: no figure, one minute. Without it, 6.2's rescue hangs in the air.
Second: is training just impossible to reproduce?
⭐ No. It can be, and it's worth it. PaLM achieved bitwise reproducibility:
restart from any checkpoint, and every later step matches the original, to the bit.
Two things. One: the compute framework is deterministic.
⭐ Two, easily missed: the data pipeline must be deterministic too.
The trick: the batch depends only on the step number,
not on device count or restarts.
⭐⭐⭐ Tie back to 6.2: the reason to cover this.
❓ Why does this matter?
Recall the rescue: on a blowup, roll back, skip those batches, keep going.
⭐⭐ That fix depends on exactly this.
With a nondeterministic pipeline you couldn't even say which batches to skip:
you wouldn't know what you just fed in.
⭐ The section's landing: the line to carry out.
🖥 Screen: pan to Ⓒ.
But the real difficulty here isn't “how to make it stable.”
Easy: a learning rate of zero never blows up.
The hard part is staying stable without losing quality.
Someone measured this cleanly:
run one configuration many times; record two columns:
how many runs finish, and how smart the model is when they do.
⭐⭐ Only the middle bar: the one thing to take from this panel. ⛔ Don't read the three numbers; not the point.
Most glaring is the middle bar:
one method finished every run. On stability it looks flawless.
⛔ But the model got a lot dumber.
⭐ On our yardstick, it went from hesitating between six words
to hesitating between dozens.
⭐⭐ So that bar comes down to one sentence:
“It never crashed” is not, by itself, evidence.
⭐ One minute of method, worth more than the result.
Also: they deliberately picked a configuration that crashes easily,
and data that makes trouble likelier.
❓ Why?
⭐ To study a rare failure, first make it common.
Small models rarely crash; large ones cost too much to rerun. How else?
⚠️ Not cheating. It's the right way.
But when reading the numbers: the crash rate was dialed up on purpose.
⭐ Last, one transferable line: also the lecture's fourth criterion, so don't stop; go straight into the close.
So remember this:
to judge any stability technique, check both columns: did it stabilize, and did quality drop?
Check only the first, and a learning rate of zero is optimal.
⭐⭐⭐ The lecture's real close. Read it, stop. Add nothing. ⛔ No more numbers. This block only sends the room home with four sentences.
We quoted a lot of numbers today. What I want you to keep is four criteria.
One: whether you need high precision comes down to two things:
how far this quantity moves per step, versus the smallest step the format resolves.
⭐ Under one step, you haven't moved.
Moving averages forget a little every step, so they can move.
Master weights build up over a hundred thousand steps, a few thousandths per step, so they can't.
Two: whether to recompute depends on how much compute you pay for each byte you save.
For a linear layer it's fixed: the input width.
For attention it grows with the sequence.
⭐ Two different slopes. So they have to cross.
Three: if two quantities grow at different rates in the same variable,
and the faster-growing one starts lower, their ranking will flip sooner or later.
⛔ You can't drop “starts lower.” If the faster one also starts higher,
the lines never cross.
⭐ So any “A is cheaper than B” must come with the range where it holds.
In this lecture it flipped several times, each in a different variable.
Four is the one I just gave: to judge any trick that stabilizes training, check both “did it stabilize?” and “did quality drop?”
⭐⭐⭐ Pause. One last line, then stop.
Numbers go stale. These four won't.
⛔ Don't derive the Pre-LN equations.⭐ Updated 2026-09-22: this used to say “skip the passage; add it if there's time.” That skipped the lecture's best turn (“same curve: a bug in pretraining, a feature in finetuning”). The one-minute formula-free version above is a must. Skip only the variance derivation (6.5).
⚠️ Don't slide past “Smaller: none seen.” It ties to 2.6's criterion: the third “don't copy across scales”.
⚠️ Nothing here is our own measurement; all from published papers. If pressed, say so; it's deliberate.
🖥 Main screen: Section 6 · Not blowing up: when the loss spikes, and how to prevent it · main figure Not blowing up
🚧 Where these notes are still thin
⭐ An honest record, so nobody mistakes this for finished.
- [ ] Never presented; every timing is an estimate. After the first run, enter actual times. ⛔ A stale table is worse than none.
- [ ] No warm-up or after-class questions. One fits at the end of Section 2: “Cut the sequence length to 4K: how does that ranking table change?” The answer is in the figure; the room must say it.
- [x] Section 3 is a dangerous length (planned 28 min).
✅ Handled 2026-09-20: the “conversion factor” shared one breathless twenty-minute block with “why descent works / saddle points / momentum”.
It now sits before the two figures
as its own fourth block, with a breather. This also matches the page's 3.3 / 3.3b
(order was reversed).
⛔ But the floor is still well over the planned 28′ (floor table above, recounted each build). The breather fixes pacing, not length. Expand, or cut as the build warns.
⚠️ 2026-09-20 added another 90 seconds (batch size: no notes before, and requested). The cut order was set first. Short of time, drop in this order: ① PaLM's β₂ conjecture (already marked “if time allows”) → ② the story of how Adam got its name → ③ the two quotes from Polyak 1964. ⛔ Don't cut the batch block. It was requested. - [ ] “6ND underestimates by five to six times” lacks third-party confirmation. It's derived from our 82.1%. Find a measurement and cite it; until then, say on stage it's derived.
📌 Settle one, cross it off, write the conclusion into its section. ⛔ Don't answer these on the course page; it shows only conclusions.