Three things: computing that number (the loss), turning it into 671B derivatives (backpropagation), sizing the step (gradient descent, optimizers). One chain, from one number to one update.
This lecture has instructor notes.
They cover how to teach it: goals, pauses, misconceptions, likely questions. Content: this page.
Presenting: lines, figure cues, timing: all written out. Self-study: give them to an AI for a TA that knows the material and how to teach it.
Left: four models. Right: four attention layouts (one cell = one layer, colored as in Topic 3's timeline table). Click a left item, then a right one to connect them; click the left one again to undo.
All matched. These four are today's four main routes:
Everything above saves at inference time. This lecture is the other half: in training, none of the four escapes the same bill.
Topic 1 followed one token forward to the model's guess. Here, the guess is wrong. Now what?
The error becomes one number. Not a table, not a vector: one number.
Training is one job: turn that one number into which way and how far each of 671B parameters moves.
✋ Someone always asks: how can one number decide 671B things? Too little information.
The answer is the lecture's thread: the 671B directions aren't read out of the number but computed from it.
It lives in the network's structure and every intermediate forward-pass value. The loss is the start; the chain rule expands it into 671B pieces. Hence Section 2's point that activations can't be discarded: to expand, keep what the expansion uses.
Three things, along that chain:
Three things, one chain. One number at one end, 671B nudges at the other.
⛔ What this lecture skips, so you don't wait for it.
⭐⭐⭐ Sections 2, 4, 5's detailed math and every collapsible block fall under these. A Topic 5 preview; skipping them costs none of the three.
Backprop and gradient descent are principles, unchanged for decades. Whether it fits is a per-generation hardware fact. Mixed, principles seem to expire too.
📐 Setting first, so each number has a known regime.
Unless stated, numbers use DeepSeek-V3's real pretraining config: sequence length 4,096, global batch ramping 3,072 → 15,360 sequences.
V3's advertised 128K is two 1,000-step extension stages after pretraining; the 14.8T tokens ran at 4K. Every 128K here is labeled context extension and converted to 4K. A number without its setting is meaningless: criterion ③ of this lecture.
Backprop isn't “running it again.” It costs two things: ~2× the forward compute and, worse, piled-up activations that can't be dropped early.
📌 First, pin down “activation.” It appears 60+ times here and has three common meanings.
“Activation” alone means the intermediate results the forward pass leaves for the backward pass.
First, the starting point: the chain runs backward from one number, the loss, which deserves two minutes.
Everything here aims to shrink one number. Start with it.
The model reads the text so far and guesses the next token, giving every vocabulary token a probability. We reveal the answer and check only its probability.
📌 Softmax turns scores into probabilities: exponentiate (all positive), divide by the sum (adds to 1). Every later use means this, as does “weights sum to 1” in Topic 3's attention figure.
loss = −ln p (p = the model's probability for the correct token)
First, where logits come from, or the figure's “row of scores” won't explain why there are exactly 129,280.
Vocab 129,280, residual width 7,168: DeepSeek-V3 config. LM head parameter count and 0.99 GiB: computed and asserted by this figure's script.
⚠️ The two blocks are not to scale (the real 18× ratio won't fit); layer internals: see Topic 1.
Where does −ln p get its p? Two steps.
The last layer outputs a row of raw scores, the logits. Any sign, any range; not probabilities yet.
⭐ Full cross-entropy is a sum over the whole vocabulary; a one-hot label (a single 1) leaves just one term nonzero. So “looking at only one” follows from one-hot labels; it isn't the definition.
⚠️ Converse check: with non-one-hot labels (label smoothing, distillation soft labels), every term appears in the loss. Numbers computed and asserted by the script; fp32 max = 3.4028e+38.
✋ First, what that 0.6964 means: it is not the loss on a training curve.
Same −ln p, but shrunk twice to fit the figure:
a 5-token vocabulary and a single position.
Compare it with random guessing over the same vocabulary:
for 5 tokens, −ln(1/5) = 1.61.
So 0.6964 says only “a bit better than guessing,”
not “already well trained.”
A training-log loss = the real 129,280-token vocabulary +
an average over all positions in a batch.
Its random-guess start, −ln(1/129,280) = 11.77, is
the 1.0 ruler's top and step 0 of the curve.
⭐ A common trap: half of the loss's size comes from the vocabulary, not model quality. A bigger vocabulary raises the same model's loss, so losses across different vocabularies aren't directly comparable (use a tokenizer-independent measure like bits per byte).
Two properties: elegant and life-saving.
① Adding the same number to every logit changes no probability. Only differences matter; exponentiation turns differences into ratios.
② Never exponentiate directly: it overflows.
Real logits hit tens or hundreds; e¹⁰² far exceeds fp32's max.
So every implementation first subtracts the row max:
by shift invariance, mathematically identical, numerically safe.
6.4's z-loss is the flip side: this defuses overflow after the fact; z-loss keeps scores small up front.
✋ Now the class question: “Why don't the wrong tokens enter the loss?”
They do, just not under their own names: they sit in the softmax denominator.
Correct answer's probability = its exponential ÷ the sum of all entries,
a sum over every vocabulary token.
⇒ Panel Ⓒ proves it: raise one “wrong” score by 1,
correct entry untouched, and loss goes 0.6964 → 0.9702.
In the gradient, all show up. Each wrong token gets +its probability: the more it holds, the harder it's pushed down. Row sum: exactly 0.
⭐ In one line: invisible in the loss formula, visible in the gradient,
and training uses only the gradient.
📌 Deeper: full cross-entropy sums over the vocabulary;
a one-hot label leaves one nonzero term.
So “looking at only one” follows from one-hot; it isn't the definition.
With soft labels (label smoothing, distillation),
every term appears in the loss.
Why this formula: it makes sense.
ln 2.d(−ln p)/dp = −1/p. The lower the probability, the larger the gradient,
so the worst mistakes get fixed first.
A draft said “otherwise it keeps nudging what it's sure of.” That doesn't follow: linear 1−p
has constant slope −1, treating “sure” and “clueless” identically.
Backward.📌 Formally: cross-entropy, or negative log-likelihood. Three names, one formula.
Drop the log for now. The model gave the correct answer 0.4. What penalty should it get?
Most natural: the reciprocal.
1 ÷ 0.4 = 2.5
2.5 means the model is as good as guessing blindly among 2.5 tokens. (With 2.5 equally likely options, each would get exactly 0.4.)
That reciprocal is perplexity: “1 ÷ your probability for the answer.” Nothing mysterious.
| p for the answer | Perplexity = 1/p | Reads as |
|---|---|---|
| 0.8 | 1.25 | Almost certain |
| 0.4 | 2.5 | Like guessing among 2–3 tokens |
| 0.1 | 10 | Like guessing among 10 tokens |
| 0.01 | 100 | Badly confused |
⛔ So why the −ln?
Purely so thousands of positions can be summed.
A sequence has 4,096 positions, each with a probability, and probabilities multiply.
0.4 to the 4,096th power is 0 on a computer: it underflows.
The logarithm fixes this: ln(a×b) = ln a + ln b.
It turns multiplication into addition: add, average, never underflow.
Expand ln(1/p) and you get −ln p. That is the whole story.
To read it, exponentiate back:
eloss = perplexity = the geometric mean of all the 1/p.
📐 Check it: three positions get 0.4 / 0.5 / 0.2.
Average ln(1/p), exponentiate: 2.92;
the geometric mean of 1/p: also 2.92. Same number, two routes.
❓ An aside people get stuck on: why is log 1 = 0?
A log asks: from 1, how many doublings reach this number?
Double each step: 1 → 2 → 4 → 8. “The log of 8 is 3”: three steps.
How many steps from 1 to 1? Zero; you're already there.
That is log 1 = 0.
One step back divides by 2, to 0.5, so log 0.5 = −1.
Greater than 1 is positive, equal to 1 is zero, less than 1 is negative.
So the loss formula holds up at both ends:
p = 1 → perplexity 1 → no steps from 1 to 1
→ loss = 0. Zero penalty for a perfect answer, not rigged: log 1 = 0 gives it.Perplexity 100 → 10,000 is 100×; the loss 4.61 → 9.21 only doubles. The logarithm compresses “how many times bigger” into “how many steps.”
Each of 129,280 tokens has a probability. The loss uses one: the correct answer's.
The answer is “cat”; the model gives “cat” 0.4, so this position's loss is −ln 0.4.
“dog,” “table,” “the”: all ignored.
Other 129,279 wasted? No: they vie for probability with the answer.
Probabilities always sum to 1. Zero-sum: others up, this one down. Absent from the formula, they decide how far the numerator's p is squeezed.
Precisely: don't mix the two levels (I once got this wrong in class).
Probabilities have no “denominator”; they always sum to 1 (zero-sum).
Softmax's internal denominator Σ exp(raw score)
varies, but at the score level.
“The denominator grew” implies the total changes. It doesn't.
Measured: one score 1.0 → 3.5; total 1.000000 → 1.000000;
the correct token squeezed 0.629 → 0.175.
Is “only one” a rule, or a consequence? A consequence.
The full formula doesn't pick one probability; it compares two distributions: the model's column vs. the reference.
The reference answer: 1 for the correct token, 0 for the other 129,279. This “a single 1, all else 0” encoding is called one-hot.
Expanded (a four-token toy example; the model gives cat 0.4, dog 0.35, table 0.15, the 0.10):
| Token | Ref. q | Model p | Term = −q × ln p |
|---|---|---|---|
| cat (right) | 1 | 0.40 | 0.9163 |
| dog | 0 | 0.35 | 0: × 0, gone |
| table | 0 | 0.15 | 0: × 0, gone |
| the | 0 | 0.10 | 0: × 0, gone |
| Total | 0.9163 = −ln 0.4 | ||
So we don't “only look at one”: the other 129,279 terms are multiplied by 0 and vanish.
Hence “cross”-entropy: two distributions compared. Entropy compares one with itself. These are the two I mixed up in the confession below.
📌 People write the reference answer, so it needn't be 100%.
Pretraining almost always uses hard one-hot labels, so here “treat it as 100%” is accurate.
Worth more than the definition: a model can be confidently wrong.
Say a wrong token gets 80%, the correct one 1%.
The perplexity at this position = 1 ÷ 0.01 = 100. Yet it doesn't hesitate: 80% on one token. Decisively wrong.
So perplexity measures accuracy, not hesitation.
Confession: I taught this backward in class. I called perplexity the spread (entropy): another quantity. One distribution, “one token 90%, nine sharing 10%”: same entropy; loss 0.11 if the 90% token is right, else 4.50: 81× in perplexity. Entropy measures confidence; loss, correctness. Low entropy, high loss: possible.
| loss nats/token | In bits = nat × 1.4427 | Perplexity = eloss | Rough level | “Torn among N tokens” |
|---|---|---|---|---|
| 11.77 computed | 16.98 | 129,280 | Fresh init, learned nothing | Uniform over vocab: the loss ceiling |
| 6.0 ballpark | 8.66 | 403 | Token frequencies | A few hundred |
| 4.0 ballpark | 5.77 | 55 | Grammar mostly right | Dozens |
| 3.0 ballpark | 4.33 | 20 | Plausible text | ~20 |
| 2.0 ballpark | 2.89 | 7 | Pretrained LLMs today, roughly | 7–8 |
| 1.0 ballpark | 1.44 | 3 | Close to memorizing | 2–3: past normal pretraining |
⛔ The top row isn't a made-up value.
A freshly initialized model gives every token the same probability,
p = 1 / 129,280. Plug it in:
loss = −ln(1 / 129,280) = ln 129,280 = 11.77
Its perplexity is exactly 129,280, the whole vocabulary: the definition of random guessing.
So the ruler's top is computed, not estimated, and moves with vocabulary size (129,280 checked in Topic 1). The bottom has no official V3 number: “around 2.0” is a rough public-model ballpark, a reference, not any model's measurement.
Column two converts nats to bits (÷ ln 2). The conversion is trivial; the reading matters.
Bits count the yes/no questions needed to pick this token from the candidates. Each halves them: log₂ of the candidate count.
At init: the whole vocabulary, 16.98 questions. Late in training: 2.89.
The candidates shrink because of the context, not because the token got easier. Compressing 129,280 to an effective 7 is the forward pass's work; the 14.09 bits saved are that 17,500×.
⛔ Precisely: “an effective 7” doesn't mean 7 candidates remain; the vocabulary stays 129,280 and probability concentrates. It means: as hard as 7 equally likely candidates.
No measured values here: perplexity = e^loss, bit = log₂(perplexity) = loss ÷ ln2; vocabulary 129,280 from the DeepSeek-V3 config.
⚠️ “Late-training loss around 2.0”: rough ballpark for public models, not one model's measurement.
⭐ Both panels were Ⓑ and Ⓒ of the last figure, split out on 2026-09-23 so the course page can fold them.
No measured values here: perplexity = e^loss, bit = log₂(perplexity); vocabulary 129,280 from the DeepSeek-V3 config.
Mental math: memorize just two numbers.
ln 2 ≈ 0.693 (think 0.7) · ln 10 ≈ 2.303 (think 2.3)
Loss falls from 3.0 to 2.3: a drop of 0.7, so perplexity halved. No calculator needed.
Check the ruler yourself:
from 11.77 down to 2.0 is a drop of 9.77; divide by 0.7 = 14 halvings;
2¹⁴ = 16,384×.
Exact: perplexity 129,280 → 7.39, a ratio of 17,496.
16,384 vs. 17,496: under 7% off; the magnitude is right.
Only the order of magnitude. Mid-run you never need two decimals; you need “is this normal?”
Why does it “get more expensive later”? The phrase recurs; settle it once here.
⛔ First, a smooth-sounding wrong explanation: “the log scale squashes the later part.” No. In ratio terms nothing changes: each 0.7 drop always halves perplexity, at 11 or at 2.
Two real reasons.
| Loss drops 1 | Perplexity | Candidates cut |
|---|---|---|
| 11.77 → 10.77 | 129,314 → 47,572 | 81,742 |
| 6 → 5 | 403 → 148 | 255 |
| 3 → 2 | 20.1 → 7.4 | 12.7 |
| 2 → 1 | 7.4 → 2.7 | 4.7 |
① The absolute gain shrinks. A “drop of 1” cuts 80,000+ candidates early, near 2 fewer than 5, since few were left.
② The root: part of what's left is inherent in language.
The next token really can be several things; no one, humans included, can call it.
Late on you're not learning but hitting a ceiling:
that entropy is irreducible, never 0.
📌 Hence two models' losses compare only on the same dataset and tokenizer. Change the data and the ceiling changes.
This ruler returns in 3.5 (is the loss falling fast enough?), 6.1 (how big a jump is a spike?) and 6.3 (reading that table). Without it, all three are gut feeling.
The ruler used two terms, nat and perplexity. With bits, they are three readings of the same quantity.
⭐ The “Just initialized · random” level (Ⓐ, far right) is a self-check: perplexity must equal the vocab size 129,280; bits, log₂(vocab). Script-asserted; a mismatch means a wrong conversion.
⚠️ “Early–mid training 7.00”: illustrative level; “near convergence 2.00”: rough ballpark for public models. References, not any one model's measurements.
Nats and bits first: they differ only in the base.
A bit is the information in one binary choice (a coin flip's result); base 2.
A nat is the same with base e.
The conversion is fixed: 1 nat = 1.4427 bits.
Why nats here?
Only because the loss is defined with ln.
With −log₂ p, everything would be identical,
every number times 1.4427. A convention, not a deep choice.
Per token is from Step 3 below: each of a sequence's thousands of positions has a loss, averaged at the end. Full unit: “nats per token, on average.”
⛔ In bits, the number becomes tangible: a compression rate.
“Average bits the model needs to encode the next token.” A language model is fundamentally a compressor; the loss is its compression bill.
ln 129,280 = 11.77 nats = 16.98 bits.
log₂ 129,280 is also exactly 16.98:
picking one of 129,000 tokens takes 17 bits.
Self-consistent, not a value someone typed in.2.0 nats = 2.89 bits.From 17 bits to under 3: everything this model learned, as compression.
📌 Three faces; pick whichever fits:
They're identities, not three measurements: change one and the others follow. Don't cite them as three proofs.
So far, the loss at one position. In a 4K sequence every position predicts its next token, so one sequence has 4,095 losses.
⛔ It is 4,095, not 4,096: the last token has no “next” to predict. The 1 matters only as the denominator for the accounting in 1.2.
Average them all and you get this batch's loss.
MaxText training script: loss = xent_sum ÷ total_weights, total_weights = target positions scored after masking padding and packing seams;
sequence length 4,096 and max global batch 15,360 come from the DeepSeek-V3 pretraining config.
“Average” is the root of much that follows.
The 1.2 figure's analogy: a room of people, each with a demand, averaged. The people: these tokens.
Batches exist first to avoid heeding one person, not to keep devices busy. This runs to Section 6 (not blowing up); math in 3.5b.
Hundreds of billions of parameters in, one number out. No vector, no table: one number.
Remember this. 1.5 shows backprop is cheap entirely thanks to this “one”.
📐 Unit: nats/token (natural log). Section 3.3b uses it for the learning rate's units.
Almost everyone asks this once: “Why do you keep saying the loss is a single number? What I see is 129,280 numbers per token, 4,095 per sequence, and even more once you multiply by the batch.”
Everything you listed exists; it just isn't all called the loss. Two compressions sit in between.
| Level | Count | Called |
|---|---|---|
| Token scores, one position | 129,280 | logits / probabilities: not the loss |
| ↓ First compression: keep only the correct answer's entry, take the negative log | ||
| One position's loss | 1 | −ln p |
| One sequence | 4,095 | One per position |
| One step (V3's global batch) | 62,899,200 (= 4,095 × 15,360 sequences) | Still one per position |
| ↓ Second compression: sum everything, divide by the number of valid tokens | ||
| Final loss | 1 | The point on your loss curve |
Why compress it to one number? Not for looks.
“One number out” is what makes “one backward pass” possible.
Per 1.5, the seed goes on the right: one number, so you plant it once.
With 4,095 objectives you'd plant 4,095 seeds,
one backward pass each. That is why multi-objective optimization costs more.
✋ Finally, an apparent contradiction:
“The loss is one number” and “the backward seed has over half a billion”: both true.
The loss as a quantity really is a scalar.
Differentiate it: one partial derivative per logit:
every position × the whole vocabulary, ~530 million numbers (4,095 × 129,280; drawn in 1.2).
⭐ “How many numbers in the loss?” asks about the quantity; “in the seed?” about its derivative. One is 1; the other, over half a billion.
The lecture's job, in one sentence.
Move this number from 11.77 to ≈ 2.
That means knowing which way to nudge each parameter: this number's partial derivative for each of 671B parameters.
The problem: how to compute 671B derivatives?
Derivatives are high-school math. The hard part: 671B of them, fast, redone every step.
“High-school math” will lose some readers, so one figure covers the lecture's three words. No limits, no formulas: three sentences.
⚠️ The 4.00 → 4.03 in Ⓐ and the ×2 / ×0.5 / ×3 in Ⓒ are made-up illustrative numbers. They just make ÷ and × visible. ⭐ The script asserts total rate = product of the three, and at least one stage shrinks; else “multiply them” reads as “it only grows,” against the vanishing/exploding-gradient section.
⭐ This figure deliberately avoids limits: strictly, a derivative is the nudge → 0 limit; this is a difference quotient. ⛔ Fine here: §1.1's brute force is exactly a difference quotient, so loose fits better.
⭐⭐ “Exchange rate” isn't decoration: two hooks, both ours: in a chain of products, the starting end shifts cost by 20,000 years (§1.5); mismatched units need one more conversion factor (§3.3).
⭐ Originally the previous figure's fourth panel; split out on 2026-09-23 so the course page could fold it away.
⚠️ 7,168 is V3's residual stream width. ⛔ The machine holds no square matrix; no real weight is equally wide on both sides.
A common mix-up: not deviation, but derivative. Deviation is departure from a reference (standard deviation), unrelated here. The term is partial derivative.
Derive is a picture: Latin dērīvāre = de (from) + rivus (stream): literally “to draw water off a stream”.
📌 Source: Etymonline (derive / derivative): Latin derivare “to lead or draw off (a stream of water) from its source,” from de rivo (de “from” + rivus “stream”); the math sense dates from the 1670s.
To derive is to draw something off. Today it suggests deduction, and the water picture is lost.
With that picture, both terms follow.
Derivative first.
A stream: input in, output out. A derivative asks not “what is it now?” but “move one notch, how far does the output move?”
Not a new stream: a second one drawn off this one. The first gives the output; the second, change per notch. One source, two streams.
Then the partial derivative (not deviation).
Now the input is 671B knobs. “Move the input one notch” breaks down: which knob?
The fix is blunt: free one knob, clamp the rest. Partial means “in part”: one variable at a time.
671B dimensions collapse to one: a partial derivative is an ordinary derivative again. Not harder: it cuts the hard problem into 671B ordinary ones.
First, brute force. Without it, backpropagation looks unremarkable.
⛔ Brute force (finite differences): nudge one parameter, rerun the whole forward pass, and see how far the loss moved.
671B parameters: 671B forward passes.
671e9 s ÷ 3.156e7 s/yr = 21,261 years
(1 s per forward pass, already wildly optimistic)
That's one step of hundreds of thousands.
Backpropagation's real job isn't differentiating; it's collapsing 671B forward passes into one.
It starts from §1.0's loss. (Just one; that's the key to what follows.)
The first gradient? Take cross-entropy, the most common loss. The network outputs a distribution (next token “the” 0.3, “a” 0.2…); the answer is 1 in one slot, 0 elsewhere.
First gradient = guess − correct answer.
Too high: positive; too low: negative. It seeds the whole backward chain.
No coincidence: it is §1.0's −ln p differentiated once.
Differentiate with respect to the logits (not probabilities)
and −ln p is exactly p − y: the model's probabilities minus the one-hot answer.
Back to §1.0:
one −ln p, two derivatives, two results.
−1/p;
lower probability, larger gradient. That's the motive: fix the worst mistakes first.p − y.
That's the implementation: the seed is one subtraction.One says “why move,” the other “how”: two faces of one loss.
So frameworks always fuse the two into one op: one step, not probabilities then derivative. Few losses are this pretty; this pair is today's default.
Precisely:
that “subtraction” is the derivative with respect to the logits, not the probabilities
(the pre-softmax output).
It matters: as the softmax step cancels,
the gradient doesn't vanish where softmax saturates, half the reason softmax and cross-entropy always pair up.
📌 Four ideas are from the official lesson notes for 3Blue1Brown's “What is backpropagation really doing?”: nudge ∝ how far off; three ways to brighten a neuron; weights scaled by upstream brightness for max return; and averaging, as you can't please every example (heed only a 2: all become 2s). Each was checked against the original; figures are our own redraws, recast for next-token prediction.
⚠️ On scaling with upstream brightness, he mentions Hebbian theory in passing (neurons that fire together wire together), but the original itself says the analogy is loose (an untrained network isn't “thinking” of that answer), so it is noted, not drawn.
⭐ The links Ⓑ→§1.8 (and folded figure→§1.6) are this lecture's own synthesis, not the original's. 3B1B explains “what backprop does”; we need “why it costs so much memory.”
📌 “Adjust weights in proportion to upstream brightness for the biggest return” is from the official 3Blue1Brown notes for “What is backpropagation really doing?” ("adjust the weights in proportion to their associated activations"). Figure: our own redraw.
⚠️ He briefly mentions Hebbian theory (neurons that fire together wire together), but the original itself says the analogy is loose, so it's noted, not drawn.
⭐ The step to “so activations can't be discarded” is our own synthesis: 3B1B explains “what backprop does”; we need “why so much memory.”
✋ A revealing class question: does backward push the right one up and the others down, or just the right one, the rest a side effect?
Both, and it's one and the same move.
The seed, “guess − answer,” is a whole row,
one slot per vocabulary token:
the correct slot: 0.40 − 1 = −0.60 (negative = push up);
every other slot: its own probability − 0 (positive = push down).
The 0.20 token gets exactly twice the push of the 0.10 one.
⭐ The row sums to exactly 0. Probability is conserved, so up and down are one redistribution. The other 120,000-plus tokens sit at probability ≈ 0, gradient ≈ 0: no share, no move.
⛔ Two conventions (the figure's “example” can mislead):
① 4,096 tokens give only 4,095 predictions: the last has no “next”;
② loss averages per token, not per sequence:
the denominator: target positions actually scored, padding and packing boundaries masked
(MaxText: total_weights; loss = xent_sum ÷ total_weights).
MaxText training script: loss = xent_sum ÷ total_weights (target positions actually scored, after masking padding and packing boundaries).
Gradient computed once (value_and_grad on that scalar). Vocab 129,280, seq length 4,096: DeepSeek-V3 pretraining config.
Ⓑ's arrows are easy to misread: a position's one arrow back is not “which way it pushes.”
Every vocabulary row pulls on a 7,168-dim vector; all 129,280 summed make that arrow: a net force, not a direction.
“Push up / down” applies only to one vocabulary slot (level Ⓐ); at “one position” it refers to nothing. So Ⓑ's arrows are neutral, not green or red.
⭐ How that net force is summed is all of §1.2c.
And “one backward pass per token, then average the 671B gradients”? Exactly the same result.
Differentiation is linear: averaging and differentiating commute. So implementations run backward once: not a shortcut, but equivalent and tens of millions of times cheaper.
⭐ The scalar is only a handle; backward doesn't start from it. It starts from a “position × full vocabulary” tensor, each entry “guess − answer” over the total. None of the 60-odd million positions' votes is erased at the loss; they add up later, in each layer's matmuls.
⛔ A natural misconception that inverts the whole chain: different positions' 7,168-dim vectors can't be summed first.
The temptation: 4,095 positions, each a 7,168-long vector, so average them and pass the average back. Do that and the whole gradient is wrong.
Position i's blame pairs only with position i's input. The weight gradient multiplies each row by its own partner, then sums; summing the blame δ first pairs position 3's blame with position 900's input, a different quantity.
Summation does happen, on the weight gradient, not δ, once per layer. δ stays a “position × width” table; the position dimension survives to the end.
Linear: weight gradient = Σ_i (δ_i ⊗ x_i), contracted over positions; input gradients aren't. ⚠️ One honest exception: attention mixes δ across positions, which already see each other going forward.
That's attention's structure, not “averaging”; in linear layers, positions stay strictly independent. Vocab 129,280, width 7,168, 61 layers: DeepSeek-V3 config.
⭐ Deeper: when does “settling up” happen? Asked in class: “Per function, or after the whole layer?”
Per function: each weighted op settles alone, independent of passing back.
Given δ, a weighted op does two things:
① Settle up: δ × this layer's input, summed over positions, updates this weight. Positions are consumed; nothing passes back.
② Pass back: δ × weight = the earlier layer's blame. Positions survive.
⭐⭐ These are 1.6's “two backward matmuls” behind training's 3× cost. One δ table, two contractions: one eats positions, one keeps.
⛔ Two common mistakes: ① The weight-gradient bucket is fixed-size: 4,000 positions just add 4,000 entries. Hence “one number per parameter,” not “one parameter set per token.”
② Layers add, never divide. The “÷ total” happened once, in the seed; dividing in all 61 layers would do it 61 times. Strictly a sum, not an average: the averaging is already in every entry.
As the class reasoned, a linear layer's weight gradient uses only same-position input; earlier tokens entered it via forward attention, not in backward.
⭐ The class got this right: “my token must use the activation that produced it, i.e., after all earlier tokens took part.” A linear layer's weight gradient uses only its own position's input,
and earlier tokens were already mixed into that input by forward attention, not pulled in by backward. ⛔ Scope: no V3 layer has a 7,168 × 7,168 square matrix. The residual stream is 7,168 wide; every matrix's other side differs (experts 7,168 → 2,048; MLA compresses KV to 512; o_proj's is 16,384). Example: expert up-projection. 61 layers total.
⭐ Originally the previous figure's fourth panel; split out on 2026-09-23 so the course page could fold it away.
How many levels does it have? Class guess: within a sequence, across local sequences, across micro-batches, across devices.
Layered, yes: three levels of addition + one division.
⛔ Why not divide per level? That's an average of averages.
Valid-token counts differ (padding, packing boundaries, the last batch). Divide separately and a small batch counts as much as a big one, with no error, no divergence, a normal loss curve.
Instead carry “loss sum” and “token count”; divide once at the end.
Source: MaxText gradient accumulation. In the loop, acc["loss"] += xent_sum and acc["total_weights"] += total_weights; after it, grads ÷ acc["total_weights"].
⚠️ One exception: auxiliary losses (MoE load balancing etc.) are instead divided by the micro-batch count, since each is already “one average per micro-batch.” Two conventions; don't mix.
⭐ Originally the previous figure's fourth panel; split out on 2026-09-23 so the course page could fold it away.
Many picture one gradient value passed down. It isn't.
What flows back is a whole tensor, shaped exactly like this layer's output: each output position's share of the loss's blame. We call it δ.
⭐ Name δ, or the rest is jargon: δ is the activations' partial derivative.
δ is the loss's partial derivative with respect to this layer's output, the activation. So backward must differentiate by activations: layer k's weight gradient needs the loss's sensitivity to layer k's output, which only layer k+1 can send. δ is that chain.
But differentiating it isn't changing it, an easy confusion:
| Take grad? | Update? | Then | |
|---|---|---|---|
| Weights | Yes | Yes | Sum across devices (÷ once, at end), to optimizer, 2 B/param |
| Activations | Yes (= δ) | No | Passed back, freed after use, never billed |
⭐ Activations aren't changed: they're not free variables. Inputs and weights determine them; only new weights change them, so optimizer state to “turn” them is pointless.
❓ δ is a full tensor too; why isn't it on 1.7's bill? It exists only at backward's current layer, freed after use. Activations live from forward to backward: 1.6's mountain. ⭐ Same size, one billed, one not: only lifetime differs.
⛔ A natural, misleading phrase: “activations are just a medium passing data along.” In forward, yes. But a mere medium would be dropped after forward, and they can't be: each layer pulls them out as input for its weight gradient. Better: forward passes activations; backward passes δ, and activations get pulled out to settle up.
Given δ from the next layer, a layer does two things (numbered as in the figure below; ① is forward):
One matmul each. One forward, two backward: hence 3× compute.
Counting isn't enough: unequal sizes would break 1 + 2 = 3. It holds because all three share one set of dimensions. For [B,in] × [in,out]: ① forward X·W, ② Xᵀ·dY, ③ dY·Wᵀ. Each is B × in × out, the same volume: exactly 3×. (Drawn to scale.)
The reason transfers: the last layer passes nothing back: drop ③, 2×. Frozen layers skip weight updates: drop ②, also 2× (3.8's LoRA). Memorizing “3×” answers neither.
⚠️ “3×” is the usual matmuls-only approximation, ignoring norm, activation functions, biases, communication. Their real share is small but not zero.
⛔ Step ②'s “input saved in forward” is the hinge of the whole topic: undiscardable activations and Section 2's recompute trade-off hang on it.
The figure above has two ᵀs: Xᵀ and Wᵀ. Unpack it first, or the rest won't make sense.
① What it is: rows ↔ columns, no number changes
A 2-row, 3-column table becomes 3 rows and 2 columns after a transpose: the first row becomes the first column. No number is lost or changed; only the reading direction changes.
So a transpose costs no compute: implementations often move nothing, just read it differently.
② Why it's unavoidable: the shapes force it
Matrix multiplication has one hard rule: columns on the left must equal rows on the right. The multiply consumes the matching dimension.
Backward's two results have fixed shapes: dW must match W (so the optimizer can add it), dX must match X (so the previous layer can take it).
| Arrangement | Consumed dim. | Why forced | |
|---|---|---|---|
| ① Forward | X[T×d_in] · W[d_in×d_out] = Y[T×d_out] |
d_in | Original: mix the input by the weights |
| ② Weight gradient | Xᵀ[d_in×T] · dY[T×d_out] = dW[d_in×d_out] |
T (positions) | Only way to get d_in×d_out |
| ③ Pass back | dY[T×d_out] · Wᵀ[d_out×d_in] = dX[T×d_in] |
d_out | Only way to get T×d_in |
With the result shapes fixed, a transpose is the only fit. It isn't an extra operation; it's the answer to “how do these have to line up?”
Two sentences, in plain words
Forward asks “how is the input mixed into the output?” ③ reverses it: “how is the output's blame split back to the input?” Same W, read backward: Wᵀ.
In ②, Xᵀ does something else: it moves the position dimension to the middle so it gets consumed.
Every position's opinion of this weight is summed here: exactly 1.2d's “settling up.” The position dimension vanishes, so this result isn't passed back.
⭐ Originally the previous figure's Ⓑ; split out on 2026-09-23 so the course page could fold it away.
⭐ The orange line (asked in class): ②'s Xᵀ isn't recomputed; it's ①'s forward input, saved. That line fully answers “why activations can't be discarded”: backward needs X for dW. The arrow follows the flow: ① saved → ② used.
📌 The “circuit diagram + three gates (distributor / swapper / router)” framing is from CS231n's backprop notes; figure and numbers are ours.
⭐ Every number is checkable: the build script asserts them (a and b both have gradient 4, c has −1) and fails on mismatch.
The class got it right: in training, earlier guesses never affect this position.
Each position's input is always the ground truth, not the model's last guess. This is teacher forcing.
⛔ But it isn't “looking things up one token at a time.”
Training computes Q / K / V and attention in one shot, all positions at once.
It's the biggest shape difference from inference, which alone goes token by token, each waiting for the last.
Strictly, “the K and V before me” is “before me and me”: the causal mask shows a position itself, not later ones.
Sources (looked up on 2026-09-22, not from memory): the term teacher forcing was introduced by Williams & Zipser (1989) for recurrent networks; the 1989 original reads “We call this technique teacher forcing” (the version adding “intuitively sensible” is from the pair's 1995 review chapter).
scheduled sampling (mixing in the model's own predictions at random, against exposure bias) is from Bengio et al., arXiv 1506.03099 (2015). ⚠️ Today's large models still use essentially pure teacher forcing: the parallelism is worth the exposure bias.
✋ So what's the cost? Exposure bias.
Training never shows the model its own mistakes; context is always right.
At inference it's all self-generated: one mistake and it's off its training distribution.
Hence long generations drift ever further. The model didn't get dumber; it's in untrained territory.
Why do it? ① Parallelism: self-fed, position t waits for t−1; 4,096 tokens means 4,000+ serial steps, impossibly slow. ② No early divergence: early guesses are awful; feeding them back blows up in one step.
A known cost that pays off.
📌 Sources (looked up): teacher forcing, coined by Williams & Zipser (1989) for RNNs; scheduled sampling (randomly feeding in the model's own predictions, against exposure bias), Bengio et al., arXiv 1506.03099 (2015). ⚠️ Today's large models still use near-pure teacher forcing.
Because partials are local.
⛔ ❓ Most people object at once—and rightly.
“A weight reaches the loss through a whole matrix, not a scalar, and its effect depends on its layer's other parameters, which interact. Everything pulls on everything: how can that be computed?”
Both hold, and it's slightly worse:
📌 Etymology: Etymonline (derive / derivative)—Latin derivare “to lead or draw off (a stream of water) from its source,” from de rivo (de “from” + rivus “stream”); the mathematical sense dates from the 1670s. ⛔ A common mix-up: deviation means a departure (standard deviation), a different word from derivative.
⚠️ Ⓒ's fan (left) is schematic: 3 hops × 6 nodes per layer (216 paths). Only the “232-digit number” (top right) uses the real model: 60 layers, 7,168 wide.
⭐ Side by side, backprop's saving is plain: paths grow exponentially; the backward pass does only 122 matmuls (two per layer), linear in depth.
The magic: the whole tangle is compressed into two numbers.
A weight's gradient needs just two things (matmul ② of 1.3's backward pair):
❓ Where did the cross-coupling go? Into the second number.
Not ignored: computed layer-wide at once, with the blame. The δ passed back already folds in everything downstream, cross-mixing too.
The complexity is real—but paid once per layer, not per parameter pair.
The price leads into 3.3b.
Pairwise interactions are second derivatives, one per pair; at 671B parameters, unstorable. So we deliberately skip them and pay with small steps.
Nobody invented the learning rate: giving up second derivatives forced it on us.
An op's backward rule needs only “how I turn input into output,” not its neighbors or the network's shape.
Hence autodiff as a general-purpose library.
One backward for matmul, one for softmax, one for addition; the framework just chains them in reverse.
Nobody hand-derives the whole network.
The alternative is hopeless: a closed form for “loss w.r.t. a first-layer weight” expands through 61 layers into astronomically many terms.
The chain rule: never expand it. Just multiply 61 small local derivatives in order.
The two matmuls of 1.3 and the chain rule's “multiply” are the same thing.
In 1.2d, matmul ③ on the incoming δ is “multiply δ by my own weights,” passed on.
That multiply is a chain-rule link; “my own weights” are the local derivative.
So “61 local derivatives multiplied in order” is literal: matmul ③ done 61 times in backward. Matmul ② (weight gradient) is off the chain: each layer reads its value off it for the optimizer, passing nothing on.
“Gradient” now means two things: the weight gradient from matmul ② (all-reduced, optimizer input, 2 bytes per parameter) and the δ from matmul ③, the activation gradient (used once, on no bill). §1.8's all-reduce carries the first.
❓ And once multiplied together? The famous answer comes straight from the chain rule.
⭐ Ⓐ's three curves, Ⓑ's two endpoint values, Ⓒ's 0.25: all script-computed, none copied: 0.8 to the 59th ≈ 1.9e-06, 1.2 to the 59th ≈ 46,956, a gap of 10 orders of magnitude; 0.25 is σ(1−σ)'s closed-form maximum, rechecked at 10,000 sample points.
⚠️ Ⓒ isolates the activation—each layer's true exchange rate also has the weight multiply (and normalization). Picked as the one factor computable here, ⛔ not the only cause.
📌 Source and date (the full story is in the instructor notes): formally identified in 1991, in Hochreiter's master's thesis, “Untersuchungen zu dynamischen neuronalen Netzen” (Technical University of Munich). ⭐ Written in German, it never appeared in an English-language journal. Three years later, Bengio, Simard and Frasconi published “Learning long-term dependencies with gradient descent is difficult” in IEEE Trans. Neural Networks 5(2):157–166—the title is the conclusion.
⚠️ Credit is still disputed; this lecture says only that “the 1991 thesis formally identified it” and stays out of the priority dispute. ⭐ The ending is the fun part: in 2001 Hochreiter, Bengio, Frasconi and Schmidhuber co-wrote a chapter, “Gradient flow in recurrent nets”—ten years on, the two groups who found it independently summed it up together.
⚠️ Also: the problem was first found in recurrent networks, not feed-forward ones like this. ⭐ But it is the same thing—a recurrent network unrolled in time is a very deep network: “depth = sequence length.” That chapter's experiments span 1,000 steps, a thousand layers.
y = x + F(x): each becomes “1 + a number”; every bracket is 1, and 1 to the 61st is 1: a non-decaying path to any layer.📌 Source: He et al., arXiv 1512.03385 (ResNet); the cleaner identity-path derivation is in the same group's arXiv 1603.05027. ⛔ This figure doesn't expand that equation—“1 to the 61st is still 1” is all it says.
⚠️ 0.9 / 1.1 are illustrative magnitudes, not any model's measurements; 61 layers as in DeepSeek-V3. Both products: script-computed, pinned with asserts.
The diagnosis first. A gradient is a product of 61 local derivatives, hypersensitive to each term's average (two left rows above).
Both are just 10% from 1, half the previous ±20%, and collapse just as fully. No middle ground: decay or grow, exponentially. Closeness to 1 makes the point: if 10% per layer can't survive 61 layers, no care is enough. That's the whole reason deep networks wouldn't train before 2015.
A common misreading: “small gradient” ≠ “unimportant layer.”
A gradient is effect per step; small means insensitive,
so it must move farther for the same effect.
It needs to move most. Yet parameter − learning rate × gradient gives it the smallest step.
This inversion opens 3.3b and is the whole reason adaptive optimizers exist.
Mathematically, a residual turns
y = F(x) into y = x + F(x).
Its local derivative goes from ∂F/∂x to
1 + ∂F/∂x.
Everything else follows.
∏ (1 + ∂Fi/∂x): one term
takes 1 from every bracket, and 161 = 1.
For any weights, the gradient has a non-decaying path to any layer.1 + tiny: effectively shallow, with F learned layer by layer.“Residuals let models go deep” is no mystery: “some number” becomes “1 + some number”. 6.0's table blames both on local derivatives drifting from 1; residuals pin them near 1 structurally, and 6.5's tricks (Pre-LN, init scaling) tighten that screw.
📌 Source: He et al., arXiv 1512.03385 (ResNet). The same group's arXiv 1603.05027, “Identity Mappings in Deep Residual Networks,” has the cleaner identity-path derivation: one equation, the gradient reaching shallow layers through no weight matrix. Not expanded: “1 to the 61st is still 1” above is all of it.
tools/manim/
computes the out-of-frame and vanishing layers, pinned by assertions.)We just said “we deliberately don't compute second derivatives.” But what is one?
A second derivative is the slope's rate of change.
The first derivative (the gradient): “move one step, the loss moves this much”—the slope.
The second: “move one step, the slope changes this much”—pictured,
how sharply it bends here.
First derivative: which way to go; second: how long this slope lasts. Two equally steep slopes, one soon flat, one never, need very different steps; the first derivative can't tell them apart.
All script-computed: Hessian 4.502e+23 entries (bf16 = 9.00e+23 bytes); at 80 GiB/device ⇒ 1.05e+13 devices; the 1.22 TiB of weights needs only 16.
Solving costs N³ = 3.02e+35 ops; a 1e18/s machine ⇒ 9.57e+09 years. ⚠️ That machine is an arbitrary yardstick for magnitude only, not a real one; universe age taken as 13.8 billion years.
Its use: the optimal step can be computed.
On a parabola, optimal step = gradient ÷ second derivative: one step down, no trial, no tuning.
It also cures 3.3b's “backwards” problem:
turn sensitive knobs less, sluggish ones more; ÷ second derivative makes them comparable.
Cauchy's 1847 paper ends on this split: first order approaches from afar; second order closes in—this lecture's second half covers what to do when second order can't.
First, its original form: find the x where f equals zero. Guess a point, draw the tangent there, see where it crosses the horizontal axis, and jump there as the next guess. Repeat.
Replace the curve with its tangent. The curve is hard; the tangent is a line whose zero is obvious. Strikingly fast: typically each iteration about doubles the correct digits.
How did it become an optimizer? A minimum is where the derivative is zero.
Run root-finding Newton on the derivative: “subtract f/f′” becomes “subtract the derivative over its derivative.”
That is, gradient ÷ second derivative. So the “optimal step” above is not another formula; it is Newton's method itself.
An equivalent, equally exact, easier picture:
Newton's method for optimization fits a parabola at your current point—matching the true curve in height, slope and curvature—and jumps straight to its bottom.
⇒ If the loss is a parabola, one step reaches the bottom—hence “one step to the bottom.”
Why not in deep learning? Two reasons, one computed above:
A middle road is quasi-Newton (the L-BFGS family): without building the table explicitly, it approximates its effect from recent gradients. Good in classical optimization, it never took off at huge scale plus mini-batch noise.
📌 Source (looked up): Newton had a similar method in the 1600s; today's iteration is Raphson's, 1690 (Analysis aequationum universalis); ⚠️ both then applied it only to polynomials.
⛔ Why not use it? Its size is the square of the parameter count.
With many parameters it is not a number but a table: one per parameter pair, the Hessian.
So “we deliberately don't compute second derivatives” isn't laziness—it was never possible.
⚠️ The “exaFLOP/s” machine is an arbitrary yardstick
for magnitude only. Numbers computed in topic04-fig-second.py,
pinned with asserts.
✋ Last, the easiest trap in the whole lecture:
Adam's “second moment” is not a “second derivative.”
| Second derivative (Hessian) | Second moment (Adam's denominator) | |
|---|---|---|
| Of what | differentiate the parameters twice | square the gradient, average |
| Yields | curvature: how sharp the bend | its gradient's usual size |
| Field | geometry | statistics |
| Count | N squared | N |
That both are called “second” is pure coincidence.
This is not pedantry; the warmup of 3.5 hinges on it:
Adam divides out “steepness,” not “curvature.”
The denominator is the gradient's root mean square, so however large this step's gradient, the update is pulled back to η's scale—“how steep” gets normalized away. But it never computes “how sharply it bends”: its N numbers don't hold that.
When you hit “Adam already handles the gradients; why still unstable?”—blame curvature, not steepness, and curvature isn't on Adam's books.
Likewise the “first moment” is not a first derivative but a moving average of the gradient; “first” means the statistical first moment (mean).
So why do people say “Adam's denominator estimates the second derivative”?
Because a link can be drawn after the fact.
But that's hindsight, not its inventors' thinking—see 3.3b, which starts from the problem each of three generations meant to fix.
⭐ The tell is size: Adam takes another road with 2N numbers,
never aiming at that N² table.
Hence “adaptive learning rate,” not “approximate Newton.”
In math, both directions work, same result. The only difference: how many passes along the chain.
⭐ This figure is about structure, not data: its only quantity is “671B,” the model size, established earlier.
📌 “Forward/reverse mode” are standard autodiff terms; backpropagation is reverse mode applied to neural networks.
tools/manim/.)The whole case for backpropagation, in one line: hundreds of billions of parameters, only one loss.
Many inputs, one output. If the loss were a million-dimensional output, not one number, this would no longer pay off; front to back would win.
So 1.1's answer: those 671B forward passes collapse into one backward pass.
This criterion transfers: whenever a problem needs many partial derivatives, first count inputs vs. outputs; that picks the starting end.
🕰️ Per public sources, it debuted in an unread master's thesis.
In 1970, University of Helsinki student Seppo Linnainmaa proposed reverse-mode automatic differentiation in his master's thesis: exactly this “push back from the output.”
And the thesis never once mentions neural networks. Pure math: efficiently differentiating a composite function.
Neural networks waited sixteen more years for the connection (the 1986 Nature paper).
We've seen this shape: 1.4's “vanishing gradient” also debuted in
a master's thesis (1991, Munich, German, never in an English journal).
Twice, a game-changer debuted as an unread master's thesis.
For 1970 I checked public biographies and surveys;
I did not read the thesis (Finnish, not digitized): solid secondhand, hence “per public sources.”
For 1991 (Hochreiter's German thesis) I also didn't read the original,
but the four-author 2001 survey is firsthand, one notch higher.
Cite neither as “I have read it.”
In 1.2d, step ② (own weight gradient) needs “the input saved in the forward pass”: the activation.
⛔ The root reason activations stay: not bad design, but the backward math itself.
Every forward intermediate stays in memory until backward needs it; at the last layer, all are still there.
Both costs stand: compute 3× (forward 1 + backward 2); memory, every activation, first layer to last. Next: trade compute for memory.
The real model: 61 layers, each with several intermediates; real slopes are far finer than drawn
⛔ The straight sides simplify: the real curve bumps at MoE dispatch and attention's big intermediates, but the summit stays at the end of forward
⚠️ Both 104.1 GiB and 5.07 GiB are our own per-op derivation (inputs: V3's config + the official reference MLA forward), not third-party confirmed. Sequence length 4,096, batch = 1, same basis as mountain Ⓐ.
📌 Ⓑ's two cards are one sequence, two configurations, not two models.
Both numbers are our own per-op derivation (V3 config + official reference MLA forward), no third-party check. Even if off by 2×, it still doesn't fit at this scale.
📌 How big is that “pile of intermediates”?
V3's real config, one 4K sequence: about 1.73 GiB per layer, 104.11 GiB over 61 layers.
V3's bf16 weights are 1,250 GiB; one sequence's activations, only about a twelfth.
But weights are fixed, while each extra sequence adds a full set of activations. At about 12 sequences activations match the weights; training batches always run well past a dozen. Hence the trade ahead.
❓ What's largest in that 1.73 GiB? It matters: Section 2 chooses what to drop.
Not attention. It's MoE's “nine copies.”
This sets Section 2's list: save on the copies first, not attention. It also explains a paradox: MoE is cheap in parameters (each token touches few weights) but pays a nine-copy tax in activations.
Per-item counts and 128K figures: block below. “Does it fit?” territory, not required.
Optional. This is where the two numbers above come from, for readers who want to check the math.
First, the base unit: one hidden-width tensor
= 4,096 × 7,168 × 2 B = 56 MiB.
Every number below is a multiple of it. (Sequence 4,096, batch 1, bf16.)
Why 4,096 and not 128K: V3 was pretrained at 4K (all 14.8T tokens); 128K came after pretraining, via two 1,000-step extension stages. Every calculation here uses 4K; 128K appears only for comparison, labeled as context extension.
To convert: activations track only total tokens (see 5.1), so at 128K every number below is ×32 (131,072 ÷ 4,096). The one exception, the attention score matrix, scales with S²: ×1,024 (caveat 2 below).
| Tensor kept | Width | Size |
|---|---|---|
| Norm input (= residual input; recomputation keeps only this) | 7,168 | 56 MiB |
| Norm output | 7,168 | 56 MiB |
| Q down-projection (pre/post layernorm) | 2 × 1,536 | 24 MiB |
| KV down-projection (incl. 64 RoPE dims) | 576 + 512 | 8.5 MiB |
| Q after up-projection | 128 heads × 192 = 24,576 | 192 MiB |
| K, V after up-projection | 128 heads × 256 = 32,768 | 256 MiB |
| Attention output | 128 heads × 128 = 16,384 | 128 MiB |
| logsumexp (fp32) | 128 | 2 MiB |
| Subtotal | ≈ 722 MiB |
| Tensor kept | Copies × width | Size |
|---|---|---|
| Norm input / output | 2 × 7,168 | 112 MiB |
| Router scores | 256 | 2 MiB |
| Copy sent to experts | = row above; all 9 share it | 0 MiB |
| gate / up / SwiGLU product | 3 × 9 copies × 2,048 | 432 MiB |
| Expert outputs (pre-combine) | 9 copies × 7,168 | 504 MiB |
| Subtotal | ≈ 1.03 GiB |
❓ Why these tensors? One criterion: does the backward formula use it?
Frameworks store only what the backward pass will read. By op, the pattern is clean:
| Op | Backward needs | Store? |
|---|---|---|
Add y=a+b |
Derivative ≡ 1, needs nothing | Zero bytes |
Element-wise multiply y=a×b |
a's gradient needs b; b's, a | Both |
Matmul y=x·W |
W's gradient needs x (1.3, ②) | Store x |
| Nonlinear SiLU / GELU | Derivative = f(input) | Store input |
| softmax | Derivative = f(output) | Store output |
⭐ Additions are free; multiplies and nonlinearities aren't. Nets are nearly all the latter, hence dozens of copies per layer.
⚠️ A common misconception: “Adding a bias creates another activation.” The new buffer briefly appears and is freed after use; it never enters the “keep until backward” ledger, all the table records.
⭐ The table's biggest driver: not every copy is as wide as the input. Up-projected Q is 24,576 wide (3.4 copies); K/V, 32,768 wide (4.6 copies). So half of “dozens of copies” is width, not just count.
The “9 copies” deserve a pause, especially where they appear and where they don't.
9 = 8 activated routed experts + 1 shared expert. The nine copies are on the output side: each expert produces its own intermediates and output. The “expert outputs” item alone is 504 MiB.
⛔ The input side has no nine copies.
All nine read the same memory: the “norm output” row above,
already counted.
⚠️ In practice, tokens are often first permuted into contiguous per-expert blocks for grouped matmul.
That buffer is a real copy: top-9 means each token appears nine times, 504 MiB at 4K.
But some redo the permutation in backward instead of storing it:
an implementation choice, so not in this required-tensor table.
(Fixed live 2026-09-22: once counted as required, overstating by 448 MiB.)
On Topic 1's parameter-count table it is invisible. MoE is a bargain on parameters (each token selects only a small fraction of the weights), but pays a nine-copy tax on activations.
Altogether: one MoE block is about 1.73 GiB, one dense block about 1.24 GiB; 58 MoE + 3 dense ≈ 104.11 GiB.
📐 At the extension length (128K): ×32 ≈ 55.39 GiB per MoE block, ≈ 3331.39 GiB for the whole model.
Many remember this number, but it is the context-extension bill. For the 14.8T-token pretraining: 104.11 GiB.
⛔ Three caveats; state them whenever you present this table.
Don't dismiss it as “only 4 GiB”: an entire MoE block's activations total just 1.73 GiB. This item alone outweighs it. At 128K it dominates; at 4K it is still the single largest tensor. The argument survives the new baseline: from “alarming” to “still number one.”
One more item isn't in either table, because it belongs to no layer.
The final projection turns hidden states into scores over the vocabulary (logits), of shape
[seq_len, vocab]. V3's vocabulary is 129,280:
4,096 × 129,280 × 2 B = 1,010 MiB
(one 4K sequence)
Next to the two tables:
It is the model's largest single tensor, and softmax usually runs in fp32, doubling it again.
So real code almost never materializes it whole: they chunk the sequence: one chunk's loss, its backward, then the next. It is Section 2's recomputation idea, along the vocabulary dimension.
Mainstream implementations chunk it, so it is left out of the total above, deliberately. But write your own loop computing the full logits in one go, and this block suddenly appears in your memory: 31.6 GiB at the 128K context-extension length, and without chunking it won't run at all.
1.3 stopped at the computed gradient. Several stages remain before it's used, none of them in inference.
⚠️ The figure has no quantities: traffic volume and how much hides are in the batch figure and Topic 5.
⭐ Stage ⑤'s “use & discard” sets up a later point: used once, the gradient can be bf16; accumulated, it's a running sum and must return to fp32.
Pure data parallelism averages the full gradient across all devices every step. Its size: params × bytes per param.
The key: this traffic doesn't depend on batch at all. Feed one sequence or a thousand, and you send the same number of bytes.
Compute, meanwhile, grows linearly with batch. So:
the bigger the batch, the thinner this traffic is spread.
That explains two seemingly unrelated things:
✋ Careful: a bigger batch does not buy throughput.
“Bigger batch is faster” has a precondition that fails at large scale: once parallelism saturates, per-step cost rises exactly as step count falls.
At this scale a bigger batch buys amortized communication and steadier gradients, not speed.
Curve and full trade-off: 3.5b, with two other folk sayings (“bigger is more stable,” “too big stops working”); the three go together.
Overlapping compute and communication is a key training optimization. Inference has no counterpart: no backward pass, no long tail to hide transfers in.
Why clipping can't hide, and how per-layer clipping differs: see the figure's takeaway. Key point: the whole step's size is what links to 3.3.
⭐⭐⭐ Gradient accumulation changes the timeline, not the total.
Scope: this stage exists, its size, whether it hides. Its form under other shardings (all-reduce or not) and volume: Topic 5.
“Parameters × bytes per parameter” is an order-of-magnitude convention: ring all-reduce goes out and back, about doubling traffic; MoE experts differ again.
The course's first real trade-off: not “is there a better way?” but “only one of two; which?”
In the core track because it follows from 1.6 (backward needs intermediates, so they can't be discarded). “Drop and recompute” is the next link, and its decision rule is valuable. How many GiB it saves and which level pays is a “does it fit?” question for Topic 5; details are collapsed.
Forward keeps every intermediate because backward needs it (1.6). Recomputation: don't.
When backward needs it, rerun the forward pass from this layer's input. Like a game save: keep a save point, not the whole playthrough, and replay from there.
The orders of magnitude to remember:
So lopsided it hardly counts as a trade-off.
Large-model training turns it on by default. The question is how far, never whether (Ⓑ's three levels).
⚠️ 104.1 GiB / 5.07 GiB are our per-op estimates (V3 config + official reference MLA forward); read them as magnitudes. Even 2× off, the asymmetry stands
⭐ “3× → 4×” is the matmul convention: recomputation reruns the forward pass, so the extra is exactly 1/3. The ZeRO paper gives the same (33% re-computation overhead)
⭐ Ⓒ's three “live at once” values come from L/k + k; the script asserts optimum at √L, both ends equal. That equality is the panel's whole point
⛔ Both percentages in Ⓑ are per step (forward + backward = 3 passes). The “selective” level once said 6%, per forward pass (old convention). Incomparable side by side; now unified.
📌 Ⓑ's “selective” 1.9% / 71%, derived and ranked: next figure and this section's text. DeepSeek-V3 (arXiv 2412.19437) says they recompute all RMSNorm ops and MLA up-projections: exactly that level
Ask every intermediate one question: drop and recompute it in backward: how many FLOPs per byte saved?
Smaller ratio, better candidate; that alone yields the ranking.
For linear layers it has an elegant closed form:
A [S,k] × [k,n] matmul
costs 2·S·k·n FLOPs to recompute and yields S·n·2 bytes (bf16).
Divide: cost per byte = k.
The closed form assumes recomputation costs only the tensor's own op, true while its input is still there. Drop the upstream tensor too, and it must be recomputed first: cost accumulates along the chain, no longer one k. Our selective list keeps inputs present; watch this in your own.
Per byte, recomputing a linear layer's output costs
exactly its input width in FLOPs.
S and n cancel:
independent of sequence length and output width.
So the best linear layer to recompute is the narrowest-input one; check the config.
Compare with 1.6: “biggest” and “most worth dropping” are two different rankings.
1.6 ranks by absolute size; this section by cost per byte saved, a ratio. No reason for them to agree.
Here they happen to agree: 1.6's two biggest items are both in the cheapest group.
Luck, not a law. Counterexample, same table: the priciest item (o_proj output, 17.59 per GiB) is only 2.5% of a layer, little saved at high cost. On another model the rankings may conflict. Then choose by ratio, not size (cost per byte saved).
Attention isn't linear: cost grows with S², output only with S, so cost per byte grows linearly in S.
⭐ The closed form for linear layers: a [S,k]×[k,n] matmul costs k per byte (recompute 2·S·k·n FLOPs ÷ output S·n·2 bytes; S and n both cancel), so on the plot it must be a horizontal line
⚠️ The two slopes are 1.25 for V3 (qk 192 / v 128) and 1.00 for GPT-3 (qk = v = 128); both use the causal-halved convention. ⛔ For another model, redraw its own two lines and recheck the causal-halving convention; the slope changes from model to model, but “it is sloped” does not.
⛔ The 13,107 crossover isn't an engineering threshold; it also depends on the reference layer (four lines, four crossovers)
📌 The 2022 paper's gains (GPT-3 saves 70% for 2.7%) use Megatron's un-halved convention; in this figure's causal one, GPT-3's attention share is 1.37%
The figure's two lines, different slopes, are the whole section: one flat (linear layers, cost = input width), one climbing (attention). Different slopes must cross.
A constant and a climbing line must cross.
Compare against the widest linear layer. For V3:
o_proj (projects the 128 concatenated heads into the residual stream), input width 128 × 128 = 16,384,
not d_model's 7,168.
⛔ Why stress “not d_model”? I made that mistake first.
The crossover once read 5,734, using d_model (7,168) as “the widest linear layer.” Standard MHA has “heads × head dim = hidden,” so that works; MLA removes the constraint: V3 is 128 heads × 128 = 16,384, twice d_model.
The counterexample sat in the 1.7 table all along (“attention output 128 heads × 128 = 16,384”), half a section away, never cross-checked.
Rule: when the architecture changes, recheck any common-sense “A = B”. It raises no error; it skews every number downstream.
Hence: 1.25·S = 16,384 → S ≈ 13,107.
13,107 is a rough crossover, not an engineering threshold: it hinges on causal halving, V3's head dims and the reference layer. Remember “different slopes must cross,” not the number.
Rank V3 at the 4K baseline: one MoE block, sorted ascending by TFLOP paid per GiB saved:
| Tensor | Saved | Recompute | Paid/GiB | Verdict |
|---|---|---|---|---|
| logsumexp (attention by-product, fp32) | 0.002 GiB | ~0 | 0 | Free |
| RMSNorm outputs (3, elementwise) | 0.121 GiB | ~0 | ~0 | Free |
| SwiGLU products × 9 (elementwise) | 0.141 GiB | ~0 | ~0 | Free |
| K/V up-proj (input width 512) | 0.250 GiB | 0.14 TFLOP | 0.55 | Worth it |
| Q up-proj (input width 1,536) | 0.188 GiB | 0.31 TFLOP | 1.65 | Worth it |
| Expert outputs × 9 (input width 2,048) | 0.492 GiB | 1.08 TFLOP | 2.20 | Worth it |
| Attention output | 0.125 GiB | 0.69 TFLOP | 5.50 | Marginal |
| gate/up/router/down-proj (input width 7,168) | 0.303 GiB | 2.33 TFLOP | 7.70 | Marginal |
| Post-MLA residual (= o_proj out, input width 16,384) | 0.055 GiB | 0.96 TFLOP | 17.59 | Marginal |
Don't memorize this table; it comes straight from the closed form above. Except the elementwise rows (cost ≈ 0) and attention (moves with S), each row's “paid/GiB” is just its input width in different units, so you can rank it from the config. A confession: one row was once hand-filled: “9.14 GiB saved × 7.70 per GiB” is 70 TFLOP, yet the column said 39.08. Yet the table's selling point is “you can rank it yourself”; anyone who did would have hit it. Now it's all generated; no hand edits.
⛔ “Is there anything that takes little memory but lots of compute?” Yes, attention. But how extreme it is depends entirely on the sequence length.
One criterion, a ratio: keep high, drop low. Here only the attention row moves with sequence length; the rest are input widths, independent of S.
| Attention, paid/GiB | 4K (our baseline) | 128K (extension) |
|---|---|---|
| Absolute | 5.50 | 175.92 |
| vs. widest linear (o_proj, 17.59) | only 31% of it | 10× pricier |
| Rank in table | 7th of 9 (o_proj, gate/up pricier) | 9th of 9 (priciest) |
| Verdict | Marginal | Never |
Same model, same rule: go from 4K to 128K and attention goes from “still behind two linear layers” to “10× more expensive than anything.”
Don't overstate it: at 4K attention's ratio is 5.50, above the “take everything below 3” line discussed below, so by our own cut it isn't in the “take” group. What changes is whether it's the extreme outlier, not whether to recompute it: the first is a fact; the second depends on where you cut.
❓ Where does “below 3” come from? I chose it; it isn't derived.
In the table, the expert-output row is 2.20 and the next row, attention output, is 5.50; 3 just sits in the gap. Put it at 2.5 or 5 and the result is identical.
What actually sets the line isn't in the table:
your current exchange rate, what a GiB of memory is worth to you in TFLOP.
Memory nearly full: move it right (drop more, pay more compute); compute-bound: move it left.
So don't memorize 3 as a threshold. The table is sorted by ratio; you only decide where to cut. The ranking is objective; the cut is your setup's call.
Cutting at 3 (taking every ratio below it): per layer you save 1.19 GiB and pay 1.53 TFLOP. Per step (one layer, forward + backward = 3 passes = 16.4 TFLOP): 9.3% more compute for a recomputable-pool saving of 71%.
Full recomputation, on the same per-step basis, is “pay 33% more, remove 95%.” Selective is still 2.7× more cost-effective, hence worth its own setting.
At 128K these numbers look very different, and much better.
Same cut: “only 1.9% more for 71%”; vs. full recomputation, 13.1× the value: truly “an order of magnitude.”
Why: the numerator (recompute cost) ∝ S, the denominator (one step) at 128K is inflated by S² attention. Longer S, cheaper-looking recomputation. “1.9%” is a flattering long-context number; don't cite it at 4K.
The direction holds, though: selective clearly pays at both baselines. And full recomputation's “pay 33% for 95%” is scale-free: it depends only on running forward twice, not on S.
⛔ One convention to tighten: attention's backward is 2.5×, not 2×.
1.3's “backward ≈ 2× forward” is exact for matmuls with weights. But this lecture assumes FlashAttention throughout (1.7, caveat ②), and to save memory it recomputes the attention scores in backward.
From the FlashAttention-2 paper: “To get the FLOPs of the backward pass, we multiply the forward pass FLOPs by 2.5 (since there are 2 matmuls in the forward pass and 5 matmuls in the backward pass, due to recomputation).” (arXiv 2307.08691 §4.1)
Another instance of this section's theme: the memory FlashAttention saves is paid for by recomputation inside FlashAttention itself. “No free lunch,” with a concrete price.
With this lecture's shares: at 128K attention is 82.1% of a layer's forward, so a step is 3.41×, not 3× (3× undercounts by 12%); at 4K it is only 12.5%, giving 3.06×: no real difference. The rest of this lecture keeps 3×, a deliberate simplification.
“Selective +9.3%” and “full +one-third” once had different denominators (a forward pass vs. a whole step), so weren't comparable. As 2.5 says, “before comparing such ratios across papers, check whether they halved for causal.” Both are now per step.
First-party evidence the rule (FLOPs per byte saved = the producing op's input width) isn't just theory.
DeepSeek-V3, plainly: they “recompute all RMSNorm operations and MLA up-projections during back-propagation,” because “with a minor overhead, this strategy significantly reduces memory requirements for storing activations.”
RMSNorm outputs and MLA up-projections are exactly this rule's two cheapest categories: elementwise ≈ 0; up-projection input widths only 512 / 1,536. Our own rule, their actual config: stronger evidence than more arithmetic.
📌 Source: arXiv 2412.19437 §3.2.3.
The idea of “selective recomputation” comes from Korthikanti et al., Reducing Activation Recomputation in Large Transformer Models, arXiv 2205.05198 (2022-05).
Same criterion as 2.3: drop parts that “take up a considerable amount of memory but are not computationally expensive to recompute.”
But it chose attention: QKᵀ, softmax, softmax dropout, and attention over V. Payoff: GPT-3 saves 70% of activation memory for only 2.7% extra compute; MT-NLG saves 65% for 1.6%.
Why reversed? Its sequence length is 2,048.
But use GPT-3's own slope: standard MHA, qk = v = 128,
slope 1.00 (not V3's 1.25): 1.00 × 2048 = 2,048.
GPT-3's widest linear layer is 49,152 (= 4h, the MLP down-projection's input width; V3: 16,384, o_proj's input width).
2,048 ≪ 49,152: far left of its own crossover. At 2K, attention really is the best to recompute. The paper was right.
Same rule, flipped conclusion. To be precise:
Not just sequence length. Slopes differ (1.00 vs. 1.25), as do reference layers (49,152 vs. 16,384): redraw both lines.
I got 49,152 wrong too: I first wrote 12,288,
GPT-3's d_model. The rule needs
the producing matmul's input width; the MLP down-projection's input is 4h.
Same mistake on V3: I took d_model 7,168 as widest, missing o_proj's 16,384 input (128 heads × 128);
the crossover was briefly wrong (fixed in 2.4). This panel's own point, and I stepped in it twice.
⭐ “Grabbing a relation that sounds like common sense” is the hardest error to catch.
No error, no blow-up; every downstream number just drifts. And since it sounds right, you've no reason to doubt it.
The same mistake, twice in this lecture: d_model as the widest linear layer, on V3 (missing o_proj's 16,384) and on GPT-3 (missing 4h's 49,152). One “common sense” rule, true on neither model.
What transfers is the shape, not the numbers. New model: redraw two lines, don't memorize numbers.
Another confession: this figure once applied V3's 1.25 to GPT-3 (2,560), in a lecture about “don't apply A's constants to B.” Rule: when two models share a figure, each line uses its own model's constants.
Mind the convention, or numbers won't match: the paper's 2.7% uses Megatron's classic formula, not halved for causal masking. In our causal convention, GPT-3's attention share is 1.37%. Both are right; before comparing such ratios across papers, check for causal halving.
Today this is handled for you. FlashAttention never stores the score matrix and recomputes it in backward: the paper's advice, built in.
⛔ I first wrote “so attention isn't even on modern frameworks' candidate lists.” Then I read the source. Exactly backwards.
Megatron-LM's selective-recomputation switch --recompute-modules:
its first candidate, core_attn,
is the default (recompute_modules = ["core_attn"]).
There are three layers here; I first covered only one.
recompute_granularity
defaults to None: nothing is recomputed.
core_attn is the default module once you enable selective.“If you are using transformer_engine as the transformer implementation, the core_attn is from transformer_engine and may be the fused version. For fused attention, you have no need to set 'core_attn' to recompute. Please check that the core_attn recompute is really needed.”
📌 transformer_config.py has
if "core_attn" in self.recompute_modules: warnings.warn(...) (main, 2026-09).
So this isn't “a stale default nobody looks after”: someone noticed, left the default alone, and pinned a warning beside it.
Why not just change it? That's the interesting part.
Changing a default has a cost: results others got with the old one become irreproducible. Frameworks prefer a suboptimal default to voiding past results.
So the default becomes a fossil: it records the 2,048-era optimum. At that length it was right; at long context, by our table, it's the last to recompute; and under today's fused attention it has essentially nothing to do.
The lesson isn't whether to turn it on, but: for any default, ask what year it was picked and if its premises still hold. Me included: I first wrote “it's still the default,” collapsing three layers into one; readers would think it recomputes attention from step one.
The rest of the list matters more:
layernorm, moe_act, mla_up_proj,
i.e. RMSNorm outputs, SwiGLU products, K/V and Q up-projections, all cheap by 2.3's rule.
All in the table's cheapest group.
But not “exactly these, no others”: the list also has mlp / moe /
shared_experts, and mlp is gate/up, the table's second-priciest row, far above the cut (the priciest is the o_proj output row).
Our closed-form list matches the framework's options.
📌 megatron/core/transformer/transformer_config.py,
field recompute_modules (main, 2026-09).
Rule: before saying “modern frameworks don't do that anymore,” grep the framework.
Such claims sound like common sense and go stale fastest.
Takeaway: backward needs intermediates, so they can't be dropped; but “drop and recompute” is a negotiable trade, settled by a rule you can derive, not by memorizing others' verdicts.
Beyond the three, folded below: recomputing can change results (random numbers, side effects, bitwise), and gains don't copy across scales (MFU / HFU).
So far, two measurable costs. Recomputation has a third, in neither account, raising no error.
Recomputation assumes a forward rerun gives exactly the same result. Three things can break that.
① Random numbers: handled for you.
Forward has dropout. If recomputation draws a different mask, backward uses a different network. The gradient is wrong, with no error.
PyTorch is blunt: save the forward-time RNG state,
restore it to recompute.
torch.utils.checkpoint's
preserve_rng_state defaults to True.
This also explains the third item on the checklist in 3.7: besides weights and optimizer state, a checkpoint must store the RNG state, for the same reason: random numbers are part of the training result, not transient runtime state.
② Side effects: nobody handles them.
Suppose the segment also modifies something on the side: updates a running statistic, increments a counter, writes a log line. Recomputation modifies it again.
The framework can't roll these back; it doesn't know what you changed. The rule: the recomputed segment should ideally be a pure function.
③ Bitwise mismatch: usually harmless, but it has one specific victim.
Run the same computation twice and the last bits may differ: reduction order or kernel choice is enough.
For training itself, differences this small barely matter. But they hinder bitwise reproducibility, exactly what PaLM achieved in 6.6 and what the “roll back + skip data” remedy in 6.2 presumes.
In short: recomputation closes the compute ↔ memory deal, but quietly demands reproducible results. The first two accounts say whether it pays; this one only shows when something breaks.
⛔ Same model, same switch, different scale: the gain can flip sign.
Why: recomputation shifts the compute-to-memory-traffic ratio, which varies anyway with parallelism and chip count.
A general rule (not only for remat): any parameter that changes shard shapes can't be copied across scales. Revalidate at the target scale.
2.5 shows the rule is harsher than it sounds: sequence length is on that list too, and it flips the gain's sign, not its size: the item most worth recomputing at 2K becomes the last to touch at 128K.
📌 Another number worth knowing, which you can compute:
Useful work is 3 units (forward 1 + backward 2); full recomputation adds 1, so 4 are executed. Recomputation is one-quarter of executed compute.
I originally wrote “one-third” here: that confused “one-third more than before” (1 ÷ 3) with “share of total executed” (1 ÷ 4). Same 1 unit, different denominator. 📌 The “one-third more” isn't our own derivation: the ZeRO paper says “33% re-computation overhead” verbatim (arXiv 1910.02054 §3.2).
That's why MFU looks so low: recomputation does real work, but by definition isn't in the numerator: not useful compute.
There are two metrics, differing by exactly the recomputation.
6ND family).With full recomputation, HFU / MFU = 4 / 3: the “1 of every 4 units” above.
So for any MFU number, first ask which numerator it uses. Two papers' MFUs can differ by 30% just because only one used recomputation.
⛔ At long context, “low MFU” is not mainly due to recomputation.
Recomputation makes MFU at most 3/4 of HFU (a factor of 1.33). But 5.2 shows that at 128K, 6ND counts only 17.9% of the real compute;
that convention gap alone is 5.6×.
⭐⭐⭐ The two factors swap places with sequence length: at 4K, 6ND undercounts by only 1.14×, so recomputation's 1.33× dominates; at 128K, the reverse.
In practice: at long context, turning off recomputation because “MFU is only 20%” recovers at most a quarter. First align the FLOP convention.
Two sections fought activations. The biggest item on the bill has stayed silent.
❓ One premise first; the bill below depends on it.
In training, each parameter sits in HBM twice:
① a compact copy (bf16, 2 bytes) for forward and backward;
② a precise copy (fp32, 4 bytes) for bookkeeping. Updates land here.
This is mixed precision. 3.2 explains why both are needed.
In mixed precision, each parameter carries five things; only the first is the model itself.
Adam: Kingma & Ba, arXiv 1412.6980 (ICLR 2015). AdamW: Loshchilov & Hutter, arXiv 1711.05101 (ICLR 2019); the sentence in the “AdamW is not a new algorithm” band paraphrases its abstract's first sentence (the original qualifies the SGD half: equivalent only when rescaled by the learning rate).
Muon: Keller Jordan et al., 2024; writeup: kellerjordan.github.io/posts/muon. Muon “only applies to 2D parameters”; “scalar and vector parameters of the network, as well as the input and output layers, should be optimized by a standard method such as AdamW”; it “has a slower per-step wallclock time than AdamW”: all three verbatim.
Momentum: Polyak 1964 (heavy ball) / Nesterov 1983. AdaGrad: Duchi, Hazan, Singer, JMLR 2011 (no arXiv version). RMSProp: Hinton, 2012 Coursera course, Lecture 6. It was never formally published.
Three state-savers not drawn: Adafactor (arXiv 1804.04235, factors v as row + column), Lion (arXiv 2302.06675, gradient sign only, one momentum), 8-bit Adam (quantized state, same algorithm).
⛔ No convergence-speed comparison: we have no head-to-head data; a curve would be invented. Only structure and each paper's claims.
Remember one line: weights 2 bytes, optimizer 12.
“Optimizer state dominates HBM” isn't rhetoric. It's 2 : 12.
All 12 bytes are fp32; the next subsection explains why.
⛔ A stubborn misconception: “Mixed precision halves memory, right?”
For resident memory, it saves not one byte. Add up both sides and you get 16 either way.
| Weights | Grads | Momentum m | 2nd moment v | Master weights | Total | |
|---|---|---|---|---|---|---|
| Pure fp32 | 4 | 4 | 4 | 4 | — | 16 B |
| Mixed precision | 2 | 2 | 4 | 4 | 4 | 16 B |
Look at that column: weights and gradients save 2 bytes each, but the master copy costs 4; the 4 eats 2 + 2.
So what does it buy? Two things, neither resident memory.
Rule of thumb: before saying “it saves memory,” ask which memory. Resident state, activations: two accounts. “Saves half” merges them and fails when it matters (e.g., will it fit one device?).
Why three fp32 boxes and two bf16? Don't memorize. One picture explains it.
📌 Every number in Ⓑ is computed by a script using a real bf16 rounding function (top 16 bits of a float32, round-to-nearest-even, stdlib struct only). Three asserts guard it: less than half a gap must round back; a full gap must land; 1,000 adds must still equal 1.0. ⭐ Rule of thumb: if a figure asks you to believe an arithmetic result, have a script do it.
📌 Ⓒ's test (not “accumulates?” but “do old contributions stay forever?”) was honed by a first-party counterexample: DeepSeek-V3 (arXiv 2412.19437 §3.3.3) keeps AdamW's moments in bf16, no degradation seen; master weights and accumulation gradients stay fp32. ⛔ Meta-rule: a rule a counterexample hits usually has coarser wording than mechanism; keep it.
⭐ Ⓐ also settles 6.6's item: fp16 needs loss scaling, not because “16 bits isn't enough.” It moves 3 of bf16's 16 bits from exponent to mantissa. With five exponent bits, not eight, the smallest representable positive number rises by over thirty orders of magnitude; small gradients underflow to zero.
The 12 fp32 bytes are exactly what keeps old contributions; the two bf16 items are used once and dropped. Not a coincidence: one test draws the line.
❓ Can gradients be fp32? Yes, in some implementations.
Usually for gradient accumulation over micro-batches. In bf16, big values swallow small ones and precision drains within a few rounds, so many implementations keep an fp32 accumulator.
That column then costs 4 bytes: 16 becomes 18.
📌 Our 16 is
the ZeRO paper's classic accounting: 2 bytes weights + 2 bytes gradients +
12 bytes for three fp32 copies, written 2Ψ + 2Ψ + KΨ = 16Ψ, K = 12.
arXiv 1910.02054.
Sourced, but a configuration, not a law; quote its assumptions.
A small correction: the paper's two 2-byte items are fp16 (2019's norm), not bf16. bf16 is standard now. Same bytes, same bill, but different formats (see “Does bf16 need loss scaling?” in 6.6).
⛔ A first-party counterexample sharpens the test.
DeepSeek-V3 keeps AdamW's two moments in bf16, “without incurring observable performance degradation,” but master weights and gradients in fp32, gradients precisely for accumulation.
Right idea, too coarse: not accumulation, but unbounded accumulation:
Tightened test: do old contributions stay forever? Yes: fp32. Decaying or single-use: low precision suffices.
“Must be fp32” isn't absolute: some have dropped it.
Master weights need fp32 because round-to-nearest erases small updates. Stochastic rounding (round up with probability set by the remainder) recovers it in expectation over many steps.
Finer still: the killer is not accumulation but accumulation + rounding the same way every time. Remove that bias and bf16 master weights are viable.
Not recommended: it needs extra support; fp32 masters are cheap and easy. Shown because knowing why a rule holds tells you when it breaks.
📌 Source: DeepSeek-V3, arXiv 2412.19437 §3.3.3 (“Low-Precision Storage and Communication,” not recomputation §3.2.3). It confirms the list nearly item by item: “We adopt the BF16 data format instead of FP32 to track the first and second moments in the AdamW … optimizer, without incurring observable performance degradation. However, the master weights … and gradients (used for batch size accumulation) are still retained in FP32 …” Accumulation reason included. The 16-byte table is no law: V3's per-param bill differs.
The test outlives this lecture. Why fp8 training is hard is the same question: as precision drops, what can go lower, what must stay? Still accumulation. Another lecture.
⭐ Gradient descent predates neural nets by over a century. Augustin-Louis Cauchy, 1847, French Academy: “General method for solving simultaneous equations”.
He wasn't optimizing; he was solving equations.
The standard method, elimination, had two flaws, Cauchy notes: it often fails outright; when it works, the last equation is hopelessly complex, even from simple ones.
⭐ So he swapped the problem: minimize a function instead.
The function must be non-negative. Merging equations takes one line: u² + v² + w² + … = 0. Squares can't be negative, so the sum is zero exactly when each is. “Solve” becomes “drive the sum to zero.”
⭐ The invention isn't calculus (Newton, Leibniz) but the reframing: “solve the equations” → “walk downhill.” Every model we train runs on it.
⭐ The paper ends: once close, switch to the linear (Newton's) method to finish fast. He knew each order's place: first order from afar, second to close. 3.3b's “why everyone must estimate the second derivative” is this split at hundreds of billions of parameters.
📌 Source: A.-L. Cauchy, Méthode générale pour la résolution des systèmes d’équations simultanées, Comptes Rendus Hebd. Séances Acad. Sci. 25, 536–538 (1847). The two flaws, the sum of squares and the Newton line paraphrase the paper (English translation).
671B partial derivatives in hand. Which way to move?
Nudge every parameter: loss changes by Σ nudge × partial derivative. For small enough steps this is derived, not a rule of thumb.
This section's one crack: “multiply and sum” isn't the partial-derivative definition (a single-variable limit) but a first-order approximation (total differential). It needs small steps: the problem for the rest of this subsection and all of 3.3b.
For loss to drop, the sum must be negative. Move each parameter against its partial derivative: every term, “negative × positive,” is negative.
“Follow the negative gradient and loss falls” is math, not folklore. Among equal-length steps, fastest: the flip side of “gradient = steepest ascent.”
⛔ How small is “small enough”? No answer.
A partial derivative is the slope where you stand; it promises nothing a meter away. Overstep and you land higher than you started.
Direction: settled. Step size: no answer. It occupies the rest of this subsection and 3.3b entirely.
📌 Ⓑ's footnote is from Su Jianlin's “Why do we prefer isotropy? An understanding via steepest descent” (in Chinese; our translation): “The negative gradient descends fastest only given a premise, chiefly a Euclidean norm; change the norm and the steepest direction changes.” ⭐ Almost all tutorials skip it; without it Muon is just “yet another optimizer.”
📌 Four framings—“a ball rolling downhill,” “step ∝ slope, so no overshoot,” “which valley depends on the start,” and “in high dimensions, read the signs and relative sizes in the column”—come from 3Blue1Brown's official lesson notes for “Gradient descent, how neural networks learn.” All checked against the original; the figure is our redraw.
⭐ The Ⓒ column of negative gradients is exactly what one backward pass produces: where Sections 1 and 3 connect.
tools/manim/.)That figure left an ugly line: your start picks your valley. Alarming in 1D. The next figure dismantles it.
📌 Four points from Hung-yi Lee's 2021 slides “When the network won't train (1): local minima and saddle points” (in Chinese): “critical points: local minima or saddle points,” “test: all Hessian eigenvalues positive?” “negative-eigenvalue directions still descend, rarely done in practice,” “measured: never reach a real local minima.” All checked at source; the figure is ours.
⭐ He also borrows Liu Cixin's “Death's End” (Three-Body III): a magician who pulls things from sealed spaces via a higher dimension. Slide: “From 3 dimensional space, it is sealed. It is not in higher dimensions.” ⛔ Novel unchecked; we relay only that line. The main metaphor (mountain pass) is ours.
⚠️ Ⓑ We deliberately give no probability: “each direction curves up or down with equal odds, so all d curve up with probability 2⁻ᵈ” sounds right, but eigenvalue signs aren't independent, so the number is made up. We keep only what holds by definition.
⭐ The slides' closing line recurs here: smaller batches and momentum both help escape critical points. Noise and inertia: what the batch figure and the two Adam figures cover.
Two cross-sections, but the shape is 3D. Let's rotate it.
tools/manim/.)❓ “Enough dimensions means no pits” is wrong by one word.
Four dimensions or a thousand: either can hold a deep, sealed pit. Nothing forbids it.
⭐ The truth: a pit both high and sealed is extremely rare in high dimensions. Sealed pits exist; sealed implies low.
Why does height go with sealedness? Sealing needs every direction curving up. Up high there are more ways down, much height left to lose. Near the bottom fewer remain, until none: only then a floor.
⭐⭐ So in high dimensions nearly every stall is at a pass. Floors aren't forbidden; they're all very low, and you're not there yet.
Not just reasoning: it was measured.
Train to zero gradient, then count: how many directions curve up?
The fraction never reached 1.
Wherever training stops, some directions still go down. Paths exist; the slope was too flat.
✋ So: “Don't more parameters make training harder?”
Memory and compute: yes. Getting stuck: the opposite. Two questions; don't merge into “big models are harder to train.”
⛔ But not stuck isn't fast. Near a pass the slope is flat, the gradient tiny; with steps “proportional to the slope,” it crawls for ages. Most optimizers ahead fix this.
📌 Dauphin et al., Identifying and attacking the saddle point problem in high-dimensional non-convex optimization (arXiv 1406.2572, NeurIPS 2014). Abstract: the usual view blames local minima far worse than the global one; they argue “a deeper and more profound difficulty originates from the proliferation of saddle points, not local minima, especially in high dimensional problems …”.
⛔ A plausible fabrication: “each direction curves up or down at even odds, so all d curve up with probability 2^(−d).” Curvature signs aren't independent: no basis. We stick to what holds by definition.
⭐ The second half is key: saddles sit in “high error plateaus” that “can dramatically slow down learning, and give the illusory impression of the existence of a local minimum”. The enemy: crawling, not sticking.
Left over: no sticking, but flat passes mean crawling. The next figure is a first answer.
⭐ Ⓐ's two trajectories, Ⓑ's three arrow lengths, Ⓒ's bars: all script-computed. Our made-up 1D loss L(x) = 0.015x² + 0.45cos(1.1x) is built for “one shallow pit, then one deep valley”; the script found the pit and its depth, nothing hand-marked. Four asserts: no-momentum run stops in the pit, gradient zero; momentum run lands in the valley; landings far apart.
⭐ Ⓑ deliberately picks the green trajectory's step right through the shallow pit, and asserts “gradient here is nearly 0 while the move is still large.” Without that, the figure says nothing.
📌 How old is this idea? 1964. Polyak, “Some methods of speeding up the convergence of iteration methods”: 22 years before the backpropagation paper in Nature. ⭐ He called it “the method of a small heavy sphere”; our ball is named after it.
⭐⭐⭐ The Ⓑ picture, “the gradient pulls back, inertia carries it away,” is in the paper verbatim: “The motion proceeds not in the direction of the force (i.e. antigradient) because of the presence of inertia”: inertia keeps it from following the negative gradient. He also says the inertia term leads to motion “along the bottom of the trough.”
⭐⭐ Best of all: in 1964 Polyak suggested ρ = 0.8–0.99; Adam's default β₁ = 0.9 is still in it, sixty years on. (Script-asserted for the figure's two βs.) ⭐ His tuning order: tune the learning rate with ρ = 0 first; add momentum once convergence slows. He reports it mostly beat the gradient method, by up to tenfold (paraphrase; the original says “up to tenfold”).
⛔ One claim left out as unverified: that the 1986 Nature backprop paper already had a momentum term. I couldn't get the original (the link was a 404), so we won't claim it until I read it.
📌 Framing from Hung-yi Lee's “When the network won't train (1)” (in Chinese): he writes momentum as “Movement: movement of last step minus gradient at present” (the last step's move, not just the gradient) and closes with “Smaller batch size and momentum help escape critical points.” ⛔ Our redraw; curves and numbers from our runs.
⭐ The figure above, escaping a shallow pit, is real but the least used of the three—sealed pits are rare in high dimensions.
① Coasting across flats. Near zero gradient, inertia remains. It treats crawling, not sticking: the high error plateaus around saddles.
② Stop bouncing off valley walls. Its main job in real training, the only one at work daily: same signs add, opposite signs cancel.
Recall 3.3b's valley, steep one way, flat the other: zigzag across, inch along.
⭐ Momentum is a filter: it amplifies the direction you keep and damps the one you swing in. Deep-net loss surfaces are usually narrow valleys; it nearly always helps.
⭐ Polyak's 1964 paper: inertia leads to motion “along the bottom of the trough.” The heading above, written in 1964.
③ Smoothing noise. Small-batch gradients are noisy; a moving average smooths jitter for free.
❓ When doesn't it help? There's a computable ceiling.
Classical momentum (Polyak's; PyTorch's SGD momentum): velocity ← β × velocity + gradient. A constant gradient grows velocity to 1 ÷ (1 − β) times it.
| β | Steady-state gain | Memory span (steps) |
|---|---|---|
| 0.80 | 5× | ~5 |
| 0.90 (most common) | 10× | ~10 |
| 0.99 | 100× | ~100 |
⭐ That is its whole budget. At β = 0.9 the step grows at most tenfold. Taller pit walls trap it.
So “sometimes it fails” isn't mystery or mistuning: finite kinetic energy, computable ceiling. It escapes above because that pit fits within tenfold.
⛔ Flip side: overshoot. Near the floor, inertia carries it past; it swings a few times, then settles. Not free: end wobbles buy no stops en route.
⚠️ Both are “momentum,” but Adam's doesn't amplify. Easy to confuse.
v ← β×v + gradient; a constant gradient grows v to 1/(1−β)×. It buys speed.m ← β×m + (1−β)×gradient: a normalized moving average; at constant gradient, m → the gradient: no amplification. It buys a stable direction, not speed.⭐ Same name, two things: one stores kinetic energy, one removes jitter. Adam's step comes from 3.3b's 0.2, not β₁'s speed.
⭐ So “escapes local minima” is its least-used side. Its daily work: coasting across plateaus and not bouncing between valley walls.
tools/manim/.)These three figures settle the direction: negative gradient lowers loss; high dimensions rarely trap; inertia crosses flats. Left: how big a step?
Direction is settled. Now: how far to step.
A line nearly everyone has written:
param ← param − lr × grad
📐 First, back to 3.3's Cauchy, 1847.
His paper closes: once close, switch to the linear, or Newtonian, method to finish fast.
That splits the job: first order from afar, second order to close.
This section: when second order can't close it.
The units argument is rigorous but slow to see. Two knobs show the result first, in grade-school arithmetic.
Two knobs, A and B; current error 1,000.
❓ How many notches each to cancel 1,000?
A: 1,000 ÷ 100 = 10 notches. B: 1,000 ÷ 1 = 1,000 notches.
Turns scale inversely with sensitivity: few for sensitive, many for sluggish.
What does param ← param − lr × grad give?
A moves η × 100, B moves η × 1—100 : 1.
⛔ You need 1 : 100; it gives 100 : 1. Not slightly off: exactly backwards.
The sensitive knob overshoots, oscillating. The sluggish one barely moves, short after thousands of steps. The slowest knob sets the pace.
The inverse rule assumes independent, locally linear directions; real valleys curve, couple. Intuition, not formula. The exact-step figure is rigorous.
This is 1.4's vanishing gradient, restated.
There, chained products shrank the first layers' gradient to 0.0016. Often misread: “early layers don't matter.” Backwards.
Small gradient = sluggish knob = farther to go. It needs to move most; SGD gives it the smallest steps.
Be precise, or the views clash.
A middle layer scaling the signal by 0.5× halves the upstream gradient; what doubles is the distance needed for the same effect. Reciprocals: one fact, two faces.
Mixed, they contradict; our convention: gradient = effect per notch, not distance to move.
The units don't match.
The gradient is loss change per parameter change: unit loss ÷ param (per 1.0, loss is in nats/token, so nats/token ÷ param). “Param” isn't physical; weights are dimensionless. The parameter's own scale is the unit; it's conserved, so the derivation holds. This is SGD; Adam's second-moment division puts updates on parameter scale, so its learning rate has other units (exact-step figure). You want a distance: in parameter units.
So something must convert: the learning rate. Its unit: param² ÷ loss, i.e. param² · token / nat.
Read the unit aloud: the learning rate is not a pure number. It bakes in parameter and loss scales; loss scale depends on vocabulary (1.0's scale tops out at ln V). New model, vocabulary or batch: retune.
One learning rate for the whole network, yet gradient scales differ by orders of magnitude. Below: same valley and rule, different rates.
📌 The dilemma (one shared learning rate: too large, the second update flies off the map; too small, a hundred updates don't reach the bottom; so each parameter gets its own rate) is from Hung-yi Lee's “Training Tip” slides (Fall 2025 GenAI-ML course).
⛔ But we did not copy his two specific numbers. Borrowing a hand-picked learning rate means borrowing its landscape. We built our own valley, ran it twice; asserts pin each path.
⚠️ The figure's steepness ratio is 25, not “orders of magnitude.” Draft one used 1,000: physically right, visually dead; dozens of steps moved the flat axis dozens of pixels, so the zigzag became a vertical line. ⭐ Rule: to show a dynamic, pick visible values; real magnitudes to the formula. Ⓑ's row 3 extrapolates.
⭐ Both step counts in Ⓑ are closed-form: each step shrinks the flat direction by (1 − 1.9 ÷ ratio), so halfway takes log½ ÷ log(that). The script checks them against a run: within 1 step.
tools/manim/. Threshold, ±5% behavior and “one third” computed at render, asserted.)Ignore gradient size, keep direction: divide out each parameter's own scale.
❓ Why divide? An exact answer, small enough to check here.
📌 “Optimal step = |f′| ÷ f″” and “estimate f″ from first derivatives” follow Hung-yi Lee's (National Taiwan University) Gradient Descent slides. The figure is our redrawing.
⭐ All four numbers check on the spot: on f = a·x², f′ ÷ f″ = 2ax ÷ 2a = x, exactly the distance to the bottom. Script-asserted.
⛔ First, a misleading, near-universal claim: “Adam's denominator estimates f″.”
It's hindsight, not the inventors' view, and can't explain why it excels on sparse data or why it turns cautious when unsure.
The real path started from statistics, not geometry.
① AdaGrad (2011): rare words never got learned.
They had text in mind, not neural nets. “The” appears millions of times, a technical term three times, yet the rare ones often say the most.
With one learning rate that term never learns: its gradient exists only on its few samples, zero elsewhere; common features set the pace.
⭐ So: give each parameter its own learning rate, bigger for the rarely seen.
How often seen? Sum its past squared gradients: large if everyday, small if rare. Then divide by the square root.
⛔ Not a word on curvature or second derivatives. Pure statistics. Provenance in 3.4: its starting point is TF-IDF, not curvature.
❓ So what exactly is the “second moment”?
Square, average, root: the gradient's RMS, i.e. this parameter's usual gradient size.
⭐ Not curvature: this knob's typical recent response size, the parameter's own ruler.
What does dividing do? Current over usual size: how big this one is versus normal, a dimensionless number.
⭐ Raw gradient 100 or 0.01: after dividing, all share one scale. That's “adaptive learning rate”: relative to its own usual size, not absolute.
First, “moment.” In statistics, first moment = mean, second = mean square. Adam's v is a moving average of squared gradients; nothing more to “second.”
A picky detail: it's uncentered.
E[g²], not the variance (which subtracts the mean); Adam holds m, the mean, yet doesn't subtract it.
Not an oversight: it's the source of the “signal-to-noise ratio.”
Constant gradient: m = √v, ratio 1, full step. Divide by variance and it blows up: a constant gradient's variance is 0.
Unsubtracted, the ratio is bounded.
Geometry? There is one, just not the assumed one.
Picture the loss surface as a long, flat ellipsoid: gradients large in some directions, small in others. Adam rescales each axis by its typical gradient size until all directions compare.
Diagonal preconditioning, genuinely geometric—but it scales by how much a direction usually moves, not how sharply it curves.
So it fixes mismatched scales, not sharp curvature. 3.5's warmup hits exactly that limit.
⭐ Also: it scales the axes but never rotates them. Coupling between axes is out of reach. Muon and Shampoo later target that gap: neither scales per coordinate (3.4).
② RMSProp: the ruler goes stale.
AdaGrad's fatal flaw: the sum only grows; steps shrink until training stalls.
Hinton changed one thing: a moving average replaces the sum.
⭐ Why: a parameter's “typical gradient size” drifts. Large early, small late. All history as ruler = a stale ruler.
③ Adam: numerator and denominator didn't match.
RMSProp had a ruler (denominator), but the numerator was the raw, noisy gradient; averaging it too gives momentum.
⭐ Now both are recent statistics. Numerator: direction; denominator: scale.
The alignment brought a bonus. The paper calls this ratio a signal-to-noise ratio:
⭐ So it does two things: evens out sensitive and sluggish, and turns cautious when unsure. The second came free, undesigned.
⭐ Finally, both paths side by side: the real lesson.
Cauchy's path is geometric: curvature sets the step. Rigorous; too costly at hundreds of billions of params.
Adam's path is statistical: ignore terrain, read the parameter's own history (usual gradient size; is this one unusual?).
⭐ Both end up close from wholly different starts. Hence “adaptive learning rate,” not “approximate Newton method”.
The key step: Adam's update ≈ “momentum ÷ √(second moment),” so the units cancel.
📌 In full: m̂ / (√v̂ + ε); ε (typically 1e−8) guards against ÷0. Strictly, units cancel only at ε = 0; ε carries gradient units, the one leak in “independent of gradient scale.” Usually negligible, but for tiny gradients (e.g. deep-layer vanishing, 6.5) ε, not √v̂, is the denominator, and Adam degrades to plain SGD.
Adam divides out the gradient's units: the step becomes a constant, whatever the gradient scale.
⛔ The constant isn't 1. It's 0.2. My earlier error: cancelled units make it constant, not 1.
Measured, it holds at 0.2–0.3, the same across model sizes.
If the gradient is mostly noise, a too-neat closed form follows:
update RMS ≈ √[(1−β₁) / (1+β₁)]
β₁ = 0.9 gives 0.2294, matching the measurements.
📌 Source: Su Jianlin, “Why Is Adam's Update RMS 0.2?” kexue.fm/archives/11267 (Scientific Spaces, in Chinese). Simulation (pure Gaussian gradients: 0.225) and a mean-field approximation agree.
The formula has only β₁, no β₂. So the βs split: β₂ sets memory length, β₁ step size (more in 3.6).
So Adam's learning rate changes role: not a conversion factor, a multiplier on a constant. Each parameter moves 0.2 × lr per step.
Hence easy tuning: decoupled from loss scale and gradient size, one coefficient fixed by β₁.
Any optimizer scaled to update RMS 0.2 inherits Adam's tuned learning rate and weight decay. Kimi K2's Adam → Muon move did this: RMS set to 0.2, all else copied.
📌 Same source (Su Jianlin). The units thread's most practical payoff: make a quantity constant and it becomes an interface.
⭐ Source of the presentation: Su Jianlin, “Appreciating the Muon Optimizer: The Essential Leap from Vectors to Matrices,” Scientific Spaces (in Chinese), spaces.ac.cn/archives/10592. Ⓐ's three special cases, Ⓑ's trace example, Ⓒ's norm view and Ⓓ's idle-window explanation: all four come from that post. ⛔ The figure is our own redrawing, not a copy
⚠️ For Ⓒ's norm view, the post cites “Old Optimizer, New Norm: An Anthology”; ⭐ and traces the lineage: Stochastic Spectral Descent (2015) had proposed roughly the same algorithm
⛔ No convergence-speed comparison: we never measured these head to head (same rule as 3.1).
⚠️ Ⓓ's “under 1%” isn't from that post but the Muon author's blog (kellerjordan.github.io/posts/muon): FLOP overhead computed from Llama 405B's width and batch tokens, not wall-clock time we measured. ⛔ Newton–Schulz chains small dependent matmuls; fewer FLOPs isn't less time. Measure on your target config (our rule: don't copy numbers across scales).
⛔ This figure once said “within 5% / the author claims 2%”; that 2% is unsourced. The Muon author's blog: FLOP overhead under 1%. Fixed 2026-09-17.
🕰️ Aside: Adam's original (2014) convergence proof was shown wrong in 2018 (Reddi et al., ICLR 2018). Fix: AMSGrad. Everyone kept using Adam. Working isn't proven. Details below.
⛔ 🕰️ This “add one more thing” lineage has a rarely mentioned stumble.
The original Adam paper (2014) came with a convergence proof. In 2018, that proof was shown wrong.
Reddi, Kale and Kumar, “On the Convergence of Adam and Beyond” (ICLR 2018, arXiv 1904.09237). In the abstract's own words, they give “a simple convex optimization setting where Adam does not converge” and “describe the precise problems with the previous analysis of Adam algorithm.”
The root cause is the exponential moving average itself: v forgets old large gradients, so steps along some directions can fail to converge indefinitely. Their fix is AMSGrad: make v never decrease.
What happened next matters most: everyone kept using Adam.
AMSGrad barely helped in practice, so the “wrongly proven” algorithm is cited over 100,000 times and trained every large model today.
The lecture's core idea in miniature: working and proven are different things. Its running lesson: don't trust conclusions; check assumptions.
The figure covers most. Three points it misses:
Ⓐ's middle column: on a diagonal matrix Muon reduces to elementwise sign: the vector approach is a special case of the matrix one. Teaching them as two things inverts that.
Between gradient and update: all-reduce average across devices → clip once by global norm (against rare huge gradients) → schedule's current learning rate → weight decay → then the update.
The gradient says which way; the optimizer, how far.
Decades of evolution answer one thing: who decides “how far”?
That's the whole thread.
An optimizer is a memory decision, not just a convergence one.
Muon's saving has two limits, both named by its author.
⚠️ This hides two kinds of exclusion:
scalars and vectors are out in principle: orthogonalization needs a matrix with several singular directions;
on a vector it degenerates to plain normalization.
But the 2D embedding and output head could take Muon;
AdamW stays because it tests better:
an empirical exclusion, not a principled one.
No conflict with Ⓓ (“fits into idle time”): “slower” is direction, “idle time” magnitude (under 1% of FLOPs, per Muon's author). Magnitude is configuration-dependent: a chain of mutually dependent small matmuls, where few FLOPs ≠ short wall-clock. Measure on your setup.
The evidence is hard: Muon raised the NanoGPT speedrun record 35%; since then (as of his blog post), twelve records by seven people all use it. Match it with tuned AdamW and you'd retake the record. Nobody has.
Link to the last lecture: DeepSeek-V4 trained most modules with Muon, one of three major upgrades in its abstract (arXiv 2606.19348).
One line, not a menu: each generation treats the instability its predecessor left. Read it against the 6.0 failure-mode table; no memorizing.
New optimizer, new disease treated. The cost always lands in one place: extra bytes per parameter.
Below: how decay is applied. What it is, we've only touched on. Three questions settle it.
① What it is, and why it differs from the textbook
Each step shrinks weights slightly toward zero.
Textbook reason: regularization, no memorizing via a few huge weights (from ridge regression).
⛔ That fails for LLM pretraining: a single pass over data ≫ parameters leaves almost no room to overfit.
Yet every config sets it, always to 0.1:
weight_decay = 0.1 in all DeepSeek models (with β₁ 0.9 / β₂ 0.95 / clipping 1.0).
So what does it do?
A 2017 paper spelled it out: with normalization, L2 barely regularizes. It sets weight scale, which sets the effective LR.
Why: scale a weight matrix feeding a Norm by 2× and the output is unchanged (the Norm undoes it). Length doesn't change the function, only how far a given step turns the direction. Longer weights, smaller angle: lower effective LR.
Weights grow during training; left alone, the effective LR drifts down. Weight decay pins the length. It is now a knob that steadies the effective LR more than a regularizer.
📌 van Laarhoven, arXiv 1706.05350 “L2 Regularization versus Batch and Weight Normalization.” ⚠️ That paper is from the BatchNorm era. The same scale invariance applies to RMSNorm, but that's our extrapolation beyond its experiments (the rule in 2.6).
② How: by multiplying, not subtracting
θ ← θ − η·λ·θ = θ·(1 − η·λ)
Each step multiplies by just under 1: exponential decay. Without gradients, weights would go to zero.
How gentle? V3's peak η = 2.2×10⁻⁴, λ = 0.1 → 2.2×10⁻⁵ shrink per step; 80% remains after 10,000 steps. Very light.
⚠️ A trap: the shrink factor contains the learning rate. Change the LR schedule and regularization strength changes too: one knob turned, two moved. Hence in 3.3b's “order of one step,” decay comes after reading the scheduled LR.
In practice decay usually skips biases and norm gains. For a specific model, check its config; don't copy blindly.
③ Why it's in the optimizer: originally, it wasn't
It began as a loss term, ½·λ·‖θ‖²;
autodiff turns it into λθ in the gradient.
It was part of “what you optimize.”
Why move it? In the loss, it runs the gradient's pipeline: momentum, second moment, denominator, every stage. We want uniform pull.
Only the update step skips the denominator, and the optimizer owns it.
It went from “what you optimize” to “how you step”: new owner, same formula.
⚠️ Cost: out of the loss, you can't claim to minimize “loss plus a regularizer.” You no longer optimize the loss you wrote. Criticized, but it won in practice.
❓ The “W” in AdamW, pictured
Weight decay nudges weights toward zero each step so they don't grow. The question is at which stage.
The old way (original Adam): mix it into the gradient.
Add λθ to the gradient; both are subtracted anyway.
But every later stage then hits it, √v division included.
Backwards, as two parameters show:
| Both at 10 | √v | Decay term | Net shrink |
|---|---|---|---|
| Common: big gradients | Large | Scaled down | Barely pulled |
| Rare: tiny gradients | Small | Scaled up | Pulled hard |
Both at 10 deserve equal restraint, yet get the reverse. We want “pull every weight back a little, uniformly,” not “penalize by gradient size.” The two got tangled.
AdamW's fix: decay skips the denominator.
The gradient path is untouched; the update separately subtracts η·λ·θ.
That's “decoupled”; W is for weight decay.
❓ Muon drops the second moment entirely. Who now treats mismatched scales?
The list owes this: Adagrad names the disease; Muon throws out the medicine.
Orthogonalization treats it at another level.
Orthogonalization equalizes the momentum matrix's singular values (how hard it pushes along each direction). Each direction's step then ignores its gradient size: scale is erased all the same.
| Adam's v | Muon's orthogonalization | |
|---|---|---|
| Where it erases | Per element, each by its own | Whole matrix, singular values → 1 |
| Erases | Its own gradient scale | Gaps between singular directions |
| Cross-axis coupling | Out of reach (axis-aligned; 3.3b) | Within reach: singular directions span axes |
That's also the cost. Adam's v also flags a parameter acting oddly “lately”; drop v, lose that. Muon keeps momentum: smoothing stays, per-element adaptivity goes.
⚠️ Orthogonalization equalizes only between directions, not the overall step size. Hence Moonlight (below) adds an overall-size rule; and once Kimi K2 calibrated Muon's update RMS to 0.2 (Adam's constant), Adam's whole hyperparameter set carried over.
📌 Each link is from the Adam paper (arXiv 1412.6980, read in the original). Its AdaGrad update, verbatim: θ_{t+1} = θ_t − α·g_t / √(Σ g²): the denominator sums squares, not divided by t: hence monotonic decay. Adam is “designed to combine the advantages of” AdaGrad (sparse gradients) and RMSProp (on-line and non-stationary settings).
⭐ On bias correction, the paper says the moving averages start at 0, so the estimates are biased towards zero, especially during the initial timesteps, worse as β nears 1. RMSProp is named in §5 (relation to AdaGrad and RMSProp): “RMSProp also lacks a bias-correction term,” and without bias correction, like in RMSProp. Ⓒ's orange line draws that sentence.
⚠️ Hung-yi Lee's course slide uses an RMS Adagrad (divides by t+1), unlike the original here. ⛔ We use the original: the effective LR's monotonic decay holds only there (a sum not divided by t). Otherwise you'll think you miscalculated.
📌 Side by side, the three sources are a fun read (full story in the instructor notes). AdaGrad is a long JMLR paper (Duchi/Hazan/Singer, 12(61):2121–2159, 2011, 39 pages). ⭐ Its title: “Adaptive Subgradient Methods for Online Learning and Stochastic Optimization”: never written for deep learning.
⭐⭐⭐ RMSProp has no paper. Its only “publication” is a slide from Hinton's Coursera course, cited as Tieleman & Hinton (2012), Lecture 6.5-rmsprop. ⭐ The slide's subtitle is itself the complete definition of the algorithm: “Divide the gradient by a running average of its recent magnitude.” One sentence became the middle link of this chain.
⭐⭐ The Adam paper (ICLR 2015) has two delightful details. The first-page footnote reads “Author ordering determined by coin flip over a Google Hangout.” The acknowledgments read “special thanks to Ivo Danihelka, and Tom Schaul for coining the name Adam”: the authors didn't name Adam themselves.
⭐ The three numbers (4.1%, 32×, 6.6×) are computed by a script with five asserts: AdaGrad decays monotonically; it's tiny by step 600; uncorrected, the first step is amplified many-fold; corrected, exactly 1 from step one; uncorrected Adam peaks below RMSProp (the momentum term's bias partly cancels).
The lineage above is hindsight. Its inventors were thinking of something else.
📌 AdaGrad (2011): the motivation was sparse features, not stability.
The paper: “infrequently occurring features are highly informative and discriminative,” so they get high learning rates, frequent ones low; “each time an infrequent feature is seen, the learner should ‘take notice.’”
Say “the” appears millions of times, a technical term three times. Under one learning rate, the term never learns. AdaGrad: discount each parameter by its accumulated gradient.
📌 Duchi, Hazan, Singer, JMLR 2011 (preliminary version at COLT 2010). The paper opens with TF-IDF as motivation (“We use this old idea as a motivation”), search's rarer-words-matter-more rule: an old information-retrieval idea, not curvature; the theory came later, as a guarantee.
📌 Adam (2014): the “equalize the size of every step” explanation starts here.
§2.1: the “effective magnitude of the steps … [is] approximately bounded by the stepsize setting α,” a “trust region … beyond which the current gradient estimate does not provide sufficient information.”
Plainly: whatever the gradient, every step is about the same size. Small gradients get lifted, large ones pressed down. That's where 3.3b's “0.2” comes from.
The paper calls m̂/√v̂ the signal-to-noise ratio:
the smaller the gradient vs. its noise, the nearer the step is to zero.
📌 Kingma & Ba, arXiv 1412.6980 (ICLR 2015).
The middle generation, RMSProp, comes from Hinton's Coursera slides: no formal paper.
⛔ While we're at it: “SGD doesn't work” overstates it.
SGD + momentum still competes on vision; on LMs Adam wins systematically. The lineage lists diseases treated, not useless predecessors.
The lesson is the mismatch: AdaGrad was built so rare words get heard; three years later Adam reread it as “equalizing step sizes,” the only framing we use today.
Rule of thumb: why a method was invented and why it stuck often differ. Tell only the second; people assume inventors knew. Usually they didn't.
Each changes a different box of the 16 bytes. Bytes per 3.1's accounting:
| Method | Box changed | Bytes per param | Cost |
|---|---|---|---|
| AdamW (baseline) | — | 2 + 2 + 4 + 4 + 4 = 16 | — |
| Muon | Drops the second moment | 2 + 2 + 4 + 4 = 12 | 2D only; made up by orthogonalization |
| Lion | Also first moment only | 2 + 2 + 4 + 4 = 12 | Sign of the gradient only |
| Adafactor | Factors v into a row + a column | That copy ≈ 0; 8 total, no first moment | Loses v's per-element info |
| 8-bit Adam | Both quantized to 1 byte | 2 + 2 + 4 + 1 + 1 = 10 | New storage, same algorithm |
Three different approaches: Muon and Lion drop the copy; Adafactor uses a cheaper representation; 8-bit stores the same thing in fewer bits. So 8-bit stacks with the first two, which exclude each other.
📌 8-bit Adam quantizes states in small independent blocks, with nonlinear quantization precise for large and small values alike. Reported: 32-bit results across tasks, hyperparameters unchanged; a two-line drop-in swap.
📌 Dettmers et al., arXiv 2110.02861 (ICLR 2022 spotlight).
But “same algorithm, different storage” isn't quite true: per the abstract, 32-bit parity needs three things together: block-wise and nonlinear quantization plus a “stable embedding layer.”
The third changes the architecture, not storage. LM input tokens are very uneven, so embedding gradients vary wildly: quantization's worst case.
Rule of thumb: for a “drop-in replacement,” count in the abstract what it changes. “Two lines” is for the caller.
3.3b gave Muon's idea; good ideas don't automatically scale. A 2025 paper added weight decay (absent originally; the AdamW entry's disease) and a separate overall update-size rule, since orthogonalization only equalizes across directions.
With both, it works out of the box. Its scaling laws report Muon at ~2× AdamW's compute efficiency (compute-optimal).
📌 arXiv 2502.16982 (Moonlight, 3B/16B MoE, 5.7T tokens). “∼2× computational efficiency” is the paper's scaling-law result, not ours. Per 2.6, re-verify at other scales. Unverified: whether Muon still needs fp32 master weights.
First, a ruler: not the optimum, just whether you're way off.
📌 CS231n, “Neural Networks Part 3,” “Ratio of weights:updates”: “A rough heuristic is that this ratio should be somewhere around 1e-3.” ⭐ The source also stresses “Note: updates, not the raw gradients”.
⚠️ Our own curves: two-layer tanh MLP (8→16→4), hand-written numpy forward/backward, fixed seed, 400 steps. The “right” LR was swept (steady-state ratio closest to 1e−3), not picked; “×2.1 again gives NaN” came from bisection.
Only three decisions: where it starts, how high it goes, how it comes down.
GPT-3: arXiv 2005.14165 App. B: “linear LR warmup over the first 375 million tokens,” “cosine decay for learning rate down to 10% of its value, over 260 billion tokens (after 260 billion tokens, training continues at 10% of the original learning rate)”; 300B total
DeepSeek-V3: arXiv 2412.19437 §4.2: linear to 2.2e-4 over 2,000 steps → flat to 10T → cosine to 2.2e-5 over 4.3T → final 500B: first 333B there, last 167B at 7.3e-6; clipping norm 1.0
⭐ What warmup actually does (not “too few samples to estimate”): arXiv 2406.09405 (NeurIPS 2024). Main benefit: a larger tolerable target LR. Systematic analysis of this shape: arXiv 2410.05192
⛔ This figure has no absolute learning rates (peaks differ 3.7×; overlaid, shapes vanish). Eight-size peak table: main text
The shapes differ completely: GPT-3 slides down after warmup; V3 holds a long, flat plateau. Plateau or not is the camps' only disagreement (third decision). Each decision below has a public config to copy, not guess.
Tutorials say: early Adam's second moment is under-sampled, so ease in.
⛔ A systematic 2024 study says plainly this is not the main reason.
Its finding: warmup's overwhelming benefit is tolerating a larger target learning rate, by moving the network to a better-conditioned region. It rebuts “too few samples” directly: preconditioned curvature starts high, as high at large batch.
So warmup's real value: a higher peak, and an easier-to-tune one (wider usable range).
📌 Source: Why Warmup the Learning Rate? Underlying Mechanisms and Improvements, arXiv 2406.09405 (NeurIPS 2024).
“Says plainly … not the main reason” overstates it. The paper's Limitations admit small experiments: a 4-layer Transformer with context 64, plus a WRN on CIFAR-10. Orders of magnitude below this lecture's scale.
So why can the peak go higher? Back to 1.4b's second derivative; otherwise “better-conditioned region” is a tautology.
Curvature, not you, sets the largest step
On a parabola, gradient descent diverges once the step exceeds 2 ÷ curvature. Not slower: each step overshoots further, and it flies off.
So local curvature caps the usable learning rate: higher curvature, smaller legal step.
The paper measured high preconditioned curvature early in training, so high your intended peak isn't legal. It's as high at large batch: their evidence against “too few samples,” since a bigger batch would fix estimation noise.
So warmup is concrete: run at a safe learning rate and let the network walk to lower curvature. Once curvature drops, the peak is legal.
Causality: the surface flattens; the network doesn't learn to tolerate big steps. Hence longer warmup buys almost nothing: once you're there, more steps barely lower curvature.
❓ “Starting at a 1-in-129,280 guess, aren't gradients huge?”
A large loss isn't a large gradient.
The initial loss, ln 129,280 = 11.77, is big.
But in 1.2b's figure the logits-layer gradient is p − y:
every entry lies between −1 and 1, the vector's length at most √2.
Vocabulary 129,280 or 1,000: no difference.
❓ “Doesn't Adam normalize the gradient? What's left to be large?”
Good question; it points at the answer. Adam divides momentum by the gradient's RMS; units cancel, the ratio is O(1). So step size ≈ the learning rate, nearly independent of the gradient.
So the danger isn't early large gradients. What's large is the ground's curvature, not your step.
Take a step η. On gentle terrain the slope barely changes. On sharp terrain the same step overshoots the bottom, landing higher on the far side; each next step flies further: the 2 ÷ curvature line, live.
1.4b's point: Adam divides out “steep,” not “curved”. Its second moment is a statistic; curvature, a second derivative, is geometry it never computes. “Preconditioned curvature” means curvature after Adam's rescaling. Still high after rescaling? That's exactly what was measured.
❓ “Hasn't the denominator just not built up yet?”
Adam guards against that with bias correction:
both moments are divided by 1 − βt to offset the short early history.
No “zero denominator.”
“Noisy early estimates, so ease in” is what the paper rejected. Evidence: curvature is as high at large batch, which would fix estimation noise.
Don't push this too far: it doesn't mean “curvature keeps falling, so the learning rate can keep rising.” The curve stops at the peak, then falls (Phases 2–3). Warmup gets you to the peak, no further.
A third explanation exists; its premise keeps it out of modern models. New angle: Adam changes what “vanishing gradients” means.
An Adam update, as the source writes it:
Δθ = −η · E[g] ⁄ √E[g²]
Top and bottom share units, so the ratio is O(1) and the update O(η). Unlike SGD, where step ∝ gradient: small gradient, small step.
With Adam, vanishing gradients stop meaning “can't learn”: while the gradient beats noise, updates stay O(1).
⛔ But “no longer stuck” isn't good news: it trades the problem for a worse one.
At noise-level gradients, top and bottom are both noise, and the ratio is still O(1).
⭐ Full-sized steps; only the direction is random.
SGD just stalls in this regime: ugly, but safe. Adam charges off in a random direction at full step size.
That's where the trouble starts. The source's reasoning (premise: real, marked vanishing gradients):
Per the source, warmup “slows the later layers' learning and gives the earlier layers more time to optimize, so all layers optimize in step”.
One symptom to match against your logs: Post-LN without warmup, the loss settles near a constant, then later diverges to NaN.
But keep its premise attached, stated plainly by the source:
“This assumes vanishing gradients. For Pre Norm-like structures, with no notable vanishing, training often works without warmup.”
So each has its own slot; they don't compete.
| Claim | Premise | Status now |
|---|---|---|
| Second moment under-sampled | None stated | Refuted by the 2024 systematic study |
| Lets the network take a larger peak LR | General case | Holds, but tested only to 4 layers |
| Hold later layers until earlier ones catch up | Clear vanishing gradients (i.e., Post-LN) |
Holds, with an observable symptom |
The question isn't “which is right” but “under what premise.” Pre-LN or Post-LN decides which applies. Modern LLMs are almost all Pre-LN (§6.5), so the second is more relevant; but on a Post-LN model showing “loss flat, then NaN,” it's the third.
📌 Source: Su Jianlin, “Notes on model optimization: why is BERT's initial standard deviation 0.02?” (in Chinese), kexue.fm/archives/8747. Both quotes are his (our translation). It is his own analysis, not peer-reviewed; we take it as a premised explanation, not a settled result.
An ironic confession: 2.6 set “re-verify across scales,” then I broke it here, writing “they disproved it” with no scale. Rules are easy to write, hard to apply to your own citations.
How long? GPT-3: the first 375M tokens (0.125% of 300B); V3: the first 2,000 steps. Too short to count as a hyperparameter.
GPT-3's Table 2.1 lists peak learning rate and batch for eight sizes: a free prior, no sweep:
📌 GPT-3 family (same data and recipe; only size varies):
The pattern: bigger model, smaller learning rate, larger batch. Mildly: 1,400× in size moves the learning rate only 10× (6.0e-4 at 125M → 0.6e-4 at 175B).
The paper: “larger models can typically use a larger batch size, but require a smaller learning rate.” They set batch by the gradient noise scale.
Batch and learning rate change together here (see the section's end). GPT-3 also ramps batch linearly from 32K tokens to target over the first 4–12B tokens. 📌 Source: arXiv 2005.14165 Table 2.1, Appendix B; the smaller-LR line is §2.3.
“Bigger batch, higher learning rate” has a ceiling many people miss.
First, the two common rules:
η* ≈ ηmax /(1 + Bnoise/B).
Monotonic and bounded:
however big the batch, LR only approaches ηmax.So “double the batch, double the learning rate” is wrong.
Stranger still: under Adam, past some batch size the optimal learning rate may fall (the surge phenomenon). Cause, premise, dispute: in the fold below. Main line: don't blindly extrapolate any scaling law.
Under Adam, stranger still: past some batch size, the optimal learning rate falls, not rises: the surge phenomenon.
⛔ Why would that happen? Intuitively: adaptive learning rates are themselves suboptimal.
Treat Adam roughly as sign(g): the larger the batch, the closer
sign(g̃) gets to the true sign(g).
But is sign(g) the best direction? Not necessarily,
especially late in training.
At a moderate batch, a little noise corrects that for you;
enlarge the batch, the noise goes, and so does the correction:
lower the learning rate more cautiously.
But this has a premise; the author deliberately added a caveat:
“(Of course, one more restriction: β is always below 1. If βnoise ≥ 1, the optimal learning rate still rises monotonically with batch size.)”
Su Jianlin faults the paper: its “the surge almost always appears” rests on a strong approximation, not very scientific; his more reasonable one gives “even if the Hessian's off-diagonal terms aren't negligible, the surge need not appear”.
📌 Su Jianlin, “How should the learning rate change as batch size grows?” (in Chinese) kexue.fm/archives/10542; on “Surge Phenomenon in Optimal Learning Rate and Batch Size Scaling.” Quotes: his words (our translation). Same caveat (his own analysis, not peer-reviewed): we take the warning “don't blindly extrapolate,” not a formula.
An apparent modern outlier: DeepSeek-V3, a 671B MoE, peak 2.2×10⁻⁴: far larger than GPT-3 175B, yet nearly 4× the learning rate.
⛔ My first explanation, which the table refuted: “V3 activates only 37B per token, so treat it as 37B.” But 13B is 1.0×10⁻⁴ and 175B 0.6×10⁻⁴, so 37B should sit near 0.8; V3 is 2.2. The new yardstick only shrank the gap from 3.7× to 2.8×.
The real answer: these numbers aren't comparable.
Rule: a trend holds only inside the controlled experiment that made it.
GPT-3's table is a prior because its eight sizes share one recipe, one knob moving,
which is also why it's no prior outside it.
Confession: I first papered over it with “active size, not total.”
The most dangerous claim: sounds like common sense, happens to fit.
Old school · cosine (GPT-3 / Llama / Mistral): cosine decay to 10% of peak over 260B tokens, then held there.
New school · warmup-stable-decay (DeepSeek-V3 has this shape)
V3's curve: 2,000 steps of linear rise to 2.2×10⁻⁴ → held constant until 10T tokens before decaying → 4.3T tokens of cosine decay to 2.2×10⁻⁵ → of the final 500B tokens, the first 333B stay at 2.2×10⁻⁵ and the last 167B switch to 7.3×10⁻⁶.
📌 arXiv 2412.19437 §4.2; clipping norm 1.0; batch 3,072 → 15,360 in the first 469B tokens, then held. A systematic analysis: arXiv 2410.05192.
The schools differ not in convergence but in when each forces you to decide.
⭐ The choice: “can I say now how many tokens I'll train?” Yes: cosine is easier. No: the plateau is your escape hatch.
Both schools end on the same number: the dashed line across the figure.
Gradient clipping is a seatbelt, not a knob. V3 uses 1.0. Clipping to tame a too-high learning rate masks the problem, not solves it.
⛔ Easiest to overlook: a learning rate alone means nothing.
Report it with batch size, schedule shape, and total tokens; change one and it no longer applies. The two configs above are copyable because they spell out all four.
3.5 ended: “a learning rate alone means nothing; report it with batch.” So what sets the batch?
“Batch” as one number misleads. A real run has four independent knobs:
| Knob | What it is | Raising it |
|---|---|---|
| micro-batch m |
Sequences per device per forward | Activations grow (1.7's table ×m) |
| Grad accum K |
Passes per update (1.8, stage ④) | Activations unchanged: only one set ever live |
| DP degree d |
Replicas, each fed its own data | Activations thin out; each replica holds all 16 B resident |
| Sequence length S |
Lecture baseline: 4,096 | Activations ∝ total tokens (5.1) |
global batch (tokens seen per update) = m × K × d × S
⛔ The link that's easiest to miss: raise TP and the global batch shrinks.
Devices are fixed, so every model-splitting dimension competes with DP:
d = total devices ÷ (TP × PP × CP)
Raise TP from 8 to 16 and the DP degree halves; all else unchanged, so does the global batch.
The formula shouldn't contain EP (I corrected this once).
Megatron-Core has two DP sizes:
data_parallel_size = world_size ÷ (TP × PP × CP) and
expert_data_parallel_size = world_size ÷ (ETP × EP × PP).
EP divides only the second: MoE reshuffles the same tokens across experts;
the step sees no fewer. CP does sit in the denominator (missing before).
📌 megatron/core/parallel_state.py:
model_size = TP × PP × CP × gtp_remat.
Batch changes, learning rate follows: the parallelism knob is wired to convergence. It only looks like model splitting.
“Same code, same data, new parallelism config, different loss curve”: this is why.
The gradient, like the loss, averages each sample's opinion (1.0).
Recall the “averaging a roomful of people” figure (1.2).
8 people in the room, one extremist: he drags the average far.
800 people: his influence is diluted to almost nothing.
Gradient “jitter” shrinks roughly as 1/√B: 4× the batch, half the noise.
📐 Sample-mean result (standard error ∝ 1/√n, independent samples). It assumes rough independence; adjacent tokens aren't, so effective samples are fewer than tokens. Order of magnitude only.
Conversely, a small batch is pushed around by noise every step: fuzzy loss, jumpy grad norm; many Section 6 spikes are lit this way by a few bad samples.
If bigger is steadier, why not enormous? The next two points sound separate but are one.
Side one · Noise falls, but so does “payoff per sample.”
While the gradient is rough, more samples sharpen it: 2× batch ≈ half the steps. Once accurate, noise isn't the bottleneck: steps barely drop, compute doubles.
The turning point is the “critical batch size”; past it, compute buys sub-proportional progress.
It isn't fixed: it grows with model size and training progress. So real configs ramp the batch: V3 from 3,072 sequences to 15,360 (last subsection's table). The ramp follows the critical point up; not for speed.
❓ Does a ramping batch mean adding devices? No. What ramps is gradient accumulation.
Recall Step 1's product:
global batch = micro-batch × grad accumulation K × DP degree × seq length.
For 5×, raise K from 1 to 5. Not one device added.
⭐ Nor does memory grow: per the accumulation row, “activations unchanged: only one set ever live.” Accumulation just postpones the update.
So what's the cost? Same devices, same tokens per second; only the tokens accumulated per update change. Fewer updates, same total time.
⭐ Ramping costs neither devices nor time; it costs update frequency. That's why it pays: early, the critical batch is small and big batches waste; later it has grown, so fewer updates cost almost no sample efficiency, buying fewer all-reduces (smaller communication share) and less noise, more stability.
📌 Three sources for batch ramp-up: mechanism, practice, framework.
--rampup-batch-size <start batch size> <batch size increment> <ramp-up samples>.
Help-text example: 16 to 1024, linear, in 126 increments.
It also wants --global-batch-size (the target),
and never asks your device count.⚠️ Assumptions: the V3 sentence is from the v2 main text (near §4.2); the Megatron flag
lives in core_r0.9.0's arguments.py;
on main it's now --step-batch-size-schedule: same mechanism, new name.
Side two · A bigger batch needs a higher LR, which has a ceiling.
Less noise warrants bigger steps, or the accuracy is wasted. 3.5's eight sizes look opposite (bigger batch, smaller LR) because scale pushes it down too.
But the learning rate can't rise forever: too far and the loss blows up (Section 6). The batch speedup is capped by that ceiling.
⛔ Together, the two sides are one constraint.
Go back to the formula from 3.5:
η* ≈ ηmax ⁄ (1 + Bnoise/B)
Side one's critical batch, quantified, is Bnoise (3.5's “they set batch size using the gradient noise scale”). Side two's ceiling has the same Bnoise in its denominator.
So both turning points coincide, with an easy anchor: at B = Bnoise, η* = half of ηmax: at the critical batch you've spent half your learning-rate budget. More batch only creeps toward the rest, never reaching it.
This box once said “two independent reasons.” Wrong. “Independent” suggests dodging one, say by raising the ceiling a bit. You can't: ηmax is the divergence threshold (3.3b's “2 ÷ curvature”), not a knob.
The two “contradictory” claims are segments of one curve.
| Bigger batch | |
|---|---|
| Stability | ✅ Monotonic: noise ∝ 1/√B; no turning point |
| Sample efficiency | Has a turning point: past the critical batch, returns diminish |
Both right, one row each. Choosing a batch means sitting between: steady enough, not yet wasteful.
Above: “bigger is steadier,” “too big stops helping.” The most widespread third: “bigger is faster”.
📌 “With parallelism, small and big batches take similar time per step; per epoch, reversed”: Hung-yi Lee, 2021 batch/momentum lecture, pt. 2 (Chinese; our translation). ⛔ Our redraw.
⚠️ No measured numbers, only shapes. The kink moves with model, devices, parallelism; measure on your target config.
This claim has a premise, and it breaks at your scale.
Left: true. Unsaturated devices: fewer steps, none costlier. Really faster.
⛔ Right: false. Saturated devices: costlier steps exactly cancel fewer steps.
Tens of billions of parameters on hundreds of devices sits almost all on the right. For you, a bigger batch buys stability and thinner communication, not speed.
The math for “thinner communication” is in 1.8: cross-device reduction doesn't grow with batch; compute does, linearly, so it thins out.
⛔ ① Forget to divide by K when accumulating = K× the learning rate.
1.8's conclusion assumes you actually took the average.
Sum K micro-batch gradients, then divide by K. Forget it and the gradient is K× too large, same as K× the learning rate.
The symptom misleads: no errors, loss falls at first, then suddenly blows up, and you suspect data or initialization. It's a learning-rate problem in disguise.
② With variable lengths, per-token and per-sequence averages differ.
One batch may mix 4,096-token and 500-token sequences (padding excluded).
They give different gradients, one line of code apart. Pretraining almost always averages per token; the point is knowing which you picked.
The live session mentioned something like “TPS”: “when TP is too large, TPS gets very large.” But by Step 1's formula, raising TP lowers DP and shrinks the global batch; of three readings (tokens per step / TP size / tokens per sequence), none fits, so no guess; skeleton only. Rule: don't complete an acronym you couldn't hear; a wrong guess teaches a relation that doesn't exist.
Takeaway: batch size is capped by sample efficiency, not memory. Memory decides whether it fits, not whether it should.
Two numbers remain: β₁ and β₂. Usually left at defaults, yet their meaning is clear.
In one line: β sets how long old terms live.
A moving average scales by β and adds the new value: window ~1/(1−β) steps.
Higher β₂: steadier denominator, slower reaction; lower: tighter tracking, more noise. No magic: a memory-length knob.
It uses β₂ = 1 − k−0.8, where k is the current step.
Plug in a few numbers:
The window grows with training (roughly as k0.8).
A fixed β₂ = 0.99 remembers only 100 steps throughout.
❓ Why? The paper's reason is concrete.
The culprit: rare embedding tokens, which may appear once in thousands of steps. In a 100-step window their second-moment estimate is garbage: mostly zero, occasionally large.
A window that grows with training fixes it. The paper: “We have found this to be more stable than the standard β2 = 0.99.”
📌 arXiv 2204.02311, hyperparameters; also
β₁ = 0.9, LR 10⁻² for the first 10,000 steps, then 1/√k decay.
⭐ Pause: you've seen this reason once already.
AdaGrad (2011) was built for exactly 3.3b's reason: rare features don't get learned.
PaLM (2022), eleven years later via another door (β₂'s memory window), hit the same problem: rare tokens are too sparse for short-window statistics to be trusted.
One gives bigger steps, the other longer memory: two doors to one problem.
⛔ But they differ: AdaGrad changes the denominator's size (bigger steps for rare parameters); PaLM, its sampling window (so noise doesn't dominate their denominators). Same disease, different medicine.
This collides with the rule in 3.2.
3.2 said: momentum and second moment are moving averages, old contributions decay, so bf16 copes.
But at β₂ = 0.9999 the “decay” barely decays;
v approaches a true accumulator.
So that rule needs a premise: β doesn't approach 1 over training. Our inference, not the paper's: PaLM uses an Adafactor-family optimizer and reports no precision config, so there's no evidence the effect shows up. A conjecture.
PaLM runs unfactorized Adafactor, per the paper “effectively equivalent to Adam with ‘parameter scaling’”: LR scaled by matrix RMS.
The tie to 3.5: “the effect of this is similar to the manual scaling down of Adam learning rate as in Brown et al. (2020)” (GPT-3).
3.5's eight configs are the manual fix; parameter scaling the automatic one. Bonus, per the paper: parameters with inherently different scales (embeddings, layer-norm scales) aren't scaled down at the same rate.
This follows directly from the bill in 3.1; no documentation needed.
Of those 16 bytes, which must go into the checkpoint? Ask, slot by slot, “if it's lost, can it be rebuilt?”
So checkpoint size = those 12 bytes × parameter count. For 671B: about 7.3 TiB per copy.
But don't pin this on V3: per 3.2 both its moments are bf16: 4+2+2 = 8 bytes, 4.9 TiB per copy, a third below the classic count. Whatever saves optimizer state saves checkpoint size too.
Two engineering facts follow:
⛔ These 12 bytes aren't enough. Miss any of three more things and “resume” becomes “restart.”
The “roll back + skip data” rescue in §6.2 depends on ① and ②. Without the data position, you can't say which batches to skip.
A real saving: the optimizer part needn't live on the accelerator. DeepSeek-V3 keeps an exponential moving average of the weights in CPU memory, updated async each step (no HBM, no training time), to preview quality after LR decay. 📌 arXiv 2412.19437 §3.2.3.
So far we trained from scratch. Most people fine-tune someone else's trained model.
No new concepts: the same 16-byte budget, redone for a different algorithm.
The pain: you can't store a 175B copy per customer.
The abstract opens with it: fully fine-tuning GPT-3 175B means a separate 175B-parameter instance per downstream task, “prohibitively expensive.”
⭐ The pain is in deployment, not in training. LoRA was built for “one base, hundreds of small add-ons”; saving training memory is a bonus.
Then the observation that prompted it.
Earlier work found that large models have huge parameter counts but a far lower effective dimension: the parameters sit in a thin subspace.
⭐ LoRA's leap of faith went one step further: if the model is low-dimensional, is the change from fine-tuning low-dimensional too?
A hypothesis, not a theorem. If it holds, the method writes itself: a low-rank change doesn't need a full-rank matrix; two skinny matrices multiplied suffice.
⭐ The shape of the reasoning: they didn't prove the change is low-rank. They built on the hypothesis, then measured it (the paper's “empirical investigation into rank-deficiency in language model adaptation”). Bet first, verify second.
📌 Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, arXiv 2106.09685. The abstract: 10,000× fewer trainable parameters, 3× less GPU memory, and “unlike adapters, no additional inference latency.” That last point is how it beat earlier methods.
📌 The method in one sentence: freeze the base and attach a pair of skinny matrices beside it.
The 4096 × 4096 matrix stays frozen. Add a 4096 × 16 and a 16 × 4096; train only those.
The key is the 16: 2 × 4096 × 16 values, just 0.8% of the original.
| Budget | Full fine-tuning | LoRA | Saved? |
|---|---|---|---|
| Resident weights + grads + optimizer |
16 B × all params | 2 B × all (frozen, read-only) + 16 B × the 0.1% |
Biggest saving |
| Compute | 3× (fwd 1 + bwd 2) | ≈ 2× | A third |
| Activations | whole chain | still whole chain | Barely |
Row one at 7B: dramatic.
(Llama-2-7B class: 32 layers, hidden 4,096; 32 × 2 sites (Q, V) × 2 matrices (A, B) × 4,096 × 16 = 8,388,608.)
A 7.9× difference, from an adapter of only 8.39M parameters, 0.124% of the model.
Why so large: 14 of the 16 bytes (gradient + master weight + m + v) serve only updated parameters. A frozen one keeps just its 2-byte weight, as at inference.
Row two, “3× → 2×,” comes from 1.3.
Per 1.3 (① is forward), backward costs two matmuls: ② the weight gradient (for the optimizer), ③ the input gradient (passed down).
Frozen layers skip ② entirely: their weights never change, so nobody needs it.
Forward (1) plus ③ (1) gives about 2×, not 3×.
Don't confuse the reverse: tuning just the last layer saves ③ (nothing flows back). LoRA has adapters in every layer, so ③ remains; it saves ②.
Row three: most misunderstood, most valuable.
“Training 0.1% of parameters = light training”? Not in memory.
Adapters sit on every layer. The first layer's adapter needs a gradient, which only flows back from the loss (1.5). So the whole chain's activations stay.
After LoRA, activations are often the largest item, so it still needs recomputation (Section 2). The two stack; neither replaces the other.
Strictly, activations shrink a little, depending on where adapters sit. Rarely spelled out:
A reads the same input, and A's gradient needs it.Counterintuitive corollary: the more adapters, the less activation memory saved. “Q and V only” saves a little: none of the three big MLP matrices has one. “More adapters work better” costs activations, not parameters.
Nonlinear ops' inputs still stay, so the total shrinks a little, not by an order of magnitude. Derived from 1.3's two matmuls; not measured.
⭐⭐⭐ Wrap-up: LoRA changes how many parameters carry 16 bytes, not that backward traverses the whole network. The parameter side shrinks; the chain doesn't: it comes from backprop, not from how many parameters you train.
A bridge to Topic 5. Rank the items by size: ZeRO's three stages follow that order.
A Topic 5 preview, not one of our three things. Three sentences; details collapsed below.
What training needs, by size:
Optimizer state (12 B/param) > weights = gradients (2 B each) > activations (batch-dependent)
Takeaway: parallelism isn't invented; this bill's size ranking forces it.
Put the items from the last three sections side by side and rank them by size. That ranking is this whole section.
| Item | /param | 671B | Share | What it is | When used |
|---|---|---|---|---|---|
| Optimizer state | 12 B | 7.32 TiB | 75% | fp32 master 4, moments m 4, v 4 | Once, at the update instant |
| Weights | 2 B | 1.22 TiB | 12.5% | bf16 copy; both passes use it | Read by every layer |
| Gradients | 2 B | 1.22 TiB | 12.5% | bf16, shaped like the weights | Written in backward; dropped after update |
| Resident total | 16 B | 9.76 TiB | 100% | ⭐ Independent of batch and seq. length: the model fixes it. | |
| Activations | — | By batch | — | Saved for backward (1.6) | ⛔ Only one you control; only one that can outgrow the rest |
The first three rows are bytes per parameter × parameter count, so the ratio is always 12 : 2 : 2, whatever the model or device count.
Optimizer state alone is 75% of the resident total, yet the only one neither pass uses: 3.1 showed it is read and written once, at the update instant.
⛔ The fourth row is different: it is the only one you control.
The first three follow the parameter count; no choice. Activations follow the tokens live at once this step: micro-batch × grad accum × DP degree × seq. length, four knobs multiplied.
So “which item is largest” is a property of the training configuration, not of the model. Push the batch far enough and the fourth row passes the first; 5.1 finds the crossover.
Remember this order: next, ZeRO works down it one stage at a time.
No need to memorize ZeRO's stages: reverse the order above and you've derived them.
📌 arXiv 1910.02054 §3 (16-byte convention, Figure 1's four numbers), §7.2 (comm volume 2Ψ / 2Ψ / 3Ψ).
⭐ The four GB figures are computed (7.5B × bytes/param), script-asserted vs. the paper's Figure 1 (120 / 31.4 / 16.6 / 1.9 GB).
One rule sets the order: shard the largest, least-used item first.
So the order is no historical accident: it falls out of size and frequency of use.
📌 Source: Rajbhandari et al., “ZeRO: Memory Optimizations Toward Training Trillion Parameter Models,” arXiv 1910.02054. Section 3's “16 bytes per parameter” comes from it too: one paper gave both the bill and how to split it.
Place the other techniques you keep hearing about on this bill and each one's place becomes obvious:
⭐⭐⭐ These are not “a few optimization tricks” but countermeasures for four different columns of the same bill. All of Topic 5 is about what the sharding itself costs.
All correct, but all at 671B parameters and TiB scale, with no reference point. Now shrink it to a model you can work out yourself.
Forward (Topic 1), backward and update in one table. The useful question isn't “how much” but “when is the peak?”
⭐ Ⓑ's three rows are computed live: 671e9 × 16 B ÷ 1024⁴ = 9.76 TiB; 9.76 TiB × 1024 ÷ 162.14 GiB = 61.7 sequences. Script-asserted: a changed number fails
⚠️ 16 B/param is the classic ZeRO-paper convention (arXiv 1910.02054); DeepSeek-V3 keeps both moments in bf16, so its per-param bill differs (3.2)
⚠️ 162.14 GiB of activations: our own per-op estimate, unchecked by others; “62 sequences” is a magnitude, not a threshold
⛔ The figure shows global totals, with no sharding. Spreading it over devices is Topic 5
tools/manim/.)Four sections, four separate accounts; now add them up. Global totals, no per-device split (next lecture's job).
📌 The resident block fits in your head:
671e9 params × 16 bytes ÷ 1024⁴ = 9.76 TiB. Broken down: weights 1.22 + gradients 1.22 + optimizer state 7.32.
For scale: at 80 GiB per device, this block alone needs 125 devices, before activations or any redundancy. The activation peak below assumes full recompute; per 2.4, V3 itself recomputes selectively (RMSNorm + MLA up-projections). Different bases: full is a clean floor; reality runs higher. (80 GiB: just a round base, not a product. Topic 1 uses v7's 94.74 GiB/device: about 105 devices. Same TiB, other divisor.)
Of these three, only the optimizer state's 7.32 TiB is pure overhead: no part in forward or backward, used once at the end of each step.
Follow that number one step further: a surprise.
The resident 9.76 TiB is fixed, whatever the sequences. How large must activations grow to catch up?
About 8.08M tokens. At this lecture's 4K baseline, about 1,973 sequences. (Full recomputation: one 4K sequence peaks at 5.07 GiB; one 128K sequence (context extension), 162.14 GiB: about 62 sequences, same token count.)
Stop: this number is easy to plug in wrong. I did, in one draft.
3.5 says V3's global batch starts at 3,072 sequences of 4K (= 12.58M tokens) and later ramps to 15,360. That looks far past the 8.08M crossover.
Global batch won't do: the crossover counts what's live at once.
global batch = DP degree × micro-batch × grad-accum K × seq length, and the K slices never coexist (point ④, 1.8). Only the first two are live at once, as in panel Ⓑ.
A safe upper bound: V3 used 2,048 devices with PP16 × EP64. Most generous DP, 2048 ÷ 16 = 128-way, micro-batch 1: only 128 sequences = 520K tokens live at once, under a fourteenth of the crossover.
So “activations overtake optimizer state from step one” is false: withdrawn. On V3's own config, 3.1's “the largest block is optimizer state” still holds.
This passage once said “V3 passes the crossover 1.7× from step one, 8.5× late in training” (vs. the old 7.37M-token crossover; today's 8.08M gives 1.56× and 7.8×), as if the example overturned the first answer. That treated global batch as live at once. Yet the figure was right all along: Ⓑ reads “live at once = DP degree × micro-batch.” Rule: when figure and text state one quantity, first check they share a definition. Figure right, text wrong, both self-consistent: no alarm.
One premise that's easy to skip: memory doesn't care how those 8.08M tokens are made up.
⛔ But compute doesn't.
The same 8.08M tokens as long sequences or as short ones need very different compute: attention grows with S², and it is on the compute bill, not the memory bill.
So the sequence-length knob behaves differently on the two bills: memory only sees the total; compute also sees the split. Hence 5.2's “6ND badly underestimates.”
The premise: all this assumes FlashAttention. A naive version storing the score matrix puts S² back on the memory bill; none of the above holds.
6ND badly underestimates long contextThe usual training-compute estimate is 6ND: about 6 FLOPs per parameter per token (2 forward, 4 backward: the 1 + 2 from Section 1).
Plugging in (for MoE, active parameters, not total): 6 × 37e9 × 131,072 ≈ 29.1 PFLOP for one 128K sequence.
⛔ But at 128K this number is badly wrong.
6ND counts weight multiplications. The attention steps QKᵀ and AV involve no weights, so the formula ignores them.
Appendix 7.2 puts attention at 82.1% of a layer's forward compute at 128K, so 6ND captures only the remaining 17.9%.
So real compute exceeds 6ND by five to six times: the 29.1 PFLOP is actually close to 160 PFLOP.
This multiplier is derived from the 82.1% (1 ÷ 0.179), our own per-op estimate: magnitude only. Remember not the multiplier but when the formula misleads: long sequences.
Conversely, 6ND works very well for short sequences. At GPT-3's 2,048, attention is just over 1%, negligible. Same lesson: the formula isn't wrong; the premise changed.
Section 1's mountain figure said the peak is at the end of forward. That holds only for activations.
Draw all four items on one timeline (the figure above) and different items peak at different moments.
Generalized, a rule you can use anywhere:
when you stack curves of different shapes, the peak of the sum need not fall on any single curve's peak. Always ask “crowded with what” along with “when is it most crowded.”
📌 In practice: debugging an OOM, “the peak is xx GiB” is useless; you need which item is largest then. Large activations: recompute or add CP. Large optimizer state: ZeRO. Mix them up and neither helps.
The three things on this machine:
−ln p each, averaged. At init it is ln(vocab), 1.0's yardstick; a smaller vocabulary starts a bit lower.Below, ③ becomes a number you can check.
📌 Setup: a 125M-parameter model, one device, batch 8 × sequence 1,024.
12 layers, width 768, MLP 3,072: row one of GPT-3's Table 2.1 (top row of 3.5's table).
The same 16 bytes as in 3.1:
| Item | Per param | 125M params |
|---|---|---|
| bf16 weights | 2 B | 0.23 GiB |
| bf16 gradients | 2 B | 0.23 GiB |
| Optimizer state (fp32 master weights + m + v) | 12 B | 1.40 GiB |
| Subtotal | 16 B | 1.86 GiB |
First checkable conclusion: weights are 0.23 GiB, training takes 1.86 GiB, eight times more, all of it training-only.
What each token leaves per layer, item by item (width 768):
| Saved tensor | Width |
|---|---|
| Outputs of the two norms | 2 × 768 = 1,536 |
| Q / K / V | 3 × 768 = 2,304 |
| Attention output | 768 |
| MLP layer-1 output | 3,072 |
| Activation output | 3,072 |
| MLP layer-2 output | 768 |
| Per layer | 11,520 values = 22.5 KiB (bf16) |
× 12 layers = 270 KiB per token. × 8,192 tokens = 2.11 GiB.
⛔ Something surprising happened.
Activations 2.11 GiB > resident 1.86 GiB. At only 125M parameters, activations exceed weights, gradients and optimizer state combined.
No long context: sequence 1,024, batch 8. Recomputation isn't just for big models.
As in 2.2: each layer keeps its input (width 768), plus all of the layer being recomputed.
0.32 GiB: down from 2.11, 6.7× less, for compute 3× → 4×.
The 0.32 GiB has a deliberate omission checkers will hit.
It counts the 12 layers only, not the final logits. GPT-3 reuses GPT-2's BPE vocabulary, about 50.3K:
8,192 tokens × 50,257 × 2 B = 0.77 GiB, 2.4× the 0.32 GiB.
Materialize all logits at once, and the largest item here is neither weights nor activations but the vocabulary score table. Real code computes the loss in chunks, so it's left out (1.7). An accounting choice; the logits still exist.
📌 50.3K, in two hops: GPT-3 paper §2.1: “We use the same model and architecture as GPT-2, including the modified initialization, pre-normalization, and reversible tokenization described therein,” so GPT-2's BPE: 50,257. GPT-3 never prints it. 30K or 60K changes nothing.
| Recompute off | Recompute on | |
|---|---|---|
| Resident | 1.86 GiB | 1.86 GiB |
| Activations | 2.11 GiB | 0.32 GiB |
| Total | 3.97 GiB | 2.18 GiB |
| Compute | 3× | 4× (+33%) |
This is the training loop on your laptop.
“Pay 33%” matches 671B, but “save 6.7×” is 3× smaller than its 20.5×.
An earlier draft said “the ratios are almost the same as for 671B.” Far from it. Why is the more valuable part.
Why does the same “full recomputation” save 6.7× here and 20.5× there?
Write it as a formula. Let r = one layer's activations ÷ its input, L = layers:
Saving = r ÷ (1 + r/L)
Numerator r: how much a layer holds; fatter layers save more.
Denominator term: the layer being recomputed, amortized over layers; more layers, less impact.
| r = activations ÷ input | L = layers | Saving | |
|---|---|---|---|
| 125M small model | 15 (11,520 ÷ 768) | 12 | 6.7 |
| The rows define r a bit differently: the small model counts only the norms' outputs; V3 (1.7's table) adds their inputs (= residual input). With inputs, small-model r = 16, saving 6.9. Same magnitude and conclusion; strictly two rulers. This cell once had a hand-typed “r ≈ 16, saving ≈ 7.1.” 16 was right; the formula gives 6.9, not 7.1. Telling: every other number here was script-computed; this one, hand-typed. | |||
| V3 671B | 31.7 (one MoE block ÷ 7,168) | 61 | 20.5 |
The small model loses on both: r 2.1× smaller, L 5× smaller. The smaller saving is inevitable, not an error.
The large r is MoE's: in 1.7's table, expert outputs and SwiGLU intermediates are each stored nine times. A fuller layer saves more. MoE saves on parameters, costs on activations, and therefore pays off especially on recomputation. One design, three bills, three directions.
For V3 the formula gives 20.8 vs. 20.5 measured: slightly lower, as the 3 dense layers are thinner than MoE. An order of magnitude, not exact.
Only three things are truly scale-independent.
The saving factor is not on this list. It follows r and L: new architecture, new math. Criterion ③ again: separate what changes from what doesn't.
Same three caveats as 1.7's table: ① derived op by op; fusion shrinks them; read as magnitudes; ② excludes the attention score matrix (FlashAttention assumed), temp, fragmentation and comm buffers; ③ “0.32 GiB” assumes a checkpoint every layer, not always the leanest density (2.2 Ⓒ).
Topic 1 ended the forward pass with “it doesn't fit.” With the bill complete, that becomes specific: “each of the four items doesn't fit, and each has its own remedy.”
Four items, four remedies:
⭐⭐⭐ Three of four lines say one word: shard. That is all of Topic 5: sharding has its own cost, to be accounted for too.
So far, accounting. It can't answer the most common question on a real run: “The loss just shot up. Now what?”
PaLM 540B spikes, ablation: arXiv 2204.02311 §5.1, all verbatim: “roughly 20 times”, “roughly 100 steps before the spike”, “roughly 200–500 data batches”, “We do not believe that the spikes were caused by ‘bad data’ per se”
Router z-loss, Ⓒ's table: arXiv 2202.08906 (ST-MoE). ⭐ Its own heading: “Many methods stabilize sparse models, but at the expense of worse quality”
QK-norm: arXiv 2302.05442 (ViT-22B) §2 — training diverged at around 8B parameters: huge attention logits collapsed attention weights to near one-hot (near-zero entropy)
Normalization placement: arXiv 2002.04745 (ICML 2020) — proves Post-LN starts with large gradients near the output layer, hence needs warmup; Pre-LN's start well-behaved
⛔ None of our own measurements: all four from papers; Ⓒ's three numbers copied from Table 4
We've met five failure modes across five sections, each under its own name. Side by side: on a real run, the hard part is diagnosis, not the fix.
| Name | Symptom | Root cause | Confirm | Remedy |
|---|---|---|---|---|
| Exploding gradients | Loss NaN/Inf, or grad norm jumps decades in one step | Local derivatives stay above 1; 61 layers compound them exponentially (1.4) | Global grad norm per step: decade jumps? | Clipping (6.4) · Pre-LN (6.5) |
| Vanishing gradients | Input-side layers barely move, as if just initialized | Same chain, local derivatives below 1 | Per-layer grad norm: decays with depth? | Pre-LN and init (6.5) |
| Rounded-away updates (“won't train”) |
Loss flat despite nonzero gradient; fp32 master weights fix it at once | Per-step relative change below mantissa resolution
(about 1/256 for bf16) |
Compare update RMS ÷ param RMS with 1/256 |
fp32 master weights (3.2) |
| Step-size divergence | Loss climbs, or oscillates with growing amplitude | η exceeds 2 ÷ sharpest curvature:
steps overshoot, amplified along it |
Halve η and rerun: instantly fine means this |
LR ceiling and warmup (3.3b · 3.5) |
| Loss spike (instability) |
Sudden spike in smooth training; heals or collapses | Not bad data: there's an ablation (6.2) | Rerun same batches: spike again? | This section, 6.1–6.6 |
Rows one and two: one chain of products, two directions.
Per 1.4, a parameter's gradient is 61 local derivatives multiplied. Above 1 on average, it grows exponentially; below 1, it decays. So the fixes are one family: keep each local derivative near 1, as 6.5's structural fixes do.
The middle rows (rounding, divergence) are both called “won't train”, with opposite remedies: bigger steps or more precision vs. smaller steps. Confusing them hurts.
📏 First, the yardstick from 1.0.
“Blew up” is a feeling; we want judgment. In numbers:
ln 129,280:
everything forgotten, uniform guessing.So spike height tells whether the run can be saved. A jump to 6 usually clears by rolling back a few dozen steps; a jump to ln V wastes that stretch.
⛔ PaLM 540B's loss spiked roughly 20 times.
With gradient clipping on. The spikes came at highly irregular times, sometimes late, and never in the smaller models.
📌 arXiv 2204.02311 §5.1. “Roughly 20 times” and “despite the fact that gradient clipping was enabled” are verbatim.
📌 arXiv 2204.02311 (PaLM) §5.1, verbatim: “roughly 20 times”, “despite the fact that gradient clipping was enabled”, “were not observed when training the smaller models”.
⚠️ The curve is schematic: spike positions were picked to match “no discernible pattern”; the paper gives no heights or widths, so the y-axis has no ticks.
Scale effect: fine when small isn't safe when large. Like “don't copy across scales” in 2.6, but what breaks is training itself, not the payoff.
First instinct: bad batch. PaLM tested it cleanly:
They replayed the batches around the spike from an earlier checkpoint. No spike.
Spikes aren't “bad data”: these batches collide with that parameter state.
Same data, other moment: nothing.
Hence their crude fix works: roll back to a checkpoint ~100 steps before the spike, skip those 200–500 batches, resume. It doesn't recur.
Not “fixed”: “worked around”. Given the cost, they “were not able to determine a principled strategy to mitigate these spikes”. Still open.
A tiny learning rate or very tight clipping stops spikes. All cost model quality.
ST-MoE measured this. Same config, different seeds: how many runs finish, at what quality:
| Method | Stability | Quality ↑ | Verdict |
|---|---|---|---|
| Baseline | 4 / 6 | −1.755 | 2 of 6 runs diverged |
| This is log-likelihood, not loss: negative, and higher is better (−1.755 beats −4.206). On the 1.0 yardstick: loss 1.755 vs. 4.206, perplexity 5.8 vs. 67. Not the same quantity as loss. | |||
| Tighter update clipping (0.1) | 3 / 3 | −4.206 | Stable, but quality wrecked |
| router z-loss | 3 / 3 | −1.741 | Stable, slightly better |
The middle row is the point: “stable 3/3” looks perfect, bought with quality. Judge stability techniques on both columns.
Two caveats against over-reading it.
Not a flaw: good design. To study a rare failure, make it common first. But remember: the failure rate was dialed up.
📌 arXiv 2202.08906 (ST-MoE), Table 4. Its heading: “Many methods stabilize sparse models, but at the expense of worse quality.”
Simple: a loss term penalizing the squared log of the softmax normalizer.
It pushes log Z toward 0, keeping logits small.
Two places, both used:
10⁻⁴ · log²Z,
which increases training stability per the paper.Why z-loss is cheap insurance: per ST-MoE, logits feed an exponential, extremely sensitive to input error. A smaller input range directly cuts that risk, and the model doesn't need the dynamic range.
Clip by global norm: treat all gradients as one vector; if its norm exceeds the threshold, scale it down. 1.0 is usual (PaLM, DeepSeek-V3: both 1.0).
But recall 6.1: PaLM spiked 20 times with clipping on. Clipping stops a one-step explosion, not parameters already drifted somewhere bad.
First, split a layer into three parts to see the difference. A layer is two parallel paths. The trunk carries x unchanged. The branch runs x through attention or the MLP for a correction F(x). They merge into x + F(x): 1.4's residual.
Normalization, part three, must sit on one of these paths. Post-LN and Pre-LN differ only in where.
Two placements
Post-LN xt+1 = Norm( xt + F(xt) )
Merge, then Norm the whole trunk, every layer.
Pre-LN xt+1 = xt + F( Norm(xt) )
Norm moves into the branch. The trunk is untouched, straight to the top.
Neither wins yet. But it's not a matter of style.
A 2020 paper proved it, with a conclusion tied to 3.5:
At init, Post-LN has large gradients near the output,
so a large learning rate is bound to be unstable; warmup holds that in check.
Pre-LN's gradients are well-behaved at init, so the paper says Pre-LN can drop warmup.
Recall 3.5's table, third row.
Its “precondition” reads “significant vanishing gradients (i.e., Post-LN)”:
this exponential decay.
So warmup doesn't have “competing explanations”:
the 2020 paper explains why it was once indispensable; 3.5's paper, why it's still around. Different questions.
📌 arXiv 2002.04745 (ICML 2020), via mean-field theory.
That says Post-LN gradients are bad, not why. The mechanism is one sentence and answers a sharper question: if it's bad, why keep it?
⛔ Post-LN is
xt+1 = Norm(xt + Ft(xt)).
At init, x and F(x) are roughly independent random vectors, each of variance 1, so their sum has variance 2. Norm pulls it back to 1: effectively ÷ √2.
Recursing,
the coefficient of input x₀ in layer l's output is 2−l/2.
The residual “shortcut” shrinks exponentially, most near the input.
Su Jianlin's verdict, harsh but well founded: “In a Post Norm BERT model, LN not only fails to mitigate vanishing gradients; it is one of the ‘culprits’ behind them.”
📌 Su Jianlin, “Notes on Model Optimization: Why Is BERT's Initial Standard Deviation 0.02?” (in Chinese), kexue.fm/archives/8747 — “In Post-Norm BERT, LN not only fails to ease vanishing gradients, it is one of their ‘culprits’” is his line (our translation); 2^(−l/2) comes from recursively unrolling its formula.
⚠️ The author's own analysis, not peer-reviewed. ⛔ The figure plots the forward residual shortcut, not a gradient multiplier: related, not the same.
So why not remove it?
Without it, x + F(x) has variance 2, growing with each residual.
The question is “where”, not “whether”:
Pre-LN uses x + F(Norm(x)) plus one final Norm,
so all residual branches weigh equally and the decay is gone.
Most counterintuitive: elsewhere, this “bug” is a feature.
Fine-tuning should adjust parameters near the output first and barely move those near the input, to preserve what pretraining learned. Vanishing gradients mean “closer to the input, weaker effect on the output”: exactly what fine-tuning wants.
So pretrained Post-LN models often fine-tune better than Pre-LN. That's the figure's two halves.
📌 Source and disclaimer per 3.5 (Su Jianlin, kexue.fm/archives/8747). It once said “see the footnote in 3.4”, wrongly: that footnote cites another post by the same author (archives/11267, on update RMS 0.2).
Of the three warmup explanations in 3.5, this confirms one precondition. The third, “hold back later layers until earlier ones catch up,” relies on this exponential decay. Pre-LN lacks it, so that explanation doesn't apply.
At around 8B parameters, ViT diverged within a few thousand steps: attention logits grew huge, so post-softmax weights collapsed to near one-hot (near-zero entropy).
The fix: LayerNorm Q and K before the dot product. Hence QK-norm, now standard in many large models.
📌 arXiv 2302.05442 (ViT-22B) §2. QK-norm was not invented by ViT-22B. The paper says “we adopt the approach of Gilmer et al. (2023)”. ViT-22B applied it at 22B parameters, with a comparison figure: hence the citation. Again a failure that only appears at scale, as in 6.1.
With every residual branch initialized at one scale, the signal grows with depth. Standard practice: scale residual-branch init down with depth. It won't prevent spikes, but sets your starting point.
❓ “Does bf16 need loss scaling?” No.
In short: same 16 bits, split differently. fp16 favors “fine resolution”; bf16 favors “reaching tiny values”. fp16's problem isn't coarseness; it can't go low enough: a small gradient becomes exactly 0, and that parameter gets no signal that step. Loss scaling is simple: shift values up, off the floor.
Loss scaling was an fp16-era necessity: fp16 has only 5 exponent bits,
so small gradients underflow to 0. Scale the loss up before backprop, down before the update.
bf16 has fp32's 8 exponent bits:
same range, less mantissa. No underflow, no loss scaling.
numpy.float16 bit patterns; “rounds to zero” was verified by rounding. No histogram: we lack that data and won't invent it.📌 Every boundary is computed from numpy.float16 bit patterns: min normal 0x0400 = 2⁻¹⁴, min subnormal 0x0001 = 2⁻²⁴; bf16 shares fp32's 8-bit exponent, floor 2⁻¹²⁶. “Below the red line rounds to 0” was verified by actually rounding.
⭐ The framing of this figure comes from the classic mixed-precision figure (a gradient histogram + a vertical fp16 floor line; see Micikevicius, Narang et al., ICLR 2018). ⛔ But we lack that histogram data, so we draw no histogram, only the real boundaries.
Pairs with 3.2: bf16 buys “enough range, poor precision”: the danger is accumulation, not underflow.
❓ “Should training be reproducible?” It can be, and it pays.
PaLM was bitwise reproducible: restart from any checkpoint and every later step matches the original run exactly.
It takes a deterministic compute framework and a deterministic data pipeline: the batch depends only on the step, not on process count or resumes.
6.2's “roll back + skip data” needs this. Otherwise, which batches to skip?
📌 Also from arXiv 2204.02311.
Back to the start.
Section 0: training uses one number to decide which way each of 671B parameters moves.
Every part is now explained:
−ln p, the model's surprise at the right answer.
From 11.77 (uniform guessing) to around 2.
(1.0)All that accounting (3× compute, undroppable intermediates, 16 B/param) just moves it from 11.8 to 2.
Near 2, each 0.01 is expensive, paid with the whole bill you just worked out.
Dozens of numbers. Not what we want you to keep.
⭐⭐⭐ Numbers go stale. These four rules won't.
Deliberately, none of our own measurements here: all four come from papers, each checked at the source. Why (§7): the lecture has many self-derived numbers; stability shouldn't add more.
This lecture's numbers are not all of one kind. Some trace to a paper; others we derived op by op, and those could be off by 2×.
What to search for in the source
| Our term | Their name | Verify in |
|---|---|---|
| Full recompute (2.2) | --recompute-granularity full |
Megatron-LMtraining/arguments.pycore/transformer/transformer_config.py |
| Selective recompute (2.3) | --recompute-granularity selective +
--recompute-modulesOptions: core_attn (default under selective) / layernorm /
moe_act / mla_up_proj / mlp … | |
| Grad clipping (1.8) | --clip-grad (default 1.0, V3's value) | |
| Recompute, by hand | torch.utils.checkpoint.checkpoint(...) |
PyTorchtorch/utils/checkpoint.py |
| Selective, per tensor | create_selective_checkpoint_contexts + CheckpointPolicy2.3's criterion in PyTorch: your rule picks which ops to save or recompute | |
| Recompute (JAX) | jax.checkpoint (alias jax.remat) |
JAXjax/_src/ad_checkpoint.py |
| Stock selective policy | jax.checkpoint_policies.dots_with_no_batch_dims_saveableThe name is the criterion: “keep matmul outputs”, 2.3's costliest rows | |
| Gradient accumulation (1.8, stage ④) | ddp.no_sync(), a context managerGradients accumulate locally, unsynced; sync “in the first forward-backward pass exiting the context” (docs) |
PyTorchtorch/nn/parallel/distributed.py |
| “Reduction hides behind compute” (1.8, stage ②) | bucket_cap_mb (default 25 MiB)Each full bucket ships at once, in flight while earlier layers compute. Stage ②'s “✅ hideable” lives here | |
| ZeRO stages (4.2) | config zero_optimization.stage = 1 / 2 / 3 |
DeepSpeedruntime/zero/config.py |
| 8-bit optimizers (3.4) | bitsandbytes.optim.Adam8bit / AdamW8bit |
bitsandbytesoptim/__init__.py |
Two rows where our criteria are the API names.
dots_with_no_batch_dims_saveable:
“save matmul outputs.” Why matmuls?
Each byte they save costs the most FLOPs to recompute: the closed form in 2.3.ddp.no_sync():
no_sync()
switches off stage ② entirely.
No communication while accumulating; one sync at the end.
Saving communication is this API's purpose, not a side effect of accumulation.core_attn is Megatron's default once selective is on
(not “on by default”: recompute_granularity defaults to None),
and the source warns it's unneeded with fused attention.
Yet by 2.4, at long context it is the last thing to recompute.
Defaults have a vintage (2.5).
Not a contradiction: 2.2's “recomputation is on by default in large-model training” is industry practice;
recompute_granularity = None is the factory setting. Different “defaults.”One option we skip: PyTorch's CheckpointPolicy
has six enum values, not two: {MUST,PREFER}_SAVE,
{MUST,PREFER}_RECOMPUTE, {MUST,PREFER}_CPU_OFFLOAD.
Each activation is a choice of three:
keep in HBM / recompute / offload to CPU and back. We cover the first two.
The third buys HBM with PCIe bandwidth instead of compute: same trade, different currency.
MUST vs. PREFER, docs:
“Use MUST_* over PREFER_* to indicate that the policy should not be overridden by other subsystems like torch.compile.”
So the table is a reverse check: if your criterion doesn't match the options, your criterion is likely wrong, or the default stale.
This table goes stale first:
Every name was grepped live from each main branch in September 2026, not from memory. Frameworks rename often.
Rule: when you give a name, give the file it lives in. Otherwise a rename makes it misleading, and nothing errors.
| Number / claim | Source |
|---|---|
| Classic 16 bytes/param (2 + 2 + 12); ZeRO's three-stage split | Rajbhandari et al., arXiv 1910.02054 |
| GPT-3 peak LR and batch for all eight sizes; 375M-token linear warmup; cosine to 10% over 260B tokens; weight decay 0.1; context 2,048 | arXiv 2005.14165 Table 2.1 + Appendix B (§2.3 defers to it) |
| DeepSeek-V3's full LR schedule (2K steps → 10T flat → 4.3T cosine → 2 final stages); clipping 1.0; batch 3,072→15,360; bf16 moments, fp32 master weights/grads; recomputes RMSNorm, MLA up-proj. | arXiv 2412.19437 §3.2.3 (recompute) / §3.3.3 (low precision) / §4.2 (hyperparams) |
| Original selective-recompute criterion; GPT-3 saves 70% at 2.7% cost, MT-NLG 65% at 1.6% | Korthikanti et al., arXiv 2205.05198 |
| Warmup's real role (permits a larger target LR; “too few samples to estimate” is not the main reason; longer warmup: diminishing returns) | arXiv 2406.09405 (NeurIPS 2024) |
| Original definitions of Adam / AdamW / Adafactor / Lion | 1412.6980 / 1711.05101 / 1804.04235 / 2302.06675 |
| Muon: 2D params only; embedding, output head on AdamW; slower per step (wall-clock) than AdamW | Author's writeup (all three stated there) |
| Number | Derivation | Weak points |
|---|---|---|
| Activations 104.11 GiB → recomputed peak 5.07 GiB (4K) | Op by op (1.7's two tables). Inputs: V3's config + the reference MLA forward | Op fusion; does MoE dispatch really make nine copies? |
| Attention = 82.1% of a layer's compute | Causal-halved. We once said Topic 1's
“independent” 81.8% agreed. Wrong; retracted.
Same numerator (one formula, S²·H·(dqk+dv), one config,
same halving); only denominators differ: whole model incl. lm_head (Topic 1)
vs. one layer's active params. The 0.463% gap = lm_head dilution:
81.8 ÷ (1−0.00463) = 82.18. |
Convention (halved?); head dims. Self-check, not an external anchor: same source, must agree. No third-party data yet |
| Crossover S ≈ 13,107 | 1.25·S = 16,384 (o_proj input = 128 heads × 128);
1.25 = V3 head dims 192/128, causal-halved.
Earlier: d_model 7,168 → 5,734.
“Widest linear = d_model” fails for MLA |
Another layer moves it. Not an engineering threshold |
| 6ND is 5–6× too low | Back-derived from the 82.1% above (1 ÷ 0.179) | Inherits the 82.1%'s doubt; no third-party check yet |
| 9.76 TiB resident block · crossover at ~1,973 4K sequences | 671e9 × 16 B ÷ 1024⁴ ÷ 5.07 GiB peak. Peak must include the layer being recomputed; checkpoints alone give 2,997 (52% high). Build-time assert | 16 B is the classic convention; V3 isn't set up that way (3.2) |
| “At 80 GiB per device, it takes 125” | 9.76 TiB ÷ 80 GiB | Arbitrary 80 GiB yardstick, not a real device; real use adds redundancy, comm buffers, fragmentation |
Why this section exists.
Both kinds look identical (a few digits and a unit), and readers copy them together.
⭐⭐⭐ Rule: a self-derived number must come with its derivation, or it is posing as a fact.
📌 A method, with a counterexample here: an isolated estimate is unverifiable; one matching an “external” anchor checks the whole formula. Our 82.1% has no anchor: as retracted, it shares a source with Topic 1's 81.8%, so had to agree. “They agree” counts only if paths are truly independent.
← Course index ·
the forward half of the bill: Topic 1 · The Life of a Token ·
want to teach it? Instructor notes for this lecture ·
how the bill gets split: Topic 5 · Parallelism
Generated by Courses/tools/topic04-build.py;
the text lives there.