Modern AI Accelerators: GPU / TPU Systems Course / Core track / Topic 4 / Backward Pass and Optimizers 🌐 中文

Backward Pass and Optimizers

A model reads a sentence and gets the next token wrong. How wrong: one number.
Training uses it to move each of 671B parameters: which way, and how far.

Three things: computing that number (the loss), turning it into 671B derivatives (backpropagation), sizing the step (gradient descent, optimizers). One chain, from one number to one update.

Prereq Topic 1 Parts need Topic 3 Next Topic 5 · Parallelism Approach: follow one loss value through

Course author Chris Yang·Google Cloud AI Infra architect

This lecture has instructor notes.

They cover how to teach it: goals, pauses, misconceptions, likely questions. Content: this page.

Presenting: lines, figure cues, timing: all written out. Self-study: give them to an AI for a TA that knows the material and how to teach it.

Section 0

One number, 671B parameters

🎯 Warm-up: match last lecture's four models to their routes Answer shown once all four are right. Optional

Left: four models. Right: four attention layouts (one cell = one layer, colored as in Topic 3's timeline table). Click a left item, then a right one to connect them; click the left one again to undo.

Kimi K3  2.8T/104B · 93 layers
DeepSeek-V4-Pro  1.6T/49B · 61 layers
Qwen3.5  397B/17B · 60 layers
Xiaomi MiMo-V2.5-Pro  1.02T/42B · 70 layers
SWASWASWASWASWASWAFULL
KDAKDAKDAgMLA
HCAHCACSAHCACSA
GDNGDNGDNgAT

Topic 1 followed one token forward to the model's guess. Here, the guess is wrong. Now what?

The error becomes one number. Not a table, not a vector: one number.

Training is one job: turn that one number into which way and how far each of 671B parameters moves.

✋ Someone always asks: how can one number decide 671B things? Too little information.

The answer is the lecture's thread: the 671B directions aren't read out of the number but computed from it.

It lives in the network's structure and every intermediate forward-pass value. The loss is the start; the chain rule expands it into 671B pieces. Hence Section 2's point that activations can't be discarded: to expand, keep what the expansion uses.

Three things, along that chain:

  • ① Where the number comes from: computing the loss (1.0).
  • ② How one number becomes 671B derivatives: backpropagation (Section 1). The core: brute force takes 20,000 years; backprop adds just 2× the forward time.
  • ③ Given the gradients, how big a step: gradient descent, optimizers (Section 3).

Three things, one chain. One number at one end, 671B nudges at the other.

⛔ What this lecture skips, so you don't wait for it.

  • Whether it fits. Enough HBM? Biggest piece? How to save? A few numbers for scale, no more.
  • Splitting it across devices. Tensor parallelism, pipelining, ZeRO's three stages: all Topic 5.

⭐⭐⭐ Sections 2, 4, 5's detailed math and every collapsible block fall under these. A Topic 5 preview; skipping them costs none of the three.

Why draw the line here Methodology, not one of the three things; safe to skip

Backprop and gradient descent are principles, unchanged for decades. Whether it fits is a per-generation hardware fact. Mixed, principles seem to expire too.

📐 Setting first, so each number has a known regime.

Unless stated, numbers use DeepSeek-V3's real pretraining config: sequence length 4,096, global batch ramping 3,072 → 15,360 sequences.

V3's advertised 128K is two 1,000-step extension stages after pretraining; the 14.8T tokens ran at 4K. Every 128K here is labeled context extension and converted to 4K. A number without its setting is meaningless: criterion ③ of this lecture.

Section 1

One number → 671B derivatives: backpropagation

Backprop isn't “running it again.” It costs two things: ~2× the forward compute and, worse, piled-up activations that can't be dropped early.

📌 First, pin down “activation.” It appears 60+ times here and has three common meanings.

  • Always the first here: what the forward pass hasn't yet discarded: intermediate results (layer inputs, norm outputs, attention scores…). §2 tallies their HBM.
  • Not “active parameters.” Last lecture's “V3 activates 37B per token” means the MoE weights selected at this step, not intermediate results. We use that sense only for MoE dispatch, written “activated experts.”
  • Nor “activation function.” SwiGLU, GELU etc. are always called “activation function” in full.

“Activation” alone means the intermediate results the forward pass leaves for the backward pass.

First, the starting point: the chain runs backward from one number, the loss, which deserves two minutes.

1.0 First: what kind of number is the loss?

Everything here aims to shrink one number. Start with it.

Step 1 · The loss is “how surprised the model is”

The model reads the text so far and guesses the next token, giving every vocabulary token a probability. We reveal the answer and check only its probability.

📌 Softmax turns scores into probabilities: exponentiate (all positive), divide by the sum (adds to 1). Every later use means this, as does “weights sum to 1” in Topic 3's attention figure.

loss = −ln p (p = the model's probability for the correct token)

Step 1b · From logits to loss, and where the “wrong guesses” went

First, where logits come from, or the figure's “row of scores” won't explain why there are exactly 129,280.

Where the logits come from In one line: forward pass done → a 7,168-dim vector → × LM head → 129,280 scores Forward LM head logits Ⓐ One position, after the forward pass Only this path; layer internals are Topic 1's job Forward done (61 layers) This position's vector 7,168 numbers × LM head 7,168 × 129,280 This weight block holds 927M numbers logits 129,280 numbers (one per vocab token) Not to scale: really 18× longer This row is the logits Any sign, unbounded Not probabilities yet. All positions at once is one matmul: [4,095 × 7,168] × [7,168 × 129,280] = [4,095 × 129,280] The result is not small: about 0.99 GiB in bf16, bigger than any single tensor in the §1.7 table. Softmax comes next: it turns these 129,280 scores into 129,280 probabilities.
In one line: forward pass done → a 7,168-dim vector → × LM head → 129,280 scores. The LM head is 7,168 × 129,280, 927M numbers on its own, and gives every vocabulary token one score. This row is the logits: any sign, unbounded, not probabilities yet.
⭐ All positions at once is one matmul whose result is about 0.99 GiB (bf16), bigger than any single tensor in the §1.7 table.
Sources and assumptions

Vocab 129,280, residual width 7,168: DeepSeek-V3 config. LM head parameter count and 0.99 GiB: computed and asserted by this figure's script.

⚠️ The two blocks are not to scale (the real 18× ratio won't fit); layer internals: see Topic 1.

Where does −ln p get its p? Two steps.

The last layer outputs a row of raw scores, the logits. Any sign, any range; not probabilities yet.

From logits to loss, and where the “wrong guesses” went Three steps: exponentiate → divide by sum → −log of the answer Ⓐ 3 steps Ⓑ 2 properties Ⓒ Wrong guesses Ⓐ From a row of scores to one number Vocab shrunk to 5 tokens (really 129,280). Same steps, but the number shrinks too (see note below) ① logits (raw scores) 2.00 the 1.00 a 1.00 in 0.00 to 0.00 is Any sign, unbounded ② Subtract the max, then exp 1.00 the 0.37 a 0.37 in 0.14 to 0.14 is All positive; top entry exactly 1 Minus 2.00 each: 0 / −1 / −1 / −2 / −2 ③ Divide by sum = probability 0.50 the 0.18 a 0.18 in 0.07 to 0.07 is Sums to exactly 1 Row total doesn't matter Shift the row: same probabilities → safe to subtract Denominator = 2.01 = 1 ÷ top entry's probability (here: perplexity) Sum = 1.00 Only this row means anything ④ Reveal the answer Correct token is “the”: probability 0.4984 (the same cell as ③, rounded there to 0.50) Loss at this position = −ln 0.4984 = 0.6964 The other four probabilities never appear in the formula So where did they go? See Ⓒ. Note the term: logits = the raw scores from the last layer. Not probabilities yet: softmax's two steps sit in between. This 0.6964 is not the loss you see on a training curve Same −ln p, but to fit the figure we shrank it twice: 5 tokens, one position. Random guessing over 5: −ln(1/5) = 1.61, so 0.6964 only means “a bit better than guessing,” not “well trained.” Training-log loss = the real 129,280-token vocab + averaged over a batch's positions. Random-guess start: 11.77 (top of the 1.0 ruler, step 0 of the curve). So half of the loss value is set by the vocab: models with different vocabs can't compare losses directly. Ⓑ Two properties: one elegant, one life-saving ① Add a constant: probabilities unchanged Add +100 to every logit: you still get 0.4984. So the absolute size of logits is irrelevant; only differences matter. What exp does: turn differences into ratios. ② Never exponentiate directly: it overflows Real logits can reach the tens or hundreds: e¹⁰² ≈ 2e+44, while fp32 tops out at 3.4e+38. So implementations subtract the row max first, then exp: that is step ② in Ⓐ The subtraction is safe because of the shift invariance on the left. Same math, no overflow. Ⓒ Where did the wrong guesses go? They are all in the denominator Correct answer's probability = its exp ÷ the sum of all exps, which spans every token Before Correct entry's probability 0.4984 loss = 0.6964 Raise just one “wrong” score by 1 raised Correct entry's probability 0.3790 loss = 0.9702 The correct entry's score didn't change at all, yet the loss rose by 0.2739. So they do take part, just not under their own names. After differentiation, they all show up the (correct) -0.5016 a +0.1834 in +0.1834 to +0.0675 is +0.0675 Sums to 0: a redistribution For each wrong one, the gradient equals its current probability. This figure in two sentences Logits = the last layer's raw scores (any sign); exp → divide by sum gives probabilities; then reveal the answer, take −log: that's the loss. Wrong guesses never left; they're in the denominator, and each reappears in the gradient. In short: invisible in the loss, visible in the gradient; training uses only the gradient.
Logits → probabilities → loss. Ⓐ Exponentiate (positive, gaps widen) → normalize (sum 1) → −ln(answer). Logits: raw last-layer scores, not probabilities.
Ⓑ Shifting all logits equally changes nothing (only differences count); raw exp overflows, so subtract the max first.
⛔ Ⓒ, the class question: wrong guesses sit in the denominator. Raise one wrong score: the loss still rises; the gradient shows them all.
Sources and assumptions

⭐ Full cross-entropy is a sum over the whole vocabulary; a one-hot label (a single 1) leaves just one term nonzero. So “looking at only one” follows from one-hot labels; it isn't the definition.

⚠️ Converse check: with non-one-hot labels (label smoothing, distillation soft labels), every term appears in the loss. Numbers computed and asserted by the script; fp32 max = 3.4028e+38.

✋ First, what that 0.6964 means: it is not the loss on a training curve.

Same −ln p, but shrunk twice to fit the figure: a 5-token vocabulary and a single position.

Compare it with random guessing over the same vocabulary: for 5 tokens, −ln(1/5) = 1.61. So 0.6964 says only “a bit better than guessing,” not “already well trained.”

A training-log loss = the real 129,280-token vocabulary + an average over all positions in a batch. Its random-guess start, −ln(1/129,280) = 11.77, is the 1.0 ruler's top and step 0 of the curve.

⭐ A common trap: half of the loss's size comes from the vocabulary, not model quality. A bigger vocabulary raises the same model's loss, so losses across different vocabularies aren't directly comparable (use a tokenizer-independent measure like bits per byte).

Two properties: elegant and life-saving.

① Adding the same number to every logit changes no probability. Only differences matter; exponentiation turns differences into ratios.

② Never exponentiate directly: it overflows. Real logits hit tens or hundreds; e¹⁰² far exceeds fp32's max. So every implementation first subtracts the row max: by shift invariance, mathematically identical, numerically safe.

6.4's z-loss is the flip side: this defuses overflow after the fact; z-loss keeps scores small up front.

✋ Now the class question: “Why don't the wrong tokens enter the loss?”

They do, just not under their own names: they sit in the softmax denominator.

Correct answer's probability = its exponential ÷ the sum of all entries, a sum over every vocabulary token.
⇒ Panel Ⓒ proves it: raise one “wrong” score by 1, correct entry untouched, and loss goes 0.6964 → 0.9702.

In the gradient, all show up. Each wrong token gets +its probability: the more it holds, the harder it's pushed down. Row sum: exactly 0.

⭐ In one line: invisible in the loss formula, visible in the gradient, and training uses only the gradient.
📌 Deeper: full cross-entropy sums over the vocabulary; a one-hot label leaves one nonzero term. So “looking at only one” follows from one-hot; it isn't the definition. With soft labels (label smoothing, distillation), every term appears in the loss.

Why this formula: it makes sense.

  • Confident and right (p → 1): −ln 1 = 0, no penalty.
  • The correct token got almost nothing (p → 0): −ln p → ∞, a huge penalty.
  • The logarithm gives “1% → 2%” and “50% → 100%” the same reward: each doubles p and cuts the loss by ln 2.
  • The log's real gift: d(−ln p)/dp = −1/p. The lower the probability, the larger the gradient, so the worst mistakes get fixed first. A draft said “otherwise it keeps nudging what it's sure of.” That doesn't follow: linear 1−p has constant slope −1, treating “sure” and “clueless” identically. Backward.

📌 Formally: cross-entropy, or negative log-likelihood. Three names, one formula.

Step 2 · Perplexity comes first; the loss is its log

Drop the log for now. The model gave the correct answer 0.4. What penalty should it get?

Most natural: the reciprocal. 1 ÷ 0.4 = 2.5

2.5 means the model is as good as guessing blindly among 2.5 tokens. (With 2.5 equally likely options, each would get exactly 0.4.)

That reciprocal is perplexity: “1 ÷ your probability for the answer.” Nothing mysterious.

p for the answerPerplexity = 1/pReads as
0.81.25Almost certain
0.42.5Like guessing among 2–3 tokens
0.110Like guessing among 10 tokens
0.01100Badly confused

⛔ So why the −ln? Purely so thousands of positions can be summed.

A sequence has 4,096 positions, each with a probability, and probabilities multiply. 0.4 to the 4,096th power is 0 on a computer: it underflows.

The logarithm fixes this: ln(a×b) = ln a + ln b. It turns multiplication into addition: add, average, never underflow.

Expand ln(1/p) and you get −ln p. That is the whole story.

To read it, exponentiate back: eloss = perplexity = the geometric mean of all the 1/p.

📐 Check it: three positions get 0.4 / 0.5 / 0.2. Average ln(1/p), exponentiate: 2.92; the geometric mean of 1/p: also 2.92. Same number, two routes.

❓ An aside people get stuck on: why is log 1 = 0?

A log asks: from 1, how many doublings reach this number?

Double each step: 1 → 2 → 4 → 8. “The log of 8 is 3”: three steps.

How many steps from 1 to 1? Zero; you're already there. That is log 1 = 0.

One step back divides by 2, to 0.5, so log 0.5 = −1. Greater than 1 is positive, equal to 1 is zero, less than 1 is negative.

So the loss formula holds up at both ends:

  • Perfectly right, p = 1 → perplexity 1 → no steps from 1 to 1 → loss = 0. Zero penalty for a perfect answer, not rigged: log 1 = 0 gives it.
  • Wrong down to 1 in 10,000 → perplexity 10,000 → loss = 9.21. The wronger, the costlier, without limit.

Perplexity 100 → 10,000 is 100×; the loss 4.61 → 9.21 only doubles. The logarithm compresses “how many times bigger” into “how many steps.”

Convention: only the answer's probability counts

Each of 129,280 tokens has a probability. The loss uses one: the correct answer's.

The answer is “cat”; the model gives “cat” 0.4, so this position's loss is −ln 0.4. “dog,” “table,” “the”: all ignored.

Other 129,279 wasted? No: they vie for probability with the answer.

Probabilities always sum to 1. Zero-sum: others up, this one down. Absent from the formula, they decide how far the numerator's p is squeezed.

Precisely: don't mix the two levels (I once got this wrong in class). Probabilities have no “denominator”; they always sum to 1 (zero-sum). Softmax's internal denominator Σ exp(raw score) varies, but at the score level. “The denominator grew” implies the total changes. It doesn't. Measured: one score 1.0 → 3.5; total 1.000000 → 1.000000; the correct token squeezed 0.629 → 0.175.

Is “only one” a rule, or a consequence? A consequence.

The full formula doesn't pick one probability; it compares two distributions: the model's column vs. the reference.

The reference answer: 1 for the correct token, 0 for the other 129,279. This “a single 1, all else 0” encoding is called one-hot.

Expanded (a four-token toy example; the model gives cat 0.4, dog 0.35, table 0.15, the 0.10):

TokenRef. qModel p Term = −q × ln p
cat (right)10.400.9163
dog00.350: × 0, gone
table00.150: × 0, gone
the00.100: × 0, gone
Total0.9163 = −ln 0.4

So we don't “only look at one”: the other 129,279 terms are multiplied by 0 and vanish.

Hence “cross”-entropy: two distributions compared. Entropy compares one with itself. These are the two I mixed up in the confession below.

📌 People write the reference answer, so it needn't be 100%.

  • Label smoothing: correct token 91%, the other 9% spread around. Same prediction, cross-entropy 0.9163 → 0.9913: the other terms really take part.
  • Distillation: the reference is a large model's full column. Every token takes part: the small model learns “cat,” and also “dog is close, table is nowhere near.”

Pretraining almost always uses hard one-hot labels, so here “treat it as 100%” is accurate.

Worth more than the definition: a model can be confidently wrong.

Say a wrong token gets 80%, the correct one 1%.

The perplexity at this position = 1 ÷ 0.01 = 100. Yet it doesn't hesitate: 80% on one token. Decisively wrong.

So perplexity measures accuracy, not hesitation.

Confession: I taught this backward in class. I called perplexity the spread (entropy): another quantity. One distribution, “one token 90%, nine sharing 10%”: same entropy; loss 0.11 if the 90% token is right, else 4.50: 81× in perplexity. Entropy measures confidence; loss, correctness. Low entropy, high loss: possible.

Step 2b · A ruler you can verify

loss
nats/token
In bits
= nat × 1.4427
Perplexity = elossRough level“Torn among N tokens”
11.77 computed16.98129,280Fresh init, learned nothingUniform over vocab: the loss ceiling
6.0 ballpark8.66403Token frequenciesA few hundred
4.0 ballpark5.7755Grammar mostly rightDozens
3.0 ballpark4.3320Plausible text~20
2.0 ballpark2.897Pretrained LLMs today, roughly7–8
1.0 ballpark1.443Close to memorizing2–3: past normal pretraining

⛔ The top row isn't a made-up value.

A freshly initialized model gives every token the same probability, p = 1 / 129,280. Plug it in:

loss = −ln(1 / 129,280) = ln 129,280 = 11.77

Its perplexity is exactly 129,280, the whole vocabulary: the definition of random guessing.

So the ruler's top is computed, not estimated, and moves with vocabulary size (129,280 checked in Topic 1). The bottom has no official V3 number: “around 2.0” is a rough public-model ballpark, a reference, not any model's measurement.

The bits column: yes/no questions needed to “pick” the token

Column two converts nats to bits (÷ ln 2). The conversion is trivial; the reading matters.

Bits count the yes/no questions needed to pick this token from the candidates. Each halves them: log₂ of the candidate count.

At init: the whole vocabulary, 16.98 questions. Late in training: 2.89.

The candidates shrink because of the context, not because the token got easier. Compressing 129,280 to an effective 7 is the forward pass's work; the 14.09 bits saved are that 17,500×.

⛔ Precisely: “an effective 7” doesn't mean 7 candidates remain; the vocabulary stays 129,280 and probability concentrates. It means: as hard as 7 equally likely candidates.

What the bits column says: yes/no questions to “pick” the token Not “bytes to store it”: how many yes/no questions to pick it from the candidates Each question halves the candidates Ⓐ One bit = one yes/no question: each one halves the candidates So “how many questions” = log₂ of the candidate count ① Just initialized: candidates = the whole vocab 129,280 left start 64,640 left 1 question 32,320 left 2 questions 16,160 left 3 questions … and so on; 1 left at question 17 ⇒ 16.98 questions (log₂ 129,280) ② Late in training: an effective 7 left 7 left start 4 left 1 question 2 left 2 questions 1 left 3 questions Done in three ⇒ 2.89 questions (log₂ 7.39) Bits measure how hard it is to pick, not how big it is to store. A token id fits in 4 bytes; unrelated to this column. The figure in one line bit = yes/no questions to pick it out of the candidates: fewer candidates, fewer questions. Context shrinks the candidates; the token didn't get easier: 16.98 down to 2.89, and those 14.09 bits are the forward pass's work.
bit = yes/no questions to “pick” this token from the candidates, not bytes to store it. Each halves them: questions = log₂(candidates).
Ⓐ At the start, the whole vocabulary: 16.98 questions; late in training, 2.89.
⭐ Key: context shrinks the candidates; the token isn't easier. Forward pass: 129,280 → effectively 7; the 14.09 bits saved are all its work (2^14.09 ≈ 17,500×).
⛔ “Effective,” precisely: the vocabulary stays; probability concentrates. (Pictured in the folded figure below.)
Sources and assumptions

No measured values here: perplexity = e^loss, bit = log₂(perplexity) = loss ÷ ln2; vocabulary 129,280 from the DeepSeek-V3 config.

⚠️ “Late-training loss around 2.0”: rough ballpark for public models, not one model's measurement.

Expand: what shrinks the candidates; “an effective 7” as a picture Picture version of the text above: no context, the whole vocabulary; with context, a handful; the right panel shows “effective”
Continued: what shrinks the candidates, and what “effective” means The text above the figure already says this; this is the picture version Ⓐ What shrinks them Ⓑ “Effective” Ⓐ Context shrinks the candidates: the token didn't get easier The key panel: those 14 bits are what the model saves No context (just initialized) Whole vocab in play: anything goes 129,280 candidates ⇒ 16.98 bits With context: “… the weather today is really” Most tokens pushed to near-0 probability: only a handful remain Effective candidates: 7 ⇒ 2.89 bits The step in between is the LLM's forward pass. It compresses 129,280 candidates to an effective 7.4: about 17,496× fewer. 16.98 − 2.89 = 14.09 bits: those 14 bits are all the work the model does. (And 2 to the 14.09th power is about that 17,496×.) Ⓑ What “an effective 7” means: the vocab didn't shrink Not “really only 7 left”: “as hard to guess as 7 equally likely ones” Real distribution (129,280 bars) Highly uneven … the other 129,275 are near 0 Equally hard 7 equally likely Each 1/7 That is all “perplexity = 7” means So perplexity is not a candidate count; it is “hesitating among about this many tokens.” The vocab stays 129,280; only the concentration changes.
Picture version of the three paragraphs above. Ⓐ Without context the whole vocabulary is in play; after “… the weather today is really,” most tokens drop to near 0; a handful remain. 16.98 − 2.89 = 14.09 bits; 2^14.09 ≈ that 17,496× (only the unrounded 14.0947 hits it exactly): all the model's work.
⛔ Ⓑ: “effective” means the vocabulary doesn't shrink; probability concentrates.
Sources and assumptions

⭐ Both panels were Ⓑ and Ⓒ of the last figure, split out on 2026-09-23 so the course page can fold them.

No measured values here: perplexity = e^loss, bit = log₂(perplexity); vocabulary 129,280 from the DeepSeek-V3 config.

Mental math: memorize just two numbers.

ln 2 ≈ 0.693 (think 0.7) · ln 10 ≈ 2.303 (think 2.3)

  • Each 0.7 off the loss halves perplexity.
  • Each 2.3 off the loss cuts perplexity 10×.

Loss falls from 3.0 to 2.3: a drop of 0.7, so perplexity halved. No calculator needed.

Check the ruler yourself: from 11.77 down to 2.0 is a drop of 9.77; divide by 0.7 = 14 halvings; 2¹⁴ = 16,384×.
Exact: perplexity 129,280 → 7.39, a ratio of 17,496. 16,384 vs. 17,496: under 7% off; the magnitude is right.

Only the order of magnitude. Mid-run you never need two decimals; you need “is this normal?”

Why does it “get more expensive later”? The phrase recurs; settle it once here.

⛔ First, a smooth-sounding wrong explanation: “the log scale squashes the later part.” No. In ratio terms nothing changes: each 0.7 drop always halves perplexity, at 11 or at 2.

Two real reasons.

Loss drops 1PerplexityCandidates cut
11.77 → 10.77129,314 → 47,57281,742
6 → 5403 → 148255
3 → 220.1 → 7.412.7
2 → 17.4 → 2.74.7

① The absolute gain shrinks. A “drop of 1” cuts 80,000+ candidates early, near 2 fewer than 5, since few were left.

② The root: part of what's left is inherent in language. The next token really can be several things; no one, humans included, can call it.
Late on you're not learning but hitting a ceiling: that entropy is irreducible, never 0.

📌 Hence two models' losses compare only on the same dataset and tokenizer. Change the data and the ceiling changes.

This ruler returns in 3.5 (is the loss falling fast enough?), 6.1 (how big a jump is a spike?) and 6.3 (reading that table). Without it, all three are gut feeling.

Aside · nats, bits, perplexity: three faces of one quantity

The ruler used two terms, nat and perplexity. With bits, they are three readings of the same quantity.

Expand: nats / bits / perplexity on one two-sided ruler Conversion in the text below (1 nat = 1.4427 bits); this draws the ruler, with ticks
Ⓐ Two-sided ruler: nats on top, bits below, same spot Like cm and inches on one ruler: different tick spacing, same length measured 0 2 4 6 8 10 12 nat The loss itself 0 2 4 6 8 10 12 14 16 18 bit × 1.4427 Just initialized · random 11.77 nats = 16.98 bits Early–mid training 7.00 nats = 10.10 bits Near convergence 2.00 nats = 2.89 bits Ⓑ One cell = one bit: length is measured, not labeled “How many bits the model needs, on average, to encode the next token” Just initialized · random 16.98 bits Near convergence 2.89 bits Compare the two bars: 5.9×. That is everything the model learned, expressed as compression. Ⓒ Perplexity = e^loss: “hesitating among N tokens” A third reading of the same quantity; again no measured values involved Near convergence = perplexity 7.4: hesitating among seven-odd tokens Just initialized · random … 129,280 in total The random-guess level is off the chart: its perplexity is exactly the whole vocab; “picking one of 129,000 tokens” is the definition of guessing. Nats and bits aren't two things: one ruler, different base A bit is the information in one binary choice, base 2; a nat is base e: a fixed factor of 1.4427. We use nats only because the loss is defined with ln. “Per token”: thousands of positions per sequence, one loss each, then averaged. Read it as “nats per token, on average.”
One quantity, three rulers: loss (nats) / compression (bits) / perplexity, on one axis.
Not three concepts but three readings: bit = nat × 1.4427 (base e → 2), perplexity = e^loss. The conversions contain no measured values.
The “Just initialized · random” level (Ⓐ, far right) is a self-check: perplexity must equal the vocab size 129,280, bits log₂(vocab). A mismatch means a wrong formula.
⛔ The takeaway is the compression: 17 bits → 2.9 bits: everything the model learned, as compression.
Sources and assumptions

⭐ The “Just initialized · random” level (Ⓐ, far right) is a self-check: perplexity must equal the vocab size 129,280; bits, log₂(vocab). Script-asserted; a mismatch means a wrong conversion.

⚠️ “Early–mid training 7.00”: illustrative level; “near convergence 2.00”: rough ballpark for public models. References, not any one model's measurements.

Nats and bits first: they differ only in the base.

A bit is the information in one binary choice (a coin flip's result); base 2.
A nat is the same with base e. The conversion is fixed: 1 nat = 1.4427 bits.

Why nats here? Only because the loss is defined with ln. With −log₂ p, everything would be identical, every number times 1.4427. A convention, not a deep choice.

Per token is from Step 3 below: each of a sequence's thousands of positions has a loss, averaged at the end. Full unit: “nats per token, on average.”

⛔ In bits, the number becomes tangible: a compression rate.

“Average bits the model needs to encode the next token.” A language model is fundamentally a compressor; the loss is its compression bill.

  • Random guessing: ln 129,280 = 11.77 nats = 16.98 bits. log₂ 129,280 is also exactly 16.98: picking one of 129,000 tokens takes 17 bits. Self-consistent, not a value someone typed in.
  • Converged: 2.0 nats = 2.89 bits.

From 17 bits to under 3: everything this model learned, as compression.

📌 Three faces; pick whichever fits:

  • nats/token: what the definition gives; what training logs print.
  • bits/token: compression; the most intuitive.
  • Perplexity: “hesitating among N tokens”; easiest for non-specialists.

They're identities, not three measurements: change one and the others follow. Don't cite them as three proofs.

Step 3 · One loss per token, averaged over a batch

So far, the loss at one position. In a 4K sequence every position predicts its next token, so one sequence has 4,095 losses.

⛔ It is 4,095, not 4,096: the last token has no “next” to predict. The 1 matters only as the denominator for the accounting in 1.2.

Average them all and you get this batch's loss.

One loss per token: 62,899,200 numbers, one average Where “how wrong it was” actually becomes one number Ⓐ The grid Ⓑ Divide by cells Ⓐ Across: positions; down: each sequence in the batch; one loss per cell Drawn as 8 × 10; really 15,360 × 4,095 Position 1 → position 4,095 (4,095 “next tokens” per sequence) 0.31 5.62 3.28 0.15 0.57 1.91 2.88 2.14 0.31 5.62 … Seq 1 0.44 7.81 1.38 0.23 0.36 0.31 5.62 3.28 0.44 7.81 … Seq 2 2.63 4.05 6.74 1.12 0.08 0.44 7.81 1.38 — — … Seq 3 1.91 2.88 2.14 1.07 0.92 — 4.05 6.74 1.91 2.88 … Seq 4 0.31 5.62 3.28 0.15 0.57 1.91 2.88 2.14 0.31 5.62 … Seq 5 0.44 7.81 1.38 0.23 0.36 0.31 5.62 3.28 0.44 — … Seq 6 2.63 4.05 6.74 1.12 0.08 0.44 7.81 1.38 2.63 4.05 … Seq 7 1.91 2.88 2.14 1.07 0.92 2.63 4.05 6.74 1.91 2.88 … Seq 8 …… … through sequence 15,360 Gray = unscored positions: they stay out of the denominator. Two kinds: ① Row-end padding: blanks fill out short rows so they stack. No text there; any guess is meaningless. ② Document seams: short documents share a row, to avoid waste. Guessing one's start from the last one's end: meaningless too. Sum everything then ÷ cell count One number This batch's loss 62,899,200 cells (15,360 × 4,095): each one a −ln p, each between 0 and 11.77. These 62,899,200 numbers end up as one. Everything later in the chain starts from this one number. Ⓑ The single division is by cell count, not sequence count The denominator counts cells, not rows Not “average each sequence, then average those.” Instead: every cell counts equally; sum, divide once. With uneven sequence lengths, the two give different results Per-row averaging overweights every token in a short sequence The figure in one line One loss per position: a batch is a 15,360 × 4,095 grid, 62,899,200 numbers. Sum them all, divide by the number of cells actually scored: one number. The whole chain after this starts from it.
Across: positions; down: sequences in the batch; one loss per cell. With V3's settings: 62,899,200 cells (15,360 × 4,095), each a −ln p. Sum, divide once: one number. This is where “how wrong” becomes one number.
⭐ Gray cells are unscored and out of the denominator: row-end padding (no text there) and document seams (guessing across a seam is meaningless).
⛔ Ⓑ in one line: divide by cells, not sequences; not each sequence first. With uneven lengths they differ: per-row averaging overweights short-row tokens.
Sources and assumptions

MaxText training script: loss = xent_sum ÷ total_weights, total_weights = target positions scored after masking padding and packing seams;

sequence length 4,096 and max global batch 15,360 come from the DeepSeek-V3 pretraining config.

“Average” is the root of much that follows.

The 1.2 figure's analogy: a room of people, each with a demand, averaged. The people: these tokens.

Batches exist first to avoid heeding one person, not to keep devices busy. This runs to Section 6 (not blowing up); math in 3.5b.

Step 4 · A single number, with units

Hundreds of billions of parameters in, one number out. No vector, no table: one number.

Remember this. 1.5 shows backprop is cheap entirely thanks to this “one”.

📐 Unit: nats/token (natural log). Section 3.3b uses it for the learning rate's units.

Expand: one number, or tens of millions? The grid draws it; this is the per-level count for self-checking. Skip when presenting

Almost everyone asks this once: “Why do you keep saying the loss is a single number? What I see is 129,280 numbers per token, 4,095 per sequence, and even more once you multiply by the batch.”

Everything you listed exists; it just isn't all called the loss. Two compressions sit in between.

LevelCountCalled
Token scores, one position129,280 logits / probabilities: not the loss
↓ First compression: keep only the correct answer's entry, take the negative log
One position's loss1−ln p
One sequence4,095One per position
One step (V3's global batch)62,899,200 (= 4,095 × 15,360 sequences)Still one per position
↓ Second compression: sum everything, divide by the number of valid tokens
Final loss1The point on your loss curve

Why compress it to one number? Not for looks.

“One number out” is what makes “one backward pass” possible.

Per 1.5, the seed goes on the right: one number, so you plant it once.
With 4,095 objectives you'd plant 4,095 seeds, one backward pass each. That is why multi-objective optimization costs more.

✋ Finally, an apparent contradiction:

“The loss is one number” and “the backward seed has over half a billion”: both true.

The loss as a quantity really is a scalar.
Differentiate it: one partial derivative per logit: every position × the whole vocabulary, ~530 million numbers (4,095 × 129,280; drawn in 1.2).

⭐ “How many numbers in the loss?” asks about the quantity; “in the seed?” about its derivative. One is 1; the other, over half a billion.

The lecture's job, in one sentence.

Move this number from 11.77 to ≈ 2.

That means knowing which way to nudge each parameter: this number's partial derivative for each of 671B parameters.

The problem: how to compute 671B derivatives?

1.1 The hard part isn't the derivative; it's 671B of them

Derivatives are high-school math. The hard part: 671B of them, fast, redone every step.

“High-school math” will lose some readers, so one figure covers the lecture's three words. No limits, no formulas: three sentences.

Derivative, partial, chain rule: one idea, an exchange rate This figure: no limits, no formulas; just what the three words mean Derivative: per notch Partial: hold rest Chain rule: multiply Ⓐ A derivative: not “what is it now?” but “move one notch, how far?” A parameter is a knob; loss, the reading on its right One parameter (one of 671B) nudge it up +0.01 → this parameter 1 notch loss 3 notches both rulers: 1 notch = 0.01 Then this knob's derivative is loss 4.00 → 4.03 (moved 0.03) 0.03 ÷ 0.01 = 3 Read as: you move 1, it moves 3 Not “reading = 4.00”: that's a value; this is a rate Note its unit, loss per parameter: the whole “how big a step” section is forced by it. Ⓑ What “partial” means, in three words: hold the rest A shower with 671B knobs: turn one at a time, hold all others held held held held only this one held held held held … the other 671B, all held Partial derivative = one knob's rate, all others held. Line up all 671B in one column, and that column is the “gradient.” Ⓒ But the knob isn't wired straight to the reading: several stages between Each stage has its own exchange rate; the total is all of them multiplied knob × weight activation slope 2.0 Intermediate A × weight activation slope 0.5 Intermediate B × weight activation slope 3.0 off by how much The orange box is the stage; the number above is the stage's slope, not the stage: weight × activation slope at this point. Total rate = all multiplied 2.0 × 0.5 × 3.0 = 3 This is just currency exchange RMB → HKD → USD, one rate per step The overall rate is a product, of course. That's all the chain rule is Start multiplying from either end: same result, but costs differ by 20,000 years. That's for the “forward vs reverse” figure coming up. Everything later in this lecture uses only these three sentences ① Derivative = move one notch, how far it moves (exchange rate). ② Partial = that rate, rest held. ③ Chain rule = every stage's rate, multiplied. No limits, no ε, no formulas to memorize. The one real trap: a rate is not a “value”. A reading of 4.00 and “move 1, it moves 3” are different; every “large gradient” below means the latter.
The way in: derivative, partial derivative and chain rule are one idea: an exchange rate.
Ⓐ A derivative asks not “what is it now?” but “move one notch, how far does it move?” Ⓑ “Partial” means hold the rest; the 671B knobs' numbers in one column are the gradient.
Ⓒ The chain rule is currency exchange: RMB → HKD → USD, one rate per step; the overall rate is their product. Which end you start from: 20,000 years apart (§1.5).
Sources and assumptions

⚠️ The 4.00 → 4.03 in Ⓐ and the ×2 / ×0.5 / ×3 in Ⓒ are made-up illustrative numbers. They just make ÷ and × visible. ⭐ The script asserts total rate = product of the three, and at least one stage shrinks; else “multiply them” reads as “it only grows,” against the vanishing/exploding-gradient section.

⭐ This figure deliberately avoids limits: strictly, a derivative is the nudge → 0 limit; this is a difference quotient. ⛔ Fine here: §1.1's brute force is exactly a difference quotient, so loose fits better.

⭐⭐ “Exchange rate” isn't decoration: two hooks, both ours: in a chain of products, the starting end shifts cost by 20,000 years (§1.5); mismatched units need one more conversion factor (§3.3).

Expand: a real stage is not a scalar Above is a 1-D toy (one knob, one reading). A real stage is 7,000-odd dims in and out; its “slope” is a whole table. We never compute that table. Expand for details
One more panel: a real stage is not a scalar The figure above is a 1-D toy; here is how a real stage differs. One conclusion: we never compute that table Slope is a whole table But a real stage is not a scalar: its “slope” is a whole table The three panels above are a 1-D toy: one knob, one reading. A real stage has 7,168 dims in, 7,168 out in 7,168 numbers stage weights + activation (e.g. 7,168→2,048; none square) out 7,168 numbers So what is this stage's “slope”? Not one number: every output dim, against every input dim, has its own exchange rate. But note: the machine holds no such square matrix. Real weights: none is square; the table exists only conceptually. The good news: we never compute that table. Backward does one thing: multiply blame by it, which directly gives the blame owed upstream. That is the second of the backward pass's two matmuls. So “multiply all the way” is true; you just multiply tables.
The panels above are a 1-D toy; here is how a real stage differs. A real stage is 7,168 dims in, 7,168 out. Its “slope” is not one number but an exchange rate per output dim per input dim: conceptually, a whole table. ⛔ But the machine holds no such square matrix; no real weight is equally wide on both sides.
⭐ The one thing to keep: we never compute that table. The backward pass only “multiplies the blame by it”: the second of its two matmuls.
Sources and assumptions

⭐ Originally the previous figure's fourth panel; split out on 2026-09-23 so the course page could fold it away.

⚠️ 7,168 is V3's residual stream width. ⛔ The machine holds no square matrix; no real weight is equally wide on both sides.

The names explain it: derive, partial

A common mix-up: not deviation, but derivative. Deviation is departure from a reference (standard deviation), unrelated here. The term is partial derivative.

Derive is a picture: Latin dērīvāre = de (from) + rivus (stream): literally “to draw water off a stream”.

📌 Source: Etymonline (derive / derivative): Latin derivare “to lead or draw off (a stream of water) from its source,” from de rivo (de “from” + rivus “stream”); the math sense dates from the 1670s.

To derive is to draw something off. Today it suggests deduction, and the water picture is lost.

With that picture, both terms follow.

Derivative first.

A stream: input in, output out. A derivative asks not “what is it now?” but “move one notch, how far does the output move?”

Not a new stream: a second one drawn off this one. The first gives the output; the second, change per notch. One source, two streams.

Then the partial derivative (not deviation).

Now the input is 671B knobs. “Move the input one notch” breaks down: which knob?

The fix is blunt: free one knob, clamp the rest. Partial means “in part”: one variable at a time.

671B dimensions collapse to one: a partial derivative is an ordinary derivative again. Not harder: it cuts the hard problem into 671B ordinary ones.

First, brute force. Without it, backpropagation looks unremarkable.

⛔ Brute force (finite differences): nudge one parameter, rerun the whole forward pass, and see how far the loss moved.

671B parameters: 671B forward passes.

671e9 s ÷ 3.156e7 s/yr = 21,261 years (1 s per forward pass, already wildly optimistic)

That's one step of hundreds of thousands.

Backpropagation's real job isn't differentiating; it's collapsing 671B forward passes into one.

1.2 The seed: the first gradient

It starts from §1.0's loss. (Just one; that's the key to what follows.)

The first gradient? Take cross-entropy, the most common loss. The network outputs a distribution (next token “the” 0.3, “a” 0.2…); the answer is 1 in one slot, 0 elsewhere.

First gradient = guess − correct answer.

Too high: positive; too low: negative. It seeds the whole backward chain.

No coincidence: it is §1.0's −ln p differentiated once.

Differentiate with respect to the logits (not probabilities) and −ln p is exactly p − y: the model's probabilities minus the one-hot answer.

Back to §1.0: one −ln p, two derivatives, two results.

  • With respect to probability p: −1/p; lower probability, larger gradient. That's the motive: fix the worst mistakes first.
  • With respect to the logits: it reduces to p − y. That's the implementation: the seed is one subtraction.

One says “why move,” the other “how”: two faces of one loss.

So frameworks always fuse the two into one op: one step, not probabilities then derivative. Few losses are this pretty; this pair is today's default.

Precisely: that “subtraction” is the derivative with respect to the logits, not the probabilities (the pre-softmax output).
It matters: as the softmax step cancels, the gradient doesn't vanish where softmax saturates, half the reason softmax and cross-entropy always pair up.

What is a gradient? Everyone asks; take the average No formulas here; just what this actually does ① One example asks ② Many voices: average Ⓐ One example sees the output and makes one request Correct answer is “the”; the network gave “the” only 0.3 off 0.70 the now 0.30 off 0.20 a now 0.20 off 0.15 in now 0.15 off 0.10 and now 0.10 gray = network now · dashed = correct · arrow = how much to push Its request, in one sentence “the” up, everything else down and push each by how far off This is the lecture's seed “prediction − truth”, in plain words It's only a “wish”: no one can edit outputs directly Ⓑ Every position makes its own request: thousands of forces add into one Note: each arrow = one position, not a token (that's Ⓐ); this level is between positions Seq 1 pos 1 pos 2 … 4,095 positions Seq 2 … 4,095 positions Each arrow = one position's request to this weight (which way, how much) They point every which way: this is what “too many voices” looks like. ⊕ Sum them all (batch 2 = 8,190 arrows) Net force Lengths sum to 6.8; net is only 2.06 most of it cancels what's left is the real signal This net force is the weight's move this step. “Much cancels, little remains” is the whole reason bigger batches are steadier: More voices drown out outliers. Cross-device reduction is this very sum: each device sums its batch, then all combine. All of backpropagation, in plain words: three steps One example makes a request → passed back layer by layer → all requests averaged. Tensors, chain rule, matmuls just implement these three steps, nothing more. This lecture's bill is all in step 2: passing a request back needs what the forward pass left behind: the whole reason activations stay and memory won't drop.
The lecture's plainest figure: “Gradient” sounds scary; it's just a roomful of requests, averaged.
Ⓐ Each example makes one request: push the right one up, the others down, each by how far off it is: plainly, “prediction − truth”.
Ⓑ Heed one example and the network answers everything with one token. So we average; the cross-device reduction is this average.
Sources and assumptions

📌 Four ideas are from the official lesson notes for 3Blue1Brown's “What is backpropagation really doing?”: nudge ∝ how far off; three ways to brighten a neuron; weights scaled by upstream brightness for max return; and averaging, as you can't please every example (heed only a 2: all become 2s). Each was checked against the original; figures are our own redraws, recast for next-token prediction.

⚠️ On scaling with upstream brightness, he mentions Hebbian theory in passing (neurons that fire together wire together), but the original itself says the analogy is loose (an untrained network isn't “thinking” of that answer), so it is noted, not drawn.

⭐ The links Ⓑ→§1.8 (and folded figure→§1.6) are this lecture's own synthesis, not the original's. 3B1B explains “what backprop does”; we need “why it costs so much memory.”

Expand: a weight's gradient scales with upstream brightness Another question: which wire is cheapest to change? Forward answers it, not backward: root of 1.6's “can't discard activations.” Expand
A weight's gradient ∝ upstream brightness Two wires, thickened equally: 20× gap in return. The return's size is this weight's gradient bright dim return ratio = brightness ratio Same +0.1 thickening: 20× the return on the bright wire A side axis: which wire is cheapest to change 4.0 upstream bright +0.1 thicker 0.2 upstream dim +0.1 thicker out output +0.40 output +0.02 Same effort 20× the return exactly the brightness ratio A weight's gradient = upstream brightness × downstream blame. This panel: the first factor. So “which wire is cheapest to change” is answered in the forward pass, not the backward: by the brightness at that moment. And that brightness is computed in the forward pass and must be saved. The whole memory bill grows from this one sentence.
A weight's gradient = upstream brightness × downstream blame: this panel: the first factor. Equal +0.1 on two wires: 20× brighter upstream, 20× the return.
Not “how to change weights”: “which wire is cheapest to change” is answered forward, not backward, by the brightness then, which the forward pass computes and must save: root of “activations can't be discarded” (1.6).
Sources and assumptions

📌 “Adjust weights in proportion to upstream brightness for the biggest return” is from the official 3Blue1Brown notes for “What is backpropagation really doing?” ("adjust the weights in proportion to their associated activations"). Figure: our own redraw.

⚠️ He briefly mentions Hebbian theory (neurons that fire together wire together), but the original itself says the analogy is loose, so it's noted, not drawn.

⭐ The step to “so activations can't be discarded” is our own synthesis: 3B1B explains “what backprop does”; we need “why so much memory.”

1.2b Push the right one up, the wrong ones down: one move

✋ A revealing class question: does backward push the right one up and the others down, or just the right one, the rest a side effect?

Both, and it's one and the same move.

The seed, “guess − answer,” is a whole row, one slot per vocabulary token:
the correct slot: 0.40 − 1 = −0.60 (negative = push up); every other slot: its own probability − 0 (positive = push down). The 0.20 token gets exactly twice the push of the 0.10 one.

⭐ The row sums to exactly 0. Probability is conserved, so up and down are one redistribution. The other 120,000-plus tokens sit at probability ≈ 0, gradient ≈ 0: no share, no move.

⛔ Two conventions (the figure's “example” can mislead): ① 4,096 tokens give only 4,095 predictions: the last has no “next”; ② loss averages per token, not per sequence: the denominator: target positions actually scored, padding and packing boundaries masked (MaxText: total_weights; loss = xent_sum ÷ total_weights).

How a whole batch of 63M losses (15,360 sequences × 4,095 positions) becomes one gradient Asked in class: “Push the wrong ones toward 0, or push only the right one, rest a side effect?” Answer: both, as one move Ⓐ 1 position Ⓑ batch = 1 Ⓒ batch = 2 Ⓐ First, one position: here every token in the vocabulary gets its own share Correct answer is “the”, given 0.40; the other 0.60 is spread over other tokens 0.40 the -0.60 0.20 a +0.20 0.20 in +0.20 0.10 and +0.10 0.10 of +0.10 gray = current probability · number below = this slot's gradient = guess − correct negative = push this slot up · positive = push this slot down So the answer to that question: both, and it's one and the same move The wrong ones aren't a side effect: each gets its own share, force = the share it holds: the 0.20 one is pushed twice as hard as the 0.10 one. The row sums to 0 (−0.60 + 0.20 + 0.20 + 0.10 + 0.10) Probability is conserved, so this is not a score bump, but a redistribution. The other 129,275 tokens: prob ≈ 0, gradient ≈ 0; they held no share Ⓑ Zoom out to a whole sequence (batch = 1): all of the above, at every position 4,096 tokens, only 4,095 predictions (the last has no “next”). Each arrow: a bundle, not a line …… 4,095 positions, each with an opinion as long as the vocab (129,280 numbers) The arrow isn't “which way this position pushes”: at this level there's no single direction One position's opinion vocab 129,280 rows; each row pulls its own way stack net force: 7,168-d So “one direction” is wrong: every vocab row pulls a 7,168-d vector; all 129,280 rows summed → this arrow. How exactly it sums: next section. batch = 1 in total 529,401,600 seed numbers each ÷ 4,095 Ⓒ batch = 2: only the denominator changes, from 4,095 to 8,190 All of the “averaging” is this one division, and it is done at the seed Seq 1 … 4,095 positions Seq 2 … 4,095 positions 8,190 opinions, each ÷ 8,190 first, then all summed into the same 671B parameter gradients So “one backward pass per token, then average the 671B gradients” vs “average the loss, one backward pass”: identical result (differentiation is linear); the latter is 8,190× cheaper. This figure in one sentence “Push the right one up” and “push the wrong ones down” are one move: the row sums to exactly 0, a redistribution; the wrong ones aren't a side effect. Loss averages over tokens, not sequences, but backward doesn't start from that scalar: it starts from a “position × vocab” tensor of (guess − correct) ÷ total. What one position sends back is a net force (129,280 rows, each pulling a 7,168-d vector), not a direction.
The class question: push the wrong ones down too? Yes, and it's the same move as pushing the right one up.
Ⓐ At one position, each vocabulary slot gets its share of “guess − answer”: negative pushes up, positive down, strength = its current share; the row sums to exactly 0: redistribution, not a score bump.
Ⓑ A sequence has 4,095 such positions (not 4,096); Ⓒ batch 1 → 2 changes only the denominator: all the “averaging” is this one division.
Sources and assumptions

MaxText training script: loss = xent_sum ÷ total_weights (target positions actually scored, after masking padding and packing boundaries).

Gradient computed once (value_and_grad on that scalar). Vocab 129,280, seq length 4,096: DeepSeek-V3 pretraining config.

Ⓑ's arrows are easy to misread: a position's one arrow back is not “which way it pushes.”

Every vocabulary row pulls on a 7,168-dim vector; all 129,280 summed make that arrow: a net force, not a direction.

“Push up / down” applies only to one vocabulary slot (level Ⓐ); at “one position” it refers to nothing. So Ⓑ's arrows are neutral, not green or red.

⭐ How that net force is summed is all of §1.2c.

And “one backward pass per token, then average the 671B gradients”? Exactly the same result.

Differentiation is linear: averaging and differentiating commute. So implementations run backward once: not a shortcut, but equivalent and tens of millions of times cheaper.

⭐ The scalar is only a handle; backward doesn't start from it. It starts from a “position × full vocabulary” tensor, each entry “guess − answer” over the total. None of the 60-odd million positions' votes is erased at the loss; they add up later, in each layer's matmuls.

1.2c How one position's request becomes 671B numbers

⛔ A natural misconception that inverts the whole chain: different positions' 7,168-dim vectors can't be summed first.

The temptation: 4,095 positions, each a 7,168-long vector, so average them and pass the average back. Do that and the whole gradient is wrong.

Position i's blame pairs only with position i's input. The weight gradient multiplies each row by its own partner, then sums; summing the blame δ first pairs position 3's blame with position 900's input, a different quantity.

Summation does happen, on the weight gradient, not δ, once per layer. δ stays a “position × width” table; the position dimension survives to the end.

How one position's request becomes 671B numbers Retold in class: steps 1–4 right. Step 5, “sum those 7,168s,” can't be done there, and it matters most Ⓐ One force Ⓑ Keep positions Ⓒ Sum too early Ⓓ Where summing happens Ⓐ Step 1: fold the vocab-long request into one direction Each token owns a direction in the model; weighting by its number gives one force 129,280 numbers one per token = “guess − correct” near-zero-prob tokens too: entry ≈ 0 through output matrix (129,280 × 7,168) 7,168 numbers its blame (δ) a direction, not a target What does this step do? Net force. Like five people pulling one ring: “the” pulls in with 0.60, “a”, “in” push out 0.20 each, “and”, “of” 0.10 each. The ring moves in one direction: that net force is δ. So “just handle the −0.60” won't do: “Go north” vs “go north, away from those three”: different paths. Ⓑ One sequence: 4,095 such vectors, stacked into a table 4,095, not 4,096: the last token has no “next”. The table's position dimension is kept to the end pos 1 pos 2 pos 4,095 width 7,168 (each position's δ) height 4,095 one row = one position's net force batch = 2: two such tables stacked 8,190 rows × 7,168 cols Each row is already ÷ 8,190 when created The “averaging” division was done at the seed. Ⓒ Step 5 from class, “sum those 7,168s”: must not happen Because each row's blame must pair with its own row's input ❌ Sum 4,095 rows into one, then pass back (sum of all δ) ⊗ (sum of all inputs) Sum like this, and “position 3's blame” ends up paired with “position 900's input”. “Who asked, in what context” is all the information in this gradient: sum it away, it's gone. A different quantity, not the correct one. Pair row with row; sum after multiplying Σ (row i's blame ⊗ row i's input) Each position multiplies its own pair, giving its own contribution to the weight; these contributions are summed. So summation does happen, just not on δ, but on the weight gradient. Once in each layer. Ⓓ The whole chain: position dim kept to the end, each layer settles once one position 129,280 numbers vocab-long request fold once 7,168 numbers net force = its δ one sequence 4,095 rows a table; rows not summed through 61 layers once per layer δ × own input → contribution settle 671B numbers sum contributions per weight So the “countless 7,168s summed into an average expectation” from class: “summed” is right; “summed at the 7,168 level” is wrong. The sum waits for the last step, and it sums contributions to the weights, not δ itself. This figure in two sentences δ is a “position × width” table from start to finish: the position dimension is kept, because position i's blame can only pair with position i's input. Summation happens on the weight gradient, once per layer: all positions' and examples' contributions to a weight, summed, give this step's move for it.
The whole chain, one position to 671B numbers. Ⓐ The vocabulary-long request folds into one net force (each token a direction, weighted by its number); Ⓑ a sequence is 4,095 rows of these, stacked.
⛔ Ⓒ is the trap: rows can't be summed first; row i's blame pairs only with row i's input; summing first mismatches them.
Ⓓ So summing happens on the weight gradient, once per layer; positions survive to the end.
Sources and assumptions

Linear: weight gradient = Σ_i (δ_i ⊗ x_i), contracted over positions; input gradients aren't. ⚠️ One honest exception: attention mixes δ across positions, which already see each other going forward.

That's attention's structure, not “averaging”; in linear layers, positions stay strictly independent. Vocab 129,280, width 7,168, 61 layers: DeepSeek-V3 config.

1.2d Settling up: where weight gradients are summed

⭐ Deeper: when does “settling up” happen? Asked in class: “Per function, or after the whole layer?”

Per function: each weighted op settles alone, independent of passing back.

Given δ, a weighted op does two things:
① Settle up: δ × this layer's input, summed over positions, updates this weight. Positions are consumed; nothing passes back.
② Pass back: δ × weight = the earlier layer's blame. Positions survive.

⭐⭐ These are 1.6's “two backward matmuls” behind training's 3× cost. One δ table, two contractions: one eats positions, one keeps.

⛔ Two common mistakes: ① The weight-gradient bucket is fixed-size: 4,000 positions just add 4,000 entries. Hence “one number per parameter,” not “one parameter set per token.”
② Layers add, never divide. The “÷ total” happened once, in the seed; dividing in all 61 layers would do it 61 times. Strictly a sum, not an average: the averaging is already in every entry.

As the class reasoned, a linear layer's weight gradient uses only same-position input; earlier tokens entered it via forward attention, not in backward.

Settling up: where dW is summed Asked: “once per function, or after the whole layer?” — once per weighted op, no waiting Ⓐ 2 matmuls Ⓑ Who settles Ⓒ Fixed bucket Ⓐ In the backward pass, a weighted op does two things: Section 1's two matmuls One δ table, two contractions: one eats position, one keeps it | a real op: expert up-proj, 7,168 → 2,048 δ (blame, output side) 4,095 rows × 2,048 cols 2,048 = output width: expert's “waist” x (input, entry side) 4,095 rows × 7,168 cols 7,168 = residual-stream width Common dim 4,095 = positions (4,096 tokens → 4,095 “next tokens”) ① Settle: position gone δ × x, summed over positions a 2,048 × 7,168 update, same size as W The 4,095 vanishes here — each position's opinion of W merges into one. Not passed back — it stays with W until the update. ② Pass: position kept δ × W itself still a 4,095-row table rows stay separate — the earlier layer's blame. Position kept all the way. Section 1's “backward does two matmuls, so training costs 3× inference”: these are the two. Ⓑ A dozen-plus ops per layer: weighted ones settle, the rest pass Answer: once per op, not after the whole layer Norm settles Q/K/V proj settles Attn scores passes softmax passes Weighted sum passes Out proj settles Residual add passes Norm settles Router score settles Expert gate/up settles SwiGLU mul passes Expert down settles Residual add passes δ flows ←, settling at each orange box Orange: own params → settles. Gray: no params (scores, softmax, elem. ×, +); reshapes blame, hands it on. So no “per-layer average”: nothing waits for a layer to end. For MoE: a token adds only to its chosen experts; the others get nothing this step. This row is schematic: a real MLA + MoE layer has more; who settles is the same. Ⓒ One fixed-size bucket per weight, all positions add in Why the gradient is “one number per param,” not “one set per token” Pos. 1 entry Pos. 2 entry Pos. 3 entry Pos. 4 entry Pos. 5 entry Pos. 6 entry Pos. 7 entry … 4,095 entries (8,190 at batch = 2) W's bucket 2,048 × 7,168 values never resized Two common traps: ① More tokens, same bucket: 4,000 positions just add 4,000 entries to it. ② Add, never divide. The “÷ total” is in the seed, done once — 61 layers would divide 61 times. Strictly a sum, not a mean: the mean is folded in already. Two takeaways A weighted op's backward does two things: δ·input, summed over positions → dW (position gone); δ·W → blame passed back (kept). That's Section 1's two matmuls. Settling: once per weighted op, not per layer; fixed bucket, add only, never divide.
The lecture's most-questioned point; this section's hinge. Ⓐ In backward, a weighted op sums over positions (weight gradient; positions consumed) and keeps them (blame passed back): 1.6's two matmuls.
Ⓑ Weighted ops settle alone, the rest pass through: no “finish the layer, then average.”
Ⓒ Fixed bucket, add-only. Nobody waits to settle, so multi-device training can hide communication in compute (folded below).
Sources and assumptions

⭐ The class got this right: “my token must use the activation that produced it, i.e., after all earlier tokens took part.” A linear layer's weight gradient uses only its own position's input,

and earlier tokens were already mixed into that input by forward attention, not pulled in by backward. ⛔ Scope: no V3 layer has a 7,168 × 7,168 square matrix. The residual stream is 7,168 wide; every matrix's other side differs (experts 7,168 → 2,048; MLA compresses KV to 512; o_proj's is 16,384). Example: expert up-projection. 61 layers total.

Expand: settling up doesn't queue Minor here; key in Topic 5 · parallelism strategies: late layers finish settling while early ones still compute, and finished parts ship meanwhile: “hiding communication behind compute.” Expand to preview.
Aside: settling never queues Not load-bearing in this lecture; pays off in Topic 5 · parallelism Order: no waiting Order: settle on arrival; no one waits loss end input end Op 1 settle Op 2 settle Op 3 settle Op 4 settle Op 5 settle Op 6 settle Why multi-device training hides comms: late layers finish while early ones run — use that window to ship what's finished.
Settling up doesn't queue: each layer settles its weight gradient once its blame arrives. ⭐ Minor in this lecture; it pays off in Topic 5 · parallelism strategies: the last layers finish settling while earlier ones still compute, and that window ships the finished part. That's “hiding communication behind compute.”
Sources and assumptions

⭐ Originally the previous figure's fourth panel; split out on 2026-09-23 so the course page could fold it away.

1.2e “Average”: how many levels?

How many levels does it have? Class guess: within a sequence, across local sequences, across micro-batches, across devices.

Layered, yes: three levels of addition + one division.

  • ① Within a sequence + across local sequences: one level. δ is “(local sequences × length) × width”; one weight-gradient matmul contracts it all.
  • ② Across micro-batches: gradient accumulation, one bucket, add only.
  • ③ Across devices (DP): all-reduce sums.
  • ④ Division: just once, by the global valid-token count.

⛔ Why not divide per level? That's an average of averages. Valid-token counts differ (padding, packing boundaries, the last batch). Divide separately and a small batch counts as much as a big one, with no error, no divergence, a normal loss curve.
Instead carry “loss sum” and “token count”; divide once at the end.

Average: how many levels? Layered, yes, but really: 3 add levels + 1 divide Ⓐ Assumed Ⓑ Actual Ⓒ Why not divide per level Ⓐ Natural guess: average per level Within sequence average once Across seqs on device average again Across micro-batches average again Across devices average again 4 divisions; really just one — and one level doesn't exist. Ⓑ Actually: 3 levels of adds, 1 divide ① Within + across seqs these two are one level δ shape: (device seqs × seq len) × width, one dW matmul contracts it all ② Across micro-batches Add to one bucket Grad accum: add each batch, never divide ③ Across devices (DP) all-reduce sum Every device's bucket added together ④ ÷ once ÷ global valid token count only this once So summing is layered, but each level only adds; division happens just once. On ①: “within-seq” and “across-seq on device” averaging are one matmul, not two steps — the two dims are already one. Ⓒ So why not divide per level? Average of averages Premise: valid token counts differ (padding, packing, last batch) micro-batch 1 valid tokens 100 · this batch's loss sum 200 → own avg 2.0 micro-batch 2 valid tokens 20 · this batch's loss sum 60 → own avg 3.0 ❌ Divide each (2.0 + 3.0) ÷ 2 = 2.50 Sum all, ÷ once (200 + 60) ÷ (100 + 20) = 2.17 A 15% gap, yet the wrong one gives no error, no divergence, a normal curve; it silently reweights each batch. Cause: dividing first means “all batches equal” — their token counts aren't. So the fix: carry just two numbers — “loss sum” and “token count”; divide at the end. In short Adds at 3 levels (matmul → accumulation → all-reduce), one division: by global valid tokens. Carry two numbers: loss sum, token count. Any early division is an average of averages: no error, just silent reweighting.
How many levels? Three of addition, one division. Ⓐ The natural guess: four averaging levels. Ⓑ But within-sequence and across-local-sequence are one matmul; accumulation and cross-device only add; one division.
Ⓒ The catch: per-level division gives an average of averages: with unequal token counts, 15% off, no error, normal curve.
Source, line by line, folded below: carry only “loss sum” and “token count,” divide once.
Sources and assumptions

Source: MaxText gradient accumulation. In the loop, acc["loss"] += xent_sum and acc["total_weights"] += total_weights; after it, grads ÷ acc["total_weights"].

⚠️ One exception: auxiliary losses (MoE load balancing etc.) are instead divided by the micro-batch count, since each is already “one average per micro-batch.” Two conventions; don't mix.

Expand: the source code; not my derivation MaxText's gradient accumulation does exactly this, line by line: sums, not averages; token count tracked separately; one division at the end, by the token total (not the micro-batch count).
Aside: the source — not my own derivation Evidence, not content: for self-checking; not used in class MaxText grad accum Not derived: the grad-accum code, line by line It adds sums, not means acc["loss"] += batch loss sum Token count kept separately acc["total_weights"] += batch valid tokens Then ÷ once, by summed token total grads ÷ acc["total_weights"] It divides not by the micro-batch count but by the summed token total: that dodges the trap above.
“Three levels of addition, one division” isn't derived: it's MaxText's gradient-accumulation code, line by line. Sums, not averages; token count summed separately; one division, by the summed token total, not “micro-batch count”: that dodges the trap.
Sources and assumptions

⭐ Originally the previous figure's fourth panel; split out on 2026-09-23 so the course page could fold it away.

1.3 The common snag: what flows back is not one number

Many picture one gradient value passed down. It isn't.

What flows back is a whole tensor, shaped exactly like this layer's output: each output position's share of the loss's blame. We call it δ.

⭐ Name δ, or the rest is jargon: δ is the activations' partial derivative.

δ is the loss's partial derivative with respect to this layer's output, the activation. So backward must differentiate by activations: layer k's weight gradient needs the loss's sensitivity to layer k's output, which only layer k+1 can send. δ is that chain.

But differentiating it isn't changing it, an easy confusion:

Take grad?Up­date?Then
WeightsYesYesSum across devices (÷ once, at end), to opti­mizer, 2 B/param
Activ­ationsYes (= δ)NoPassed back, freed after use, never billed

⭐ Activations aren't changed: they're not free variables. Inputs and weights determine them; only new weights change them, so optimizer state to “turn” them is pointless.

❓ δ is a full tensor too; why isn't it on 1.7's bill? It exists only at backward's current layer, freed after use. Activations live from forward to backward: 1.6's mountain. ⭐ Same size, one billed, one not: only lifetime differs.

⛔ A natural, misleading phrase: “activations are just a medium passing data along.” In forward, yes. But a mere medium would be dropped after forward, and they can't be: each layer pulls them out as input for its weight gradient. Better: forward passes activations; backward passes δ, and activations get pulled out to settle up.

Given δ from the next layer, a layer does two things (numbered as in the figure below; ① is forward):

  • ② Its weight gradient: δ × the input saved in forward.
  • ③ The δ to pass back: δ × its own weights.

One matmul each. One forward, two backward: hence 3× compute.

Counting isn't enough: unequal sizes would break 1 + 2 = 3. It holds because all three share one set of dimensions. For [B,in] × [in,out]: ① forward X·W, ② Xᵀ·dY, ③ dY·Wᵀ. Each is B × in × out, the same volume: exactly 3×. (Drawn to scale.)

The reason transfers: the last layer passes nothing back: drop ③, 2×. Frozen layers skip weight updates: drop ②, also 2× (3.8's LoRA). Memorizing “3×” answers neither.

3× is counted: 1 matmul forward, 2 backward No chain rule, just count matmuls 1 forward 2 backward 3 total Ⓐ One layer unrolled: 3 matmuls (1 forward, 2 backward) Matmuls only; norm / activations / bias ignored All squares same size; shapes differ by square count (schematic: T = 6 · d_in = 4 · d_out = 3). Transpose = same squares, rearranged. ① Forward compute output input X 6 × 4 × weight W 4 × 3 = output Y 6 × 3 → next layer ② Backward · dW “how to change my W” input Xᵀ 4 × 6 × blame dY 6 × 3 = gradient dW 4 × 3 → optimizer This isn't new: it's ①'s input, saved ③ Backward · pass “what to pass back” blame dY 6 × 3 × weight Wᵀ 3 × 4 = new blame dX 6 × 4 → prev layer All three: same MAC count (positions × d_in × d_out), so 1 + 2 = 3 isn't a metaphor; it's counted. ②'s dashed block = why activations can't be discarded. · The two ᵀ? See the folded panel below. Ⓑ Key: one line in, two branches out Not vague “backward is slower”: two branches, both required incoming blame δ (just this one) × saved input one matmul W's gradient to optimizer ends here What the step really needs · branch stops here × this weight one matmul new blame δ fed back continues ← The chain; drop it and it breaks Different operands (input vs. W): can't merge; lose one, chain breaks; lose the other, no update. Both needed: hence the 2. Ⓒ So the tally closes Inference: fwd only 1 pass = 1× Training: fwd + bwd 1 pass 1 pass 1 pass = 3× Training + full recompute 1 pass 1 pass 1 pass 1 pass = 4× 1 pass = every matmul in the network, once. Inference buys 1; training buys 3 Two other common questions “Why does training cost so much more than inference?” Compute: 3×. But the real gap isn't that; it's activations held in HBM front to back (next fig.). “Can backward be one matmul?” Yes, if you don't pass further back (e.g., tune only the last layer): skip ③. Not for frozen layers: LoRA, with an adapter per layer, still needs every ③; it skips ② (dW). See 3.8.
Ⓑ is the key: backward isn't just slower; it answers two questions: how to change my weight, and what to send upstream. Two questions, two matmuls.
⛔ Ⓐ Step ②'s “input saved in forward” is the topic's hinge: undiscardable activations and Section 2's trade-off hang on it.
Sources and assumptions

⚠️ “3×” is the usual matmuls-only approximation, ignoring norm, activation functions, biases, communication. Their real share is small but not zero.

⛔ Step ②'s “input saved in forward” is the hinge of the whole topic: undiscardable activations and Section 2's recompute trade-off hang on it.

Expand: what are the two “ᵀ”s? A transpose costs no compute (rows ↔ columns, no number changes, data often not even moved), and the shapes force it: dW must match W and dX X, so only a transpose fits. Expand for details.

The figure above has two ᵀs: Xᵀ and Wᵀ. Unpack it first, or the rest won't make sense.

① What it is: rows ↔ columns, no number changes

A 2-row, 3-column table becomes 3 rows and 2 columns after a transpose: the first row becomes the first column. No number is lost or changed; only the reading direction changes.

So a transpose costs no compute: implementations often move nothing, just read it differently.

② Why it's unavoidable: the shapes force it

Matrix multiplication has one hard rule: columns on the left must equal rows on the right. The multiply consumes the matching dimension.

Backward's two results have fixed shapes: dW must match W (so the optimizer can add it), dX must match X (so the previous layer can take it).

ArrangementConsumed dim.Why forced
① Forward X[T×d_in] · W[d_in×d_out] = Y[T×d_out] d_inOriginal: mix the input by the weights
② Weight gradient Xᵀ[d_in×T] · dY[T×d_out] = dW[d_in×d_out] T (positions)Only way to get d_in×d_out
③ Pass back dY[T×d_out] · Wᵀ[d_out×d_in] = dX[T×d_in] d_outOnly way to get T×d_in

With the result shapes fixed, a transpose is the only fit. It isn't an extra operation; it's the answer to “how do these have to line up?”

Two sentences, in plain words

Forward asks “how is the input mixed into the output?” ③ reverses it: “how is the output's blame split back to the input?” Same W, read backward: Wᵀ.

In ②, Xᵀ does something else: it moves the position dimension to the middle so it gets consumed. Every position's opinion of this weight is summed here: exactly 1.2d's “settling up.” The position dimension vanishes, so this result isn't passed back.

Aside: the transpose In short: transposing is free (rows ↔ cols, nothing changes); shapes force it What it is, then why it's needed The transposes: no extra step, same table read the other way Two parts: what it is, then why it's forced ① What it is Rows ↔ cols: nothing changes 1 2 3 4 5 6 A (2 × 3) transpose 1 4 2 5 3 6 Aᵀ (3 × 2) Yellow row → yellow column. No value lost or changed; only the read direction. So it costs no compute; often nothing even moves. ② Why it's forced Matmul's one rule: left's columns = right's rows ① X [T × d_in] × W [d_in × d_out] = Y [T × d_out] Matching middle dim: d_in, which gets consumed Original: input mixed by W ② Xᵀ [d_in × T] × dY [T × d_out] = dW [d_in × d_out] Matching middle dim: T (positions), which gets consumed For a W-shaped result, the only way ③ dY [T × d_out] × Wᵀ [d_out × d_in] = dX [T × d_in] Matching middle dim: d_out, which gets consumed For an X-shaped result, the only way So a transpose isn't an extra op: once result shapes are fixed (dW like W, dX like X), it's the only fit. Plainly: forward asks “how input mixes into output”; ③ asks “how output blame splits back”. Same W, read the other way, is Wᵀ. · ② uses Xᵀ to put positions in the middle to be consumed — that's 1.2d's “settling up.”
The short version. ① A transpose costs no compute: rows become columns, no number changes (often not even moved in practice). ② The shapes force it: matmul requires “columns on the left = rows on the right,” and backward's two results have fixed shapes, dW must have W's shape, dX X's, so a transpose is the only fit.
Sources and assumptions

⭐ Originally the previous figure's Ⓑ; split out on 2026-09-23 so the course page could fold it away.

⭐ The orange line (asked in class): ②'s Xᵀ isn't recomputed; it's ①'s forward input, saved. That line fully answers “why activations can't be discarded”: backward needs X for dW. The arrow follows the flow: ① saved → ② used.

Walk the numbers: 3 nodes, one forward, one backward All principle so far. Nodding is easy; the numbers show where you're stuck Black: forward Red: gradient Ⓐ The formula: f = (a + b) × c Each wire carries two numbers: black on top (forward), red below (backward) a = 2 grad 4 b = -3 grad 4 c = 4 grad -1 + × f = -4 grad 1 q = a + b -1 ↤ 4 Forward →, backward ← on the same wires; red = “nudge this, how much f moves” Ⓑ Common gates have just three personalities: three wiring shapes No formulas: thicker = larger grad, dashed = negative; same skeleton, follow the red + Add gate: distributor a b + upstream 4 4 4 Both same width: each a copy × Mul gate: swapper a b × upstream 1 4 -1 swap Wires cross: take the other's value max gate: router a b max upstream 4 4 ✕ 0 One wire cut: winner takes all Ⓒ The middle gate: source of the whole bill Don't skip: all later talk of activations starts here × backward needs it so it stays Just one gate every layer every step …becomes all of this Every square = the same thing: a saved forward value (layer, step) So activations aren't a casual cache: backward's math demands them Recompute makes sense even on this tiny circuit Drop q; when backward needs it, redo a + b: one add saves one number's storage. That's recompute; no logic changes. Hence adds cheap, matmuls costly: redo an add ≈ free; redo a matmul = all those multiplies again.
The lecture's only example small enough to follow by eye: two numbers per wire, black forward, red backward.
Ⓑ No memorizing gates: add distributes, multiply swaps, max routes; that's enough to hand-derive any circuit.
⛔ Ⓒ Don't skip: the multiply gate needs forward values in backward, the source of the whole activation bill. Backward's math, not the framework, demands them.
Sources and assumptions

📌 The “circuit diagram + three gates (distributor / swapper / router)” framing is from CS231n's backprop notes; figure and numbers are ours.

⭐ Every number is checkable: the build script asserts them (a and b both have gradient 4, c has −1) and fails on mismatch.

1.3b Do wrong early guesses hurt later ones? Not in training

The class got it right: in training, earlier guesses never affect this position.

Each position's input is always the ground truth, not the model's last guess. This is teacher forcing.

⛔ But it isn't “looking things up one token at a time.”

Training computes Q / K / V and attention in one shot, all positions at once.
It's the biggest shape difference from inference, which alone goes token by token, each waiting for the last.

Strictly, “the K and V before me” is “before me and me”: the causal mask shows a position itself, not later ones.

Do wrong guesses hurt later positions? Not in training This is teacher forcing: each input is always ground truth, not the model's guess Ⓐ Train Ⓑ Infer Ⓒ Trade-off Ⓐ Training: inputs all ground truth All positions at once: one matmul, not one by one truth The weather is very nice today guesses weather is cold ✘ wrong nice today . ✘ no such path So errors don't carry The next input comes from the truth row, not the wrong guess. Positions don't contaminate each other — so all 4,095 run at once. Ⓑ Inference: context is its own output It must go one at a time: each token waits for the last The weather is cold and rainy ✘ wrong here all later tokens build on it What each one is fed Train: truth · Infer: own output Train: one pass · Infer: 4,095 serial The biggest shape difference. Ⓒ The cost has a name: exposure bias It never trains on its own errors Its world: context is always right Inference context is all its own. One slip takes it into a distribution unseen in training: no experience there. That's why long generations drift. Not dumber: it's in untrained territory. Why do it anyway? ① Parallel: with self-feeding, position t awaits t−1: 4,095 positions, 4,095 serial steps. ② Early stability: early guesses are junk, self-feeding blows up instantly. A trade: known cost, still worth it. Two takeaways Train on truth, infer on self: no error leaks in training, and 4,095 positions run at once; inference is serial. The cost, exposure bias: never trained on its own errors, one slip goes off-distribution; hence long-text drift.
Training feeds the truth; inference feeds itself: Ⓐ In training, the “prediction → next input” path doesn't exist: wrong guesses can't taint later positions; all compute at once. Ⓑ Inference feeds itself; one mistake and everything after builds on it.
Ⓒ The cost, exposure bias: the model never trains on its mistakes, the root of long-text drift. Why accept it? Parallelism pays too well.
Sources and assumptions

Sources (looked up on 2026-09-22, not from memory): the term teacher forcing was introduced by Williams & Zipser (1989) for recurrent networks; the 1989 original reads “We call this technique teacher forcing” (the version adding “intuitively sensible” is from the pair's 1995 review chapter).

scheduled sampling (mixing in the model's own predictions at random, against exposure bias) is from Bengio et al., arXiv 1506.03099 (2015). ⚠️ Today's large models still use essentially pure teacher forcing: the parallelism is worth the exposure bias.

✋ So what's the cost? Exposure bias.

Training never shows the model its own mistakes; context is always right.
At inference it's all self-generated: one mistake and it's off its training distribution.

Hence long generations drift ever further. The model didn't get dumber; it's in untrained territory.

Why do it? ① Parallelism: self-fed, position t waits for t−1; 4,096 tokens means 4,000+ serial steps, impossibly slow. ② No early divergence: early guesses are awful; feeding them back blows up in one step.
A known cost that pays off.

📌 Sources (looked up): teacher forcing, coined by Williams & Zipser (1989) for RNNs; scheduled sampling (randomly feeding in the model's own predictions, against exposure bias), Bengio et al., arXiv 1506.03099 (2015). ⚠️ Today's large models still use near-pure teacher forcing.

1.4 Why partial derivatives make this easy

Because partials are local.

⛔ ❓ Most people object at once—and rightly.

“A weight reaches the loss through a whole matrix, not a scalar, and its effect depends on its layer's other parameters, which interact. Everything pulls on everything: how can that be computed?”

Both hold, and it's slightly worse:

  • Across a layer, effects add. Weights feeding one output are summed first.
  • Downstream, its effect depends on the other outputs. The next layer mixes them: “everything pulls on everything,” literally.
Ⓐ “Derive” = draw off: dērīvāre = de (from) + rivus (stream) Tangent slope on the left = curve height on the right, same x input the river input → output input drawn off in +1 → out +? slope 1.2 height 1.2 same number slope on the left, height on the right Ⓑ Partial: free one knob, clamp the rest partial = differentiate one variable at a time w1 w2 w3 w4 w5 w6 w7 w8 w9 9 knobs 8 of them clamped 9-D collapses to 1-D—a partial is an ordinary derivative Ⓒ The fan folds into one number—the magic of the partial derivative Left: “the complexity is real.” Right: “it only takes two numbers.” w gradient wanted same-layer neighbors are summed in here layer 2 layer 3 layer 4 3 hops, already 216 paths Real model: 60 layers × 7,168 wide—a 232-digit path count all folded in input x saved in forward × blame δ passed back = grad of w The fan isn't ignored—it's already inside δ Backward runs once per layer, not over every parameter pair The complexity doesn't vanish—it is “paid once” Too many paths to write down, yet its gradient needs just two numbers: the saved input and the returned blame. All mixing and branching was done per layer, in one pass, while computing δ. The price: pairwise terms (2nd derivatives) are skipped on purpose—too many to store. Skipping them forces small steps—hence a learning rate.
Ⓒ is the main course: with more paths to the loss than you can write down, how is a weight's gradient computable?
Left: the complexity is real. Same-layer weights add into one output, which fans out: 216 paths in 3 hops. Right: two cells multiplied. The arrow between is the whole answer: the fan is folded into δ.
Ⓐ: “derivative” is Latin for drawing off a stream; the dashed line verifies: tangent slope (left) = curve height (right). Ⓑ: “partial” is a clamp, not a deviation: free one variable, clamp the rest, N dimensions collapse to one.
Sources and assumptions

📌 Etymology: Etymonline (derive / derivative)—Latin derivare “to lead or draw off (a stream of water) from its source,” from de rivo (de “from” + rivus “stream”); the mathematical sense dates from the 1670s. ⛔ A common mix-up: deviation means a departure (standard deviation), a different word from derivative.

⚠️ Ⓒ's fan (left) is schematic: 3 hops × 6 nodes per layer (216 paths). Only the “232-digit number” (top right) uses the real model: 60 layers, 7,168 wide.

⭐ Side by side, backprop's saving is plain: paths grow exponentially; the backward pass does only 122 matmuls (two per layer), linear in depth.

The magic: the whole tangle is compressed into two numbers.

A weight's gradient needs just two things (matmul ② of 1.3's backward pair):

  • the input it saw in the forward pass;
  • the backward blame at its output.

❓ Where did the cross-coupling go? Into the second number.

Not ignored: computed layer-wide at once, with the blame. The δ passed back already folds in everything downstream, cross-mixing too.

The complexity is real—but paid once per layer, not per parameter pair.

The price leads into 3.3b. Pairwise interactions are second derivatives, one per pair; at 671B parameters, unstorable. So we deliberately skip them and pay with small steps.
Nobody invented the learning rate: giving up second derivatives forced it on us.

An op's backward rule needs only “how I turn input into output,” not its neighbors or the network's shape.

Hence autodiff as a general-purpose library.

One backward for matmul, one for softmax, one for addition; the framework just chains them in reverse.

Nobody hand-derives the whole network.

The alternative is hopeless: a closed form for “loss w.r.t. a first-layer weight” expands through 61 layers into astronomically many terms.

The chain rule: never expand it. Just multiply 61 small local derivatives in order.

The two matmuls of 1.3 and the chain rule's “multiply” are the same thing.

In 1.2d, matmul ③ on the incoming δ is “multiply δ by my own weights,” passed on.

That multiply is a chain-rule link; “my own weights” are the local derivative.

So “61 local derivatives multiplied in order” is literal: matmul ③ done 61 times in backward. Matmul ② (weight gradient) is off the chain: each layer reads its value off it for the optimizer, passing nothing on.

“Gradient” now means two things: the weight gradient from matmul ② (all-reduced, optimizer input, 2 bytes per parameter) and the δ from matmul ③, the activation gradient (used once, on no bill). §1.8's all-reduce carries the first.

❓ And once multiplied together? The famous answer comes straight from the chain rule.

Multiply all the way down—vanishing and exploding gradients The last figure said “multiply 61 small derivatives in order”—this one shows what that product does ×0.8: vanish ×1.0: steady ×1.2: explode Ⓐ One 60-layer net; only the per-layer rate changes—all three curves truly multiplied x: layer, input → output; y: gradient it receives 1 0 layer 1 (nearest input) layer 60 (nearest output) off the chart This one hugs the axis—not missing, just too small to draw Ⓑ Only a log axis fits both—ends 10 orders apart though the two rates differ by just 20% 10^-6 10^-4 10^-2 10^0 10^2 10^4 ×0.8 per layer ≈ 1.9 × 10⁻⁶ ×1.0 per layer = 1 ×1.2 per layer ≈ 47K 10 orders of magnitude What's amplified isn't the gradient but each layer's tiny deviation from 1: a little off, sixty times over, no way back. Ⓒ Where the per-layer rate comes from—one factor is the activation's derivative Both are real plotted functions, not sketches derivative x → 1.0 0.25 ReLU derivative always 1 for x > 0 sigmoid derivative peaks this low Max of sigmoid's derivative? σ(1 − σ) peaks at σ = ½ = 0.25 This alone multiplies each layer by ≤ ¼ The weight multiply comes on top—so not “always 4× decay”; it just pushes down by design That's why deep nets suddenly trained once ReLU arrived: it raised each layer's built-in “push down” factor from ¼ back to 1. This figure only answers “what happens” and “why”—the fixes come later, each in its own place Remember: the chain rule is a product; products are exponential. Watch each layer's rate vs. 1, not the gradient's size—tiny drifts over dozens of layers never recover. Symptoms are opposite: vanishing is silent (loss stalls, early layers never learn, no error); exploding is loud (loss diverges, NaN)—far easier to catch.
The chain rule is a product; products are exponential. One 60-layer net, exchange rate 0.8 vs. 1.2: 20% apart, yet the input-most gradient is 10 orders off.
Ⓐ's red line on the axis isn't missing, just too small to draw. Ⓑ needs a log axis for both ends.
Ⓒ is the one cause you can check on the spot: sigmoid's derivative peaks at 0.25 (closed-form max of σ(1−σ)), a factor ≤ ¼ per layer; ReLU is exactly 1 for positive inputs. That's why deep nets suddenly trained.
Sources and assumptions

⭐ Ⓐ's three curves, Ⓑ's two endpoint values, Ⓒ's 0.25: all script-computed, none copied: 0.8 to the 59th ≈ 1.9e-06, 1.2 to the 59th ≈ 46,956, a gap of 10 orders of magnitude; 0.25 is σ(1−σ)'s closed-form maximum, rechecked at 10,000 sample points.

⚠️ Ⓒ isolates the activation—each layer's true exchange rate also has the weight multiply (and normalization). Picked as the one factor computable here, ⛔ not the only cause.

📌 Source and date (the full story is in the instructor notes): formally identified in 1991, in Hochreiter's master's thesis, “Untersuchungen zu dynamischen neuronalen Netzen” (Technical University of Munich). ⭐ Written in German, it never appeared in an English-language journal. Three years later, Bengio, Simard and Frasconi published “Learning long-term dependencies with gradient descent is difficult” in IEEE Trans. Neural Networks 5(2):157–166—the title is the conclusion.

⚠️ Credit is still disputed; this lecture says only that “the 1991 thesis formally identified it” and stays out of the priority dispute. ⭐ The ending is the fun part: in 2001 Hochreiter, Bengio, Frasconi and Schmidhuber co-wrote a chapter, “Gradient flow in recurrent nets”—ten years on, the two groups who found it independently summed it up together.

⚠️ Also: the problem was first found in recurrent networks, not feed-forward ones like this. ⭐ But it is the same thing—a recurrent network unrolled in time is a very deep network: “depth = sequence length.” That chapter's experiments span 1,000 steps, a thousand layers.

So how do deep networks train? Residuals: one plus sign

How do deep nets train at all? Residuals' whole secret: one plus sign Last figure posed it (products collapse or explode exponentially); this one is the answer plain: product residual: sum Change one symbol in the output—local derivative: “a number” → “1 + a number” Only this point; norm placement and init scaling: 6.5 ❌ No residual y = F(x) per-layer local derivative = a number 0.9 per layer, × 61 layers 0.0016 (vanishes) 1.1 per layer, × 61 layers 335 (explodes) Only 10% off 1, yet total collapse With residual y = x + F(x) per-layer local derivative = 1 + a number all brackets 1, × 61 layers 1 (steady) F is small at init, so it starts at 1 + tiny Gradient always has a non-decaying path to every layer The figure in one sentence Residuals swap “multiply” for “add.” Without them, layer 3 depends on the product of layers 4–61; with them, layers add, so a bad layer drops one term instead of zeroing the whole path. No mystery, then: residuals pin each local derivative near 1—and the 6.0 table traces both failures to local derivatives drifting from 1.
Previous figure: the problem. This one: the answer. No residuals: each local derivative is “a number”; 10% off 1 over 61 layers gives 0.0016 (vanish) or 335 (explode).
Residuals, y = x + F(x): each becomes “1 + a number”; every bracket is 1, and 1 to the 61st is 1: a non-decaying path to any layer.
⭐ Residuals swap “multiply” for “add”: a bad layer adds one term less; it can't zero the path.
Sources and assumptions

📌 Source: He et al., arXiv 1512.03385 (ResNet); the cleaner identity-path derivation is in the same group's arXiv 1603.05027. ⛔ This figure doesn't expand that equation—“1 to the 61st is still 1” is all it says.

⚠️ 0.9 / 1.1 are illustrative magnitudes, not any model's measurements; 61 layers as in DeepSeek-V3. Both products: script-computed, pinned with asserts.

The diagnosis first. A gradient is a product of 61 local derivatives, hypersensitive to each term's average (two left rows above).

Both are just 10% from 1, half the previous ±20%, and collapse just as fully. No middle ground: decay or grow, exponentially. Closeness to 1 makes the point: if 10% per layer can't survive 61 layers, no care is enough. That's the whole reason deep networks wouldn't train before 2015.

A common misreading: “small gradient” ≠ “unimportant layer.” A gradient is effect per step; small means insensitive, so it must move farther for the same effect. It needs to move most. Yet parameter − learning rate × gradient gives it the smallest step. This inversion opens 3.3b and is the whole reason adaptive optimizers exist.

Mathematically, a residual turns y = F(x) into y = x + F(x).

Its local derivative goes from ∂F/∂x to 1 + ∂F/∂x.

Everything else follows.

  • The product gains an identity path. Expand the 61-layer ∏ (1 + ∂Fi/∂x): one term takes 1 from every bracket, and 161 = 1. For any weights, the gradient has a non-decaying path to any layer.
  • It swaps “multiply” for “add.” Without residuals, layer 3's gradient hinges on the product of layers 4–61; with them, contributions add: a bad layer adds one term less; it can't zero the path.
  • At init it is 1 by construction. F starts small (many codebases zero the last projection), so local derivatives start at 1 + tiny: effectively shallow, with F learned layer by layer.

“Residuals let models go deep” is no mystery: “some number” becomes “1 + some number”. 6.0's table blames both on local derivatives drifting from 1; residuals pin them near 1 structurally, and 6.5's tricks (Pre-LN, init scaling) tighten that screw.

📌 Source: He et al., arXiv 1512.03385 (ResNet). The same group's arXiv 1603.05027, “Identity Mappings in Deep Residual Networks,” has the cleaner identity-path derivation: one equation, the gradient reaching shallow layers through no weight matrix. Not expanded: “1 to the 61st is still 1” above is all of it.

A log axis shows ten orders of magnitude and hides their size. Both tracks share one axis and scan line: always the same layer.
Upper, log scale: three lines barely spread, “close enough.”
⭐ Lower, linear scale, the numbers as they are: blue exits the top by layer 8; red vanishes on the floor by layer 18; the fan above is only 30% open.
⛔ Rates 20% apart outgrow one plot within a dozen layers. (14-second silent Manim loop; the script in tools/manim/ computes the out-of-frame and vanishing layers, pinned by assertions.)

1.4b Aside: what a second derivative is (not Adam's “second moment”)

Expand: second derivative vs. Adam's “second moment” Off the main line (a live question). A second derivative is the slope's rate of change (first: which way; second: how long the slope lasts). Optimal step = gradient ÷ second derivative: Newton's method. But it's N² numbers, 10 trillion devices for V3; skipping it isn't laziness. ⛔ Adam's “second moment” is unrelated: geometry vs. statistics; “second” is coincidence.

We just said “we deliberately don't compute second derivatives.” But what is one?

A second derivative is the slope's rate of change.

The first derivative (the gradient): “move one step, the loss moves this much”—the slope.
The second: “move one step, the slope changes this much”—pictured, how sharply it bends here.

First derivative: which way to go; second: how long this slope lasts. Two equally steep slopes, one soon flat, one never, need very different steps; the first derivative can't tell them apart.

What a second derivative really is—and why it isn't Adam's “second moment” First derivative: which way to go; second: how long this slope will last Ⓐ Meaning Ⓑ Use Ⓒ Why not Ⓓ vs. moment Ⓐ Two parameters, identical gradients—yet very different situations The first derivative sees no difference; only the second can tell them apart slope 2 to min 2.00 ① gentle bend 2nd deriv. = 1 slope 2 to min 0.25 ② sharp bend 2nd deriv. = 8 Both points have exactly the same slope, yet one is 2.00 from the bottom and the other 0.25—8× apart. From the gradient alone you can't tell them apart. Ⓑ So its use is concrete: the optimal step can be computed optimal step = gradient ÷ 2nd deriv. On a parabola, one step to the bottom—no trial, no tuning. The two points above: 2 ÷ 1 = 2.00 · 2 ÷ 8 = 0.25 And it fixes the “backwards” problem turn sensitive knobs less, sluggish ones more— divide by it and parameters become comparable. Ⓒ Then why not use it? Its size is the square of the parameter count With many parameters it is a table: one number per parameter pair Can't store it The weights: 1.22 TiB—16 devices of 80 GiB This table—10 trillion devices Not 100 thousand. 10 trillion. (4.5e+23 numbers, in bf16) Can't compute it Solving with it naively costs the cube of the parameter count On an exaFLOP/s machine—9.6 billion years The universe is 13.8 billion years old. That machine is an arbitrary yardstick; magnitude only So “we deliberately skip second derivatives” isn't laziness—it was never possible in the first place. Ⓓ One last thing: Adam's “second moment” is not a “second derivative” They share the word “second” by pure coincidence—the easiest trap in this lecture Second derivative (Hessian) Second moment (Adam denominator) Operation differentiate twice w.r.t. params square the gradient, then average Yields curvature—how sharply it bends how big its gradient usually is Field geometry statistics Count N squared N Likewise, the “first moment” is not a first derivative—it is a moving average of the gradient; “first” means the statistical first moment (the mean). So Adam doesn't “approximate the Hessian”—with 2N numbers it takes another road rather than approximating that N² table. Two sentences to remember 2nd deriv. = how fast the slope changes = how sharply it bends. 1st: which way; 2nd: how long a slope lasts—optimal step = gradient ÷ 2nd deriv. One step to the bottom, but it is N² numbers: impossible to store or compute, so we skip it on purpose. Adam's “second moment” is unrelated: parameters differentiated twice (geometry) vs. squared gradients averaged (statistics). Same word, by coincidence.
“Second derivative” is load-bearing, yet never defined. Ⓐ Two points, same gradient, 2.00 vs. 0.25 from the bottom: gradient alone can't tell them apart. Ⓑ Optimal step = gradient ÷ 2nd derivative: one step down.
Ⓒ But it's N² numbers: weights need 16 devices, this table 10 trillion; solving, over 9 billion years. Not laziness.
⛔ Ⓓ The crux: Adam's “second moment” is unrelated to the “second derivative”: one differentiates parameters twice (geometry), one averages squared gradients (statistics). “Second” is pure coincidence.
Sources and assumptions

All script-computed: Hessian 4.502e+23 entries (bf16 = 9.00e+23 bytes); at 80 GiB/device ⇒ 1.05e+13 devices; the 1.22 TiB of weights needs only 16.

Solving costs N³ = 3.02e+35 ops; a 1e18/s machine ⇒ 9.57e+09 years. ⚠️ That machine is an arbitrary yardstick for magnitude only, not a real one; universe age taken as 13.8 billion years.

Its use: the optimal step can be computed.

On a parabola, optimal step = gradient ÷ second derivative: one step down, no trial, no tuning.
It also cures 3.3b's “backwards” problem: turn sensitive knobs less, sluggish ones more; ÷ second derivative makes them comparable.

Cauchy's 1847 paper ends on this split: first order approaches from afar; second order closes in—this lecture's second half covers what to do when second order can't.

What is “Newton's method,” really? Built for solving equations, not optimization. 3 min; safe to skip

First, its original form: find the x where f equals zero. Guess a point, draw the tangent there, see where it crosses the horizontal axis, and jump there as the next guess. Repeat.

Replace the curve with its tangent. The curve is hard; the tangent is a line whose zero is obvious. Strikingly fast: typically each iteration about doubles the correct digits.

How did it become an optimizer? A minimum is where the derivative is zero.

Run root-finding Newton on the derivative: “subtract f/f′” becomes “subtract the derivative over its derivative.”

That is, gradient ÷ second derivative. So the “optimal step” above is not another formula; it is Newton's method itself.

An equivalent, equally exact, easier picture: Newton's method for optimization fits a parabola at your current point—matching the true curve in height, slope and curvature—and jumps straight to its bottom.
⇒ If the loss is a parabola, one step reaches the bottom—hence “one step to the bottom.”

Why not in deep learning? Two reasons, one computed above:

  • It can't be stored or solved. With many parameters the “second derivative” is N² numbers, and you must also invert it: N³.
  • And the loss surface is not a bowl. Near a saddle the table may be “curving up in some directions, down in others”—then Newton heads for the saddle, or even a peak. It only seeks zero derivative, valley or peak.

A middle road is quasi-Newton (the L-BFGS family): without building the table explicitly, it approximates its effect from recent gradients. Good in classical optimization, it never took off at huge scale plus mini-batch noise.

📌 Source (looked up): Newton had a similar method in the 1600s; today's iteration is Raphson's, 1690 (Analysis aequationum universalis); ⚠️ both then applied it only to polynomials.

⛔ Why not use it? Its size is the square of the parameter count.

With many parameters it is not a number but a table: one per parameter pair, the Hessian.

  • It can't be stored. The weights, 1.22 TiB, fit on 16 devices of 80 GiB; this table needs 10 trillion. Not a hundred thousand: ten trillion.
  • Nor can it be computed. Naive solving costs parameters cubed—at an exaFLOP/s, 9.6 billion years. The universe is 13.8 billion years old.

So “we deliberately don't compute second derivatives” isn't laziness—it was never possible.

⚠️ The “exaFLOP/s” machine is an arbitrary yardstick for magnitude only. Numbers computed in topic04-fig-second.py, pinned with asserts.

✋ Last, the easiest trap in the whole lecture:
Adam's “second moment” is not a “second derivative.”

Second derivative (Hessian)Second moment (Adam's de­nominator)
Of whatdiffer­entiate the parameters twice square the gradient, average
Yieldscurvature: how sharp the bend its gradient's usual size
Fieldgeometrystatistics
CountN squaredN

That both are called “second” is pure coincidence.

This is not pedantry; the warmup of 3.5 hinges on it:

Adam divides out “steepness,” not “curvature.”

The denominator is the gradient's root mean square, so however large this step's gradient, the update is pulled back to η's scale—“how steep” gets normalized away. But it never computes “how sharply it bends”: its N numbers don't hold that.

When you hit “Adam already handles the gradients; why still unstable?”—blame curvature, not steepness, and curvature isn't on Adam's books.

Likewise the “first moment” is not a first derivative but a moving average of the gradient; “first” means the statistical first moment (mean).

So why do people say “Adam's denominator estimates the second derivative”? Because a link can be drawn after the fact. But that's hindsight, not its inventors' thinking—see 3.3b, which starts from the problem each of three generations meant to fix.
⭐ The tell is size: Adam takes another road with 2N numbers, never aiming at that N² table. Hence “adaptive learning rate,” not “approximate Newton.”

1.5 The deepest point: why back to front

Expand: why back to front, not the reverse Off the main line: nobody asks about forward-mode gradients now. Right: one number, so one seeding pass. Left: hundreds of billions, so as many passes: at 1 s per pass, ~20,000 years apart (see 1.1). Expand for why one right-side seed suffices

In math, both directions work, same result. The only difference: how many passes along the chain.

Why back to front—because that end is a single number The key is not how the chain rule multiplies (either end works)—it's how many passes you need forward: input end reverse: output end Ⓐ Forward mode—seed on the left, once per parameter Each seed means walking the whole chain end to end once embed layer 1 layer 2 … last layer loss pass 1 pass 2 pass 3 ⋮ 671B passes Each pass yields the gradient of one parameter—the only one it perturbed So the cost scales with the parameter count. 671B parameters means 671B passes over the whole chain. Seeds drawn at the far left for tidiness—really each goes in its parameter's layer. Still “one pass per parameter.” Ⓑ Reverse mode—seed on the right, where there's one number One seed, and every gradient is collected on the way back embed layer 1 layer 2 … last layer loss seed = 1 just this one ✓ got grad ✓ got grad ✓ got grad ✓ got grad ✓ got grad By the far left, all 671B gradients are in hand—after a single pass. What travels back is always one thing (the seed, rewritten en route), so the huge intermediate Jacobians are never computed. Ⓒ Another view: the backward pass is the same network run in reverse Not “a different algorithm”—same shape, three changes W W Wᵀ Wᵀ forward backward same net three changes this constant is σ′(z) computed in forward ① every edge reversed ② same W, transposed—no new params ③ nodes: “apply nonlinearity” → “× const” The point is ③: that constant is fixed in forward—so forward results must stay around for backward to multiply by. Ⓓ So there is one rule, and it has nothing to do with neural nets Don't read it as “backprop always wins”—change the shape and it flips Start from the narrow end. 671B params → 1 loss many in, few out → back to front (backprop) our case 10 params → 1M-dim output few in, many out → front to back here backprop loses Rule: whenever a problem needs lots of partials, first count inputs vs. outputs It decides not “is it correct”—both directions give identical results—but how many times you walk the chain. And this is the half inference doesn't have: inference is one forward pass; there is no “walking back,” so no “collecting gradients on the way.”
The rows differ only in which end the seed goes in—cost falls from “671B passes” to “one pass.”
Ⓑ's ✓ under each op is key: gradients are collected on the way back, not all at the end.
Ⓒ, another view: the backward pass is the same network reversed—same shape, three changes. The third matters most: each node's nonlinearity becomes a constant multiplier, fixed in the forward pass.
⛔ Ⓓ's counterexample: “backprop is better” is conditional: many inputs, few outputs.
Sources and assumptions

⭐ This figure is about structure, not data: its only quantity is “671B,” the model size, established earlier.

📌 “Forward/reverse mode” are standard autodiff terms; backpropagation is reverse mode applied to neural networks.

Two chains, one start—who finishes first? Top, forward mode: one parameter per pass, lit only at the end; eight parameters, eight passes (eight blocks, right).
Bottom, reverse mode: the ball walks back once from the loss, lighting every parameter en route—one pass, one block at right, always.
⭐ It finishes early and sits idle while the top grinds on—that wait is what a 671-billion-fold cost looks like. (9-second silent Manim loop; script in tools/manim/.)

The whole case for backpropagation, in one line: hundreds of billions of parameters, only one loss.

Many inputs, one output. If the loss were a million-dimensional output, not one number, this would no longer pay off; front to back would win.

So 1.1's answer: those 671B forward passes collapse into one backward pass.

This criterion transfers: whenever a problem needs many partial derivatives, first count inputs vs. outputs; that picks the starting end.

🕰️ Per public sources, it debuted in an unread master's thesis.

In 1970, University of Helsinki student Seppo Linnainmaa proposed reverse-mode automatic differentiation in his master's thesis: exactly this “push back from the output.”

And the thesis never once mentions neural networks. Pure math: efficiently differentiating a composite function.

Neural networks waited sixteen more years for the connection (the 1986 Nature paper).

We've seen this shape: 1.4's “vanishing gradient” also debuted in a master's thesis (1991, Munich, German, never in an English journal).
Twice, a game-changer debuted as an unread master's thesis.

How far each historical claim was checked

For 1970 I checked public biographies and surveys; I did not read the thesis (Finnish, not digitized): solid secondhand, hence “per public sources.”
For 1991 (Hochreiter's German thesis) I also didn't read the original, but the four-author 2001 survey is firsthand, one notch higher.
Cite neither as “I have read it.”

1.6 Closing the loop: backward reads forward values, so activations stay

In 1.2d, step ② (own weight gradient) needs “the input saved in the forward pass”: the activation.

⛔ The root reason activations stay: not bad design, but the backward math itself.

Every forward intermediate stays in memory until backward needs it; at the last layer, all are still there.

Both costs stand: compute 3× (forward 1 + backward 2); memory, every activation, first layer to last. Next: trade compute for memory.

Activations form a mountain: piled up going forward, summit at the loss x = time, not layer · weights & optimizer state omitted (flat; would squash the shape) Forward · pile up Peak Backward · tear down Full recompute Ⓐ Each layer adds a result — none can be freed early Why keep: the weight grad needs the forward input Activations (GiB) 0 25 50 75 100 Forward: layer 1 → 61 Backward: layer 61 → 1 Real V3 layer mix: 3 dense (1.24 GiB each) + 58 MoE (1.73 GiB each) Peak 104.1 GiB forward done, backward not begun 🔍 Orange: same run, full recompute · own y-scale Same sequence & model; nothing kept, recomputed on demand loss Peak 5.07 GiB — mid-backward Each drop: a checkpoint freed; jump: the layer being recomputed ↓ With full recompute: peak 5.07 GiB, 21× less Ⓑ How tall is it? One 4K sequence (batch = 1), V3 scale Same basis as Ⓐ · own per-op estimate, ballpark only No recompute 104.1 GiB everything kept An 80 GiB device can't hold even one Full recompute 5.07 GiB only layer inputs kept Same device fits 15 (activations only) Heights to scale: the sliver is what's left after recompute Times batch, linearly: per-device batch 8 → 833 GiB and 40.6 GiB All at 4K, the pretraining length; long-context extension follows pretraining (before post-training) Two bonus answers Why training costs more: only 3× the compute, but inference has no mountain; it keeps only the KV cache. The gap is memory, not compute. When it peaks: at the summit, between forward and backward. Section 5 adds weights & optimizer state as flat bands: same shape, raised.
The x-axis is time, not layer: by layer, you couldn't ask “when is memory tightest?”
The summit answers Section 5's “when is the peak?”: forward done, backward not begun.
Sources and assumptions

The real model: 61 layers, each with several intermediates; real slopes are far finer than drawn

⛔ The straight sides simplify: the real curve bumps at MoE dispatch and attention's big intermediates, but the summit stays at the end of forward

⚠️ Both 104.1 GiB and 5.07 GiB are our own per-op derivation (inputs: V3's config + the official reference MLA forward), not third-party confirmed. Sequence length 4,096, batch = 1, same basis as mountain Ⓐ.

📌 Ⓑ's two cards are one sequence, two configurations, not two models.

Both numbers are our own per-op derivation (V3 config + official reference MLA forward), no third-party check. Even if off by 2×, it still doesn't fit at this scale.

📌 How big is that “pile of intermediates”?

V3's real config, one 4K sequence: about 1.73 GiB per layer, 104.11 GiB over 61 layers.

V3's bf16 weights are 1,250 GiB; one sequence's activations, only about a twelfth.

But weights are fixed, while each extra sequence adds a full set of activations. At about 12 sequences activations match the weights; training batches always run well past a dozen. Hence the trade ahead.

❓ What's largest in that 1.73 GiB? It matters: Section 2 chooses what to drop.

Not attention. It's MoE's “nine copies.”

  • Expert outputs 28% + SwiGLU products 8%: one cause: each token goes to 9 experts (8 routed + 1 shared), each storing its own intermediates: nine copies. Together 37%, a layer's largest block.
  • ⚠️ Inputs don't count: the nine experts share one buffer (the upstream norm's output). Many implementations store a permuted copy for the grouped matmul (nine copies, 504 MiB at 4K); others redo the permutation in backward. An implementation choice, not a math necessity, so excluded.
  • Attention output: only 7%, a notch below most guesses.

This sets Section 2's list: save on the copies first, not attention. It also explains a paradox: MoE is cheap in parameters (each token touches few weights) but pays a nine-copy tax in activations.

Per-item counts and 128K figures: block below. “Does it fit?” territory, not required.

The activation bill, itemized: what a layer holds Safe to skip: for doing the math yourself (“does it fit?”)

1.7 The raw bill, itemized: what a layer actually holds

Optional. This is where the two numbers above come from, for readers who want to check the math.

First, the base unit: one hidden-width tensor = 4,096 × 7,168 × 2 B = 56 MiB. Every number below is a multiple of it. (Sequence 4,096, batch 1, bf16.)

Why 4,096 and not 128K: V3 was pretrained at 4K (all 14.8T tokens); 128K came after pretraining, via two 1,000-step extension stages. Every calculation here uses 4K; 128K appears only for comparison, labeled as context extension.

To convert: activations track only total tokens (see 5.1), so at 128K every number below is ×32 (131,072 ÷ 4,096). The one exception, the attention score matrix, scales with S²: ×1,024 (caveat 2 below).

What the MLA sublayer keeps

Tensor keptWidthSize
Norm input (= residual input; recomputation keeps only this)7,16856 MiB
Norm output7,16856 MiB
Q down-projection (pre/post layernorm)2 × 1,53624 MiB
KV down-projection (incl. 64 RoPE dims)576 + 5128.5 MiB
Q after up-projection128 heads × 192 = 24,576192 MiB
K, V after up-projection128 heads × 256 = 32,768256 MiB
Attention output128 heads × 128 = 16,384128 MiB
logsumexp (fp32)1282 MiB
Subtotal≈ 722 MiB

What the MoE sublayer keeps

Tensor keptCopies × widthSize
Norm input / output2 × 7,168112 MiB
Router scores2562 MiB
Copy sent to experts= row above; all 9 share it0 MiB
gate / up / SwiGLU product3 × 9 copies × 2,048432 MiB
Expert outputs (pre-combine)9 copies × 7,168504 MiB
Subtotal≈ 1.03 GiB

❓ Why these tensors? One criterion: does the backward formula use it?

Frameworks store only what the backward pass will read. By op, the pattern is clean:

OpBackward needsStore?
Add y=a+b Derivative ≡ 1, needs nothingZero bytes
Element-wise multiply y=a×b a's gradient needs b; b's, aBoth
Matmul y=x·W W's gradient needs x (1.3, ②)Store x
Nonlinear SiLU / GELU Derivative = f(input)Store input
softmaxDerivative = f(output)Store output

⭐ Additions are free; multiplies and nonlinearities aren't. Nets are nearly all the latter, hence dozens of copies per layer.

⚠️ A common misconception: “Adding a bias creates another activation.” The new buffer briefly appears and is freed after use; it never enters the “keep until backward” ledger, all the table records.

⭐ The table's biggest driver: not every copy is as wide as the input. Up-projected Q is 24,576 wide (3.4 copies); K/V, 32,768 wide (4.6 copies). So half of “dozens of copies” is width, not just count.

The “9 copies” deserve a pause, especially where they appear and where they don't.

9 = 8 activated routed experts + 1 shared expert. The nine copies are on the output side: each expert produces its own intermediates and output. The “expert outputs” item alone is 504 MiB.

⛔ The input side has no nine copies. All nine read the same memory: the “norm output” row above, already counted.
⚠️ In practice, tokens are often first permuted into contiguous per-expert blocks for grouped matmul. That buffer is a real copy: top-9 means each token appears nine times, 504 MiB at 4K. But some redo the permutation in backward instead of storing it: an implementation choice, so not in this required-tensor table. (Fixed live 2026-09-22: once counted as required, overstating by 448 MiB.)

On Topic 1's parameter-count table it is invisible. MoE is a bargain on parameters (each token selects only a small fraction of the weights), but pays a nine-copy tax on activations.

Altogether: one MoE block is about 1.73 GiB, one dense block about 1.24 GiB; 58 MoE + 3 dense ≈ 104.11 GiB.

📐 At the extension length (128K): ×32 ≈ 55.39 GiB per MoE block, ≈ 3331.39 GiB for the whole model.

Many remember this number, but it is the context-extension bill. For the 14.8T-token pretraining: 104.11 GiB.

⛔ Three caveats; state them whenever you present this table.

  • Derived op by op, not from a report. Frameworks vary widely in fusion, in quietly recomputing cheap ops, and in whether MoE dispatch truly makes 9 copies. Magnitudes only.
  • The attention score matrix is not included. It scales with S², the only item here not tracking the total token count. At 4K: 4.00 GiB per layer (128 heads × 4,096² × 2 B); at 128K: 4 TiB per layer (×1,024, not ×32). FlashAttention never writes it to memory, hence its absence. A naive non-flash implementation voids the whole table.

    Don't dismiss it as “only 4 GiB”: an entire MoE block's activations total just 1.73 GiB. This item alone outweighs it. At 128K it dominates; at 4K it is still the single largest tensor. The argument survives the new baseline: from “alarming” to “still number one.”

  • The 5.07 GiB peak is the bill “after the trade has already been made once.” It assumes each layer keeps only its input, plus the full activations of the layer being recomputed (3.34 GiB + 1.73 GiB), which is exactly recomputation. The second term is easy to miss: omit it and the peak is 34% low, memory ≈ a third short. It also assumes checkpoints sit exactly on layer boundaries and only one layer is recomputed at a time. Pipeline bubbles may hold more; the peak then scales with that count. The number holds only if both do. So raw bill first, Section 2 second: order matters.

One more item isn't in either table, because it belongs to no layer.

The final projection turns hidden states into scores over the vocabulary (logits), of shape [seq_len, vocab]. V3's vocabulary is 129,280:

4,096 × 129,280 × 2 B = 1,010 MiB (one 4K sequence)

Next to the two tables:

  • Largest in the tables: “K/V after up-projection,” 256 MiB.
  • Logits are 1,010 MiB, almost four times that.
  • A whole MLA sublayer totals just 722.5 MiB. Logits alone are bigger.

It is the model's largest single tensor, and softmax usually runs in fp32, doubling it again.

So real code almost never materializes it whole: they chunk the sequence: one chunk's loss, its backward, then the next. It is Section 2's recomputation idea, along the vocabulary dimension.

Mainstream implementations chunk it, so it is left out of the total above, deliberately. But write your own loop computing the full logits in one go, and this block suddenly appears in your memory: 31.6 GiB at the 128K context-extension length, and without chunking it won't run at all.

1.8 Gradient in hand, not done

1.3 stopped at the computed gradient. Several stages remain before it's used, none of them in inference.

Life of a gradient: five stages from computed to discarded Three lanes: what it does, can it hide?, in inference? Can hide Can't hide Not in inference Ⓐ Five stages in sequence; just one overlaps compute and only one stalls every device ① Computed Gradients appear layer by layer in backward → — Arrive in turn, last layer first Inference: none ② All-reduce Each device sees different data; average them → Can hide Later layers' grads ship while earlier ones compute Inference: none ③ Clip By global norm; rescale if over → Can't hide All layers/devices: forced sync Inference: none ④ Accumulate Sum several micro-batches (optional) → — Changes the timeline, not totals Inference: none ⑤ Use & discard Handed to optimizer, then done — Discarded right away, so it can be stored in bf16 Inference: none Lane 3 is the point: inference has none of the five; they're training-only. Training's premium isn't “one more pass” but a whole pipeline. Only ③ stalls all devices, for a single number Global norm = one scalar over all params: sum squares, reduce, sqrt, then decide the scale. Tiny traffic; the cost is the sync. So global-norm clipping is inherently once per step. Per-layer clipping exists but bounds each layer's norm, not the whole step's size.
Read all three lanes: “what happens,” “can it overlap compute,” and the point.
The third is all purple: the class line made visible (what inference lacks).
⛔ Of five stages, only clipping stalls all devices, for one number.
Sources and assumptions

⚠️ The figure has no quantities: traffic volume and how much hides are in the batch figure and Topic 5.

⭐ Stage ⑤'s “use & discard” sets up a later point: used once, the gradient can be bf16; accumulated, it's a running sum and must return to fp32.

Traffic size, and why batch doesn't change it Communication is parallelism: Topic 5

Stage ②'s traffic has a counterintuitive property

Pure data parallelism averages the full gradient across all devices every step. Its size: params × bytes per param.

The key: this traffic doesn't depend on batch at all. Feed one sequence or a thousand, and you send the same number of bytes.

Compute, meanwhile, grows linearly with batch. So:

the bigger the batch, the thinner this traffic is spread.

That explains two seemingly unrelated things:

  • Why large models keep raising batch size. GPT-3 ramped from 32K tokens to 3.2M, PaLM from 1M to 4M, V3 from 3,072 sequences to 15,360 (4K context, i.e. 12.6M → 62.9M tokens). All of them grow the batch as training goes on.
  • Why gradient accumulation saves more than memory. Updating once every K micro-batches divides all-reduces by K.

✋ Careful: a bigger batch does not buy throughput.

“Bigger batch is faster” has a precondition that fails at large scale: once parallelism saturates, per-step cost rises exactly as step count falls.

At this scale a bigger batch buys amortized communication and steadier gradients, not speed.

Curve and full trade-off: 3.5b, with two other folk sayings (“bigger is more stable,” “too big stops working”); the three go together.

Why the “can hide” lane matters

Overlapping compute and communication is a key training optimization. Inference has no counterpart: no backward pass, no long tail to hide transfers in.

Why clipping can't hide, and how per-layer clipping differs: see the figure's takeaway. Key point: the whole step's size is what links to 3.3.

Aside on ④: what gradient accumulation saves

⭐⭐⭐ Gradient accumulation changes the timeline, not the total.

Scope: this stage exists, its size, whether it hides. Its form under other shardings (all-reduce or not) and volume: Topic 5.

“Parameters × bytes per parameter” is an order-of-magnitude convention: ring all-reduce goes out and back, about doubling traffic; MoE experts differ again.

Section 2

The cost: what can't be dropped, and the first real trade-off

The course's first real trade-off: not “is there a better way?” but “only one of two; which?”

In the core track because it follows from 1.6 (backward needs intermediates, so they can't be discarded). “Drop and recompute” is the next link, and its decision rule is valuable. How many GiB it saves and which level pays is a “does it fit?” question for Topic 5; details are collapsed.

2.1 Recomputation: store save points, not the whole run

Forward keeps every intermediate because backward needs it (1.6). Recomputation: don't.

When backward needs it, rerun the forward pass from this layer's input. Like a game save: keep a save point, not the whole playthrough, and replay from there.

2.2 Full recomputation: is the trade worth it?

The orders of magnitude to remember:

So lopsided it hardly counts as a trade-off.

Large-model training turns it on by default. The question is how far, never whether (Ⓑ's three levels).

Full recompute: 1/3 more compute, 19/20 less memory Both normalized to original = 100% · otherwise TiB vs TFLOPs: no common scale Cost: compute Saved: memory Ⓐ The two sides: too lopsided to call a trade-off Bar = % of original (remaining / grown) Cost: compute Before: 3× (forward 1 + backward 2) Now: 4× (one extra forward) +33% The extra sliver is “one more forward pass” Saved: memory Before: 104.1 GiB activations Now: 5.07 GiB — only 4.9% left This bar is almost invisible: that is the whole point Ⓑ The question isn't “on or off” but which level Off Keep all Only sensible with memory to spare Selective Drop by cost/benefit 128K: ~1.9% compute saves ~71% (4K: 9.3%); best ratio Full Keep only layer inputs 33% compute saves 95%; when memory is truly short With memory to spare (small model, short sequence), recomputation is pure loss Ⓒ Another axis: a checkpoint every k layers Note: not the same as Ⓑ (Ⓑ: what a layer keeps; here: how often) Store all keep every layer 💾 💾 💾 💾 💾 💾 💾 💾 💾 💾 💾 💾 💾 💾 💾 💾 16 stored + 1 in segment = 17 Every 4 layers L = 16, √16 = 4 💾 💾 💾 💾 4 stored + 4 in segment = 8 First one only all 16 live during recompute 💾 1 stored + 16 in segment = 17 Live at once One square = one layer. 💾 = a kept checkpoint Both ends equal: storing less doesn't save more. Drop a checkpoint and you carry a longer segment during recompute.
Note the asymmetry: cost grows a sliver; savings shrink almost out of sight.
⚠️ Both bars are normalized: raw quantities in different units, side by side, invite a meaningless length comparison.
Ⓒ switches axis: Ⓑ asks “what to keep within a layer,” Ⓒ “how many layers between checkpoints.” Its three “live at once” values: 17 / 8 / 17, equal at both ends.
⛔ The optimum is in the middle, at √L. From Chen et al. 2016 (arXiv 1604.06174; the title is the conclusion: Training Deep Nets with Sublinear Memory Cost): “an algorithm that costs O(sqrt(n)) memory to train a n layer network, with only the computational cost of an extra forward pass”. ⚠️ ZeRO (1910.02054 §3.2) also cites roughly square-root memory at 33% recomputation, but restates this result; it isn't the source.
Sources and assumptions

⚠️ 104.1 GiB / 5.07 GiB are our per-op estimates (V3 config + official reference MLA forward); read them as magnitudes. Even 2× off, the asymmetry stands

⭐ “3× → 4×” is the matmul convention: recomputation reruns the forward pass, so the extra is exactly 1/3. The ZeRO paper gives the same (33% re-computation overhead)

⭐ Ⓒ's three “live at once” values come from L/k + k; the script asserts optimum at √L, both ends equal. That equality is the panel's whole point

⛔ Both percentages in Ⓑ are per step (forward + backward = 3 passes). The “selective” level once said 6%, per forward pass (old convention). Incomparable side by side; now unified.

📌 Ⓑ's “selective” 1.9% / 71%, derived and ranked: next figure and this section's text. DeepSeek-V3 (arXiv 2412.19437) says they recompute all RMSNorm ops and MLA up-projections: exactly that level

2.3 Selective recomputation: the rule is one number

Ask every intermediate one question: drop and recompute it in backward: how many FLOPs per byte saved?

Smaller ratio, better candidate; that alone yields the ranking.

For linear layers it has an elegant closed form:

A [S,k] × [k,n] matmul costs 2·S·k·n FLOPs to recompute and yields S·n·2 bytes (bf16). Divide: cost per byte = k.

The closed form assumes recomputation costs only the tensor's own op, true while its input is still there. Drop the upstream tensor too, and it must be recomputed first: cost accumulates along the chain, no longer one k. Our selective list keeps inputs present; watch this in your own.

Per byte, recomputing a linear layer's output costs exactly its input width in FLOPs. S and n cancel: independent of sequence length and output width.

So the best linear layer to recompute is the narrowest-input one; check the config.

Compare with 1.6: “biggest” and “most worth dropping” are two different rankings.

1.6 ranks by absolute size; this section by cost per byte saved, a ratio. No reason for them to agree.

Here they happen to agree: 1.6's two biggest items are both in the cheapest group.

  • SwiGLU products (8%): elementwise, cost per byte ≈ 0
  • Expert outputs (28%): input width 2,048, 2.20 TFLOP per GiB, under the “take all below 3” line

Luck, not a law. Counterexample, same table: the priciest item (o_proj output, 17.59 per GiB) is only 2.5% of a layer, little saved at high cost. On another model the rankings may conflict. Then choose by ratio, not size (cost per byte saved).

2.4 Attention breaks the pattern, and it must cross the linear layers

Attention isn't linear: cost grows with S², output only with S, so cost per byte grows linearly in S.

Why one rule gives opposite answers: different slopes must cross y-axis: cost per byte to drop & recompute · log-log: constant = flat, ∝ S = straight Cheap → drop Costly → keep attention Ⓐ x-axis: sequence length — the only variable that moves Linear-layer lines are flat: their cost is independent of length 1K 4K 16K 64K 256K Sequence length (tokens) 1K 10K 100K FLOPs/byte K/V up-proj (input 512) Q up-proj (input 1,536) Expert output (input 2,048) gate / up / router (input 7,168) o_proj (input 16,384): widest V3 attention (purple) = 1.25 × seq length GPT-3 attention = 1.00 × seq length GPT-3's widest linear (49,152 = 4h) GPT-3's own crossover 49,152 Crossover S ≈ 13,107 Right of it: attention costliest GPT-3 2,048 → 2,048 FLOPs/byte V3 pretrain 4,096 → 5,120 FLOPs/byte V3 extension 131,072 → 163,840 FLOPs/byte Long-context extension: same model, 32× to the right Same rule, verdict flips; each model has its own lines The 2022 paper (arXiv 2205.05198) drops what's big yet cheap; it chose attention. In orange, GPT-3 at 2,048 costs only 2,048/B, with its own crossover at 49,152. 2,048 is far to its left: attention really is cheapest. The paper was right. What matters is the red group's own path. V3 pretrains at 4,096, at 5,120/B — below gate/up; an ordinary candidate. Extension goes to 131,072; the same attention costs 163,840, past 13,107 by far: now the one to keep. Same model, same rule, one knob moved: the verdict flipped. Not a word changed. And note: the two models' slopes and reference widths both differ; the flip comes from “different slopes must cross,” not from any specific numbers.
This topic's most valuable figure: “different slopes must cross” takes a paragraph in words, a second in a picture.
Log-log isn't for looks: only here is a constant flat and “proportional to S” straight.
⛔ The two vertical lines are the whole flip: same rule, only S changed.
Sources and assumptions

⭐ The closed form for linear layers: a [S,k]×[k,n] matmul costs k per byte (recompute 2·S·k·n FLOPs ÷ output S·n·2 bytes; S and n both cancel), so on the plot it must be a horizontal line

⚠️ The two slopes are 1.25 for V3 (qk 192 / v 128) and 1.00 for GPT-3 (qk = v = 128); both use the causal-halved convention. ⛔ For another model, redraw its own two lines and recheck the causal-halving convention; the slope changes from model to model, but “it is sloped” does not.

⛔ The 13,107 crossover isn't an engineering threshold; it also depends on the reference layer (four lines, four crossovers)

📌 The 2022 paper's gains (GPT-3 saves 70% for 2.7%) use Megatron's un-halved convention; in this figure's causal one, GPT-3's attention share is 1.37%

The figure's two lines, different slopes, are the whole section: one flat (linear layers, cost = input width), one climbing (attention). Different slopes must cross.

A constant and a climbing line must cross.

Compare against the widest linear layer. For V3: o_proj (projects the 128 concatenated heads into the residual stream), input width 128 × 128 = 16,384, not d_model's 7,168.

⛔ Why stress “not d_model”? I made that mistake first.

The crossover once read 5,734, using d_model (7,168) as “the widest linear layer.” Standard MHA has “heads × head dim = hidden,” so that works; MLA removes the constraint: V3 is 128 heads × 128 = 16,384, twice d_model.

The counterexample sat in the 1.7 table all along (“attention output 128 heads × 128 = 16,384”), half a section away, never cross-checked.

Rule: when the architecture changes, recheck any common-sense “A = B”. It raises no error; it skews every number downstream.

Hence: 1.25·S = 16,384 → S ≈ 13,107.

  • Below ~13K: attention isn't yet the priciest; dropping it isn't the layer's worst deal.
  • Above ~13K: past the widest linear layer, it stays priciest and grows linearly pricier.

13,107 is a rough crossover, not an engineering threshold: it hinges on causal halving, V3's head dims and the reference layer. Remember “different slopes must cross,” not the number.

Every tensor in a layer, ranked: the 9-row table, where to cut, save vs. pay Rule is above; this is the worked accounting (“does it fit?”)

Rank V3 at the 4K baseline: one MoE block, sorted ascending by TFLOP paid per GiB saved:

TensorSavedRecomputePaid/GiBVerdict
logsumexp (attention by-product, fp32)0.002 GiB~00Free
RMSNorm outputs (3, elementwise)0.121 GiB~0~0Free
SwiGLU products × 9 (elementwise)0.141 GiB~0~0Free
K/V up-proj (input width 512)0.250 GiB0.14 TFLOP0.55Worth it
Q up-proj (input width 1,536)0.188 GiB0.31 TFLOP1.65Worth it
Expert outputs × 9 (input width 2,048)0.492 GiB1.08 TFLOP2.20Worth it
Attention output0.125 GiB0.69 TFLOP5.50Marginal
gate/up/router/down-proj (input width 7,168)0.303 GiB2.33 TFLOP7.70Marginal
Post-MLA residual (= o_proj out, input width 16,384)0.055 GiB0.96 TFLOP17.59Marginal

Don't memorize this table; it comes straight from the closed form above. Except the elementwise rows (cost ≈ 0) and attention (moves with S), each row's “paid/GiB” is just its input width in different units, so you can rank it from the config. A confession: one row was once hand-filled: “9.14 GiB saved × 7.70 per GiB” is 70 TFLOP, yet the column said 39.08. Yet the table's selling point is “you can rank it yourself”; anyone who did would have hit it. Now it's all generated; no hand edits.

⛔ “Is there anything that takes little memory but lots of compute?” Yes, attention. But how extreme it is depends entirely on the sequence length.

One criterion, a ratio: keep high, drop low. Here only the attention row moves with sequence length; the rest are input widths, independent of S.

Attention, paid/GiB4K (our baseline) 128K (extension)
Absolute5.50175.92
vs. widest linear (o_proj, 17.59) only 31% of it10× pricier
Rank in table7th of 9 (o_proj, gate/up pricier) 9th of 9 (priciest)
VerdictMarginalNever

Same model, same rule: go from 4K to 128K and attention goes from “still behind two linear layers” to “10× more expensive than anything.”

Don't overstate it: at 4K attention's ratio is 5.50, above the “take everything below 3” line discussed below, so by our own cut it isn't in the “take” group. What changes is whether it's the extreme outlier, not whether to recompute it: the first is a fact; the second depends on where you cut.

❓ Where does “below 3” come from? I chose it; it isn't derived.

In the table, the expert-output row is 2.20 and the next row, attention output, is 5.50; 3 just sits in the gap. Put it at 2.5 or 5 and the result is identical.

What actually sets the line isn't in the table: your current exchange rate, what a GiB of memory is worth to you in TFLOP.
Memory nearly full: move it right (drop more, pay more compute); compute-bound: move it left.

So don't memorize 3 as a threshold. The table is sorted by ratio; you only decide where to cut. The ranking is objective; the cut is your setup's call.

Cutting at 3 (taking every ratio below it): per layer you save 1.19 GiB and pay 1.53 TFLOP. Per step (one layer, forward + backward = 3 passes = 16.4 TFLOP): 9.3% more compute for a recomputable-pool saving of 71%.

Full recomputation, on the same per-step basis, is “pay 33% more, remove 95%.” Selective is still 2.7× more cost-effective, hence worth its own setting.

At 128K these numbers look very different, and much better.

Same cut: “only 1.9% more for 71%”; vs. full recomputation, 13.1× the value: truly “an order of magnitude.”

Why: the numerator (recompute cost) ∝ S, the denominator (one step) at 128K is inflated by S² attention. Longer S, cheaper-looking recomputation. “1.9%” is a flattering long-context number; don't cite it at 4K.

The direction holds, though: selective clearly pays at both baselines. And full recomputation's “pay 33% for 95%” is scale-free: it depends only on running forward twice, not on S.

Tighter: with FlashAttention, attention's backward is 2.5×, not 2× More precise; no conclusion flips

⛔ One convention to tighten: attention's backward is 2.5×, not 2×.

1.3's “backward ≈ 2× forward” is exact for matmuls with weights. But this lecture assumes FlashAttention throughout (1.7, caveat ②), and to save memory it recomputes the attention scores in backward.

From the FlashAttention-2 paper: “To get the FLOPs of the backward pass, we multiply the forward pass FLOPs by 2.5 (since there are 2 matmuls in the forward pass and 5 matmuls in the backward pass, due to recomputation).” (arXiv 2307.08691 §4.1)

Another instance of this section's theme: the memory FlashAttention saves is paid for by recomputation inside FlashAttention itself. “No free lunch,” with a concrete price.

With this lecture's shares: at 128K attention is 82.1% of a layer's forward, so a step is 3.41×, not 3× (3× undercounts by 12%); at 4K it is only 12.5%, giving 3.06×: no real difference. The rest of this lecture keeps 3×, a deliberate simplification.

“Selective +9.3%” and “full +one-third” once had different denominators (a forward pass vs. a whole step), so weren't comparable. As 2.5 says, “before comparing such ratios across papers, check whether they halved for causal.” Both are now per step.

First-party evidence the rule (FLOPs per byte saved = the producing op's input width) isn't just theory.

DeepSeek-V3, plainly: they “recompute all RMSNorm operations and MLA up-projections during back-propagation,” because “with a minor overhead, this strategy significantly reduces memory requirements for storing activations.”

RMSNorm outputs and MLA up-projections are exactly this rule's two cheapest categories: elementwise ≈ 0; up-projection input widths only 512 / 1,536. Our own rule, their actual config: stronger evidence than more arithmetic.

📌 Source: arXiv 2412.19437 §3.2.3.

2.5 The same rule gave the opposite answer in 2022

The idea of “selective recomputation” comes from Korthikanti et al., Reducing Activation Recomputation in Large Transformer Models, arXiv 2205.05198 (2022-05).

Same criterion as 2.3: drop parts that “take up a considerable amount of memory but are not computationally expensive to recompute.”

But it chose attention: QKᵀ, softmax, softmax dropout, and attention over V. Payoff: GPT-3 saves 70% of activation memory for only 2.7% extra compute; MT-NLG saves 65% for 1.6%.

Why reversed? Its sequence length is 2,048.

But use GPT-3's own slope: standard MHA, qk = v = 128, slope 1.00 (not V3's 1.25): 1.00 × 2048 = 2,048. GPT-3's widest linear layer is 49,152 (= 4h, the MLP down-projection's input width; V3: 16,384, o_proj's input width).

2,048 ≪ 49,152: far left of its own crossover. At 2K, attention really is the best to recompute. The paper was right.

Same rule, flipped conclusion. To be precise:

Not just sequence length. Slopes differ (1.00 vs. 1.25), as do reference layers (49,152 vs. 16,384): redraw both lines.

I got 49,152 wrong too: I first wrote 12,288, GPT-3's d_model. The rule needs the producing matmul's input width; the MLP down-projection's input is 4h. Same mistake on V3: I took d_model 7,168 as widest, missing o_proj's 16,384 input (128 heads × 128); the crossover was briefly wrong (fixed in 2.4). This panel's own point, and I stepped in it twice.

⭐ “Grabbing a relation that sounds like common sense” is the hardest error to catch.

No error, no blow-up; every downstream number just drifts. And since it sounds right, you've no reason to doubt it.

The same mistake, twice in this lecture: d_model as the widest linear layer, on V3 (missing o_proj's 16,384) and on GPT-3 (missing 4h's 49,152). One “common sense” rule, true on neither model.

What transfers is the shape, not the numbers. New model: redraw two lines, don't memorize numbers.

Another confession: this figure once applied V3's 1.25 to GPT-3 (2,560), in a lecture about “don't apply A's constants to B.” Rule: when two models share a figure, each line uses its own model's constants.

Mind the convention, or numbers won't match: the paper's 2.7% uses Megatron's classic formula, not halved for causal masking. In our causal convention, GPT-3's attention share is 1.37%. Both are right; before comparing such ratios across papers, check for causal halving.

Today this is handled for you. FlashAttention never stores the score matrix and recomputes it in backward: the paper's advice, built in.

In Megatron's source the default survives, with a maintainers' warning Fun, but a framework detail

⛔ I first wrote “so attention isn't even on modern frameworks' candidate lists.” Then I read the source. Exactly backwards.

Megatron-LM's selective-recomputation switch --recompute-modules: its first candidate, core_attn, is the default (recompute_modules = ["core_attn"]).

There are three layers here; I first covered only one.

  • ① “Default” doesn't mean “on by default.” recompute_granularity defaults to None: nothing is recomputed. core_attn is the default module once you enable selective.
  • ② With FlashAttention / fused attention, recomputing it is nearly pointless. The scores are never stored; what never materialized can't be “dropped and recomputed”.
  • ③ Most interesting: Megatron knows this, and wrote the warning into its source.

“If you are using transformer_engine as the transformer implementation, the core_attn is from transformer_engine and may be the fused version. For fused attention, you have no need to set 'core_attn' to recompute. Please check that the core_attn recompute is really needed.”

📌 transformer_config.py has if "core_attn" in self.recompute_modules: warnings.warn(...) (main, 2026-09).

So this isn't “a stale default nobody looks after”: someone noticed, left the default alone, and pinned a warning beside it.

Why not just change it? That's the interesting part.

Changing a default has a cost: results others got with the old one become irreproducible. Frameworks prefer a suboptimal default to voiding past results.

So the default becomes a fossil: it records the 2,048-era optimum. At that length it was right; at long context, by our table, it's the last to recompute; and under today's fused attention it has essentially nothing to do.

The lesson isn't whether to turn it on, but: for any default, ask what year it was picked and if its premises still hold. Me included: I first wrote “it's still the default,” collapsing three layers into one; readers would think it recomputes attention from step one.

The rest of the list matters more: layernorm, moe_act, mla_up_proj, i.e. RMSNorm outputs, SwiGLU products, K/V and Q up-projections, all cheap by 2.3's rule. All in the table's cheapest group. But not “exactly these, no others”: the list also has mlp / moe / shared_experts, and mlp is gate/up, the table's second-priciest row, far above the cut (the priciest is the o_proj output row). Our closed-form list matches the framework's options.

📌 megatron/core/transformer/transformer_config.py, field recompute_modules (main, 2026-09). Rule: before saying “modern frameworks don't do that anymore,” grep the framework. Such claims sound like common sense and go stale fastest.

Takeaway: backward needs intermediates, so they can't be dropped; but “drop and recompute” is a negotiable trade, settled by a rule you can derive, not by memorizing others' verdicts.

Beyond the three, folded below: recomputing can change results (random numbers, side effects, bitwise), and gains don't copy across scales (MFU / HFU).

Two off-the-books costs: results may differ; gains don't transfer Matters in practice, not for the principles of backprop, gradient descent or the loss

2.5b A cost off the books: recomputing may give other results

So far, two measurable costs. Recomputation has a third, in neither account, raising no error.

Recomputation assumes a forward rerun gives exactly the same result. Three things can break that.

① Random numbers: handled for you.

Forward has dropout. If recomputation draws a different mask, backward uses a different network. The gradient is wrong, with no error.

PyTorch is blunt: save the forward-time RNG state, restore it to recompute. torch.utils.checkpoint's preserve_rng_state defaults to True.

This also explains the third item on the checklist in 3.7: besides weights and optimizer state, a checkpoint must store the RNG state, for the same reason: random numbers are part of the training result, not transient runtime state.

② Side effects: nobody handles them.

Suppose the segment also modifies something on the side: updates a running statistic, increments a counter, writes a log line. Recomputation modifies it again.

The framework can't roll these back; it doesn't know what you changed. The rule: the recomputed segment should ideally be a pure function.

③ Bitwise mismatch: usually harmless, but it has one specific victim.

Run the same computation twice and the last bits may differ: reduction order or kernel choice is enough.

For training itself, differences this small barely matter. But they hinder bitwise reproducibility, exactly what PaLM achieved in 6.6 and what the “roll back + skip data” remedy in 6.2 presumes.

In short: recomputation closes the compute ↔ memory deal, but quietly demands reproducible results. The first two accounts say whether it pays; this one only shows when something breaks.

2.6 Takeaway: you can't copy others' gains

⛔ Same model, same switch, different scale: the gain can flip sign.

Why: recomputation shifts the compute-to-memory-traffic ratio, which varies anyway with parallelism and chip count.

A general rule (not only for remat): any parameter that changes shard shapes can't be copied across scales. Revalidate at the target scale.

2.5 shows the rule is harsher than it sounds: sequence length is on that list too, and it flips the gain's sign, not its size: the item most worth recomputing at 2K becomes the last to touch at 128K.

📌 Another number worth knowing, which you can compute:

Useful work is 3 units (forward 1 + backward 2); full recomputation adds 1, so 4 are executed. Recomputation is one-quarter of executed compute.

I originally wrote “one-third” here: that confused “one-third more than before” (1 ÷ 3) with “share of total executed” (1 ÷ 4). Same 1 unit, different denominator. 📌 The “one-third more” isn't our own derivation: the ZeRO paper says “33% re-computation overhead” verbatim (arXiv 1910.02054 §3.2).

That's why MFU looks so low: recomputation does real work, but by definition isn't in the numerator: not useful compute.

There are two metrics, differing by exactly the recomputation.

  • MFU (Model FLOPs Utilization): numerator: recomputation-free model FLOPs (the 6ND family).
  • HFU (Hardware FLOPs Utilization): numerator: FLOPs actually executed, recomputation included.

With full recomputation, HFU / MFU = 4 / 3: the “1 of every 4 units” above.

So for any MFU number, first ask which numerator it uses. Two papers' MFUs can differ by 30% just because only one used recomputation.

⛔ At long context, “low MFU” is not mainly due to recomputation.

Recomputation makes MFU at most 3/4 of HFU (a factor of 1.33). But 5.2 shows that at 128K, 6ND counts only 17.9% of the real compute; that convention gap alone is 5.6×.

⭐⭐⭐ The two factors swap places with sequence length: at 4K, 6ND undercounts by only 1.14×, so recomputation's 1.33× dominates; at 128K, the reverse.

In practice: at long context, turning off recomputation because “MFU is only 20%” recovers at most a quarter. First align the FLOP convention.

Section 3

With gradients, how big a step? Gradient descent and optimizers

Two sections fought activations. The biggest item on the bill has stayed silent.

3.1 How many bytes per param?

❓ One premise first; the bill below depends on it.

In training, each parameter sits in HBM twice:
① a compact copy (bf16, 2 bytes) for forward and backward;
② a precise copy (fp32, 4 bytes) for bookkeeping. Updates land here.

This is mixed precision. 3.2 explains why both are needed.

In mixed precision, each parameter carries five things; only the first is the model itself.

Optimizer lineage: how many state copies each generation adds per parameter Ordered by the memory bill, not by the math · x-axis is order only, not a linear time scale 0 copies 1 copy 2 copies a step back Ⓐ Sixty years of adding the same thing, until 2024 took one away A “copy” = one extra tensor the same size as the parameters 0 copies 1 copy 2 copies SGD step along the gradient + Momentum 1964 remembers last direction AdaGrad 2011 a step size per param RMSProp 2012 sum → moving average Adam 2014 momentum + adaptive: both AdamW 2017 fixes weight-decay bug Muon 2024 drop v, orthogonalize x-axis shows order only, not a linear time scale Muon is the step back 60 years of adding; in 2024 the second moment was dropped AdamW is not a new algorithm: it fixes a bug The abstract opens: L2 and weight decay are equivalent for SGD, but not for adaptive methods like Adam. L2 rides in the gradient and gets divided by v: params with large gradients get their weight decay diluted. AdamW's fix: decouple weight decay from the gradient and subtract it separately in the update. That is what the “W” stands for. Ⓑ The five items in “16 bytes per parameter”: 12 B are the three fp32 copies standard mixed-precision setup bf16 weights 2 B used fwd & bwd bf16 grads 2 B from backward fp32 master weights 4 B the real copy; accumulates fp32 momentum m 4 B Adam state #1 fp32 2nd moment v 4 B what Muon drops = 16 B 2 B: inference too the weights you download 14 B: training only this lecture is all about this span Here is where “optimizer state is the biggest item” comes from, in numbers: weights take 2 B; the optimizer side (master + m + v) takes 12 B: six times as much. Switch to Muon and the second moment is gone: 16 B becomes 12 B, a quarter less. But it only handles 2D parameters; embedding and output head stay on AdamW, so the whole model saves less than a quarter.
Ⓐ's shape is the whole figure: sixty years of adding state, then a step back in 2024.
Ⓑ makes “optimizer state is biggest” a ruler: weights 2 B, optimizer 12 B.
Sources and assumptions

Adam: Kingma & Ba, arXiv 1412.6980 (ICLR 2015). AdamW: Loshchilov & Hutter, arXiv 1711.05101 (ICLR 2019); the sentence in the “AdamW is not a new algorithm” band paraphrases its abstract's first sentence (the original qualifies the SGD half: equivalent only when rescaled by the learning rate).

Muon: Keller Jordan et al., 2024; writeup: kellerjordan.github.io/posts/muon. Muon “only applies to 2D parameters”; “scalar and vector parameters of the network, as well as the input and output layers, should be optimized by a standard method such as AdamW”; it “has a slower per-step wallclock time than AdamW”: all three verbatim.

Momentum: Polyak 1964 (heavy ball) / Nesterov 1983. AdaGrad: Duchi, Hazan, Singer, JMLR 2011 (no arXiv version). RMSProp: Hinton, 2012 Coursera course, Lecture 6. It was never formally published.

Three state-savers not drawn: Adafactor (arXiv 1804.04235, factors v as row + column), Lion (arXiv 2302.06675, gradient sign only, one momentum), 8-bit Adam (quantized state, same algorithm).

⛔ No convergence-speed comparison: we have no head-to-head data; a curve would be invented. Only structure and each paper's claims.

Remember one line: weights 2 bytes, optimizer 12.

“Optimizer state dominates HBM” isn't rhetoric. It's 2 : 12.

All 12 bytes are fp32; the next subsection explains why.

The 16 bytes: does mixed precision save resident memory? “Not one byte”: counterintuitive, pure accounting. Remember: Adam keeps two moments

⛔ A stubborn misconception: “Mixed precision halves memory, right?”

For resident memory, it saves not one byte. Add up both sides and you get 16 either way.

WeightsGradsMomentum m2nd moment vMaster weightsTotal
Pure fp324444—16 B
Mixed precision2244416 B

Look at that column: weights and gradients save 2 bytes each, but the master copy costs 4; the 4 eats 2 + 2.

So what does it buy? Two things, neither resident memory.

  • Speed (the main point). Matmuls run in bf16 on the chip's dedicated matrix units.
  • Activations. Stored in bf16: here you really save half. Per 5.4's toy example, often above resident state.

Rule of thumb: before saying “it saves memory,” ask which memory. Resident state, activations: two accounts. “Saves half” merges them and fails when it matters (e.g., will it fit one device?).

3.2 What needs high precision? One test: does it accumulate?

Why three fp32 boxes and two bf16? Don't memorize. One picture explains it.

Precision, in one picture: big numbers swallow small ones Every number here is computed by a script; run it yourself bf16: wide but coarse fp16: fine but narrow fp32: both, 2× the bytes Ⓐ A float has two fields: exponent = how big, mantissa = how fine Note: bf16 exponent = fp32 exponent fp32 32-bit 8-bit exp 23-bit mantissa bf16 16-bit 8-bit exp 7-bit mant. same exp as fp32 → same range, coarser fp16 16-bit 5-bit exp 10-bit mant. exp 3 bits short → small grads flush to 0 bf16, fp16: same 16 bits, split differently. fp16 needs loss scaling not for lack of bits, but because it gave them to the mantissa. Ⓑ So the addition literally does nothing: the weight never moves weight 1.0, +0.0003/step (typical late in training) 1.0000000 1.0078125 1.0156250 1.0234375 nothing in between 1 gap = 0.0078125 update 0.0003 only 3.8% of a gap: rounds back to start it takes 26 adds to cross one gap but never does 1,000 adds in bf16 1.0 → 1.0 never moved: rounded back every time same 1,000 adds in fp32 1.0 → 1.3 every add landed So the fp32 master copy is not just insurance: without it, late-stage training really does stall. Ⓒ Who needs fp32? One test: do old contributions stay forever? Not “does it accumulate?”: too coarse; a DeepSeek-V3 result refutes it 10⁰ 10¹ 10² 10³ 10⁴ 10⁵ steps share of one step's contribution still left half master must be fp32 v (β₂ = 0.999, Adam default): half-life 693 steps m (β₁ = 0.9): 6.6 steps grads/acts: gone next step Only the flat line needs fp32: it still holds a tiny increment from 100,000 steps ago. The other three forget, and what fades can be stored coarsely. 12 fp32 bytes hold what stays; 2 bf16 bytes, what fades. Not a guess: this curve draws the line. Accumulated grads persist a while, so back to fp32. The rule can be broken: remove the one-sided rounding The killer isn't the small step but always rounding the same way. Stochastic rounding (round up with prob. = remainder) makes the expected step exact: what's lost comes back over many steps. (The script checks this identity.) Not a recommendation: it needs HW/framework support, and fp32 masters are cheap and simple. Why show it? Knowing why a rule holds tells you when it can fail.
Ⓑ is the whole subsection: weight 1.0, plus 0.0003 per step, 1,000 times in bf16: still 1.0. Not slow: it never moved. In fp32: 1.0 → 1.3.
The numbers are script-computed, not quoted: a real bf16 rounding function, three asserts. Run it.
Ⓐ fixes a misconception: fp16 needs loss scaling, but not because “16 bits isn't enough.” Same 16 bits as bf16, with 3 moved from exponent to mantissa, so the range shrinks and small gradients flush to 0.
Sources and assumptions

📌 Every number in Ⓑ is computed by a script using a real bf16 rounding function (top 16 bits of a float32, round-to-nearest-even, stdlib struct only). Three asserts guard it: less than half a gap must round back; a full gap must land; 1,000 adds must still equal 1.0. ⭐ Rule of thumb: if a figure asks you to believe an arithmetic result, have a script do it.

📌 Ⓒ's test (not “accumulates?” but “do old contributions stay forever?”) was honed by a first-party counterexample: DeepSeek-V3 (arXiv 2412.19437 §3.3.3) keeps AdamW's moments in bf16, no degradation seen; master weights and accumulation gradients stay fp32. ⛔ Meta-rule: a rule a counterexample hits usually has coarser wording than mechanism; keep it.

⭐ Ⓐ also settles 6.6's item: fp16 needs loss scaling, not because “16 bits isn't enough.” It moves 3 of bf16's 16 bits from exponent to mantissa. With five exponent bits, not eight, the smallest representable positive number rises by over thirty orders of magnitude; small gradients underflow to zero.

The 12 fp32 bytes are exactly what keeps old contributions; the two bf16 items are used once and dropped. Not a coincidence: one test draws the line.

❓ Can gradients be fp32? Yes, in some implementations.

Usually for gradient accumulation over micro-batches. In bf16, big values swallow small ones and precision drains within a few rounds, so many implementations keep an fp32 accumulator.

That column then costs 4 bytes: 16 becomes 18.

📌 Our 16 is the ZeRO paper's classic accounting: 2 bytes weights + 2 bytes gradients + 12 bytes for three fp32 copies, written 2Ψ + 2Ψ + KΨ = 16Ψ, K = 12. arXiv 1910.02054. Sourced, but a configuration, not a law; quote its assumptions.

A small correction: the paper's two 2-byte items are fp16 (2019's norm), not bf16. bf16 is standard now. Same bytes, same bill, but different formats (see “Does bf16 need loss scaling?” in 6.6).

⛔ A first-party counterexample sharpens the test.

DeepSeek-V3 keeps AdamW's two moments in bf16, “without incurring observable performance degradation,” but master weights and gradients in fp32, gradients precisely for accumulation.

Right idea, too coarse: not accumulation, but unbounded accumulation:

  • Master weights: tiny increments over 100,000 steps; old contributions stay forever. Truly unbounded: fp32.
  • Momentum m / second moment v: moving averages. Each step scales by a factor below 1, then mixes in the new value; old contributions decay. Not an accumulator, so bf16 copes. (If β₂ isn't too close to 1: the closer, the smaller each step's change, until bf16 rounds it away. V3 uses β₂ = 0.95; see PaLM's step-dependent β₂ in §3.6. Corrected 2026-09-25: caveat added.)
  • Gradients: normally single-use. Accumulated, they become an accumulator and go back to fp32.

Tightened test: do old contributions stay forever? Yes: fp32. Decaying or single-use: low precision suffices.

“Must be fp32” isn't absolute: some have dropped it.

Master weights need fp32 because round-to-nearest erases small updates. Stochastic rounding (round up with probability set by the remainder) recovers it in expectation over many steps.

Finer still: the killer is not accumulation but accumulation + rounding the same way every time. Remove that bias and bf16 master weights are viable.

Not recommended: it needs extra support; fp32 masters are cheap and easy. Shown because knowing why a rule holds tells you when it breaks.

📌 Source: DeepSeek-V3, arXiv 2412.19437 §3.3.3 (“Low-Precision Storage and Communication,” not recomputation §3.2.3). It confirms the list nearly item by item: “We adopt the BF16 data format instead of FP32 to track the first and second moments in the AdamW … optimizer, without incurring observable performance degradation. However, the master weights … and gradients (used for batch size accumulation) are still retained in FP32 …” Accumulation reason included. The 16-byte table is no law: V3's per-param bill differs.

The test outlives this lecture. Why fp8 training is hard is the same question: as precision drops, what can go lower, what must stay? Still accumulation. Another lecture.

3.3 Why following the gradient lowers the loss

First, whose idea was this? Cauchy, 1847

⭐ Gradient descent predates neural nets by over a century. Augustin-Louis Cauchy, 1847, French Academy: “General method for solving simultaneous equations”.

He wasn't optimizing; he was solving equations.

The standard method, elimination, had two flaws, Cauchy notes: it often fails outright; when it works, the last equation is hopelessly complex, even from simple ones.

⭐ So he swapped the problem: minimize a function instead.

The function must be non-negative. Merging equations takes one line: u² + v² + w² + … = 0. Squares can't be negative, so the sum is zero exactly when each is. “Solve” becomes “drive the sum to zero.”

⭐ The invention isn't calculus (Newton, Leibniz) but the reframing: “solve the equations” → “walk downhill.” Every model we train runs on it.

⭐ The paper ends: once close, switch to the linear (Newton's) method to finish fast. He knew each order's place: first order from afar, second to close. 3.3b's “why everyone must estimate the second derivative” is this split at hundreds of billions of parameters.

📌 Source: A.-L. Cauchy, Méthode générale pour la résolution des systèmes d’équations simultanées, Comptes Rendus Hebd. Séances Acad. Sci. 25, 536–538 (1847). The two flaws, the sum of squares and the Newton line paraphrase the paper (English translation).

671B partial derivatives in hand. Which way to move?

Nudge every parameter: loss changes by Σ nudge × partial derivative. For small enough steps this is derived, not a rule of thumb.

This section's one crack: “multiply and sum” isn't the partial-derivative definition (a single-variable limit) but a first-order approximation (total differential). It needs small steps: the problem for the rest of this subsection and all of 3.3b.

For loss to drop, the sum must be negative. Move each parameter against its partial derivative: every term, “negative × positive,” is negative.

“Follow the negative gradient and loss falls” is math, not folklore. Among equal-length steps, fastest: the flip side of “gradient = steepest ascent.”

⛔ How small is “small enough”? No answer.

A partial derivative is the slope where you stand; it promises nothing a meter away. Overstep and you land higher than you started.

Direction: settled. Step size: no answer. It occupies the rest of this subsection and 3.3b entirely.

Why gradient descent works: everything else rests on it Earlier: how the gradient is computed. Here: why following it makes things better 1D: a line 2D: a landscape 671B dims: can't draw Ⓐ Start in 1D—picture a ball rolling downhill One rule: slope positive, move left; negative, move right a parameter → loss start A start B lands here lands here each step shorter nobody slows it: the slope flattens A lands in the shallower valley only because it started left, not on merit Auto-brake step ∝ slope nearer the floor, flatter slope flatter slope, smaller step so it won't overshoot the valley floor comes free if steps are small enough Ⓑ Two parameters: one slope number isn't enough One number can't give a direction; you need a vector 1D: “slope + or −?” one number suffices 2D: “which way drops fastest?” one number can't say: it's a direction → That direction is the gradient It points steepest uphill so downhill takes its negative and its length tells you how steep Note for later: “fastest” depends on how you measure step size; change the ruler and the fastest direction changes (Muon figure Ⓒ). Ⓒ 671B parameters: forget the hill, read it differently Can't draw a 671B-dim hill, but the column of numbers is readable all params in a column 0.31 −1.24 0.07 ⋮ 671B numbers + −gradient same column +0.002 −0.910 +0.004 ⋮ one-to-one → each entry says two things ① sign: push this param up or down ② relative size: which change matters more the second is the point: not just where to go, but who moves first So why does it work? Part is guaranteed, part isn't Guaranteed: every step follows the locally steepest descent, with step ∝ slope; unless steps are absurdly big, loss keeps falling. Not guaranteed: which valley you land in depends on where you start, maybe not the lowest. Gradient descent never promised optimal, only that every step gets better.
Everything grows from this figure. Before: how the gradient is computed. Now: why following it helps.
Ⓐ Both trails are a real run. Steps shrink: nobody slows the ball; the slope flattens.
⛔ Trail A lands in the shallower valley—only because it started left. Compare landings, then starts.
Sources and assumptions

📌 Ⓑ's footnote is from Su Jianlin's “Why do we prefer isotropy? An understanding via steepest descent” (in Chinese; our translation): “The negative gradient descends fastest only given a premise, chiefly a Euclidean norm; change the norm and the steepest direction changes.” ⭐ Almost all tutorials skip it; without it Muon is just “yet another optimizer.”

📌 Four framings—“a ball rolling downhill,” “step ∝ slope, so no overshoot,” “which valley depends on the start,” and “in high dimensions, read the signs and relative sizes in the column”—come from 3Blue1Brown's official lesson notes for “Gradient descent, how neural networks learn.” All checked against the original; the figure is our redraw.

⭐ The Ⓒ column of negative gradients is exactly what one backward pass produces: where Sections 1 and 3 connect.

1D → 2D → stop drawing space. Act 1: same curve, same rule, two balls, two valleys.
Act 2: 2D contours from above; shape still visible.
⭐ Act 3, the key: 671B dimensions can't be pictured, so drop space. Parameters become bars (color = sign, length = magnitude); shrinking bars, falling loss.
(15 s silent loop, Manim. Structure from 3Blue1Brown's neural-network series (source, CC BY-NC-SA 4.0); idea only; visuals and data ours. Script: tools/manim/.)

That figure left an ugly line: your start picks your valley. Alarming in 1D. The next figure dismantles it.

Zero slope ≠ stuck: getting trapped in a pit is a 1D illusion This figure answers: why more parameters make descent work better True min: stuck Saddle: go on Measured: no true min Ⓐ Zero-slope points come in several kinds—check two directions to tell them apart Each panel: one point, sliced along two directions True min (local min) dir 1 up ↑ dir 2 up ↑ both up every way is worse: stuck Peak (local max) dir 1 down ↓ dir 2 down ↓ both down rolls off; rare in training Saddle (pass) dir 1 up ↑ dir 2 down ↓ one up, one down follow the down one A mountain pass is a saddle point: along the road over the mountain it is the highest point; along the ridge it is the lowest. Geographers call this landform a saddle. Ⓑ A true min needs every direction up—but if just one points down, you can go on 1D: only two directions 1D plot 2 dirs (2 drawn) both up is easy, so pits everywhere small model ~10⁴ dirs (40 drawn) all 40 up is already strict, let alone ~10⁴ our scale ~10¹¹ dirs … the row ends here—the real count is a billion times more just this one miss one and it's a pass: go on This is a definition, not a probability argument: curvature signs aren't independent, so we put no number on “how rare.” Ⓒ Not just reasoning—it was measured Train to zero gradient, then count how many directions curve up 0 1 x-axis: share of up directions (minimum ratio) where trained points actually land = 1: all directions up = a true local minimum measured: never reached In other words: where training stops, some directions still go down it stopped not for lack of a path, but because it's flat Schematic axis, not measured data; it shows one fact, “every observation stops short of 1.” Original plot: Hung-yi Lee's slides. The deeper half of why it works: the more parameters, the harder it is to truly get stuck Good news: the “trapped in a shallow pit” picture is a 1D artifact. In high dimensions, nearly every zero-slope point is a pass; some direction still goes down. Bad news: not stuck doesn't mean fast. Near a pass the slope is tiny, so plain descent (step ∝ slope) crawls there. Most of the optimizers ahead exist to fix this.
This lifts the last figure's shadow—“trapped in a shallow pit” exists only in 1D.
Ⓐ One word: pass, highest on the road over, lowest along the ridge. Zero slope, yet a way through.
Ⓑ A true floor needs every direction curving up; among hundreds of billions, miss one and it's a pass. Ⓒ Measured: no stopping point was a true floor.
⛔ Not “training is easy.” Not stuck isn't fast: near a pass the slope is too flat; most optimizers ahead fix that.
Sources and assumptions

📌 Four points from Hung-yi Lee's 2021 slides “When the network won't train (1): local minima and saddle points” (in Chinese): “critical points: local minima or saddle points,” “test: all Hessian eigenvalues positive?” “negative-eigenvalue directions still descend, rarely done in practice,” “measured: never reach a real local minima.” All checked at source; the figure is ours.

⭐ He also borrows Liu Cixin's “Death's End” (Three-Body III): a magician who pulls things from sealed spaces via a higher dimension. Slide: “From 3 dimensional space, it is sealed. It is not in higher dimensions.” ⛔ Novel unchecked; we relay only that line. The main metaphor (mountain pass) is ours.

⚠️ Ⓑ We deliberately give no probability: “each direction curves up or down with equal odds, so all d curve up with probability 2⁻ᵈ” sounds right, but eigenvalue signs aren't independent, so the number is made up. We keep only what holds by definition.

⭐ The slides' closing line recurs here: smaller batches and momentum both help escape critical points. Noise and inertia: what the batch figure and the two Adam figures cover.

Two cross-sections, but the shape is 3D. Let's rotate it.

The same saddle, rotating. The real shape behind the cross-sections. Red curves up one way (loss rises), green curves down the other—crossing in the middle.
⭐ Flat underfoot at the black dot, but the green path is always there. (8 s silent loop; Manim, community edition of 3Blue1Brown's tool. Script: tools/manim/.)

❓ “Enough dimensions means no pits” is wrong by one word.

Four dimensions or a thousand: either can hold a deep, sealed pit. Nothing forbids it.

⭐ The truth: a pit both high and sealed is extremely rare in high dimensions. Sealed pits exist; sealed implies low.

Why does height go with sealedness? Sealing needs every direction curving up. Up high there are more ways down, much height left to lose. Near the bottom fewer remain, until none: only then a floor.

⭐⭐ So in high dimensions nearly every stall is at a pass. Floors aren't forbidden; they're all very low, and you're not there yet.

Not just reasoning: it was measured.

Train to zero gradient, then count: how many directions curve up?
The fraction never reached 1.

Wherever training stops, some directions still go down. Paths exist; the slope was too flat.

✋ So: “Don't more parameters make training harder?”

Memory and compute: yes. Getting stuck: the opposite. Two questions; don't merge into “big models are harder to train.”

⛔ But not stuck isn't fast. Near a pass the slope is flat, the gradient tiny; with steps “proportional to the slope,” it crawls for ages. Most optimizers ahead fix this.

📌 Dauphin et al., Identifying and attacking the saddle point problem in high-dimensional non-convex optimization (arXiv 1406.2572, NeurIPS 2014). Abstract: the usual view blames local minima far worse than the global one; they argue “a deeper and more profound difficulty originates from the proliferation of saddle points, not local minima, especially in high dimensional problems …”.

⛔ A plausible fabrication: “each direction curves up or down at even odds, so all d curve up with probability 2^(−d).” Curvature signs aren't independent: no basis. We stick to what holds by definition.

⭐ The second half is key: saddles sit in “high error plateaus” that “can dramatically slow down learning, and give the illusory impression of the existence of a local minimum”. The enemy: crawling, not sticking.

Left over: no sticking, but flat passes mean crawling. The next figure is a first answer.

Momentum: same start, plus a little inertia Last figure: “not stuck, but slow.” This is the first answer to it No momentum: stuck Momentum: escapes β: memory span Ⓐ Same loss, same start: both trajectories actually simulated Only difference: the right one remembers its last move shallow pit deep valley stops here rolls here same start Gray: no momentum rolls into the pit and truly stops: gradient there is already zero Green: with momentum carries last step's velocity, gets past overshoots, then swings back To be precise: it gets past because the pit is shallow. Momentum makes small pits less likely to trap you; it never promised the lowest valley. Ⓑ How it gets past: just past the pit floor, two forces fight Arrow length and direction to scale, from the green step crossing the pit from this point ← toward deep valley back → Gradient term − lr × gradient 0.092 it pulls back Inertia term β × last move 1.304 it keeps going Net move sum of both 1.213 inertia wins 14× Look closely: the gradient is against moving on; just past the floor, the slope pulls it back. Inertia carries it out. Ⓒ What β is: “last move” is really all past moves, older ones lighter Each step back multiplies by β: a row of bars decaying by β now 1 back 9 back 17 back effective window ≈ 10 steps window = 1 ÷ (1 − β); β = 0.9 gives ~10 steps. β isn't magic: it's a “how long to remember” knob. Ⓓ Last, and easy to miss: lowering the learning rate quietly raises friction bar length = friction λ, the real physical quantity; β just mixes it with step size baseline η = 0.9 β = 0.85 λ = 0.158 memory 7 steps only η cut to 1/10 η = 0.09 β = 0.85 (same) λ = 0.500 friction up 3.16× memory 7 steps also β → 0.953 β ← 1−(1−β)√r λ = 0.158 friction unchanged memory 21 steps Friction should be a constant; move η without β and it changes. After the fix, memory goes from 7 to 21 steps (exactly 1/√r×). Momentum answers that second half, but it fixes slowness, not optimality Last figure: “not stuck, but slow”; momentum is the first fix for slow. Where the slope is flat, the gradient term nearly vanishes, but the last move remains, so it keeps sliding. But don't sell it as “finds the global optimum”. It escapes here because the pit is shallow; a deeper pit would trap it all the same. It weakens “the start decides everything”; it doesn't remove it.
Same loss, same start, plus a little inertia. Gray stops in the shallow pit halfway (gradient truly zero); green reaches the deeper valley. Both simulated.
Ⓑ, the trick, is starker than expected: just past the pit floor the gradient pulls back—inertia, over ten times stronger, carries it out. Arrows to scale.
⛔ Not “finds the global optimum”: it escapes because the pit is shallow. “The start decides everything” is weakened, not removed.
Sources and assumptions

⭐ Ⓐ's two trajectories, Ⓑ's three arrow lengths, Ⓒ's bars: all script-computed. Our made-up 1D loss L(x) = 0.015x² + 0.45cos(1.1x) is built for “one shallow pit, then one deep valley”; the script found the pit and its depth, nothing hand-marked. Four asserts: no-momentum run stops in the pit, gradient zero; momentum run lands in the valley; landings far apart.

⭐ Ⓑ deliberately picks the green trajectory's step right through the shallow pit, and asserts “gradient here is nearly 0 while the move is still large.” Without that, the figure says nothing.

📌 How old is this idea? 1964. Polyak, “Some methods of speeding up the convergence of iteration methods”: 22 years before the backpropagation paper in Nature. ⭐ He called it “the method of a small heavy sphere”; our ball is named after it.

⭐⭐⭐ The Ⓑ picture, “the gradient pulls back, inertia carries it away,” is in the paper verbatim: “The motion proceeds not in the direction of the force (i.e. antigradient) because of the presence of inertia”: inertia keeps it from following the negative gradient. He also says the inertia term leads to motion “along the bottom of the trough.”

⭐⭐ Best of all: in 1964 Polyak suggested ρ = 0.8–0.99; Adam's default β₁ = 0.9 is still in it, sixty years on. (Script-asserted for the figure's two βs.) ⭐ His tuning order: tune the learning rate with ρ = 0 first; add momentum once convergence slows. He reports it mostly beat the gradient method, by up to tenfold (paraphrase; the original says “up to tenfold”).

⛔ One claim left out as unverified: that the 1986 Nature backprop paper already had a momentum term. I couldn't get the original (the link was a 404), so we won't claim it until I read it.

📌 Framing from Hung-yi Lee's “When the network won't train (1)” (in Chinese): he writes momentum as “Movement: movement of last step minus gradient at present” (the last step's move, not just the gradient) and closes with “Smaller batch size and momentum help escape critical points.” ⛔ Our redraw; curves and numbers from our runs.

Why momentum works: three mechanisms; the famous one matters least

⭐ The figure above, escaping a shallow pit, is real but the least used of the three—sealed pits are rare in high dimensions.

① Coasting across flats. Near zero gradient, inertia remains. It treats crawling, not sticking: the high error plateaus around saddles.

② Stop bouncing off valley walls. Its main job in real training, the only one at work daily: same signs add, opposite signs cancel.

Recall 3.3b's valley, steep one way, flat the other: zigzag across, inch along.

  • Across the steep direction the gradient flips sign each step—summed, signs cancel, the swing shrinks.
  • Along the flat one it keeps its sign—the sum rolls ever faster.

⭐ Momentum is a filter: it amplifies the direction you keep and damps the one you swing in. Deep-net loss surfaces are usually narrow valleys; it nearly always helps.

⭐ Polyak's 1964 paper: inertia leads to motion “along the bottom of the trough.” The heading above, written in 1964.

③ Smoothing noise. Small-batch gradients are noisy; a moving average smooths jitter for free.

❓ When doesn't it help? There's a computable ceiling.

Classical momentum (Polyak's; PyTorch's SGD momentum): velocity ← β × velocity + gradient. A constant gradient grows velocity to 1 ÷ (1 − β) times it.

βSteady-state gainMemory span (steps)
0.805×~5
0.90 (most common)10×~10
0.99100×~100

⭐ That is its whole budget. At β = 0.9 the step grows at most tenfold. Taller pit walls trap it.

So “sometimes it fails” isn't mystery or mistuning: finite kinetic energy, computable ceiling. It escapes above because that pit fits within tenfold.

⛔ Flip side: overshoot. Near the floor, inertia carries it past; it swings a few times, then settles. Not free: end wobbles buy no stops en route.

⚠️ Both are “momentum,” but Adam's doesn't amplify. Easy to confuse.

  • Classical momentum: v ← β×v + gradient; a constant gradient grows v to 1/(1−β)×. It buys speed.
  • Adam's first moment: m ← β×m + (1−β)×gradient: a normalized moving average; at constant gradient, m → the gradient: no amplification. It buys a stable direction, not speed.

⭐ Same name, two things: one stores kinetic energy, one removes jitter. Adam's step comes from 3.3b's 0.2, not β₁'s speed.

⭐ So “escapes local minima” is its least-used side. Its daily work: coasting across plateaus and not bouncing between valley walls.

Same curve, same start; only difference: keep the velocity or not. Gray (no momentum) stops in the first shallow pit; blue goes straight past to the deep valley.
⭐ It overshoots and swings back: not a bug, a real side effect, fixed next.
⛔ Same code; only β changes, 0 → 0.9. (24.6 s, silent loop, Manim. Script-computed terrain and paths; asserts pin “first pit shallower,” “gray stops there,” “blue reaches the deep valley,” “peak velocity nearly 3× higher.” Script: tools/manim/.)

These three figures settle the direction: negative gradient lowers loss; high dimensions rarely trap; inertia crosses flats. Left: how big a step?

3.3b How big a step? Start with a units problem

Direction is settled. Now: how far to step.

A line nearly everyone has written: param ← param − lr × grad

Before units: two knobs show this line is backwards

📐 First, back to 3.3's Cauchy, 1847.

His paper closes: once close, switch to the linear, or Newtonian, method to finish fast.

That splits the job: first order from afar, second order to close.

This section: when second order can't close it.

The units argument is rigorous but slow to see. Two knobs show the result first, in grade-school arithmetic.

Two knobs, A and B; current error 1,000.

  • A is sensitive: a notch moves the error 100.
  • B is sluggish: a notch moves it just 1.

❓ How many notches each to cancel 1,000?

A: 1,000 ÷ 100 = 10 notches. B: 1,000 ÷ 1 = 1,000 notches.

Turns scale inversely with sensitivity: few for sensitive, many for sluggish.

What does param ← param − lr × grad give? A moves η × 100, B moves η × 1—100 : 1.

⛔ You need 1 : 100; it gives 100 : 1. Not slightly off: exactly backwards.

The sensitive knob overshoots, oscillating. The sluggish one barely moves, short after thousands of steps. The slowest knob sets the pace.

The inverse rule assumes independent, locally linear directions; real valleys curve, couple. Intuition, not formula. The exact-step figure is rigorous.

This is 1.4's vanishing gradient, restated.

There, chained products shrank the first layers' gradient to 0.0016. Often misread: “early layers don't matter.” Backwards.

Small gradient = sluggish knob = farther to go. It needs to move most; SGD gives it the smallest steps.

Be precise, or the views clash. A middle layer scaling the signal by 0.5× halves the upstream gradient; what doubles is the distance needed for the same effect. Reciprocals: one fact, two faces.
Mixed, they contradict; our convention: gradient = effect per notch, not distance to move.

The same point in units: why there must be a conversion factor

The units don't match.

The gradient is loss change per parameter change: unit loss ÷ param (per 1.0, loss is in nats/token, so nats/token ÷ param). “Param” isn't physical; weights are dimensionless. The parameter's own scale is the unit; it's conserved, so the derivation holds. This is SGD; Adam's second-moment division puts updates on parameter scale, so its learning rate has other units (exact-step figure). You want a distance: in parameter units.

So something must convert: the learning rate. Its unit: param² ÷ loss, i.e. param² · token / nat.

Read the unit aloud: the learning rate is not a pure number. It bakes in parameter and loss scales; loss scale depends on vocabulary (1.0's scale tops out at ln V). New model, vocabulary or batch: retune.

Units exposed, SGD's root flaw shows

One learning rate for the whole network, yet gradient scales differ by orders of magnitude. Below: same valley and rule, different rates.

One LR can't serve all params; no middle value works Same valley & rule, LR varies — both runs real +5%: diverges −5%: slow Middle: fails Ⓐ A valley steep one way, flat the other: ratio 25× Vertical: steep; horizontal: flat Min Off the chart Start Red: LR +5% Bounces keep growing ↑ up: steep param → across: flat param Blue: LR −5% Stable, but see the zigzag: Each step mostly bounces Only a sliver heads to the min LRs 10% apart: one explodes, one crawls. The window is that narrow. Ⓑ Split the difference? No value in between works Computed, not estimated LR +5% Steep axis diverges Hard cap: 2 ÷ steepness, no matter what Max safe LR Flat axis: halfway Takes 0.37 × ratio steps; here (25): 9 steps Real model Scales span decades Ratio 1000 → 364 steps, and that's the best case Not bad tuning: the valley shape decides, and the model, not you, sets it. Only way out: stop seeking a “best global LR”; none exists One LR per parameter. Which value? Next: the exact answer (f′ ÷ f″), and why we can only estimate it. The cost: one per param = state per param; the 12 bytes of optimizer state trace back here.
This figure replaces a bare claim—“gradient scales differ by orders of magnitude” sounds trite; drawn, it's red bouncing off the frame.
Ⓑ is the point: no middle ground. Even the largest stable learning rate needs hundreds of steps for half the flat direction. The valley's shape, not bad tuning.
Both paths are real runs; all three numbers asserted.
Sources and assumptions

📌 The dilemma (one shared learning rate: too large, the second update flies off the map; too small, a hundred updates don't reach the bottom; so each parameter gets its own rate) is from Hung-yi Lee's “Training Tip” slides (Fall 2025 GenAI-ML course).

⛔ But we did not copy his two specific numbers. Borrowing a hand-picked learning rate means borrowing its landscape. We built our own valley, ran it twice; asserts pin each path.

⚠️ The figure's steepness ratio is 25, not “orders of magnitude.” Draft one used 1,000: physically right, visually dead; dozens of steps moved the flat axis dozens of pixels, so the zigzag became a vertical line. ⭐ Rule: to show a dynamic, pick visible values; real magnitudes to the formula. Ⓑ's row 3 extrapolates.

⭐ Both step counts in Ⓑ are closed-form: each step shrinks the flat direction by (1 − 1.9 ÷ ratio), so halfway takes log½ ÷ log(that). The script checks them against a run: within 1 step.

Make the learning rate an axis. Slide right to raise η; the path morphs:
creeping along the floor → just right → bouncing → off-screen.
⭐ The ruler's three bunched lines: red is the divergence threshold (closed form: 2 ÷ steep-direction curvature), flanked by ±5%. Only 10% apart, yet crossing flips convergence to divergence.
⛔ The left end fails too: once η stops the bouncing, the flat direction covers only a third in 26 steps. Neither end works: the one-learning-rate dilemma. (15 s silent loop, Manim; tools/manim/. Threshold, ±5% behavior and “one third” computed at render, asserted.)

Adaptive methods come down to one idea

Ignore gradient size, keep direction: divide out each parameter's own scale.

❓ Why divide? An exact answer, small enough to check here.

Step size: there is an exact answer Earlier: what dividing gives. Here: why divide Best step = |f′| ÷ f″ Ⓐ Parabola: f′ ÷ f″ = distance to the bottom In theory one step, no tuning You're here 0.5 left Min Steep valley f′ = 2, f″ = 4 2 ÷ 4 = 0.5 = “0.5 left” exactly You're here 2 left Min Flat valley f′ = 1, f″ = 0.5 1 ÷ 0.5 = 2 = “2 left” exactly Ⓑ Counterintuitive: big gradient ≠ far away So raw gradients mislead Gradient only Steep 2.0 > flat 1.0 → “steep is farther; go big” Wrong: it's closer (0.5 vs 2.0) → After ÷ f″ 0.5 and 2.0 → both right Now comparable Ⓒ Why not compute f″? Too expensive Brings the formula back to reality Hessian One per param pair 10¹¹ params: can't store, let alone compute Estimate From past f′ Big grads → probably steep → In code AdaGrad: sum of g² Adam: EMA That's the denominator → So the adaptive denominator isn't a trick: it stands in for f″ To be precise: it isn't f″, not even in units. f″ is loss ÷ param²; √v ≈ gradient RMS, in loss ÷ param: one “param” short. So a true Newton step g ÷ H is already in param units, needs no LR; Adam's m ÷ √v is unitless. This explains 3.3: Adam strips grad scale; each step ≈ 0.2 × LR. As m ÷ √v is unitless, Adam's LR is in “param” units, unlike SGD's “param² ÷ loss”: different. The true reason Adam LRs tune & transfer well.
This figure answers “why divide.” Ⓐ On a parabola, f′ ÷ f″ is exactly the distance to the bottom: the best step is computable.
Ⓑ Counterintuitive: big gradient ≠ far. Steep-valley point: gradient 2.0, distance 0.5. Without ÷ f″, no comparison.
Ⓒ Reality: f″ is unaffordable at 100B+ params; adaptive optimizers estimate it from gradient history—hence Adam's denominator.
Sources and assumptions

📌 “Optimal step = |f′| ÷ f″” and “estimate f″ from first derivatives” follow Hung-yi Lee's (National Taiwan University) Gradient Descent slides. The figure is our redrawing.

⭐ All four numbers check on the spot: on f = a·x², f′ ÷ f″ = 2ax ÷ 2a = x, exactly the distance to the bottom. Script-asserted.

⛔ First, a misleading, near-universal claim: “Adam's denominator estimates f″.”

It's hindsight, not the inventors' view, and can't explain why it excels on sparse data or why it turns cautious when unsure.

The real path started from statistics, not geometry.

How it was found: three generations, one problem each

① AdaGrad (2011): rare words never got learned.

They had text in mind, not neural nets. “The” appears millions of times, a technical term three times, yet the rare ones often say the most.

With one learning rate that term never learns: its gradient exists only on its few samples, zero elsewhere; common features set the pace.

⭐ So: give each parameter its own learning rate, bigger for the rarely seen.

How often seen? Sum its past squared gradients: large if everyday, small if rare. Then divide by the square root.

⛔ Not a word on curvature or second derivatives. Pure statistics. Provenance in 3.4: its starting point is TF-IDF, not curvature.

❓ So what exactly is the “second moment”?

Square, average, root: the gradient's RMS, i.e. this parameter's usual gradient size.

⭐ Not curvature: this knob's typical recent response size, the parameter's own ruler.

What does dividing do? Current over usual size: how big this one is versus normal, a dimensionless number.

⭐ Raw gradient 100 or 0.01: after dividing, all share one scale. That's “adaptive learning rate”: relative to its own usual size, not absolute.

The full picture: “moment” and its real geometry

First, “moment.” In statistics, first moment = mean, second = mean square. Adam's v is a moving average of squared gradients; nothing more to “second.”

A picky detail: it's uncentered. E[g²], not the variance (which subtracts the mean); Adam holds m, the mean, yet doesn't subtract it.

Not an oversight: it's the source of the “signal-to-noise ratio.” Constant gradient: m = √v, ratio 1, full step. Divide by variance and it blows up: a constant gradient's variance is 0. Unsubtracted, the ratio is bounded.

Geometry? There is one, just not the assumed one.

Picture the loss surface as a long, flat ellipsoid: gradients large in some directions, small in others. Adam rescales each axis by its typical gradient size until all directions compare.

Diagonal preconditioning, genuinely geometric—but it scales by how much a direction usually moves, not how sharply it curves.

So it fixes mismatched scales, not sharp curvature. 3.5's warmup hits exactly that limit.

⭐ Also: it scales the axes but never rotates them. Coupling between axes is out of reach. Muon and Shampoo later target that gap: neither scales per coordinate (3.4).

② RMSProp: the ruler goes stale.

AdaGrad's fatal flaw: the sum only grows; steps shrink until training stalls.

Hinton changed one thing: a moving average replaces the sum.

⭐ Why: a parameter's “typical gradient size” drifts. Large early, small late. All history as ruler = a stale ruler.

③ Adam: numerator and denominator didn't match.

RMSProp had a ruler (denominator), but the numerator was the raw, noisy gradient; averaging it too gives momentum.

⭐ Now both are recent statistics. Numerator: direction; denominator: scale.

The alignment brought a bonus. The paper calls this ratio a signal-to-noise ratio:

  • Gradient steady, one direction: numerator builds, denominator holds, ratio → 1, a full step;
  • Gradient jumpy, sign-flipping: numerator cancels, the squared denominator still grows. Step shrinks itself.

⭐ So it does two things: evens out sensitive and sluggish, and turns cautious when unsure. The second came free, undesigned.

⭐ Finally, both paths side by side: the real lesson.

Cauchy's path is geometric: curvature sets the step. Rigorous; too costly at hundreds of billions of params.

Adam's path is statistical: ignore terrain, read the parameter's own history (usual gradient size; is this one unusual?).

⭐ Both end up close from wholly different starts. Hence “adaptive learning rate,” not “approximate Newton method”.

The key step: Adam's update ≈ “momentum ÷ √(second moment),” so the units cancel.

📌 In full: m̂ / (√v̂ + ε); ε (typically 1e−8) guards against ÷0. Strictly, units cancel only at ε = 0; ε carries gradient units, the one leak in “independent of gradient scale.” Usually negligible, but for tiny gradients (e.g. deep-layer vanishing, 6.5) ε, not √v̂, is the denominator, and Adam degrades to plain SGD.

Adam divides out the gradient's units: the step becomes a constant, whatever the gradient scale.

⛔ The constant isn't 1. It's 0.2. My earlier error: cancelled units make it constant, not 1.

Measured, it holds at 0.2–0.3, the same across model sizes.

If the gradient is mostly noise, a too-neat closed form follows:
update RMS ≈ √[(1−β₁) / (1+β₁)]
β₁ = 0.9 gives 0.2294, matching the measurements.

📌 Source: Su Jianlin, “Why Is Adam's Update RMS 0.2?” kexue.fm/archives/11267 (Scientific Spaces, in Chinese). Simulation (pure Gaussian gradients: 0.225) and a mean-field approximation agree.

The formula has only β₁, no β₂. So the βs split: β₂ sets memory length, β₁ step size (more in 3.6).

So Adam's learning rate changes role: not a conversion factor, a multiplier on a constant. Each parameter moves 0.2 × lr per step. Hence easy tuning: decoupled from loss scale and gradient size, one coefficient fixed by β₁.

The 0.2 also settles “retune when switching optimizers?”

Any optimizer scaled to update RMS 0.2 inherits Adam's tuned learning rate and weight decay. Kimi K2's Adam → Muon move did this: RMS set to 0.2, all else copied.

📌 Same source (Su Jianlin). The units thread's most practical payoff: make a quantity constant and it becomes an interface.

Muon: “drop size, keep direction” on three shapes Read side by side: one idea, not three topics Scalar Diagonal General Ⓐ Why it's “matrix sign” Each column: raw in, direction out Scalar −1 0 +1 0.42 Whether 0.42 or 42 Output: +1 Diagonal +0.8 +1 -0.3 -1 +0.5 +1 On diagonal Each on its own Flattening loses nothing Limit of vector methods General matrix Singular values All → 1 Keep direction, drop size Ⓑ “A matrix is just numbers?” One example says no Flattening erases this A matrix Diagonal is special: sum = trace, trace: similarity-invariant, = Σλ. flatten After flattening All slots alike; ticks mark the six diagonal ones, lost to the optimizer. SGD / Adam do this (elementwise) — Muon refuses. Ⓒ Change the ruler, change the optimizer: two shapes Same task: go farthest along grad, step capped σ₁ σ₂ grad Lands here ○ F norm = √(σ₁² + σ₂²) (flattened L2 length) On the gradient ray → ratios kept: big stays big That's SGD σ₁ σ₂ grad Lands here 1 1 □ Spectral = max(σ₁, σ₂) (induced by matrix × vector) At the corner → both σ become 1: that's msign That's Muon Same grad Same goal ⬇ Only the shape Ⓓ “A dozen-odd extra matmuls per step: slow?” They fill idle time Muon author's blog: FLOP overhead under 1%, FLOPs not wall-clock This step's grad Busy Gap Nearly idle Next step's grad Busy msign runs here These matmuls are fixed-size, parallel: no stalls It spends idle compute to buy 4 fewer bytes/param (16 → 12)
Read Ⓐ's three columns side by side. Apart, three facts; together, one operation.
Ⓑ: the trace separates matrix from vector; Ⓒ: new ruler, new optimizer; Ⓓ: why a dozen-odd matmuls cost almost nothing.
📌 All four from Su Jianlin's Muon post (Scientific Spaces); redrawn by us.
Sources and assumptions

⭐ Source of the presentation: Su Jianlin, “Appreciating the Muon Optimizer: The Essential Leap from Vectors to Matrices,” Scientific Spaces (in Chinese), spaces.ac.cn/archives/10592. Ⓐ's three special cases, Ⓑ's trace example, Ⓒ's norm view and Ⓓ's idle-window explanation: all four come from that post. ⛔ The figure is our own redrawing, not a copy

⚠️ For Ⓒ's norm view, the post cites “Old Optimizer, New Norm: An Anthology”; ⭐ and traces the lineage: Stochastic Spectral Descent (2015) had proposed roughly the same algorithm

⛔ No convergence-speed comparison: we never measured these head to head (same rule as 3.1).

⚠️ Ⓓ's “under 1%” isn't from that post but the Muon author's blog (kellerjordan.github.io/posts/muon): FLOP overhead computed from Llama 405B's width and batch tokens, not wall-clock time we measured. ⛔ Newton–Schulz chains small dependent matmuls; fewer FLOPs isn't less time. Measure on your target config (our rule: don't copy numbers across scales).

⛔ This figure once said “within 5% / the author claims 2%”; that 2% is unsourced. The Muon author's blog: FLOP overhead under 1%. Fixed 2026-09-17.

🕰️ Aside: Adam's original (2014) convergence proof was shown wrong in 2018 (Reddi et al., ICLR 2018). Fix: AMSGrad. Everyone kept using Adam. Working isn't proven. Details below.

That stumble: the proof's flaw, AMSGrad, and why nobody switched

⛔ 🕰️ This “add one more thing” lineage has a rarely mentioned stumble.

The original Adam paper (2014) came with a convergence proof. In 2018, that proof was shown wrong.

Reddi, Kale and Kumar, “On the Convergence of Adam and Beyond” (ICLR 2018, arXiv 1904.09237). In the abstract's own words, they give “a simple convex optimization setting where Adam does not converge” and “describe the precise problems with the previous analysis of Adam algorithm.”

The root cause is the exponential moving average itself: v forgets old large gradients, so steps along some directions can fail to converge indefinitely. Their fix is AMSGrad: make v never decrease.

What happened next matters most: everyone kept using Adam.

AMSGrad barely helped in practice, so the “wrongly proven” algorithm is cited over 100,000 times and trained every large model today.

The lecture's core idea in miniature: working and proven are different things. Its running lesson: don't trust conclusions; check assumptions.

Muon goes one step further

The figure covers most. Three points it misses:

Ⓐ's middle column: on a diagonal matrix Muon reduces to elementwise sign: the vector approach is a special case of the matrix one. Teaching them as two things inverts that.

Steps the formula omits but training takes

Between gradient and update: all-reduce average across devices → clip once by global norm (against rare huge gradients) → schedule's current learning rate → weight decay → then the update.

The gradient says which way; the optimizer, how far.

Decades of evolution answer one thing: who decides “how far”?

  • SGD: you decide, so go tune it.
  • AdaGrad / Adam: the gradient's own history of scale decides.
  • Muon: the matrix's geometry decides.

That's the whole thread.

3.4 New optimizer, new bill

An optimizer is a memory decision, not just a convergence one.

Muon's saving has two limits, both named by its author.

  • 2D parameters only. Scalars, vectors, the embedding, and the output head still use AdamW; the writeup says input and output layers work best with AdamW. So the model saves less than that quarter.

    ⚠️ This hides two kinds of exclusion: scalars and vectors are out in principle: orthogonalization needs a matrix with several singular directions; on a vector it degenerates to plain normalization.
    But the 2D embedding and output head could take Muon; AdamW stays because it tests better: an empirical exclusion, not a principled one.

  • Each step is slower (per the writeup). Not “faster and smaller”: it trades memory for per-step time.

No conflict with Ⓓ (“fits into idle time”): “slower” is direction, “idle time” magnitude (under 1% of FLOPs, per Muon's author). Magnitude is configuration-dependent: a chain of mutually dependent small matmuls, where few FLOPs ≠ short wall-clock. Measure on your setup.

The evidence is hard: Muon raised the NanoGPT speedrun record 35%; since then (as of his blog post), twelve records by seven people all use it. Match it with tuned AdamW and you'd retake the record. Nobody has.

Link to the last lecture: DeepSeek-V4 trained most modules with Muon, one of three major upgrades in its abstract (arXiv 2606.19348).

One line, not a menu: each generation treats the instability its predecessor left. Read it against the 6.0 failure-mode table; no memorizing.

  • SGD: direction, no memory. Bounces on steep directions, stalls on flat ones.
  • + Momentum: treats the bouncing. Averaging past gradients cancels opposing parts, adds aligned ones.
  • Adagrad / RMSProp: treat mismatched parameter scales. One shared learning rate, but gradients span orders of magnitude (the two trajectories in 3.3b). Fix: divide each step by its own gradient RMS.
  • Adam: combines both: first moment for direction, second for scale. Price: 8 extra bytes (3.1).
  • AdamW: treats a side effect. The second moment scales weight decay, weakening it on large-gradient parameters. The fix applies decay to the weights directly: the decoupled in the title, and essentially the paper's one contribution.
  • Muon: treats the update direction itself. Elementwise Adam lets a few directions dominate a matrix update; Muon orthogonalizes it so steps across directions even out.

New optimizer, new disease treated. The cost always lands in one place: extra bytes per parameter.

Aside: what weight decay is, and why it lives in the optimizer

Below: how decay is applied. What it is, we've only touched on. Three questions settle it.

① What it is, and why it differs from the textbook

Each step shrinks weights slightly toward zero.

Textbook reason: regularization, no memorizing via a few huge weights (from ridge regression).

⛔ That fails for LLM pretraining: a single pass over data ≫ parameters leaves almost no room to overfit.

Yet every config sets it, always to 0.1: weight_decay = 0.1 in all DeepSeek models (with β₁ 0.9 / β₂ 0.95 / clipping 1.0). So what does it do?

A 2017 paper spelled it out: with normalization, L2 barely regularizes. It sets weight scale, which sets the effective LR.

Why: scale a weight matrix feeding a Norm by 2× and the output is unchanged (the Norm undoes it). Length doesn't change the function, only how far a given step turns the direction. Longer weights, smaller angle: lower effective LR.

Weights grow during training; left alone, the effective LR drifts down. Weight decay pins the length. It is now a knob that steadies the effective LR more than a regularizer.

📌 van Laarhoven, arXiv 1706.05350 “L2 Regularization versus Batch and Weight Normalization.” ⚠️ That paper is from the BatchNorm era. The same scale invariance applies to RMSNorm, but that's our extrapolation beyond its experiments (the rule in 2.6).

② How: by multiplying, not subtracting

θ ← θ − η·λ·θ = θ·(1 − η·λ)

Each step multiplies by just under 1: exponential decay. Without gradients, weights would go to zero.

How gentle? V3's peak η = 2.2×10⁻⁴, λ = 0.1 → 2.2×10⁻⁵ shrink per step; 80% remains after 10,000 steps. Very light.

⚠️ A trap: the shrink factor contains the learning rate. Change the LR schedule and regularization strength changes too: one knob turned, two moved. Hence in 3.3b's “order of one step,” decay comes after reading the scheduled LR.

In practice decay usually skips biases and norm gains. For a specific model, check its config; don't copy blindly.

③ Why it's in the optimizer: originally, it wasn't

It began as a loss term, ½·λ·‖θ‖²; autodiff turns it into λθ in the gradient. It was part of “what you optimize.”

Why move it? In the loss, it runs the gradient's pipeline: momentum, second moment, denominator, every stage. We want uniform pull.

Only the update step skips the denominator, and the optimizer owns it.

It went from “what you optimize” to “how you step”: new owner, same formula.

⚠️ Cost: out of the loss, you can't claim to minimize “loss plus a regularizer.” You no longer optimize the loss you wrote. Criticized, but it won in practice.

❓ The “W” in AdamW, pictured

Weight decay nudges weights toward zero each step so they don't grow. The question is at which stage.

The old way (original Adam): mix it into the gradient. Add λθ to the gradient; both are subtracted anyway. But every later stage then hits it, √v division included.

Backwards, as two parameters show:

Both at 10√vDecay termNet shrink
Common: big gradi­entsLargeScaled down Barely pulled
Rare: tiny gradi­entsSmallScaled up Pulled hard

Both at 10 deserve equal restraint, yet get the reverse. We want “pull every weight back a little, uniformly,” not “penalize by gradient size.” The two got tangled.

AdamW's fix: decay skips the denominator. The gradient path is untouched; the update separately subtracts η·λ·θ. That's “decoupled”; W is for weight decay.

❓ Muon drops the second moment entirely. Who now treats mismatched scales?

The list owes this: Adagrad names the disease; Muon throws out the medicine.

Orthogonalization treats it at another level.

Orthogonalization equalizes the momentum matrix's singular values (how hard it pushes along each direction). Each direction's step then ignores its gradient size: scale is erased all the same.

Adam's vMuon's ortho­gonal­ization
Where it erasesPer element, each by its own Whole matrix, singular values → 1
ErasesIts own gradient scale Gaps between singular directions
Cross-axis coupling Out of reach (axis-aligned; 3.3b) Within reach: singular directions span axes

That's also the cost. Adam's v also flags a parameter acting oddly “lately”; drop v, lose that. Muon keeps momentum: smoothing stays, per-element adaptivity goes.

⚠️ Orthogonalization equalizes only between directions, not the overall step size. Hence Moonlight (below) adds an overall-size rule; and once Kimi K2 calibrated Muon's update RMS to 0.2 (Adam's constant), Adam's whole hyperparameter set carried over.

From AdaGrad to Adam: not four options but one causal chain Feed one constant gradient, plot each effective LR: a healthy one is flat AdaGrad: → 0 RMSProp: early spike Adam: flat Ⓐ Each link patches the last hole and leaves a new one Curve in each box: its effective LR (real runs) SGD One global learning rate can't fit mismatched scales ÷ RMS of its own past gradients → one scale per parameter AdaGrad 2011 Denominator only grows → effective LR falls monotonically to 0 sum → moving average → denom. forgets, no monotone decay RMSProp 2012 No momentum, no bias corr. → first steps spike +momentum +bias corr. → smooth, stable start Adam 2014 Constant grad: flat from step 1 Ⓑ AdaGrad fails late: the denominator only grows, LR sinks to 0 x: steps; y: effective LR (steady = 1) 1 0 step 600 SGD (ref.) AdaGrad by step 600: 4.1% of steady state Not the data: the input gradient never changes; the ever-growing denominator crushes it by itself. Ⓒ RMSProp fails early: the average starts at 0, far too low at first Small denom → big update. Adam's bias correction fixes it 1× 4× 8× step 400 RMSProp (no corr.) Adam, no corr. Adam (corrected) step 1 amplified 32× (off chart) uncorrected Adam peaks at 6.6×: momentum's bias partly cancels it Green is flat from step 1. Bias correction is no minor patch: it makes the first few dozen steps usable. Strongest evidence: a later link contains an earlier one as a special case The Adam paper proves AdaGrad is a special case of Adam (β₁ = 0, infinitesimal (1−β₂), step size annealed as 1/√t). So it is a chain, not four parallel options. Each link leaves a debt: Adam's are decay caught in the adaptive denominator (AdamW fixes) and two states per parameter (Muon trims). Both come later.
The lineage names each disease; this figure is its oscilloscope: each disease has a shape.
One causal chain, not four options: each link patches one hole and opens another.
Feed in a constant gradient: a healthy optimizer draws a flat line; anything else is a defect. AdaGrad fails late (its ever-growing denominator sinks the effective LR to 4% of steady state); RMSProp fails early (its average starts at 0, so step one is 32× too big); Adam is flat from step one.
Strongest evidence (the Adam paper): AdaGrad is a special case of Adam (β₁ = 0, infinitesimal (1−β₂), LR annealed as t^−1/2).
Sources and assumptions

📌 Each link is from the Adam paper (arXiv 1412.6980, read in the original). Its AdaGrad update, verbatim: θ_{t+1} = θ_t − α·g_t / √(Σ g²): the denominator sums squares, not divided by t: hence monotonic decay. Adam is “designed to combine the advantages of” AdaGrad (sparse gradients) and RMSProp (on-line and non-stationary settings).

⭐ On bias correction, the paper says the moving averages start at 0, so the estimates are biased towards zero, especially during the initial timesteps, worse as β nears 1. RMSProp is named in §5 (relation to AdaGrad and RMSProp): “RMSProp also lacks a bias-correction term,” and without bias correction, like in RMSProp. Ⓒ's orange line draws that sentence.

⚠️ Hung-yi Lee's course slide uses an RMS Adagrad (divides by t+1), unlike the original here. ⛔ We use the original: the effective LR's monotonic decay holds only there (a sum not divided by t). Otherwise you'll think you miscalculated.

📌 Side by side, the three sources are a fun read (full story in the instructor notes). AdaGrad is a long JMLR paper (Duchi/Hazan/Singer, 12(61):2121–2159, 2011, 39 pages). ⭐ Its title: “Adaptive Subgradient Methods for Online Learning and Stochastic Optimization”: never written for deep learning.

⭐⭐⭐ RMSProp has no paper. Its only “publication” is a slide from Hinton's Coursera course, cited as Tieleman & Hinton (2012), Lecture 6.5-rmsprop. ⭐ The slide's subtitle is itself the complete definition of the algorithm: “Divide the gradient by a running average of its recent magnitude.” One sentence became the middle link of this chain.

⭐⭐ The Adam paper (ICLR 2015) has two delightful details. The first-page footnote reads “Author ordering determined by coin flip over a Google Hangout.” The acknowledgments read “special thanks to Ivo Danihelka, and Tom Schaul for coining the name Adam”: the authors didn't name Adam themselves.

⭐ The three numbers (4.1%, 32×, 6.6×) are computed by a script with five asserts: AdaGrad decays monotonically; it's tiny by step 600; uncorrected, the first step is amplified many-fold; corrected, exactly 1 from step one; uncorrected Adam peaks below RMSProp (the momentum term's bias partly cancels).

Built for one reason, kept for another

The lineage above is hindsight. Its inventors were thinking of something else.

📌 AdaGrad (2011): the motivation was sparse features, not stability.

The paper: “infrequently occurring features are highly informative and discriminative,” so they get high learning rates, frequent ones low; “each time an infrequent feature is seen, the learner should ‘take notice.’”

Say “the” appears millions of times, a technical term three times. Under one learning rate, the term never learns. AdaGrad: discount each parameter by its accumulated gradient.

📌 Duchi, Hazan, Singer, JMLR 2011 (preliminary version at COLT 2010). The paper opens with TF-IDF as motivation (“We use this old idea as a motivation”), search's rarer-words-matter-more rule: an old information-retrieval idea, not curvature; the theory came later, as a guarantee.

📌 Adam (2014): the “equalize the size of every step” explanation starts here.

§2.1: the “effective magnitude of the steps … [is] approximately bounded by the stepsize setting α,” a “trust region … beyond which the current gradient estimate does not provide sufficient information.”

Plainly: whatever the gradient, every step is about the same size. Small gradients get lifted, large ones pressed down. That's where 3.3b's “0.2” comes from.

The paper calls m̂/√v̂ the signal-to-noise ratio: the smaller the gradient vs. its noise, the nearer the step is to zero. 📌 Kingma & Ba, arXiv 1412.6980 (ICLR 2015). The middle generation, RMSProp, comes from Hinton's Coursera slides: no formal paper.

⛔ While we're at it: “SGD doesn't work” overstates it.

SGD + momentum still competes on vision; on LMs Adam wins systematically. The lineage lists diseases treated, not useless predecessors.

The lesson is the mismatch: AdaGrad was built so rare words get heard; three years later Adam reread it as “equalizing step sizes,” the only framing we use today.

Rule of thumb: why a method was invented and why it stuck often differ. Tell only the second; people assume inventors knew. Usually they didn't.

Three state-saving branches, each cutting different bytes of the 16

Each changes a different box of the 16 bytes. Bytes per 3.1's accounting:

MethodBox changedBytes per paramCost
AdamW (baseline)—2 + 2 + 4 + 4 + 4 = 16—
MuonDrops the second moment2 + 2 + 4 + 4 = 12 2D only; made up by ortho­gonal­ization
LionAlso first moment only2 + 2 + 4 + 4 = 12 Sign of the gradient only
AdafactorFactors v into a row + a column That copy ≈ 0; 8 total, no first moment Loses v's per-element info
8-bit AdamBoth quan­tized to 1 byte 2 + 2 + 4 + 1 + 1 = 10 New storage, same algorithm

Three different approaches: Muon and Lion drop the copy; Adafactor uses a cheaper representation; 8-bit stores the same thing in fewer bits. So 8-bit stacks with the first two, which exclude each other.

📌 8-bit Adam quantizes states in small independent blocks, with nonlinear quantization precise for large and small values alike. Reported: 32-bit results across tasks, hyperparameters unchanged; a two-line drop-in swap.

📌 Dettmers et al., arXiv 2110.02861 (ICLR 2022 spotlight).

But “same algorithm, different storage” isn't quite true: per the abstract, 32-bit parity needs three things together: block-wise and nonlinear quantization plus a “stable embedding layer.”

The third changes the architecture, not storage. LM input tokens are very uneven, so embedding gradients vary wildly: quantization's worst case.

Rule of thumb: for a “drop-in replacement,” count in the abstract what it changes. “Two lines” is for the caller.

Scaling Muon: two additions

3.3b gave Muon's idea; good ideas don't automatically scale. A 2025 paper added weight decay (absent originally; the AdamW entry's disease) and a separate overall update-size rule, since orthogonalization only equalizes across directions.

With both, it works out of the box. Its scaling laws report Muon at ~2× AdamW's compute efficiency (compute-optimal).

📌 arXiv 2502.16982 (Moonlight, 3B/16B MoE, 5.7T tokens). “∼2× computational efficiency” is the paper's scaling-law result, not ours. Per 2.6, re-verify at other scales. Unverified: whether Muon still needs fp32 master weights.

3.5 How to draw the LR schedule

First, a ruler: not the optimum, just whether you're way off.

“Is my learning rate right?” A line you can check by eye Measures update ÷ parameter — the update, not the gradient (source's emphasis) low right high black line = 1e−3 Ⓐ Three learning rates, one network: which side of the line each falls on Log y-axis; black line = CS231n's rule of thumb 1e−3 — a rough heuristic, not a law 10⁻5 10⁻4 10⁻3 10⁻2 10⁻1 10⁰ steps → 1e−3 CS231n heuristic too low η=0.000961 right η=0.0288 too high η=0.865 The source reads it in three lines: on the line = about right; far below = LR maybe too low; far above = LR maybe too high. Its virtue: independent of model size — same units top and bottom, so they cancel. Change models; the line stays put. Ⓑ Yet the “too high” run has lower loss — the cost shows up elsewhere That's what the run gave us; no hiding it. So this ratio is not a “lowest loss” metric 0 5 10 15 first-layer weight norm ‖W‖ right / too low nearly overlap too high The too-high run inflates the weights 4.2×, step-to-step loss jitter grows ~9× — and LR ×2.1 again gives NaN outright. Verdict: this ratio measures not “lowest loss” but “how close to the cliff.” A diagnostic is not an objective. This is a toy problem, so “lower loss” doesn't count here anyway Random X → random Y, no val set: it's memorizing noise (faster = lower loss). In real runs, bigger weights + jitter usually mean worse generalization. But this panel's point doesn't depend on the toy: the ratio only asks “is my step a sane size for this parameter?” A thermometer, not a treatment.
“Is my learning rate right?” One line checks it by eye. It plots update ÷ parameter (update, not gradient, as the source stresses); CS231n's rule of thumb: 1e−3.
Ⓐ Three learning rates, one network: too low under the black line, “right” on it, too high far above.
⛔ Ⓑ matters most: the too-high run has lower loss, yet inflates weights 4.2× and jitter ~9×, and 2.1× more LR gives NaN.
The ratio measures not lowest loss but distance to the cliff: a diagnostic, not an objective.
Sources and assumptions

📌 CS231n, “Neural Networks Part 3,” “Ratio of weights:updates”: “A rough heuristic is that this ratio should be somewhere around 1e-3.” ⭐ The source also stresses “Note: updates, not the raw gradients”.

⚠️ Our own curves: two-layer tanh MLP (8→16→4), hand-written numpy forward/backward, fixed seed, 400 steps. The “right” LR was swept (steady-state ratio closest to 1e−3), not picked; “×2.1 again gives NaN” came from bisection.

Only three decisions: where it starts, how high it goes, how it comes down.

Two LR schedules: very different shapes, same landing point Y-axis is % of peak, not absolute LR · the warmup sliver would be a vertical line at true scale; widened here GPT-3 · cosine V3 · WSD 10% of peak Ⓐ X-axis is training progress (fraction of total tokens) Both normalized, so they overlay directly 0% 25% 50% 75% 100% Training progress (tokens consumed / total) 100% 50% 10% % of peak 10% of peak GPT-3 cosine hits 10% at 260B tokens DeepSeek-V3 long plateau decay starts only at 10T Warmup widened GPT-3's real share: 0.125% Ⓑ WSD's appeal is practical Not “it converges better”: it postpones a decision Cosine's hard constraint Curve shape depends on the endpoint So you must know total steps up front Train longer midway? Redraw the whole curve The plateau's freedom The constant phase can stop anytime Add a short fast decay → usable checkpoint More data? Resume from the plateau In one line: “how long to train” moves from day one to any time Two landing points, one coincidence Warmup is a few ‰ of the run, not a knob to tune: GPT-3 0.125%, V3 first 2,000 steps. A paper found that with the target LR fixed, longer warmup buys almost nothing; the peak itself is what matters. Both curves land near 1/10 of peak: GPT-3 states “10%”; V3 goes 2.2e-4 → 2.2e-5, exactly 1/10 (a last stretch drops to 7.3e-6, ~3.3%). Two samples don't make a law, but they show “decay to zero” isn't the default.
Shape, not height: the peaks differ 3.7×, so they're normalized to overlay. Absolute peaks: an eight-size comparison in the text.
Look for V3's long plateau and both curves hitting the same “10%” dashed line.
Sources and assumptions

GPT-3: arXiv 2005.14165 App. B: “linear LR warmup over the first 375 million tokens,” “cosine decay for learning rate down to 10% of its value, over 260 billion tokens (after 260 billion tokens, training continues at 10% of the original learning rate)”; 300B total

DeepSeek-V3: arXiv 2412.19437 §4.2: linear to 2.2e-4 over 2,000 steps → flat to 10T → cosine to 2.2e-5 over 4.3T → final 500B: first 333B there, last 167B at 7.3e-6; clipping norm 1.0

⭐ What warmup actually does (not “too few samples to estimate”): arXiv 2406.09405 (NeurIPS 2024). Main benefit: a larger tolerable target LR. Systematic analysis of this shape: arXiv 2410.05192

⛔ This figure has no absolute learning rates (peaks differ 3.7×; overlaid, shapes vanish). Eight-size peak table: main text

The shapes differ completely: GPT-3 slides down after warmup; V3 holds a long, flat plateau. Plateau or not is the camps' only disagreement (third decision). Each decision below has a public config to copy, not guess.

Phase 1 · Warmup: the popular explanation is wrong

Tutorials say: early Adam's second moment is under-sampled, so ease in.

⛔ A systematic 2024 study says plainly this is not the main reason.

Its finding: warmup's overwhelming benefit is tolerating a larger target learning rate, by moving the network to a better-conditioned region. It rebuts “too few samples” directly: preconditioned curvature starts high, as high at large batch.

So warmup's real value: a higher peak, and an easier-to-tune one (wider usable range).

📌 Source: Why Warmup the Learning Rate? Underlying Mechanisms and Improvements, arXiv 2406.09405 (NeurIPS 2024).

“Says plainly … not the main reason” overstates it. The paper's Limitations admit small experiments: a 4-layer Transformer with context 64, plus a WRN on CIFAR-10. Orders of magnitude below this lecture's scale.

So why can the peak go higher? Back to 1.4b's second derivative; otherwise “better-conditioned region” is a tautology.

Curvature, not you, sets the largest step

On a parabola, gradient descent diverges once the step exceeds 2 ÷ curvature. Not slower: each step overshoots further, and it flies off.

So local curvature caps the usable learning rate: higher curvature, smaller legal step.

The paper measured high preconditioned curvature early in training, so high your intended peak isn't legal. It's as high at large batch: their evidence against “too few samples,” since a bigger batch would fix estimation noise.

So warmup is concrete: run at a safe learning rate and let the network walk to lower curvature. Once curvature drops, the peak is legal.

Causality: the surface flattens; the network doesn't learn to tolerate big steps. Hence longer warmup buys almost nothing: once you're there, more steps barely lower curvature.

Three misconceptions, all from the false premise “danger = large gradients”

❓ “Starting at a 1-in-129,280 guess, aren't gradients huge?”

A large loss isn't a large gradient. The initial loss, ln 129,280 = 11.77, is big. But in 1.2b's figure the logits-layer gradient is p − y: every entry lies between −1 and 1, the vector's length at most √2. Vocabulary 129,280 or 1,000: no difference.

❓ “Doesn't Adam normalize the gradient? What's left to be large?”

Good question; it points at the answer. Adam divides momentum by the gradient's RMS; units cancel, the ratio is O(1). So step size ≈ the learning rate, nearly independent of the gradient.

So the danger isn't early large gradients. What's large is the ground's curvature, not your step.

Take a step η. On gentle terrain the slope barely changes. On sharp terrain the same step overshoots the bottom, landing higher on the far side; each next step flies further: the 2 ÷ curvature line, live.

1.4b's point: Adam divides out “steep,” not “curved”. Its second moment is a statistic; curvature, a second derivative, is geometry it never computes. “Preconditioned curvature” means curvature after Adam's rescaling. Still high after rescaling? That's exactly what was measured.

❓ “Hasn't the denominator just not built up yet?”

Adam guards against that with bias correction: both moments are divided by 1 − βt to offset the short early history. No “zero denominator.”

“Noisy early estimates, so ease in” is what the paper rejected. Evidence: curvature is as high at large batch, which would fix estimation noise.

Don't push this too far: it doesn't mean “curvature keeps falling, so the learning rate can keep rising.” The curve stops at the peak, then falls (Phases 2–3). Warmup gets you to the peak, no further.

A third explanation exists; its premise keeps it out of modern models. New angle: Adam changes what “vanishing gradients” means.

The third explanation in full Premise: Post-LN (6.5 explains). On Pre-LN it doesn't apply; skip to the table below

An Adam update, as the source writes it:

Δθ = −η · E[g] ⁄ √E[g²]

Top and bottom share units, so the ratio is O(1) and the update O(η). Unlike SGD, where step ∝ gradient: small gradient, small step.

With Adam, vanishing gradients stop meaning “can't learn”: while the gradient beats noise, updates stay O(1).

⛔ But “no longer stuck” isn't good news: it trades the problem for a worse one.

At noise-level gradients, top and bottom are both noise, and the ratio is still O(1).

⭐ Full-sized steps; only the direction is random.

SGD just stalls in this regime: ugly, but safe. Adam charges off in a random direction at full step size.

That's where the trouble starts. The source's reasoning (premise: real, marked vanishing gradients):

  1. No warmup: fast from the start. Later, more sensitive layers learn faster.
  2. But the later layers take the earlier layers' outputs as input, and the earlier layers haven't learned yet, so they race ahead on bad inputs.
  3. They soon hit a bad local optimum; learning slows; gradients reaching earlier layers weaken further.
  4. Gradients go bad, but Adam's step stays constant-order: constant-order random noise. Direction scrambles; layers break front to back.

Per the source, warmup “slows the later layers' learning and gives the earlier layers more time to optimize, so all layers optimize in step”.

One symptom to match against your logs: Post-LN without warmup, the loss settles near a constant, then later diverges to NaN.

But keep its premise attached, stated plainly by the source:

“This assumes vanishing gradients. For Pre Norm-like structures, with no notable vanishing, training often works without warmup.”

So each has its own slot; they don't compete.

ClaimPremiseStatus now
Second moment under-sampledNone stated Refuted by the 2024 systematic study
Lets the network take a larger peak LRGeneral case Holds, but tested only to 4 layers
Hold later layers until earlier ones catch up Clear vanishing gradients
(i.e., Post-LN)
Holds, with an observable symptom

The question isn't “which is right” but “under what premise.” Pre-LN or Post-LN decides which applies. Modern LLMs are almost all Pre-LN (§6.5), so the second is more relevant; but on a Post-LN model showing “loss flat, then NaN,” it's the third.

📌 Source: Su Jianlin, “Notes on model optimization: why is BERT's initial standard deviation 0.02?” (in Chinese), kexue.fm/archives/8747. Both quotes are his (our translation). It is his own analysis, not peer-reviewed; we take it as a premised explanation, not a settled result.

An ironic confession: 2.6 set “re-verify across scales,” then I broke it here, writing “they disproved it” with no scale. Rules are easy to write, hard to apply to your own citations.

How long? GPT-3: the first 375M tokens (0.125% of 300B); V3: the first 2,000 steps. Too short to count as a hyperparameter.

Phase 2 · How high to peak: copy a ready-made table

GPT-3's Table 2.1 lists peak learning rate and batch for eight sizes: a free prior, no sweep:

📌 GPT-3 family (same data and recipe; only size varies):

  • 125M → 6.0×10⁻⁴, batch 0.5M tokens | 350M → 3.0×10⁻⁴, batch 0.5M | 760M → 2.5×10⁻⁴, batch 0.5M
  • 1.3B → 2.0×10⁻⁴, batch 1M | 2.7B → 1.6×10⁻⁴, batch 1M
  • 6.7B → 1.2×10⁻⁴, batch 2M | 13B → 1.0×10⁻⁴, batch 2M
  • 175B → 0.6×10⁻⁴, batch 3.2M

The pattern: bigger model, smaller learning rate, larger batch. Mildly: 1,400× in size moves the learning rate only 10× (6.0e-4 at 125M → 0.6e-4 at 175B).

The paper: “larger models can typically use a larger batch size, but require a smaller learning rate.” They set batch by the gradient noise scale.

Batch and learning rate change together here (see the section's end). GPT-3 also ramps batch linearly from 32K tokens to target over the first 4–12B tokens. 📌 Source: arXiv 2005.14165 Table 2.1, Appendix B; the smaller-LR line is §2.3.

“Bigger batch, higher learning rate” has a ceiling many people miss.

First, the two common rules:

  • Square-root scaling: n× batch, √n× learning rate. Goal: constant update noise (update covariance ∝ η²/B, so η ∝ √B).
  • OpenAI, via gradient noise scale Bnoise: η* ≈ ηmax /(1 + Bnoise/B). Monotonic and bounded: however big the batch, LR only approaches ηmax.

So “double the batch, double the learning rate” is wrong.

Stranger still: under Adam, past some batch size the optimal learning rate may fall (the surge phenomenon). Cause, premise, dispute: in the fold below. Main line: don't blindly extrapolate any scaling law.

The surge phenomenon: cause, premise, dispute A disputed, advanced result (Su Jianlin criticizes the paper's approximation). Main text needs only “don't blindly extrapolate”; safe to skip

Under Adam, stranger still: past some batch size, the optimal learning rate falls, not rises: the surge phenomenon.

⛔ Why would that happen? Intuitively: adaptive learning rates are themselves suboptimal.

Treat Adam roughly as sign(g): the larger the batch, the closer sign(g̃) gets to the true sign(g). But is sign(g) the best direction? Not necessarily, especially late in training.
At a moderate batch, a little noise corrects that for you; enlarge the batch, the noise goes, and so does the correction: lower the learning rate more cautiously.

But this has a premise; the author deliberately added a caveat:

“(Of course, one more restriction: β is always below 1. If βnoise ≥ 1, the optimal learning rate still rises monotonically with batch size.)”

Su Jianlin faults the paper: its “the surge almost always appears” rests on a strong approximation, not very scientific; his more reasonable one gives “even if the Hessian's off-diagonal terms aren't negligible, the surge need not appear”.

📌 Su Jianlin, “How should the learning rate change as batch size grows?” (in Chinese) kexue.fm/archives/10542; on “Surge Phenomenon in Optimal Learning Rate and Batch Size Scaling.” Quotes: his words (our translation). Same caveat (his own analysis, not peer-reviewed): we take the warning “don't blindly extrapolate,” not a formula.

An apparent modern outlier: DeepSeek-V3, a 671B MoE, peak 2.2×10⁻⁴: far larger than GPT-3 175B, yet nearly 4× the learning rate.

⛔ My first explanation, which the table refuted: “V3 activates only 37B per token, so treat it as 37B.” But 13B is 1.0×10⁻⁴ and 175B 0.6×10⁻⁴, so 37B should sit near 0.8; V3 is 2.2. The new yardstick only shrank the gap from 3.7× to 2.8×.

The real answer: these numbers aren't comparable.

  • Batch sizes differ by leagues. GPT-3 175B: 3.2M tokens; V3: 12.6M ramping to 62.9M, 20× apart. And learning rate and batch move together.
  • Shapes differ. GPT-3's cosine decays early; V3's WSD holds its peak through 10T tokens. “Peak” means different things on the two curves.
  • Even the parameters differ. Most MoE parameters see few tokens per step: an order of magnitude fewer updates than dense.

Rule: a trend holds only inside the controlled experiment that made it. GPT-3's table is a prior because its eight sizes share one recipe, one knob moving, which is also why it's no prior outside it.
Confession: I first papered over it with “active size, not total.” The most dangerous claim: sounds like common sense, happens to fit.

Phase 3 · How to decay: two schools, same endpoint

Old school · cosine (GPT-3 / Llama / Mistral): cosine decay to 10% of peak over 260B tokens, then held there.

New school · warmup-stable-decay (DeepSeek-V3 has this shape)

V3's curve: 2,000 steps of linear rise to 2.2×10⁻⁴ → held constant until 10T tokens before decaying → 4.3T tokens of cosine decay to 2.2×10⁻⁵ → of the final 500B tokens, the first 333B stay at 2.2×10⁻⁵ and the last 167B switch to 7.3×10⁻⁶.

📌 arXiv 2412.19437 §4.2; clipping norm 1.0; batch 3,072 → 15,360 in the first 469B tokens, then held. A systematic analysis: arXiv 2410.05192.

The schools differ not in convergence but in when each forces you to decide.

⭐ The choice: “can I say now how many tokens I'll train?” Yes: cosine is easier. No: the plateau is your escape hatch.

Both schools end on the same number: the dashed line across the figure.

Takeaway: three tuning rules

  • ① Don't guess. Start from a public config close in size and architecture; the table above and V3's report are a free prior.
  • ② Don't copy across scales. Per 3.3b, LR has units: a small-scale value needs a transferable parameterization or a quick re-check at target scale.
  • ③ Judge by two curves. Loss blows up or grad norm spikes: too high; loss crawls, grad norm on the floor: too low.
    “Slow” has an absolute scale (the ruler from 1.0): falling from 11.8 is a normal start; near 2, every 0.01 is expensive.

Gradient clipping is a seatbelt, not a knob. V3 uses 1.0. Clipping to tame a too-high learning rate masks the problem, not solves it.

⛔ Easiest to overlook: a learning rate alone means nothing.

Report it with batch size, schedule shape, and total tokens; change one and it no longer applies. The two configs above are copyable because they spell out all four.

3.5b How big a batch? Why bigger is steadier, and too big stops helping

3.5 ended: “a learning rate alone means nothing; report it with batch.” So what sets the batch?

Step 1 · Unpack “batch”: four knobs multiplied

“Batch” as one number misleads. A real run has four independent knobs:

KnobWhat it isRaising it
micro-batch
m
Sequences per device per forward Activations grow (1.7's table ×m)
Grad accum
K
Passes per update (1.8, stage ④) Activations unchanged: only one set ever live
DP degree
d
Replicas, each fed its own data Activations thin out; each replica holds all 16 B resident
Sequence length
S
Lecture baseline: 4,096 Activations ∝ total tokens (5.1)

global batch (tokens seen per update) = m × K × d × S

⛔ The link that's easiest to miss: raise TP and the global batch shrinks.

Devices are fixed, so every model-splitting dimension competes with DP:

d = total devices ÷ (TP × PP × CP)

Raise TP from 8 to 16 and the DP degree halves; all else unchanged, so does the global batch.

The formula shouldn't contain EP (I corrected this once). Megatron-Core has two DP sizes: data_parallel_size = world_size ÷ (TP × PP × CP) and expert_data_parallel_size = world_size ÷ (ETP × EP × PP). EP divides only the second: MoE reshuffles the same tokens across experts; the step sees no fewer. CP does sit in the denominator (missing before). 📌 megatron/core/parallel_state.py: model_size = TP × PP × CP × gtp_remat.

Batch changes, learning rate follows: the parallelism knob is wired to convergence. It only looks like model splitting.

“Same code, same data, new parallelism config, different loss curve”: this is why.

Step 2 · Why a bigger batch is steadier

The gradient, like the loss, averages each sample's opinion (1.0).

Recall the “averaging a roomful of people” figure (1.2).

8 people in the room, one extremist: he drags the average far.
800 people: his influence is diluted to almost nothing.

Gradient “jitter” shrinks roughly as 1/√B: 4× the batch, half the noise.

📐 Sample-mean result (standard error ∝ 1/√n, independent samples). It assumes rough independence; adjacent tokens aren't, so effective samples are fewer than tokens. Order of magnitude only.

Conversely, a small batch is pushed around by noise every step: fuzzy loss, jumpy grad norm; many Section 6 spikes are lit this way by a few bad samples.

Step 3 · Why not keep growing it?

If bigger is steadier, why not enormous? The next two points sound separate but are one.

Side one · Noise falls, but so does “payoff per sample.”

While the gradient is rough, more samples sharpen it: 2× batch ≈ half the steps. Once accurate, noise isn't the bottleneck: steps barely drop, compute doubles.

The turning point is the “critical batch size”; past it, compute buys sub-proportional progress.

It isn't fixed: it grows with model size and training progress. So real configs ramp the batch: V3 from 3,072 sequences to 15,360 (last subsection's table). The ramp follows the critical point up; not for speed.

❓ Does a ramping batch mean adding devices? No. What ramps is gradient accumulation.

Recall Step 1's product: global batch = micro-batch × grad accumulation K × DP degree × seq length. For 5×, raise K from 1 to 5. Not one device added.

⭐ Nor does memory grow: per the accumulation row, “activations unchanged: only one set ever live.” Accumulation just postpones the update.

So what's the cost? Same devices, same tokens per second; only the tokens accumulated per update change. Fewer updates, same total time.

⭐ Ramping costs neither devices nor time; it costs update frequency. That's why it pays: early, the critical batch is small and big batches waste; later it has grown, so fewer updates cost almost no sample efficiency, buying fewer all-reduces (smaller communication share) and less noise, more stability.

📌 Three sources for batch ramp-up: mechanism, practice, framework.

  • Mechanism: McCandlish et al., An Empirical Model of Large-Batch Training, arXiv 1812.06162 (OpenAI, 2018). The abstract: “We find that the noise scale increases as the loss decreases over a training run”; the theory offers “a rough model of the benefits of adaptive batch-size training.” Source of the growing critical batch.
  • Practice: DeepSeek-V3 (arXiv 2412.19437): “We employ a batch size scheduling strategy, where the batch size is gradually increased from 3072 to 15360 in the training of the first 469B tokens, and then keeps 15360 in the remaining training.” Yet V3 kept the same 2,048 GPUs.
  • Framework: Megatron-LM has a first-class flag --rampup-batch-size <start batch size> <batch size increment> <ramp-up samples>. Help-text example: 16 to 1024, linear, in 126 increments. It also wants --global-batch-size (the target), and never asks your device count.

⚠️ Assumptions: the V3 sentence is from the v2 main text (near §4.2); the Megatron flag lives in core_r0.9.0's arguments.py; on main it's now --step-batch-size-schedule: same mechanism, new name.

Side two · A bigger batch needs a higher LR, which has a ceiling.

Less noise warrants bigger steps, or the accuracy is wasted. 3.5's eight sizes look opposite (bigger batch, smaller LR) because scale pushes it down too.

But the learning rate can't rise forever: too far and the loss blows up (Section 6). The batch speedup is capped by that ceiling.

⛔ Together, the two sides are one constraint.

Go back to the formula from 3.5:

η* ≈ ηmax ⁄ (1 + Bnoise/B)

Side one's critical batch, quantified, is Bnoise (3.5's “they set batch size using the gradient noise scale”). Side two's ceiling has the same Bnoise in its denominator.

So both turning points coincide, with an easy anchor: at B = Bnoise, η* = half of ηmax: at the critical batch you've spent half your learning-rate budget. More batch only creeps toward the rest, never reaching it.

This box once said “two independent reasons.” Wrong. “Independent” suggests dodging one, say by raising the ceiling a bit. You can't: ηmax is the divergence threshold (3.3b's “2 ÷ curvature”), not a knob.

The two “contradictory” claims are segments of one curve.

Bigger batch
Stability ✅ Monotonic: noise ∝ 1/√B; no turning point
Sample efficiency Has a turning point: past the critical batch, returns diminish

Both right, one row each. Choosing a batch means sitting between: steady enough, not yet wasteful.

Step 3b · Checking a third folk claim

Above: “bigger is steadier,” “too big stops helping.” The most widespread third: “bigger is faster”.

“A bigger batch is faster”: it has a premise nobody writes down This panel shows not the conclusion but where the conclusion applies Left: slack Right: full Comms: no kink Ⓐ x: batch, y: time per update — the line bends; it isn't straight Most accounts draw only the left half batch → Time per update saturates here Nearly flat 2× batch, ~same time ∝ batch 2× batch, 2× time devices not full devices full Left: a bigger batch is free (fewer steps, none costlier); right: not free (fewer steps, each costlier). Ⓑ Same claim, two segments: true in one, false in the other And the tutorial line “bigger batch is faster” is nearly always left Left: not saturated “Fixed data: big batch is faster” True: fewer steps, none costlier Mostly here: small models, 1 device, teaching demos Right: saturated Same claim False: step-cost rise cancels step-count drop Nearly all here: tens of billions of params, hundreds of devices Ⓒ But communication ignores the segments: it doesn't depend on batch So “bigger batch, thinner spread” always holds for comms Gradients synced per step = params × bytes/param 1 sequence or 1,000: same traffic → So it can only be amortized 2× batch → half comms per token Holds in both segments What to keep: not “is a big batch fast,” but how to read “it depends” Compute and comms differ: compute has two phases (free, then not); comms has one (always: thinner is better). So batch size isn't a single number; it depends on which segment you're in. Rule (for ourselves): borrow someone's explanation with its premise attached. Moving only the conclusion is a trap this lecture already fell into.
The line bends: that kink is the whole point.
Order: find the kink in Ⓐ, compare sides in Ⓑ, then see in Ⓒ why communication differs.
⚠️ No measured numbers, only shapes. The kink depends on your model, devices and parallelism config; measure it.
Sources and assumptions

📌 “With parallelism, small and big batches take similar time per step; per epoch, reversed”: Hung-yi Lee, 2021 batch/momentum lecture, pt. 2 (Chinese; our translation). ⛔ Our redraw.

⚠️ No measured numbers, only shapes. The kink moves with model, devices, parallelism; measure on your target config.

This claim has a premise, and it breaks at your scale.

Left: true. Unsaturated devices: fewer steps, none costlier. Really faster.
⛔ Right: false. Saturated devices: costlier steps exactly cancel fewer steps.

Tens of billions of parameters on hundreds of devices sits almost all on the right. For you, a bigger batch buys stability and thinner communication, not speed.

The math for “thinner communication” is in 1.8: cross-device reduction doesn't grow with batch; compute does, linearly, so it thins out.

Step 4 · Two details that bite silently

⛔ ① Forget to divide by K when accumulating = K× the learning rate.

1.8's conclusion assumes you actually took the average.

Sum K micro-batch gradients, then divide by K. Forget it and the gradient is K× too large, same as K× the learning rate.

The symptom misleads: no errors, loss falls at first, then suddenly blows up, and you suspect data or initialization. It's a learning-rate problem in disguise.

② With variable lengths, per-token and per-sequence averages differ.

One batch may mix 4,096-token and 500-token sequences (padding excluded).

  • Per-token average: every valid token counts equally; long sequences, with more tokens, speak louder.
  • Per-sequence average: average within each sequence, then across; equal say per sequence, so short-sequence tokens weigh more.

They give different gradients, one line of code apart. Pretraining almost always averages per token; the point is knowing which you picked.

Left out: an acronym we couldn't make out Author's note for the record; safe to skip

The live session mentioned something like “TPS”: “when TP is too large, TPS gets very large.” But by Step 1's formula, raising TP lowers DP and shrinks the global batch; of three readings (tokens per step / TP size / tokens per sequence), none fits, so no guess; skeleton only. Rule: don't complete an acronym you couldn't hear; a wrong guess teaches a relation that doesn't exist.

Takeaway: batch size is capped by sample efficiency, not memory. Memory decides whether it fits, not whether it should.

3.6 The other two knobs: what β₁ and β₂ do

Two numbers remain: β₁ and β₂. Usually left at defaults, yet their meaning is clear.

In one line: β sets how long old terms live.

A moving average scales by β and adds the new value: window ~1/(1−β) steps.

  • β₁ = 0.9 → momentum window ~10 steps, enough to smooth directional jitter.
  • β₂ = 0.95 → about 20 steps; β₂ = 0.99 → about 100 steps.

Higher β₂: steadier denominator, slower reaction; lower: tighter tracking, more noise. No magic: a memory-length knob.

PaLM's twist: β₂ changes with the step

It uses β₂ = 1 − k−0.8, where k is the current step.

Plug in a few numbers:

The window grows with training (roughly as k0.8). A fixed β₂ = 0.99 remembers only 100 steps throughout.

❓ Why? The paper's reason is concrete.

The culprit: rare embedding tokens, which may appear once in thousands of steps. In a 100-step window their second-moment estimate is garbage: mostly zero, occasionally large.

A window that grows with training fixes it. The paper: “We have found this to be more stable than the standard β2 = 0.99.”

📌 arXiv 2204.02311, hyperparameters; also β₁ = 0.9, LR 10⁻² for the first 10,000 steps, then 1/√k decay.

⭐ Pause: you've seen this reason once already.

AdaGrad (2011) was built for exactly 3.3b's reason: rare features don't get learned.

PaLM (2022), eleven years later via another door (β₂'s memory window), hit the same problem: rare tokens are too sparse for short-window statistics to be trusted.

One gives bigger steps, the other longer memory: two doors to one problem.

⛔ But they differ: AdaGrad changes the denominator's size (bigger steps for rare parameters); PaLM, its sampling window (so noise doesn't dominate their denominators). Same disease, different medicine.

This collides with the rule in 3.2.

3.2 said: momentum and second moment are moving averages, old contributions decay, so bf16 copes.
But at β₂ = 0.9999 the “decay” barely decays; v approaches a true accumulator.

So that rule needs a premise: β doesn't approach 1 over training. Our inference, not the paper's: PaLM uses an Adafactor-family optimizer and reports no precision config, so there's no evidence the effect shows up. A conjecture.

One more find: GPT-3's hand-tuned table, automated by PaLM

PaLM runs unfactorized Adafactor, per the paper “effectively equivalent to Adam with ‘parameter scaling’”: LR scaled by matrix RMS.

The tie to 3.5: “the effect of this is similar to the manual scaling down of Adam learning rate as in Brown et al. (2020)” (GPT-3).

3.5's eight configs are the manual fix; parameter scaling the automatic one. Bonus, per the paper: parameters with inherently different scales (embeddings, layer-norm scales) aren't scaled down at the same rate.

What a checkpoint stores; why 12 bytes Storage, not one of the three things

3.7 A checkpoint stores exactly those 12 bytes

This follows directly from the bill in 3.1; no documentation needed.

Of those 16 bytes, which must go into the checkpoint? Ask, slot by slot, “if it's lost, can it be rebuilt?”

  • bf16 weights (2 bytes): no need to store; one cast of the fp32 master rebuilds them.
  • bf16 gradients (2 bytes): no need to store. The next step recomputes them.
  • fp32 master weights (4 bytes): must be stored, and not just as bf16. Saving as bf16 throws away all the low-order bits accumulated so far.
  • Momentum m, second moment v (4 B each): must be stored. They're history: lost, they reset to zero.

So checkpoint size = those 12 bytes × parameter count. For 671B: about 7.3 TiB per copy.

But don't pin this on V3: per 3.2 both its moments are bf16: 4+2+2 = 8 bytes, 4.9 TiB per copy, a third below the classic count. Whatever saves optimizer state saves checkpoint size too.

Two engineering facts follow:

  • Why checkpoints are so big and slow to save. They store mostly the optimizer; the model is only a third.
  • Why released weights are so much smaller than a checkpoint. A release needs only the 2-byte inference weights; the checkpoint's 12 bytes are all for training: one to six.

⛔ These 12 bytes aren't enough. Miss any of three more things and “resume” becomes “restart.”

  • ① Step count: the learning rate, β₂ and weight decay may all be functions of the step (3.5, 3.6).
  • ② Data-reader position: easiest to miss, hardest to spot: resuming retrains the same batches while the loss curve looks perfectly normal.
  • ③ RNG state: needed for dropout and data shuffling.

The “roll back + skip data” rescue in §6.2 depends on ① and ②. Without the data position, you can't say which batches to skip.

A real saving: the optimizer part needn't live on the accelerator. DeepSeek-V3 keeps an exponential moving average of the weights in CPU memory, updated async each step (no HBM, no training time), to preview quality after LR decay. 📌 arXiv 2412.19437 §3.2.3.

3.8 LoRA: the training most people actually do

So far we trained from scratch. Most people fine-tune someone else's trained model.

No new concepts: the same 16-byte budget, redone for a different algorithm.

The authors' reasoning first, or the “skinny matrices” look like a mere trick

The pain: you can't store a 175B copy per customer.

The abstract opens with it: fully fine-tuning GPT-3 175B means a separate 175B-parameter instance per downstream task, “prohibitively expensive.”

⭐ The pain is in deployment, not in training. LoRA was built for “one base, hundreds of small add-ons”; saving training memory is a bonus.

Then the observation that prompted it.

Earlier work found that large models have huge parameter counts but a far lower effective dimension: the parameters sit in a thin subspace.

⭐ LoRA's leap of faith went one step further: if the model is low-dimensional, is the change from fine-tuning low-dimensional too?

A hypothesis, not a theorem. If it holds, the method writes itself: a low-rank change doesn't need a full-rank matrix; two skinny matrices multiplied suffice.

⭐ The shape of the reasoning: they didn't prove the change is low-rank. They built on the hypothesis, then measured it (the paper's “empirical investigation into rank-deficiency in language model adaptation”). Bet first, verify second.

📌 Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, arXiv 2106.09685. The abstract: 10,000× fewer trainable parameters, 3× less GPU memory, and “unlike adapters, no additional inference latency.” That last point is how it beat earlier methods.

📌 The method in one sentence: freeze the base and attach a pair of skinny matrices beside it.

The 4096 × 4096 matrix stays frozen. Add a 4096 × 16 and a 16 × 4096; train only those.

The key is the 16: 2 × 4096 × 16 values, just 0.8% of the original.

What happens to the three budgets

BudgetFull fine-tuningLoRASaved?
Resident
weights + grads + optimizer
16 B × all params 2 B × all (frozen, read-only)
+ 16 B × the 0.1%
Biggest saving
Compute3× (fwd 1 + bwd 2) ≈ 2×A third
Activationswhole chain still whole chainBarely

Row one at 7B: dramatic. (Llama-2-7B class: 32 layers, hidden 4,096; 32 × 2 sites (Q, V) × 2 matrices (A, B) × 4,096 × 16 = 8,388,608.)

  • Full fine-tuning: 6.74e9 × 16 B = 100.4 GiB.
  • LoRA (a pair on Q and on V per layer, rank 16): frozen base 12.55 GiB + adapter 0.125 GiB = 12.68 GiB.

A 7.9× difference, from an adapter of only 8.39M parameters, 0.124% of the model.

Why so large: 14 of the 16 bytes (gradient + master weight + m + v) serve only updated parameters. A frozen one keeps just its 2-byte weight, as at inference.

Row two, “3× → 2×,” comes from 1.3.

Per 1.3 (① is forward), backward costs two matmuls: ② the weight gradient (for the optimizer), ③ the input gradient (passed down).

Frozen layers skip ② entirely: their weights never change, so nobody needs it.

Forward (1) plus ③ (1) gives about 2×, not 3×.

Don't confuse the reverse: tuning just the last layer saves ③ (nothing flows back). LoRA has adapters in every layer, so ③ remains; it saves ②.

Row three: most misunderstood, most valuable.

“Training 0.1% of parameters = light training”? Not in memory.

Adapters sit on every layer. The first layer's adapter needs a gradient, which only flows back from the loss (1.5). So the whole chain's activations stay.

After LoRA, activations are often the largest item, so it still needs recomputation (Section 2). The two stack; neither replaces the other.

Strictly, activations shrink a little, depending on where adapters sit. Rarely spelled out:

  • A fully frozen linear layer: its input can go: only ② in 1.3 (weight gradient) needs it, and ② is skipped.
  • An adapter pins the input back: A reads the same input, and A's gradient needs it.

Counterintuitive corollary: the more adapters, the less activation memory saved. “Q and V only” saves a little: none of the three big MLP matrices has one. “More adapters work better” costs activations, not parameters.

Nonlinear ops' inputs still stay, so the total shrinks a little, not by an order of magnitude. Derived from 1.3's two matmuls; not measured.

⭐⭐⭐ Wrap-up: LoRA changes how many parameters carry 16 bytes, not that backward traverses the whole network. The parameter side shrinks; the chain doesn't: it comes from backprop, not from how many parameters you train.

Section 4

The bill goes to Topic 5 (not covered here)

A bridge to Topic 5. Rank the items by size: ZeRO's three stages follow that order.

A Topic 5 preview, not one of our three things. Three sentences; details collapsed below.

What training needs, by size:

Optimizer state (12 B/param) > weights = gradients (2 B each) > activations (batch-dependent)

  • When it doesn't fit, shard in this order: largest, least-used first.
  • That is ZeRO's three stages: optimizer state, then gradients, weights last.
  • How, onto whom, at what communication cost: Topic 5's job.

Takeaway: parallelism isn't invented; this bill's size ranking forces it.

The bill: the four items' sizes, how ZeRO's stages follow, what each other technique shards Topic 5 preview, not required

4.1 First, rank the bill by size

Put the items from the last three sections side by side and rank them by size. That ranking is this whole section.

Item/param671BShareWhat it isWhen used
Optimizer state12 B7.32 TiB75%fp32 master 4, moments m 4, v 4Once, at the update instant
Weights2 B1.22 TiB12.5%bf16 copy; both passes use itRead by every layer
Gradients2 B1.22 TiB12.5%bf16, shaped like the weightsWritten in backward; dropped after update
Resident total16 B9.76 TiB100%⭐ Independent of batch and seq. length: the model fixes it.
Activations—By batch—Saved for backward (1.6)⛔ Only one you control; only one that can outgrow the rest

The first three rows are bytes per parameter × parameter count, so the ratio is always 12 : 2 : 2, whatever the model or device count.

Optimizer state alone is 75% of the resident total, yet the only one neither pass uses: 3.1 showed it is read and written once, at the update instant.

⛔ The fourth row is different: it is the only one you control.

The first three follow the parameter count; no choice. Activations follow the tokens live at once this step: micro-batch × grad accum × DP degree × seq. length, four knobs multiplied.

So “which item is largest” is a property of the training configuration, not of the model. Push the batch far enough and the fourth row passes the first; 5.1 finds the crossover.

Remember this order: next, ZeRO works down it one stage at a time.

4.2 ZeRO's three stages follow exactly this order

No need to memorize ZeRO's stages: reverse the order above and you've derived them.

ZeRO's three stages — one bar, cut three times No need to memorize the order: shard the largest, least-used item first Weights Grads Optimizer state Ⓐ Bar length = what each device must actually hold For 7.5B params, 64-way data parallel, the setting of the paper's Figure 1 Baseline · pure data parallel full copy on every device Weights Gradients Optimizer state 120.0 GB per device 2Ψ — ZeRO-1 · shard optimizer state Pos Weights Gradients sharded away 31.4 GB per device 2Ψ same as baseline ZeRO-2 · then gradients Pos+g Weights sharded away 16.6 GB per device 2Ψ same as baseline ZeRO-3 · weights last Pos+g+p ≈ FSDP sharded away 1.9 GB per device 3Ψ 1.5× Memory held Comm. volume All bars use one scale, so lengths compare directly The four numbers are computed, not copied: a script asserts them against the paper's Figure 1 and fails the build on mismatch. Compute, don't copy. Ⓑ The order is no historical accident — it falls out of size × frequency of use So the stages can't be reordered, and needn't be memorized Uses per step → ↑ Bytes per param 1 10 100 2 B 12 B 1 · Optimizer state used once per step comm. unchanged 2 · Grads reduced once after backward comm. unchanged 3 · Weights needed in every layer's forward comm. ×1.5 Empty corner: big and busy Don't memorize: follow “big first, busy last” and out come ZeRO's three stages. Beyond ZeRO too: move the big, idle item first, the small, busy one last. Bonus: unpack the baseline's 2Ψ — it shows why sharding optimizer state is free Pure DP's per-step all-reduce is really two steps: reduce-scatter (Ψ) reduces and scatters the gradients, then all-gather (Ψ) collects them. Total 2Ψ. ZeRO-1 needs exactly that scattered intermediate: each device updates only its own shard. No extra traffic; it just keeps the intermediate.
Ⓐ is one bar cut three times. Dashed boxes are what was sharded away; the GB figure on the right is what each device still has to hold.
Ⓑ gives the order: size × frequency of use. Optimizer state: largest, used only at the update, so cut first.
Compute, don't copy: a copied number that's off may be your typo or theirs; a computed one fails on the spot.
Sources and assumptions

📌 arXiv 1910.02054 §3 (16-byte convention, Figure 1's four numbers), §7.2 (comm volume 2Ψ / 2Ψ / 3Ψ).

⭐ The four GB figures are computed (7.5B × bytes/param), script-asserted vs. the paper's Figure 1 (120 / 31.4 / 16.6 / 1.9 GB).

One rule sets the order: shard the largest, least-used item first.

  • ZeRO-1 shards optimizer state. The largest, used only once, at the update, so sharding it adds almost no communication. Best value.
  • ZeRO-2 then shards gradients. Weight-sized, but they can stay scattered until backward finishes.
  • ZeRO-3 (≈ FSDP) shards the parameters last. Every layer's forward needs them, so once sharded each layer must all-gather them first: the most communication of the three.

So the order is no historical accident: it falls out of size and frequency of use.

📌 Source: Rajbhandari et al., “ZeRO: Memory Optimizations Toward Training Trillion Parameter Models,” arXiv 1910.02054. Section 3's “16 bytes per parameter” comes from it too: one paper gave both the bill and how to split it.

4.3 Same bill: what each other technique shards

Place the other techniques you keep hearing about on this bill and each one's place becomes obvious:

  • Recomputation (Section 2)—targets activations.
  • offload—targets whatever is not needed right now, slicing by stretch of time, not by item.
  • CP (context parallelism)—shards activations that grow with seq. length, overtaking optimizer state at long context.

⭐⭐⭐ These are not “a few optimization tricks” but countermeasures for four different columns of the same bill. All of Topic 5 is about what the sharding itself costs.

Section 5

All three, on a machine at hand

All correct, but all at 671B parameters and TiB scale, with no reference point. Now shrink it to a model you can work out yourself.

Forward (Topic 1), backward and update in one table. The useful question isn't “how much” but “when is the peak?”

One step, all items — before “when is the peak,” ask “the peak of what” These are global totals, not per device · x-axis marks phases, not true time proportions Weights · flat Optimizer · flat Activations · fwd Gradients · bwd Ⓐ Two fixed bands, plus two with opposite shapes The four layers below are drawn to true scale (global batch = 1 seq): that's why optimizer state is thickest Start End of fwd End of bwd Updated Timeline of one step Weights 1.22 TiB Optimizer state 7.32 TiB Never moves, largest of the four — yet used once, at the very end Gradients 1.22 TiB + activations 162.14 GiB — really this thin ↑ The two layers above magnified 20×; at true scale, too thin to show the peaks Activation peak fwd just done, bwd not started Gradient peak bwd just done, not yet updated The two peaks are at different moments Ask “peak of what” before “at what moment” Ⓑ “Are activations or optimizer state bigger?” — no fixed answer Because the resident block doesn't change with batch, while activations grow linearly with batch Resident (batch-independent) 9.76 TiB weights 2 + grads 2 + optimizer 12 = 16 B/param, × 671B Activations · 1 seq live 0.16 TiB One 128K sequence, full recomputation Activations · 62 seqs live 9.76 TiB Only now do they catch up — live at once = DP degree × micro-batch So “largest” is a property not of the model but of this training config — change the global batch, change the answer.
The closing figure: the four sections' pieces on one timeline at last.
Ⓐ Opposite shapes: blue piles up in forward, orange in backward; highs over half a step apart.
Ⓑ All three bars are computed live, asserted: change an input, a mismatch fails.
📌 What grad accumulation saves and doesn't: all five points in 1.8.
Sources and assumptions

⭐ Ⓑ's three rows are computed live: 671e9 × 16 B ÷ 1024⁴ = 9.76 TiB; 9.76 TiB × 1024 ÷ 162.14 GiB = 61.7 sequences. Script-asserted: a changed number fails

⚠️ 16 B/param is the classic ZeRO-paper convention (arXiv 1910.02054); DeepSeek-V3 keeps both moments in bf16, so its per-param bill differs (3.2)

⚠️ 162.14 GiB of activations: our own per-op estimate, unchecked by others; “62 sequences” is a magnitude, not a threshold

⛔ The figure shows global totals, with no sharding. Spreading it over devices is Topic 5

The same bill, in motion. Above, the mountain's shape; here, how it builds and collapses.
■ Resident (weights + optimizer state, 14 B/param): never moves | ■ Activations: built in forward, freed in backward | ■ Gradients (2 B/param): appear in backward, full at its end | Bottom row: 61 layers, lit as reached
⭐⭐ Watch the vertical line: without recomputation, the peak is when forward ends, before backward starts (with recomputation, it moves to backward's first instant: 61 layer inputs + the layer being recomputed; see 1.6's 🔍 orange inset). Not a phase; an instant. (True byte counts, same basis as above. 14.6 s silent loop, Manim; script in tools/manim/.)
Full step bill: four-item total, why 6ND underestimates, when it peaks “Does it fit?”: Topic 5 preview, optional

5.1 Memory: four items added

Four sections, four separate accounts; now add them up. Global totals, no per-device split (next lecture's job).

📌 The resident block fits in your head:

671e9 params × 16 bytes ÷ 1024⁴ = 9.76 TiB. Broken down: weights 1.22 + gradients 1.22 + optimizer state 7.32.

For scale: at 80 GiB per device, this block alone needs 125 devices, before activations or any redundancy. The activation peak below assumes full recompute; per 2.4, V3 itself recomputes selectively (RMSNorm + MLA up-projections). Different bases: full is a clean floor; reality runs higher. (80 GiB: just a round base, not a product. Topic 1 uses v7's 94.74 GiB/device: about 105 devices. Same TiB, other divisor.)

Of these three, only the optimizer state's 7.32 TiB is pure overhead: no part in forward or backward, used once at the end of each step.

Follow that number one step further: a surprise.

The resident 9.76 TiB is fixed, whatever the sequences. How large must activations grow to catch up?

About 8.08M tokens. At this lecture's 4K baseline, about 1,973 sequences. (Full recomputation: one 4K sequence peaks at 5.07 GiB; one 128K sequence (context extension), 162.14 GiB: about 62 sequences, same token count.)

Stop: this number is easy to plug in wrong. I did, in one draft.

3.5 says V3's global batch starts at 3,072 sequences of 4K (= 12.58M tokens) and later ramps to 15,360. That looks far past the 8.08M crossover.

Global batch won't do: the crossover counts what's live at once.

global batch = DP degree × micro-batch × grad-accum K × seq length, and the K slices never coexist (point ④, 1.8). Only the first two are live at once, as in panel Ⓑ.

A safe upper bound: V3 used 2,048 devices with PP16 × EP64. Most generous DP, 2048 ÷ 16 = 128-way, micro-batch 1: only 128 sequences = 520K tokens live at once, under a fourteenth of the crossover.

So “activations overtake optimizer state from step one” is false: withdrawn. On V3's own config, 3.1's “the largest block is optimizer state” still holds.

This passage once said “V3 passes the crossover 1.7× from step one, 8.5× late in training” (vs. the old 7.37M-token crossover; today's 8.08M gives 1.56× and 7.8×), as if the example overturned the first answer. That treated global batch as live at once. Yet the figure was right all along: Ⓑ reads “live at once = DP degree × micro-batch.” Rule: when figure and text state one quantity, first check they share a definition. Figure right, text wrong, both self-consistent: no alarm.

One premise that's easy to skip: memory doesn't care how those 8.08M tokens are made up.

  • About 1,973 sequences of 4K or about 62 of 128K: for memory, the same.
  • Why: the activation bill has no S² term: the attention score matrix never lands in memory (the second caveat in 1.7). Every remaining item is “one per token,” so only total tokens count.

⛔ But compute doesn't.

The same 8.08M tokens as long sequences or as short ones need very different compute: attention grows with S², and it is on the compute bill, not the memory bill.

So the sequence-length knob behaves differently on the two bills: memory only sees the total; compute also sees the split. Hence 5.2's “6ND badly underestimates.”

The premise: all this assumes FlashAttention. A naive version storing the score matrix puts S² back on the memory bill; none of the above holds.

5.2 Compute: 6ND badly underestimates long context

The usual training-compute estimate is 6ND: about 6 FLOPs per parameter per token (2 forward, 4 backward: the 1 + 2 from Section 1).

Plugging in (for MoE, active parameters, not total): 6 × 37e9 × 131,072 ≈ 29.1 PFLOP for one 128K sequence.

⛔ But at 128K this number is badly wrong.

6ND counts weight multiplications. The attention steps QKᵀ and AV involve no weights, so the formula ignores them.

Appendix 7.2 puts attention at 82.1% of a layer's forward compute at 128K, so 6ND captures only the remaining 17.9%.

So real compute exceeds 6ND by five to six times: the 29.1 PFLOP is actually close to 160 PFLOP.

This multiplier is derived from the 82.1% (1 ÷ 0.179), our own per-op estimate: magnitude only. Remember not the multiplier but when the formula misleads: long sequences.

Conversely, 6ND works very well for short sequences. At GPT-3's 2,048, attention is just over 1%, negligible. Same lesson: the formula isn't wrong; the premise changed.

5.3 Peak: first ask “the peak of what”

Section 1's mountain figure said the peak is at the end of forward. That holds only for activations.

Draw all four items on one timeline (the figure above) and different items peak at different moments.

Generalized, a rule you can use anywhere:

when you stack curves of different shapes, the peak of the sum need not fall on any single curve's peak. Always ask “crowded with what” along with “when is it most crowded.”

📌 In practice: debugging an OOM, “the peak is xx GiB” is useless; you need which item is largest then. Large activations: recompute or add CP. Large optimizer state: ZeRO. Mix them up and neither helps.

5.4 One step of a 125M-parameter model

The three things on this machine:

  • ① How the loss is computed: 8,192 positions, one −ln p each, averaged. At init it is ln(vocab), 1.0's yardstick; a smaller vocabulary starts a bit lower.
  • ② How many derivatives one number becomes: 125M, for “two more forward passes,” not 125M (1.1).
  • ③ How big a step to take: Adam keeps two moving averages for every parameter: 8 of the 12 bytes. The other 4, fp32 master weights, come from mixed precision, not Adam (3.1).

Below, ③ becomes a number you can check.

📌 Setup: a 125M-parameter model, one device, batch 8 × sequence 1,024.

12 layers, width 768, MLP 3,072: row one of GPT-3's Table 2.1 (top row of 3.5's table).

Step 1 · The resident block

The same 16 bytes as in 3.1:

ItemPer param125M params
bf16 weights2 B0.23 GiB
bf16 gradients2 B0.23 GiB
Optimizer state (fp32 master weights + m + v)12 B1.40 GiB
Subtotal16 B1.86 GiB

First checkable conclusion: weights are 0.23 GiB, training takes 1.86 GiB, eight times more, all of it training-only.

Step 2 · The activations

What each token leaves per layer, item by item (width 768):

Saved tensorWidth
Outputs of the two norms2 × 768 = 1,536
Q / K / V3 × 768 = 2,304
Attention output768
MLP layer-1 output3,072
Activation output3,072
MLP layer-2 output768
Per layer11,520 values = 22.5 KiB (bf16)

× 12 layers = 270 KiB per token. × 8,192 tokens = 2.11 GiB.

⛔ Something surprising happened.

Activations 2.11 GiB > resident 1.86 GiB. At only 125M parameters, activations exceed weights, gradients and optimizer state combined.

No long context: sequence 1,024, batch 8. Recomputation isn't just for big models.

Step 3 · With recomputation

As in 2.2: each layer keeps its input (width 768), plus all of the layer being recomputed.

0.32 GiB: down from 2.11, 6.7× less, for compute 3× → 4×.

The 0.32 GiB has a deliberate omission checkers will hit.

It counts the 12 layers only, not the final logits. GPT-3 reuses GPT-2's BPE vocabulary, about 50.3K:

8,192 tokens × 50,257 × 2 B = 0.77 GiB, 2.4× the 0.32 GiB.

Materialize all logits at once, and the largest item here is neither weights nor activations but the vocabulary score table. Real code computes the loss in chunks, so it's left out (1.7). An accounting choice; the logits still exist.

📌 50.3K, in two hops: GPT-3 paper §2.1: “We use the same model and architecture as GPT-2, including the modified initialization, pre-normalization, and reversible tokenization described therein,” so GPT-2's BPE: 50,257. GPT-3 never prints it. 30K or 60K changes nothing.

Recompute offRecompute on
Resident1.86 GiB1.86 GiB
Activations2.11 GiB0.32 GiB
Total3.97 GiB2.18 GiB
Compute3×4× (+33%)

This is the training loop on your laptop.

  • The 16-byte budget (3.1): gives the 1.86 GiB.
  • Activations grow with total tokens (5.1, this section): the 2.11 GiB's only variable is the 8,192 tokens.
  • Trading compute for memory (2.2): pay 33% more, save 6.7×.

“Pay 33%” matches 671B, but “save 6.7×” is 3× smaller than its 20.5×.

An earlier draft said “the ratios are almost the same as for 671B.” Far from it. Why is the more valuable part.

Why does the same “full recomputation” save 6.7× here and 20.5× there?

Write it as a formula. Let r = one layer's activations ÷ its input, L = layers:

Saving = r ÷ (1 + r/L)

Numerator r: how much a layer holds; fatter layers save more.
Denominator term: the layer being recomputed, amortized over layers; more layers, less impact.

r = activations ÷ inputL = layersSaving
125M small model15 (11,520 ÷ 768)126.7
The rows define r a bit differently: the small model counts only the norms' outputs; V3 (1.7's table) adds their inputs (= residual input). With inputs, small-model r = 16, saving 6.9. Same magnitude and conclusion; strictly two rulers.
This cell once had a hand-typed “r ≈ 16, saving ≈ 7.1.” 16 was right; the formula gives 6.9, not 7.1. Telling: every other number here was script-computed; this one, hand-typed.
V3 671B31.7 (one MoE block ÷ 7,168)6120.5

The small model loses on both: r 2.1× smaller, L 5× smaller. The smaller saving is inevitable, not an error.

The large r is MoE's: in 1.7's table, expert outputs and SwiGLU intermediates are each stored nine times. A fuller layer saves more. MoE saves on parameters, costs on activations, and therefore pays off especially on recomputation. One design, three bills, three directions.

For V3 the formula gives 20.8 vs. 20.5 measured: slightly lower, as the 3 dense layers are thinner than MoE. An order of magnitude, not exact.

Only three things are truly scale-independent.

  • Pay 33% more. It depends only on “forward twice, not once,” not model size or sequence length.
  • 16 bytes per parameter. 125M or 671B, the breakdown is identical (3.1).
  • Activations count only total tokens. 8,192 tokens or 8.08M, same rule.

The saving factor is not on this list. It follows r and L: new architecture, new math. Criterion ③ again: separate what changes from what doesn't.

Same three caveats as 1.7's table: ① derived op by op; fusion shrinks them; read as magnitudes; ② excludes the attention score matrix (FlashAttention assumed), temp, fragmentation and comm buffers; ③ “0.32 GiB” assumes a checkpoint every layer, not always the leanest density (2.2 Ⓒ).

What goes to Topic 5 Handoff to parallelism

5.5 What this bill hands to the next lecture

Topic 1 ended the forward pass with “it doesn't fit.” With the bill complete, that becomes specific: “each of the four items doesn't fit, and each has its own remedy.”

Four items, four remedies:

  • Optimizer state (largest, least used) → shard (ZeRO-1)
  • Gradients (weight-sized; scattered till backward ends) → shard (ZeRO-2)
  • Weights (used by all layers) → shard last (ZeRO-3 / FSDP): costliest comms
  • Activations (∝ batch × seq. length) → recompute, or shard by seq. (CP)

⭐⭐⭐ Three of four lines say one word: shard. That is all of Topic 5: sharding has its own cost, to be accounted for too.

Section 6

Not blowing up: when the loss spikes, and how to prevent it

So far, accounting. It can't answer the most common question on a real run: “The loss just shot up. Now what?”

Not blowing up — think of it as a river, fixed in three stretches This section easily becomes a list of tricks — and you still won't know which to try first Upstream · reshape Midstream · gates Downstream · salvage Ⓐ Picture it as a river — one flood, three places to act Upstream to downstream: higher = root cause, lower = first aid Upstream Reshape Wider, straighter channel — less likely to overflow · Norm inside the residual block (Pre-LN) · QK-norm — keep attention scores sane · Scale init with depth One-off: done before training starts Midstream Floodgates Close the gate when water rises · z-loss — keep pre-softmax logits in check · Grad clipping — global norm, often 1.0 · High precision where it counts Always on: cheap, mostly no quality hit Downstream Salvage Already capsized — haul it back, restart · Roll back to a pre-spike checkpoint · Skip that stretch of data, continue · Log the step and the data position Repeat each time — root cause unfixed Flow → loss spike The further downstream, the more it's firefighting. The upstream fixes are done once, up front; the downstream ones recur with every incident. Ⓑ A PaLM ablation that contradicts everyone's first instinct It cost one retraining run and ruled out a whole class of guesses Test the “bad data” hypothesis Same data + different step ↓ Result: no spike. So: not bad data Needs both at once Either alone: no spike Why rollback + data skip works Ⓒ The real difficulty: stability is easy; stability without losing quality is hard ST-MoE Table 4, raw: baseline 6 seeds, the other two 3 each Baseline -1.755 stable 4/6 2 of 6 runs diverged Tighter update clip -4.206 stable 3/3 Far shorter bar — look here router z-loss -1.741 stable 3/3 Longest bar, zero failures Longer bar = better quality (higher is better). The middle bar, “stable 3/3”, looks perfect, but it wrecked quality.
Ⓐ Ordered by where a fix acts in the chain: a list of tricks doesn't say which to try first.
Ⓑ PaLM's ablation: the same batches fed at another point don't spike. Spikes are data meeting parameter state, not bad data.
⛔ Ⓒ: the middle bar looks perfect (stable 3/3) but wrecked quality.
Sources and assumptions

PaLM 540B spikes, ablation: arXiv 2204.02311 §5.1, all verbatim: “roughly 20 times”, “roughly 100 steps before the spike”, “roughly 200–500 data batches”, “We do not believe that the spikes were caused by ‘bad data’ per se”

Router z-loss, Ⓒ's table: arXiv 2202.08906 (ST-MoE). ⭐ Its own heading: “Many methods stabilize sparse models, but at the expense of worse quality”

QK-norm: arXiv 2302.05442 (ViT-22B) §2 — training diverged at around 8B parameters: huge attention logits collapsed attention weights to near one-hot (near-zero entropy)

Normalization placement: arXiv 2002.04745 (ICML 2020) — proves Post-LN starts with large gradients near the output layer, hence needs warmup; Pre-LN's start well-behaved

⛔ None of our own measurements: all four from papers; Ⓒ's three numbers copied from Table 4

6.0 Naming the failure modes

We've met five failure modes across five sections, each under its own name. Side by side: on a real run, the hard part is diagnosis, not the fix.

NameSymptomRoot cause ConfirmRemedy
Exploding gradients Loss NaN/Inf, or grad norm jumps decades in one step Local derivatives stay above 1; 61 layers compound them expo­nentially (1.4) Global grad norm per step: decade jumps? Clipping (6.4) · Pre-LN (6.5)
Vanishing gradients Input-side layers barely move, as if just initialized Same chain, local derivatives below 1 Per-layer grad norm: decays with depth? Pre-LN and init (6.5)
Rounded-away updates
(“won't train”)
Loss flat despite nonzero gradient; fp32 master weights fix it at once Per-step relative change below mantissa resolution (about 1/256 for bf16) Compare update RMS ÷ param RMS with 1/256 fp32 master weights (3.2)
Step-size diver­gence Loss climbs, or oscillates with growing amplitude η exceeds 2 ÷ sharpest curvature: steps overshoot, amplified along it Halve η and rerun: instantly fine means this LR ceiling and warmup (3.3b · 3.5)
Loss spike
(instability)
Sudden spike in smooth training; heals or collapses Not bad data: there's an ablation (6.2) Rerun same batches: spike again? This section, 6.1⁠–⁠6.6

Rows one and two: one chain of products, two directions.

Per 1.4, a parameter's gradient is 61 local derivatives multiplied. Above 1 on average, it grows exponentially; below 1, it decays. So the fixes are one family: keep each local derivative near 1, as 6.5's structural fixes do.

The middle rows (rounding, divergence) are both called “won't train”, with opposite remedies: bigger steps or more precision vs. smaller steps. Confusing them hurts.

6.1 First, a real case

📏 First, the yardstick from 1.0.

“Blew up” is a feeling; we want judgment. In numbers:

  • Late pretraining, loss hovers around 2: normal.
  • A jump to 6: perplexity 7 → 400. Back to “knows only word frequencies”, but still knows the words.
  • A jump to 11.8, ln 129,280: everything forgotten, uniform guessing.

So spike height tells whether the run can be saved. A jump to 6 usually clears by rolling back a few dozen steps; a jump to ln V wastes that stretch.

⛔ PaLM 540B's loss spiked roughly 20 times.

With gradient clipping on. The spikes came at highly irregular times, sometimes late, and never in the smaller models.

📌 arXiv 2204.02311 §5.1. “Roughly 20 times” and “despite the fact that gradient clipping was enabled” are verbatim.

Loss spikes — what they look like, and no one can say why Curve is a schematic drawn from the paper's text, not measured data — so no y-axis ticks 540B: ~20 spikes Smaller: none seen Clipping on Ⓐ Two curves on one sheet: the difference is obvious x: training steps; y: loss — lower is better, so the trend heads down loss steps → 540B ~20 spikes Smaller model none at all Lower curve has its own y-axis — compare shape, not level Only three claims, all stated in the paper: ① No pattern in timing — some clustered, some far apart ② Can come late — not “shaky start, then fine” ③ Small model is flat — small-scale OK ≠ safe at scale Ⓑ “Safeguard on, still spiked” sounds paradoxical — two different quantities Zoom in: the clipped curve is flattened; the spiking one still spikes ① What clipping controls: gradient norm (log y-axis, spans one decade) clip this part is cut step it wanted flat tops Red line touches the threshold everywhere, never crosses — safeguard worked 100%, no misses ② But what spikes is another quantity: loss (same stretch, same steps; y-axis zoomed to fit) Six points, every one — step size capped, loss not No contradiction Safeguard on “how big a step” Incident in “the loss” It did not fail; it guards something else So why the spikes? Paper: unknown So everything later in §6 is “work around” or “make it rarer”; nothing is a “cure” Knowing a problem is still unsolved matters as much as knowing three fixes — otherwise you'll blame your own config. Rule: saying “this is an open problem” beats glossing over it. Gloss over it, and students hunt for a config bug that doesn't exist.
Ⓐ Compare the shapes: top full of spikes, bottom flat. Bottom: a smaller model, same recipe.
Ⓑ is why this section exists: the safeguard was on; incidents happened anyway. Every remedy below is a workaround, not a cure.
⚠️ Ⓐ has no y-axis ticks, on purpose. No ticks = schematic; ticks = data. The paper describes the shape but gives no numbers.
Sources and assumptions

📌 arXiv 2204.02311 (PaLM) §5.1, verbatim: “roughly 20 times”, “despite the fact that gradient clipping was enabled”, “were not observed when training the smaller models”.

⚠️ The curve is schematic: spike positions were picked to match “no discernible pattern”; the paper gives no heights or widths, so the y-axis has no ticks.

Scale effect: fine when small isn't safe when large. Like “don't copy across scales” in 2.6, but what breaks is training itself, not the payoff.

6.2 The surprising ablation: not bad data

First instinct: bad batch. PaLM tested it cleanly:

They replayed the batches around the spike from an earlier checkpoint. No spike.

Spikes aren't “bad data”: these batches collide with that parameter state.

Same data, other moment: nothing.

Hence their crude fix works: roll back to a checkpoint ~100 steps before the spike, skip those 200–500 batches, resume. It doesn't recur.

Not “fixed”: “worked around”. Given the cost, they “were not able to determine a principled strategy to mitigate these spikes”. Still open.

6.3 The hard part: stable without losing quality

A tiny learning rate or very tight clipping stops spikes. All cost model quality.

ST-MoE measured this. Same config, different seeds: how many runs finish, at what quality:

MethodStabilityQuality ↑Verdict
Baseline4 / 6−1.7552 of 6 runs diverged
This is log-likelihood, not loss: negative, and higher is better (−1.755 beats −4.206). On the 1.0 yardstick: loss 1.755 vs. 4.206, perplexity 5.8 vs. 67. Not the same quantity as loss.
Tighter update clipping (0.1)3 / 3−4.206 Stable, but quality wrecked
router z-loss3 / 3−1.741 Stable, slightly better

The middle row is the point: “stable 3/3” looks perfect, bought with quality. Judge stability techniques on both columns.

Two caveats against over-reading it.

  • Different denominators. Baseline: 6 seeds; each technique: 3 seeds. “4/6” vs. “3/3” is not like-for-like. Three runs are far too few for “guaranteed stable”.
  • The setup was deliberately chosen. Per the authors, small models rarely go unstable; large ones are too costly for enough seeds. So: a config diverging about one run in three, on multilingual data, found to worsen instability, so slightly smaller models suffice.

Not a flaw: good design. To study a rare failure, make it common first. But remember: the failure rate was dialed up.

📌 arXiv 2202.08906 (ST-MoE), Table 4. Its heading: “Many methods stabilize sparse models, but at the expense of worse quality.”

6.4 Fix 1 (numerics): cap what grows too large

z-loss: rein in pre-softmax logits

Simple: a loss term penalizing the squared log of the softmax normalizer. It pushes log Z toward 0, keeping logits small.

Two places, both used:

Why z-loss is cheap insurance: per ST-MoE, logits feed an exponential, extremely sensitive to input error. A smaller input range directly cuts that risk, and the model doesn't need the dynamic range.

Gradient clipping: a seatbelt, not a knob

Clip by global norm: treat all gradients as one vector; if its norm exceeds the threshold, scale it down. 1.0 is usual (PaLM, DeepSeek-V3: both 1.0).

But recall 6.1: PaLM spiked 20 times with clipping on. Clipping stops a one-step explosion, not parameters already drifted somewhere bad.

6.5 Fix 2 (structure): hard to blow up by design

Where the norm goes, which settles 3.5's open question

First, split a layer into three parts to see the difference. A layer is two parallel paths. The trunk carries x unchanged. The branch runs x through attention or the MLP for a correction F(x). They merge into x + F(x): 1.4's residual.

Normalization, part three, must sit on one of these paths. Post-LN and Pre-LN differ only in where.

Two placements

Post-LN xt+1 = Norm( xt + F(xt) )
Merge, then Norm the whole trunk, every layer.

Pre-LN xt+1 = xt + F( Norm(xt) )
Norm moves into the branch. The trunk is untouched, straight to the top.

Neither wins yet. But it's not a matter of style.

A 2020 paper proved it, with a conclusion tied to 3.5:

At init, Post-LN has large gradients near the output, so a large learning rate is bound to be unstable; warmup holds that in check.
Pre-LN's gradients are well-behaved at init, so the paper says Pre-LN can drop warmup.

Recall 3.5's table, third row. Its “precondition” reads “significant vanishing gradients (i.e., Post-LN)”: this exponential decay.
So warmup doesn't have “competing explanations”: the 2020 paper explains why it was once indispensable; 3.5's paper, why it's still around. Different questions.

📌 arXiv 2002.04745 (ICML 2020), via mean-field theory.

That says Post-LN gradients are bad, not why. The mechanism is one sentence and answers a sharper question: if it's bad, why keep it?

⛔ Post-LN is xt+1 = Norm(xt + Ft(xt)).

At init, x and F(x) are roughly independent random vectors, each of variance 1, so their sum has variance 2. Norm pulls it back to 1: effectively ÷ √2.

Recursing, the coefficient of input x₀ in layer l's output is 2−l/2. The residual “shortcut” shrinks exponentially, most near the input.

Su Jianlin's verdict, harsh but well founded: “In a Post Norm BERT model, LN not only fails to mitigate vanishing gradients; it is one of the ‘culprits’ behind them.”

In Post-LN, the norm itself is a culprit of vanishing gradients — yet it stays, for good reason Plotted: forward residual shortcut term — the “so gradients vanish too” step is argued in the text, not drawn Pre-LN: flat Post-LN: 2^(−l/2) Same curve, two values Both halves are the same chart — only the verdict changes x: layer; y (log) = share of the original input left in this layer's output share left layer → 1 10⁻¹ 10⁻² 10⁻³ 10⁻⁴ Pre-LN Post-LN Ⓐ In pretraining Residual in name only The nearer the input, the harder the cut Early layers barely learn share left layer → 1 10⁻¹ 10⁻² 10⁻³ 10⁻⁴ Pre-LN Post-LN Ⓑ But in fine-tuning Exactly what you want Fine-tune only the top layers Holds early layers in place Identical curves, same axes — only “what you use it for” changes. At layer 24, Post-LN's shortcut term is 1/4096 of Pre-LN's. Why not just remove LN? Because the variance keeps growing Without it, x + F(x) has variance 2, growing with every residual, so you still need a Norm. The question was never “whether”, but where. Pre-LN adds x + F(Norm(x)), plus one Norm on the final output — so every residual branch is weighted equally, and the decay above disappears.
Same chart, same axes, both halves. Only the conclusion box changes.
Red, Post-LN: the input's share in layer l's output is 2−l/2, 1/4096 at layer 24. Green, Pre-LN: flat at 1.
Ⓐ: “residual in name only”; Ⓑ: it conveniently pins early layers. Same curve: pretraining bug, fine-tuning feature.
⛔ This is the forward residual shortcut term, not a gradient multiplier: related, not the same.
Sources and assumptions

📌 Su Jianlin, “Notes on Model Optimization: Why Is BERT's Initial Standard Deviation 0.02?” (in Chinese), kexue.fm/archives/8747 — “In Post-Norm BERT, LN not only fails to ease vanishing gradients, it is one of their ‘culprits’” is his line (our translation); 2^(−l/2) comes from recursively unrolling its formula.

⚠️ The author's own analysis, not peer-reviewed. ⛔ The figure plots the forward residual shortcut, not a gradient multiplier: related, not the same.

So why not remove it? Without it, x + F(x) has variance 2, growing with each residual. The question is “where”, not “whether”: Pre-LN uses x + F(Norm(x)) plus one final Norm, so all residual branches weigh equally and the decay is gone.

Most counterintuitive: elsewhere, this “bug” is a feature.

Fine-tuning should adjust parameters near the output first and barely move those near the input, to preserve what pretraining learned. Vanishing gradients mean “closer to the input, weaker effect on the output”: exactly what fine-tuning wants.

So pretrained Post-LN models often fine-tune better than Pre-LN. That's the figure's two halves.

📌 Source and disclaimer per 3.5 (Su Jianlin, kexue.fm/archives/8747). It once said “see the footnote in 3.4”, wrongly: that footnote cites another post by the same author (archives/11267, on update RMS 0.2).

Of the three warmup explanations in 3.5, this confirms one precondition. The third, “hold back later layers until earlier ones catch up,” relies on this exponential decay. Pre-LN lacks it, so that explanation doesn't apply.

QK-norm: runaway attention scores

At around 8B parameters, ViT diverged within a few thousand steps: attention logits grew huge, so post-softmax weights collapsed to near one-hot (near-zero entropy).

The fix: LayerNorm Q and K before the dot product. Hence QK-norm, now standard in many large models.

📌 arXiv 2302.05442 (ViT-22B) §2. QK-norm was not invented by ViT-22B. The paper says “we adopt the approach of Gilmer et al. (2023)”. ViT-22B applied it at 22B parameters, with a comparison figure: hence the citation. Again a failure that only appears at scale, as in 6.1.

Initialization: brief, but don't skip it

With every residual branch initialized at one scale, the signal grows with depth. Standard practice: scale residual-branch init down with depth. It won't prevent spikes, but sets your starting point.

6.6 Two common misconceptions

❓ “Does bf16 need loss scaling?” No.

In short: same 16 bits, split differently. fp16 favors “fine resolution”; bf16 favors “reaching tiny values”. fp16's problem isn't coarseness; it can't go low enough: a small gradient becomes exactly 0, and that parameter gets no signal that step. Loss scaling is simple: shift values up, off the floor.

Loss scaling was an fp16-era necessity: fp16 has only 5 exponent bits, so small gradients underflow to 0. Scale the loss up before backprop, down before the update.
bf16 has fp32's 8 exponent bits: same range, less mantissa. No underflow, no loss scaling.

fp16's problem isn't “too coarse”; it's “can't go low” This panel has one real number line and real boundaries — each value computed from IEEE 754 bit patterns fp16 min normal fp16 zero line bf16 min Ⓐ On one log axis, the two 16-bit formats' floors are 30 decades apart Left = smaller; past the red line, the value is 0 in fp16 This whole span: bf16 holds it, fp16 can't 30 decades 10⁻⁴⁰ 10⁻³⁵ 10⁻³⁰ 10⁻²⁵ 10⁻²⁰ 10⁻¹⁵ 10⁻¹⁰ 10⁻⁵ 10⁰ larger → bf16 1.18e-38 fp16 zero line 5.96e-08 fp16 min normal 6.10e-05 So fp16's problem was never “16 bits too coarse” — bf16 is 16 bits too, with a shorter mantissa. The problem: only 5 exponent bits, so the floor is too high. Ⓑ What loss scaling does — push the whole distribution right on this axis ×1024 = shift of 3.01 units on the log axis — enough if it crosses the red line gradient 2e-08 in fp16 → 0 × 1024 becomes 2.05e-05 survives Honestly: once scaled it lands in fp16's subnormal range (right of red, left of orange) — stored, but at reduced precision. The normal range needs a bigger scale. bf16 never needs this — its floor is 30 decades further left, so these gradients are already in range. So whether bf16 needs loss scaling comes down to exponent bits, not total bits bf16 and fp16 are both 16 bits, but bf16 spends them on exponent (8 bits, like fp32), fp16 on mantissa (10 bits). So bf16 is coarser but wider. Rule: “not precise enough” is an adjective; “everything left of this line is gone” is a countable loss. Swap a vague worry for a line you can point at, and the myth dissolves.
fp16 isn't “too coarse”; it “can't go low”.
Ⓐ Three real boundaries on a log axis: fp16 min normal 6.10e−5, fp16 zero line 5.96e−8 (left of it is truly 0), bf16 min 1.18e−38. The green span, 30 orders of magnitude, is what bf16 stores and fp16 can't.
Ⓑ Loss scaling is a rigid right shift on this axis: ×1024 = 3.01 units, rescuing gradients below the red line.
⛔ Boundaries come from numpy.float16 bit patterns; “rounds to zero” was verified by rounding. No histogram: we lack that data and won't invent it.
Sources and assumptions

📌 Every boundary is computed from numpy.float16 bit patterns: min normal 0x0400 = 2⁻¹⁴, min subnormal 0x0001 = 2⁻²⁴; bf16 shares fp32's 8-bit exponent, floor 2⁻¹²⁶. “Below the red line rounds to 0” was verified by actually rounding.

⭐ The framing of this figure comes from the classic mixed-precision figure (a gradient histogram + a vertical fp16 floor line; see Micikevicius, Narang et al., ICLR 2018). ⛔ But we lack that histogram data, so we draw no histogram, only the real boundaries.

Pairs with 3.2: bf16 buys “enough range, poor precision”: the danger is accumulation, not underflow.

❓ “Should training be reproducible?” It can be, and it pays.

PaLM was bitwise reproducible: restart from any checkpoint and every later step matches the original run exactly.

It takes a deterministic compute framework and a deterministic data pipeline: the batch depends only on the step, not on process count or resumes.

6.2's “roll back + skip data” needs this. Otherwise, which batches to skip?

📌 Also from arXiv 2204.02311.

6.7 Takeaway: four rules

Back to the start.

Section 0: training uses one number to decide which way each of 671B parameters moves.

Every part is now explained:

  • That number — −ln p, the model's surprise at the right answer. From 11.77 (uniform guessing) to around 2. (1.0)
  • How one number becomes 671B derivatives — many in, few out, so going backward from the output is cheapest. 671B forward passes become one backward pass. (1.5)
  • How big a step to take — direction is computed; step size isn't. Fifty years of optimizers fight that unanswerable question. (3.3b)

All that accounting (3× compute, undroppable intermediates, 16 B/param) just moves it from 11.8 to 2.

Near 2, each 0.01 is expensive, paid with the whole bill you just worked out.

Dozens of numbers. Not what we want you to keep.

  • ① High precision or not compares two numbers: “this quantity's relative change per step” vs. “the smallest step the mantissa resolves”. Adjacent bf16 values differ by 1/128 (7 mantissa bits, 2⁻⁷), so the “added vs. rounded away” threshold is half that, about 1/256.
    Both right, different quantities: 1/128 is the grid spacing (the 0.0078125 in Ⓑ of 3.2's precision figure); 1/256 is the rounding threshold (under half a step, nothing changes). Use the latter.
    Moving averages forget each step, so they move; master weights sum 100K steps of a few thousandths each, so they can't. (3.2)
  • ② Recompute or not: “FLOPs paid per byte saved”. For a linear layer, the input width; for attention, linear in sequence length. Different slopes, so they cross. (2.3)
  • ③ If two quantities grow at different rates in one variable, and the faster one starts lower, their ranking must flip somewhere. So any “A beats B” needs the range where it holds; without it, the ranking is meaningless.
    The underlined condition was added on 2026-09-20; without it the rule is wrong. If the faster one also starts higher, the lines never cross on the positive axis; the ranking never changes. Our example (2.4, crossing at S ≈ 13,107) is the “slower one starts higher” case, so it held. A rule always holds on its own example, where it's hardest to falsify.
    Fixed, it's more useful: to predict a flip, check which grows faster and which starts lower.

    Here it flipped several times, each in a different variable: sequence length (2.4), model scale (6.1, this section), and one switch whose payoff even changes sign across scales (2.6).
  • ④ Judge stability techniques on “stable?” and “quality lost?”. On the first alone, a zero learning rate is optimal. (6.3, this section)

⭐⭐⭐ Numbers go stale. These four rules won't.

Deliberately, none of our own measurements here: all four come from papers, each checked at the source. Why (§7): the lecture has many self-derived numbers; stability shouldn't add more.

Appendix

Checking the numbers

This lecture's numbers are not all of one kind. Some trace to a paper; others we derived op by op, and those could be off by 2×.

7.0 What frameworks call these

What to search for in the source

Our termTheir nameVerify in
Full recompute (2.2) --recompute-granularity full Megatron-LM
training/arguments.py
core/transformer/transformer_config.py
Selective recompute (2.3) --recompute-granularity selective + --recompute-modules
Options: core_attn (default under selective) / layernorm / moe_act / mla_up_proj / mlp …
Grad clipping (1.8) --clip-grad (default 1.0, V3's value)
Recompute, by hand torch.utils.checkpoint.checkpoint(...) PyTorch
torch/utils/checkpoint.py
Selective, per tensor create_selective_checkpoint_contexts + CheckpointPolicy
2.3's criterion in PyTorch: your rule picks which ops to save or recompute
Recompute (JAX) jax.checkpoint (alias jax.remat) JAX
jax/_src/ad_checkpoint.py
Stock selective policy jax.checkpoint_policies.dots_with_no_batch_dims_saveable
The name is the criterion: “keep matmul outputs”, 2.3's costliest rows
Gradient accumu­lation (1.8, stage ④) ddp.no_sync(), a context manager
Gradients accumulate locally, unsynced; sync “in the first forward-backward pass exiting the context” (docs)
PyTorch
torch/nn/parallel/distributed.py
“Reduction hides behind compute” (1.8, stage ②) bucket_cap_mb (default 25 MiB)
Each full bucket ships at once, in flight while earlier layers compute. Stage ②'s “✅ hideable” lives here
ZeRO stages (4.2) config zero_optimization.stage = 1 / 2 / 3 DeepSpeed
runtime/zero/config.py
8-bit optimizers (3.4) bitsandbytes.optim.Adam8bit / AdamW8bit bitsandbytes
optim/__init__.py

Two rows where our criteria are the API names.

  • dots_with_no_batch_dims_saveable: “save matmul outputs.” Why matmuls? Each byte they save costs the most FLOPs to recompute: the closed form in 2.3.
  • ddp.no_sync(): no_sync() switches off stage ② entirely. No communication while accumulating; one sync at the end. Saving communication is this API's purpose, not a side effect of accumulation.
  • core_attn is Megatron's default once selective is on (not “on by default”: recompute_granularity defaults to None), and the source warns it's unneeded with fused attention. Yet by 2.4, at long context it is the last thing to recompute. Defaults have a vintage (2.5). Not a contradiction: 2.2's “recomputation is on by default in large-model training” is industry practice; recompute_granularity = None is the factory setting. Different “defaults.”

One option we skip: PyTorch's CheckpointPolicy has six enum values, not two: {MUST,PREFER}_SAVE, {MUST,PREFER}_RECOMPUTE, {MUST,PREFER}_CPU_OFFLOAD.
Each activation is a choice of three: keep in HBM / recompute / offload to CPU and back. We cover the first two. The third buys HBM with PCIe bandwidth instead of compute: same trade, different currency.
MUST vs. PREFER, docs: “Use MUST_* over PREFER_* to indicate that the policy should not be overridden by other subsystems like torch.compile.”

So the table is a reverse check: if your criterion doesn't match the options, your criterion is likely wrong, or the default stale.

This table goes stale first:

Every name was grepped live from each main branch in September 2026, not from memory. Frameworks rename often.

Rule: when you give a name, give the file it lives in. Otherwise a rename makes it misleading, and nothing errors.

7.1 Cited numbers (safe to copy)

Number / claimSource
Classic 16 bytes/param (2 + 2 + 12); ZeRO's three-stage split Rajbhandari et al., arXiv 1910.02054
GPT-3 peak LR and batch for all eight sizes; 375M-token linear warmup; cosine to 10% over 260B tokens; weight decay 0.1; context 2,048 arXiv 2005.14165 Table 2.1 + Appendix B (§2.3 defers to it)
DeepSeek-V3's full LR schedule (2K steps → 10T flat → 4.3T cosine → 2 final stages); clipping 1.0; batch 3,072→15,360; bf16 moments, fp32 master weights/grads; recomputes RMSNorm, MLA up-proj. arXiv 2412.19437 §3.2.3 (recompute) / §3.3.3 (low precision) / §4.2 (hyper­params)
Original selective-recompute criterion; GPT-3 saves 70% at 2.7% cost, MT-NLG 65% at 1.6% Korthikanti et al., arXiv 2205.05198
Warmup's real role (permits a larger target LR; “too few samples to estimate” is not the main reason; longer warmup: diminishing returns) arXiv 2406.09405 (NeurIPS 2024)
Original definitions of Adam / AdamW / Adafactor / Lion 1412.6980 / 1711.05101 / 1804.04235 / 2302.06675
Muon: 2D params only; embedding, output head on AdamW; slower per step (wall-clock) than AdamW Author's writeup (all three stated there)

7.2 Our derivations (magnitudes, not thresholds)

NumberDerivationWeak points
Activations 104.11 GiB → recom­puted peak 5.07 GiB (4K) Op by op (1.7's two tables). Inputs: V3's config + the reference MLA forward Op fusion; does MoE dispatch really make nine copies?
Attention = 82.1% of a layer's compute Causal-halved. We once said Topic 1's “independent” 81.8% agreed. Wrong; retracted. Same numerator (one formula, S²·H·(dqk+dv), one config, same halving); only denominators differ: whole model incl. lm_head (Topic 1) vs. one layer's active params. The 0.463% gap = lm_head dilution: 81.8 ÷ (1−0.00463) = 82.18. Convention (halved?); head dims. Self-check, not an external anchor: same source, must agree. No third-party data yet
Crossover S ≈ 13,107 1.25·S = 16,384 (o_proj input = 128 heads × 128); 1.25 = V3 head dims 192/128, causal-halved. Earlier: d_model 7,168 → 5,734. “Widest linear = d_model” fails for MLA Another layer moves it. Not an engineering threshold
6ND is 5–6× too low Back-derived from the 82.1% above (1 ÷ 0.179) Inherits the 82.1%'s doubt; no third-party check yet
9.76 TiB resident block · crossover at ~1,973 4K sequences 671e9 × 16 B ÷ 1024⁴ ÷ 5.07 GiB peak. Peak must include the layer being recomputed; checkpoints alone give 2,997 (52% high). Build-time assert 16 B is the classic convention; V3 isn't set up that way (3.2)
“At 80 GiB per device, it takes 125” 9.76 TiB ÷ 80 GiB Arbitrary 80 GiB yardstick, not a real device; real use adds redundancy, comm buffers, frag­mentation

Why this section exists.

Both kinds look identical (a few digits and a unit), and readers copy them together.

⭐⭐⭐ Rule: a self-derived number must come with its derivation, or it is posing as a fact.

📌 A method, with a counterexample here: an isolated estimate is unverifiable; one matching an “external” anchor checks the whole formula. Our 82.1% has no anchor: as retracted, it shares a source with Topic 1's 81.8%, so had to agree. “They agree” counts only if paths are truly independent.

← Course index · the forward half of the bill: Topic 1 · The Life of a Token · want to teach it? Instructor notes for this lecture · how the bill gets split: Topic 5 · Parallelism
Generated by Courses/tools/topic04-build.py; the text lives there.