Topic 1 · The Life of a Token 16 segments · 67 min

How to use these notes

These are the presenter's notes, not for students. They read the course page; you read this.
Course author: Chris Yang · Google Cloud AI Infra architect.

🎯What this segment must leave behind The one sentence students should keep. Just one. Scattered means not landed.
🗣Script Near-verbatim. Say yellow lines word for word; reword the rest freely.
🖥Screen Where to scroll on the course page, and whether to move the slider at the top.
❓Likely questions Real questions, with answers. No answer? It says so: “I'm not sure” beats making one up.
⚠️What not to say Where this segment drifts. That's where the time goes.

How the hour is split

The course page holds more than an hour can cover, on purpose. In class, keep this pace on the main thread; details stay on the page for later. The sidebar uses the same timings.

MinContentLands
0–5Opening: all guess the KV cache one 128K prefill leaves; then assumptions, model card Frame it: “we're doing the math.” The opening computes only the KV cache: that number is certain. Activations wait for the tally
5–12Step 1 · Embedding: one lookup, 1.75 GiB gone The first “sit up” moment
12–27⭐ Step 2 · MLA: four generations → the compression → three blowups The heaviest 15 minutes. Cut anything else, never this
27–40Steps 3–4 · Dense → MoE: dense's four virtues, then what MoE trades them for Say outright: “MoE saves compute, not HBM”
40–48Steps 5–6 · Between layers, exit: residual stream as bus; logits blowup at the exit Raise “why not keep all activations?” Don't answer
48–60⭐ Steps 7 + 7.5 · Total + we ran it Land on “at least 14 devices,” then hook Topic 2

Using the course page live: project the course page; scroll with the steps. The console at the top can be dragged any time: in the attention square-term segment, push sequence length from 4K to 1M on the spot. Beats ten sentences.

⛔ Before you go on stage, remember this Not sure? Say so on the spot: “I'm not sure; I'll check and get back to you.” Never improvise a plausible-sounding explanation.

An invented answer looks the most real: it has the shape of correct knowledge, so you won't catch it yourself. And if one person in the room knows better, your credibility for the whole lecture is gone.

Wherever these notes say “not sure,” say exactly that. A real example: this course once taught “7168 = 128 heads × 56” as an architectural constraint. 128 × 56 does equal 7168, but V3's per-head dimension is 128, not 56. The constraint doesn't hold for MLA at all. That's the cost of making things up.
00

Fix the ruler: how we do the math

⏱ 3 min
What this segment must leave behind Fix the ruler first, then measure, and never swap it midway. Students must accept: no conclusions here, only math, with the assumptions nailed down first.

Script

Stay on the first screen. Don't scroll yet.

First, the method, because it's not how most material on large models works.

Most material hands you conclusions: MoE saves compute, MLA saves memory, FlashAttention is faster. All true, but heard. And with things you heard, change the setting and you can't tell if they still hold.

This lecture does one thing: the math.

How? Take a real model, DeepSeek V3, and follow one token from input to output. At every component, ask the same four questions on the spot: how big are the weights, the activations, how many multiplications? And the fourth, the one that matters most: does anything blow up here?

You'll find that last question is usually the point.

One beat. The real content of this segment starts here.

But measuring needs a ruler. The usual mistake isn't bad arithmetic; it's switching rulers halfway. So we nail the assumptions down now, for good.

Scroll to the four “Assumptions” cards.

Five. One: sequence length 128K, i.e. 131,072 tokens, roughly a short book. Two: batch of one; get one sequence right, multiply later. Three: bf16 precision, two bytes per number. Four: forward only, no backward. Five: the yardstick is one TPU v7 device, 94.74 GiB; every number we compute gets divided by it, or you have no feel for it.

Number four needs explaining. Why forward only?

Optionally ask “Anyone want to guess why?”, then answer yourself.

Because forward is fact; backward is mixed with decisions.

Forward is matmul after matmul, each step a fixed size. No choice. Backward: first decide. Keep intermediates or recompute them? Which device holds optimizer state? Accumulate gradients over steps? Strategies. Choose differently, get completely different numbers.

Mix decisions into facts and the math gets muddy. So we count only the part everyone pays. Backward gets Topic 4, when we know the math and strategy is worth discussing.

If the room is impatient, jump to the “First, guess a number” quiz, then come back for the assumptions.

OK, the ruler's set. Now, Step 1: how a word turns into a row of numbers.

Likely questions

Why 128K? Why not 4K or 1M?

Because 128K is today's norm, and it sits right at the tipping point. At 4K problems hide: attention takes only about 10% of compute (computed later). At 128K, about 80%: the bottleneck has moved.

At the key moment, drag the slider at the top in front of them.

Batch of 1? Isn't that unrealistic?

On purpose. Batch is a multiplier: it scales every term equally and only hides the proportions.

Batch=1 shows the structure. For realism, move the slider at the top to 8.

What's bf16? Why not fp32 or fp8?

bf16 is today's default precision for training and inference: two bytes per number. fp32 is four bytes, more precise but twice the cost. fp8 is one byte, cheaper but numerically risky.

Stop there. Precision is Topic 8; “two bytes per number” is all the math needs.

Where does 94.74 come from?

It's the usable HBM of one TPU v7 device: a hardware constant, not something we measured.

If pressed on what's inside the device: that's Topic 2. Note it and move on, or it eats your time.

Where does 576 come from? Why not 512?

512 is the compressed KV, 64 the RoPE slice: together, what one token stores per layer. In the config: kv_lora_rank = 512 and qk_rope_head_dim = 64.

Why not compress the 64 RoPE dims too? One line, “they carry position; compress them and they're useless,” then stop. Step 2 covers this asymmetry; expanding now blows the time.

MQA needs only 3.81 GiB, even less than MLA. Why not just use MQA?

The best question you'll get, and the crux of Step 2. Don't fully answer yet.

Say only: MQA saves by making all heads share one K/V copy, and the lost expressiveness is gone for good. MLA saves almost as much without any head paying. Step 2 is the how.

Then park it on the board; point back at MLA. In the opening, an open hook beats an answered question.

So how much does backward cost? Can you give a rough number now?

A directional answer settles the room: compute roughly triples, and the biggest memory chunk is optimizer state, larger than the model itself.

Then stop: the line signals “more math is coming,” not Topic 4.

What not to say

The three easiest ways to drift here ① Don't start explaining what a Transformer is. Newcomers pick it up along the “follow the token” thread; a crash course takes ten minutes and wrecks the pace.
② Don't explain bf16's bit layout. Say “two bytes” and move on.
③ Don't linger on “why 94.74.” That's Topic 2.
Pacing Three minutes, max. It's the foundation, not the house. What grabs them is the Step 1 moment: “we haven't entered layer 1, and a chunk of HBM is already gone.” Get there early.
01

The model card: reading the table

⏱ 2 min
What this segment must leave behind Every number here comes from the officially published config file. None is estimated, none measured by us. It decides whether the room believes your later math. Say it out loud.

Script

Scroll to the three “The model card” tables.

The table has over twenty rows. Don't read them one by one. People tune out and won't remember them anyway.

Pull out just four numbers. Point back to the rest as they come up.

Click each of the next four on the table as you say it.

First: hidden_size is 7168. The width of the residual stream, the main road through the whole model. Every activation later is token count times this number. If the room remembers one number, make it this one.

Second: 61 layers. Nothing unusual. The next row is: first_k_dense_replace is 3, so the first 3 layers are ordinary MLPs and MoE starts at layer 4. Precisely: “3 dense layers plus 58 MoE layers.” We'll reuse that 3 and 58 when counting parameters.

The third number is actually a false: tie_word_embeddings = false. The big lookup matrix at the entrance and the exit matrix that turns vectors back into words don't share weights. Two separate copies. Many models share to save a copy. V3 doesn't, and pays over 900 million extra parameters.

Plant a hook here. Don't answer it.

Ask in passing: “Why do some models share and some don't?” Then: “We'll get to that at the exit, the last step.” Not now.

The fourth isn't a number but a sentence, the most important one:

Every number in this table comes from DeepSeek's officially published config file. None is estimated, and none is our own measurement. Every later derivation starts from here.

Say it: the final tally gives 671 billion total parameters and 36.6 billion active, matching the official figures. That's when this sentence pays off.

Likely questions

How was the width 7168 chosen?

Two hard factors: align with the hardware's matrix units (7168 is a multiple of 128), and balance against layer count: same parameter budget, wider or deeper? Each model redoes it; no standard answer.

⚠️ Don't say “7168 must divide by the head count.” Not true for V3. 128 heads × 128 dims make attention 16,384 wide inside, 2.29× of 7168; o_proj folds it back. Standard MHA needs the two equal; MLA drops that constraint.

Hence o_proj is the largest weight in an MLA layer (117.44M, 63% of it): it spans the 16384 → 7168 gap. Point back here at Step 2's five-row table.

Why is the vocabulary such an odd number, 129,280?

Certain: vocabulary size is set when training the tokenizer, as a trade-off. Bigger: one word per token, shorter and faster sequences, but both big matrices at entrance and exit grow. Smaller: the reverse. 129,280 is in between.

Checkable: 129,280 ÷ 128 = 1,010 exactly, friendly to the matrix units.

Unknown, so say so: DeepSeek has never publicly explained the choice. If asked: “They haven't said; I'm not sure.” Don't improvise a plausible reason.

What's the MTP row?

num_nextn_predict_layers = 1: besides the next token, the model also predicts the one after. Just mention it; Step 6 goes deeper. Its uses in training and inference differ completely; that's the interesting part.

Why aren't the first 3 layers MoE?

Good question; hold it. Never answer offhand “early layers are more general.” That's everywhere, but it isn't the paper's reason.

For now, one line: the earliest public reason was training stability (DeepSeekMoE saw layer-1 load balancing converge especially slowly), and the paper doesn't explain why V3 went from 1 layer to 3.

Step 3 takes it apart, with its own figure.

What not to say

The two easiest ways to drift here ① Don't explain fields one by one, least of all MLA's six (q_lora_rank, kv_lora_rank and the rest). They only make sense in Step 2. Explain now and you explain twice; the first won't land.
② Don't explain YaRN. If asked about the max_position_embeddings row, say “an extrapolation method stretches it from 4K to about 160K; that's Topic 3” and move on.
Pacing Two minutes. Hand out the model card as a later reference; don't tour the model. Past three minutes, you've drifted.
02

How a word becomes numbers

⏱ 8 min
What this segment must leave behind Before the first layer, a chunk of HBM is already gone. Nothing “computed” yet, just one table lookup, but 3.48 GiB is already spent: the lecture's first “so that's how the bill builds up” moment.

Script

Scroll to “Step 1 · From text to vectors”: vocabulary matrix on the left, residual-stream tensor on the right, a “lookup” arrow between.

First, what happens at the input. A sentence comes in and is cut into tokens.

A token isn't a character, nor a word. Common Chinese characters are mostly one token each; a long English word may split into two or three. Once cut, every token is just an integer ID, on V3 from 0 to 129,279: the vocabulary is 129,280.

Scroll to the axis figure under “Tokenization: why 129,280”: green band on top, red below, axis between: the tug-of-war over vocabulary size. Talk to the figure. Don't read the text.

The room usually asks first: “where does 129,280 come from?” The answer first: it's not special at all. Look at this axis. Llama 2 is 32K. Llama 3 is 128K, V3 is 129K, Qwen3 is 152K, gpt-oss about 200K, and Gemma 2 is 256K. V3 and Llama 3 are practically the same point.

Point at the green bandA bigger vocabulary buys this: fewer tokens per sentence. Shorter sequences, so attention's square term drops. That pays off in Step 2.

Point at the red bandIt costs three things: both tables grow, rare words get too few samples to learn, and the exit computes softmax over the whole vocabulary.

Get this one exactly right. The “both tables get bigger” cost barely applies to V3. Table size is vocabulary × width only, regardless of the 61 layers. So the bigger the model, the smaller its share: both tables together are only 0.28% of 671B. The other two costs are what really cap vocabulary size.

Pause. Next: this segment's first counterintuitive point.

By default, going from ID to vector isn't a multiplication. It's a lookup.

The model stores a big 129,280 × 7,168 table. Whatever your token's ID, just pull out that row: no matmul, no additions, one row fetched by index.

It's the plainest step in the model, the only one with almost no compute. Not cheap, though, because of the table: 129,280 times 7,168 is 926.68 million parameters, which is 1.73 GiB in bf16.

Point at the “Residual stream” tensor.

Now the second cost. After the lookup, each of the 128K tokens is a 7,168-dim vector. Stacked: a 131,072 × 7,168 tensor. Two bytes each in bf16: 1.75 GiB.

Add the two together here. Slow down.

A table, 1.73. A tensor, 1.75. Together, 3.48 GiB. And we haven't entered layer 1 yet.

This segment lands here. Stop two seconds; let it sink in.

Point at the main road running through the page (the residual stream figure).

One last thing; it shapes how you see the next 61 layers.

That 1.75 GiB tensor isn't scratch you throw away. It is the residual stream: one main road from layer 1 to layer 61. Every layer reads it once, computes, and writes back once.

So 7,168 isn't some layer's internal size but the main road's width, end to end. Hence “if you remember one number, make it this one.” Every later step's activation memory is tokens × 7,168 × bytes per number.

If someone asks how 7,168 was chosen, open the “hard constraint vs. DeepSeek's own call” block and talk from it. Otherwise leave it shut.

The block in one sentence: 7,168 divides by 128 and by 256. TPU matrix units are 128 or 256 a side; NVIDIA FP16 wants dimensions aligned to 8, 64 on A100. Neither side needs padding.

Then add at once: that only explains why not 7,000 or 7,200, not why not 6,144 or 8,192, which divide just as evenly. Wide versus deep: DeepSeek has never publicly said how they decided.

Likely questions

Isn't a lookup the same as multiplying a one-hot vector by a matrix?

Same math; in engineering, worlds apart. A one-hot vector is 129,280 long. Stack 128K tokens and you get a 131,072 × 129,280 matrix. That intermediate alone is 31.56 GiB, and 99.999% of it is zeros.

So the default fetches rows by index: same math, different algorithm, costs orders of magnitude apart. FlashAttention is the same idea.

(Don't say “always”: the page's Expand block notes 34 MaxText production configs turn on use_iota_embed, which really compiles to a one-hot matmul. The denominator is so large that it's only 0.46% of total compute.)

What about position? Is “which token is this” encoded here?

No. On V3 this step carries content only. RoPE injects position on the fly in Step 2's attention (config row qk_rope_head_dim = 64).

Early Transformers did add a positional encoding to the embedding; mainstream models have mostly moved it into attention. Why is Topic 3. Just mention it.

Does that 1.73 GiB table stay in HBM the whole time?

Yes. It's a weight, resident from load to unload.

The exit table is a separate copy, not the same memory. That's what tie_word_embeddings = false means. Two 926.68M tables, entrance and exit, 3.45 GiB together. This number comes back in Step 6.

If a bigger vocabulary has so many benefits, why not go to a million?

What holds up first. A vocabulary scaling-law paper (Scaling Laws with Vocabulary, NeurIPS 2024, arXiv 2407.13623) concludes “most LLMs' vocabularies are too small.” Its fit puts Llama2-70B's optimum at 216K or more, vs. an actual 32K. Same compute, vocabulary 32K → 43K lifts ARC-Challenge 29.1 → 32.0.

⚠️ But state the limits: that paper fits dense models from 33M to 3B. There's no basis for extrapolating it to a 671B MoE. It shows the direction. It can't prove 129,280 is optimal for V3.

As for “why not a million”: rare words end up as dead words. It has a name, under-trained / glitch tokens (arXiv 2405.05417; the famous SolidGoldMagikarp is one; hit one and the model outputs garbage). Plus the exit softmax keeps getting more expensive.

You only get 1.75 GiB if you cram in the full 128K, right? Who has inputs that long?

Fair, and exactly the pattern we want them to find. Drag the console's sequence length to 4K; let them watch it fall. 56 MiB at 4K, 1.75 GiB at 128K, exactly 32×, because it's linear.

Linear is actually the good news. In Step 2 we'll see something that grows with the square, and that's the real trouble. Plant that hook here.

What not to say

The four easiest ways to drift here ① Skip tokenization algorithms (BPE merges, vocabulary training): off our thread (the math); ten minutes gone. If asked: “the tokenizer is trained separately; not covered.”
② Skip “why embeddings capture meaning.” King − man + woman is nice, but it's why the model works, not how to count its resources; we stay out of that.
③ Don't say “7,168 must divide by the head count.” (Flagged last segment.) V3's 128 heads × 128 dims make attention 16,384 wide inside, nowhere near 7,168. MLA removed that old constraint.
④ Don't turn “hardware-aligned” into “so they chose 7168.” Alignment is a hard constraint; width vs. depth, a trade-off: 6,144 and 8,192 divide by 128 and 256 just as evenly. Call a constraint a reason and any expert stops you with one question.
Pacing Eight minutes, but the real weight is in the last two. One minute on the vocabulary axis (“129,280 isn't special”), five on the lookup and the two costs, the last two on the residual stream as one main road. That's the skeleton of the whole topic. Go slowly.
By the end the room should think: “Only Step 1, and already 3.48 GiB?” If they feel that, Step 2 is easy to teach.
About the “Expand” blocks on the page All collapsed by default. Don't open any while presenting. They hold derivations, citations, and precision caveats, for reading after class.
Open one only when pressed, and talk from it. (PDF export expands them all, so the archive loses nothing.)
03

Why the KV cache runs away

⏱ 4 min
What this segment must leave behind Multiply two linear terms and you get a disaster. Don't give the answer here. Lay out the math and let the room conclude on its own that “this has to change.”

Script

Scroll to “The weak spot: a runaway KV cache.”

First, why it exists. Text is generated one token at a time, and each new token looks back at every token before it.

Store nothing, and each new token recomputes the tens of thousands before it. Again for the second, again for the third. The further you go, the worse the deal.

The natural fix: store each earlier token's intermediate results and reuse them. The two stored things are K and V; where they live is the KV cache. Nothing wrong there; every model does it.

Stop. Now: “where the trouble is.”

The trouble is how big it gets, and it grows in two directions at once.

One: the longer the sequence, the more you store. 100K tokens, 100K copies.
Two: every layer stores its own copy. 61 layers, 61 times.

Neither is scary alone. But they multiply. Grow linearly in two directions at once, and the product is a disaster.

Point at the log-scale staircase; walk it bar by bar.

“Disaster” is vague. Let's build the number step by step, assuming for now V3 uses the most traditional attention, unoptimized.

First bar: one token, one layer. 32,768 values to store: K and V × 128 heads × 128 dims per head. In bf16, 64 KiB. Negligible.

Second bar: × 61 layers. One token, whole model: 3.8 MiB. Still harmless.

Third bar: × 131,072 tokens. 488 GiB.

Stop for two seconds. This segment's first blowup.

Point at the vertical axis: log scale, 10× per gridline. It has to be, or the first bar would be invisible. Left to right spans nearly 8 million times, from two very ordinary steps: multiply by layers, multiply by tokens.

Point at the red dashed line, then at the number in the red box below.

Make it tangible. One TPU v7 device has 94.74 GiB of HBM. This pile needs more than five devices just to hold. For one user, one conversation. Serve a hundred people at once: times a hundred.

The sharper version, worth memorizing:

Stored this way, a single conversation of 25,446 tokens fills an entire device with cache alone. And not one byte of model weights has been loaded yet.

Scroll to the flare figure.

Last, how it changes with length. X is sequence length, Y is KV cache. Red is the traditional scheme; blue is MLA.

Both are straight, both linear, but their slopes differ by 56.9×. From one origin they keep spreading apart. The red gap between them is what a different storage scheme saves.

Say it out loud: not a fixed amount, but a gap that widens with length. The longer the context, the bigger the payoff from storing it differently. That's why, once long context arrived, everyone started changing attention.

Segment ends here. Don't go on to how MLA works.

Closing line: “At this point, I don't need to tell you what has to happen next.” Pause, and let them fill it in.

Likely questions

Why not skip the cache and recompute every time? Trade compute for memory.

Possible, but so costly nobody does it. Without a cache, token n recomputes the n before it: total compute goes from linear to quadratic.

And it's the wrong direction: generation is already bound by memory bandwidth, not compute. You'd spend the scarcer resource to save the cheaper one.

Partial recomputation is real; in training it's called recomputation. Different topic; skip it.

If it doesn't fit, can you move it to CPU memory or disk?

Yes: offloading, a mature practice. But it swaps “doesn't fit” for “moves too slowly”: every generated token hauls this pile across once, and bandwidth becomes the new bottleneck.

That's inference systems; this topic does the math, not solutions. Say “that's another route, and the cost is bandwidth,” and move on.

Store the KV in fp8 and you halve it, right?

Right, and widely used. Switch the console to fp8: 488 GiB becomes 244 GiB.

Make clear it's not the same thing as MLA: lowering precision stores each number smaller; MLA stores fewer numbers. The two stack, and real systems often use both.

Lower precision costs quality. How far you can go is an experimental question; this course draws no conclusion.

What about bigger batches? Serving lots of people at once?

One more multiplier, linear. Drag the batch slider to 8: the number goes up 8×.

The real consequence: the KV cache is one copy per user, the weights one copy shared by all. So the more people you serve, the bigger the KV cache's share of HBM, until it overtakes the weights themselves.

What not to say

The three easiest ways to drift here ① Never reveal how MLA works early. This segment's value is pushing people to “this has to change,” then stopping. Answer early and the next segment falls flat.
② Skip PagedAttention and memory fragmentation. That's inference-system engineering, not “why is the number so big,” and it won't stop once started.
③ Don't blur KV cache and weights. Keep repeating: the weights are one copy shared by everyone; the KV cache is one copy per user. The Step 7 tally needs it again.
Pacing Four minutes, but the turning point of the whole topic. The first three segments introduce. Here, for the first time, something goes wrong.
One minute per figure. The staircase shows how the number builds; the flare, how it spreads with length. After the closing line, stop. Two seconds of silence beat three more sentences.
04

Three knobs, four generations

⏱ 4 min
What this segment must leave behind The first three generations trade quality for memory. MLA trades compute for memory. Last segment: “something has to change”; this one: only a handful of places can. Narrower search, and the next segment's details land.

Script

Scroll to “Three knobs, four generations.”

We stopped at “I don't need to tell you what comes next.” Lay it out, with a reassurance: only a few places can change; you can count them.

Split the last formula's middle term: values per token per layer = 2 × KV heads × head dim. The 2 is K and V, fixed. The other two plus layer count are all the knobs.

Walk the three-row table row by row.

① Copies: KV heads; V3 has 128. The knob people keep turning, the strongest: to 8, divide by 16; to 1, by 128. The cost is as blunt: many Q heads share one K and V, and lost expressiveness never comes back.

② Width: the head dim. The industry settled on 128, for two reasons. It sets how much one head holds; narrow it and every head gets dumber. And kernels are tuned for few widths. You save memory but run slowly.

③ Layers: neighboring layers share one copy, or only some see the full length. This knob differs: it changes whether you store, not how much, so it's not in the four-generation table. If asked, one sentence.

Stop. Three knobs done; now four generations. Scroll to the four-generation figure. Point at each card, left to right.

The four generations: knob one turned from 128 down to 1.

MHA, 2017. Original Transformer: one KV per head. Best quality, biggest cache.

MQA, 2019. A very short Shazeer paper, “One write-head is all you need”: all heads share one copy, dividing straight by the head count.

Say this; it explains the next decade. Shazeer solved not “it doesn't fit” but “it can't move fast enough”: each generated token rereads K and V from HBM. Bandwidth was the bottleneck. “Doesn't fit” came later, with long context.

GQA, 2023. Ainslie et al. Neither one copy nor one per head: shared within groups. Default for a practical reason: the paper's recipe converts an MHA model to GQA with 5% of the original pre-training compute. No training from scratch.

Short pause. The punchline. Point at the blue card, far right.

Now look at MLA. It turns none of these knobs.

Still 128 heads. Still 128 dims each. All 61 layers still there. It changes what's stored: not expanded K and V, but a 512-dim compressed vector plus 64 position dims on their own path. 576 values in all.

Back to the numbers on the four-generation cards.

576 sits between GQA's 2,048 and MQA's 256. Yet its quality beats both. (Inferred from two tables: MHA beats GQA and MQA; MLA beats MHA.)

Stop; let them spot what's wrong. On one axis, that's impossible. Save more than GQA and beat its quality? No free lunch. It works because MLA isn't on that axis at all.

Two-second pause. Land it.

So the takeaway isn't the four names. It's one sentence:

The first three trade quality for memory; MLA, compute for memory.

The first three discard information for good. MLA compresses it, expanding into 128 heads on the spot. Nothing lost; the price is extra compute per use.

Trainers know this: don't store the intermediate, recompute it: recomputation. In a sense, MLA is recomputation for the KV cache.

Point at the yellow warning box.

One boundary you must add. Omit it and you mislead; get it wrong, worse. “MLA saves memory” is an inference statement: training has no KV cache at all. Caching exists only in decode, keeping earlier K and V for the next token. Training runs the whole sequence in one forward pass: no next step, nothing to cache.

Training's bill is activation memory, and at this scale recomputation is maxed out: a layer keeps only its input residual stream; 128-head K and V are recomputed, used, discarded. So MHA and MLA store the same residual stream: a tie, not an MLA win.

Don't flip to “MLA is useless for training”, the opposite mistake. Training gains twice, both in Topic 4: relax recomputation a notch and store just the 512-dim latent; context parallelism passes the latent, not K and V, cutting communication by the same factor.

Hand off: “Compress to 512, expand to 128 heads. Sounds like magic. Next: how, and why 64 position dims need their own path.”

Likely questions

MQA existed in 2019. Why did large-scale use wait until 2023?

Answer confidently; separate paper from interpretation.

Paper: Shazeer's motivation is the memory bandwidth of reloading K/V during generation, not running out of HBM.

Interpretation: 2019 sequences ran a few hundred to two thousand tokens; the KV cache barely registered. The knob was there, not worth turning. Long context and large-batch inference raised its payoff by orders of magnitude; then it was turned.

Close: the knob was always there; whether to turn it depends on what's most expensive at the time.

If MLA is better, why do Llama, Qwen and others still use GQA?

Say up front: engineering judgment, not experiment. Three practical reasons:

① GQA has an uptraining recipe: 5% of the compute converts an MHA model; MLA trains from scratch. ② MLA is far harder to implement (esp. inference-side matrix absorption); ecosystem support came late. ③ GQA is about one config line; MLA, a set of structural changes.

“Better” and “easier to adopt” differ. Infrastructure is full of this. Don't predict a winner.

Is the head dim really untouchable?

It can; it rarely does. Some models use 64, some 256.

Theory allows it, but both directions are a bad deal. Narrower holds less per head; wider regrows the cache. And kernels are tuned for few widths: pick an odd one and you save memory but run slowly.

Can you turn all three knobs at once?

Yes; real systems do. GQA + KV quantization + sliding window is a common stack. MLA can add fp8 too.

One caveat: the memory savings multiply, but the quality losses don't necessarily just add. How far to stack is empirical. This course gives no numbers.

Linear attention and sliding windows: which generation are they?

They're not on this line. The table compares values per token per layer; they change whether you store and for how long. Linear attention compresses history into a fixed-size state; sliding windows drop the distant part.

A separate branch; one sentence, move on.

What not to say

The three easiest ways to drift here ① Don't explain how MLA is implemented. This segment answers only “which axis changed.” Compression and RoPE routing are the whole next segment. Say them early, it's empty.
② Don't call MQA's quality flatly bad. Shazeer's paper says only minor quality degradation; the GQA paper stressed the loss. The figure's “clear quality loss” is for contrast; out loud, leave room: “good enough, but clearly behind MHA, which is why GQA came along.”
③ Don't present “MLA saves memory” as universal. Always add the inference-only boundary. Anyone who trains will catch it; say it first.
Pacing Four minutes: knobs 1, four generations 1.5, MLA's change of axis 1.5. Setup goes fast. Slow down on “it turns none of these knobs,” then pause two seconds.
Paired with the last: “something has to change,” then “only these places can.” Narrower search, details land.
MLA-savvy listeners are most likely to jump in here. Invite them to explain, then add the inference-only boundary. They probably won't.
05

How MLA works · RoPE's own path

⏱ 4 min
What this segment must leave behind Store 576 values; expand into 128 heads on the spot when used. Compute traded for memory. 64 position dims go separately: mix position into the compressed part and a very valuable inference shortcut is gone.
Where this segment sits Ideas only, no shapes. Tensor shapes and path branching are the entire next segment (05⁺⁺). Cover them here and it's empty.
It pays every IOU you leave. Say “we'll get to that next” freely.

Script

Scroll to “How MLA actually works.” Talk across text (left) and small figure (right).

Last time: MLA changed the axis. Now, what that axis is. In one line: compress on write, expand on read.

Concretely: a token's KV isn't stored as 128 heads but as a compressed 512-dim latent; at attention time a matrix expands it back into 128 heads.

Stress: the expansion happens on every use. That's “trade compute for memory” in practice. HBM holds 512 values; 128 heads appear at compute time.

Stop. Now the harder half.

Why can't position be compressed too? MLA's twistiest part.

Point at the blue box, “First, what RoPE is.”

Twenty seconds on what RoPE is. Most only know the name.

Attention itself is blind to order. Shuffle a sentence; the result is identical. So you must tell it what comes first.

RoPE does it neatly. It adds nothing; it rotates the vector by a position-set angle. Token 1 turns 1 degree, token 100 turns 100. In a dot product, the result carries their angle difference: relative distance.

The line that sticks: no new numbers; it just turns the vector.

Short pause. The course's easiest place to get muddled. Slow down.

The trouble: how far you turn depends on the token's position.

Rotate, then compress into the latent, and position is mixed in, the rotation stuck between two matrices. Matrices don't commute; what's stuck in the middle can't move out.

Get this next sentence exactly right:

What breaks is not correctness. The model still trains, runs, answers correctly. What breaks is a very valuable inference shortcut. The price: for every new token, you recompute K for every earlier token.

Don't name the shortcut yet. The room just needs to accept: once position mixes in, some cost later gets very expensive.

Point at the small figure's gold side path.

So V3 splits each head in two: 128 dims without position for content, compressible; 64 dims with position for order, stored raw.

Point at the “576 = 512 + 64” box.

576 = 512 + 64. That's everything MLA stores per token per layer.

Two-second pause. Now the real takeaway.

This 576 has nothing to do with the head count. 128 heads, 576. 256 heads, still 576. The head count drops out of the math entirely.

The fundamental difference from the first three: they divide by the head count; MLA removes it from the formula. Division hits a floor; a removed term has none.

Bonus: all 128 heads share one 64-dim copy. Per-head copies would need 128 × 64 = 8,192 RoPE values, growing the cache from 576 to 8,704. MLA, for nothing.

Likely questions

Are the expanded K and V the same as computing them directly?

No. It's lossy. 512 dims can't hold all of 32,768.

Yet that's why it works. A later segment shows “those 32,768 values never had 32,768 degrees of freedom.”

One line: lossy, acceptably so, and measured quality is actually better; why comes later. Don't expand.

If it expands on every use, doesn't compute go up?

Yes, but it's the right trade. Generation is bound by memory bandwidth, not compute. Spend the plentiful to save the scarce.

Restate the boundary: this is inference. In training the expanded K and V are recomputed and discarded, so MHA and MLA keep the same residual stream. No training memory saved.

Why 512? And how was the 64 chosen?

No public source derives either number; the paper reports results only.

You can offer only scale: 512 is about a fourteenth of 7,168, and 64 is a common RoPE dimension.

Here you can openly say “I'm not sure.” Far better than inventing a plausible reason.

All heads share those 64 RoPE dims. Don't they conflict?

The paper makes it a shared key, not one per head.

Intuitively: position is a property of the token, not of a head, the same for all 128 heads.

Boundary: a sensible design rationale; don't present it as the paper's argument.

Likely follow-up: why not share Q's 64 dims too? Because Q's part isn't pure position: it projects a vector from content, then rotates it. Same angle for every head, but what gets rotated differs, so each head gets a different “distance → score” curve.

But the real reason is cost: Q never enters the cache, so sharing saves nothing, just loses expressiveness. K does, and sharing squeezes 8,192 to 64, worth the loss. Cost, not math, decides the asymmetry. Derivation: the page's “Expand: why Q's 64 dims aren't shared” Touch on it only.

Q is low-rank too (1,536). Why?

Not for the cache: no byte of Q enters it.

The paper's reason: reducing activation memory in training, adding, pointedly, “even if it cannot reduce the KV cache.”

We also find it saves weights as a side effect (the third bill in the cost table). Both hold; say which is the paper's and which is ours.

What not to say

The three easiest ways to drift here ① Don't walk the shape chain. Shapes and the three paths are the next segment. See “Where this segment sits.”
② Don't unpack RoPE. Complex rotations, frequency allocation, YaRN extrapolation: ten minutes we don't need. “Rotate the vector by a position-dependent angle” is enough.
③ “Compression and rotation can't swap order” isn't about correctness. It's performance. Say “the result comes out wrong” and an expert corrects you on the spot, collapsing your “so it must be isolated” argument.
Pacing Four minutes: compression 2, RoPE 2.
The compression half is easy: “compress on write, expand on read” carries it. Don't drag it out.
Slow down for RoPE; stop at “some cost later gets very expensive”. Go no further; that's the next segment's hook.
Hit “the head count drops out of the math” hard. It closes Step 2 and is the line most likely to go home with them.
05+

7,168 and 16,384: a constraint that never was

⏱ 7 min
What this segment must leave behind Every head reads the full 7,168. The input is never split. “7,168 ÷ 128” means the 128 heads concatenate to exactly the input width. It describes concatenation, not dividing the input. A convention, not a law; after MLA, not even a convention.
Up front: this segment grew out of a real mistake This course got this wrong twice running, the second time more fundamentally.
First time: treating “7,168 = 128 heads × 56 dims” as an architectural constraint. 128 × 56 does equal 7,168, but each V3 head is 128-dim.
Second time: right number, but phrased as “7,168 is cut into 128 heads, each gets 7,168 ÷ 128.” That puts the “cut” on the input side: wrong. Each head's projection matrix is 7,168 × d, so every head sees the full input. The 128 pieces come from concatenation, not division.
Tell the room both mistakes. They convince better than any abstract warning, and most of the room is likely picturing the same slices.

Script

Scroll to “⭐ An easy trap: 7,168 and 16,384.”

First, mental math for the room: 7,168 divided by 128 heads. What do you get?

Wait two seconds; someone says 56. Don't correct it; say the next line first.

That division hides two misconceptions, and the second is worse.

The first is simple: 56 isn't V3's head dim. Each V3 head is 128-dim. 128 heads concatenate to 16,384, 2.29× the residual stream.

Stop. The section's foundation. Slow down.

The second: thinking the input gets split. Many picture 7,168 dims in a row, chopped into 128 segments of 56, one per head.

That's not how it works. The input is never split.

Point at the fan on the figure's left.

Each of the 128 heads reads the full 7,168 dims. Each has its own projection matrix, 7,168 by that head's output width, projecting the full 7,168 into its own working space.

So with the textbook choice, each head's matrix is 7,168 × 56. On V3, each head's matrix is 7,168 × 128. The 7,168 on the left never changes: every head sees everything.

So where's the “split”? On the output side. Each head outputs a short slice; concatenating them gives the total width.

So the division really says: “the 128 heads, concatenated, exactly equal the input width.” It describes concatenation, not dividing the input.

Pause and ask: is “exactly equal” required?

If that landed, it answers itself: obviously not. Each head's matrix is 7,168 by d, so d was always free. Set it to anything; the 128 heads concatenate to whatever width results.

Point at the two rows, especially each box's matrix shape.

Top row, textbook: 7,168 × 56 per head, concatenated exactly 7,168.
Bottom row, V3: 7,168 × 128 per head, concatenated 16,384; o_proj maps it back.

Same input in both rows: every head reads the full 7,168. Only output width differs.

Turn: so why has everyone chosen “exactly equal” for years?

The 2017 paper chose one value, and copying turned it into intuition.

The original paper defines the output projection as “heads × head dim” by “model width”. From day one, the two could differ.

The next sentence: “In this work we use 8 heads, each with d_model divided by h.” “In this work”: it's a choice. With a reason: total compute then about matches full-width single-head attention. A compute budget, not a mathematical necessity.

Now the real point: if it was always allowed, why did nobody do it?

Because wider heads cost you KV cache.

In the standard scheme, every head stores its own K and V. Widen the heads, every copy grows: the cache grows linearly. 2.29× wider, 2.29× more cache.

So in the old setup, attention width and cache cost are the same knob. Want expressiveness? Pay in memory. Most people chose not to pay.

Point at the figure's right side, where “Locked” turns into “Link broken.”

MLA breaks the link. It caches only the compressed vector, so cache size is (compressed dim + the 64 position dims) × layers. No head count, no head dim in that formula.

Not my inference: it's the paper's formula. The row in Table 1 of the V2 paper reads (d_c + d_h^R) · l, which on V3 is (512 + 64) × 61. (Don't read the symbols aloud; “576 times 61 layers” is enough. If someone checks the paper, the 64 is included; “compressed dim × layers” alone leaves it out.)

So one knob becomes two: expressiveness, and cache cost. What V3 did was turn the expressiveness knob up by 2.29×.

Evidence: V2, where MLA debuted, has a residual stream of only 5,120, yet attention is also 128 × 128 = 16,384: 3.2×, bolder than V3. DeepSeek never calls it deliberate; two generations alike is hardly chance.

Finally, the cost. Never skip it, or the segment becomes a sales pitch. Scroll to the figure under “The cost: three bills, V3 pays two.” Don't speak yet; let them look at the red dashed vertical line.

Every bar is measured against that red dashed line, the cost of staying equal-width, defined as 1.00×. Longer than the line: you paid more.

Pause. That's how to read the figure; if it doesn't land, the three bills are wasted.

On paper, widening adds three bills. V3 only really pays two.

Point at the first red bar.

First bill: compute, 2.86×. Width grows 7,168 → 16,384, QK runs at 192 per head: attention compute grows. Unavoidable; saving it is exactly why the original paper chose “exactly equal.” V3 knowingly traded compute for expressiveness.

Point at the second red bar (2.29×, shorter than the first).

Second bill: o_proj weights, 2.29×. It bridges 16,384 back to 7,168; one side 2.29× wider, the matrix 2.29× bigger. 51.38M becomes 117.44M, 63% of one layer's MLA weights. Also unavoidable.

The second bar is pure widening. The first runs longer because QK works at 192 per head, not 128.

Point at the third bar: yellow, dashed outline, the longest one.

The third bill should be scariest: 128 heads × a 7,168 × 192 Q projection, done directly, 176.16M. 3.43×, the bar that reaches furthest on screen.

But it's dashed: “what it would cost; V3 didn't do it.”

Move down one row to the short green bar.

V3 doesn't project directly: it compresses 7,168 to 1,536, then expands to 128 heads. A narrow waist. The two stages total 48.76M.

Pause here; it's counterintuitive.

0.95×. Green is shorter than the red line: 5% under equal width.

So V3 made attention 2.29× wider, and the Q projection still has slightly fewer parameters than without widening.

That drop on screen, 3.43× dashed to 0.95× green, is one field's work: q_lora_rank = 1536. It also shows the two LoRAs do different jobs:

kv_lora's 512 saves KV cache, because that's what gets stored.
q_lora's 1536 saves weights. Q never goes into the cache. It's compressed purely so that widening the heads is affordable in parameters.

The figure's bottom line says exactly this. Short on time? Read it out.

Likely questions

If every head reads the full 7,168, aren't 128 heads doing the same work 128 times?

Same input, but each head's projection is learned independently. One input, 128 matrices, 128 different subspaces.

Analogy: 128 people read one passage, each through a different lens, and each notices something different. Really split the input and each head sees only 1/128 of the passage: crippling.

The original paper's line: multi-head attention lets the model jointly attend to different representation subspaces.

Then what are q_lora_rank 1536 and kv_lora_rank 512? How do they relate to 16,384?

Narrow waists on the way, not the destination. Three numbers:

7,168 is the way in and out: the residual stream.
512 / 1536 is the narrowest point on the way. That's what gets stored.
16,384 is the widest point, for attention. Used, then discarded, never stored.

The whole path goes wide → narrow → wide → back to residual width. Save memory where it's narrow, get expressiveness where it's wide. All of MLA's cleverness is in that offset.

Do wider heads really buy quality? Is there evidence?

Honestly: I haven't seen a controlled experiment varying only head width. Labs rarely publish such ablations.

Inferable: wider heads express finer patterns (standard capacity reasoning), and V2 and V3 both used this freedom fully once they had it, which suggests the trade paid off in their own experiments.

Don't put words in DeepSeek's mouth. If asked: “I haven't seen a public controlled experiment, so I'm not sure.”

If you can widen freely, why stop at 2.29× and not go to 10×?

The other two costs are real. Wider means costlier attention compute and bigger o_proj weights. o_proj is already 63% of a layer; wider is more lopsided.

So it's not “widen freely.” It's “a constraint was removed, leaving a knob you can tune.” 2.29 is a trade-off point they chose, not a ceiling.

At inference, do you really expand 512 into 16,384? Isn't that wasteful?

Sharp question; the answer is “not necessarily.” The MLA paper describes a trick: the up-projections can be mathematically absorbed into neighboring matrices. K's folds into the Q projection, V's into o_proj.

After absorption, inference doesn't even need to compute K and V explicitly.

(To quote the paper, use this, from the end of the V2 paper's Appendix C: “due to the associative law of matrix multiplication, we can absorb WUK into WUQ, and WUV into WO. Therefore, we do not need to compute keys and values out for each query.” It's on the page, in the “Caveat: the fold is an inference optimization” box.)

An implementation optimization; not covered. Answer “it's absorbed into neighbors, so you don't really expand,” and stop.

What not to say

The easiest ways to go off track ① Don't call the convention “wrong.” Equal width is a reasonable, proven default; most models do fine with it. We take apart the belief it's a law, not the practice.
② Don't claim causation for DeepSeek. “MLA removed the constraint” follows from the formula; “that's why they chose 128 × 128” is interpretation. Weigh them differently.
③ Skip the matrix-absorption derivation. Elegant, but an implementation optimization: ten minutes off the main thread (the accounting).
④ Never say “split” or “divided among the heads.” Even in passing, the room pictures slices, and then “why heads can be widened” never makes sense. Always say “each head projects on its own” and “concatenated.”
Pacing Seven minutes, five parts: the puzzle → break the “split” picture → where the constraint came from → how MLA unties it → the cost. Don't reorder.
Part two (every head reads the full width) gets two minutes; it's the foundation. Land it and “why heads can be widened” explains itself.
The opening math is the hook. Really make the room do it. Don't say 56 yourself. Their own mistake sticks ten times better than one you describe.
Ending on the q_lora bill leads into the later five-bar weight figure.
About the “Expand” blocks on the page From this segment on, Step 2 hides 7 of them, all collapsed by default. No need to open any. They hold the three bills' full math, full ablation data, paper quotes, and the five-matrix shape table: after-class reading.
Open one only when pressed, and talk from it. Later segments' screen cues mark each block “skip” or “open only if pressed.” (PDF export expands them all, so the archive misses nothing.)
05⁺⁺

The shape chain: the steps Q, K and V each take

⏱ 8 min
What this segment must leave behind Only 576 values per layer are ever stored. Everything else is computed on the fly. Q, K and V each take their own path, and RoPE rides only a short stretch of two of them. That isn't a convenience, it's forced: let position encoding touch the main path and the cache you just saved is handed right back.
Where this segment sits The last segment covered why this design is possible; this one, how it actually runs. Don't flip the order. Every step here rests on that foundation: the input isn't split, and the width is free.
The only place in the course you walk a figure box by box. Don't recite shapes; point at the figure. Nobody remembers nine shapes, but they'll remember “three paths, two gold boxes.”

Script

Scroll to “⭐ The full shape chain”. Five seconds of silence first.

It looks scary but says only three things. A matrix enters at the top; one the same shape leaves at the bottom. Between, it splits into Q, K, V.

We'll walk it path by path. Each box is one transformation. Its three lines are the op, the weight shape and the output shape.

Pin down the start, or everything after floats.

In comes a batch of tokens, say 128K, each 7,168-dim. So a 128K by 7,168 matrix. That's last segment's residual stream.

Point at the Q column, top to bottom.

① Q path: down, then up

Step one: q_a_proj takes 7,168 down to 1,536. That matrix is 7,168 by 1,536. What comes out is Q's latent representation.

Step two, a norm. Shape unchanged, skip it.

Step three: q_b_proj lifts 1,536 up to 24,576. Reshape those 24,576 and you get 128 heads of 192 dims each.

Stop and spell out 192: the key to all that follows.

Why 192 per head, not 128? 192 is 128 plus 64. The first 128 dims are content; the last 64 are reserved for position encoding.

So step four splits each head's 192 dims in two: the first 128, q_nope, stay untouched. The last 64, q_pe, get rotated by RoPE.

Join them: 192 again. Q is final. 128K, 128 heads, 192 dims each.

One more line, so no one confuses Q with KV.

No byte of the Q path enters the KV cache. Squeezing to 1,536 saves weights and training activations, not cache. Hence the cost table.

Switch to the big KV column.

② KV path: store 576, compute the rest on the fly

K and V share a down-projection: 7,168 by 576, stream down to 576.

Then it's split right away: the first 512 are c_KV, the last 64 are k_pe.

Point at the gold dashed boxes: the key sentence.

At inference, these two blocks are all the model stores. 512 plus 64 is 576. That's it.

Stop for two seconds. Let it land.

Every box below, K, V and all built from them, is computed on use. None is stored.

Keep going down: c_KV goes through a norm, then kv_b_proj lifts 512 to 32,768. Reshape that into 128 heads of 256 dims.

Split again: first 128 are K's content part, last 128 are V itself. V is final here: 128K, 128 heads, 128 dims each. V never touches RoPE. Not once.

Go back to the narrow k_pe column.

The narrow 64-dim path? Into RoPE. Once rotated, one copy goes to all 128 heads. Figure shape: [128K, 1, 64]. The 1 is key: one head.

Joined to each head's K content part, K is final: 128 heads, 192 dims each. That lines up exactly with Q; now we can score.

Now the hardest part. Ask; don't answer yet.

③ Why RoPE needs its own 64-dim branch

Ask: why not position-encode the whole vector? Why carve out 64?

Wait. Most can't answer but get curious: the state you want.

Because otherwise, the cache we just saved gets handed right back.

Inference has a neat shortcut: pre-fold K's up-projection into Q's matrix. K is never computed; Q scores straight against the cached 512-dim latent.

But RoPE, a position-dependent rotation, would sit between the two.

Matmul doesn't commute; the middle can't move, so the fold breaks.

Concretely: each new token recomputes K for every earlier token. The memory you saved turns into repeated compute.

Point at the small “RoPE?” table.

So they did something clean: they put position on a side path.

Each head's vector splits in two. First 128, unrotated: content, and foldable matrices. Last 64, rotated: all position info.

Why? Dot products add. Dotting two concatenated vectors gives the front halves' dot product plus the back halves'.

One 192-dim dot product is content similarity plus position relevance, computed apart. No crosstalk; each half keeps its math.

Last punch, the segment's peak: how cheap this is.

This position path is only 64 dims, and all 128 heads share one copy.

In other words: store 64 more values, never recompute K. That's the 64 in 576.

Run it backwards: unshared, one copy per head, those 64 become 128 times 64: 8,192. The cache jumps from 576 to 8,704, and MLA is pointless.

Below is a collapsed block titled “Expand: why Q's 64 dims aren't shared”. Don't open it; cover it aloud, unless pressed on details.

Nine times out of ten, someone asks: position belongs to the token, not to a head. So why don't Q's 64 dims share one copy too?

They aren't pure position. They're content projections, then rotated. Same angle per head; what's rotated differs. So each head has its own “distance → score” curve; sharing locks all to one.

Cost decides. Q isn't cached; sharing saves nothing. K is cached; sharing shrinks 8,192 to 64: a loss worth taking. Cost, not math, sets the asymmetry. Compromise only on the side that costs money.

Point at the bottom row of the figure.

Wrap-up: Q, K score, over root 192, softmax, times V, concat 128 heads to 16,384, o_proj back to 7,168. Into the stream, then MoE.

Note the red box: scores are 128 heads times sequence length squared. Uncached, but it eats HBM quadratically. Next segment's second blowup.

The three numbers people get wrong most

Check before class. Someone will do the math. ① q_b_proj is 1,536 × 24,576, not 12,288. 24,576 = 128 heads × 192, and 192 = 128 unrotated + 64 rotated.
② 128 × 128 = 16,384 is the output-side width. It shows up at the input of o_proj (v_head_dim × heads), not at the output of q_b_proj. Easy to mix up.
③ k_pe has 1 head, not 128. So the cache is 512 + 64 = 576, not 512 + 128×64 = 8,704. The whole design hinges on this. Get it wrong and the segment falls apart.

Likely questions

Q is compressed too. Why no cache saving?

Because the cache holds history, and Q belongs only to the current token.

For token N, its Q attends to the previous N-1 tokens' K and V. K and V must be kept. Q is thrown away after use.

Compressing Q saves weights (smaller matrices) and training activations, not inference KV cache. MLA's paper compresses Q “even if it cannot reduce the KV cache.”

k_pe has one head shared by all 128. Doesn't that limit expressiveness?

There's a loss, but only in the position part, which never needed separate heads.

The 128 content dims stay per-head: all 128 filters intact. What's shared is “how far apart are token 5 and token 300,” the same fact for every head.

It's the MQA idea, borrowed locally: the figure's function name, kv_a_proj_with_mqa, gets its “mqa” from this.

If the matrices can be folded, why does the figure still show K and V being computed?

The figure shows the math definition, also how the public reference implementation computes it.

Absorbing the up-projections into Q and o_proj is an equivalent rewrite by inference engines. It doesn't change the result, only the order of computation.

Stress this: to keep that rewrite possible, RoPE had to be isolated. The design made room for an optimization that hadn't happened yet: this section's most intriguing point.

The weights are a few big matrices. So are the 128 heads real, or just bookkeeping?

Half right. In projections, bookkeeping. In scoring, heads are physical, unmergeable.

The right half: q_b_proj, kv_b_proj, o_proj are each one block, not 128. view: zero data movement, zero compute. “Head i's matrix” is its i-th column group.

The wrong half, as a counterfactual: no heads: one dot product of full 24,576-dim Q and K, one score per token pair. With 128 heads, each pair gets 128 scores, each with its own softmax, each weighting its own slice of V.

Totally different, and the second can't be one big matmul: softmax is nonlinear. Linear ops merge any way; apply a nonlinearity per group and they can't be re-stitched.

So a head has a lifecycle: at view it gets a name, it truly lives in scoring and softmax, at o_proj it dies. That matrix is the only place the 128 heads ever exchange info.

Hardware angle (infra folks love it): projections are big regular matmuls, accelerators' favorite; scoring, many small ones, low arithmetic intensity: one reason attention was long a bottleneck.

Why divide by root 192, and not root 128?

Because the dot product runs over 192 dims. The scale follows the dot product's dims, not the content part.

In code it's q_head_dim ** -0.5, with q_head_dim = 128 + 64 = 192.

(V3 also adjusts the factor for YaRN long-context extrapolation. Don't go there. If asked: “a separate correction; the main line holds.”)

Are the two matrices just equivalent to one 7,168 × 24,576 matrix?

No, it's a downgrade. Make this clear, or the later “cost” part means nothing.

One 7,168 by 24,576 matrix can express more. Pinch a 1,536 waist in the middle, and it gets at most 1,536 independent directions. In math terms, its rank is capped.

Analogy: a junction box with 7,168 wires in and 24,576 wires out, but only 1,536 between. Anything gets through, just not as many patterns.

Stricter: there's an RMSNorm in the middle. So it isn't even a product of two matrices; the middle step is nonlinear.

Real savings: direct takes 176M parameters, split 48.76M, 3.61× fewer. DeepSeek bets the transformation never needed that much rank.

Nine shapes is too many. A one-liner?

Yes, and it's the segment's real takeaway:

“Wide in → squeeze to 576 and store → expand into 128 heads on use → back to wide.”

Q follows the same pattern; it just isn't stored. RoPE alone gets its own lane.

What not to say

How this segment usually goes off the rails ① Don't read the shapes one by one. Reading a dozen-plus boxes runs over time, and nobody keeps up. Say only four numbers: 7,168 in, 576 stored, 192 for scoring, 7,168 out. Point for the rest.
② Don't explain how RoPE rotates. Complex rotations, frequencies, extrapolation: another course. You need only “it depends on position, it's a rotation, it can't be moved.”
③ Don't present matrix absorption as part of the model. It's an inference optimization. Call it architecture and readers of the reference code find a mismatch.
④ Don't mention, for completeness, that kv_a_proj is two matrices fused. (The c_KV compressor and the k_pe producer are merged into one 7,168 × 576 in the implementation.) An implementation detail; it only blurs “576 = 512 + 64.”
Pacing Eight minutes, three parts: Q path 2 min → KV path 3 min → RoPE 3 min. Fast through the first two; slow on the third.
The peak of the segment is “store 64 more values, and you never have to recompute K.” Every shape before it serves this line. Stop there.
Short on time: squeeze the Q path to one minute (four lines: “down, back up, split in two, rotate one part”), but cut nothing from the KV path or RoPE.
End on the quadratic matrix in the red box, handing off to the next segment's second blowup.
05⁺³

Why so much compression? The 32,768 never existed

⏱ 7 min
What this segment must leave behind Of the 56.9×, 4.57× isn't compression at all. It's just not being wasteful. The real compression is only 12.44×, and what it compresses is rank, not value count.
The bigger meta-lesson: the paper gives no theoretical proof here, only ablations. Provable, derivable and experiment-only claims carry different weights; keep them apart.
Where this segment sits The last three segments covered how; this one covers why they dared. The only place in the course that truly answers “why does this work,” and Topic 1's best showcase of “if you're not sure, check; never make it up.”
The one line to keep: “the 32,768 never existed.”

Script

Scroll to “⭐ Why so much compression?”.

MLA saves 56.9×. Sounds like magic. And every trick has a sleight of hand.

Take it apart, and most of it isn't compression at all.

Point at the two-cuts figure.

Cut 1: 32,768 values, 7,168 degrees of freedom

The standard approach stores 32,768 values per token per layer. 128 heads, each 128 dims of K and 128 of V. Multiply.

Ask; give the room three seconds.

Here's my question: where do those 32,768 values come from?

All come from the 7,168-dim input. K is a 7,168 by 16,384 matrix on it; so is V.

And a 7,168 by 16,384 matrix has rank at most 7,168.

Stop. Next line is the core. Slow down.

So the 32,768 dims always live in a subspace of at most 7,168.

Store just the 7,168 input values, and K and V rebuild bit for bit.

Provable, not a hunch. 4.57× of standard storage is pure redundancy. Cutting it isn't compression, just not being wasteful.

Pre-empt “why did nobody do this before?”

Why did nobody just store the input before? Recompute: each new token redoes all earlier tokens' K and V. Memory saved, compute paid.

MLA's real feat: that recompute almost never happens. Last segment's matrix absorption. Only now is the 4.57× collectible.

Point at the second cut.

Cut 2: cap the rank at 576

What's left, 7,168 to 576, is 12.44×. Real, and lossy. But it compresses rank, not value count.

Switch to the bus figure.

Think of the 512 as an information bus. All 128 heads read off it.

Each head reads via its own matrix, getting something else. No head diversity is lost. But nobody can get what isn't on the bus.

MLA's bet in one line: all 128 heads need no more than 512 information channels.

Point at the gold dash.

This settles last segment's question: why the 64-dim position path isn't on the bus. The dashed line shows it branching off the input, bypassing the bus. Squeeze position onto the shared bus, and matrix absorption breaks.

The key part: here the room decides whether to trust you.

The paper's case? It doesn't make one

Note: “two cuts” and “bus” are my derivation, not the paper's.

DeepSeek's paper gives no theory here. No rank analysis, no information theory. Just “here's what we do.”

Pause. Honesty carries weight.

But Appendix D has a hard head-to-head ablation: the real answer.

Point at the green box, “⭐ The paper's case”. Cover only this box; leave “Expand: both ablation tables” closed.

A large 250B MoE, same architecture, only attention swapped. Standard caches 860K values per token; MLA, 34,600. That's 4%.

And on quality, BBH, MMLU, C-Eval and CMMLU, it wins all four.

Pause. This line founds Step 2.

No pushover: it beats the baseline that saves no cache and has the best quality.

Only if someone looks skeptical, open that block and walk its four-column table with the next three lines.

Here are the four scores: BBH 46.6 to 50.7, MMLU 57.5 to 59.0, C-Eval 57.9 to 59.2, CMMLU 60.7 to 62.5.

The small model wins three and loses one. C-Eval drops from 51.6 to 50.9. Point that cell out honestly; it's red for exactly this.

The paper first compared standard attention, GQA and MQA. On MMLU: standard 45.2, GQA 41.2, MQA 37.9. The old ways of saving cache really cost quality.

Last, most uncomfortable question.

57× compression, so why is quality better?

The paper doesn't explain. Two guesses of mine; hear them as guesses.

Guess one: the low-rank bottleneck may regularize, one gap for all heads forcing cleaner representations. No controlled test: “maybe.”

Guess two (likelier, I think): savings spent elsewhere. Freed cache let attention widen to 16,384, 2.29× the residual stream.

The real contest may be not “compressed vs not,” but “bottleneck + extra-wide heads” vs “no bottleneck + regular heads.”

Wrap up: 512's three roles.

Last: 512 isn't arbitrary. One knob sets three things.

One, cache size: what's stored. Two, expressiveness: the rank cap. Three, if “never compute K” pays: narrower bus, smaller folded matrix.

RoPE's 64 dims stay off the bus precisely to protect number three. One number, three things, locked.

Three claims, three weights: keep them apart

The easiest way to crash: blurring these three ① What can be proven: “32,768 values carry at most 7,168 degrees of freedom.” That's linear algebra: rank can't exceed the input dimension. Say it flatly.
② What can be derived: “so 4.57× of the 56.9× is pure redundancy” and “512 is a shared information bus.” They follow from ①; the logic holds, but say “this is the course's breakdown, not the paper's.”
③ What only experiments support: “rank 512 is enough” and “MLA beats MHA.” These have only ablation evidence, no theory at all. Prefix: “the paper's experiments show.”
Presenting ③ as ① is the one fatal mistake.

Likely questions

If storing the 7,168-dim input suffices, why not store 7,168? Why squeeze to 512?

Storing 7,168 fixes memory, not compute.

Storing the input, each step recomputes all past tokens' K and V. To avoid it, absorb the up-projection into Q; the folded matrix scales with bottleneck width.

At 7,168, the folded matrix is too big; at 512, much smaller. So 512 isn't just memory; it makes “no recomputation” pay off in compute.

(A derivation, not in the paper. Say so.)

Capping rank at 512 loses information. How do we know the loss doesn't matter?

We don't. It's a bet the paper doesn't defend.

The only evidence is ablations: at two scales, only attention swapped, MLA is no worse than MHA; the large model is better across the board.

That's validation after the fact, not proof in advance. Say so; much of deep learning design stands on this footing.

Don't invent a reason to round off the story. “The paper gives no theory here, only experiments” convinces more than any plausible explanation.

How was 512 chosen? Tuning?

A ratio, not an origin. It says “d_c is set to 4 times the per-head dimension, and the position path to half of it”. V3: 128 per head, hence 512 and 64. V2, same ratio.

How they picked it, the paper doesn't say. I've seen no public sweep records, so I'm not sure.

You can say where it lands: the paper works out that MLA's cache equals GQA with only 2.25 groups. Smaller than three-group GQA, yet quality beats MHA.

Is this “rank” idea the same as LoRA?

Same root idea, completely different use.

What they share: “a big matrix doesn't use its full rank, so a narrow waist is enough.”

How they differ: LoRA compresses weight updates, for less fine-tuning optimizer state. MLA compresses activations: per-token intermediates, for less KV cache.

One acts in training, the other in inference.

If the 4.57× cut is free, why doesn't every model do it?

It isn't free. It needs supporting pieces. Alone, it costs compute (you recompute). It pays off only with matrix absorption.

And absorption in turn requires RoPE to be isolated. So these three are one bound design. Pull out any one and it stops working.

That's also why MLA is an architecture, not a trick.

What not to say

Four red lines ① Don't present the “two cuts” breakdown as the paper's. It's the course's derivation. Add “the paper doesn't put it this way.”
② Don't give rigorous definitions of rank, subspaces or linear maps. Not a math department. “Put 7,168 numbers into a matrix, and what comes out has at most 7,168 kinds of variation” is enough.
③ Don't skip the small model's C-Eval drop. Reporting three cells of four is the worst dishonesty; people read the table.
④ Don't compare MLA with other labs' approaches. (GQA variants, KV quantization, sliding windows, and so on.) Another course; it invites unbacked claims.
Pacing Seven minutes, four parts: Cut 1, 2 min → Cut 2, 2 min → the paper's evidence, 2 min → why it's better, 1 min.
The peak is “the 32,768 never existed.” Say it twice: at the start as a hook, after Cut 1 as the conclusion.
After “the paper gives no theoretical proof,” always pause two seconds. In that pause the room starts believing your earlier numbers.
Short on time? Cut the two guesses about why quality is better. The first three parts are the bones.
05⁺⁴

Scoring: from dot products back to the residual stream

⏱ 7 min
What this segment must leave behind So far: how Q, K and V get prepared. This segment: what they do once they're ready.
In one line: attention gives each position a probability distribution that decides how much it takes from each earlier position.
Last puzzle piece: the 128 heads exchange information at o_proj, for the first and only time.
Final minute: the weights. Five blocks per layer. The largest is irrelevant to memory; the smallest sets the whole memory bill.
Where this segment sits The only “mechanism” segment in Step 2. Everything before it was about saving. After hearing only about compression, the room asks: how is it used? This answers.
It also sets up the next segment's 4 TiB. After these five steps, “why it's quadratic in sequence length” explains itself.

Script

Scroll to the five-step “⭐ Q, K, V in hand: how attention plays out”.

Start with an intuition, not matrices.

Reading a sentence, at the fifth word, you glance back at the first four and judge which matter most. That's scoring.

One difference: 128 heads, 128 sets of criteria looking at once.

One second to let it land.

① Dot product: who should look at whom

Dot the current position's Q with every position's K. Same direction, large: “look closer here.” Unrelated or opposite: small or negative.

Each head does this alone: 128 square matrices, each 128K by 128K.

② Divide by √192: no magic

Summed over 192 dims, the dot product grows with dimension.

Raw into softmax, it saturates. One position takes nearly all weight; the rest and their gradients vanish. Dividing by √d fixes the range.

Below: collapsed block, Expand: two details of “÷ √192”. Skip it. Open only if someone pushes on the constant.

Two things to get exactly right, neither to unpack. One: the paper says “we suspect,” not “we prove”; a heuristic. Two: in V3's code the factor isn't 1 over √192. Long-context extrapolation adds a factor: 0.1352, not 0.0722. Same principle, different constant.

③ Mask the future: correctness, not optimization

Position five can't see six. Upper triangle set to minus infinity; softmax zeroes it.

Stress it; it's often filed as a detail.

Saves nothing; it's for correctness. Unmasked, training copies answers; beautiful loss. At inference later tokens don't exist: collapse.

④ Softmax: scores into “how much to take”

Point at the 8-by-8 miniature matrix, then the bars on its right.

Normalize each row; every row now sums to exactly 1.

So each row is a probability distribution. Plainly: “I take 29% of my information from position 1, and 47% from position 5.”

Last segment's cliffhanger, closed: why the 128 heads can't merge here. Softmax is nonlinear, per row, per head; it can't fold into one matmul.

⑤⑥ Weighted sum, then concatenate and exit

Those weights sum V. Each head emits a 128-dim vector.

Concat 128 heads: 16,384. o_proj folds it to 7,168, into the stream.

Pause two seconds after this line.

o_proj is the layer's largest matrix: 117M parameters, about 60% of an MLA layer. Big and important: the only place the 128 heads exchange information. Before it, each head computes alone.

Point at “A different kind of compute.”

An aside, since it recurs: a different kind of compute. Projections are big, regular, fixed-shape matmuls; accelerators love them. Scoring: 128 small matmuls batched, a row-wise reduction mid-way.

And it's quadratic in sequence length: double it, 4× the work. The number that will startle you in a minute grows out of this.

Close: line up one MLA layer's weights

Scroll one screen to the bar chart “Five projections: one layer's size.” Let the room see the lengths before any numbers.

Lots of MLA, but a layer has only five weight blocks: the five bars.

Two seconds. Let the blue bar speak.

The top one, o_proj, outweighs the other four combined: 63%. The flip side of what I just said: it's where the 128 heads exchange information. One end 16,384 wide, the other 7,168: both wide, so it's biggest.

Finger to the short yellow bar at the bottom. Barely visible.

The bottom one, kv_a_proj, is just 4.13M. Two percent. The biggest is 28× larger.

Next line is why this figure exists. Slow down.

Yet this shortest bar sets the entire KV cache bill. The cache holds the 576 values it writes. The other four add not one byte.

The largest weight block has nothing to do with the memory bill. The smallest one sets all of it. The figure shows it at a glance.

The five add up to 187M. Times 61 layers: 11.41B. In a 671B-parameter model, that's all of attention: about 1.7%. The rest is in the MLP and MoE. That's Steps 3 and 4.

Likely questions

Why divide by √d? Why not by d, or something else?

A footnote in the paper argues heuristically: if Q and K components are independent, mean 0, variance 1, their dot product sums d terms: mean 0, variance exactly d.

Variance d: standard deviation √d. Divide by √d, back to 1. Dividing by d overshoots: scores go flat and attention becomes roughly an average.

(Wording: the paper says “We suspect.” Useful intuition, not a theorem.)

Why 192? Isn't each head 128-dim?

128 is V's per-head dimension, not Q's or K's.

Each Q and K head is concatenated: 128 dims without RoPE plus 64 with, 192 in total. Scoring dots all 192 at once, so the denominator is 192.

In the code it's q_head_dim, which equals qk_nope_head_dim + qk_rope_head_dim.

128 matrices of 128K × 128K: fit in HBM?

No. That's the blowup in the next section.

So nobody stores it in full. Compute tile by tile, accumulate into the output, discard each tile.

(Stop there. Tiling is Topic 2.)

Mask: training only, or inference too?

Both, in different forms.

Training, and inference's first step (reading the whole prompt), use the full upper-triangular mask: all positions computed at once.

Generating one token at a time needs no explicit mask. One new Q; its KV cache holds only history. The future doesn't exist yet.

Do the 128 heads never talk on the way?

Within a layer, yes. Projections to weighted sum, 128 paths run fully parallel.

The only meeting point is o_proj. 16,384 dims in, 7,168 out; every output component combines all 128 heads.

Plus the residual stream: the next layer's Q, K and V regrow from the merged result. So “the heads don't talk” holds only inside a layer.

All of attention is only 11.41B, 1.7% of 671B. So it's not that important?

Few parameters ≠ low cost. Correct this on the spot: the course's most misleading intuition.

Attention's three bills are fully decoupled: weights 11.41B (tiny), KV cache 8.58 GiB (linear in length), compute at 128K 81.8% of the forward pass (quadratic).

MoE is the opposite: hundreds of billions of parameters, 37B active per token, none tied to length.

So “share of parameters” measures only the weights column. Next segment's three numbers knock it out.

o_proj is so big. Couldn't it get a narrow waist too, like Q?

Structurally, yes: low-rank factorization works on any matrix. But V3 didn't, and the paper doesn't say why. So this is speculation; say so.

One plausible reason: o_proj is the only place the 128 heads exchange information. A waist there squeezes all 128 heads through r dims, cutting exactly the expressiveness this layer can least afford.

(If pressed, stop there. “The paper doesn't say; my guess” beats inventing.)

What not to say

Three red lines ① Don't present “divide by √d” as rigorously proven. The paper itself says “we suspect.” Don't upgrade it.
② Don't explain how FlashAttention tiles. Not kernel engineers; “it never stores the whole thing” is enough. Topic 2.
③ Don't wave softmax off as “just a normalization.” “Each row is a probability distribution” is this segment's core. It explains what attention does and why the heads can't merge back.
Pacing Seven minutes, four parts: intuition and dot product (2) → scaling and masking (2) → softmax to exit (2) → five bars and the weights (1).
Two high points, both figures. Don't fill either with talk. First, the 8-by-8 miniature matrix. At softmax, point at the highlighted row, then the bars on the right. “It adds up to 100%” at a glance beats three sentences.
Second, the five-bar figure. “The longest bar is irrelevant to memory; the shortest sets the whole memory bill.” Pause two seconds, then move on.
If time is short, cut the “two details” paragraph, but if asked about the constant, answer exactly as the box says. Don't cut the five bars. They set up Steps 3 and 4's “the parameters are all in MoE.”
This segment's two expanders Both stay collapsed: two details of “÷ √192” and each matrix's shape and role. Open the first only if pushed on the constant, the second only if asked for exact numbers. The five bars say it all.
06

Three blowups

⏱ 3 min
What this segment must leave behind After five segments of accounting, close Step 2 with three memorable numbers.
In one line: the 57× saving is beautiful, but what decides whether long context is possible is the 4 TiB that is never stored.
The numbers escalate. Demo the last one live with the slider.
Where this segment sits Step 2's curtain, and the course's first emotional beat. After five dense segments, the room needs a takeaway.
Three minutes, three numbers, no mechanisms. Mechanisms are done; let the magnitudes speak.

Script

Scroll to the four-scheme KV cache figure, red dashed line visible.

That's Step 2. I'll close it with three numbers, each bigger than the last.

Blowup one · 57×

Same setup: 128K context, batch 1, bf16. Standard MHA's KV cache is 488 GiB. MLA's is 8.58 GiB. That's 56.9× smaller.

Point at the red dashed line.

This red line is one TPU v7 device: 94.74 GiB. MHA needs over five devices, before a byte of weights. MLA leaves 90% of one device free.

Pause. First “doesn't fit.”

Blowup two · 4 TiB

Scroll to “⚠️ The real blowup: the score matrix,” the red-square grid.

That one is stored. There's something far scarier, and it never shows up in any memory report, because it's never stored at all.

Attention scores every token against every token. That's a 128K-by-128K matrix, and there's one per head. 128 of them.

It comes to 4.00 TiB. One layer, one pass.

Two seconds, then compare.

Two comparisons to land it. First, all weights are 1.22 TiB; this one-layer, one-pass intermediate is over three times that. Second, at 94.74 GiB per device, it's 44 devices.

Point at the red squares: “Each square = one TPU v7 device.”

So FlashAttention isn't a speed-up. It makes this possible at all. The idea: never materialize the whole matrix; compute in tiles, accumulate, throw each away.

Blowup three · 81.8%, drag the slider

Scroll to the curve. Hand on the length slider.

The third number I won't say. I'll drag it.

Two parts of attention are quadratic in sequence length: scoring and the weighted sum. All else, projections, MLP, MoE, is linear.

Drag to 4K. Then push right notch by notch, reading each number.

At 4K, the quadratic term is 12.3% of forward compute. No problem.

32K: 52.9%. More than half. 128K: 81.8%. Push it to 1M: 97.3%.

Hands off. Three seconds of silence: the course's peak.

Notice: the model didn't change. Parameters, layers, experts: same. Only the input grew, and the bottleneck moved elsewhere.

At this scale, compute is driven less by parameter count than by sequence length.

Go to the green box “What happened next,” then Step 3.

With this curve, the next move is obvious: if attention eats over 80% of compute, stop every query looking at every token.

Skipping tokens, which to see less, at what cost: a whole lecture, Topic 3. This one computes why change is unavoidable: the line on screen.

Likely questions

4 TiB is bigger than all the weights. Is that even possible? Did you get it wrong?

It's right. That's why it's worth telling. Weight size depends only on the model, not input length. The score matrix is quadratic in sequence length.

With a long enough sequence, an “intermediate result” bigger than the model is inevitable. 128K is far past the crossover.

(Write the formula: 128 × 131,072 × 131,072 × 2 B. Really write it; people checking it beats ten assurances.)

Never materialized, so why “4 TiB”?

It explains why something like FlashAttention must exist.

“Implement it from the textbook and you need 4 TiB.” A counterfactual, but the design constraint's source. Without it, FlashAttention sounds like a speed trick, as if things would just run a bit slower without it.

Not a bit slower. It wouldn't run at all.

12% to 97%: measured or computed?

Computed from FLOP formulas, not measured. Say so.

Numerator: attention's two quadratic terms; denominator: the pass's total FLOPs. Parameters: V3's public config. The line sweeps that ratio over sequence length.

(Versus measurement: memory access, kernel efficiency, fusion mean time share won't match FLOP share exactly. But direction and magnitude are right, which makes it useful.)

How do the three numbers change with batch?

The first two scale with batch; the third barely.

KV cache and score matrix are per sample: batch 8 is eight times, so 4 TiB becomes 32 TiB.

The share is a ratio; batch scales both sides, cancels. Attention's compute share depends only on sequence length, not batch. Mention it: people think batch tuning gets around it.

So how did DeepSeek solve it later? What did V3.2 and V4 do?

Always comes up. It's on the page; give three sentences, then stop.

One: in V3.2 each query attends only to its 2,048 most relevant tokens, not all of them. At 128K that's 1.6%; quadratic becomes linear. The 2,048 is in V3.2's own config, field index_topk.

Two: V4 hybridizes two kinds of attention: one compresses, then selects; the other compresses harder, no selecting, sees everything. Two kinds, not three.

Three: at a 1M-token context, V4-Pro needs 27% of V3.2's per-token inference FLOPs and 10% of its KV cache. Source: the V4 paper's abstract.

(Then stop. How it selects, wrong picks, how training teaches picking: Topic 3's substance; opening any costs twenty minutes. Close with: “This lecture only computes why change was unavoidable. How to change it is a whole section next lecture.”)

What not to say

Three red lines ① Don't explain how FlashAttention tiles. Only “it never stores the whole thing.” Tiling is Topic 2; it turns three minutes into ten.
② Don't present blowup three as one static number. 81.8% alone has no punch. The punch is the contrast: nothing changed, the input just got longer. Really drag the slider, 4K to 1M.
③ Don't go into Topic 3's solutions. This establishes the problem; answering it steals Topic 3's opening.
Note: the page's closing green box now states DSA's 2,048, V4's two kinds of attention, and the 27% and 10%. After-class reading, not script. Read at most its first sentence, “stop every query seeing every token,” then hands off the screen. If pressed, use the Q&A entry below: three sentences, no further.
Pacing Three minutes, a minute each, each slower than the last. 57× fastest (a recap). 4 TiB medium, with room for pauses. The slider slowest.
The high point: three seconds of silence after the slider hits 1M. Don't fill it.
If time is short, cut blowup one (just covered), never two or three. They're the door to everything after.
07

The dense MLP's four hardware-friendly properties

⏱ 7 min
What this segment must leave behind In one line: the compute MoE saves is paid for with these four properties.
Skip them, and MoE sounds like a free lunch. Then every problem in the next segment looks self-inflicted.
This segment is the control group. The dense MLP isn't hard; making the room remember “these things cost money” is.
Second takeaway, the closing chain: on “why aren't the first three layers MoE,” the popular answer and the paper's answer differ. It's the course's model for handling explanations that sound smooth.
Where this segment sits V3's first 3 layers are ordinary MLPs (first_k_dense_replace = 3), so the control group isn't hypothetical: it's three real layers.
A structural freebie too: every MoE expert is the same three-matrix SwiGLU. Nail it here; next segment only tracks width and count. Saves three minutes.

Script

Scroll to Step 3, the whole three-matrix SwiGLU figure on screen.

Step 3, the dense MLP. Simplest-looking, but the control group for all MoE ahead. Remember it.

Structure: two widen, one projects back

7,168 dims in, widened to 18,432, projected back to 7,168. That's all.

Point at the gray bars: “drawn to scale.”

Widening happens twice: the one thing to explain. Upper copy through SiLU: the valve. Lower as is: the flow. Multiply elementwise, project back.

That's SwiGLU: three matrices, each 7,168 by 18,432. 396M parameters a layer, 1.19B in all.

A hand goes up here almost every time; good question.

The inevitable question: why three matrices?

If nobody asks, I will. A classic MLP has two: widen, nonlinearity, project back. So what's the third for?

It makes the nonlinearity adaptive.

In the classic form, the nonlinearity is fixed. Every input and channel squashed by one hard-coded rule; training can't change it.

SwiGLU makes “which channels pass, how much” learned and input-dependent. A channel may be open for this token, shut for the next.

Easiest line to get wrong. Pause first.

The multiply buys expressiveness, not SiLU.

Scroll to the eight-bar ablation. No numbers; let them see two piles.

One figure nails it down

SwiGLU's original experiment: eight variants, matched parameters and compute, same steps, perplexity compared. Longer is better.

Point at the two red bars on top first.

Top: “swap the activation, still two matrices.” GELU and Swish. Both left of the baseline: slightly worse than the original ReLU.

Hand down to the five blue and yellow bars.

Bottom: “add a multiply, three matrices.” Five variants; all win.

Let it speak.

The gap between groups beats any gap within one. So the dividing line is clear: a multiply or not, not which activation.

Point at the yellow bar, Bilinear: the crux.

Most telling: Bilinear. No activation on either path, just two linear maps multiplied. It still beats the two on top with their chosen activations.

No activation, just a multiply, and it wins.

Point at GEGLU at the bottom; heads off a misreading.

One more: SwiGLU 1.636, GEGLU 1.633. Basically a tie, GEGLU even a hair better. So V3 didn't pick SwiGLU as the best. Within this group, any pick is about as good. What matters is being in it.

Point at “Honestly: even SwiGLU's authors can't say why it works.”

Must add: the author says he can't explain why gating works. His words: “We offer no explanation as to why these architectures seem to work; we attribute their success, as all else, to divine benevolence.”

So “the multiply adds interactions” is plausible intuition, not proven cause. The table shows only: gating really wins, not via activation.

And it doesn't cost more

Scroll to the three-row figure, red “parity line” on screen.

Most missed: an extra matrix sounds like 50% more parameters. No.

Point at the three rows, top to bottom.

Row one, classic: two matrices, 4× by convention, 28,672: 411M.

Row two, the paper's rule: three matrices, inner width times two-thirds. Three cells then match the two above: still 411M. Parameters and compute exactly equal; the rule is in the paper, not our fudge.

Row three is what V3 actually uses: 18,432, a bit below the rule. 396M.

Slow down.

One more matrix, 3.57% fewer parameters. Gating's power is free.

If asked “So that's how they back-solved 18,432?”, careful: see the Q&A below.

You can say: 18,432 lands near that rule, even leaner. You can't say: “so they set it by that rule.” The paper doesn't say so. Most models today ignore the rule: Llama 3 and Mistral use 3.5×, Gemma 2 a full 4×, all far wider. If anything, V3 is closest to it.

Four properties: the real takeaway

Scroll to “Four hardware-friendly properties”; point at each.

Note these four. In three minutes, MoE will take them away one by one.

One: every token takes exactly the same path. Shapes are static, predictable, known at compile time.

Two: one big matmul. High utilization: the MXU's favorite shape.

Three: no communication to assign work. No scheduling.

Four: the compiler plans everything ahead; no runtime uncertainty.

Pause, then the takeaway.

MoE keeps none of these four.

So MoE is no free lunch. It saves a lot of compute, paid for with these four. Every problem in the next segment maps back to one of them.

Scroll to the last figure, three boxes: the close, and a demo of method.

The last inevitable question: why the first three layers?

There's a widely repeated answer. I'll say it, then why it can't be the answer.

The popular version: layers near the input do generic work, basic lexical and local syntax every token needs; nothing to specialize.

Two seconds: it sounds so smooth.

Perfectly reasonable. But it's not the reason DeepSeek gave.

Point at the three boxes, left to right.

The papers show a chain. It began in DeepSeekMoE, early 2024, which kept the first layer dense, with a reason. In their words: they observed that load balancing converges especially slowly in the first layer.

Training stability, not “generic versus specialized.”

V2 also kept the first layer, saying only “Following Dai et al. (2024).”

Then V3 went to three layers, the paper stating the fact with no reason. Why 1 became 3, nothing public says.

Two seconds. Core takeaway next.

Say “the original public reason was training stability,” not “V3 picked 3 layers as the most generic.” That's our unbacked guess.

“Load balancing converges slowly” → “early layers are generic” swaps a training phenomenon for a story. Only the first is checkable.

Point at the blue line at the bottom. One line.

A bonus: V3's deployment puts these shallow dense MLPs apart at 1-way tensor parallelism, to save that communication. That backs the four properties: dense pays off in real money when deployed.

Likely questions

Why three matrices? Aren't two enough?

The script answers it; this goes one level deeper for follow-ups.

Short answer: the third matrix lets each token decide how far the valve opens.

“Why not just pick a better activation?”: even the best activation is a fixed one-variable function. Its per-channel behavior is frozen once training ends. Gating gives an input-dependent, per-channel multiplier. A different kind of thing.

“Doesn't one more matrix mean 50% more parameters?”: only at the same inner width. V3 shrinks the width from the conventional 4× to 2.57×, so it's 3.57% smaller than the classic form. (3 × 7,168 × 18,432 = 396.36 M vs 2 × 7,168 × 28,672 = 411.04 M)

“Multiply adds quadratic terms”: rigorous?

No. Say so: it's the easiest point to slide past.

Just one of SwiGLU's paths is linear. The other passes SiLU, so the product isn't strictly quadratic.

The strictly quadratic one is the same paper's Bilinear variant: no activation on either path, exactly (xW) ⊙ (xV).

And Bilinear still beats every activation-only variant (1.648 vs 1.679 / 1.683). That's the real evidence: the multiply matters, not the activation.

(Wording: “the multiply introduces interactions between input components”: fine. “Output quadratic in x”: only Bilinear. SwiGLU: “it keeps multiplicative interactions.”)

SiLU, Swish, GLU, SwiGLU: how do these names relate?

Three nested layers, three sentences:

SiLU is an activation function (Swish, renamed). GLU is the structure “multiply two paths”, whatever the activation. SwiGLU is the GLU with SiLU as its gate.

(Unless pressed, skip. “Two paths + one multiply” beats a family tree of names.)

Are the three dense layers big? Worth a section?

In parameters, negligible: 1.19B, about 0.2% of 671B.

But the section is about the four properties, not 0.2%. A free teaching aid: fully explainable in one pass, and the same three-matrix SwiGLU as the MoE experts.

Seven minutes here saves three next segment and buys a control group.

Is there anything else behind 18,432, beyond parameter parity?

It meets three constraints at once. All are checkable on the spot; the paper claims none.

① Most worth mentioning: 9 × 2,048 = 18,432. 2,048 is one MoE expert's width, and V3 activates 8 routed plus 1 shared expert per token: exactly 9.
So a token's MLP in the first 3 dense layers is as wide, matrix for matrix, as its 9 active experts in the other 58 layers. Per-token MLP compute is an almost flat line from first layer to last.

Keep the word “almost.” An MoE layer also has a router, 7,168 × 256 = 1.84M, 0.46% of 396.36M. Say “the experts match exactly, plus 0.46% for the router.” Volunteer tails like this before someone drags them out.

② Aligned to the accelerator: 18,432 ÷ 256 = 72, divides evenly. The paper's 2/3 rule gives 19,114.67, not even an integer.

③ Close to the 2/3 rule, but not on it: V3 is 0.643×, the rule is 0.667×.

(Wording: “equal” is fact; “so that's how they chose it” is interpretation. The V3 paper gives only the two numbers, intermediate_size and moe_intermediate_size, never saying the relationship was deliberate. Comfortably say “I'm not sure.” Separating computed from guessed beats a neat answer.)

Why the first three layers? Not one, or five?

The script covers it; compressed version:

Dense early layers began in DeepSeekMoE. The reason: “load balancing converges especially slowly in the first layer,” a training-stability issue.

V2 kept 1 layer. V3 went to 3, with no reason. Why 3 and not 1 or 5 isn't in anything public.

(Don't fill the hole with “early layers are more generic.” Smooth, but a semantic story replacing the training phenomenon reported. Just say, “they didn't explain this step.”)

Are MoE experts structurally the same?

Identical, just much narrower.

Same three-matrix SwiGLU, inner width cut from 18,432 to 2,048 (moe_intermediate_size in the config).

So next segment needn't relearn the structure. Just watch width and count.

What not to say

Four red lines ① Don't treat this as “a quick pass over dense.” The 0.2% of parameters really doesn't matter, but the four properties are the next segment's control group. If they fall flat, MoE has no tension.
② Don't compare activation functions. The ablation figure says the opposite: “swapping the activation doesn't help.” “SiLU beats GELU” inverts the conclusion. The point is the gating structure, not the shape of the S-curve.
③ Don't present 18,432 as known design intent. “9 × 2,048 is exactly equal” is checkable fact. “So they set it by that” is interpretation. Don't present inference as fact.
④ Don't explain the dense first three layers with “early layers are more generic.” Once taught that way; no longer. The paper's reason is “load balancing converges especially slowly in the first layer,” a training phenomenon. Why V3 went from 1 layer to 3 is never explained. Turning “not explained” into “because they're generic” is the one place here that would truly mislead.
Pacing Seven minutes, four parts: structure (1) → “why three matrices” (2½, two figures) → four properties (1½) → the first-three-layers chain (2).
“Why three matrices” will almost certainly be interrupted, so raise it yourself. Posing it is calmer and you control the length.
Two takeaways. Leave space after both. First, “MoE keeps none of these four”: the switch from “dense is boring” to “oh, that was a setup.”
Second, the final chain: what you can say, what you can't, and a question the paper never answers. It shows how the course treats smooth explanations.
If time is short, cut the ablation bars other than Bilinear (read the two groups' conclusions), never the four properties or the final chain.
This segment's three expanders “The formula,” “full table; does it hold at 671B?” and “18,432 meets three constraints” stay collapsed. Normally none. Open the first for the formula, the second if the figure is doubted, the third if asked about 18,432.
The second also has a width table for today's mainstream models (Llama 3 / Mistral / Qwen3 / Gemma 2), each read from its own config. Open it on “isn't this just a DeepSeek thing?”
08

MoE basics · “Experts” aren't that kind of expert

⏱ 6 min
What this segment must leave behind In one line: MoE decouples “how big the model is” from “how much each token computes.”
On V3 that's an identity, not a metaphor: each token's FLOPs in that layer exactly equal last segment's plain dense MLP layer.
Plus one misconception to correct: experts don't split by subject. Uncorrected, it skews how routing and load balancing land.
Where this segment sits Last segment set up four good properties; this one takes them away one by one. Savings first, and the room will listen to the cost.
Don't re-teach the structure. An expert is last segment's three-matrix SwiGLU, intermediate width cut from 18,432 to 2,048. One sentence.

Script

Scroll to Step 4, the dense-vs-MoE side-by-side figure.

MoE has exactly one core idea. State it first; everything follows from it.

The core: decouple parameters from compute

In the three dense layers, each token runs all the weights. More parameters, more cost per token: welded together.

MoE sets out to break that weld.

Split one wide MLP into many narrow ones. V3: 256 routed experts plus 1 shared. Same three-matrix SwiGLU, intermediate 18,432 → 2,048.

Scroll down to “⭐ Each token takes its own path” (below the red-and-green figure).

Per token, a router picks just 8 routed experts. Plus the shared expert every token passes: 9. The other 248? This token never touches them.

High point. Slow down.

⭐ The exact match

Keep going to the nine-segment bar at the end of 4a, both red dashed lines on screen.

The math: one layer holds 257 experts, 11.32B parameters. But each token activates only 9: 396,361,728 parameters.

I gave you this exact number in the last segment, for the dense layer.

Two seconds. Let them catch it.

Not close. Identical. Because 9 times 2,048 is exactly 18,432.

Sweep along the two red dashed lines.

Nine segments end to end equal the dense bar: by value, not drawing.

Flatly: per token, an MoE layer's expert compute exactly equals one dense MLP layer. But it holds 28.6× the parameters.

Half a second, but say it: it's the course's credibility.

Strictly, a hair more: MoE first runs the router matrix, 7,168 by 256. Next to 396M, that's 0.46%. Precisely: not one extra expert FLOP, plus a half-percent triage fee.

Takeaway. Pause.

That's “decoupling parameters from compute,” literally. The model grows; the per-token bill stays put.

Scale it to the whole model

58 of 61 layers are MoE: 656B of weights, 98% of the 671B.

Each token uses only 3.5% of them. “A 671B model that computes only 37B per token”: this is where that line comes from.

Scroll back up a little to the three “Fine-grained” cards above it.

Why small and many? Same activation budget: 1 of 8 big experts gives 8 options; 8 of 256 small ones, ~10¹⁴ combinations. More, smaller experts, more ways to split the work: fine-grained MoE's whole case.

The one thing to correct on the spot

Change tone: a correction, not an add-on.

One last thing, and it matters: fixing an idea that just took root.

“Experts” aren't that kind of expert.

Hear “expert” and you picture doctors, lawyers, translators: a split by subject, one for code, one for Chinese. No.

Just 256 trained narrow nets, numbered 0–255, meaningless to us.

Don't stop at “I say it's not.” Go to the evidence: once a bare claim, now data. Back up to the red-and-green figure under the dense-vs-MoE one. Left panel only, first.

The Mixtral paper measured it: took routing apart and counted.

They fed in math papers, biology abstracts, philosophy papers, and checked where each domain's tokens went. If experts split by subject, the three should differ sharply.

The three distributions nearly overlap. The paper: “we do not observe obvious patterns in the assignment of experts based on the topic.” Just before, “surprisingly”: the authors were surprised too.

Right panel's three green bars.

Then what does routing follow? Ask instead: do adjacent tokens share an expert?

At random, one time in eight: 12.5%. Layer 0: 14.1%, basically random. But middle layers hit 26.4%, over twice random.

So routing follows local structure, not subject. The paper's examples: “self” in Python and “Question” in English always go to the same expert; indentation tokens “always” go to the same experts.

Takeaway next.

And that turns straight into a systems problem; that's why we care.

Under expert parallelism, locality overloads a few experts. Flip side: you can cache. One property: load balancing's foe, caching's friend.

Point at the scope line under the figure.

Scope: Mixtral: 8 experts, top-2; V3: 256, top-8. Numbers don't transfer. It proves the negative: measured, “split by subject” didn't show up. How V3's 256 split work: no public analysis this rigorous that I've found.

Close with triage, then 4b.

Think triage desk: it knows no medicine, only “this symptom goes to that door.” Behind the door? Not its concern, nor ours.

Why correct this? Believe in subject experts and the rest seems pointless: load imbalance, auxiliary losses, group limits. “Just send each job to its expert. How hard can it be?” Hard because there is no subject.

Likely questions

9 × 2,048 = 18,432. Did the authors design them to be equal?

The equality is fact; the intent is a guess.

Certain: all four are in V3's public config (intermediate_size, moe_intermediate_size, num_experts_per_tok, n_shared_experts). Anyone can check.

Not certain: no public source says they picked these values to make them equal. Say “these two numbers are equal,” not “they designed it to be equal.”

(Still a useful yardstick: whatever MoE costs beyond dense, it isn't FLOPs. Routing, communication, load imbalance, memory: every later problem traces here.)

Why have a separate shared expert? Why not pick one of the routed experts?

It solves redundancy, not division of labor.

Every token needs grammar and common knowledge. Without a shared expert, all 256 routed experts would relearn those basics: 256 copies of the same knowledge.

Put them in 1 expert every token passes, and the routed experts are free to specialize.

So the 248 inactive experts cost nothing?

No compute, but their full share of HBM. A top MoE misconception.

Weights stay in HBM throughout. The next token may need any; no loading on demand.

So MoE saves compute, not memory. Every byte of the 671B stays. That's Segment 11's math.

Why sparsify the MLP and not attention?

Two reasons.

One: the MLP holds most parameters. MLA layer 187M, dense MLP layer 396M, MoE layer 11.32B.

Two: the MLP is naturally many independent problems. It acts on each token alone, no cross-token interaction, so tokens can safely take different routes.

Attention exists to let tokens see each other. It can't be split.

Has nobody studied what the experts learn?

Yes: the paper I just cited, Mixtral, arXiv 2401.04088, §5.

One negative, one positive. Negative: no obvious expert–topic pattern. Positive: routing tracks local syntax; mid-layer neighbors share an expert twice as often as random.

Wording: “No obvious topic pattern observed” ≠ “proven experts have no interpretability at all.” The second, far stronger, I'm unsure of; don't pin it on the paper.

(Safe version: “Measured, routing follows local syntactic structure, not subject. The numbers mean nothing; don't expect to read expert 137's job.”)

Does the Mixtral result carry over to V3?

Very different architectures; two-part answer.

The negative result carries over. “Experts split by subject” is something the audience reads into the word “expert,” and the only public measurement refutes it. Absent contrary evidence, don't let it stay in the room.

The specific numbers don't. Mixtral picks 2 of 8; V3, 8 of 256, an order of magnitude sparser. V3's version of 26.4%: unmeasured.

(If pressed, “did V3 run this analysis itself?”: I haven't found one. Say exactly that.)

What not to say

Five red lines ① Don't sell the “exact match” as design intent. Arithmetic is fact; motive isn't public. The prettier the point, the firmer the line: get it wrong, the segment's credibility goes.
② Don't explain routing or load balancing here. That's the next segment, MoE's hardest part; it needs four full minutes.
③ Don't casually say “MoE saves memory.” It saves compute, not memory. The room's easiest self-made error. Clear it up unprompted before Segment 11.
④ Don't make “not one extra FLOP” absolute (once taught so; fixed). Experts are exactly equal, but MoE also runs the router: +0.46%. Tiny, but proof we did the math; volunteer it.
⑤ Don't state “experts don't split by subject” as a bare assertion. Measurements back it now. Point at the figure, not “take my word for it.”
Pacing Six minutes, five parts: core idea (1) → exact match (1½) → whole-model math (½) → fixing “expert” with the measurement figure (2) → triage close (1).
Two high points, three minutes apart; don't let them compete. First, the two-second silence at “Not close. Identical.” Deliver the line before it, “I gave you this exact number in the last segment,” flat. Let the room get there itself; it sticks better.
Second, the red-and-green figure's right panel: “26.4%, twice random.” Land it on “so routing follows local structure,” not on the number.
Never cut the measurement figure. Short on time? Cut “why small and many,” then the left panel's domain distributions (the right's three bars stand alone). But leave “experts don't split by subject” unsettled and the next two segments go wrong.
09

Routing: how the 8 are chosen

⏱ 5 min
What this segment must leave behind Main: n_group=8 and topk_group=4 have nothing to do with model quality. Communication forced them. The theme's sharpest case: hardware shapes the model.
Secondary: the bias decides only who gets picked, never how much each weighs. Scheduling stays out of the output.
Both lead to the close: almost every design here exists to make the model run, not smarter.
Where this segment sits Last segment showed the deal; this one pays the bill. The property lost most badly: “No communication to assign work.”
Five minutes, three figures. Talk less. The routing dataflow figure's blue and green paths never cross: “the bias picks, it doesn't weigh” in one glance. The four-bar figure shows “who's waiting”; the two-box figure settles “free doesn't mean none.”
One deliberate admission. “Auxiliary-loss-free” is tidier than the truth; say so yourself. In a course about limits, this earns trust.

Script

Scroll to 4b: the whole routing dataflow figure.

First, the router, and a number you may not expect.

It's the smallest thing in the layer

It's a single matrix, 7,168 by 256: 1.84 million parameters.

Its layer has 11.32B parameters. The router is 0.016% of it.

Pause. Great contrast.

The most troublesome part of MoE is the smallest thing in the layer.

Point at the yellow sigmoid box.

A counterintuitive scoring detail: sigmoid, not softmax. Softmax makes the 256 scores sum to 1, so experts compete. Sigmoid scores each on its own, 0 to 1; none affects another.

The real problem: load balancing

The router is learned, so it develops favorites. A few experts do well early, get picked more, train better, get picked even more. Winner takes all.

200-plus experts can sit mostly idle, compute piled on a handful.

Scroll down to the four-bar figure at the end of 4b, “⭐ Imbalance: one device busy, all wait”: red top, green below.

First, how expensive is this? Else the fix looks like overkill.

Experts sit on different devices. Top half: the four get 40, 20, 25 and 15 percent of tokens. The hatched cells are just waiting.

When does the layer finish? When the slowest device does. So 37.5% of the four devices' compute you paid for goes unused.

Next line carries the weight.

And it repeats every layer: 58 layers, each waiting for stragglers.

Not “a few percent of efficiency”: a weakest-link problem; the cluster runs at its slowest device's speed.

(Those four numbers are made up, just to show “who's waiting.” Real imbalance depends on data and routing; no fixed number. Say so.)

The old fix adds a “how evenly did you spread” penalty to the loss. Now the model optimizes two unrelated things, answering correctly and spreading evenly. They pull against each other; both end up worse.

Scroll back to the routing dataflow figure at the top of 4b. Point at the blue “bias b” box on the left; trace the blue arrows.

V3: give each expert a bias. Overloaded lately? Nudge it down so it loses a little in the competition. Nobody picking it? Nudge it up.

Not learned by gradient: set from observed load, never in the main loss.

Secondary takeaway. Slow; let the figure talk.

⭐ The bias picks, it doesn't weigh

Circle the red dashed box, then the cut.

Look at the two paths in this figure.

Blue carries the bias through grouping and selection, outputs 8 indices, and is cut off.

Green path: trace from the sigmoid to the green box at the bottom.

Green, the raw score, runs from sigmoid to end, never touching the bias.

The final weighting uses the green one.

Two seconds, then this.

So the bias can even out the load without changing one byte of the model's output. Scheduling stays out of the output.

Three wins. Push the bias as hard as you like; the output stays true. Outside backprop, it steals no gradient from the main objective. Outside the training loop, the policy changes without touching the model.

Honesty: thirty seconds, but say it.

⭐ But “no auxiliary loss at all” overstates it

Scroll down to the blue-and-yellow two-box figure (above the four-bar one): “V3 actually has two balancers, for two scales.”

The method is called auxiliary-loss-free: literally, no auxiliary loss. The name is tidy. The truth isn't quite.

The paper: though V3 mainly relies on the auxiliary-loss-free strategy, to prevent extreme imbalance within one sequence it also uses a complementary sequence-wise balance loss. Its factor: an extremely small value.

Point at the left box, then the right box.

So there are really two mechanisms, for problems at two scales.

The bias on the left works across steps: after each, it checks global load and adjusts, for “a few experts chronically overloaded.”

The tiny loss on the right works within a single step. It handles one sequence that happens to pile its tokens onto a few experts. That happens inside a step; the bias, adjusting once per step, can't reach it.

Takeaway.

Left is the policy, right the fuse. “Free” means the left does the main work, not that the right is absent.

Scope: “two mechanisms, one per scale” is in the paper. “That's why the factor can be so small” is our reading.

The last piece: group limits, this segment's main takeaway

Scroll back up to the routing dataflow figure. Point at the blue box “256 → 8 groups of 32; pick only 4 groups.”

256 experts, 8 groups of 32. Each token may touch at most 4 groups.

A question: is this limit there to make the model learn better?

Really wait: this pause is the pivot.

It has nothing to do with it.

Because a group is a machine.

A token picks 8 experts. Spread over 8 machines, its activations go to 8 places and come back. That's all-to-all communication.

A 4-group cap fixes how many machines a token can bother.

The theme's sharpest line. Hold three seconds.

A communication constraint, written straight into the architecture.

Not an optimization bolted on outside: part of the model definition. Open V3's config and the two numbers are right there.

With thirty seconds spare, add this: plainest evidence hardware draws the line first.

One sentence in the paper stopped me the first time I read it.

It says: although V3 selects only 8 routed experts in practice, it could go up to 13 at exactly the same communication cost.

One second.

Read it so: first set the box, “at most 4 nodes,” from the machines' shape, then see how many experts fit. It holds 13; they filled it with 8.

The order is reversed: not the model asking for 8 and the system delivering. The system drew a line, and the model picked a number inside it.

Close; on to 4c.

The takeaway: almost every routing design exists not to make the model smarter, but to make it run. Sigmoid, the bias, the group limit, the final 2.5 scaling: none adds expressive power.

Likely questions

Why sigmoid and not softmax? Isn't softmax more standard?

For sure: softmax forces the 256 scores to sum to 1, a zero-sum constraint. One scoring high pushes the others down, even if all fit. Sigmoid lets each expert judge “does this token suit me?” on its own.

It also pairs better with the bias. A bias on an independent score has a clear meaning (“make it easier to pick”); on a normalized distribution, a muddled one.

(Scope: a design rationale. Not “sigmoid is proven better.” I've seen no controlled comparison.)

If the bias isn't learned, how does it get updated?

By counting actual load and adjusting directly. Too many tokens this batch, nudge its bias down; too few, up.

A feedback controller, not an optimized parameter, so no gradient, never in the loss.

(Step size and frequency: training details, not needed here. If asked: “adjusted from load feedback; step size depends on the implementation.”)

Doesn't the group limit stop the model from picking the best experts?

Yes. That's its cost; admit it.

If this token's 8 best experts are spread over 5 groups, it can't get them all. The expressiveness given up buys controlled communication.

That's this segment's point. Some designs exist for one reason only: the hardware. Not every hyperparameter serves quality.

How bad can load imbalance really get? Isn't it just a bit slower?

Not a bit: the cluster runs at its slowest device's pace.

Back to the figure: the four devices get 40/20/25/15, and the 40 sets the layer time. 37.5% of paid compute sits idle; evened out, the same work runs 1.6× faster.

And it repeats every layer: 58 layers, each waiting for stragglers. Not “a few percent of efficiency”: a weakest-link problem, 58 times in a row.

(Those four numbers are illustrative; real imbalance depends on the data and routing. Pressed for the real figure: not sure; there's no general answer.)

Why is the group score “the sum of the top 2 in the group”? Where does the 2 come from?

No guessing: the 2 is derived from a formula the paper gives.

The paper: each token goes to at most M nodes, chosen by “the sum of the highest Kr/M affinity scores of the experts on each node.” Kr, activated routed experts, is 8; M, max nodes per token, is 4.

Write on the whiteboard: 8 ÷ 4 = 2.

So “top 2” is just 8 ÷ 4. Allow only 2 nodes, and it becomes “the sum of the top 4.”

And the rule's shape makes sense: at most 4 nodes, 8 experts, so 2 per node on average. Scoring by the top 2 asks, “if I pick this node, what do its best 2 give me?”: exactly the share you'll take from it.

(Scope: the formula is the paper's; the reading “so it matches the share” is ours. The course once answered “public sources don't explain it; not sure”: not looking hard enough; fixed.)

Why did V3 switch from V2's softmax to sigmoid?

Certain: the paper says so explicitly. It says, “Slightly different from DeepSeek-V2, DeepSeek-V3 uses the sigmoid function to compute the affinity scores, and applies a normalization among all selected affinity scores to produce the gating values.”

An easy catch; raise it yourself: if you normalize to 1 at the end anyway, aren't they the same?

No: they act at different moments. Softmax's denominator spans all 256, baking competition into scoring: raise one, the other 255 drop. Sigmoid has none; each expert can say alone, “I'm good at this token.” Competition waits for the “rank and take the top 8” step.

(Scope: not “sigmoid is proven better.” V3 published no controlled comparison that I've seen.)

What not to say

Five red lines ① Don't call the group limit “an optimization trick.” It's in the config: part of the model definition. That distinction is the whole point; soften it and it's wasted.
② Don't go into how all-to-all is implemented here. “How many machines a token sends to” is enough. Traffic math is 4c.
③ Don't call the bias “a learned parameter.” It isn't in the loss and doesn't go through backprop. Otherwise “it steals no gradient from the main objective” contradicts itself.
④ Don't say “V3 uses no auxiliary loss at all.” Once taught so; fixed. The paper says it “also employ[s] a complementary sequence-wise balance loss,” with a tiny factor. Truth: “not the main tool,” not “none at all.” The line an expert is likeliest to correct on the spot.
⑤ No more “public sources don't explain the 2.” It's Kr/M = 8 ÷ 4, the paper's formula; the old “not sure” was not looking hard enough.
Pacing Five minutes, six parts (min): router size 0.5 → load-balancing disease 0.5 → bias picks, doesn't weigh 1 → two-mechanism admission 0.5 → group limits 1.5 → close 1.
The pivot: “Is this limit there to make the model learn better?” Really wait for a reaction. Let them think “probably, yes”; then “nothing to do with it” lands.
Never cut: “the bias picks, it doesn't weigh,” “a group is a machine,” “free doesn't mean none.” Short on time? Cut sigmoid (a detail), then “the box holds 13” (lovely, but a bonus).
This segment's two “Expand” blocks Normally, open neither. First: “Expand: sigmoid scores ‘don't sum to 1,’ yet the 8 final weights sum to 1. Contradiction?” Experts raise this; inside: the two moments, and the 2.5.
Second: “Expand: the rule in the paper, and where the 2 in ‘top 2 per group’ comes from.” Open it if asked about the 2: Kr/M = 8 ÷ 4, plus the paper's “the box holds 13” passage.
10

Move tokens, or move weights?

⏱ 2 min
What this segment must leave behind Main point: every earlier calculation had one right answer. This one is a genuine either/or. “It depends on the workload” isn't a dodge: slope 1 vs slope 0 must cross.
Secondary: inference decode and training sit on opposite sides of the crossover; same model, opposite answers.
It also plants the first seed of Topic 6's “why prefill and decode have opposite bottlenecks.” Plant it, don't grow it.
Where this segment sits 4b is “how experts get picked.” 4c is “once picked, which goes to which: data or weights?” The fourth good property MoE takes from dense, where every token uses the same weights.
Two minutes, carried by the log-log chart; if it lands, you barely need words.

Script

Scroll to 4c: the crossing-lines chart, fully on screen.

Expert weights don't fit on one device: 21 GiB a layer, 1.19 TiB for all 58. So experts always span many devices.

The token is on this device; the expert it needs is on that one. Who goes to whom?

Two routes, and both work

Point at the flat green line, then the blue diagonal.

Route one: send the token over. It picked 8 experts, but those live on at most 4 nodes. So it sends 4 copies of its activations out, and gets the results back. Out once, back once: 112 KiB per token per layer.

Route two: bring the weights here. Every device broadcasts its share of expert weights, then each computes locally. The token never moves. 21.08 GiB per layer.

Next line is the pivot. Slow down.

⭐ It's not which number is bigger. It's the shape of the two lines

Route one scales with the token count. That's the diagonal.

Route two doesn't depend on the token count. Whether the batch has 10 tokens or 10 million, it's the same 21.08 GiB. That's the flat line.

Pause. Let the room get there first.

One has slope 1. The other has slope 0. So they have to cross.

That's why “it depends on the workload” isn't a dodge here. It's geometry.

Point at the red circle, the crossover.

They cross at roughly 197,000 tokens.

Who lives on each side

Point at the three blue dots: bottom left, middle, top right.

Inference decode produces a few dozen tokens per step. Far to the left. Moving tokens wins, by several orders of magnitude.

A training step easily tops a million tokens. Far right. Moving weights wins.

And one 128K prefill is 131,072 tokens. Still left of the crossover, but right up against it.

Hold two seconds: the Topic 6 seed.

So: same model, same inference request, yet decode sits orders of magnitude from the crossover and prefill almost on top of it. Why they come out opposite on so many things is Topic 6's job. For now, just remember it.

State the limits. This matters

Point at the gray line under the chart and the yellow arrow.

This 197,000 is not an engineering threshold. Don't use it.

It's a rough comparison: bytes moved per layer only. I didn't account for all-to-all versus all-gather efficiency, and those two don't cost the same.

And the crossover moves. Spread experts over more devices and move-tokens traffic splits thinner: the crossover shifts right.

So what's it good for? It says the two schemes are in the same order of magnitude.

Last line. Hold three seconds, then move on.

A hundred times apart, there'd be nothing to choose. This close, the either/or is real. Every earlier calculation had one right answer. This one doesn't.

If time allows, the small print below. Names only.

A whole line of work lives here; just know the names: DeepEP, EPLB, dropless. All attack this problem; we don't go into them.

Likely questions

Why not just put every expert on every device? Then nothing has to move

It doesn't fit. One layer's expert weights are 21.08 GiB; all 58 layers, 1.19 TiB: nearly the whole model (1.22 TiB).

One device has tens of GiB of HBM: over an order of magnitude short.

That's why this problem must exist: while one device can't hold every expert, something must cross devices.

Is the move-weights line really independent of the token count?

Yes. You're moving weights, and weights don't know about tokens.

Expert count and size are fixed by the model. 10 tokens or 10 million, the same 257 experts' weights.

That flat line is why the chart works. If it grew with token count too, the lines might never cross.

Can you mix the two schemes?

Yes, and real systems often do. Don't present the chart as all-or-nothing.

Mix by dimension: move weights in one device group, tokens in another. Mix by phase: one for prefill, the other for decode.

(How to mix is out of scope. If asked: “many combinations; it depends on how the parallelism is sharded.” Don't invent a design on the spot.)

So can I use 197,000 to make decisions or not?

Not as a threshold; as a sense of scale, yes.

Bytes per layer only, ignoring the collectives' efficiency gap and the experts' device count. Wider spread, crossover further right.

The only takeaway: same order of magnitude. For a real system, redo it with your own sharding and measured network numbers.

Is this the same thing as the group limit from earlier (pick at most 4 groups)?

Same chain, different links.

The group limit caps how many machines one token can bother, holding the move-tokens slope down (4 copies, not 8).

This section asks the higher-level question: should tokens move at all?

Together: 4b makes this route's cost acceptable; 4c asks whether to take it at all.

What not to say

Three red lines ① Don't present 197,376 as an engineering threshold. It's rough, and moves with how many devices the experts span. It supports only “same order of magnitude.” Say more and you hand the room a number that will hurt them.
② Don't say “training always moves weights, inference always moves tokens.” It's one continuous line, not two boxes; real systems mix.
③ Don't go into DeepEP / EPLB / dropless here. Names only, so those who've heard them can place them. Expanding one turns two minutes into ten, for an audience that won't use it.
Pacing Two minutes, three parts: two routes 20 s each → slopes must cross 40 s → three landing points + limits 40 s.
The pivot is “slope 1, slope 0, so they must cross.” Pause after it. If the room reaches “so there must be a crossing” on its own, it worked.
Short on time? Cut the three names and the prefill seed. Never cut the limits: a pretty number without saying it can't be used is worse than none.
10⁺

Dropless: how static shapes were won back

⏱ 3 min
What this segment must leave behind Main point: the first good property MoE takes away is static shapes; the old fix was brute force. Give every expert a fixed-size box, drop what doesn't fit, pad empty slots with zeros. Shapes are static again, but the same layer loses at both ends.
Secondary point: this gives 4b's load balancing its real justification. Segment 9 framed imbalance as a weakest-link speed cost. With capacity boxes, it makes tokens disappear outright.
Where this segment sits The running thread's last payoff: the “four good properties.” Segment 7 set up all four; Segment 9 repaid “no communication to assign work,” Segment 10 the communication volume; this one repays “static shapes” and “the compiler plans it all ahead.”
One of the few places an abstract constraint becomes a picture: the capacity-box figure makes “static shapes” real.
Cuttable whole if time is short, but it's two high-payoff minutes.

Script

End of 4c: the capacity-box figure.

I just mentioned dropless. Two minutes on it: one of the course's running threads is behind it.

Back to Segment 7's four good properties

First: static shapes. The compiler knows every matrix size and buffer location before the run, so it can schedule the whole graph.

MoE breaks this one hardest.

How many tokens in this batch land on expert #37? You only know after routing, and the next batch differs.

Pause: this is the root of the problem.

But hardware wants it all fixed up front.

The old way: give each expert a fixed-size box

Left figure: the three experts, left to right.

The box size is a hyperparameter called the capacity factor.

Once it's set, every expert always computes exactly that many tokens. Shapes are static; the compiler is happy.

But the layer loses in two places at once.

Point at the two red dashed boxes above expert A.

This expert on the left got 6 tokens. The box holds 4. The extra 2 are simply dropped.

Dropped: their output from this layer is zeroed; only the residual passes. For those two, the layer might as well not exist.

Switch to expert C on the right, the hatched one.

The expert on the right got only 1 token; the other 3 slots are zero padding, costing compute and HBM, producing zero.

Next line is the main point. Slow down.

So the capacity factor is a miserable knob. Too small, you drop real data. Too big, you burn compute. You lose either way.

Three numbers that show this isn't “just tune a parameter”

Scroll to the blue box. Read its three items one by one.

One: dropping tokens really hurts quality. In one controlled comparison, dropping cut validation loss by only 0.15; never dropping, by 0.26, 1.73× as much.

Two: a drop-proof box is costly; there, MoE-layer math more than doubled.

Three: you don't even know the right size. Some models need a capacity factor of 11 to stop dropping; in others it spikes mid-training without warning.

Pause two seconds.

A knob you can't set, costly either way, shifting on its own mid-run.

If challenged, give the scope: that experiment is Transformer-Small scale, 64 experts, top-1 routing, not V3's 256 experts, top-8. It proves the knob itself is a problem; it doesn't convert into “how much V3 would lose by dropping tokens.”

What dropless does

Point at the figure's right half.

The idea is simple: no box.

Rewrite the MoE layer as one block-sparse matmul. That kind of op handles blocks of different sizes natively. 6 tokens come in, it computes 6. 1 comes in, it computes 1.

No drops, no wasted compute.

But the real win isn't the compute you save. It's that the knob is gone: nothing to tune, nothing shifting mid-run.

What about V3?

Scroll to the green box.

V3's technical report is blunt: load stayed well balanced throughout training, so it drops no tokens in training.

Pause, then tie back to Segment 9.

Notice the reason: the load was balanced well enough.

So Segment 9's load-balancing bias matters more than I said. I called imbalance a weakest-link problem: all wait for the slowest device. The other half: with capacity boxes, it makes tokens disappear.

It isn't just keeping things from slowing down. It's keeping data from vanishing.

Thirty seconds on the half most people miss.

And after deployment?

That passage is really two sentences, and the second matters as much: specific deployment strategies keep inference balanced, so inference drops no tokens either.

But training and inference use different mechanisms. Training's loop lets the bias adjust every step; deployed, the loop is gone.

Inference moves experts: extra copies of those seen hot in production. Prefill has 32 redundant experts; each GPU hosts 1 beyond its usual 8. Hotness: online statistics, re-adjusted roughly every ten minutes.

The next line is the keeper.

Note the boundary: rearranged only within a node, explicitly adding no cross-node all-to-all. Again: topology first, then optimize.

Wrap up and move to 4d.

Likely questions

Is a dropped token gone for good? Won't the model learn garbage?

The token isn't gone; its expert output for this layer is. Via the residual, its information passes unchanged.

It skips this layer's MLP, keeping only the attention result.

So the model doesn't collapse, but quality drops. The 1.73× gap measures exactly that.

Why not make the box huge? There's plenty of HBM

Two reasons.

One, it doesn't pay: the box's empty part really is computed, multiplying zeros. There, avoiding drops more than doubled the math.

Two, you don't know how big to make it: some models need 11×, and that can jump mid-training. Tune it for today, still drop tomorrow.

The second is the real pain: not “a bit more expensive” but “you can't know in advance.”

If dropless is so good, why does anyone still use capacity boxes?

Dropless needs ops for irregular shapes, and those aren't free.

Capacity boxes need only plain dense matmuls. Static shapes keep compiler and hardware happiest; every framework runs them.

Dropless trades kernel complexity for not tuning the knob. It pays off for large models; at small scale maybe not.

(Don't go into block-sparse internals. If pressed: “that's kernel-level work; dedicated libraries do it.”)

Does V3 use dropless?

Careful: don't reflexively say yes.

The result is certain: the report says it drops no tokens in training. The stated reason is certain: good load balancing.

The mechanism isn't: whether its training kernels still have a notion of “capacity,” I found in no public source.

So say “V3 drops no tokens in training,” not “V3 uses dropless kernels.” Others reproducing V3 on other hardware with dropless kernels is separate from how V3 was trained.

Are tokens dropped at inference too?

For V3, the report says explicitly no, by another mechanism. Training adjusts the bias each step; deployed, with no loop, hot experts are duplicated instead (prefill: 32 redundant experts, rearranged about every ten minutes from online statistics, only within a node).

Elsewhere, it depends on whether a capacity is set, not on the model.

One structural difference: a decode step has a few dozen tokens, extremely sparse over 256 experts. “One expert gets overloaded” looks completely different in decode.

The real inference headache is the opposite: most experts receive no tokens, and their weights move for nothing. The other side of the previous segment's chart.

What not to say

Three red lines ① Don't say “V3 uses dropless.” The report states only that no tokens are dropped and why, nothing about the kernels. The easy slip here; it breaks the course's line between checked and inferred.
② Don't explain how block-sparse is implemented. “That kind of op handles blocks of different sizes natively” is enough. Below that is kernel engineering.
③ Don't describe a dropped token as “the token disappears.” The residual remains; it only skips this layer's MLP. Say it vanishes and the room finds the system absurd and stops believing you.
Pacing Three minutes, five parts: why shapes can't stay static 20 s → the box loses both ways 40 s → three numbers 30 s → dropless + Segment 9 tie-back 30 s → post-deployment redundant experts 30 s.
The weight is the tie-back: “It isn't just keeping things from slowing down. It's keeping data from vanishing.” Let it land. Don't rush.
Short on time? Cut it whole; the main thread survives. One minute? Keep only the “box loses both ways” figure and the tie-back.
11

671B and 37B: the mountain and the stream

⏱ 3 min
What this segment must leave behind Main point: MoE saves compute, not HBM. The course's most misunderstood point. “37B active” is not as cheap to run as a 37B dense model: it needs exactly the HBM of a 671B dense model.
Secondary point: parameters, compute and HBM are three accounts; one component can rank very differently in each. Attention: 1.7% of parameters, 81.8% of compute at 128K. This way of thinking is the main take-home, and the bridge to Step 7's grand total.
Along the way, this segment closes the books. 671.03B matches the official 671B, which checks every formula from the past two hours at once.
Where this segment sits Step 4's close: the course's first “settle the books” moment. Earlier steps filled the table; now sum it, check it against an outside figure.
Three parts: the total → the 61 : 1 comparison → the mountain and the stream. Two analytical, one visceral: a natural wind-down.
Give the mountain picture two more seconds: after so many numbers, the ears rest here.

Script

Scroll to 4d: the full parameter table.

First, close the books

We derived every number in this table. Now we add them up.

One expert, 44.04M. 257 experts plus router: 11.32B a layer. Times 58: 656.57B.

One attention layer, 187.11M; times 61, 11.41B. Three dense layers, 1.19B. The entrance and exit tables, 1.85B.

Read the next number slowly, digit by digit.

Total: 671.03B. Official: 671B.

Pause two seconds.

Be clear what this means. It isn't that one number matched.

It checks every formula we've used over these two hours, all at once. Hidden size 7,168, expert width 2,048, SwiGLU's three matrices, MLA's ranks. Get any one of them wrong, and the total doesn't match.

That's how to check your work: not number by number, but against an outside anchor that ties all the formulas together.

⭐ The starkest row in the table

The comparison figure: 61 blue slices against one green block.

Look at these two bars. The top one is all 61 attention layers combined: 11.41B. The bottom one is any single MoE layer: 11.32B.

Same length.

Hold three seconds.

61 attention layers equal one MoE layer.

And in Segment 6 we found attention takes 81.8% of the compute at 128K.

So attention is 1.7% of the parameters and 81.8% of the compute.

Parameters and compute are two accounts. They don't even rank things alike.

If time allows, the MLA breakdown box below.

A side finding, same lesson: different accounts, different answers. Of MLA's 187M parameters, the biggest piece isn't the KV compression but the output projection stitching the 128 heads back into the residual stream: 62.8% in one matrix.

For all the cache compression, the output side holds most parameters. No contradiction: MLA shrinks memory per token; this counts parameters.

Mountain and stream: the main point

The blue bar / green bar figure; the mountain illustration is just below.

The blue bar on top is 671.03B, all the parameters. The short green piece below is 36.62B, what one token actually uses.

About 18 times.

The course's most misheard line. Hit it hard.

All of that 18× is compute. HBM saves nothing. Not one byte.

The 634B in between really are idle this step. But they must stay in HBM, because the next token might need some.

Point at the mountain illustration.

The mountain is what HBM must carry. The stream is what compute must pay for.

MoE separates the mountain from the stream. Before it, a bigger mountain meant paying for more water.

Pause, then give the correction.

So “37B active” does not mean “as cheap to run as a 37B dense model.” You need exactly the HBM of a 671B dense model.

MoE buys “more knowledge for the same compute,” not “a bigger model in the same memory.”

Wrap up and move to Step 5.

In one line: parameters, compute and HBM are three accounts. A component that's negligible in one can be the star of another. In Step 7 we'll put all three side by side.

Likely questions

Is it 36.62B or 37B? Why do the two numbers differ?

The gap: is embedding counted as active?

Item by item, 36.62B: 9 experts × 58 layers, all 61 attention layers, the 3 dense layers, plus the output matrix. Add the embedding: 37.55B.

The official 37 falls between; the report doesn't say how it's counted, and I couldn't find out. So the course uses 36.62, and “about 37B” when matching the official figure.

A gap of under 1B changes no conclusion. 18× is still 18×.

Why not move rarely used experts to CPU memory or disk to save HBM?

You can, and people do, but it trades on a different account.

The problem: you can't know in advance which experts are rarely used. Which 8 experts the next token picks is known only after routing, too late to fetch weights from outside.

So it trades latency for HBM. It pays off when throughput isn't critical; for step-by-step decode, it hurts.

(Don't go into offload strategies; “it trades latency for HBM” is enough.)

If it saves no HBM, what is MoE for?

It's for fitting more knowledge into the compute you can afford.

Flip the question: in a 671B dense model, every token computes through all 671B. Nobody can pay that bill.

MoE gives you 671B of capacity on a 37B compute budget. The price: HBM for 671B.

An explicit trade: HBM for compute. Whether it pays depends on which you're shorter on.

Why do 61 attention layers only equal one MoE layer? Is attention just small?

Attention isn't small; one MoE layer is huge.

One MoE layer has 257 experts at 44M each. Essentially 257 MLPs side by side, only 9 used at a time.

One attention layer is 187M, about half of one dense MLP layer (396M). Attention was never the parameter hog. It's the compute hog.

This segment's point: “big” means different things in the parameter and compute accounts.

What's the 0.03B difference between 671.03 and the official 671?

Just rounding: the official 671 was reported rounded.

More interesting: my table excludes the MTP (multi-token prediction) module. Without it, the total already matches 671. So the official 671B excludes it too.

(The course skips MTP. If asked: “an auxiliary training module, not part of the 671B,” and leave it.)

What not to say

Three red lines ① Never turn “37B active” into “as easy to deploy as a 37B model.” The course's most misread point, and why this segment exists. Go soft and they leave with the wrong cost model.
② Don't present 671.03 matching 671 as “look how precise my math is.” It's the method: one outside anchor checks every formula at once. A method travels. A pretty number doesn't.
③ Don't go into offload, quantization or how to shard expert parallelism here. This segment only sets up the three accounts. Saving is for later topics; open that door and it won't close.
Pacing Three minutes, one minute per part: the total → 61 : 1 → the mountain and the stream.
Must-keep pauses: two seconds after 671.03; three seconds after “same length.” Skip them, lose the impact.
Short on time, cut the MLA breakdown (a bonus). Never cut mountain and stream: it closes Step 4; the course's most-repeated line.
12

The residual stream is a bus, not a pipe

⏱ 4 min
What this segment must leave behind Main point: the network isn't a pipe, it's a bus. The difference is one symbol, the “+”: a pipe replaces; a bus accumulates. The only time the course overturns a picture the audience already holds: draw the wrong one first, then knock it down.
Secondary point: raise “why not discard the 61 layers' activations?” Don't answer. Topic 4's job; leaving with a question beats an answer on the spot.
A free bonus: why the layer count can be an arbitrary number like 61. Constant shape: add or drop a layer and no other is affected.
Where this segment sits The first four steps were inside one layer; this is the first about what's between layers. A gear change; slow down a little.
Four minutes, on the side-by-side figure. Wrong on the left, right on the right. The wrong picture came from seeing, not reasoning, so only a picture can replace it.
Not hard, but easy to deliver flat; it needs one clear flip: “notice, it's add, not replace.”

Script

Scroll to Step 5: the side-by-side figure, fully on screen.

Let me ask you something first. What does a Transformer look like in your head?

Really wait. Let them picture it. Point at the left half of the figure.

Most people picture the left side: a pipe. Data goes into layer 1, out into layer 2, then layer 3, to the end.

That picture is wrong.

Switch to the right half.

It's actually a bus

There's one vector, 7,168 wide, running all the way through.

Every layer does three things: read a copy, compute something, add the result back.

Scroll to the formula box below.

As a formula, it's one line: x equals x plus MLA of x. Next line: x equals x plus MoE of x.

Repeat 61×: that's the whole model.

The flip. Stress “add,” then pause two seconds.

Notice: it's add, not replace. That one word carries this whole segment.

Why it has to be addition

Two reasons: one about gradients, one about meaning.

Gradients: addition gives backprop a highway. Otherwise it passes through 61 layers of multiplication and fades to nothing.

Meaning is the more interesting one: addition makes “do nothing” the default.

If a layer hasn't learned anything useful yet, its output is near 0. Adding it changes nothing; what's already there survives.

Pause. The next three points are the payload.

⭐ Three observable consequences

One: every layer sees everything so far, not just the previous layer's output.

So layer 40 reads directly what the embedding wrote. The 39 layers between didn't wash it away; they stacked more on top.

Two: the shape never changes. 7,168 in, 7,168 out, the same for all 61 layers.

So layers can be added or dropped. 61 is empirical, not derived.

It matters for hardware too: the width never changes, so you can shard along it all the way through without replanning each layer.

Read the notes under both figures.

Three: in a pipe, a broken layer 30 makes the next 31 useless. On a bus, layer 30 outputting 0 loses one contribution; the rest run on.

Where the norm sits: one sentence

V3 uses pre-norm: normalize, run the sublayer, add its output straight back to the trunk. The trunk itself is untouched.

The other placement is post-norm: add first, then normalize the sum. That puts a norm across the bus every segment, cutting the direct highway.

Modern models have pretty much all gone pre-norm. For experts in the room: Xiong et al.'s 2020 paper on Layer Normalization proved that post-norm has large gradients near the output layer at initialization, so learning-rate warm-up is required; pre-norm can drop warm-up entirely. That's “trains unstably” in a form you can verify.

One last number. It only raises the question

Scroll to the box “1.75 GiB/layer × 61, all kept = 106.75 GiB.”

At 128K, the residual stream has to be kept once per layer: 131,072 times 7,168 times 2 bytes, exactly 1.75 GiB.

Keep all 61 layers, and that's 106.75 GiB. One device holds 94.74 GiB.

Pause, then add this; it hits harder.

Not a single weight in that number: it's all intermediate results. The 1.22 TiB of weights is a separate account.

So: why keep them at all? Why not discard them once computed?

Stop. Let the question hang.

Why backward needs these intermediates is Topic 4's job. Not today.

But the screen has half an answer. Read it, don't explain: the V3 paper says all RMSNorm and MLA up-projections are recomputed in backward, not stored.

So when something doesn't fit, a real system's first reflex isn't more devices but: must this be stored? If not, don't; recompute in backward, time for space, not free.

Enough. Don't slide into recomputation strategies; that's Topic 4. If time allows: the mHC box at the end and the figure above it. Thirty seconds.

An aside: V3 has one residual stream; V4's mHC replaces it with four parallel ones.

At the figure: the bus gets wider, not the workshops. Layers still read and write 7,168 dims; MLA and MoE don't change a byte.

Others widened the residual stream before DeepSeek, but deep stacks went unstable. mHC makes the mixing matrix doubly stochastic (rows and columns each sum to 1), so the signal neither grows nor vanishes. Not “wider,” but “wider, made trainable.”

Careful with the cost line. Don't say “4× wider, so 4× the memory.” The V4 paper absorbs most of it with fused kernels and selective recompute; its only absolute number: 6.7% wall-clock overhead. Details are in the expandable box; skip them in class.

Likely questions

Is “bus” a metaphor, or literally true?

Close to literal, which is why I like it.

That 7,168-dim vector sits in HBM the whole forward pass. Every layer reads and adds to that same block of memory.

Unlike a bus, layers access it strictly in order: no contention. Most accurately: “a bus accessed sequentially.”

If every layer is an optional increment, can you just prune some layers?

Structurally, yes; that's this segment's property. The shape doesn't change, so remove a layer and the rest still run.

But quality drops, by a hard-to-predict amount. What it learned isn't backed up elsewhere.

“Can be pruned” isn't “pruning is harmless.” Layer pruning is real, but that's model compression.

Pre-norm vs. post-norm: what differs?

Whether the norm touches the trunk.

Pre-norm: normalize, run the sublayer, add its output straight back. The trunk is pure addition, never rescaled.

Post-norm: add, then normalize the whole trunk, so every layer touches that gradient highway once.

With enough layers, post-norm trains unstably. Same as the first reason earlier: no toll booths on the highway.

For something harder: Xiong et al. 2020 (On Layer Normalization in the Transformer Architecture) analyzed gradients at initialization. Post-norm has large gradients near the output layer, so warm-up is required; pre-norm's are well-behaved, so warm-up can go entirely.

Note the claim's shape: not “post-norm can't be trained,” but “post-norm adds one more thing to get right, harder to tune with depth.” At tens to hundreds of layers that means “unstable training,” but repeat it precisely.

(Skip norm variants; the course needs one intuition: keep the trunk clean.)

How is 106.75 GiB computed? Why only one copy per layer?

131,072 tokens × 7,168 dims × 2 bytes (bf16): one copy is 1.75 GiB. Times 61 layers, 106.75 GiB.

“One copy per layer” is a floor. Real implementations also keep intermediates inside each layer, so the true number is only higher.

The floor persuades better: even the leanest accounting doesn't fit.

If asked “how much higher”: about 4.15 TiB, roughly forty times. The bulk: MoE copies each token's activations nine times, and MLA decompresses K and V back to over 30,000 dims. That compression saves inference KV cache, not the training forward pass. That's Topic 4: give the magnitude and stop.

Do V4's four residual streams make the model four times wider?

No. Be clear, or people hear “four times the parameters.”

The residual stream widens, not each layer's compute width. Sublayers still read and write 7,168; only the channels between layers multiply. The V4 paper's own words: “the expanded residual width does not influence the design of the inner layers.”

Numbers you can give: both sizes use nhc = 4 and 20 Sinkhorn-Knopp iterations. V4-Pro has 61 layers and hidden 7,168, identical to V3, so this page's math applies directly. V4-Flash has 43 layers and 4,096.

4× wider, so 4× activation memory?

The easiest one to get wrong; we've done so ourselves.

“1.75 GiB × 4 = 7.00 GiB per layer” is an upper bound, not the cost paid. It assumes all four streams are kept whole at every layer.

The V4 paper is explicit: fused kernels, selective recompute, and a reworked DualPipe overlap. In their words, they “recompute most hidden states between layers and all normalized layer inputs.” A large share is never stored.

So the paper gives no absolute memory increase; it depends on recompute settings. Its one absolute number is time: mHC's wall-clock overhead is 6.7% of the overlapped 1F1B pipeline stage.

(Always quote 6.7% with its denominator. Not “training is 6.7% slower” but “the overlapped pipeline stage is 6.7% slower.”)

What not to say

Four red lines ① Don't skip drawing the wrong picture first. Go straight to the bus and people nod, then keep thinking with the pipe. A wrong picture can only be replaced by a picture, not argued away.
② Having raised “why keep them,” don't answer “backward needs them.” Answer and the suspense dies, and it's the ticket into Topic 4. The on-screen line “V3 chooses not to store them and recomputes in backward” is fine to read: it answers “what they did,” not “why backward needs it.” Read it and stop.
③ Don't call mHC “a 4× wider model.” The residual stream widens, not compute width; layer I/O is still 7,168-dim.
④ Don't say “4× wider, so each layer goes from 1.75 GiB to 7.00 GiB” anymore. We once taught that: an upper bound engineering largely cancels, presented as the cost paid. Fixed. The paper's only absolute figure: wall-clock overhead, 6.7% of the overlapped 1F1B pipeline stage. V4 report readers will likely correct you.
Pacing Four minutes, four parts: wrong vs. right 1 min → why addition 1 min → three consequences 1.5 min → number + open question 0.5 min. The mHC thirty seconds are overflow; cut if over time.
The opening question needs a real pause. Let them picture the pipe, then: “that's wrong.”
Keep only one consequence? The third (what happens when a layer breaks): most concrete, best at making “add vs. replace” stick.
13

The exit: the scariest temporary tensor

⏱ 5 min
What this segment must leave behind Main point: count only weights and your memory estimate is wrong. Hinted at twice; this third time hits hardest. Say it outright to close the forward pass.
The evidence: a tensor that doesn't exist in the parameter table outweighs all 61 layers of attention weights. 31.56 GiB vs. 21.26 GiB, 1.48×.
Secondary point: two “one thing, two uses” cases: split input/output weights, and MTP. One sentence each.
Where this segment sits Last stop: simplest operation, scariest number. That contrast is the content. Every step so far got more complicated; this single matmul yields the trip's largest tensor. Use that drop.
Five minutes: one on the mechanism, four on that number. Simple mechanism; don't linger.
Once this lands, the Step 7 total follows. “There's more than weights” has now been shown three times.

Script

Scroll to the top of Step 6.

After 61 layers, we hold one 7,168-dim vector. Now it becomes “what comes next?”

Two moves: a last RMSNorm, then a matmul fanning 7,168 dims out to the 129,280-entry vocabulary. Every token gets a score.

Softmax it and you have a probability distribution. This is where temperature acts.

An easy-to-miss config detail

Point at the red box: tie_word_embeddings.

This matrix has exactly the shape of Step 1's embedding table: 7,168 by 129,280.

Many models share one matrix for both: weight tying. Intuitively, representing and predicting a token feel like one job.

But V3's config sets it to false: reading and writing are different jobs, each with its own copy. The cost is 926M extra parameters.

That also explains a number in the Step 7 table: the embedding-related total is 1.85B, not 0.93B, because it counts two copies.

Breathe. The heavy part starts here.

⭐ Now look at what this matrix outputs

Scroll to the red logits figure, fully on screen.

131,072 positions. 129,280 scores at each one. Two bytes per score.

That's 31.56 GiB.

Two seconds, then compare.

Three comparisons. Bigger than all 61 layers of MLA weights (21.26 GiB). Bigger than one MoE layer's 257 experts. 18× one copy of the residual stream.

Point at the heading below: “Three details that make the logits look worse.”

Three details that make the logits look worse

First: this step is just 0.46% of the forward pass's compute, under half a percent.

242.92 TFLOP, out of 52.5 PFLOP for the whole forward pass. The cheapest step of the trip makes the trip's largest tensor.

Second: cross-entropy usually upcasts logits to fp32, or the numerics are unstable.

That's 63.13 GiB: 67% of one 94.74 GiB device. Something purely temporary eats two-thirds of a device.

The third one is my favorite. That 31.56 is the training number, because training computes a loss at every position.

Inference prefill only needs the logits for the last position: 252.5 KiB.

Scroll to the side-by-side figure: red slab left, thin green line right.

Same op. Training and inference differ 131,072×: exactly the sequence length.

And that green line lies. At true scale it's 1/131,072 of the dashed box's height: 0.0013 pixels. Undrawable, so it's drawn about 2,300 times thicker, just to be seen.

That line is a memory hook. Pause.

So “is lm_head expensive?” has no answer until you say whether you mean training or inference.

So what do real implementations do?

Nobody materializes it. The standard fix is chunking: a few thousand positions at a time, compute the loss, accumulate the gradient, drop the chunk, move on.

In other words, fuse the loss in, so the big tensor never exists in full.

It's the same trick as FlashAttention: don't write intermediates to memory. Second time this trick shows up in the course.

The wrap-up. Slow down.

⭐ The takeaway

Scroll to the last green box.

Notice: this is the third time the course has hit the same thing.

Step 2 was the KV cache. Step 5 was activations. Now it's logits.

Each time, something absent from the parameter table outgrows the parameters.

Stop. Say the next line word by word.

So the real lesson of this step isn't 31.56. It's the rule: count only weights and your memory estimate is wrong.

If there's time, the MTP table. 30 seconds.

A quick aside: MTP

One more config entry. Besides the next token, the model also predicts the one after it. Note: one extra token. The depth is 1.

Built for training; the paper says so: “we utilize MTP to improve training.” Trained only on the next token, a model can coast; looking further ahead forces it to plan.

At inference you can just drop it; the main model works as usual. The paper's ablations do exactly that, so both arms cost the same at inference.

But dropping it wastes it, so it's reused as the speculative-decoding draft. V3 measured 85 to 90% acceptance on the second token, and 1.8× the end-to-end token rate.

Built for training, a windfall for inference. Why that trade pays off is Topic 6's business. Just name it.

Likely questions

How can logits be over 30 GiB? Isn't the output matrix just 7168 × 129280?

The lecture's most common sticking point; someone will ask. Root cause: “position” is used in two places, and listeners merge them.

Agree first: that 7168 × 129280 matrix exists, and there's only one copy. It's the weights: 1.73 GiB, already counted in the 671B. They're right, but that's one token's story.

Then pull the dimensions apart: you fed in 128K tokens. Each token multiplies its own 7168-dim vector by that matrix and gets its own 129280 scores. So the result isn't one row but a table of 128K rows by about 129K columns. 252.5 KiB a row × 128K rows = 31.56 GiB.

On the whiteboard, draw two edges: horizontal “vocab 129,280”, vertical “tokens 131,072”. Say: “About the same size. A coincidence.” Always say this. Otherwise half the room thinks they're the same thing.

Close with a twist: in every earlier layer, a position is just 7168 numbers. At the exit it becomes 129,280: 18× fatter. The model is widest not in the middle but on the way out.

So is that why pretraining usually uses 4K, to keep the logits small?

Right conclusion, wrong reason. Correct it now or it sticks.

Logits depend only on the total token count, not on sequence length. Total tokens = sequence length × batch. 128K × batch 1 and 4K × batch 32 are both 131,072 tokens, so the logits are identical: 31.56 GiB either way. Shorter sequence, bigger batch: nothing saved.

What short sequences save is attention. It costs batch × the square of sequence length, so at fixed total tokens it's proportional to sequence length: the 128K split pays 32× the 4K split.

For an expert, give a checkable anchor: in V3's config, original_max_position_embeddings is 4096, and YaRN's factor is 40. 4096 × 40 = 163840. The base model was trained at 4K. 128K was extended afterward.

Stop there. Training upcasting logits to fp32, hence standard chunked cross-entropy, is Topic 4. Just name it.

Does that 31.56 GiB really get allocated, or is it just theoretical?

Not in a normal implementation, and that's the point, not a disclaimer.

It's the literal coding of the math definition. Nobody does that, because it's this big.

Its value isn't “a block this size sits in memory”; it explains why every framework chunks and fuses here. Without it, you don't know what that code is avoiding.

(Don't put it in a capacity plan. The real peak depends on your chunk size, not on this number.)

Why doesn't V3 share the input and output weights? Wouldn't sharing save memory?

It really does save: 926M parameters, 1.73 GiB. Sizable on small models, so nearly all of them tie.

At 671B, that copy is 0.14% of the total. Hardly worth it.

The case for not tying: reading and writing aren't really the same job. “What does this token mean?” and “should the next position be this token?” want different representations. Tying forces two jobs onto one parameter set.

(Don't claim this is why DeepSeek didn't tie. The report doesn't say; it's the common explanation.)

Does chunking the logits give a different result than computing them all at once?

Math: identical. Floating point: tiny differences.

Why identical: cross-entropy is independent per position. Slice by position, add up the losses, and it's the same expression as one pass.

The differences: summation order. Float addition isn't associative, so chunking may flip the last bits; harmless to training.

What needs care is softmax's own numerical stability. That's why you upcast to fp32 and subtract the max first.

It's only 0.46% of compute. Doesn't that make this step unimportant?

The opposite: that contrast is what this segment teaches.

Compute and memory are separate ledgers; here's the course's most extreme case. Near zero on compute, number one on memory.

Allocate by compute and you miss it, then OOM here.

(Point back to Step 4's “parameters and compute are separate ledgers.” Same idea, different ledgers.)

“tie = false, nothing shared,” but I heard V3's output head is shared?

Both true; different things are shared. The segment's easiest tangle.

Not shared: embedding table ↔ output head. That's tie_word_embeddings = false, which is why we count 926M twice.

Shared: the main model's output head ↔ the MTP module's output head. The paper puts the shallowest (embedding) and deepest (output head) layers on one pipeline rank, so MTP and the main model physically share those parameters and gradients.

So: MTP doesn't add another 926M to the total. That answers challenges to the Step 7 table.

63.13 GiB ÷ 94.74 gives me 67%. Didn't the page say 72%?

67% is right. The 72% was an old error on the page, kept there as a warning.

GiB is 1024³, GB is 1000³: a 7.4% gap. The old text assumed 94.74 was decimal GB, converted 63.13 GiB to 67.78 GB, and got 72%.

But 94.74 was GiB all along: the compiler string 95.38G − 94.74G = 656.93M only works in base 1,024. Both numbers share a base, so divide directly: 63.13 ÷ 94.74 = 67%.

Every percentage on the page is computed from bytes, not from the displayed numbers. Worth saying live: the trap isn't ignorance of units but knowing the trap and fixing the wrong side. Check the denominator's base.

Why is the vocabulary as big as 129,280? Wouldn't a smaller one save memory?

Vocabulary size is a two-sided trade-off.

Bigger: a word fits in one token, sequences get shorter and faster, but the embedding table and logits both grow.

Smaller: you save on those two, but the same text needs more tokens, and compute grows with them; the attention term quadratically.

So shrinking it usually loses. You save a linear term and pay a quadratic one.

(Don't compare vendors' vocabularies. “A trade-off, and the sides aren't symmetric” is enough.)

What not to say

Four red lines ① Don't call 31.56 GiB “the memory it uses when it runs.” It exists only in a literal coding of the definition; real code never materializes it. Get this wrong and people plan capacity with it. Say: “it's so big that engineering routes around it.”
② Don't present the fp32 63.13 GiB as an official V3 number. fp32 cross-entropy is common practice, not specified by the V3 report. Here only 129,280, 7,168 and tie=false come from the config; the rest is computed under the course's assumptions.
③ For MTP, give the two numbers and stop; skip how speculative decoding works. That's ten minutes, and Topic 6. Leave just the impression “built for training, a windfall for inference,” plus two checkable numbers: 85–90% acceptance, 1.8× TPS. If asked why: “the draft reads the main model's own hidden state, not a separate small model.” Then stop.
④ Don't say MTP “predicts two extra tokens.” Depth is 1: it predicts one extra token. The main model verifies two positions at once. They sound alike; the first is wrong.
Pacing Five minutes, light start, heavy finish: mechanism 1 min → number 1 min → three details 1.5 min → training vs. inference figure 0.5 min → wrap-up 0.5 min. The 30-second MTP aside is overflow; cut if over.
After “31.56,” two seconds, then compare. Without it, it's just another number.
Say “count only weights and your memory estimate is wrong” word by word. It closes the forward pass and opens Step 7's total. If the room remembers one sentence, make it this.
14

The total · computed vs. measured · the hook

⏱ 13 min
What this segment must leave behind Main point: from today, “why parallelism?” is a division's result, not a slogan. 1.23 TiB ÷ 94.74 GiB = 13.3. Six stops of parts, collected to give this one division meaning.
Second point: static analysis is trustworthy, yet least accurate where you need it most. The three AOT lessons we learned the hard way beat any published claim.
Third point (the hook): 94.74 was our denominator all day, yet never explained. Topic 2's opener; today just make them itch.
Where this segment sits Thirteen minutes: the longest segment, and the only one with no new concept. Every piece has appeared before. Three jobs: add up, verify, hand off. Go slow; let people breathe.
Six stops collected; this one ties up. Shift your tone. Drop the new-material push; use “let's see what we've built.”
Biggest risk: reading the table aloud, not time. Leave it on screen; say three conclusions.

Script

Scroll to Step 7, the full total table on screen.

That's all seven steps. This table is everything we collected.

Three seconds to scan. Don't read.

I won't read it; it's on screen. I'll use it to answer three questions.

Question one: compute for a 128K forward pass?

52.5 PFLOP.

And 81.8% of that is attention's quadratic term. Not the 671B parameters: the term that scales with sequence length squared.

Question two: how much resident memory?

Weights 1.22 TiB, plus KV cache 8.58 GiB: 1.23 TiB total.

Question three: how many devices?

Point at the third card, the orange one.

1.23 TiB over one device's 94.74 GiB.

Beat, then the number. The lecture lands here.

13.3.

Devices don't come in tenths: at least 14.

Longer pause. Next line is the real point.

The number itself doesn't matter. What matters is that it came out of a division.

I never said “big models need parallelism” today. Not once. We computed, step by step, and it showed up on its own.

⭐ But don't remember 14 as “the answer”

Scroll to the yellow warning box.

14 devices is the most optimistic value possible. It rests on five assumptions, and none of them holds in real training.

Forward only. Batch 1. No activations or temp buffers. Ideally packed weights.

Scroll to the three-bar figure below, fully on screen.

Add the 41.89 GiB of peak activations, and 13.3 becomes 13.7. That's the second bar, almost as long as the first. This step is fine.

Setup done. Next number is the point.

But training is a different league. Look at the third bar.

16 bytes per parameter. Two for weights, two for gradients, four for the fp32 master copy, and four each for the two moments.

The third bar's five colored blocks.

Look at the proportions. The model is just the two bytes on the left; the 12 on the right are all optimizer.

671B times 16 is 9.76 TiB. Divide by 94.74 and you get about 105 devices.

Nearly eight times more than here. Topic 4 does the math; cutting the orange, purple and red blocks is basically all of it.

Say it: the message is the shape, not 105.

What this figure really says isn't 105. It's the shape: the first two bars are about equal, and the third is eight times longer.

Because ①, ② and ③ ask different questions: “do the weights fit?”, “does one forward pass fit?”, “can it be trained?”

Yet everyday talk lumps all three into “how many accelerators does this model need?” When you hear it, ask: which accounting?

So even the most frugal accounting needs 14 devices; real training, eight times that.

Another angle: how long is this compute?

Scroll to the three time cards.

PFLOP has no intuitive scale. Divide by hardware compute to get seconds.

A single TPU v7 device peaks at about 1,153 TFLOP per second in bf16. Officially 2,307 per chip, two devices per chip, so halve it. You'll meet that 2-to-1 again.

Theoretical limit, 100% utilization: 45.5 seconds.

Realistically, at 35% utilization: 2.2 minutes.

Slow. Let it sink in.

Over two minutes, just to push one 128K input through once. Forward only. No backward, no parameter update.

Spread over those 14 devices: 9.3 seconds.

The small print under the third card.

But that 9.3 comes with a clause: “assuming perfect, lossless parallelism.”

Two seconds: the doorway to the whole course.

That clause is the rest of this course.

14 devices don't make it 14 times faster. They swap data, wait for the slowest, and recompute to save memory.

How big that gap is, where it comes from, how to close it: Topics 5 and 7.

Breathe. Shift from computing to verifying.

⭐ Step 7.5: can we trust these derivations?

Scroll to Step 7.5.

Everything so far is derivation. So: can you trust it?

Yes, and checking doesn't need a device.

The compiler is a static analyzer: compile the model and parallelism config ahead of time (AOT), and it reports the memory needed, with no accelerator allocated.

You know the answer before queuing for hardware: the workflow's cheapest step.

Say this, or they'll expect a tutorial.

To be clear: this is not an AOT how-to. The course doesn't cover running it.

The three points are about how far to trust a tool that answers without running. Not specific to this tool: any performance model or capacity sheet gets the same verdict.

Setup done. Three points carry the segment.

We learned these three the hard way. They're not from a book.

① Compiles ≠ fits

A clean compile means only that the graph is legal. We've seen all-green compiles needing far more memory than a device has.

Next line is the real content.

Worse: the errors mislead.

Real compile failures often get reported as “out of memory,” so people shrink the batch. Exactly the wrong way.

A wrong error message is more dangerous than none. It sends you confidently the wrong way.

② The number isn't everything

It's only temp buffers. Parameters and optimizer state are a separate ledger.

And it doesn't shrink with parallelism: shard as wide as you like, temp buffers stay the same. The easiest to get wrong: you expect more devices to shrink it; it doesn't budge.

③ ⭐ It fails right near the limit

Far from the limit, AOT is very reliable.

But in a narrow band by the limit, it can't tell. Our misses all fell there.

Scroll to the three-color bar figure.

Big green zone left, small green zone right, a red band hugging the limit between. Green is accurate. Red isn't.

The key line. Say it slowly.

Now the text under the red band: configs worth trying nearly all squeeze into it.

Naturally. Who tries a config that obviously won't fit, or wastes memory?

Stop. Say the next line word by word.

In other words: it's least reliable exactly where you need it most.

That's not a bug in the tool but a property. Any tool would do the same.

So don't treat static analysis as an oracle. It rules out most obviously bad configs, but won't make the final call.

One more surprise

Same model, same parallelism. Swap one op's implementation and the memory needed changes, in no fixed direction: up or down.

Scroll to the dark takeaway box.

So Step 7.5 in one line: memory isn't a property of the model. It's a property of model × implementation × compiler.

Last part. Relax your tone.

Finally, a hook

All day, we've used 94.74 as the denominator.

For where the KV cache hits the wall, whether activations fit, and the 14 devices.

Beat.

But I never explained where it's from.

Why 94.74? Not 96, or 128?

Why does a single device peak at 1,153 when the official number is 2,307?

Why are there two devices in one chip?

Don't answer. Two seconds, then close.

That's the next lecture.

Today's job was only to compute that it doesn't fit, not to assert it.

Likely questions

In the peak-activations table, why is it “residual stream ×2”?

Heads-up: “×2” appears three times in that row, for different reasons. Don't lump them. Q/K one each, V/output one each: just two same-size tensors. Only the residual stream's ×2 needs explaining.

A layer first keeps a copy of the incoming x as is, then layer-norms x into a separate buffer. The sublayer computes on that, and the result is added back onto the kept copy.

The add-back is key: because of it, the original x can't be dropped. From the norm until the addition, two 1.75 GiB copies are live at once. That's the ×2.

The MoE layer repeats the same structure, so it's also ×2. The output step has no add-back, so it's one copy, as the table shows. Sharp eyes will spot it; use it as your opening.

13.3 devices: in practice, just use 16?

Almost never 14, and not just from rounding.

First, slice topologies have shape constraints. Not every count forms a valid interconnect topology; real sizes are powers of two or specific length × width × height shapes.

Second, and more important: 14 is a lower bound, not to be taken literally. As the yellow box said, training needs about 105; real planning starts there.

(Skip topology rules; that's Topics 2 and 5. “There are shape constraints” is enough.)

9.3 seconds: what do you get in practice?

This course hasn't measured it, so no number.

I can say where the gap comes from: devices swap data, synchronize, wait for the slowest one, and recompute to save memory. Together, not a few-percent effect.

That's the value of the clause “assuming perfect, lossless parallelism”: it labels an unknown instead of hiding it.

(If pressed, hand it to Topic 7, which measures and attributes it. Don't toss out a percentage. Every number in this lecture is computed or looked up; break that once and all earlier credibility goes.)

Attention: 81.8% of compute but 1.7% of parameters. A contradiction?

No contradiction; it's a contrast the course most wants remembered.

Parameter count sets how much you store, regardless of input length. MLA's 11.41B weights are the same whether you feed 4K or 128K.

Attention compute grows with sequence length squared. Long sequences let it eat the compute table; parameters don't budge.

In one line: parameters are static; compute varies with input. Infer one from the other and you're wrong; we saw it three times.

105 devices, 14 devices: are those chips or devices?

Devices, not chips: a factor of two.

94.74 GiB is per-device capacity, so anything divided by it counts devices.

A TPU v7 chip holds two devices. So 105 devices is 53 chips; 14 devices is 7 chips. Both right, different things.

You planted this 2-to-1 with 1,153 vs. 2,307; point back to it. It will bite again in later topics; make it clear now.

(Every count on the course page means devices; the figure captions say so too.)

How wide is that red band? Within what percentage counts as “near the limit”?

No reproducible number, so I won't give one.

I vouch only for the shape: narrow, hugging the limit, reliable either side, not within. Drawn width is illustrative; position isn't.

Width depends on model, op implementations and compiler version. That's the next point, “swap an op implementation and memory changes.” If it moves with the implementation, “a few percent” has no stable answer.

(Don't make up a number here. Every number in this lecture is computed or looked up; break that once and all earlier credibility goes.)

What exactly is AOT? What does it need?

It's “compile, don't execute.” From model + parallelism config, the compiler lays out the graph and memory; no hardware.

Its value is time, not precision. Devices mean queuing; compiling doesn't. Screen a dozen configs meanwhile.

(Don't teach running AOT here; out of scope. The audience needs only “this exists, and it has three known pitfalls.”)

Inaccurate where it counts: what good is it?

A lot, used in the right place.

It's good at ruling out, not confirming. Of a dozen candidate configs, it reliably flags the eight that clearly won't work, saving much queuing and trial and error.

It's bad at the final pick between two, which are exactly the ones hugging the limit.

So cut the bulk with it, then run the last two or three on hardware. A sieve, not a judge.

What not to say

Three red lines ① Never turn “13.3” into “so 14 devices can run V3.” It's a lower bound: forward only, batch=1, no activations. Someone will plan with it, off by 8×. Every time, follow at once with “this is the most optimistic value possible.”
② Present the three AOT points as “we hit these,” not general theorems. Measured on our own models and configs. Call them theorems, and the first failed reproduction elsewhere sinks the lecture's credibility.
③ After the 94.74 hook, don't explain it. Even “some is reserved for the system” kills the hook. That opens Topic 2; send people out with the question.
Pacing 13 min, five parts: three questions 3 min → three-accountings figure 3 min → timing 2.5 min → static analysis 3 min → hook 1.5 min.
Three mandatory pauses: a beat before “13.3”; two seconds after “assuming perfect, lossless parallelism”; two seconds after the three “whys.” The segment's power is in those pauses.
Don't read the total table row by row. Six rows by five columns: three minutes to read, and nobody remembers a number.
If the lecture runs over, cut “Another angle” (time conversion). But keep “assuming perfect, lossless parallelism” alone: it's the doorway to the course's second half.