The Life of a Token

No Transformer tour. A real model, taken apart.
One token, end to end, doing the math.

By the end, “why it needs a machine room” isn't a slogan but numbers you computed yourself.

Model DeepSeek V3 Setup 128K · batch 1 · bf16 · forward only Yardstick v7 device · 94.74 GiB ⏱ 60 min

Course author Chris Yang·Google Cloud AI Infra architect

This lecture has instructor notes (a script).

They cover how to teach: each segment's goal, where to pause, common misconceptions, answers to likely questions. The content is here.

Two uses. Presenting: the notes give lines, figure cues and timing. Self-study: hand the whole script to an AI; it becomes a TA that knows the full background and how to teach it, and walks you through.

◉ Live
128K
1
Presets

First, guess a number

Feed DeepSeek V3 a 128K-token document, one prefill: it reads everything, generates nothing yet.
After that pass, one thing must stay in HBM: every later token looks back at the K and V of every earlier token in every layer. That is the KV cache.
How large is the one this prefill leaves?
For scale: one TPU v7 device has ~94.74 GiB of usable HBM. (Throughout, “device” means one such device; each v7 chip holds two.)

Answer: 8.58 GiB, 9% of one device.
The other three are real too: four ways to store K and V, four generations of attention. Same model and document; only “values per token per layer” changes, yet the answer spans 128×. All of Step 2 lives in the gaps between them.

Where 8.58 GiB comes from: three steps, one multiplication each.

① One token, one layer

1.1 KiB

576 values × 2 B
576 = 512 (compressed KV) + 64 (RoPE)

② One token, all 61 layers

68.6 KiB

× 61 Per-layer copies; unavoidable

③ 131,072 tokens

8.58 GiB

× 131,072 9% of one device

Why the KV cache alone multiplies by layer count Activations live one layer: layer 3's are dead once layer 4 reads them, so no layer multiplier.
The KV cache is different. Token n+1 needs the K and V of all n earlier tokens in every layer. So it lives the whole conversation, × 61 layers, growing linearly with length, never freed.
The test: does the next token still need it? Yes → KV cache; no → drop it.

Four options, four storage schemes, all at 128K, 61 layers, bf16:

OptionSchemePer token/layer Full lengthvs MHA
~500 GiBMHA · each of 128 heads stores its own (classic)32,768 488.00 GiB1×
~30 GiBGQA · 128 heads in 8 groups, 16 heads share one copy (Llama-style)2,048 30.50 GiB16×
~9 GiB ✓MLA · compressed into one 512-dim latent vector (V3)576 8.58 GiB56.9×
~4 GiBMQA · all heads share one copy (smallest)256 3.81 GiB128×

MLA sits between GQA and MQA in size, yet should beat both on quality. (The paper never tests MLA against GQA or MQA directly; this chains two of its ablation tables. See Step 2.) MQA is smaller only because all heads share one copy; that expressiveness is gone for good. MLA saves nearly as much without sacrificing any head. That is Step 2's thread. (The GQA row assumes Llama's common 8 groups, not a V3 setting. Linear and sliding-window attention are omitted: they change whether each token is stored, not how much.)

This 56.9× is not a modest saving but a new order of magnitude At 488 GiB the KV cache alone needs 5.2 devices, before any weights, for one user.
8.58 GiB is 9% of one device.
One means “first, find more machines.” The other just fits.

The point isn't the numbers but deriving them yourself, and the counterintuitive things on the way. This lecture follows one token end to end, doing the math at each step.

Why DeepSeek V3

The GPT-3 175B era is long gone.

The gold standard

Most later open models start from it. To show what a modern LLM looks like, dissecting this one beats surveying ten.

Three wins at once

MLA shrinks the KV cache, fine-grained MoE the active parameters, FP8 the training cost.

Each hits a hardware bottleneck

HBM, compute, bandwidth: three bottlenecks, three answers. This course builds that chain.

⭐ “The standard” is checkable, not an opinion Tencent's Hy3 (Hunyuan 3, 295B/21B), open-sourced in 2026, copies V3's MoE field by field: sigmoid routing, auxiliary-loss-free expert bias, a shared expert, MTP, even the config field for “keep the first k layers dense” has the same name. Only attention changed: GQA instead of MLA.

So this lecture dissects V3. Understand it, and later open models are the same parts rearranged.

How big is 671B?

A parameter count means nothing alone, only side by side.

Full bar = total parameters (HBM holds them). Dark left end = active (compute pays for them). (The V3 row uses blue and green instead.) Dense models: equal. MoE splits them, the root reason LLMs kept growing these past two years.
⚠️ Bottom bar: Kimi K3 leads in total parameters (2.8T), not sparsity: 2800 ÷ 104 = 26.9×, below V4's 32.7×. It spends sparsity on expert granularity: 16 of 896 experts, 1.8% lit (V3: 8 of 256, 3.1%). Size, sparsity and granularity are three separate rankings.
One takeaway from this figure V3 has 3.8× GPT-3's total parameters, yet each token costs 0.21× the compute. Since 2024, “bigger” no longer means “more expensive.” Every later calculation rests on that split.

Assumptions

Every step uses these, unchanged. Edit the top console and every number follows.

Sequence length

128K

131,072 tokens

batch

1

One sequence first, then multiply

Precision

bf16

2 B for weights and activations

Direction

Forward only

Backward, optimizer state: Topic 4

Why forward only, for now Forward is fact, derivable step by step. Backward is decisions: what to recompute, where optimizer state goes, gradient accumulation. They'd muddy the thread.

The model card

All later math draws from this table, straight from the official config.json; nothing estimated.

BackboneValueNotes
hidden_size7,168d: residual stream width
num_hidden_layers61
first_k_dense_replace3First 3 dense, other 58 MoE
vocab_size129,280
max_position_embeddings163,8404K, YaRN-extended
tie_word_embeddingsfalse⚠️ In, out: two matrices
num_nextn_predict_layers1MTP
Attention · MLAValueNotes
num_attention_heads128
q_lora_rank1536Q low-rank too
kv_lora_rank512⭐ KV → 512
qk_nope_head_dim128QK/head: 192
qk_rope_head_dim64
v_head_dim128
MoEValueNotes MoEValueNotes
n_routed_experts256 moe_intermediate_size2048⭐ Fine-grain
n_shared_experts1Every token intermediate_size18,432First 3 layers
num_experts_per_tok8top-8 n_group / topk_group8 / 4Grouped
scoring_funcsigmoidNot softmax topk_methodnoaux_tcNo aux loss

The route

Text in, probability distribution out. Seven steps between.

Residual stream · d = 7,168 · through all 61 layers Text "unhappy" Tokenize → subwords → token id no weights · no FLOPs ① Embed 129,280 × 7,168 id lookup × 61 layers ② MLA 128 heads latent KV cache either/or ③ MLP · first 3 ④ MoE · last 58 ⑤ residual · norm ⑥ lm_head 7,168 → 129,280 separate copy Probabilities 129,280-d Ask at every step: ① weights ② activations ③ FLOPs ④ what blows up here? ←often the key
The seven-step route. The residual stream is the bus: every layer reads and writes it.
Left two white boxes: outside the model. Tokenization (text → subwords → token ids) happens before ①: no weights, no FLOPs (Step 1 explains the 129,280 vocab).
③ and ④ share a slot: either/or, not sequential. First 3 layers dense MLP, the other 58 MoE.
① Embed
② MLA
③ Dense
④ MoE
⑤ Bus
⑥ Exit
⑦ Total
So far— weights— HBM— compute
An ancient scroll hangs above a pool of light; one character leaves the page and ignites into a golden spark, sending teal ripples across the water
Step 1 · Where the journey starts From text to vectors A word becomes a list of numbers. One table lookup, no compute, yet HBM is already being spent.
Step 1

From text to vectors

No compute, one table lookup. Yet a chunk of HBM is gone.

Tokenization: why 129,280

First, context: on one axis with its contemporaries' vocabularies, it is nothing special.

Bigger vocab: gains grow → · Same text, fewer tokens → shorter sequence, attention's N² term drops · Rare words stay whole → names, code, rare glyphs 32K 64K 128K 256K Vocab (log scale, 2×/tick) Llama 2 32,000 Llama 3 128,256 gpt-oss 201,088 Qwen3 151,936 Gemma 2 256,000 DeepSeek V3 129,280 Bigger vocab: costs grow → · Both vocab tables grow → negligible at V3 scale, see below · Rare words: fewer samples → undertrained; at worst glitch tokens · Output softmax spans the vocab → cost grows with it ⭐ Mainstream vocabs grew 4–8× in 2+ years; 129,280 is mainstream, not magic
All six are the vocab_size in each model's public config.json. Llama 3's 128,256 and V3's 129,280 are practically one point; their markers overlap.
Expand: what “two bigger tables” costs— why V3 can afford 129K, a small model can't

A vocab matrix's size depends only on vocab × d, not on layer count. So the deeper and larger the model, the smaller its share.

// V3: in + out tables, unshared (tie_word_embeddings = false) 129,280 × 7,168 × 2 = 1.853B → of 671B: 0.28% // Counterexample: same 129,280 vocab, d = 2,048, 12-layer model 129,280 × 2,048 × 2 = 530M // 2 tables 12 × 12 × 2,048² = 604M // body at 12d²/layer, ballpark only → vocab is almost half

So the “bigger tables” cost in the figure barely holds V3 back. The real costs are undertrained rare words and a more expensive output softmax.

What the research says A paper on vocabulary scaling laws (Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies, NeurIPS 2024, arXiv 2407.13623) concludes that most LLMs' vocabularies are too small. Its fitted formula puts Llama2-70B's optimum at 216K or more, against an actual 32K; at the same compute budget, raising the vocabulary from 32K to 43K lifts ARC-Challenge from 29.1 to 32.0.
⚠️ One limit: the paper fits dense models from 33M to 3B. Extrapolating it to a 671B MoE has no basis. It points toward “bigger,” not that 129,280 is optimal for V3.
Rare words that never get learned have a name: under-trained / glitch tokens (arXiv 2405.05417; the famous SolidGoldMagikarp is one).

Why the width is 7,168

Remember d = 7,168, the model's main highway width; every later step uses it. Why: 7,168 = 128 × 56 = 256 × 28. Divisible both ways: no padding waste on TPU or GPU.

Expand: hard constraint vs. DeepSeek's own call— easy to conflate

Hard constraint: align with the hardware's matrix units. A TPU MXU is a systolic array, 256 × 256 on v6e and TPU7x, 128 × 128 on earlier generations. NVIDIA recommends dimensions in multiples of 8 for FP16 and of 64 on A100. 7,168 divides by both 128 and 256, so neither side pads. No one picking this number escapes it.

The trade-off (soft): for a fixed parameter budget, wider or deeper. Every model makes that call afresh, and DeepSeek has never publicly said how it arrived at 7,168.

A constraint isn't a reason Alignment explains only why it isn't 7,000 or 7,200, not why it isn't 6,144 or 8,192, which divide just as evenly. Merge them into “so they chose 7,168” on stage, and any expert will call you on it.

Embedding: a lookup, by default

Fetch the token id's row from the 129,280 × 7,168 table. On this path, compute is zero. Real frameworks don't always look it up; see below.

// embedding matrix parameters 129,280 × 7,168 = 926.68M → 1.73 GiB // size of the tensor after the lookup 131,072 × 7,168 × 2 B = 1.75 GiB
Expand: how true is “a lookup, not a matmul”?— in production it's often a real matmul

Mathematically it equals one-hot vector × embedding matrix. Nobody would really do that multiply: 129,280 multiply-adds per value. Yet real training frameworks run both, behind an explicit switch.

// MaxText layers/embeddings.py: both if cfg.use_iota_embed: iota = lax.iota(jnp.int32, self.num_embeddings) one_hot = jnp.array(inputs[..., None] == iota, dtype=self.dtype) output = jnp.dot(one_hot, embedding) // ← real matmul else: output = embedding.at[inputs].get() // ← gather, zero compute

In base.yml the switch defaults to false, i.e. gather. But 34 production configs in the repo set it on explicitly, none off. So do GPU model configs (llama3, mixtral), not just TPU ones. Default and actual usage are opposites.

Switched on, what does this step cost? Exactly as much as lm_head in the summary table at the page's end: same shape, one 129,280 → 7,168, the other 7,168 → 129,280. Later we call them mirror images. With a lookup the mirror holds only for parameters; with a matmul, for compute too:
2 × 129,280 × 7,168 = 1.85 GFLOP / token // sequence = 242.92 TFLOP of the step's 52.50 PFLOP: 0.46%

Precisely: “zero compute” holds only for the gather path. Compiled the other way it isn't zero; the denominator just buries 0.46% in the noise. Why pay for it? It turns an irregular sparse row fetch into a regular dense matmul, shardable along vocab, with a matmul backward too. The most plausible reason, not one any public source states; don't treat it as settled.

Same memory either way: the table stays resident; fetched rows take space too (right).

Vocab table 129,280 × 7,168 token #8492's row ↕ 129,280 rows ↔ 7,168 cols lookup Residual stream 1.75 GiB ↕ 128K tokens × batch ↔ 7,168 Read/written by every layer

⚠️ Easy to assume: d needn't divide by the head count

In standard MHA, hidden = heads × head_dim must hold. In V3 it doesn't, by a wide margin.

① Standard MHA: heads × head dim = d heads × head dim = d exactly; heads just slice the stream (8 drawn, any count works) ② V3 MLA — 128 heads × 128-dim output = 16,384 1.29 × d wider than the stream 16,384 = attention width, not a multiple of 7,168 o_proj 16,384 × 7,168 117.44M · largest MLA weight per layer (63%) ↙ folds the 1.29× back back to residual stream 7,168 ↑ d = 7,168 · shared baseline, drawn to scale ⭐ MLA lifts the old rule “attention width = residual width”; cost: 117.44M o_proj
Both rows share the d = 7,168 baseline, true scale. Top, old rule: concatenated heads fill the stream exactly. Bottom, V3: attention widens to 16,384, then o_proj folds back to 7,168.
Expand: this figure also explains a recurring fact— why o_proj is a layer's largest block

o_proj is the largest weight in an MLA layer (117.44M, 63% of the layer). Nothing special: it just spans the gap above, 16,384 in and 7,168 out.

o_proj = 16,384 × 7,168 = 117.44M MLA weights per layer = 187.11M → 117.44 / 187.11 = 62.76%

More important: once that gap opens, the head count and d can be set independently. That is what lets Step 2 keep all 128 heads.

The first eye-opening number Before layer 1, turning tokens into vectors already takes 1.75 GiB. And this tensor isn't discarded after use: it is the residual stream, read and written once by each of 61 layers.
① Embed
② MLA
③ Dense
④ MoE
⑤ Bus
⑥ Exit
⑦ Total
So far— weights— HBM— compute
A churning sea of teal cloud being drawn into a jade tablet, while a single gold thread skirts the vortex and flies off on its own
Step 2 · Heaviest of the course MLA: attention and the KV cache war Squeeze a whole cloud into one bead to store; expand it on use. Position, the gold thread, must route around the squeeze alone.
Step 2 ⭐

MLA: attention and the KV cache war

The heart of the lecture and of V3. Budget 15 minutes.

The weak spot: a runaway KV cache

Attention lets every token look at all earlier tokens. To avoid recomputing their K and V, we keep them around. That is the KV cache. Its size:

KV cache = seq length × layers × values per token per layer × bytes ↑ linear ↑ linear

Two linear terms multiplied: disaster. 4K → 128K tokens is 32×, then × 61 layers. That is why long context forces attention to change.

“Disaster” is vague. Build it step by step.

Log scale, 10× per gridline, or the first bar vanishes. Left to right spans 8M×, from two ordinary steps: × layers, × tokens. Blue bar: same tokens, stored with MLA.

① One token, one layer

64.0 KiB

32,768 values × 2 B negligible

② One token, all 61 layers

3.8 MiB

× 61 still harmless

③ 131,072 tokens

488.00 GiB

× 131,072 alone fills 5.2 devices

Startling enough to memorize Stored this way, one conversation of 25,446 tokens fills a whole TPU v7 device (94.74 GiB) with cache alone. No weights are loaded yet, and that is one user.
With MLA, the same device holds 1,447,608 tokens.

The flare: why long context breaks it

Draw one line per storage scheme. Both are straight, but their slopes differ by 56.9×: same origin, ever-widening gap.

X: sequence length; Y: KV cache; both axes linear, no visual squashing. Red area: what a different storage scheme saves, a gap that widens with length, not a fixed amount. Drag the slider to open it.

Three knobs, four generations

Every attention variant turns the same knobs; each generation changes how much a token stores. Split “values per token per layer” once more and the knobs appear:

per token per layer = 2 × KV heads × head dim // 2 = one K + one V; fixed // V3 with the most traditional scheme: 2 × 128 × 128 = 32,768
KnobEffectWho
① Copies
KV heads
128 → 8 → 1. Straight division, the strongest. Cost: Q heads share one K/V; lost expressiveness never returns MQA, GQA
② Width
head dim
Rarely changed; most use 128. Sets what one head holds: narrower is dumber. Kernels suit few widths; odd ones save memory, run slowly —
③ Layers
layers
Layers share KV, or only some see full length (sliding window). Whether to store, not how much: so not one of the four layer sharing
window attn.
⭐ Up front: MLA turns none of these knobs 128 heads × 128 dims, 61 layers: unchanged. It changes what is stored: not expanded K/V but a 512-dim compressed vector plus 64 separate position dims, 576 in all.
MLA's 576 sits between GQA's 2,048 and MQA's 256, yet beats both on quality (inferred, not measured directly: one table in the paper shows MHA beating GQA and MQA, another MLA beating MHA). Impossible on one axis; MLA is off that axis.
MHA · KV per head 2017 · Transformer 128 Q heads 128 Q heads ↔ 128 KV 32,768 1× 488.00 GiB @ 128K ctx best quality max cache MQA · one shared KV 2019 · Shazeer 128 Q heads 128 Q heads ↔ 1 KV 256 ÷128 ≈3.81 GiB cache plunges clear quality loss GQA · grouped 2023 · Ainslie et al. 128 Q heads 128 Q heads ↔ 8 KV (16 per KV) 2,048 ÷16 30.50 GiB compromise, industry default MLA · one latent vector ⭐ 2024 · DeepSeek 128 Q heads latent 128 Q heads ↔ 1 latent 576 ÷57 8.58 GiB = 512 latent + 64 RoPE cache ÷57, better quality Further right, less stored per token → Every column: 128 heads × 128 dims × 2 (K, V) = 32,768, as in the table above First three share; MLA compresses instead
Four generations. The first three share KV: less memory, less expressiveness. MLA compresses, not shares.
⭐ The point: not the number but what pays for it The first three trade quality for memory; shared KV loses expressiveness for good.
MLA trades compute for memory. Store 576, expand to 128 heads on use. Nothing lost; each use costs extra compute.
Same shape as training's recomputation: don't store intermediates, recompute on demand. In a sense, MLA is recomputation applied to the KV cache.
⚠️ “MLA saves memory” holds only for inference Training has no KV cache. Only decode caches: one token at a time, earlier K/V kept for the next step. Training is one forward pass over the sequence; no next step, nothing to cache.
Training's bill is activation memory, and at this scale recomputation is maxed: a layer keeps only its input residual stream; the 128-head K/V are recomputed, used, discarded. MHA and MLA store the same stream: a tie.
Not “MLA saves training memory too”, nor “useless for training”: it pays off in two other places. With recomputation relaxed a notch, store the 512-dim latent, not 32,768 values; context parallelism passes the latent around its ring instead of K/V: a communication saving, not memory. Both in Topic 4.

How MLA actually works

Compress on write, expand on read. Not 128 per-head copies of a token's KV but one 512-dim latent; at attention time a matrix expands it back to 128 heads.

First, what RoPE is Attention is blind to order: shuffle a sentence, same result. Order must be supplied. RoPE rotates the vector by a position-dependent angle instead of adding to it: token 1 by 1°, token 100 by 100°. (Simplified: really, dimensions pair up, each pair at its own rate. The fastest turns 1 radian per position, the rest ever slower; the slowest takes many positions per turn.) A dot product then carries the angle difference: relative distance.

Why position needs its own path. The angle depends on position. Rotate, then compress, and position leaks into the latent; stuck between two matrices, the rotation can't be moved out: matmul doesn't commute.

Precisely: correctness survives; it trains and runs. A valuable inference shortcut breaks: each new token recomputes every earlier K. Which shortcut: see the shape chain below.

So V3 splits each head: 128 dims without position (compressed), 64 dims with it (stored raw).

576 = 512 + 64 All MLA stores per token per layer, independent of head count: 128 heads or 256, 576 values.
input h · 7,168 kv_a_proj separate path c_KV · 512 · cached k_rope 64 ↓ these 576 are all that's stored ↓ expanded on the fly (not cached) kv_b_proj 128 heads × (K 128 + V 128) 128 heads share 64 RoPE QK dim per head = 128 non-RoPE + 64 RoPE = 192 store 576, expand to 128 heads on use · compute for HBM

⭐ An easy trap: 7,168 and 16,384

Almost everyone first computes 7,168 ÷ 128 heads = 56. That hides two misconceptions; the second is deeper.

Misconception 1: V3 heads are 56-dim Each head is 128-dim. 128 of them give 16,384, 2.29× the residual width. On V3 the two differ.
Misconception 2: “128 slices of 56” (the worse one) The input is never split. Each head reads all 7,168 dims through its own 7,168 × head_dim matrix into its own space.
The “split” is on the output side: the heads' outputs concatenate, and that width is 7,168 or 16,384.
Even in a standard model the division means “the concatenation equals the input width,” not “the input was divided.”

The section rests on this. See it, and “why wider heads?” answers itself: each head's matrix is 7,168 × d_h; d_h was always free.

The fan, left: every head reads the full 7,168. The rows differ only in output width per head: the textbook choice concatenates to exactly 7,168; V3 to 16,384, and o_proj maps that back into the residual stream. Right: the KV cache each design pays.

Where the rule came from

A choice, not a definition, in the 2017 Transformer paper. §3.2.2 writes W^O ∈ R^(h·d_v × d_model): the two never had to match.

The next sentence: “In this work we employ h = 8 … we use d_k = d_v = d_model/h = 64.” “In this work”: a chosen value, not a rule. The reason given is plain:

The paper's reason (Vaswani et al., 2017 · arXiv 1706.03762) “Due to the reduced dimension of each head, the total computational cost is similar to that of single-head attention with full dimensionality.”
A compute budget, not a mathematical necessity.

Seven years of copying made it intuition. It was only a convention.

Why V3 can widen

In standard attention, wider heads cost cache: each head stores K and V, 2 × heads × head_dim. Width and cache are locked, linearly.

MLA cuts the lock. The MLA paper's Table 1 gives all four cache formulas; MLA's is (d_c + d_h^R) × layers, no head count, no head dim.

A new degree of freedom Attention as wide as you like; KV cache not a byte bigger. Expressiveness and cache: separate knobs, a first.

Evidence: DeepSeek-V2, MLA's debut, has a 5,120 residual stream but 128 × 128 = 16,384 attention, 3.2×, bolder than V3. (DeepSeek never claims causation, but two generations alike is hardly coincidence.)

The cost: three bills, V3 pays two

Benefits are half the story. Widening adds three costs; another design cancels the third.

Equal-width design = 1.00×; what widening costs ① attn FLOPs 2.86× QK at 24,576, AV at 16,384 — unavoidable ② o_proj weights 2.29× 51.38M → 117.44M — unavoidable ③ Q proj · one step 3.43× 51.38M → 176.16M, 2.29× wider ×1.5 RoPE split ③ Q proj · real 0.95× 48.76M via q_lora: 7,168 → 1,536 → 24,576 · 5.1% below equal width ↑ baseline = 1.00× ⭐ Costs ① ② are paid; ③ is absorbed by q_lora's 1,536 waist: 2.29× wider, yet fewer Q params One reason q_lora_rank exists: it saves weights, not KV cache (Q is never cached)
Three bars, one 1.00× baseline, linear length. Dashed yellow = “V3's shape without the q_lora waist,” not another attention. Green, below, is what shipped.
The line inside yellow: 3.43× is two independent decisions multiplied: 2.29× for widening, 1.5× (192 ÷ 128) for MLA's separate 64 RoPE dims.
⚠️ Two different “what ifs.” The right segment exists only with MLA. Plain MHA rotates all 128 dims, so no 192: Q is 7,168 × (128×128) = 117.44M, 2.29×, same as o_proj, and yellow stops at the line.
Expand: full math of the three costs— source of the figure's four multiples
CostEqual widthWidened Result
① Attention FLOPs
QK and AV widths differ; count each
QK 56 + AV 56
= 112
QK 192 + AV 128
= 320
2.86×
really paid
② o_proj weights
must bridge the 16,384 → 7,168 gap
7,168 × 7,168
51.38M
16,384 × 7,168
117.44M
2.29×
63% of a layer
③ Q projection weights
128 heads, each a 7,168 × d_h matrix
7,168 × (128×56)
51.38M
Widen only (plain MHA)
7,168 × (128×128)
117.44M; RoPE works inside these 128; no 192
2.29×
widening alone
③ cont.: but V3 uses MLA. RoPE skips those 128 dims; it gets its own 64 — V3's shape, one step
7,168 × (128×192)
176.16M
another ×1.5
3.43× in all
What ③ really does: detour via q_lora
7,168 → 1,536 → 128×192: a narrow waist
11.01M + 37.75M
= 48.76M
÷3.61 vs 1-step
−5.1% vs equal

Check: 7168×7168 = 51,380,224 (note 128×56 = 7,168, so at equal width Q projection and o_proj match); 7168×16384 = 117,440,512, 117.44 ÷ 51.38 = 2.29; 7168×24576 = 176,160,768, 176.16 ÷ 51.38 = 3.43; 11.01 + 37.75 = 48.76; 176.16 ÷ 48.76 = 3.61; 48.76 ÷ 51.38 = 0.949.
How the two parts of ③ multiply: 2.29 × 1.5 = 3.43, where 1.5 = 192 ÷ 128. Only MLA has that part.
Why ① is 2.86, not 2.29: attention's two matmuls have different widths. QKT runs at qk_head_dim = 192, AV at v_head_dim = 128, so (192+128) ÷ (56+56) = 320 ÷ 112 = 2.857. Every FLOP figure here is computed per segment; a single 16,384 under-reports attention compute.

So that's what q_lora_rank is for Attention 2.29× wider, yet Q has slightly fewer parameters than equal width. The 1,536 waist absorbs it.
The two LoRAs differ:
kv_lora_rank = 512 saves KV cache (it is what's stored);
q_lora_rank = 1536 doesn't: Q is never cached.
Hence “saves parameters, not cache” in the five-row table's q_a_proj row.
⚠️ Attribution. The weight saving is our calculation, not the paper's. The paper: “to reduce the activation memory during training” (V2, end of §2.1.2), even though it cannot reduce the KV cache. Both true; don't credit the paper with ours.
Cost ① is real Attention compute rises; the original paper chose d_model/h to save it. V3 knowingly buys expressiveness with compute: no free lunch.
② too: o_proj, 117.44M, is a layer's largest MLA block, hence top of the five-row table.

⭐ The full shape chain: [128K, 7168] to Q / K / V

Now every step in the layer. In and out: [128K, 7168]. Between, three very different paths: Q, K, V.

One box, one transformation: op / weight shape / output shape. The gold-dashed pair is all inference caches: 512 + 64 = 576 values per token per layer. k_pe has 1 head, shared by all 128: 64, not 128×64.
Drawing notes: ① kv_a_proj actually outputs [c_KV 512 | k_pe 64]; k_pe is drawn left to sit by k_nope without crossing lines. ② Weights are [in_dim, out_dim], as in the code's nn.Linear(in, out).

① Q path: down, then up

Q: two projections, narrow waist. q_a_proj takes 7,168 to 1,536, Q's latent; after RMSNorm, q_b_proj lifts it to 24,576; a view makes 128 heads × 192.

Easy to misremember: 24,576, not 12,288 Each head is 192-dim, not 96 or 128. 192 = 128 (unrotated) + 64 (rotated).
So q_b_proj has shape [1536, 128×192] = [1536, 24576]. 128×128 = 16,384 is the output-side width (v_head_dim × heads), at o_proj's input, not here.

Then each head's 192 dims split: first 128 (q_nope) untouched, last 64 (q_pe) get RoPE. Rejoined, query_states is final: [128K, 128, 192].

No byte of Q is cached. Its compression only saves weights and training activations (see the cost table).

② KV path: store 576, compute the rest on the fly

K and V share one down-projection. kv_a_proj_with_mqa takes 7,168 to 576, split at once: first 512 c_KV (shared K/V compression), last 64 k_pe (the position strip).

⭐ The page's key sentence Only these two blocks are cached: 512 + 64 = 576. Everything downstream (k_nope, value, key_states) is computed on the fly, never stored.
The MLA paper's formula: cached elements = (d_c + d_h^R) × layers, with neither head count nor head dim in it.

After RMSNorm, kv_b_proj lifts c_KV to 32,768; a view makes 128 heads × 256, halved: first 128 k_nope, last 128 are V. value_states is final: [128K, 128, 128], never touching RoPE.

k_pe, after RoPE, is broadcast, one copy, to all 128 heads and joined to each head's k_nope: key_states is final: [128K, 128, 192], matching Q.

③ Why RoPE needs its own 64-dim branch

MLA's twistiest, most-skipped part. Why not encode position across the whole vector? Because the cache just saved would be given right back.

If all of K carried RoPE Inference's elegant shortcut: fold K's up-projection into Q's matrix in advance. K is never materialized; Q scores directly against the cached 512-dim c_KV.
But RoPE, a position-dependent rotation, would sit between them. Matmul doesn't commute; the middle can't be moved out, so the fold breaks.
Result: every new token must recompute every earlier token's K.
Expand: the paper's words— DeepSeek-V2 §2.1.3
“RoPE is incompatible with low-rank KV compression … W^UK cannot be absorbed into W^Q any more during inference, since a RoPE matrix related to the currently generating token will lie between W^Q and W^UK and matrix multiplication does not obey a commutative law. As a result, we must recompute the keys for all the prefix tokens during inference …”

Note the last sentence: we must recompute the keys. “Must recompute,” not “computes it wrong.” What breaks is efficiency, not correctness. At the podium, say “the result is wrong” and experts will correct you.

The fix: position on a side path

Split each head's vector into two jobs:

PartDimsRoPE?Role
q_nope / k_nope128 ❌ noContent; keeps matrices foldable
q_pe / k_pe64 ✅ yesPosition; all in these 64
Concat & score192— score = content + position, one dot product

Dot products add: [a;b]·[c;d] = a·c + b·d. One 192-dim dot product equals “content similarity” plus “positional relevance,” computed apart. No interference; each half keeps its math.

Nearly free The position strip is 64 dims, one copy for all 128 heads (hence [128K, 1, 64]).
64 cached values buy “no K recomputation”: the 64 in 576.
One per head would add 128 × 64 = 8,192: 576 → 8,704, and MLA would be pointless.
Expand: why Q's 64 dims aren't shared— cost, not math, decides
Position is per token, not per head: all heads rotate by the same angle. So why does K share one copy while Q has 128 (128 × 64 = 8,192)?
Those 64 dims aren't pure position. The paper's eq. (14) is RoPE(W^QR · c_Q): project a vector from the content, then rotate. The 128 heads use different rows: same angle, different thing rotated.
Expand the position term (rotations are orthogonal: R_t^T × R_s = R_(s−t)):
// head i, query at t, key at s (R_t · a_i) · (R_s · b) = a_i^T × R_(s−t) × b // middle: relative distance
The middle is distance only; the sides are content. a_i differs per head, so each head has its own “distance → score” curve. Some heads watch neighbors; others look far back for a match. Share one copy and all 128 heads get the same curve.
⭐ But cost decides it, not expressiveness: Q isn't cached, so sharing saves nothing: expressiveness lost for free. K is, and sharing shrinks 8,192 to 64: a loss worth taking.
The asymmetry is a cost decision, not a math requirement: compromise only where it pays.
Caveat: the fold is an inference optimization, not the model definition The reference implementation (Hugging Face's modeling_deepseek.py) materializes K and V, then runs attention. The figure draws that path: the math definition.
Absorbing the up-projections into Q and o_proj is an equivalent rewrite done only by fast inference engines. Leaving room for it is why RoPE was isolated at design time.
The paper (V2, end of Appendix C): “due to the associative law of matrix multiplication, we can absorb W^UK into W^UQ, and W^UV into W^O. Therefore, we do not need to compute keys and values out for each query.”
Expand: are the “128 heads” real or bookkeeping?— heads have a lifecycle

A natural question after the chain: throughout, the weights are a few big matrices, and view moves no value. So are the 128 heads just bookkeeping?

Projections: yes, pure bookkeeping q_b_proj is one block, 1,536 × 24,576; kv_b_proj is one block, 512 × 32,768; o_proj is one block, 16,384 × 7,168.
Not 128 small matrices: three big ones. view / reshape just re-reads: no data moved, no compute.
“Head i's projection” is the big matrix's i-th column group.
In scoring: heads are physical, unmergeable Without heads, one dot product of full 24,576-dim Q and K: one score per pair.
With 128 heads, each pair gets 128 scores, then each runs its own softmax and weights its own V slice.
Results differ completely, and the second isn't one big matmul: softmax is nonlinear, run per head. Linear steps merge however arranged; a per-group nonlinearity can't be rejoined.

Precisely: a head has a lifecycle. At the view it only gets a name. It is truly alive during scoring and softmax. At o_proj it dies: that 16,384 × 7,168 matrix is the only place the 128 heads ever exchange information. Both ends are single blocks; only the short stretch in the middle is real.

The hardware shows it too: projections are large, regular matmuls, accelerators' favorite. Scoring is batch × 128 small batched matmuls, inherently lower arithmetic intensity: one reason attention has long been a bottleneck.

⭐ Why so much compression? The 32,768 never existed

56.9× looks like magic; magic has a trick. Most of it isn't compression, just not being wasteful.

Linear in element count. The bottom bar is 12 pixels: the true ratio, not an error.
⚠️ The “two cuts” split is our derivation, not the paper's. Check: 32,768 ÷ 7,168 = 4.57, 7,168 ÷ 576 = 12.44, 4.57 × 12.44 = 56.9.

Cut 1: 32,768 values, 7,168 degrees of freedom

Standard MHA stores 2 × 128 heads × 128 dims = 32,768 values per token per layer. Where from?

All from the same 7,168-dim input. K and V are each a 7,168 × 16,384 matrix times h. A 7,168 × 16,384 matrix has rank at most 7,168.

Provable, not a hunch The 32,768-dim vector always lies in a ≤7,168-dim subspace.
Store those 7,168 values (the input) and K and V rebuild bit for bit.
So 4.57× of standard storage is pure redundancy; cutting it isn't compression.
Why did nobody store just the input? Recomputation. Store only h and each new token re-multiplies every earlier K and V. Memory saved, compute paid.
MLA's real contribution: that recomputation almost never happens (matrix absorption), so the 4.57× is collectible at last.

Cut 2: cap the rank at 576, the real compression

The rest, 7,168 → 576, 12.44×, is real and lossy, but compresses rank, not value count.

Picture c_KV as a 512-wide bus all 128 heads read from. Each reads via its own matrix and gets something different (diversity intact), but none gets what isn't on the bus.
Gold dashes: k_pe's 64 dims bypass the bus, straight from h, the paper's eq. (15), k^R = RoPE(W^KR · h). Position stays off the bus, for the absorption reason explained above.

MLA's bet: the 128 heads together need at most 512 information channels.

⭐ The paper's case: no theory, just a clean experiment Same scale and architecture, only attention swapped (V2 paper, Appendix D, 250B MoE):
KV cache 860.2 K → 34.6 K, 4% of MHA; BBH, MMLU, C-Eval, CMMLU: wins all four.
And MHA is the baseline that saves nothing and has the best quality.
Expand: both ablation tables, and why 57× smaller wins— one cell loses; we say so
Be clear: the paper shows what works, not why DeepSeek-V2 §2.1.2 offers no theory: no rank analysis, no information theory, just “here's what we do.” The “two cuts / bus” story above is ours, not the paper's.
Instead: a head-to-head ablation (Appendix D). That data is the real answer.
Benchmark Small MoE
MHA
Small MoE
MLA
Large MoE
MHA
Large MoE
MLA
Total params15.8B15.7B 250.8B247.4B
KV cache / token
elements
110.6 K15.6 K 860.2 K34.6 K
4% of MHA
BBH37.939.0 46.650.7
MMLU48.750.0 57.559.0
C-Eval51.650.9 57.959.2
CMMLU52.353.4 60.762.5

Source: DeepSeek-V2 paper, Table 9. Same scale, same architecture, only attention swapped. Training tokens: small 1.33T, large 420B. Large wins all four; small wins three, loses one (C-Eval 51.6 → 50.9; say this out loud).

It beats the strongest baseline, not a pushover Table 8 of the same paper first compares a 7B dense model, MHA vs GQA (8 groups) vs MQA.
MMLU: MHA 45.2 > GQA 41.2 > MQA 37.9; C-Eval: MHA 42.9 > GQA 37.7 > MQA 30.0.
So the traditional cache savers (GQA / MQA) really do cost quality. MLA beats that no-savings, best-quality MHA.
⚠️ This is an inference that chains two tables: Table 8 compares MHA / GQA / MQA on a 7B dense model, Table 9 compares MHA / MLA on MoE models, and no table in the paper puts MLA directly against GQA or MQA. “MLA beats both GQA and MQA” rests on a transitive step: MHA beats them, and MLA beats MHA.
Hardest question: 57× smaller, yet better? The paper doesn't say; both points below are speculation Speculation 1 · The low-rank bottleneck itself regularizes. One narrow opening for every head may force cleaner, more shared representations. Seen elsewhere; no controlled experiment here backs it.
Speculation 2 · Savings went elsewhere: not “compressed vs. uncompressed.” The freed cache let attention widen to 128 × 128 = 16,384, 2.29× the residual stream. Really: “narrow bottleneck + extra-wide heads” vs. “no bottleneck + regular heads.”
Don't claim causation DeepSeek never claimed. What's certain: in their own controlled tests, the trade paid off.
The three roles of 512 One knob sets three things:
① Cache size: it's what is stored;
② Expressiveness: the rank cap from h to K/V;
③ Whether “never materialize K” pays: narrower bus, smaller folded matrix.
RoPE's 64 dims stay off the bus to protect ③.

⭐ Q, K, V in hand: how attention plays out

Everything so far prepared these three tensors. Now attention does its actual work.

Top: five-step pipeline. Bottom: the score matrix in miniature. Masked above the diagonal; softmaxed rows sum to 1: a probability distribution.
⚠️ Shading and percentages are illustrative of rows summing to 1, not model outputs.

Intuition: at a sentence's fifth word you glance back at the first four and judge which matter most. Scoring computes that, with 128 heads applying their own criteria at once.

① Dot product: who looks at whom Dot the current Q with every K. Aligned: large (“look closer here”); unrelated or opposite: small or negative.
Each head separately: 128 matrices of 128K × 128K.
② ÷ √192: no magic, anti-saturation A sum over 192 dims grows with dimension. Raw into softmax, it saturates: one position takes nearly all weight, the rest go to 0, and gradients vanish.
Dividing by √d restores the range. 192, not 128: each Q/K head is “128 unrotated + 64 rotated.”
Expand: two details of “÷ √192”— skippable in class, but don't get them wrong
Two details to nitpick ① The original paper says “we suspect,” not “we prove.” Attention Is All You Need §3.2.1: “We suspect that for large values of d_k, the dot products grow large in magnitude, pushing the softmax function into regions where it has extremely small gradients”. A footnote gives the intuition: with independent components of mean 0 and variance 1, the dot product has variance d_k, so dividing by √d_k restores standard deviation 1. Useful intuition, but a conjecture, not a theorem.
② In V3's actual code the factor is not 1/√192. With YaRN extrapolation (original_max_position_embeddings = 4096, factor = 40, 4,096 × 40 = 163,840), the scale is also multiplied by mscale²:
mscale = 0.1 × ln(40) + 1 = 1.3689, 1/√192 × 1.3689² = 0.1352, not 0.0722: a 1.87× difference. Same principle; the constant is tied to long-context extrapolation.
Source: the * mscale * mscale after softmax_scale = q_head_dim^(-0.5) in the reference modeling_deepseek.py, and the definition of yarn_get_mscale(). Also, 163,840 is the config's position limit, 128K the advertised usable length; they differ.
③ Mask: correctness, not optimization Position 5 can't see 6: upper triangle = −∞, zero after softmax.
Unmasked, the model copies the answers: training looks great; inference, lacking later tokens, collapses at once.
④ Softmax: scores into “how much to take” Rows normalized to sum to 1: each a probability distribution, “29% of my info from position 1, 47% from position 5.”
Softmax is nonlinear, per row and per head, so the 128 heads can never merge back (the previous section's point).

⑤ Weighted sum, ⑥ concatenate, exit. Step ④'s weights sum V; each head emits 128 dims. Concatenated: 16,384, which o_proj folds to 7,168 into the residual stream. o_proj is the largest matrix in the layer (117.44M, about 60% of an MLA layer), and the only place heads exchange information; before it, each computes alone.

A different kind of compute Projections (Q/K/V, o_proj) are big regular matmuls, [128K, 7168] × [7168, N]: static shapes, high arithmetic intensity, accelerator heaven.
Scoring is 128 small batched matmuls around a row-wise reduction (softmax), and quadratic in length: 2× sequence, 4× work.
The next section's startling number grows from here.

Five projections: one layer's size

Five weight matrices per layer; one is ~60% alone. The bars tell it; numbers are in the expander below.

Bar ∝ params · per-layer total 187.11M o_proj 117.44M  62.8% — more than the other 4 combined 16,384 × 7,168 q_b_proj 37.75M 20.2% 1,536 × 24,576 kv_b_proj 16.78M 9.0% 512 × 32,768 q_a_proj 11.01M 5.9% 7,168 × 1,536 kv_a_proj 4.13M 2.2% ⭐ alone sets the KV cache, yet smallest of all 7,168 × 576 ⭐ o_proj is big: spans 16,384 → 7,168, wide on both ends ⭐ kv_a_proj is small: it writes just 576, the entire KV cache
One layer's five MLA matrices, linear in params. Largest = 28× smallest, yet the smallest sets the memory bill. × 61 layers = 11.41B: attention's whole share of 671B.
Expand: each matrix's shape and role— source of the five numbers
MatrixShapeParamsRole
q_a_proj7,168 × 1,53611.01MQ down to low rank. Saves params, not cache
q_b_proj1,536 × (128×192)37.75MQ back up to 128 heads
kv_a_proj7,168 × 5764.13M⭐ Compress to 512 + separate 64-dim RoPE
kv_b_proj512 × (128×256)16.78M⭐ Back up to K and V for 128 heads
o_proj(128×128) × 7,168117.44MOutput projection, the layer's largest: folds the 128 heads' joined 16,384 dims back to 7,168
Total per layer187.11M× 61 layers = 11.41B

Check: 16384×7168 = 117,440,512; 1536×24576 = 37,748,736; 512×32768 = 16,777,216; 7168×1536 = 11,010,048; 7168×576 = 4,128,768. Total 187,105,280 = 187.11M, 117.44 / 187.11 = 62.76%.

Bottom line: what MLA saves

MLA's KV cache

8.58 GiB

131,072 × 61 layers × 576 × 2 B

Standard MHA, same config

488.00 GiB

32,768 values per token per layer

The gap

56.9×

This gap is why MLA exists.

one device 94.74 GiB
KV cache, four schemes, same sequence length. Red dashes: one TPU v7 device's HBM. MHA's cache alone needs several devices, before any weights.

⚠️ The real blowup: the score matrix

That's what is stored. Something transient, made during compute, is far scarier.

Every token scored against every token: a sequence length × sequence length matrix, per head.

128 heads × 131,072 × 131,072 × 2 B = 4.00 TiB ← per layer, per pass
Why FlashAttention must exist Not speed but fitting at all. It never materializes the matrix: tile by tile, accumulate, discard. It saves not time but these 4.00 TiB.
Each square = one TPU v7 device (94.74 GiB). This is just one layer, one pass, batch 1.

FLOPs: when the square term takes over

Two parts of attention are quadratic in length: scores (QKᵀ) and the weighted sum (AV). All else (projections, MLP, MoE) is linear.

Negligible when short, all-consuming when long. The crossover is computable.

X: sequence length (log). Y: attention's quadratic share of forward compute. Line: current length; drag the slider.
At 128K, the attention quadratic term takes 81.8% of forward compute. At this scale, sequence length, not parameter count, dominates compute. Push the slider at the top to 1M and it becomes 97%.
What happened next The curve dictates the next move: if attention eats more than 80% of compute (82% at 128K, 97% at 1M), stop every query seeing every token. V3.2's DSA: each query picks its 2,048 most relevant tokens and attends to those (index_topk = 2048, in V3.2's own config). At 128K that's 1.6%; quadratic becomes linear.
V4 is a hybrid of two attentions: Compressed Sparse Attention (compression + sparse selection) and Heavily Compressed Attention (harder compression, no selection, global view). At 1M tokens, V4-Pro needs 27% of V3.2's per-token inference FLOPs and 10% of its KV cache. (DeepSeek-V4 paper abstract, arXiv 2606.19348. Two kinds, not three; an earlier version erred.)

Their designs and trade-offs belong to Topic 3. This lecture only shows why change was unavoidable: the curve above.
① Embed
② MLA
③ Dense
④ MoE
⑤ Bus
⑥ Exit
⑦ Total
So far— weights— HBM— compute
Step 3

Dense MLP: what “not sparse” looks like

first_k_dense_replace = 3: V3's first 3 layers are ordinary MLPs. Dense first: it shows what MoE saves.

Three matrices + one gate

Widen, open a valve, project back. Three matrices, each 7,168 × 18,432: 396.36M parameters per layer, 1.19B for all three.

The shapes are fixed, independent of input and sequence length. All four advantages below follow.

Expand: the formula— the gold × sign, written out
gate = W_gate · x // 7168 → 18432 up = W_up · x // 7168 → 18432 h = SiLU(gate) ⊙ up // per-elem out = W_down · h // 18432 → 7168

⊙ is elementwise, not a matmul: two 18,432-dim vectors, multiplied entry by entry, give 18,432 dims. SiLU(z) = z · σ(z); config field: hidden_act: "silu".

Four hardware-friendly properties

① Every token takes the same path→ static shapes
② One big matmul→ high utilization, MXU's favorite
③ No communication to assign work→ no scheduling overhead
④ Compiler plans it all ahead→ no runtime uncertainty

MoE keeps none of them; we'll revisit each. In Step 4 they become: shapes unknown until routing ends (the dropless section wins this back); one matmul shattered into 257; two extra all-to-alls; a compiler reserving for the worst case.

SwiGLU gating = two paths + a multiply input x 7,168 inner 18,432 2.57× wide output 7,168 gate_proj 7,168 → 18,432 SiLU channel valve up_proj 7,168 → 18,432 flow: raw, no activation ⊙ pointwise down_proj 18,432 → 7,168 Gray bars drawn to scale: widened 2.57×, then projected back 7,168 × 18,432 per matrix · 396.36M/layer · 1.19B for first 3
Division of labor: two matrices widen, one projects back. Upper: SiLU, the valve. Lower: raw, the flow. Multiplied elementwise. A classic MLP has only up and down; the extra matrix buys a valve each token sets itself.

Why three matrices, not two?

The most common interruption here, and a fair one. A classic MLP has only two: widen, fixed nonlinearity, project back. But that nonlinearity is fixed: one rule for every input and channel.

SwiGLU widens twice: one copy, through SiLU, is the valve, the other the raw flow, multiplied elementwise. Which channels pass, and how much, becomes learned and input-dependent.

The key isn't SiLU. It's the multiply. The same paper's Bilinear variant multiplies two paths with no activation: a strict quadratic form in x. It still beats every activation-only swap. Expressiveness comes from multiplying two linear maps, not a cleverer activation.

One experimental table backs this. Eight variants, equal parameters and compute: watch the two groups.

Bar = gain over ReLU baseline (heldout log-perplexity, lower is better) · all eight matched in params & compute Swap activation still 2 matrices: none wins GELU 1.679 a bit worse Swish 1.683 even worse Add a multiply 3 matrices, width × 2/3: all win GLU 1.663 ⭐ Bilinear no activation: linear × linear 1.648 ReGLU 1.645 SwiGLU what V3 uses 1.636 GEGLU 1.633 the best ↑ ReLU baseline 1.677 ⭐ Gap between groups > any gap within one: the divide is the multiply, not the activation The four GLU variants differ only in gate function: GLU sigmoid, ReGLU ReLU, GEGLU GELU, SwiGLU SiLU. Bilinear has none
Source: GLU Variants Improve Transformer, arXiv 2002.05202, Table 1. T5-scale span filling, 524,288 steps; all eight match in params and compute.
Swapping activations (top two): no gain or worse. Adding a multiply (bottom five): always better. SwiGLU 1.636 ≈ GEGLU 1.633; V3 didn't pick SwiGLU as best.
Honestly: even SwiGLU's authors can't say why it works The paper's conclusion, verbatim: “We offer no explanation as to why these architectures seem to work; we attribute their success, as all else, to divine benevolence.”
Plainly: no explanation. So “the multiply adds interaction terms” is plausible intuition, not proven cause. The table shows only this: gating wins, and not because of the activation.
Expand: full table; does it hold at 671B?— the figure shows one of two columns
VariantMatrices 65,536 steps524,288 steps
FFNReLU baseline2 1.9971.677
FFNGELU2 1.9831.679
FFNSwish2 1.9941.683
FFNGLU3 1.9821.663
FFNBilinear ⭐ no activation3 1.9601.648
FFNReGLU3 1.9531.645
FFNSwiGLU in V33 1.9441.636
FFNGEGLU3 1.9421.633

Held-out log-perplexity; lower is better. Columns: 65,536 vs 524,288 training steps. Both give the same two groups: it doesn't hinge on one budget.

Be clear about what this table covers It ran on a T5-scale span-filling task, not a 671B causal LM. Strictly, the gated group wins in that setting.
What backs us up is different: today every mainstream config is in the GLU family. A strong consensus signal, not a direct extrapolation from the table.
Modelhidden_actVariant Inner width ÷ residual stream
Llama 3 8BsiluSwiGLU3.50×
Mistral 7BsiluSwiGLU3.50×
Qwen3 8BsiluSwiGLU3.00×
Gemma 2 9Bgelu_pytorch_tanhGEGLU4.00×
DeepSeek V3 (first 3 layers)siluSwiGLU 2.57×

All from each model's config.json. It also shows something non-obvious: few models follow the “× 2/3” rule (4× → 2.67×) today. Llama 3 and Mistral use 3.5×, Gemma 2 a full 4×, all much wider. V3's 2.57× is the closest to the rule, and the leanest here. So “V3 back-solved 18,432 from parameter parity” is plausible, but it is our reading. Models following the rule are now the minority.

One more matrix, no extra cost?

The other sure question. The inner width shrinks, by a rule fixed in that paper.

Cell = one weight matrix, length = its params · row totals compare directly Plain MLP 2 matrices · 4× = 28,672 up 205.52M down 205.52M 411.04M The 2/3 rule 3 matrices · 19,114.67 (exact) gate 137.01M up 137.01M down 137.01M 411.04M V3 uses 3 matrices · 2.57× = 18,432 gate 132.12M up 132.12M down 132.12M 396.36M parity line V3 −3.57% ↘ ⭐ Paper's fixed rule: 3 matrices × 2/3 inner width = same params & compute as 2 matrices. V3 goes a bit lower: 0.643, not 0.667
Compare row totals. A third matrix costs nothing extra: the width shrinks, so the row is shorter. Gating is free in parameters; V3 even saves 3.57%.
“× 2/3” isn't our fudge; the paper states it “All of these layers have three weight matrices, as opposed to two for the original FFN. To keep the number of parameters and the amount of computation constant, we reduce the number of hidden units dff … by a factor of 2/3 when comparing these layers to the original two-matrix version.”
— GLU Variants Improve Transformer, arXiv 2002.05202, Section 2. The ablation table used this rule, so its eight variants are comparable.
Expand: 18,432 meets three constraints— the paper doesn't say which mattered

A clean number that fits all three. Each checks out arithmetically; the paper claims none.

TestCheckFit
① Exactly 9 experts
= 8 routed + 1 shared
9 × 2,048 = 18,432 Exactly equal
dense = MoE’s per-token active width
② Aligned to matrix units 18,432 ÷ 256 = 72 Divides evenly
2/3 rule: 19,114.67, not whole
③ The paper's 2/3 rule 28,672 × 2/3 = 19,114.67 Close, not exact
V3 0.643×, rule 0.667×

① matters most: it is too exact to dismiss as coincidence. The MLP a token passes through in V3's first three layers and the 9 experts it activates in the other 58 layers are equal in parameters and compute, matrix for matrix. From first layer to last, per-token MLP compute is an almost flat line.

Keep the word “almost”: an MoE layer also has a router, 7,168 × 256 = 1.84M, 0.46% of 396.36M. Precisely: “the experts match exactly, plus 0.46% for the router,” not “not one parameter more.”

Mind the wording. “Equal” is checkable; “so that's why they chose it” is interpretation. The V3 paper only lists intermediate_size and moe_intermediate_size; it never calls the match deliberate.

This structure returns unchanged Every MoE expert uses the identical three-matrix SwiGLU; only the inner width shrinks from 18,432 to 2,048 (moe_intermediate_size). Nothing new to learn next step; watch width and count.
The control group for every MoE problem ahead MoE is no free lunch. Of dense's four properties, it keeps none; every Step 4 problem maps back to one:
① Static shapes → unknown until routing runs, different every batch
② One big matmul, the MXU's favorite → 257 small matmuls, none fills the MXU
③ No communication to assign work → two extra all-to-alls per layer, there and back
④ Compiler plans ahead → reserves for the worst case; too little and tokens drop
These four pay for the compute MoE saves.

Why are the first layers not MoE?

The popular answer: “Early layers do generic work; nothing to specialize.” Plausible, not DeepSeek's reason. The real story: a chain with a broken last link.

One design, three papers, in chronological order DeepSeekMoE · 2024 layer 1 dense ✅ Reason: layer-1 load balancing is slow DeepSeek-V2 · 2024 layer 1 dense ↩ only cites the earlier paper “Following Dai et al. (2024)” DeepSeek-V3 · 2024 layers 1–3 dense ❓ no reason given why 1→3: not in the paper Fact Original reason: training stability (slow load balancing), not “generic vs. specialized” Guess “V3 chose 3 as most generic”: our guess, not in the paper Aside V3 deploys the shallow dense MLPs at 1-way TP, separately, to save communication Echoes property ③ (no communication to route work): dense pays off in real deployments
A verifiable chain, and a verifiable gap. “Early layers are generic” swaps a training fact (slow load-balance convergence) for a semantic story. Both may be true; only the first is in the paper.
Expand: the paper's “only reason,” verbatim— one sentence

In the whole chain, only the DeepSeekMoE paper gives a reason, in one sentence:

// DeepSeekMoE (arXiv:2401.06066), §5.1 Experimental Setup We substitute all FFNs except for the first layer with MoE layers, since we observe that the load balance status converges especially slower for the first layer.

Note we observe: empirical, not derived. The paper never says why it's the first layer, and the next two generations never elaborate.

The paper's 145B model does the same, without restating the reason. DeepSeek-V2 says Following Dai et al. (2024), we substitute all FFNs except for the first layer; DeepSeek-V3 says We substitute all FFNs except for the first three layers. The change from 1 to 3 gets not a word of explanation.

Across labs: no consensus

That chain is DeepSeek's alone. Contemporary MoE config.json files disagree widely: some keep 3 layers, some 1, many none.

ModelLayers Leading denseSource
DeepSeekMoE 16B281 Paper: the only one giving a reason
DeepSeek-V2601 first_k_dense_replace: 1; paper: “following” prior work
DeepSeek-V2-Lite271 first_k_dense_replace: 1
DeepSeek-V3613 first_k_dense_replace: 3, no explanation
GLM-4.5923 first_k_dense_replace: 3, same as V3
Kimi K2 ⭐61 1 first_k_dense_replace: 1: copies V3, but changed this back
Qwen3-235B-A22B940 mlp_only_layers: []: all 94 layers are MoE
Mixtral 8x7B320 No such field
MiniMax-M2620 No dense prefix; shared_intermediate_size: 0 no shared expert

Every row except DeepSeekMoE 16B (paper text) is read directly from that model's Hugging Face config.json (the DeepSeek-V2 row also quotes the paper).

The Kimi K2 row is worth thirty seconds Its architectures field reads DeepseekV3ForCausalLM, with hidden_size 7168, intermediate_size 18432, moe_intermediate_size 2048, q_lora_rank 1536, kv_lora_rank 512, qk_nope_head_dim 128, qk_rope_head_dim 64, num_hidden_layers 61. Not one shape on this page changed.
Yet first_k_dense_replace went from 3 back to 1. A team copying this closely changed only this: not an architectural necessity, just a value each lab finds by experiment.
Expand: why the first layer?— labeled speculation

Up front: this is our inference, not the paper’s.

A router is a linear projection of the hidden state plus top-k, so similar hidden states must pick the same experts. In the first layer, before attention mixes anything in, the hidden state is essentially the embedding plus some position info: a word looks nearly the same in any context.

So in layer 1 the router degenerates into assignment by vocabulary ID. Frequent words (commas, the, “of”) dominate and flood one expert in batches. The data distribution itself skews the load; the aux loss fights it, so slow convergence is no surprise.

⚠️ It has a single source; don't treat it as a law “The first layer is hardest to balance” comes only from DeepSeekMoE's own experiments; we found no public cross-model, cross-data check.
So all we can say is “in that setup, the first layer was hardest.” Another model, dataset or aux-loss coefficient may differ: which layer is hard to balance isn't a general rule yet. It also explains why labs keep 0 to 3 dense layers: their experiments really differed.
Wrap-up: what is this design for? ① The sole public origin: DeepSeekMoE saw first-layer load balance converge especially slowly.
② No one has explained the mechanism; V3's move from 1 to 3 is unexplained too.
③ Far from consensus: at least three mainstream peers keep no dense layer.
④ Free perk: no all-to-all, so shallow dense layers deploy apart at 1-way TP, saving communication.
Honestly: an engineering choice with an empirical start, no mechanism, no consensus. 0–3 dense layers? Trial and error, for now.
① Embed
② MLA
③ Dense
④ MoE
⑤ Bus
⑥ Exit
⑦ Total
So far— weights— HBM— compute
A cliff face carved with hundreds of meditation cells; only nine glow gold, and a single mote of spirit drifts along the lit path
Step 4 · 97.8% of all parameters MoE: the 671B's bulk All 256 doors are there, each taking HBM; a passing token lights up only 9.
Step 4 ⭐

MoE: the 671B's bulk

These 58 layers hold 97.8% of all parameters.

Triage first, then the specialists. Not every doctor sees every patient: triage sends each one to the few specialists who fit, so the total workload drops.

4a · Where the sparsity lies

Dense · layers 1–3 18,432 wide 396.36M · 100% static, no comm MXU full MoE · last 58 layers 256 small experts; token uses only 8 + 1 shared Router 7,168 × 256 score ■ 8 picked: 3×7,168×2,048 = 44.04M each ■ shared □ 248 idle: weights still in HBM layer total 11.32B used by one token 396.36M · sparsity 28.6×
Left: one wide dense MLP; right: 256 narrow experts. A token does the same multiplies either way: the 9 on the right total exactly 396.36M.
The only thing MoE buys you It decouples parameters from compute: 671B parameters (more knowledge), dense-level compute per token (affordable).
⚠️ “Expert” is MoE's most misleading word It suggests specialties: one for math, one for code, one for medicine. Measured, experts don't split that way.

The triage analogy explains why total compute drops; don't extend it to “so some expert knows medicine.” Measurements refute that.
Measured MoE routing: not by subject, but by syntax ✗ Not by subject Expert usage per domain: math, biology, philosophy: near-identical ArXiv PubMed Philosophy ← same shape Source:“we do not observe obvious patterns in the assignment of experts based on the topic” ✓ By local syntax Metric: adjacent tokens, same expert rand 12.5% 14.1% layer 0 ≈ random 26.4% layer 15 2× random 22.7% layer 31 E.g. Python's self and English Question always hit one expert, even across tokens; indentation tokens “always” go to the same experts ⭐ That makes it a systems problem, our core concern Paper: high locality → under EP, a few experts overload easily (over-subscription); yet it also enables caching Foe of load balance, friend of caching: depends where you stand
Source: Mixtral of Experts, arXiv 2401.04088, §5 (Routing analysis), Table 5. Right-hand bars are averages over 8 domains (layer 0: 13.6–⁠14.9%, layer 15: 23.6–⁠28.4%, layer 31: 19.7–⁠26.3%).
Scope: Mixtral is 8 experts, top-2; V3 is 256, top-8. Not directly comparable. It supports only the negative result (no subject specialization). For V3's 256 experts we found no public analysis this rigorous.

⭐ Each token takes its own path

That is what “the life of a token” means: all 58 MoE layers triage afresh, so each token traces its own path.

Schematic: three tokens crossing MoE layers. Each layer triages anew; paths differ. The yellow band is the shared expert: unrouted, every token passes it in every layer, hence the one straight line; the zigzags above are top-8 picks.
Scaled down: 16 rows for 256 routed experts, 3 lit cells for the 8 picked. Really: 58 layers, 8 of 256 plus 1 shared each. Paths are astronomically many; the model's capacity lives in them.

Why sparsify the MLP and not attention

First, the MLP holds most parameters: per the tables above, 187M per MLA layer, 396M per dense MLP, 11.32B per MoE layer. Second, the MLP is a pile of independent problems: it acts on each token alone, so tokens can safely take different routes. Attention exists to let tokens see each other; it can't be split.

Fine-grained: why many small experts

dense intermediate

18,432

first 3 layers' width

each expert

2,048

just 1/9 of dense

possible combinations

C(256,8)

≈ 4×10¹⁴ ways to split work

Same activation budget: smaller, more numerous experts, more combinations. 8 big experts, top-1: 8 options. 256 small ones, top-8: astronomical. That is the whole case for fine-grained MoE.

Shared expert: less redundancy

Every token needs grammar and common knowledge. Without a shared expert, all 256 routed experts would each learn them: 256 copies of the same knowledge. V3 puts them in 1 shared expert that every token passes, freeing the routed experts to specialize.

⭐ Per token, an MoE layer computes exactly as much as a dense one

Dense MLP first 3 layers intermediate width 18,432 MoE layer used by this token 8 routed experts · 2,048 each shared 2,048 Other 248 routed experts here: this token touches none 9 / 257 = 3.50% 8 × 2,048 + 1 × 2,048 = 18,432  Same length by arithmetic, not drawing
The nine segments end to end are exactly the dense bar above. Width sets all three matrix sizes: equal width ⇒ equal parameters ⇒ equal compute. A token's FLOPs on MoE experts equal a dense MLP layer's.
(One small extra: the router matrix, 7,168 × 256 = 1.84M, 0.46% of 396.36M. So: not one extra expert FLOP, plus a half-percent triage fee.)
Check it in two lines
// One dense layer (first 3 layers) 3 × 7,168 × 18,432 = 396,361,728 // One MoE layer: the 8 + 1 experts this token activates 9 × 3 × 7,168 × 2,048 = 396,361,728
Not close: identical. Because 9 × 2,048 = 18,432, and intermediate width alone fixes all three matrix shapes.
The same layer holds 11.32B parameters, 28.6× a dense layer. Yet each token computes the same 396M, plus a half-percent triage fee. That is “decoupling parameters from compute,” literally: the model grows while the bill per token stays put. The price: the extra 28× must stay in HBM. Section 4d does that math.
The equality's origin, and its limits Certain: 9 × 2,048 = 18,432 is an identity; both lines use V3's public config (intermediate_size, moe_intermediate_size, num_experts_per_tok, n_shared_experts).
Not certain: whether the authors chose these values to make them equal; no source says so. Say “they are equal,” not “they designed it so.”

But the equality is a ready-made yardstick: whatever MoE costs beyond dense, it isn't FLOPs. Routing, communication, load imbalance, memory: every later problem traces back here.

4b · Routing: how the 8 are chosen

token on residual stream · 7,168 dims router 7,168 × 256 1.84M params: 0.016% of the layer's 11.32B sigmoid (not softmax) 256 independent s: no competition, sum ≠ 1 raw s untouched to the end ↓ bias acts only here ↓ bias b per expert set by load, not learned selection score = s + b overloaded ↓, never picked ↑ 256 → 8 groups of 32; pick only 4 groups group score = top-2 sum · 4 groups out, 128 left top-8 of 128 candidates → just 8 indices bias stops here use these 8 indices to fetch unbiased s normalize to 1, × 2.5: the actual weights
The two paths never cross. Blue carries the bias but is cut off at “8 indices.” Green bypasses from the sigmoid to the end, never touching the bias. So the bias evens out load without changing one byte of output.
⭐ Keep this: scheduling stays out of the output Load balancing is scheduling; quality is learning. V3 puts them on disjoint paths.

Three consequences: ① push the bias as hard as you like, the output stays true; ② outside backprop, it steals no gradient from the main objective; ③ it acts outside the training loop, so policy changes leave the model alone.

(The paper, verbatim: “Note that the bias term is only used for routing. The gating value, which will be multiplied with the FFN output, is still derived from the original affinity score.” —arXiv 2412.19437, Section 2.1.2. Two scores, two paths.)
Expand: sigmoid scores “don't sum to 1,” yet the 8 final weights sum to 1. Contradiction?— you will be asked

No: they happen at two different moments. “Doesn't sum to 1” is scoring: each of the 256 is scored alone, unaffected by the rest. “Normalized to 1” comes after selection: only the chosen 8 share the weight.

// ① Score: 256 independent sigmoids, any total is fine s_i = sigmoid(u · e_i) i = 1 … 256 // ② Select: bias used only to rank; pick 8 (bias stops here) // ③ Weight: normalize over these 8, × routed_scaling_factor g_i = s_i / Σ(s over the 8) × 2.5

So what does sigmoid change? Step ①. Softmax's denominator spans all 256 experts, so raising one score pushes down the other 255: forced zero-sum. Sigmoid has no such denominator: each expert says “I'm good at this token” on its own.
Competition moves to the “rank, take top 8” step, not into scoring. V2 used softmax; V3 uses sigmoid, per the paper.

Where 2.5 comes from: the config's routed_scaling_factor. Normalization pins the 8 weights' sum to 1 and the shared branch's weight is 1, both small vs. the residual stream; a constant above 1 scales them back up. Our reading: the paper's formulas omit it; only the open-source config and code have it.

DesignValueWhy
Scoringsigmoid Softmax: 256 scores compete, sum to 1. Sigmoid scores each alone: no zero-sum
Groupsn_group=8
topk_group=4
⭐ Forced by hardware. Paper: node-limited routing, “we ensure that each token will be sent to at most M nodes.” M = 4; group = node: caps machines per token
Top-ktop-8 8 routed + 1 shared = 9 experts per token
Balancenoaux_tc Main: a dynamic per-expert bias, lowered if busy, raised if idle.
⭐ The bias picks who, not their weights: load evens, output unchanged. Not learned by gradient: set from observed load.
But “no aux loss at all” overstates it: a tiny sequence-wise loss backs it up (below).
⭐ This course's theme at its sharpest n_group=8, topk_group=4 have nothing to do with model quality. They exist only to keep a token's experts on few machines, so all-to-all crosses as few as possible. A communication constraint written into the architecture.
Expand: the rule in the paper, and where the 2 in “top 2 per group” comes from— a number you can derive
The paper, verbatim “Like the device-limited routing used by DeepSeek-V2, DeepSeek-V3 also uses a restricted routing mechanism to limit communication costs during training. In short, we ensure that each token will be sent to at most M nodes, which are selected according to the sum of the highest Kr/M affinity scores of the experts distributed on each node.”
—DeepSeek-V3 Technical Report, arXiv 2412.19437, Section 2.1.2. The stated reason leaves no ambiguity: “to limit communication costs.”

The “top 2” isn't arbitrary; it's computed. The paper writes Kr/M: Kr = activated routed experts, M = most nodes a token may reach:

// K_r = num_experts_per_tok = 8 M = topk_group = 4 K_r / M = 8 / 4 = 2 ← hence group score = “sum of its top 2”

The rule makes sense: if a token can reach at most 4 nodes and must pick 8 experts, each node has to contribute 2 on average. Scoring a group by the sum of its top 2 asks, “if I pick this node, how much do its best 2 give me?” That is exactly the share you will take from it.

An aside: this communication design has headroom The paper, verbatim: “although DeepSeek-V3 selects only 8 routed experts in practice, it can scale up this number to a maximum of 13 experts (4 nodes × 3.2 experts/node) while preserving the same communication cost.” Same passage: the IB/NVLink overlap needs only 20 SMs.
Read it as: the machines set “at most 4 nodes”; then see how many experts fit in that box. They used 8; it holds 13. The plainest case of “hardware draws the box; the model picks parameters inside.”
Why this design matters The old way adds an imbalance penalty to the loss. That makes the model optimize two unrelated goals at once, “answer correctly” and “spread evenly.” They pull against each other; both lose. The paper agrees: a large auxiliary loss impairs model performance.

V3 moves the main burden of balancing out of the loss, into scheduling outside the training loop, independent of what the model learns. Good engineering instinct: a constraint that isn't the objective stays out of it.
⚠️ “Auxiliary-loss-free” is tidier than the truth We said “no auxiliary loss”: overstated. The paper: “Although DeepSeek-V3 mainly relies on the auxiliary-loss-free strategy for load balance, to prevent extreme imbalance within any single sequence, we also employ a complementary sequence-wise balance loss.” Its α “will be assigned an extremely small value.”
Accurately: the bias does the main work; a tiny auxiliary loss is the backstop. Not “none at all.”
V3 actually has two balancers, for two scales Primary · dynamic bias b Scale: batch, across steps after each step: overloaded −γ, underloaded +γ ✓ no loss term, no grad ✓ affects picks, not output Fixes: chronic hot experts Backstop · sequence-wise loss (tiny α) Scale: within one sequence per sequence: don't pile into few experts ✗ It is in the loss but paper: α “extremely small” Fixes: extreme skew within one sequence ⭐ Bias can't fix the right side: it's a global value updated once per step; useless when one sequence piles up. That's inside a step Left = policy, right = fuse. “Aux-loss-free” = left is primary, not right absent
Not redundant: the two mechanisms work at different scales. The bias evens long-term load across steps; the tiny loss stops one sequence collapsing within a step. Hence α can be tiny: a backstop, not the main balancer.
(Scope: “two mechanisms, one per scale” is in the paper; “hence α can be so small” is a fair reading, not the paper's.)

⭐ Imbalance: one device busy, all wait

Experts live on different devices. If a few are hot, their devices take more tokens while the rest idle. The layer ends when the slowest device does.

One MoE layer, four devices, % = token share. Bar length = compute time Skewed layer ends here = busiest device device 0 · 40% busy device 1 · 20% idle device 2 · 25% idle device 3 · 15% idle Hatched = paid-for idle compute: 37.5% of all four Balanced time saved same work, layer 1.6× faster 25% each Schematic, made-up ratios — 40/20/25/15 just show who waits. Real imbalance varies with data and routing
Not “a few percent lost”: the cluster runs at its slowest device's speed. Again in all 58 layers.
The 4b bias turns the top picture into the bottom one. “Keep devices busy” sounds like ops, but it multiplies into training time and cost.

4c · Move tokens, or weights?

The systems highlight of the topic. Experts sit on different devices; a token needs expert #137. Two options: a genuine either/or.

Scheme A · Dispatch / Combine: tokens go to experts device 0 exp 0–63 device 1 exp 64–127 device 2 device 3 all-to-all ×2 (there + back) traffic ∝ tokens × hidden ✓ cheap with few tokens (decode) ✗ costly with many tokens (training) Scheme B · EP = 1 + FSDP: fetch weights device 0 device 1 device 2 device 3 all-gather weights → compute locally tokens stay put traffic ∝ weight size (token-independent) ✓ amortized over many tokens ✗ wasted on few tokens
Neither always wins. Many tokens, relatively small weights → move weights; few tokens (e.g. decode, a few per step) → move tokens. Topology too: cross-node bandwidth is far below intra-node, so group routing keeps all-to-all in-node.
B shows forward only; backward all-gathers weights again and reduce-scatters gradients to shards (standard FSDP).
⚠️ An easy misnomer: Scheme B is not AG‑RS All-gathering expert weights to every device is real, and right on TPU, but it is called “EP = 1 + FSDP,” or “no expert parallelism.”
In MoE, AG-RS is something else: an all-to-all alternative that also moves tokens: replicate every token to every device, route locally, reduce-scatter results back. Two independent implementations agree: in Megatron-Core's flag --moe-token-dispatcher-type, all three values, allgather / alltoall / flex, are token dispatchers; MaxText calls this path ring of experts.

The “move tokens or weights” split stands; only the second label was wrong. All three paths' mechanics and traffic, and why TPU picks the most-bytes one: appendix (Chinese edition) §14 How tokens meet their experts

Traffic, computed

// Dispatch + Combine, 1 layer, 4 copies 2 × 131,072 tok × 4 × 7,168 × 2 B = 14.00 GiB // × 58 layers = 812.00 GiB ← moved in a single forward pass
⭐ 4 copies, not 8: the unit of movement is the destination, not the expert The tempting intuition: 8 experts selected, so send 8 copies. Wrong.
The copies sent depend on how many places a token lands. Two selected experts on the same node share one copy. Once there, it is fanned out over the intra-node interconnect, not across nodes.
And the number of destinations is fixed: group routing (topk_group = 4) guarantees a token lands on at most 4 nodes. So the multiplier is 4, not 8.
One-line test: it doesn't depend on top-k. Top-k 8 → 16, still at most 4 nodes: cross-node traffic grows not one byte.
The paper settles it— 8 to 13 experts, same communication cost

“…we limit each token to be dispatched to at most 4 nodes, thereby reducing IB traffic. For each token, when its routing decision is made, it will first be transmitted via IB to the GPUs with the same in-node index on its target nodes. Once it reaches the target nodes, we will endeavor to ensure that it is instantaneously forwarded via NVLink to specific GPUs that host their target experts… This implies that, although DeepSeek-V3 selects only 8 routed experts in practice, it can scale up this number to a maximum of 13 experts (4 nodes × 3.2 experts/node) while preserving the same communication cost.”
—DeepSeek-V3 Technical Report, arXiv 2412.19437, Section 3.2.2.

The last sentence decides it: 8 → 13 for free. If traffic scaled with experts, 5 more couldn't be free. It must be proportional to nodes, locked by M = 4.

The same section gives the layout: 256 routed experts evenly over 64 GPUs on 8 nodes (4 per GPU, 32 per node), M = 4. So the “8 groups” are the 8 nodes; a token reaches at most 4.

Refinement: one of the 4 destinations may be the token's own node; that copy skips the network. 4 of 8 nodes include home with chance 1/2, so expected cross-node copies: 3.5, not 4. Above we conservatively use 4, an upper bound.

Against the layer's compute it gives the comm/compute ratio: can compute hide it? Topic 7's performance lecture rests on it.

⭐ Why it's a real either/or: two slopes

110 1001K 10K100K 1M10M tokens per step / forward pass (log) 1 MiB1 GiB1 TiB traffic per layer (log) Scheme B · move weights 21.08 GiB: flat, no token dependence Scheme A · move tokens 112 KiB per token: linear crossover ≈ 197,376 tokens 21.08 GiB ÷ 112 KiB decode step (tens of tokens) → far left: move tokens one 128K prefill = 131,072 tokens 14.00 GiB: left of crossover, A wins training step (millions) → far right: move weights wider EP → A cheaper, crossover → Rough: bytes per layer only; ignores all-to-all vs. all-gather efficiency
Slopes differ (1 and 0), so the lines must cross. “It depends on the workload” is geometry, not a dodge. Decode and training sit on opposite sides of the crossover, hence opposite answers.
Where the crossover comes from: two steps
// A: bytes moved per token per layer (there + back) // 4 = max nodes a token lands on (topk_group), not 8 experts; see the red box above 2 × 4 copies × 7,168 × 2 B = 112 KiB // B: a layer's experts (256 routed + 1 shared, 3-matrix SwiGLU) 257 × 3 × 7,168 × 2,048 × 2 B = 21.08 GiB // token count where they're equal 21.08 GiB ÷ 112 KiB = 197,376
Mind the limits: a rough comparison reducing both to “bytes moved per layer.” Real all-to-all and all-gather differ in efficiency, and it depends how many devices hold the experts: wider spread, thinner A traffic, crossover further right.
So don't treat 197,376 as an engineering threshold. It shows the two schemes share an order of magnitude, which is what makes it an either/or.

⭐ Dropless: how static shapes were lost and won back

📄 Appendix (Chinese edition) Dropless and grouped matmul On one device, both approaches with realistic numbers (16,384 rows per expert per step): why shapes must be regular, what MegaBlocks changed, whether to call it block-sparse or grouped GEMM, how padding to 128 relates to PagedAttention pages, and how far the evidence for V3's “no token dropping” goes.
Then three levels down: how the MXU stays fed (the HBM → VMEM → VREG pipeline), how tokens meet their experts (all-to-all / AG-RS / no EP, and why TPU picks the most traffic), and how much memory “rows × 8” takes, which few people compute.

Back to the dense MLP's first virtue: static shapes. Knowing all matrix sizes and buffers up front, the compiler schedules the whole graph.

MoE breaks that. How many tokens hit expert #37 is known only after routing, and changes next batch. But hardware wants it fixed up front.

The old fix is blunt: a fixed-size box per expert, and shapes are static again. The price: overflow is discarded, empty slots zero-padded.

① Old way: a fixed-size box per expert static, but loses both ways capacity ✕ ✕ 2 dropped output zeroed; only the residual passes 3 zero slots still cost compute + HBM expert A · 6 in expert B · 4 in expert C · 1 in ② Dropless: no box, compute what comes block-sparse matmul: ragged shapes, one pass no ✕, no hatching no drops, no waste 641 Capacity factor = box-size knob: too small drops tokens, too big wastes compute; hand-tuned. What dropless buys: no knob at all.
Left: one MoE layer loses at both ends: hot experts discard real data, cold ones burn compute on zeros. What drops depends on how this batch happens to split. So the 4b balancing bias also keeps tokens from vanishing, not just devices busy.
Three MegaBlocks numbers (arXiv 2211.15841) ① Dropping tokens really hurts quality. In one controlled test, capacity factor 1 (drops) cut validation loss by only 0.15; no drops cut it by 0.26, 1.73× as much.
② Avoiding drops with a bigger box is expensive. The paper: “MoE-layer math operations increased by over 2× in order to avoid dropping tokens.”
③ And you can't know how big. Per Tutel: some models need capacity factor 11 to avoid drops; in others it spikes without warning mid-training.
So it isn't “just tune a parameter.” It is a hyperparameter you can't set, costly if wrong either way. Dropless deletes it.
Expand: the setup behind these numbers— know their scope first

Not 671B, and not top-8. A decoder-only model at Transformer-Small scale, every FFN replaced by 64 experts, top-1 routing, trained on 10 billion tokens of The Pile on a single A100.

AspectMegaBlocks runDeepSeek-V3
Experts64256 + 1 shared
Experts per tokentop-1top-8
Model sizeTransformer-Small scale671B
Balancingstandard auxiliary lossdynamic bias + tiny auxiliary loss

The biggest difference is top-1. With top-1 a token bets on one expert; if it's full, the token is dropped outright. Top-8 spreads the bet, so the same imbalance drops less.
So these numbers show the capacity-factor knob itself is a problem, not “what V3 would lose by dropping tokens.” Teach the former.

Two positive results, for the record (paper's experiments section, same setup; abstract: “up to 40% over … Tutel and 2.4x over … Megatron-LM” dense models): end-to-end training 1.38× / 2.0× / 4.35× faster than padding-based methods (three sizes); 1.8×–2.4× faster than equal-quality dense Transformers.

What DeepSeek-V3 says: training and inference The report's “No Token-Dropping” passage: “Due to the effective load balancing strategy, DeepSeek-V3 keeps a good load balance during its full training. Therefore, DeepSeek-V3 does not drop any tokens during training. In addition, we also implement specific deployment strategies to ensure inference load balance, so DeepSeek-V3 also does not drop tokens during inference.”
Stated reason: good load balance. Back to 4b: the bias keeps devices busy and no token lost.
“No drops at inference” rests on redundant experts Training balances with the bias; deployed, with no training loop, experts get moved. Per the report, hot experts get duplicate deployments (prefill: 32 redundant experts; each GPU hosts 1 beyond its 8), found from online statistics, re-adjusted periodically (e.g., every 10 minutes).
Note: experts are rearranged within a node, explicitly avoiding extra cross-node all-to-all. Again: topology first, then optimize.
One thing not to say Say “V3 drops no tokens in training.” Don't say “V3 uses dropless kernels.” The report states the result (no drops) and its stated cause (good balance). Whether its training kernels still have a “capacity,” we found nothing public.
(Very different: “balanced enough never to hit capacity” and “no capacity limit at all” are different implementations, easily conflated.)

In passing: DeepEP (a communication library tuned for this all-to-all) and EPLB (expert load rebalancing).

4d · Accounting for the 671B

LevelFormulaParams%
One expert3 × 7,168 × 2,04844.04M—
Layer (256 + 1 shared + router)257 × 44.04M + 1.84M11.32B—
58 MoE layers58 × 11.32B656.57B97.8%
61 MLA layers61 × 187.11M11.41B1.7%
3 dense MLP layers3 × 396.36M1.19B0.2%
Embedding + lm_head2 × 926.68M1.85B0.3%
Total671.03Bofficial 671B ✓

⭐ Starkest row: all of attention = one MoE layer

All 61 MLA layers 11.41B ↑ 61 slices, one per layer Any single MoE layer 11.32B ↑ 1 block = 1 layer same length within 0.8% — 61 : 1
Attention got a whole step, yet holds 1.7% of parameters. Step 2's curve: it takes 81.8% of compute at 128K. Parameters and compute are separate ledgers, ranked differently.
Expand: surprise, MLA's largest matrix isn't a compression one— the flip side of Step 2's math
How the 187.11M breaks down
q down-proj 7168 × 1536 = 11.01M q up-proj 1536 × (128 × 192) = 37.75M kv down-proj 7168 × (512 + 64) = 4.13M kv up-proj 512 × (128 × 256) = 16.78M // out: 128 heads → residual o_proj (128 × 128) × 7168 = 117.44M ← 62.8% // total = 187.11M (= table ✓)
Despite all the KV compression, the bulk sits on the output side. No contradiction: MLA shrinks per-token storage (memory account), not the layer's parameter count (parameter account). Two accounts, two answers.
All params · 671.03B · all in HBM MoE 656.57B (97.8%) · MLA 11.41B · dense 1.19B · in/out 1.85B Used per token · 36.62B · compute pays only for this 37B 634B params idle this step, yet not one byte can leave HBM ⚠️ MoE saves compute (18.3×), not HBM (0×)
Expand: what “37B” counts— why the official figure and the itemized sum disagree
Two computable numbers, official one between Summing the table, a token computes with 36.62B parameters (9 experts × 58 layers + all 61 MLA + 3 dense layers + the big output matrix). Counting the embedding table as “active”: 37.55B.
The official 37B sits between. The report doesn't say how it counted; we couldn't find out. We use 36.62B, and “about 37B” next to the official figure. Under 1B apart, no conclusion changes: 18× is still 18×.

The section in one line

HBM holds the mountain; compute pays only for the stream.

671B of parameters sit on HBM like an immovable mountain; each token flows only through the near-invisible stream at its foot: 37B.

MoE separates mountain and stream. Before it, a bigger mountain meant more water to pay for.

A vast mountain of spirit ore, with only a thin thread of cyan spirit energy trickling from its foot
Spell out this counterintuitive point HBM holds 671B; compute pays for 37B. Many hear “37B active” and picture a 37B dense model. No: it needs a 671B dense model's memory; only compute is saved. MoE buys “more knowledge for the same compute,” not “a bigger model in the same memory.”

4e · Sparse per token, dense per batch

“Compute pays only for 37B” needs one more twist: that is per token, and machines never process one token at a time.

Both true, about different things One token passes only 9 experts, not 257: sparse.
A batch, if large enough, lights up all 256: dense.
No single token activates every expert; the batch does.
So “MoE is fully activated after all” is exactly right on “must the weight bytes be read,” exactly wrong on “how many multiplies.” Every byte of the 671B weights crosses HBM; the FLOPs really are only 9/257.

Turning “dense” into a number

Density is a division, not an adjective. Rows per expert set the shape of its matmul:

// rows one expert receives this step rows per expert = total tokens in this step × 8 ÷ 256 = total tokens ÷ 32 // and what makes up “total tokens” training / prefill = batch × sequence length ← both count decode = batch × 1 ← the sequence-length leg is gone
32: this configuration's “dilution factor” 256 experts, 8 per token: each expert sees just 1/32 of the tokens fed in. More experts or smaller top-k dilute further: 384 experts, top-8, gives 48.

The hardware threshold: 128 rows

Matmuls run in tiles, not rows. In TPU's grouped matmul kernel, the bf16 tile height is 128 rows, hard-coded in the public source:

// MaxText megablox Pallas kernel (v2), calculate_tiling() bf16_bf16_tile_m = 128 // comment, verbatim: "128 is the largest tile_m value that is safe to use // for most scenarios" (raised for lower bit widths)

To hardware, “16,384 rows per expert” and “8 rows per expert” are different worlds: 128 full tiles versus not even one.

ScenarioTokens Rows/expert TilesStatus
Training (64 dev × 8,192)524,288 16,384128 tiles ✅ well fed
Prefill (one 8K chunk)8,192 2562 tiles ⚠️ marginal
Decode (256 requests)256 8under 1 tile ❌ collapsed
Decode row: root of all MoE inference pain Giving decode training's 16,384 rows takes 16,384 × 32 = 524,288 concurrent requests. No engineering gets there.
Sequence length doesn't help decode: each request emits one token per step, one row however long its context. That leaves concurrency, which falls far short.
Same model: compute-bound in training, memory-bound in decode. Different physics.

Back to 4c: EP changes segments, not traffic

This math settles something 4c left open. Per-device work in the two schemes has identical row counts:

// A · EP: 4 experts/device, tokens all-to-all'd from 64 devices rows per device = 16,384 × 4 = 65,536 in 4 segments of 16,384 rows // B · no EP (EP=1 + FSDP): all 256 experts per device; own 8,192 tokens only rows per device = 8,192 × 8 = 65,536 in 256 segments of 256 rows // exactly the same 65,536 rows; the only difference is how many segments
Same M axis · 65,536 rows = 512 tiles of 128 rows A · with EP: 4 segments 3 seams → ≤ 3 extra tiles → +0.59% B · without EP: 256 segments 255 seams → ≤ 255 extra tiles → ≤ +49.8% ⭐ Same 65,536 rows, same FLOPs: the seams cost, not the work seams drawn schematically; really 255
Why seams cost: TPU revisits tiles instead of padding Segment boundaries rarely hit multiples of 128. GPUs zero-pad to the tile edge; TPU's megablox runs the straddling tile twice, with the left, then the right expert's weights, into one output.
The cost changes form: not “≤ 127 zero rows per segment” but “≤ (segments − 1) extra tiles in total,” per the source comments. Mechanism: appendix (Chinese edition) §11 · No padding on TPU.
// bound: tiles_m ≤ tiles run ≤ tiles_m + segs − 1 tiles_m = 65,536 ÷ 128 = 512 A · EP ≤ 512 + 3 = 515 → +0.59% B · FSDP ≤ 512 + 255 = 767 → +49.8% ← upper bound
49.8% is an upper bound, not a given, but dropless can't escape it If all 256 segments were exactly 256 rows, every boundary would land on a multiple of 128: overhead 0. But equal segments are exactly the assumption dropless gives up. Segments are ragged, so nearly every boundary misaligns. The real value sits near the bound, not 0.
The takeaway EP's first value is long segments, not saved communication. It regathers each expert's tokens from 64 devices, growing each segment from 256 rows back to 16,384: 64× the segment length.
(64× is segment length, not efficiency: 0.59% versus a 49.8% upper bound is about 1.5×, far less, since in-tile waste was bounded anyway.)
FSDP's truly fatal cost lies elsewhere: each device assembles all 256 experts' weights, 21.00 GiB per layer (256 × 44.04M × 2 bytes), versus 336 MiB under EP.
① Embed
② MLA
③ Dense
④ MoE
⑤ Bus
⑥ Exit
⑦ Total
So far— weights— HBM— compute
Step 5

The residual bus

So far, all inside one layer. Now: how 61 layers connect.

✗ Wrong picture: a pipe Data flows from layer 1 to 2 to 3 Embedding Layer 1 Layer 2 ⋮ Layer 61 Sees only the layer before Break layer 30, lose all 31 after ✓ What it really is: a bus One 7,168-dim vector runs through; each layer reads, adds Residual stream · d = 7,168 Layer 1 norm → MLA + Layer 2 norm → MoE + Layer 61 norm → MoE + ⋮ read add back Each layer reads “all so far” Output 0 → one term less; rest run on
The difference isn't the drawing; it's the “+”. A pipe replaces: each output supersedes the last. A bus accumulates: each layer adds a term to one shared vector. So “do nothing” is the free default, which lets you stack 61 layers.
Snag: the middle line is the trunk, so why “residual” stream? Instinct says the added-back loop deserves the name, not the trunk. By the original definition, instinct is right. ResNet (He et al., 2015) has each layer learn residual functions with reference to the layer inputs. The residual is the branch, H(x) minus x; the trunk is the paper's shortcut connection (identity path).
“Residual stream” comes from another tradition, interpretability, which defines it as “simply the sum of the output of all the previous layers and the original embedding.”
So it isn't “residual = trunk”; it's “the stream of residuals,” possessive dropped. The trunk is the sum of the residuals, not a residual: x = embedding + layer 1's residual + layer 2's + … + layer 61's. Precise, just compressed.
Don't read “residual” as “leftover” Everyday “residual” means what remains; here it means “difference from the identity mapping”: an increment, not a remainder. Read “increment stream” and the section clicks: each layer adds an increment to a shared ledger, which is the sum of all increments.
Expand: both sources verbatim, and the trunk's other name—“residual stream” was chosen on purpose; its rival was “embedding”

① Residual = the branch. He et al. 2015, Deep Residual Learning for Image Recognition (arXiv 1512.03385), abstract:

"We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions." // the main text calls the trunk: "The formulation of F(x)+x can be realized by feedforward neural networks with ‘shortcut connections’."

② Residual stream = sum of residuals. Elhage et al. 2021, A Mathematical Framework for Transformer Circuits:

"each layer adds its results into what we call the ‘residual stream.’" "The residual stream is simply the sum of the output of all the previous layers and the original embedding."
Bonus: “embedding,” the trunk's other name, avoided on purpose The same paper also says: “In transformers, the residual stream vectors are often called the ‘embedding.’ We prefer the residual stream terminology, both because it emphasizes the residual nature ... and also because we believe the residual stream often dedicates subspaces to tokens other than the present token, breaking the intuitions the embedding terminology suggests.”
So the trunk does deserve its own name, and it already had one: embedding. They dropped it on purpose: to stress that the stream holds residuals, and because it often reserves subspaces for other tokens, while “embedding” implies it describes only the current token.

Lineage: the same paper traces models built around a residual stream back to Schmidhuber's line (highway networks, the LSTM); ResNet brought their modern success.

The residual stream is a bus, not a pipeline

The usual picture is an assembly line: layer 1 → 2 → 3. It's closer to a bus. One 7,168-wide vector runs through the whole network; each layer reads it, computes, and adds the result back.

x = x + MLA(norm(x)) // +=, not = x = x + MoE(norm(x)) // ↑ these two lines × 61

Why addition. For gradients, it gives backprop a direct highway that doesn't fade across 61 layers. For meaning, it makes “do nothing” the default: a layer that hasn't learned yet outputs ≈0, and existing information survives.

Where RMSNorm sits: one placement decides whether you need warm-up

V3 uses pre-norm: normalize, run the sublayer, add its output straight to the trunk. Post-norm adds first, then normalizes the sum. That puts the norm on the bus, cutting the figure's “direct highway.”

Not taste: proven, and far more precisely than “it trains unstably.”

Expand: the verifiable version of “post-norm is unstable”—the finding is “need warm-up?”, not “converges?”

The source is Xiong et al. 2020, On Layer Normalization in the Transformer Architecture (arXiv 2002.04745). They analyzed both placements' gradients at initialization, with two parallel conclusions:

post-normAt initialization, gradients near the output layer are large. Per the paper, this explains “why the learning rate warm-up stage is essential in training the Post-LN Transformer”: warm-up is required, not a tuning trick
pre-normGradients behave well. The paper claims it's “the first to show that the learning-rate warm-up stage can be removed for the Pre-LN Transformer”: no warm-up needed

So the claim isn't “post-norm can't be trained”; it's “post-norm adds one more thing you must get right.” A setting that blows up if wrong and gets harder to tune with depth is, at tens to hundreds of layers, “unstable training” in practice. When you repeat the claim, say which one you mean.

An oft-missed bonus: with the norm off the trunk, the residual stream is pure addition end to end; the gradient reaches layer 1 unchanged. The “bus vs. pipeline” figure above draws this path. Post-norm puts a norm in every bus segment and breaks it.

Residual stream · d = 7,168 norm → MLA + norm → MoE + Layer 1 Layer 2 ⋮ × 61 layers ⋮ Layer 61 1.75 GiB/layer × 61, all kept = 106.75 GiB
A number that half-answers its question Saving the residual stream at all 61 layer boundaries takes 106.75 GiB: more than a single device, before any intermediate inside a layer. So why keep them?
Half the answer is in the V3 paper: it doesn't. Verbatim: “We recompute all RMSNorm operations and MLA up-projections during back-propagation, thereby eliminating the need to persistently store their output activations.” If it doesn't fit, don't store it; buy it back with compute in backward. That's trading time for space, not free. The other half (why backward needs these intermediates, how to price the trade) is in Topic 4. The reflex: when activations don't fit, ask not “how do I add devices?” but “must this be stored?”

A short detour: mHC, or how V4 widens the bus

So far: “one 7,168-dim bus.” The bus itself is a design choice you can change.  In V4, DeepSeek replaced it with mHC (Manifold-Constrained Hyper-Connections). The topic's one excursion beyond V3; it sits here because you need the bus to see what changes.

V3 · one stream xℓ+1 = xℓ + Fℓ(norm(xℓ)) xℓ 7,168 norm → Fℓ (MLA / MoE) + xℓ+1 7,168 V4 · mHC, nhc = 4 Xℓ+1 = BℓXℓ + CℓFℓ(AℓXℓ) Xℓ · 4 × 7,168 Bℓ · 4×4 doubly stoch. Aℓ · 1×4 norm → Fℓ  still 7,168-dim Cℓ · 4×1 + Xℓ+1 ⭐ A wider bus, same layers: AℓXℓ is still 7,168-dim, so MLA and MoE don't change a byte Paper: wider residual “does not influence the design of the inner layers.” The new part isn't Bℓ but making it doubly stochastic
Widening the residual stream wasn't DeepSeek's idea. Hyper-Connections (Zhu et al., 2025) first widened it from d to n×d. It helped, but broke the identity mapping, destabilizing deep stacks. mHC constrains Bℓ to a doubly stochastic matrix (Birkhoff polytope): rows and columns sum to 1, so spectral norm ≤ 1; the signal neither grows nor vanishes. mHC isn't “wider”; it's “wider made trainable.”

① What it actually changes

Residual state goes from Rd to R4×d. But each layer's input and output stay d-dimensional (“still 7,168-dim” above). Width is decoupled from hidden size: a new knob.

② What “doubly stochastic” buys

Rows and columns sum to 1: mixing is a convex combination, mean conserved, norm bounded. Products stay doubly stochastic, so 61 layers don't blow up. Sinkhorn-Knopp projects onto the set; V4 runs 20 passes.

③ ⚠️ The cost isn't where you'd expect

“4× wider → 4× activation memory” skips the engineering section. Per V4's paper, mHC raises activation memory and pipeline traffic; fused kernels, selective recompute and a reworked DualPipe overlap cut wall-clock overhead to 6.7%.

Expand: every number's source, and why V4-Pro maps directly onto this page—61 layers, 7,168 dims, identical to V3

First, a coincidence that makes the numbers directly comparable. V4 comes in two sizes. V4-Pro has 61 layers and hidden 7,168, exactly as V3; V4-Flash has 43 layers and hidden 4,096. Both use nhc = 4 and 20 Sinkhorn-Knopp iterations. So the 1.75 GiB bus above is exactly V4-Pro's.

// DeepSeek-V4 report (arXiv 2606.19348) §4.2.1, config V4-Pro layers = 61 d = 7168 n_hc = 4 t_max = 20 1.6T total / 49B active V4-Flash layers = 43 d = 4096 n_hc = 4 t_max = 20 284B total / 13B active // source: mHC paper (arXiv 2512.24880), experimental setup HC and mHC: expansion rate n = 4; main model 27B

How much memory does “4× wider” add? No number here: none can be given. Naively, 1.75 GiB × 4 = 7.00 GiB per layer, but that assumes all four streams' activations are kept at every layer. The V4 paper: “we recompute most hidden states between layers and all normalized layer inputs, while avoiding recomputation of compute-intensive operations.” Most between-layer hidden states are recomputed in backward, not stored. Store vs. recompute is an engineering setting, not an architectural constant; the real increase depends on it; the paper gives no absolute memory figure.

The only cost the paper quantifies in absolute terms is time: “Collectively, these optimizations constrain the wall-time overhead of mHC to only 6.7% of the overlapped 1F1B pipeline stage.” The 6.7% is of the overlapped 1F1B pipeline stage, not total training time. Quote the number with its denominator.

⚠️ Corrected from an earlier version This page once said “4× wider → 1.75 GiB becomes 7.00 GiB per layer.” Not wrong alone, but it passed off an upper bound engineering largely cancels as the cost paid. The real increase depends on recomputation; the paper gives only the 6.7% wall-clock overhead.
① Embed
② MLA
③ Dense
④ MoE
⑤ Bus
⑥ Exit
⑦ Total
So far— weights— HBM— compute
A golden spark of thought bursts at a cave mouth into a sky full of scattering runes
Step 6 · End of the journey Output: from vector back to text After 61 layers, one last move fans a single vector out into 129,280 possibilities: the most memory-hungry instant of the trip.
Step 6

Output: back to text

After 61 layers: one 7,168-dim vector. Turn it back into “what comes next?”

The inverse of the embedding

A final RMSNorm, then a 7,168 → 129,280 matrix scores every token in the vocabulary. It mirrors Step 1's embedding matrix; if Step 1 compiles to a matmul (see its expandable note), even the FLOPs match.

tie_word_embeddings = false, yet one thing is shared Many models tie input and output to one matrix (one less copy; representing and predicting a token feel alike). V3 does not. Separate read and write weights: 926M extra parameters.
The trap: V3 does share its output head, but with MTP, not the embedding table. The paper puts the first layers (with embedding) and last (with output head) on one pipeline rank, so MTP and the main model physically share those parameters and gradients. So at MTP below, don't add another 926M.

MTP: built for training, a windfall for inference

num_nextn_predict_layers = 1: depth 1, meaning it predicts one extra token (the one after next). Not two: this number is the depth.

Why build
it (train)
Extra supervision. Trained only on the next token, a model can coast; predicting the one after forces it to plan ahead. The V3 paper names this the main purpose of MTP: “we utilize MTP to improve training”
Not needed
to serve
“during inference, we can directly discard the MTP modules and the main model can function independently and normally”: it's training-time scaffolding. Ablations drop it, so both arms cost the same to serve
Pity to drop
(spec. dec.)
Keep it as a cheap draft model for the token after next; the main model checks both at once. Hit: free token; miss: no loss.
V3 measured: 85%–90% 2nd-token acceptance, 1.8× end-to-end TPS

Speculative decoding pays off because of Topic 6: decode is bandwidth-bound, not compute-bound. With the weights already read, one more token is nearly free to verify. Acceptance hits 85%–90% because the draft isn't a separate model: it reads the main model's hidden state.

Last residual vector 7,168 lm_head · 926.68M logits tensor: the exit step 31.56 GiB 131,072 tokens × 129,280 vocab × 2 B For scale: all MLA weights (61 layers) 21.26 GiB — smaller than this one temp tensor (1.48×)
Another easy-to-miss blow-up The logits tensor is 31.56 GiB: scarier than any middle layer, and purely temporary. Real code computes logits in chunks with the loss fused in, never materializing it. Same trick as FlashAttention: don't write intermediates out.

⭐ Three details that make the logits look worse

AngleNumberWhy it matters
Few FLOPs 0.46% lm_head: 242.92 TFLOP of the 52.50 PFLOP forward pass, under 0.5%. Cheapest step, biggest tensor
Loss in fp32 63.13 GiB Cross-entropy is unstable in bf16; logits usually go fp32. This purely temporary tensor takes 67% of one device.
⚠️ This cell once had units backwards; kept as a warning. It said: “63.13 GiB vs. 94.74 GB: 67% is wrong; converted, 67.78 GB ÷ 94.74 = 72%.” But 94.74 was GiB all along (the compiler's 95.38G − 94.74G = 656.93M only adds up in base 1,024): 63.13 ÷ 94.74 = 67% was right; the “corrected” 72% was wrong.
The pattern: knowing the unit trap, then fixing the wrong side: assume a decimal denominator, convert the numerator. Check the denominator's base first.
Percentages here use bytes, not displayed numbers.
Train vs. infer 252.5 KiB Training: loss at every position; prefill: only the last. Same op, 131,072× apart
Training · loss at every position All 131,072 positions needed 31.56 GiB 131,072 × 129,280 × 2 B That's bf16; fp32 for the loss doubles it Inference prefill · last position only Logits for the first 131,071: none needed (dashed: size if computed as in training) ↑ only this · 252.5 KiB This line lies: true height is 1/131,072 of the box, i.e. 0.0013 px. Drawn ~2,300× thicker to be visible Same op, same 926.68M weights, same code: a 131,072× gap. “Costly to train” and “cheap to serve” describe the same lm_head
Half this figure can't be drawn; that's the point. lm_head has the least compute, yet makes training's largest tensor; in inference it shrinks to a hairline. “Is this op expensive?” has no answer until you say training or inference. Hence Topic 6 treats inference on its own, not as a forward-pass special case.

⭐ Three questions nearly everyone trips on

① Why are training logits 131,072× larger? Positions, not vocabulary Same 129,280 vocabulary both ways; what differs is how many positions are scored.
Training uses teacher forcing: the whole sequence goes in at once and every position predicts. Position 5 predicts token 6, 6 predicts 7, all in parallel: one forward pass, 131,072 losses, none skippable.
Inference samples only the last position; the other 131,071 need no logits. 131,072 : 1. That's the whole gap.
An easy ratio: the logits are “18 residual streams stacked”
// at the output, width jumps from 7,168 to 129,280 129,280 ÷ 7,168 = 18.04 // so one stream 1.75 GiB × 18.04 = 31.56 GiB ← the number above
Everything else is “positions × 7,168”; only the output is “positions × 129,280.” Same positions, 18× the width: hence it stands out.
② Inference samples from the top-k anyway; why compute the small scores? True, but order matters: you need all 129,280 scores to find the top 64. Top-k selects after computing; it skips no compute.
Inference needn't save anyway: one position's full logits are just 252.5 KiB.
The pressure is in training, which can't use top-k. The cross-entropy denominator sums the entire vocabulary; drop a term and it's wrong. (Older approximations like sampled and hierarchical softmax pay with biased gradients.)
The modern fix is chunking: logits for a small slice of positions, take the loss, discard. It's exact, and saves peak memory, not compute.

With a real training config: why chunking is mandatory

The 31.56 GiB above uses the top panel's setting (128K). Try a common training shape, sequence length 4,096 with a per-device batch of 12:

// positions needing logits this step 4,096 × 12 = 49,152 // the logits tensor itself bf16 49,152 × 129,280 × 2 B = 11.84 GiB fp32 49,152 × 129,280 × 4 B = 23.67 GiB ← cross-entropy is unstable; usually fp32 // vs. residual stream, same positions 49,152 × 7,168 × 2 B = 0.66 GiB ← that 18× again
On one device, this temporary tensor takes a quarter of HBM Of 94.74 GiB HBM per device, bf16 logits take 12.5%; fp32 for the loss, 25.0%. Yet it's absent from the parameter table, costs about 2.2% of compute (at this 4K shape; 0.46% at 128K), and is dropped the next step.
Why chunking works: positions are independent The loss is a sum of per-position terms: position i's logits don't depend on position j's. So stream: slice logits → loss → discard → next slice.
slices of 1,024 → peak 1,024 × 129,280 × 2 B = 252 MiB // 11.84 GiB → 252 MiB: 1/48
And you save twice. The backward gradient tensor is as large as the logits (just p − y, elementwise, same shape). Fused, it never hits memory either: each slice is contracted at once to the “1,024 × 7,168” hidden-state gradient.
Cost: near zero. lm_head is ~2.2% of total compute; recomputing it barely registers.

Where does “the next token” come from? Almost everyone asks

At layer 0, the vector is the embedding of the current token. How does it become the next token after 61 layers? Is the “shift by one” rule written into lm_head's parameters?

No. It lives in the loss, not the architecture lm_head is trivially simple: a table, one row per token, each a 7,168-dim direction.
row w of lm_head = a direction for token w in 7,168-dim space score of w at position i = top vectorᵢ · row w // a dot product ↑ best-aligned wins
So lm_head stores only “what each token looks like,” nothing about “next.”
The guessing happens in the 61 layers below. Attention lets position i see all earlier ones, so at the top the vector means not “what token i is” but “given the first i tokens, what comes next.”
“Next” comes from shifting the training labels by one.
// corpus: it is a fine day today pos 1 2 3 4 5 input it is a fine day ← always from the corpus ↓ ↓ ↓ ↓ ↓ 61 layers + lm_head, every pos ↓ ↓ ↓ ↓ ↓ label is a fine day today ← same row, shifted left by one
Best evidence: MTP shifts labels by two, no code change MTP above predicts the token after next. Same structure, same scoring; labels shift by two. If “shift by one” lived in the architecture, this couldn't be so cheap.
③ So in training, is each position's loss computed on its most probable token? No. Training neither samples nor takes the argmax; it never “picks” a token.
It takes the corpus's true next token and the model's probability for that token; loss = −log(that). The model's own favorite plays no part.
Crucially, predictions are never fed back. Position i+1's input is the true token i+1, whatever position i guessed. Hence teacher forcing: the teacher resets you every step.
Why it mustOtherwise
Parallelism: all 131,072 positions in one forward pass Feed predictions back and position i+1 waits for position i: serial training, 131,072× slower
Differentiable: cross-entropy is smooth everywhere argmax isn't differentiable; backprop can't pass
Gradient covers the whole vocabulary: true token up, the other 129,279 down Argmax alone needs no denominator, which is why ② rules out top-k
The price: exposure bias Training always shows a correct prefix; inference shows the model's own, possibly wrong, output. That gap is exposure bias. Scheduled sampling and the like once tried to ease it; today's large models mostly fix it in post-training, with real rollouts.

④ Isn't a 0/1 label far from the real probability distribution?

After “The weather today is,” “nice,” “great,” “lovely” all work. Why does the label give one 1, the others 0? Fair point: for any single example, the label is wrong.

Key point: a label isn't a claim about the distribution; it's one sample from it Flip a coin: always 0 or 1, never 0.5. Average ten thousand flips and you get 0.5.
“The weather today is” recurs in the corpus, followed by “nice,” “great,” or “lovely.” Each pushes its own way; the averaged gradient is the empirical frequency. Each label is hard; their sum is soft.
// why it is right in expectation minimize E(x,y)~data [ −log pθ(y|x) ] ≡ minimize KL( Pdata(·|x) ‖ pθ(·|x) ) + H( Pdata(·|x) ) ↑ constant, independent of θ // so the optimum lands exactly at pθ = Pdata ← the true conditional distribution, not one-hot
Corollary 1: the model never reaches 1; gradient descent holds it back Say the model gives “nice” probability 1 after “The weather today is.” Every example followed by “great” then has loss −log(0) = ∞.
So the prefix's other occurrences rein in the push to 0 or 1. The equilibrium is the empirical distribution, not one-hot.
Corollary 2: the loss floor isn't 0; it's the entropy of language KL can at best reach 0, but H(Pdata) can't be reduced: it doesn't depend on the parameters. Loss leveling off above zero isn't failure; it measures how many sensible next words there really are.
Loss heading to 0 means memorizing the corpus, not learning language.

What if I have better soft labels? There are two routes, and their very existence admits one-hot loses information:

MethodNew labelCost
label smoothing 1 → 1−ε, rest share ε Transformer: ε=0.1; most LLM pretraining skips it (hurts calibration)
Knowledge distillation Teacher's full distribution Needs a stronger teacher. Data-efficient: it restores what one-hot discards
The original paper names the trade “During training, we employed label smoothing of value εls = 0.1. This hurts perplexity, as the model learns to be more unsure, but improves accuracy and BLEU score.” —Vaswani et al. 2017, Attention Is All You Need §5.4
“Learns to be more unsure” flips your worry: one-hot doesn't teach confusion; it teaches overconfidence, which smoothing pulls back.
In one sentence One-hot says only “this time, this token.” Soft labels add “cat is a bit like dog, nothing like car.”
One-hot averages it out of vast data: slow, but teacher-free. That's pretraining vs. distillation.

⑤ Distillation in depth: why it beats one-hot, and by how much

One-hot is correct in expectation. Its only flaw: it's slow. Distillation does one thing: replace the hard label with the teacher's full output distribution. It buys more than intuition suggests. All numbers below are from Hinton / Vinyals / Dean 2015, Distilling the Knowledge in a Neural Network (arXiv:1503.02531).

First: the teacher really teaches ratios between “negligible” probabilities “one version of a 2 may be given a probability of 10−6 of being a 3 and 10−9 of being a 7 whereas for another version it may be the other way around. This is valuable information that defines a rich similarity structure over the data.”
One-in-a-million and one-in-a-billion are both negligible, but their ratio means something: “this 2 leans toward 3, not 7.” This became known as dark knowledge: beyond the answer, the teacher hands over a map of what resembles what.
One-hot throws the whole map away.
// same position, both labels one-hot ∂L/∂logiti = pi − yi y is 0/1 → all 129,279 wrong tokens target 0, already ~0 ⇒ almost no gradient this step soft ∂L/∂logiti = pi − qi q = teacher ⇒ every token off q gets pushed some up, some down; not a single winner
Second: one-hot is a sample, so it's noisy; the teacher's distribution isn't “When the soft targets have high entropy, they provide much more information per training case than hard targets and much less variance in the gradient between training cases, so the small model can often be trained on much less data than the original cumbersome model and using a much higher learning rate.”
Back to the coin: one-hot is one flip each time. The same prefix draws “nice,” then “great,” so the gradient swings until vast data averages it out. The teacher just states the coin's bias: identical every time, no noise to average.
Hence the paper's “a much higher learning rate.”
Aside: an information estimate (my derivation, not the paper's)
one-hot, per position: at most log2(129,280) ≈ 17 bits // “which of 129K tokens” soft, per position: all 129,280 dims // each a constraint
Same forward pass, same 31.56 GiB of logits, a supervision signal orders of magnitude richer. No extra FLOPs, far more information: that's the speedup.

So how much faster? Three measured results

SetupHardDistilledTakeaway
Speech · 3% data
paper Table 5
Test acc. 44.5%
overfits; must early-stop
Test acc. 57.0%
converges, no early stop
Full-data baseline: 58.9%
1/33 the data, 1.9 points behind
MNIST
1,200 vs. 800 units
Small net alone: 146 errors
large net: 67
Small net, soft labels (T=20)
74 errors
Gap 146−67 = 79,
distilled back 146−74 = 72
= 91% (my math)
Ensemble of 10
speech frame acc.
Ensemble's gain distilled into one model “More than 80% of the improvement … is transferred”
The hardest number: 33× data efficiency 3% of the data is 1/33. Same model, same 3%: only new labels lift test accuracy from 44.5% to 57.0%, near the 33×-data baseline (58.9%). Hard labels score higher in training (67.3% vs. 65.4%): not failing to learn, but memorizing the 3%.
Distillation saves data, not compute; it buys generalization, not fit.
🪄 The most counterintuitive experiment: learning a class it has never seen Delete every 3 from the training set. The distilled model knows 3 only by hearsay.
raw 206 errors, 133 of them on the 1,010 3s // cause: class 3's bias is too low (no positive example raised it) bias +3.5 109 errors, only 14 of them on 3s → 98.6% of 3s correct, without ever having seen a 3
The teacher never taught 3 directly. But every 8 or 5 it taught came with “a bit like a 3.” In that similarity map, the classes around 3 carve out its place.
⚠️ No high temperature, no distillation Trained on one-hot labels, the teacher at T=1 is near one-hot too: the 10−6 and 10−9 sit on the floor, teaching nothing. Heating lights up the floor.
// high-T limit: distillation = matching logits (paper Eq. 4; zero-mean logits per example) ∂C/∂zi ≈ (1 / N·T²) · (zi − vi) // grads ∝ 1/T² ⇒ mixing losses, multiply soft by T² or it drowns
Does it matter? The paper cites a group that distilled at T=1: they closed only 28% of the large–small gap; heated, over 80%. Same method, wrong temperature: a third of the gain.
But distillation needs a teacher first Soft labels aren't free: the teacher averaged them out of vast one-hot data. Distillation hands over that average, sparing you the trip.
Pretraining is still the only step that creates from nothing: slow, but teacher-free. Hence V3-class models train the hard way on one-hot labels; their smaller versions can be distilled.
Aside: what the LM in lm_head stands for LM = Language Model. A head is the task-specific output layer atop the backbone. Swap it for another job: classification, scoring, reward-model heads. lm_head is the head that predicts the next token.
Third time, same thing Step 2: KV cache. Step 5: activations. Now: logits. Same shape each time: something absent from the parameter table outgrows the parameters.
The lesson isn't 31.56; it's the rule “count only weights and your memory estimate is wrong.”
① Embed
② MLA
③ Dense
④ MoE
⑤ Bus
⑥ Exit
⑦ Total
So far— weights— HBM— compute
Step 7 ⭐

Adding it all up

Seven steps done. Now, the three questions.

PartWeights MemoryForward FLOPsNotes
Embed926.68M1.73 GiB —0 as a lookup; as matmul, = lm_head
MLA × 6111.41B21.26 GiB 45.92 PFLOP81.8% quadratic
KV cache—8.58 GiB —Linear in length
Dense MLP × 31.19B2.21 GiB 311.71 TFLOP
MoE × 58656.57B1.19 TiB 6.03 PFLOP+ all-to-all 812.00 GiB
lm_head926.68M1.73 GiB 242.92 TFLOP⚠️ logits 31.56 GiB
Total671.03B 1.23 TiB 52.50 PFLOPOnly 36.62B active
Resident memory; each cell is one TPU v7 device's 94.74 GiB. Count them. Forward only, batch=1, no activations or temp buffers.

① Compute for one forward pass

52.50 PFLOP

81.8% of it is the attention quadratic term

② Resident memory

1.23 TiB

Weights 1.22 TiB + KV 8.58 GiB

③ How many devices

13.3 devices

Divide by 94.74 GiB. Only a lower bound.

⚠️ “13.3 devices” is the most optimistic value, not the answer It rests on five assumptions, none true in real training: forward only, batch = 1, no activations, no temporary buffers, perfectly packed weights.
Add 41.89 GiB of peak activations and 13.3 becomes 13.7. Training is another league: at 16 bytes per parameter (weights + gradients + fp32 master + two moments), 671B is 9.76 TiB, about 105 devices (at our 94.74 GiB/device; Topic 4 rounds to 80 GiB for mental math: 125), nearly 8× more. Details in Topic 4.
So: the most frugal accounting needs 14 devices; real training, eight times that.

Where the 41.89 GiB of peak activations comes from

The opening question counted only the KV cache: its cost is deterministic; what's stored, and how long, is clear at every step. Activations need a full trip to cost. Now we've made one.

Two terms people keep confusing KV cache = kept. Token n+1 needs K and V of all n before it. It lives all conversation, multiplies by layer count, grows linearly, never freed.
Activations = discarded after use. Layer 3's intermediates die when layer 4 starts: they live within one layer. So they don't multiply by layer count; only the fattest layer counts. Hence 61 layers peak at just tens of GiB.
(Training keeps activations for backward, so they dominate memory. We count only forward, so the two separate cleanly.)

Activations don't add: take the max of 3 peaks:

Peak atWhat's liveTotal
MLA layerKV 8.58 GiB + Q/K 6.00 GiB each + V/output 4.00 GiB each + residual ×2 32.08 GiB
MoE layerKV + dispatch/combine buffers 14.00 GiB (half each) + residual ×2 26.08 GiB
Output ← largestKV + logits 31.56 GiB + residual 41.89 GiB

The scariest item isn't attention; it's the output. Times a 129K vocabulary, logits exceed 3× the whole KV cache. (Assumes logits at all positions; pure generation needs only the last, so ignore it there. Activations vary with implementation; magnitudes, not exact values.)

The table's three “×2”s have three reasons Q and K: two same-size tensors, 128 heads × 192.
V and output: two same-size tensors, 128 heads × 128.
Residual stream ×2 differs. The incoming x must be kept, since the layer ends by adding it back; its layer-normed copy is a separate buffer. From norm to addition, two copies live, each 1.75 GiB.
MoE repeats the structure, so also ×2; the output step has no add-back: one copy.
Aside: why pretraining usually uses 4K, not 128K—what it saves isn't logits, it's attention

It doesn't save on logits. Logits depend only on total tokens (= sequence length × batch):  128K × batch 1 and 4K × batch 32 are both 131,072 tokens, and the logits are 31.56 GiB either way.

Short sequences save attention: its cost is batch × the square of sequence length. Same 131,072 tokens: the 128K split costs 32× the 4K one.

In V3's config, original_max_position_embeddings is 4,096 and YaRN's factor is 40; 4,096 × 40 = 163,840. The base model was trained at 4K and extended to 128K afterward. Also, training usually upcasts logits to fp32 for cross-entropy, doubling them; hence chunked cross-entropy is standard: compute a slice, discard it, never materialize the table.

Same model; new accounting, new magnitude. Bar = TPU v7 devices needed ① Forward · batch=1 · weights 13.3 ② + peak activations 41.89 GiB 13.7 ← nearly same as ① ③ Training · 16 B per param 2 B 2 B 4 B 4 B 4 B ~105 bf16 weights grad fp32 master moment m moment v 671.03B params × 16 B = 9.76 TiB ÷ 94.74 GiB/device (both binary; divide directly) ⭐ Not “105” but the shape: first two bars ≈ equal, third 8× longer Orange/purple/red (fp32 master + 2 moments = 12 B): all optimizer; the model is only 2 B. Cutting them is all of Topic 4
①②③ differ not in precision but in the question. ① “do the weights fit?” ② “does one forward pass fit?” ③ “can it be trained?” Answers an order of magnitude apart hide in one question: “how many accelerators does this model need?” Next time, ask back: which accounting?
⚠️ Units: every count here is in devices, since 94.74 GiB is per-device capacity. A TPU v7 chip holds two. So 105 devices = 53 chips. Both right, different things. This 1:2 will bite again in later topics.

Another angle: compute in seconds

PFLOP has no intuitive scale; divide by hardware compute to get seconds. A single TPU v7 device peaks at about 1,153 TFLOP/s in bf16: the official 2,307 BF16 TFLOPS per chip divided by its two devices.

Theoretical limit (100% utilization)

45.5 s

One device, one pass. Unreachable in practice.

More realistic (35% utilization)

2.2 min

Typical real utilization in large-model training

Spread over the devices that hold it

9.3 s

At 14 devices (13.3 rounded up; no half devices), 35% utilization, assuming perfect, lossless parallelism.  That clause is the rest of the course.

“Assuming perfect, lossless parallelism” is the doorway to the course 14 devices are not 14× faster: they swap data, wait for the slowest, recompute to save memory. How big that gap is, why, and how to close it: Topics 5 and 7.
Now “why parallelism?” needs no argument: it's a computed conclusion. How to parallelize, what to split, how to hide communication: Topic 5. This lecture only had to compute that it doesn't fit, not assert it.
Step 7.5

Computed vs. measured numbers

Seven steps of derivation. One question: can you trust it?

You can check it without occupying a device The compiler is a static analyzer: compile model plus parallelism config ahead of time (AOT) and it reports the memory without allocating an accelerator. You get the answer before queuing. Cross-checking the table above is the cheapest step.
Below is no AOT how-to; we don't cover running it. It asks how far to trust a tool that answers without running. Any performance model or capacity sheet gets the same verdict.

Run it for real and you learn three things, the hard way, not from a book.

① Compiles ≠ fits

A clean compile means only a legal graph. We've seen green builds far exceed device memory.

Worse, errors mislead. Compile failures may pose as “out of memory,” so you shrink the batch: exactly wrong. A wrong error is worse than none.

② The number isn't everything

Only temp buffers; parameters and optimizer state are separate (see Topic 4).

And it doesn't shrink with parallelism: shard wider, temp buffers stay put. The easiest to miss.

③ ⭐ It fails right near the limit

Far from the limit, very reliable. Near the limit, in a narrow band, it can't tell.  All our wrong calls fell there.

Static analysis is no oracle. It rules out most obviously bad configs; the final call is yours.

x = config memory ÷ device limit. Color = can static analysis be trusted here? Device limit Far below: accurate Way over: accurate ⚠️ shaky 0 0.5× 1.0× 1.3× Configs worth trying: nearly all here — who'd try an obvious OOM or obvious waste? ⭐ So: static analysis is least reliable exactly where you need it most. A property, not a bug; true of any tool
The red band's width is illustrative; its position isn't. Narrow, at the limit; no reproducible “a few percent”: model, implementation, compiler decide. We vouch only for the shape: solid ends, shaky middle, where decisions happen.
One more surprise Same model, same parallelism: swap one op's implementation and memory changes, in no fixed direction, up or down.
Memory belongs not to the model but to model × implementation × compiler. This recurs throughout Topic 7.

Takeaways

Numbers fade. Keep these counterintuitive conclusions: every later lecture uses them.

01 · MoE

It saves compute, not memory.

“Only 37B active” doesn't mean 37B-dense cheap: HBM still holds all 671B.

02 · Long context

Long sequences move the bottleneck from parameters to length.

Attention is 12% of compute at 4K, 82% at 128K, 97% at 1M. Same model, new length, new optimization target.

03 · “Blowups”

Scariest tensors: temporaries, not weights.

Attention scores and logits aren't even parameters. Used once and discarded, they outpeak any layer's weights. Weights-only memory math is wrong.

04 · Architecture

Some architecture numbers are set by hardware, not quality.

“4 groups first, then experts” only caps how many machines traffic spans. Separate what makes a model smart from what makes it run.

05 · “Experts”

In the only publicly measured model, experts don't split by domain.

Mixtral's authors, searching by topic, found no obvious patterns. Yet consecutive tokens repeat an expert 26.4% of the time in middle layers (random: 12.5%). Routing seems to track syntax and token, not subject.
⚠️ Mixtral is top-2 of 8; V3 is top-8 of 256: no direct transfer. Only this is certain: across 58 MoE layers, each token takes its own path.

06 · The accounting itself

Memory belongs to model × implementation × compiler, not the model.

Same config, new kernel: memory changes, either way. Static analysis prunes most non-starters but doesn't make the final call.

Next lecture
Topic 2 · TPU vs GPU
This lecture measured everything against one number: 94.74 GiB.
“Doesn't fit,” “needs 14 devices,” “logits take 0.33 of a device”: each came from dividing by it.

But what does that device look like? Why exactly that number?
Why are some chips called GPUs and others TPUs, and how differently do they see the same model?

Topic 1: what the model needs. Topic 2: what the machine can give.
(Topic 2's course page is finished: more than 40 figures, with instructor notes.)
Topic 2 → Course home

Left out on purpose

A boundary, not an oversight.

Backprop & optimizer state

Forward is fact; backward mixes in policy. Together they blur → Topic 4

Parallelism

We only show it doesn't fit → Topic 5

FP8 training

Key V3 contribution; a precision/numerics topic → Topic 8

Inference

Prefill/decode, continuous batching; here: training forward only → Topic 6