Topic 6 · Inference

How to use these notes

Two uses.
① Presenting: read it aloud: lines, screen cues, minutes per segment.
② Self-study: hand the whole file to an AI to walk you through as a TA.

The content lives on the course page; these notes cover only how to teach it. Course author: Chris Yang · Google Cloud AI Infra architect.

⏱ Time: core track 90′

⛔ Never cut Sections 0 and 1. Section 0 plants “every decode step makes one full trip”; Section 1 shows why it's expensive; every later section builds on it.
Ten sections, about 90 minutes, no break. To go deeper, each section on the course page ends with a link to the Chinese full edition (L300).
In the script, the K in 4K, 128K means thousand tokens.
The floor below, with interaction, comes to about 76 minutes (recounted each build; the table rules): reading only, at 140 words a minute; pointing, waiting, follow-ups aren't in it. Slack varies: most in Section 7 (~3′), under 1′ in 6, 8, 9 (8 almost none): take questions in roomy sections, theirs after class. Section 0's opening recap question gets one minute only; don't reteach Topic 5.

SegmentMinCuttable?The one thread
0 Why inference is expensive, and how it got cheaper8—Every decode step hauls the whole load (weights + KV) once; only two ways to save: more riders per trip, or haul less
1 A long prompt in one pass, but output one token at a time?9—The bus: fixed fuel per trip; prefill rides full, decode carries one rider a trip
2 Just add users? How big can the batch get?9—The bus stops at every step: board and get off anytime (continuous batching); seats are KV; the price is a longer wait per user
3 KV cache: how big, how to shrink?8—Each request's notebook: how thick a line is (MLA / GQA), no wasted paper (paging), never recompute (prefix reuse)
4 Many devices: TEP or DEP?11—Who copied the notebook, how many times; test: TP degree ÷ KV heads; capacity and speed apart: SGLang (DEP on both ends) gains throughput only with full seats; large-scale EP fills the expert buses
5 Long context: DCP vs. PCP10—Who moves: PCP moves KV, no answers to merge; DCP moves Q, answers must merge; move whichever is smaller
6 PD disaggregation: one more KV trip, yet more users?9—Two jobs drag each other (switch to a kitchen: one stove for prep, one for plating); split, each runs its own batch
7 Speculative decoding: many tokens per trip?11—Guess, then verify: hits are free, misses cost little; bigger batch, smaller gain; drafts move ever closer to the big model (EAGLE needs training; MTP differs in who trains it)
8 Quantization: fewer bytes, accuracy drop?8—Halve the bits, halve the data-movement floor; experts to 4 bits, attention 8+ bits; KV can be compressed too
9 Day one: which questions first?7—Back to the trip: the eight questions are one question: where is this trip stuck?
Total90

⛔ Above is the plan; below is the floor (script at 140 words/min + 15 s per screen switch + half a minute per interaction; recounted at every build).

SectionScript (words)ScreensFloorPlannedInteractions / skippableWith interactionAfter skips
Section 0 · Why inference is expensive, and how it got cheaper59455.5′8′2 / 06.5′6.5′
Section 1 · A long prompt in one pass, but output one token at a time?52385.7′9′2 / 06.7′6.7′
Section 2 · Just add users? How big can the batch get?68366.4′9′2 / 07.4′7.4′
Section 3 · KV cache: how big, how to shrink?57955.4′8′2 / 06.4′6.4′
Section 4 · Many devices: TEP or DEP?92988.6′11′2 / 09.6′9.6′
Section 5 · Long context: DCP vs. PCP74477.1′10′2 / 08.1′8.1′
Section 6 · PD disaggregation: one more KV trip, yet more users?80898.0′9′1 / 08.5′8.5′
Section 7 · Speculative decoding: many tokens per trip?71887.1′11′2 / 08.1′8.1′
Section 8 · Quantization: fewer bytes, accuracy drop?75566.9′8′2 / 07.9′7.9′
Section 9 · Day one: which questions first?56255.3′7′2 / 06.3′6.3′
Total66′90′19 / 076′76′

Section 0 · Why inference is expensive, and how it got cheaper (8′)Planned 8′ · with interaction 6.5′

🖥 Main screen for this segment: course page Section 0 · Why inference is expensive, and how it got cheaper · main figure Four years cheaper: each fix leaves a new gap

🎯 What this segment must leave behind: each token hauls the whole load once; each recent technique fills the last one's gap.

🖥 Screen: open the collapsed “🎯 Warm-up: which collectives per parallelism?” under the Section 0 heading; hold off on “Check answers”.

First, a recap. Topic 5 made five cuts: DP, FSDP, TP, PP, EP. Each cut adds one kind of communication.
Across the top, five kinds: AllReduce, AllGather, ReduceScatter, AllToAll, point-to-point. For each row, tick what a training step needs; more than one is fine. One minute.

💬 Interaction: everyone ticks on the page (or read rows aloud for a show of hands). After a minute, click “Check answers”; ask someone about a red row. Expected: DP: AllReduce; FSDP: AllGather + ReduceScatter; TP: AllReduce; PP: point-to-point; EP: AllToAll.

For DP and TP, AllGather plus ReduceScatter also counts: AllReduce is those two halves joined.
This table is all training, where the answers already exist: compute the whole sequence at once, watch only throughput. Inference differs: someone's at the screen, waiting.

Something you do daily: Paste a four-thousand-token document into a chat box, ask a question, hit Enter.
A short wait, then the first token. Then tokens pop out one by one, steadily. A reasoning model first thinks a long passage to itself.

On the other end, a very expensive chip works for you. Its headline spec: operations per second.
But as those tokens trickle out, that compute mostly sits idle, waiting for data. Two things today: why this is expensive, and how it got cheaper in recent years.

🖥 Screen: stop on Only two waits: first token, then each next. Point at the red dot, then the green boxes.

That scene, as a figure: First the model reads your whole document: prefill. Then it emits the first token: this red dot. Each green box after it is one more token: decode.
As a user you feel just two things: the wait for the first token, and the gaps after.

One term: what the model emits is a token. Today, think of it as roughly a word.
The purple staircase climbs one step per token. That's the KV, coming up: it remembers every earlier token and keeps growing.

🖥 Screen: “The cost: one full haul per token”: the three bullets and the green box, “In one sentence.”

Where's the cost? Our running example: DeepSeek-V3. Each token touches about 37 billion parameters; at one byte each, 37 GB. Why only a slice? Section 1.
Those 37 GB live in HBM: a warehouse. The multiplying happens at a small workbench nearby, about one fifteen-hundredth of the warehouse.

It won't fit, so every token hauls those thirty-odd GB out again.
The second cargo is KV. To write the next token, the model looks back at every earlier one and saves what it found, so the next step reads instead of recomputing.
That's each user's notebook, thicker the longer the chat: at a hundred-odd thousand tokens, over 8 GiB.

Picture a bus. The weights are its own cargo, hauled every trip: fixed fuel, about the same for one rider or a full bus. The notes are each rider's luggage: more riders, more weight.
So only two ways to save: more riders per trip, or a lighter load. Just one leaves the trip alone and splits the two kinds of work: Section 6.

💬 Interaction: ask “More riders, lighter load: which technique comes to mind?” Take two or three. Expected: more people per bus (later: batching) = more riders; smaller numbers (quantization) = lighter load. No answers? The table below answers it.

🖥 Screen: Four years cheaper: each fix leaves a new gap: sweep from 2022 at left, then stop on the seven-row table.

Most savings came step by step along those two roads. Many names; read only the “Bottleneck” column.
In 2022 the bus made no stops: everyone rode to the end with the slowest passenger. Let people on and off at every stop, and it fills up. Once full, notes eat the seats.
Every later row: fix one gap, expose the next. Each section takes one link; don't memorize names today.

Two outside forces kept pushing. Context grew to a million tokens, thickening everyone's notes. Reasoning models think hundreds to thousands of tokens first, so one-at-a-time output became most of the work. They forced several later techniques.

🖥 Screen: scroll to the blue “Guess first” box under the table; point at DeepSeek-V4-Pro in it.

A question for the last section. Same GB300s, same model, DeepSeek-V4-Pro, same per-user speed; only software changes. How many times launch day's per-GPU throughput? One and a half, two, or five? Remember your guess.
Section 1 returns to the opening: four thousand tokens read in a moment; why does output come one at a time?

⚠️ Where this segment drifts⚠️ Warm-up: one minute only; call on no one, explain no collective (Topic 5). “Which does inference use?”: same table, new weights. TP's few all-reduces are small and latency-bound; EP need not use all-to-all (small print under the answers); Section 4, Chinese full edition Section 6.
⛔ Don't reveal the multiple before Section 9.
⚠️ Only the map here: expand no table row (later sections own them), or it becomes a chronology.
⚠️ “Why only 37 billion?”: MoE has hundreds of experts per layer; each token picks a few (Section 1). Total: 671B.
⚠️ “V3's 600-odd billion parameters won't fit one chip; who hauls the 37 GB?”: one chip is for the picture; really, several chips each store and read a share; the floor divides by device count (Section 8 backup: eight v7s, FP8, ≥ 0.63 ms, exactly 1/8 of the single-chip 5 ms; our derivation). Strictly, “every token” means “every step”: each rider emits one per step (Section 1).
⚠️ “Does a GPU re-haul every step?”: yes, same reason; the page gives no GPU on-chip cache size, so quote none.
⚠️ “Is the price drop all systems work?”: no: the AI Index credits mostly stronger small models; the opening question counts software only. Quote no price-drop multiple: all price-based (reclassified in the full-edition ledger). Use only the 5.1× (same GPU, model, interactivity tier).
⚠️ Spoken roundings: workbench = one v7 chip's 128 MiB on-chip (2 TensorCores × 64 MiB), ~1/1,500 of the chip's 192 GiB HBM (same per device: 64 MiB vs 94.75 GiB; don't pit 128 MiB against one device's HBM); “over 8 GiB” = 8.58 GiB at 128K; “a million tokens” = Gemini 1.5 Pro preview, 1M, 2024-02 (GPT-4 Turbo hit 128K first, 2023-11); reasoning models = o1 (2024-09), R1 (2025-01).
⚠️ “Four thousand tokens” just matches Section 1's 4K prefill. “Mostly idle” means one user (Section 1: 0.32% used).
⚠️ “Two hollow dots at the axis's right end”: new model-side directions. One swaps most layers for linear attention: one fixed-size notebook per user, not per-token notes. The other, a recurrent model, loops the same layers several times. Both ideas are old; the figure dates their mainstream arrival. Engine impact: Chinese full edition, Sections 13 and 0.

Section 1 · A long prompt in one pass, but output one token at a time? (9′)Planned 9′ · with interaction 6.7′

🖥 Main screen for this segment: course page Section 1 · A long prompt in one pass, but output one token at a time? · main figure One user, one token: 5 ms moving, 0.016 ms computing

🎯 What this segment must leave behind: prefill is compute-bound, decode is bound by data movement; decode carries one rider a trip and still pays full fuel.

Back to Section 0's scene: a four-thousand-token document, a short wait, the first token, then the rest one at a time.
Four thousand tokens read in one pass; output, one pass per token. Same chip; why the difference?

🖥 Screen: “Rows per pass decide: compute- or memory-bound”: the three bullets.

Same bus. Each trip hauls the weights out of HBM once. Each seat holds one token: one row of the matmul.
Prefill is a tour group: the document's ready, all four thousand tokens board at once: a full bus. In decode, each user boards one token per trip: rows on the bus equal users.

Why can't later tokens board early? The second token isn't known until the first arrives.
Training has the answer written, so it computes whole sequences at once. Inference doesn't: one at a time.

So how many riders fill the bus? TPU v7's compute ÷ its bandwidth = 313: the ridge point. In the time to read one byte from HBM, the chip does 313 operations.
Each extra rider adds one operation per byte hauled. Compute stops waiting only at 313 riders. A four-thousand tour group is far past that; one rider uses under a three-hundredth of the compute.

🖥 Screen: the plot under Rows per pass decide: red line, orange dots top right, green dots bottom left.

Orange is prefill: above the line, compute-bound. Green is decode: all below the line, bound by data movement.

💬 Interaction: ask “DeepSeek-V3 reads 37 GB per token; one v7 reads 7.37 TB a second. How long per read?” Wait five seconds. Expected: about five milliseconds. Five seconds? It's TB.

🖥 Screen: One user, one token: 5 ms moving, 0.016 ms computing. Untick “Autoplay”, click “⟲ Reset”, then “Start”: weights load onto the workbench block by block. At the first stop, point at the blue bar, then the near-invisible orange.

Right, five milliseconds. And the compute? 0.016 milliseconds: this orange bar, too thin to see.
A very expensive chip, over 99% of its time waiting on the warehouse.

🖥 Screen: click “Next”: the second token hauls the full load again.

Next token? The workbench can't hold 37 GB; that load is gone, so haul it all again. Another five milliseconds, one token.

🖥 Screen: “Next” again: 313 rows a trip; orange catches up with blue.

Same load, 313 riders. Only now does compute catch up with data movement.
Flip it, more interesting: below that, one more rider barely lengthens the trip. The bus runs anyway; extra riders ride free.
So faster compute won't help. Bandwidth sets the floor.

🖥 Screen: More users: weights amortize, KV doesn't: blue segment, then purple.

The fix: more riders per trip. The bus's cargo is shared, so each rider's blue segment shrinks.
But purple doesn't move: each user reads their own notes; no pooling spreads that.

As chats get long, purple takes over. A few hundred riders, each with a hundred-odd-thousand-token chat: notes are nearly 80% of the load.
In the end, notes fill the seats: Section 3.

🖥 Screen: MoE breaks the batch apart again: filled square left, then thin line right.

One more cut. Each layer has two steps: attention, which looks back through the notes, then a big feed-forward network. MoE splits it into many pieces: experts.
In bus terms, each layer is a transfer station. DeepSeek-V3 has 256 branch lines per station; each token rides only eight. That's the earlier “each token touches only 37 billion.”

💬 Interaction: ask “You finally got 256 people aboard. At the transfer, each picks eight branch lines. How many ride each line, on average?” Wait five seconds. Expected: eight (256 × 8 ÷ 256).

Right, just eight. Each branch line is its own bus, hauling its own weights for eight riders. Empty again: back to a few riders a trip.
To fill every line, by the full edition's H100 math, a step needs over nine thousand tokens. Section 4 solves it: spread the experts over more devices.

🖥 Screen: the green box, “Takeaway.”

Take away one line: prefill is compute-bound, decode is bound by data movement. To earn back each trip's fuel, carry more riders.
The next three sections take that road: keep the bus full, free seats from the notes, regather the riders MoE scattered. Step one: why not just add more users?

⚠️ Where this segment drifts⚠️ Spoken roundings: ridge point 313 = v7 bf16 2,307 TFLOP/s ÷ 7.37 TB/s; decode really needs about 343 riders to cross (inputs and outputs move too); “over 99% waiting” = 0.32% of compute used; “nearly 80%” = KV is 78% of bytes read per token at 256 users × 128K; “fill every line” = about 300 rows per expert (Chinese full edition, Section 3, H100).
⚠️ “0.016 milliseconds” ≈ 74 billion operations per token ÷ 4,614 trillion/s (W8A8, FP8 compute), not the ridge point's bf16 2,307. Dividing by bf16 gives 0.032 ms; the ratio stays 0.32% (weights at two bytes: 10 ms; both our derivation). Answer if asked; keep the line.
⚠️ Five ms and 0.32% are ideal lower bounds from specs, not measurements: weights only, no KV, communication or kernel overhead.
⚠️ “One more operation per byte per rider” assumes 16 bits: two bytes per weight, and a multiply-add per row is two operations; they cancel. Say “in the time to read one byte, 313 operations,” not “each byte brings 313 operations.”
⚠️ Ridge point: intuition only (operations per byte-read time). Don't derive the intensity formula (L300 Section 1).
⚠️ “Each user pays 9.14 GB” stays in the full edition. “More users, more experts read?”: yes, the cargo grows some; 64 users touch about 87% of experts (full edition, Section 1 caption: 1 − (1 − 8/256)64 ≈ 87%). Don't walk through the formula.
⚠️ “Five ms a token is 200 a second; why does Section 2 say 70-plus ms apart?”: other hardware and model, hundreds of users per bus, so more notes per step and longer steps. Five ms is one user's weights-only ideal floor.
⚠️ “Over nine thousand” uses H100's ridge (about 300; full edition, Section 3). No v7 figure on the page; don't improvise.
⚠️ “V3 is FP8; why a bf16 ridge line?”: FP8 doubles compute (ridge about 626), but each parameter is one byte, so operations per byte double too; still 0.32%.
⚠️ “Nearly 80%” (78%) is a byte ratio, not a claim that eight chips hold notes for 256 users × 128K.
⚠️ “Same on H100?”: same logic, different ratio. The page computes only v7; quote no GPU ridge point.
❓ Likely questions (backup answers, not counted in the floor)
🛠 GPU veteran: “FlashAttention already fills compute for attention, so gathering 300-plus users is just for EP, right?” → First half backward. FlashAttention keeps the score matrix out of HBM; that helps prefill and training (thousands of query rows at once). In decode each user has one query row and reads a whole private notebook, unpooled (the figure's unmoving purple). Attention's operations per byte depend on how many query heads share a KV copy: one (MHA), a group (GQA), all (MLA; Section 3). The 300-plus users serve the shared weights: projections around attention, shared experts, dense layers. Routed experts get eight rows each and need over nine thousand tokens to cross: Section 4's large-scale EP.

Section 2 · Just add users? How big can the batch get? (9′)Planned 9′ · with interaction 7.4′

🖥 Main screen for this segment: course page Section 2 · Just add users? How big can the batch get? · main figure Gray cells pile up…

🎯 What this segment must leave behind: continuous batching keeps the seats full; KV sets how many; more riders, slower each.

Last section: one rider per decode trip is a waste, so carry more. This trick changes no model code or kernels, only the queue, and serves tens of times more users.
What's hard? Riders don't board or leave together. One asks “what day is it?” and goes. Another wants a 3,000-word report.

🖥 Screen: scroll to “Old way”; stop on its two bullets.

The earliest way is static batching: fill the bus, leave and arrive together. Early finishers can't get off; their seats ride empty.
Every step still hauls the weights once; empty seats ride for nothing.
Why not swap riders midway? Not for lack of ideas. Old engines had one button, run the whole batch; the scheduler never saw which step it was on.

🖥 Screen: scroll to “Stop at every step”, play Gray cells pile up…; point at the gray cells growing on top, then below.

Then Orca, in 2022. First idea: after every step, stop and look; whoever's done gets off at once, whoever's at the door gets on.
Watch: gray cells, empty seats, pile up on top; below, none. Same steps, the bottom finishes clearly more.

This became continuous batching. vLLM and SGLang both do it today.
Same bus: the old way is a charter: fill it, arrive together. Continuous batching is a real bus: every stop, some off, some on.

But riders are at different points: one on token three, another on token 3,000. How can they share a trip? Orca's second idea.
The weight matmul doesn't care whose row it is. Line everyone up in one strip and multiply; the weights are read once, shared.

Only attention can't merge. It means rereading your own notes, and everyone's differ in length and content, so each reads their own.
Most of a step is merged; only note-reading runs separately.

Boarding needs care too. Swallow a prompt of tens of thousands of tokens at once, and that step runs long, stalling everyone mid-output.
So cut it into chunks, one per step; the rider boards gradually. That's chunked prefill. After the last chunk, the first token comes out at once; next step, they're a regular rider.

One or two board per step. So how does the bus reach hundreds? Boarding takes one step; emitting keeps you aboard hundreds or thousands. Add a rider every few steps and it stays full.
The door isn't the bottleneck. The seats are.

🖥 Screen: scroll to “Seats are KV”; stop on the first paragraph. Don't scroll on: the figure below answers question two.

How many can ride at most? Each rider takes a block of KV: last section's notebook. As many notebooks as memory holds, that many seats.
One equation links seats and throughput: tokens per second is roughly riders over the time between tokens.

💬 Interaction: ask “At a steady 50 ms between tokens, how many riders does one device need to emit 10,000 tokens a second?” Pause five seconds. Reveal: 500 riders (10,000 × 0.05 s).

What if memory holds only 300 notebooks? Throughput is pinned around 6,000. Everyone else at the door just queues.
That's running out of seats.

💬 Interaction: ask “So add seats. On GB300 with V4-Pro, a bit more KV-pool memory made room for 13% more riders. By the formula, how much should throughput rise? How much did it?” Pause five seconds; most say 13. Reveal: only 0.8%.

🖥 Screen: down to Over 10% more seats…: top and bottom bars, then the middle two.

The formula's fine. Our mistake: assuming the time between tokens stays put. The extra riders cost everyone: the gap grew almost 10%.
Why? The weights were already shared; a newcomer brings only their own notes. More riders, more notes to read, slower steps.
A bit over 10% more riders, almost 10% slower steps. Divide, and it nearly cancels.

🖥 Screen: scroll to “So batch size”; stop on Throughput vs. interactivity…, trace the green line right to left.

The whole tradeoff, drawn, is this line. X-axis: tokens per second per user; y-axis: total tokens per second per GPU.
Move left: a fuller bus, a better-paid GPU, slower users. Cut each user's speed by more than half, and each GPU emits ten times the tokens.
So fix “how fast per user” first. From here on, that's the interactivity tier.

When someone says “our per-GPU throughput is double yours,” ask first: how fast per user? They may just be further left on the line.
Production does pick left: DeepSeek gives each user only twenty-odd tokens a second, for high throughput.

🖥 Screen: the green “Takeaway” box.

Three steps, then. Carry more: continuous batching keeps seats full. How many at most: as many notebooks as fit in memory. Carry too many: each rider slows a bit.
What's the trick worth? In the Orca paper, at the same latency, it served 36.9 times the old way's requests.

Notebooks set the seats, so naturally: how thick is each one, and can it be thinner?
A thinner notebook wins twice: more riders fit, and less to read each step makes everyone faster. This section was all tradeoffs; this is a rare free win.

⚠️ Where this segment drifts⚠️ 36.9× basis: GPT-3 175B, median per-token latency 190 ms for both, vs. FasterTransformer static batching; not today's engines.
⚠️ Animation lengths are illustrative (15 steps: static finishes 8, continuous 14), not a measured ratio: say “clearly more.”
⚠️ Curve endpoints (course page, Section 2): 895 per GPU at 173 tokens/s per user, 9,759 at 68. “More than half, ten times” rounds 68/173 ≈ 0.39, 9,759/895 ≈ 10.9.
⚠️ DeepSeek: about 14,800 per 8-GPU node, other hardware and model; don't compare with the per-GPU curve.
⚠️ The curve's three points: InferenceX's vLLM. Save SGLang's 11,200 and the Day-0 2,200 for Section 9.
⚠️ Chunked prefill: one line; step assembly, chunk size: Chinese full edition, §3.
⚠️ The 13% case, as recorded: concurrent requests per rank 886 → 999 (in dep8 a rank is one GPU; spoken as “riders”); time between tokens 77.7 → 85.3 ms (“almost 10%”: 85.3/77.7 ≈ 1.10); throughput 10,614 → 10,704.
⚠️ “Riders ÷ time between tokens” is mental math only; don't derive or name it (Little's law, if asked). The figure's “≈ +3%” ceiling is our 999/886 × 77.7/85.3 (“nearly cancels”). 886 → 999: mem-fraction 0.85 → 0.88, +13% (Chinese full edition); “room for” is hedging. The record doesn't say why throughput stalled; blame no one.
⚠️ “500 riders” and “pinned around 6,000” are our derivation, not measurements: 10,000 × 0.05 s = 500; 300 × 1000 ÷ 50 ms = 6,000.
⚠️ “How fast must the door add riders?”: 1,000 tokens each, 300 aboard: one every three steps on average (illustrative, not measured; Chinese full edition, §3).
⚠️ “Does chunked prefill remove the stall entirely?”: later chunks read all earlier notes, so they're heavier (128K in 8K chunks: last ≈ 7× first). Chinese full edition, §3.
❓ Likely questions (backup answers, not counted in the floor)
🛠 GPU veteran: “If the CPU picks riders every step, doesn't the GPU sit idle?” → Run the CPU a step ahead: it schedules the next step while the GPU runs this one, assuming everyone keeps writing rather than waiting for results. The new token stays on the GPU and feeds straight in; finished riders are spotted a step late and leave next round. SGLang: 1.1× throughput (Chinese full edition, §3.3). (Added 2026-10-22 after a trial run.)

Section 3 · KV cache: how big, how to shrink? (8′)Planned 8′ · with interaction 6.4′

🖥 Main screen for this segment: course page Section 3 · KV cache: how big, how to shrink? · main figure 8× the context…

🎯 What this segment must leave behind: seats = KV pool ÷ notebook thickness. To seat more: thinner lines, then no wasted paper, then no redoing.

Last section ended here: seats are notebooks. Every token gets a line, and writing the next one means rereading every line before it.
How thick is it? Work it out the oldest way first. It'll shock you.

🖥 Screen: scroll to “Step one: same model shape, line thickness differs over 50×”; stop on the three bullets.

The model reads its notes from many angles at once. One watches grammar; another tracks who “he” is. Each angle is an attention head.
The oldest scheme, MHA, gives every head its own copy. At V3's shape, one user filling the context needs over 500 GB of notes alone; two and a half chips couldn't hold it. That's our derivation; stored that way for real, it won't run.

So step one: thinner lines. GQA has several heads share a copy: Llama's eight groups mean only eight copies. DeepSeek's MLA goes further: each layer stores one compressed summary, and every head computes against it.
For the notes, MLA counts as one KV head; we'll need that next section. Divide, and the notes shrink to about 1/57 of MHA. That division is ours.

Does thinner make the model dumber? The GQA paper tested one model: decoding over five times faster, average score barely moved.
One model, one machine. Trust the direction; don't copy the multiplier.

🖥 Screen: scroll to “Thinner isn't enough: longer notes, fewer riders”; stop on 8× the context…. Point at the three-segment bar, then the green line.

But thinner lines still grow in number with every token. Look: half a v7 chip, which software calls one device. Minus weights and a small reserve, forty-odd GiB is left for notes. Green is MLA: at 4K context, over 170 riders fit.

💬 Interaction: ask “Same device, same V3, context from 4K to 32K, eight times longer. How many riders?” Pause five seconds. Reveal: exactly one eighth, 172 → 21. Then the red X: MHA seats no one before 32K.

Double the tokens, halve the riders. So long context costs twice: fewer riders share each trip, and each reads thicker notes every step.
That's it for thickness. What's left is how the paper is laid out.

🖥 Screen: scroll to “Step two: no wasted paper, a page at a time”; play Request 2 leaves…. When request 2 ends, point at blocks 2 and 5.

Step two: don't waste paper. The old way couldn't know who'd write how much, so it reserved everyone the thickest notebook, in one contiguous chunk. Unwritten space sits empty; no one can borrow it.

💬 Interaction: ask “Reserving like that, how much of the KV memory is actually used?” Pause five seconds. Reveal: only 20–40% (vLLM paper, measured: 20.4%–38.2%).

vLLM's PagedAttention makes it a loose-leaf binder. A page holds a fixed number of lines; fill one, take whichever page is free next. A page table keeps the order.
Watch: request 2 leaves; its two pages go straight to request 4. Utilization reaches 96%.

The cost: notes scatter across memory, so every read starts with the page table.
It buys more than no waste: two requests with an identical opening share one page, no copies. Step three grows from that.

🖥 Screen: scroll to “Step three: reuse a shared opening and keep writing”; stop on A cached prefix isn't recomputed….

Step three: don't redo what's done. Many requests open alike: same system prompt, a chat's first turns. Those notes are already written; keep them, and the next request carries on.
Notes turn from scratch paper into a cache. When it's full, evict whatever has gone unread longest.

DeepSeek even keeps these notes on disk; over half a day's input hits. For one very long prompt, the first token goes from 13 seconds to half a second.
Isn't disk the slowest? It is. But reading finished notes back still far beats recomputing.

But only an exact prefix counts. Each line is written looking at all tokens before it. Change one token at the start and every later line changes; all must be rewritten.

🖥 Screen: stop on the green “Takeaway” box.

Each step saves one thing: thinner lines save thickness, paging saves empty paper, prefix reuse saves recompute. In the vLLM paper, one baseline already did continuous batching, yet vLLM still got two to four times their throughput at equal latency.
But that's one device. When a model needs several, is each notebook stored once, or copied to every device? That's next: TEP or DEP.

⚠️ Where this segment drifts⚠️ “Over 500 GB, two and a half chips”: our derivation at V3's shape (128 heads; 3,997,696 bytes ≈ 4 MB per token; “filling the context” = 128K tokens ≈ 524 GB ≈ 488 GiB, vs. 192 GiB per chip). No model stores KV this way.
⚠️ The GQA score is the paper's average (T5-XXL converted to 8 groups, TPU v4; 47.1 vs. 47.2 original = “barely moved”). Not a general multiplier; conversion and retraining: see the paper.
⚠️ Asked “doesn't MLA's decompression cost compute?”: in decode each head works off the summary, at about 3.4× the MACs of the expanded form (Chinese full edition, §5). Decode lacks bandwidth, not compute, so it pays.
⚠️ “About 1/57” and “about 5.4× faster” are our derivation (dividing two public numbers); the MHA row is hypothetical.
⚠️ The GQA and MLA rows are different models (Llama-3.1-70B vs. V3): don't say “MLA is N× smaller than GQA”.
⚠️ MLA's summary: 576 numbers, 1,152 bytes per layer, 61 layers; 1/57 = MHA row ÷ this.
⚠️ “Forty-odd GiB” = 94.75 GiB × 0.9 (10% reserve) − 39.1 GiB (1/16 of the weights) = 46.2 GiB per device (same sum as the course figure's top bar), rough, assuming all weights sharded. Each device really holds its own non-expert weights, so the pool is smaller (split: Chinese full edition).
⚠️ vLLM's 2–4× is whole-system, not paging alone. “One baseline already did continuous batching” = the authors' Orca reimplementation. Basis: vLLM paper (SOSP 2023), equal latency, vs. FasterTransformer and that Orca, not today's engines.
⚠️ “One very long prompt” is 128K and high-reuse in the source. The two disk panels come from documents of different dates: “over half hit” and “13 s to half a second” aren't one measurement.
⚠️ Out of scope (if asked: Chinese full edition, §4): the purple line (linear-attention hybrid), how DSA / V4 save reads and storage, per-block prefix hashing.
❓ Likely questions (backup answers, not counted in the floor)
🛠 GPU veteran: “If a long chat drops early turns to fit the context, does the cache still help?” → Mostly not. Once the opening changes, no later notes match; the LMCache paper reports this can halve the prefix hit rate. (Moved off the page 2026-10-20.)
🛠 GPU veteran: “How is KDA different from MLA? Isn't it also read once, used by many heads?” → Two routes. MLA is still notes: a line per token, growing with length, thin because all heads share one compressed summary (more work per byte). KDA is linear attention, no notes: each head keeps a fixed head_dim × head_dim state, updated every token, same size at any length. But each head's block is its own, unshared, read, updated and rewritten every step, each number used only a few times: still movement-bound. MLA wins on compute per byte, KDA on not crowding seats at long context. Real models mix them: K3 has 69 KDA + 24 MLA layers (config). KDA's state pool and prefix hits have their own issues (Section 5's warning box; Chinese full edition, §4).
🛠 GPU veteran: “Does KDA always have the same head count as MLA?” → No, a design choice: K3's config has two separate fields (linear_attn_config.num_heads and num_attention_heads) both happening to be 96, head_dim 128. Matching lets TP split heads by one number and kernels line up; the math doesn't need it. Aside: more KDA heads, more state blocks to move; more MLA heads just reuse the one summary more.

Section 4 · Many devices: TEP or DEP? (11′)Planned 11′ · with interaction 9.6′

🖥 Main screen for this segment: course page Section 4 · Many devices: TEP or DEP? · main figure TEP → DEP

🎯 What this segment must leave behind: TP copies each user's notes “TP degree ÷ KV heads” times; DEP doesn't, winning on shorter steps; large-scale EP fills the expert bus.

Last section made the notes thinner. But a model as big as DeepSeek-V3 doesn't fit on one device; it takes several.
Experts, everyone splits across devices. The fight is over attention: do devices team up on one batch of users, or each take its own?

🖥 Screen: stop on the “Reading the names” box.

First, the names. TEP and DEP both end in EP, the experts: same on both sides. Only the first letter changes. T is TP, split by heads. D is data parallel: in attention each device runs its own users; at the expert stop, all pool together.
Next section's DCP and PCP also have three letters, but there the first letter is a phase, decode or prefill: a different scheme.

🖥 Screen: scroll to “How many notebook copies?”, stop on TEP vs DEP in attention.

Recall: each angle you look from is a query head. Notes come in sets; one set is a KV head. Typical GQA keeps 8; MLA keeps one, so 1 KV head.
TP splits by heads: each device takes a few heads and their notes. More KV heads than devices? Each gets its own. Fewer? Devices share a head; its notes are copied onto each.

How many copies? One division: TP degree over KV heads. DeepSeek-style models: one KV head; at 8-way TP all 8 devices share it, so each user's notes are copied 8 times.
Copies don't just take space: every step, all 8 devices read the same notebook end to end, 8 times the reading.

💬 Interaction: ask “Qwen3.5-397B's full-attention layers: 32 query heads, 2 KV heads, also 8-way TP. How many copies of each user's notes?” Wait five seconds. Expected: four (8 ÷ 2). Add: 8 KV heads, one per device: no copies.

DEP is different: each device carries its own users and keeps their notes once. Two axes: is there enough room, and is each step fast?

🖥 Screen: scroll to “Capacity vs. speed: empty seats earn nothing”, stop on TEP → DEP; cover the bottom half.

Room first. TEP4 puts one batch on four devices: one original, three copies. DEP gives each device its own batch: per device, room for up to 4× the users.
⭐ But 4× is a ceiling: if too few users show up, extra seats sit empty and earn nothing.

Do you really get that 4×? Here's a case we cracked ourselves.
16 GB300 machines, V4-Pro on SGLang. Prefill and decode run on separate machines; why is Section 6. Most do prefill; two form the decode group. Both sides run attention as DEP. Target: the official blog's 11,200 per GPU.

🖥 Screen: reveal the bottom half. Point at the topology bar, then the first row, 727.

First working run: only 6,900. Way short.
Clue one: each GPU wrote for just 727 users at once. Seats weren't full.
Why? V4-Pro has two notebooks per user (whole text, recent window), memory split by a ratio. It was off: one couldn't fit newcomers; the other was still largely empty.

💬 Interaction: ask “Even out the ratio: seats go from 727 to 886, about 20% more. How much did throughput rise?” Wait five seconds. Reveal (second row): up more than 50%, more than the seats: prefill keeps feeding, so freed seats fill at once.

Encouraged, we kept adding note memory. Remember Section 2's puzzle? Over 10% more seats, throughput up just 0.8%. That's this very step.
New culprit: seats were plenty; more users slowed each step, cancelling them out.

Case closed: seats buy throughput only if filled and steps don't slow down. We ended at 95.6% of the official target.
Widening the decode group made it worse; the trouble was prefill. Section 6.

The other axis, step speed, has one corroboration, vLLM: decode from TP4 to dep8 cut the time between tokens to about a quarter; over all GPUs, per-GPU throughput a bit more than doubled. The denominator matters; Section 9 does the math.

There's a cost, too. DEP isn't just more buses. Expert cargo is still spread, not duplicated; but each device carries its own attention weights and state. That's where Kimi K3 tripped. Most of its layers keep no verbatim notes; each user has one fixed-size sticky note, rewritten as it goes. Under DEP every device must hold all heads' sticky notes for everyone. DEP8 ran out of memory at load and fell back to TP. Next section picks that up.

🖥 Screen: scroll to “Large-scale EP: regather the users”, play Two lanes, same columns. Point at EP8's 8 rows on top, then the ~128 rows below.

Remember Section 1's figure? Hundreds of experts per layer, each user picks only 8. Fill a bus at last, and each expert still sees just 8 rows.
The fix: spread experts past one machine: large-scale EP. DeepSeek's live decode uses 144 GPUs, two experts each, plus extra copies of hot ones to even the load.

Read it column by column. Spreading buys three things.
The bus fills: users per expert, 8 → over 120.
Less reading: expert weights read per device per layer drop to a sixteenth.
Memory frees up: experts take just over 5 GB, the rest goes to notes.

🖥 Screen: scroll to “Why it works: waits, not volume”, stop on Large-scale EP leans on the network.

Won't the network choke? Only users travel; notes and experts stay. Per token, a step's round trips total about 10 MB at most, over 10 times less than reading its notes.
Not volume, but waiting. 58 expert layers, a round trip each: one token waits 116 times.

Unhidden, waiting eats everything saved on reading weights. So split riders in two and alternate: half compute, half travel. Too few users? Don't: it's slower.
DeepSeek did it on plain 8-GPU boxes over InfiniBand, GPUs sending data themselves.

🖥 Screen: scroll to “Why big domains like NVL72 suit inference”, stop on Whole rack as one NVLink domain. Point at the 1× dashed line; at “it's not all large-scale EP”, the yellow box below.

NVL72 makes it easier: a whole rack in one NVLink domain, so experts spread over dozens of GPUs, all in-rack. SGLang ran DeepSeek on it: per-GPU decode at nearly three times an H100 large-scale EP deployment; with lower precision, nearly five times.
But as the yellow box says, that mixes in memory, precision and kernels; it's not all large-scale EP. All you can say: a big NVLink domain gives large-scale EP ample headroom.

So is TP useless? DEP has no edge in two cases: a KV head for every device, no copies anyway; or few users: notes aren't crowded anyway. DEP is built for crowds.

🖥 Screen: stop on the green “Takeaway” box.

The takeaway: DEP doesn't copy the notes, but only filled seats buy throughput; large-scale EP fills the expert bus.
But a model that falls back to TP still has its notes copied, and at long context they stop fitting. Next: long context, and how DCP and PCP differ.

⚠️ Where this segment drifts⚠️ Framing: K3's “sticky note” = KDA linear attention, one fixed-size state per request, in 69 of its 93 layers (Chinese full edition, §6). Under DEP each rank holds all heads' state pool; both loads ran out of memory. TP8 splits it by heads. “Sticky note” recurs in Sections 6 and 9: set it up. “Separate machines” = PD disaggregation: one line, don't explain why here.
⚠️ Main data: SGLang, GB300, 14 prefill nodes at dep4 each + decode dep8, 4096 input, 14 streams × concurrency 1050 (runbook :851–858). 727 → 886 gains 54%, yet the self-reported peak only goes 11,851 → 12,063. Don't say “20% more seats, 50% more compute utilization”; quote only the end-to-end gain.
⚠️ Spoken rounding: “6,900” = 6,887; “up more than 50%” = 10,614 (+54%); “still largely empty” = full-text pool 93%, sliding-window pool 63%; “a sixteenth” = 1.41 GB → 88 MB (our derivation; same 16× as 8 → 128 uses per byte). 10,704 tok/s = end-to-end per decode GPU at 95.6%; 11,200's denominator is unpublished: per decode GPU is our reading (page says so), not official. Section 2's 13% → 0.8% = this case's 886 → 999 step.
⚠️ Speed: our SGLang setup was never compared with TP. vLLM, same concurrency: time between tokens 46.8 → 11.8 ms (“about a quarter”); “a bit more than doubled” = 2.09×. Decode GPUs also went 4 → 8; denominator: Section 9.
⚠️ NVL72 multiples: decode was throttled to fill with requests, then released and measured; one H100 baseline had simulated MTP, GB200 didn't.
⚠️ 2.09× is corroboration only: V4-Pro, GB300, vLLM, concurrency 512, over all GPUs (incl. 12 prefill GPUs), our derivation; over decode GPUs, 1.31×. Never “DEP is always twice as fast”; don't set it against the SGLang runs.
⚠️ Don't argue capacity with “riders on board” (output × interval): it's an outcome of speed, not pool capacity. dep16 had 12 prefill nodes vs. dep8's 14, so the GPU ratio changed too: not “dep16 is worse than dep8”.
⚠️ EP144, two per GPU: 256 experts + 32 redundant = 288 copies. 8 → ~128 rows = 144 GPUs × 32 users × 8 experts ÷ 288, assuming 32 requests per GPU, even load. 1.41 GB → 88 MB and 81.7 → 5.1 GB use V3's FP8 expert size. All derived, not production measurements.
⚠️ Compare 128 only with 8, never with Section 1's 313: different hardware and precision; we never claim it clears that line.
⚠️ “The rest of the 512 aren't on the decode bus” is our derivation; the records don't say where those few hundred requests sat.
⚠️ Asked “how long is one wait?”: DeepEP at EP256, round trip ≈ 554 µs (dispatch 194 + combine 360); × 58 layers ≈ 32 ms, no-overlap upper bound (our derivation). Don't compare it with V4-Pro's 11.8 ms. Asked “how small is too small?”: SGLang at 32 tokens per GPU, micro-batch splitting is 27% slower. A group ties up to 128 GPUs together (one fails, all go offline).
⚠️ “About 10 MB”: our derived upper bound (172 KB per token × 58 layers). “Over 10 times less” is vs. 288 MB of notes per step at 4K context (Chinese full edition, §6.4); shorter context, smaller gap (state it if asked).
⚠️ NVL72 multiples are only vs. H100 large-scale EP deployments, mixing in memory, precision, kernels and PD disaggregation. 3.8× prefill, 4.8× decode; spoken “nearly three / nearly five times” = decode 2.7× (high precision) / 4.8× (rounded). Quote no multiple vs. B200.
⚠️ TEP/DEP are TensorRT-LLM's terms, ≠ Megatron's ETP/EDP. Details: Chinese full edition, §6.
⚠️ Two-batch overlap: one line only. EPLB and DP attention on TPU: not covered (Chinese full edition, §6).
❓ Likely questions (backup answers, not counted in the floor)
🛠 GPU veteran: “Which two pools are the two notebooks? Must the ratio be hand-tuned?” → Every layer has both. Full-length pool: the whole sequence compressed (CSA merges 4 into 1, HCA 128 into 1); sliding-window pool: the latest 128 tokens as-is. Ratio ≈ fixed window slots per user ÷ average length, so it tracks input length. October's SGLang sets it if none is passed; we hand-tuned it in July (Chinese full edition, §4.2). (Added after the 2026-10-22 rehearsal.)
🛠 GPU veteran: “Hundreds of GPUs in one instance. What if one dies?” → The whole group goes offline, a cost of large-scale EP. Prefill goes the other way (Chinese full edition, §6.4): 2–4 GPUs per instance, a smaller blast radius among the cited benefits. DeepSeek's production decode still spreads wide and lives with slow restarts: they take minutes, so keep spares, reroute requests; on restart, EP size resets to the healthy GPUs left. A retiring instance gets 1 minute's grace; unfinished requests move with their KV (§6.4). (Moved off the main line 2026-10-20.)
🛠 GPU veteran: “Seats rose only 20%. How did throughput rise over 50%?” → The records don't break down the cause. Two readings: decode's self-reported peak barely moved (11,851 → 12,063), so the gain is end to end; the fourteen prefill nodes kept feeding, so freed seats filled at once. Not “compute utilization rose 50%”; invent no causes.
🛠 GPU veteran: “With large-scale EP, do you still need 9,000-plus concurrency? 128 GPUs means 900?” → Global tokens are unchanged; EP spreads them. At the Chinese full edition's ~300 rows per expert on H100: 300 × 256 ÷ 8 ≈ 9,600 tokens per step; per GPU ≈ 9,600 ÷ GPU count. 32 GPUs × 8 experts: about 300 each; 128 GPUs × 2: about 75, not 900 (our derivation, no redundant experts). What's saved: expert weights read per GPU. 32 GPUs × ~300 happens to sit near Section 1's line, but on a different hardware basis: not “clearing the line too”.
💰 Budget owner: “A few premium users, very low concurrency, each needing speed. TEP or DEP?” → TEP (attention on TP) plus speculation. Few users: a step is nearly all weight reading. TP splits attention and dense weights by heads; each device reads its share of one step, pooling bandwidth for one batch. DEP gives each device full attention weights; a user's attention runs on one device, no faster for few users (our explanation). TP's copying bites only with many users: in the Chinese full edition's §6.2 TPU runs, DEP leads by over 60% at concurrency 512 and ties on the decode-heavy concurrency-64 line. TP's ceiling is the per-layer sum: small, latency-bound data, so doubling devices doesn't halve it. Stay inside the NVLink/ICI domain; heads must divide evenly by devices. Idle compute makes verifying guesses almost free (magnitude: Section 7's TensorRT-LLM backup). Speculation isn't just for low concurrency: this section's V4-Pro runs (GB300, vLLM) had DSpark on both sides (7 guesses per step), and the top measured throughput: DSpark + dep8 (concurrency 1,536, 65,132 tok/s); no DSpark-off control, so its share is unknown. Fixed-guess MTP backfires: V4-Flash, DEP8 + MTP on B200, slower at all ten concurrencies (Section 7). DSpark verifies less when busy, more when idle, so it stacks with DEP (our explanation). Tiered pools: premium, TP + speculation at a high price; bulk, DEP + large-scale EP (adaptive speculation like DSpark still on): the two ends of Section 2's curve.

Section 5 · Long context: DCP vs. PCP (10′)Planned 10′ · with interaction 8.1′

🖥 Main screen for this segment: course page Section 5 · Long context: DCP vs. PCP · main figure Who moves: notes or question?

🎯 What this segment must leave behind: split across devices, the smaller moves. PCP moves notes: earlier first token. DCP moves the question: fit, net win only at long context.

Picture an agent dropping a 128,000-token context on you at once, roughly a novella. Two things happen right away.
First, the first token is slow: the model must read the whole novella. Second, the bus is full. Last section K3 fell back to TP, so every user's notes are copied several times; at this length memory holds only a few.
One slow, one crowded; this section has a move for each. Underneath, one action: cut the novella by token into segments across devices. The only difference: who moves.

🖥 Screen: stop on the “Don't go by the letters” box.

The moves are PCP and DCP: P for prefill, D for decode, named for their usual phase. But the mechanism isn't tied to a phase; Meta's million-token inference paper names them by what moves: notes or question. So track only who moves.

🖥 Screen: scroll to “Who moves: notes or question?”, stop on Who moves. Untick “Autoplay”, click “⟲ Reset”, then “Start”. At the first pause, point to Dev 0–2's gray cells.

Why must anyone move? Recall attention: each new token carries a question, checks it against every earlier token's notes, scores them, takes a weighted average.
Each device stores only its segment of notes. Question here, notes there: nothing to check. Either the notes travel or the question does. The trick: move whichever is smaller.

Prefill first. Picture a group exam: one book's questions split among four people, a segment each, but any question may need notes from anywhere earlier.
Thousands of questions, thousands of note lines: same count. Compare thickness: a token's question has a copy per head, and V3 has 128 heads; its notes under MLA are one shared summary line. The question is dozens of times thicker, so you move the notes.

🖥 Screen: click “Next” three times. At each stop, point at Dev 0's dimmed, just-returned segment (passing, not checked), then the new green cells elsewhere.

Four people in a circle pass the notes around, a volume at a time. Each checks a volume as it arrives, the next already on its way: the moving hides behind the checking.
That's PCP: questions stay, notes circulate, several read at once, first token sooner. How much? Meta read a million tokens on over a hundred GPUs in 77 seconds.

💬 Interaction: before “Next” into DCP, ask “Decode: each step, one new token, one question. Behind it, over 120,000 note lines on eight devices. Which do you move?” Wait 5 seconds. Expected: move the one question.

Right, the question. Notes stay put, round-robin by token on eight devices; each step it's copied eight ways, each checks its volume, returns a result to combine. That's DCP.

🖥 Screen: click “Next”: the question flies into four devices (four drawn, read as eight); again: numerators and denominators combine at the end.

Combining has one catch: each device returns a numerator and a denominator. Don't divide yet; sum all, divide once. Averaging averages is wrong. PCP merges as each volume comes round the ring, DCP once at the end. Same trick.

🖥 Screen: scroll to “Same 8-way split, different payoffs” , stop on CP at inference. Point first at the eight equal top bars: one green, seven red.

What does DCP buy? Section 1: decode is movement-bound. MLA: one notes head per layer; TP8's eight devices each store a full copy, seven duplicates, reread every step.
DCP turns copies into volumes: each device stores and reads only an eighth. For one 128K V3 request, just over 8 GiB per device drops to just over 1 GiB.

The fare: a trip per layer, question out, numerators and denominators back.
K3 with one user: as the input grows, TP8 slows per token, while DCP8 barely moves. The note reading really is shared. But DCP8 starts about four milliseconds higher.

Where did the four milliseconds go? Per token, TP8's eight devices settle up over a hundred times, each tiny: it's all waiting. A switch, symm-mem: each device reserves an identical region and writes straight into the others', no handshake. One user: worth 35%.

But on K3, SGLang forces it off whenever DCP is on: it makes decode's captured CUDA Graph miscompute. DCP's four extra milliseconds are mostly that lost switch. So on K3, DCP mainly buys fit.

🖥 Screen: scroll to “When DCP wins and when it loses”; keep DCP wins only at long context hidden.

So when to turn it on? Ask whether the notes fit: total notes = length × users.

💬 Interaction: ask “Customer-service bot, 4K inputs, eight users: DCP8 or not? A 128K agent chat?” Pause, then reveal. Expected: 4K, no: it fits; fare for nothing. 128K, yes: without it, long queues.

128K peak first: K3, eight users at once. TP8 can't fit them, some queue; first token: half a minute. DCP8 fits all: just over two seconds, throughput more than triples.
Far left, 4K: notes fit anyway, DCP adds only the fare. All four points below the line.

Last point: DCP covers only attention. It sits on attention's TP, same devices, just turning copies into volumes. Experts stay as they were, EP is fine: the usual setup is TEP plus DCP.
Not with DEP: DCP borrows its degree from attention's TP group; DEP's is one device, nothing to borrow. On K3, DEP won't start, attention falls back to TP: hence DCP works.

🖥 Screen: stop on the green “Takeaway” box.

Takeaway: move whichever is smaller. Prefill moves notes (PCP): earlier first token; decode, the question (DCP): fit, net win only with long inputs, many users; short ones even lose.
But prefill and decode still fight over the same devices. Next section splits them onto two sets: an extra trip for the notes, so how does it serve more users?

⚠️ Where this segment drifts⚠️ Basis for 108 → 365 tok/s (+237%; always say “more than triples”, our arithmetic): K3, GB300, SGLang, TP8 across 2 nodes, 128K input / 1K output, concurrency 8, TP8 best-tuned. Not “DCP is three times faster”: at 4K all four concurrencies lose; concurrency 1 loses 26% (exactly the symm-mem loss).
⚠️ “Move whichever is smaller” is our intuition. Meta picks pass-KV/pass-Q by whether communication can hide behind attention compute: Q only below ~3.25% new tokens. For veterans.
⚠️ Every DCP device needs this step's Q. Gathering questions, exchanging denominators, merging results: all extra traffic (3 collectives/layer, vLLM default backend).
⚠️ 8.58 → 1.07 GiB (spoken “just over 8 → just over 1 GiB”) is paper math for one 128K V3 request (70,272 bytes per token). “TP8 can't fit them”: TP8 default, no speculation, holds only ~790K tokens (measured); 128K × 8 = over a million is our derivation.
⚠️ Meta's 77 seconds: 128 H100s (spoken “over a hundred GPUs”), Llama 3 405B, 1M-token prefill. Page also: “each doubling from 1 to 8 nodes about halves it; 8 → 16 cuts only ~35%”: unspoken unless asked “does more splitting keep helping?” Never “linear”.
⚠️ Not covered: linear-attention state relay, the HiCache conflict, Ring vs. Ulysses, §7.3's side-by-side memory math (Chinese full edition, §7).
⚠️ “Half a minute, just over two seconds” = median TTFT, 128K, concurrency 8: TP8 best tuned 32.1 s (default 39.3 s) vs. DCP8 2.3 s, queueing included. Officially DCP8 also lifts group KV capacity from 1.5M to 12.2M tokens.
⚠️ DCP doesn't split K3's KDA state pool, only the MLA layers' notes.
⚠️ “Named by phase”: DCP/PCP are framework names (SGLang flag --dcp-size). Meta's paper (arXiv 2411.01783) says pass-KV / pass-Q: full prefill uses pass-KV; decode and high-cache-hit partial prefill use pass-Q. NVIDIA's Helix paper calls the latter KV parallelism (KVP).
⚠️ Behind “as input grows, barely moves, starts higher”: K3, SGLang, concurrency 1. TP8 per token 4K 11.1 ms → 128K 12.0 ms; DCP8 15.2 → 15.3 ms. At 4K, TP8 with symm-mem off is 15.1 ms, so DCP8's ~4 ms higher start is mostly symm-mem forced off (our inference from two rows). The official large-scale preset: DCP8 gives “+72% concurrency ceiling, same engine throughput, ITL ×1.8”. Not “DCP makes each token faster”.
⚠️ Rewritten after 10-22 rehearsal: symm-mem (symmetric memory) = each device reserves a same-shaped region the others write into directly, saving the small-message all-reduce handshake. K3 at bs=1: +35% (87.8 vs. 64.9 tok/s, runbook §11.2). “Over a hundred times” = 93 layers × at least 2 all-reduces ≈ 186 (estimate, runbook §11.0.4). Off under DCP is SGLang's model setting for K3 (“due to decode CUDA graph correctness issues”), not a DCP limit: never “DCP can't use it at all”. “Mostly that lost switch”: our two-row inference, not “all of it”.
⚠️ “Dozens of times thicker”: V3's per-token question is 128 heads × 192 dims = 24,576; an MLA notes line is 512 + 64 = 576: ~43× (our derivation). In the absorbed form the question is 128 × 576, ~128×. q_lora_rank 1536 isn't the question compared, just the projection bottleneck.
⚠️ “Throughput more than triples”: now in the 128K passage, not the takeaway.
❓ Likely questions (backup answers, not counted in the floor)
🧑‍🎓 Newcomer: “Why not just average the averages?” → Open “The arithmetic” fold: Dev 0 has two tokens, 24 ÷ 3 = 8; Dev 1 has one, 20 ÷ 1 = 20. Averaging: 14, wrong: Dev 1's single token gets half the vote. Sum numerators and denominators: 44 ÷ 4 = 11 is right. The denominator is those tokens' weight sum, not their count. (Moved to fold/backup after 10-22 rehearsal.)
🛠 GPU veteran: “Doesn't MLA compress the question to 1536 dims first? Why not send that and expand on arrival?” → Possible, but not worth it. Expanding needs the up-projection matrix; under TP each holds only its heads' eighth. Expanding all heads locally, each device stores and reads the whole matrix every step: ~40 million extra parameters per layer, to spare a bit over 100 KB of question. Decode is read-bound; below ~300 tokens on the bus, sending the question is cheaper (our derivation, V3 shapes). And all eight would redo it. SGLang: each device computes its heads' question, then all-gathers within DCP. Even against 1536, notes at 576 are smaller, so PCP's “move the notes” stands. (Added after 10-22 rehearsal.)
🛠 GPU veteran: “How much does 4K lose? Same length, does user count matter?” → At 4K all four concurrencies lose, up to −26%. At 8K, concurrency 8 loses 11%, concurrency 32 gains 24%. Judge by total notes, not length (page bullets under “When DCP wins and when it loses”; moved from script).
🛠 GPU veteran: “DCP's full topology?” → TEP + DCP: attention splits by heads (TP), copies become volumes; experts still split by EP (moved from the script).
🛠 GPU veteran: “At what length does attention out-compute the experts?” → Whole Qwen3.5-397B: ~34K decode, ~68K prefill (our derivation; page bullet “Ignore the compute crossover”). One full-attention layer alone: 8.4K / 16.9K. DCP hinges not on this but on whether the notes fit.
🛠 GPU veteran: “Does PCP throw an error when it goes wrong?” → Often not. A device missing another segment's KV still reads fluently, just unlike unsplit output; less compute runs faster: suspiciously fast? Check output first. Linear-attention layers carry a rolling per-token state; segments starting from zero go wrong; relay the state. (Off main line since 10-20 rehearsal.)
🛠 GPU veteran: “DCP also runs attention on eight devices in parallel. Isn't that faster? Why only space?” → Right, both. MLA has one notes head, so under TP8 every device reads the whole notebook each step; under DCP8 each reads an eighth, sharing the read time. Cost: an extra communication round per layer (gather questions, exchange denominators, merge results), and on K3 symm-mem goes off. Measured at concurrency 1: DCP8 15.3 ms per token vs. TP8 12.0 ms, but from 4K to 128K DCP8 rises 0.1 ms, TP8 0.9 ms. The larger reading's share (longer context, more attention layers), the more speedup shows. K3 has just 24 MLA layers, so its net gain is mainly fit.
🧑‍🎓 Newcomer: “Is DCP only for decode and PCP only for prefill?” → No; the names follow the commonest phase. The test: which side is smaller. Meta passes the question even in prefill when the cached prefix is long and new input short.
🧑‍🎓 Newcomer: “So does PCP split the KV too?” → Yes. Both split by token, each device storing its segment's KV. In prefill questions are long too, split along; decode's lone question can't. Only difference: KV or Q travels (Meta's pass-KV / pass-Q: one split, two ways to pass). Detail: Meta cuts the sequence into 2N chunks, paired end to end to balance the causal triangle; DCP places tokens round-robin.
🛠 GPU veteran: “Both are eight-way. Which one adds devices?” → Both are. “Adding devices” is deployment habit, not mechanism. SGLang's DCP8 sits on the existing TP8 group (TP8+DCP8), same eight devices; Meta's CP stacks another dimension over TP8 machines: 128 GPUs = 16 × 8. On stage, say what each buys.
🛠 GPU veteran: “How many ways can DCP go?” → At most TP degree ÷ KV heads. DEP's TP is 1: no stacking.
🛠 GPU veteran: “Under DEP, a long request's notes all sit on one device. Isn't that what should be split?” → Not stacking is today's practice. The Chinese full edition, §7.3, gives “degree at most TP ÷ KV heads”; SGLang's DCP borrows the attention TP group (TP8+DCP8 is still those eight), so at DEP's TP of 1 it can't be enabled. Meta's CP is a separate dimension stacked across machines (§7.2 compares two CP2 machines with TP16, as we read it). §7 has no CP-on-DEP measurements: claim neither way. (The old spoken “nothing to split” overstated it.)
🛠 GPU veteran: “With DCP on, what EP do the experts use?” → TEP. DCP sits on attention's TP, same devices; experts split by EP as before (TEP16 + DCP16 in the Chinese full edition's §7.3 table). DEP's long-context problem isn't copying but a full set of attention weights and state per device, which DCP can't fix.
🛠 GPU veteran: “What are the three extra collectives per layer?” → vLLM's default backend (dcp_comm_backend="ag_rs"): ① all-gather Q: weights split by heads, so each device projects only its heads' Q, yet all heads must check its local KV segment; ② all-gather the LSE: the kernel divides locally but keeps the log-sum-exp (one number per user per head, very light), from which each device computes its share and rescales its local result; ③ reduce-scatter the output: summed, it's the full result, split back by heads for the head-split output projection. “Don't divide yet” is the classroom version; the code does “divide, keep the denominator”. The cost is waiting, not volume: V3's 61 MLA layers × 3 = 183 extra collectives per token, K3's 24 × 3 = 72 (our arithmetic). Two ready savings: the a2a backend merges ② and ③ into one all-to-all (source comment: MLA 3 → 2 per layer); dcp_q_replicate keeps a full question projection per device and drops ①, at the cost of duplicate projection compute. No public measurements for either: quote no speedups. (Source: vLLM main, vllm/config/parallel.py, vllm/v1/attention/ops/dcp.py, 2026-10-17.)
🛠 GPU veteran: “Doesn't PCP merge too? Why “no combine”?” → No cross-device combine: Ring merges locally on arrival (“don't divide yet”), DCP at the end. Same trick.
🛠 GPU veteran: “93% parallel efficiency: doesn't that contradict 8 → 16 machines cutting only 35%?” → The Chinese full edition has both (1M tokens in 77 s, 93% efficiency; 128K at 5.85 s on 8 machines, 3.8 s on 16); the source doesn't say at what length or device counts 93% holds. Don't force an answer.
🛠 GPU veteran: “Can DCP run with HiCache or speculation?” → With HiCache L3 (remote storage), or L1 + L2 plus speculation, fall back to pure TP.
🛠 GPU veteran: “Under PD disaggregation, is DCP better for decode?” → Officially TP8 is a bit higher: 1P1D decode TP8 2,808 tok/s vs. 2P2D decode DCP8 2,633. Denominator and input length unstated: don't compare with ours.
🛠 GPU veteran: “Why does 4K at concurrency 1 lose exactly 26%?” → Exactly the symm-mem loss: on K3, SGLang forces it off under DCP (a model setting, not a DCP limit).
🧑‍🎓 Newcomer: “Why store notes round-robin by token, not first/second half?” → New tokens keep arriving; round-robin keeps devices evenly loaded (our explanation).

Section 6 · PD disaggregation: one more KV trip, yet more users? (9′)Planned 9′ · with interaction 8.5′

🖥 Main screen for this segment: course page Section 6 · PD disaggregation: one more KV trip, yet more users? · main figure Once split, 3 devices do the work of 7 colocated

🎯 What this segment must leave behind: Splitting PD doesn't make decode haul less; it stops interruptions. Each side batches to its limit, so an extra KV trip serves more users.

Section 5 left a gap: prefill and decode still share devices. The fuller each device, the more users one long prompt holds up.
The fix: split, each job on its own machines. Section 0 promised a trick that leaves the trip alone; this is it. Cost: one more KV trip per request. So how does that win more users?

🖥 Screen: scroll to “Each job drags the other”, stop on Top, red box: a whole batch waits for a loop. Red box, then purple arrow.

Each cell is one step. For the first four, A, B and C each emit a token. Then a long prompt arrives: orange, five cells.
Top, colocated: the prompt hogs the machine; red box, five cells, no token. Bottom, split: prefill has its own machine, decode never skips. Price: the purple arrow, a KV transfer.

🖥 Screen: the “One kitchen: prep and service” box.

Back to the kitchen from Topic 5. Prefill is prep: the whole prompt at once. Decode is service: plates go out one at a time, and it pays to carry several at once.
A big party orders, prep ties up the stove for several minutes, and every table waits. No restaurant makes one stove do both. Split, and you have two stoves; the prepped ingredients get handed over once: the KV, the notebook from earlier.

What about Section 2's trick, slipping a long prompt in as chunks? That just turns one big order into a few small ones; the stove still has two jobs fighting over it.
Split, service hauls no less; it's just never interrupted, so it dares carry a full tray.

🖥 Screen: scroll to “Why more users: each side at its own limit”, stop on Once split, 3 devices do the work of 7 colocated. Point at the top seven devices, then the bottom three.

DistServe's example. One device doing both while holding latency takes just 1.6 requests a second. Single-job devices, bottom row, are much faster.
By workload, mixing should manage over three (our derivation). Why only 1.6?

The catch: “holding latency” is two targets. The first token can't be too slow; tokens can't stall in between. Count only those meeting both: goodput. Cold food doesn't count.
One stove must guard the worst case on both ends: a big order arrives, service stops. Guarding both means fewer orders.

💬 Interaction: at the bottom row, ask: “Prefill devices take 5.6 requests a second, decode devices 10. For 10 a second, how few devices, split how?” Expected: three, two prefill and one decode.

Right. Both sides fill up at about the same time. Colocated, seven devices handle just over 11. Now three handle 10. Requests per device more than double.
The extra trip doesn't matter; both sides running at their own limits does.

🖥 Screen: scroll to “What the extra trip costs”, stop on One KV transfer is just 5–10% of a prefill. Point at the long top bar, then the thin three-color one (zoomed 10× below).

Costly handoff? Our Qwen3-Coder on TPU: an 8K-token prompt preps in one to two seconds; the handoff, by bandwidth estimate, about 100 ms: a rounding error.
Why so small? TP talks every layer and needs the fastest links. The handoff is once per request; a slower network is fine.

🖥 Screen: Thinner KV lets PD span datacenters: orange bars, then green.

Cheap per trip isn't cheap in total: a prep stove hands over many batches a second. On one dedicated link, a standard model's thick KV feeds only one or two prefill machines. Hybrid models have layers remembering only nearby tokens, or a sticky note; their thin KV lets it feed twelve to twenty-one.
Only thin KV lets prep move data centers; thick KV is best kept inside the NVLink domain.

🖥 Screen: scroll to “After the split: sharding, ratios, when not to split”, stop on “Sharding: many narrow prefills, one wide decode”. Prefill item, then decode.

A bonus: each stove gets its own setup. Our V4-Pro deployment: both sides run DEP, differing in width and replicas.
Prefill: fourteen small groups of four devices, one machine each, never talking. Decode: one big group, eight devices on two machines.

Prefill is compute-bound: one long prompt saturates it alone. Small groups mean little cross-machine traffic; to serve more, add replicas.
Decode is movement-bound: only a big batch makes each expert-weight read pay. EP goes wide, each expert gathering more users; attention runs DP, storing each notebook once. Like many small stoves prepping alone, feeding one big serving line.

Section 4 said a wider decode group can backfire; here's where. At sixteen devices step time fell almost half, but two fewer prep stoves couldn't keep decode fed: output per decode device fell about 40%.
Wide needs feeding, so the stove count must be worked out.

🖥 Screen: “Compute a starting ratio, then sweep.”

A start is computable: decode's finished requests a second, times input length, is the tokens a second prefill must chew. Divide by one prefill instance's rate.
But for one V4-Pro, the formula, official recipe and our sweep give three different numbers. A start is just a start: sweep until output per decode device stops rising.

🖥 Screen: stop on An extra prefill machine buys a faster first token. Point at “Per GPU” (left), then the TTFT bars (right).

One more stove? Per GPU output drops about 30%; the slowest first-token wait almost halves.
Why no more total output? A request takes over a minute end to end; the first token's sub-second saving barely shows. Another stove buys the first token, not throughput.

Sometimes don't split: few users, no collision, a wasted handoff. At concurrency 1 on TPU, the first token got nearly three times slower, for only about 12% more output.
Don't copy someone else's ratio. Measure your own load.

🖥 Screen: the green “Takeaway” box.

Takeaway: decode hauls no less, just uninterrupted, and each stove runs at its limit. The gain? Mooncake, serving Kimi live, handles more than twice the requests of prior systems.
But split or not, decode still emits one token per step. Next: can one trip yield several?

⚠️ Where this segment drifts⚠️ Goodput: one plain sentence on stage. Full definition (full edition §2): max per-device request rate with a set share (e.g. 90%) meeting both TTFT and time-between-tokens targets.
⚠️ “More than twice”: Mooncake (FAST '25) vs. prior systems, +115% on A800, +107% on H800. DistServe's 7.4× is “up to”: per-device request rate, ≥90% on target, vs. vLLM and DeepSpeed-MII. Splitwise's 2.35× is throughput at equal cost and power. Different bases: don't compare.
⚠️ 1.6 / 5.6 / 10: DistServe's OPT-13B, single-A100 example (the Topic 5 figure). “7 devices, ~11.2”, “3 devices, 10”: computed from it.
⚠️ The ~100 ms trip is a bandwidth estimate, not a timing: Qwen3-Coder-480B (thicker KV than V3), 8K prompt, KV ~1.04 GB, HBM → host memory → 100 Gb/s network → peer HBM. “Rounding error” = 5–10% of the 1–2 s prefill; recount for much bigger KV. “Over five seconds” = this 1.04 GB at V4-Pro's measured 200 MB/s (our conversion), not V4-Pro's own KV.
⚠️ “Why only 1.6”: two constraints, TTFT and time between tokens. Source: DistServe abstract (full edition §9.1, §16 ledger). “Over three” = 1 ÷ (1/5.6 + 1/10) ≈ 3.6, our derivation, just to flag the oddity.
⚠️ Ratio numbers (on stage only “three different numbers”). Formula: 8 devices × 11,200 ÷ 1,024 ≈ 87.5 req/s; × 8,192 ≈ 717K tok/s; ÷ 72.8K per prefill ≈ 9.8 (full edition §9.1). Official recipe: 8. Our SGLang runbook's early sweep: 14; 14 → 16 added only 2% per decode device.
⚠️ PrfaaS figure: KV egress/s from one 8×H200 at 32K input. “Room for N” = ⌊100 ÷ per-machine egress⌋ (full-edition derivation).
⚠️ “Over a minute, under a second”: mean end-to-end ~65 s; mean TTFT 2.40 → 1.73 s (−0.67 s).
⚠️ 2P1D figure: DeepSeek-V3, B200, SGLang + NIXL, concurrency 128; TTFT client-side, queueing included; per GPU −31%, P99 TTFT −48%.
⚠️ “Nearly three times slower, about 12% more” = 95 → 281 ms (≈2.96×), output +12%: Qwen3-Coder, two v7x-8s, 1K in / 1K out, concurrency 1, medians, one measurement. Whether 12% is throughput or per-token speed, and where the extra 186 ms goes, the full edition doesn't say: don't invent.
⚠️ Not covered (full edition §8–9): three software layers; two transfer libraries with opposite results on two engines; hybrid-model tail; ratio-formula estimate; AFD / EPD.
⚠️ “Sharding”: 14 × dep4 + 1 × dep8 on 16 GB300 machines is our own SGLang deployment (§6.1); dep16: TPOT 60 → 33 ms, per decode device 9,032 → 5,270 (§6.1). “Compute-bound, so small groups, many replicas” and “movement-bound, so wide EP” are our reasoning from Sections 1 and 4, not SGLang's words. SGLang GB200 evidence: §8.5. TEP + DCP and prefill PCP are inferred, unused by us: say so.
❓ Likely questions (backup answers, not counted in the floor)
🛠 GPU veteran: “Does a misconfigured network error out? What to check before launch?” → No error, just slow first tokens. vLLM, V4-Pro on GB300: a missing config routed KV via host memory, ~200 MB/s, over 5 s per trip; fixed, two orders of magnitude faster. Before launch, measure KV bandwidth, diff split vs. unsplit token by token at temperature 0: early divergence is most likely a bug; after a few hundred tokens, maybe floating-point noise. (Off stage since the 2026-10-20 rehearsal.)
🧑‍🎓 Newcomer: “Are weights or KV transferred?” → KV. Weights stay resident; each request ships only its own KV.
🛠 GPU veteran: “NIXL or Mooncake?” → Ask instead which path works on this engine and network. Common ground: inside one NVLink domain, KV stays in it (full edition §8.2).
🛠 GPU veteran: “Isn't 100 Gb/s conservative?” → Yes. GPU NICs: A4X 400 Gb/s, ~21 ms; GB300 800 Gb/s, ~10 ms (full-edition derivation).
🛠 GPU veteran: “Do PD per-device numbers count prefill devices?” → Many PD reports divide by decode devices only; include prefill and the conclusion can flip (Section 9, question two).
🛠 GPU veteran: “Anything special about PD for hybrid models?” → Decode can't recompute the tail (the recurrent state would absorb those tokens twice), so transfer everything. Qwen3.5 moves ~349 MB per request.
🛠 GPU veteran: “What if a KV pull fails?” → vLLM policy “fail”: one transient rejection kills the engine. “Recompute”: stable, ~17% tax (12 prefills: per device 3,187 → 2,645).

Section 7 · Speculative decoding: many tokens per trip? (11′)Planned 11′ · with interaction 8.1′

🖥 Main screen for this segment: course page Section 7 · Speculative decoding: many tokens per trip? · main figure Small guesses 4, big checks them in one trip

🎯 What this segment must leave behind: one weight trip yields every right guess; more users, no freebies. Drafts keep nearing the big model; EAGLE vs. MTP: who trains it.

Section 6 left a gap: split or not, decode emits one token per step, and every token hauls the weights through once.
If each user needs fast tokens, the batch stays small, and reasoning models write thousands of tokens per answer. Can one trip yield several?

🖥 Screen: scroll to “Guess, then verify: hits free, misses cheap”, stop on Small guesses 4, big checks them in one trip. Untick “Autoplay”, click “⟲ Reset”, then “Start”. Speak after the first auto-pause.

We've written “So if it is.” The small model guesses four tokens: four small purple cells, fast. The big model: one weight trip, one big gray cell, all those positions at once.
Left to right: two match, the third doesn't. That cell becomes the big model's token; the rest are dropped. One trip, three tokens.

🖥 Screen: click “Next ▶” to the second pause; then re-tick “Autoplay”.

Step two, all four right; the big model's last position comes free. Five out.

💬 Interaction: ask: “If the small model gets all four wrong, how many tokens does this trip yield?” Expected: one, same as without speculation.

Right. Until the first miss: each right guess, plus the big model's. At worst, one.

🖥 Screen: the “Apprentice drafts, master grades” box, then Cost of verifying more; point at the two time bars on the right.

Picture the small model as an apprentice drafting, the big model a master grading in one pass. The draft is ready-made input, so the master reads it at once, like a prompt.
With few users, step time is all weight hauling: a dense model grading five tokens hauls the weights once, as for writing one. The bus is running anyway.
MoE isn't that cheap: each position picks its own experts, so verifying more reads more experts. K3 is MoE: with speculation on, each trip got over 40% slower.

Random sampling: what if apprentice and master draw different tokens? Compare confidence, not tokens: if the master is at least as confident, keep it; if less, keep proportionally.
The math guarantees the big model's own output: speed changes, the answer doesn't.

The payoff is a division: tokens per step ÷ step slowdown. K3, eight devices, one user: almost seven tokens per trip. Take off the 40-plus percent slower trip: each token 4.8 times faster.
Measured with randomness off, always picking the top token: optimistic. Turn randomness on, you keep by confidence ratio: keep fewer. Official numbers, normal sampling: a notch lower.

🖥 Screen: scroll to “More users, smaller gain”, stop on Speculation's gain shrinks. Leftmost green bar to the 4th (64 users); skip the far-right purple (long context, Q&A).

💬 Interaction: ask: “Over four times faster for one user; 64 users at once, how much is left?” Wait five seconds; take 2–3 guesses. Expected: just over 1.7×; still a gain, a smaller one.

Smaller: free tokens need idle compute.
64 users, each verifying seven or eight positions: a trip is four or five hundred rows. Compute isn't idle; every draft token costs real work. The draft also takes memory: the notebook pool shrinks almost a fifth.

Worse: DEP for big batches, eight B200s, built-in draft: slower at all ten concurrencies.
So enable it first for the few-users, fast-tokens tier; big batches: measure first.

🖥 Screen: scroll to “Drafts: ever nearer the big model”, stop on Where drafts come from; sweep the timeline left to right.

Where do drafts come from? One story: ever nearer the big model.
First, a separate small model. Then it grew onto the big model: EAGLE peeks at the vector the big model holds just before emitting, its “what I'm about to say,” to guess.
DeepSeek then trained it into the main model: MTP. EAGLE-3 (2025) is in SGLang and vLLM.

The newest step isn't on the figure: draft a whole block in one pass. Fast, but no slot sees what the one before it picked.
DeepSeek used to run one-step MTP live; DSpark, from this line, replaced it: two cuts.

🖥 Screen: skip the collapsed DSpark paper data (don't open it); stop on DSpark cut one, let it autoplay; then DSpark cut two.

Cut one fixes the draft. After “Yes,” comes “of course” or “no problem.” Slot 2 can't see if slot 1 picked “of” or “no,” so it bets half on each, and you may get “of problem.” A very light head glancing left at the previous slot ends the mix-ups, at almost no extra time.
Cut two fixes verification. Each slot gets a confidence: idle, verify more; busy, fewer; low-confidence tokens take no seat. With few users, each gets four to six verified.

🖥 Screen: scroll to “EAGLE and MTP: same shape, different trainer”, stop on Inside EAGLE and MTP. Legend: orange, blue; chaining: gray/green, lower right.

Common misconception: EAGLE isn't training-free; it needs little training.
Big model frozen, input and output layers borrowed; only a small orange middle is trained. For 70B, it's under 1B parameters: one or two days on one A100 server.

MTP is nearly the same structure; the difference is who trains it. The vendor trains MTP in pretraining and ships it; anyone can attach and train EAGLE after release.
Can that one MTP layer chain guesses? It was trained to guess one step; forced to chain, the third position is almost always wrong. Fine-tuned for chaining, over 30% still gets accepted there. Guessing several steps takes a draft trained to chain.

🖥 Screen: the green “Takeaway” box.

Takeaway: one weight trip yields every right guess; at worst one token. K3, one user: each token 4.8 times faster; 64 users: 1.7 times.
More users, it helps less; every trip stays as heavy. Next lever: lighten the load.

⚠️ Where this segment drifts⚠️ Don't mix up 4.81× and 4.22×: 4.81×: time between tokens (K3, GB300, 8 devices, one user); 4.22×: end-to-end throughput, first token included; 1.74× at 64 users: also throughput. The ~16 ms trip is our back-calculation (2.31 × 6.96); on stage, “under 50% slower.”
⚠️ B200 set (V4-Flash, vLLM, DEP8 + MTP, −3% to −37%): a high-throughput EP + DP setup; doesn't carry to TP-only low-latency setups.
⚠️ Temperature set: V4-Pro, GB300, greedy DSpark draft, 4K in / 1K out, concurrency 256: acceptance 1.16% vs. 34%, 944 vs. 2,911 tok/s (3.1×). 1.16% below theory: our derivation (Q&A).
⚠️ Draft-lineage speedups use different bases: don't compare. EAGLE 2.7–3.5×, EAGLE-3 up to 6.5× (1.38× throughput at batch 64): each vs. no speculation, on its team's own models.
⚠️ EAGLE training: 0.99B trainable params (70B), 68K ShareGPT chats, 1–2 days, one A100 server (EAGLE paper). MiMo-7B chained 70% / 10% / 2% → FastMTP 81% / 56% / 36%, 2.03× avg.
⚠️ Trip 11.1 → 16.1 ms (1.45×): K3, GB300, bs=1; full edition: not all from extra expert reads. Left figure's V3 “Verifying 8 = 4.4× verifying 1”: our derivation, uniform routing, overestimate.
⚠️ 6.96 and 4.81×: SGLang, temperature 0, random dataset, block 7 (optimistic). Distance from the theoretical max unstated: never “right at the ceiling.” 4.73 / 2.61: vLLM K3 runbook reference acceptance lengths at the official temperature 1.0, top-p 0.95. Different engines: don't divide.
⚠️ Cut from the script: “same order as Section 1's compute line” (different hardware and model); the full edition says only “the workbench fills up first.” “64 × seven or eight positions ≈ four or five hundred rows”: our derivation, magnitude only (draft block 7, plus the big model's position = 8; 64 × 7 = 448, 64 × 8 = 512). Section 1's ~313–343-row line is dense weights on v7; K3 is MoE on GB300 (far fewer rows per expert). Never “already past the line.”
⚠️ Rejection resampling (Newcomer Q&A): standard speculative sampling; go no deeper.
⚠️ KV pool 787,504 → 637,504 tokens (−19.0%, “almost a fifth”), K3 with DSpark. The B200 set is 12% slower even at concurrency 1; no reason given.
⚠️ DSpark: DeepSeek, 2026-07, DFlash backbone plus a small sequential head; survival probability and load set how many to verify. Replaced MTP-1 (2 tokens verified each time) two weeks after the V4 preview went live. V4-Flash under concurrency 200: ~4–6 verified per request.
⚠️ DSpark data figures (arXiv 2607.05147). Left: Qwen3-4B per-position acceptance, numbers from the text only; 0.74, 0.63, 0.78 are “later in the block,” position not stated: never “the last position.” Math, position 1: EAGLE-3 0.81, DFlash 0.88, DSpark 0.93. Right: Qwen3-8B, Table 1; math / code / chat each a three-benchmark mean, incl. the big model's 1 added token; all nine averaged give “18.4% above DFlash, 26.7% above EAGLE-3.”
⚠️ DSpark animations: scores (of 55%, course 95%, etc.), slot confidences and the throughput table are schematic; “16–18%” (Qwen3 4B–14B vs. DFlash) and “batch 128, drafts of 4–16 only 0.2–1.3% slower” are the paper's measurements; queueing and stop-when-it-drops follow its Algorithm 1. Production (§5.2) drops the stop rule, sizing the total from confidence two steps back: don't expand (full edition §11.3).
⚠️ Not covered (full edition §11.3–11.4): n-gram drafting; PD + speculation falling out of CUDA Graph; max-running-requests reset to 48.
⚠️ “Hundred numbers, match once in a hundred”: illustrative (uniform counterexample, Google's speculative-decoding retrospective blog), not measured. “Keep if the master is at least as confident, else proportionally” = acceptance prob. min(1, big-model prob. ÷ draft prob.). A greedy draft bets 100%, so keep probability = the big model's for that token (our derivation).
⚠️ Script shorthand: “randomness off, always picking the top token” = temperature 0, greedy. “Official numbers a notch lower” = temperature 1.0, top-p 0.95: 4.73 (code), 2.61 (creative) vs. 6.96. “Almost no extra time” = batch 128, drafts of 4–16 only 0.2–1.3% slower, buying 16–18% more accepted per round. “Third position almost always wrong” = 2%; “over 30%” = FastMTP's 36%. “Over 40% slower” = 11.1 → 16.1 ms. Don't say Medusa or DFlash aloud.
⚠️ “Take off the 40-plus percent” = 6.96 ÷ 1.45 ≈ 4.8, divided only so the script flows. In the ledger, 4.81 = 11.1 ÷ 2.31, two measured per-token intervals; 16 ms is back-calculated (2.31 × 6.96). Never “we measured the slowdown, then derived 4.8.” “Turn randomness on … keep fewer”: direction only; pitfall in Q&A. “Used to run one-step MTP,” not “only dared to”: the source gives no reason; don't tie it to the MiMo chaining numbers. “Helps less,” not “doesn't help”: 64 users still get 1.74×; only the DEP set slowed throughout.
❓ Likely questions (backup answers, not counted in the floor)
🧑‍🎓 Newcomer: “With random sampling, why not just check both drew the same token?” → Two people picking at random from a hundred numbers match once in a hundred, though they think alike. So compare confidence; a rejection is resampled from the two distributions' difference. Temperature 0 always picks the likeliest token (greedy); 1 rolls dice by probability. V4-Pro server default: 1.
🧑‍🎓 Newcomer: “In cut two, why does chat get one seat when busy?” → Chat rarely survives past slot 2, so its later slots mostly waste verification; coding gets them. Animation: idle, 14/15 admitted; busy, 7 (illustrative).
🧑‍🎓 Newcomer: “DFlash drafts a block per pass: what's in it, how is it made?” → A fill-in-the-blanks sheet. On Qwen3, 16 slots: slot 1 is the token just confirmed (the master's, last round), then 15 blanks. The 5-layer apprentice borrows the master's input and output layers, frozen. Hidden states from 5 master layers, shallow to deep, are concatenated into a note on “what the master is thinking,” projected into every apprentice layer's KV (EAGLE-3: input only). In one pass the blanks see each other, read the note, and fill together. “Diffusion” = filling blanks; ordinary diffusion refills repeatedly, DFlash once. Training: a random token of a real answer as slot 1, the next 15 masked. Weakness: blanks see only blanks, not what neighbors became; hence “of problem,” and worse accuracy toward the block's end. (DFlash, arXiv 2602.06036 §4–5; LLaMA 3.1: block 10; Qwen3 Coder: 8-layer draft.)
🛠 GPU veteran: “What if draft sampling and temperature don't match?” → A greedy draft bets 100%; at temperature 1.0 the big model gives any token partial confidence, so most get rejected. Load-testing V4-Pro on GB300, the tool sent no temperature; the server used 1.0: acceptance 1.16%. Temperature 0: 34%, 3.1× throughput. K3's draft samples by probability: fine at 1.0. Falling acceptance throws no error: alert on it as a top-level metric. (Off stage since the 2026-10-20 rehearsal.)
🧑‍🎓 Newcomer: “Can misses make it slower than no speculation?” → In tokens, no: at worst one. In time, yes, if verification slows and guesses miss (the DEP set).
🛠 GPU veteran: “Is benchmarking at temperature 0 cheating?” → No, but both sides must match and reports must state it.
💰 Budget owner: “Turn it on in production?” → Low-concurrency interactive tiers first; big batches: measure first; once on, alert on acceptance.
🛠 GPU veteran: “Greedy draft at temperature 1.0: why 1.16%? Theory isn't that low.” → Theoretical acceptance is the big model's probability for the token: low, not zero. Measured = accepted ÷ guessed: 7 guesses per step, and after the first rejection the rest are void (our derivation).
🛠 GPU veteran: “Acceptance differs ~30×, throughput only 3×?” → Throughput tracks extra tokens per trip, not the rate; even at 1%, each trip yields at least one.
🛠 GPU veteran: “Are temperature-0 numbers inflated for production?” → Depends on the draft: a greedy one collapses at temperature 1; sampling drafts like K3's suit production.
🛠 GPU veteran: “Why does longer context shrink it too?” → 4.22× at 4K, 1.32× at 32K. Without speculation, time between tokens only goes 11.1 → 11.4 ms, so verification shouldn't cost much more; likelier, acceptance length fell or the draft slowed on long KV (derivation, unmeasured).
🛠 GPU veteran: “How many MTP layers?” → TensorRT-LLM, 8 B200s, R1, one user: 111 tok/s; 1 layer 198, 3 layers 253, 5 layers down to 239. Different hardware and model: magnitude only.
🛠 GPU veteran: “What if I don't want to train a draft?” → n-gram (Prompt Lookup): copy matching prompt text. vLLM, 4 H100s, Llama 3 70B: up to 2.8× summarizing; a small draft model, same setup, 1.5× chat.

Section 8 · Quantization: fewer bytes, accuracy drop? (8′)Planned 8′ · with interaction 7.9′

🖥 Main screen for this segment: course page Section 8 · Quantization: fewer bytes, accuracy drop? · main figure Halve the bits, halve the per-step floor…

🎯 What this segment must leave behind: fewer bytes, accuracy drop? Saved: half the bits, half each trip's floor; 4-bit experts, attention ≤ 8 bits: about half a step. Lost: most error dies en route; softmax amplifies it, so protect attention. Measure the loss.

Section 7 ended here: more users, less help from speculation.
Section 0 gave two ways to cut cost: more riders per trip, or less cargo. So far, mostly riders. Now the other road: same bus, lighter cargo.
Lighter cargo, and everyone asks: does the model get dumber? Two halves: how much you save, and whether accuracy drops.

🖥 Screen: scroll to “Halve the bits, halve the floor”, stop at Halve the bits…; point at the four left bars, top to bottom.

Lighter cargo means fewer bits per weight: 16, 8, 4. Why the direct payoff? Section 1 showed that with few users a step is almost all weight reads: half the bytes, half the floor.
Many users, same: with MoE each expert gets only a few rows, so weight reads still dominate. Freed memory also holds more notes.

💬 Interaction: Ask “4 bits: only 16 codes. One block of weights runs from a thousand down to a thousandth. How do you fit them in?” Take a few answers; wait five seconds. Expected answer: a multiplier per small group; divide, then store.

Right: that multiplier is a scale. A few values share one: divide on store, multiply back on read. Why only a few? Put one long steel pipe in a crate of small screws: the ruler is sized for the pipe, and every screw measures zero. Smaller group, better-fitting ruler. Scales cost bytes too, so 4 bits isn't a clean halving again.

🖥 Screen: scroll to “4-bit experts, attention 8+ bits”; point at the three parameter counts up top.

So everything to 4 bits? First, what's in the cargo? The gpt-oss 120B model card lists the parts separately: experts are nearly all of it, attention under 1%. OpenAI took only the experts to 4 bits, and the whole model fits on one GPU.

🖥 Screen: down to Quantize only the experts…. Point at the top bar only; on the reveal, sweep all three, then blue: “it doesn't move.”

💬 Interaction: Ask “Experts go from 16 bits to 4: a quarter. How much of a step's bytes is left?” Take two or three; wait five seconds. Expected answer: about half, not a quarter.

About half. Blue, attention, doesn't move. By storage it's under 1%, but MoE reads only a few experts per token while attention is read in full at every layer. Shrink the experts and attention is almost 40% of the step.
Why not 4-bit attention too? Too small to save much, and the most error-sensitive part. Hence the second half: does accuracy drop?

🖥 Screen: scroll to “Accuracy: error crosses four gates…”; point at the four gates. At “how much you lose”, down to Same 4 bits…: red dot (right), gray (left), green (middle), then the two lines below.

Follow one error on its trip: from a rounded weight to a token coming out, it crosses four gates.
Gate one: land somewhere flat. Training walks downhill toward lower error, so a trained model sits on a valley floor, mostly flat. Nudge a weight a little, and most directions don't hurt. GPTQ pushes the error toward the flat directions.
Gate two: errors cancel. Rounding goes up about as often as down; add a few thousand values and most of it cancels. The danger is that long steel pipe, hence small groups.

Gate three: only first place counts. Small noise jiggles every word's score; the winner changes only if the top two were already close.
Gate four is the amplifier: softmax. People often say softmax recovers the error; it's the opposite. A small score difference becomes an exponential probability difference. Expert outputs are added straight back, so their error stays linear. Attention scores and expert routing both pass through softmax, so protect them. One paper took attention computation straight to 8-bit integers: Llama2 fell to about chance on a multiple-choice test. Smoothing outliers first, the same paper got 8 bits nearly lossless.
So everyone keeps attention at 8 bits at most, with special handling.

Absorbed error doesn't mean no drop: like any medicine, quantization has side effects. At the same 4 bits, the loss depends on how you quantize. Averages lie too: accuracy can stay flat while right and wrong answers swap and cancel.
The figure shows perplexity: roughly, how many words the model hesitates between per token; lower is better. Red, plain rounding, sits far from gray, 16-bit. Green picks its ruler more carefully and wins back half the gap.

One step further: a format designed for 4 bits. Per NVIDIA's blog, R1 from 8 bits to NVFP4 lost under 1% on main language tasks.
Furthest of all: train for 4 bits from the start. Kimi K2 Thinking officially says “lossless,” naming no baseline precision. So when you hear “lossless,” ask: compared with what?

🖥 Screen: scroll to “Quantize the KV too”; point at the top lines, then down to Half the bits for KV…: the two green bars.

Last, the notes compress too. With long context, KV is the bulk. Storing it at 8 bits is one flag; Kimi K3's high-throughput recipe makes it required, or that many users don't fit.
Two examples: one stores 8 bits, the other 4. Both green bars stop right of the red dashed line: neither saves a full half. On the left, part was never quantized; on the right, the extra is the scales.

🖥 Screen: stop at the green box “Takeaway”.

Takeaway: about half a step saved. Most error dies en route; protect softmax, and measure the loss on your own benchmarks.
The gain? vLLM's FP8 weights cut Llama 3 70B's time between tokens by up to half; gpt-oss quantized only its experts and fits on one GPU.
That's the toolkit. The last segment returns to that one trip: given a model and some devices, where do you start?

⚠️ Where this segment drifts⚠️ Framing: “straight to 8 bits gives chance → so 8 bits at most” contradicted itself; fix from the full edition (§12.1): SageAttention smooths K's outliers first, then 8 bits is nearly lossless. Chance = direct quantization, no outlier handling; “8 bits at most” = FP8 attention weights (V4, R1 on TPU).
⚠️ Framing: “about chance” = SageAttention, Llama2 MMLU 25.5% with Q, K, P, V straight to INT8 (chance: 25%). “Averages can lie” moved from gate three to “how you quantize”; flip source unchanged.
⚠️ The floor is our derived ideal: DeepSeek-V3, 37B active, 8 TPU v7 chips, one user, full bandwidth, weights only. Real steps are slower; 0.63 ms = 1/8 of Section 1's 5 ms on one chip.
⚠️ vLLM FP8: 2 H100s, 2024-07. Dense Llama 3 70B's time between tokens fell to as little as 1/2; MoE Mixtral 8x7B only to 1/1.6 (only if asked; no reason given).
⚠️ gpt-oss-120b: experts 114.71B, attention 0.96B, embedding + output 1.16B. 98% is derived (114.71 ÷ 116.83). So is the step's 10.2 → 5.0 GB (BF16 → MXFP4 experts; attention, output in BF16).
⚠️ Only “within 1%” has a source (NVIDIA blog, R1-0528, FP8 → NVFP4, main language tasks). K2 Thinking's “lossless” names no baseline. No official like-for-like FP4 vs. FP8 speed exists; don't estimate one.
⚠️ FP8 KV's 1.63× in the same memory is derived for V3.2's layout (1,284 ÷ 788 bytes), not general. NVFP4 KV roughly halving FP8 again, under 1% loss: NVIDIA's blog.
⚠️ “Almost 40%” is derived: 1.9 ÷ 5.0 GB ≈ 38% (expert-only chart, MXFP4 experts). Attention stays above 4 bits: little to save, more error-sensitive (full edition §12, point four). FP8 attention: DeepSeek-V4 model card (“mostly FP8 elsewhere”); full edition: R1 and GLM-5.1 on TPU are FP4 experts, FP8 attention.
⚠️ V3.2's 1.63×: MLA alone 1,152 ÷ 656 ≈ 1.76; the 132 B FP8 indexer key per layer brings it to 1,284 ÷ 788 ≈ 1.63 (derived).
⚠️ “Only a few rows” = the 8 rows of Sections 1 and 4 (derived: 256 × 8 ÷ 256).
⚠️ AWQ, WikiText-2 perplexity: 16-bit 5.47, AWQ 4-bit 5.60, RTN 5.73 (the page's numbers). “Wins back half” is derived: excess over 16-bit, 0.26 → 0.13.
⚠️ Don't expand on MXFP4/NVFP4 scale formats, R1's FP8 not fitting on v7x-8, CUDA Graphs or kernel fusion; point to the full edition, §12.
⚠️ The “steel pipe” is illustrative: one huge value inflates the shared scale and small ones round to zero (LLM.int8() blog's outlier example). “Smaller group, better fit” is direction only; sizes only if asked (MX 32, NVFP4 16). Perplexity as “words it hesitates between” is intuition, not an identity.
⚠️ Four gates: full edition §12 text, source table. Flat valleys: AdaRound, Sagun et al., GPTQ. Cancellation is our explanation; outliers: LLM.int8(), Super Weight. Flips: Dutta et al. 2024. Softmax's exponential amplification is our derivation, backed by DeepSeek-V3 (training) and SageAttention (Q, K, P, V, not weights; MMLU 25.5%). Never say “softmax recovers precision.”
⚠️ Framing (main line now two halves, savings / accuracy): “TPU v7 is fast without FP4 compute” is back in its backup answer; K3's FP8 KV “128 requests per device group” is page-only; the softmax interaction was cut; gate four corrects the misconception aloud. Shorthand: “nearly all of it” = 98%; “one GPU” = one 80 GB GPU; 10.2 → 5.0 GB = “about half”; AWQ dots pointed at, not read (red 5.73, green 5.60, gray 5.47). K2 Thinking's ~2× low-latency speedup and INT4-only benchmarks: page-only. KV figure: left, V3.2 BF16 → FP8, 1,284 → 788 bytes per token per layer (extra: unquantized positional encoding, added scales, indexer keys already FP8); right, V4.1-Flash main KV in native FP4, 45% under its FP8 (shortfall = scales: FlashMLA README, per token FP4 256 B + 32 B scales = 288 B vs. FP8 512 + 16 = 528 B, derived; spoken “the extra is the scales”. README: FP4 only for extra_k_cache, k_cache stays FP8; never “all KV is FP4”). “Four gates” orders the four points along the error's path. “8-bit integers” = INT8.
❓ Likely questions (backup answers, not counted in the floor)
🛠 GPU veteran: “Take a real model. What's the actual floor?” → DeepSeek-V3, 37B parameters per token, on 8 TPU v7 chips: FP8 needs at least 0.63 ms, 1/8 of Section 1's 5 ms on one chip (derived; real steps slower). (Moved from the main line.)
🛠 GPU veteran: “Accuracy dropped after quantizing. What do I check first?” → Two rulers: ① INT4 spans −8 to 7; files store it +8, and loading must subtract it. Kimi K2.6 logs: a loader skipped it → repeated gibberish, tokens flowing, no error. ② KV set to fp8 without calibration: vLLM uses scale 1.0 everywhere. Check output and benchmarks before going live. (Cut from the main line after the 2026-10-20 dry run.)
🛠 GPU veteran: “Softmax has a bounded derivative. How is it an amplifier?” → Ratios, not the derivative: score error δ multiplies probability by eδ (full edition §12.1, point four): a multiplicative skew. Expert outputs join the residual: linear error. Softmax scores and top-k routing can also swap the chosen tokens or experts outright.
🧑‍🎓 Newcomer: “Can quantization and speculation be combined?” → Yes. One lightens each trip; the other gets several tokens per trip.
🛠 GPU veteran: “TPU has no FP4 compute. What's FP4 good for?” → Fewer HBM bytes, more capacity; unpacking on chip is cheap. Don't say FP4 computes faster on TPU.
💰 Budget owner: “Does 4 bits halve the devices?” → Depends on fit and step speed; step bytes only roughly halve. Measure accuracy on your own benchmarks.
🛠 GPU veteran: “MXFP4 vs. NVFP4?” → MX: one E8M0 scale per 32 values. NVFP4: one FP8 E4M3 scale per 16, plus a per-tensor FP32 second-level scale. NVIDIA, 8B hybrid, 1T tokens: loss above BF16 about 2.5% (MXFP4), 1.5% (NVFP4).
🛠 GPU veteran: “Why must R1 be FP4 on TPU?” → On v7x-8, FP8 experts need about 101.8 per device, over the 94.75 limit; FP4 about 60.9, ~70 with attention: fits. Units mixed: README in GB, 94.75 GiB from run logs (about 101.7 GB). FP8 doesn't fit either way.
🛠 GPU veteran: “Are INT4 and FP4 the same?” → No. FP4 (E2M1) is floating point: dense near zero, sparse farther out. INT4 is evenly spaced, −8 to 7. The subtract-8 trap is INT4-only.
🛠 GPU veteran: “Are reasoning models this robust?” → Not entirely. A 2025 study: 8 bits, and 4-bit weights with 16-bit activations, essentially lossless; below that, risk rises clearly, more for smaller models and harder problems. Thousand-token chains of thought may accumulate small flips (our conjecture).
🧑‍🎓 Newcomer: “Why do experts tolerate quantization better?” → MoQE (2023) measured expert layers as more robust than dense FFNs; 2-bit experts still usable. No mechanism given; don't invent one.
🧑‍🎓 Newcomer: “Can you train at 4 bits directly?” → Yes: K2 Thinking does quantization-aware training in post-training. Catch: only the model maker can.

Section 9 · Day one: which questions first? (7′)Planned 7′ · with interaction 6.3′

🖥 Main screen for this segment: course page Section 9 · Day one: which questions first? · main figure Eight day-one questions: set the yardstick, do the math, then pick techniques

🎯 What this segment must leave behind: the eight questions are one: where is this trip stuck? Set the yardstick, measure; the answer points to a fix. Fixes stack on one bus: software alone, 5.1× per device.

Switch roles. Day one: you inherit a model and a pile of devices; your boss asks if it can get cheaper. Where do you start?
The whole course gave one answer: every decode step hauls the weights and KV through once. So day one has one question: where is this trip stuck?

🖥 Screen: all of Eight day-one questions. Skip the numbers; point at the bus in the middle.

The whole course in one picture. The warehouse is HBM; the workstation is Section 0's little workbench, where the chip computes; between them, the bus. The pins split that one question into eight; question seven takes two. Don't memorize them. Let's ride the bus once.

First set the yardstick: which tier, and how many devices in the denominator.
The tier is how fast tokens must come out. Change it, and one model's per-device throughput can differ over 10×. A multiple won on a tier too slow to bear is worthless.
The denominator: Section 4's DEP gain of just over 2× counts all devices. That run also added decode devices; count only those: about 1.3×. Without a matching yardstick, you can't tell if you gained.

Yardstick set, weigh two things: cargo and riders.
Cargo is Section 1: bytes per step over bandwidth is the fastest possible step. Measured several times slower? Not too few devices: check layout and scheduling first.
Riders are Section 3: how many users' notes fit in memory caps the batch.

💬 Interaction: Ask “You've measured: step time is near the floor, but only eight riders are aboard and memory is full. Where do you start?” Take two or three; wait five seconds. Expected answer: seats: thinner notes, no duplicate notes, or lighter cargo to free space.

Right: stuck on seats. Speculation won't help; more devices just run another equally empty bus. Measure, and the answer points to the fix.

🖥 Screen: scroll to “Find the jam, pick a fix”; point line by line.

By bottleneck, then.
Few riders, trip too empty: add riders. Continuous batching lets users board and leave any step. MoE scatters users across experts: add DEP and large EP to fill the expert bus.
Too few seats: thinner notes, paging, prefix reuse. Don't copy notes across devices. Very long contexts: split by sequence.

A long prompt stalls output: separate prep from serving: PD disaggregation.
Few users wanting fast tokens: speculation, several tokens per trip.
Wherever you're stuck, fewer bytes always help: quantization.

Don't reverse the order. Same vLLM runs as Section 4: tuning parameters alone gained a bit over 20%; the right split first, just over 2×.

🖥 Screen: scroll to “Opener: software-only speedup?”; stop on its first-paragraph question.

💬 Interaction: Reread the opening question. Ask “Hands up: one and a half times? Two? Five?” Count; wait five seconds. Expected answer: about 5×; our derivation gives 5.1×.

🖥 Screen: down to Same GPUs, same model, same per-user speed: software alone gives 5.1× per GPU; point at the two left bars.

Launch day: about 2,200 per GPU. By June: 11,200. Same GPUs, same model, same per-user speed.
That's 5.1×, our derivation; the blog title says 5×. If you guessed five, you win.

The blog doesn't split the 5× by technique; June also had MTP on, and the denominator isn't stated. So don't multiply the sections' speedups: each starts where the last ended. This is the only like-for-like total.
When someone quotes a speedup, ask: against what, which tier, how many devices?

The road isn't finished. In some layers, new models keep one sticky note rewritten as they go, not token-by-token notes. No first half to cut out for reuse, so prefix caching gets hard. That's the next gap.

🖥 Screen: stop at the green box “Takeaway”.

One last look at the bus. Section 1: every step hauls the weights and KV through once; with few users, it runs empty.
Each later section added a fix: fill the seats, free seats, keep prep from blocking serving, several tokens per trip, lighter cargo.
Stack them on one bus, same devices, same model, every user just as fast: the opener's 5×.

The whole course in one sentence: every decode step hauls the weights and KV through once; every technique either shares that trip among more users, or moves less.

⚠️ Where this segment drifts⚠️ 5.1× holds only on this basis: GB300, DeepSeek-V4-Pro, SGLang, same interactivity tier; ~2,200 (launch day) → ~11,200 (June) per GPU, derived; blog title 5×. Only June had MTP; share not broken out. Market price cuts: no multiple; we quote no price-based numbers.
⚠️ “Cost falls to about one-fifth” is derived (full edition §2.3: cost/token = device-hour price ÷ per-device throughput; 2,200 ÷ 11,200 ≈ 0.196), assuming ① same device price, ② all devices, prefill included. Denominator unstated: if decode GPUs only and prefill didn't shrink, all-device cost falls less than 5×. The 5.1 ratio is unaffected (one chart, one basis).
⚠️ +28% (concurrency alone) and 2.09× share one vLLM record (GB300, V4-Pro, VLLM-V4PRO-RUNBOOK.md:446-489; 3p1d, concurrency 256 → 512, total 16,459 → 21,100), hence “same runs as Section 4”; Section 4's text lacks 28% (page: see full edition §10). Say only “a bit over 20%.”
⚠️ “Tens of times”: full edition §15, question two (→ §2); here only Section 4's 2.09 → 1.31. Compile 60–90 s / under 1 s same shape: full edition §14 (and the “central kitchen”).
⚠️ 111 → 198 (1 MTP layer) → 253 (3) → 239 (5): TensorRT-LLM, single-user tokens/s, R1, 8 × B200. Other model, hardware: magnitude only; too many layers goes backward.
⚠️ “Sticky note” = a linear-attention layer's recurrent state: fixed size, rewritten every token, so no per-prefix copy to cut out.
⚠️ Section speedups don't multiply: Orca's baseline was FasterTransformer; paging's already had Orca. Answer “how much faster altogether?” with this.
⚠️ After the closing line, stop.
⚠️ Framing: decode devices 4 → 8 (“also added decode devices”); 1.31 spoken “about 1.3×”; “just over 2×” = 2.09. “Stack them” follows full edition §15's last line (“①–⑪ stacked”) and takeaway chain: say they stack, not which make up the 5× (blog doesn't say).
⚠️ “Too slow to bear is worthless” is reasoning, no number. Example if asked: the Section 2 curve (tier change: 10×+).
❓ Likely questions (backup answers, not counted in the floor)
🛠 GPU veteran: “What's pin 8 on the cab?” → Validate before launch. TPU compiles whole graphs per fixed shape, a central kitchen pre-planning portion sizes. Uncompiled size: first request 60–90 s, later under 1 s. Warm up common sizes first (full edition §14). (Moved here after the 2026-10-20 dry run.)
🧑‍🎓 Newcomer: “Must I memorize the eight questions?” → No, they're in the figure. Remember the order: yardstick and workload, then fixes.
💰 Budget owner: “Only one thing first: which?” → The page doesn't rank by cost. We suggest the eight-question order: yardstick, split, parameters last.
🛠 GPU veteran: “Which devices are in the 5.1× denominator?” → Decode or all: not stated; full edition ledger: “not stated officially.” One-fifth cost assumes all devices, same price.
🛠 GPU veteran: “How much of the 5.1× is MTP?” → Not broken out. The blog's two curves (no MTP: 2.1× at the 40 tier; MTP: 2.6× at 80) are at different tiers; can't subtract.

🚧 Where these notes are still thin

  • [x] Full run-through: handoffs aligned; one bus metaphor (Section 6's kitchen, Section 7's apprentice flagged as switches), plan re-split against the floor.
  • [x] Layperson + inference expert, Sections 0–4: added layers, heads, KV heads, how Little's law counts users, the 2.09× and 1.31× denominators.
  • [x] Same readers, Sections 5–9: added weighted averages, temperature and sampling, quantization's two budgets (capacity, step bytes), why compile ahead, the 5.1× denominator; segments near planned minutes.
  • [x] Restored Sections 0–4's “why”: a Topic 5 recap question opens Section 0; Sections 5–9 smoothed for speech; with-interaction times 0.3–1 min under plan.
  • [ ] Not yet timed end to end; progress in Courses/reviews/T6-L200-STATE.md.