Inference

Training computes a big batch per step; inference emits one token.
Every decode step hauls the weights and KV cache once. Every technique either shares that trip among more users or moves fewer bytes.
Prereqs Topic 3 (KV cache), Topic 5 (parallelism) Data as of 2026-10 ⏱ about 90 min 📖 Going deeper: L300 full edition

Course author Chris Yang·Google Cloud AI Infra architect

Section 0

Why inference is expensive, and how it got cheaper

🎯 Warm-up: which collectives per parallelism? — Topic 5 recap; open in class, not before. Safe to skip

Per row, tick every collective needed in one training step (possibly several). Fill all five rows, then click “Check answers”.

ParallelismAllReduceAllGatherReduceScatterAllToAllP2P
DP (data parallel)Full model per device, own data
FSDP (ZeRO-3)Weights, grads, optimizer state sharded
TP (tensor parallel)Layer matrices split across devices
PP (pipeline parallel)Layers cut into stages
EP (expert parallel)MoE experts on different devices

Training does a batch per step; inference, one token. The cost: each token hauls the whole load from HBM.

Only two waits: first token, then each next

Life of a request: one big prefill, then token-by-token decode Schematic, not to scale. Top: activity and token timing · middle: KV length in HBM · bottom: what each decode step reads from HBM prefill: whole prompt at once decode: 1 token/step weights KV cache Arrival to finish Activity token timing Queue prefill whole prompt, one pass Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 Step 7 Step 8 token 1 token 2 token 3 token 4 token 5 token 6 token 7 token 8 end Request arrives TTFT: arrival → first token (incl. queue) TPOT: mean gap between tokens KV length in HBM prefill writes the whole prompt's KV at once each decode token adds one KV slot Each step reads from HBM weights: read once per step KV: all of it, every step The two phases hit opposite limits prefill reads the weights once and shares them across thousands of prompt tokens: compute-bound. decode reads the weights and all the KV to emit one token: bandwidth-bound, and each trip gets heavier as the KV grows.
Red dot, then green boxes: one token, one weight read each; purple staircase grows every step.
Schematic, not to scale.
Sources and assumptions

⚠️ Schematic timeline, boxes not to scale: prompt drawn as 6 KV slots, decode as 8 steps, for legibility.

📌 TTFT and TPOT follow vLLM's benchmark_serving: TTFT runs from sending the request to the first output token; TPOT is the mean time per output token, excluding the first. Section 2 covers how they're measured (with or without queueing, mean or p99).

⚠️ “Prefill yields the first token” is standard: its last position gives the first output token's distribution.

The cost: one full haul per token

In one sentence Each decode step hauls the weights and KV once. Every technique shares the trip with more users or moves less.
One family leaves the trip alone: each kind of work gets its own hardware.

Four years cheaper: each fix leaves a new gap

The problem chain: when each of the 13 techniques appeared Dot = year first published (⑫⑬: year they reached mainstream models; ideas older); spacing within a year shows order only; “→” = active until; circle = link number system hollow: model driver: context driver: outputs 2022 2023 2024 2025 2026 Longer context 128K 1M Longer outputs o1 R1 2022 Orca cont. batching ① 2022 →2026 GPTQ quantization ② 2022 →2025 Speculative decoding ③ 2023 →2025 GQA shrinks KV ④ 2023 PagedAttention ⑤ 2023 Chunked prefill ⑥ 2023 →2025 PD disaggregation ⑦ 2023 RadixAttention ⑧ 2024 →2025 KV tiered offload ⑨ 2024 →2025 Sequence split (CP) ⑩ 2024 →2025 MoE large EP ⑪ 2025 →2026 Linear-attn hybrid ⑫ 2026 Recurrent models ⑬
What each link does: see the table below.
Savings and baselines: Chinese full edition, Section 0.
Sources and assumptions

First published: Orca OSDI'22; GPTQ arXiv 2210.17323; speculative decoding, Leviathan et al., arXiv 2211.17192; GQA arXiv 2305.13245; PagedAttention blog 2023-06-20; Splitwise arXiv 2311.18677; RadixAttention arXiv 2312.07104; SARATHI arXiv 2308.16369; Mooncake arXiv 2407.00079; Meta context parallelism arXiv 2411.01783; SGLang v0.4 blog 2024-12-04 (DP attention), DeepSeek-V3 arXiv 2412.19437; ⑫ Qwen3-Next blog 2025-09-10 (earlier: H3 arXiv 2212.14052, Jamba 2403.19887); ⑬ Koi-4B arXiv 2610.12582, NVIDIA-Nemotron-Labs-Dory1.0-4B 2026-10-16 (earlier: Universal Transformers arXiv 1807.03819). End years: DeepSeek-V3.2-Exp technical report (DSA, 2025-09); SGLang HiCache blog 2025-09-10; NVIDIA Dynamo press release 2025-03-18; SGLang large-scale EP blog 2025-05-05; vLLM v0.10.2 release (DCP, 2025-09); EAGLE-3 arXiv 2503.01840; Inferact TPU megakernel blog 2026-09, vLLM DeepSeek-V4.1-Flash blog 2026-10-07 (FP4 KV). External drivers: OpenAI DevDay 2023-11-06, Google 2024-02-15; OpenAI 2024-09-12, DeepSeek-R1 arXiv 2501.12948. Dots are evenly spaced within a year and show order only.

DebutBottleneckFixLeft behind§
2022Batch waits on the slowestContinuous batching: on/off each stepKV caps seats; max-length reserve2
2023–2024KV wastes HBM; rows too fatPaging; GQA/MLA slim; prefix reuseContexts keep growing3
2024–2025MoE scatters users over hundreds of expertsDEP + large EP: 100+ devicesPer-device state; long KV copied4
2024–2025Long KV won't fit; slow TTFTDCP / PCP: split the sequencePays only for long contexts5
2023–2025Long prompts stall outputPD disaggregation: split the jobsOne more KV trip per request6
2022–2025Reasoning: thousands of tokens, one a stepSpeculative decoding: several per stepGains shrink with many users7
2022–2026Only bytes left to saveFP8 / FP4 quantization, fused kernelsAccuracy drop must be measured8
Guess first Same GB300, DeepSeek-V4-Pro and per-user speed; only software changes. Per-GPU throughput is how many times launch day's? (Section 9)
Market prices fell more, mostly from stronger small models (Stanford AI Index 2025); this is software only.
Section 1

A long prompt in one pass, but output one token at a time?

The same chip is flat out reading a prompt, mostly idle emitting tokens. The only difference: rows per pass.

Rows per pass decide: compute- or memory-bound

Rows per pass set the bottleneck: decode sits below the ridge point, prefill above A 7168×7168 bf16 matmul. X: rows fed per pass, M (tokens); Y: operations per byte moved. Both axes log scale decode: M = concurrent users prefill: M = prompt length v7 ridge point Arithmetic intensity = operations ÷ bytes moved 1 4 16 64 256 1,024 4,096 16,384 M: rows fed per pass (tokens) 1 10 100 1,000 ops / byte v7 ridge (bf16): below 313 ops/byte, HBM-bound ↑ compute-bound on this side ↓ HBM-bandwidth-bound on this side Needs ~343 rows per pass to cross the ridge decode, 1 user: 1 op per byte decode, 64 users: 63 ops per byte decode, 256 users: 239 ops per byte prefill, one 4K prompt: far above the line Decode struggles to cross the line however many users it batches; one prompt puts prefill far past it Decode feeds 1 row per user per step, so even 256 users batched together give only 239 ops per byte; one 4K prompt is 4,096 rows at once. On the same chip, prefill is limited by compute and decode by HBM bandwidth.
Red line at 313: green dots (decode) all below, orange (prefill) above.
X axis: rows per pass; our derivation.
Sources and assumptions

📌 7168 = hidden_size in DeepSeek-V3's config.json. Ridge point 313 = v7 per-chip bf16 2,307 TFLOP/s ÷ HBM 7.37 TB/s (Google Cloud public specs; bf16 = half of FP8's 4,614, our derivation; same as Topic 2).

📌 Our derivation: ops 2·M·H²; bytes 2·(H² + 2·M·H) (weights, input read once, output written once, 2 bytes per bf16); intensity = M·H ÷ (H + 2M) = 313 at M ≈ 343.

⚠️ Weights only, intensity is just M and the crossover exactly 313 (Topic 5's fig5-decode-ai convention); adding input and output moves it to 343.

⚠️ Deliberately left out: attention (it reads KV, so its intensity doesn't grow with users; Section 5) and MoE expert layers (see this section's figure on rows per expert).

One user, one token: 5 ms moving, 0.016 ms computing

Blue fills step by step; orange barely moves until step 3 (313 rows), where it catches up. (27 s, Manim; untick “Autoplay” to step. Ideal lower bound, weights only; “compute = data movement at 313 rows” is our derivation from the same specs.)

37 GB ÷ 7.37 TB/s ≈ 5 ms; compute 0.32% used (ideal lower bound). Faster compute won't help; bandwidth sets the floor.

Back to the bus Fixed fuel per trip: prefill is a full bus, decode one rider. Up to 313, extra riders ride free.

More users: weights amortize, KV doesn't

Bytes moved per token: weights amortize, KV is per user DeepSeek-V3, FP8 weights, bf16 MLA KV. Each bar = weights read per step ÷ users + this user's own KV. Same scale throughout weights per token this user's own KV Bytes per token (GB) 1 concurrent user all 37 GB for a single user 4K KV 0.8% 32K KV 6% 128K KV 20% 64 concurrent users 9.14 GB each (87% of experts hit) 4K KV 3% 32K KV 20% 128K KV 50% 256 concurrent users 2.62 GB each 4K KV 10% 32K KV 47% 128K KV 78% 0 8 16 24 32 40 48 GB/token More users and longer context make KV the heavier part of each trip One weight read is shared by everyone; KV is each user's own history, 9.21 GB per token at 128K, however many users you batch. At 256 users and 128K, KV is already 78%: beyond that, the savings have to come from the KV itself.
More users shrink blue; only context grows purple.
Uniform routing, equal contexts; no activations/comms. Our derivation.
Sources and assumptions

📌 Our derivation: KV = 70,272 bytes/token (MLA stores 512 + 64 values per layer, bf16, 61 layers, config.json) × context. At 128K: 9.21 GB = 8.58 GiB, same as Topic 5.

📌 Our derivation: weights read per step, b users = non-routed + 653.9B routed-expert params × [1 − (1 − 8/256)^b] (FP8, 1 B each). b = 1: 37 GB exactly; large b → 670.5 GB, i.e. 671B.

⚠️ At 64 users each pays 9.14 GB, not 37 ÷ 64: users pick different experts, so a step reads 87% of them.

⚠️ Uniform routing, equal contexts; activation and comms bytes not counted. FP8 KV halves the purple.

MoE breaks the batch apart again

MoE breaks up the batch: 256 users in, 8 rows per expert One decode step for a batch of 256 users. Squares are v7 MXUs (256 per side); the filled part is the rows actually fed this step rows fed to the MXU this step (decode tokens) Dense layer vs one expert Dense layer: 256 users, one weight matrix MXU 256 × 256 256 rows: one side full Intensity ≈ 239 ridge point 313 MoE layer: one expert MXU 256 × 256 8 rows avg: 1/32 of a side Intensity ≈ 8 ridge point 313 256 × 8 ÷ 256 = 8 still below, but close The 256 users you batched get split 256 ways again at the experts Each user picks only 8 experts, so an expert gets 8 rows on average: intensity about 8, versus about 239 for dense, a 30× gap. To keep experts fed, you have to gather users from many devices into one place: that's the job of large-scale EP.
Left fills one side; right, a sliver. Like 256 branch lines, each rider on 8: 8 riders a line.
8: uniform-routing average; shared experts unaffected.
Sources and assumptions

📌 256 routed experts, 8 picked per token: DeepSeek-V3 config.json (n_routed_experts, num_experts_per_tok). MXU side of 256: public JAX source jax/_src/pallas/mosaic/tpu_info.py, mxu_column_size = 256 for TPU 7.

📌 Our derivation: expert matrix 7168×2048; at M = 8 the intensity = 2·8·7168·2048 ÷ [2·(7168·2048 + 8·(7168 + 2048))] ≈ 8. Dense layer's 239: this section's first figure.

⚠️ 8 is an average: uneven routing gives some experts dozens of rows, some none. Shared experts see every token, unaffected.

The 313 line holds only per whole matmul; to fill every expert, a step needs 9,000+ tokens a step (Chinese full edition, Section 3, for H100).

Takeaway Prefill is compute-bound; decode is bound by data movement. To earn back the fuel, carry more riders: keep the bus full (Section 2), free seats for KV (Section 3), regather users MoE scattered (Section 4).
Three-column comparison, intensity derivation: Chinese full edition, Section 1.
Section 2

Just add users? How big can the batch get?

Riders don't board or leave together: one asks and goes, another wants 3,000 words. Scheduling sets how full a trip runs, seats set the cap, and fuller means slower per rider.

Old way: fill up, wait for the slowest

Stop at every step and reschedule

Gray cells pile up on top, none below. (18 s silent loop; request lengths illustrative, steps equal.)

Orca (OSDI 2022), first idea: stop after every step and look again. Now called continuous batching; vLLM and SGLang both do it.

Still the same bus Static batching is a charter: fill it, arrive together. Continuous batching is a real bus: off and on at every stop.

Seats are KV: more seats, not always more tokens

One seat per request the KV pool can hold. Per-device throughput ≈ riders ÷ TPOT (time between tokens); full seats cap it.

Add seats? TPOT won't stay put:

Over 10% more seats: throughput nearly flat, every step slower GB300 · DeepSeek-V4-Pro · more KV memory only · bar = % change Riders TPOT Ceiling Measured Change, after vs. before Riders 886 → 999 +13% TPOT 77.7 → 85.3 ms every rider slower Throughput ceiling riders ÷ TPOT ≈ +3% (our derivation) Measured throughput end-to-end +0.8% 0
Top vs. bottom: over 10% more riders, throughput nearly flat.
One measured setup; only KV-pool memory changed.
Sources and assumptions

Measured: GB300, DeepSeek-V4-Pro, SGLang, mem-fraction-static 0.85 → 0.88; requests per rank 886 → 999, TPOT 77.7 → 85.3 ms, end-to-end throughput 10,614 → 10,704 (SGLANG-V4PRO-RUNBOOK.md:849–858, 885–894).

Our derivation: 0.8% = 10,704 ÷ 10,614 − 1; ceiling = 999/886 × 77.7/85.3 − 1 ≈ 2.7%, assuming every new seat is filled and TPOT is as measured.

Weights are already amortized; newcomers add only KV, lengthening every step.

So batch size starts with “how fast per user?”

Throughput vs. interactivity is one curve: set the tier, then compare GB300 · DeepSeek-V4-Pro · 8K in, 1K out · y = output tokens ÷ decode GPUs only (11,200: denominator unpublished) InferenceX: Dynamo + vLLM (one curve) PyTorch blog: SGLang This repo, measured: SGLang dep8 Per-GPU throughput vs. per-user speed 0 50 100 150 200 0 3,000 6,000 9,000 12,000 Tokens per second per user (tok/s/user) → more responsive Tokens/s per GPU ↑ Set the tier: 50 tok/s per user 9,759 @ 68 3,816 @ 121 895 @ 173 11,200 (2026-06, MTP on) 2,200 (day 0, no MTP) 9,168 @ ≈ 17 (TPOT ≈ 60 ms) Same tier: 11,200 vs. 2,200. Ratio? ≈ 224 concurrent users per GPU (11,200 ÷ 50) A per-GPU throughput number alone tells you nothing On one vLLM curve, per-user speed 68 → 173 cuts per-GPU throughput 9,759 → 895. Always report throughput with its tier. This repo's 9,168 sits at a low tier, ≈ 17 per user; it can't be ranked directly against 11,200 at the 50 tier.
Move left along the green line: more throughput per GPU, slower tokens per user.
Only the three green points form one curve; others mix frameworks and configs.
Sources and assumptions

InferenceX, three points: gpu-tpu-pedia gpu/inference/a4x-max/deepseek-v4/VLLM-V4PRO-RUNBOOK.md:683–695 (GB300 Dynamo + vLLM, V4-Pro FP4, 8K/1K, output ÷ decode GPU).

PyTorch blog, 2026-06-23, “Serving DeepSeek-V4 on GB300 with SGLang: 5x Higher Throughput at the Same Interactivity Since Day-0”: ~11,200 tok/s/GPU at roughly 50 tok/s/user; the Day-0 curve without MTP is about 2,200.

⚠️ Public material doesn't settle whether 11,200 is output ÷ decode GPUs or (in + out) ÷ all GPUs (SGLANG-V4PRO-RUNBOOK.md:745). We plot the former; if the latter, the point moves down.

Measured in this repo, 9,168: SGLANG-V4PRO-RUNBOOK.md:596 (conc600, TPOT 59–61 ms), 723–725. The x-coordinate 1000 ÷ 60 ≈ 17 is our derivation.

Goodput: DistServe (arXiv 2401.09670).

Takeaway Continuous batching fills the seats; KV sets their number; more riders, slower each. The gain: at the same 190 ms per-token latency, Orca served GPT-3 175B at 36.9× the requests/s of FasterTransformer's static batching.
Next: how big is each request's KV, and can it shrink?
Going deeper: Chinese full edition, §3.
Section 3

KV cache: how big, how to shrink?

Seats are KV: each request keeps a notebook, one line per token. Seats = KV pool ÷ notebook thickness. Three steps seat more: thinner lines, no wasted paper, no redoing.

Step one: same model shape, line thickness differs over 50×

Does thinning cost accuracy? The GQA paper converted T5-XXL to 8 groups: about 5.4× faster decoding (our derivation), average 47.1 vs. 47.2 originally. Only T5-XXL on TPU v4.

Thinner isn't enough: longer notes, fewer riders

8× the context: full attention fits 1/8 as many, the linear hybrid over 1/4. How KV grows caps the batch One v7 device; V3 weights over 8 chips = 16 devices, rest goes to KV. x: context length; y: sequences that fit at once (log, rounded down) MLA · DeepSeek-V3 Linear + full attn · Qwen3.5 shape GQA · Llama-3.1-70B MHA · V3 shape (hypothetical) 94.75 GiB usable per device (1 chip = 2 devices) 1/16 of V3 weights (FP8): 39.1 GiB Keep 10% 9.5 GiB KV pool: 46.2 GiB Sequences that fit at once 1 10 100 1000 1K 2K 4K 8K 16K 32K 64K 128K Context length (log; each step doubles; 1K = 1024 tokens) seqs 172 21 5 seqs · MLA 70,272 B/token 157 41 11 seqs · linear + full attn 30,720 B/token + 180 MiB/seq 36 4 1 seq · GQA 327,680 B/token MHA: 0 fit from 16K 12 3 Sequences that fit = users sharing each trip MLA fits 172 at 4K but only 21 at 32K. Linear layers charge every sequence a fixed 180 MiB of state up front: at 4K the hybrid actually fits fewer than MLA (157 < 172); at 128K, over twice as many (11 > 5).
Each step right doubles context and halves the green, blue and red counts.
One v7 device (half a chip, 94.75 GiB), bf16 KV, weights roughly fully sharded. Purple (a linear-attention hybrid) isn't detailed in this section. Our derivation.
Sources and assumptions

Sources: KV per token: V3 (512+64)×61×2 B = 70,272 (config.json); Llama-3.1-70B 2×8×128×80×2 = 327,680 (HF config.json); Qwen3.5 full attention 15 layers × 2 heads × 256 dims × K, V × 2 B = 30,720; linear-layer state 45×64×128×128×4 B = 180 MiB (HF config.json, mamba_ssm_dtype float32).

Assumptions: the MHA row is a hypothetical at V3's shape (128 heads × 128 dims, Topic 3's first problem): 3,997,696 B/token.

Our derivation: one v7 chip appears to software as two devices, each managing its own HBM, so we account per device. All V3 weights at FP8, 1 B/parameter: 671 GB ÷ 16 devices = 41.9 GB = 39.1 GiB. Measured usable HBM is 94.75 GiB per device (tpu-inference R1 FP4 README); a further 10% is held back for activations (our assumption). KV pool = 94.75 GiB × 0.9 − 39.1 GiB = 46.2 GiB. A rough estimate with every weight sharded; under DP attention each device holds its own copy of the non-expert weights, so the real pool is smaller. We assume attention runs data-parallel (each device stores only its own sequences); if attention runs TP, MLA's KV is replicated on every device (Topic 5).

Not counted: activations and communication buffers (folded into the 10%), the linear layers' convolution state, and the engine's actual state dtype (may not be fp32).

Fewer riders per trip, each reading more KV per step. Long context pays on both.

Step two: no wasted paper, a page at a time

Request 2 leaves; its two blocks go right to request 4. (18 s silent loop; block size, count and request lengths illustrative.)

Step three: reuse a shared opening and keep writing

A cached prefix isn't recomputed: first token 13 s → 0.5 s DeepSeek production, computed KV kept on disk. Top: TTFT of one 128K high-reuse prompt (linear, 0–14 s); bottom: share of a day's input that hits Miss: whole prefix redone Hit: KV read back from disk Time to first token (TTFT), 128K prompt 0 s 2 s 4 s 6 s 8 s 10 s 12 s 14 s Miss Hit Recompute 128K prefix 13 s 0.5 s, incl. disk read ≈ 1/26 (our derivation: 13 ÷ 0.5) Input tokens over 24 hours (separate statistic, 2025-02) 608B 56.3% hit on-disk KV 43.7% miss (our derivation: 100 − 56.3) 0% 100%
Top: one request's saving on a hit. Bottom: how much of a day's input hits.
DeepSeek production; TTFT and hit rate come from two public sources, different periods.
Sources and assumptions

Source: TTFT 13 s → 0.5 s, from the DeepSeek API docs' “Context Caching on Disk” announcement (api-docs.deepseek.com/news/news0802, 2024-08; original wording “128K prompt with high reference”).

Source: 608B input tokens and the 56.3% hit rate are from DeepSeek open-infra-index 202502OpenSourceWeek day_6 (window 2025-02-27 12:00 to 02-28 12:00 UTC+8; 342B of 608B hit; V3 and R1 combined).

Our derivation: 13 ÷ 0.5 = 26; 100% − 56.3% = 43.7%. The two numbers come from two documents at two points in time, not from one measurement. Prices are not shown.

Takeaway Thinner lines (MLA/GQA), no wasted pages, reused prefixes: same memory, more riders, less recompute.
The gain: in the vLLM paper (SOSP 2023), 2–4× the baselines' throughput at equal latency, even against continuous batching (the authors' Orca reimplementation). Whole systems, not paging alone.
So far, one device. With many: one notebook each, or a copy per device?
Going deeper: Chinese full edition, §4.
Section 4

Many devices: TEP or DEP?

Too big for one device, V3 runs on several. Experts go EP either way; attention is the fork: one shared batch, or own users per device? This decides how many copies of each notebook exist.

Reading the names TEP = T + EP, DEP = D + EP. EP (experts) in both; the first letter is how attention splits: T = TP (by heads), D = DP (own users per device). dep8 = 8-way DEP.
In DCP/PCP (next section) the first letter is a phase, a different axis. Nor is either Megatron-LM's ETP/EDP.

How many notebook copies?

TEP vs DEP in attention: one shared batch, or one each Same 4 devices, same 4 request batches (color = batch); experts the same on both sides: 8 over 4 devices, 2 each Req 1 Req 2 Req 3 Req 4 TEP4: TP attention, all four on one batch Requests Attention KV cache Experts Dev 0 1/4 heads All KV (full) E0 E1 Dev 1 1/4 heads All KV (full) E2 E3 Dev 2 1/4 heads All KV (full) E4 E5 Dev 3 1/4 heads All KV (full) E6 E7 Attention out AllReduce: sum the four partials Expert step All devices have all tokens; local experts, then AllReduce Pro Four devices per request: each request runs faster Con KV can't be split, full copy per device: 3 of 4 wasted DEP4: data-parallel attention, one batch each Requests Attention KV cache Experts Dev 0 All heads Own only E0 E1 Dev 1 All heads Own only E2 E3 Dev 2 All heads Own only E4 E5 Dev 3 All heads Own only E6 E7 Attention Each on its own, no communication Expert step AllToAll dispatch to expert devices, then combine back Pro No duplicated KV: more requests fit, high throughput Con Uneven request lengths, uneven load; idle devices still run The difference is all in attention: one batch together, or one each TEP: four devices share each batch, so a single request is fast; but MLA-style models have one KV head, so every device stores the full KV. DEP: each device serves its own requests, no duplicated KV, bigger batches; the price is an AllToAll in the expert step and guarding against uneven load.
Attention only: left, four devices share a batch; right, each has its own.
Expert row: EP on both sides.
Sources and assumptions

📌 TensorRT-LLM definitions: TEP<N> shards both attention (TP) and experts (EP) across N ranks; DEP<N> keeps attention data-parallel (ADP) while distributing experts across N ranks.

⚠️ Expert-step communication as usually implemented: with TP attention every device holds the same tokens, so expert outputs are combined by all-reduce; with data-parallel attention the tokens differ, so it takes an all-to-all. Illustrative: 4 devices, 8 experts.

Real models: MLA and V4-Pro have 1 KV head: TEP8 makes 8 copies. Qwen3.5-397B, 32 query heads on 2 KV heads: 4 copies. GQA with 8 KV heads: none, so DEP gains nothing.

Capacity vs. speed: empty seats earn nothing

V4-Pro (1 KV head), our own SGLang setup on GB300: prefill and decode on separate machines (why: Section 6), DEP on both, 14 prefill nodes at dep4, decode at dep8. 10,704 tok/s per decode GPU end to end: 95.6% of the official target (the PyTorch blog's ~11,200 per GPU; denominator unstated, we read it as decode GPUs).

TEP → DEP: weigh capacity and speed separately. SGLang, DEP on both ends: filled seats buy throughput Top: capacity axis (illustrative, every KV pool full). Bottom: SGLang, measured (14 prefill nodes × dep4 → decode dep8) Distinct batches' KV Copies of one batch's KV ① Capacity: distinct batches held when KV pools are full (illustrative) TEP4 4 devices, 1 batch Dev 0 Batch A Dev 1 Batch A copy Dev 2 Batch A copy Dev 3 Batch A copy 4 devices hold just 1 batch 1 KV head can't be split: each device stores a full copy DEP8 1 batch each Dev 0 Batch 1 Dev 1 Batch 2 Dev 2 Batch 3 Dev 3 Batch 4 Dev 4 Batch 5 Dev 5 Batch 6 Dev 6 Batch 7 Dev 7 Batch 8 8 devices, 8 batches: 4× ceiling per device 4 = TP degree 4 ÷ KV heads 1 A ceiling: extra seats pay only once concurrent requests fill TEP4's pool; long context alone won't do it. ② SGLang, measured: 14 prefill × dep4 → decode dep8, input 4096; throughput tracks filled seats prefill × 14 nodes, dep4 each (attn DP4 + EP4) decode dep8 (2 nodes, DP8 + EP8) 16 GB300 nodes, one NVL72 domain Concurrent per GPU Usage of 2 KV pools End-to-end per decode GPU Time between tokens Pools mis-split (ratio 0.20) 727 0.93 / 0.63 6,887 58.0 ms Evened out (ratio 0.15) 886 0.92 / 0.89 10,614 77.7 ms 13% more memory 999 0.92 / 0.67 10,704 85.3 ms +54% → +0.8% Seats buy throughput on two conditions: they get filled, and steps don't slow down 727 → 886: pools evened out, more seats; 14 prefill nodes keep them filled, so end to end +54%. 886 → 999: 13% more seats, only +0.8%: the step stretches from 77.7 to 85.3 ms, cancelling the extra riders.
Top: capacity ceiling; 3 of TEP4's 4 batches are copies. Bottom (SGLang, measured): throughput rises only as seats fill.
GB300, DEP on both ends, 4096 input; top half illustrative, not measured.
Sources and assumptions

SGLang deployment: gpu/inference/a4x-max/deepseek-v4/SGLANG-V4PRO-RUNBOOK.md:33–34 (14 prefill × dep4 + dep8 decode, 16 nodes in one NVL72 domain), scripts/prefill-dep4.sh, decode-dep8.sh (--enable-dp-attention, EP 4/8); the three measured rows :851–858 (ISL 4096, 14 streams × concurrency 1050, #1/#3/#7).

Corroboration: VLLM-V4PRO-RUNBOOK.md:298–302 (num_key_value_heads 1), :464–465, :476 (concurrency 512: TP4 decode output 4,220, time between tokens 46.8 ms; dep8 11,031 and 11.8 ms; 16 and 20 GPUs).

Our derivation: capacity-ceiling multiple = TP ÷ min(TP, KV heads); the top half is illustrative, and bar lengths are not drawn to measured request counts. +54% = 10,614 ÷ 6,887, +0.8% = 10,704 ÷ 10,614; 2.09× = (55,153 ÷ 20) ÷ (21,100 ÷ 16), where 55,153 and 21,100 are total throughput from the same table (input + output; 4K in, 1K out, so roughly 5× the output alone), 1.31× = (11,031 ÷ 8) ÷ (4,220 ÷ 4).

DEP's cost: each device packs its own kit

Large-scale EP: regather the users

Section 1's gap: only 8 rows per expert on average. Fix: spread the experts beyond one 8-GPU machine (large-scale EP; TensorRT-LLM's line: EP above 8). Attention stays per-device: DEP scaled up.

Two lanes, same columns: more users per expert copy, less read per device. (24 s, Manim; untick “Autoplay” to step. Expert sizes, layer count: V3 config; 32 requests per device, even load: illustrative.)

DeepSeek's production decode: EP144 on 18 machines; 256 experts + 32 redundant = 288 copies; per GPU, 2 routed + 1 shared.

Why it works: waits, not volume

Large-scale EP leans on the network: more GPUs, slower transfers Left: DeepEP low-latency kernel, measured (H800, one 400 Gb InfiniBand NIC per GPU; 128 tokens per batch). Right: three interconnect domains Dispatch: tokens to experts' GPUs Combine: results sent back Time per transfer (µs) 0 100 200 300 400 EP8 EP16 EP32 EP64 EP128 EP256 77 118 155 173 192 194 114 195 273 314 369 360 32× the GPUs: dispatch 77 → 194 (×2.5), combine 114 → 360 (×3.2) At EP256: one round trip per layer × 58 MoE layers ≈ 32 ms a step, exposed unless hidden (our derivation; production hides it under compute with two-batch overlap) Three interconnect domains 8-GPU machines + InfiniBand NVLink ~160 GB/s in a node, ~50 GB/s per GPU between nodes DeepSeek's EP144 runs on this, across 18 machines GB200 NVL72 72 GPUs in one NVLink domain SGLang: latency low enough to drop two-batch overlap Ironwood TPU 64-chip cubes, ICI 3D torus, 6 neighbors per chip Optical switches reach 9,216 chips; beyond that, DCN NVL72 isn't required Plain 8-GPU nodes work with one RDMA NIC per GPU, full mesh and GPU-initiated sends. Bigger domain: faster trips, less to hide.
Left x-axis: 32× the GPUs, only 2–3× the latency, but paid every layer.
Left: measured, H800, one 400 Gb NIC per GPU. Right: vendors' public specs.
Sources and assumptions

Left: DeepEP v1.2.1 README, low-latency kernel table (H800 NVLink ~160 GB/s; CX7 400 Gb/s IB ~50 GB/s; 128 tokens/batch, hidden 7168, top-8, FP8 dispatch, BF16 combine). Right: DeepSeek-V3 technical report §3.4; LMSYS 2025-06-16, “Deploying DeepSeek on GB200 NVL72 with PD and Large Scale EP (Part I)” (“the pure-NVLink solution … allows two-batch overlap to be disabled”); Google Cloud, “Inside the Ironwood TPU codesigned AI stack” (64-chip cube, 3D torus, 9,216 chips, DCN).

Our derivation: (194 + 360) µs × 58 layers ≈ 32 ms, an upper bound with no overlap. Of V3's 61 layers the first 3 are dense, leaving 58 MoE layers (config.json first_k_dense_replace 3). No bandwidth figures are given for NVL72 or ICI.

One more cost: a small batch can't fill the bus, yet pays the full transfer cost.

Why big domains like NVL72 suit inference

Whole rack as one NVLink domain for large EP: 2.7–4.8× per GPU vs. H100 large EP SGLang · GB200 NVL72 · DeepSeek-V3/R1 · 2K input · PD disaggregation, 48 decode GPUs; H100 = 1 H100 large EP Part I: high-prec. Part II: FP8 attention + NVFP4 experts Per-GPU throughput vs. H100 large-EP deployment H100 large EP Baseline 1× NVL72 · high precision decode, 7,583 tok/s per GPU 2.7× NVL72 · low precision prefill 3.8× NVL72 · low precision decode 4.8× Don't credit it all to large EP The multiples mix memory, precision, kernels and PD disaggregation. Decode throttled to fill up, then measured; one H100 baseline used simulated MTP, GB200 didn't.
Start at the dashed line, where H100 ends; the bottom bar is nearly 5× as long.
Both posts: per GPU vs. H100 large-scale EP.
Sources and assumptions

LMSYS 2025-06-16, “Deploying DeepSeek on GB200 NVL72 with PD and Large Scale EP (Part I)”: decode 7,583 tok/s per GPU, 2.7× H100 per GPU (2,000-token input; the H100 bar includes simulated MTP; decode saturated via the slow-down API, then released).

LMSYS 2025-09-25, “Deploying DeepSeek on GB200 NVL72 with PD and Large Scale EP (Part II)”: FP8 attention + NVFP4 experts, prefill 26,156 and decode 13,386 tok/s per GPU, 3.8× and 4.8× H100; 48 decode GPUs (“For decode, we use 48 ranks”).

The H100 baselines are the H100 PD + large-scale EP deployments each post cites; multiples are as reported, not broken down at equal precision.

Takeaway DEP copies no notes, so steps shorten; large-scale EP fills the expert bus and frees memory for notes.
The gain: SGLang, DEP both sides, 95.6% of the official target; vLLM to dep8, 2.09× per-GPU throughput (all GPUs).
But DEP needs per-device state; some models fall back to TP, whose copies eat memory at long context. Next: how do DCP and PCP differ?
Going deeper: Chinese full edition, §6.
Section 5

Long context: DCP vs. PCP

Section 4's gap: K3-like models can't run DEP; falling back to TP copies notes several times, so long contexts exhaust memory. The other problem: long prompts mean a long wait for the first token. Both fixes split the sequence; the names differ by one letter.

Don't go by the letters DCP, PCP: framework names, after each one's usual phase: D is decode, P prefill. CP is context parallel: one long sequence across devices.
The mechanism isn't phase-bound: Meta names them by what moves (pass-KV, pass-Q) and passes Q in prefill too when cache hits are high and new input short. We track only who moves: PCP passes notes, DCP the question.

Who moves: notes or question?

Attention is a table: rows are new tokens' questions (Q), columns earlier notes (K, V); looking only back, a triangle. Split it: one side moves.

PCP: blue notes circle the ring, questions stay. DCP: reversed; the yellow question flies to every device, answers combine at the end. (34 s, Manim; untick “Autoplay” to step. 4 devices, 4 segments: illustrative; ring load balancing not drawn.)

Same split: by token, each device holding its segment's KV. Only what travels differs.

Why opposite: smaller side moves

Combining: don't divide yet. Attention is a weighted average: devices send numerators and denominators; sum, then divide once.

The arithmetic: merging two devices' averages
Merging blockwise attention: add numerators, add denominators, divide only once, at the end Attention = score-weighted average. Three tokens: exp scores 2, 1, 1; values 10, 4, 20. Dev 0 holds the first two, Dev 1 the third Dev 0 Dev 1 ✗ ✓ Each device works on its own notes Dev 0: 2×10 + 1×4 Numerator 24 Denominator 3 Dev 1: 1×20 Numerator 20 Denominator 1 numerator = Σ weight×value, denominator = Σ weight (log of the denominator = the familiar LSE) Divide each, then average Dev 0: 24 ÷ 3 = 8 Dev 1: 20 ÷ 1 = 20 Average: (8 + 20) ÷ 2 = 14 ✗ Dev 1 has one token but half the vote Don't divide yet: once at the end Numerator: 24 + 20 = 44 Denominator: 3 + 1 = 4 44 ÷ 4 = 11 ✓ Same as all 3 tokens on one device Each device hands over its denominator with its result Like merging two classes' average scores: bring each class size; don't average the averages. Ring merges this way chunk by chunk en route; DCP merges at the end. Same trick.
Three tokens: the middle fails, the right checks out.
Like merging two class averages: you need each headcount.
Sources and assumptions

📌 Framing: Ring Attention appendix code (Liu et al., arXiv 2310.01889) sums numerator and denominator separately; numerically stable blockwise softmax: FlashAttention (arXiv 2205.14135).

⚠️ Our example: arbitrary numbers, computed from the definition, asserted.

Same 8-way split, different payoffs

CP at inference: TP copies the KV cache; DCP turns copies back into capacity DeepSeek-V3, one 128K-context request. KV per token = (512 + 64) × 61 layers × 2 bytes = 70,272 bytes Duplicates The one copy needed This request's KV on each of 8 devices TP 8 Dev 0 Dev 1 Dev 2 Dev 3 Dev 4 Dev 5 Dev 6 Dev 7 8.58 GiB/device Total 68.6 GiB (8 identical) TP 8 + DCP 8 Dev 0 Dev 1 Dev 2 Dev 3 Dev 4 Dev 5 Dev 6 Dev 7 1.07 GiB/device Total 8.58 GiB (exactly one) TP beyond the KV head count adds only copies MLA's KV has one head. With 8-way TP every device stores a full copy; 7 copies are wasted. DCP spreads KV round-robin by token over 8 devices, 1/8 each; it adds no devices, reusing the TP ones.
Top: eight equal blocks, one green, seven red. Bottom: eight slivers, one copy.
MLA has one KV head per layer; TP can't split it.
Sources and assumptions

📌 vLLM docs, “Context Parallel Deployment”: KV cache duplicated tp_size / H times; DeepSeek-R1 (MLA) has 1 KV head, so -tp 8 = 8× copies; add -dcp 8. KV sizes: V3's config.json, computed by this script.

When DCP wins and when it loses

DCP wins only at long context: 128K, concurrency 8: 365 vs 108, +237% Kimi K3 · GB300 · SGLang · TP8 across 2 nodes, 1K output. Y-axis: DCP8 ÷ TP8 output throughput (log scale); one line per concurrency Conc. 8 Conc. 1 Conc. 32 Conc. 64 Input length → DCP8 vs. TP8 0.6× 0.8× 1× 1.5× 2× 3× 4× DCP wins DCP loses 4K 8K 32K 128K Input length (tokens) 4K: all four lose Conc. 64: −7% Conc. 8: −9% Conc. 32: −11% Conc. 1: −26% Conc. 32: +24% −11% −17% Conc. 1: +12% Conc. 1 untested at 8K, 32K; dashed line joins ends +237% 365 vs 108 tok/s
See how high the highlighted concurrency-8 line climbs at 128K, then the four 4K points, all below 1×.
Y: output-throughput ratio (log); TP8 at its best tuned setting.
Sources and assumptions

gpu/inference/a4x-max/kimi-k3/SGLANG-K3-RUNBOOK.md §11.0.3 (:1010–1023, eight operating points), §11.0.0 wide table (:921–938; 4K c8/c32 computed here against TP8's best config, symm-mem on), :721–723 (logical KV 1.5M → 12.2M tokens), :1024–1026 (−26% = the cost of turning symm-mem off), :962 (symm-mem is unavailable under DCP).

Per-point ratios: 4K c1 −26%, c8 −9%, c32 −11%, c64 −7%; 8K c8 −11%, c32 +24%; 32K c8 −17%, c32 +7%; 128K c1 +12%, c8 +237%.

Throughput: output tok/s, warm runs; 32K c32 uses TP8 = 408.2 per §11.0.3 (the wide table's ratio-0.4 row gives 420: ratio 1.04, same conclusion). K3's MLA layers have one KV head; DCP doesn't split its 69-layer KDA state pool (same file, :725–726). The x-axis isn't converted to total KV: the source gives only input length and concurrency, not measured KV usage.

Before enabling: DCP doesn't stack on DEP

DCP covers only attention and adds no devices: on attention's TP, copies become shards. Experts can use EP or TP, so the usual setup is TEP + DCP (our K3 run: TP8 + DCP8, no expert EP). Today DCP borrows its degree from the attention TP group (at most TP degree ÷ KV heads); DEP attention has one device per group: nothing to borrow. EP is fine; DEP is not. It rescues exactly the models that can't run DEP and fall back to TP attention.

Takeaway Smaller side moves. PCP moves notes: compute-bound prefill gets its first token sooner. DCP moves the question, so memory-bound decode gains capacity; a net win only at long context, after per-layer communication.
The gain: K3 / GB300 / SGLang, TP8 over two machines, 128K in / 1K out, concurrency 8: DCP8 vs. tuned TP8, output 108 → 365 tok/s (+237%). Short inputs lose.
But prefill and decode still share devices; the more users, the more each long incoming prompt delays. Next: PD disaggregation adds a KV move; why does it serve more users?
Going deeper: Chinese full edition, §7.
Section 6

PD disaggregation: one more KV trip, yet more users?

Section 5's gap: prefill and decode still share devices; the fuller each one, the more users a long new prompt delays. Splitting them across machines costs one extra KV trip per request. So why more users?

Each job drags the other

Top, red box: a whole batch waits. Bottom, purple arrow: one extra trip. (22 s silent loop, Manim. Cells: relative length, not measured ms; KV transfer drawn shorter than prefill; scale below.)
One kitchen: prep and service Trip size isn't the problem; two jobs share one stove. Prefill preps the whole prompt at once; decode serves plate by plate, best several at a time. A big party orders, prep piles up, service halts.
Split: two stoves, prepped ingredients (KV) handed over once.

Why more users: each side at its own limit

Once split, 3 devices do the work of 7 colocated: neither side holds the other back Requests/s one device sustains under the same latency targets (DistServe paper example, 13B model on one A100) Both Prefill only Decode only Requests/s per device Colocated Both 1.6 req/s Both 1.6 req/s Both 1.6 req/s Both 1.6 req/s Both 1.6 req/s Both 1.6 req/s Both 1.6 req/s 7 devices ≈ 11.2 req/s Split prefill 5.6 req/s prefill 5.6 req/s decode 10 req/s 3 devices = min(2 × 5.6, 10) = 10 req/s Requests per device more than double; this is how to set the ratio Each side has its own ceiling: one prefill device handles 5.6, one decode device 10, so 2:1 keeps both about equally busy. Long-prompt workloads load prefill harder, so add prefill; long-answer workloads add decode.
Top: seven devices doing both. Bottom: three, one job each.
From Topic 5; rates: the paper's example.
Sources and assumptions

📌 DistServe (Zhong et al., arXiv 2401.09670), Sec. 1 and Fig. 1 example (13B model, one A100, 90% of requests within SLO): per-device goodput 1.6 / 5.6 / 10 rps under the same SLO; 2P1D: 3.3 rps per device, 2.1×. The authors' blog “Throughput is Not All You Need” uses the same numbers.

⚠️ Our derivation: 2P1D handles min(2 × 5.6, 10) = 10/s, ~3.3 per device; colocated, 10/s takes ⌈10 ÷ 1.6⌉ = 7 devices.

DistServe: up to 7.4× the unsplit per-device request rate (≥90% on target). Mooncake serves Kimi live; NVIDIA Dynamo 1.0 shipped March 2026. PD splitting is now standard.

What the extra trip costs

One KV transfer is just 5–10% of a prefill: the network isn't the bottleneck Our 1P1D on TPU v7 (Qwen3-Coder-480B), 8K-token prompt. KV transfer is a bandwidth estimate, not a timed measurement prefill (1–2 s) ① ② ③ Same time axis: 0 to 2 s 0 500 ms 1000 ms 1500 ms 2000 ms One prefill 1–2 s One KV transfer 10 + 83 + 10 ≈ 100 ms Zoomed 10×: 10 ms ① HBM → host (PCIe) 83 ms ② host → peer (DC network, 100 Gb/s) 10 ms ③ peer → HBM (PCIe) It can take the slow link because each request transfers only once This model has few KV heads (8) and stores KV in FP8: ≈1.04 GB for an 8K prompt. A much larger KV, or another network, means redoing the math. TP talks every layer, so it needs the fastest tier; one transfer per request can hide even across machines.
Long 1–2 s bar on top; thin three-color bar below.
KV time: bandwidth estimate, not a timing.
Sources and assumptions

📌 Our TPU v7 1P1D record (wiki qwen3-coder-480b-pd-disagg-tpuv7x-20260425): three legs, bandwidth-estimated, ~10 / 80 / 10 ms, ~100 ms total, 5–10% of prefill (hence prefill 1–2 s, inferred).

⚠️ Our derivation: KV = 2 × 62 layers × 8 KV heads × 128 × 8,192 × 1 byte ≈ 1.04 GB; 100 Gb/s = 12.5 GB/s → about 83 ms.

Thinner KV lets PD span datacenters: hybrid models send 1/4 to 1/13 the bytes of full attention PrfaaS paper, Table 3: KV egress per 8×H200 prefill machine (SGLang v0.5.9) at 32K input, Gb/s Full attn Hybrid (linear or SWA + a few full-attn layers) KV each prefill machine sends per second (Gb/s, 32K input) 0 20 40 60 80 100 Case study: whole prefill cluster averages 13 Gb/s egress = 13% of the link Qwen3-235B Full attention 33.35 Room for 2 Qwen3.5-397B 3 of every 4 layers linear 8.25 Room for 12 MiniMax-M2.5 Full attention 59.93 Room for 1 MiMo-V2-Flash 5 of every 6 layers local-only 4.66 Room for 21 Cross-cluster link ≈ 100 Gb/s total (shared)
Orange bars: room for only 1–2 machines. Green: 12–21.
KV/s from one 8×H200 at 32K input; “room for N” = 100 Gb/s ÷ per-machine egress (our derivation).
Sources and assumptions

📌 PrfaaS, arXiv 2604.15039 (Moonshot AI + Tsinghua): Φ_kv from Table 3. Case study: throughput +54%, P90 TTFT −64%, ~+15% at equal cost, +32% over naive heterogeneous serving without scheduling; length threshold 19.4K, ~half of requests sent out; average egress 13 Gb/s, 13% of the Ethernet link (§4.3.1, hence a ~100 Gb/s link). “Room for N” = ⌊100 ÷ per-machine egress⌋ (our derivation).

📌 Case study: in-house 1T hybrid model, 32 H200s for prefill + 64 H20s for local PD, vs. homogeneous PD on 96 H20s. “3 of every 4 layers linear” = the paper's 3:1 hybrid; “5 of every 6 layers local-only” = 5:1 sliding-window attention (SWA).

⚠️ The 4× and 13× pair models of similar size (the paper's words: “dense models of similar size”), not strict equal-parameter controls.

After the split: sharding, ratios, when not to split

Sharding: many narrow prefills, one wide decode Our V4-Pro on GB300 with SGLang, 16 machines, runs DEP on both sides (each attention device serves its own users; experts split by EP). They differ in width and replica count:
  • Prefill: 14 × dep4. 4-device groups on one machine, never talking. Compute-bound: one long prompt alone saturates it. Small groups, little cross-machine traffic; add replicas to serve more.
  • Decode: 1 × dep8, 8 devices on two machines. Movement-bound: needs a big batch to amortize each expert-weight read; large-scale EP gives each expert more users. DP attention: V4-Pro has 1 KV head, so each user's KV is stored once.
  • Wide needs feeding. Decode at dep16 left 12 prefill machines, too few; output per decode device fell ~40% (Section 4).
  • Not just us. SGLang's public GB200 NVL72 runs match: one 48-GPU decode instance (DP attention + EP48), 2–4-GPU prefill instances (Chinese full edition, §8.5).
  • New model, new mix. No DEP? Decode falls back to TEP + DCP (K3, Section 5). Slow first token? Prefill can add PCP. Both inferred from Sections 4–5, unused in our deployment.
An extra prefill machine buys a faster first token, not throughput: per-GPU output falls ~30% DeepSeek-V3, B200 (8 GPUs per machine), SGLang + NIXL; 1P1D (16 GPUs) → 2P1D (24 GPUs); input 3,000–4,000, output 400–600, concurrency 128 1P1D 2P1D Machine view: tokens per GPU spent Total output (tok/s) 1,118.8 1,159.0 +3.6% Per GPU (tok/s) 69.9 48.3 −31% User view: time to first token Mean TTFT (s) 2.40 1.73 −28% P99 TTFT (s) 12.07 6.23 −48% Before adding prefill, ask: are you short on first-token speed, or on tokens per second? The extra prefill lets queued requests start sooner; total output barely moves. Per-GPU throughput always makes 2P1D look bad: judge it on a TTFT SLO.
Left: total nearly flat, per GPU down ~30%. Right: both TTFTs fall.
Per GPU = total ÷ all GPUs; TTFT client-side, incl. queueing.
Sources and assumptions

📌 gpu/inference/a4/deepseek-v3/REPORT-2P1D-DeepSeek-V3.md: tables at :69, :85; comparison at :100–142 (per GPU = total output ÷ GPU count, the report's method, recomputed by script). c256: 1,798.1 → 1,775.6.

⚠️ TTFT is client-side, queueing included. Both runs used --disable-cuda-graph (REPORT-1P1D :129), so TPOT isn't shown.

Takeaway Splitting stops interruptions, not hauling; each side batches to its limit. The extra KV moves once per request, and one trip is cheap; each side shards its own way.
The gain: Mooncake's FAST '25 paper reports 115% (A800) / 107% (H800) more requests than prior systems.
Split or not, decode still emits one token per step. Next: several per trip?
Going deeper: Chinese full edition, §8–9.
Section 7

Speculative decoding: many tokens per trip?

Section 6's gap: split or not, decode makes one trip per token. Tiers needing fast tokens can't batch big; reasoning models write thousands of tokens. Can one trip yield several?

Guess, then verify: hits free, misses cheap

Step 1, red token: the miss becomes the big model's token; the rest is dropped. Step 2: a free token. (23 s, Manim; untick “Autoplay” to step. Schematic: greedy, temperature 0.)
Apprentice drafts, master grades The apprentice (small model) drafts a few tokens; the master (big model) checks them as one input, in one trip.
Matches stay; the first miss becomes the master's token; the rest go. Worst case: one token, as without it.
Cost of verifying more: dense part free, MoE experts not Left: V3 weight bytes per verify step (our derivation, FP8). Right: Kimi K3, measured; slower trip, far more tokens Shared by every token Routed experts Speculation on V3: GB read per step, verifying n tokens 0 60 120 180 GB 1 37.5 2 3 4 5 6 7 8 163.8 Tokens verified per step, n Verifying 8 = 4.4× verifying 1 Light blue: 17.1 GB at any n Kimi K3, measured: tokens per trip Spec off One trip 11.1 ms 1 token back Spec on One trip 16.1 ms 6.96 tokens back (avg.) 1.45× vs. off Amortized per token 11.1 ms → 2.31 ms, down to 1/4.81 A slower trip is fine: it brings back far more tokens
Left: dark blue grows with n as more experts are read. Right: each trip a bit slower, far more tokens.
Left: our derivation (V3, FP8, uniform routing; overestimate). Right: K3, GB300, one user, measured.
Sources and assumptions

V3 architecture from the HF deepseek-ai/DeepSeek-V3 config.json: hidden 7,168, moe_intermediate 2,048, 61 layers with the first 3 dense, 8 of 256 routed experts selected. Each expert has 3 × 7,168 × 2,048 ≈ 44.04M parameters.

⚠️ Our derivation: everything else = 671B − 256 × 58 × expert size ≈ 17.1B. That gives ≈ 37.5B active per token, within 2% of the technical report's 37B (external anchor).

⚠️ Our derivation: if n tokens pick experts independently and uniformly, one layer touches 256 × (1 − (1 − 8/256)^n) experts on average. Real routing is uneven and neighboring tokens often pick the same experts, so the dark-blue segment is an overestimate.

K3: gpu/inference/a4x-max/kimi-k3/SGLANG-K3-RUNBOOK.md:1089 (GB300, TP8 across two nodes, ISL 4096 / OSL 1024, temperature 0, bs=1, TPOT 11.1 → 2.31 ms), :979 (accept len 6.96, block 7).

⚠️ Our derivation: time per trip with speculation = 2.31 ms × 6.96 ≈ 16.1 ms; the inter-token interval drops to 1/4.81 (11.1 ÷ 2.31 ≈ 4.81). The runbook's end-to-end 4.22× includes time to first token, hence below 4.81. The 7th square on the right stands for the 0.96 in the average; schematic.

Note: K3 runs bs=1 on 8 GPUs, so a step also includes communication and inter-kernel gaps, not just weight reads; the “1.45×” trip time isn't all extra experts.

K3 on GB300, one user: trips slow 11.1 → ~16 ms (back-calculated) but carry 6.96 tokens on average: 4.81× faster per token. Optimistic (temperature 0, random prompts): official temperature-1.0 acceptance is ~4.73 code, ~2.61 creative; different engine, don't divide.

More users, smaller gain

Speculation's gain shrinks: with more users or longer context, less comes free Kimi K3 · GB300 · SGLang TP8. Bar = output throughput with speculation ÷ without; dashed line = 1, break-even 4K context, 1 → 64 users 32K context, 1 user Throughput with speculation ÷ without 1× 4.22× 1 user 2.86× 8 users 1.89× 32 users 1.74× 64 users 1.32× 32K · 1 user Same single user, context 4K → 32K: 4.22× → 1.32×
Bars shrink left to right: more users, less gain.
K3 output throughput on GB300 vs. no speculation, same setup. Rightmost: one user, 32K context.
Sources and assumptions

K3: gpu/inference/a4x-max/kimi-k3/SGLANG-K3-RUNBOOK.md:1089–1092 (GB300, TP8 across two nodes, ISL 4096 / OSL 1024, random, temperature 0, open loop, warm runs only; total output tok/s), :1057–1058 (32K: only concurrency 1 was measured; 8 / 32 were not, and the cluster has since been deleted).

Note: at concurrency 64, K3's TPOT with speculation is still only 40% of that without, so throughput and latency both improve here.

Sources for the other two related sets of measurements. B200: gpu/inference/a4/deepseek-v4/README.vllm.md:265–277 (DeepSeek-V4-Flash, 8× B200, vLLM v0.20.0, EP+DP=8, MTP num_speculative_tokens=2; acceptance rate about 54%; −3% to −37% across ten concurrency levels).

V4-Pro: gpu/inference/a4x-max/deepseek-v4/SGLANG-V4PRO-RUNBOOK.md:900–911 (steps 1 / draft 2 → steps 2 / draft 3; acceptance length 1.88 → 2.36 (+26%), implied step time 115.3 → 162.5 ms (+41%); decode peak −13%, end-to-end +28%). The B200 setup is an EP+DP high-throughput configuration; its conclusion doesn't carry over to TP-only low-latency configurations.

So: low-concurrency, interactive tiers first; big batches: measure first.

Drafts: ever nearer the big model

Where drafts come from: grown into the big model, or no model at all x-axis: first public release (arXiv v1; Prompt Lookup has no paper, so its GitHub repo date). Cards show only where the draft comes from Main: trained (model / head / layer) Side branch: no draft model Drafts grow ever closer to the big model 2022-11 Speculative decoding Separate small model drafts 2024-01 Medusa Extra decoding heads on big model 2024-01 EAGLE Small model reads big model's states 2024-12 MTP · DeepSeek-V3 +1 layer in pretraining; shares output head 2025-03 EAGLE-3 Fuses hidden states from several layers 2026-07 DSpark Short block per pass; confidence sets how many 2022 2023 2024 2025 2026 2023-11 Prompt Lookup No draft model: copy from the prompt
Left to right, drafts move step by step from a separate small model onto the big one.
Speedups omitted on purpose: each team measured its own models, so they don't compare.
Sources and assumptions

Dates are arXiv v1 (each abstract page checked on 2026-10-15). Speculative decoding: Leviathan et al. 2211.17192; Chen et al. 2302.01318; Medusa 2401.10774; EAGLE 2401.15077, EAGLE-2 2406.16858; MTP · DeepSeek-V3 2412.19437; EAGLE-3 2503.01840; DSpark 2607.05147. Prompt Lookup has no paper; its date is the creation of GitHub repo apoorvumang/prompt-lookup-decoding (2023-11, per the GitHub API; since integrated into vLLM and transformers). Medusa and EAGLE are both 2024-01, so they sit side by side on the axis.

Not in the figure: SpecInfer 2305.09781 arranges drafts as a tree; SuffixDecoding 2411.04975 copies from past outputs; diffusion drafters DiffuSpec 2510.02358, DFlash 2602.06036; Lookahead 2402.02057. P-EAGLE, PARD, LayerSkip and others are also left out.

Note: each method's speedup was measured by its authors on their own models and workloads, under different conventions; we don't compare them, so the figure doesn't show them.

Speedups use different baselines and models, so they don't compare; the Chinese full edition lists each (Section 11).

DSpark's fix: two cuts. Whole-block drafts start strong, fade fast; chat is inherently harder to guess than problem solving (paper data folded below). One cut each.

Paper data: which positions do one-by-one vs. whole-block drafts get right?
DSpark paper, measured: depth wins early, coherence later Left: accuracy at each position on its own; right: avg. tokens accepted per round EAGLE-3 (one by one) DFlash (whole block) DSpark ① Per-position acceptance: position 1 → later in block Qwen3-4B 0.5 0.6 0.7 0.8 0.9 1.0 Chat EAGLE-3 0.53 0.74 Chat DFlash 0.72 0.63 Code DFlash 0.87 0.78 Math Position 1 only EAGLE-3 0.81 DFlash 0.88 DSpark 0.93 Arrows: position 1 → later. One-by-one drafts firm up, whole-block drafts slip; at position 1 the shallow draft is weakest. ② Avg. tokens accepted per round Qwen3-8B, incl. the big model's 1 token 0 1 2 3 4 5 6 4.66 4.77 5.65 Math 4.15 4.46 5.28 Code 2.58 2.97 3.50 Chat Same draft: chat accepts ~2 fewer than math. How many to verify should depend on content and load. Early positions need a deep draft, later ones need coherence; DSpark gets both Block drafts stack deeper, so position 1 is sharper; the sequential head stops the slide. Low chat counts are why cut two sets how many to verify by confidence.
Left: see which way the arrows point. Right: see how short the chat group is.
All numbers measured in the DSpark paper; on the right, each category averages three benchmarks.
Sources and assumptions

DSpark: arXiv 2607.05147. Left: numbers from the text of §4.3.1 (Figure 2's curves aren't published point by point, so we draw only what the text states; the right end is the text's “deeper into the block”, no position named). Right: Table 1, Qwen3-8B; math = GSM8K / MATH500 / AIME25, code = MBPP / HumanEval / LiveCodeBench, chat = MT-Bench / Alpaca / Arena-Hard; each category averages its three benchmarks (our calculation).

Cross-check: over all nine benchmarks DSpark averages 26.7% above EAGLE-3 and 18.4% above DFlash, matching the paper's text. Temperature 1.0, block length 7, 5-layer draft (EAGLE-3: 1 layer).

Watch slot 2's score bars: same backbone, but with the head they track slot 1. (24 s, Manim; untick “Autoplay” to step. Scores schematic; 0.2–1.3% and 16–18% measured in the paper.)
Watch the same drafts: 14 admitted when idle, only 7 when busy. (26 s, Manim; untick “Autoplay” to step. Confidences and throughput table schematic; queueing and stop rule per the paper's Algorithm 1.)

EAGLE and MTP: same shape, different trainer

Inside EAGLE and MTP: both ends borrowed, only a small middle trained Orange boxes are trained; blue boxes are borrowed from the big model as is Trained Borrowed, frozen How data is joined EAGLE (2024-01): big model frozen, draft is a small add-on Last hidden vector already computed Next token already drawn Embedding borrowed Concat 2× width FC back to 1× 1 decoder layer next hidden vector LM head borrowed → token Draft Predicted vector + just-sampled token feed back to guess the next: one draft pass per token Why add the drawn token: the vector alone can't say which was sampled The 70B model's draft: 0.99B trainable params, A100 server, 1–2 days EAGLE-3 (2025-03): two changes Change 1: concat three layers, predict the token Shallow Middle Deep Concat 3× width FC back to 1× No vector target: decoder → borrowed LM head → token Change 2: from token 2 on, the draft's own output stands in Step 1 big model's how big model's can + I do Step 2 big model's how big model's can draft's I + do it “I” isn't verified yet, so there's no big-model vector for it; the draft's previous output fills in. So training feeds the draft its own outputs too: training-time test. MTP (since DeepSeek-V3): nearly the same skeleton, different trainer DeepSeek-V3's MTP module Main-model vector + next embedding Norm + proj. transformer block Output head shared w/ main When trained Big model EAGLE After release; anyone can add one frozen MTP In vendor pretraining; ships with it trained jointly Also MTP: Qwen3-Next, GLM-4.5, Xiaomi MiMo; Step 3.5 Flash has 3 modules (MTP-3) One MTP layer, chained: output fed back for another pass MiMo-7B, acceptance of draft tokens 1 / 2 / 3 70% 81% 1st 10% 56% 2nd 2% 36% 3rd As is (1 step ahead only) Chain-tuned (FastMTP) Similar shape; the difference is who trains it and whether it learned to chain EAGLE isn't training-free, just cheap to train; MTP can guess several steps ahead, but only if trained to chain, not by a flag change.
Orange is trained; blue is borrowed from the big model. Each panel trains only a small middle. Bottom right: chaining untrained (gray) vs. trained (green).
Structure per the papers' text; how / can / I / do / it is from the EAGLE-3 paper.
Sources and assumptions

EAGLE: arXiv 2401.15077 v3 §3 (Figure 6 description: embedding layer and LM head use the target model's parameters, untrained; FC + one decoder layer; features concatenated with the token embeddings advanced by one step to 2× width), §4 Training (70B: 0.99B trainable, 1–2 days on an A100 40G server). EAGLE-3: arXiv 2503.01840 v3 §3.1–3.2 (low / mid / high features concatenated to 3k, compressed back to k by an FC; from step 2 the draft output a stands in; training-time test); how / can / I / do / it is the example in the paper's Figure 5.

MTP: the MTP section of the DeepSeek-V3 technical report; adoption by other vendors per the MTP page of the architecture gallery on sebastianraschka.com and the FastMTP introduction. Acceptance rates: FastMTP arXiv 2509.18362 (MiMo-7B single-layer MTP; main model frozen, MTP head fine-tuned on self-distilled data; as is, about 10% at position 2 and about 2% at position 3).

Takeaway One weight trip yields every right guess, at worst one token. Biggest with few users; shrinks as more join.
The gain vs. no speculation (K3, GB300, one user): 4.81× faster per token; 4.22× end to end, first token (not sped up) included; 64 users: 1.74×.
With many users speculation helps less; each trip stays as heavy. Next section, a new lever: lighten the load.
Going deeper: Chinese full edition, §11.
Section 8

Quantization: fewer bytes, accuracy drop?

Section 7's gap: more users, less help from speculation, trips as heavy. Mostly we added riders; now: same bus, lighter cargo, fewer bits for weights and KV. Savings? Accuracy loss?

Halve the bits, halve the floor

Halve the bits, halve the per-step floor: FP4 is just over a quarter of BF16 DeepSeek-V3 · 8 TPU v7 chips · batch 1 · weights only (our derivation) BF16 FP8 MXFP4 NVFP4 Weights read per token (37B active params) BF16 16 bit 74.00 GB → ≥ 1.26 ms FP8 8 bit 37.00 GB → ≥ 0.63 ms MXFP4 4.25 bit 19.66 GB → ≥ 0.33 ms NVFP4 4.5 bit 20.81 GB → ≥ 0.35 ms Does the whole model fit? (671B, raw HBM) BF16 1,342 GB FP8 671 GB MXFP4 356 GB NVFP4 377 GB 8 chips: 1,649 GB
Left: each bar is shorter than the last. Halve the bits and the minimum step time halves too.
Ideal (our derivation): full bandwidth, no KV or communication; real steps are slower.
Sources and assumptions

V3: technical report, arXiv 2412.19437, abstract: 671B total parameters, 37B active per token. TPU v7: Google Cloud public specs, 192 GiB of HBM and 7.37 TB/s of bandwidth per chip. The right panel uses decimal GB throughout: 8 × 192 GiB = 1,536 GiB ≈ 1,649 GB (our derivation).

MXFP4: OCP Microscaling Formats (MX) Specification v1.0. NVFP4: NVIDIA technical blog, “Introducing NVFP4 for Efficient and Accurate Low-Precision Inference.”

⚠️ Our derivation: floor = active parameters × bits per parameter ÷ 8 ÷ (8 × 7.37 TB/s). Assumes weights spread evenly and bandwidth saturated; excludes KV, activations and communication, so real steps are slower.

Note: real deployments rarely take the whole model to FP4. DeepSeek-V4 puts only the experts in FP4 and keeps most of the rest in FP8 (HF model card).

Same cargo, packed tighter Same bus; each item drops from 16 bits to 8 or 4.
Few users: steps are nearly all weight reads; half the bytes, half the floor.

4-bit experts, attention 8+ bits

gpt-oss-120b's model card lists experts 114.71B, attention 0.96B, embedding + output 1.16B: experts ≈98%, attention under 1% (our derivation). OpenAI put only experts in MXFP4; it fits one 80 GB GPU.

Our TPU v7 runs: fitting on one machine R1, GLM-5.1: official FP8, experts only; one scale per output channel, rounded to FP4, no calibration data; converted once offline, reused.
First win: it fits. R1's FP8 experts take ~101.8 GB per device (R1 FP4 README); the usable 94.75 GiB is only ~101.7 GB, no room for activations or KV; FP4, ~60.9 GB: one four-chip machine runs it. Accuracy didn't collapse (R1 ≈95% on GSM8K), but without a same-machine FP8 baseline the drop is unknown.
Quantize only the experts: step bytes roughly halve, not drop to a quarter gpt-oss-120b · batch 1 · weight GB read per generated token (our derivation; excl. KV and activations) Experts (top-4 × 36 layers) Attention BF16 Output layer BF16 Only the expert format changes Experts BF16 7.2 1.9 1.2 Total 10.2 · experts 70% Experts FP8 3.6 1.9 1.2 Total 6.7 · experts 54% Experts MXFP4 1.9 1.9 1.2 Total 5.0 · experts 38% BF16 → MXFP4: ~2.06× fewer bytes per step; FP8 → MXFP4: ~1.34× The two fixed segments (blue, gray) match across all three bars; long context adds a KV segment on top.
Top to bottom: orange experts keep shrinking; blue and gray stay put.
gpt-oss-120b weight bytes per token, from config; one user, no KV.
Sources and assumptions

gpt-oss-120b config.json (hidden 2880, intermediate 2880, 128 experts with top-4 routing, 36 layers, 64 heads, 8 KV heads, head_dim 64, vocabulary 201,088); model card Table 1 (attention 0.96B, active 5.13B) as the cross-check anchor; MXFP4 at 4.25 bits per parameter (model card).

Computed from config, attention is 0.955B and active parameters 5.12B (plus about 0.01B for the router), matching the model card. Bytes, shares and ratios are our derivation and exclude KV, activations and any overhead beyond the scales.

Net: quarter-size experts save only about half the step: attention is then about 38% of its bytes (our derivation), plus KV at long context.

Accuracy: error crosses four gates; method sets the loss

Error crosses four gates; softmax doesn't recover it. Follow one rounding error, weight to token (sources: Chinese full edition, §12):

The method sets the loss. At the same 4 bits, plain rounding clearly trails searched scales:

Same 4 bits: how you pick the scales decides the loss AWQ paper's test · Llama-2-7B · WikiText-2 perplexity (further right is worse) 16-bit AWQ Round-to-nearest Perplexity scale 5.40 5.45 5.50 5.55 5.60 5.65 5.70 5.75 5.80 16-bit 5.47 unquantized AWQ 4-bit 5.60 searched scales RTN 4-bit 5.73 plain rounding extra perplexity vs 16-bit
Red sits far from gray; green recovers half: same 4 bits, different scales.
Lower perplexity is better; one model and dataset only.
Sources and assumptions

AWQ: arXiv 2306.00978v5, Llama-2-7B, WikiText-2 perplexity: FP16 5.47, RTN 5.73, AWQ 5.60 (INT4, one scale per group of 128 values). One model and dataset only.

Quantize the KV too

Half the bits for KV, but neither example saves a full half Bar = bytes per KV entry, before = 1; red dashed line = exact half; two models, not comparable Before After DeepSeek-V3.2: BF16 → FP8 BF16 FP8 ≈1.63× users per GB (our derivation) exact half The extra: unquantized RoPE, added scales, FP8 indexer keys DeepSeek-V4.1-Flash: FP8 → FP4 FP8 FP4 45% smaller exact half The shortfall is scales: one per 16 values, 2× FP8's scale bytes
Both green bars pass the red dashed line.
Two models, two schemes: compare only each one's savings.
Sources and assumptions

V3.2: FlashMLA README (“FP8 with scale,” the version before commit 2e5429f of 2026-09-30): per token per layer, 512 FP8 values + 4 FP32 scales + 64-dim BF16 RoPE = 656 B, plus a 132 B FP8 indexer key; BF16 = (512 + 64) × 2 + 132 = 1,284 B. Our derivation: 1,284 ÷ 788 ≈ 1.63, excluding weights and activations.

V4.1-Flash: in 2026-09 the model card said the main KV is stored in FP4 (one scale per 16 values), and FlashMLA released a kernel that reads it at the same time; but the README says this format is valid only for extra_k_cache, and the k_cache copy stays FP8. Per token: FP4 256 B data + 32 B scales = 288 B; FP8 512 B data + 16 B scales = 528 B; 45% smaller (our derivation).

Takeaway Saved: halve the bits, halve the floor; 4-bit experts, attention at 8+ bits: about half a step saved. Lost: most error dies en route, but softmax amplifies it, so protect attention; measure the loss.
The gain: vLLM's FP8 weights at best halved Llama 3 70B's inter-token latency on 2 H100s; expert-only quantization puts gpt-oss-120b on one 80 GB GPU.
Toolkit complete. Next, back to the trip: given a model and devices, where to start?
Going deeper: Chinese full edition, §12.
Section 9

Day one: which questions first?

Toolkit done. Back to Section 0: model, devices in hand, what first on day one? Eight questions: four on paper, then four at the knobs.

Eight day-one questions: set the yardstick, do the math, then pick techniques The same trip as Section 0: each question is pinned to the part it asks about Yardstick / launch Share the trip Move less Split and place Eight pins ingest prompt KV warehouse = HBM weights KV weights workstations = compute 1 2 3 4 5 6 7a 8 7b Math 1 What's the workload? 2 Which yardstick? 3 Minimum bytes per step? 4 How many sequences fit? Knobs 5 How to split attention? 6 Split PD? How many each? 7a Quantize? 7b Speculate? 8 Validated before launch?
On the bus: pins 3, 4, 7 on cargo and seats, 8 on the cab; all ask about that one trip.
Two roads: blue = share the trip, green = move less. Orange = split and place; gray = yardstick and launch.
Sources and assumptions

⚠️ Our synthesis: the order is calibrate, do the math, then tune techniques. How to answer each question, and how to judge the answer, are in the main text and the corresponding sections.

Set the yardstick, weigh the trip

Find the jam, pick a fix (⑤–⑦)

Order matters: same vLLM tests as Section 4: DEP first, 2.09× per device; tuning concurrency alone (256 → 512), just +28% (Chinese full edition, §10).

⭐ Eight questions in one: where is this trip stuck? Each fix shares it or lightens it.

Opener: software-only speedup?

Section 0 asked: same GB300, DeepSeek-V4-Pro and per-user speed; software alone, how many times launch-day per-GPU throughput?

Same GPUs, same model, same per-user speed: software alone gives 5.1× per GPU Left: GB300 · DeepSeek-V4-Pro · SGLang. Right: different model and GPUs, for MTP's order of magnitude only Day 0 June (MTP on) MTP layers Tokens/s per GPU (same interactivity tier) ~2,200 Day 0 2026-04 · no MTP ~11,200 June 2026-06 · MTP on ×5.1 our derivation Separate setup: 8 × B200 · R1 · 1 user 111 no MTP 198 1 layer 253 3 layers 239 5 layers drops Cost per token falls to about one-fifth (our derivation: all GPUs counted, GPU price unchanged) The blog doesn't say whether “per GPU” divides by decode GPUs or all GPUs; both numbers come from one chart, so the ratio holds. 5.1× is the whole software stack combined; the blog doesn't isolate MTP's share. The right panel is an order-of-magnitude reference only.
Left two bars: same GPUs, model, tier.
Right: other model and GPUs; MTP ballpark only.
Sources and assumptions

Left: PyTorch blog, 2026-06-23, “Serving DeepSeek-V4 on GB300 with SGLang: 5x Higher Throughput at the Same Interactivity Since Day-0” (“the June 2026 MTP curve delivers ~11,200 tok/s/GPU … versus ~2,200 tok/s/GPU for the Day-0 (April 2026) no-MTP curve”).

Our derivation: 11,200 ÷ 2,200 ≈ 5.1; cost 2,200 ÷ 11,200 ≈ 0.196.

Right: TensorRT-LLM tech blog, “Pushing Latency Boundaries: Optimizing DeepSeek-R1 Performance on NVIDIA B200 GPUs”: 8 × B200, 1k input / 2k output; without MTP 111 TPS/user; MTP 1: 198, MTP 3: 253, MTP 5: 239. Both model and GPUs differ from the left.

Don't multiply per-section speedups Orca's baseline was FasterTransformer; paging's already ran Orca. Each baseline is the one before: multiplying counts Orca twice. Only the 5.1× is like-for-like.
Takeaway Eight questions in one: where is this trip stuck? Set the yardstick, measure; the fix follows.
The gain: same GB300, V4-Pro and tier; software alone, 5.1× launch-day per-GPU throughput (our derivation).
Next gap: linear-attention layers keep one ever-rewritten sticky note (Section 4), so no prefix can be cut out; Qwen3.5 on tpu-inference still runs prefix caching off.
Quoted a speedup? Ask: against what, which tier, how many devices?
Going deeper: Chinese full edition, §15.

← Course overview · Previous: Topic 5 · Parallelism Strategies · Going deeper: Topic 6 full edition (L300) (Chinese)
Built by Courses/tools/topic06-build-L200.py; text in topic06_L200_s*.py.