Per row, tick every collective needed in one training step (possibly several). Fill all five rows, then click “Check answers”.
| Parallelism | AllReduce | AllGather | ReduceScatter | AllToAll | P2P |
|---|---|---|---|---|---|
| DP (data parallel)Full model per device, own data | |||||
| FSDP (ZeRO-3)Weights, grads, optimizer state sharded | |||||
| TP (tensor parallel)Layer matrices split across devices | |||||
| PP (pipeline parallel)Layers cut into stages | |||||
| EP (expert parallel)MoE experts on different devices |
| Parallelism | Answer | Where it's used |
|---|---|---|
| DP | AllReduce | Sums the gradients from every device each step and gives everyone a copy. all-reduce decomposes into reduce-scatter + all-gather, so ticking those two also counts |
| FSDP | AllGather + ReduceScatter | In the forward pass, all-gather each layer's weights just before using them, and again in the backward pass; reduce-scatter the finished gradients so each device keeps only its own shard |
| TP | AllReduce | Two per layer forward and two backward, summing the partial matmul results; with sequence parallelism these become all-gather + reduce-scatter, which also counts |
| PP | Point-to-point | Activations pass only between adjacent stages (gradients in the backward pass); no collective across all devices |
| EP | AllToAll | Two per layer: dispatch each token to the experts it picked, then bring the results back the same way |
Inference keeps only the forward pass. The table still holds, but the emphasis shifts: TP's few all-reduces per layer carry little data, so latency is what hurts; and EP doesn't have to use all-to-all—the published TPU inference work does the expert step with all-gather plus reduce-scatter (Chinese full edition, §6.4).
Training does a batch per step; inference, one token. The cost: each token hauls the whole load from HBM.
⚠️ Schematic timeline, boxes not to scale: prompt drawn as 6 KV slots, decode as 8 steps, for legibility.
📌 TTFT and TPOT follow vLLM's benchmark_serving: TTFT runs from sending the request to the first output token; TPOT is the mean time per output token, excluding the first. Section 2 covers how they're measured (with or without queueing, mean or p99).
⚠️ “Prefill yields the first token” is standard: its last position gives the first output token's distribution.
First published: Orca OSDI'22; GPTQ arXiv 2210.17323; speculative decoding, Leviathan et al., arXiv 2211.17192; GQA arXiv 2305.13245; PagedAttention blog 2023-06-20; Splitwise arXiv 2311.18677; RadixAttention arXiv 2312.07104; SARATHI arXiv 2308.16369; Mooncake arXiv 2407.00079; Meta context parallelism arXiv 2411.01783; SGLang v0.4 blog 2024-12-04 (DP attention), DeepSeek-V3 arXiv 2412.19437; ⑫ Qwen3-Next blog 2025-09-10 (earlier: H3 arXiv 2212.14052, Jamba 2403.19887); ⑬ Koi-4B arXiv 2610.12582, NVIDIA-Nemotron-Labs-Dory1.0-4B 2026-10-16 (earlier: Universal Transformers arXiv 1807.03819). End years: DeepSeek-V3.2-Exp technical report (DSA, 2025-09); SGLang HiCache blog 2025-09-10; NVIDIA Dynamo press release 2025-03-18; SGLang large-scale EP blog 2025-05-05; vLLM v0.10.2 release (DCP, 2025-09); EAGLE-3 arXiv 2503.01840; Inferact TPU megakernel blog 2026-09, vLLM DeepSeek-V4.1-Flash blog 2026-10-07 (FP4 KV). External drivers: OpenAI DevDay 2023-11-06, Google 2024-02-15; OpenAI 2024-09-12, DeepSeek-R1 arXiv 2501.12948. Dots are evenly spaced within a year and show order only.
| Debut | Bottleneck | Fix | Left behind | § |
|---|---|---|---|---|
| 2022 | Batch waits on the slowest | Continuous batching: on/off each step | KV caps seats; max-length reserve | 2 |
| 2023–2024 | KV wastes HBM; rows too fat | Paging; GQA/MLA slim; prefix reuse | Contexts keep growing | 3 |
| 2024–2025 | MoE scatters users over hundreds of experts | DEP + large EP: 100+ devices | Per-device state; long KV copied | 4 |
| 2024–2025 | Long KV won't fit; slow TTFT | DCP / PCP: split the sequence | Pays only for long contexts | 5 |
| 2023–2025 | Long prompts stall output | PD disaggregation: split the jobs | One more KV trip per request | 6 |
| 2022–2025 | Reasoning: thousands of tokens, one a step | Speculative decoding: several per step | Gains shrink with many users | 7 |
| 2022–2026 | Only bytes left to save | FP8 / FP4 quantization, fused kernels | Accuracy drop must be measured | 8 |
The same chip is flat out reading a prompt, mostly idle emitting tokens. The only difference: rows per pass.
📌 7168 = hidden_size in DeepSeek-V3's config.json. Ridge point 313 = v7 per-chip bf16 2,307 TFLOP/s ÷ HBM 7.37 TB/s (Google Cloud public specs; bf16 = half of FP8's 4,614, our derivation; same as Topic 2).
📌 Our derivation: ops 2·M·H²; bytes 2·(H² + 2·M·H) (weights, input read once, output written once, 2 bytes per bf16); intensity = M·H ÷ (H + 2M) = 313 at M ≈ 343.
⚠️ Weights only, intensity is just M and the crossover exactly 313 (Topic 5's fig5-decode-ai convention); adding input and output moves it to 343.
⚠️ Deliberately left out: attention (it reads KV, so its intensity doesn't grow with users; Section 5) and MoE expert layers (see this section's figure on rows per expert).
37 GB ÷ 7.37 TB/s ≈ 5 ms; compute 0.32% used (ideal lower bound). Faster compute won't help; bandwidth sets the floor.
📌 Our derivation: KV = 70,272 bytes/token (MLA stores 512 + 64 values per layer, bf16, 61 layers, config.json) × context. At 128K: 9.21 GB = 8.58 GiB, same as Topic 5.
📌 Our derivation: weights read per step, b users = non-routed + 653.9B routed-expert params × [1 − (1 − 8/256)^b] (FP8, 1 B each). b = 1: 37 GB exactly; large b → 670.5 GB, i.e. 671B.
⚠️ At 64 users each pays 9.14 GB, not 37 ÷ 64: users pick different experts, so a step reads 87% of them.
⚠️ Uniform routing, equal contexts; activation and comms bytes not counted. FP8 KV halves the purple.
📌 256 routed experts, 8 picked per token: DeepSeek-V3 config.json (n_routed_experts, num_experts_per_tok). MXU side of 256: public JAX source jax/_src/pallas/mosaic/tpu_info.py, mxu_column_size = 256 for TPU 7.
📌 Our derivation: expert matrix 7168×2048; at M = 8 the intensity = 2·8·7168·2048 ÷ [2·(7168·2048 + 8·(7168 + 2048))] ≈ 8. Dense layer's 239: this section's first figure.
⚠️ 8 is an average: uneven routing gives some experts dozens of rows, some none. Shared experts see every token, unaffected.
The 313 line holds only per whole matmul; to fill every expert, a step needs 9,000+ tokens a step (Chinese full edition, Section 3, for H100).
Riders don't board or leave together: one asks and goes, another wants 3,000 words. Scheduling sets how full a trip runs, seats set the cap, and fuller means slower per rider.
Orca (OSDI 2022), first idea: stop after every step and look again. Now called continuous batching; vLLM and SGLang both do it.
One seat per request the KV pool can hold. Per-device throughput ≈ riders ÷ TPOT (time between tokens); full seats cap it.
Add seats? TPOT won't stay put:
Measured: GB300, DeepSeek-V4-Pro, SGLang, mem-fraction-static 0.85 → 0.88; requests per rank 886 → 999, TPOT 77.7 → 85.3 ms, end-to-end throughput 10,614 → 10,704 (SGLANG-V4PRO-RUNBOOK.md:849–858, 885–894).
Our derivation: 0.8% = 10,704 ÷ 10,614 − 1; ceiling = 999/886 × 77.7/85.3 − 1 ≈ 2.7%, assuming every new seat is filled and TPOT is as measured.
Weights are already amortized; newcomers add only KV, lengthening every step.
InferenceX, three points: gpu-tpu-pedia gpu/inference/a4x-max/deepseek-v4/VLLM-V4PRO-RUNBOOK.md:683–695 (GB300 Dynamo + vLLM, V4-Pro FP4, 8K/1K, output ÷ decode GPU).
PyTorch blog, 2026-06-23, “Serving DeepSeek-V4 on GB300 with SGLang: 5x Higher Throughput at the Same Interactivity Since Day-0”: ~11,200 tok/s/GPU at roughly 50 tok/s/user; the Day-0 curve without MTP is about 2,200.
⚠️ Public material doesn't settle whether 11,200 is output ÷ decode GPUs or (in + out) ÷ all GPUs (SGLANG-V4PRO-RUNBOOK.md:745). We plot the former; if the latter, the point moves down.
Measured in this repo, 9,168: SGLANG-V4PRO-RUNBOOK.md:596 (conc600, TPOT 59–61 ms), 723–725. The x-coordinate 1000 ÷ 60 ≈ 17 is our derivation.
Goodput: DistServe (arXiv 2401.09670).
Seats are KV: each request keeps a notebook, one line per token. Seats = KV pool ÷ notebook thickness. Three steps seat more: thinner lines, no wasted paper, no redoing.
Does thinning cost accuracy? The GQA paper converted T5-XXL to 8 groups: about 5.4× faster decoding (our derivation), average 47.1 vs. 47.2 originally. Only T5-XXL on TPU v4.
Sources: KV per token: V3 (512+64)×61×2 B = 70,272 (config.json); Llama-3.1-70B 2×8×128×80×2 = 327,680 (HF config.json); Qwen3.5 full attention 15 layers × 2 heads × 256 dims × K, V × 2 B = 30,720; linear-layer state 45×64×128×128×4 B = 180 MiB (HF config.json, mamba_ssm_dtype float32).
Assumptions: the MHA row is a hypothetical at V3's shape (128 heads × 128 dims, Topic 3's first problem): 3,997,696 B/token.
Our derivation: one v7 chip appears to software as two devices, each managing its own HBM, so we account per device. All V3 weights at FP8, 1 B/parameter: 671 GB ÷ 16 devices = 41.9 GB = 39.1 GiB. Measured usable HBM is 94.75 GiB per device (tpu-inference R1 FP4 README); a further 10% is held back for activations (our assumption). KV pool = 94.75 GiB × 0.9 − 39.1 GiB = 46.2 GiB. A rough estimate with every weight sharded; under DP attention each device holds its own copy of the non-expert weights, so the real pool is smaller. We assume attention runs data-parallel (each device stores only its own sequences); if attention runs TP, MLA's KV is replicated on every device (Topic 5).
Not counted: activations and communication buffers (folded into the 10%), the linear layers' convolution state, and the engine's actual state dtype (may not be fp32).
Fewer riders per trip, each reading more KV per step. Long context pays on both.
Source: TTFT 13 s → 0.5 s, from the DeepSeek API docs' “Context Caching on Disk” announcement (api-docs.deepseek.com/news/news0802, 2024-08; original wording “128K prompt with high reference”).
Source: 608B input tokens and the 56.3% hit rate are from DeepSeek open-infra-index 202502OpenSourceWeek day_6 (window 2025-02-27 12:00 to 02-28 12:00 UTC+8; 342B of 608B hit; V3 and R1 combined).
Our derivation: 13 ÷ 0.5 = 26; 100% − 56.3% = 43.7%. The two numbers come from two documents at two points in time, not from one measurement. Prices are not shown.
Too big for one device, V3 runs on several. Experts go EP either way; attention is the fork: one shared batch, or own users per device? This decides how many copies of each notebook exist.
📌 TensorRT-LLM definitions: TEP<N> shards both attention (TP) and experts (EP) across N ranks; DEP<N> keeps attention data-parallel (ADP) while distributing experts across N ranks.
⚠️ Expert-step communication as usually implemented: with TP attention every device holds the same tokens, so expert outputs are combined by all-reduce; with data-parallel attention the tokens differ, so it takes an all-to-all. Illustrative: 4 devices, 8 experts.
Real models: MLA and V4-Pro have 1 KV head: TEP8 makes 8 copies. Qwen3.5-397B, 32 query heads on 2 KV heads: 4 copies. GQA with 8 KV heads: none, so DEP gains nothing.
V4-Pro (1 KV head), our own SGLang setup on GB300: prefill and decode on separate machines (why: Section 6), DEP on both, 14 prefill nodes at dep4, decode at dep8. 10,704 tok/s per decode GPU end to end: 95.6% of the official target (the PyTorch blog's ~11,200 per GPU; denominator unstated, we read it as decode GPUs).
SGLang deployment: gpu/inference/a4x-max/deepseek-v4/SGLANG-V4PRO-RUNBOOK.md:33–34 (14 prefill × dep4 + dep8 decode, 16 nodes in one NVL72 domain), scripts/prefill-dep4.sh, decode-dep8.sh (--enable-dp-attention, EP 4/8); the three measured rows :851–858 (ISL 4096, 14 streams × concurrency 1050, #1/#3/#7).
Corroboration: VLLM-V4PRO-RUNBOOK.md:298–302 (num_key_value_heads 1), :464–465, :476 (concurrency 512: TP4 decode output 4,220, time between tokens 46.8 ms; dep8 11,031 and 11.8 ms; 16 and 20 GPUs).
Our derivation: capacity-ceiling multiple = TP ÷ min(TP, KV heads); the top half is illustrative, and bar lengths are not drawn to measured request counts. +54% = 10,614 ÷ 6,887, +0.8% = 10,704 ÷ 10,614; 2.09× = (55,153 ÷ 20) ÷ (21,100 ÷ 16), where 55,153 and 21,100 are total throughput from the same table (input + output; 4K in, 1K out, so roughly 5× the output alone), 1.31× = (11,031 ÷ 8) ÷ (4,220 ÷ 4).
Section 1's gap: only 8 rows per expert on average. Fix: spread the experts beyond one 8-GPU machine (large-scale EP; TensorRT-LLM's line: EP above 8). Attention stays per-device: DEP scaled up.
DeepSeek's production decode: EP144 on 18 machines; 256 experts + 32 redundant = 288 copies; per GPU, 2 routed + 1 shared.
Left: DeepEP v1.2.1 README, low-latency kernel table (H800 NVLink ~160 GB/s; CX7 400 Gb/s IB ~50 GB/s; 128 tokens/batch, hidden 7168, top-8, FP8 dispatch, BF16 combine). Right: DeepSeek-V3 technical report §3.4; LMSYS 2025-06-16, “Deploying DeepSeek on GB200 NVL72 with PD and Large Scale EP (Part I)” (“the pure-NVLink solution … allows two-batch overlap to be disabled”); Google Cloud, “Inside the Ironwood TPU codesigned AI stack” (64-chip cube, 3D torus, 9,216 chips, DCN).
Our derivation: (194 + 360) µs × 58 layers ≈ 32 ms, an upper bound with no overlap. Of V3's 61 layers the first 3 are dense, leaving 58 MoE layers (config.json first_k_dense_replace 3). No bandwidth figures are given for NVL72 or ICI.
One more cost: a small batch can't fill the bus, yet pays the full transfer cost.
LMSYS 2025-06-16, “Deploying DeepSeek on GB200 NVL72 with PD and Large Scale EP (Part I)”: decode 7,583 tok/s per GPU, 2.7× H100 per GPU (2,000-token input; the H100 bar includes simulated MTP; decode saturated via the slow-down API, then released).
LMSYS 2025-09-25, “Deploying DeepSeek on GB200 NVL72 with PD and Large Scale EP (Part II)”: FP8 attention + NVFP4 experts, prefill 26,156 and decode 13,386 tok/s per GPU, 3.8× and 4.8× H100; 48 decode GPUs (“For decode, we use 48 ranks”).
The H100 baselines are the H100 PD + large-scale EP deployments each post cites; multiples are as reported, not broken down at equal precision.
Section 4's gap: K3-like models can't run DEP; falling back to TP copies notes several times, so long contexts exhaust memory. The other problem: long prompts mean a long wait for the first token. Both fixes split the sequence; the names differ by one letter.
Attention is a table: rows are new tokens' questions (Q), columns earlier notes (K, V); looking only back, a triangle. Split it: one side moves.
Same split: by token, each device holding its segment's KV. Only what travels differs.
Combining: don't divide yet. Attention is a weighted average: devices send numerators and denominators; sum, then divide once.
📌 Framing: Ring Attention appendix code (Liu et al., arXiv 2310.01889) sums numerator and denominator separately; numerically stable blockwise softmax: FlashAttention (arXiv 2205.14135).
⚠️ Our example: arbitrary numbers, computed from the definition, asserted.
📌 vLLM docs, “Context Parallel Deployment”: KV cache duplicated tp_size / H times; DeepSeek-R1 (MLA) has 1 KV head, so -tp 8 = 8× copies; add -dcp 8. KV sizes: V3's config.json, computed by this script.
gpu/inference/a4x-max/kimi-k3/SGLANG-K3-RUNBOOK.md §11.0.3 (:1010–1023, eight operating points), §11.0.0 wide table (:921–938; 4K c8/c32 computed here against TP8's best config, symm-mem on), :721–723 (logical KV 1.5M → 12.2M tokens), :1024–1026 (−26% = the cost of turning symm-mem off), :962 (symm-mem is unavailable under DCP).
Per-point ratios: 4K c1 −26%, c8 −9%, c32 −11%, c64 −7%; 8K c8 −11%, c32 +24%; 32K c8 −17%, c32 +7%; 128K c1 +12%, c8 +237%.
Throughput: output tok/s, warm runs; 32K c32 uses TP8 = 408.2 per §11.0.3 (the wide table's ratio-0.4 row gives 420: ratio 1.04, same conclusion). K3's MLA layers have one KV head; DCP doesn't split its 69-layer KDA state pool (same file, :725–726). The x-axis isn't converted to total KV: the source gives only input length and concurrency, not measured KV usage.
DCP covers only attention and adds no devices: on attention's TP, copies become shards. Experts can use EP or TP, so the usual setup is TEP + DCP (our K3 run: TP8 + DCP8, no expert EP). Today DCP borrows its degree from the attention TP group (at most TP degree ÷ KV heads); DEP attention has one device per group: nothing to borrow. EP is fine; DEP is not. It rescues exactly the models that can't run DEP and fall back to TP attention.
Section 5's gap: prefill and decode still share devices; the fuller each one, the more users a long new prompt delays. Splitting them across machines costs one extra KV trip per request. So why more users?
📌 DistServe (Zhong et al., arXiv 2401.09670), Sec. 1 and Fig. 1 example (13B model, one A100, 90% of requests within SLO): per-device goodput 1.6 / 5.6 / 10 rps under the same SLO; 2P1D: 3.3 rps per device, 2.1×. The authors' blog “Throughput is Not All You Need” uses the same numbers.
⚠️ Our derivation: 2P1D handles min(2 × 5.6, 10) = 10/s, ~3.3 per device; colocated, 10/s takes ⌈10 ÷ 1.6⌉ = 7 devices.
DistServe: up to 7.4× the unsplit per-device request rate (≥90% on target). Mooncake serves Kimi live; NVIDIA Dynamo 1.0 shipped March 2026. PD splitting is now standard.
📌 Our TPU v7 1P1D record (wiki qwen3-coder-480b-pd-disagg-tpuv7x-20260425): three legs, bandwidth-estimated, ~10 / 80 / 10 ms, ~100 ms total, 5–10% of prefill (hence prefill 1–2 s, inferred).
⚠️ Our derivation: KV = 2 × 62 layers × 8 KV heads × 128 × 8,192 × 1 byte ≈ 1.04 GB; 100 Gb/s = 12.5 GB/s → about 83 ms.
📌 PrfaaS, arXiv 2604.15039 (Moonshot AI + Tsinghua): Φ_kv from Table 3. Case study: throughput +54%, P90 TTFT −64%, ~+15% at equal cost, +32% over naive heterogeneous serving without scheduling; length threshold 19.4K, ~half of requests sent out; average egress 13 Gb/s, 13% of the Ethernet link (§4.3.1, hence a ~100 Gb/s link). “Room for N” = ⌊100 ÷ per-machine egress⌋ (our derivation).
📌 Case study: in-house 1T hybrid model, 32 H200s for prefill + 64 H20s for local PD, vs. homogeneous PD on 96 H20s. “3 of every 4 layers linear” = the paper's 3:1 hybrid; “5 of every 6 layers local-only” = 5:1 sliding-window attention (SWA).
⚠️ The 4× and 13× pair models of similar size (the paper's words: “dense models of similar size”), not strict equal-parameter controls.
📌 gpu/inference/a4/deepseek-v3/REPORT-2P1D-DeepSeek-V3.md: tables at :69, :85; comparison at :100–142 (per GPU = total output ÷ GPU count, the report's method, recomputed by script). c256: 1,798.1 → 1,775.6.
⚠️ TTFT is client-side, queueing included. Both runs used --disable-cuda-graph (REPORT-1P1D :129), so TPOT isn't shown.
Section 6's gap: split or not, decode makes one trip per token. Tiers needing fast tokens can't batch big; reasoning models write thousands of tokens. Can one trip yield several?
V3 architecture from the HF deepseek-ai/DeepSeek-V3 config.json: hidden 7,168, moe_intermediate 2,048, 61 layers with the first 3 dense, 8 of 256 routed experts selected. Each expert has 3 × 7,168 × 2,048 ≈ 44.04M parameters.
⚠️ Our derivation: everything else = 671B − 256 × 58 × expert size ≈ 17.1B. That gives ≈ 37.5B active per token, within 2% of the technical report's 37B (external anchor).
⚠️ Our derivation: if n tokens pick experts independently and uniformly, one layer touches 256 × (1 − (1 − 8/256)^n) experts on average. Real routing is uneven and neighboring tokens often pick the same experts, so the dark-blue segment is an overestimate.
K3: gpu/inference/a4x-max/kimi-k3/SGLANG-K3-RUNBOOK.md:1089 (GB300, TP8 across two nodes, ISL 4096 / OSL 1024, temperature 0, bs=1, TPOT 11.1 → 2.31 ms), :979 (accept len 6.96, block 7).
⚠️ Our derivation: time per trip with speculation = 2.31 ms × 6.96 ≈ 16.1 ms; the inter-token interval drops to 1/4.81 (11.1 ÷ 2.31 ≈ 4.81). The runbook's end-to-end 4.22× includes time to first token, hence below 4.81. The 7th square on the right stands for the 0.96 in the average; schematic.
Note: K3 runs bs=1 on 8 GPUs, so a step also includes communication and inter-kernel gaps, not just weight reads; the “1.45×” trip time isn't all extra experts.
K3 on GB300, one user: trips slow 11.1 → ~16 ms (back-calculated) but carry 6.96 tokens on average: 4.81× faster per token. Optimistic (temperature 0, random prompts): official temperature-1.0 acceptance is ~4.73 code, ~2.61 creative; different engine, don't divide.
K3: gpu/inference/a4x-max/kimi-k3/SGLANG-K3-RUNBOOK.md:1089–1092 (GB300, TP8 across two nodes, ISL 4096 / OSL 1024, random, temperature 0, open loop, warm runs only; total output tok/s), :1057–1058 (32K: only concurrency 1 was measured; 8 / 32 were not, and the cluster has since been deleted).
Note: at concurrency 64, K3's TPOT with speculation is still only 40% of that without, so throughput and latency both improve here.
Sources for the other two related sets of measurements. B200: gpu/inference/a4/deepseek-v4/README.vllm.md:265–277 (DeepSeek-V4-Flash, 8× B200, vLLM v0.20.0, EP+DP=8, MTP num_speculative_tokens=2; acceptance rate about 54%; −3% to −37% across ten concurrency levels).
V4-Pro: gpu/inference/a4x-max/deepseek-v4/SGLANG-V4PRO-RUNBOOK.md:900–911 (steps 1 / draft 2 → steps 2 / draft 3; acceptance length 1.88 → 2.36 (+26%), implied step time 115.3 → 162.5 ms (+41%); decode peak −13%, end-to-end +28%). The B200 setup is an EP+DP high-throughput configuration; its conclusion doesn't carry over to TP-only low-latency configurations.
So: low-concurrency, interactive tiers first; big batches: measure first.
Dates are arXiv v1 (each abstract page checked on 2026-10-15). Speculative decoding: Leviathan et al. 2211.17192; Chen et al. 2302.01318; Medusa 2401.10774; EAGLE 2401.15077, EAGLE-2 2406.16858; MTP · DeepSeek-V3 2412.19437; EAGLE-3 2503.01840; DSpark 2607.05147. Prompt Lookup has no paper; its date is the creation of GitHub repo apoorvumang/prompt-lookup-decoding (2023-11, per the GitHub API; since integrated into vLLM and transformers). Medusa and EAGLE are both 2024-01, so they sit side by side on the axis.
Not in the figure: SpecInfer 2305.09781 arranges drafts as a tree; SuffixDecoding 2411.04975 copies from past outputs; diffusion drafters DiffuSpec 2510.02358, DFlash 2602.06036; Lookahead 2402.02057. P-EAGLE, PARD, LayerSkip and others are also left out.
Note: each method's speedup was measured by its authors on their own models and workloads, under different conventions; we don't compare them, so the figure doesn't show them.
Speedups use different baselines and models, so they don't compare; the Chinese full edition lists each (Section 11).
DSpark's fix: two cuts. Whole-block drafts start strong, fade fast; chat is inherently harder to guess than problem solving (paper data folded below). One cut each.
DSpark: arXiv 2607.05147. Left: numbers from the text of §4.3.1 (Figure 2's curves aren't published point by point, so we draw only what the text states; the right end is the text's “deeper into the block”, no position named). Right: Table 1, Qwen3-8B; math = GSM8K / MATH500 / AIME25, code = MBPP / HumanEval / LiveCodeBench, chat = MT-Bench / Alpaca / Arena-Hard; each category averages its three benchmarks (our calculation).
Cross-check: over all nine benchmarks DSpark averages 26.7% above EAGLE-3 and 18.4% above DFlash, matching the paper's text. Temperature 1.0, block length 7, 5-layer draft (EAGLE-3: 1 layer).
EAGLE: arXiv 2401.15077 v3 §3 (Figure 6 description: embedding layer and LM head use the target model's parameters, untrained; FC + one decoder layer; features concatenated with the token embeddings advanced by one step to 2× width), §4 Training (70B: 0.99B trainable, 1–2 days on an A100 40G server). EAGLE-3: arXiv 2503.01840 v3 §3.1–3.2 (low / mid / high features concatenated to 3k, compressed back to k by an FC; from step 2 the draft output a stands in; training-time test); how / can / I / do / it is the example in the paper's Figure 5.
MTP: the MTP section of the DeepSeek-V3 technical report; adoption by other vendors per the MTP page of the architecture gallery on sebastianraschka.com and the FastMTP introduction. Acceptance rates: FastMTP arXiv 2509.18362 (MiMo-7B single-layer MTP; main model frozen, MTP head fine-tuned on self-distilled data; as is, about 10% at position 2 and about 2% at position 3).
Section 7's gap: more users, less help from speculation, trips as heavy. Mostly we added riders; now: same bus, lighter cargo, fewer bits for weights and KV. Savings? Accuracy loss?
V3: technical report, arXiv 2412.19437, abstract: 671B total parameters, 37B active per token. TPU v7: Google Cloud public specs, 192 GiB of HBM and 7.37 TB/s of bandwidth per chip. The right panel uses decimal GB throughout: 8 × 192 GiB = 1,536 GiB ≈ 1,649 GB (our derivation).
MXFP4: OCP Microscaling Formats (MX) Specification v1.0. NVFP4: NVIDIA technical blog, “Introducing NVFP4 for Efficient and Accurate Low-Precision Inference.”
⚠️ Our derivation: floor = active parameters × bits per parameter ÷ 8 ÷ (8 × 7.37 TB/s). Assumes weights spread evenly and bandwidth saturated; excludes KV, activations and communication, so real steps are slower.
Note: real deployments rarely take the whole model to FP4. DeepSeek-V4 puts only the experts in FP4 and keeps most of the rest in FP8 (HF model card).
gpt-oss-120b's model card lists experts 114.71B, attention 0.96B, embedding + output 1.16B: experts ≈98%, attention under 1% (our derivation). OpenAI put only experts in MXFP4; it fits one 80 GB GPU.
gpt-oss-120b config.json (hidden 2880, intermediate 2880, 128 experts with top-4 routing, 36 layers, 64 heads, 8 KV heads, head_dim 64, vocabulary 201,088); model card Table 1 (attention 0.96B, active 5.13B) as the cross-check anchor; MXFP4 at 4.25 bits per parameter (model card).
Computed from config, attention is 0.955B and active parameters 5.12B (plus about 0.01B for the router), matching the model card. Bytes, shares and ratios are our derivation and exclude KV, activations and any overhead beyond the scales.
Net: quarter-size experts save only about half the step: attention is then about 38% of its bytes (our derivation), plus KV at long context.
Error crosses four gates; softmax doesn't recover it. Follow one rounding error, weight to token (sources: Chinese full edition, §12):
The method sets the loss. At the same 4 bits, plain rounding clearly trails searched scales:
AWQ: arXiv 2306.00978v5, Llama-2-7B, WikiText-2 perplexity: FP16 5.47, RTN 5.73, AWQ 5.60 (INT4, one scale per group of 128 values). One model and dataset only.
--kv-cache-dtype in both vLLM and SGLang. Kimi K3's high-throughput recipe requires it: BF16 KV for 128 requests per device group doesn't fit.V3.2: FlashMLA README (“FP8 with scale,” the version before commit 2e5429f of 2026-09-30): per token per layer, 512 FP8 values + 4 FP32 scales + 64-dim BF16 RoPE = 656 B, plus a 132 B FP8 indexer key; BF16 = (512 + 64) × 2 + 132 = 1,284 B. Our derivation: 1,284 ÷ 788 ≈ 1.63, excluding weights and activations.
V4.1-Flash: in 2026-09 the model card said the main KV is stored in FP4 (one scale per 16 values), and FlashMLA released a kernel that reads it at the same time; but the README says this format is valid only for extra_k_cache, and the k_cache copy stays FP8. Per token: FP4 256 B data + 32 B scales = 288 B; FP8 512 B data + 16 B scales = 528 B; 45% smaller (our derivation).
Toolkit done. Back to Section 0: model, devices in hand, what first on day one? Eight questions: four on paper, then four at the knobs.
⚠️ Our synthesis: the order is calibrate, do the math, then tune techniques. How to answer each question, and how to judge the answer, are in the main text and the corresponding sections.
Order matters: same vLLM tests as Section 4: DEP first, 2.09× per device; tuning concurrency alone (256 → 512), just +28% (Chinese full edition, §10).
⭐ Eight questions in one: where is this trip stuck? Each fix shares it or lightens it.
Section 0 asked: same GB300, DeepSeek-V4-Pro and per-user speed; software alone, how many times launch-day per-GPU throughput?
Left: PyTorch blog, 2026-06-23, “Serving DeepSeek-V4 on GB300 with SGLang: 5x Higher Throughput at the Same Interactivity Since Day-0” (“the June 2026 MTP curve delivers ~11,200 tok/s/GPU … versus ~2,200 tok/s/GPU for the Day-0 (April 2026) no-MTP curve”).
Our derivation: 11,200 ÷ 2,200 ≈ 5.1; cost 2,200 ÷ 11,200 ≈ 0.196.
Right: TensorRT-LLM tech blog, “Pushing Latency Boundaries: Optimizing DeepSeek-R1 Performance on NVIDIA B200 GPUs”: 8 × B200, 1k input / 2k output; without MTP 111 TPS/user; MTP 1: 198, MTP 3: 253, MTP 5: 239. Both model and GPUs differ from the left.
← Course overview ·
Previous: Topic 5 · Parallelism Strategies ·
Going deeper: Topic 6 full edition (L300) (Chinese)
Built by Courses/tools/topic06-build-L200.py; text in topic06_L200_s*.py.