Five collectives, then five cuts: data, weights, experts, sequence, the work itself. Each fills an earlier gap.
There's also an instructor script.
It covers how to teach: goals, pauses, misconceptions, likely questions. The content is here.
Presenting: lines, figure cues and timing are scripted. Self-study: give the script to an AI for a TA that knows the material and how to teach it.
Section 9 sources every number; ours say “our derivation.”
Question 1: In standard mixed-precision AdamW, each parameter keeps five items resident in HBM. Bytes for each?
| Weights (the copy for compute) | bytes |
| Gradients | bytes |
| Master weights (the optimizer's copy) | bytes |
| First moment m | bytes |
| Second moment v | bytes |
2 + 2 + 4 + 4 + 4 = 16 bytes. The first two are half precision (bf16), for the matmuls; the last three full precision (fp32), for the optimizer. Times V3's 671B params: 9.76 TiB (~10,000 GB), before activations.
16 bytes is the standard count, with bf16 gradients. In practice frameworks often keep the gradient accumulation buffer in fp32 (Megatron's default; V3 too): 18 bytes.
Preview (think first; answers in §2.2):
Topic 4's bill: 671B params × 16 bytes (weights 2 + gradients 2 + optimizer state 12) = 9.76 TiB (~10,000 GB) of resident training state. Top devices hold 200–300 GB, so storage alone takes 30–60 devices before any compute. So cut. But along which dimension?
📌 V3 full training: 2.788M H800 GPU hours (DeepSeek-V3 technical report, arXiv 2412.19437, abstract; covers pre-training, context extension and post-training); 2,048 H800s (sec. 3.1).
📌 B300: 288 GB per GPU (NVIDIA DGX B300 user guide: 8 × 288 GB); TPU v7: 192 GiB HBM3e per chip.
⚠️ Our derivation: 9.76 TiB = 671B × 16 bytes; 318 years = 2.788M ÷ 8,760 hours; 57 days = 2.788M ÷ 2,048 ÷ 24.
Tensors can be cut along batch, sequence, hidden, layer or expert. The whole lecture is one idea:
The lecture is a relay: cut, pay one communication; find the gap, cut again.
Two rulers per cut: how often it talks (→ which link) and FLOPs per byte moved (→ stall or not; Section 3).
Token and batch: a token is the smallest unit of text, about a word piece; a batch is a step's input, often counted in tokens here.
Tensor and hidden dimension: tensors are a model's multidimensional arrays; a token's vector is hidden-dimension long (7,168 in V3).
Activations: each forward layer's intermediate results, kept for backward.
Attention head: attention runs as dozens of small, independent attentions in parallel, each a head.
Micro-batch: one step's batch cut into a few pieces processed one at a time.
V3: DeepSeek-V3, the 671B-parameter MoE model, our running example.
Q, K, V: every token computes three vectors: Q is “what I'm looking for,” K is “what I am,” V is “what I carry.”
A new token compares its Q with every earlier token's K and sums their V, weighted by similarity.
Query heads and KV heads: each head has its own Q; K and V may have fewer, several query heads sharing one KV group (at the extreme, one KV head). Inference caches K and V, so the KV-head count decides whether the cache splits by head.
Softmax: attention scores every earlier token; softmax normalizes those scores into weights that sum to 1.
MLA: DeepSeek's attention; K and V shrink to one small latent all heads share: far smaller KV cache, but only “one head” to split.
Node: one machine, usually 4 to 8 GPUs joined by a high-speed interconnect (NVLink on NVIDIA).
HBM: on-device fast memory; the model and intermediates must sit here.
Compute and bandwidth: operations vs. bytes moved, per second. Moving much and computing little is bandwidth-bound; the reverse, compute-bound.
ICI and slices: ICI is TPU chips' dedicated fast link; chips joined by ICI form a slice; slices talk via the regular data-center network (DCN).
Devices and axes: software sees one v7 chip as 2 devices, and parallelism degree counts devices; a slice's chips form a 3D grid whose x, y, z directions are the axes.
Rank: one process in the parallel job, usually one GPU (on TPU, one device).
v7 and v7x: two names for the same TPU generation (Ironwood); both appear here.
N-way: “8-way TP” means TP over groups of 8 devices. TP4 × DP2 means 4-device TP groups, 2 of them doing data parallelism: 4 × 2 = 8 devices.
MoE and experts: many parallel feed-forward blocks (experts); a small router picks a few per token, the rest idle.
The parameter count is large, but each token touches only a small fraction.
KV cache: in generation, each processed token leaves a K and V that later tokens attend to;
it is kept throughout, growing with context.
Prefill and decode: inference's two phases. Prefill reads the full prompt at once for token one; decode emits one per step.
Decode steps are light, so requests batch up: read weights once, use them across the batch; bigger pays more.
Each later cut adds a communication; meet them now. Five see daily use: all-reduce, all-gather, reduce-scatter, all-to-all, plus point-to-point send/receive. Each answers: who sends to whom, and is arriving data concatenated, summed or kept as is?
A collective: devices sending and receiving under one shared rule. Figure conventions:
Device 0 is the root, the hub of all traffic.
All four funnel through the root's link; more devices, more jam. Training runs on the next group.
Naively (the root serving each in turn), broadcast and reduce pile all traffic on the root, worse with every device. Libraries chain them as a relay, so the root moves one copy. Gather and scatter can't: the root holds n distinct pieces; the total stays flat, but it all crosses its one link.
“All” = everyone gets it: all-gather = gather for all, all-reduce = reduce for all, reduce-scatter = reduce, a block each; all-to-all: no sum, no concat.
Other work fits in between; later free wins start here: ZeRO (Section 2), tensor + sequence parallelism (Section 3).
⚠️ AGRS / RSAG are our shorthand for the order, not standard terms.
With a root, device 0 receives and sends n−1 copies each: a bottleneck. Instead, link devices head to tail in a ring, no root.
The ring = reduce-scatter + all-gather. Another build shows why it wins:
📌 The two assemblies: NVIDIA NCCL documentation, Collective Operations (AllReduce = Reduce + Broadcast = ReduceScatter + AllGather).
⚠️ Our derivation: naively rooted, device 0 sends n−1 copies; on the ring each sends 2(n−1)/n. Libraries use a chain or tree; we compare only the naive forms.
The price: steps grow with devices. For small messages, per-step waits dominate, the ring loses; libraries pick by size.
Large messages are bandwidth-bound, where the ring is near optimal; small ones by per-step waits, where many steps hurt. So libraries like NCCL pick ring, tree, etc. by message size. The NVLink switch (NVSwitch) can also sum in-switch, nearly halving each device's sends.
TPUs are more direct: ICI itself is a torus. In big enough slices (every dimension a multiple of 4) each is a closed ring; in smaller ones it's a line, the ring doubles back, and bandwidth roughly halves.
The torus adds more: each chip sits on three rings, X, Y, Z. Gathering X, then Y, then Z keeps one direction busy per step, the last heaviest. Start three data chunks on different axes (XYZ / YZX / ZXY) and all three stay saturated: the same all-gather is up to 3× faster counting bandwidth alone. Reduce-scatter runs in reverse (big step first); all-reduce's halves likewise. ⚠️ Our derivation: per-step latency ignored; XLA's chunking/scheduling unchecked.
📌 How to Scale Your Model, sharding chapter: gathering over several axes at once “increases the available bandwidth by a factor of N_axes”.
⚠️ Our derivation: bandwidth only, per-step latency ignored; equal bandwidth on all axes, every dimension a closed ring (slice dims multiples of 4). XLA's chunking and scheduling unchecked.
⚠️ Illustrative: token counts between devices are hand-picked, not measured routing; in real V3 each token picks 8 experts.
Expert parallelism (Section 4) uses the general case: unequal counts, same-route combine. Ulysses (Section 5) uses the equal-size case: a transpose.
Of eight, the four rooted are building blocks; five see daily use (send/receive: Section 3's layer cut).
| Collective | Does | Sends | Used by |
|---|---|---|---|
| AllReduce | Sum; all get it | 2(n−1)/n · S | Data-parallel grad sync; TP: 2× fwd, 2× bwd per layer |
| ReduceScatter | Sum; one block each | (n−1)/n · S | FSDP bwd grads; TP + sequence parallel |
| AllGather | Concat; all get it | (n−1)/n · S | FSDP fwd weights; TP + sequence parallel; Q in KV-split inference |
| AllToAll | Just moves data | (n−1)/n · S | Expert-parallel dispatch and return; Ulysses |
| Broadcast / Reduce | Fan-out / fan-in | S | Initial weights out, metric sums, etc. |
| Send / Recv | One-to-one | Varies | Pipeline stages pass activations; Ring Attention passes KV |
Ring: reduce-scatter, 3 steps of 1/4 copy; all-gather, 3 more: 1.5 copies. Rooted: device 0 sends 3. More devices, wider gap.
The simplest cut: each device trains on different samples. At first it doesn't fix “it doesn't fit”; only FSDP does, at a price.
Every device: full model, own samples. After backward, one all-reduce averages gradients; all update. One exchange per step: rare but big (pure DP at V3's size: ~2.7 TB per device per step). Each still stores 16 bytes per parameter: still too big.
Analogy: V3's weights are a library of 61 reading rooms, one per layer, across V3's 128 branches (one per device). DP gives every branch a full copy; below, we strip duplicates.
Three pieces: weights (the model), gradients (each parameter's move this step), optimizer state (largest: master weights plus two moments per parameter). Ψ = parameter count.
Per ZeRO's paper, optimizer state is what stays resident across steps, checkpointed. Gradients, recomputed and zeroed each step, count separately (own ZeRO stage). Megatron sets accumulator precision (--main-grads-dtype) with master weights and moments (“precision-aware optimizer”), so some count it there.
📌 Source: Rajbhandari et al., ZeRO, arXiv 1910.02054 secs. 5, 7. Per the paper, Pos and Pos+g communicate as much as DP (2Ψ); Pos+g+p at most 1.5×. Ψ = parameter count, in elements.
⚠️ Per-device bytes computed and asserted by our script; 9.76 TiB as in Topic 4 §5.1 (671e9 × 16 ÷ 1024⁴). 128-way = V3's 2,048 devices ÷ 16 pipeline stages (report, arXiv 2412.19437 sec. 3.2: PP16; the division is our derivation). Resident only, no activations.
Warm-up Q2: optimizer state is 12 of 16 bytes, so it goes first. It is also the idlest:
ZeRO-1 shards optimizer state, ZeRO-2 adds gradients, ZeRO-3 weights; each device keeps only its 1/n. ZeRO-1 also saves compute: each updates its 1/n, not all parameters.
Why are the first two stages free? The §1.4 identity:
📌 ZeRO paper, arXiv 1910.02054 sec. 7: Pos and Pos+g communicate as much as DP (2Ψ). Elementwise Adam: Kingma and Ba, arXiv 1412.6980 Algorithm 1.
⚠️ With micro-batches, only ZeRO-1 is strictly free: ZeRO-2 keeps no full gradient, so can't accumulate first; it reduce-scatters after each micro-batch. So V3 and Megatron both pair ZeRO-1 with pipeline parallelism (PP, which needs micro-batches).
📌 V3 report, arXiv 2412.19437 sec. 3.3.3: BF16, not FP32, tracks AdamW's two moments “without incurring observable performance degradation”; master weights, batch-accumulation gradients stay FP32. Sec. 4.2: β₁ = 0.9, β₂ = 0.95.
📌 Not the default: optimizer state normally matches the parameter precision; Megatron-LM must explicitly enable --use-precision-aware-optimizer plus --exp-avg-dtype bf16 --exp-avg-sq-dtype bf16 (flags in megatron/training/arguments.py; without the former, megatron/core/optimizer/optimizer_config.py asserts both must be fp32). PyTorch AdamW's default β₂ is 0.999.
⚠️ Our derivation: real bf16 ticks and rounding (8 sig. bits, round-to-nearest). V3 tested its FP8 recipe (incl. bf16 moments) vs. BF16: rel. loss error < 0.25% at 16B, 230B; no bf16-moment ablation.
After two stages, 2 bytes of weights per parameter remain: 1.29 TiB per device for V3 at 128-way (~1,415 GB; a device has 200–300 GB). Too big. More devices can't push ZeRO-2 lower: what's left is full weights. Cutting data alone, large models need ZeRO-3, PyTorch's FSDP (Fully Sharded Data Parallel): all-gather each layer before use, discard after.
Back to the library. FSDP cuts blind to content: a room's books laid page by page in a strip, cut into 128 equal lengths, one per branch. Cuts often land mid-book: one book's tail, the next's start. Alone, a piece is unreadable. To use a room, a branch copies the other 127 (all-gather), reads, discards the copies. After backward, each piece's edits merge at its owner (reduce-scatter).
📌 V3 config: technical report, arXiv 2412.19437 sec. 3.1–3.2 (2,048 H800s; PP 16, EP 64 over 8 nodes, ZeRO-1 DP); config.json (61 layers, first 3 dense, 256 routed + 1 shared expert).
⚠️ Our derivation: DP 128 = 2,048 ÷ 16. V3 uses ZeRO-1, not FSDP; we borrow its degrees. PyTorch FSDP v1 flattens a layer's parameters into one strip, split evenly; FSDP2 shards each parameter on dim 0. Either way a lone shard can't compute.
EP (Section 4) deals whole shelves, each a self-sufficient expert; FSDP's fragments serve even communication, useless until reassembled.
Communication: 3Ψ, 1.5× DP. In return, per-device resident memory drops 9.76 TiB → 78.11 GiB (~84 GB, before activations): already over V3's 80 GB H800. The data cut alone isn't enough; hence the next cuts.
DP talks once a step: slow links work. FSDP talks every layer: fast lane only.
FSDP moves weights each step, an amount set by parameter count, not batch; compute scales with each device's tokens.
The global batch can't grow forever (too large hurts learning): more devices, fewer tokens each. Small batches cut compute, not traffic; weight movement dominates. How else to cut, adding devices without each computing less?
Versus the next cut: FSDP splits the model in form, but at heart is DP—weights are whole at compute time, each device on its own batch. The next cut keeps them split while computing.
ZeRO-1 and ZeRO-2 are free (all-reduce = RS + AG anyway; the local update between sends nothing). Optimizer state: largest and idlest, used only to update.
FSDP rebuilds each layer's weights before computing; a big per-device batch hides that traffic, a small one can't. The second cut splits the weights instead: devices share tokens, so adding devices doesn't thin them out. Three walls: small batch stalls FSDP → TP; TP can't leave one machine → PP; PP leaves devices idle.
Once data arrives, how often is it used? Often: the transfer hides. Once: the device waits. As a ratio, FLOPs per byte moved over the network must beat the hardware's FLOP/s ÷ bytes/s, or the device idles.
FSDP's math is clean (⚠️ our derivation, dense model): ~6Ψ bytes per step (bf16 weights gathered in forward and backward, gradients scattered once, ~2Ψ each) for 6ΨT FLOPs (Topic 4: ~6 FLOPs per parameter per token; T = tokens per device). The ratio is exactly T, whatever the model size. Intuition: each arriving 2-byte parameter is used once per local token: one multiply-add, 2 FLOPs.
Analogy: a door workshop. Craftsmen are devices; the runner, the network, carries tools (planers, jigs = weights) and door panels (activations, one per token, passed between stages). FSDP ships tools: each tool works every panel at the bench, more panels, better trips.
⚠️ Our derivation, not from a paper: FSDP moves ≈ 6Ψ bytes per step (2 bf16 weight AGs + 1 bf16 gradient RS) for 6ΨT FLOPs; TP does 4 all-reduces per layer, each sending ≈ 4Th bytes, computing 72h²T/n. Both assume dense layers, perfect overlap.
📌 v7: 2,307 TFLOP/s bf16 per chip. ICI 1,200 GB/s is the send + receive total over 6 links; we use 600 GB/s per direction (on a ring each device receives what it sends: 600 too). HBM's 7.38 TB/s is 12×+ the network, so the network binds (spec table in Inferact's TPU megakernel blog and wiki ici-dcn, both citing Google's TPU7x docs; the 6-link split and the 600 per direction are our derivation). One axis only: hardware line ~3× higher. V3's hidden size of 7,168 is from config.json.
⚠️ The denominator should be all-gather busbw measured on your own network (nccl-tests on GPUs; = data size ÷ time × (n−1)/n). The thin dashed line assumes 70% of nominal, lifting the threshold to about 5,493; the 70% is not a measured v7 number.
Red dashed line ≈ 3,845: below that many tokens per device per step, FSDP stalls. A v7 chip is 2 devices: ~7,700 per chip.
Mind the denominator: your measured bandwidth, not the spec sheet. Threshold = FLOP/s ÷ weight bytes actually gathered per second. Run a real all-gather (nccl-tests on GPUs): its busbw = size ÷ time × (n−1)/n excludes each device's own shard, so it equals actual receive bytes/s per device. At an assumed 70% of nominal, it rises 3,845 → ~5,493 (thin dashed line); an FSDP axis with few links, shared with other axes, goes higher.
Red dashed line: 3,845 = v7's 2,307 TFLOP/s (bf16) per chip ÷ ~0.6 TB/s received per chip (the usual 1.2 TB/s counts send + receive, equal on a ring; ⚠️ this split is our estimate, see sources). The limit is the network: arriving weights are also read from HBM, but at 7.38 TB/s (12×+ the network) that line is ~313 (gray dashed, bottom), not binding. This is the optimistic line; one axis: ~3× higher. MoE on FSDP alone (no EP) moves all parameters, computes only the chosen ones: another ~18× (V3's 671B ÷ 37B).
async_op=True returns a handle, then work.wait().limit_all_gathers caps AGs in flight.forward_prefetch / backward_prefetch).TP (tensor parallelism) cuts weight matrices; a device holds and computes a piece. NVIDIA Megatron-LM:
📌 Source: Shoeybi et al., Megatron-LM, arXiv 1909.08053 sec. 3 (MLP and self-attention splits; f / g communication operators).
Two confusions. First, every device has a share of both: device 0 has half of UP and of DOWN, not one matrix each (that's splitting by layer, PP). Nothing moves between them; DOWN's partial sums are added once, moving activations. Attention too: split by head, add at the exit. So per layer: two all-reduces forward, two backward.
Second, “horizontal”/“vertical” is just drawing. We draw W1 as input × output (X·W1), so an output-dim cut looks vertical; PyTorch's nn.Linear stores output × input, so it looks horizontal. Remember: first cut on output, second on input. Megatron, drawing X·A, calls them ColumnParallelLinear and RowParallelLinear.
The order can't flip: a nonlinear GeLU sits in between:
📌 Shoeybi et al., Megatron-LM, arXiv 1909.08053 sec. 3: GeLU is nonlinear: a row-split first matrix must sync before it. f: identity fwd, all-reduce bwd; g: reverse.
⚠️ Our counterexample: 2 and −3 are arbitrary values; GeLU is computed with the exact formula.
SP splits TP's leftover small ops by sequence: same traffic, less activation memory. Not a fourth cut, despite the name (§8.8).
TP can't split the small per-token ops (normalization, dropout, residual add), so there every device holds full activations. Megatron's sequence parallelism (SP) splits these by sequence, and each all-reduce in two: all-gather to rejoin before the TP block, reduce-scatter to split after. By §1's identity, traffic is unchanged.
With TP, the runner carries panels (activations): craftsmen share panels, each doing part, then combine. Double the panels, and handoffs and work both double: batch cancels. Panel size is hidden size: bigger panels, more work per handoff. Tools can come early (next layer's during this one); fresh panels by default wait for the stage to end: TP traffic is hard to hide. What's left: FLOPs per byte ≈ 4.5 × hidden ÷ TP degree (⚠️ our derivation, dense layers). V3 (hidden 7,168) at 8-way TP: ~4,032. The drawn red line is most optimistic, so 8-way seems to touch it (7,168 × 4.5 ÷ 3,845 ≈ 8.4); the real line is higher, with 8-way below it (see below).
Small batch leans on TP, large batch on FSDP: left, blue is below red, so use orange; right, it has crossed, so FSDP. This assumes TP's line clears red; when both sit on or below it, as for V3, TP doesn't pay (§3.7).
Large models often use both: they cut different dimensions. Grid the devices: rows are TP groups, columns FSDP groups. TP slices weights into output-dim strips; FSDP splits each on the input dim.
📌 Scaling Book (How to Scale Your Model), training chapter, “Combining FSDP and Tensor Parallelism”: In[B_X, D_Y] · W_in[D_X, F_Y] · W_out[F_Y, D_X], with FSDP on the X axis and TP on Y. The book moves activations over the TP axis with AllGather + ReduceScatter, the same volume as one AllReduce.
⚠️ Our derivation: a row shares tokens, so FLOPs per byte on the FSDP axis = the token count of that row; 961 = 3,845 ÷ 4. The axes use different links, so each threshold must be recomputed from its own links' measured bandwidth, and can only go up.
Per layer: all-gather in the column to rebuild your strip (1/TP of the matrix), compute alone, add partial sums once in the row. Backward reduce-scatters gradients along columns to owners.
The math shifts: a row shares tokens but each device gathers only a 1/TP strip, so the FSDP threshold counts a whole row's tokens: ~961 per device at 4-way TP (⚠️ our derivation; measured per-axis bandwidth not applied). Hence “small batch leans on TP”: TP lends a row's tokens to FSDP. The price: TP syncs every layer, needing the fastest links.
Many machines, slow links, no per-layer weight traffic like FSDP: split by layer. TP cuts wide, PP deep:
Stages pass activations only at boundaries, point-to-point: the least traffic of any cut, so PP can cross slow links. Naively: “4 devices' memory, 1 device's speed”, one stage working at a time. So feed in micro-batches, production-line style. Planing, drilling, painting: one station each; panels flow in stacks. At first later stations wait; at the end earlier ones idle. That idle time is the bubble:
📌 Bubble fraction: Narayanan et al., arXiv 2104.04473 sec. 2.2 (bubble time fraction, GPipe / 1F1B); interleaving further divides the bubble by v, the virtual stages per device. Zero Bubble: arXiv 2401.10241. DualPipe: github.com/deepseek-ai/DualPipe.
Bubble ÷ ideal compute = (p−1) ÷ m (p stages, m micro-batches). More help, but too-small ones underfeed the device.
Every row idles the same total: p−1 forwards + p−1 backwards. Original GPipe keeps each micro-batch's activations until its backward; at fixed micro-batch size, more of them store more. 1F1B caps those in flight at p: same bubble, less memory. Oddly, PP splits weights, yet per-device activations barely shrink: each stage holds several micro-batches.
VPP gives each device non-adjacent layer chunks: bubble ÷ chunks per device. Zero Bubble splits backward into input/weight gradients; the latter can wait to fill gaps. DeepSeek-V3's DualPipe feeds both ends of the pipeline, mainly to hide EP traffic under paired forward–backward compute (tech report sec. 3.2.1).
Micro-batches weren't invented for pipelines. They began as gradient accumulation: a replica that can't fit its batch share at once runs it piece by piece in time, accumulating gradients, then updates once. Pipelines borrowed this to keep stages busy. Three levels: global batch = micro-batch size × count × DP replicas. With PP, “per-device batch” is just bookkeeping: the stages form one replica on the same data. In MaxText, global batch = per_device_batch_size × device count; micro-batch = global batch ÷ num_pipeline_microbatches (must be a multiple of the stage count; defaults to it).
GPipe's extra memory: a precondition. At fixed total batch, finer splits don't change its activations (more, smaller micro-batches). The problem is a fixed micro-batch size: more of them mean a bigger batch, more activations. 1F1B's depend only on stages × micro-batch size.
How 1F1B is scheduled. Stage k (from 0) runs p−1−k warm-up forwards, then alternates one backward, one new forward (one out, one in), then drains the remaining backwards. So the first stage holds at most p micro-batches, the last 1. Why not have stage 0 send more while waiting? Steady state finishes one micro-batch per forward + backward, and a stage-0 round trip takes about p such periods. In flight = rate × time ≈ p. Extras just queue: more memory, no speed.
Uneven memory across stages. First stage holds p, last 1. Fixes: unequal layers in first/last stages (the last also has output layer, loss); recompute only early stages; park activations in host memory; or a V-shaped placement (each device owns an early and a late stage) to flatten activation peaks (Qi et al., Pipeline Parallelism with Controllable Memory, arXiv 2405.15362, Zero Bubble's sequel).
How big a micro-batch. Not “one that fills HBM”: under 1F1B, stage 0 holds p at once. Pick the smallest that fills one stage's compute. An LLM sequence is thousands of tokens, matrices are big; one or two often do; measure the knee. p = 4, 64 sequences per replica: 16 each → 4 micro-batches, bubble 3/4 of ideal time; 1 each → 64, just 3/64. Rule: micro-batches ≥ 4× stages; else VPP, fewer stages, or Zero Bubble-style ones.
What DualPipe costs. Tech report Table 2: each device stores two parameter copies (stage i forward, stage i from the end in reverse); peak activations p+1 micro-batches (1F1B: p). Aim: hide EP's ~1:1 cross-node traffic, trim the bubble. V3 can: wide EP leaves few weights per device. Chimera (2021) proposed bidirectional pipelines; DualPipe needs only even stage and micro-batch counts.
PP is for scale over slow links. Where the fast interconnect is big enough, skip it: a TPU slice puts 1,000+ chips on high-speed ICI—room for FSDP and TP—with DP between slices. So MaxText's ici_pipeline_parallelism and dcn_pipeline_parallelism default to 1; typical configs skip PP and its headaches. GPUs need PP because TP can't leave NVLink; slow cross-node links go to PP.
Our running example, DeepSeek-V3, uses no TP; per its report, memory optimizations made costly TP unnecessary. Nearly all parameters sit in narrow expert matrices; splitting them shrinks pieces, not activation traffic. Cut along experts instead: the third cut.
FSDP: FLOPs per byte moved = tokens per device; few tokens, transfer can't hide. TP syncs every layer; panels pass only after the stage ends, so it can't hide and must stay on the fastest links.
MoE parameters sit almost all in the experts; the third cut puts different experts on different devices. It adds one kind of traffic, all-to-all, but brings the worst problem of any cut.
Apart from its first 3 layers, V3's other 58 are MoE: 256 routed experts (8 picked per token) plus 1 shared expert all tokens use.
📌 config.json: hidden_size 7,168, moe_intermediate_size 2,048, n_routed_experts 256, num_hidden_layers 61, first_k_dense_replace 3. 671B = main-model total, no MTP module. “Rest” = 671B − routed experts, computed by this script.
256 routed experts × 58 layers ≈ 653.9B parameters, ~97% of the total. TP targets one wide matrix; V3's bulk is many narrow ones, best handed out whole to devices: move the patients (tokens), don't split the doctors (experts). V3's real config: PP16 + EP64, no TP (last line).
Section 2's library: EP deals a room's 256 expert shelves to 64 branches, 4 whole shelves each; PP cuts the 61 rooms into 16 ordered stages. Whole units, usable on arrival, not FSDP's shred-and-rebuild.
Expert parallelism (EP) communicates only in MoE layers, twice per layer, via §1.6's all-to-all, the pattern networks dread most: dispatch sends each token to its expert's device; combine sends results home.
V3 tames both: at most 4 machines per token, one copy each, fanned out on arrival; dispatch in FP8, combine in BF16.
📌 DeepSeek-V3 tech report, arXiv 2412.19437: sec. 2.1.2, node-limited routing (machines ranked by top-2 affinity sum); sec. 3.2.2, IB to the target machine's same-index device, then NVLink forwarding, NVLink 160 / IB 50 GB/s, 13 experts; sec. 3.3.3, FP8 dispatch, BF16 combine; sec. 4.2, M = 4, 64 devices, 8 machines.
⚠️ Our derivation: 28.7 KB = 4 × 7,168 × 1 byte; 115 KB = 8 × 7,168 × 2 bytes (no node limit, spread wide: 8 experts may be on 8 other machines; V3 puts each layer's experts on just 8 machines incl. source, so ≤ 7 cross-machine copies). Red box: same-index device reached first across machines; picks illustrative.
Cost tracks machines visited: one copy each, which NVLink forwards to its experts. The report puts NVLink at ~3.2× cross-machine speed, so each machine feeds 3.2 experts on average for free: 4 × 3.2 ≈ 13, and equal traffic allows up to 13 experts (V3 picks 8).
That's training and prefill. Latency-bound decode skips in-machine forwarding: one copy per expert, so copies grow with experts picked.
EP's imbalance is worst: unlike other cuts, who is busy shows only after routing, and changes every batch. All wait for the busiest expert. It snowballs: popular experts get more tokens, learn more, draw more; others wither. Big-model memory buys a small-model result: a quality problem, not just speed. A two-part cure:
📌 Auxiliary-loss-free load balancing: Wang et al., arXiv 2408.15664; V3 tech report, arXiv 2412.19437 sec. 2.1.2, sec. 4.2 (the report calls this step the bias update speed γ: 0.001 for the first 14.3T tokens, 0 for the last 500B). Writing the step as u follows Algorithm 1 of Wang et al. (bias update rate u, also 0.001 in their experiments); this figure keeps that notation.
⚠️ Illustrative simulation (fixed random seed, pick 2 of 8, u = 0.02), not V3's measured load.
Attention and experts differ entirely in shape (outside MoE, the EP dimension is just DP): attention's load scales with sequence and KV cache, the experts' with sheer parameter count. So the same devices can split them differently.
📌 Megatron-Core moe/README.md (MoE Parallel Folding, example TP4·CP2·DP8·PP4 → ETP1·EP64·EDP1); paper: arXiv 2504.14960. TEP / DEP definitions: TensorRT-LLM tech blog 26.
How much does it matter? Two names, then one measurement (box below).
The two most common inference setups: TEP (TP attention, EP experts) and DEP (DP attention, EP experts), suffixed by device count. Experts match; attention differs: one shared request batch, or one per device?
📌 TensorRT-LLM definitions: TEP<N> shards both attention (TP) and experts (EP) across N ranks; DEP<N> keeps attention data-parallel (ADP) while distributing experts across N ranks.
⚠️ Expert-step comms as usually built: TP attention, all devices hold the same tokens, so expert outputs all-reduce; data-parallel attention, tokens differ, so all-to-all. Illustrative: 4 devices, 8 experts.
TEP spreads each request over devices: fast generation. But attention splits by head; V3's KV has one head, so every device holds it in full (§5.4). DEP keeps only each device's own requests' KV: more requests fit, experts get bigger batches. The price: all-to-all for experts, uneven load from varied request lengths, idle devices running empty forward passes. The “same devices, two mappings” figure's left setup (attention TP4 × DP2) sits between. Not Megatron's ETP/EDP (§8.6).
The first three cuts split batch, weights and experts, never the sequence. But activations (training) and KV cache (inference) grow with context. What if one sample outgrows one device?
Load imbalance: all wait for the busiest expert. V3 biases expert scores, down if overloaded, up if under; it alters only routing, not the computed weights.
One sample outgrows a device? Split its sequence: the fourth cut. Training and inference split different things.
Context parallelism (CP) cuts a long sequence into chunks, one chunk's activations per device. Only attention is hard: every token must see all earlier ones, held elsewhere. Two fixes:
Why no note-passing in Ulysses? In a “chunk × head” table, later chunks see earlier ones down a column; heads in a row never interact. Ring splits rows, so dependencies cross devices; Ulysses transposes, giving each device whole columns.
📌 DeepSpeed-Ulysses: Jacobs et al., arXiv 2309.14509 (all-to-all around attention; per-device traffic is flat as length and devices scale together). Ring Attention: Liu et al., arXiv 2310.01889.
⚠️ Schematic: 4 chunks × 4 heads, 4 devices; each real cell is a whole token chunk's Q, K, V on one head.
Ring: people stay, notes rotate. Ulysses: re-cut so each holds a few heads of every note.
Stacked, they make USP: block the table over a 2D device grid, Ulysses' all-to-all inside a machine, a ring across; device count is the product. Megatron's CP supports this hierarchy too.
📌 USP: Fang & Zhao, arXiv 2405.07719; xDiT docs: ulysses-degree × ring-degree = sp-degree, lifting the rule that sp-degree must be below the head count; friendlier to mixed networks.
⚠️ “All-to-all inside the machine, ring across machines”: a common layout we infer from each traffic type, not the paper's rule.
Generative attention is causal: tokens see only the past. Later chunks work harder, so a contiguous split swamps the last device. Fix: one light, one heavy block per device (zigzag, below).
📌 Megatron-LM megatron/core/utils.py: sequence cut into 2×cp_size blocks; rank r gets blocks r and 2·cp−r−1; per docs/user-guide/features/context_parallel.md no redundant lower-triangle compute, balanced load. Script counts cells, diagonals as whole; halving diagonals: 2/6/10/14 vs. 8/8/8/8, contiguous split: a 7× gap.
In long-context, high-concurrency decode, KV often uses the most memory. TP splits by head, DCP by token. TP deals heads to devices: V3's 128 query heads split easily, but MLA's KV is one compressed copy shared by all heads: one unsplittable head.
Many models (Llama 2 70B) have 8 KV heads: 8-way TP gives each device one. Once TP exceeds the KV head count H, each head is duplicated TP ÷ H times (vLLM docs: “duplicated for tp_size / H times”). MLA's single KV head under 8-way TP: 8 identical full copies, one per device. Hence V4-Pro on TP4 (§4.4) wastes 3 copies.
📌 vLLM context-parallel deployment docs: KV cache duplicated tp_size / H times; DeepSeek-R1 with MLA has 1 KV head, so -tp 8 = 8× copies; add -dcp 8. KV sizes from V3's config.json, script-computed.
DCP (decode context parallelism) stores KV round-robin by token across existing devices. Token, not chunk, because notes keep growing and new ones must spread evenly too.
The new token's Q goes to every device; each computes attention on its KV share, and the partials merge. Attention is a score-weighted average, so the rule is “don't divide yet”:
📌 Framing: Ring Attention appendix code (Liu et al., arXiv 2310.01889) sums numerator, denominator apart; numerically stable blockwise softmax: FlashAttention (arXiv 2205.14135).
⚠️ Our example: arbitrary numbers, computed from the definition and asserted.
Three extra collectives per layer (vLLM default: gather Q, swap softmax denominators, combine outputs) reclaim the KV that TP duplicated; DCP degree maxes at TP ÷ KV heads, exactly the duplicates. Again: communication for memory.
Batching still works: 1,000 requests = 1,000 rows, split alike; each device does its 1/TP of all 1,000 in one step. It fits: 1/TP of the KV per device means longer contexts, more requests, bigger batches. It reads less: under MLA, plain TP has every device read the whole KV (different heads, same copy); DCP reads 1/TP, so long-context attention speeds up too.
DCP is for decode; splitting a long prompt in prefill, for a sooner first token, is PCP (prefill context parallelism). One letter apart, they're consecutive steps. In a “question × notes” table, prefill's thousands of questions split by row; decode's lone question leaves no rows, so DCP further splits that row.
Section 6 puts prefill and decode on separate machines, one each (our synthesis).
Prefill and decode now want different splits: compute-hungry prefill a split prompt, bandwidth-hungry decode spread-out KV. Shared devices fit neither well. So stop splitting tensors; give them separate machines: the fifth cut.
No. “Don't divide yet”: each device sends numerator and denominator; sum each, then divide once.
Cuts 1–4 split tensors (batch, weights, experts, sequence); cut 5 splits work. A request's stages differ in character: put them on different machines, each with its own sharding.
Two phases per request. Prefill runs thousands of prompt tokens through matmuls at once: compute-bound. Decode emits one token a step yet reads all weights and KV from HBM: bandwidth-bound.
Your answer is streaming word by word; someone pastes a hundred-page manual, and it stops. Voice assistants suffer more: halting speech feels slow at any average. On shared devices the engine must pick one job per step; a long prefill holds several steps while every request mid-generation waits: new requests' TTFT (time to first token) and others' TPOT (time per output token) worsen.
Decode rereads the weights from HBM each step for one use by the batch. How big a batch makes that pay?
⚠️ Our derivation: per step a dense layer reads 2P bytes (bf16) for 2Pb FLOPs (b/byte); an expert gets b × 8 ÷ 256 tokens. Weights only; KV raises the bar.
📌 v7: 2,307 TFLOP/s bf16 per chip; HBM 3,433 GiB/s per TensorCore, two per chip (author's wiki: entities/tpu-v7). V3's 8/256 comes from its config.json.
MLA helps by shrinking the KV, so a device fits more requests. MegaScale-Infer's A100 math: a batch needs at least 156 requests; each Mixtral expert gets just 39.
📌 DistServe arXiv 2401.09670: prefill is compute-heavy, decode is bound by HBM bandwidth; split, they serve 7.4× more requests or meet a 12.6× tighter SLO.
📌 Course author's 1P1D on TPU v7x (Qwen3-Coder-480B): the three-leg KV transfer, estimated from bandwidth, ~100 ms, 5–10% of a 1–2 s prefill. 8K-prompt KV ≈ 1.04 GB (FP8, computed from config), ~83 ms over a 100 Gb/s network; long prompts favor 2P:1D, long outputs 1P:2D.
Chunked prefill helps: small prompt chunks ride along with decode steps. But small chunks slow prefill (each rereads all earlier KV); the workloads still share devices and one parallelism. PD disaggregation splits them: prefill machines hand the KV cache to decode machines, which only generate.
Our TPU v7x 1P1D setup (Qwen3-Coder-480B, a v7x-8 each for prefill and decode) moves KV from prefill HBM to decode HBM:
📌 Our TPU v7x 1P1D record (wiki qwen3-coder-480b-pd-disagg-tpuv7x-20260425): three legs, estimated from bandwidth, ~10 / 80 / 10 ms, ~100 ms total, 5–10% of prefill (prefill 1–2 s inferred from this).
⚠️ Our derivation: KV = 2 × 62 layers × 8 KV heads × 128 × 8,192 × 1 byte ≈ 1.04 GB; 100 Gb/s = 12.5 GB/s → about 83 ms.
| Leg | Path | Estimate |
|---|---|---|
| ① | HBM → host memory (PCIe) | ~10 ms |
| ② | Host memory → peer memory (data-center network, conservatively one chip's 100 Gb/s share) | ~83 ms |
| ③ | Peer memory → HBM (PCIe), into decode's KV pool | ~10 ms |
| Total | ~100 ms (10 + 83 + 10) |
The middle leg: the KV for an 8K prompt is 2 (one K, one V) × 62 layers × 8 KV heads × 128 (head dim) × 8,192 × 1 byte (FP8) ≈ 1.04 GB. 100 Gb/s is 12.5 GB/s, so one trip takes about 83 ms. A CPU-side proxy also routes each request to both sides.
It is Section 1's point-to-point send/receive over the slow data-center network, affordable as each request goes once. From here on, scheduling centers on those notes: KV moves with its request. Kimi's Mooncake calls its inference system “KVCache-centric.”
Short requests, light traffic? The extra transfer and machines may not pay; stay colocated with chunked prefill.
Splitting adds a knob, the prefill-to-decode ratio; balance the load. DistServe's clean example:
📌 DistServe (Zhong et al., arXiv 2401.09670), example from Sec. 1 and Fig. 1 (13B model, one A100, 90% of requests within SLO): per-device goodput of 1.6 / 5.6 / 10 rps under the same SLO; 2P1D gives 3.3 rps per device, 2.1×. The authors' blog “Throughput is Not All You Need” walks through these numbers.
⚠️ Our derivation: 2P1D handles min(2 × 5.6, 10) = 10 per second, about 3.3 per device; colocated, 10 per second takes ⌈10 ÷ 1.6⌉ = 7 devices.
Rule of thumb: long prompts run 2P:1D, long outputs 1P:2D.
DistServe searches the ratio jointly with each side's parallelism. Versus unsplit systems: up to 7.4× more requests at the same latency targets (vs. DeepSpeed-MII), or up to 12.6× tighter targets (vs. vLLM) (OPT models, 90% SLO attainment). The figure's “more than double” is the paper's single-device example; 7.4× is the best over all ratios and shardings.
The real payoff: the two sides stop sharing one parallelism.
Decode splits further: attention reads each request's own KV, binding it to the request. Experts don't care whose token it is; they want big batches, since each step reads their weights once (§6.1). AFD (Attention-FFN disaggregation) separates them: M machines only run attention, N only hold experts.
📌 MegaScale-Infer arXiv 2504.02263 (disaggregated expert parallelism, ping-pong pipeline, up to 1.90× per-GPU throughput); Step-3 arXiv 2507.19427; vLLM released an experimental AFD plugin in 2026-07.
Each layer ships tokens to the experts (M → N) and back (N → M). Micro-batches rotate to hide this: one in attention, one in experts, one in flight. They split the batch (requests grouped), not the sequence: each decode request emits one token per step, so 3,000 requests make three 1,000-token groups. MegaScale-Infer needs at least three (four if communication is slow), reporting up to 1.90× per-GPU throughput.
ByteDance's MegaScale-Infer and StepFun's Step-3 do this; vLLM got an experimental plugin in July 2026.
Same idea, one stage earlier: the vision encoder gets its own machines and ships embeddings to the LM (encoder disaggregation; with PD, often EPD). Training's version: heterogeneous multi-module parallelism, each with its own scheme.
Five cuts, each with its own communication and uses, coexist on a real cluster: which stay in a machine, which cross machines, whose degree comes first?
Neither side stalls the other, so both run full: in the paper, 2 prefill + 1 decode serve 10 req/s; colocated needs 7 devices. The KV transfer is a sliver of prefill.
Real configs often stack three or four parallelisms, leaving one question: which cut rides which link. Right, and the savings hold; wrong, and the network eats them.
Links differ. GB300 NVL72 joins 72 GPUs in one NVLink domain at 1.8 TB/s each (bidirectional throughout); the NIC beyond gives 200 GB/s, 9× slower. TPU slices use ICI inside (large ones form a 3D torus, §1.5); the data-center network across slices is about 50× slower like for like (conversion in Sources), far wider than GPUs' 9×.
Chattiness varies too: per-step counts (our derivation, illustrative config) span 3 orders of magnitude:
📌 Counts are our derivation: TP 2 all-reduces per layer each in forward and backward (arXiv 1909.08053 sec. 3); FSDP 3 per layer (ZeRO's 3Ψ, arXiv 1910.02054 sec. 7); EP, PP, DP from the schedule; with PP, divide TP's count by stages; with recomputation, TP 6 per layer. Links: GB300 NVLink 5 1.8 TB/s/GPU; A4X Max: 4 GPUs, 4 × CX-8 800 Gb/s/node.
Guess first: which of TP, PP, DP gets the fast link? Picture an office building:
📌 Llama 3 technical report arXiv 2407.21783 sec. 3.3.2 (dimensions ordered inside out: [TP, CP, PP, DP]); V3 technical report sec. 3.2 (EP 64 across 8 machines).
⚠️ Office-building analogy: ours (loosely after CERFACS); ring sizes don't show bandwidth ratios.
Default placement: TP, EP, FSDP and CP talk every layer, so they fill the fastest tier first; PP (stage boundaries only) and DP (once a step) take slow links. Also ask whether communication overlaps compute. TP's can't (each block's closing all-reduce stalls the next), so TP stays on fast links, its degree capped by that tier.
EP and FSDP can hide traffic by sending early, so some cross machines: V3's EP 64 spans 8 machines (H800's intra/inter-node gap is just 3–4×, unlike TPU's); Llama 3 puts FSDP outermost. DP moves the most data, but once a step; backward runs last layer first, so late gradients ship early. Section 6's once-per-request KV can likewise take the slow data-center network (TPU v7x).
Same 8 devices and TP4 × DP2; only the TP groups move. Schematic: a transfer is 1 cell on a fast link, 9 on a slow one.
One color per TP group (placement 2: one per row). A step = 8 TP rounds + 1 DP: placement 1, 8 × 1 + 9 = 17 cells; placement 2, 8 × 9 + 1 = 73 (its DP pair shares a machine: 1 cell).
Hence TP rarely leaves an 8-GPU server. GB300 stretches NVLink across a rack, so EP can go wide (TP, limited by head count and matmul efficiency, rarely tops 8 or 16 even there): the fast tier's size sets how deep these cuts go.
Wrong placement and sharding leave tuning circling under the wrong ceiling, as we found with DeepSeek-V4-Pro on GB300. 1P1D = one prefill + one decode machine; 3P = three prefill; DEP8 = each decode device's attention serves its own requests, experts over 8 devices:
📌 Measured by the course author: gpu-tpu-pedia gpu/inference/a4x-max/deepseek-v4/VLLM-V4PRO-RUNBOOK.md (1p1d 14,563; 3p1d 21,100; 3p + DEP8 at concurrency 512, 55,153; at concurrency 1,536, 65,132). GPU counts assume 4 per node, 1 node each for prefill and TP4 decode, 2 nodes for DEP8; per-GPU figures computed by this script.
Tuning everything on TP4 decode (no eager mode, more prefill machines, concurrency) lifted total throughput 45%, but on twice the GPUs: per-GPU fell, TPOT stayed near 50 ms. DEP8 quartered TPOT at the same concurrency and more than halved TTFT.
Resharding alone bought two bigger-batch wins: no KV copy on each of 4 devices (§5.4), and twice the decode GPUs (4 → 8), halving experts per GPU. The 8 GPUs span two trays, inside the rack's fast NVLink. Check the sharding before touching parameters.
Picking parallelism: five gates, each a yes/no question:
Two real configurations; mind the two “not used”: V3 skips TP on 8-GPU H800 servers, Hunyuan 3 skips EP on TPU. “DP 4 × FSDP 128” is a product of shard counts.
| Step | Hunyuan 3 (295B MoE, TPU v7, 256 chips) | DeepSeek-V3 (671B MoE, 2,048 H800s) |
|---|---|---|
| ① Fits | FSDP 128 (FSDP shards experts too); any narrower OOMs | ZeRO-1 shards optimizer state, EP 64 experts |
| ② Fast tier | 256 chips, one fast slice; FSDP already spreads experts: no EP; a small EP trial lost throughput (note below) | 8 GPUs/server: no TP; EP 64 spans 8 machines (slow-link exception); ≤ 4 machines per token |
| ③ Sequence | 4K / 8K, no CP | 4K pre-training; no CP reported |
| ④ PP | Fits in one slice, not used | PP 16; DualPipe hides comms under compute |
| ⑤ DP | Leftover 4× to DP: DP 4 × FSDP 128, the default (v7 chip = 2 devices; degrees count devices: 256 chips = 512). Best: FSDP 256; freed HBM lifts per-device batch 12 → 16, +3% vs. default; widening alone costs <1%: the gain is all batch | The rest: ZeRO-1 data parallelism |
Same five steps, nearly disjoint results: the two machines' fast links differ in shape. Don't copy another team's config; check your own links first.
Note: Hunyuan 3's EP test (4-way, 16 chips) lost 71% in one run, but with the batch halved (a confound); another setup lost 37%. The record blames extra all-to-all hops on the torus; at 4-way that cost is small, so the cause is open. V3: PP 16 × DP 128 = 2,048; EP 64 shards experts within those 128 ranks, no extra GPUs (our derivation).
Last question: adding devices, how much per-device efficiency survives? Two measures:
The metric is scaling efficiency: per-device throughput after ÷ before. We lack clean strong-scaling data; closest: the same DP 4 × FSDP 128 at per-device batch 12 → 8 fell from 580 to 453 per chip. Less work per device hurts.
Our two measurement sets; compare within a set only. Left: weak scaling, devices and global batch ×4. Right: same devices, varying FSDP width.
📌 Measured by the course author: gpu-tpu-pedia tpu/Hunyuan3-295B-Pretraining/TUNING-v7 sec. 3.7 (five splits, pdbs 8), sec. 4.1 (64 and 256 chips both at 580, pdbs 12). Numbers are TFLOP/s per chip.
Left: DP is nearly free. Four 64-chip groups do one gradient all-reduce per step: under 1% of the 23.5 s step by a rough v7 ICI estimate, even discounted for shared links (our derivation). But the groups share one slice (DP on ICI); this doesn't carry over directly across slices.
Right is the warning: same chips and batch, but FSDP over all chips instead of a quarter means more gather steps on smaller pieces, so latency stops amortizing: 11% lost. Narrower FSDP (64 or 32 shards) can't fit shards beside activations. While the batch can grow, grow it with the devices; extra devices become replicas first.
Every parallelism now has a story: what it cuts, what it sends, which link it rides. Next, a panorama and tables decode TEP8, DEP16 and DP4 × FSDP128.
TP syncs every layer, unhideable. PP: once per stage boundary, can cross machines. DP: once a step, can span data centers.
So many names: DP, FSDP, TP, SP, CP, PP, EP, and for inference DCP, PCP, DEP, TEP… Just four families. Of the five cuts, weights and experts are both model parallelism. Learn the families and the names fall into place.
One criterion: what does it split? A figure, then a table:
📌 Families, collectives summarized from §1–7; “can cross slow links” = default placement; exceptions (V3's cross-node EP, Llama 3's outermost FSDP) in §7.
| Family | Splits | Each holds | Examples |
|---|---|---|---|
| DataData parallelism | The batch (requests) | Own samples; logically full model | DP, FSDP, Attention DP |
| SeqSequence parallelism | Tokens within one sample | A chunk: activations (training), KV cache (inference) | CP, Ulysses, DCP |
| ModelModel parallelism | The weights | Some columns, layers or experts | TP, PP, EP |
| DisaggDisaggregation | Work phases or modules | One job: prefill or experts | PD disaggregation, AFD |
Tables follow the four families; “New” = appeared or went mainstream in 2025–26. For reference; on a first read, skip to 8.6. Many names are new; come back as needed.
| Name | Splits | Solves | Extra comms | Use |
|---|---|---|---|---|
| DP / DDP | The batch; full model replicated | Linear throughput scaling | Train: gradient all-reduce once a step. Infer: none, router splits load | Train · Infer |
| ZeRO-1 / 2 / 3 | Each stage adds one: optimizer state → gradients → parameters | Under DP every device holds the full state: pure redundancy | RS + AG; ZeRO-1/2 match DP (ZeRO-2 with micro-batches: one RS per micro-batch); ZeRO-3 re-gathers parameters every layer | Train |
| FSDP / FSDP2 | ZeRO-3, PyTorch's native version | 1.5× DP's traffic; memory falls linearly with devices | Forward: AG weights; backward: AG again, then RS gradients | Train |
| HSDP | Shard in-node, replicate across nodes (e.g. Hunyuan 3's DP 4 × FSDP 128, though in one slice) | Keeps the most frequent all-gathers in the fast domain | AG / RS within a node, all-reduce across nodes | Train |
| ZeRO++ | Three tricks on top of ZeRO-3 | Cross-node comms too costly | INT8 weights pre-AG; extra param copy in-node; quantized grads | Train |
| DP across DCN | DP spanning multiple slices or data centers | DP talks once per step, the best fit for slow links (PP works too) | Gradient all-reduce over DCN | Train |
| DiLoCo New | Replicas run hundreds of steps alone, then sync once | Cross-data-center training: syncing every 500 steps cuts syncs several-hundredfold | One outer gradient sync (Streaming variant: shards in turn, overlapped with compute) | Train |
| Attention DP New | By request, attention layers only; experts configured separately | MLA has 1 KV head, so TP8 copies KV 8×; with DP each device keeps only its requests' KV | All-to-all when followed by EP; idle devices run a dummy forward to stay in sync | Infer |
-dp without -ep puts expert layers on TP, not EP.When a sample is too long for one device, split the sequence. Training and inference differ in why: training splits activations, inference the KV cache. Hence two tables.
Training (and prefill): split activations
| Name | Splits | Solves | Extra comms | Use |
|---|---|---|---|---|
| Megatron SP | LayerNorm, Dropout, residual activations only; needs TP (filed by what it splits; by role TP's partner, §3.3) | Activations TP misses, fully copied on each device | Splits TP's all-reduce into all-gather + reduce-scatter; same total volume | Train · Infer |
| CP (context parallelism) | All activations, along the sequence | Long-context training; near-mandatory past tens of K | KV via ring, all-to-all or all-gather; combinable hierarchically | Train |
| Ulysses | Switches between sequence split and head split | Long sequences: per-device comm flat as length and devices scale together | All-to-all before and after attention; degree ≤ head count (KV heads for GQA) | Train |
| Ring Attention | KV blocks circle a ring | Degree not limited by head count | Ring P2P overlapping compute | Train · Infer |
| Striped / USP / 2D-Attention | Reassign tokens; stack Ulysses and Ring in 2D | Causal masks load the ring unevenly; the two limits complement | Both of the above | Train |
| PCP (prefill CP) New | Query tokens during prefill | Time to first token (TTFT) on long prompts | All-gather KV, or a ring | Infer |
| Chunked PP New | Prompt chunks pipelined in turn | Prefill startup ∝ first chunk only | Pipeline point-to-point | Infer |
Inference decode: split the KV cache. The hottest area in 2025–26
| Name | Splits | Solves | Extra comms | Use |
|---|---|---|---|---|
| DCP (decode CP) New | KV cache sequence, tokens round-robin across devices | TP > KV heads copies KV; MLA has 1 head: TP8 = 8 identical copies. DCP turns that into capacity | Three per layer (vLLM default): all-gather Q, exchange LSE, merge outputs | Infer |
| Helix New | Same devices, two layouts per layer: attention by KV sequence (plus TP ≤ KV head count), FFN by TP × EP | Million-token decode: both KV and weight reads must spread | All-to-all after attention swaps partial results; FFN keeps TP's all-reduce or EP's all-to-all | Infer |
MaxText context_autoregressive | Decode: KV split by sequence, FFN by expert | Same as above, the TPU way | Inserted automatically by XLA | Infer |
This family truly splits weights, in three places: within a matrix, across layers, across experts.
Inside the matrix (tensor parallelism)
| Name | Splits | Solves | Extra comms | Use |
|---|---|---|---|---|
| TP | Matrices split column-then-row, in pairs | One layer's weights or compute don't fit | 2 all-reduces per layer each way, very frequent, so it stays in the fastest ring (NVLink domain, ICI in one slice); on v7, TP8 is just 4 chips, already below the line | Train · Infer |
| 2D / 2.5D / 3D TP | Matrix split into a grid | 1D TP comms grow with the degree | Broadcasts and reductions along grid rows/columns | Train, largely abandoned |
| GTP New | Reshards weights along the TP axis, gathered on use | Megatron docs: “GTP_remat is an implementation of ZeRO-3,” but on the model-parallel axis | Per-weight async all-gather; gradient reduce-scatter | Train, experimental |
Across layers (pipeline parallelism)
| Name | Splits | Solves | Extra comms | Use |
|---|---|---|---|---|
| PP | Layers, grouped into stages | Model too deep, or only slow links between machines | Point-to-point at stage boundaries, least volume; the price: the bubble | Train · Infer |
| VPP (interleaved 1F1B) | Several non-contiguous layer groups per device | Shrinks the bubble | More point-to-point transfers | Train |
| Zero Bubble New | Backward split: activation vs. weight gradients | Fills the bubble with the weight-gradient half | Point-to-point | Train |
| DualPipe | Feeds the pipeline from both ends at once | V3's heavy expert comms must fully overlap forward/backward compute | Point-to-point, overlapped with EP's all-to-all | Train |
Across experts (MoE)
| Name | Splits | Solves | Extra comms | Use |
|---|---|---|---|---|
| EP | Different experts on different devices | Total expert parameters too large | Dispatch, combine: two all-to-alls. Data picks destinations, so load is uneven: EP's worst problem | Train · Infer |
| Wide-EP New | EP spread over 32, 64 or more devices | Few experts per device, so each gets a bigger decode batch | Large-scale all-to-all; needs a dedicated library | Infer |
| ETP | TP inside one expert | Expert too big; fine-grained: usually 1 | All-reduce inside the expert | Train |
| EDP | Leftover expert-side DP, derived from the others | Where expert gradients are summed | Expert-gradient all-reduce | Train |
| Parallel Folding New | Attention, expert layers: same devices, two splits | Breaks the old “EP ≤ DP” constraint. E.g. attention at TP4 · CP2 · DP8, experts straight at EP64 | Two sets of comm groups | Train |
The first three split tensors; this one splits the work: unlike jobs go to separate machines, each with its own parallelism.
| Name | Separates | Solves | Extra comms | Use |
|---|---|---|---|---|
| PD disaggregation | Prefill, decode on separate machines | Prefill (compute) and decode (bandwidth) hurt TTFT, TPOT | KV cache transfer between machines | Infer |
| AFD New | Attention on one machine group, experts on another | Experts pool several attention groups' tokens; ratio tunable | Per-layer round trip, attention → experts → attention; hidden with 3–4 micro-batches | Infer |
| Encoder disaggregation | Multimodal vision encoder runs alone | Encoder and LM compete for resources | Embedding transfer | Infer |
| Heterogeneous multi-module parallelism | Encoder, LM: own parallelism each | Multimodal training: parts too unlike in shape | Bridging between the two meshes | Train |
Inference MoE configurations are usually written as shorthand plus a number, like TEP8 or DEP16. The first letter says how attention is split; EP says how experts are split.
| Shorthand | Attention layers | Expert layers |
|---|---|---|
| TEP<N> | TP across N devices | EP over the same N devices |
| DEP<N> | Attention DP: each device its own requests | EP over the same N devices |
TensorRT-LLM's definition, verbatim: TEP<N> shards both attention (TP) and experts (EP) across N ranks. DEP<N> keeps attention data-parallel (ADP) while distributing experts across N ranks. vLLM uses the terms the same way.
With PD disaggregation each side often picks one; no fixed rule says which gets which: it depends on model and load. vLLM's Kimi K3 deployment uses TEP8 for prefill, DEP16 for decode; TensorRT-LLM's example is the reverse: DEP4 for prefill, TEP8 for decode.
Other combined terms you'll hear:
The test: does it split a new dimension? If not, it isn't new parallelism.
| Name | What it actually is |
|---|---|
| TBO / SBO / DBO | Comm hidden in compute: two staggered batches, one talks, one computes |
| Async TP, TP comm overlap, collective matmul | Still TP, with comm and matmul chopped small and interleaved |
| EPLB, redundant experts | Load balancing: replicate and re-place hot experts. An EP fix, no new dimension |
| Offload, recomputation | Trading time for HBM, or host memory for HBM |
| Chunked prefill | Inference scheduling: long prompts chunked, mixed with decode, less blocking (§6.1). Not §8.3's Chunked PP (prefill too): its chunks flow down the pipeline: a new axis |
| Term | What it means where |
|---|---|
| SP | At least three: Megatron's SP (LayerNorm-type segments only, with TP); DeepSpeed's SP (Ulysses); any sequence splitting. Megatron: splitting all activations = CP |
| CP | Training: split activations. vLLM has two switches: PCP adds devices and cuts TTFT; DCP adds none and expands KV capacity |
| DP | Dense: separate replicas; MoE inference: really Attention DP, per-step sync |
| ETP | Megatron: in-expert TP; TensorRT-LLM Hybrid ETP: expert layers mix TP + EP |
| hierarchical | ZeRO++ hierarchical sharding ≠ Megatron hierarchical CP |
| GPU / chip / device | 1 GPU = 1 device; a TPU v7 chip = 2 to software; degrees count devices |
| Bandwidth | One-way or send + receive: here 1.8 TB/s (NVLink) and 1.2 TB/s (ICI) are combined; 800 Gb/s and 100 Gb/s (NICs) one-way |
| Concurrency/batch | Concurrency: requests queued; batch: computed at once |
Try the opening's V3: PP 16, EP 64, ZeRO-1 data parallelism (report sec. 3.2). PP cuts depth, sends only at stage boundaries, so it crosses machines; EP cuts experts, two all-to-alls per layer, so ≤ 4 machines per token; ZeRO-1 cuts data: one reduce-scatter + all-gather a step, cheapest. Three cuts, three collectives, three links: V3's cryptic config (PP 16, EP 64, ZeRO-1) now reads plainly.
⭐ Back to the opening: where to cut. The five cuts split different things but pay one price: spend communication to buy memory or compute. Picking a parallelism strategy is picking which communication to pay, how often, on which link.
Last, with the ruler from Sections 3 and 6: the hardware line is compute ÷ bandwidth; how you cut sets the workload line. Each cut here tries to lift a workload line above it.
① An MoE model will decode on one GB300 rack (72 GPUs, one NVLink fast domain). How would you split its attention and expert halves? Answer the three questions, then compare with 7.3.
② TEP8 vs. Megatron's ETP8: only the letter order differs. How do the meanings differ? Look it up, then see 8.6's red box.
Claim ← source. Section 8 checked 2026-09-23, Section 1 2026-09-24, the rest 2026-09-25 after ten review rounds and two trial lectures.
| Claim | Source |
|---|---|
| Bytes per device per collective (Section 1 table) | NVIDIA/nccl-tests: bus bandwidth correction factors in doc/PERFORMANCE.md: AllReduce 2(n−1)/n, ReduceScatter / AllGather / AlltoAll (n−1)/n, Broadcast / Reduce 1 |
| Step-by-step ring ReduceScatter; the root pattern | wanghonglei, “Collective communication primitives for distributed deep learning, from zero to mastery” (Chinese, 2026-06-27), ch. 1–2; steps script-computed, asserted; shifted one so device k gets chunk k |
| Warm-up: 16 B/param standard; V3 keeps m, v in bf16, master weights and accumulated grads in fp32 | DeepSeek-V3 report, arXiv 2412.19437 sec. 3.3.3 (AdamW's 1st and 2nd moments in BF16, not FP32, “without incurring observable performance degradation”; master weights and batch-accumulation gradients stay FP32); PyTorch: optimizer state at parameter precision; Megatron-LM needs --use-precision-aware-optimizer plus --exp-avg-dtype bf16 --exp-avg-sq-dtype bf16 (Megatron Core MoE docs; flags in megatron/training/arguments.py; without it, megatron/core/optimizer/optimizer_config.py asserts both are fp32); that group lists --main-grads-dtype, --main-params-dtype, --exp-avg-dtype, --exp-avg-sq-dtype together (arguments.py); we still count gradients separately; V3's ~14 B/param is derived (compute-weight precision unstated; BF16 assumed); Megatron's bf16 training defaults to fp32 gradient accumulation and all-reduce (arguments.py: “bfloat16 requires gradient accumulation and all-reduce to be done in fp32”), so gradients are 4 B: 18 B/param |
| V3's full FP8 recipe validated: relative loss error < 0.25% | Same report, sec. 3.3, appendix B.1: ~16B and ~230B models, ~1T tokens each, full FP8 recipe (bf16 optimizer state included) vs. BF16 baseline; no separate “moments in bf16” ablation |
| Moments in bf16? Depends on β₂ | V3's AdamW: β₁ = 0.9, β₂ = 0.95 (report sec. 4.2); PyTorch default 0.999. ⚠️ Our derivation: bf16 has 8 significant bits, relative spacing 2⁻⁷–2⁻⁸; rounding drops changes under half a gap (≥ ~0.2%); at β₂ = 0.999 a step adds only 0.1%. Cross-check: Dettmers et al., arXiv 2110.02861: block-wise 8-bit optimizer state matches 32-bit |
| V3 full training: 2.788M H800 GPU hours; about 318 years on one GPU | DeepSeek-V3 technical report, abstract (covers pre-training, context extension and post-training); 318 years = 2.788M ÷ 8,760 hours, our derivation |
| ZeRO local update: no comms; clipping: one scalar all-reduce | Adam is element-wise (Kingma & Ba, arXiv 1412.6980 Alg. 1); clipping uses the global norm (Pascanu et al., arXiv 1211.5063); sharded: devices sum local squares, then all-reduce (our summary) |
| Memory and communication at each ZeRO stage (Section 2) | Rajbhandari et al., ZeRO, arXiv 1910.02054 sec. 5, 7: Pos and Pos+g move the same volume as DP (2Ψ), Pos+g+p ≤ 1.5×; memory 16Ψ → 16Ψ/Nd |
| How TP splits and how often it talks; SP adds none (Section 3) | Megatron-LM arXiv 1909.08053 sec. 3 (2 all-reduces forward, 2 backward); arXiv 2205.05198 sec. 4.2.2 (AG + RS replace all-reduce, no extra comms); TP must divide the query heads, KV groups and TP must divide one another (checks in megatron/core/transformer/transformer_config.py) |
| PP bubble (p−1)/m; interleaved ÷ v | Narayanan et al., arXiv 2104.04473 sec. 2.2.1–2.2.2; Zero Bubble arXiv 2401.10241; DualPipe README |
| DualPipe: 2× params, activation peak p+1; after Chimera | DeepSeek-V3 technical report, arXiv 2412.19437 sec. 3.2.1, Table 2 |
| V-shapes level device peaks | Qi et al., arXiv 2405.15362 |
| MaxText: PP defaults to 1; micro-batch = global batch ÷ num_pipeline_microbatches | Comments in MaxText configs/base.yml |
| V3's training parallelism; parameter distribution | DeepSeek-V3 technical report, arXiv 2412.19437 sec. 3.2 (16-way PP, 64-way EP, ZeRO-1, no TP); config.json (61 layers, first 3 dense, 256 experts, moe_intermediate_size 2048, hidden 7168) |
| FLOPs per byte moved; v7 hardware line about 3,845 | ⚠️ Our derivation (dense approx., perfect overlap); v7 2,307 TFLOP/s bf16; spec: ICI 1,200 GB/s/chip, plus a 200 GB/s tier; only reading that as per link, send + receive (“6 links × 200, 600 sent”) fits (derived; Scaling Book's 9e10/link/direction gives ~540, line ~4,270, hence 3,800–4,300); per device also ~3,845 (2 devices share a chip's links, halving both) (wiki ici-dcn, Inferact blog spec table). Fixed 2026-09-25: earlier misused 1,200 → 1,922; TP line's 4.5 assumes standard attention + 4×-wide MLP (V3: MLA + MoE; illustrative) |
| V3's EP: ≤ 4 nodes, FP8 dispatch / BF16 combine, aux-loss-free balancing | DeepSeek-V3 technical report, arXiv 2412.19437 sec. 2.1.2, sec. 3.2.2, sec. 3.3.3; ≈ 28.7 KB of cross-node dispatch per token is our derivation |
| V3's balancing hyperparameters, no token dropping; NVLink / IB 3.2×, ≤ 13 experts at equal traffic | Ibid. sec. 4.2 (bias update speed γ = 0.001, 0 for last 500B tokens; seq-wise balance loss α = 0.0001; M = 4, 64 GPUs/8 nodes), sec. 2.1.2 (No Token-Dropping), sec. 3.2.2 (160/50 GB/s, 4 × 3.2 ≈ 13) |
| The Parallel Folding example | Megatron-Core megatron/core/transformer/moe/README.md; arXiv 2504.14960 |
| TP4 → DEP8 on GB300: at equal concurrency 512, 2.61× total, 2.09× per GPU (DEP8 at 1,536: 2.47× per GPU, TTFT 95 s); tuning +45% | Course author's measurements: gpu-tpu-pedia gpu/inference/a4x-max/deepseek-v4/README.md and VLLM-V4PRO-RUNBOOK.md (TP4 decode 14,563 → 21,100 after tuning, 16 GPUs, 1,319 per GPU; DEP8 65,132, 20 GPUs, 3,257 tok/s per GPU) |
| Activations grow with seq. length | Korthikanti et al., arXiv 2205.05198 Eq. (1): sbh(34 + 5as/h) per layer |
| Ring Attention; Ulysses: constant traffic, degree ≤ head count | arXiv 2310.01889; arXiv 2309.14509 sec. 3.2 (4Nh/P, constant when N and P scale together); head-count cap in USP, arXiv 2405.07719 sec. 3 |
| CP's zigzag split | Megatron-LM megatron/core/utils.py (2×cp chunks; rank r takes chunks r and 2·cp−r−1); docs/user-guide/features/context_parallel.md |
| TP replicates KV tp/H times; DCP reuses TP ranks | vLLM context parallel deployment docs; vllm/config/parallel.py docstring. V3's 70,272 KV B/token computed from config.json, whose num_key_value_heads=128 counts decompressed MLA heads; cached compressed (vLLM: 1 KV head) |
| Motivation and gains of PD disaggregation (Section 6) | DistServe arXiv 2401.09670 (prefill compute-leaning, decode bandwidth-bound; 7.4× requests vs. DeepSpeed-MII, 12.6× tighter SLO vs. vLLM) |
| v7x 1P1D: 3 KV stages ~100 ms (bandwidth est.); 2P:1D / 1P:2D | Author's deployment notes (KV time: bandwidth estimate): wiki qwen3-coder-480b-pd-disagg-tpuv7x-20260425. 8K KV ≈ 1.04 GB, ~83 ms at 100 Gb/s, derived (Qwen3-Coder config: 62 layers, 8 KV heads, head_dim 128) |
| Communications per step for each cut (Section 7) | ⚠️ Our derivation (illustrative config: 60 layers, 8 micro-batches): TP 4 per layer (arXiv 1909.08053 sec. 3), FSDP 3 per layer (arXiv 1910.02054 sec. 7), EP / PP / DP counted from the schedule |
| GB300 NVLink 1.8 TB/s / 800 Gb/s NIC per GPU; 9× | Author's wiki: sources/nvidia-gpu-comparison-20260311, analyses/gb300-a4x-max-network-congestion-control (A4X Max: 4 GPUs per node, 4 × CX-8 800 Gb/s); 9× is our bidirectional conversion |
| Hunyuan 3 scaling, five-step test | Author's measurements: gpu-tpu-pedia tpu/Hunyuan3-295B-Pretraining/TUNING-v7 sec. 3.7 (five splits: 404 / 450 / 453 / OOM / OOM), sec. 4.1 (580 at 64 and 256 chips), sec. 3.6 (DP2 × FSDP256, pdbs 16: 599), 4-way EP on 16 chips −71% (single run). Sec. 4.1: 23.54 s/step; inter-group all-reduce < 1%, derived (source's 12 ms math inconsistent); EP reconfig −37%, wider FSDP under 1% slower (450 vs. 453): TUNING-v7 sec. 3.7 |
| DCP: 3 collectives per layer | dcp_comm_backend docstring in vllm/config/parallel.py (default ag_rs: 3 NCCL calls per layer; a2a backend: 2) |
| ZeRO-2 is no longer free with micro-batches; with PP, choose ZeRO-1 | DeepSpeed docs: pipeline parallelism is incompatible with ZeRO-2/3; V3 technical report sec. 3.2 (ZeRO-1); Megatron distributed optimizer |
| TP FLOPs per byte ≈ 4.5 × hidden ÷ TP; V3 at TP8 ≈ 4,032 | ⚠️ Our derivation: 4 all-reduces per layer, each ~4Th bytes, vs. 72h²T/n compute (standard attention + 4×-wide MLP); for V3 illustrative only |
| Decode EP: direct sends per expert | DeepEP low-latency mode (deepseek-ai/DeepEP README); V3 technical report sec. 3.4 |
| V3's cluster and at most 4 nodes per token | DeepSeek-V3 technical report, arXiv 2412.19437 sec. 3.1 (2,048 H800s, NVLink within nodes, IB between nodes), sec. 2.1.2, sec. 3.2 |
| Strong / weak scaling | Amdahl 1967; Gustafson 1988 (Reevaluating Amdahl's Law); critical batch size: arXiv 1812.06162 |
| NVSwitch does the additions in the switch (NVLS) | NVLS algorithm (NVIDIA NCCL docs); “nearly half”: our derivation, ~S sent per GPU vs. 2(n−1)/n·S |
| Which TPU slices wrap around; small ones get ~half bandwidth | How to Scale Your Model (the Scaling Book), TPU chapter: only full cubes (multiples of 4) get wraparound; Google Cloud TPU7x topology docs |
| 3D torus rotating three blocks; all-gather 3× faster | Scaling Book, sharding chapter: multi-axis gathers multiply bandwidth; 63 vs. 21: our bandwidth-only derivation for 4×4×4, XLA schedule unchecked |
| Llama 3 puts FSDP outermost | Llama 3 technical report, arXiv 2407.21783 sec. 3.3.2 (parallelism dimension order [TP, CP, PP, DP]) |
| V3: 37B active params per token; pre-trained at 4K | DeepSeek-V3 technical report, arXiv 2412.19437 abstract and sec. 4.1 |
| TorchTitan's “4D” | TorchTitan paper, arXiv 2410.06511 sec. 2.1.5 and Table 6 (4D = FSDP + TP + PP + CP); the abstract stops at 3D (Llama 3.1 405B, 512 GPUs); today's pytorch/torchtitan README no longer says “4D” |
| “Even the largest devices carry only two to three hundred GB of HBM” | NVIDIA B300: 288 GB HBM3e/GPU; Google TPU7x: 192 GiB HBM/chip (source code: 206 × 10⁹ B ≈ 191.85 GiB); 9.76 TiB ≈ 10,700 GB (decimal) |
| DCN 100 Gb/s per chip; cross-slice ~50× slower, same basis | Google Cloud TPU7x docs (checked in TPU expert review round 5); 100 Gb/s read as one-way, ×2 ≈ 25 GB/s bidirectional, ~48× below ICI's 1,200 GB/s bidirectional, our derivation (one-way only: ~96×) |
| Definitions of TEP / DEP | TensorRT-LLM tech blog 26 (DeepSeek V4 on Blackwell), verbatim; vLLM Kimi K3 blog (2026-07-27) |
| No TEP / DEP in Megatron; ETP / EDP / Parallel Folding | NVIDIA/Megatron-LM main: megatron/core/transformer/moe/README.md; paper arXiv 2504.14960 |
| PCP adds devices; DCP doesn't, reusing TP ranks | vllm-project/vllm main: vllm/config/parallel.py docstring; vLLM context parallel deployment docs |
| Attention DP: motive, flags | SGLang DP / DPA guide; TensorRT-LLM docs/source/features/parallel-strategy.md; vLLM DP deployment docs |
| Helix | arXiv 2507.07120; TensorRT-LLM tech blog 22 |
| GTP quote | Megatron-LM docs/api-guide/core/generalized_tensor_parallel.md |
| Zero Bubble / DualPipe | arXiv 2401.10241; github.com/deepseek-ai/DualPipe |
| DiLoCo | arXiv 2311.08105; Streaming DiLoCo arXiv 2501.18512; the diloco mesh axis in MaxText |
| AFD | MegaScale-Infer arXiv 2504.02263; Step-3 arXiv 2507.19427; vLLM AFD Plugin blog (2026-07-23) |
| Chunked PP | LMSYS blog (2026-01-15); Mooncake arXiv 2407.00079 |
| MaxText's parallelism axes | AI-Hypercomputer/maxtext: src/maxtext/configs/base.yml, docs/guides/optimization/sharding.md |
← Course overview ·
Previous: Topic 4 · Backward pass and optimizers ·
To teach it: instructor script
Built by Courses/tools/topic05-build.py; the main text lives in that script.