Parallelism Strategies

A model that doesn't fit on one device must be split across many.
Each cut spends traffic along its dimension to buy memory or compute.

Five collectives, then five cuts: data, weights, experts, sequence, the work itself. Each fills an earlier gap.

Prerequisite Topic 4 (the 16-byte bill) Data as of 2026-09 ⏱ about 84 min

Course author Chris Yang·Google Cloud AI Infra architect

There's also an instructor script.

It covers how to teach: goals, pauses, misconceptions, likely questions. The content is here.

Presenting: lines, figure cues and timing are scripted. Self-study: give the script to an AI for a TA that knows the material and how to teach it.

Section 9 sources every number; ours say “our derivation.”

Section 0

One device can't hold it

🎯 Warm-up: in DeepSeek-V3 training, what are a parameter's 16 bytes? — Q1 reviews Topic 4, Q2–3 preview. Safe to skip

Question 1: In standard mixed-precision AdamW, each parameter keeps five items resident in HBM. Bytes for each?

Weights (the copy for compute) bytes
Gradients bytes
Master weights (the optimizer's copy) bytes
First moment m bytes
Second moment v bytes

Preview (think first; answers in §2.2):

  1. Question 2: Which of the 16 bytes is biggest? Which to cut first?
  2. Question 3: DeepSeek-V3 doesn't store 16 bytes: two parts drop to half precision. Which two? Why not the master weights?

Topic 4's bill: 671B params × 16 bytes (weights 2 + gradients 2 + optimizer state 12) = 9.76 TiB (~10,000 GB) of resident training state. Top devices hold 200–300 GB, so storage alone takes 30–60 devices before any compute. So cut. But along which dimension?

One device is too small and too slow: we cut to fit, and to go faster DeepSeek-V3, 671B params, standard mixed precision (16 bytes/param) B300 TPU v7 2,048 GPUs at once Too big: devices needed to hold 9.76 TiB B300 (288 GB each): 38 TPU v7 (192 GiB/chip): 53 Too slow: 2.788M H800 GPU hours One GPU: 318 years 2,048 GPUs at once: ~57 days ← same scale, down to a sliver GPU hours include pre-training, long-context extension, post-training. We have to cut, but every cut adds new device-to-device traffic The 10,000 GB at left is model state only; activations scale with batch × sequence length and outgrow it at long sequences. After the split, each device holds what the other lacks, so they must exchange. That is the whole lecture.
Left: it doesn't fit. Right: it won't finish.
Square = device; right bars share one scale.
Sources and assumptions

📌 V3 full training: 2.788M H800 GPU hours (DeepSeek-V3 technical report, arXiv 2412.19437, abstract; covers pre-training, context extension and post-training); 2,048 H800s (sec. 3.1).

📌 B300: 288 GB per GPU (NVIDIA DGX B300 user guide: 8 × 288 GB); TPU v7: 192 GiB HBM3e per chip.

⚠️ Our derivation: 9.76 TiB = 671B × 16 bytes; 318 years = 2.788M ÷ 8,760 hours; 57 days = 2.788M ÷ 2,048 ÷ 24.

Tensors can be cut along batch, sequence, hidden, layer or expert. The whole lecture is one idea:

In one sentence Traffic buys memory or compute.
A strategy picks which communication, how often, on which link.

The lecture is a relay: cut, pay one communication; find the gap, cut again.

A relay: each red note is why the next cut exists Box: what the cut splits and its toll (extra communication). Red below: the problem left, forcing the next cut Problem left → reason for next cut Five collectives first, then one cut at a time Cut 1 · data Split batch; drop copies Toll: AllReduce, split in two ↳ Leaves open Small batch: weight traffic exposed Cut 2 · weights Split matrices; split layers Toll: AllReduce/layer; stage P2P ↳ Leaves open Params mostly in narrow experts Cut 3 · experts Whole experts per device Toll: 2× AllToAll/layer ↳ Leaves open Long context: 1 sample won't fit Cut 4 · sequence Train: activations; infer: KV Toll: Ring KV; gather Q, merge ↳ Leaves open Prefill, decode: different cuts Cut 5 · the work Prefill, decode apart Toll: One KV transfer per request ↳ Leaves open Who gets the fast lane? Onto the machine Chattiest traffic → fast lane Toll: Misplaced: several× slower Every cut earns its keep, and none is the last Each adds one kind of communication and buys back memory or speed. What it leaves uncovered is where the next section opens.
Red notes are the plot: each opens the next section.
Each section ends on one: “What this cut leaves open.”

Two rulers per cut: how often it talks (→ which link) and FLOPs per byte moved (→ stall or not; Section 3).

Glossary (from Topics 1 and 3, one line each): tensor, activations, attention head, MoE, KV cache, prefill and decode, Q/K/V, MLA, HBM, ICI, “N-way”…

Token and batch: a token is the smallest unit of text, about a word piece; a batch is a step's input, often counted in tokens here.
Tensor and hidden dimension: tensors are a model's multidimensional arrays; a token's vector is hidden-dimension long (7,168 in V3).
Activations: each forward layer's intermediate results, kept for backward.
Attention head: attention runs as dozens of small, independent attentions in parallel, each a head.
Micro-batch: one step's batch cut into a few pieces processed one at a time.
V3: DeepSeek-V3, the 671B-parameter MoE model, our running example.
Q, K, V: every token computes three vectors: Q is “what I'm looking for,” K is “what I am,” V is “what I carry.” A new token compares its Q with every earlier token's K and sums their V, weighted by similarity.
Query heads and KV heads: each head has its own Q; K and V may have fewer, several query heads sharing one KV group (at the extreme, one KV head). Inference caches K and V, so the KV-head count decides whether the cache splits by head.
Softmax: attention scores every earlier token; softmax normalizes those scores into weights that sum to 1.
MLA: DeepSeek's attention; K and V shrink to one small latent all heads share: far smaller KV cache, but only “one head” to split.
Node: one machine, usually 4 to 8 GPUs joined by a high-speed interconnect (NVLink on NVIDIA).
HBM: on-device fast memory; the model and intermediates must sit here.
Compute and bandwidth: operations vs. bytes moved, per second. Moving much and computing little is bandwidth-bound; the reverse, compute-bound.
ICI and slices: ICI is TPU chips' dedicated fast link; chips joined by ICI form a slice; slices talk via the regular data-center network (DCN).
Devices and axes: software sees one v7 chip as 2 devices, and parallelism degree counts devices; a slice's chips form a 3D grid whose x, y, z directions are the axes.
Rank: one process in the parallel job, usually one GPU (on TPU, one device).
v7 and v7x: two names for the same TPU generation (Ironwood); both appear here.
N-way: “8-way TP” means TP over groups of 8 devices. TP4 × DP2 means 4-device TP groups, 2 of them doing data parallelism: 4 × 2 = 8 devices.
MoE and experts: many parallel feed-forward blocks (experts); a small router picks a few per token, the rest idle. The parameter count is large, but each token touches only a small fraction.
KV cache: in generation, each processed token leaves a K and V that later tokens attend to; it is kept throughout, growing with context.
Prefill and decode: inference's two phases. Prefill reads the full prompt at once for token one; decode emits one per step. Decode steps are light, so requests batch up: read weights once, use them across the batch; bigger pays more.

Section 1

Five collectives

Each later cut adds a communication; meet them now. Five see daily use: all-reduce, all-gather, reduce-scatter, all-to-all, plus point-to-point send/receive. Each answers: who sends to whom, and is arriving data concatenated, summed or kept as is?

1.1 Reading the figures

A collective: devices sending and receiving under one shared rule. Figure conventions:

1.2 One to everyone: four basic moves

Device 0 is the root, the hub of all traffic.

Four basic moves: one to everyone Four devices, one color each; data in four blocks. Striped block = several devices' data summed; dashed box = empty Device 0's data A Device 1's B Device 2's C Device 3's D Broadcast one copy → a copy for all Before After Dev 0 A0 A1 A2 A3 A0 A1 A2 A3 Dev 1 A0 A1 A2 A3 Dev 2 A0 A1 A2 A3 Dev 3 A0 A1 A2 A3 Device 0's whole copy, duplicated to everyone Scatter one copy split → a block each Before After Dev 0 A0 A1 A2 A3 A0 Dev 1 A1 Dev 2 A2 Dev 3 A3 Device 0 cuts its copy in four, one block each Gather a block each → one copy Before After Dev 0 A0 A0 B1 C2 D3 Dev 1 B1 Dev 2 C2 Dev 3 D3 Each sends one block to dev 0, which concatenates Reduce a copy each → one sum Before After Dev 0 A0 A1 A2 A3 Σ0 Σ1 Σ2 Σ3 Dev 1 B0 B1 B2 B3 Dev 2 C0 C1 C2 C3 Dev 3 D0 D1 D2 D3 Each sends a whole copy to dev 0, summed blockwise Two pairs, each differing in one move Broadcast and scatter are one-to-many; gather and reduce are many-to-one. Gather and reduce differ only in whether arrivals are concatenated or summed. All four have a root (device 0); done naively, all traffic squeezes through its one link. Training runs the rootless ones daily: AllGather, ReduceScatter, AllReduce; AllToAll is just every device doing its own scatter.
Four rooted moves: broadcast, scatter, gather, reduce.
Gather vs. reduce: concatenate or sum on arrival.
See them move: four animations, one pause per step
Broadcast: device 0's whole copy, duplicated to everyone (7 s, Manim; untick “Autoplay” to step.)
Scatter: device 0 sends block j to device j and keeps only block 0 (7 s, Manim; untick “Autoplay” to step.)
Gather: everyone hands its block to device 0, which concatenates them in order (7 s, Manim; untick “Autoplay” to step.)
Reduce: everyone hands its whole copy to device 0, which sums them block by block (7 s, Manim; untick “Autoplay” to step.)

All four funnel through the root's link; more devices, more jam. Training runs on the next group.

A closer look: how the root jams, how libraries ease it

Naively (the root serving each in turn), broadcast and reduce pile all traffic on the root, worse with every device. Libraries chain them as a relay, so the root moves one copy. Gather and scatter can't: the root holds n distinct pieces; the total stays flat, but it all crosses its one link.

1.3 Everyone to everyone: training's workhorses

Training's daily workhorses: everyone to everyone, no root An “All” prefix means everyone ends up with the result. Same conventions: striped = summed, dashed = empty Dev 0 Dev 1 Dev 2 Dev 3 AllGather a block each → everyone a full copy Before After Dev 0 A A B C D Dev 1 B A B C D Dev 2 C A B C D Dev 3 D A B C D Concat, no sum. FSDP forward uses it to rebuild layer weights ReduceScatter a copy each → each gets one summed block Before After Dev 0 A0 A1 A2 A3 Σ0 Dev 1 B0 B1 B2 B3 Σ1 Dev 2 C0 C1 C2 C3 Σ2 Dev 3 D0 D1 D2 D3 Σ3 Sum, then split. Device k gets only the sum of block k AllReduce a copy each → everyone gets the sum Before After Dev 0 A0 A1 A2 A3 Σ0 Σ1 Σ2 Σ3 Dev 1 B0 B1 B2 B3 Σ0 Σ1 Σ2 Σ3 Dev 2 C0 C1 C2 C3 Σ0 Σ1 Σ2 Σ3 Dev 3 D0 D1 D2 D3 Σ0 Σ1 Σ2 Σ3 Data parallelism syncs gradients with it AllToAll dev k sends dev j a piece (equal sizes shown here) Before After Dev 0 A0 A1 A2 A3 A0 B0 C0 D0 Dev 1 B0 B1 B2 B3 A1 B1 C1 D1 Dev 2 C0 C1 C2 C3 A2 B2 C2 D2 Dev 3 D0 D1 D2 D3 A3 B3 C3 D3 No sum, no concat, just moves. MoE dispatches tokens with it The first three are one family; the fourth is different AllGather concatenates; ReduceScatter sums then splits; AllReduce sums and gives everyone the result. Taken apart, AllReduce is ReduceScatter, then AllGather (order: see 1.4). AllToAll neither sums nor concatenates: every device sends every device a different piece. Every pair has traffic, so it is the hardest on the network.
“All” in the name: everyone gets the result.
All-to-all just moves data, no sum or concat; shown in the equal-size case.
AllGather: three ring steps, each passing the newest block right. Concat only, no sum (20 s, Manim; untick “Autoplay” to step.)
ReduceScatter: three ring steps; each sends a block right, the receiver adds it. Device j ends with block j's sum (23 s, Manim; untick “Autoplay” to step.)
AllReduce: same ring, two laps: three sum steps (ReduceScatter), three concat steps (AllGather) (41 s, Manim; untick “Autoplay” to step.)

“All” = everyone gets it: all-gather = gather for all, all-reduce = reduce for all, reduce-scatter = reduce, a block each; all-to-all: no sum, no concat.

1.4 All-reduce splits in two

The course's most important equation: AllReduce = ReduceScatter + AllGather One AllReduce, done in two halves, gives the identical result. Several later parallelisms each use one of the halves Dev 0 Dev 1 Dev 2 Dev 3 Two halves Start: a whole copy each Dev 0 A0 A1 A2 A3 Dev 1 B0 B1 B2 B3 Dev 2 C0 C1 C2 C3 Dev 3 D0 D1 D2 D3 After ReduceScatter: one summed block each Σ0 Σ1 Σ2 Σ3 After AllGather: everyone has the sum Σ0 Σ1 Σ2 Σ3 Σ0 Σ1 Σ2 Σ3 Σ0 Σ1 Σ2 Σ3 Σ0 Σ1 Σ2 Σ3 ① ReduceScatter sum first, keep one block ② AllGather then concat to everyone Each step, every device sends (n−1)/n of a whole copy; both steps total 2(n−1)/n—the same factor NCCL uses for AllReduce's bus bandwidth Where the later free wins come from Data parallelism already does one AllReduce. ZeRO splits it: ReduceScatter the gradients, update locally, then AllGather the new weights—ZeRO-1/2: same total traffic, less memory. Likewise, tensor + sequence parallelism splits each layer's AllReduce into these two halves, run in different places.
Sum then concat = one AllReduce.
Once split, the halves can run at different times.

Other work fits in between; later free wins start here: ZeRO (Section 2), tensor + sequence parallelism (Section 3).

A closer look: same AG and RS; order and what's between decide the result
  • All-reduce: reduce-scatter (RS), then all-gather (AG), nothing between; two laps of one ring (§1.5).
  • AG then RS, compute between: gather in, compute, scatter out. FSDP's weight gather (§2.3), SP's sequence gather (§3.3) and expert parallelism's all-gather token dispatch (Megatron's allgather dispatcher) fit it. We call it AGRS.
  • RS then AG, optimizer between: ZeRO scatters gradients, each device updates its share, then gathers the new weights (§2.2). All-reduce's order, plus a middle update. We call it RSAG.
  • AG and RS are transposes: a forward AG means a backward RS at the same spot, and vice versa. So SP's backward swaps their positions, yet still runs AG first: it meets the forward's exit first.
  • FSDP differs: forward has only an AG. In backward, RS is its transpose; the AG just re-gathers dropped weights.

⚠️ AGRS / RSAG are our shorthand for the order, not standard terms.

1.5 No root: the ring

With a root, device 0 receives and sends n−1 copies each: a bottleneck. Instead, link devices head to tail in a ring, no root.

Same ring, two laps: three steps that sum (ReduceScatter), three that concatenate (AllGather), making one AllReduce. Device 3's block exits right and wraps to device 0. (41 s, Manim; untick “Autoplay” to step.)

The ring = reduce-scatter + all-gather. Another build shows why it wins:

One AllReduce, two ways to build it: overload the root, or share the load S = one whole copy. Bar = how many copies the device sends in total (naive rooted version, no chain) Dev 0 Dev 1 Dev 2 Dev 3 Option 1: Reduce + Broadcast (root) Dev 0 3 copies Dev 1 1 copy Dev 2 1 copy Dev 3 1 copy Dev 0's link carries it all; jams as n grows Option 2: ReduceScatter + AllGather (ring) Dev 0 1.5 copies Dev 1 1.5 copies Dev 2 1.5 copies Dev 3 1.5 copies Solid = 3 RS steps, hollow = 3 AG steps; 3/4 copy each Everyone equally busy, every link in use at once Any device count: under two copies Devices (n) Root: n − 1 Ring: 2(n−1)/n 2 1 1 4 3 1.5 8 7 1.75 1,000 999 1.998 The −1: you never send your own block to yourself Why collective libraries almost always pick option 2 Same result either way; only who works differs: option 1 piles it on device 0, option 2 spreads it over everyone. Option 2's two halves are ReduceScatter and AllGather, and every free win in the later cuts comes from this split.
Same result; who works differs.
Right table: the root column grows with devices; the ring stays under 2.
Sources and assumptions

📌 The two assemblies: NVIDIA NCCL documentation, Collective Operations (AllReduce = Reduce + Broadcast = ReduceScatter + AllGather).

⚠️ Our derivation: naively rooted, device 0 sends n−1 copies; on the ring each sends 2(n−1)/n. Libraries use a chain or tree; we compare only the naive forms.

The price: steps grow with devices. For small messages, per-step waits dominate, the ring loses; libraries pick by size.

A closer look: ring vs. tree, NVSwitch in-switch sums, the TPU torus

Large messages are bandwidth-bound, where the ring is near optimal; small ones by per-step waits, where many steps hurt. So libraries like NCCL pick ring, tree, etc. by message size. The NVLink switch (NVSwitch) can also sum in-switch, nearly halving each device's sends.

TPUs are more direct: ICI itself is a torus. In big enough slices (every dimension a multiple of 4) each is a closed ring; in smaller ones it's a line, the ring doubles back, and bandwidth roughly halves.

The torus adds more: each chip sits on three rings, X, Y, Z. Gathering X, then Y, then Z keeps one direction busy per step, the last heaviest. Start three data chunks on different axes (XYZ / YZX / ZXY) and all three stay saturated: the same all-gather is up to 3× faster counting bandwidth alone. Reduce-scatter runs in reverse (big step first); all-reduce's halves likewise. ⚠️ Our derivation: per-step latency ignored; XLA's chunking/scheduling unchecked.

AllGather on a 3D torus: stagger the routes, keep all three axes busy 4×4×4 slice; each chip starts with 1/64. Bar = time on that axis (unit: 1/64 of the data over one axis); same scale on both sides Block A Block B Block C Axis by axis: X, then Y, then Z 3 15 63 X idle Y idle Z idle Time 63 Only one axis busy per step; step 3 alone takes 48/63 Rotated: A goes XYZ, B YZX, C ZXY 1 5 21 X Y Z Time 21 Same blocks, spread out: all three axes busy throughout, 1/3 the time Same data; the difference is how many axes work at once X, Y, Z are each a ring: axis by axis uses one direction per step; rotating keeps all three saturated, tripling bandwidth. ReduceScatter runs in reverse (big step first); AllReduce = the two back to back; counting bandwidth only, also 3× faster.
Same blocks: queued left, spread out right.
4×4×4 slice; bandwidth only, no per-step latency.
Sources and assumptions

📌 How to Scale Your Model, sharding chapter: gathering over several axes at once “increases the available bandwidth by a factor of N_axes”.

⚠️ Our derivation: bandwidth only, per-step latency ignored; equal bandwidth on all axes, every dimension a closed ring (slice dims multiples of 4). XLA's chunking and scheduling unchecked.

The same 4×4×4 cube, played twice: axis by axis lights one direction per step, rotating lights all three; the two time bars at bottom right share a scale, 63 vs. 21. (47 s, Manim; untick “Autoplay” to step.)

1.6 All-to-all: a different piece for everyone

AllToAll: everyone mails everyone; pieces differ in content and size Expert parallelism: the router assigns each token a device (number in cell); color = source device From dev 0 From dev 1 From dev 2 From dev 3 ① Dispatch: all send in all directions; thicker = more Sent: 8 tokens per device, each to its assigned device Received: sorted by source Dev 0 1 0 2 3 1 3 1 0 Dev 1 0 0 0 2 1 2 3 3 Dev 2 0 0 0 2 1 1 0 3 Dev 3 1 0 2 3 0 2 1 2 Dev 0 gets 11 Dev 1 gets 8 Dev 2 gets 7 Dev 3 gets 6 Dashed = sent to self, no network. Every other line crosses devices, all leaving at once Dev 0 gets 11, dev 3 only 6: everyone waits for dev 0 to finish ② Combine: experts done, back the same way Dev 0 Dev 0 Dev 1 Dev 1 Dev 2 Dev 2 Dev 3 Dev 3 Same lines, reversed: each token returns to its source slot Twice per layer: dispatch, then combine ③ Special case: equal sizes = one transpose dev k's row dev j's row transpose Ulysses swaps chunks ↔ heads with this equal-size case Why the network dreads it Sizes are known only after routing (unequal counts: often called AllToAllv); receivers are unevenly loaded; the busiest sets everyone's wait. Every piece is a private letter: no merging en route as in AllReduce, no neighbors-only ring. Every pair has traffic, so it tests the network's full bisection.
Dispatch scatters, combine retraces; only equal sizes make a transpose.
Counts illustrative; real ones depend on routing.
Sources and assumptions

⚠️ Illustrative: token counts between devices are hand-picked, not measured routing; in real V3 each token picks 8 experts.

Dispatch fires every way at once, unequal counts; after compute, everything returns along the same lines. (20 s, Manim; untick “Autoplay” to step; manual mode pauses each step.)

Expert parallelism (Section 4) uses the general case: unequal counts, same-route combine. Ulysses (Section 5) uses the equal-size case: a transpose.

1.7 One table to sum up

Of eight, the four rooted are building blocks; five see daily use (send/receive: Section 3's layer cut).

“Sends” = per device, ring; S = one whole copy (AllGather: assembled; ReduceScatter: before summing); Broadcast / Reduce assume a chained relay.
CollectiveDoesSendsUsed by
AllReduceSum; all get it2(n−1)/n · SData-parallel grad sync; TP: 2× fwd, 2× bwd per layer
ReduceScatterSum; one block each(n−1)/n · SFSDP bwd grads; TP + sequence parallel
AllGatherConcat; all get it(n−1)/n · SFSDP fwd weights; TP + sequence parallel; Q in KV-split inference
AllToAllJust moves data(n−1)/n · SExpert-parallel dispatch and return; Ulysses
Broadcast / ReduceFan-out / fan-inSInitial weights out, metric sums, etc.
Send / RecvOne-to-oneVariesPipeline stages pass activations; Ring Attention passes KV
Two takeaways ① All-reduce = reduce-scatter + all-gather; several later cuts get free wins from it.
② Only all-to-all sends everyone different data; expert parallelism needs it, networks fear it most.
Each later cut's extra traffic is a row here. Next: the first cut, data.
Check yourself: 4 devices, one all-reduce. Ring: copies sent per device? Naive rooted: device 0?

Ring: reduce-scatter, 3 steps of 1/4 copy; all-gather, 3 more: 1.5 copies. Rooted: device 0 sends 3. More devices, wider gap.

Section 2

First cut: data

The simplest cut: each device trains on different samples. At first it doesn't fix “it doesn't fit”; only FSDP does, at a price.

2.1 Data parallelism (DP): full model per device

Every device: full model, own samples. After backward, one all-reduce averages gradients; all update. One exchange per step: rare but big (pure DP at V3's size: ~2.7 TB per device per step). Each still stores 16 bytes per parameter: still too big.

Analogy: V3's weights are a library of 61 reading rooms, one per layer, across V3's 128 branches (one per device). DP gives every branch a full copy; below, we strip duplicates.

2.2 ZeRO: shave the idlest first

Three pieces: weights (the model), gradients (each parameter's move this step), optimizer state (largest: master weights plus two moments per parameter). Ψ = parameter count.

Per ZeRO's paper, optimizer state is what stays resident across steps, checkpointed. Gradients, recomputed and zeroed each step, count separately (own ZeRO stage). Megatron sets accumulator precision (--main-grads-dtype) with master weights and moments (“precision-aware optimizer”), so some count it there.

ZeRO: remove the duplicates, one kind at a time—the first two stages are free; the last costs 50% more communication 16 bytes per parameter, split three ways as in Topic 4's table. Example: DeepSeek-V3, 671B params, 128-way DP (2,048 devices ÷ 16 pipeline stages) weights 2 B (bf16) gradients 2 B (bf16) optimizer state 12 B (fp32) Resident bytes per parameter, per device bar length ∝ bytes/param Bytes/param (after 128-way split) V3 per device Comm/step Data parallel 16.00 bytes 9.76 TiB 2Ψ ZeRO-1 4.09 bytes 2.50 TiB 2Ψ (= data parallel) ZeRO-2 2.11 bytes 1.29 TiB 2Ψ (= data parallel) ZeRO-3 = FSDP ← zoom 20× to see it 0.125 bytes 78.11 GiB 3Ψ (1.5×) The removed pieces haven't vanished: each device keeps only its own 1/128. First two stages free, the last one costs ZeRO-1/2 add zero bytes of comm over DP (with one sync/step), yet cut 16 B to ~2. V3 still needs 1.29 TiB/device: doesn't fit. Only ZeRO-3 reaches 78.11 GiB/device; comm 2Ψ → 3Ψ (+50%). ~84 GB overflows an 80 GB H800: splitting data isn't enough.
Optimizer state alone takes 12 of 16 bytes: cut it first.
The ZeRO-3 bar needs zooming in: 16 bytes split 128 ways leaves 0.125 each.
Sources and assumptions

📌 Source: Rajbhandari et al., ZeRO, arXiv 1910.02054 secs. 5, 7. Per the paper, Pos and Pos+g communicate as much as DP (2Ψ); Pos+g+p at most 1.5×. Ψ = parameter count, in elements.

⚠️ Per-device bytes computed and asserted by our script; 9.76 TiB as in Topic 4 §5.1 (671e9 × 16 ÷ 1024⁴). 128-way = V3's 2,048 devices ÷ 16 pipeline stages (report, arXiv 2412.19437 sec. 3.2: PP16; the division is our derivation). Resident only, no activations.

Warm-up Q2: optimizer state is 12 of 16 bytes, so it goes first. It is also the idlest:

Shave the idlest first—the order of ZeRO's stages is the order of idleness When each piece is read or written in one step (schematic timeline: forward 1 unit, backward 2, then a short update) weights 2 B gradients 2 B optimizer state 12 B One step: forward → backward → update Forward Backward Update Weights ZeRO-3 last: busiest; each layer must gather it first +50% comm Gradients ZeRO-2 next: appear layer by layer during backward free Optimizer state idle in memory the whole step ZeRO-1 first: largest and idlest, used only at the update free solid = being read/written; dashed = holds memory, untouched The largest is also the idlest, so shave it first Optimizer state is 12 of the 16 bytes and is used only in the final update: each device owns its 1/n, nobody waits. Busier toward the top: every layer uses the weights; shard them, and each layer must first copy in the other slices. That is FSDP's extra 50%.
The long dashed bar: ZeRO-1's free win.
Schematic: only when each piece is read or written.

ZeRO-1 shards optimizer state, ZeRO-2 adds gradients, ZeRO-3 weights; each device keeps only its 1/n. ZeRO-1 also saves compute: each updates its 1/n, not all parameters.

Why are the first two stages free? The §1.4 identity:

One ZeRO step: the update in the middle stays on each device—a split AllReduce sandwiches it Four devices; gradients split into four slices by parameter, device k owns slice k. Striped = summed across devices; dashed box = not stored here Dev 0 Dev 1 Dev 2 Dev 3 Three things in one step After backward: each has full grads Dev 0 g0 g1 g2 g3 Dev 1 g0 g1 g2 g3 Dev 2 g0 g1 g2 g3 Dev 3 g0 g1 g2 g3 ReduceScatter: sum of slice k only Σ0 Σ1 Σ2 Σ3 Update slice k locally (no comm) W0 W1 W2 W3 AllGather: full new weights W0 W1 W2 W3 W0 W1 W2 W3 W0 W1 W2 W3 W0 W1 W2 W3 ① ② ③ No communication in the middle: Adam is elementwise Parameter i's update uses only its own gradient, two moments and master weight. Four devices each updating a slice = exactly one updating all. ① + ③ = the original AllReduce, not one byte more. Sole exception, grad clipping: each device sums its slice's squares, then AllReduces one scalar.
① ReduceScatter + ③ AllGather = the original AllReduce.
Only the middle panel is new: each device updates just its slice.
Sources and assumptions

📌 ZeRO paper, arXiv 1910.02054 sec. 7: Pos and Pos+g communicate as much as DP (2Ψ). Elementwise Adam: Kingma and Ba, arXiv 1412.6980 Algorithm 1.

⚠️ With micro-batches, only ZeRO-1 is strictly free: ZeRO-2 keeps no full gradient, so can't accumulate first; it reduce-scatters after each micro-batch. So V3 and Megatron both pair ZeRO-1 with pipeline parallelism (PP, which needs micro-batches).

Warm-up Q3: V3's moments are half precision; master weights (and accumulated gradients) stay full Optimizer state holds no gradients (per-step input, zeroed after use); the full-precision one is the micro-batch accumulator.
Per our accounting (2.2): optimizer state 12 → 8 bytes (master 4 + moments 2 + 2), plus 4 for the accumulator. Add compute weights (unstated; assume the usual BF16, 2 bytes): ~14 bytes per parameter (our derivation).
Master weights grow in steps too tiny to survive compression. Moments: depends on β₂ (bf16 ruler below; V3: 0.95, PyTorch default: 0.999).
A closer look: a bf16 ruler—moments compress, master weights don't
Can the moments live in bf16? Depends how far one step moves—under half a tick, it rounds back Ticks = bf16 values near 1 (spacing 1/128). v ← β₂·v + (1−β₂)·g²; assume current v = 1 and new g² is 2× that β₂ = 0.95 (V3) β₂ = 0.999 (PyTorch default) A bf16 ruler 1.0000 1.0078 1.0156 1.0234 1.0312 1.0391 1.0469 1.0547 1.0625 new value weighted 5%: moves to 1.050, stored as 1.0469: survives new value weighted 0.1%: moves to 1.001, under half a tick (0.0039), stored as 1.000: rounded away So V3 can store both moments in bf16, but not the master weights Moments are moving averages; when β₂ isn't too close to 1, each step moves far enough for bf16 to hold. V3 uses 0.95. Master weights get a tiny increment each step and keep accumulating: anything under half a tick is lost, so they stay fp32.
Green step: several ticks. Red step: not even half a tick.
Rounding computed in real bf16; assumes the new g² is 2× the current v.
Sources and assumptions

📌 V3 report, arXiv 2412.19437 sec. 3.3.3: BF16, not FP32, tracks AdamW's two moments “without incurring observable performance degradation”; master weights, batch-accumulation gradients stay FP32. Sec. 4.2: β₁ = 0.9, β₂ = 0.95.

📌 Not the default: optimizer state normally matches the parameter precision; Megatron-LM must explicitly enable --use-precision-aware-optimizer plus --exp-avg-dtype bf16 --exp-avg-sq-dtype bf16 (flags in megatron/training/arguments.py; without the former, megatron/core/optimizer/optimizer_config.py asserts both must be fp32). PyTorch AdamW's default β₂ is 0.999.

⚠️ Our derivation: real bf16 ticks and rounding (8 sig. bits, round-to-nearest). V3 tested its FP8 recipe (incl. bf16 moments) vs. BF16: rel. loss error < 0.25% at 16B, 230B; no bf16-moment ablation.

2.3 ZeRO-3 = FSDP: the last 2 bytes cost 50%

After two stages, 2 bytes of weights per parameter remain: 1.29 TiB per device for V3 at 128-way (~1,415 GB; a device has 200–300 GB). Too big. More devices can't push ZeRO-2 lower: what's left is full weights. Cutting data alone, large models need ZeRO-3, PyTorch's FSDP (Fully Sharded Data Parallel): all-gather each layer before use, discard after.

Back to the library. FSDP cuts blind to content: a room's books laid page by page in a strip, cut into 128 equal lengths, one per branch. Cuts often land mid-book: one book's tail, the next's start. Alone, a piece is unreadable. To use a room, a branch copies the other 127 (all-gather), reads, discards the copies. After backward, each piece's edits merge at its owner (reduce-scatter).

One library, three ways to split it—FSDP shreds, readable only once reassembled; EP hands out whole shelves, usable as is Think of V3's full weights as one library collection: one reading room = one layer, one branch = one device. Branch counts follow V3's real config Dense room catalog desk shared shelf expert shelf ① Library = model: 61 rooms = 61 layers 3 Dense 58 MoE, mostly identical Room 1 at far left; data passes room by room, left to right ~671B params in all: the whole collection, not one book Zoom into one MoE room desk shared 256 expert shelves, each a complete expert ② Data parallel: 128 branches, each a full copy Branch 1 Branch 2 Branch 3 Branch 4 … Branch 128 Complete, but every branch holds the same books 128 branches, 128 identical copies ③ FSDP: flatten each room, cut into equal lengths, one piece per branch drawn as 8 pieces; really 128 Room k's books, as one strip Book 1 Book 2 Book 3 Book 4 Book 5 Book 6 Book 7 Branch 1 Branch 2 Branch 3 Branch 4 … Branch 126 Branch 127 Branch 128 Even red cuts, ignore book edges Branch 3: end of one book + start of next Alone, unreadable To use room k Copy all 128, rebuild (AllGather) Read: layer computes Drop copies, keep own piece Copy again in backward ④ Contrast: EP and PP hand out whole units, usable as is EP 256 → 64 branches Branch 1: 1–4 Branch 2: 5–8 Branch 3: 9–12 … Branch 64: 253–256 4 whole shelves each; a shelf = one full expert Usable as is; to look up, go to that branch PP 61 rooms → 16 stages Each stage takes whole rooms, in order A room is a whole layer, ready to compute ~4 rooms per stage (schematic; the report doesn't give V3's layers per stage) The difference: can the piece you hold be used on its own? FSDP ignores book edges, cutting by length so communication splits evenly; a piece alone is useless and must be all-gathered before every use. EP/PP split whole units (shelf = expert, room = layer), ready to compute. They pay differently: tokens travel to experts; layers queue to hand off work.
A lone piece is unreadable: FSDP's key difference from EP and PP.
Branch counts: V3's real config (DP 128, EP 64, PP 16).
Sources and assumptions

📌 V3 config: technical report, arXiv 2412.19437 sec. 3.1–3.2 (2,048 H800s; PP 16, EP 64 over 8 nodes, ZeRO-1 DP); config.json (61 layers, first 3 dense, 256 routed + 1 shared expert).

⚠️ Our derivation: DP 128 = 2,048 ÷ 16. V3 uses ZeRO-1, not FSDP; we borrow its degrees. PyTorch FSDP v1 flattens a layer's parameters into one strip, split evenly; FSDP2 shards each parameter on dim 0. Either way a lone shard can't compute.

EP (Section 4) deals whole shelves, each a self-sufficient expert; FSDP's fragments serve even communication, useless until reassembled.

Why FSDP costs 50% more—backward must reassemble the discarded weights Same layer, same step. Each colored block is one communication of (n−1)/n of the layer's weights AllGather (rebuild) ReduceScatter (sum, split) compute One layer · one step Data parallel Forward Backward RS AG 2 units (RS + AG = one AllReduce) FSDP AG Forward discard AG Backward RS 3 units = 1.5× Gather, discard, gather again FSDP gathers a layer before forward and drops it after, so only one layer is ever whole; backward needs it, so gather again—that's the extra unit. Conversely, PyTorch's reshard_after_forward=False keeps it: back to 2 units, full weights held all through backward. Trading comm for memory, again.
DP talks twice per layer, FSDP three times.
The extra one: backward re-gathers weights dropped after forward.
A full layer exists only in use; otherwise each device holds a quarter. (14-s silent Manim loop.)

Communication: 3Ψ, 1.5× DP. In return, per-device resident memory drops 9.76 TiB → 78.11 GiB (~84 GB, before activations): already over V3's 80 GB H800. The data cut alone isn't enough; hence the next cuts.

⚠️ Which stages are free, which costs ZeRO-1 and ZeRO-2 are free (with micro-batches, only ZeRO-1; see “One ZeRO step” sources). ZeRO-3 costs 50% more; large models must pay.

2.4 Reach depends on frequency

DP talks once a step: slow links work. FSDP talks every layer: fast lane only.

A closer look: how far DP, FSDP, HSDP and DiLoCo each reach
  • Data parallelism, ZeRO-1: once per step, so slow links work, even across data centers.
  • FSDP: gathers weights every layer, hundreds to thousands a step (counted in §7); needs a high-bandwidth domain.
  • HSDP (hybrid sharded data parallel): FSDP in-node, DP across. Frequent gathers stay in; only gradient sync crosses (÷ in-node shards).
  • DiLoCo goes further: replicas sync only every few hundred steps, built for cross-data-center training.
⚠️ Written wrong, HSDP gradient sync can become one all-device all-reduce HSDP sums over two axes, then splits along the FSDP axis back to owners.
Cheap: reduce-scatter on the intra-node FSDP axis (1/F each), then all-reduce that 1/F on the inter-node data axis.
Expensive: merge both axes into one group, all-reduce the full gradients, keep your slice. The all-gather half rebuilds data only to discard it; the cross-node leg moves full gradients, not 1/F.
Auto-sharding (e.g. JAX's GSPMD) can emit the expensive form; we've hit it. Check: in the compiled HLO, do the all-reduce's replica groups span both axes?
Pure FSDP has one axis: no issue. Data-center boundaries don't matter; the extra data axis does.

2.5 What this cut leaves open

FSDP moves weights each step, an amount set by parameter count, not batch; compute scales with each device's tokens.

The global batch can't grow forever (too large hurts learning): more devices, fewer tokens each. Small batches cut compute, not traffic; weight movement dominates. How else to cut, adding devices without each computing less?

Versus the next cut: FSDP splits the model in form, but at heart is DP—weights are whole at compute time, each device on its own batch. The next cut keeps them split while computing.

Check yourself: which two ZeRO stages are free? Why optimizer state first?

ZeRO-1 and ZeRO-2 are free (all-reduce = RS + AG anyway; the local update between sends nothing). Optimizer state: largest and idlest, used only to update.

Section 3

Second cut: weights

FSDP rebuilds each layer's weights before computing; a big per-device batch hides that traffic, a small one can't. The second cut splits the weights instead: devices share tokens, so adding devices doesn't thin them out. Three walls: small batch stalls FSDP → TP; TP can't leave one machine → PP; PP leaves devices idle.

3.1 Where FSDP runs out: FLOPs per byte moved

Once data arrives, how often is it used? Often: the transfer hides. Once: the device waits. As a ratio, FLOPs per byte moved over the network must beat the hardware's FLOP/s ÷ bytes/s, or the device idles.

FSDP's math is clean (⚠️ our derivation, dense model): ~6Ψ bytes per step (bf16 weights gathered in forward and backward, gradients scattered once, ~2Ψ each) for 6ΨT FLOPs (Topic 4: ~6 FLOPs per parameter per token; T = tokens per device). The ratio is exactly T, whatever the model size. Intuition: each arriving 2-byte parameter is used once per local token: one multiply-add, 2 FLOPs.

Analogy: a door workshop. Craftsmen are devices; the runner, the network, carries tools (planers, jigs = weights) and door panels (activations, one per token, passed between stages). FSDP ships tools: each tool works every panel at the bench, more panels, better trips.

Two cuts, two ratios: FSDP tracks batch, TP tracks hidden size Y axis: FLOPs per byte moved over the network (our derivation, dense-layer approx.). Below the hardware line = stalled on communication FSDP (moves weights) = tokens/device T TP (moves activations) = 4.5 × hidden ÷ TP degree v7 network line (ICI) ≈ 3,845 Below: devices wait on the network Above: devices keep up 0 2,048 4,096 6,144 8,192 Tokens per device per step, T 0 2,048 4,096 6,144 8,192 ← T < 3,845: FSDP stalls FSDP 8-way TP ≈ 4,032 32-way TP ≈ 1,008 v7 network ≈ 3,845 HBM line ≈ 313 (not binding) At 70% measured ≈ 5,493 (assumed, illustrative) Small batch: switch to TP. Large batch: FSDP FSDP FLOPs/byte = tokens/device: on v7, under ~3,845 it can't keep up. More devices, same batch: FSDP eventually sinks below. TP ignores batch: only hidden ÷ TP degree. V3 (hidden 7,168): 8-way ≈ 4,032 barely clears the optimistic line; 32-way ≈ 1,008 doesn't. TP has a batch-independent ceiling.
FSDP's slope tracks batch; TP's flat lines, hidden size.
Below the red line, transfer outlasts compute.
Sources and assumptions

⚠️ Our derivation, not from a paper: FSDP moves ≈ 6Ψ bytes per step (2 bf16 weight AGs + 1 bf16 gradient RS) for 6ΨT FLOPs; TP does 4 all-reduces per layer, each sending ≈ 4Th bytes, computing 72h²T/n. Both assume dense layers, perfect overlap.

📌 v7: 2,307 TFLOP/s bf16 per chip. ICI 1,200 GB/s is the send + receive total over 6 links; we use 600 GB/s per direction (on a ring each device receives what it sends: 600 too). HBM's 7.38 TB/s is 12×+ the network, so the network binds (spec table in Inferact's TPU megakernel blog and wiki ici-dcn, both citing Google's TPU7x docs; the 6-link split and the 600 per direction are our derivation). One axis only: hardware line ~3× higher. V3's hidden size of 7,168 is from config.json.

⚠️ The denominator should be all-gather busbw measured on your own network (nccl-tests on GPUs; = data size ÷ time × (n−1)/n). The thin dashed line assumes 70% of nominal, lifting the threshold to about 5,493; the 70% is not a measured v7 number.

Red dashed line ≈ 3,845: below that many tokens per device per step, FSDP stalls. A v7 chip is 2 devices: ~7,700 per chip.

Mind the denominator: your measured bandwidth, not the spec sheet. Threshold = FLOP/s ÷ weight bytes actually gathered per second. Run a real all-gather (nccl-tests on GPUs): its busbw = size ÷ time × (n−1)/n excludes each device's own shard, so it equals actual receive bytes/s per device. At an assumed 70% of nominal, it rises 3,845 → ~5,493 (thin dashed line); an FSDP axis with few links, shared with other axes, goes higher.

A closer look: where 3,845 comes from; when it's higher

Red dashed line: 3,845 = v7's 2,307 TFLOP/s (bf16) per chip ÷ ~0.6 TB/s received per chip (the usual 1.2 TB/s counts send + receive, equal on a ring; ⚠️ this split is our estimate, see sources). The limit is the network: arriving weights are also read from HBM, but at 7.38 TB/s (12×+ the network) that line is ~313 (gray dashed, bottom), not binding. This is the optimistic line; one axis: ~3× higher. MoE on FSDP alone (no EP) moves all parameters, computes only the chosen ones: another ~18× (V3's 671B ÷ 37B).

A closer look: how “hiding behind compute” works
  • One collective, two instructions. start hands off, compute continues; done: “I need the result; wait if not here.” XLA HLO: all-gather-start / all-gather-done; PyTorch: async_op=True returns a handle, then work.wait().
  • done waits only for you: no global barrier, only your own data. Yet on a ring a slow device's data arrives late, delaying you indirectly.
  • Hiding = start earlier, done later, compute between. Prefetch: FSDP starts layer i+1's AG during layer i; backward RS is done a few layers later.
  • The cost is memory. The receive buffer is allocated at start and held until used after done; the earlier you start, the more layers are in flight. So prefetching must be throttled: PyTorch FSDP's limit_all_gathers caps AGs in flight.
  • Issued early ≠ transferred early. Channels are few; collectives on one channel queue up, so an early one behind an unfinished big one just waits. Schedule by channel queue, not just issue time.
  • Loops get in the way. JAX often writes each layer (or layer group) as a loop body (scan). start/done must pair in one iteration, so moves stay within a layer. To prefetch the next layer's weights, the compiler must move it into the previous iteration: XLA's collective pipeliner pass. ⚠️ Measured: it ignores memory when moving; overdoing it can exhaust HBM.
  • PyTorch FSDP: no loop; the runtime prefetches by layer (forward_prefetch / backward_prefetch).
  • Likewise, TP is hard to hide (§3.4): it sends activations this layer just computed, so it can't start early.

3.2 TP: cut inside the matrix

TP (tensor parallelism) cuts weight matrices; a device holds and computes a piece. NVIDIA Megatron-LM:

TP on an MLP: columns, then rows, no comm in between Two devices. Gray = full copy on both; blue / orange = each device's half; dashed = cut. Blue meets only blue, so nothing is exchanged Device 0's half Device 1's half Full copy Y = GeLU(X · W1) · W2 Megatron-LM scheme X Input X Replicated × W1: column split L → dev 0, R → dev 1 = Halved intermediate GeLU elementwise, local × W2: row split Top → dev 0, bottom → dev 1 = Partial + Partial AllReduce the only comm Y Full Y on each The trick: defer all comm to the end Columns, then rows: halves stay local through GeLU; zero comm between the matmuls. Attention: each device takes its heads, one AllReduce at the exit. So 2 forward and 2 backward, every layer, very often.
Columns, then rows: the middle stays local.
Megatron-LM's scheme; attention, split by head, is the same.
Sources and assumptions

📌 Source: Shoeybi et al., Megatron-LM, arXiv 1909.08053 sec. 3 (MLP and self-attention splits; f / g communication operators).

Communication only at the end, but every layer has one. (9-s silent Manim loop.)

Two confusions. First, every device has a share of both: device 0 has half of UP and of DOWN, not one matrix each (that's splitting by layer, PP). Nothing moves between them; DOWN's partial sums are added once, moving activations. Attention too: split by head, add at the exit. So per layer: two all-reduces forward, two backward.

Second, “horizontal”/“vertical” is just drawing. We draw W1 as input × output (X·W1), so an output-dim cut looks vertical; PyTorch's nn.Linear stores output × input, so it looks horizontal. Remember: first cut on output, second on input. Megatron, drawing X·A, calls them ColumnParallelLinear and RowParallelLinear.

The order can't flip: a nonlinear GeLU sits in between:

Why columns first: GeLU is nonlinear; add partial sums before it Same MLP: Y = GeLU(X · W1) · W2. Follow one element of the intermediate On device 0 On device 1 Bad Good Rows first: a partial sum per device (e.g. 2, −3; true −1) input after GeLU Dev 0: GeLU(2) = 1.954 Dev 1: GeLU(−3) = −0.004 True: GeLU(−1) = −0.159 GeLU each, then add: 1.954 + (−0.004) = 1.950 ✗ Over 10× off, sign flipped: sum before GeLU This element, two ways Rows first 2 −3 AllReduce −1 GeLU −0.159 Extra comm in every MLP Columns first (Megatron) −1 none no comm GeLU −0.159 All on dev 0, zero comm So 4 AllReduces per layer: 2 forward, 2 backward Forward: one at the MLP exit, one at the attention exit. Backward mirrors it: X's gradient is summed at each entry, one more each.
One element, two devices, two orders.
Rows first: communicate before GeLU.
Sources and assumptions

📌 Shoeybi et al., Megatron-LM, arXiv 1909.08053 sec. 3: GeLU is nonlinear: a row-split first matrix must sync before it. f: identity fwd, all-reduce bwd; g: reverse.

⚠️ Our counterexample: 2 and −3 are arbitrary values; GeLU is computed with the exact formula.

3.3 SP: TP's partner

SP splits TP's leftover small ops by sequence: same traffic, less activation memory. Not a fourth cut, despite the name (§8.8).

A closer look: what SP splits; why traffic stays equal

TP can't split the small per-token ops (normalization, dropout, residual add), so there every device holds full activations. Megatron's sequence parallelism (SP) splits these by sequence, and each all-reduce in two: all-gather to rejoin before the TP block, reduce-scatter to split after. By §1's identity, traffic is unchanged.

3.4 TP's ceiling ignores batch

With TP, the runner carries panels (activations): craftsmen share panels, each doing part, then combine. Double the panels, and handoffs and work both double: batch cancels. Panel size is hidden size: bigger panels, more work per handoff. Tools can come early (next layer's during this one); fresh panels by default wait for the stage to end: TP traffic is hard to hide. What's left: FLOPs per byte ≈ 4.5 × hidden ÷ TP degree (⚠️ our derivation, dense layers). V3 (hidden 7,168) at 8-way TP: ~4,032. The drawn red line is most optimistic, so 8-way seems to touch it (7,168 × 4.5 ÷ 3,845 ≈ 8.4); the real line is higher, with 8-way below it (see below).

Small batch leans on TP, large batch on FSDP: left, blue is below red, so use orange; right, it has crossed, so FSDP. This assumes TP's line clears red; when both sit on or below it, as for V3, TP doesn't pay (§3.7).

A closer look: why 8-way TP is really below; two more degree limits
  • ① 3,845 assumes all three axes fully used, each a closed ring. 8-way TP spans only 4 chips (one v7 chip is 2 devices), so two discounts stack: the slice is too small for a ring (§1.5) and uses only one or two axes (§3.1). Real bar: several times higher.
  • ② Megatron by default waits for TP traffic before computing: it can't hide.
  • Attention heads split whole: Megatron needs query heads divisible by TP degree (fewer KV heads: one of KV heads/TP divides the other; bigger TP copies KV).
  • TP syncs every layer: it must stay in the fastest interconnect tier (GPUs: usually one NVLink domain).

3.5 FSDP + TP: a 2D split

Large models often use both: they cut different dimensions. Grid the devices: rows are TP groups, columns FSDP groups. TP slices weights into output-dim strips; FSDP splits each on the input dim.

FSDP + TP cut different dims—1/8 resident, gather only your strip 8 devices, 2 rows × 4 columns: row = TP group (same data), column = FSDP group (two halves of one strip). Example: MLP's W1 TP 0 strip TP 1 TP 2 TP 3 ① The split: TP cuts 4 vertical strips, FSDP halves each Full W1 (in dim × out dim) Out dim: TP, 4 strips In dim: FSDP, two halves TP 0 TP 1 TP 2 TP 3 Data shard 1 Holds 1/8 row 1 · col 1 Holds 1/8 row 1 · col 2 Holds 1/8 row 1 · col 3 Holds 1/8 row 1 · col 4 Data shard 2 Holds 1/8 row 2 · col 1 Holds 1/8 row 2 · col 2 Holds 1/8 row 2 · col 3 Holds 1/8 row 2 · col 4 Same row: same data, different strips (TP). Same column: same strip, different data (FSDP) ② One layer: gather in column, compute, sum in row ① AllGather in column (FSDP) ② Local compute ③ AllReduce in row (TP) gather Gathers strip 2: 1/4 of the matrix, not all of it Same data X (shared by row) Each computes 1/4 of the intermediate; GeLU local No comm here Partial sum Partial sum Partial sum Partial sum Sum = Y W2 gives partial sums; add once Moves activations, not weights ③ The math: TP lends a row's tokens to FSDP FSDP only Each gathers a full layer, used only by its own T tokens Threshold: T ≥ 3,845 per device FSDP × TP 4 Each gathers 1/4 strip, used by all of the row's tokens Threshold: ≥ 3,845 tokens per row, ~961 per device Compatible and complementary: at small batch, TP props up FSDP FSDP splits the input dim, gathers in columns; TP splits the output dim, sums in rows. No conflict: 1/8 resident, 1/4 strip gathered. The cost sits on the TP axis: 2 syncs forward, 2 backward, per layer, unhidden, so TP takes the fastest links and FSDP the rest.
FSDP gathers in columns, TP sums in rows; you gather only your strip.
Per the Scaling Book (training); 961 is our derivation.
Sources and assumptions

📌 Scaling Book (How to Scale Your Model), training chapter, “Combining FSDP and Tensor Parallelism”: In[B_X, D_Y] · W_in[D_X, F_Y] · W_out[F_Y, D_X], with FSDP on the X axis and TP on Y. The book moves activations over the TP axis with AllGather + ReduceScatter, the same volume as one AllReduce.

⚠️ Our derivation: a row shares tokens, so FLOPs per byte on the FSDP axis = the token count of that row; 961 = 3,845 ÷ 4. The axes use different links, so each threshold must be recomputed from its own links' measured bandwidth, and can only go up.

Per layer: all-gather in the column to rebuild your strip (1/TP of the matrix), compute alone, add partial sums once in the row. Backward reduce-scatters gradients along columns to owners.

The math shifts: a row shares tokens but each device gathers only a 1/TP strip, so the FSDP threshold counts a whole row's tokens: ~961 per device at 4-way TP (⚠️ our derivation; measured per-axis bandwidth not applied). Hence “small batch leans on TP”: TP lends a row's tokens to FSDP. The price: TP syncs every layer, needing the fastest links.

3.6 PP: split by layer

Many machines, slow links, no per-layer weight traffic like FSDP: split by layer. TP cuts wide, PP deep:

TP cuts wide, PP cuts deep: one syncs every layer, one at boundaries Same 8-layer model, 4 devices. Color = owning device Dev 0 Dev 1 Dev 2 Dev 3 Communication Cut wide (TP): every layer in four strips L1 L2 L3 L4 L5 L6 L7 L8 Every layer, all four sync (1 AllReduce each: attn, MLP) Comm every layer → fastest links only Cut deep (PP): two layers per stage, one stage per device L1 L2 L3 L4 L5 L6 L7 L8 Hand off only at stage boundaries (point-to-point) Only 3 comm points → slow links OK; cost: idle waits One cuts wide, one cuts deep; each pays its own price Wide: each device does a quarter of every layer but syncs with the other three every layer, so TP stays in one machine. Deep: devices barely talk, so PP can span many machines; the cost is idle time while the pipeline fills and drains.
Red dots = communication: one per layer left, three right.
Sources swap horizontal and vertical; we say wide/deep.

Stages pass activations only at boundaries, point-to-point: the least traffic of any cut, so PP can cross slow links. Naively: “4 devices' memory, 1 device's speed”, one stage working at a time. So feed in micro-batches, production-line style. Planing, drilling, painting: one station each; panels flow in stacks. At first later stations wait; at the end earlier ones idle. That idle time is the bubble:

PP's cost, the bubble: wait to fill, wait to drain 4 stages, 8 micro-batches, GPipe schedule (all forwards, then all backwards). Backward drawn 2× as long Fwd Bwd Bubble: stage idle What each stage does in one step stage 0 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 stage 1 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 stage 2 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 stage 3 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 time → Bubble ÷ ideal compute = (p−1) ÷ m = 3 ÷ 8 ≈ 38% The bubble is pure loss: it can be amortized, not eliminated More stages, fewer micro-batches, bigger bubble: (p−1) ÷ m. So PP needs enough micro-batches; at a fixed batch, more of them means smaller ones; too small starves the device. Later schedules attack it: interleaved (VPP) splits stages finer, Zero Bubble fills it with weight grads, DualPipe feeds both ends. The price: more complex scheduling, more P2P traffic.
Light gray is the bubble: an idle stage.
4 stages, 8 micro-batches: bubble = 3/8 of ideal compute.
Sources and assumptions

📌 Bubble fraction: Narayanan et al., arXiv 2104.04473 sec. 2.2 (bubble time fraction, GPipe / 1F1B); interleaving further divides the bubble by v, the virtual stages per device. Zero Bubble: arXiv 2401.10241. DualPipe: github.com/deepseek-ai/DualPipe.

4 → 8 micro-batches halves the gray bubble. (14‑s silent Manim loop.)

Bubble ÷ ideal compute = (p−1) ÷ m (p stages, m micro-batches). More help, but too-small ones underfeed the device.

A closer look: why activations pile up; three common fixes

Every row idles the same total: p−1 forwards + p−1 backwards. Original GPipe keeps each micro-batch's activations until its backward; at fixed micro-batch size, more of them store more. 1F1B caps those in flight at p: same bubble, less memory. Oddly, PP splits weights, yet per-device activations barely shrink: each stage holds several micro-batches.

VPP gives each device non-adjacent layer chunks: bubble ÷ chunks per device. Zero Bubble splits backward into input/weight gradients; the latter can wait to fill gaps. DeepSeek-V3's DualPipe feeds both ends of the pipeline, mainly to hide EP traffic under paired forward–backward compute (tech report sec. 3.2.1).

Going deeper: micro-batch source and size; 1F1B scheduling; uneven stage memory; DualPipe's cost

Micro-batches weren't invented for pipelines. They began as gradient accumulation: a replica that can't fit its batch share at once runs it piece by piece in time, accumulating gradients, then updates once. Pipelines borrowed this to keep stages busy. Three levels: global batch = micro-batch size × count × DP replicas. With PP, “per-device batch” is just bookkeeping: the stages form one replica on the same data. In MaxText, global batch = per_device_batch_size × device count; micro-batch = global batch ÷ num_pipeline_microbatches (must be a multiple of the stage count; defaults to it).

GPipe's extra memory: a precondition. At fixed total batch, finer splits don't change its activations (more, smaller micro-batches). The problem is a fixed micro-batch size: more of them mean a bigger batch, more activations. 1F1B's depend only on stages × micro-batch size.

How 1F1B is scheduled. Stage k (from 0) runs p−1−k warm-up forwards, then alternates one backward, one new forward (one out, one in), then drains the remaining backwards. So the first stage holds at most p micro-batches, the last 1. Why not have stage 0 send more while waiting? Steady state finishes one micro-batch per forward + backward, and a stage-0 round trip takes about p such periods. In flight = rate × time ≈ p. Extras just queue: more memory, no speed.

Uneven memory across stages. First stage holds p, last 1. Fixes: unequal layers in first/last stages (the last also has output layer, loss); recompute only early stages; park activations in host memory; or a V-shaped placement (each device owns an early and a late stage) to flatten activation peaks (Qi et al., Pipeline Parallelism with Controllable Memory, arXiv 2405.15362, Zero Bubble's sequel).

How big a micro-batch. Not “one that fills HBM”: under 1F1B, stage 0 holds p at once. Pick the smallest that fills one stage's compute. An LLM sequence is thousands of tokens, matrices are big; one or two often do; measure the knee. p = 4, 64 sequences per replica: 16 each → 4 micro-batches, bubble 3/4 of ideal time; 1 each → 64, just 3/64. Rule: micro-batches ≥ 4× stages; else VPP, fewer stages, or Zero Bubble-style ones.

What DualPipe costs. Tech report Table 2: each device stores two parameter copies (stage i forward, stage i from the end in reverse); peak activations p+1 micro-batches (1F1B: p). Aim: hide EP's ~1:1 cross-node traffic, trim the bubble. V3 can: wide EP leaves few weights per device. Chimera (2021) proposed bidirectional pipelines; DualPipe needs only even stage and micro-batch counts.

PP is for scale over slow links. Where the fast interconnect is big enough, skip it: a TPU slice puts 1,000+ chips on high-speed ICI—room for FSDP and TP—with DP between slices. So MaxText's ici_pipeline_parallelism and dcn_pipeline_parallelism default to 1; typical configs skip PP and its headaches. GPUs need PP because TP can't leave NVLink; slow cross-node links go to PP.

3.7 What this cut leaves open

Our running example, DeepSeek-V3, uses no TP; per its report, memory optimizations made costly TP unnecessary. Nearly all parameters sit in narrow expert matrices; splitting them shrinks pieces, not activation traffic. Cut along experts instead: the third cut.

Check yourself: few tokens per device—FSDP or TP: which stalls on traffic first? Why is TP stuck in one machine?

FSDP: FLOPs per byte moved = tokens per device; few tokens, transfer can't hide. TP syncs every layer; panels pass only after the stage ends, so it can't hide and must stay on the fastest links.

Section 4

Third cut: experts

MoE parameters sit almost all in the experts; the third cut puts different experts on different devices. It adds one kind of traffic, all-to-all, but brings the worst problem of any cut.

4.1 Why split the experts

Apart from its first 3 layers, V3's other 58 are MoE: 256 routed experts (8 picked per token) plus 1 shared expert all tokens use.

V3 is almost all experts, so cut along the experts Bar to scale. Each expert = 3 matrices, 7,168 × 2,048 (gate / up / down) Routed Rest: attn, shared expert, dense MLP, vocab 671B, to scale Routed experts ≈ 653.9B (≈97%) Rest ≈ 17.1B (≈3%) ↑ One expert ≈ 44.04M params × 256 × 58 layers V3 trained with no TP at all Report: careful memory savings made costly TP unnecessary. We add: 2,048-wide experts make TP worse still; 256 experts split naturally. V3 training: 16-way PP + 64-way expert parallelism + ZeRO-1 data parallelism, no TP (tech report, sec. 3.2).
Gray sliver = attention, shared expert, dense MLPs and vocabulary combined.
Routed share computed from config.json sizes.
Sources and assumptions

📌 config.json: hidden_size 7,168, moe_intermediate_size 2,048, n_routed_experts 256, num_hidden_layers 61, first_k_dense_replace 3. 671B = main-model total, no MTP module. “Rest” = 671B − routed experts, computed by this script.

256 routed experts × 58 layers ≈ 653.9B parameters, ~97% of the total. TP targets one wide matrix; V3's bulk is many narrow ones, best handed out whole to devices: move the patients (tokens), don't split the doctors (experts). V3's real config: PP16 + EP64, no TP (last line).

Section 2's library: EP deals a room's 256 expert shelves to 64 branches, 4 whole shelves each; PP cuts the 61 rooms into 16 ordered stages. Whole units, usable on arrival, not FSDP's shred-and-rebuild.

4.2 EP: tokens fly to the experts

Expert parallelism (EP) communicates only in MoE layers, twice per layer, via §1.6's all-to-all, the pattern networks dread most: dispatch sends each token to its expert's device; combine sends results home.

Dispatch, queue, combine: the longest queue stalls all. Color = home device; below: its experts. (10 s silent Manim loop.)

V3 tames both: at most 4 machines per token, one copy each, fanned out on arrival; dispatch in FP8, combine in BF16.

Dispatch: one copy per machine, fan out inside Each token picks 8 routed experts; 8 devices per machine. Left: generic case, experts widely spread. Right: V3 Source device Has the expert Slow: cross-machine IB Fast: in-machine NVLink Red box = first hop (same index) No limit, BF16: 8 experts, 8 machines, 8 copies Source machine 8 copies × 14,336 B ≈ 115 KB V3: ≤ 4 machines, FP8: 4 copies Source machine (dev 3) 4 copies × 7,168 B ≈ 28.7 KB (1/4 of left) Why: send less across, fan out inside NVLink 160 GB/s ≈ 3.2× cross-machine IB at 50 GB/s: one copy that lands can be forwarded to 3.2 experts on average at no extra cost. So the report says the same traffic allows up to 13 experts (4 machines × 3.2): cost depends on machines visited, not experts chosen.
Same token: 8 cross-machine copies left, 4 right at half size.
Green in-machine arcs are nearly free.
Sources and assumptions

📌 DeepSeek-V3 tech report, arXiv 2412.19437: sec. 2.1.2, node-limited routing (machines ranked by top-2 affinity sum); sec. 3.2.2, IB to the target machine's same-index device, then NVLink forwarding, NVLink 160 / IB 50 GB/s, 13 experts; sec. 3.3.3, FP8 dispatch, BF16 combine; sec. 4.2, M = 4, 64 devices, 8 machines.

⚠️ Our derivation: 28.7 KB = 4 × 7,168 × 1 byte; 115 KB = 8 × 7,168 × 2 bytes (no node limit, spread wide: 8 experts may be on 8 other machines; V3 puts each layer's experts on just 8 machines incl. source, so ≤ 7 cross-machine copies). Red box: same-index device reached first across machines; picks illustrative.

Cost tracks machines visited: one copy each, which NVLink forwards to its experts. The report puts NVLink at ~3.2× cross-machine speed, so each machine feeds 3.2 experts on average for free: 4 × 3.2 ≈ 13, and equal traffic allows up to 13 experts (V3 picks 8).

That's training and prefill. Latency-bound decode skips in-machine forwarding: one copy per expert, so copies grow with experts picked.

4.3 EP's worst problem: data-driven load

EP's imbalance is worst: unlike other cuts, who is busy shows only after routing, and changes every batch. All wait for the busiest expert. It snowballs: popular experts get more tokens, learn more, draw more; others wither. Big-model memory buys a small-model result: a quality problem, not just speed. A two-part cure:

Bias as a ticket at the door: sets the order, not the diagnosis Sim: pick 2 of 8, 1,024 tokens/step, popularity falls from expert 0 to 7. Each step: overloaded bias down, underloaded up Tokens taken this step Average Step 0: busiest = 2.26× average Expert 0 1 2 3 4 5 6 7 Step 30: busiest = 1.30× average Expert 0 1 2 3 4 5 6 7 Step 150: busiest = 1.08× average Expert 0 1 2 3 4 5 6 7 The busiest expert sets everyone's wait The old fix penalizes imbalance in the training loss: like forcing triage to change its diagnoses, it hurts what the model learns. V3 biases only the selection scores, not the gating weights: it sets the queue order, not the diagnosis. Bias frozen for the last 500B tokens.
Red dashes = average: flatter bars, less waiting.
Illustrative simulation: 2 of 8, faster than V3 so it shows in tens of steps.
Sources and assumptions

📌 Auxiliary-loss-free load balancing: Wang et al., arXiv 2408.15664; V3 tech report, arXiv 2412.19437 sec. 2.1.2, sec. 4.2 (the report calls this step the bias update speed γ: 0.001 for the first 14.3T tokens, 0 for the last 500B). Writing the step as u follows Algorithm 1 of Wang et al. (bias update rate u, also 0.001 in their experiments); this figure keeps that notation.

⚠️ Illustrative simulation (fixed random seed, pick 2 of 8, u = 0.02), not V3's measured load.

4.4 Attention and experts split separately

Attention and experts differ entirely in shape (outside MoE, the EP dimension is just DP): attention's load scales with sequence and KV cache, the experts' with sheer parameter count. So the same devices can split them differently.

Attention vs experts: same devices, two mappings 8-device sketch: in attention a device belongs to a TP group; in the expert layer it takes another role Group 1: request A Group 2: request B Square = one expert Each device plays two roles per layer Attention: TP groups of 4, each with its own requests Dev 0 Dev 1 Dev 2 Dev 3 Dev 4 Dev 5 Dev 6 Dev 7 Request A Request B AllToAll Expert layer: 8 devices × 32 experts; all requests' tokens come here Dev 0 Dev 1 Dev 2 Dev 3 Dev 4 Dev 5 Dev 6 Dev 7 One big matrix vs many small: split each its own way Attention's big matrix is split into strips over 4 devices; each group serves its own requests. The 256 experts are simply handed out whole. Megatron: Parallel Folding (training). Inference: TP attention = TEP, data-parallel attention = DEP; half-and-half setups like this exist too.
Same 8 devices on both sides.
TP groups in attention; 32 experts each in the expert layer.
Sources and assumptions

📌 Megatron-Core moe/README.md (MoE Parallel Folding, example TP4·CP2·DP8·PP4 → ETP1·EP64·EDP1); paper: arXiv 2504.14960. TEP / DEP definitions: TensorRT-LLM tech blog 26.

How much does it matter? Two names, then one measurement (box below).

The two most common inference setups: TEP (TP attention, EP experts) and DEP (DP attention, EP experts), suffixed by device count. Experts match; attention differs: one shared request batch, or one per device?

TEP vs DEP in attention: one shared batch, or one each Same 4 devices, same 4 request batches (color = batch); experts the same on both sides: 8 over 4 devices, 2 each Req 1 Req 2 Req 3 Req 4 TEP4: TP attention, all four on one batch Requests Attention KV cache Experts Dev 0 1/4 heads All KV (full) E0 E1 Dev 1 1/4 heads All KV (full) E2 E3 Dev 2 1/4 heads All KV (full) E4 E5 Dev 3 1/4 heads All KV (full) E6 E7 Attention out AllReduce: sum the four partials Expert step All devices have all tokens; local experts, then AllReduce Pro Four devices per request: each request runs faster Con KV can't be split, full copy per device: 3 of 4 wasted DEP4: data-parallel attention, one batch each Requests Attention KV cache Experts Dev 0 All heads Own only E0 E1 Dev 1 All heads Own only E2 E3 Dev 2 All heads Own only E4 E5 Dev 3 All heads Own only E6 E7 Attention Independent, no comm. Expert step AllToAll dispatch to expert devices, then combine back Pro No duplicated KV: more requests fit, high throughput Con Uneven request lengths, uneven load; idle devices still run The difference is all in attention: one batch together, or one each TEP: four devices share each batch, so a single request is fast; but MLA-style models have one KV head, so every device stores the full KV. DEP: each device serves its own requests, no duplicated KV, bigger batches; the price is an AllToAll in the expert step and guarding against uneven load.
Same experts; in attention, TEP's devices work together, DEP's alone.
Illustrative: 4 devices, 4 batches, 8 experts.
Sources and assumptions

📌 TensorRT-LLM definitions: TEP<N> shards both attention (TP) and experts (EP) across N ranks; DEP<N> keeps attention data-parallel (ADP) while distributing experts across N ranks.

⚠️ Expert-step comms as usually built: TP attention, all devices hold the same tokens, so expert outputs all-reduce; data-parallel attention, tokens differ, so all-to-all. Illustrative: 4 devices, 8 experts.

TEP spreads each request over devices: fast generation. But attention splits by head; V3's KV has one head, so every device holds it in full (§5.4). DEP keeps only each device's own requests' KV: more requests fit, experts get bigger batches. The price: all-to-all for experts, uneven load from varied request lengths, idle devices running empty forward passes. The “same devices, two mappings” figure's left setup (attention TP4 × DP2) sits between. Not Megatron's ETP/EDP (§8.6).

One measurement: a new split doubles per-device throughput DeepSeek-V4-Pro, GB300, vLLM: moving decode from TP4 to DEP8 (DP8 attention, EP8 experts) gave 2.09× a tuned TP4's per-device throughput at equal concurrency. The main gain is attention's KV, no longer copied to each of 4 devices (§5.4). Per-device attention weights are a few percent of an MoE model. Figures show V3 (32 experts/device); the test used V4-Pro.
Full accounting (devices, concurrency, time to first token) in §7.3.

4.5 What this cut leaves open

The first three cuts split batch, weights and experts, never the sequence. But activations (training) and KV cache (inference) grow with context. What if one sample outgrows one device?

Check yourself: EP's worst problem? How does V3 treat it, and why doesn't that hurt learning?

Load imbalance: all wait for the busiest expert. V3 biases expert scores, down if overloaded, up if under; it alters only routing, not the computed weights.

Section 5

Fourth cut: sequence

One sample outgrows a device? Split its sequence: the fourth cut. Training and inference split different things.

5.1 Why one sample won't fit

5.2 Training: CP cuts activations by sequence

Context parallelism (CP) cuts a long sequence into chunks, one chunk's activations per device. Only attention is hard: every token must see all earlier ones, held elsewhere. Two fixes:

Q stays put, KV circles the ring once, each device fills its row. (9 s silent Manim loop; no causal mask.)

Why no note-passing in Ulysses? In a “chunk × head” table, later chunks see earlier ones down a column; heads in a row never interact. Ring splits rows, so dependencies cross devices; Ulysses transposes, giving each device whole columns.

Ulysses: transpose once and the notes stay home. Ring splits by row, Ulysses by column Attention work as a table: row = sequence chunk (S, a run of tokens), column = head (H). Four colors = four devices Dev 0 Dev 1 Dev 2 Dev 3 ① Meet the table: columns have dependencies, rows don't Attention work H0 H1 H2 H3 S0 S1 S2 S3 Same column: later chunks need earlier ones Across chunks → pass notes (K, V) Same row: each head on its own Across heads → separate, no talking Ring: split by row H0 H1 H2 H3 S0 S1 S2 S3 Each device: 1 chunk × all heads To see other chunks, the notes make a lap of the ring The key: split by column, and each column's dependencies land on one device. That is what Ulysses does. ② Ulysses: one AllToAll before, one after On entry: split by chunk H0 H1 H2 H3 S0 S1 S2 S3 After AllToAll: split by head H0 H1 H2 H3 S0 S1 S2 S3 AllToAll again: back to chunks H0 H1 H2 H3 S0 S1 S2 S3 AllToAll AllToAll Each device: 1 head × all chunks Column fully local: attention runs here Dev k sends cell j of its row to dev j: keeps 1, sends 3 Per-token ops after that are split by chunk again Ring passes notes; Ulysses changes the split Upside: double the sequence and the devices, and per-device traffic stays the same; attention itself never waits on anyone. Cost: an AllToAll before and after every layer (the pattern networks hate most), and devices ≤ heads; with fewer KV heads than query heads, the KV head count is the cap.
Columns depend, rows don't: split by column; attention asks no one.
Schematic: 4 chunks × 4 heads, 4 devices.
Sources and assumptions

📌 DeepSpeed-Ulysses: Jacobs et al., arXiv 2309.14509 (all-to-all around attention; per-device traffic is flat as length and devices scale together). Ring Attention: Liu et al., arXiv 2310.01889.

⚠️ Schematic: 4 chunks × 4 heads, 4 devices; each real cell is a whole token chunk's Q, K, V on one head.

All-to-all, step by step: first how device 0 sends, then everyone at once; afterward each device holds every chunk of one head. (26 s, Manim; untick “Autoplay” to step.)

Ring: people stay, notes rotate. Ulysses: re-cut so each holds a few heads of every note.

Stacked, they make USP: block the table over a 2D device grid, Ulysses' all-to-all inside a machine, a ring across; device count is the product. Megatron's CP supports this hierarchy too.

USP stacks Ulysses and Ring. Table in blocks; transpose inside, ring across machines Same “chunk × head” table. 4 devices, 2 machines: total devices = Ulysses dim (2) × ring dim (2) Dev 0 (machine 1) Dev 1 (machine 1) Dev 2 (machine 2) Dev 3 (machine 2) ① Two steps: AllToAll inside, then ring across On entry: split by chunk H0 H1 H2 H3 S0 S1 S2 S3 After in-machine AllToAll: blocks H0 H1 H2 H3 S0 S1 S2 S3 In-machine AllToAll (Ulysses 2) Dev 0↔1, dev 2↔3 swap columns Ring across machines (Ring 2) Blocks with the same heads swap notes: Dev 0 ↔ dev 2 (H 0, 1), dev 1 ↔ dev 3 (H 2, 3) Each device: 2 chunks × 2 heads Heads already split; ring handles the other chunks After attention, one more in-machine AllToAll restores the chunk split. ② All three side by side Split Communication Limit Best placed Ring By row (chunk) Notes passed along the ring None Can span machines Ulysses By column (head) AllToAll before + after Devices ≤ heads In-machine (fast lane) USP By block In-machine AllToAll + cross-machine ring Only Ulysses dim ≤ heads Each on its own tier USP = two cuts stacked: total devices = Ulysses dim × ring dim The head-count cap binds only the Ulysses dim; the ring dim can grow freely, so a model with few heads can still be split across many devices. AllToAll suffers most on slow links, so it stays in-machine; a ring step talks only to a neighbor and overlaps with compute, so it goes across machines.
Transpose inside, ring across: the head cap binds only one dimension.
“AllToAll inside, ring across” is our summary of common practice.
Sources and assumptions

📌 USP: Fang & Zhao, arXiv 2405.07719; xDiT docs: ulysses-degree × ring-degree = sp-degree, lifting the rule that sp-degree must be below the head count; friendlier to mixed networks.

⚠️ “All-to-all inside the machine, ring across machines”: a common layout we infer from each traffic type, not the paper's rule.

Swap heads inside each machine first, then pass notes between machines: AllToAll inside, only the ring across. (27 s, Manim; untick “Autoplay” to step.)

5.3 Causality unbalances the load

Generative attention is causal: tokens see only the past. Later chunks work harder, so a contiguous split swamps the last device. Fix: one light, one heavy block per device (zigzag, below).

Causal attention split by sequence: contiguous is uneven, zigzag evens it out exactly Sequence cut into 2 × 4 = 8 blocks. Row = query block, column = KV block, each cell = one attention pair; causal mask leaves the lower triangle Dev 0 Dev 1 Dev 2 Dev 3 8 × 8 causal attention blocks: who computes which rows Contiguous: dev r takes blocks 2r, 2r+1 block 0 → dev 0 block 1 → dev 0 block 2 → dev 1 block 3 → dev 1 block 4 → dev 2 block 5 → dev 2 block 6 → dev 3 block 7 → dev 3 Dev 0: 3 cells Dev 1: 7 cells Dev 2: 11 cells Dev 3: 15 cells Zigzag: dev r takes blocks r and 7−r block 0 → dev 0 block 1 → dev 1 block 2 → dev 2 block 3 → dev 3 block 4 → dev 3 block 5 → dev 2 block 6 → dev 1 block 7 → dev 0 Dev 0: 9 cells Dev 1: 9 cells Dev 2: 9 cells Dev 3: 9 cells Contiguous: several-fold gap; zigzag: perfectly even Causal mask: later blocks do more work. Contiguous, dev 3 computes 15 cells, dev 0 only 3; everyone waits for dev 3. Zigzag pairs the lightest block with the heaviest on one device, so each gets 9 cells. This is how Megatron's CP splits.
Same 8 blocks, reassigned: busiest vs. idlest, several-fold → equal.
Cells counted by script from the causal mask.
Sources and assumptions

📌 Megatron-LM megatron/core/utils.py: sequence cut into 2×cp_size blocks; rank r gets blocks r and 2·cp−r−1; per docs/user-guide/features/context_parallel.md no redundant lower-triangle compute, balanced load. Script counts cells, diagonals as whole; halving diagonals: 2/6/10/14 vs. 8/8/8/8, contiguous split: a 7× gap.

5.4 Inference: TP copies the KV cache

In long-context, high-concurrency decode, KV often uses the most memory. TP splits by head, DCP by token. TP deals heads to devices: V3's 128 query heads split easily, but MLA's KV is one compressed copy shared by all heads: one unsplittable head.

Many models (Llama 2 70B) have 8 KV heads: 8-way TP gives each device one. Once TP exceeds the KV head count H, each head is duplicated TP ÷ H times (vLLM docs: “duplicated for tp_size / H times”). MLA's single KV head under 8-way TP: 8 identical full copies, one per device. Hence V4-Pro on TP4 (§4.4) wastes 3 copies.

Inference: TP duplicates the KV cache; DCP turns copies back into capacity DeepSeek-V3, one 128K-context request. KV per token = (512 + 64) × 61 layers × 2 bytes = 70,272 bytes Duplicate copies The one copy needed This request's KV on each of 8 devices TP 8 Dev 0 Dev 1 Dev 2 Dev 3 Dev 4 Dev 5 Dev 6 Dev 7 8.58 GiB/device Total 68.6 GiB (8 identical) TP 8 + DCP 8 Dev 0 Dev 1 Dev 2 Dev 3 Dev 4 Dev 5 Dev 6 Dev 7 1.07 GiB/device Total 8.58 GiB (exactly one) TP beyond the KV head count adds only copies MLA's KV has one head. With 8-way TP every device stores a full copy; 7 copies are wasted. DCP spreads KV round-robin by token over 8 devices, 1/8 each; it adds no devices, reusing the TP ones.
The 7 red copies hold identical data.
KV sizes from V3's config.json.
Sources and assumptions

📌 vLLM context-parallel deployment docs: KV cache duplicated tp_size / H times; DeepSeek-R1 with MLA has 1 KV head, so -tp 8 = 8× copies; add -dcp 8. KV sizes from V3's config.json, script-computed.

DCP (decode context parallelism) stores KV round-robin by token across existing devices. Token, not chunk, because notes keep growing and new ones must spread evenly too.

KV lands on four devices in turn; a few extra collectives per layer gather Q, merge results. (11 s silent Manim loop; TP alone first, for contrast.)

The new token's Q goes to every device; each computes attention on its KV share, and the partials merge. Attention is a score-weighted average, so the rule is “don't divide yet”:

Merging blockwise attention, don't divide yet: add numerators, add denominators, divide once at the end Attention = score-weighted average. Three tokens: exp scores 2, 1, 1; values 10, 4, 20. Dev 0 holds the first two, dev 1 the third Dev 0 Dev 1 ✗ ✓ Each device works on its own notes Dev 0: 2×10 + 1×4 Numerator 24 Denominator 3 Dev 1: 1×20 Numerator 20 Denominator 1 numerator = Σ weight×value, denominator = Σ weight (log of the denominator = the familiar LSE) Divide each, then average Dev 0: 24 ÷ 3 = 8 Dev 1: 20 ÷ 1 = 20 Average: (8 + 20) ÷ 2 = 14 ✗ Dev 1 has one token but half the vote Don't divide yet: once at the end Numerator: 24 + 20 = 44 Denominator: 3 + 1 = 4 44 ÷ 4 = 11 ✓ Same as all 3 tokens on one device Each device hands over its denominator with its result Like merging two classes' average scores: bring each class size; don't average the averages. Ring merges this way chunk by chunk en route; DCP merges at the end. Same trick.
Averages don't average without their “headcounts.”
Send numerators and denominators; divide once.
Sources and assumptions

📌 Framing: Ring Attention appendix code (Liu et al., arXiv 2310.01889) sums numerator, denominator apart; numerically stable blockwise softmax: FlashAttention (arXiv 2205.14135).

⚠️ Our example: arbitrary numbers, computed from the definition and asserted.

Three extra collectives per layer (vLLM default: gather Q, swap softmax denominators, combine outputs) reclaim the KV that TP duplicated; DCP degree maxes at TP ÷ KV heads, exactly the duplicates. Again: communication for memory.

Batching still works: 1,000 requests = 1,000 rows, split alike; each device does its 1/TP of all 1,000 in one step. It fits: 1/TP of the KV per device means longer contexts, more requests, bigger batches. It reads less: under MLA, plain TP has every device read the whole KV (different heads, same copy); DCP reads 1/TP, so long-context attention speeds up too.

5.5 Prefill splits its sequence too

DCP is for decode; splitting a long prompt in prefill, for a sooner first token, is PCP (prefill context parallelism). One letter apart, they're consecutive steps. In a “question × notes” table, prefill's thousands of questions split by row; decode's lone question leaves no rows, so DCP further splits that row.

PCP and DCP: split rows; at one row, split the row. DCP isn't PCP's peer but the step after rows run out Attention is a “question × notes” table: row = a new token's question (Q), column = an earlier token's notes (K, V); tokens look only back, hence the triangle Dev 0 Dev 1 Dev 2 Dev 3 PCP (prefill): thousands of Qs, split by row (e.g. 4,000 rows → 1,000 per device) Notes (K, V) → Qs Dev 0 Dev 1 Dev 2 Dev 3 Prefill: thousands of Qs at once Tall, wide table: lots of work Rows split over 4 devices: each row finishes on one device, no merging needed But dev 3's two rows (red) need every note to the left: gather earlier KV first Goal Speed: first token sooner Devices New dim on top of TP: more devices Extra comm. Gather earlier chunks' KV DCP (decode): only 1 Q left, no rows to split; split that row by column This step: 1 new Q per request (a one-row table), plus a long run of stored notes Notes round-robin by token: note 0 → dev 0, note 1 → dev 1… (same 4 TP devices, none added) Dev 0 → part of the row: numerator + denominator Dev 1 → part of the row: numerator + denominator Dev 2 → part of the row: numerator + denominator Dev 3 → part of the row: numerator + denominator 4 partials → don't divide yet: sum numerators and denominators, divide once In practice: 1,000 requests = 1,000 rows, all split alike; each device holds 1/4 of each one's notes Goal Fit, read less: 1/4 of KV per device, 1/4 read per step Devices Reuses the TP devices; none added Extra comm. Gather Q, swap denominators, merge Split rows first; down to one row, split that row PCP: prefill has thousands of Qs, split by row; each row finishes on one device with no merge, but must gather earlier notes; extra devices, to get the first token out sooner. DCP: decode has 1 Q per step, no rows to split, so it splits that row by column (notes), computes pieces and merges by don't-divide-yet; same devices, to make it fit.
Many Qs: split rows. Decode's single Q: split the row itself.
Schematic: 8 questions, 16 notes, 4 devices; real PCP also zigzags rows for balance.

Section 6 puts prefill and decode on separate machines, one each (our synthesis).

5.6 What this cut leaves open

Prefill and decode now want different splits: compute-hungry prefill a split prompt, bandwidth-hungry decode spread-out KV. Shared devices fit neither well. So stop splitting tensors; give them separate machines: the fifth cut.

Check yourself: two devices each hold a partial attention result. Just average?

No. “Don't divide yet”: each device sends numerator and denominator; sum each, then divide once.

Section 6

Fifth cut: split the work, not the tensors

Cuts 1–4 split tensors (batch, weights, experts, sequence); cut 5 splits work. A request's stages differ in character: put them on different machines, each with its own sharding.

6.1 Why prefill and decode fight

Two phases per request. Prefill runs thousands of prompt tokens through matmuls at once: compute-bound. Decode emits one token a step yet reads all weights and KV from HBM: bandwidth-bound.

Your answer is streaming word by word; someone pastes a hundred-page manual, and it stops. Voice assistants suffer more: halting speech feels slow at any average. On shared devices the engine must pick one job per step; a long prefill holds several steps while every request mid-generation waits: new requests' TTFT (time to first token) and others' TPOT (time per output token) worsen.

Decode rereads the weights from HBM each step for one use by the batch. How big a batch makes that pay?

Why decode insists on big batches: each weight read feeds just one batch x: requests generating together in one step, b (log scale). y: FLOPs per byte of weights read from HBM (our derivation; weight reads only) Dense: = b One MoE expert: = b ÷ 32 (V3 picks 8/256) v7 HBM line ≈ 313 1 4 16 64 256 1,024 4,096 16,384 Requests per step, b 1/16 1 16 256 HBM line ≈ 313 Dense: saturates at b ≈ 313 MoE: saturates at b ≈ 10,000 Above: compute keeps up Below: device waits on HBM to read weights So every decode technique is about assembling a big batch PD disaggregation: decode machines only generate, so batches grow. DEP: each attention rank keeps its requests; experts gather tokens from all devices. Wide-EP spreads experts over more devices, a few per device; AFD lets one set of expert machines take tokens from several attention groups at once.
Blue counts requests per batch; orange divides that by 32.
Section 3's FLOPs-per-byte ruler; bytes from HBM.
Sources and assumptions

⚠️ Our derivation: per step a dense layer reads 2P bytes (bf16) for 2Pb FLOPs (b/byte); an expert gets b × 8 ÷ 256 tokens. Weights only; KV raises the bar.

📌 v7: 2,307 TFLOP/s bf16 per chip; HBM 3,433 GiB/s per TensorCore, two per chip (author's wiki: entities/tpu-v7). V3's 8/256 comes from its config.json.

MLA helps by shrinking the KV, so a device fits more requests. MegaScale-Infer's A100 math: a batch needs at least 156 requests; each Mixtral expert gets just 39.

Cut 5 · two jobs on two sets of machines: decode no longer stalls behind prefill Schematic timeline (not measured timing). Each cell is one step; green decode emits a token per cell, orange is one long prompt's prefill decode: 1 token/step prefill: whole prompt KV cache transfer Same devices vs two separate sets Colocated Long prompt's prefill Everyone's decode stalls for these 5 steps Prefill machines Decode machines prefill KV cache handed over (≈100 ms, bandwidth estimate) Steady token stream What splitting buys: generation no longer stalls Colocated, one long prompt's prefill can stall everyone for several steps. Split, the decode lane doesn't lose a single cell. The only cost is one extra KV transfer, from the prefill machine's HBM to the decode machine's HBM.
Splitting wins back the cells cut from the top lane.
The bottom lane adds prefill machines; the aim is no gaps, not more tokens. Schematic; 100 ms is our v7x bandwidth estimate.
Sources and assumptions

📌 DistServe arXiv 2401.09670: prefill is compute-heavy, decode is bound by HBM bandwidth; split, they serve 7.4× more requests or meet a 12.6× tighter SLO.

📌 Course author's 1P1D on TPU v7x (Qwen3-Coder-480B): the three-leg KV transfer, estimated from bandwidth, ~100 ms, 5–10% of a 1–2 s prefill. 8K-prompt KV ≈ 1.04 GB (FP8, computed from config), ~83 ms over a 100 Gb/s network; long prompts favor 2P:1D, long outputs 1P:2D.

Same 16 steps: the long prefill takes 5 from the top lane, none from the bottom. (11 s silent loop, Manim; cell counts schematic, not measured.)

Chunked prefill helps: small prompt chunks ride along with decode steps. But small chunks slow prefill (each rereads all earlier KV); the workloads still share devices and one parallelism. PD disaggregation splits them: prefill machines hand the KV cache to decode machines, which only generate.

6.2 The cost: one KV transfer

Our TPU v7x 1P1D setup (Qwen3-Coder-480B, a v7x-8 each for prefill and decode) moves KV from prefill HBM to decode HBM:

One KV transfer is just 5–10% of a prefill: the network isn't the bottleneck Our 1P1D on TPU v7x (Qwen3-Coder-480B), 8K-token prompt. KV transfer is a bandwidth estimate, not a timed measurement prefill (1–2 s) ① ② ③ Same time axis: 0 to 2 s 0 500 ms 1000 ms 1500 ms 2000 ms One prefill 1–2 s One KV transfer 10 + 83 + 10 ≈ 100 ms Zoomed 10×: 10 ms ① HBM → host (PCIe) 83 ms ② host → peer (datacenter network, 100 Gb/s) 10 ms ③ peer → HBM (PCIe) It can take the slow link because each request transfers only once This model has few KV heads (8) and stores KV in FP8: ≈1.04 GB for an 8K prompt. A much larger KV, or another network, means redoing the math. Section 3's TP talks every layer, so it must stay in the fastest tier; one trip per request hides even across machines.
That thin three-color bar is the whole price of splitting.
The 10× zoom shows its three legs.
Sources and assumptions

📌 Our TPU v7x 1P1D record (wiki qwen3-coder-480b-pd-disagg-tpuv7x-20260425): three legs, estimated from bandwidth, ~10 / 80 / 10 ms, ~100 ms total, 5–10% of prefill (prefill 1–2 s inferred from this).

⚠️ Our derivation: KV = 2 × 62 layers × 8 KV heads × 128 × 8,192 × 1 byte ≈ 1.04 GB; 100 Gb/s = 12.5 GB/s → about 83 ms.

A closer look: estimating each leg
LegPathEstimate
①HBM → host memory (PCIe)~10 ms
②Host memory → peer memory (data-center network, conservatively one chip's 100 Gb/s share)~83 ms
③Peer memory → HBM (PCIe), into decode's KV pool~10 ms
Total~100 ms (10 + 83 + 10)

The middle leg: the KV for an 8K prompt is 2 (one K, one V) × 62 layers × 8 KV heads × 128 (head dim) × 8,192 × 1 byte (FP8) ≈ 1.04 GB. 100 Gb/s is 12.5 GB/s, so one trip takes about 83 ms. A CPU-side proxy also routes each request to both sides.

It is Section 1's point-to-point send/receive over the slow data-center network, affordable as each request goes once. From here on, scheduling centers on those notes: KV moves with its request. Kimi's Mooncake calls its inference system “KVCache-centric.”

Short requests, light traffic? The extra transfer and machines may not pay; stay colocated with chunked prefill.

6.3 How many machines per side

Splitting adds a knob, the prefill-to-decode ratio; balance the load. DistServe's clean example:

Once split, 3 devices do the work of 7 colocated: neither side holds the other back Requests/s one device sustains under the same latency targets (DistServe paper example, 13B model on one A100) Both Prefill only Decode only Requests/s per device Colocated Both 1.6 req/s Both 1.6 req/s Both 1.6 req/s Both 1.6 req/s Both 1.6 req/s Both 1.6 req/s Both 1.6 req/s 7 devices ≈ 11.2 req/s Split prefill 5.6 req/s prefill 5.6 req/s decode 10 req/s 3 devices = min(2 × 5.6, 10) = 10 req/s Per-device requests more than double: here's how the ratio is set Each side has its own ceiling: one prefill device handles 5.6, one decode device 10, so 2:1 keeps both about equally busy. Long-prompt workloads load prefill harder, so add prefill; long-answer workloads add decode.
A second kind of machine saves devices.
Per-device requests more than double: neither side stalls the other.
Sources and assumptions

📌 DistServe (Zhong et al., arXiv 2401.09670), example from Sec. 1 and Fig. 1 (13B model, one A100, 90% of requests within SLO): per-device goodput of 1.6 / 5.6 / 10 rps under the same SLO; 2P1D gives 3.3 rps per device, 2.1×. The authors' blog “Throughput is Not All You Need” walks through these numbers.

⚠️ Our derivation: 2P1D handles min(2 × 5.6, 10) = 10 per second, about 3.3 per device; colocated, 10 per second takes ⌈10 ÷ 1.6⌉ = 7 devices.

Rule of thumb: long prompts run 2P:1D, long outputs 1P:2D.

A closer look: the paper's best case

DistServe searches the ratio jointly with each side's parallelism. Versus unsplit systems: up to 7.4× more requests at the same latency targets (vs. DeepSpeed-MII), or up to 12.6× tighter targets (vs. vLLM) (OPT models, 90% SLO attainment). The figure's “more than double” is the paper's single-device example; 7.4× is the best over all ratios and shardings.

6.4 Separate sharding per side

The real payoff: the two sides stop sharing one parallelism.

6.5 AFD: attention and experts on separate machines

Decode splits further: attention reads each request's own KV, binding it to the request. Experts don't care whose token it is; they want big batches, since each step reads their weights once (§6.1). AFD (Attention-FFN disaggregation) separates them: M machines only run attention, N only hold experts.

Split attention from experts too: experts take tokens from several sources at once AFD (Attention-FFN disaggregation). Schematic: M attention machines, N expert machines, three micro-batches taking turns Attention Experts Batch A (follow it) Batch B Batch C Every layer makes one M → N → M round trip Attention machine 1 Attention machine 2 Attention machine 3 Expert machine 1 Expert machine 2 Out (M → N): tokens to experts Back (N → M): results returned Three micro-batches take turns; out and back take half a cell each Attention A A A B B B C C Out A A A B B C C Experts A A A B B C C Back A A B B C C 0 1 2 3 4 5 6 7 8 A: attention in cell 0, out ½, experts 1 cell, back ½; attention again in cell 3. Comm over ½ cell each way → cycle > 3 cells → 4 micro-batches (MegaScale-Infer). The finer the split, the more each side can pick its own machines and sharding Experts take tokens from several attention machines at once, a far bigger batch than one machine gives; each side's machine count is tuned separately. The cost is a round trip every layer, hidden by rotating micro-batches (the paper needs three or four), and the network between the sides must be fast.
It splits a layer's two kinds of work, not a tensor.
Machine and cell counts schematic.
Sources and assumptions

📌 MegaScale-Infer arXiv 2504.02263 (disaggregated expert parallelism, ping-pong pipeline, up to 1.90× per-GPU throughput); Step-3 arXiv 2507.19427; vLLM released an experimental AFD plugin in 2026-07.

Each layer ships tokens to the experts (M → N) and back (N → M). Micro-batches rotate to hide this: one in attention, one in experts, one in flight. They split the batch (requests grouped), not the sequence: each decode request emits one token per step, so 3,000 requests make three 1,000-token groups. MegaScale-Infer needs at least three (four if communication is slow), reporting up to 1.90× per-GPU throughput.

Who uses it

ByteDance's MegaScale-Infer and StepFun's Step-3 do this; vLLM got an experimental plugin in July 2026.

6.6 Multimodal: encoder, too

Same idea, one stage earlier: the vision encoder gets its own machines and ships embeddings to the LM (encoder disaggregation; with PD, often EPD). Training's version: heterogeneous multi-module parallelism, each with its own scheme.

6.7 What this cut leaves open

Five cuts, each with its own communication and uses, coexist on a real cluster: which stay in a machine, which cross machines, whose degree comes first?

Check yourself: splitting prefill and decode adds a KV transfer. Why does it save devices?

Neither side stalls the other, so both run full: in the paper, 2 prefill + 1 decode serve 10 req/s; colocated needs 7 devices. The KV transfer is a sliver of prefill.

Section 7

Onto the machine

Real configs often stack three or four parallelisms, leaving one question: which cut rides which link. Right, and the savings hold; wrong, and the network eats them.

7.1 Fast or slow links, chatty cuts

Links differ. GB300 NVL72 joins 72 GPUs in one NVLink domain at 1.8 TB/s each (bidirectional throughout); the NIC beyond gives 200 GB/s, 9× slower. TPU slices use ICI inside (large ones form a 3D torus, §1.5); the data-center network across slices is about 50× slower like for like (conversion in Sources), far wider than GPUs' 9×.

Chattiness varies too: per-step counts (our derivation, illustrative config) span 3 orders of magnitude:

How often each cut talks: chatty ones ride fast links, only quiet ones go slow Communications per step (to scale, our derivation). Illustrative config: 60 layers, 8 micro-batches, every layer MoE Fastest tier only Can cross to slow links Communications per step TP 1,920 4 all-reduces per layer EP 1,920 4 all-to-alls per MoE layer FSDP 1,440 All-gather in fwd, again in bwd, then reduce-scatter PP 16 Each micro-batch crosses a stage boundary, there and back DP 1 One gradient all-reduce per step 0 500 1,000 1,500 How far apart the two links are (GB300 NVL72, per GPU, bidirectional) Within a rack: NVLink 1,800 GB/s Outside the rack: NIC 200 GB/s ← 9× apart Pair the two charts and the placement falls out TP talks every layer and can't hide it: it must stay in the NVLink domain (a 72-GPU rack on GB300); EP and FSDP leave only if comm overlaps compute. PP talks a dozen-odd times a step and DP once, so slow links are fine. Same on TPU: ICI within a slice, datacenter network across slices.
Pair three orders of magnitude in frequency with one in link speed: that's the placement.
Counts: derived for an illustrative config. Bandwidth: GB300 per GPU, bidirectional.
Sources and assumptions

📌 Counts are our derivation: TP 2 all-reduces per layer each in forward and backward (arXiv 1909.08053 sec. 3); FSDP 3 per layer (ZeRO's 3Ψ, arXiv 1910.02054 sec. 7); EP, PP, DP from the schedule; with PP, divide TP's count by stages; with recomputation, TP 6 per layer. Links: GB300 NVLink 5 1.8 TB/s/GPU; A4X Max: 4 GPUs, 4 × CX-8 800 Gb/s/node.

Guess first: which of TP, PP, DP gets the fast link? Picture an office building:

A cluster is an office building: who should share a desk? Each ring's links are slower than the one inside it. The inside-out order matches Llama 3's [TP, CP, PP, DP] Desk: fastest tier Building: network Other buildings: slow Inside out, each ring slower Same desk: TP, EP, FSDP, CP Co-writing a contract, clause by clause Same building: PP Passing drafts by chapter (stage edges only) Other building, other city: DP Settling accounts at day's end (once a step) Which ring is the desk? · 8-GPU server: that machine · GB300 NVL72: a 72-GPU rack · TPU: one slice Why this order Comms per step span 3 orders of magnitude, link speed spans one: what each link carries must match its speed. This is the default placement; some move cuts outward on purpose Only cuts that overlap comm with compute can move out: V3's EP spans 8 machines; Llama 3 puts FSDP in the outermost tier. TP can't hide: until the sum at the end of each block is done, the next block can't start, so it almost never leaves the inner ring.
One desk to co-write a contract; drafts can cross floors; accounts can settle from another city.
Ring sizes show only that links slow outward.
Sources and assumptions

📌 Llama 3 technical report arXiv 2407.21783 sec. 3.3.2 (dimensions ordered inside out: [TP, CP, PP, DP]); V3 technical report sec. 3.2 (EP 64 across 8 machines).

⚠️ Office-building analogy: ours (loosely after CERFACS); ring sizes don't show bandwidth ratios.

Default placement: TP, EP, FSDP and CP talk every layer, so they fill the fastest tier first; PP (stage boundaries only) and DP (once a step) take slow links. Also ask whether communication overlaps compute. TP's can't (each block's closing all-reduce stalls the next), so TP stays on fast links, its degree capped by that tier.

EP and FSDP can hide traffic by sending early, so some cross machines: V3's EP 64 spans 8 machines (H800's intra/inter-node gap is just 3–4×, unlike TPU's); Llama 3 puts FSDP outermost. DP moves the most data, but once a step; backward runs last layer first, so late gradients ship early. Section 6's once-per-request KV can likewise take the slow data-center network (TPU v7x).

7.2 A misplacement, up close

Same 8 devices and TP4 × DP2; only the TP groups move. Schematic: a transfer is 1 cell on a fast link, 9 on a slow one.

Rounds wait for the slowest hop, so placement 2 runs at slow-link speed every round; red arrows mark slow hops. (29 s, Manim; untick “Autoplay” to step. 8 TP rounds + 1 DP per step; fast:slow 1:9, schematic.)

One color per TP group (placement 2: one per row). A step = 8 TP rounds + 1 DP: placement 1, 8 × 1 + 9 = 17 cells; placement 2, 8 × 9 + 1 = 73 (its DP pair shares a machine: 1 cell).

Hence TP rarely leaves an 8-GPU server. GB300 stretches NVLink across a rack, so EP can go wide (TP, limited by head count and matmul efficiency, rarely tops 8 or 16 even there): the fast tier's size sets how deep these cuts go.

7.3 Right cut first, then tune

Wrong placement and sharding leave tuning circling under the wrong ceiling, as we found with DeepSeek-V4-Pro on GB300. 1P1D = one prefill + one decode machine; 3P = three prefill; DEP8 = each decode device's attention serves its own requests, experts over 8 devices:

Resharding beats retuning by a mile: compare per GPU, not totals GB300 · DeepSeek-V4-Pro · vLLM, 4K in / 1K out. Bar length is per-GPU throughput (tok/s); totals and GPU counts in parentheses TP4 decode DEP8 decode Tokens per second per GPU 1P1D, TP4 decode Original script (concurrency 256) 1,820 (total 14,563, 8 GPUs) 3P1D, TP4 decode Best: +prefill, tuned concurrency 1,319 (total 21,100, 16 GPUs) Total +45%, per GPU −28% 3P + DEP8 decode Decode GPUs 4→8, concurrency 512 2,758 (total 55,153, 20 GPUs) Longer is better, per GPU However well you tune, you only reach that sharding's ceiling On TP4, adding machines and tuning concurrency gave +45% total, but paid for with twice the GPUs; TPOT stayed pinned at about 46.8–53 ms. DEP8 gives 2.09× per GPU at the same concurrency and TPOT drops to ~12 ms; at concurrency 1,536 per-GPU reaches 2.47×, but TTFT is 95 s.
Compare per-GPU throughput: runs used 8, 16 and 20 GPUs.
Course author's measurement, 4K in / 1K out.
Sources and assumptions

📌 Measured by the course author: gpu-tpu-pedia gpu/inference/a4x-max/deepseek-v4/VLLM-V4PRO-RUNBOOK.md (1p1d 14,563; 3p1d 21,100; 3p + DEP8 at concurrency 512, 55,153; at concurrency 1,536, 65,132). GPU counts assume 4 per node, 1 node each for prefill and TP4 decode, 2 nodes for DEP8; per-GPU figures computed by this script.

Tuning everything on TP4 decode (no eager mode, more prefill machines, concurrency) lifted total throughput 45%, but on twice the GPUs: per-GPU fell, TPOT stayed near 50 ms. DEP8 quartered TPOT at the same concurrency and more than halved TTFT.

Resharding alone bought two bigger-batch wins: no KV copy on each of 4 devices (§5.4), and twice the decode GPUs (4 → 8), halving experts per GPU. The 8 GPUs span two trays, inside the rack's fast NVLink. Check the sharding before touching parameters.

The fine print: every number, assumption and gap in the record
  • Eager mode off (CUDA Graphs on): only +2.6%; more prefill machines, tuned concurrency: total 14,563 → 21,100 (+45%); TPOT stays 46.8–53 ms.
  • Switching to DEP8 at the same concurrency of 512: TPOT 46.8 ms → 11.8 ms, TTFT 55.8 s → 22.8 s. At concurrency 256 the same setup's TTFT is only 7.8 s, so most of the 512 figure is queuing.
  • The original record doesn't break down why TPOT dropped too.
  • A GB300 tray: 4 GPUs; prefill→decode KV also rides in-rack NVLink, not NICs. Concurrency: requests queued; batch: computed at once.
  • The figure's last two rows run at concurrency 512, the first at the original script's 256. At 1,536, DEP8 reaches 65,132 total (2.47× per GPU), but TTFT hits 95 s: prefill is the bottleneck again.
  • Throughput counts prompt and output tokens. Against the first original script (1,820 per GPU), DEP8 is ~1.5× per GPU.

7.4 Choosing in five steps

Picking parallelism: five gates, each a yes/no question:

  1. Gate 1: does it fit? Weights and optimizer state too big: FSDP / ZeRO (Section 2). Too many MoE experts: EP (Section 4). KV too big at inference: DCP (Section 5).
  2. Gate 2: how big is the fast tier? Keep TP and EP within it (8-GPU server: 8; GB300 rack: 72; TPU: the slice).
  3. Gate 3: long sequence? CP for long-context training, PCP for long inference prompts.
  4. Gate 4: still short, or on a very slow link? Only now PP: small, rare boundary activations cross slow links; cost: the bubble (Section 3).
  5. Gate 5: devices left? All to DP: one communication per step, the cheapest.

Two real configurations; mind the two “not used”: V3 skips TP on 8-GPU H800 servers, Hunyuan 3 skips EP on TPU. “DP 4 × FSDP 128” is a product of shard counts.

StepHunyuan 3 (295B MoE, TPU v7, 256 chips)DeepSeek-V3 (671B MoE, 2,048 H800s)
① FitsFSDP 128 (FSDP shards experts too); any narrower OOMsZeRO-1 shards optimizer state, EP 64 experts
② Fast tier256 chips, one fast slice; FSDP already spreads experts: no EP; a small EP trial lost throughput (note below) 8 GPUs/server: no TP; EP 64 spans 8 machines (slow-link exception); ≤ 4 machines per token
③ Sequence4K / 8K, no CP4K pre-training; no CP reported
④ PPFits in one slice, not usedPP 16; DualPipe hides comms under compute
⑤ DPLeftover 4× to DP: DP 4 × FSDP 128, the default (v7 chip = 2 devices; degrees count devices: 256 chips = 512). Best: FSDP 256; freed HBM lifts per-device batch 12 → 16, +3% vs. default; widening alone costs <1%: the gain is all batchThe rest: ZeRO-1 data parallelism

Same five steps, nearly disjoint results: the two machines' fast links differ in shape. Don't copy another team's config; check your own links first.

Note: Hunyuan 3's EP test (4-way, 16 chips) lost 71% in one run, but with the batch halved (a confound); another setup lost 37%. The record blames extra all-to-all hops on the torus; at 4-way that cost is small, so the cause is open. V3: PP 16 × DP 128 = 2,048; EP 64 shards experts within those 128 ranks, no extra GPUs (our derivation).

7.5 What adding devices costs

Last question: adding devices, how much per-device efficiency survives? Two measures:

The metric is scaling efficiency: per-device throughput after ÷ before. We lack clean strong-scaling data; closest: the same DP 4 × FSDP 128 at per-device batch 12 → 8 fell from 580 to 453 per chip. Less work per device hurts.

Our two measurement sets; compare within a set only. Left: weak scaling, devices and global batch ×4. Right: same devices, varying FSDP width.

Adding devices: as replicas, almost no loss; spread thinner, you lose Hunyuan 3, TPU v7, measured TFLOP/s per chip. 1 chip = 2 devices; DP × FSDP in devices (64 chips = 128). Per-device batch differs between sets Extras as DP replicas Wider FSDP Doesn't fit Same recipe scaled 4× (per-device batch 12) 580 64 chips FSDP 128 580 256 chips DP 4 × FSDP 128 Same 256 chips, only the split changes (per-device batch 8; not comparable to left) 404 DP 1 FSDP 512 450 DP 2 FSDP 256 453 DP 4 FSDP 128 OOM DP 8 FSDP 64 OOM DP 16 FSDP 32 Grow the batch along with the devices Left: work per device unchanged, 4× the devices gives 4× the throughput: 100% weak scaling. Between groups: one gradient all-reduce per step. Right: one FSDP group grows from 128 to 512 devices; more gather steps, smaller pieces, fixed latency not amortized: 404 is 11% below 453.
Same chips, replicas vs. spread thin: 11% apart.
Course author's measurements; per-device batch differs by set.
Sources and assumptions

📌 Measured by the course author: gpu-tpu-pedia tpu/Hunyuan3-295B-Pretraining/TUNING-v7 sec. 3.7 (five splits, pdbs 8), sec. 4.1 (64 and 256 chips both at 580, pdbs 12). Numbers are TFLOP/s per chip.

Left: DP is nearly free. Four 64-chip groups do one gradient all-reduce per step: under 1% of the 23.5 s step by a rough v7 ICI estimate, even discounted for shared links (our derivation). But the groups share one slice (DP on ICI); this doesn't carry over directly across slices.

Right is the warning: same chips and batch, but FSDP over all chips instead of a quarter means more gather steps on smaller pieces, so latency stops amortizing: 11% lost. Narrower FSDP (64 or 32 shards) can't fit shards beside activations. While the batch can grow, grow it with the devices; extra devices become replicas first.

7.6 What this section leaves open

Every parallelism now has a story: what it cuts, what it sends, which link it rides. Next, a panorama and tables decode TEP8, DEP16 and DP4 × FSDP128.

Check yourself: TP, PP, DP: which goes innermost? Which can cross slow links?

TP syncs every layer, unhideable. PP: once per stage boundary, can cross machines. DP: once a step, can span data centers.

Section 8

Panorama: every parallelism in use today

So many names: DP, FSDP, TP, SP, CP, PP, EP, and for inference DCP, PCP, DEP, TEP… Just four families. Of the five cuts, weights and experts are both model parallelism. Learn the families and the names fall into place.

8.1 Four families

One criterion: what does it split? A figure, then a table:

Four families; each name carries two tags: its collective, and whether it can cross slow links Weights and experts both count as model parallelism. ● slow links OK   ○ fast only (AFD, Wide-EP span machines but talk every layer, so fast too; exceptions in Section 7) Data Sequence Model Disaggregation Data parallel · split batch DP AllReduce ● ZeRO-1 RS + AG ● FSDP AG ×2 + RS ○ HSDP inter-node dimension on slow links ● Attention DP AllToAll (follows EP) ○ Sequence parallel · split a sample Megatron SP AG + RS (TP's partner) ○ Ring Attention Send/Recv around ring ○ Ulysses AllToAll ○ DCP Inference: AG + merge, 3×/layer ○ PCP Inference: split long prompt ○ Model parallel · split weights TP AllReduce, every layer ○ PP Send/Recv at stage edges ● EP AllToAll, every MoE layer ○ Wide-EP EP over dozens of devices ○ Disaggregation · split work PD disaggregation KV sent across once ● AFD Per layer: attn→experts→attn ○ Encoder disaggregation embeddings sent once ● For any name, ask two things: what does it split, and what extra collective does it add What it splits picks the column; the extra collective picks the link. Only the quiet ones dare cross slow links. Inference shorthands like TEP8, DEP16 combine two splits: the first letter is for attention, EP for the experts.
Any name: which column, and is its dot filled?
Dots: default placement; exceptions in Section 7.
Sources and assumptions

📌 Families, collectives summarized from §1–7; “can cross slow links” = default placement; exceptions (V3's cross-node EP, Llama 3's outermost FSDP) in §7.

FamilySplitsEach holdsExamples
DataData parallelismThe batch (requests) Own samples; logically full modelDP, FSDP, Attention DP
SeqSequence parallelismTokens within one sample A chunk: activations (training), KV cache (inference)CP, Ulysses, DCP
ModelModel parallelismThe weights Some columns, layers or expertsTP, PP, EP
DisaggDisaggre­gationWork phases or modules One job: prefill or expertsPD disaggre­gation, AFD

Tables follow the four families; “New” = appeared or went mainstream in 2025–26. For reference; on a first read, skip to 8.6. Many names are new; come back as needed.

8.2 Data parallelism

NameSplitsSolvesExtra commsUse
DP / DDPThe batch; full model replicatedLinear throughput scaling Train: gradient all-reduce once a step. Infer: none, router splits loadTrain · Infer
ZeRO-1 / 2 / 3Each stage adds one: optimizer state → gradients → parameters Under DP every device holds the full state: pure redundancyRS + AG; ZeRO-1/2 match DP (ZeRO-2 with micro-batches: one RS per micro-batch); ZeRO-3 re-gathers parameters every layerTrain
FSDP / FSDP2ZeRO-3, PyTorch's native version 1.5× DP's traffic; memory falls linearly with devicesForward: AG weights; backward: AG again, then RS gradientsTrain
HSDPShard in-node, replicate across nodes (e.g. Hunyuan 3's DP 4 × FSDP 128, though in one slice)Keeps the most frequent all-gathers in the fast domain AG / RS within a node, all-reduce across nodesTrain
ZeRO++Three tricks on top of ZeRO-3Cross-node comms too costly INT8 weights pre-AG; extra param copy in-node; quantized gradsTrain
DP across DCNDP spanning multiple slices or data centers DP talks once per step, the best fit for slow links (PP works too)Gradient all-reduce over DCNTrain
DiLoCo NewReplicas run hundreds of steps alone, then sync once Cross-data-center training: syncing every 500 steps cuts syncs several-hundredfoldOne outer gradient sync (Streaming variant: shards in turn, overlapped with compute)Train
Attention DP NewBy request, attention layers only; experts configured separately MLA has 1 KV head, so TP8 copies KV 8×; with DP each device keeps only its requests' KV All-to-all when followed by EP; idle devices run a dummy forward to stay in syncInfer
⚠️ “DP” means something else in MoE inference On dense models DP means independent replicas; on MoE in vLLM / SGLang it turns on Attention DP: expert layers still run jointly, so every rank syncs every step.
Another vLLM trap: -dp without -ep puts expert layers on TP, not EP.

8.3 Sequence parallelism

When a sample is too long for one device, split the sequence. Training and inference differ in why: training splits activations, inference the KV cache. Hence two tables.

Training (and prefill): split activations

NameSplitsSolvesExtra commsUse
Megatron SPLayerNorm, Dropout, residual activations only; needs TP (filed by what it splits; by role TP's partner, §3.3) Activations TP misses, fully copied on each deviceSplits TP's all-reduce into all-gather + reduce-scatter; same total volumeTrain · Infer
CP (context parallelism)All activations, along the sequenceLong-context training; near-mandatory past tens of K KV via ring, all-to-all or all-gather; combinable hierarchicallyTrain
UlyssesSwitches between sequence split and head splitLong sequences: per-device comm flat as length and devices scale together All-to-all before and after attention; degree ≤ head count (KV heads for GQA)Train
Ring AttentionKV blocks circle a ringDegree not limited by head countRing P2P overlapping computeTrain · Infer
Striped / USP / 2D-AttentionReassign tokens; stack Ulysses and Ring in 2D Causal masks load the ring unevenly; the two limits complementBoth of the aboveTrain
PCP (prefill CP) NewQuery tokens during prefill Time to first token (TTFT) on long promptsAll-gather KV, or a ringInfer
Chunked PP NewPrompt chunks pipelined in turn Prefill startup ∝ first chunk onlyPipeline point-to-pointInfer

Inference decode: split the KV cache. The hottest area in 2025–26

NameSplitsSolvesExtra commsUse
DCP (decode CP) NewKV cache sequence, tokens round-robin across devices TP > KV heads copies KV; MLA has 1 head: TP8 = 8 identical copies. DCP turns that into capacity Three per layer (vLLM default): all-gather Q, exchange LSE, merge outputsInfer
Helix NewSame devices, two layouts per layer: attention by KV sequence (plus TP ≤ KV head count), FFN by TP × EP Million-token decode: both KV and weight reads must spreadAll-to-all after attention swaps partial results; FFN keeps TP's all-reduce or EP's all-to-allInfer
MaxText context_autoregressiveDecode: KV split by sequence, FFN by expert Same as above, the TPU wayInserted automatically by XLAInfer

8.4 Model parallelism

This family truly splits weights, in three places: within a matrix, across layers, across experts.

Inside the matrix (tensor parallelism)

NameSplitsSolvesExtra commsUse
TPMatrices split column-then-row, in pairsOne layer's weights or compute don't fit 2 all-reduces per layer each way, very frequent, so it stays in the fastest ring (NVLink domain, ICI in one slice); on v7, TP8 is just 4 chips, already below the lineTrain · Infer
2D / 2.5D / 3D TPMatrix split into a grid1D TP comms grow with the degree Broadcasts and reductions along grid rows/columnsTrain, largely abandoned
GTP NewReshards weights along the TP axis, gathered on use Megatron docs: “GTP_remat is an implementation of ZeRO-3,” but on the model-parallel axis Per-weight async all-gather; gradient reduce-scatterTrain, experimental

Across layers (pipeline parallelism)

NameSplitsSolvesExtra commsUse
PPLayers, grouped into stagesModel too deep, or only slow links between machines Point-to-point at stage boundaries, least volume; the price: the bubbleTrain · Infer
VPP (interleaved 1F1B)Several non-contiguous layer groups per deviceShrinks the bubbleMore point-to-point transfersTrain
Zero Bubble NewBackward split: activation vs. weight gradients Fills the bubble with the weight-gradient halfPoint-to-pointTrain
DualPipeFeeds the pipeline from both ends at onceV3's heavy expert comms must fully overlap forward/backward compute Point-to-point, overlapped with EP's all-to-allTrain

Across experts (MoE)

NameSplitsSolvesExtra commsUse
EPDifferent experts on different devicesTotal expert parameters too large Dispatch, combine: two all-to-alls. Data picks destinations, so load is uneven: EP's worst problemTrain · Infer
Wide-EP NewEP spread over 32, 64 or more devices Few experts per device, so each gets a bigger decode batchLarge-scale all-to-all; needs a dedicated libraryInfer
ETPTP inside one expertExpert too big; fine-grained: usually 1All-reduce inside the expertTrain
EDPLeftover expert-side DP, derived from the othersWhere expert gradients are summedExpert-gradient all-reduceTrain
Parallel Folding NewAttention, expert layers: same devices, two splits Breaks the old “EP ≤ DP” constraint. E.g. attention at TP4 · CP2 · DP8, experts straight at EP64 Two sets of comm groupsTrain
MoE layers get their own parallel axes Megatron training writes attention as TP × CP × DP × PP and experts as ETP × EP × EDP × PP, configured separately. Inference does the same under other names (§8.6).
Here MoE tuning has the most freedom, and the most room for error.

8.5 Disaggregation

The first three split tensors; this one splits the work: unlike jobs go to separate machines, each with its own parallelism.

NameSeparatesSolvesExtra commsUse
PD disaggregationPrefill, decode on separate machines Prefill (compute) and decode (bandwidth) hurt TTFT, TPOTKV cache transfer between machinesInfer
AFD NewAttention on one machine group, experts on another Experts pool several attention groups' tokens; ratio tunable Per-layer round trip, attention → experts → attention; hidden with 3–4 micro-batchesInfer
Encoder disaggregationMultimodal vision encoder runs aloneEncoder and LM compete for resourcesEmbedding transferInfer
Heterogeneous multi-module parallelismEncoder, LM: own parallelism eachMultimodal training: parts too unlike in shapeBridging between the two meshesTrain

8.6 Shorthands: TEP and DEP

Inference MoE configurations are usually written as shorthand plus a number, like TEP8 or DEP16. The first letter says how attention is split; EP says how experts are split.

ShorthandAttention layersExpert layers
TEP<N>TP across N devicesEP over the same N devices
DEP<N>Attention DP: each device its own requestsEP over the same N devices

TensorRT-LLM's definition, verbatim: TEP<N> shards both attention (TP) and experts (EP) across N ranks. DEP<N> keeps attention data-parallel (ADP) while distributing experts across N ranks. vLLM uses the terms the same way.

With PD disaggregation each side often picks one; no fixed rule says which gets which: it depends on model and load. vLLM's Kimi K3 deployment uses TEP8 for prefill, DEP16 for decode; TensorRT-LLM's example is the reverse: DEP4 for prefill, TEP8 for decode.

⛔ TEP / DEP are not ETP / EDP Same letters, different order, different things.
TEP / DEP (inference) describe how attention is split; DEP's D is Attention DP.
ETP / EDP (Megatron training) describe the TP degree inside an expert and which devices expert gradients are summed across.
The Megatron repo has zero hits for TEP / DEP. Treat them as two vocabularies; don't translate between them.

Other combined terms you'll hear:

8.7 Mistaken for parallelism

The test: does it split a new dimension? If not, it isn't new parallelism.

NameWhat it actually is
TBO / SBO / DBOComm hidden in compute: two staggered batches, one talks, one computes
Async TP, TP comm overlap, collective matmulStill TP, with comm and matmul chopped small and interleaved
EPLB, redundant expertsLoad balancing: replicate and re-place hot experts. An EP fix, no new dimension
Offload, recomputationTrading time for HBM, or host memory for HBM
Chunked prefillInference scheduling: long prompts chunked, mixed with decode, less blocking (§6.1). Not §8.3's Chunked PP (prefill too): its chunks flow down the pipeline: a new axis

8.8 One name, many meanings

TermWhat it means where
SPAt least three: Megatron's SP (LayerNorm-type segments only, with TP); DeepSpeed's SP (Ulysses); any sequence splitting. Megatron: splitting all activations = CP
CPTraining: split activations. vLLM has two switches: PCP adds devices and cuts TTFT; DCP adds none and expands KV capacity
DPDense: separate replicas; MoE inference: really Attention DP, per-step sync
ETPMegatron: in-expert TP; TensorRT-LLM Hybrid ETP: expert layers mix TP + EP
hierarchicalZeRO++ hierarchical sharding ≠ Megatron hierarchical CP
GPU / chip / device1 GPU = 1 device; a TPU v7 chip = 2 to software; degrees count devices
BandwidthOne-way or send + receive: here 1.8 TB/s (NVLink) and 1.2 TB/s (ICI) are combined; 800 Gb/s and 100 Gb/s (NICs) one-way
Concurrency/batchConcurrency: requests queued; batch: computed at once

8.9 Three questions to take away

Unfamiliar parallelism name? Ask three questions ① What does it split? (data, sequence, model, or the work itself)
② What communication does it add? (one of Section 1's five)
③ Which link should it ride? (chattier per step → further inside; §7's office building)

Try the opening's V3: PP 16, EP 64, ZeRO-1 data parallelism (report sec. 3.2). PP cuts depth, sends only at stage boundaries, so it crosses machines; EP cuts experts, two all-to-alls per layer, so ≤ 4 machines per token; ZeRO-1 cuts data: one reduce-scatter + all-gather a step, cheapest. Three cuts, three collectives, three links: V3's cryptic config (PP 16, EP 64, ZeRO-1) now reads plainly.

⭐ Back to the opening: where to cut. The five cuts split different things but pay one price: spend communication to buy memory or compute. Picking a parallelism strategy is picking which communication to pay, how often, on which link.

Last, with the ruler from Sections 3 and 6: the hardware line is compute ÷ bandwidth; how you cut sets the workload line. Each cut here tries to lift a workload line above it.

Two after-lecture questions (answers are above; think first. Sections 1–7 each end with a “Check yourself” too)

① An MoE model will decode on one GB300 rack (72 GPUs, one NVLink fast domain). How would you split its attention and expert halves? Answer the three questions, then compare with 7.3.
② TEP8 vs. Megatron's ETP8: only the letter order differs. How do the meanings differ? Look it up, then see 8.6's red box.

Section 9

Sources

Claim ← source. Section 8 checked 2026-09-23, Section 1 2026-09-24, the rest 2026-09-25 after ten review rounds and two trial lectures.

ClaimSource
Bytes per device per collective (Section 1 table)NVIDIA/nccl-tests: bus bandwidth correction factors in doc/PERFORMANCE.md: AllReduce 2(n−1)/n, ReduceScatter / AllGather / AlltoAll (n−1)/n, Broadcast / Reduce 1
Step-by-step ring ReduceScatter; the root patternwanghonglei, “Collective communication primitives for distributed deep learning, from zero to mastery” (Chinese, 2026-06-27), ch. 1–2; steps script-computed, asserted; shifted one so device k gets chunk k
Warm-up: 16 B/param standard; V3 keeps m, v in bf16, master weights and accumulated grads in fp32DeepSeek-V3 report, arXiv 2412.19437 sec. 3.3.3 (AdamW's 1st and 2nd moments in BF16, not FP32, “without incurring observable performance degradation”; master weights and batch-accumulation gradients stay FP32); PyTorch: optimizer state at parameter precision; Megatron-LM needs --use-precision-aware-optimizer plus --exp-avg-dtype bf16 --exp-avg-sq-dtype bf16 (Megatron Core MoE docs; flags in megatron/training/arguments.py; without it, megatron/core/optimizer/optimizer_config.py asserts both are fp32); that group lists --main-grads-dtype, --main-params-dtype, --exp-avg-dtype, --exp-avg-sq-dtype together (arguments.py); we still count gradients separately; V3's ~14 B/param is derived (compute-weight precision unstated; BF16 assumed); Megatron's bf16 training defaults to fp32 gradient accumulation and all-reduce (arguments.py: “bfloat16 requires gradient accumulation and all-reduce to be done in fp32”), so gradients are 4 B: 18 B/param
V3's full FP8 recipe validated: relative loss error < 0.25%Same report, sec. 3.3, appendix B.1: ~16B and ~230B models, ~1T tokens each, full FP8 recipe (bf16 optimizer state included) vs. BF16 baseline; no separate “moments in bf16” ablation
Moments in bf16? Depends on β₂V3's AdamW: β₁ = 0.9, β₂ = 0.95 (report sec. 4.2); PyTorch default 0.999. ⚠️ Our derivation: bf16 has 8 significant bits, relative spacing 2⁻⁷–2⁻⁸; rounding drops changes under half a gap (≥ ~0.2%); at β₂ = 0.999 a step adds only 0.1%. Cross-check: Dettmers et al., arXiv 2110.02861: block-wise 8-bit optimizer state matches 32-bit
V3 full training: 2.788M H800 GPU hours; about 318 years on one GPUDeepSeek-V3 technical report, abstract (covers pre-training, context extension and post-training); 318 years = 2.788M ÷ 8,760 hours, our derivation
ZeRO local update: no comms; clipping: one scalar all-reduceAdam is element-wise (Kingma & Ba, arXiv 1412.6980 Alg. 1); clipping uses the global norm (Pascanu et al., arXiv 1211.5063); sharded: devices sum local squares, then all-reduce (our summary)
Memory and communication at each ZeRO stage (Section 2)Rajbhandari et al., ZeRO, arXiv 1910.02054 sec. 5, 7: Pos and Pos+g move the same volume as DP (2Ψ), Pos+g+p ≤ 1.5×; memory 16Ψ → 16Ψ/Nd
How TP splits and how often it talks; SP adds none (Section 3)Megatron-LM arXiv 1909.08053 sec. 3 (2 all-reduces forward, 2 backward); arXiv 2205.05198 sec. 4.2.2 (AG + RS replace all-reduce, no extra comms); TP must divide the query heads, KV groups and TP must divide one another (checks in megatron/core/transformer/transformer_config.py)
PP bubble (p−1)/m; interleaved ÷ vNarayanan et al., arXiv 2104.04473 sec. 2.2.1–2.2.2; Zero Bubble arXiv 2401.10241; DualPipe README
DualPipe: 2× params, activation peak p+1; after ChimeraDeepSeek-V3 technical report, arXiv 2412.19437 sec. 3.2.1, Table 2
V-shapes level device peaksQi et al., arXiv 2405.15362
MaxText: PP defaults to 1; micro-batch = global batch ÷ num_pipeline_microbatchesComments in MaxText configs/base.yml
V3's training parallelism; parameter distributionDeepSeek-V3 technical report, arXiv 2412.19437 sec. 3.2 (16-way PP, 64-way EP, ZeRO-1, no TP); config.json (61 layers, first 3 dense, 256 experts, moe_intermediate_size 2048, hidden 7168)
FLOPs per byte moved; v7 hardware line about 3,845⚠️ Our derivation (dense approx., perfect overlap); v7 2,307 TFLOP/s bf16; spec: ICI 1,200 GB/s/chip, plus a 200 GB/s tier; only reading that as per link, send + receive (“6 links × 200, 600 sent”) fits (derived; Scaling Book's 9e10/link/direction gives ~540, line ~4,270, hence 3,800–4,300); per device also ~3,845 (2 devices share a chip's links, halving both) (wiki ici-dcn, Inferact blog spec table). Fixed 2026-09-25: earlier misused 1,200 → 1,922; TP line's 4.5 assumes standard attention + 4×-wide MLP (V3: MLA + MoE; illustrative)
V3's EP: ≤ 4 nodes, FP8 dispatch / BF16 combine, aux-loss-free balancingDeepSeek-V3 technical report, arXiv 2412.19437 sec. 2.1.2, sec. 3.2.2, sec. 3.3.3; ≈ 28.7 KB of cross-node dispatch per token is our derivation
V3's balancing hyperparameters, no token dropping; NVLink / IB 3.2×, ≤ 13 experts at equal trafficIbid. sec. 4.2 (bias update speed γ = 0.001, 0 for last 500B tokens; seq-wise balance loss α = 0.0001; M = 4, 64 GPUs/8 nodes), sec. 2.1.2 (No Token-Dropping), sec. 3.2.2 (160/50 GB/s, 4 × 3.2 ≈ 13)
The Parallel Folding exampleMegatron-Core megatron/core/transformer/moe/README.md; arXiv 2504.14960
TP4 → DEP8 on GB300: at equal concurrency 512, 2.61× total, 2.09× per GPU (DEP8 at 1,536: 2.47× per GPU, TTFT 95 s); tuning +45%Course author's measurements: gpu-tpu-pedia gpu/inference/a4x-max/deepseek-v4/README.md and VLLM-V4PRO-RUNBOOK.md (TP4 decode 14,563 → 21,100 after tuning, 16 GPUs, 1,319 per GPU; DEP8 65,132, 20 GPUs, 3,257 tok/s per GPU)
Activations grow with seq. lengthKorthikanti et al., arXiv 2205.05198 Eq. (1): sbh(34 + 5as/h) per layer
Ring Attention; Ulysses: constant traffic, degree ≤ head countarXiv 2310.01889; arXiv 2309.14509 sec. 3.2 (4Nh/P, constant when N and P scale together); head-count cap in USP, arXiv 2405.07719 sec. 3
CP's zigzag splitMegatron-LM megatron/core/utils.py (2×cp chunks; rank r takes chunks r and 2·cp−r−1); docs/user-guide/features/context_parallel.md
TP replicates KV tp/H times; DCP reuses TP ranksvLLM context parallel deployment docs; vllm/config/parallel.py docstring. V3's 70,272 KV B/token computed from config.json, whose num_key_value_heads=128 counts decompressed MLA heads; cached compressed (vLLM: 1 KV head)
Motivation and gains of PD disaggregation (Section 6)DistServe arXiv 2401.09670 (prefill compute-leaning, decode bandwidth-bound; 7.4× requests vs. DeepSpeed-MII, 12.6× tighter SLO vs. vLLM)
v7x 1P1D: 3 KV stages ~100 ms (bandwidth est.); 2P:1D / 1P:2DAuthor's deployment notes (KV time: bandwidth estimate): wiki qwen3-coder-480b-pd-disagg-tpuv7x-20260425. 8K KV ≈ 1.04 GB, ~83 ms at 100 Gb/s, derived (Qwen3-Coder config: 62 layers, 8 KV heads, head_dim 128)
Communications per step for each cut (Section 7)⚠️ Our derivation (illustrative config: 60 layers, 8 micro-batches): TP 4 per layer (arXiv 1909.08053 sec. 3), FSDP 3 per layer (arXiv 1910.02054 sec. 7), EP / PP / DP counted from the schedule
GB300 NVLink 1.8 TB/s / 800 Gb/s NIC per GPU; 9×Author's wiki: sources/nvidia-gpu-comparison-20260311, analyses/gb300-a4x-max-network-congestion-control (A4X Max: 4 GPUs per node, 4 × CX-8 800 Gb/s); 9× is our bidirectional conversion
Hunyuan 3 scaling, five-step testAuthor's measurements: gpu-tpu-pedia tpu/Hunyuan3-295B-Pretraining/TUNING-v7 sec. 3.7 (five splits: 404 / 450 / 453 / OOM / OOM), sec. 4.1 (580 at 64 and 256 chips), sec. 3.6 (DP2 × FSDP256, pdbs 16: 599), 4-way EP on 16 chips −71% (single run). Sec. 4.1: 23.54 s/step; inter-group all-reduce < 1%, derived (source's 12 ms math inconsistent); EP reconfig −37%, wider FSDP under 1% slower (450 vs. 453): TUNING-v7 sec. 3.7
DCP: 3 collectives per layerdcp_comm_backend docstring in vllm/config/parallel.py (default ag_rs: 3 NCCL calls per layer; a2a backend: 2)
ZeRO-2 is no longer free with micro-batches; with PP, choose ZeRO-1DeepSpeed docs: pipeline parallelism is incompatible with ZeRO-2/3; V3 technical report sec. 3.2 (ZeRO-1); Megatron distributed optimizer
TP FLOPs per byte ≈ 4.5 × hidden ÷ TP; V3 at TP8 ≈ 4,032⚠️ Our derivation: 4 all-reduces per layer, each ~4Th bytes, vs. 72h²T/n compute (standard attention + 4×-wide MLP); for V3 illustrative only
Decode EP: direct sends per expertDeepEP low-latency mode (deepseek-ai/DeepEP README); V3 technical report sec. 3.4
V3's cluster and at most 4 nodes per tokenDeepSeek-V3 technical report, arXiv 2412.19437 sec. 3.1 (2,048 H800s, NVLink within nodes, IB between nodes), sec. 2.1.2, sec. 3.2
Strong / weak scalingAmdahl 1967; Gustafson 1988 (Reevaluating Amdahl's Law); critical batch size: arXiv 1812.06162
NVSwitch does the additions in the switch (NVLS)NVLS algorithm (NVIDIA NCCL docs); “nearly half”: our derivation, ~S sent per GPU vs. 2(n−1)/n·S
Which TPU slices wrap around; small ones get ~half bandwidthHow to Scale Your Model (the Scaling Book), TPU chapter: only full cubes (multiples of 4) get wraparound; Google Cloud TPU7x topology docs
3D torus rotating three blocks; all-gather 3× fasterScaling Book, sharding chapter: multi-axis gathers multiply bandwidth; 63 vs. 21: our bandwidth-only derivation for 4×4×4, XLA schedule unchecked
Llama 3 puts FSDP outermostLlama 3 technical report, arXiv 2407.21783 sec. 3.3.2 (parallelism dimension order [TP, CP, PP, DP])
V3: 37B active params per token; pre-trained at 4KDeepSeek-V3 technical report, arXiv 2412.19437 abstract and sec. 4.1
TorchTitan's “4D”TorchTitan paper, arXiv 2410.06511 sec. 2.1.5 and Table 6 (4D = FSDP + TP + PP + CP); the abstract stops at 3D (Llama 3.1 405B, 512 GPUs); today's pytorch/torchtitan README no longer says “4D”
“Even the largest devices carry only two to three hundred GB of HBM”NVIDIA B300: 288 GB HBM3e/GPU; Google TPU7x: 192 GiB HBM/chip (source code: 206 × 10⁹ B ≈ 191.85 GiB); 9.76 TiB ≈ 10,700 GB (decimal)
DCN 100 Gb/s per chip; cross-slice ~50× slower, same basisGoogle Cloud TPU7x docs (checked in TPU expert review round 5); 100 Gb/s read as one-way, ×2 ≈ 25 GB/s bidirectional, ~48× below ICI's 1,200 GB/s bidirectional, our derivation (one-way only: ~96×)
Definitions of TEP / DEPTensorRT-LLM tech blog 26 (DeepSeek V4 on Blackwell), verbatim; vLLM Kimi K3 blog (2026-07-27)
No TEP / DEP in Megatron; ETP / EDP / Parallel Folding NVIDIA/Megatron-LM main: megatron/core/transformer/moe/README.md; paper arXiv 2504.14960
PCP adds devices; DCP doesn't, reusing TP ranksvllm-project/vllm main: vllm/config/parallel.py docstring; vLLM context parallel deployment docs
Attention DP: motive, flagsSGLang DP / DPA guide; TensorRT-LLM docs/source/features/parallel-strategy.md; vLLM DP deployment docs
HelixarXiv 2507.07120; TensorRT-LLM tech blog 22
GTP quoteMegatron-LM docs/api-guide/core/generalized_tensor_parallel.md
Zero Bubble / DualPipearXiv 2401.10241; github.com/deepseek-ai/DualPipe
DiLoCoarXiv 2311.08105; Streaming DiLoCo arXiv 2501.18512; the diloco mesh axis in MaxText
AFDMegaScale-Infer arXiv 2504.02263; Step-3 arXiv 2507.19427; vLLM AFD Plugin blog (2026-07-23)
Chunked PPLMSYS blog (2026-01-15); Mooncake arXiv 2407.00079
MaxText's parallelism axesAI-Hypercomputer/maxtext: src/maxtext/configs/base.yml, docs/guides/optimization/sharding.md

← Course overview · Previous: Topic 4 · Backward pass and optimizers · To teach it: instructor script
Built by Courses/tools/topic05-build.py; the main text lives in that script.