How to use these notes
Two uses.
① Presenting: lines, cues and minutes are all here.
② Self-study: give the file to an AI for a TA who knows the background and how to teach it.
Content is on the course page; these notes cover only how to teach it.
Course author: Chris Yang · Google Cloud AI Infra architect.
| 🎯 | What this segment must leave behind The line the room keeps. |
| 🗣 | Script Yellow: read verbatim. |
| 🖥 | Screen Where to scroll, which animation to play.
⭐ Cues carry class="board"; the build flags a missing figure or section number. |
| ⚠️ | What not to say Where this segment drifts. |
⏱ Time: core track 84′
⛔ This lecture is a relay; skip no leg. Each section's closer, “What this cut leaves open,” opens the next: always say it. Short on time? Squeeze mid-section examples, not the ends.
| Segment | Min | Cuttable? | Notes |
|---|---|---|---|
| 0 Opening | 5 | No | ⭐ 16-byte warm-up a must; plant only: communication buys memory or compute |
| 1 The five collectives | 10 | No | ⭐ The equation is a must; skip the 1.7 table |
| 2 First cut: data | 10 | No | ⭐ First two stages free; the last costs 50% |
| 3 Second cut: weights | 14 | No | TP, FSDP each cover a side; columns before rows (GeLU counterexample); FSDP × TP 2D; PP: bubble only |
| 4 Third cut: experts | 10 | No | ⭐ Load imbalance is EP's worst disease |
| 5 Fourth cut: sequence | 13 | No | ⭐ Always show the KV-replication figure |
| 6 Fifth cut: the work | 11 | No | PD's kitchen analogy; ratio arithmetic; why AFD needs three micro-batches |
| 7 Onto the machine | 7 | No | ⭐ Chattiest traffic on the fast lane; pick the cut, then tune |
| 8 Panorama and close | 4 | Close: no | Model-read one row; flag three traps; return to the opening line |
| Total | 84 | Section 9 (Sources): not in class; open if pressed |
⛔ Above is the plan; below is the floor. “Floor” = verbatim script plus figure switches, excluding animation-loop waits (about 3 min in all). Recounted every build.
| Section | Script (words) | Figures | Floor | Planned |
|---|---|---|---|---|
| Section 0 · Opening: one device can't hold it | 359 | 3 | 3.3′ | 5′ |
| Section 1 · Five collectives | 837 | 8 | 8.0′ | 10′ |
| Section 2 · First cut: data | 903 | 8 | 8.4′ | 10′ |
| Section 3 · Second cut: weights | 1,244 | 10 | 11.4′ | 14′ |
| Section 4 · Third cut: experts | 875 | 7 | 8.0′ | 10′ |
| Section 5 · Fourth cut: sequence | 1,265 | 10 | 11.5′ | 13′ |
| Section 6 · Fifth cut: split the work, not the tensors | 990 | 6 | 8.6′ | 11′ |
| Section 7 · Onto the machine | 632 | 6 | 6.0′ | 7′ |
| Section 8 · Panorama and wrap-up | 418 | 1 | 3.2′ | 4′ |
| Total | 68′ | 84′ | ||
🧵 Before you go on stage: two threads
① Every cut pays for one kind of communication. Name it at each cut: which of Section 1's five? By Section 7 the room asks it: “so, fast lane or slow?”
② Every cut covers what the earlier cuts missed. Small batch, FSDP can't cope: add TP. V3 is nearly all experts: add EP. Long context, one sample won't fit: add CP. Prefill and decode want different cuts: add PD disaggregation. ⛔ Say “next, another kind of parallelism” and the thread snaps.
⭐ If the room keeps one sentence: choosing a parallelism strategy is choosing which communication to pay for, how often, on which link.🖥 Setting up the screen
⭐ A dozen-plus animations, all silent loops. Watch each through in silence, then talk; talking over it splits eyes and ears.
⭐ Press “Hide narration” (bottom right, or T) before projecting. Or the room reads, not listens.
Section 0 · Opening: one device can't hold it (5′)
🎯 Leave only one line: spend one kind of communication to buy memory or compute.
🖥 Screen: course page Section 0, “🎯 Warm-up” at top. Open it; show only Question 1's table; don't scroll (the lead gives the answer). Questions 2–3: previews.
A warm-up on last time. DeepSeek-V3, 671 billion parameters. In standard mixed precision, each parameter keeps five things in HBM: compute weights, gradients, master weights for the optimizer, first moment, second moment. How many bytes each?
💬 Interaction: take the room's numbers, fill the table, click “Check.”
Two, two, four, four, four: sixteen bytes. The first two half precision, for matmuls; the last three full precision, for the optimizer.
Times 671 billion: 9.76 TiB, ~10,000 GB.
Sixteen is the standard count, half-precision gradients. bf16 training often accumulates them in full precision (Megatron's default): eighteen.
Two previews; just think.
Question 2: which of the sixteen bytes is biggest? What would you cut first?
Question 3: V3 actually drops two parts to half precision. Which two? Why can't the master weights go too?
Answers in Section 2.
🖥 Screen: collapse warm-up, scroll to lead.
Today's biggest devices carry two to three hundred GB of HBM. Holding 10,000 GB needs thirty or forty to fifty or sixty of them.
And not just big models: an 8B model at sixteen bytes a parameter is 128 GB, too big for an 80 GB H100.
That's before activations, what each layer keeps for the backward pass. With long sequences they outgrow it all.
Even if it fits, one device can't finish. V3's report: 2.788 million H800 GPU hours. On one device, 318 years.
So we cut not just to fit, but to go faster.
V3's report sums its cuts in one line: PP 16, EP 64, ZeRO-1. Gibberish now; by the end you'll read it.
Each device holds only part, so they swap: a new communication per cut.
This course rests on one sentence: every cut spends one kind of communication to buy memory or compute.
🖥 Screen: the roadmap, Red notes are the plot. Point only at box titles and red notes; skip the “Toll” row.
Today's plot is one thread: cut, pay for one kind of communication; find what it missed, cut again.
Three questions per cut: what does it split, what does it pay, which link?
Five cuts: data, weights, experts, sequence, and finally the work itself. Each box's red note is what that cut leaves open, forcing the next.
Where would you cut first? Most split the data: a few samples each. Right. We start there.
But before cutting, meet the kinds of conversation devices have. Every cut adds one.
🔍 Going deeper: why these five things, the two mountains, how 318 years becomes two months (if pressed or time allows; not in the floor)
Why five, not one weight copy? Weights and gradients go half precision for speed: matmuls are the heaviest work; it computes faster and halves the bytes, saving memory and bandwidth.
The last three serve the optimizer, which picks each parameter's move. Adam keeps two numbers per parameter: the first moment tracks recent direction, so noise doesn't throw it off; the second tracks usual gradient size and sets each step: small for steadily large gradients, big for small.
So Adam's cleverness costs two numbers per parameter. With the full-precision master weights: three at four bytes, twelve of the sixteen.
Why eighteen? A step often sums gradients over many micro-batches; half precision runs out of digits, and small numbers added to big ones vanish. So Megatron accumulates in full precision by default: two becomes four, sixteen eighteen.
These sixteen bytes depend only on parameter count, not batch or sequence length. Activations scale with batch, length, width and depth. Stretch the sequence from 4K to a hundred-odd K and they grow tens of times.
So HBM holds two mountains: one follows the model, one the data. Some cuts shave the first, some the second.
Many think 8B is small: one device serves it. But inference needs only the two bytes of weights; training needs sixteen, eight times as much.
Now the other way. Over V3's 2,048 H800s, 2.788 million GPU hours is about 1,300-odd hours, under two months. Only if two thousand devices stay busy, never waiting on each other.
Two reasons to cut: to fit, split the model state; to go faster, spread the compute. Either way devices must talk.
Names up front are a wall. Each grows from the last cut's problem; name it then and it sticks.
🖥 Main screen for this segment: course page Section 0 · One device can't hold it · main figure Red notes are the plot
Section 1 · Five collectives (10′)
🎯 Leave two things: all-reduce splits into two halves; all-to-all is everyone sending everyone something different.
🖥 Screen: press “Hide narration” (bottom right, or T), then the whole 1.2 figure, Four rooted moves.
A thousand devices must add up their results. One collects it all? It sends and receives 999 copies. In a ring, each passes just 1.998.
Each name answers two questions: who sends to whom, and on arrival, is the data concatenated, summed, or left as is.
Ten seconds on the figures, all drawn alike. Four devices: device 0 blue, 1 orange, 2 green, 3 purple, each with data cut into four blocks; a dashed box is empty.
Key rule: vertical stripes mean a block was summed; the number of stripe colors is how many devices were added.
Picture a class: four students, homework in four pieces. The simple moves have a class rep: the root, device 0.
Top left: the rep gives everyone a full copy: Broadcast. Top right: it splits one copy, a piece each: Scatter.
Bottom left: each hands the rep a piece to assemble: Gather. Bottom right: each hands in a whole copy; the rep adds matching scores: Reduce.
🖥 Screen: open “See them move” (four animations). Watch one loop in silence, then compare gather and reduce full screen.
Watch one thing: gather and reduce differ by one move. Both send everything to the root; one concatenates, one sums. After the reduce, the root's blocks show all four stripe colors: all four added.
The root's flaw: done naively, everything squeezes through its one link; more people, more jam.
So training's workhorses have “All” in front. “All” means everyone ends up with the result; nobody asks the root.
🖥 Screen: the whole 1.3 figure, “All” in the name, then the three animations below. They pause at each ring step: point at the thick-outlined block, click “Next.”
Top left, AllGather: one block each, then all four. Concatenate, no sum.
Top right, ReduceScatter: a whole copy each; at the end device 0 holds only block 0's sum, device 1 block 1's. Sum, then split.
Bottom left, AllReduce: a whole copy in, all four blocks' sums out; data parallelism syncs gradients with it. Bottom right looks different; save it for last.
🖥 Screen: the whole 1.4 figure, Sum then concat.
Today's key equation: all-reduce equals “sum, each keeps one block,” then “concatenate, everyone gets a copy”. AllReduce = ReduceScatter + AllGather.
Sum: each device gets a block of the total; concatenate: each gets all. Same as one go.
Why split it? So the halves can run at different times, with work between.
Next section's first cut slips an “update” between them: every device stores a big chunk less, for zero extra communication.
🖥 Screen: the 1.5 ring animation, Same ring, two laps, step by step. At each pause: “the thick-outlined block picked up one more person,” then “Next.” At step 1, point at device 3's block: it exits right and wraps under to device 0.
No root: how does everyone get a copy? A ring; each talks only to its right neighbor.
ReduceScatter first. Each step, each device sends a block right, gets one from the left, adds it to its matching block. After step one some blocks sum two people, after step two three. Four devices, three steps: each holds one block with all four colors, the full sum.
Three more steps round the same ring, concatenating, not summing, give each total to everyone. That's an AllGather. Together: one AllReduce.
🖥 Screen: next figure, Same result: left, middle, then the table at right.
One AllReduce, two ways. Left, the root way: sum onto device 0, send it back. Device 0 sends three copies, others one.
Middle, the ring you just saw: ReduceScatter, then AllGather. Each sends one and a half copies. Same result; who does the work differs.
Right: the opening numbers, 999 versus 1.998. That's why it's the most important equation: it doesn't just hold, it wins.
The ring's cost: steps grow with device count. Big messages: per-device bandwidth decides; the ring is near optimal. Tiny ones: the wait per step decides; many steps hurt.
So libraries choose by message size; small messages go over a tree. TPUs are more direct: chips on dedicated fast links form a slice; large slices come wired as rings.
🖥 Screen (optional; if the room knows TPUs): in “A closer look,” play The same 4×4×4 cube. Silent through one loop (47 s; steppable). Axis by axis lights one direction per step; rotated lights all three. The static Same blocks: self-study.
TPUs add a layer: each chip sits on three rings, X, Y, Z. Gather along X, then Y, then Z: one direction works per step, and the last is heaviest.
Split the data in three, each on its own starting axis: all three directions busy, AllGather up to three times faster (bandwidth alone). ReduceScatter reverses; AllReduce, both halves.
🖥 Screen: the 1.6 figure, Dispatch scatters, combine retraces (① crossing lines, ② reversed, ③ transpose inset, bottom right), then the animation Dispatch fires, manual mode. Stop at “Dev 0 gets 11,” then watch it fly back.
The odd one out: AllToAll. Everyone holds a different piece for each other person: everyone scattering at once.
The classic case: expert parallelism. The router picks each token's device. In dispatch, all devices send every way at once, counts differing per pair: some receivers swamped, some idle. When the experts finish, it all goes back the same way: combine.
With equal-size pieces it's exactly a table transpose, a special case.
Why does the network dread it? Each piece is a private letter; nothing merges en route. The others run as neighbor-only rings; this can't: every pair has traffic, testing the network's whole cross-section. And sizes are known only once the model reaches that layer.
Remember: when we split the experts, every layer uses it twice, out as animated, back as the last caption says.
Count: eight names; the four with a root are building blocks. Five see daily use: AllReduce, AllGather, ReduceScatter, AllToAll, and point-to-point send and receive.
When we cut the model into stages by layer, it carries data between them.
Take away two things: all-reduce splits into two halves; all-to-all is everyone sending everyone something different.
On the opening roadmap, each box's “Toll” is one of these five.
Vocabulary done. The first cut: split the data.
🔍 Going deeper: why the ring is cheap, when it loses, why all-to-all is hard; the TPU 2D/3D torus (if pressed or time allows; not in the floor)
1. Why the ring is cheap
From four devices to n: n blocks, each step sends one n-th of a copy. ReduceScatter takes n − 1 steps, AllGather n − 1 more. For a thousand: 999 times two over a thousand, the opening 1.998. More devices only creep toward two.
The ring's trick: what each device sends barely depends on headcount. The root's work grows tenfold with ten times the people.
ReduceScatter costs about one copy, AllGather one. The equation holds in cost too; next section, needing one half, you pay one copy.
2. When the ring loses
We counted bytes, not steps. A thousand-device ring takes nearly two thousand steps, each waiting on the last; a tiny fixed cost per step adds up.
A collective's time has two terms: steps times fixed cost, plus data over bandwidth. Big messages: the second dominates; the ring is near optimal. Tiny ones, say one number: the first dominates; many steps are fatal.
A tree attacks the step count: a thousand devices merge in pairs, about ten rounds in, ten back out. More bytes than the ring, but twenty steps, not two thousand. So libraries pick by size: big ring, small tree.
3. Why all-to-all strains the network
Split the cluster down the middle. A ring AllReduce talks only to neighbors: two links cross the split. In AllToAll every device sends to every other: about half the data crosses. What counts is the gap's width, not one link's speed.
Machines differ. In one NVLink domain, switches link all GPUs, any to any in one hop: a wide gap. On a ring-connected machine, a letter to the far side passes several neighbors on crowded links.
More headaches. Sizes are known only after routing; devices often swap counts first. The device with an overloaded expert is extra busy; everyone waits. And letters can't merge: AllReduce sums partway and shrinks as it goes; AllToAll saves not one byte.
Last, point-to-point send/receive: one device hands data to a neighbor; of the five, easiest on the network.
4. A flat ring: the 2D torus
Sixteen chips, four rows of four, each with four links: left, right, up, down. Rows and columns wrap end to end, each a ring of four. A grid wrapped at both ends is a torus.
One big ring of sixteen works too, but uses only left and right links and takes thirty steps. The torus goes one direction at a time. First every row ReduceScatters at once; each chip keeps a quarter copy, its row's sum.
Then each column ReduceScatters that quarter: each chip keeps a sixteenth, the sum of all sixteen. The second half reverses, AllGather along columns then rows: everyone has the total.
Count. Bytes: about three quarters of a copy in the row phase; the column phase handles a quarter, sends three sixteenths. Fifteen sixteenths in all; the second half matches: about two copies, like the big ring. Steps: four phases of three, twelve, not thirty. No extra bytes, far fewer steps.
One waste: rows idle the vertical links, columns the horizontal. Halve the data, one half rows first, one columns first: all four links work. Links are bidirectional: one share each way, twice as fast again; in a ring of four the farthest chip is two hops.
A standalone AllGather or ReduceScatter runs half the recipe: AllGather small blocks first, big last; ReduceScatter the reverse, shrinking as it goes.
This assumes the rings wrap. When every dimension of a TPU slice is a multiple of four, each closes. In small slices some don't: a line, where the farthest data walks all the way, much slower.
5. A ring in 3D: the 3D torus
3D adds a direction: four by four by four, sixty-four chips, six links each. ReduceScatter along X leaves a quarter, along Y a sixteenth, along Z a sixty-fourth: the full sum. Then AllGather back along Z, Y, X.
Six small rings of three steps: eighteen. One big ring of sixty-four: a hundred and twenty-six. Bytes: still about two copies.
Idle links, same fix: split the data in three, rotate the order: X, Y, Z; Y, Z, X; Z, X, Y. All three directions work at once: triple one direction's bandwidth. The more axes a cut spans, the more links it uses; more in Section 7.
AllToAll gets no such deal on a torus. It can go one direction at a time, but letters can't merge; far ones take several hops, crowding links. The middle gap limits it, and a torus's is narrower than a big switch's. Section 4 settles that.
6. How many directions per link? Full duplex, and the v7x math
A common misconception: a ring sends on the right link and receives on the left, so each link carries one direction. In fact every ICI link is full duplex: two one-way streets, both full speed at once.
A one-way ring uses half: each chip sends right, receives from the left, leaving the right link's return lane and the left's outbound lane idle. Six links: twelve one-way streets. Rotating three pieces fills six; split each again, half each way: all twelve fill, usable bandwidth doubles again on top of 3×.
The v7x numbers. Officially, only a per-chip ICI total: 1,200 GB/s. The breakdown is our derivation: six links at 200 for both directions, 100 per direction; 600 out, 600 in, 1,200 together. External anchor: the Scaling Book's 90 GB/s per link per direction for v7. It lines up.
So for any ICI bandwidth, ask: one direction, or send plus receive? Time an AllGather with outbound bandwidth; divide by 1,200 and you come out twice too fast.
⚠️ Ring vs tree: one line only, “small messages go over a tree” (relaxed after this framing passed live on 2026-09-25).
🖥 Main screen for this segment: course page Section 1 · Five collectives · main figure “All” in the name
Section 2 · First cut: data (10′)
🎯 Leave one line: shave the idlest first; the first two stages are free, the last costs 50% more communication.
🖥 Screen: scroll to the Section 2 heading.
A teaser: ZeRO-3 knowingly pays 50% more communication, yet nearly every large model uses it. What does that half buy?
The first cut is the simplest: each device gets different samples: data parallelism.
Every device holds the full model and works alone, finding which way and how far each parameter should move: the gradient. Different samples, different gradients; one AllReduce averages them, and all update together.
It talks once per step, the least chatty of today's cuts. Not small, though: pure data parallelism at V3's size sends about 2.7 TB per device per step, just once.
And it does nothing for “doesn't fit.” Each device still holds the full 9.76 TiB, not a byte less.
Picture V3's weights as a library in 61 reading rooms, one per layer. V3 runs 128-way: 128 branches, each with the full collection. Nothing missing, but 128 identical copies.
💬 Interaction: don't switch screens yet (title and takeaway give it away). Opening question 2: “128 identical copies. Of the 16 bytes, which goes first?” Wait for “the 12.”
🖥 Screen: switch to the ZeRO: remove the duplicates figure. Hold on the long top bar, then go row by row. Save the red 1.29 TiB (“V3 per device” column) for “Even shaved this far.”
So shave the duplicates, one kind at a time, by size. That's ZeRO. Each parameter takes 16 bytes: blue, the weights, 2; orange, the gradients, 2; this long green stretch, 12, the optimizer's books: optimizer state.
Largest first. ZeRO-1 shards optimizer state: green gone. ZeRO-2, gradients too: orange gone. ZeRO-3 adds the weights; the bar nearly vanishes.
🖥 Screen: the Shave the idlest first figure. The long green dashed bar, then bottom to top.
Why this order? Who's busy in a step? Green, optimizer state, is largest but used only in the final update, idle all step. Shave it; nobody waits.
Orange, gradients, come per layer in backward; blue, weights, always busy. Shave the idlest first, the busiest last: shard it, and each layer first copies in others' slices.
🖥 Screen: the One ZeRO step figure. Panels ①, ②, ③, one at a time.
The first two stages are easy math. Each device keeps the optimizer state for one slice, so it needs only that slice's gradient sum: a ReduceScatter, not an AllReduce. It updates in place, then an AllGather rebuilds the new weights for all.
Together that's exactly one AllReduce. So ZeRO-1 or ZeRO-2 costs not one byte more than plain data parallelism, and each device keeps books only for its share.
Why is the in-place update free? Adam is per-parameter. Parameter i's new value depends only on its own gradient, two moments and master weight. So four devices updating a quarter each equal one updating all.
One exception: gradient clipping needs the whole gradient's norm. Each device sums its slice's squares; those are summed once. Negligible.
🖥 Screen: the green box answering opening question 3; open “A closer look” below it for the bf16 ruler.
Answer to question 3: V3 keeps the two moments in half precision, the master weights in full. Note: optimizer state holds no gradient; gradients are per-step raw material, zeroed after use. Full precision is for the gradient accumulator. So V3's optimizer state drops from 12 bytes to 8, plus 4 for the accumulator.
The ruler's ticks: values half precision stores near 1, 1/128 apart. Green, β₂ = 0.95: the new value weighs 5%, so a step moves several ticks and sticks. Red, the default 0.999: the new value weighs 0.1%, under half a tick: storing snaps it back. The moment is stuck.
Why not compress the master weights? Their tiny per-step increments accumulate; anything under half a tick rounds away. So V3 compresses only the moments.
Even shaved this far, the “V3 per device” column shows 1.29 TiB, over 1,400 GB. Still doesn't fit.
🖥 Screen: the One library, three ways to split it figure. Point at ③'s red cuts, the red box on Branch 3, then ④.
So to the last stage: shard the weights too. In PyTorch, FSDP. This one isn't free.
Back to the library. FSDP ignores what the books are: it lays a room's books page by page in one strip and cuts 128 equal lengths, one per branch. Branch 3 holds the end of one book and the start of the next. Alone, nobody can read it.
To use the room, each branch copies all 128 pieces and rebuilds it; after reading, it drops the copies, keeping its own piece. That copying is the AllGather.
Bottom: EP, later, is opposite: whole shelves, each a whole expert, usable as is. FSDP shreds only to even out communication; useless until rejoined.
🖥 Screen: the whole Why FSDP costs 50% more figure.
Two rows. Data parallelism: two colored blocks per layer, sum once, gather once. FSDP copies in the other slices before a layer, then discards them. Backward, computing gradients in reverse, needs them again: another copy; then one gradient sum.
Three blocks; the extra one, this blue block in the middle, is the 50%.
🖥 Screen: the FSDP: copy in the other slices animation. Silent through one loop (14 s).
Watch the counter. Forward copies each layer in once; backward copies it again and splits the gradients once. Three layers, nine transfers.
In return, each device's resident data drops from about 10,000 GB to roughly 84 GB, before activations.
But V3's H800 has only 80 GB: still no fit. Data alone isn't enough; hence the next cuts.
One more, for experts. With micro-batches, ZeRO-2 is no longer free: without a full gradient, it can't add micro-batches up first, so it sums after each. ZeRO-3 includes ZeRO-2, shares the problem, and reassembles the weights every micro-batch too. So with pipelining, V3 picked ZeRO-1, not ZeRO-3; memory came down through pipelining and expert parallelism.
🔍 Going deeper: a ZeRO-1 step, where the accumulator counts, reduction precision, why V3 uses only ZeRO-1 (if pressed or time allows; not in the floor)
1. What does ZeRO-1 shard?
Master weights count as optimizer state: the 12 bytes are master weights 4 plus two moments at 4 each. ZeRO-1 shards all three.
Each device keeps full half-precision weights and gradients, plus its slice of master weights and moments. Slices are equal cuts of one long strip of all parameters, not whole layers.
2. How a step runs
One update per step, after backward finishes every layer, not per layer. Forward and backward use the full half-precision weights and leave a full gradient set.
Normally you'd AllReduce now. But each device owns one slice and needs no other slice's sum: it only ReduceScatters.
It updates its slice of master weights and moments and converts to half precision; only its slice of the half-precision copy is now new.
Then an AllGather: each device's new slice overwrites the old values everywhere, so all have full new weights for the next forward. The ReduceScatter can go bucket by bucket during backward; the update stays once per step.
3. Where the accumulator counts
The accumulator isn't optimizer state; it takes the gradient's slot. The 16 bytes assume one batch per step. Several micro-batches must be summed first; in half precision small ones round away, so a full-precision accumulator takes the slot: 2 bytes become 4.
ZeRO-1 doesn't shard it. For V3: half-precision weights about 1.3 TB, accumulator about 2.7 TB, exactly twice; master weights and moments split 128 ways, only tens of GB. After shaving, the accumulator is the biggest thing on the device.
Since the accumulator needs a full copy, ZeRO-2, sharding gradients, has nowhere to accumulate: it must ReduceScatter every micro-batch. With the V3 paper's precisions, its bill: weights 2, accumulator 4, master weights 4, moments 2 each; 14 total.
4. Half-precision gradient reduction?
Intuition: you've accumulated locally, so send half precision and halve the bandwidth. The catch: ReduceScatter is itself a sum. Each ring hop adds its part to the partial sum and passes it on: over a hundred additions to one number.
In half precision, every hop stores and rounds that growing partial sum, so small values added later get swallowed. The problem is storing it at each hop, not the transfer.
Both are used. Megatron's grad_reduce_in_fp32 picks full or half precision for the gradient buffer and the reduction. NVIDIA's recipes for DeepSeek-style models turn it off: half precision throughout, halving memory and bandwidth, at some accuracy cost. The V3 paper doesn't say which it uses.
It's also the main reason data parallelism and FSDP losses drift slightly apart in long runs: a different addition order, different floating-point results, amplified by training.
5. Gradient life; who shards what
Strictly, gradients needn't live as long as weights: they appear layer by layer in backward, can be zeroed after the update, and forward never uses them. But Megatron-style frameworks keep one contiguous gradient buffer all step, so the memory math counts it resident: 2 + 2 + three 4s, 16 bytes.
The accumulator doesn't sit beside the half-precision gradient; it replaces it. Each layer's half-precision gradient is added in and dropped at once; only the accumulator stays. 2 becomes 4, so 16 becomes 18, not 20.
Who shards what: ZeRO-1, only those 12 bytes; the accumulator, in the gradient slot, stays. ZeRO-2 shards the gradients, accumulator included. ZeRO-3, i.e. FSDP, adds the weights; all three split 128 ways.
In stages that shard gradients, when backward reaches a layer, its full gradient appears briefly on every device. A ReduceScatter leaves each device its 1/128 of the sum and frees the rest. With micro-batches, that repeats per micro-batch, into the device's accumulator slice.
6. Which stage does V3 actually use?
The V3 paper: 16-way pipeline parallelism, 64-way expert parallelism, ZeRO-1 data parallelism. 2048 devices ÷ 16 stages = 128-way data parallelism, the lecture's 128. But those 128 ways run ZeRO-1, not FSDP.
ZeRO-3 includes ZeRO-2, so it inherits the micro-batch problem, worse. Besides a ReduceScatter per micro-batch, it re-gathers every layer's weights in forward and backward, and a pipeline step runs many micro-batches.
Memory comes from the other cuts: 16-way pipelining leaves 1/16 of the layers per device; 64-way EP spreads the experts. ZeRO-1 takes the largest piece left, optimizer state, at no extra communication. The main text's 128-way FSDP at 84 GB/device is a data-parallel-only what-if showing that alone isn't enough, not V3's real setup.
Also remember: data parallelism talks once per step, FSDP hundreds or thousands of times. The chattier it is, the more it needs the fast lane; more in Section 7.
What does this cut leave open? Each step FSDP moves the weights, an amount set by model size alone. Compute scales with each device's tokens; a token is roughly a word.
Add devices, and the total fed in at once can't grow without limit (too big, and the model learns worse), so each device gets fewer.
Less compute, same traffic: moving weights eats all the time, forcing the second cut.
2.4's two items cover how far this cut reaches. Skip; if asked: “FSDP within a machine, data parallelism across machines.”
⚠️ Data parallelism is “least chatty,” not “cheapest”: per-step traffic is large (about 2.7 TB per device for V3); only the frequency is low.
⚠️ Easiest slip: “FSDP is free.” ZeRO-1 and ZeRO-2 are free; ZeRO-3 pays 50% more.
With micro-batches, only ZeRO-1 stays free (said on stage; why: the “One ZeRO step” figure's sources).
🖥 Main screen for this segment: course page Section 2 · First cut: data · main figure ZeRO: remove the duplicates
Section 3 · Second cut: weights (14′)
🎯 Leave only one line: FSDP ships tools, tracks batch; TP passes door panels, tracks hidden size.
🖥 Screen: scroll to the Section 3 heading.
How few is “few”? Simple question: move something once, how often is it used?
Picture a door workshop: the craftsman takes two seconds per job, the runner five per delivery; even a fast craftsman waits. Device = craftsman; network = runner.
The runner carries two things. Tools, like planers and molds: the weights. Door panels, half-made doors passed between stages: the activations, one per token.
So: how long does each delivered byte keep him busy?
One end of the ruler is hardware: hand speed over leg speed. A TPU v7 chip does over 2,300 trillion operations a second; the network brings in about 600 billion bytes a second. Divide: over 3,800.
Note “brings in”: FSDP must receive the others' weight slices before a layer. On a ring, each device sends what it receives, 600 billion each way; the usual 1.2 trillion adds both.
So each byte brought in needs over 3,800 operations queued, or he waits.
The bottleneck is the network, not HBM. Arriving weights are still read from HBM, but at over 7.3 trillion bytes a second, over 12× the network; its line is only about 310, the gray dashed line at the bottom.
🖥 Screen: the whole Two cuts, two ratios figure; the red dashed line first.
X: tokens per device per step. Y: FLOPs per byte brought in.
The red line is that 3,800-odd; the red area below is the craftsman waiting.
The blue diagonal is FSDP. Its runner ships tools: the weights, Section 2's reading room each layer reassembles. Each arriving tool works every panel in his hands: more panels, longer busy.
The math is clean: a parameter is two bytes; each token does one multiply-add with it, two operations. They cancel: FLOPs per byte = tokens per device. Under about 3,800 tokens per device per step, FSDP is held back. Our own, most optimistic derivation.
What's 3,800 tokens? One V3 pre-training sample is 4K tokens: roughly one sample per device. Any fewer and weight-moving shows.
🖥 Screen: same Two cuts, two ratios; the thin dashed line above the red.
A trap: the red line's denominator, the runner's speed, can't come from the spec sheet; that's physical bandwidth, and real gathers lose some.
Measure instead: run a real AllGather on your network (nccl-tests on GPUs) and read its busbw, the bytes each device actually receives per second, net of the algorithm's own accounting.
At 70% of spec, the threshold rises from about 3,800 to about 5,500: this thin dashed line. The 70% is my assumption. With few links, or links shared with other parallelism, it rises further.
The second cut stops moving weights and cuts the matrices: tensor parallelism, TP.
🖥 Screen: the whole TP on an MLP figure.
An MLP's two weight matrices: UP expands, DOWN shrinks back (W1, W2 on screen). NVIDIA Megatron's trick: cut the first along its output dim, the second along its input.
Two devices, half each: blue meets only blue, orange only orange; no talk in between. All communication moves to the exit: one AllReduce at the end to sum.
Two craftsmen, one batch, half the stages each; pooling the panels is the AllReduce.
🖥 Screen: the Tensor parallelism splits an MLP animation. Silent through one loop (9 s).
Two traps. First, each device holds half of UP and of DOWN. Not one matrix per device; that's splitting by layer, later. They don't talk; one sum after DOWN, sending activations.
Second, “horizontal” or “vertical” is just drawing. We draw weights as input × output, so the output-dim cut looks vertical; PyTorch stores output × input, where it looks horizontal. Remember: first matrix by output dim, second by input dim.
🖥 Screen: the Why columns first figure, left, then right.
Why columns first? Try rows first: each device holds a share of a number; only the sum is real. Device 0 has 2, device 1 minus 3; the true value is minus 1.
GeLU isn't a straight line. Add, then GeLU: minus 0.159. GeLU each, then add: 1.950. Even the sign flips. So you must sum first: an extra communication from nowhere.
Cut columns, and each number lands whole on one device. GeLU runs locally.
How often? Forward: once at the MLP exit, once at the attention exit. Backward mirrors it: X is shared, each device gets half its gradient, so the entry sums again.
Two forward, two backward, per layer. So TP stays in the fastest tier: on GPUs, usually one machine.
🖥 Screen: back to Two cuts, two ratios; the two flat orange lines.
The TP line is flat. Why? TP's runner passes panels, not tools: the activations, one per token. Twice the tokens, twice the passing, twice the work: they cancel.
It depends on two things. One, the numbers per token: the hidden size, the panel size. Bigger panels, more work per delivery, a higher line. Two, the split count. Finer splits, less work each, same delivery: the line drops.
V3 at 8 ways is this solid line: touching the optimistic red line, below it in reality. 32 ways: the dashed line below.
So each cut covers one side. The data fed in at once is the batch: small batch, lean on TP; large batch, lean on FSDP.
Figure: left, blue is still low, orange carries it; right, blue passes it; FSDP takes over.
🖥 Screen: the FSDP + TP figure. ①: the big matrix, then the 8 devices; ② left to right; then ③'s two boxes.
Can FSDP and TP combine? Yes, often, in large-model training: they cut different dimensions.
These 8 devices form two rows of four. TP cuts the weights along the output dim into four vertical strips, one per column; FSDP halves each strip. Each holds 1/8.
A row of four is a TP group, same data; a column of two, an FSDP group, different data.
One layer, three steps. One: AllGather within the column, joining the two halves: only your strip, a quarter of the matrix.
Two: each computes a quarter locally, on the row's shared data. Three: AllReduce within the row to sum partial sums, sending activations.
The best part is the math. FSDP alone: each device gathers a whole layer for its own tokens. Combined: each gathers a quarter strip, used by all tokens in its row of four. So FSDP's threshold counts the whole row's tokens: only around 960 per device.
That's what “small batch, lean on TP” really means: TP lends a row's tokens to FSDP. The price: the TP axis syncs every layer and needs the fastest links.
🖥 Screen: the TP cuts wide, PP cuts deep figure, left, then right; count the red dots.
TP can't leave the fastest tier. To span slow links without moving weights each layer, cut the other way.
TP cuts wide: four strips per layer, every layer syncs; a red dot per layer, left. PP cuts deep: layers 1–2 to device 0, 3–4 to device 1; panels pass only between stages, by Section 1's point-to-point send/receive. Just three red dots, right: slow links suffice.
But a naive deep cut gives four devices' memory at one device's speed: only one stage works at a time.
So cut a batch into pieces and feed them in turn. Same workshop, now an assembly line: planing, drilling, painting, one station each; stacks of panels flow down. While the first stack is planed, the other two stations wait.
🖥 Screen: the Pipeline parallelism animation. Silent through both runs, 4 and 8 micro-batches (14 s). The static PP's cost, the bubble: self-study.
That waiting is the gray cells: the bubble, waiting for the pipe to fill at the start and drain at the end.
Finer cuts, smaller bubble: the same work in 8 pieces halves the gray. But not always better: tiny pieces make matrices too narrow to fill the device, and hand-off waits grow with the piece count. Overdo it and you eat the savings back.
🔍 Going deeper: what was new in GPipe (if pressed or time allows; not in the floor)
1. Not pipelining, but making it synchronous
GPipe is a 2018 Google paper. Relaying layer stages was old; GPipe made pipelining synchronous and general. The core is this section's trick: cut a batch into micro-batches, feed them in turn, accumulate their gradients, update once at the end.
Mathematically, the result equals no pipeline: one big device start to finish. Microsoft's PipeDream, same era, went asynchronous, micro-batches on different weight versions: no bubble, but a different result and convergence worries. GPipe kept the result; the price is the bubble.
2. Tied to recomputation
Each stage keeps only its boundary activations; inner ones are dropped after forward and recomputed in backward: a little compute for a lot of activation memory. Pairing it with pipelining, GPipe popularized today's activation checkpointing.
3. Bubble math, and the scale then
The paper's conclusion: at least four times as many micro-batches as stages, the bubble is negligible; hence the course page's rule of thumb. They trained a 6-billion-parameter, 128-layer Transformer with it, translating over a hundred languages.
The static figure is GPipe's schedule: all forwards, then all backwards. Later 1F1B: same bubble, less activation memory.
What this cut leaves open starts with an oddity: V3 trained with no TP at all; the report says memory was optimized enough without it.
And the parameters? Nearly all sit in narrow “expert” networks. Cut a very narrow matrix and each device gets even less work, yet passes as many panels: why the flat line dropped.
So don't split matrices; split by expert.
Conclusion only: TP's math ignores batch; it depends on hidden size and split count. Formula: 3.4.
⚠️ The three PP improvements (VPP, Zero Bubble, DualPipe): name only; don't explain mechanisms.
🖥 Main screen for this segment: course page Section 3 · Second cut: weights · main figure Two cuts, two ratios
Section 4 · Third cut: experts (10′)
🎯 Leave two things: EP's worst problem is load imbalance; attention and the experts can each get their own split.
🖥 Screen: scroll to the Section 4 heading.
97% of V3's parameters live in its experts. But patients don't book evenly: the busiest expert decides how long everyone waits.
This is MoE. Past the first three layers, every V3 layer has 256 small networks, called experts; each token picks just 8, never touching the rest.
🖥 Screen: V3 is almost all experts, whole figure; then the green box at the bottom.
Of every 100 parameters, 97 sit in the experts. Attention, the vocabulary and each layer's shared expert, which every token passes, are that sliver on the right.
Could the last cut, TP, split the experts? It doesn't pay. A V3 expert's middle layer is only 2,048 wide. Split 8 ways, each device keeps 256. Tiny matrices, every door panel still to pass.
So MoE's cheapest split doesn't cleave experts; it hands whole experts to different devices. Move the patients, don't split the doctors. That's expert parallelism, EP.
In the library: 256 expert shelves go whole to 64 wings, 4 each, usable on arrival, the opposite of FSDP's shred-and-reassemble.
The bottom line is V3's real training setup: 16-way pipeline, 64-way expert parallelism, no TP at all.
🖥 Screen: Dispatch, queue, combine (the EP animation). Stay quiet for one full loop (10 seconds). Replay for the problem; point when expert 0 has 7 queued.
Its communication is the last one from Section 1: AllToAll.
Like hospital triage: the router picks each token's experts; it's dispatched and queues at their door; once seen, it goes home. Twice per layer.
Networks hate AllToAll most. V3 tames it with two tricks.
🖥 Screen: Dispatch: one copy per machine, fan out inside, left then right; finish on the short green arcs.
Left, the naive way: a token picks 8 experts; on 8 different machines, that's 8 cross-machine copies.
Right, V3: the router picks machines before experts, at most 4. Each machine gets one cross-machine copy, landing on the same-index device; NVLink then forwards it to the real experts. Dispatch also drops from two bytes to one. Net: a quarter of the left.
Why are extra hops inside fine? NVLink is about 3.2 times faster than the cross-machine network, so while one copy crawls across, 3.2 can be forwarded inside. No extra time.
4 machines, 3.2 experts each: about 13. So per the report, the same traffic could cover 13 experts; V3 uses 8. Cost depends on machines reached, not experts picked.
But this cut's real problem isn't communication; it's this queue. Other cuts are mostly even by config, known in advance.
Not EP: which expert is busy shows only after routing, and changes every batch. Expert 0 has 7 queued, the others one or two. Every device waits for that queue.
Worse, V3 drops no tokens. Others often cap each queue and discard overflow; for V3, imbalance turns straight into waiting.
And it snowballs: a popular expert gets more tokens, learns better, gets more popular; the rest wither. You pay for a big model's memory and train a small one.
🖥 Screen: Bias as a ticket at the door, three panels left to right; point at the red dashed line.
The old fix penalized imbalance in the training objective: pressuring the triage nurse to change the diagnosis, which hurts learning. V3 hangs a ticket dispenser at the door instead.
After every step it nudges scores: overloaded experts 0.001 less, idle ones 0.001 more.
Why is the diagnosis untouched? The router's score does two jobs: pick the top 8, and weight each chosen expert. The ticket touches only the picking; weights use the original score. Which door changes; the diagnosis never does. In this illustrative simulation, the busiest expert falls from over 2× the average to just over 1×.
At inference, hot experts are also replicated across devices.
🖥 Screen: Attention vs experts: same devices, two mappings.
One more thing: in one layer, attention and experts are shaped completely differently.
Left: each group of 4 devices computes one attention on its own batch of requests. Right: the same 8 devices, 32 experts each.
🖥 Screen: TEP vs DEP in attention, left then right, row by row (requests, attention, KV, experts); end on the bottom four rows.
The two commonest inference setups are TEP and DEP; the number counts all devices. Experts are spread out in both; the difference is attention: do devices share one batch of requests, or take one each?
TEP4 on the left. Four request batches, all on every device. Attention splits by head, a quarter each. But V3-like models have one unsplittable KV head, so every device stores every request's KV. Four red boxes, three wasted.
After attention, an AllReduce sums the partial results; now every device has all the tokens. Each computes its own experts; another AllReduce combines. Upside: four devices per request, so one request generates fast.
DEP4 on the right. D is data parallel: each device runs only its own batch and stores only its KV, no duplication. Attention runs locally, no communication. The cost: full attention weights on every device, only a few percent in MoE.
For experts, tokens go out and come back by AllToAll.
Upside: no duplicated KV, so memory fits far more requests, experts get bigger batches, throughput rises. Catch: uneven request lengths mean uneven load; an idle device must still run an empty forward pass, or everyone's AllToAll waits.
In one line: TEP makes one request fast; DEP runs many at once. Decode is a throughput game and the KV takes the most space, so DEP is often the pick.
How big a difference? On GB300, switching decode from TP4 to DEP8 doubled per-device throughput at the same concurrency. Section 7.3 has the accounting.
The biggest piece is the left's row of red boxes: unsplittable KV, copied to all four devices by TP4. The next cut explains why.
Still open: the experts are spread out, but attention holds something that grows with conversation length, untouched by the first three cuts: the sequence.
If pressed: the runbook's “2.85×” is total throughput, DEP8 at concurrency 768 vs. TP4 at 512 (60,065 ÷ 21,100); “3.09×” is the 1,536 row. At equal concurrency 512 it's 2.61×, the page's number.
⛔ Don't pair TEP / DEP with Megatron's ETP / EDP.
Near-identical letters, different meanings; it costs three minutes. After class, point to the 8.6 warning box.
⚠️ For GB300, say “per device” and “same concurrency”: 2.09× per device (2.61× total). Concurrency 1,536 reaches 2.47× per device, but time to first token is 95 s; don't lead with that.
🖥 Main screen for this segment: course page Section 4 · Third cut: experts · main figure V3 is almost all experts
Section 5 · Fourth cut: sequence (13′)
🎯 Leave one line: training splits activations, inference splits the KV; once TP exceeds the KV head count, every extra way is pure duplication.
🖥 Screen: scroll to the Section 5 heading.
One 128K request carries 8.58 GiB of notes. Over 8 devices, the notes split 8 ways, right? Not necessarily. One kind of split stores it 8 times: seven copies wasted.
In training, activations grow with the sequence; in inference, these notes.
First, why one sample stops fitting.
Training: activations grow at least linearly with length. Without FlashAttention, the attention term grows with the square.
Inference is more concrete. Per token V3 stores one compressed note: 512 numbers plus 64 positional, 576 at 2 bytes each, across 61 layers. Over seventy thousand bytes per token.
Times 128K tokens: 8.58 GiB, a ninth of an 80 GB GPU, for one request. Production serves dozens to hundreds.
In training this is context parallelism, CP: cut one long sequence into chunks, one per device.
The hard part is attention: every token must see all earlier tokens, now on other devices.
🖥 Screen: the Ring Attention animation. Stay quiet for half a loop (loop 9 s).
One approach is Ring Attention. Each token's question, Q, searches everyone's notes.
Your Q stays put. The others' notes, the KV, come round the ring a chunk at a time; compute on each as it arrives, and pass it on. After one lap your row is full: Section 1's ring again.
Why can it hide communication? Each step sends one chunk of notes, linear in chunk length, and computes questions × notes, quadratic in it.
The longer the chunk, the more compute leads: long sequences suit this ring best.
How does a device get the full answer? Its questions meet each chunk once, each giving a partial result; after a lap, merge them. Merging has a trap, met later with DCP: same trick.
The other approach is Ulysses. One term first: attention splits into many independent heads; V3 has 128.
Ulysses uses AllToAll's equal-pieces case, a transpose, to go from chunk to head splitting: a few heads per device, whole sequence, attention done locally, then back.
Remember: Ring keeps people put, circulating notes; Ulysses re-cuts the table.
🖥 Screen: Ulysses: transpose once and the notes stay home. ①'s gray table at left (red vertical arrow, then one row), then its Ring at right; then ②'s three tables, left to right.
Lay out attention as a table: each row a chunk of the sequence, each column a head.
In a column, later chunks must see earlier ones: crossing chunks means passing notes. Within a row, heads never talk.
Ring splits by row: one chunk, all heads per device. Column dependencies cross devices, so notes must lap the ring.
The Ulysses idea: columns have dependencies, rows don't. So split by column!
Data arrives split by chunk, as the per-token operations before attention are. One AllToAll: device k sends cell j of its row to device j, keeping 1 cell, sending 3.
Middle table: each device holds one head, all chunks. Its column's needs are all local; attention finishes without asking anyone. Another AllToAll switches back to chunks for the rest of the layer.
🖥 Screen: the Ulysses animation. Uncheck “Autoplay”; click “Next ▶” one step at a time. Stop where head 0 is scattered over all four devices, where device 0 sends alone, where all send (26 s total).
Now by device: left column split by chunk, right by head. Watch device 0: head 0 stays; heads 1, 2, 3 fly to the right columns of devices 1, 2, 3. The others do likewise at once. Each right column then holds all four chunks of one head.
Ulysses' upside: double sequence and devices, and per-device communication stays flat.
Its ceiling is the head count: at least one whole head per device; with fewer KV heads than query heads, the KV head count is the limit.
Ring has no such cap, but every step waits for a chunk of notes.
🔍 Going deeper: Ulysses vs. TP; V3 step by step; when Ulysses, when USP; Ulysses vs Ring (if pressed or time allows; not in the floor time)
1. Inside attention, it's exactly TP
After the swap, each device computes a few heads over the whole sequence: exactly TP's shape on attention. It doesn't “degenerate into TP” with one chunk: chunks and head groups both equal the device count, so heads can't split without the sequence splitting.
2. The whole difference is outside attention
Three things. Weights: TP splits them by head; each device uses only its heads' projections. Ulysses projects, then swaps: each device computes all heads' Q, K, V for its tokens, so at the moment of use it needs every head's projection weights. Alone it saves activations, not weights; the paper adds ZeRO-3: weights sharded, AllGathered when needed.
Outside attention: each TP device holds the full sequence; a Ulysses device, only its chunk, with full weights. The MLP moves not one byte.
What's sent: TP, partial sums, an AllReduce after the output projection over the whole sequence's activations. Ulysses, the activations, each device one over the device count of its share.
Closest relative: Megatron's TP plus sequence parallelism, which AllGathers the whole sequence's input to every device before attention. Ulysses computes Q, K, V locally, then swaps. The paper compares exactly this: Megatron sends 4Nh per device per layer, Ulysses 4Nh ÷ device count.
3. Walking through V3: 128 heads, a million tokens, 128 devices
V3's config has 128 heads. Set MLA aside; pretend each head's Q, K, V are 128 dims. Take 2 to the 20th tokens, just over 1,048,000, cut into 128 chunks of exactly 8,192, one per device.
Step one: each device projects its 8,192 tokens into Q, K, V with all heads' weights, so it holds all 128 heads for its own chunk. Q alone is 8,192 × 128 heads × 128 dims, 256 MiB in half precision; K and V likewise.
Step two: the first AllToAll. Each device cuts Q by head into 128 blocks, 8,192 tokens × one head, 2 MiB each. Head n goes to device n, its own kept. Same for K, V.
Device 5 now has one block from each of the 128 devices: all head 5, chunks 0 through 127. Join them in chunk order: head 5's Q, K, V over the whole million tokens. That's the concatenation; order matters.
Step three: device 5 runs FlashAttention with the causal mask for head 5, asking no one. The million-squared score matrix is computed only block by block. Bonus: identical work per device; the causal mask leaves no one busier.
Step four: the second AllToAll retraces the route. Device 5 cuts head 5's output into 128 chunk blocks, chunk n to device n.
Now device 0 holds heads 0 through 127, all for chunk 0. Join them along the head dimension: chunk 0's complete attention output. That's the merge.
Step five, back to normal: device 0's full output projection compresses 128 heads to 7,168 dims; residual, LayerNorm and MLP finish its 8,192 tokens locally. Repeat per layer.
4. The numbers
Per device per layer, outbound: Q, K, V and output, four 256 MiB tensors, 127 blocks of each sent. About 1 GiB, exactly 4Nh divided by 128. Backward: the same again.
Versus 128-way TP: each device holds all million tokens, so one layer's input activations alone are 14 GiB. Each AllReduce sums that 14 GiB; on a ring each device sends nearly twice that, once for attention, once for the MLP.
In one line: inside attention, Ulysses is TP by head; but it projects before swapping (needing every head's weights), and outside attention handles only its chunk. Price: one swap before, one after, each moving one over the device count.
5. When to use Ulysses, and when to combine it into USP
Weights first: Ulysses doesn't save them; that's ZeRO-3's or FSDP's job. The paper's section 3.3 merges the sequence-parallel group into the ZeRO-3 sharding group; the video example below pairs FSDP with USP. So its limits are head count and network, not weights.
Fits: tiny batch, very long sequence; precisely, a single sample doesn't fit on one device. Data parallelism splits by sample, stopping at one per device; below that, split inside the sample. Short sequences, big batches: data parallelism suffices.
Case one: long-context training. Extend a model from thousands of tokens to hundreds of thousands or a million, and no device holds one sample's activations. Ulysses handles activations, ZeRO-3 weights.
Case two: video-generation inference. The Wan 2.1 README runs 8-GPU inference as FSDP plus USP, Ulysses = 8 for the 14B model. One request, one clip: batch 1, and high-resolution video is a very long token sequence. Attention is bidirectional: no causal mask, no KV cache. The user wants this one request fast, beyond data parallelism's help, so the sequence is split across eight GPUs.
The same README has a counterexample: the 1.3B model has only 12 heads, not divisible by 8, so its 8-GPU runs switch to Ring.
Doesn't fit: LLM decode, one token per step, nothing long to split; splitting the KV cache is DCP's job (later). And TP, if on, also splits by head, leaving Ulysses fewer heads.
Use USP in three situations.
More devices than heads. The USP paper's example: Llama 3 8B has only 8 KV heads, so 16-way means Ulysses 8 × Ring 2.
Heads don't divide. For that 12-head model the README recommends pure Ring; by USP's rules Ulysses 4 × Ring 2 also works (4 divides 12): derived from the rules, not in the README.
Crossing machines. AllToAll hates slow links: keep the Ulysses dimension on one machine's 8 devices, over NVLink. Ring, neighbor-only and overlapping, goes between.
Back to V3, counting bytes (per device per layer, forward, head dims as assumed). Pure 128-way Ulysses: about 1 GiB, under 1 GiB of it cross-machine. USP with Ulysses 8 inside and Ring 16 across: the cross-machine ring alone is about 7.5 GiB.
So with 128 heads to go around and a network that handles the AllToAll, pure Ulysses sends least. USP is the fallback when heads run short or cross-machine AllToAll can't run at full speed. Formula-derived, not measured.
In one line: a sample doesn't fit and one request must be faster, Ulysses; heads run short or you cross machines, USP. Weights always go to FSDP or ZeRO-3.
6. What Ulysses is really good at, compared with Ring
Misconception: AllToAll sends more. The reverse. Same V3 example, per device per layer, forward: under Ring each device collects the other 127 chunks' K and V, about 64 GiB. Ulysses: about 1 GiB, over sixty times less.
Why? Ring: each device always sees the whole sequence's K and V, about 2Nh, at any device count. Ulysses: each swaps one over the device count of its share, 4Nh divided by the device count. More devices, less sent.
Three more strengths.
Attention in one piece. After the swap each device has one head over the whole sequence: off-the-shelf FlashAttention works, sparse and sliding-window variants unchanged. Ring merges softmax block by block: a custom kernel, and small blocks can't saturate compute.
Balanced by construction: one head over the whole sequence each, identical work. Under a causal mask, Ring leaves later chunks busy, earlier ones idle, unless you shuffle the sequence.
Fewer steps: two AllToAlls per layer, before and after; Ring passes 127 times.
Ring's two strengths: no head-count cap, and communication overlapped with compute. Compute grows with the square of length, so at very long sequences each step computes far more than it sends, hiding those sixty-plus GiB. Ulysses' AllToAll can't hide: compute waits on it, on the critical path.
So at 128 devices and a million tokens, plain Ring works; compute probably covers communication. Price: sixty-plus times more bytes on the NICs, fighting data and expert parallelism for links. With enough heads and a network that handles the AllToAll, Ulysses is cheaper.
🖥 Screen: USP stacks Ulysses and Ring. First ①, left table to right; then the red up-down arrows in the right table; finish on ②'s comparison table.
The two stack: USP. Same table, in blocks. 4 devices, 2 machines.
Step one: AllToAll inside each machine, Ulysses between just 2 devices. In machine 1, device 0 takes heads 0 and 1 of the first two chunks, device 1 heads 2 and 3. Machine 2 likewise with the last two chunks.
Each device now holds two chunks × two heads; the ring crosses chunks: upper and lower blocks with the same heads swap notes, device 0 with 2, device 1 with 3. This step crosses machines.
Table below: USP's devices = Ulysses dimension × ring dimension. The head cap binds only the Ulysses dimension; the ring grows freely, so few-head models still spread wide.
Placement: AllToAll hates slow links, so it stays inside; the neighbor-only, overlapping ring goes between. A common layout inferred from how the two behave, not a hard rule from the paper.
🖥 Screen: the USP animation, manual, one click at a time (27 s total). Linger where device 0 owns heads 0 and 1 but chunks 2 and 3's notes sit on device 2.
First, devices 0 and 1 swap heads in machine 1; machine 2 likewise. The red box is device 0's gap: chunks 2 and 3's notes, on the other machine.
Step two: translucent copies fly across, device 0 swapping with 2, device 1 with 3. Each computes, discards the copies, swaps back to chunks inside the machine.
🖥 Screen: Causal attention split by sequence. Point only at the left triangle at first.
Generative models add a headache: tokens look only backward, so later chunks do more work. Split contiguously, device 3 computes 15 cells, device 0 just 3; all wait for device 3.
💬 Interaction: cover the right half, title and takeaway band too (the title says “zigzag evens it out exactly”). Ask “8 blocks, 4 people: fairest split?” Wait for “pair light with heavy.”
Pair the lightest block with the heaviest, on the same device. Split like the right side, every device gets 9 cells.
🖥 Screen: Inference: TP duplicates the KV cache.
Inference is even more counterintuitive. V3 has 128 question heads, one shared, compressed note head.
TP splits by head. The questions split; the single note head can't, so every device stores an identical copy. With 8 devices, the 7 red copies are waste.
Last section's V4-Pro on GB300 also has one note head: TP4 wasted 3 copies, one of the reasons.
The general rule: TP splits by head, so the notes go to at most as many devices as they have heads.
Many models, like Llama 2 70B, give the notes 8 heads: 8-way TP, one per device, no waste. Once TP exceeds the note heads, each head is copied “TP divided by heads” times.
MLA compresses the notes to one head, the ugliest case: TP N ways, N copies.
🖥 Screen: the DCP animation. Stay quiet through the first scene (TP only); speak once it switches to DCP (loop 11 s).
DCP fixes it: notes stored round-robin by token across the same devices, none added.
The new token's question goes to every device; each searches its share; then merge. Three extra communications per layer: gather questions, exchange denominators, combine results. The duplicated memory comes back. Communication for memory, again.
Two details. Why round-robin by token, not chunk? Notes grow by one per new token; round-robin spreads them evenly on its own.
How many ways for DCP? At most TP divided by note heads, exactly reclaiming the copies; beyond that, nothing's left.
🖥 Screen: Merging blockwise attention, don't divide yet. Middle (wrong), then right (right).
The merge has a trap. Attention is a score-weighted average, divided by the sum of all scores, but each device sees only its own notes.
Divide on each device, then average: 14. Wrong: device 1 has one token but half the vote. Right: don't divide yet. Add numerators, add denominators, divide once at the end: 11.
Like merging two classes' averages: you need the class sizes. So each device also hands over its denominator: the middle communication. Ring merges so en route, DCP at the end: same trick.
Inference has two phases: prefill reads the whole prompt in one go; decode outputs one token at a time. DCP is for decode.
Prefill's counterpart is PCP: devices read one long prompt together, so the first token comes sooner.
One letter apart, they're consecutive steps. As a table: rows are questions, columns notes. Prefill has thousands of questions, so it splits by row: four thousand rows, a thousand per device. Each row finishes on one device, given all earlier notes. PCP is for speed: it adds devices.
Decode has one question per step, no rows to split. Only now does DCP go further: split that row by notes, a piece per device, merged don't-divide-yet. DCP is for fitting: it uses the devices you have.
Left open: prefill, reading a long prompt, wants the sequence split; decode wants DCP to spread notes. One device set holds one split; cram both on, neither fits well.
⚠️ Ulysses: conclusions only (communication doesn't grow with scale; parallelism ≤ head count). No formulas.
⚠️ The Ring Attention animation shows no causal mask. Don't claim it balances load by itself.
The 2026-09-26 live session found Sections 5 and 6 skimmed; the page's derivations (where 8.58 GiB comes from, Ulysses' cap, TP ÷ heads, the DCP degree, PCP) are now scripted.
🖥 Main screen for this segment: course page Section 5 · Fourth cut: sequence · main figure Inference: TP duplicates the KV cache
Section 6 · Fifth cut: split the work, not the tensors (11′)
🎯 Leave one line: put two different kinds of work on two machine sets, and each side picks its own sharding.
🖥 Screen: scroll to the Section 6 heading.
Two machine sets shipping data back and forth: more devices? The opposite: 3 devices do the work of 7.
Section 5 left us here: prefill and decode want different sharding, even different resources.
Four cuts split tensors. The fifth: one request already holds two very different jobs, so put them on two sets of machines.
Two phases, remember. Prefill reads the whole prompt at once, thousands of tokens; math can't keep up. Compute-bound.
Decode emits one token at a time, yet each step reads weights and notes from HBM; reading can't keep up. Bandwidth-bound.
💬 Interaction: “Chatbots: who hates waiting for the first word more? Hands up. Or words trickling out?” Hands only; no cold-calling.
Both, and together they infect each other. Your answer streams; someone drops in a hundred-page manual; your words stop.
Serve the newcomer, and everyone mid-answer stalls; serve them, the newcomer waits.
Names: the wait for the first token is TTFT; the gap between tokens, TPOT. Voice assistants show it best: however fast on average, stop-start speech feels sluggish.
On shared devices, the engine picks who goes first each step. A long prefill takes several steps; everyone mid-generation waits. TTFT and TPOT both worsen.
🖥 Screen: Same 16 steps animation. Stay quiet first as the top lane gets cut (11 s loop). Skip the static Cut 5 · two jobs on two sets of machines.
A kitchen: some cooks prep, some plate. On one station, a big order's prep takes minutes, and every table waits. Top lane: the long orange block drops in; those cells emit nothing.
Split: prep works elsewhere and hands over the ingredients; plating never stops. The ingredients: Section 5's notes, the KV cache.
The bottom lane uses extra machines. What matters: do tokens keep coming?
Someone will ask: why not chop the big order and interleave it with plating? That's chunked prefill; it helps.
Two problems. Small chunks slow prep: each rereads all earlier notes. Worse, both crews share one station, so one sharding.
PD disaggregation splits them: prefill machines prefill and hand over the notes; decode machines only generate.
🖥 Screen: One KV transfer is just 5–10% of a prefill. Gray bar on top, then the thin three-color bar below.
How long is the handoff? We built one on TPU v7x: one prefill machine, one decode, running Qwen3-Coder-480B. 8K-token prompt: by bandwidth, about a hundred milliseconds, 5 to 10 percent of the prefill. This model's notes are small; redo it for others.
It can take the slow link: each request transfers once. Hold that for the next section.
Why a hundred? Mostly the middle leg, between machines. Notes for 8K tokens: K and V, 62 layers, 8 KV heads, 128 numbers per head, 1 byte each: about 1.04 GB.
Conservatively, 100 Gb/s per chip between machines: 12.5 GB/s, a bit over eighty milliseconds. About ten more at each end, HBM to host memory. Roughly a hundred.
From here on, scheduling centers on the notes: they follow the request. Kimi's Mooncake calls its whole system “KVCache-centric.”
Flip side: short requests, light traffic? The transfer and extra machines may not pay; stay colocated with chunked prefill.
🖥 Screen: Once split, 3 devices do the work of 7 colocated. Top row of seven gray devices, then the bottom three.
Does splitting save anything? One paper's math is clean. Doing both, a device handles 1.6 requests a second; prefill only, 5.6; decode only, 10.
Two on prefill, one on decode: prefill a bit over 11, decode 10, so three devices do 10. Colocated, 10 takes 7. A second machine kind saves over half the devices.
Keep both sides equally busy. Rule of thumb: long prompts, two prefill per decode; long answers, one to two.
Searching the ratio with each side's sharding, the paper serves up to 7.4 times more requests than DeepSpeed-MII at the same latency targets. Three-for-seven is one single-device example.
Better, each side shards its own way. Prefill uses Section 5's PCP: thousands of query rows split across devices for an earlier first token, paying in extra devices. Decode uses DCP: one query per request, so split the notes further. More fits; each device reads less.
🖥 Screen: scroll back up to Why decode insists on big batches (Section 6's first figure, saved for here). Red line, blue dot, orange dot.
First, decode's temperament. Each step reads the weights from HBM, and each read serves the batch once. So batch size is your FLOPs per byte.
Section 3's craftsman and runner again; now the runner fetches tools from the warehouse, HBM. One trip per token, used once per door panel in the batch. v7 does over 2,300 trillion operations a second and reads over 7 trillion bytes a second from HBM: a bit over 310. Under about 300 requests a batch, the craftsman waits.
MoE is harder. Each expert gets a thirty-second of the batch: V3 picks 8 of 256. Feeding every expert takes a batch of about ten thousand.
ByteDance's MegaScale-Infer did this math on A100: at least 156 requests a batch, just 39 per Mixtral expert. The model helps: MLA shrinks the notes, so a device holds more requests and batches grow.
No device has ten thousand requests. Many devices must send tokens to the same experts: spread experts wider, more requests feed them, each expert's batch grows. DEP, Wide-EP, and AFD, next, all assemble that batch.
🖥 Screen: Split attention from experts too. Links on the left, then the three lanes on the right.
Decode splits one level further. Attention is tied to requests; experts just need enough tokens. Separate machines: that's AFD. Experts take tokens from several attention groups at once: a big batch.
The cost: a round trip every layer, so the batch splits into three groups taking turns. The batch, not the sequence: decode has one token per request per step; three thousand requests make three groups of a thousand. Follow A: a cell of attention, half a cell out, a cell of experts, half back; in cell 3 it's A's turn again. Both sides stay busy.
Why three? MegaScale-Infer showed hiding the round trip takes at least three, four if communication is slow. It reports up to 1.9 times the per-GPU throughput. ByteDance's MegaScale-Infer and StepFun's Step-3 both work this way.
The same idea reaches one stage earlier: deploy a multimodal model's vision encoder separately, shipping its embeddings to the language model; with PD, often written EPD.
So the fifth cut says: don't cram different kinds of work together. Split them, and each side picks its own sharding.
What this leaves open: all five cuts are done, but on a real cluster they coexist.
Which stay inside a machine, which cross? Misplace them, and the network eats back all each cut saved.
⚠️ For the KV trip, work only the middle leg (1.04 GB ÷ 12.5 GB/s); give each end's “about ten milliseconds” in passing.
⚠️ Animation cell counts are schematic: don't present “11 tokens in 16 steps” as measured.
🖥 Main screen for this segment: course page Section 6 · Fifth cut: split the work, not the tensors · main figure: the Same 16 steps animation
Section 7 · Onto the machine (7′)
🎯 Leave one line: chatty traffic on fast links, only quiet traffic on slow ones. Right sharding first, then tune.
🖥 Screen: course page Section 7; hold on the heading.
💬 Interaction: hold off on the “How often each cut talks” figure (its title is the answer). Ask: “The cluster is an office building. TP: co-writing a contract clause by clause. PP: passing drafts chapter by chapter. DP: another team squaring accounts once at day's end. Who must share a desk?” Note the split for later.
🖥 Screen: How often each cut talks, whole figure; bottom half first.
Same devices, same sharding, different placement: 17 cells vs 73.
Links differ. Bottom half: a GB300 rack joins 72 GPUs in one NVLink domain, 1.8 TB/s per GPU, send plus receive. Off the rack, the NIC: 200 GB/s per GPU, nine times slower. TPU is more lopsided: ICI inside a slice, data-center network across, about fifty times slower on the same basis.
Top half: cuts differ in chattiness too. Talks per step, illustrative config: TP, EP and FSDP talk every layer, over a thousand times a step. PP, a dozen or so. DP, once.
🖥 Screen: A cluster is an office building. The three rings, inside out.
Your vote? Co-writing: one desk; drafts: a few floors; accounts: another city. Chatty ones sit inside; quiet ones go outward.
Counting isn't enough: can traffic hide behind compute? TP can't: the next block waits for the sum ending this one, so TP stays on the fast lane.
EP and FSDP can send early and hide some, so some setups move them out. V3's EP spans 8 machines: on H800, inside vs. outside a machine differ only three to four times, far below TPU's gap. Llama 3 puts FSDP outermost.
DP's one transfer is big but easy: backward runs last layer first, so gradients go out as they're ready.
🖥 Screen: the animation in 7.2, step by step. Placement 1, round 1: same-color arrows; the ring stays in the machine. Placement 2, round 1: red arrows; two hops cross the slow link, one looping back outside.
8 devices, TP4 × DP2; each color is a TP group.
Placement 1: each TP group in one machine, eight fast rounds; only the final DP crosses the slow link. Placement 2: each group spans two machines; each round waits for its slowest hop, so all are slow.
The math. Placement 1: 8 fast cells, plus 9 for the slow DP: 17. Placement 2: eight slow rounds, 8 × 9, 72; the DP pair shares a machine, 1 cell: 73.
17 vs 73: over four times slower. The one-to-nine ratio is schematic, not measured.
🔍 Going deeper: how placement is written on TPU: GSPMD vs. plain SPMD (if pressed or time allows; not in the floor time)
1. Plain SPMD: you write each device's program
SPMD, a programming model: single program, multiple data. Every device runs the same program on its own data. Plain style: you hand-write each device's piece, when to talk, which AllReduce to call. Megatron's TP was written that way.
2. GSPMD: you write one big device's program; the compiler splits it
GSPMD: a 2021 Google paper and an XLA compiler feature. Write for one big device; annotate a few key tensors: shard this dimension along that mesh axis. The compiler propagates the annotations, picks each step's sharding, inserts collectives like AllGather and ReduceScatter, and emits one program for every device.
Two payoffs. Generality: data, tensor, FSDP and expert parallelism are just different annotations; model code stays put. Section 3's two-by-four FSDP-by-TP grid is annotated so on TPU. Scale: any device count, one compiled program, so compile time doesn't grow.
3. How it ties to this section
On TPU, this section's placement lives here: lay the chips out as a multi-dimensional mesh, then pick which axis is TP, which FSDP. Wrong axis: placement 2. JAX's jit-plus-sharding and MaxText sit on it. Its predecessor GShard trained a 600-billion-parameter MoE translation model in 2020.
In short: plain SPMD, you write each device's program; GSPMD, one big device's, split by the compiler.
Section 3's yardstick again: each link has its own hardware line. A link nine times slower, a threshold nine times higher. At V3's hidden width, 8-way TP sits right on TPU's fast-lane line; on a link ten times slower, far below.
TP rarely leaves an 8-GPU server. GB300's fast lane spans a rack, so EP goes wide.
🖥 Screen: Resharding beats retuning by a mile.
Wrong sharding: tuning can't help. The experts cut's promised GB300 numbers, per GPU.
Original script: 1,820 per GPU. On TP4 we tuned everything: prefill from one machine to three, total from about 14,500 to 21,100, up 45 percent. But GPUs doubled: per GPU, 1,319. TPOT held around 50 milliseconds.
The last row: decode on DEP8, 4 GPUs to 8. Same concurrency: 2,758 per GPU, over double. TPOT 46.8 to 11.8 milliseconds; TTFT 55.8 to 22.8 seconds. The original record doesn't attribute the TPOT drop.
The gain? As the sequence cut showed, TP4 copies the notes onto all four devices. DEP8 ends that, spreading experts over 8 GPUs; freed HBM means a bigger batch.
8 GPUs: two machines, one rack's fast lane. Sharding first, parameters second.
🖥 Screen: scroll to the table in 7.4.
Choosing parallelism is five gates. Does it fit? How big is the fast tier? Keep TP and EP in it. Long sequence? Still short, or forced across a slow link? Only then PP. Devices left? All to data parallelism.
In the table, only the two “not used” entries. V3, on 8-GPU H800 servers: no TP; its EP spans 8 machines.
Tencent's Hunyuan 3 on TPU: all 256 chips in one slice, all fast links; FSDP already spreads the experts. A small EP trial, 4-way on 16 chips, lost seventy percent of throughput, but the batch was halved too, so the cause isn't pinned down. So: no EP; FSDP shards experts.
Same five steps, little overlap: the fast links differ in shape. Don't copy others' configs; look at your own links first.
That's everything: what each cut splits, what talking it adds, which link. Next, one figure holds them all, and TEP8-style shorthand makes sense.
Adding devices on stage (cut after the 2026-09-25 run-through: ~1 min, off the relay; self-study). If asked: grow the batch with the devices; extra devices go to replicas first.⚠️ Don't follow Hunyuan with “GPU is one hop; the TPU torus takes detours”: the page says that cost is small at 4-way; paired, it sounds causal.
⚠️ The GB300 setup's KV rides in-rack NVLink; don't call it “the slow link.” That's the TPU v7x PD setup.
⛔ Don't read the 7.4 table row by row.
Just the two “not used” cells: Hunyuan 3 on TPU no EP, V3 on H800 no TP (8 GPUs per server); together they make the point.
⚠️ 17 vs 73 cells is schematic, not measured.
🖥 Main screen for this segment: course page Section 7 · Onto the machine · main figure How often each cut talks
Section 8 · Panorama and wrap-up (4′)
🎯 Leave only one line: for any name, first ask what it splits, what communication it adds, which link it goes on.
🖥 Screen: course page Section 8, the panorama in 8.1, Four families, narration folded (button reads “Show narration”; don't press). The segment's only screen change; don't scroll to the tables.
Last, the panorama. Four columns: data, sequence, model, disaggregation.
Five cuts, four columns: weights and experts are both model parallelism, sharing one.
One criterion: what does it split? Data: each device computes different samples. Sequence: each holds a chunk of a sample. Model: each holds part of the model. Work: each does one kind of job.
Two misfilings. FSDP is data parallelism, not model: weights are stored sharded, but each layer reassembles the whole reading room first, and each device still computes its own samples. It saves memory; who computes what stays.
Sequence parallelism is its own family. Strictly it splits data too, within a sample, but it exists for long context and talks nothing like DP. Lump them, both blur.
How to read it? Try EP. Column? Model, the third cut: splitting experts.
What talking does it add? AllToAll, the last one in Section 1. Which link? The dot at top right: filled can take slow links; hollow stays on the fast lane. EP is hollow.
Ask every name these three.
💬 Interaction: “Now you read PP.” Wait for “model, point-to-point send/receive, filled: it can take slow links.”
Under the figure, inference shorthand: TEP8, DEP16. From the experts cut: the leading letter is how attention splits, EP the cell we just read, the number total devices.
Under PD each side often picks one, no fixed rule: vLLM deploys Kimi K3 with TEP8 for prefill, DEP16 for decode; TensorRT-LLM's example does the reverse.
The tables below are an after-class map; just three traps.
One: DP means something else in MoE inference. In vLLM or SGLang it's Attention DP: experts still run jointly, every step syncs, idle devices run dummy forwards. DP without EP leaves experts on TP.
Two: TEP and DEP aren't ETP and EDP. The first pair, inference-side: how attention splits. The second, Megatron training-side: TP degree inside an expert, which devices sum expert gradients. Megatron's repo has zero hits for TEP or DEP. Don't convert.
Three: not parallelism at all: TBO, Async TP, EPLB, offload, chunked prefill add no dimension; they hide communication, balance load, trade memory or schedule.
Back to the opening relay: each cut patches what earlier ones missed, paying one kind of communication to buy back memory or compute.
The opening V3 config again: PP 16, EP 64, ZeRO-1 data parallel. Cut deep, cut experts, cut data. Stage-boundary sends, AllToAll, once a step. Crosses machines, capped at 4, cheapest of all. That gibberish now reads plainly.
So choosing parallelism comes down to: which communication you'll pay for, how often, and on which link.
Chips change, numbers age. This line won't.
⛔ Don't walk the panorama tables row by row (2026-09-26 live run: name the three traps, skip the rest).
Demonstrate reading one entry, once.
⚠️ After the closing line, stop. Nothing beyond “Any questions?”
🖥 Main screen for this segment: course page Section 8 · Panorama: every parallelism in use today · main figure Four families
🚧 Where these notes are still thin
- [ ] Never delivered; timings are estimates. Revise after the first real run.
- [ ] No pre- or post-class questions. Section 7's end suits one: “On a GB300-style rack where all 72 GPUs share one fast link, which cut in V3's config is most worth changing?”