How to use these notes
Two uses.
① Presenting: read it as written: lines, screen cues, interactions, minutes per segment. 🧭 opens each section: where to bring everyone first; 🖥: which figure, which number to point at; yellow lines: what you say aloud; 💬: what to ask, which answer to wait for; ⚠️ at the end: answers only if asked.
② Self-study: hand the whole file to an AI to walk you through as a TA.
The content lives on the course page; these notes cover only how to teach it. By default it shows only headings, takeaways, figures and tables (the “Show text” switch, bottom right, expands the text); project it that way, so the audience watches the figures and listens to you. Course author: Chris Yang · Google Cloud AI Infra architect.
🧵 Before you go on stage
Three hidden threads: ① Estimate first, then look: measure good news with a ruler, work out what to look for, then open the profile. ② One question per page: Overview Page, was it idle? Trace Viewer, which part? The books, is it work never done? HLO, what should be there? The code, why? ③ A real speedup does the same work faster; a fake one does less work.
⭐ If they take away one line: Too good to be true? Look at the profile before you celebrate.
⏱ Time: planned 82′
⛔ Never cut Sections 0 and 6. Section 0 plants “estimate good news, then look”; 6 closes the five steps; each middle one opens on the last closing question. Squeeze the middle.
No time limit: teach what's needed. Each section's minutes are counted from these notes: the script at 140 words a minute, 15 s per screen switch, half a minute per interaction (about 71 minutes with interaction; the floor table below rules),
plus the full played time of the recording and animation stops these notes cite (about 4.9 minutes: Section 2, 2.0 recording stops 3–7 and 2.5 animation stops 1–5, about 1.0; Section 3, 3.2 Roofline recording stops 2–3 and 3.6 memory recording stops 1–11, about 2.7; Section 4, 4.3 Graph Viewer recording stops 1, 2, 9–11 and 4.4 animation stops 1–7, about 1.2; Section 5, 3.6 memory recording stop 12, under 0.1),
for a total of about 76 minutes (each section's minutes = with interaction + its stops, rounded). Follow-ups and live XProf excluded; for an hour, squeeze the middle per “Cuttable?”.
Clips: Section 2, 2.0 recording (stops 3–9: tidy, read rows, open a layer), optional basics recording, 2.5 animation (1–5). Section 3, 3.0 recording only without live XProf; 3.2 Roofline recording (2, 3); 3.6 memory recording (1–11 in four parts, 12 for Section 5; live: click for real). Section 4, 4.3 Graph Viewer recording (pick stops), 4.4 animation (point as it plays).
| Segment | Min | Cuttable? | The one thread |
|---|---|---|---|
| 0 One step got 50% faster. Do you believe it? | 4 | Can't cut | Two step times, the diff; symptom only, estimate first |
| 1 Estimate first: is this number even possible? | 10 | Down to 9 (conventions, “Compute it yourself”: point only; all 3 asks: no wait) | Two rulers, MFU and roofline; an MoE got a dense model's gain: suspicious |
| 2 Open the profile: where did the time go? | 17 | Down to 15 (no clips, static figures; one sentence each: other Overview tiles, down a column, four leaf kinds) | Overview: TensorCore never idle → one step, one layer; down a column; four leaf kinds → layer side by side: SparseCore lacks two kinds of communication |
| 3 Reconcile the books: is the missing time work that was never done? | 21 | Down to 17 (no Roofline clip; memory clip stops 1, 5, 10, 11 only; 375 s aside: two sentences, no bet) | custom-call 117.45% → one kernel call at 25× peak → Roofline's step average inflated too → counts unchanged → reconcile both ends → memory |
| 4 Down to HLO and the code: why wasn't it done? | 14 | Down to 11 (Graph Viewer stops 1, 2, 11 only; animation nonstop; bet: ask, don't wait) | Count collectives by shape: 12 / 6 → 0; Graph Viewer; op_name points to the tokamax branch; zero-fill, no error |
| 5 A control: what does a real speedup look like? | 9 | Already at the floor (7.2′); only option: ask both interactions, don't wait | BF16, same change: no collective missing, every kernel call shorter |
| 6 Wrap-up: how to investigate next time | 7 | Can't cut (floor 5.8′; just point at the sources figure: saves half a minute) | Five-step order, three checks before celebrating, capture your own profile |
| Total | 82 |
⛔ Above is the plan; below is the floor (script at 140 words/min + 15 s per screen switch + half a minute per interaction; recounted at every build).
| Section | Script (words) | Screens | Floor | Planned | Interactions / skippable | With interaction | After skips |
|---|---|---|---|---|---|---|---|
| Section 0 · One step got 50% faster. Do you believe it? | 340 | 4 | 3.4′ | 4′ | 1 / 0 | 3.9′ | 3.9′ |
| Section 1 · Estimate first: is this number even possible? | 744 | 8 | 7.3′ | 10′ | 3 / 0 | 8.8′ | 8.8′ |
| Section 2 · Open the profile: where did the time go? | 1,204 | 15 | 12.3′ | 17′ | 3 / 0 | 13.8′ | 13.8′ |
| Section 3 · Reconcile the books: is the missing time work that was never done? | 1,595 | 17 | 15.6′ | 21′ | 4 / 0 | 17.6′ | 17.6′ |
| Section 4 · Down to HLO and the code: why wasn't it done? | 1,113 | 11 | 10.7′ | 14′ | 3 / 0 | 12.2′ | 12.2′ |
| Section 5 · A control: what does a real speedup look like? | 721 | 8 | 7.2′ | 9′ | 2 / 0 | 8.2′ | 8.2′ |
| Section 6 · Wrap-up: how to investigate next time | 527 | 8 | 5.8′ | 7′ | 1 / 0 | 6.3′ | 6.3′ |
| Total | 62′ | 82′ | 17 / 0 | 71′ | 71′ | ||
Section 0 · One step got 50% faster. Do you believe it? (4′)Planned 4′ · with interaction 3.9′
🖥 Main screen for this segment: course page Section 0 · One step got 50% faster. Do you believe it?
🎯 What this segment must leave behind: good news? Don't celebrate it or doubt it yet. Spend a minute estimating: is this number possible?
🧭 Get everyone there first: “We're in Section 0 of the course page, ‘One step got 50% faster. Do you believe it?’” Switch there, let everyone find it, then begin.
🖥 Screen: switch to 0.1 Two step times. Point at the two step times, then the 1.50 line.
A story from our own work. We were training Tencent Hunyuan3, Hy3 for short. One day we changed a set of parameters and reran. A step dropped from 11.76 seconds to 7.84. We'll call the old config the original recipe. 11.76 divided by 7.84 is 1.5. Same time, half again as many tokens.
Just a lucky run? We ran the same config twice more: 7.846 and 7.840. Under a tenth of a percent apart: not noise.
💬 Interaction: ask the audience: do you believe this number? Hands up three times: believe, don't, not sure; then one reason per group. Once you hear “new parameters, of course it's faster” and “half again at once is too much,” move on. Don't judge yet.
🖥 Screen: still on 0.1. Point at the “Setup” table below the step times, rows top to bottom.
First, what this ran on. Hy3 is an MoE model with 295 billion parameters and 80 layers. Every layer except the first has 192 experts. An expert is a feed-forward network; each token picks only 8.
The machine is 64 TPU v7 chips. Each chip has two TensorCores, which the profile counts as two devices, so 128 in all. The weights don't fit on one device. So they're split into 64 shards, one per device, and gathered back from the others when needed. That's FSDP; it comes up throughout.
We train in FP8, one byte per number, on open-source MaxText.
🖥 Screen: switch to 0.2 What changed; point at its three rows, top to bottom.
So what changed? We diffed both configs line by line: only 13 lines differ. Twelve are expert-matmul tile sizes. A kernel is the code the chip actually runs. A tile is how big a block it loads into on-chip memory to compute at once. Going by the config, they went from 1024 to 2048 and 1536.
The last line, use_tokamax_gmm, goes from True to False: expert matmuls drop the tokamax library's kernel for MaxText's own.
A new implementation, bigger tiles, faster matmuls. Completely plausible, and at the time we saw nothing wrong either. But we weren't sure. Really faster, or had something gone wrong? I couldn't tell. When you're not sure, don't rush to open the profile, the run's record of every op's start and end. First, spend a minute estimating.
🖥 Screen: stop on the ⭐ line in 0.3 When in doubt, estimate first.
Estimate what? Just one question: for this machine and model type, is a 50% faster FP8 training step normal? With that range, you know what to look for in the profile.
- Asked “Is the 50% about throughput or step time?”: throughput. 11,758.2 ÷ 7,842.1 = 1.499; the step is one third shorter (0.667). 0.1 collapsible block.
- Asked “Do profile step times match the logs?”: profile median steps, 11,758.2 / 7,842.1 ms; MaxText's host timer: 11.765 / 7.853 s, including dispatch, so about 0.1% longer. Reruns: 7,846.1 / 7,840.1 ms. All in the 0.1 collapsible block.
- Asked “Is tile 1024 what the original recipe really used?”: no. On tokamax the launch script injects the original's tile separately; the compiled program says (512, 2048, 1536). So “1024 → 2048” doesn't mean the kernel's real tiling changed (0.2 collapsible block).
- Asked “Why 64 shards but 128 devices?”: besides 64-way FSDP there's a data axis of 2; 64 × 2 = 128, so each shard sits on two devices (Section 3, memory).
- Asked “How was 295B checked?”: the tencent/Hy3 model card on Hugging Face lists Total 295B and the MTP layer's 3.8B on two separate rows (its layer row says 80 layers, excluding the MTP layer); Section 3 back-calculates 298.85B from memory, exactly the two added together.
- Terms like gmm, tile, FSDP, all-gather: the collapsible block “Unfamiliar term?” ending 0.3 has ten. Explain each as it comes up; don't read the list.
- ⛔ In this section, don't judge or hint at the conclusion. “Believe it or not” only gets the audience to place a bet and give a reason; Section 1's rulers decide.
Section 1 · Estimate first: is this number even possible? (10′)Planned 10′ · with interaction 8.8′
🖥 Main screen for this segment: course page Section 1 · Estimate first: is this number even possible? · main figure Same switch to FP8…
🎯 What this segment must leave behind: DeepSeek-V3 (MoE) gains only about 20% from FP8; the modified run got a dense model's gain. Suspicious: find the 3.9 s in the profile.
🧭 Get everyone there first: “We're in Section 1 of the course page, ‘Estimate first: is this number even possible?’” Switch there, let everyone find it, then begin.
Estimates take a ruler. Ruler one is MFU: a chip's real floating-point operations per second, divided by its peak. TFLOP/s is trillions per second. A v7 chip's BF16 peak is 2,307 TFLOP/s. BF16 is two bytes a number, the most common in training.
For FP8 we divide by the same number. FLOPs come from the model's shapes: one multiply-add is two operations, however many bytes a number takes. The denominator doesn't change either. So both precisions sit on one ruler.
🖥 Screen: switch to the table in 1.1 Ruler one: MFU. Point at the two public MoE rows, Hy3 BF16 and the original, the red row, then the 405B at the bottom.
Public MoEs Qwen3 and DeepSeek-V3 both sit near 27%. Hy3 BF16 is 28.7, the FP8 original 29.3: same cluster. The modified row leaps to 44, just shy of dense Llama 405B.
Where does it come from? MaxText logs each step's FLOPs: 3,981.54 TFLOP per device. Two devices per chip: times 2, then divide by step time. Both FP8 runs log identical FLOPs, so the whole MFU gain is the shorter step. MFU alone says the number jumped, not whether it should have.
🖥 Screen: open the collapsible block “MFU denominators” under 1.1; point at 4,614.
A common trap here: FP8 peak is twice BF16's, 4,614 per chip. Use that as the denominator, and the modified run's MFU drops to 22%, artificially halved. Same run, new denominator, new MFU. So for FP8 MFU, always name the peak you divided by.
🖥 Screen: open the collapsible block “Compute it yourself” under 1.1. Of its eight code lines, point at the step-time and FLOPs ones, then 676.2 and 1,013.7.
Don't take our word for this ruler; measure it yourself. MaxText logs step times and FLOPs as TensorBoard events. Eight lines of Python read them, take the median step, double and divide. Our two runs: 676.2 and 1,013.7, 0.1% off the table, only from picking different steps.
🖥 Screen: switch to the “Five conventions” table at the end of 1.1. Skim only the “Pitfall” column of rows 1–4; row 5, HBM, isn't used until Section 3: skip it.
We use these conventions throughout; we hit or nearly hit each pitfall. Step time is a median, to dodge the step slowed by writing the profile. FLOPs come from the shapes, FP8 included. The denominator: always 2,307. One-line sanity check: MFU above 100% means a convention is wrong.
💬 Interaction: ask the audience: FP8 makes a number one byte, not two. At most how much faster is one matmul? A whole step? Accept “2×” or “the step too”; don't correct. Move on at the first answer; the roofline settles it.
🖥 Screen: switch to the 1.2 figure “FP8: at most 2×, and only on matmuls”. Left only: slope, flat roof, ridge point; then the example point, blue roof to orange.
Ruler two, the roofline, gives an op's top speed. The x-axis is arithmetic intensity, operations per byte moved from HBM. On the left slope, HBM bandwidth sets the speed; on the flat roof to the right, peak compute does. They meet at the ridge point.
Switch to FP8, and both ends double. A matmul on the flat roof gets a roof twice as high. A matmul on the slope moves half the bytes, so its point shifts 2 times right. Up the slope, that's exactly 2 times faster too. So one matmul gets at most 2 times faster.
But a step isn't just matmuls. Routing, permuting tokens by expert, elementwise ops, communication: none are on this chart, and FP8 doesn't help them. So, that question: one matmul tops out at 2 times, so a whole step can't reach 2 times.
🖥 Screen: same figure. Point at the right-hand curve and the red dot above it. Skip the takeaway band below for now: it answers a later interaction.
The right curve is our own derivation. Say matmuls are a share p of the step. At best, FP8 halves that share. So a step is at most 1 divided by (1 minus p over 2) times faster: 2 times only at p equals 100%.
Note the baseline: Hy3's own BF16, not Section 0's original, since others all report BF16-to-FP8 gains. The modified run is 1.53 times Hy3 BF16. Working back, at least 70% of the step must be matmuls.
The same formula on others gives a reference. DeepSeek-V3 with FP8 is 1.214 times its BF16, so matmuls are at least 35% of its step. The dense 405B is about 1.53 times: 70% again. A dense layer is just a few big matmuls, so 70% is believable.
💬 Interaction: ask the audience: every layer of an MoE also routes, permutes tokens and gathers expert weights from other devices. Can matmuls be 70% of its step? On “no” or “not sure,” switch.
🖥 Screen: switch to the 1.3 figure “Same switch to FP8…”. Point at dense +52.8% and DeepSeek-V3 +21.4% up top; Hy3 Original +2.2%; the red row, +53.2%.
Now peers: same chip, same FP8 switch. The dense 405B gains 52.8%. DeepSeek-V3, an MoE, only 21.4%. Hy3's own original: only 2.2% over its BF16. The modified run: 53.2%, like the dense model.
The three Hy3 points are our own measurements on 64 chips; the other rows cite published ones. The ruler isn't precise, but it measures one thing: what a change like this usually gains on similar models.
💬 Interaction: ask the audience: recall Section 0's show of hands? Bet again: an MoE got a dense model's gain. Is the change that good, or was something missed? Mind changed? Hands up. No reveal; carry the bets on.
Either this change cured all of MoE's old problems in one go, or something's wrong. The estimate can't say which. But now we know what to look for in the profile.
🖥 Screen: switch to the two-column table in 1.4 Open the profile with questions in hand.
Back to Section 0's step times: the missing 3.9 seconds must come off some ops. If it's real, the same ops are all there, just shorter, mostly expert matmuls. Communication is unchanged. If not, a whole class of ops is gone or absurdly short, and communication shrinks too.
This table isn't a conclusion; it's a checklist. Left column matches, we celebrate. Right column matches, we dig. Take it with you, and open the profile.
- Asked “Which MFU denominator is right?”: 2,307 is the per-chip BF16 peak (MaxText and this course use it). XProf's Roofline Model page gives 1,028.75 per TensorCore; × 2 = 2,057.5, 10.8% smaller. Dividing by the FP8 peak, 4,614, artificially halves MFU (1,014.8 ÷ 4,614 = 22.0%). Always state FP8 MFU's denominator (1.1 collapsible block).
- Asked “How do you get p ≥ 70%?”: new step time = p/2 + (1 − p); a measured speedup of s gives p ≥ 2 × (1 − 1/s). DeepSeek-V3: s = 1.214 → 35%; 405B and Hy3 modified: s ≈ 1.53 → 69%. It's our derivation's upper bound, not a measurement (1.2 collapsible block).
- Asked “The original gathers expert weights in FP8, halving those bytes: FP8 touches more than matmuls?”: yes, but the original's exposed communication is only about 0.2 s a step (195.9 ms; whole-step accounting, 2.5 collapsible block “Reading start/done…”). Counting it leaves p's lower bound unchanged (1.2 collapsible block).
- Asked “Why is it 1.53× here and 1.50× in Section 0?”: different baselines. 1.53 = modified 1,014.8 ÷ Hy3 BF16 662.3 (“BF16 to FP8”, like the public table); 1.50 = original ÷ modified step time. BF16 and FP8 batches differ (12 vs 7), so compare TFLOP/s per chip, not step times.
- The public measurements (Qwen3, DeepSeek-V3, 405B) are cited from the Hy3 tuning log, not measured by this course; if pressed on the source, say so.
- ⛔ No “it's fake” verdict here; say only “suspicious; we know what to look for.”
Section 2 · Open the profile: where did the time go? (17′)Planned 17′ · with interaction 13.8′
🖥 Main screen for this segment: course page Section 2 · Open the profile: where did the time go? · main figure Figure 2.4A
🎯 What this segment must leave behind: neither TensorCore idles. At the same layer, the modified SparseCore row lacks two kinds of communication and expert matmuls shrink to 1/75–1/93: a big chunk of work is gone.
🧭 Get everyone there first: “We're in Section 2 of the course page, ‘Open the profile: where did the time go?’ Start at 2.1, Overview.” Switch there, let everyone find it, then begin.
Take Section 1's table and open the profile. The tool is XProf: open source, pip install, runs locally. Its first page, the Overview Page, is like a checkup's first page. Just one question for it: did the missing 3.9 seconds come from an idle TensorCore, or from different work? The TensorCore is the chip's matmul unit.
🖥 Screen: switch to Figure 2.1, Overview; only the left column: ① Average Step Time, ② TPU Duty Cycle. Hold back key row ②: it answers the interaction.
① Step time: 11,793.8 milliseconds to 7,857.0, ratio still 1.5. ② Duty Cycle, the share of time the TensorCore has work, is 99.9% in both: 11 or 12 milliseconds idle per step on average; step by step, 5 to 15, similar in both. The missing 3,900 milliseconds are unrelated to idle time.
💬 Interaction: ask the audience: does Duty Cycle 99.9% mean the TensorCore works flat out all the time? Wait for “Not necessarily: stalling for data counts as work too,” then move on.
Right. Stalled waiting for data still counts as busy. So the Overview says just one thing: neither TensorCore idles. The missing time is in their work.
🖥 Screen: switch to Figure 2.1B; sweep ① to ⑦, stop only on the “0.0” pairs ② and ④.
Five more cells on the left; newcomers always wonder. Scan both runs. FLOPS and HBM read 0.0: this open-source version doesn't compute them; they aren't really zero. Op placement and eager execution match exactly, as JAX training always does. Only step time and Duty Cycle differ.
The chip also has SparseCores, the TensorCore's data movers. Is the SparseCore busy? No cell says. What the TensorCore did and the SparseCore moved shows only on the timeline. Next page: Trace Viewer.
🖥 Screen: switch to the 2.0 recording; untick “Autoplay”; at stops 3–7 read each caption's action. If you can open XProf live, skip the recording and click these stops for real (2.1, “Opening it locally”).
It opens the full session: 8 TensorCores, two SparseCores each, dozens of rows. A mess. Collapse All: one row per process. Keep only TPU:0 and its two SparseCores; close the rest with the X at right; their rows are identical. Then Expand All.
On the XLA Modules row, click train 7's jit_train_step, the median of three captured steps. Press f: full screen. Gathering or summing across devices is a collective. Here all run on SparseCore rows. The TensorCore issues one and keeps computing while the SparseCore moves it.
🖥 Screen: for Trace Viewer newcomers, open the collapsible block “Basics: W/S/A/D, …” at the end of 2.2 Trace Viewer and play the basics recording; pause at stops 2 (W zooms in), 4 (click, Details), 5 (m marks a width), 7 (Ctrl multi-select, Totals). Narrate: W/S zoom around the mouse, A/D pan; click for Details, f fills, m marks a width; Ctrl-click for Totals; Find Events (top) searches names. If all know it, skip (about 1 minute, not counted in the floor below).
🖥 Screen: switch to Figure 2.2A and Figure 2.2B; point at ② forward, ③ backward scan. Then the to-scale timeline above (The 3.9 s saved…): three off-loop bits.
A step's skeleton: a big forward block, a big backward block about three times as long. These are scans. MaxText compiles the 79 identical MoE layers as one loop. On the trace, an iteration is a layer. It runs 78; the compiler peels the other one out.
Both runs, same skeleton. Forward: 2,704 milliseconds to 1,998; backward: 8,313 to 5,168. Both shrink, backward by over 3 seconds. Outside the loops, 700-odd milliseconds: layer 0, output head, optimizer, barely changed. All change is in the loops. But a 10,000-plus millisecond screen hides names. Can't see what shrank.
🖥 Screen: switch to the table in 2.3 Reading the rows; point only at the XLA Ops and Sparse Core Ops rows. Below that, One device in Trace Viewer…: left half only, the 2 “Time” rows, the “Tag” rows (right half: 2.5). Recording stop 8 shows it for real: 2 green solid boxes measured, 3 gray dashed boxes labels.
Before zooming, learn the rows. Only two record what was done: the TensorCore's XLA Ops and the SparseCore's Sparse Core Ops. A slice is one instruction run once. The rest are labels for navigation: Steps says which step, XLA Modules which program. Read durations only from those two.
🖥 Screen: switch to Figure 2.3C; point down ① to ⑤, then ⑦.
Zoomed in, a column can stack a dozen or more slices; read it top to bottom. Stacked means contained. On top, the big forward loop, 2,700 milliseconds, holds the 3.4-millisecond expert matmul below. Count time only on the bottom slice, the leaf; the containers above already include it. The rows below aren't more nesting but the leaf's labels: name path, op name, line of Python. The left neighbor's path adds sort_activations: at a glance, the MoE step sorting tokens by expert.
💬 Interaction: ask the audience: per Section 1's table, what do we watch first at one layer? Wait for both, “expert matmuls” and “is the SparseCore traffic still there”, then move on.
🖥 Screen: switch to Figure 2.4A; point at ① three gmm, ② three SparseCore all-gathers.
At one layer, a screen is tens of milliseconds. Zooming over 300 times with W from a full step drifts; safest is writing the layer's start and end into the link, as recording stop 9 does. Both runs: layer 40, same window width, each left-aligned to its start. The original, forward. ① Three expert matmuls on the TensorCore, gmm on the trace, a bit over 3 milliseconds each. ② Three all-gathers on the SparseCore.
What all-gather does here: each device holds only a 64th of the 192 experts and gathers the full set from all 64 when needed. Its shape leads with 192: the whole set.
🖥 Screen: switch to Figure 2.4B; point at the same ① and the ② dashed box.
The modified run, same layer. ① Each gmm is down to 0.04 milliseconds. ② That SparseCore stretch is empty. Dashed: in the original, not here. The layer's expert matmuls: 10.1 milliseconds to 0.1.
It isn't completely empty: a few tiny all-gathers remain, none for all 192 experts, plus gather_offload moving tokens by expert, in both runs. What's missing is exactly one kind: gathering expert weights.
💬 Interaction: ask the audience: forward gathers weights before use. Once backward has the expert-weight gradients, what other collective do FSDP's rules call for? Wait for “sum the gradients, then split them back to each device”, then switch figures.
🖥 Screen: switch to Figure 2.5A; point at ③, three reduce-scatters. Then Figure 2.5B: dashed boxes ② ③.
Backward is even clearer. The original's SparseCore adds ③: three reduce-scatters, about 10 milliseconds each. Reduce-scatter is all-gather reversed: gradients are summed over the 64 devices, and each keeps only its share. Its shape leads with 3: 3 experts per device, 192 divided by 64.
The modified run: ② ③ both empty, nine expert matmuls from 35.7 milliseconds to 0.5. Section 1's table: same ops, just shorter? No. Two whole kinds of SparseCore traffic vanished; expert matmuls are 75 to 93 times shorter. A big chunk of work is gone.
🖥 Screen: switch to Layer 40 collectives in order…; point at dashed cells ④ ⑤, sweep the rest left to right, then ⑦ at the end of backward row A and in row B.
The screenshots box only the longest few. The rest? Here's every collective on this layer's SparseCore, in order. Attention, router and shared-expert weights are gathered in both, about equally often; the modified backward even gathers the router once more. Only two kinds are dashed: ④ expert-weight gathers, ⑤ expert-gradient scatters. Neither program has a single all-to-all: tokens never leave the device.
Now ⑦: both runs sum expert gradients over the 2 data-axis replicas. The original first has ⑤, a sum over the 64 shards; the modified run doesn't. So it sums over only 2? Note that puzzle.
🖥 Screen: switch to Figure 2.4C; point at ①, slivers ② in the lower zoom, then ③.
Look back. A layer has over 500 leaves; why watch only these few? Sort by name. The 13 big compute ops take 68.3% of the layer; the 500-plus slivers under 0.1 milliseconds total less than 3%. Done is the TensorCore's mark for waiting on communication or a SparseCore transfer. This layer has 27, only 4 with width; the rest had data early. So per layer, watch only the widest compute and dones with width.
The missing two: gathering all 192 experts, scattering their gradients back.
🖥 Screen: switch to the done screenshot in 2.5, Figure 2.6; point at the vertical dashed line at the right edges of ① ②, then the two brackets.
Next: how much TensorCore time did the original's communication actually take? It runs on the SparseCore; the TensorCore issues it and computes on, leaving two marks on its row. Start is the issue; done, where the data is needed. If the data's there, done has no width. If not, the TensorCore stalls on done; its width is the wait.
The TensorCore is the head chef, the SparseCore a kitchen hand fetching ingredients. Start: the chef calls “go get it”, keeps cooking. Done: the chef reaches out. If they're there, they're in hand; if not, the chef waits idle.
🖥 Screen: switch to the 2.5 animation; untick “Autoplay” and pause at each of stops 1–5.
Replay this one from the issue. On the TensorCore it's 548; the SparseCore's move is one lower, 547. Stop 1: the TensorCore issues and keeps going. Stop 2: the SparseCore is still moving tokens by expert, so 547 starts over 4 milliseconds late. Stop 3: as it moves, the TensorCore quantizes; 1.23 milliseconds hidden.
Stop 4: the TensorCore reaches done before 547 finishes, and has to wait 1.18 milliseconds. Done and 547 end only 0.1 microseconds apart: released the moment the transfer ends. Stop 5: the gmm right after consumes exactly these weights.
🖥 Screen: open the collapsible block “Reading start/done: hidden vs exposed…” under 2.5; point at “The whole step”: over 2,600 dones, most under 1 µs.
Per step, the original's TensorCore stalls on communication dones only 161 milliseconds; the modified run, 41.6. Saving 119 milliseconds, nowhere near 3.9 seconds. Dropping the communication saves little by itself; the bulk is elsewhere. And one layer is just one of 78 iterations: is the whole missing 3.9 seconds this undone work?
- Asked “Why is Overview's step time a bit above Section 0's?”: Overview averages 3 steps cut at host-recorded boundaries (11,793.8 / 7,857.0); Section 0 is TPU:0's median jit_train_step (11,758.2 / 7,842.1). A 0.2–0.3% gap; the ratio is 1.50 either way (Figure 2.1B, key row ①).
- Asked “Why 78 iterations for 79 layers, and which forward and backward iterations match?”: MoE layer 0 is peeled out. Forward k = layer k+1, backward k = layer 78−k; forward 39 and backward 38 are both layer 40, checked via the HLO while counters (2.2 collapsible block). The 8 expert matmuls between the two passes are the MTP layer, checked in 4.6.
- Asked “Each gmm is 1/93. Why not 1/64?”: kernels have fixed overhead, so it isn't the expert-count ratio. That's inferred, not measured separately.
- Asked “What are those Details fields?”: open the 2.3 collapsible block (Figure 2.3A, a compute op; Figure 2.3B, a collective done). Just one point:
model_flops/bytes_accessedare booked from shapes; modified gmm.27 also shows 2,886,218,022,912. Only Wall Duration is measured. - Asked “Find shows 1,872. Does TPU:0 have that many?”: Find also counts closed processes: 1,872 = 78 iterations × 3 steps × 8 TensorCores. Enter only highlights without jumping; clicking → jumps; and Enter brings all closed ones back (basics recording, stops 8–9).
- Asked “Another estimate of the done waits?”: SparseCore communication not hidden by TensorCore compute: original 195.9 ms, modified 67.6, same magnitude (2.5 collapsible block).
- Asked “Is SparseCore load on the Overview?”: open-source 2.23.1 has no SparseCore cell on the left. The data endpoint has sc_* fields (original: step time 8606.2 ms, idle 5234.02 ms). “Compute” is the idlest of the 16 (the source takes the minimum); step time and idle are averages, so step time − idle ≠ compute. The original's 16 are very unevenly loaded: side evidence only (collapsible block under Figure 2.1B).
- Asked “How were leaves classified?”: by name and HLO category. gmm / tgmm / splash, convolution fusion: big compute; collective names: communication; copy, gather_offload, dynamic-slice start/done: data movement; the rest: small compute. Rules: collapsible block under the 2.4 count table. Backward, same layer: 14 of 63 dones have width.
- Asked “What are the three small whiles in a layer?”: a binary search placing tokens by expert (op_name contains searchsorted), 9 iterations each. On the trace, the sub-level's first slice appears once per iteration (Figure 2.3D).
- Slice colors mean nothing: same name, same color. Key (solid = measured, dashed = nothing here, vertical dashed line = layer end): “Screenshot conventions” box under 2.0.
- Opening XProf live: “Opening it locally” in 2.1. Download the 7 profiles from the GitHub Release, unpack with zstd, pip install xprof, set up directories; a few script lines print links with a window. Trace Viewer opened via Tools on the home page ignores window parameters.
Section 3 · Reconcile the books: is the missing time work that was never done? (21′)Planned 21′ · with interaction 17.6′
🖥 Main screen for this segment: course page Section 3 · Reconcile the books: is the missing time work that was never done? · main figure Figure 3.2
🎯 What this segment must leave behind: 90% of the missing 3.9 seconds is expert matmuls never computed: 117.45% is the alarm, one kernel at 25× peak is the evidence.
🧭 Get everyone there first: “We're in Section 3 of the course page, ‘Reconcile the books: is the missing time work that was never done?’ Start at 3.1, HLO Op Profile.” Switch there, let everyone find it, then begin.
Trace Viewer shows only a small slice at once. For a whole step's totals, switch to two pages that count totals. First, Op Profile. It sorts every TensorCore instruction into a few piles by category, each with two numbers: time and utilization.
💬 Interaction: ask the audience: if a category's time drops by more than half, what does its utilization do? Bet: down, unchanged, up; then switch.
🖥 Screen: switch to Figure 3.1; point at ② the custom-call row in both runs (name and leftmost TIME %). Leave ③ FLOPS and ① overall utilization for now.
custom-call means external kernels XLA, the compiler behind JAX, calls rather than generates: here, Pallas kernels for attention and the expert matmuls. Pallas is JAX's tool for hand-written kernels. custom-call drops from just over 6,500 milliseconds a step to just over 2,900: 3.6 seconds from one category; the rest together barely moves.
🖥 Screen: now point at ③ in both runs.
③, utilization, goes from 53% up to 117.45%. Above 100% means this category does more operations per second than the page's own peak. Don't conclude yet. Let's expand it and see who pulled it up.
🖥 Screen: switch to Figure 3.2; point at FLOPS ①②③ (right), then ④⑤⑥ (left).
Expand custom-call. Most time is three attention kernels at 34 to 40% utilization, almost as in the original. The absurd rows are two expert matmuls with only 0.2% of the time. The gmm.27 group is over 5,100%; tgmm.8, over 2,700%. So 117 is just an average: a big honest attention block plus a small absurd matmul block.
Click gmm.27; read the left panel. 49 microseconds a call, 58.5 PFLOP a second; peta means a million billion. TPU v7's FP8 peak is 4,614 TFLOP a second per chip, 2,307 per TensorCore. That's 25 times over. No chip can do that.
How was this number computed? Utilization is booked FLOPs over measured time. A Pallas kernel's author reports its FLOPs in code. They're from the input shapes only, however many experts actually ran. All three FP8 runs book exactly the same, yet time shrank by more than half. The booking didn't shrink. The work did.
Careful: over 100% alone doesn't prove it's impossible. We infer this page's peak is the BF16 tier, and it doesn't rescale Pallas kernels by precision. So a saturated FP8 kernel could in theory show around 200%. That's why 117 is only the alarm; the single kernel at 25 times is the evidence.
🖥 Screen: switch to the 3.2 Roofline recording; pause briefly at stop 2 and at stop 3.
Same kernel, Roofline Model page. It plots each op on Section 1's roofline. In the original, gmm_megablox.16 sits right on the flat roof, at 81.7% of peak. That's what real work looks like. The modified gmm.27 has no point at all: it's 57 times above the flat roof, off the chart. 57 doesn't contradict the earlier 25. Its flat roof is 1,028.75 per core, less than half the official FP8 2,307. A smaller divisor gives a bigger multiple.
🖥 Screen: switch to Figure 3.2B; point at ⑦ in both, then at ②, the point that rose.
Section1 here treats the whole program as one point: the original at 43% of peak, the modified run at 65%. It looks like a real speedup. But booked FLOPs per step differ by a tenth of a percent. Time drops by a third, so of course utilization rises by half. Take the expert matmuls out of both numerator and denominator, and the rest barely moves, 29.9 to 31.4. The same booking lifts the step average.
🖥 Screen: if asked “Are other ops normal on Roofline?”, open the 3.2 collapsible block “Reading Roofline Section2”. In Figure 3.2C, point at ②, green and blue on the flat roof, then ③, the orange cloud on the slope. Then Figure 3.2D, top half: ④, every done row is 0 FLOPs, marked HBM. Otherwise, skip it.
Section2 plots one point per op. Matmuls hug the flat roof, elementwise loop fusions the bandwidth slope. That's normal. The done category does no math, so it has no point. The table marks it HBM-bound, but that's really the TensorCore waiting for the SparseCore; check Trace Viewer.
🖥 Screen: switch to Figure 3.3; ranks 4–10, then top 3.
So were the expert matmuls simply never called? Switch to HLO Op Stats, a big table ranked by op. In the original, 7 of the top 10 are expert matmuls; the modified run has none. Ranks 4 to 10 all change hands. In come loop fusions, chains of elementwise ops XLA fuses into one, and gather_offload, which moves tokens to their experts. The top three are all attention, and their time per call differs by less than a tenth of a percent. What wasn't touched didn't move at all. This change touched only the expert matmuls.
🖥 Screen: switch to Figure 3.4; point at ② the call count and ③ the time per call.
Filter for gmm, both runs side by side. In the modified run, the first expert matmul ranks 124th. Same count, 1,872: 78 iterations times 3 steps times 8 TensorCores, so every iteration called as usual. But time per call fell from 5,692 microseconds to 94.5. Called, but almost nothing done.
🖥 Screen: switch to Figure 3.5; ① ② in the original, then ③ modified.
At the bottom of Op Stats is a SparseCore table, one row per instruction. Filter for reduce-scatter. The original has six expert-gradient reduce-scatters: three in the loop, 78 times a step each, and three outside, once a step. Each takes 9 to 11 milliseconds, over 2,350 a step in all. In the modified run, neither table has a single one. The matching TensorCore done takes only 0.02 milliseconds a call. These collectives were already hidden under compute, so without them the TensorCore saves almost nothing.
🖥 Screen: switch to the table in 3.4 Reconcile both ends; point at the “Saved” column in three rows: “One step”, “of which expert matmuls”, “Waiting on dones”.
Reconcile both ends, starting with time. A step is 3,916 milliseconds shorter; expert matmuls are 3,570 of it, 90%. The TensorCore waits about 485 milliseconds less, only 119 of it on collectives.
The other end is work. 192 experts over 64 devices is 3 per device. If each computed only its own 3, 189 of 192 shares went undone. All 192 took the original 3,669 milliseconds. Scaled by expert count, the undone part is worth 3,611. The actual saving, 3,570, is 1% off. Both ends match.
One row goes backwards: in the modified run, loop fusion takes 200-odd milliseconds more. Less work, yet something grew? Note it; we'll hit it later.
🖥 Screen: switch to Figure 3.6 (3.5, the 375-second run); point at ② ③ on top, then column ⑤ below.
Here's the opposite case. Another profile takes 375 seconds a step, 31.9 times slower. The same page's first row names the culprit: custom-call, 98.5% of the time at under 1% utilization. Below, Op Stats' top 10 are all expert matmuls, 0.37 to 0.42 seconds per call. It's the original recipe, run without injected tiles. A tile is the matrix block a kernel loads on chip and computes in one go. Here every tile fell back to the default 128 by 128 by 128. With tiles that small, each kernel ran 109 times slower.
💬 Interaction: ask the audience: same page, 117% utilization vs under 1%. What to check in each? Wait for “who skipped the work” and “kernel config”.
The reading is the same for both. Put a category's time share next to its utilization, then expand down to the specific kernel.
🖥 Screen: show the 3.5 box “From a category down to one kernel” (unread, for later).
Time adds up. There's a second ledger: memory. The original must gather all 192 experts before computing. Backward yields a full gradient for all 192, which needs a home until scattered back. Each block is 2,304 MiB. If the modified run really gathers and scatters nothing, these should all vanish and the peak drop by several GiB.
💬 Interaction: ask the audience: how much lower is the modified peak? Take bets: several GiB, a few hundred MiB, or about the same. Then play the clip.
🖥 Screen: switch to the 3.6 memory recording, untick “Autoplay”, play stops 1–3. If you can open XProf live, click through them for real.
Two pages show memory. Memory Profile is the runtime view, time on the x-axis. Stop 1: a 94.74 GiB device, 93.3 in use, only 1.5 left. But it peaks at 1.4 milliseconds, before the training step even starts. Stop 2: Memory ID 1, another device; the peak moves to step 2. Stop 3: put the peak time in HLO's at time; only the training step runs then. The original peaks at 93.2, the modified run at 91.6: only 1.7 GiB less.
💬 Interaction: ask the audience: can this page tell who holds the memory at the peak? Wait for “No, the lines are flat” or “We need another page”, then go on.
All three lines are flat end to end. The program reserves its workspace in one block at launch, so the allocator can't see the ups and downs inside. So we need another page.
🖥 Screen: play stops 4 and 5, then switch to Figure 3.8 (Memory Viewer, three runs); point at ① ② ③ top to bottom, then the green band on curves A and B.
Memory Viewer is the compiler's allocation plan for one program; the x-axis is instruction index. Stop 4: by default it opens a trivial program. Stop 5: switch to the training step, jit_train_step. This version doesn't refresh the header's module name; trust the left panel. The header has three numbers: the peak; entry parameters, the parameters and optimizer state passed in at launch; and the temporaries live at the peak.
The FP8 runs have identical entry parameters, just over 44,500 MiB. BF16 is almost exactly half, because it shards parameters 128 ways; FP8 only 64, each on two devices. The peaks differ by only 169 MiB. We guessed several GiB. Why so little? The green band moved too: late in the original, early in the modified run. Who's at the peak?
🖥 Screen: play stops 6–10, switch to Figure 3.9 (by Buffer Size); A's ① ②, end on ③.
Scroll down. Each buffer live at the peak is one bar, sorted by size; hover to see its name on the left. Stops 6 and 7: the six widest bars are the three expert weights and nu, Adam's second moment for them. That 3 in the shape is each device's three experts.
Stop 8: the half-width bars are mu, the first moment, in bf16. Adam keeps both moving averages per parameter. Together that's two and a half times the parameters: exactly the header's entry parameters. Stop 9: some blocks have no Framework Op in the left panel at all. The compiler inserted those; no code change removes them.
Stop 10, the key one in the original: ③. Five 2,304 MiB gradient blocks, shape 192 by 4096 by 1536. Three are this layer's gradients in the backward loop, just computed, not yet scattered back to 3. The other two belong to the MTP layer, the extra layer after the trunk that predicts one more token. It's outside the loop, and its expert gradients are summed across all devices. The peak lands while these are alive.
🖥 Screen: play stop 11; switch to B in the by Buffer Size figure; ④, then ⑦. Skip stop 12 (BF16) until Section 5.
Stop 11, the modified run. The loop's three 192-expert gradient blocks are nowhere to be found. The widest three are now the output head's logits, a score per vocabulary entry per position, 6,608 MiB each. The three blocks at ⑦ also start with 192, but they're the MTP layer's gathered weights, in both runs, not loop gradients. The savings really are gone. But the peak only sees the single highest moment; the top spot changed hands, so it's only 169 lower.
🖥 Screen: show the 3.6 box “From the peak to the code in four steps” (for later, like 3.5's).
The time and memory ledgers agree: the loop gathers and scatters nothing. But a profile only records what happened. Did each device really get only 3 experts, and why? We need the program itself for that.
- Asked “Isn't FP8 twice as fast as BF16? So over 100% is normal?”: this page's peak is the BF16 tier [inferred: the same page rescales XLA's FP8 matmuls by half] (1,137.15 per core = 1,028.75 × Time Scale Multiplier 1.10537), and it doesn't rescale Pallas kernels by precision, so a full-speed FP8 Pallas kernel could in theory show about 200%. So 117% is only an alarm; the evidence is one kernel at 58.53 PFLOP/s, 25× over even the FP8 peak (3.2 collapsible block).
- Asked “What are the self-reported FLOPs?”: gmm.27 books 2,886,218,022,912 per call = 2 × 229,376 × 4,096 × 1,536; 229,376 = 7 × 4,096 tokens × 8 experts per token (3.2 main text).
- Asked “Why is WASTED 0.0%?”: WASTED = time share × (1 − max(FLOPS utilization, HBM utilization)). Past 100% the bracket is negative, shown as 0: a do-nothing kernel scores zero waste.
- Asked “Any modified gmm that didn't shrink?”: those run once per step (outside the 78-iteration loop) still take 4.2–5.0 ms: the MTP layer, its weights gathered separately (3.3, 4.6).
- Asked “Why does loop fusion take 276 ms more?”: the modified path pads uncomputed rows with 0, found in HLO in 4.6: the two new fusions total about 321 ms, of which about 136 ms is an existing add that changed category and about 185 ms the zero-fill itself; other loop fusions shrink by about 45 ms in total, so the category nets 276 more.
- Asked “How much less waiting?”: waiting on dones drops about 485 ms; collective dones are 119 ms of it (2.5), the rest gather_offload and other async transfers, not split further.
- Asked “Why no tiles at 375 s, and no tuning-log note?”: no launch script or logs, so unknown. Tuning log §3.4.1 has the same mechanism, “without tiles, 12.4× slower”, at another scale.
- Asked “Which Op Stats time column?”: self time, excluding nested sub-instructions. Big blocks like a while wrap a whole iteration, so Total time double-counts (3.3 collapsible block).
- Asked “Op Stats counts per step?”: the TensorCore table sums this host's 8 TensorCores × 3 steps, so counts ÷ 24; the SparseCore table, one SparseCore. Don't mix them: times overlap.
- Asked “Why do the two memory pages' peaks differ by 3.6 GiB?”: runtime peak = all of Stack + Heap, wherever the program is; Memory Viewer = entry parameters + temporaries at the peak. The gap: Stack's excess over those temporaries plus Heap's 0.36 GiB over the entry parameters (3.6 collapsible block). OOM follows the runtime.
- Asked “Why is that Heap block called unused preallocated?”: the allocator's name for a block allocated before capture. It's within 0.36 GiB of the entry parameters (parameters + optimizer state); [inferred] it holds them.
- Asked “What's in the entry parameters?”: f32 parameters 17,813 + Adam nu 17,813 (f32) + mu 8,906.5 (bf16) MiB. f32 parameters × 64 shards ÷ 4 bytes ≈ 298.85B; Hy3's model card lists 295B (main) + 3.8B (MTP layer) = 298.8B, a match (3.6 collapsible block).
- Asked “No [192,…] blocks at all in the modified run?”: at the peak, the loop's [192,…] gradients from tgmm are gone. The MTP layer's expert-gradient all-reduce remains (all-reduce.364/366/368), just not live at the peak. The 3 [192,…] bf16 blocks left are the MTP layer's compiler-inserted weight gather (op_name convert_element_type).
- Asked “How much does Roofline Section2 cut off?”: it plots only the 1000 most time-consuming ops. Of this run's 3,897, the cut ones are only 0.21% of time; custom-call is 55.5% vs 55.4% in Op Stats. Open-source 2.23.1 has no pie chart here (3.2 collapsible block).
- Asked “Do the attention kernels report bytes? Can Bound by be trusted?”: all 18 cost_estimate report bytes; no row has “HBM BW 0, Bound by meaningless”. The two splash kernels' HBM BW is 70.20/56.99 GiB/s (Figure 3.2D).
- Asked “Can Utilization Viewer show the MXU running FP8?”: in open-source 2.23.1 it shows No Data for every profile, a routing bug (log: No ProfileProcessor registered for tool: utilization_viewer), fixed in 2.23.2; the 7 profiles have hardware counters. Our screenshots use 2.23.1, so check the kernel's operand types in Details and its speed instead.
- Asked “Why 5,674.9% on Roofline, 5,147.80% in Op Profile?”: Roofline divides by 1,028.75 directly; Op Profile also multiplies by the Time Scale Multiplier (about 1.105), so about 1.1× apart (3.2 collapsible block).
- Recording: the
3.0 recordingwalks 3.1–3.3 on the real pages in seven stops; play it if you can't open XProf live.
Section 4 · Down to HLO and the code: why wasn't it done? (14′)Planned 14′ · with interaction 12.2′
🖥 Main screen for this segment: course page Section 4 · Down to HLO and the code: why wasn't it done? · main figure 4.4 animation
🎯 What this segment must leave behind: the original has 12 expert all-gathers and 6 gradient reduce-scatters; the modified run has none. Both live only in the tokamax branch. A sharded-down expert dimension is still legal input, so nothing complains.
🧭 Get everyone there first: “We're in Section 4 of the course page, ‘Down to HLO and the code: why wasn't it done?’ Start at 4.1.” Switch there, let everyone find it, then begin.
Something that never happened leaves no trace in a profile. To know what should be there, read what the compiler finally hands the chip: the HLO. You write one train_step function. XLA splits it across 64 devices and fuses elementwise chains into fusions. It inserts collectives where needed and fixes the order. That's the final HLO.
🖥 Screen: switch to the two code lines in 4.1 HLO.
One instruction per line, a shape on every value. Name, equals sign, type and shape, opcode, operands in parentheses. The second line is gmm.27: an output of 229,376 rows by 1,536 columns. Opcode custom-call: a call into an external kernel. Every slice name in the trace is one instruction name here. The trace says when it runs and how long. The HLO says what goes in and out, and which instructions exist at all.
Where do you get the HLO? Three places. In XProf, Graph Viewer and HLO Op Stats show one instruction. The profile files hold all of it, ready to extract and count. Or have XLA dump it on a fresh compile. To count, use the second.
💬 Interaction: ask the audience: what HLO count proves the modified run gathers no experts? Wait for “count the all-gathers, by shape”.
🖥 Screen: switch to the count table in 4.2: first-row totals, then the two red rows.
Totals first: all-gathers go from 92 to 82, reduce-scatters from 6 to 0. That only says something is missing. By shape, it's clear. The original has 12 all-gathers taking the expert weights from this device's 3 to 192. And 6 reduce-scatters taking the expert gradients from 192 back to 3. In the modified run, not one of these 18 exists.
💬 Interaction: ask the audience: each device holds 3; gathered, it's 192. Across how many devices? Wait for “64”, then switch to the excerpt.
🖥 Screen: scroll to the 4.2 HLO excerpt: line 1's async-start input [1,3,…] and output [1,192,…], 'fsdp'=64, then the modified [3,4096,1536].
It's in the first line. Input: this device's 1 by 3 by 4096 by 1536. Output: 1 by 192 by, and so on. The all-gather doing the work groups by fsdp, 64 devices, along the first dimension: 3 times 64, 192. In the modified run, gmm.27's weights come straight from a fusion that only quantizes, shape 3. Nothing is gathered in between.
🖥 Screen: go to the 4.2 figure “FP8 quantization is the same step in both…”. Point at top-row ① to ④, the two ⑤ after the fork, then the two right-hand bars.
Section 1 said the original quantizes, then gathers. Did the modified run just move quantization? Trace both quantization fusions upward. f32 master weights, cast to bf16 before the loop, one layer per iteration. Multiply by 2, clamp to plus or minus 448, cast to FP8. These four steps are identical in both. Then they fork: the original gathers all 64 shards in FP8, 1,152 MiB. Gathering bf16 first would be 2,304 MiB. The modified run gathers nothing; the kernel gets this device's 18 MiB.
🖥 Screen: go to the 4.3 Graph Viewer recording, untick “Autoplay”. Pause at stops 1 and 2, click 3–8 silently, pause at 9, 10, 11. Then Figure 4.2: ⑥ on both sides.
You can trace it on the page too. In Trace Viewer, click the original's gmm_megablox.16. The first line in Details is a Graph Viewer link. It centers on this instruction and draws who feeds it and whom it feeds. Stop 2: below is its HLO line. The weights are 192 experts. There's also an s32 193; hold on to that.
Follow operand 7 upward: bitcast, done, start. One level in is the all-gather itself: this device's 3, 64 devices, gathered into 192. In the modified run, centered on gmm.27, operand 7 is the quantization fusion directly. 3 experts.
Now that ⑥: s32 193. It's the group offsets of 192 token groups, identical on both sides. Tokens still arrive sorted into 192 expert groups. Yet the modified run holds weights for only 3 experts.
💬 Interaction: ask the audience: the kernel multiplies 3 experts' weights by 192 token groups. Error or result? Take bets; don't reveal.
🖥 Screen: switch to the 4.4 animation and untick “Autoplay”. Point through stops 1–7 as they play; no full loop first. Answer the bet at stop 5.
The animation shows device 0. The original gathers all 192 experts from the other 63 devices and computes all 192 groups. It scatters the gradients back; each device keeps 3. The modified run holds only 3, so it computes only 3 groups. The other 189 groups' rows are zeroed. No error.
🖥 Screen: switch to the 4.4 figure “192 experts in 64 shards of 3…”. Point at the same numbers left and right: ①, ②'s blue cells, ④'s dashed outline, ⑤.
This figure lays the animation flat: 192 experts cut in order into 64 shards, 3 per device. The original gathers all, computes all, sums the 64 shards, scatters back to 3. Then one more add across the 2 data-axis copies. The modified run holds only 3, computes only 3 groups, outputs 3 gradients directly. No ④ at all.
Section 2's open question from the communication figure lands here: its ⑤ is ④ here, its ⑦ is ⑤. The modified run keeps only the sum across the 2 copies. It only ever had 3 gradients; nothing to scatter.
Remember the last stop. 189 of 192 shares of the work are gone. Expert matmuls fall from 3,668.8 milliseconds a step to 98.3. Section 3's books now have a picture.
🖥 Screen: go to the op_name table in 4.5 Back to the code, then code ① ④ and ③ ⑤.
Which code was it? Every HLO instruction carries an op_name: the name path JAX recorded while tracing your program. The weight all-gather ends in shard_map/all_gather, handwritten in the code. The original's gmm carries tokamax. The modified gmm.27 carries jit(gmm)/pallas_call: another branch. And of Section 0's 13 changed lines, one is exactly use_tokamax_gmm.
Open ops.py in public MaxText. The forward all_gather of the weights is ①. The backward psum_scatter of the gradients, the code's reduce-scatter, is ④. Both live only in the tokamax branch. The other branch calls the kernel directly, with neither. No forward gather, no backward scatter: matching the all-gather and reduce-scatter that vanished together from the SparseCore row.
The “gather along the fsdp axis” argument reaches both branches. If experts shard on FSDP and weight calibration starts with fixed, the MoE layer passes it on. The other branch ignores it: the argument arrives, but the code using it isn't there.
🖥 Screen: scroll down to the backend.py code: point at num_current_groups, num_total_groups and the L555 line.
The animation's “the rest are written as zero” is these lines. backend.gmm already supports a device owning only some expert groups. Seeing 3 weight groups and a 192-group table, it builds the table for 3, computes only their rows, zero-fills the rest. Every shape is legal; nothing errors. If downstream still accepts a sharded-down dimension as valid, the error never surfaces. The only alarm left is “too good to be true”.
🖥 Screen: switch to the two-row table in 4.6 Section 3’s two puzzles.
Section 3's puzzle: loop fusion grew 276 milliseconds in the modified run. The HLO shows two extra fusions per layer, named with jit(gmm)'s where: the zero-fill we just saw. Each layer it zeroes 189 of 192 shares of the 229,376 rows. The two new fusions total about 321 milliseconds: about 136 is an existing add, fused in under another category, and the remaining 185 or so is the zero-fill. Other loop fusions shrink by about 45, so the category nets 276 more. The work wasn't done, yet zero-filling added work.
The other puzzle: why didn't the expert matmuls outside the loop shrink? That's the MTP layer. The compiler gathers its 192 experts itself, bypassing those two lines of gather and scatter. Nothing was skipped; both runs are equally slow there.
🖥 Screen: stop on the ⭐ line in 4.7 After the fix.
After the fix: move the gather and scatter outside that if, so both branches share them. A step then takes 12.501 seconds, 6% slower than the original's 11.761. That number is from the tuning log; that run kept no profile. The 50% speedup came entirely from skipped work. So what does a really faster change look like on these same pages?
- Asked “Why 92 → 82, not 92 → 80?”: the modified run still gathers the router's small [4096,192] weight and bf16 copies of the MTP layer's 3 expert matrices (compiler-inserted for the sharding: op_name convert_element_type, is_spmd_generated in HLO). The original gathers the outside-loop layer's 3 weights in one instruction with others: 12 by shape, 10 by instruction (4.2 collapsible block).
- Asked “Where do I get the HLO?”: HLO op text in Graph Viewer / Op Stats; the HloProto in the profile (
<module>.hlo_proto.pb); or recompile withXLA_FLAGS=--xla_dump_to(4.1 table). Hard to read? Strip the noise first (4.1 collapsible block). - Asked “What is is_spmd_generated?”: only communication the compiler inserted for the weights' sharding carries it. The original's 573 doesn't: it's handwritten in the code. BF16's 580 does (4.1 collapsible block “One HLO line, symbol by symbol”; Section 5).
- Asked “Open-source Graph Viewer: not found via link”: 2.23.1 ignores the link's module name; the drop-down stays on jit_add. Pick jit_train_step once by hand (recording stops 3–4). Check HLO Full Text showed no full text locally; not chased.
- Asked “Which 3 groups do the other devices compute?”: per the code, without ring-of-experts group_offset is 0 on every device, so every device computes groups 0–2's tokens. Right for device 0; device r multiplies the experts from 3r on by groups 0–2's tokens: [inferred from the code, not verified]. Say exactly this; never call it measured.
- Asked “Which version of the code?”: the yangwhale/maxtext hunyuan3 branch @1afaa0fb named in the Hy3 tuning log. ops.py and backend.py are byte-identical to upstream AI-Hypercomputer/maxtext @188af874, same line numbers; the Python call-stack line numbers in the xplane match.
- Asked “9 unshrunk outside the loop?”: MTP layer. No while/body in op_name, call stack via multi_token_prediction.py, compiler gathers weights in bf16, both equally slow (4.6).
- 12.501 s is a documented number, no profile; we can't recompute it. If pressed, say exactly that.
Section 5 · A control: what does a real speedup look like? (9′)Planned 9′ · with interaction 8.2′
🖥 Main screen for this segment: course page Section 5 · A control: what does a real speedup look like? · main figure Figure 5.1B
🎯 What this segment must leave behind: a real speedup leaves the count and shapes of the work unchanged; the same kernels each get a bit shorter; utilization stays under 100%.
🧭 Get everyone there first: “We're in Section 5 of the course page, ‘A control: what does a real speedup look like?’ Start at 5.1.” Switch there, let everyone find it, then begin.
Section 4 settled FP8's 3.9 seconds: skipped work. Flip it: what does a real speedup look like on these same pages? No new example. BF16's recipe got the same change: the same switch from tokamax to the native branch, the same 12 tile values.
💬 Interaction: ask the audience: in the 1.4 Open the profile table, will BF16 land in “If real” or “If wrong”? Hands up.
🖥 Screen: switch to the table in 5.1 The same change: only the BF16 “One step” row.
Measured, one step goes from 21,747.2 milliseconds to 20,693.8: 1,053.4 milliseconds faster, 4.8%. Is that plausible? Expert matmuls are about 38% of a BF16 step, and the table has them 13% faster. 38% times 13% is right around 5% of the step. The books balance. That's the first signal.
🖥 Screen: scan the right column, then the left FP8 one from the “SparseCore all-gather” row down.
Down the right column, only two rows really move. The step, and expert matmuls: 8,183.8 milliseconds to 7,116.2, 13% less. Communication time, HLO counts, the kernel's weight shape: unchanged. Now FP8, on the left. From the SparseCore communication row down, every row is an alarm. BF16 lands in “If real”.
🖥 Screen: go to Figure 5.1A and Figure 5.1B. Point at the SparseCore's solid ② expert gather and ③ gradient scatter, then back at the dashed boxes in the same spot in Section 2's Figure 2.5B, then the TensorCore's ①, 9 expert matmuls.
Backward layer 40, side by side. SparseCore first. ② expert gather and ③ gradient scatter are solid on both sides, almost the same width. Each all-gather, about 4.9 milliseconds. In Section 2's Figure 2.5B, FP8's modified run had an empty dashed box there. Now the TensorCore. ① all 9 expert matmuls are there, each still 6 to 10 milliseconds. Together, 79.11 to 66.89. Same work, done faster.
This backward layer loses 12.22 milliseconds. The same forward layer's 3 expert matmuls lose 1.07 more: 13.29 per layer. Times 78 layers is about 1,037 milliseconds. Just a dozen or so short of the step's 1,053.4. It adds up layer by layer.
🖥 Screen: switch to the 5.3 Where the 1.05 s went table: the three Δ, then Model GFLOP/s.
The whole step reconciles too. The step loses 1,053.4. The custom-call category loses 1,054.1, and the expert matmuls inside it lose 1,067.6. The call counts are nearly identical: 957 versus 956.
Is the speed plausible? The same expert matmul runs 513,223 GFLOP per second after the change: 0.51 PFLOP, a bit over 40% of BF16's per-core peak. FP8's modified gmm.27? 58.53 PFLOP per second: 25 times even the FP8 peak. A truly faster kernel always stays inside the physical limit.
What wasn't touched shouldn't move. HLO Op Stats' top three are all attention, and their time per call differs by under 0.1%. The change touched only expert matmuls.
🖥 Screen: go to the 3.6 memory recording: stop 12, skipped in Section 3 (bf16old's Memory Viewer): ③ the 3 widest logits, then ② the hovered parameter.
Stop 12, skipped in Section 3: the BF16 original's Memory Viewer. The three widest are again the output head's logits, 11,328 MiB each. The two BF16 peaks differ by only 2 MiB, with identical largest buffers at the peak. Same work, same storage.
But the hovered parameter in stop 12 has an odd shape: f32 192 by 79 by 1536 by 32. All 192 experts are there, each with only 32 rows. On the FP8 side, each device holds 3 complete experts. Same model. How does it end up cut two different ways?
🖥 Screen: go to the three-condition table in 5.4 Why BF16 avoided the trap. Point at BF16's ① and ③ cells.
The tuning log gives three trigger conditions for the FP8 skipped work; all must hold. BF16 fails ①: it doesn't shard on the expert dimension, and isn't quantized. It fails ③ too: 128 shards, and 192 doesn't divide by 128. So BF16 shards the expert weights along 4096. Each device holds 192 experts of 32 rows: the shape in stop 12.
🖥 Screen: scroll to the 5.4 HLO excerpt: [1,192,32,1536] → [1,192,4096,1536], dimensions={2}, is_spmd_generated.
The kernel needs full weights. Who gathers them? An all-gather in the HLO. Input 1 by 192 by 32 by 1536, output 1 by 192 by 4096 by 1536. It concatenates 128 shards along 4096: 32 times 128 is exactly 4096. It carries is_spmd_generated: the compiler inserted it for the sharding, so you won't find it in the code.
So whichever branch runs, the kernel gets all 192 experts. The expert dimension isn't cut, so the “3 experts is also a valid input” trap can't happen. The gather doesn't rely on those two lines, so they can't drop it.
💬 Interaction: ask the audience: you have both examples. Shown one page, no before-run, which page alone exposes skipped work? Wait for “utilization in HLO Op Profile” or “one kernel's speed, past the physical limit”, then close: other pages need two runs side by side; beyond-peak needs no control, one number is simply wrong.
🖥 Screen: go to the five-row table in 5.5 Real speedup vs. skipped work. Point only at the “Op Profile / Op Stats” and “Memory” rows.
Two rows. The Op Profile row: real speedup, 49.62% utilization; skipped work, 117.45%, one kernel past peak. That row needs no control. The memory row is easiest to misread. Both peaks moved only a little. But on the skipped-work side, what's at the peak changed: the 3 in-loop gradient buffers of 192 experts are gone. Read only the peak and you'd think nothing changed.
The rest say the same. Real: the same things, narrower. Skipped work: something gone whole, or past a physical limit. Real is the same work done faster; fake is less work. Next time a number looks too good to be true, in what order do you check?
- Asked “Which three trigger conditions?”: tuning log TUNING-v7 §3.4.10: ① shard_exp_on_fsdp=True and weight calibration begins with fixed; ② use_tokamax_gmm=False; ③ expert count divisible by FSDP shard count. BF16 fails ① (shard_exp_on_fsdp=False, no quantization) and ③ (ici_fsdp=-1 = 128 shards; 192 ÷ 128 isn't whole: our derivation) (5.4 table).
- Asked “What does the gather in BF16's HLO look like?”: all-gather.580,
bf16[1,192,32,1536] → [1,192,4096,1536], concatenating 128 shards along dimension 2 (counting from 0, the 4096 one), with is_spmd_generated; gmm.24 gets[192,4096,1536](5.4 excerpt). - Asked “Why do the two BF16 runs differ by one call?”: 957 vs 956; we didn't chase which one (5.3 table and text).
- Asked “Why is each call of the new kernel about 10% faster?”: kernel and tile details, not covered here. The profile shows the same work, done faster.
- Asked “BF16 top 10, before and after?”: same expert-matmul count; top three (attention) differ by under 0.1% per call. Untouched ops not moving corroborates too (5.3 table, text).
- BF16's same-layer figures and whole-step skeleton: the 5.2 collapsible block. Same conclusion; skip it on stage.
- Asked “How is 0.51 PFLOP ‘a bit over 40%’?”: Op Stats, same gmm.19, 513,223 GFLOP/s ÷ Op Profile's per-core peak 1,137,150 ≈ 45% (original 475,232 ≈ 42%); the Roofline Model page's 1,028.75 gives 49.9% / 46.2%. gmm.27's “25×” is against the official FP8 peak, 2,307 TFLOP/s per core (Section 3).
- Asked “13.29 ms/layer × 78 = 1,037; where's the dozen-odd ms to 1,053?”: expert matmuls lose 1,067.6 per step; the 78 loop layers give ~1,037, the other ~30 ms is outside it (layer 0, MTP) [inferred: not broken down kernel by kernel]. All other ops combined actually gain about 14 ms (1,067.6 − 1,053.4). This layer's Ctrl multi-select readings: Exercise 13.
Section 6 · Wrap-up: how to investigate next time (7′)Planned 7′ · with interaction 6.3′
🖥 Main screen for this segment: course page Section 6 · Wrap-up: how to investigate next time · main figure Next time a number looks too good…
🎯 What this segment must leave behind: too good to be true? Read the profile before celebrating: five steps, each confirmed before the next, each with a readable number.
🧭 Get everyone there first: “We're in Section 6 of the course page, ‘Wrap-up: how to investigate next time.’” Switch there, let everyone find it, then begin.
🖥 Screen: switch to “Next time a number looks too good…”: the orange “Start”, then the five steps left to right, each down to its red dashed box (what we saw).
Look back over the path. A number looked too good to be true. We neither doubted nor celebrated first; we estimated with the rulers. An MoE got a dense model's gain: not necessarily wrong, but worth a look. Then, page by page, in a fixed order.
① The Overview Page characterizes. Both Duty Cycles are 99.9%, so the lost time isn't idle. ② Trace Viewer locates. Same layer side by side: the SparseCore's expert gathers and gradient scatters are all gone. ③ Reconcile the two summary pages. Custom-call utilization: 117.45%. The first expert matmul falls from 4th to 124th.
④ HLO finds the cause. Expert all-gathers go from 12 to 0; the kernel's weights from 192 experts to 3. ⑤ In the code, gather and scatter live only in the tokamax branch.
Why this order? Each step answers one question with one readable number, in its red box. Confirm each before the next. Skip one, and what follows has no footing.
💬 Interaction: ask the audience: would the 13-line config diff alone show it? Wait for “every line looks reasonable; can't tell”.
🖥 Screen: scroll to the 6.1 symptom decision tree “Symptom → where to look…”: the five black entries on top, then the box colors and legend.
A new profile, new symptoms, same reading. First, which kind is it? Step time moved, you don't know where. A number looks too good to be true. You changed something; did it take? Memory is tight. A slice you can't identify. Enter there, walk down. The box color is where you end up: gray Overview, blue Trace Viewer, green summary pages, purple HLO, orange code.
🖥 Screen: still on the tree, point down the column under the second entry: ④ → ⑧ → ⑪.
Our case: a number looks too good to be true, so second entry. ④ absurd utilization: expand to single kernels, and the gmm.27 group reports 5,147.80%. Each box down means the one above isn't fully answered. ⑧ how far from the roof? In the Roofline Model it's off the chart. Next down is ⑪: two pages disagree.
🖥 Screen: scroll to the twelve-row table in 6.1 “Where to look for common symptoms”: # rows 4, 8, 11.
Each circled number is the same-numbered row here: which page, what to check, how to confirm, where our case is. Not now; look up by number when a symptom hits.
🖥 Screen: scroll to the end-of-6.1 figure “When two pages disagree, ask first…”: four data kinds on top, four clashes below; skip the tool cards.
Row ⑪ hit us four times. 117.45% versus Wall Duration. Roofline and HLO Op Profile about 10% apart on one kernel's utilization. Two memory peaks 3.6 GiB apart. A point left of the ridge, labeled Compute. No page was wrong: the two read different data from one file. Durations are recorded at run time; FLOPs and bytes are booked by shape at compile time. When numbers clash, first ask what each reads.
🖥 Screen: switch to the three-row table in 6.2 Three checks before celebrating. Scan the left column, then compare the real-speedup and skipped-work columns.
So before celebrating a great number, check three things, distilled from Section 5's comparison table. Do the rulers agree? Is the same work still there? Does any number exceed a physical limit? Even if all three pass, one thing is beyond a profile: does the loss curve match the old one? That takes enough steps to compare.
🖥 Screen: switch to 6.3 Capture and open your own and point at the profiler, skip_first_n_steps_for_profiler and profiler_steps rows in the table.
Download this course's 7 profiles from the GitHub Release. To open them, see the 2.1 collapsible block “Opening it locally”. For your own, set profiler to xplane in MaxText. Skip the first 5 steps, compile and warm-up, and capture 3. Give xprof the output's tensorboard directory, and every screen here opens on your machine.
🖥 Screen: point at the page-top “✍️ Exercises” button.
There are 24 exercises. The first 12 retrace the main text; the last 12 swap in another collective, profile and kernel. Open-source XProf alone answers them all.
One line: too good to be true? Estimate, then the profile, then celebrate.
- Asked “How to investigate another profile?”: enter the 6.1 tree at one of five entries; look up the number reached in the 12-row “Where to look for common symptoms” table below, each with our example. Event names: the 2.3 collapsible block. Just point to it.
- Asked “The data-source figure's middle row?”: which data kinds each of the 9 tool pages reads (top-right badge). Utilization Viewer is No Data on all 7; its kind is unchecked here (6.1).
- Asked “Pitfalls in open-source XProf?”: we hit four. Graph Viewer from a link ignores the module name; Overview shows 0.0% FLOPS / HBM Utilization; Trace Viewer from the home page ignores the window parameters; Utilization Viewer shows No Data for any profile (fixed in 2.23.2) (6.3 list).
- Asked “What is profile_cleanly?”: on by default: each step waits until the training state is computed, so the profile's step boundaries line up.
- Asked “Profiles from multiple hosts?”: only the first host uploads by default (
upload_all_profiler_results=false); we read one host's file per profile. - Asked “Is this the right entry command?”: it follows the public repo's local-run doc (maxtext.trainers.pre_train.train). It may differ by version; go by yours.
- Asked “One command that produces all these numbers?”: not in this course. Our own scripts read them from the xplane; the pages show the same numbers.
🚧 Where these notes are still thin
- [x] All seven sections match the course page's sub-section and figure numbers; every cue anchor exists there (checked at build time).
- [x] Run-through timing: floor, interactions and played recording and animation stops, about 76 minutes (see the time budget above).
- [ ] No live rehearsal yet: time spent switching screens and waiting for interaction answers is unmeasured.