Performance Tuning and Tooling

New kernel parameters cut an FP8 training step from 11.76 s to 7.84 s: 50% faster.
Don't celebrate yet. Is that even possible? Estimate with a ruler first, then open the profile and work down layer by layer.
Case Tencent Hunyuan3 · 64 TPU v7 chips · FP8 Tool open-source XProf Prereqs Topic 2 (TPU), Topic 5 (parallelism) ⏱ about 82 min

Course author Chris Yang·Google Cloud AI Infra architect

There's also an instructor script.

It covers how to teach: each part's goal, where to pause, common misconceptions, how to answer questions. The content is on this page.

Presenting: follow the script; lines, figure cues and timing are all there. Self-study: give the whole script to an AI for a TA that knows the lecture's background and how to teach it, and have it walk you through.

Section 0

One step got 50% faster. Do you believe it?

📝 Notes for this section

A kernel-parameter change cut a Hunyuan3 FP8 training step from 11.76 s to 7.84 s. Throughput up 50%. Do you believe it?

0.1 Two step times

Original FP811.76 s/step
Modified7.84 s/step
11.76 ÷ 7.84 = 1.50: half again as many tokens in the same time
Setup
ModelTencent Hunyuan3 (Hy3), a 295B MoE: 80 layers, 192 experts per layer (all but the first), 8 per token
Hardware64 TPU v7 chips; each has 2 TensorCores, counted as 2 devices in the profile: 128 in all
ShardingFSDP, 64-way: 64 weight shards, one per device, gathered from the others when needed
Precision, frameworkFP8 training (1 byte per number); MaxText (open-source JAX training framework)

From our own tuning log: real configs, step times and profiles.

How the step times were measured (profile median / MaxText log: they agree)
OriginalModifiedHow measured
Profile step11,758.2 ms7,842.1 msOne jit_train_step on the TPU:0 XLA Modules row; median of 3 complete steps
MaxText step time11.765 s7.853 sHost timer (perf/step_time_seconds) incl. dispatch; about 0.1% longer than the profile
Two reruns—7,846.1 / 7,840.1 msSame config, two more runs, <0.1% apart: not noise

“50% faster” is throughput: 11,758.2 ÷ 7,842.1 = 1.499. The step is one third shorter (7,842.1 ÷ 11,758.2 = 0.667).

0.2 What changed

Config keyOriginal (before)Modified
Expert-matmul tile: w{i,o}_tile_{fwd,dlhs,drhs}_embed_dim (6 keys)10242048
Expert-matmul tile: w{i,o}_tile_{fwd,dlhs,drhs}_mlp_dim (6 keys)10241536
use_tokamax_gmm (the expert-matmul kernel: True = Tokamax library's, False = MaxText's own)TrueFalse

Line by line, only these 13 lines differ. New kernel, bigger tiles, faster matmuls: entirely plausible. At the time, we saw nothing wrong either.

Config diff in detail (18 tile keys; why the original's 1024 isn't its real tile)
  • Both configs have all 18 tile keys; the other 6, *_batch_seq, are 512 on both sides. Only the 12 above and use_tokamax_gmm actually changed.
  • On the tokamax path, which the original recipe takes, the tile isn't read from these 12 keys; the launch script injects it, and the compiled program says (512, 2048, 1536). So “1024 → 2048” doesn't mean the kernel's real tiling changed.
  • Source: MaxText writes the full config to the TensorBoard event file; we diffed both line by line. The injected tile is from the profile's HLO.

0.3 When in doubt, estimate first

⭐ Good news isn't necessarily fake, or necessarily real. Before opening the profile, spend a minute estimating: is this number possible on this machine, for this kind of model? Estimate it, and you'll know what to look for once it's open.

Unfamiliar term? Look it up here (the lecture's ten terms)
TermIn one line
StepOne training iteration: forward + backward + weight update. A shorter step means higher throughput.
MoE, expertsMany parallel feed-forward networks (experts) per layer; a token uses only a few. Hy3: 192 per layer but the first, 8 per token.
Expert matmul (gmm / tgmm)A grouped matmul over the tokens routed to each expert. gmm computes the forward pass and input gradient, tgmm the expert-weight gradient; these are the names in the trace.
tileThe block a kernel loads into on-chip memory and computes before loading the next.
BF16, FP8Two number formats: BF16 is 2 bytes per number, FP8 is 1 byte. On v7, FP8 peak compute is twice BF16's.
TFLOP/s, MFUTrillions of floating-point operations per second. MFU = achieved TFLOP/s ÷ peak (details in Section 1).
TensorCore, SparseCoreA v7 chip has 2 TensorCores (main cores that do the matmuls), counted as 2 devices in the profile, plus SparseCores (small cores); in this lecture's traces the collectives run on the SparseCore rows.
FSDPWeights split into N shards across N devices, gathered back when needed. 64-way here.
all-gather, reduce-scatterTwo collectives. All-gather: every device assembles the full tensor from the shards. Reduce-scatter: sum first, then each device keeps one shard.
profile, XProfA profile records every op's start and end time during a run. XProf is the viewer: open source, runs locally (Section 6).
Section 1

Estimate first: is this number even possible?

📝 Notes for this section

An estimate takes under a minute: measure with two rulers and see which cluster it lands in.

1.1 Ruler one: MFU

MFU = achieved TFLOP/s per chip ÷ peak. v7 BF16 peak: 2,307 TFLOP/s per chip. FP8 FLOPs are still counted from model shapes (a multiply-add = 2 ops, regardless of bytes) over the same 2,307, so both precisions share one ruler.

TFLOP/s per chipMFU
Public MoE (BF16, Qwen3-235B / DeepSeek-V3)629.8 / 612.727.3% / 26.6%
Hy3 BF16 (64 chips, tuned tiles)662.328.7%
Hy3 FP8 original677.029.3%
Hy3 FP8 modified1,014.844.0%
Dense llama3.1-405B (BF16)1,261.454.7%

The modified row leaps out of the MoE cluster, a notch below dense. How to tell: TFLOP/s per chip = MaxText's per-device FLOPs per step (3,981.54 TFLOP) × 2 (devices per chip) ÷ step time. Both FP8 runs log identical FLOPs, so the MFU gain is all from the shorter step.

MFU denominators: 2,307, 2,057.5, 4,614 (same run, new denominator, new number)
  • 2,307: BF16 peak per chip, used by MaxText and this course; per device, half that: 1,153.5.
  • 2,057.5: XProf's Roofline Model page gives 1,028.75 TFLOP/s per TensorCore; × 2 that is 2,057.5 per chip, 10.8% below 2,307. Unify denominators before comparing XProf utilization with MaxText MFU.
  • 4,614: FP8 peak per chip. Divide by it and FP8 MFU is artificially halved (1,014.8 ÷ 4,614 = 22.0%). Always state FP8 MFU's denominator.
  • Sources: TPU7x (Ironwood) specs (per chip BF16 2,307 and FP8 4,614 TFLOP/s, 2 TensorCores) from public Google Cloud docs, cloud.google.com/tpu/docs/tpu7x; TUNING-v7 §⁠1.1 (denominator), §⁠1.2 (public measured table), §⁠2.7 (XProf constants); 1,014.8 = 2 × 3,981.54 ÷ 7.847 and 677.0 = 2 × 3,981.54 ÷ 11.761 (our recomputation).
Compute it yourself (two numbers from MaxText's TensorBoard events; eight lines of Python)
import sys, statistics
from tensorboard.backend.event_processing.event_accumulator import EventAccumulator
ea = EventAccumulator(sys.argv[1]); ea.Reload()
v = lambda k: [e.value for e in ea.Scalars(k)]
t = statistics.median(v("perf/step_time_seconds")[2:])      # skip 2 steps; median avoids warm-up and the profile write
f = v("perf/per_device_tflops")[-1]                         # FLOPs per device per step (TFLOP)
chip = 2 * f / t                                            # 2 devices per v7 chip
print(f"step time {t:.3f} s  per chip {chip:,.1f} TFLOP/s  MFU {chip / 2307:.1%}")
  • Run it on the original and modified event files: 676.2 (29.3%) and 1,013.7 (43.9%). The 0.1% gap from the table is from picking different steps (the table uses the tuning log's step times).
  • Sanity check: an MFU above 100% means a wrong timing basis or overcounted FLOPs.

Five conventions used throughout; we hit or nearly hit each pitfall:

QuantityOur methodPitfall
Step timejit_train_step on the profile's TPU:0 XLA Modules row, median of 3 complete steps (11,758.2 / 7,842.1 ms); or MaxText's perf/step_time_seconds, median after skipping the first 2 stepsOverview Page's “Average Step Time” averages over host step boundaries: 0.2–0.3% longer (§2.1). File writes slow profiled steps: median, not mean
TFLOP/s per deviceMaxText's per-device FLOPs per step (3,981.54 TFLOP) ÷ step time; same in both FP8 runsComputed from model shapes, not measured: it only shows work booked, not work done
MFUTFLOP/s per chip ÷ 2,307 (BF16 peak per chip; 2 devices per chip)XProf: 1,028.75 per TensorCore, even × 2 only 2,057.5 per chip. Unify denominators first (fold above)
FP8 TFLOP/sFLOPs from model shapes, independent of precision; still ÷ 2,307Dividing by the FP8 peak 4,614 artificially halves MFU (1,014.8 ÷ 4,614 = 22.0%)
HBMWhether it fits: the runtime peak. What fills that peak: the compiler's allocation plan (§3.6)The two peaks differ by several GiB; don't mix them (§3.6)

1.2 Ruler two: the roofline

The roofline bounds an op's speed. X-axis: arithmetic intensity (operations per byte moved from HBM). Bandwidth caps the left slope, peak compute the flat roof; they meet at the ridge point.

FP8: at most 2×, and only on matmuls Left: roofline (log–log, per TensorCore). Right: the lower the matmul share p, the less FP8 helps BF16 FP8 (peak ×2) Hy3 Modified Roofline: a matmul's top speed 30 100 300 1,000 3,000 100 300 1,000 3,000 Arithmetic intensity (FLOP / byte) TFLOP/s Roof 1,028.75 Roof ×2 Slope = HBM bandwidth 279 558 e.g. 1,000 FLOP/B FP8 halves bytes → 2,000 Only a step's matmuls gain from FP8 0% 25% 50% 75% 100% 1.00× 1.25× 1.50× 1.75× 2.00× Matmul share of step time, p 2× only at p = 100% DeepSeek-V3 +21% Dense 405B +53% Hy3 Modified +53% ≥35% ≥69% Much of an MoE step isn't matmul: FP8 usually adds only about 20% Hy3 Modified gains as much as dense 405B. That needs 70% of the step in matmuls. Doubtful for now.
Left: the example point rises from blue roof to orange, exactly 2×. Right: the red dot (Hy3 modified, +53% over its own BF16) needs 70% of the step in matmuls.
Roofline constants: open-source XProf's Roofline page. Right curve: an upper bound we derived.
Sources and assumptions

📌 Roofline constants: open-source XProf Roofline page, Device Information (TPU v7x, peak 1,028.75 TFLOP/s per TensorCore, HBM 3,433 GiB/s); the ridge point, 279.1 FLOP/B, matches the 279.085 read in TUNING-v7 §⁠2.7.

📌 FP8 peak ×2: per chip FP8 4,614 ÷ BF16 2,307 (TUNING-v7 §⁠1.1's MFU denominator).

📌 Gains: llama3.1-405B BF16→FP8 +52.8%, DeepSeek-V3 612.7→743.5 (+21.4%), from the official Ironwood measured table as cited in TUNING-v7 §⁠1.2; Hy3 modified 1,014.8 ÷ BF16 662.3 = 1.532 (64 chips, table in TUNING-v7 §⁠3.4.6).

⚠️ Our derivation: FP8 ridge point 558 = 279 × 2 (roof ×2, same bandwidth). A step speeds up at most 1 ÷ (1 − p/2), assuming FP8 at best halves matmul time and leaves the rest; inverting, p ≥ 2 × (1 − 1/gain). Quantization also halves unhidden communication's bytes, so that counts toward p; the original recipe exposes about 0.2 s per step, which doesn't change the conclusion.

⚠️ Illustrative: the intensity-1,000 point is illustrative, not a measured Hy3 kernel. Hy3's BF16 and FP8 runs differ in batch size (12 vs 7), so we compare TFLOP/s per chip.

FP8 shrinks a number from 2 bytes to 1, a 2× gain at either end of the chart. Under the flat roof, the roof rises 2×; on the slope, half the bytes moves the point 2× right, also exactly 2× faster. So a matmul gains at most 2×, and only matmuls gain: routing, per-expert token permutes, elementwise ops and communication aren't on this chart.

Where the right curve comes from (1 ÷ (1 − p/2), solved for p)
  • Step time = matmuls p + the rest (1 − p). At best FP8 halves matmuls, rest unchanged: new step = p/2 + (1 − p), a 1 ÷ (1 − p/2) speedup.
  • Conversely, a measured speedup s means matmuls take at least p ≥ 2 × (1 − 1/s): DeepSeek-V3 s = 1.214 → p ≥ 35%; dense 405B s = 1.528 → p ≥ 69%; Hy3 modified s = 1,014.8 ÷ 662.3 = 1.532 → p ≥ 69%.
  • This assumes FP8 touches only matmuls. The original recipe also quantizes expert weights to FP8 before the all-gather, halving those bytes too. But in this profile, exposed communication is only about 0.2 s per step; counting it doesn't change the conclusion.

1.3 Compare with peers

Public results report the per-chip TFLOP/s gain from BF16 to FP8 on the same model, so Hy3 also starts from its own BF16 (662.3). Section 0's 1.50× starts from the FP8 original recipe instead.

Same switch to FP8: dense +53%, MoE +21%, Hy3 Modified +53% One ruler: TFLOP/s per chip (BF16-equivalent FLOPs); right end 2,307 = BF16 peak = MFU 100% BF16 FP8 The suspect TFLOP/s per chip and MFU 0 500 1,000 1,500 2,000 2,307 MFU 25% MFU 50% MFU 75% MFU 100% MoE BF16 level 600–630 Dense llama3.1-405B 256 chips +52.8% 1,261.4 ≈1,927 MoE DeepSeek-V3 256 chips +21.4% 612.7 743.5 MoE Hy3 Original 64 chips +2.2% 662.3 677.0 MoE Hy3 Modified 64 chips +53.2% 662.3 1,014.8 Too good to be true: an MoE got a dense model's FP8 gain Not proof yet, just a reason to open the profile.
Top two rows first: dense +53%, MoE +21%. Then the red row.
Public numbers as cited in the Hy3 tuning log; the three Hy3 points measured on 64 chips.
Sources and assumptions

📌 llama3.1-405B BF16 1,261.4 (MFU 54.7%), FP8 +52.8%; DeepSeek-V3 612.7 → 743.5; MoE BF16 level 600–630 (Qwen3-235B 629.8, DeepSeek-V3 612.7): official Ironwood measured table, all in bf16 terms, synthetic data, per chip, as cited in TUNING-v7 §⁠1.2.

📌 Hy3 (64 chips, 80 layers, seq 4096): BF16 with tuned tiles 662.3 (20.61 s), FP8 original 677.0 (11.761 s), FP8 modified 1,014.8 (7.847 s), table in TUNING-v7 §⁠3.4.6; we recomputed 677.0 and 1,014.8 from MaxText's per-device FLOPs per step: 3,981.54 TFLOP × 2 ÷ step time.

⚠️ Our derivation: the 405B FP8 point ≈ 1,261.4 × 1.528 ≈ 1,927 (the source table gives only the gain); Hy3's two gains = 677.0 ÷ 662.3 and 1,014.8 ÷ 662.3.

⚠️ FP8 TFLOP/s uses BF16-equivalent FLOPs, still ÷ 2,307; dividing by the FP8 peak 4,614 artificially halves MFU (TUNING-v7 §⁠1.2 warns). Hy3's BF16 and FP8 runs differ in batch (12 vs 7).

⭐ FP8 gives DeepSeek-V3 (MoE) only ~20%, Hy3's original recipe only 2%. The modified run gains 53% over Hy3 BF16, like the dense 405B. Either it cured all MoE's old ills, or something's wrong.

1.4 Open the profile with questions in hand

The estimate can't tell real from fake, but it tells you what to look for. The missing 3.9 s must come off some ops:

If realIf wrong
Same ops, each shorter, mostly the expert matmulsA whole class of ops gone, or absurdly short
Same communicationLess communication too

⭐ Too good to be true? Estimate, then look. Take this table to the profile: which ops lost 3.9 s? Is communication intact?

Section 2

Open the profile: where did the time go?

📝 Notes for this section

Open the profile with Section 1’s question: which ops lost the 3.9 s? Is the same work faster, or is a whole kind of work gone?

Try it yourself The 7 profiles are public in a GitHub Release; “Opening it locally” at the end of 2.1 has every command, download to first view. “Open it yourself” under each figure gives run, tool and window, plus a “↗ Open” link to your local XProf (port 6311).

2.0 Walk the whole path once

⭐ The whole path in 90 seconds: both Overviews → tidy Trace Viewer to one step → read the rows → same layer, A over B.

Local open-source XProf 2.23.1. Each stop is the real page after one action (DPR2 screenshot); boxes from xplane event start/end at page zoom. Autoplays by default; untick “Autoplay” to step; stops 1–12 match the table. Before stops 7 and 9, the old frame zooms to the new real time window.
StopActionYou got it right if you see
1–2Left column of both OverviewsStep time 11,793.8 → 7,857.0 ms; TPU Duty Cycle 99.9% in both
3Open Trace ViewerWhole session: 8 TensorCore processes, two SparseCores under each, dozens of rows
4Click Collapse AllEach process folds into one row
5Keep only /device:TPU:0, SparseCore 0 / 1; Expand AllTPU:0 has Steps, XLA Modules, XLA Ops; each SparseCore has Sparse Core Ops
6Click the jit_train_step block of train 7Details: Wall Duration 11,758,171,824 ns (unit ns; drop the last 6 digits to read ms)
7Press f (zooms from the session to this step)One step fills the screen: forward scan 2.70 s, backward scan 8.31 s
8Read the rowsTake durations only from XLA Ops and Sparse Core Ops; the other rows are labels
9Open layer 40 with view_start/view_end in the link (the view zooms from the step to this layer)This layer, original recipe, forward: 3 expert matmuls at 3.3–3.4 ms each; 3 all-gathers of the 192 experts on the SparseCore
10–11Same layer, A over B: forward, backwardOriginal: SparseCore all-gathers of the 192 experts (forward, backward) and expert-gradient reduce-scatters (backward). Modified: those stretches are empty
12SummaryTwo jobs missing: gathering all 192 experts, scattering expert gradients back

Full video: topic07-xrec-open.en.mp4 (last frame png). The next four subsections unpack the stops with stills of the same windows.

Screenshot conventions ①②③ = reading order, row for row with the key below. Solid box = event on a measured row (only XLA Ops and Sparse Core Ops are measured; 2.3). Dashed box = nothing here: there at the same spot in the original, empty in the modified. Gray dashed vertical line = layer end; next layer to its right (in 2.5’s small example: two right edges aligned). XProf slice colors mean nothing: same name, same color.

2.1 Overview: the verdict first

⭐ The TensorCore idles in neither run (Duty Cycle 99.9%): the missing 3.9 s is in its work. Trace Viewer shows what it did.

Figure 2.1 · Overview left column: Original / Modified. ① Average Step Time; ② TPU Duty Cycle
Left column only: ① step time, ② TensorCore busy?
#WhatOn the page (boxes drawn at its text)
①Average Step TimeOriginal / Modified: 11,793.8 / 7,857.0 ms (XProf’s mean of 3 host-recorded steps; 0.2–0.3% above the median jit_train_step)
②TPU Duty CycleTensorCore busy share: 99.9% in both. Stalling on a done for communication counts as “busy”, so it only rules out long idle gaps

A profile opens on Overview by default. Two questions: how long is a step, and does the TensorCore sit idle for long? Duty Cycle also counts stalling on a done for communication (2.5) as work, so it only rules out long idle stretches; the wait itself is in Trace Viewer. The rest of the column: 2.1B.

Opening it locally (fetch 7 profiles → unpack → pip install xprof → pick a run; window in the URL)

The lecture’s 7 profiles are public in GitHub Release topic07-hy3-profiles-v1 (Hy3 training on 64 TPU v7 chips; each is 3 steps from one host, 1.2–1.45 GB unpacked; only host names changed, to gke-node-N). Four copy-ready steps from download to first view:

# 1. Download (7 assets, 0.86 GB in total; for one or two runs, fetch only those)
mkdir -p ~/hy3-prof && cd ~/hy3-prof
B=https://github.com/yangwhale/gpu-tpu-pedia/releases/download/topic07-hy3-profiles-v1
for r in fp8old fp8new fp8nat r3anat fp8tk bf16old bf16new; do curl -fLO $B/hy3-$r.tar.zst; done
curl -fLO $B/SHA256SUMS && sha256sum -c SHA256SUMS
#   fetched only some? add --ignore-missing; on macOS use shasum -a 256 -c SHA256SUMS

# 2. Unpack (9.2 GB in total): each tar already holds <run>/plugins/profile/<timestamp>/<host>.xplane.pb
#   no zstd command? install it first: apt install zstd / brew install zstd
for f in hy3-*.tar.zst; do zstd -dc "$f" | tar -xf -; done

# 3. Install open-source XProf and point it at this directory
#   ==2.23.1 is the version behind the screenshots; drop it for the latest
pip install xprof==2.23.1
xprof --logdir ~/hy3-prof --port 6311
#   or use TensorBoard: pip install tensorboard tensorboard-plugin-profile
#   then tensorboard --logdir ~/hy3-prof --port 6311 and open the PROFILE page

# 4. Open http://127.0.0.1:6311/ in a browser
#   pick a run in the Sessions dropdown on the left (e.g. fp8old/2026_08_15_03_57_28), and a tool under Tools
  • XProf finds runs by the layout <logdir>/<run>/plugins/profile/<timestamp>/<host>.xplane.pb: each logdir subdirectory is one run. Unpacked, ~/hy3-prof holds 7 run directories; your own profiles laid out the same way appear alongside.
  • Each profile’s first open is slow (measured locally: Overview about 50 s, Trace Viewer about 55 s, while XProf converts the over-1 GB xplane; it caches alongside, so later opens are fast). Port taken? Use another; change 6311 in the links below.
  • Same files as the screenshots: run names, hosts, timestamps match the table, so the “Open it yourself” links work as is.
runWhatMedian step (ms)Unpacked file
fp8oldFP8 original recipe (tokamax, injected tiles 512 / 2048 / 1536)11,758.2fp8old/plugins/profile/2026_08_15_03_57_28/gke-node-1.xplane.pb
fp8newFP8 modified recipe (native + 12 new tile values): 50% faster7,842.1fp8new/plugins/profile/2026_08_15_04_11_46/gke-node-1.xplane.pb
fp8natrerun of fp8new7,846.1fp8nat/plugins/profile/2026_08_15_16_59_57/gke-node-4.xplane.pb
r3anatsecond rerun of fp8new7,840.1r3anat/plugins/profile/2026_08_16_01_29_28/gke-node-5.xplane.pb
fp8tkoriginal recipe, no tiles injected (default 128×128×128): 375 s run375,160.0fp8tk/plugins/profile/2026_08_15_17_48_55/gke-node-4.xplane.pb
bf16oldBF16 before the change (tokamax, injected tiles)21,747.2bf16old/plugins/profile/2026_08_15_09_13_02/gke-node-2.xplane.pb
bf16newBF16, after the same change (native + the same 12 tile values)20,693.8bf16new/plugins/profile/2026_08_15_09_28_25/gke-node-3.xplane.pb
  • The lecture mainly uses two runs: original fp8old, modified fp8new; the other five appear in 3.5 (the 375 s run), Section 5 (BF16) and the exercises page. For your own profile, swap in its run name, timestamp, host and the windows below.
  • Overview: http://127.0.0.1:6311/?run=fp8old/2026_08_15_03_57_28&tag=overview_page&host=gke-node-1, or pick Overview Page under Tools in the home page’s left column.
  • Trace Viewer at a given window: open its page directly, window in view_start/view_end (ms from the profile start). These lines print the link; change run and window to open any screen in this lecture:
import urllib.parse as u
D = {"fp8old": ("2026_08_15_03_57_28", "gke-node-1"),
     "fp8new": ("2026_08_15_04_11_46", "gke-node-1"),
     "fp8nat": ("2026_08_15_16_59_57", "gke-node-4"),
     "r3anat": ("2026_08_16_01_29_28", "gke-node-5"),
     "fp8tk": ("2026_08_15_17_48_55", "gke-node-4"),
     "bf16old": ("2026_08_15_09_13_02", "gke-node-2"),
     "bf16new": ("2026_08_15_09_28_25", "gke-node-3")}
run, vs, ve = "fp8old", 25476.963, 25512.640          # original recipe, forward, layer 40 (Figure 2.4A)
d, host = D[run]
data = f"/data/plugin/profile/data?run={run}/{d}&tag=trace_viewer@&host={host}&hosts={host}"
print("http://127.0.0.1:6311/data/plugin/profile/trace_viewer_index.html?is_streaming=true&is_oss=true"
      f"&trace_data_url={u.quote(data, safe='')}&view_start={vs}&view_end={ve}")
  • Then tidy up as in 2.2: Collapse All → close all but /device:TPU:0 and its two SparseCores → Expand All. Trace Viewer opened from the home page’s Tools ignores the link’s window: you get the whole session.
Open it yourself (same screen in local open-source XProf; first time, see “Opening it locally” in §2.1)
  • run fp8old / fp8new, tool overview_page; read only the left column, Performance Summary.

The rest of the Overview left column

⭐ Of the 7 left-column cells in open-source 2.23.1’s Overview, two are worth reading: step time and Duty Cycle. FLOPS and HBM show 0.0%; Op Placement and Eager are fixed for JAX training; SparseCore step time and idle get no cell.

Figure 2.1B · Overview left column, cell by cell: Original / Modified (open-source XProf 2.23.1; no SparseCore cell in the left column). ① Average Step Time; ② FLOPS Utilization, two cells; ③ TPU Duty Cycle; ④ Memory Bandwidth Utilization, two cells; ⑤ Framework Op Placement; ⑥ Op Time Spent on Eager Execution; ⑦ Run Environment
Same page, both runs: ① and ③ differ, the other five match.
#WhatBoth readings (verbatim page text; extras in parentheses from the same page’s data endpoint)
①Average Step Time11,793.8 / 7,857.0 ms (σ 30.1 / 4.7 ms): 3-step mean, steps cut at host-recorded boundaries; 0.2–0.3% above §0.1’s median jit_train_step
②FLOPS Utilization ×20.0% in both: uncomputed in open-source 2.23.1, not really zero. Read utilization in Roofline and HLO Op Profile (Section 3)
③TPU Duty Cycle99.9% / 99.9%: TensorCore busy share. Only 11.09 / 12.00 ms idle per step (data endpoint tc_idle_ms_average: the Step-time Graph’s TensorCore idle, right)
④Memory Bandwidth Utilization ×20.0% in both: as ②, not computed by open-source XProf
⑤Framework Op PlacementHost 0.0%, Device 100.0% (both): all ops on the device, as always in JAX training; no help here
⑥Op Time Spent on Eager ExecutionHost and Device 0.0%: nothing runs eagerly op by op (JAX jit-compiles the whole step); no help here
⑦Run EnvironmentTraining job, 1 host, TPU v7x, 8 device cores (4 chips × 2 TensorCores). Same in both

2.1 read only ① and ③; first-timers ask whether the other five cells matter. Side by side, ② and ④ are 0.0% in both and ⑤ ⑥ ⑦ match word for word: only ① and ③ differ. Is the SparseCore busy? The column has no cell for it, and the Step-time Graph on the right plots only the TensorCore; its busy time and traffic are in its Trace Viewer row (2.4, 2.5).

SparseCore data-endpoint fields (off-page; compute is the idlest core’s, so compare A with B only)

The page’s data endpoint has these fields (command in “Open it yourself”):

Field in the data endpointMeaning (from the field name)OriginalModified
sc_step_time_ms_averageSparseCore step time8,606.2 ms5,001.9 ms
sc_idle_ms_averageSparseCore idle5,234.02 ms3,341.18 ms
sc_compute_ms_averageSparseCore compute1,582.65 ms1,622.52 ms
sc_infeed_ms_averageSparseCore infeed0.00 ms0.00 ms
sc_outfeed_ms_averageSparseCore outfeed0.00 ms0.00 ms
tc_idle_ms_averageTensorCore idle (for reference)11.09 ms12.00 ms
  • In open-source 2.23.1’s xprof/convert/op_stats_to_input_pipeline_analysis.cc, each SparseCore’s compute per step = step time − idle. Within a step, step time and idle are averaged over the 16 SparseCores, but compute takes the minimum of the 16; then all are averaged over steps. So step time − idle = 8,606.2 − 5,234.02 = 3,372.2 is the mean compute; sc_compute’s 1,582.65 is the idlest core’s. SparseCore steps follow its own Sparse Core Steps row, not TensorCore steps.
  • The original’s 16 SparseCores are lopsided (under one TensorCore, one busy about 5.0–5.6 s, the other about 1.6 s); the modified’s 16 all sit at 1.62–1.70 s, so comparing minima understates the gap. Side evidence only; see their Trace Viewer rows for busy time and traffic.
  • Per-step TensorCore idle in the Step-time Graph on the right (data endpoint tcIdleTimeMs): Original 5.5 / 13.9 / 13.9 ms, Modified 5.8 / 15.3 / 15.0 ms. Step 0 is just over 5 ms in both runs, the next two in the teens [cause not checked].
  • The 8 device cores in ⑦ = 4 v7 chips × 2 TensorCores each: each published run holds only one host’s xplane, so you see only its 4 chips here (64 in the whole job).
Open it yourself (same screen in local open-source XProf; first time, see “Opening it locally” in §2.1)
  • run fp8old / fp8new, tool overview_page: http://127.0.0.1:6311/?run=fp8old/2026_08_15_03_57_28&tag=overview_page&host=gke-node-1; the left column reads ①→⑦ from top to bottom (scroll down for Run Environment).
  • Data endpoint (the sc_* fields the page doesn’t show): curl -s --compressed "http://127.0.0.1:6311/data/plugin/profile/data?run=fp8old/2026_08_15_03_57_28&tag=overview_page&host=gke-node-1" | python3 -m json.tool | grep -E "sc_|tc_idle"; change the run name and timestamp to read the modified run.

2.2 Trace Viewer: down to one step, one layer

⭐ One step per screen: a forward loop and a backward loop, 78 iterations each, one layer per iteration. Both runs share this skeleton; the modified one is a third shorter.

Trace Viewer draws each op as a slice on a time axis and opens on the whole session, dozens of rows (recording stop 3). Four moves: Collapse All → close the rest (X at the right end), keeping only /device:TPU:0 and its SparseCore 0 and SparseCore 1 (other devices look the same) → Expand All → on XLA Modules, click the median step’s jit_train_step and press f to fill the screen.

Figure 2.2A · FP8 Original · Skeleton of one step. ① One step: jit_train_step; ② Forward scan (78 iterations); ③ Backward scan (78 iterations)
Original, one step: ② forward scan, ③ backward scan, about 3× the forward.
#WhatHow to tell (names, times from the xplane)
①One step: jit_train_stepjit_train_step on the XLA Modules row, 11,758.17 ms (median of 3 complete steps)
②Forward scan (78 iterations)while.457, 2,704.03 ms, 78 iterations (the compiler peels layer 0 off before the loop and runs it separately; fold below)
③Backward scan (78 iterations)while.453, 8,313.25 ms, 78 iterations, layers counting down
Figure 2.2B · FP8 Modified (50% faster) · Skeleton of one step. ① One step: jit_train_step; ② Forward scan (78 iterations); ③ Backward scan (78 iterations)
Modified: same skeleton, every part shorter.
#WhatHow to tell (names, times from the xplane)
①One step: jit_train_stepjit_train_step on the XLA Modules row, 7,842.07 ms (median of 3 complete steps)
②Forward scan (78 iterations)while.480, 1,998.05 ms, 78 iterations (the compiler peels layer 0 off before the loop and runs it separately; fold below)
③Backward scan (78 iterations)while.477, 5,168.26 ms, 78 iterations, layers counting down

A v7 chip has 2 TensorCores, which the profile counts as 2 devices; keeping one is enough.

scan: with scan_layers=True in the config, MaxText compiles the 79 structurally identical MoE layers into one while loop, one layer per iteration; on the trace it is the two big blocks ② ③. Forward 2,704.0 → 1,998.1 ms, backward 8,313.2 → 5,168.3 ms. Both shrink, the backward far more (by 3.1 s).

The 3.9 s saved: ~80% from the backward scan; outside the loops barely changes To scale, A/B on one ruler, left-aligned. Bounds from xplane: jit_train_step on XLA Modules, the two top-level whiles on XLA Ops Outside loops Forward scan (while) Backward scan (while) A Original 11,758.2 ms Forward scan 2,704.0 ms Backward scan 8,313.2 ms (3.1× forward) Before loops 142.0 Between loops 317.5 After loops 281.4 B Modified 7,842.1 ms Forward 1,998.1 Backward scan 5,168.3 ms 124.3 302.9 248.6 Backward saves 3,145 Forward saves 706 Outside loops (3 parts) save 65 in all B’s step ends here, 3,916 ms shorter than A What each part holds (from xplane leaf names and tf_op: jvp() = forward, transpose(jvp()) = backward) Before loops 142.0 → 124.3 ms Embedding lookup Forward: dense layer, MoE layer 0 (2 splash, 3 gmm calls) Fwd scan 2,704.0 → 1,998.1 ms Forward of MoE layers 1–78 78 iterations, one layer each Between loops 317.5 → 302.9 ms MTP layer, output head, loss forward and backward (MTP layer: 8 expert matmuls) Bwd scan 8,313.2 → 5,168.3 ms Backward of layers 78–1 78 iterations, counting down After loops 281.4 → 248.6 ms Backward: dense layer, MoE layer 0 AdamW update, all params (multiply_reduce_fusion) Both scans do one layer per iteration: a little saved per layer × 78 = seconds Saved: backward scan 3,145 ms, forward scan 706 ms; the 3 parts outside the loops (embedding, dense layer, MoE layer 0, MTP, output head, optimizer) only 65 ms together. Next: zoom into one iteration (one layer) of a scan, A over B on the same layer.
One ruler, two steps: nearly all saving is in the two scans, about 80% backward. Compare the orange bars; trace the 3 dashed boxes right of B to their parts.
Sources and assumptions

📌 Step bounds: the median jit_train_step on TPU:0 XLA Modules (median of 3 complete steps), fp8old 23,983.849–35,742.021 ms, fp8new 8,262.067–16,104.132 ms (from profile start).

📌 Bounds of the two whiles: fp8old while.457 24,125.829–26,829.855, while.453 27,147.384–35,460.634; fp8new while.480 8,386.321–10,384.372, while.477 10,687.259–15,855.523 ms. The five parts sum to the step time within 0.1 ms.

📌 What each part holds, told by top-level leaf names, tf_op and shapes: fusion.502 before the loops reads the vocabulary (bf16[120832,4096]); the dense layer’s intermediate dim is 13,312 (config mlp_dim); between the loops are two bf16[7,4096,120832] output-head matmuls and 8 expert matmuls (gmm 3, gmm_t 3, tgmm 2) of the MTP layer; the multiply_reduce_fusion after the backward loop holds AdamW’s first and second moments and weight decay (0.9 / 0.95 / 1e-8 / 0.1, matching the config).

📌 Why MoE layer 0 is outside the loops: the compiler peeled it off the scan, forward before, backward after (sources in the main-text fold “Why 79 layers loop only 78 times”).

Why 79 layers loop only 78 times, and which forward and backward iterations match (checked in the compiled program)
  • Of the 80 layers, the first is dense and the other 79 are MoE, weights stacked with leading dim 79. “Layer N” in this course always counts MoE layers (0–78), leaving out the dense one. The compiler peels MoE layer 0 out of the loop to run alone: forward takes layers 1–78 (layer 0 before), backward 78–1 in reverse (layer 0 after).
  • So forward iteration k = layer k+1, backward iteration k = layer 78−k; forward 39 and backward 38 are both layer 40, the one 2.4’s A/B pairs use. Source: the two whiles’ counter initial values, conditions and indexing in the compiled HLO.
  • Outside the two loops a step still has 741 ms (6.3%): 142.0 ms before (MoE layer 0 forward); 317.5 ms between (output head, loss, 8 expert matmuls of the MTP layer, checked in 4.6; MTP is multi-token prediction, an extra layer after the trunk, covered in 3.3); 281.4 ms after (MoE layer 0 backward, optimizer update) [inferred: assigned by the op names in each part]. Modified: 124.3 / 302.9 / 248.6 ms, almost unchanged; all the change is inside the two loops.
Basics: W/S/A/D, click, f, m, Ctrl multi-select, Find (new to Trace Viewer? watch this 1-minute-plus recording first)
Local open-source XProf 2.23.1, original recipe; each stop: the real page after one action (DPR2 screenshot); boxes from xplane event start/end at page zoom. Stops 1–9 match the table.
StopActionWhat you see
1The window in the link opens the original recipe’s forward layer 40, already tidied to TPU:0 + two SparseCores35.7 ms on screen
2Hover over gmm_megablox.16, press W 3 timesZooms around the mouse to 10.6 ms on screen; the name appears
3Press S 3 times to zoom back, D 2 times to pan rightWindow moves right by about 10 ms; A pans left
4A to come back, click gmm_megablox.16Details: Wall Duration 3,425,196 ns (drop the last 6 digits to read ms)
5Click all-gather.575 on the SparseCore, press mInterval marked on the ruler (3.981 ms); over the same span the TensorCore computes gmm_megablox.16
6With it still selected, press fThe slice fills the screen; the 3.981 ms marked by m is exactly one screen wide
7Same layer in a new page: click one, hold Ctrl (Command on a Mac) and click two more“3 items selected.”, Totals 10,106,678 ns = 10.1 ms for all three
8Type gmm_megablox.16 into Find Events at the top, press Enter“0 of 1872”, view doesn’t move; 1,872 = 78 iterations × 3 steps × 8 TensorCores, closed processes included. Gotcha: closed processes all come back
9Click the → right of the boxJumps to the first match, the one in train 5: 3,329,133 ns

Shortcuts need the mouse over the timeline; durations are in ns; clicks while loading don’t count. Full video: topic07-xrec-basics.en.mp4.

Open it yourself (same screen in local open-source XProf; first time, see “Opening it locally” in §2.1)
  • Original, one step: run fp8old, view_start=23933.8, view_end=35792.0 ↗ Open; or, from the whole session, click train 7’s jit_train_step and press f.
  • Modified, one step: run fp8new, view_start=8212.1, view_end=20070.3 ↗ Open (both windows 11,858.2 ms wide).

2.3 Reading the rows: only two carry time

⭐ Only two rows are measured: XLA Ops (TensorCore) and Sparse Core Ops (SparseCore), where a slice = one instruction run once. The other rows are labels.

RowWhat it isWhat to read
StepsTraining step (train 5/6/7), host-loggedLabel: find the step; not timed
XLA ModulesCompiled program running; one jit_train_step = one stepA step’s start and end
XLA Ops (measured)Every TensorCore instruction: big loop slices on top, their instructions belowTime ops on the bottom level only
Sparse Core ModulesProgram running on the SparseCoreLabel
Sparse Core Ops (measured)Every SparseCore instruction: collectives (all-gather, reduce-scatter, all-reduce) and per-expert token moves (gather_offload)Communication time

Collectives run on the SparseCore here: the TensorCore issues one and keeps computing while the SparseCore moves data. So to see if communication shrank, check the SparseCore row; the TensorCore keeps only start (issue) and done (wait), as in the small 2.5 example.

One device in Trace Viewer: 3 processes, 2 measured rows; a collective leaves only 2 marks on the TC Left: rows kept after cleanup, names as in the UI. Right: the all-gather in layer 40 of the Original backward pass, at real times (screen 8.6 ms) Time: measured durations Tag: locate only, not timed TC waits 31,220 31,221 31,222 31,223 31,224 31,225 31,226 31,227 ms from profile start ▼ /device:TPU:0 (TensorCore) Steps tag XLA Modules tag XLA Ops upper time XLA Ops lower time Framework Name Scope +2 rows tag ▼ /device:TPU:0 SparseCore 0 Sparse Core Modules tag Sparse Core Ops time ▼ /device:TPU:0 SparseCore 1 Sparse Core Modules tag Sparse Core Ops time train 7 jit_train_step (one slice = one step) Leaf’s name path, tf_op, code line: read down, not timed jit_train_step jit_train_step while.453 · backward loop (container: width includes all leaves below) abs_reduce_fusion clamp_convert_fusion.144 clamp_convert… …548…call-done gmm_megablox.14 gather_offload (token moves) all-gather.547 all-gather.545 gather_offload (token moves) 1 2 3 hid 1.23 ms 4 exposed 1.18 ms SC starts moving 4.27 ms after start Other devices and the host collapsed (rows identical) 1 start: TC issues, near-zero width (a vertical line), then keeps computing all-gather.548.cloned.1.call-start 2 Execution bar: the actual move, on the SC’s Sparse Core Ops row; its number is the TC’s minus 1 547 ↔ 548 3 Hidden: SC moves while TC computes other work (here FP8 quantization, clamp_convert) 1.225 ms 4 Exposed: TC reaches done before the move ends and stalls; done’s width is the wait, its right edge aligns with the execution bar’s end 1.181 ms Comms shrank? Check the SC row. Did the TC wait, and for how long? Check done’s width A collective leaves only start and done on the TC; the transfer time isn’t on it. done width 0 = data arrived early; any width = exposed wait.
Rows on the left, marks on the right. Only two rows are measured; on the TensorCore a collective leaves only start and done. Right: the 2.5 example from issue to release, at real times; ②③④ return in 2.5 and its animation.
Sources and assumptions

📌 Every event on the right is from the fp8old xplane (TPU:0’s XLA Ops; Sparse Core Ops on both SparseCores), the same data as the static figure and agdone animation in §2.5: start 31,219.512 ms; all-gather.547 on SparseCore 0, 31,223.777–31,226.183 ms (2.406 ms); TensorCore all-gather.548.cloned.1.call-done, 31,225.002–31,226.183 ms (1.181 ms); after release, gmm_megablox.14 starts at 31,226.184 ms.

📌 The small cluster right after start on the TensorCore is a few dozen tiny slices within a dozen or so µs (sort, slice, copy, etc.), not drawn individually. The slices on SparseCores 0 and 1 are gather_offload_custom_fusion.68/.64 (per-expert token moves).

📌 Row names come from Trace Viewer in our local open-source XProf 2.23.1. Framework Name Scope is really several rows (one per name-path segment), merged into one here. The xplane also has an Async XLA Ops row, not used in the text, so not drawn.

Click a slice: what Details shows (one compute op, one collective done)

Click a slice for Details below: fields on the left, links and up/downstream on the right. We try a compute op and a collective done.

Figure 2.3A · Click a compute op: what to read in the Details panel (Original, gmm_megablox.16). ① Title: slice name; ② Wall Duration: how long; ③ Start Stack Trace: the HLO text; ④ hlo_category: which category; ⑤ model_flops: FLOPs booked; ⑥ bytes_accessed: bytes booked; ⑦ source: which Python line; ⑧ Right column: links, up/downstream
Compute op: ② how long, ③ the HLO text, ⑤ FLOPs booked, ⑦ which Python line.
#WhatWhat to read (boxes mark where the text sits)
①Title: slice namegmm_megablox.16: its name in the HLO (minus the %)
②Wall Duration: how long3,425,196 ns = 3.43 ms: the slice’s width, in ns. The Start line above is the time since profile start
③Start Stack Trace: the HLO textThe compiled line: output shape plus each operand’s type and name. Section 4 counts shapes from it (Figure 4.1 reads it symbol by symbol)
④hlo_category: which category"custom-call": a call to an external kernel (here the Pallas expert matmul). Section 3’s Op Profile groups categories by this field
⑤model_flops: FLOPs booked2,886,218,022,912 = 2 × 229,376 × 4,096 × 1,536: computed from the shapes. The modified recipe’s gmm.27 runs only 0.04 ms yet shows the same number (we checked it locally); the 117% in §3.2 comes from this booking
⑥bytes_accessed: bytes booked5,664,411,648: the compile-time estimate of bytes read and written, also from the shapes (modified recipe’s gmm.27: 5,664,407,552, almost unchanged)
⑦source: which Python linemegablox/ops.py:106: the return gmm_fwd_bwd(…) line in public MaxText (gmm’s entry point). The source_stack below is the full call stack, which §4.5 follows back to the code
⑧Right column: links, up/downstreamThe Graph Viewer link (§4.3 enters here); operands: whose outputs it reads; consumers: who uses its output
Figure 2.3B · Click a collective’s done: which fields change (Original, the done of all-gather.548). ① Title: …call-done; ② Category: sparsecore_offload; ③ Wall Duration: how long the TensorCore waited; ④ HLO: what is moved; ⑤ hlo_category + model_flops; ⑥ bytes_accessed: bytes moved; ⑦ operands: the paired start
Collective done: ② Category appears, ③ is the TensorCore’s wait, ⑤ FLOPs is 0, and ⑥ bytes are the moved weights.
#WhatWhat to read (boxes mark where the text sits)
①Title: …call-doneall-gather.548.cloned.1.call-done: a collective splits into two instructions, issue (start) and wait for completion (done); this is the done, the subject of the 2.5 example
②Category: sparsecore_offloadThis collective is handed to the SparseCore; compute ops lack this line
③Wall Duration: how long the TensorCore waited1,181,016 ns = 1.18 ms: how long the TensorCore waits on done for the SparseCore to finish, i.e. this collective’s exposed part
④HLO: what is movedResult f8e4m3fn[1,192,4096,1536], input [1,3,4096,1536]: each device gathers its own 3 experts’ FP8 weights into all 192
⑤hlo_category + model_flops"async-done", 0: nothing booked, just waiting
⑥bytes_accessed: bytes moved1,207,959,552 = 192 × 4,096 × 1,536 × 1 byte: the FP8 weights of 192 experts, about 1.21 GB
⑦operands: the paired startall-gather.548.cloned.1.call-start: its issuing instruction; Find it by name in Trace Viewer
⭐ Don’t confuse these two fieldsFor compute ops, model_flops / bytes_accessed are booked at compile time from the shapes, not measured: they ignore how many rows the kernel actually computed. Only Wall Duration is measured.
Open it yourself (same screen in local open-source XProf; first time, see “Opening it locally” in §2.1)
  • Compute op: forward layer 40, fp8old view_start=25476.963, view_end=25512.640 ↗ Open; click gmm_megablox.16 on the TensorCore (the same one as stop 4 of the basics recording); scroll down to Args at the bottom of Details.
  • Collective done: fp8old view_start=31222.3, view_end=31227.3 ↗ Open (backward layer 40, the 2.5 window); click all-gather.548.cloned.1.call-done, the slice in the middle of the TensorCore’s second XLA Ops level.
Reading event names left to right (this course’s names; public conventions only)
NameHow to read it
all-gather.548.cloned.1.call-doneall-gather is the operation, .548 the instruction’s number in the program (a different number is a different instruction). Middle suffixes like .cloned.1 come from compiler cloning or rewriting; skip them [inferred from the naming pattern]. Trailing call-start/call-done are one async run’s issue and wait. The SparseCore slice doing the move is 1 lower (all-gather.547, 2.5)
gmm_megablox_transpose_rhs.10, gmm.27, splash_mha_fwd…Pallas kernels: a self-chosen name (expert matmul, attention) plus a number. Same-named slices are one kernel called many times
gather_offload_custom_fusion.68, gather_offload_async_done.6Per-expert token moves offloaded to the SparseCore: the first runs there, the second is the TensorCore waiting for it
broadcast_select_fusion, select_add_fusion, clamp_convert_fusionWhen XLA fuses elementwise ops, the name lists them [inferred from the naming pattern]: clamp_convert is FP8 quantization (clamp, then cast); select picks one of two by condition
fusion.1176{1} (in Memory Viewer)The number in braces is which of the instruction’s outputs this is: one instruction can produce several buffers

Reading down a column: the leaf and its code

⭐ Zoom to a few ms around a kernel and expand all rows under TPU:0: a column stacks a dozen-plus slices. Which level holds the duration? Is each level nested in the one above, or independent?

Figure 2.3C · Reading down a column: gmm_megablox.16 from the loop to the line of code (Original, forward layer 40). ① Top level: while.457; ② Next level: leaf op gmm_megablox.16; ③ Framework Name Scope: the name path; ④ Framework Ops: the leaf’s tf_op; ⑤ Source code, upper: which Python line; ⑥ Source code, lower: empty here; ⑦ Left neighbor’s extra level: sort_activations
Read ① down to ⑤: main loop → leaf → name path → tf_op → code line. ⑥ is empty; ⑦ is the left neighbor’s extra level.
#WhatHow to tell (boxes from xplane event times and DOM rows)
①Top level: while.457The forward main loop: from 24,125.829, 2,704.03 ms, 78 iterations, one layer each. This screen spans only 6.6 ms, so the loop runs in from the left and out the right; neither end is visible. A container: the level below lies within its span
②Next level: leaf op gmm_megablox.16From 25,497.811 ms, 3.425 ms, nothing below: only a slice with nothing under it is one instruction executing once (a leaf). Time only this level: while.457’s 2,704 ms already includes it, so adding both double-counts
③Framework Name Scope: the name pathjit(train_step) → jvp() → while → body → closed_call → shard_map: the leaf’s tf_op split into a path, one segment per row, outer around inner. jvp() = forward (backward: transpose(jvp())), while/body = scan loop body, shard_map = per-device part. No iteration (layer) index
④Framework Ops: the leaf’s tf_opThe full tf_op string: the first half is ③’s path; the trailing tokamax:{…} is the kernel name and config (the Pallas expert matmul). Left neighbors show only op names like gather, convert_element_type
⑤Source code, upper: which Python line/deps/src/maxtext/kernels/megablox/ops.py:106: the leaf’s Details source (innermost source-stack frame). Adjacent leaves from the same line merge into one, so it’s wider than ②: from clamp_convert_fusion.165 (FP8 quantization) past gmm_megablox.17 and off screen. The quantization happens in gmm’s entry point
⑥Source code, lower: empty hereSource code stacks by time too: a container with a source takes the upper row, its leaves the lower (like the three small whiles in Figure 2.3D). while.457 has no source or tf_op, so ③④⑤ lack its level and the lower row here is empty
⑦Left neighbor’s extra level: sort_activationsgather_offload_async_done.6 (1.43 ms) has an extra path segment, sort_activations; Source code: /deps/src/maxtext/layers/moe.py:142, the MoE step that sorts tokens by expert. A run of slivers usually belongs to one module; read down the column to see it.

⭐ Stacked = contained: XLA Ops is a single row of events in the profile; XProf stacks it into two levels by which time range encloses which. Time only the bottom slice, a leaf op: nothing under it, one instruction run once. The container above (here the forward loop while.457) already spans all its leaves. The bottom three rows are the leaf’s name path, tf_op (JAX’s tag for the op and its code) and code line, not more nesting.

Same for ⑦: its extra segment sort_activations and line maxtext/layers/moe.py:142 put the left neighbor gather_offload_async_done.6 in the MoE step sorting tokens by expert. Tiny slices nearby often share a module; read down a column.

Why only while gets sub-levels in XLA Ops, and how to count its iterations (three small whiles inside one layer)

Most HLO instructions run on the hardware as one block: a fusion compiles to one kernel, and the sub-computations of sort or reduce (just rules for comparing or adding two values) go into that same kernel. Only while runs its body iteration by iteration at runtime; each body instruction logs an event, so they stack beneath it as sub-levels. JAX’s lax.scan, fori_loop and while_loop all compile to while, as do library functions that use scan inside. A Python for unrolls at compile time: no while.

Figure 2.3D · Loops inside a layer: count a small while’s iterations on the trace (Original, forward layer 40, 0.27 ms). ① Level 1: while.457; ② Level 2: three small whiles; ③ Level 3: loop body, once per iteration; ④ Source code, upper: the container’s; ⑤ Source code, lower: each leaf’s
Three levels: ① main loop → ② three small whiles → ③ loop body per iteration. Source code rows: ④ container’s, ⑤ each leaf’s.
#WhatHow to tell (boxes from xplane event times and DOM rows)
①Level 1: while.457Forward main loop: the first slice beneath it, prepare_start_all-gather.570…, appears 78 times inside it = 78 iterations (forward layers 1–78); backward while.453 counts to 78 the same way
②Level 2: three small whileswhile.489/488/487, about 80 µs each, back to back just before the three expert matmuls: level-2 containers, not leaf ops
③Level 3: loop body, once per iterationCount any slice of the sub-level: the boxes mark the easy-to-spot gather_fusion.22, 9 times under while.489 (9 boxes); the first slice, copy.2136, also appears 9 times. The loop body repeats once per iteration → 9 iterations. All 936 small whiles in this step run 9 iterations, with 72 leaf ops each
④Source code, upper: the container’smegablox/ops.py:106: the small while’s own source, same start and end as ②, in the upper row
⑤Source code, lower: each leaf’sRead a leaf’s origin here; the slice above only says which line the whole while came from
  • Counting iterations on the trace: the sub-level is the loop body laid out in order, once per iteration, so count any slice of it under its while (the figure boxes gather_fusion.22). The first, copy.2136, appears 9 times under while.489 = 9 iterations. By the same count, forward while.457 and backward while.453 both run 78.
  • 958 whiles in one step: 22 outermost (the forward and backward main loops plus 20 small whiles outside them) and 936 inside them = 78 iterations × (3 per forward layer + 9 per backward layer). All 936 run 9 iterations.
  • Where the small whiles come from: their op_name in the compiled HLO is jit(_searchsorted_scan_impl), inside the Tokamax expert matmul’s entry point: the jnp.searchsorted that finds each token’s position by expert. By default it binary-searches with scan, in ⌈log₂(n+1)⌉ iterations (JAX source jax/_src/numpy/hijax.py). This while’s state holds an f32[449]; n = 449 gives exactly 9, matching the count on the trace [inferred: this f32[449] is the array being searched].
  • Source code stacks too: the small while’s source takes the upper row (④), its leaves’ the lower (⑤), so read a leaf’s origin from the lower row only. while.457 has no source or tf_op, so its column’s lower row is empty (⑥ in Figure 2.3C).
Open it yourself (same screen in local open-source XProf; first time, see “Opening it locally” in §2.1)
  • Reading down a column: run fp8old, view_start=25,494.800, view_end=25,501.400. Expand All opens the Framework Name Scope, Framework Ops and Source code rows under TPU:0; click gmm_megablox.16 and check tf_op and source in Details.
  • The three small whiles: run fp8old, view_start=25,493.375, view_end=25,493.645 (0.27 ms; names show as ga; click for full names).

2.4 The same layer side by side: what the SparseCore row lacks

⭐ In modified layer 40 the SparseCore’s all-gather of the 192 experts is gone, as is the backward expert-gradient reduce-scatter. The expert matmuls shrink from 10.1 to 0.1 ms.

A whole step (11.76 s) is too wide for op names; zoom to one layer (tens of ms). Zooming 300-odd× with W from a full step drifts; a window written into the link is most reliable (see Open it yourself).

Both runs show layer 40 at the same window width, each left-aligned to the layer’s start.

Figure 2.4A · FP8 Original · Forward, same layer (MoE layer 40). ① Expert matmuls ×3; ② All-gather of the 192 experts
Original, forward: ① three expert matmuls of over 3 ms each; ② three expert all-gathers on the SparseCore.
#WhatHow to tell (names and times from the xplane)
①Expert matmuls ×3TC: the layer’s 3 expert matmuls (gmm) total 10.11 ms: 3.43/3.33/3.35 ms
Two boxes in the main image: gmm_megablox.16 + gmm_megablox.17 (from 25,497.812 ms, 3.425/3.328 ms); gmm_megablox.18 (from 25,505.087 ms, 3.354 ms)
②All-gather of the 192 expertsSC0: all-gathers 573/575/577 of the 192 experts’ weights, 2.38/3.98/2.21 ms; shapes like f8e4m3fn[1,192,4096,1536]
Two boxes in the main image: all-gather.573 (from 25,477.978 ms, 2.380 ms); all-gather.575 + all-gather.577 (from 25,496.737 ms, 3.981/2.209 ms)
BracketLayer: 34.68 msWindow = this layer plus 0.5 ms each side; the modified image has the same width, left-aligned to its own layer
Figure 2.4B · FP8 Modified (50% faster) · Forward, same layer (MoE layer 40). ① Expert matmuls ×3; ② No expert all-gather here
Modified, forward: ① the three expert matmuls are down to only 0.04 ms each; ② no expert all-gather on the SparseCore in that stretch.
#WhatHow to tell (names and times from the xplane)
①Expert matmuls ×3TC: the layer’s 3 expert matmuls (gmm) total 0.11 ms: 0.04/0.04/0.04 ms; boxes a few pixels wide (original, same layer: 10.11 ms)
Two boxes in the main image: gmm.27 + gmm.26 (from 9,404.251 ms, 0.037/0.037 ms); gmm.28 (from 9,405.287 ms, 0.035 ms)
②No expert all-gather hereDashed: layer start to the first gmm, the original’s ② slot. Here SC0 has no expert-weight all-gather, only 5 other small all-gathers (none the full [1,192,…] expert set, each ≤0.18 ms) and gather_offload (gathering tokens by route)
Zoom 1Window 9,404.18–9,404.42 msgmm.27 (0.037 ms), gmm.26 (0.037 ms); SC0 has no events in this span, so its row is absent
Zoom 2Window 9,405.18–9,405.42 msgmm.28 (0.035 ms)
BracketLayer: 25.59 msGray dashed line: end of this layer; to its right is the next layer (A and B share one window width = the original’s layer)

What all-gather does here: each device stores only 1/64 of the 192 experts and, when needed, gathers all of them from the 64 devices in the FSDP group. The original’s three are shaped like f8e4m3fn[1,192,4096,1536]: the leading 192 = all 192 experts. In the modified run that stretch is empty but for a few tiny all-gathers (none covering the 192 experts) and gather_offload moving tokens per expert.

Figure 2.5A · FP8 Original · Backward, same layer (MoE layer 40). ① Expert matmuls ×9; ② All-gather of the 192 experts; ③ Expert-gradient reduce-scatter
Original, backward: plus ③, the expert-gradient reduce-scatters, about 10 ms each.
#WhatHow to tell (names and times from the xplane)
①Expert matmuls ×9TC: the layer’s 9 expert matmuls (gmm/tgmm) total 35.69 ms
Five boxes in the main image: gmm_megablox.14 + gmm_megablox.13 + gmm_megablox_transpose_rhs.10 (from 31,226.184 ms, 3.530/2.982/3.631 ms); gmm_megablox.15 (from 31,237.982 ms, 3.357 ms); gmm_megablox_transpose_rhs.12 + gmm_megablox_transpose_rhs.11 (from 31,245.398 ms, 4.050/4.043 ms); tgmm_megablox.10 + tgmm_megablox.11 (from 31,255.554 ms, 4.522/3.911 ms); tgmm_megablox.12 (from 31,265.379 ms, 5.660 ms)
②All-gather of the 192 expertsSC0: all-gathers 543/547/545 of the 192 experts’ weights, 2.37/2.41/2.21 ms; shapes like f8e4m3fn[1,192,4096,1536]
Two boxes in the main image: all-gather.543 (from 31,200.121 ms, 2.370 ms); all-gather.547 + all-gather.545 (from 31,223.777 ms, 2.406/2.207 ms)
③Expert-gradient reduce-scatterSC0: expert-gradient reduce-scatters 12/14/16, 10.45/10.25/9.13 ms; shapes like bf16[3,4096,1536]
Two boxes in the main image: reduce-scatter.12 (from 31,260.076 ms, 10.447 ms); reduce-scatter.14 + reduce-scatter.16 (from 31,270.953 ms, 10.246/9.134 ms)
BracketLayer: 106.12 msWindow = this layer plus 0.5 ms each side; the modified image has the same width, left-aligned to its own layer
Figure 2.5B · FP8 Modified (50% faster) · Backward, same layer (MoE layer 40). ① Expert matmuls ×9; ② No expert all-gather here; ③ No gradient reduce-scatter here
Modified, backward: ② and ③ both empty.
#WhatHow to tell (names and times from the xplane)
①Expert matmuls ×9TC: the layer’s 9 expert matmuls (gmm/tgmm) total 0.47 ms; 0.03–0.08 ms each, boxes a few pixels wide (original, same layer: 35.69 ms)
Five boxes in the main image: gmm.20 (from 13,233.506 ms, 0.060 ms); gmm.22 + gmm.21 (from 13,234.596 ms, 0.037/0.037 ms); gmm.25 (from 13,236.223 ms, 0.035 ms); gmm.24 + gmm.23 (from 13,237.125 ms, 0.036/0.036 ms); tgmm.10 + tgmm.9 + tgmm.8 (from 13,246.101 ms, 0.076/0.079/0.079 ms)
②No expert all-gather hereDashed: layer start to the first gmm, the original’s ② slot. Here SC0 has no expert-weight all-gather, only 5 other small all-gathers (none the full [1,192,…] expert set, each ≤0.16 ms) and gather_offload (gathering tokens by route)
③No gradient reduce-scatter hereDashed: from tgmm finishing the expert gradients to the layer end, where the original’s ③ sits. Here SC0 has not a single reduce-scatter, only 3 all-reduces (longest 0.56 ms) and 4 small all-gathers
Zoom 1Window 13,246.04–13,246.39 mstgmm.10 (0.076 ms), tgmm.9 (0.079 ms), tgmm.8 (0.079 ms)
The zoom shows only some① has 9 in all; the zoom boxes 3. The others are likewise tens of µs each; see row ①
BracketLayer: 66.20 msGray dashed line: end of this layer; to its right is the next layer (A and B share one window width = the original’s layer)

reduce-scatter is the reverse: finished gradients are summed across the 64 devices in the FSDP group, and each keeps only its own share. The original’s three, in backward, have shapes like bf16[3,4096,1536]: each device keeps its 3 experts’ gradients (192 ÷ 64 = 3). The modified backward has none at all.

The screenshots box only the longest. Listing all SparseCore collectives in layer 40 in order shows if only these two kinds are missing:

Layer 40 collectives in order: Modified lacks only ④ expert-weight gathers and ⑤ expert-gradient scatters Cell = run of same-kind SparseCore collectives: count, time (④⑤ each, rest total) · above cell: ms from layer start · dashed: Original only ①②③ attention/router/shared weights ④ expert-weight all-gather ⑤ expert-grad reduce-scatter ⑥⑦ gradient all-reduce Forward (layer 34.68 → 25.59 ms) A Original 11 ops 1 ×4 0.50 ms 0.0 4 ×1 2.38 ms 0.5 2 ×1 0.09 ms 3.0 4 ×2 3.98/2.21 ms 19.3 3 ×3 0.30 ms 32.9 B Modified 8 ops 1 ×4 0.42 ms 0.0 4 ×1 absent 2 ×1 0.05 ms 0.5 4 ×2 absent 3 ×3 0.28 ms 23.9 Backward (layer 106.12 → 66.20 ms; it first recomputes the forward, so weights are gathered again) A Original 22 ops 1 ×4 0.33 ms 0.0 4 ×1 2.37 ms 0.4 2 ×1 0.08 ms 2.8 4 ×2 2.41/2.21 ms 24.0 1 ×1 0.18 ms 44.6 3 ×3 0.23 ms 44.8 5 ×1 10.45 ms 60.3 1 ×3 0.28 ms 70.8 5 ×2 10.25/9.13 ms 71.2 6 ×3 1.01 ms 97.7 7 ×1 0.29 ms 101.2 B Modified 17 ops 1 ×4 0.42 ms 0.0 4 ×1 absent 2 ×1 0.05 ms 0.5 4 ×2 absent 1 ×1 0.28 ms 33.2 3 ×3 0.19 ms 33.5 2 ×1 0.09 ms 36.6 5 ×1 absent 1 ×3 0.32 ms 37.7 5 ×2 absent 7 ×1 0.40 ms 37.7 6 ×3 1.11 ms 60.1 Not drawn: 4 tiny synchronous all-reduces on the TensorCore (each under 0.25 ms), in both runs. Neither program has any all-to-all or collective-permute: tokens never leave the device. All other kinds match (Modified backward gathers router weights once more); only ④ and ⑤ differ, the longest ops in the layer SparseCore comms per layer, Original: fwd 9.46 ms, bwd 39.22 ms; Modified: 0.75 ms, 2.85 ms. ⑦ is in both: Modified still sums expert grads over the 2 data-axis replicas, only without ⑤ first.
Dashed cells first: only orange ④ and red ⑤ are in the original but not the modified run. The rest, kind by kind: same counts, all a few tenths of a ms.
Sources and assumptions

📌 Windows: layer 40, forward fp8old 25,477.463–25,512.140, fp8new 9,384.636–9,410.223 ms; backward fp8old 31,199.767–31,305.887, fp8new 13,209.876–13,276.078 ms (from profile start; as in Figures 2.4 and 2.5). Counted: execution bars named with all-gather/reduce-scatter/all-reduce on the Sparse Core Ops rows of TPU:0’s two SparseCores; a same-named all-reduce on both counts once.

📌 Which op carries which weight, matched by shape against the config: [1,4096,64,128], two [1,4096,8,128], [1,64,128,4096] = attention (64 query heads, 8 KV heads, 128 per head); [1,4096,192] = router (192 experts); [1,4096,1536] ×2, [1,1536,4096] = shared expert (one, intermediate size 1,536); [1,192,4096,1536] ×2, [1,192,1536,4096] = the fully gathered expert weights. All f8e4m3fn.

📌 Reduction scope, from replica_groups in the final HLO: ⑤ reduce-scatter over {'fsdp'} (64 shards); ⑦ expert-gradient all-reduce (original all-reduce.384, modified .394, shapes bf16[3,4096,1536] ×2 + [3,1536,4096]) over {'data'} (2 replicas); the three in ⑥ (router f32[4096,192], attention, shared expert) span all 128 devices.

📌 The modified backward adds one router-weight all-gather (.499, 0.085 ms): the same weights are gathered for the recompute and again for the gradient, a compiler scheduling choice unlike the original’s [inferred]. The TensorCore’s 4 synchronous all-reduces: original all-reduce.306/307/308/312, modified .316/317/318/322 (f32[192] and a few small bf16 vectors).

#What is gathered or summedShape (f8e4m3fn gathered / bf16 gradient)Ops per layer: Original → Modified
①Attention’s four weights (q, k, v, output)[1,4096,64,128], [1,4096,8,128] ×2, [1,64,128,4096]Fwd 4 → 4, bwd 8 → 8
②Router weights[1,4096,192]Fwd 1 → 1, bwd 1 → 2
③Shared expert’s three weights[1,4096,1536] ×2, [1,1536,4096]Fwd 3 → 3, bwd 3 → 3
④The 192 experts’ weights[1,192,4096,1536] ×2, [1,192,1536,4096]Fwd 3 → 0, bwd 3 → 0
⑤Expert-gradient reduce-scatterbf16[192,…] → [3,…]Bwd 3 → 0
⑥Gradient sum for attention, router, shared expert (all 128 devices)three all-reducesBwd 3 → 3
⑦Expert-gradient sum (data axis, 2 replicas)bf16[3,4096,1536] ×2, [3,1536,4096]Bwd 1 → 1

Apart from ④ and ⑤, both runs match kind for kind, and neither program has a single all-to-all: tokens never leave the device (the gather_offload moving tokens per expert is the SparseCore moving data in-device). A new question: ⑦ is in both. The original sums first over 64 shards (⑤), then over 2 (⑦); the modified run has no ⑤, so its expert gradients are summed over only 2. Note it; Section 4 returns to the program.

One layer: four leaf kinds, only two worth watching

⭐ Zoomed to one layer, XLA Ops level 2 is a long run of leaves: a few big labeled ones, the rest a mass of slivers. Should you click each one?

Figure 2.4C · Zoomed to one layer: four kinds of leaf ops; watch only the widest compute and the dones with width (Original, forward, layer 40). ① Big compute; ② narrow small compute (zoom window); ③ data movement; ④ collectives; ⑤ small whiles (containers, not leaves)
Top, one layer: ① big compute, ③ data-movement dones, ④ the only communication done with width. Bottom, the 0.65 ms between the gray dashed lines, zoomed: ② a mass of slivers, ⑤ small whiles nested in the layer.
#WhatHow to tell (classified by xplane event name; counts and ms computed by r12-trace.py)
①Big computePallas kernels (attention splash_mha… 9.89 ms; three expert matmuls gmm_megablox.16/17/18, 3.3–3.4 ms each) and compiler-generated matmuls, convolution fusion (9; the widest 3 boxed). This kind: 13 ops, 23.70 ms, 68.3% of the layer
②Narrow small compute (zoom window)Elementwise, type conversion, norm, routing compares and sorts, slicing: loop fusion, reduce-window, slice, sort and the like. 315 ops, 6.79 ms, 19.6% of the layer. Layer-wide, 504 leaves are under 0.1 ms (all four kinds), totaling just 0.98 ms (2.8%). Widest in the zoom window: sort.284 (0.201 ms, the green box in the middle)
③Data movementgather_offload_async_done.6/.7: the TensorCore stalls here while the SparseCore moves tokens per expert (1.43/1.87 ms). Also copy-start/copy-done, relayouts like copy (data formatting; e.g. the copy.2030 block at the zoom window’s left), scatter_offload and others: 165 ops, 3.97 ms, 11.4% of the layer
④Collectivesall-gather.570…call-done, 0.22 ms: the layer’s only collective done with width. Leaves named all-gather/reduce-scatter/all-reduce: 43 ops, 0.22 ms, 0.6% of the layer, almost all zero-width starts, dones and helper instructions around the issue
⑤Small whiles (containers, not leaves)while.489/488/487, about 80 µs each: loops nested in the layer, leaves one level below (Figure 2.3D counts their iterations)

Sort this layer’s 536 leaf ops by name into the four kinds above (XLA files matmuls under convolution fusion, so they count as ①) and total each:

Layer 40Layer ms① Big compute② Small compute③ Data movement④ CollectivesOf which <0.1 msAsync dones ≥1 µs wide
Original, forward34.6813 ops, 23.70 ms (68.3%)315 ops, 6.79 ms (19.6%)165 ops, 3.97 ms (11.4%)43 ops, 0.22 ms (0.6%)504, 0.98 ms4 / 27
Original, backward106.1236 ops, 72.13 ms (68.0%)875 ops, 24.68 ms (23.3%)485 ops, 7.19 ms (6.8%)92 ops, 2.11 ms (2.0%)1403, 3.42 ms14 / 63

⭐ 13 big compute ops take 68.3%; 504 slivers under 0.1 ms, just 2.8%. Only 4 of 27 async dones have width (the rest: zero width, data came early). Watch only two things: the widest compute and the dones with width.

So, three steps. Orient at the top level: click a leaf in the big loop and read tf_op: a second path segment of jvp() means forward, transpose(jvp()) backward. Find the widest compute (① above). Find the dones with width (③ ④). To place a sliver in its module, read down a column (Figure 2.3C). The trace answers “where, and next to what”; for “a kind’s total per step”, read Op Profile / HLO Op Stats (Section 3), not slice sums on the trace.

How the kinds are assigned; the same layer, modified (definitions and four numbers)
  • Definitions: a leaf = any non-while event in this iteration’s (layer 40’s) time range, classed by name and HLO category. Names starting with gmm/tgmm/splash, or category convolution fusion, go to ①; names containing a collective go to ④; copy-start/copy-done, category data formatting, names containing gather_offload/scatter_offload/sparse-core-data-format, and dynamic-slice/dynamic-update-slice starts and dones go to ③ (plain copy ops of category data formatting too); the rest go to ②. Custom-calls named prepare_… follow the communication or transfer in their name; all under 1 µs wide.
  • Why the slivers are so narrow: the layer’s 504 leaves under 0.1 ms have a median of 0.94 µs each and a median compile-time byte count (reads + writes) of only 2,308 bytes (57 record 0); the 32 blocks over 0.1 ms, a median of 683,204,608 bytes. Booked bytes ÷ duration: a median of about 2 GB/s for slivers, about 1,598 GB/s for wide blocks. Width tracks data volume; a sliver’s time is mostly per-instruction fixed start/finish cost [inferred: from comparing bytes and durations; overhead not measured separately]. So hundreds of slivers total under 3% of a layer: not worth clicking one by one.
  • The leaves add up to ≈ the layer: forward 34.67/34.68 ms, backward 106.11/106.12 ms. The TensorCore has almost no gaps in this layer, matching the Overview’s 99.9% Duty Cycle.
  • Counting dones: forward, 61 dones = 27 async dones + 34 copy-done (each under 1.1 µs); of the async dones with width, only 1 is a collective. Backward, 14 of 63 async dones exceed 1 µs; the widest are a dynamic-slice-done at 4.68 ms and the all-gather.548…call-done from 2.5 at 1.18 ms.
  • The same layer, modified:
Layer 40Layer ms① Big compute② Small compute③ Data movement④ CollectivesOf which <0.1 msAsync dones ≥1 µs wide
Modified, forward25.5913 ops, 13.68 ms (53.5%)262 ops, 7.80 ms (30.5%)90 ops, 3.92 ms (15.3%)32 ops, 0.18 ms (0.7%)369, 1.14 ms4 / 23
Modified, backward66.2036 ops, 37.27 ms (56.3%)872 ops, 25.56 ms (38.6%)485 ops, 2.71 ms (4.1%)72 ops, 0.66 ms (1.0%)1392, 3.85 ms12 / 57

Modified: ① is still 13/36 ops (names still start with gmm), but each expert matmul is down to tens of µs; ①’s share falls from 68.3% to 53.5% forward, 68.0% to 56.3% backward. The kind table shows the same work got thinner, not why; for why, see 2.4 and Section 3.

Open it yourself (same screen in local open-source XProf; first time, see “Opening it locally” in §2.1)
  • One layer: run fp8old, view_start=25,476.963, view_end=25,512.640 (as in 2.4).
  • Zoom window: run fp8old, view_start=25,493.300, view_end=25,493.950; click sort.284 and gather_offload_async_done.6 (one-layer screen); read Wall Duration.

⭐ The modified layer’s SparseCore lacks two kinds of communication; expert matmuls shrink to 1/75–1/93: a big chunk of work is gone. Is all the missing 3.9 s this undone work? Section 3 checks the totals.

Open it yourself (same screen in local open-source XProf; first time, see “Opening it locally” in §2.1)
  • Forward, layer 40: fp8old view_start=25476.963, view_end=25512.640 ↗ Open; fp8new 9384.136–9419.813 ↗ Open. Modified, zoom windows: 9404.183–9404.423 ↗ Open, 9405.183–9405.423 ↗ Open.
  • Backward, layer 40: fp8old 31199.267–31306.387 ↗ Open; fp8new 13209.376–13316.496 ↗ Open; modified zoom window 13246.043–13246.393 ↗ Open.
  • Click gmm_megablox.16 (TensorCore) and all-gather.575 (SparseCore); read Wall Duration (ns) in Details.

2.5 Example: is the original’s communication hidden?

⭐ Per step the original’s TensorCore waits on communication dones only 161 ms (1.4%): compute hides most transfers. The modified recipe saves just 119 ms of that wait, far short of 3.9 s.

Figure 2.6 · Original, backward, layer 40: one exposed done. ① Done on the TC; ② execution bar on SC0; ③ hidden part; ④ exposed part
The dashed line joins the right edges of ① and ②; then read the two brackets.
#WhatHow to tell (names and times from the xplane)
①Done on the TCTC: all-gather.548.cloned.1.call-done, 31,225.002–31,226.183 ms, 1.181 ms
②Execution bar on SC0SC0: all-gather.547 (one below the TC’s number), 31,223.777–31,226.183 ms, 2.406 ms, shape f8e4m3fn[1,192,4096,1536]: a gather of all 192 experts’ weights
③Hidden partSC starts moving → TC reaches done: 1.225 ms, overlapped with TC compute
④Exposed partDone width 1.181 ms: the TC stalls until the SC finishes. Dashed line: right edges 0.1 µs apart

The static figure shows only the outcome. Replayed from start: the SparseCore row is still busy, so 547 starts moving over 4 ms late; the TensorCore keeps computing, reaches done before the transfer ends, and waits.

Figure 2.6’s transfer, start to finish (xplane times; zero = TensorCore start): start issues → SparseCore starts 4.27 ms late → TensorCore reaches done, released only when the transfer ends. Autoplays by default; untick “Autoplay” to step; stops 1–6 match the table.
StopWhat happensReading
10 ms: TensorCore issues start, keeps computingStart has almost no width
2SparseCore row still moving tokens per expert (two gather_offload); 547 starts only at 4.27 msStarts only at 4.27 ms (why the two ahead block it: [inferred: the SparseCore row runs one at a time])
3SparseCore moves 547; TensorCore quantizes1.23 ms hidden
4TensorCore reaches done; 547 unfinished1.18 ms exposed: done and 547 right edges 0.1 µs apart
5Released, runs gmm_megablox.14Its operand 7 is the done of all-gather.548 (HLO)
6Per-step totalTensorCore wait on communication dones 161.0 → 41.6 ms: 119 ms saved, far short of 3.9 s
Reading start/done: hidden vs exposed (this transfer and the whole step)
  • An async collective leaves two markers on the TensorCore: start issues it (almost zero width); the SparseCore moves the data. Done is where the TensorCore needs it: if arrived, done is zero-width; if not, the TensorCore stalls, and done’s width is exactly its idle time.
  • Pairing: the SparseCore bar’s number is one below the TensorCore done’s (all-gather.547 ↔ all-gather.548…call-done), and their right edges align: the TensorCore’s wait ends as the SparseCore finishes. Here the gap is 0.1 µs.
  • This transfer: 2.41 ms of SparseCore moving: the first 1.23 ms the TensorCore computed other work (hidden), the last 1.18 ms it stalled on done (exposed).
  • The whole step: in the median step the TensorCore waits on collective dones 161.0 ms in total (2,658 dones, 2,339 under 1 µs wide); modified: 41.6 ms. Another estimate (SparseCore communication not covered by TensorCore compute): 195.9 ms original, 67.6 ms modified, same order of magnitude.
  • Slow transfer, or stuck in a queue? Fully gathered, 547 is 192 × 4,096 × 1,536 FP8 values = 1.21 GB, moved in 2.406 ms: about 0.50 TB/s. Same-shaped 543 (this layer) and 573 (forward layer 40) both take 2.37–2.38 ms (about 0.51 TB/s): same speed, so this one isn’t slow. Just before it on the SparseCore row sit two gather_offload ops; 547 starts 3 µs after the previous one ends: it looks like a queue [inferred: the SparseCore row runs one at a time]. Start is at 31,219.51 ms, left of the Figure 2.6 window (the animation’s time zero).
  • So the root cause here: the communication itself is gone. The 2.4 transfers were mostly hidden under compute in the original, and never happen in the modified run.
Open it yourself (same screen in local open-source XProf; first time, see “Opening it locally” in §2.1)
  • run fp8old, view_start=31222.3, view_end=31227.3 ↗ Open; click all-gather.548.cloned.1.call-done and press m to mark its span on the ruler: its right edge lines up with all-gather.547 on the SparseCore.
Section 3

Reconcile the books: is the missing time work that was never done?

📝 Notes for this section

In one layer, Section 2 saw the SparseCore’s expert gather and gradient scatter vanish and expert matmuls shrink to 1/75–1/93. But a layer is one of 78 iterations. Is the step’s missing 3.9 s all undone work? Check on two pages that report totals: Op Profile by category, Op Stats by op.

3.0 The route at a glance

⭐ One minute: Op Profile by category, both runs → expand custom-call → click for details → Op Stats: rank, count.

Real pages of local open-source XProf 2.23.1 (DPR2 captures), boxed at page text. Autoplays by default; untick “Autoplay” to step; stops 1–7 match the table.
StopPage, actionReading
1Original, HLO Op Profile, Group by: CategoryOverall FLOPS utilization 20.20%; custom-call 55.44% of time, 53.39% utilization
2Modified, same pagecustom-call 37.86% of time, utilization 117.45%, WASTED 0.0%
3Click custom-call to expandAttention, three groups: 34–40%, as in the original; gmm.27 group 5,147.80%, tgmm.8 group 2,703.29%
4Click the gmm.27 row; read the left panel49.31 µs per call, 58.53 PFLOP/s: 25× FP8 peak
5Original, expanded the same waySame expert matmuls: 2.4–8.8% of time per group, 45–77% utilization
6Modified, HLO Op Stats, name filter gmmFirst expert matmul ranks 124th; 1,872 calls; 94.5 µs per call
7Original, same filterRanks 4th; the same 1,872 calls; 5,692 µs per call

Full video: topic07-xrec-opprof.en.mp4 (last frame png). Below, stop by stop.

3.1 Op Profile: which category shrank?

⭐ Nearly all the missing time is in custom-call: 6.5 s → 3.0 s per step. Yet its utilization rises instead, to 117.45%.

Figure 3.1 · HLO Op Profile by category: Original / Modified (control row omitted). ① Overall FLOPS utilization; ② the custom-call category; ③ custom-call FLOPS
Same custom-call row: ② time share down, ③ utilization more than doubled.
#WhatOn the page (boxes at the on-page text)
①Overall FLOPS utilizationOverall TPU FLOPS utilization: original 20.20%, modified 30.70%
②The custom-call categoryPallas kernels: TIME 55.44% → 37.86% (in ms, Op Stats basis, mean of 8 TensorCores × 3 steps: 6532.9 → 2969.9 ms per step); WASTED 25.84% → 0.0% (wasted = time share × (1 − max(FLOPS utilization, HBM utilization)); above 100% utilization it is scored as no waste at all)
③custom-call FLOPSCategory FLOPS utilization 53.39% → 117.45%, above the page’s own peak

HLO Op Profile buckets every TensorCore instruction by category, with two numbers each: TIME % (share of TensorCore time) and FLOPS (utilization; formula below). custom-call is an external kernel XLA calls rather than generates: here, hand-written Pallas kernels, both attention (splash) and the expert matmuls (gmm: forward and input gradient; tgmm: expert-weight gradient; glossary in 0.3). custom-call alone loses 3.6 s per step (6,532.9 → 2,969.9 ms, table ②); the rest together barely move.

The odd part is the third column: time fell by over half, yet utilization rose from 53.39% to 117.45%. Utilization above 100% means more operations per second than the page’s own peak. Don’t conclude yet; expand it.

How Op Profile computes FLOPS and WASTED (two formulas, matching the page exactly)
  • FLOPS utilization = booked FLOPs ÷ (measured time × peak). Measured time is from the trace. Booked FLOPs are not measured: the compiler counts them from shapes for XLA-generated ops; a Pallas kernel’s author reports them in code (pl.CostEstimate, stored in the compiled HLO as cost_estimate).
  • This page’s peak, 1,137.15 TFLOP/s per core = the Roofline page’s 1,028.75 TFLOP/s × Time Scale Multiplier 1.10537. Time Scale Multiplier is a conversion factor atop the Roofline page: each op event here carries 1.15789 or 1.0 (about 2:1 by duration); weighted by duration, 1.10537 (per the tooltip of the Apply Dvfs Scaling Factor toggle at the top, it scales to the device’s performance state at the time; switch it on and the Roofline peak becomes 1,137.15, as here). Stop 4’s left panel yields the same number: 58.53 PFLOP/s ÷ 5,147.80%.
  • Check it yourself: all three FP8 profiles book the same 3,966.1 TFLOP per step per core on custom-call (broken down in the next subsection). Original custom-call, measured 6,541.8 ms → 53.31%; modified, 2,972.8 ms → 117.32%. The page: 53.39% and 117.45%, less than 0.2 percentage points apart (we use TPU:0’s 3 steps, the page 8 cores).
  • WASTED = time share × (1 − max(FLOPS utilization, HBM utilization)): time is wasted only if spent without saturating the hardware. Above 100% utilization the bracket goes negative and the page shows 0.0%: a kernel that did no work scores as wasting nothing at all.
Open it yourself (same screen in local open-source XProf; first time, see “Opening it locally” in §2.1)

3.2 117.45%: an impossible utilization

⭐ Two expert-matmul kernels drag the utilization up. The gmm.27 group takes 49.31 µs per call, i.e. 58.53 PFLOP/s: 25× the FP8 peak. No chip can do that; the kernel did not do all the work it reports.

Figure 3.2 · Modified, custom-call expanded: which kernels push utilization past 100%. ① Three attention kernels; ② the gmm.27 group; ③ the tgmm.8 group; ④ Total Time Avg; ⑤ FLOPS utilization; ⑥ FLOP rate
Read the FLOPS at ①②③ on the right first, then ④⑤⑥ in the left panel, which details the gmm.27 group.
#WhatOn the page (boxes at the on-page text)
①Three attention kernelsThree splash_mha_dkv/fwd groups: FLOPS 40.23%, 34.24%, 34.24%, as in the original (Op Stats per call: 15,788/9,886 µs original, 15,773/9,885 µs modified)
②The gmm.27 groupgmm.27 and its duplicate(s): TIME 0.20%, FLOPS 5,147.80% (expanded: gmm.27/22/26/21, 5,134–5,158% each)
③The tgmm.8 grouptgmm.8 and its duplicate(s): FLOPS 2,703.29%
④Total Time AvgLeft panel: this group averages 49.31 µs per call
⑤FLOPS utilizationLeft panel: 5,147.80%. HBM bandwidth utilization is also 100.00% (next line: 33.73 TB/s vs a per-core peak of 3,433 GiB/s ≈ 3.69 TB/s, also impossible)
⑥FLOP rateLeft panel: 58.53 PFLOP/s = self-reported 2.886 TFLOP per call ÷ 49.31 µs. The bf16 normalized rate is also 58.53: Pallas kernels aren't rescaled by precision

Expanded, custom-call's time goes mostly to three attention kernels at 34–40% utilization, nearly as in the original recipe. The absurd rows are two expert matmuls with only 0.2% of the time (②③). The 117% is an average: a big honest attention block (about 37%) plus a small absurd expert-matmul block (about 2,500%) lands above 100%.

Why it is impossible. The numerator is self-reported: the compiled program records 2,886,218,022,912 FLOPs per gmm.27 call = 2 × 229,376 × 4,096 × 1,536. The 229,376 rows are this device's tokens (7 × 4,096), each routed to 8 experts; 4,096 and 1,536 are an expert matrix's two dimensions. The count comes purely from input shapes, however many experts were actually computed. The denominator is measured: 49.31 µs per call, so 58.53 PFLOP/s. TPU v7's official FP8 peak is 4,614 TFLOP/s per chip, 2,307 TFLOP/s per core (2 TensorCores per chip): 25× over. HBM likewise: 33.73 TB/s against about 3.69 TB/s per core (⑤), 9× over.

Same kernel, another page: the Roofline Model page plots each op as a point on §1's roofline. A kernel doing real work at best touches the roof; gmm.27 lands off the chart (recording stop 3).

Roofline page, local open-source XProf 2.23.1 (DPR2 screenshots). Boxes at the on-page text, the blue dot at its pixel position; the table's Program ID column (a program fingerprint) hidden before capture. Stops 1–4 match the table below.
StopActionWhat you see
1Modified, Roofline page, Device Information at the topPeak 1,028.75 TFLOP/s per core; Time Scale Multiplier 1.10529 (Op Profile's peak ≈ the product)
2Original, down to Section2, type gmm_megablox.16 in the Operation boxOne blue dot on the flat roof; 840,976 GFLOP/s, 81.7% of peak
3Modified, same box, type gmm.27No dot: 58,380,995 GFLOP/s is 57× above the flat roof, off the chart; the table shows 5,674.9%, memory bandwidth 912.6%
4Just that table row58,380,995 ÷ 1,028,750 = 56.7×; code at MaxText backend.py (modified branch)

⭐ Same FLOPs booked, less work done: both FP8 runs book identical custom-call FLOPs, yet its time fell from 6,541.8 ms to 2,972.8 ms. Utilization = same numerator ÷ shrunken denominator, hence the 117%.

Per step, per TensorCoreOriginalModifiedSource
custom-call booked FLOPs (TFLOP)3,966.13,966.118 cost_estimate (hlo_proto) × TPU:0 calls/step
 of which expert matmul / attention2,762.1/1,204.02,762.1/1,204.0
custom-call measured time (ms)6,541.82,972.8TPU:0, mean of 3 steps
Utilization on the page53.39%117.45%Op Profile
One expert matmul: time per call3,431.1 µs (gmm_megablox.16)48.6 µs (gmm.27)Same
 self-reported 2.886 TFLOP ÷ that time841 TFLOP/s59,387 TFLOP/sArithmetic
If FP8 is twice as fast as BF16, why isn't “over 100%” enough on its own? (117% is an alarm; the single kernel is the evidence)
  • The page's 1,137 TFLOP/s per-core peak is the BF16 tier [inferred: the same page rescales XLA's FP8 matmuls by half, see the next bullet] (official BF16: 2,307 TFLOP/s per chip, 1,153.5 per core); FP8: 2,307 per core. It doesn't rescale Pallas kernels by precision (left panel: bf16 normalized FLOP rate = FLOP rate = 58.53 PFLOP/s), so a saturated FP8 Pallas kernel could in theory show around 200%.
  • Control: for FP8 matmuls XLA generates itself, Op Stats shows Model GFLOP/s up to 2,235,944 (≈ 2,236 TFLOP/s, right at the FP8 peak), and Normalized, rescaled to BF16, 1,154,733. The page rescales those; it cannot know what precision runs inside a Pallas kernel.
  • So 117% is an alarm, not evidence; the evidence is the single-kernel row, 25× even the FP8 peak, impossible at any precision.
  • The 5,674.9% at Roofline stop 3 is about 1.1× Op Profile's 5,147.80%: the Roofline page divides by 1,028.75 directly, while Op Profile also multiplies by the Time Scale Multiplier.
  • This page's peak of 1,028.75 is more than 10% below the official BF16 1,153.5 per core. Times 1.10537 (1.10529 modified) it is 1,137, still 1.4% short. We have not verified how the two relate.

The table's per-call time for one expert matmul is a 3-step mean on TPU:0 (gmm.27: 48.6 µs); the left panel's 49.31 µs averages the group over 8 cores: 1–2% apart.

Open it yourself (same screen in local open-source XProf; first time, see “Opening it locally” in §2.1)
  • On the §3.1 page, click the custom-call row to expand it, then gmm.27 and its duplicate(s): the left panel shows ④⑤⑥.
  • Roofline recording: tool roofline_model, in Section2 type gmm.27 in the Operation box (original: gmm_megablox.16), press Enter.
  • “and its duplicate(s)”: the call sites of one kernel (gmm.27/22/26/21…) grouped; click ▶ to expand further, up to Ops Limit at top right (default 10).

The Roofline page: an inflated step average, and which roof each op sits on

⭐ The step average is inflated too: Roofline Section1 shows 43.4% → 65.2%. Same cause as the 117%: both runs book nearly equal FLOPs per step (5,261 vs 5,267 TFLOP) in a step 66.6% as long. Without expert matmuls the rest goes 29.9% → 31.4%, barely moving.

Recall §1's roofline: arithmetic intensity (operations per byte moved) on x, operations per second on y. The left slope is the bandwidth limit, the flat right part the compute limit; they meet at its ridge point. XProf's Roofline page draws it over a profile: Section1 plots the whole program as one point, Section2 each of the 1000 most expensive ops. The y numerator is booked FLOPs (same source as Op Profile), the denominator measured time. That is why gmm.27 (§3.2) can land off the chart, and the step average cannot escape the same problem. Section1 first, the two runs stacked:

Figure 3.2B · Roofline Section1 (whole program): Original / Modified. ① Peak (denominator); ② where the step lands; ③ Total (HW); ④ Normalized FLOP Rate; ⑤ Bound by; ⑥ HBM BW; ⑦ FLOP Rate / Peak
Top vs bottom: ④ FLOPs per second up by half, ⑦ utilization 43.4% → 65.2%; ② the point rises, its x barely moves. Right: zoom of area ②; red dashed line = ridge point.
#WhatOn the page (boxes at the on-page text; points read off the axes)
①Peak (denominator)Per TensorCore 1,028.75 TFLOP/s, HBM 3,433 GiB/s; their ratio is the ridge point, 279.1 FLOP/Byte (page's HBM Roofline ridge point: 279.085). Time Scale Multiplier 1.10537 original, 1.10529 modified (see below)
②Where the step landsThree points overlap (Total, Average, each Step): x = 253.5 → 279.4 FLOP/Byte, right by the 279.1 ridge point; y = 446,098 → 670,406 GFLOP/s, a clear step up
③Total (HW)In the legend, yet no point and no table row: it needs whole-session hardware-counter FLOPs (page note (5)), which open-source XProf 2.23.1 gives for neither run. Only the Total/Average rows are usable
④Normalized FLOP RateAverage row (complete steps only): 446,097.87 → 670,406.39 GFLOP/s. Times the step time (11.794 s / 7.857 s): 5,261 / 5,267 TFLOP booked per step per core, almost identical
⑤Bound byBoth Compute: XProf picks the higher utilization (compute 43.4%/65.2% vs HBM 31.3%/40.2%)
⑥HBM BW1,075.76 → 1,381.46 GiB/s; ÷ 3,433 = 31.3% → 40.2%, the rightmost column, Max memory BW utilization
⑦FLOP Rate / Peak43.4% → 65.2%: 446,098 ÷ 1,028,750 = 43.36%, 670,406 ÷ 1,028,750 = 65.17%. Numerator (FLOPs booked per step) unchanged, denominator (step time) down to 66.6%, so the ratio rose 1.50×

Read the Average row (complete steps only; Total includes partially captured ones). Normalized FLOP Rate is booked FLOPs per second; Bound by, whether compute or bandwidth is closer to its limit; FLOP Rate / Peak, the rate ÷ the 1,028.75 TFLOP/s per-core peak. Both runs say Compute and utilization jumps, as if the whole step really got much faster.

Work out the numerator first (FLOPs per second × step time): 5,261 vs 5,267 TFLOP per step per core, 0.1% apart. Same work, a third less time: utilization up 1.50×. Same accounting as §3.2's 117%: each expert matmul is booked at 2.886 TFLOP from its shapes; the modified calls shrink to tens of µs, the booking doesn't. Remove the expert-matmul family from numerator and denominator and the rest runs at 29.9% of peak original, 31.4% modified: the 21.8-point step gain is almost all this one family.

Per step, per TensorCoreOriginalModifiedHow derived
Booked FLOPs (TFLOP)5,2615,267Average row Normalized FLOP Rate × step time
Step time (ms)11,7947,857Average row Total Time per core
FLOP Rate / Peak43.36%65.17%÷ 1,028,750 GFLOP/s; page: 43.4%/65.2%
 of which expert matmul: FLOPs / time2,762.1/3,665.7 ms2,762.1/98.3 msxplane, 3 complete TensorCore steps (table in §3.2)
 everything except expert matmul29.9%31.4%(total FLOPs − family) ÷ (total time − family) ÷ peak our derivation
Section1 in detail (why Total (HW) is missing, why a point left of the ridge says Compute, where the Time Scale Multiplier comes from)
  • No Total (HW): page note (5): this row uses hardware performance counters, the others XLA's cost analysis. Both runs list it in the legend, yet the chart has no such point and the table no such row (the data endpoint returns only Total, Average and the three steps). So this page offers only booked FLOPs ÷ measured time; no row has counter-measured FLOPs.
  • Left of the ridge point, yet Compute: the original's point is at x = 253.5 FLOP/Byte, just below the 279.1 ridge (left of the red dashed line in the zoom). The x-axis sums each instruction's self-reported bytes_accessed (table's Memory BW, 1,638.91 GiB/s); Bound by uses the HBM bytes of a second, per-memory-space count (table's HBM BW, 1,075.76 GiB/s). By that count intensity is 386.2, right of the ridge, and compute utilization (43.4%) beats HBM (31.3%), hence Compute. The two byte counts differ, for single kernels in Section2 too.
  • How these columns are computed: SetRooflineMetrics in open-source XProf's xprof/convert/op_metrics_to_record.h (openxla/xprof, tag xprof-v2.23.1), summarized:
    operational_intensity = flops_v2 / bytes_accessed       # x-axis: self-reported bytes
    hbm_bytes = sum of each instruction's HBM entries in memory_accessed_breakdown   # else falls back to bytes_accessed
    hbm_bw    = hbm_bytes / time                              # the table's HBM BW
    flops_utilization  = measured_flop_rate / peak_flops
    hbm_bw_utilization = hbm_bw / peak_hbm_bw
    Bound by  = the highest utilization (Compute, HBM; on TPU also CMEM and VMEM when they have bytes)
    So Bound by only says which term is closer to its limit; both numerators are booked, not a measured bottleneck. Division by zero yields 0 (SafeDivide).
  • HBM bytes fell too: HBM BW × step time gives 12,687 → 10,854 GiB per step per core, 14% less. This count too is fixed at compile time (the program lost the expert all-gather and other instructions, and the booking shrank with them), not measured traffic.
  • Time Scale Multiplier: each TensorCore op event carries one (this run has only two values, 1.15789 and 1.0); their duration-weighted mean, 1.10536, matches the 1.10537 atop the page (1.10529 modified). Op Profile's peak includes it (§3.2); Roofline's doesn't by default. The Apply Dvfs Scaling Factor switch at the top is off by default; its tooltip reads “Show flops utilization scaled with the evaluated device capability (due to pstate changes).” Switched on locally, the peak goes from 1,028.75 to 1,137.15 TFLOP/s (exactly × 1.10537, Op Profile's number); the two VMEM read/write peak bandwidths scale too, HBM doesn't.
  • Model FLOP Rate (a Section2 table column) is about 17.5% above Normalized (523,982 vs 446,098 GFLOP/s). As the §3.2 fold explained, Normalized rescales XLA's own FP8 matmuls to BF16 [inferred: the gap comes from this kind of rescaling, not checked entry by entry]. We read only Normalized, the basis of the page's default utilization.
Open it yourself (same screen in local open-source XProf; first time, see “Opening it locally” in §2.1)
Reading Roofline Section2 (original recipe: who hugs which roof, which points to distrust, is the top 1000 enough)

⭐ Per op, the original's points all behave: matmuls hug the flat roof, elementwise fusions the slope. Two kinds of “fake points” sit far from the roof: 0-FLOP done ops (they can't be plotted at all) and done rows over 100% bandwidth. Attention kernels report bytes, so their Bound by is trustworthy.

Scroll down to Section2. One point per op, colored by category; the table is sorted by total time, 10 rows a page. On the original, read the chart's shape first, then the top rows:

Figure 3.2C · Roofline Section2 (per op): Original scatter plot and top 10 table rows. ① Red star: whole program; ② a string along the flat roof; ③ a cloud along the slope; ④ top three: attention kernels; ⑤ their HBM BW; ⑥ ranks 4–9: expert matmuls
② The string by the flat roof is matmuls, ③ the cloud under the slope elementwise fusions. In the table: ④⑤ attention kernels, ⑥ expert matmuls.
#WhatWhat the page shows (boxes at the on-page text; points read off the axes)
①Red star: whole programThe same point as in Section1 (intensity 253.5, 446,133 GFLOP/s)
②A string along the flat roofBlue custom-call points (44, intensity 240–3,210) and green convolution fusions (XLA's own matmuls; 112 of 134 marked Compute) crowd right of the ridge point, along the flat roof. Time-weighted, custom-call averages 59.0% of peak, convolution fusion 83.2%. Matmuls doing real work belong exactly here
③A cloud along the slopeOrange loop fusions (elementwise fusions, 435): intensity only 0.08–4.6, 425 marked HBM, time-weighted at 61.3% of HBM peak. Little compute, lots of movement: the only lever is fewer reads and writes
④Top three: attention kernelssplash_mha_dkv/fwd (Pallas kernels): Bound by Compute, FLOP Rate/Peak 44.4%/37.8%
⑤Their HBM BW70.20/56.99 GiB/s, not 0: splash reports bytes in cost_estimate (details below), so Bound by for these can be trusted
⑥Ranks 4–9: expert matmulstgmm/gmm_megablox (Pallas): FLOP Rate/Peak 49.3%–77.6%, all below 100%. Modified, the same family's gmm.27 is at 5,674.9% (§3.2 recording stop 3)

A convolution fusion is a matmul XLA fused with its neighboring small ops; a loop fusion, a chain of elementwise ops fused into one; a custom-call, a Pallas kernel. The matmul string sits right of the ridge: much compute, little movement, so at best it nears the flat roof (custom-call 59.0%, convolution fusion 83.2%, ②). The elementwise cloud has intensity below 5: each value loaded gets only a few operations, so only fewer reads and writes help. On the original, this chart rules things out: ordinary ops already sit on the roof; no category “clearly could go faster”.

Two more readings. First, done ops (§2: where the TensorCore waits for an async job, mostly a SparseCore transfer): select only async-done under Category, and the chart is empty. Second, the attention kernels: type splash in the Operation box.

Figure 3.2D · Two readings of Section2: 0-FLOP done ops, attention kernels that report bytes (Original). ① Only the top 1000 plotted; ② 0 FLOPs not plotted; ③ empty chart; ④ FLOPs 0, Bound by HBM; ⑤ the done's HBM BW; ⑥ over 100%; ⑦ points of the two attention kernels; ⑧ HBM BW not 0
Top: ① only the top 1000 are plotted, ② 0-FLOP ops aren't, so ③ is empty; ④ every row has 0 FLOPs; ⑤⑥ one done's bandwidth exceeds the peak. Bottom: ⑦ both attention kernels plot fine; ⑧ their HBM BW is not 0.
#WhatWhat the page shows (boxes at the on-page text; points read off the axes)
①Only the top 1000 plottedPage note (3): Section2 takes only the 1000 most time-consuming ops. Of the original's 3,897 TensorCore ops the table has 999 rows, dropping only 0.21% of self time (74 ms per core of the whole profile's 35,379 ms), so category shares hold: custom-call 55.5% here vs 55.4% in the full Op Stats. Open-source 2.23.1 has no pie chart on this page
②0 FLOPs not plottedPage note (7): ops with 0 FLOPs get no point. The top 1000 rows include 263 of them (loop fusion 78, data formatting 35, slice 34, async-done 32…), 10.3% of total time
③Empty chartWith only async-done selected, the chart below shows no points at all: every done has 0 FLOPs
④FLOPs 0, Bound by HBMAll 32 done rows show 0.00 GFLOP/s and Bound by HBM, the only nonzero term. A done is where the TensorCore waits for an async job to finish (mostly a SparseCore transfer). Skip it; see in Trace Viewer what it waited on
⑤The done's HBM BWgather_offload_async_done.2: 3,542.46 GiB/s, above the 3,433 peak
⑥Over 100%103.2%: the numerator is the 1,879,048,192 bytes booked to this done from its shapes, the denominator only the TensorCore's short wait on it (494.0 µs per call): two different things
⑦Points of the two attention kernelsdkv (intensity 3,210) and fwd (1,304) both plot as blue points: the kernels report bytes, so intensity = FLOPs ÷ bytes has a value
⑧HBM BW not 070.20/56.99 GiB/s. For a Pallas kernel, check whether this cell is 0 before reading Bound by. Here it isn't, so Compute is credible

Three rules (details in the two tables above). Skip done ops: FLOPs are 0 and the bandwidth cell is meaningless too (gather_offload_async_done.2: shape-booked bytes ÷ the TensorCore's 494 µs wait, ⑤⑥); see what it waits on in Trace Viewer (§2.5). For a Pallas kernel, first check whether HBM BW is 0. If the author left bytes out of pl.CostEstimate, Bound by has only compute to go on and Compute means nothing. Ours all fill them in (none of either run's 18 cost_estimate entries has 0 bytes), so splash's 44.4%/37.8% reads at face value. Is the top 1000 enough? The table holds only 999 of the original's 3,897 TensorCore ops, yet drops just 0.21% of time (0.31% modified); custom-call's share is 55.5% vs 55.4% in full Op Stats. A profile with many small ops may not be covered as well; for shares, Op Stats is safer.

Section2 in detail (two byte counts checked digit by digit, XLA's own matmuls a bit over 100%, modified kernels changed bytes, not FLOPs)
  • Two byte counts: in the table, Memory BW = self-reported bytes_accessed ÷ time per call; HBM BW = the HBM part of the per-memory-space byte count each instruction carries in the xplane (memory_access_breakdown) ÷ time per call. Recomputed, both match the page digit for digit: splash_mha_dkv…5 self-reports 2,247,884,800 bytes → 132.60 GiB/s, HBM count 1,190,133,764 → 70.20 GiB/s; gmm_megablox.16 self-reports 5,664,411,648 → 1,537.13 GiB/s, HBM count (two entries combined) 2,852,140,804 → 773.97 GiB/s. The point's x uses the first (dkv: 3,209.9 FLOP/Byte), Bound by the second.
  • Some of XLA's own convolution fusions are slightly over 100% too: 31 in the original, at most 112.2% (38 modified). About 1.1×, nothing like gmm.27's 5,674.9% (56.7×): these are XLA's own FP8 matmuls, Model FLOP Rate up to 2,236 TFLOP/s, about 97% of the official FP8 per-core peak, so physically possible. They slightly exceed 100% because this page's denominator, 1,028.75, is about 10% below the official BF16 1,153.5 per core [inferred: we have not verified how the denominator relates to the official peak]. Modified: 19 custom-calls exceed 100%, topped by gmm.28 at 6,116.3%.
  • The modified kernels changed bytes, not FLOPs: the modified program's 18 cost_estimate entries keep the original's flops exactly, but several expert-matmul entries report fewer bytes (smallest 1,681,922,048 vs 4,060,184,576 original). Once the input shape becomes [3,…], the shape-derived byte count follows; the FLOP formula doesn't [inferred: from comparing the two sets of values; we did not read how the kernel source computes either].
Open it yourself (same screen in local open-source XProf; first time, see “Opening it locally” in §2.1)
  • Same page, scroll down to Section2: you land on Figure 3.2C (10 table rows per page, pager at the bottom).
  • Figure 3.2D, top: pick async-done in the Category drop-down. Bottom: set Category back to All, type splash in the Operation box and press Enter. Ignore the table's Program ID column (hidden in the screenshots).

Utilization Viewer (per the official docs, it charts how busy each execution unit is) shows No Data for every profile in open-source 2.23.1: this version routes it to an unregistered processor (log: No ProfileProcessor registered for tool: utilization_viewer), fixed in 2.23.2. The raw data is in these 7 profiles (hardware-counter events on the TPU plane), but our screenshots use 2.23.1, so we skip the page. Its question, does the MXU really run FP8, we answer from the kernel's HLO operand types (f8e4m3fn[3,4096,1536] in Details) and the single kernel's FLOP rate (§3.2).

3.3 Op Stats: rank, call count, time per call

⭐ Expert matmul calls didn't drop (1,872 in both), but each fell from a few ms to tens of µs, 100 at most. The SparseCore's expert-gradient reduce-scatters are gone.

Figure 3.3 · HLO Op Stats, TensorCore top 10: Original / Modified. ① Top three: attention; ② ranks 4–10: expert matmuls
Same top three; ranks 4–10 all replaced.
#WhatOn the page (boxes at the on-page text)
①Top three: attentionBoth runs: splash attention kernels, 15,788/9,886/9,886 vs 15,773/9,885/9,885 µs per call. Same work, same time (the control)
②Ranks 4–10: expert matmuls7 of the original's top 10 are expert matmuls (tgmm_megablox/gmm_megablox_transpose_rhs/gmm_megablox), 3.5–5.7 ms per call. Modified: none in the top 10; ranks 4–10 are loop fusions and gather_offload async-dones, and the first expert matmul, tgmm.8, is 124th (Figure 3.4). The modified ② is dashed: no expert matmuls left at those ranks

HLO Op Stats is one big table, a row per op: call count (#Occurrences), self time per call (excluding nested sub-instructions), share and FLOP rate. The modified top 10 has no expert matmuls (②); ranks 4–10 go to loop fusions (a chain of elementwise ops fused into one) and gather_offload, which moves tokens to their experts. The top three, the attention kernels, barely differ; as the control, they show nothing else moved.

Filter for gmm, with the two runs stacked:

Figure 3.4 · HLO Op Stats, names containing gmm: Original / Modified. ① Rank; ② #Occurrences; ③ Avg. self time; ④ Model GFLOP/s; ⑤ the few that run once per step
Down each column: ② same count, ③ time per call 60× apart, ④ and so is the rate.
#WhatOn the page (boxes at the on-page text)
①RankRank of the first expert matmul: 4 in the original (tgmm_megablox.12), 124 modified (tgmm.8)
②#Occurrences1,872 in both = 78 iterations × 3 steps × 8 TensorCores (4 chips on this host): not one call fewer
③Avg. self time5,692.2 µs → 94.5 µs per call (first row)
④Model GFLOP/s507,049 → 30,526,213 GFLOP/s (507 TFLOP/s → 30.5 PFLOP/s): same numerator (self-reported FLOPs), the time 60× smaller
⑤The few that run once per stepModified: gmm.6, tgmm.2… appear 24 times (once per step per core, outside the 78-iteration loop), 4.2–5.0 ms and 573–686 TFLOP/s per call. Not shrunk (outputs shaped like bf16[192,1536,4096], all 192 experts): the MTP layer (§4.6)

Same count (②): every iteration made its calls. Time per call fell from 5,692 µs to 94.5 µs; Model GFLOP/s, self-reported FLOPs ÷ time per call, rose 60× with it, to 30.5M GFLOP/s (30.5 PFLOP/s). Not all shrank: the once-per-step calls (⑤, outside the loop) still take 4.2–5.0 ms at 573–686 TFLOP/s, the same order of magnitude as the original's kernels of the same kind. They are the MTP layer's expert matmuls (multi-token prediction: an extra layer after the trunk that predicts one more token), whose weights are gathered by a separate path (§4.6).

A SparseCore table sits at the bottom of the page; filter for reduce-scatter:

Figure 3.5 · HLO Op Stats, names containing reduce-scatter: Original / Modified. ① Dones on the TensorCore; ② execution on the SparseCore; ③ Modified: both tables empty
Original: ① TensorCore dones take only 0.02 ms, ② SparseCore execution 9–11 ms per call (top three rows in the loop, bottom three outside). Modified: ③ both tables empty.
#WhatOn the page (boxes at the on-page text)
①Dones on the TensorCoreOriginal TensorCore table: reduce-scatter.13/17….call-done average 0.021/0.020 ms per call. When the TensorCore reaches the done, the SparseCore is long finished: these collectives were already hidden under compute
②Execution on the SparseCoreOriginal SparseCore table, 6 reduce-scatters: .16/.12/.14 run 234 times each (78 iterations × 3 steps; one SparseCore only), 9.1–10.4 ms per call, 2,354.7 ms per step in all (trace, by type: 2,354.0); .10/.6/.8 3 times each (once per step, outside the loop)
③Modified: both tables emptySame filter: no reduce-scatter in either table. The expert gradients were never scattered back

The Sparse Core Ops table records each instruction the SparseCore executes: the very collectives of the §2 trace. The original has six expert-gradient reduce-scatters, three in the loop (78 per step each, 9.1–10.4 ms per call) and three outside (one per step, 9.5–10.6 ms), 2,354.7 ms per step; the modified run has none in either table. Their dones on the original's TensorCore take only 0.02 ms: already hidden under compute (as §2.5 concluded), so losing them saves the TensorCore almost nothing; the savings are mostly expert matmuls.

Op Stats conventions (dividing counts, self time, don't add the two tables)
  • The TensorCore table sums this host's 8 TensorCores over the profile's 3 complete steps: count ÷ 24 = per step per core. Avg. self time averages the 8 cores, 1–2% off our TPU:0-only numbers.
  • This version's SparseCore table records only one SparseCore: reduce-scatter.16's 234 calls = 78 iterations × 3 steps. Our SparseCore numbers are from the trace; they agree (2,354.7 vs 2,354.0 ms).
  • Sum only self time: total time includes nested sub-instructions (a while holds its loop body) and double-counts. Don't add the two tables: the SparseCore moves data while the TensorCore computes, so their times overlap.
  • The HLO op name filter box defaults to Exact Match: click the = at its right for “Contains”, and gmm then catches gmm and tgmm.
Open it yourself (same screen in local open-source XProf; first time, see “Opening it locally” in §2.1)
  • Tool hlo_stats (HLO Op Stats under Tools on the start page). In Visible Columns above the table, tick only Rank, HLO op category, HLO op name, #Occurrences, Avg. self time (us), Total self time (%), Model GFLOP/s, Normalized GFLOP/s: Program id is a program fingerprint, and HLO op text runs to several hundred characters a row.
  • Figure 3.4: click = in the HLO op name box for “Contains” and enter gmm. Figure 3.5: same with reduce-scatter; scroll down to Sparse Core Ops.

3.4 Reconcile both ends: missing time ≈ work never done

⭐ The step is 3,916.1 ms shorter: expert matmuls 3,570.5 ms less, waiting on dones 485 ms less; elsewhere actually took a little more. If each device computed only its own 3/192, the skipped matmuls should save 3,611 ms. Both ends match.

TPU:0 median step (ms)OriginalModifiedSaved
One step11,758.27,842.13,916.1
custom-call6,545.02,971.93,573.1
 of which expert matmuls (gmm/tgmm, 957 ops)3,668.898.33,570.5
Waiting on dones (collectives, gather_offload, other async transfers)969.0483.6485.4
Elementwise ops (non-fusion elementwise)155.922.2133.7
loop fusion2,166.52,442.4−275.9
Other categories−0.2

One end is time: 91% of the 3,916.1 ms saved is expert matmuls (same 957 ops, 97% less time). The other end is work: 192 experts over the FSDP group's 64 devices is 3 per device. If each modified device computed only its own 3, it skipped 189/192 of the work (whether it did must be checked in the HLO). All 192 took the original 3,668.8 ms; if time scales with expert count, the skipped part is worth 3,668.8 × 189/192 = 3,611.5 ms [inferred], vs an actual 3,570.5 ms: 1% off.

The SparseCore's skipped gradient scatter and weight gather (2,354 ms of reduce-scatter, about 1,200 ms of all-gather) aren't in this table: they overlapped TensorCore compute anyway and show up only as the 485 ms less waiting (119 ms of it collective dones, §2.5; the rest other async dones such as gather_offload, not split further). Loop fusion actually takes 276 ms more in the modified run. Less work, yet a category grew? Note it; §4.6 finds the cause in the HLO.

⭐ 90% of the 3.9 seconds saved is expert matmuls never computed.

3.5 Interlude: a 375 s step the same pages expose at a glance

⭐ One profile's step takes 375 s, 31.9× the original. Op Profile's first row names it: custom-call, 98.49% of time at 0.94% utilization. Op Stats' top 10 are all expert matmuls at 0.37–0.42 s per call.

Figure 3.6 · The 375-second run (Original, tiles not injected): Op Profile and Op Stats. ① Overall FLOPS utilization; ② the custom-call category; ③ custom-call FLOPS; ④ top 10 all expert matmuls; ⑤ time per call
The opposite of 117%: one category takes all the time, barely using compute.
#WhatOn the page (boxes at the on-page text)
①Overall FLOPS utilization0.62% (original: 20.20%)
②The custom-call categoryTIME 98.49%, WASTED 97.56%: the 375-second step is almost all Pallas kernels
③custom-call FLOPS0.94% (original: 53.39%): tens of times the time, same booked FLOPs (all three FP8 runs: 3,966 TFLOP per step per core)
④Top 10 all expert matmulstgmm_megablox/gmm_megablox/gmm_megablox_transpose_rhs fill the top 10, still 1,872 calls each
⑤Time per call367,386–418,121 µs (0.37–0.42 s) per call vs 3.4–5.7 ms in the original: about 109× slower (gmm_megablox.16: 373,915.2 ÷ 3,432.0 µs). Model GFLOP/s only 7,719

This is the original recipe rerun with a byte-identical config; that run just didn't inject the expert-matmul tile sizes, so every kernel used the default 128 × 128 × 128. A tile is the block of matrix a kernel loads on-chip and computes at once. Too small, and movement and scheduling overhead swamp the math, idling most compute. The compiled program sets all these kernels to tile 128×128×128 (7,511 sites); the original uses sizes like 512×2048×1536. Each expert matmul ran about 109× slower (⑤), the step 31.9×.

Two cases, one page. Modified: custom-call has little time at over 100% utilization. More work booked than done: find who skipped it. 375-second run: custom-call has all the time at under 1%. Nothing missing, just extremely slow: check the kernel config. One reading for both: put a category's time share next to its utilization, then expand to the kernel.

Per step, per TensorCoreOriginalModified375-second run
custom-call booked FLOPs (TFLOP)3,966.13,966.13,966.1
custom-call measured time (ms)6,541.82,972.8369,525.1
Utilization on the page53.39%117.45%0.94%
One gmm_megablox.16/gmm.27: time per call3,431.1 µs48.6 µs373,755 µs
 self-reported 2.886 TFLOP ÷ that time841 TFLOP/s59,387 TFLOP/s7.7 TFLOP/s

All three book the same FLOPs; only the denominator differs: absurdly shrunk when modified, ballooned in the 375-second run.

From a category down to one kernel ① Op Profile, Group by = Category; ② read each category's TIME and FLOPS together: lots of time at low FLOPS is slowness, little time at absurd FLOPS is booked work not matching work done; ③ expand the category to find the kernel; ④ click the kernel and read the left panel: time per call, FLOPs per second, against peak. Then check call count and per-call time in Op Stats, two profiles side by side.
How the 375-second run was traced (config diff, where tiles come from)
  • MaxText writes the full config into the TensorBoard event file. Diffed line by line with the original, the 375-second run differs only in step count, steps skipped before profiling and output directory.
  • The original's tiles aren't in the config: it uses the Tokamax kernel, its tiles injected separately at runtime. The compiled program shows each kernel's tile: (512, 2048, 1536) at 4,203 sites and (512, 1536, 1536) at 3,380 in the original (only these two, 7,583 in all); the 375-second run: all default (7,511 sites). A “site” is one occurrence of this kernel-config string in the compiled program (HloProto), not one kernel; the two programs differ slightly, the totals by 72, and this course didn’t match them site by site.
  • Why that run didn't inject them, and why the tuning log doesn't say: we found neither its launch script nor logs, so this is not established. The Hy3 tuning log §⁠3.4.1 has the same mechanism: without tiles, 12.4× slower (at another scale, hence another factor).
Open it yourself (same screen in local open-source XProf; first time, see “Opening it locally” in §2.1)
  • Run fp8tk: Op Profile as in §3.1 (Group by = Category); Op Stats as in §3.3, unfiltered, TensorCore table top 10.

3.6 Memory: do the never-gathered experts show up?

⭐ The modified recipe neither gathers the 192 experts' weights nor scatters their gradients: memory should fall a lot. Did it?

§3.4 settled time; memory is a separate ledger. The original recipe must gather all 192 experts' weights before computing, and the backward pass yields a full 192-expert gradient that must be held until scattered back to the devices: one bf16[192,4096,1536] block is 2,304 MiB (192 × 4,096 × 1,536 × 2 bytes). If the modified recipe truly skips both, these blocks should be gone and the peak several GiB lower. XProf shows memory on two pages: Memory Profile, the runtime allocator's view, over time; Memory Viewer, the compiler's allocation plan for one program, over instruction index. Step through the recording first:

Memory Profile and Memory Viewer in local open-source XProf 2.23.1 (DPR2 screenshots); boxes follow page text and bars. Stops 1–12 match the table below.
StopPage and actionWhat you read
1fp8old Memory Profile, as it opensCapacity 94.74 GiB, window peak 93.28 GiB, peak time 1.4 ms; Stack 49.39 + Heap 43.89 + Free 1.46; all three lines flat
2Set the Memory ID drop-down to 1Peak time 23,606.7 ms, 93.23 GiB; the default 0 (TPU:0) logs 128 extra small allocations, putting its peak at the window's start
3Enter 23606.7 in “HLO's at time” above HLO ModulesOnly jit_train_step is running (8 rows = 8 TensorCores, 7 on screen); Memory Breakdown Table: stack 49.387, unused preallocated 43.846 GiB
4Switch to Memory ViewerDefault module jit_add, peak 0.00 MiB: switch modules first
5Pick jit_train_step in the Hlo Modules drop-downPeak 91,766.56 MiB; entry parameters 44,533.22; temporaries 47,230.39; x-axis is Program Order. The header's Module Name stays jit_add (not refreshed in this version); trust the drop-down
6Scroll down to HLO Ops at Peak; under by Buffer Size, hover the widest barA 5,688 MiB f32[3,79,…]: a parameter or its nu (6 equal-width bars, reordered each time the page opens)
7Hover the second barThe other kind: nu, or an expert weight under state['model']
8Hover the half-width barsmu in bf16, 2,844 MiB; params + nu + mu = 44,533 MiB
9Next bar of the same widthbitcast_convert_fusion.11: Temporary, no Framework Op, a bf16 weight copy the compiler inserted
10Further right, the 2,304 MiB barstgmm_megablox.10, bf16[192,4096,1536]: this layer's full gradient for all 192 experts. 5 blocks of this shape, 11.25 GiB: 3 from this layer, 2 from the MTP layer's expert-gradient all-reduce
11Same page, fp8newPeak 91,597.20 MiB; the three widest are now the output-head logits bf16[7,4096,120832]; no [192,…] gradients left in the peak list
12Same page, bf16oldParameter param.329 (wo) f32[192,79,1536,32]: all 192 experts present, the 4,096 dimension cut to 32; entry parameters 22,271 MiB, half of FP8

Full video: topic07-xrec-mem.en.mp4 (the poster is the stop-10 frame, png).

Figure 3.7 · Memory Profile: peak, time, Stack/Heap (fp8old). ① Memory ID; ② Window peak; ③ Peak time; ④ Stack/Heap/Free; ⑤ Timeline; ⑥ HLO's at time; ⑦ Running at that moment; ⑧ stack; ⑨ unused preallocated
The runtime view: peak, time, two big blocks. Read ② and ③ first (not on Memory ID 0), then ⑦ below for the program running then.
#WhatWhat the page shows (boxes follow page text and bars)
①Memory IDDefaults to 0 (TPU:0), which logs 128 extra small allocations (#Allocation 196 vs 68), so its peak lands at 1.4 ms, the window's start; the training step doesn't begin until 380 ms. Switch to 1; Memory IDs 1–7 all match
②Window peak93.23 GiB (93.28 on Memory ID 0) against a capacity of 94.74 GiB: only 1.50 GiB left
③Peak time23,606.7 ms, inside the profile's second step
④Stack/Heap/Free49.39 + 43.85 + 1.50 = 94.74 GiB. Stack is workspace reserved in one block at launch; Heap is arrays kept across steps (⑨)
⑤TimelineAll three lines are nearly flat: memory is allocated in one block at launch, so the allocator never sees the program's ups and downs. This chart can show only peak and headroom
⑥HLO's at timeEnter the peak time, 23606.7; the module table keeps only the program then running
⑦Running at that momentOnly jit_train_step. 8 rows (8 TensorCores), 7 on screen; scroll the table for TPU:6. It spans about 12,213–23,970 ms: the peak belongs to the training step
⑧stackFirst entry in the Memory Breakdown Table: 49.387 GiB, the Stack from ④
⑨unused preallocatedSecond entry, “unused preallocated device memory”, 43.846 GiB, allocated before the profile started. Despite the name, it is within 0.36 GiB of Memory Viewer's entry parameters (parameters + optimizer state, 44,533 MiB = 43.49 GiB) [inferred: the parameters and optimizer state were allocated before the window opened, and the allocator records them as one block]

The lines stay flat because the training program reserves its whole workspace in one block at launch (Stack), and parameters and optimizer state predate the profile (Heap). The peak time is just the moment a 512-byte allocation landed: enter it in HLO's at time, confirm jit_train_step is running, and move on. The original peaks at 93.23 GiB, the modified run at 91.57 GiB: only 1.66 GiB less. To see what holds it, open this program in Memory Viewer.

Figure 3.8 · Memory Viewer: peak, entry parameters and temporaries of jit_train_step. ① Peak; ② Entry parameters; ③ Temporaries
The compile-time view: three runs side by side. A and B have equal entry parameters, C half; B's green band (peak span) is earlier.
#WhatWhat the page shows (boxes follow page text and bars)
①PeakA 91,766.56 MiB (89.6 GiB), B 91,597.20, C 88,401.89 MiB. This program only, from the compile-time plan, so it is below Memory Profile's runtime peak (§3.6 text; fold “Where the two peaks differ”). The header's Module Name still says jit_add: open-source 2.23.1 doesn't refresh it after a module switch; trust jit_train_step in the left-panel drop-down
②Entry parametersA and B both 44,533.22 MiB, C 22,271.03 MiB: BF16 is almost exactly half, 4.4 MiB over (fold “Why BF16's entry parameters are half of FP8's”)
③TemporariesTemporaries alive at the peak: A 47,230.39, B 47,061.12, C 66,127.62 MiB. The curve's x-axis is Program Order (instruction index), not time; the green band is the peak span: instructions 3,204–3,319 in A, earlier in B

The header has three numbers: the peak; entry parameters, arrays passed in at program start (parameters, optimizer state, this step's data); temporaries, buffers allocated during the run and freed after use (activations, gathered weights, gradients not yet scattered). The green band (peak span) sits at instructions 3,204–3,319 in the original, earlier in the modified run (near instruction 1,687). The two peaks differ by only 169 MiB, against an estimated saving of several GiB, and the peak has moved. See what is live there.

Figure 3.9 · Largest buffers at the peak: by Buffer Size (equal-width bars boxed together). ① Params ×3 + nu ×3 (f32); ② mu ×3 + bf16 weight copies ×3 + grad accumulators ×3; ③ [192,…] expert grads ×5 (3 from the loop + 2 from MTP); ④ Logits ×3; ⑤ Params ×3 + nu ×3 (f32); ⑥ mu ×3 + bf16 weight copies ×3; ⑦ MTP layer's gathered [192,…] ×3; ⑧ Logits ×3; ⑨ Sorted tokens ×2; ⑩ Params ×3 + nu ×3 (f32); ⑪ Gathered [192,…] ×3
The largest buffers at the peak. Equal-width bars share a box; A's ③ has no match in B.
#WhatWhat the page shows (boxes follow page text and bars)
①Params ×3 + nu ×3 (f32)FP8 Original: 6 bars × 5,688 MiB. param.467, param.393, param.471, param.391, param.469, param.392
②mu ×3 + bf16 weight copies ×3 + grad accumulators ×3FP8 Original: 9 bars × 2,844 MiB. bitcast_convert_fusion.11, broadcast.4714.clone, broadcast.4714.clone2, broadcast_in_dim.1367.clone, param.468, param.466
③[192,…] expert grads ×5 (3 from the loop + 2 from MTP)FP8 Original: 5 bars × 2,304 MiB. all-reduce.358, tgmm_megablox.11, tgmm_megablox.10, tgmm_megablox.12, all-reduce.354
④Logits ×3FP8 Modified: 3 bars × 6,608 MiB. multiply_convert_fusion.1, fusion.1176{1}, select_reduce_fusion.3{1}
⑤Params ×3 + nu ×3 (f32)FP8 Modified: 6 bars × 5,688 MiB. param.619, param.620, param.695, param.697, param.699, param.621
⑥mu ×3 + bf16 weight copies ×3FP8 Modified: 6 bars × 2,844 MiB. copy.2313, copy.2312, copy.2311, param.694, param.698, param.696
⑦MTP layer's gathered [192,…] ×3FP8 Modified: 3 bars × 2,304 MiB. all-gather.537, all-gather.539, all-gather.538
⑧Logits ×3BF16 Original: 3 bars × 11,328 MiB. multiply_convert_fusion.1, select_reduce_fusion.2{1}, fusion.1248{1}
⑨Sorted tokens ×2BF16 Original: 2 bars × 3,072 MiB. gather_offload_custom_fusion.81, gather_offload_custom_fusion.79
⑩Params ×3 + nu ×3 (f32)BF16 Original: 6 bars × 2,844 MiB. param.403, param.327, param.329, param.328, param.405, param.407
⑪Gathered [192,…] ×3BF16 Original: 3 bars × 2,304 MiB. all-gather.578, all-gather.577, copy.1476.args_refs_4_.args_refs_4_

Scroll down to HLO Ops at Peak Memory Allocation Time: one bar per buffer live at the peak, widest first under by Buffer Size. Hover a bar for its name, size, shape and Framework Op (the name path JAX recorded) in the left panel. One by one, they fall into three kinds:

In B (modified), the three widest are now bf16[7,4096,120832], 6,608 MiB each: the output head's logits, a score per vocabulary entry per position (7 sequences per device × 4,096 tokens × 120,832 entries). None of the loop's 3 [192,…] gradient blocks appear in its peak list: tgmm emits [3,…] directly; this layer's gradient scatter is gone. The MTP all-reduce remains (all-reduce.364/366/368), just not at the peak. The 3 [192,…] left in the list are the MTP layer's gathered weights, op_name convert_element_type (again in §4.2, §4.6).

⭐ Those blocks did vanish, but the peak is just the top moment, now the output-head logits, not the backward span: only 169 MiB lower. Memory and time agree: the loop gathers and scatters nothing.

From the peak to the code in four steps ① Memory Profile: read peak and headroom (not on Memory ID 0); enter the peak time in HLO's at time to confirm the program. ② Memory Viewer: switch to it; the header's three numbers give entry parameters vs temporaries. ③ by Buffer Size: hover widest first, identifying each equal-width group once (name, shape, Allocation Type). ④ The left panel's Framework Op points to the code; blocks without one are compiler-inserted, and no code change removes them.
What Stack and Heap contain (fp8old, Memory ID 1, matched line by line against Memory Viewer's entry parameters and temporaries)
BlockSizeContentsHow checked
Heap44,899 MiB (43.85 GiB)Arrays that persist across steps, allocated before the profile started (unused preallocated device memory in the Memory Breakdown Table)Memory Profile
├ Expert and other weights (f32)17,813.0 MiBstate['model']…, 44 blocksMemory Viewer entry parameters, split by Framework Op
├ Adam second moment nu (f32)17,813.0 MiBstate['optimizer']…['nu']…Same
├ Adam first moment mu (bf16)8,906.5 MiB…['mu']…, stored as bf16, hence halfSame
├ Data batch and step count0.6 MiBdata['inputs'] etc.Same
└ Unaccounted366 MiBHeap beyond the entry parameters; the allocator records Heap as one block, with no per-buffer breakdownnot checked
Stack50,573 MiB (49.39 GiB)Workspace reserved in one block at program start: activations, temporarily gathered weights, gradients not yet scattered, compiler-inserted copies, logitsMemory Profile
├ Compile-time temporary area (incl. fragmentation)49,473.0 MiBFourth line of the Memory Viewer header, “HLO lifetime temporary variables with fragmentation”Memory Viewer
└ Unaccounted1,100 MiBStack beyond the compile-time temporary area; reserved as one block at runtime, with only a total in the profilenot checked
  • Where the two peaks differ: the runtime peak = all of Stack + Heap, whatever is running. Memory Viewer's peak = entry parameters + temporaries live at the peak (peak − entry parameters = 47,233 MiB; the header lists 47,230 MiB of temporaries, leaving 3 MiB unclassified). The 3,705 MiB gap = (Stack − temporaries at peak) 3,339 + (Heap − entry parameters) 366 (from the unrounded bytes; subtracting the rounded numbers above is off by a MiB or two). The runtime number decides OOM.
  • The modified run's Stack is 1,704 MiB smaller (49.39 → 47.72 GiB): the compile-time temporary area drops from 49,473 to 48,434 MiB, while parameters and optimizer state are identical. Runtime peak 93.23 → 91.57 GiB.
Why BF16's entry parameters are half of FP8's (sharding differs; external parameter check)
  • FP8 uses ici_fsdp_parallelism=64 × ici_data_parallelism=2: parameters are cut into 64 shards, each stored on 2 devices, so each device holds 1/64 and the expert weights have shape [3,79,4096,1536] (sharded along the expert dimension).
  • BF16 uses ici_fsdp_parallelism=-1: 128 devices, 128 shards, 1/128 each. Expert weights are f32: wi_0/wi_1 are [192,79,32,1536] and wo is [192,79,1536,32] (param.329, hovered at stop 12, is wo; all three are split along the 4,096 dimension, 32 per shard: what Section 5 calls the expert dimension left unsharded). Entry parameters: 22,271 MiB, almost exactly half of FP8 (4.4 MiB over).
  • External anchor: FP8's per-device f32 parameters, 17,813 MiB × 64 shards ÷ 4 bytes each = 298.85B parameters. The Hy3 model card lists 295B (main) + 3.8B (MTP layer) = 298.8B, 0.02% off. The parameter line matches an outside number, so the breakdown holds.
  • BF16 runs 12 sequences per device (FP8 runs 7), so each of its logits blocks is 11,328 MiB, and its peak also falls on the output head. The two BF16 peaks, 88,401.89 vs 88,399.85 MiB, are almost identical (Section 5).
Open it yourself (same screen in local open-source XProf; first time, see “Opening it locally” in §2.1)

⭐ Time and memory both add up: 90% of the missing 3.9 s is expert matmuls never computed. But a profile records only what happened; why each device got only 3 experts takes the HLO and the code.

Section 4

Down to HLO and the code: why wasn't it done?

📝 Notes for this section

Section 3 balanced the books: 90% of the missing 3.9 s is expert matmuls never computed. Which part of the program is gone? A profile records only what happened, not what didn't. To see what should be there, read the program itself: the final HLO the compiler hands the chip.

4.1 HLO: the program after compilation

⭐ HLO is your JAX program after XLA has partitioned, fused and scheduled it: one instruction per line, a shape on every value. Every slice name in the trace is an HLO instruction name.

HLO (the IR of XLA's High Level Optimizer) is the compiler's internal program form. You write a train_step; the compiler splits it across 64 devices, fuses elementwise chains into fusions, inserts collectives and fixes the order, giving the final HLO, one line like this per instruction:

name = type[shape] opcode(operands, …), attributes…
gmm.27 = bf16[229376,1536] custom-call(…, get-tuple-element.14549, …), custom_call_target="tpu_custom_call"

The second line is the gmm.27 from Sections 2 and 3: a bf16 output of 229,376 × 1,536, and custom-call means a call into an external kernel (the Pallas expert matmul). The trace tells you when each instruction runs and for how long; the HLO, what goes in and out and which instructions exist at all, including ones never generated, which no profile shows.

WhereHowGood for
Open-source XProf pagesGraph Viewer: pick a module and an instruction to draw what feeds it and what it feeds (§4.3). Op Stats' HLO op text column: the full textOne instruction's inputs and outputs, in the browser
From the profile filesThe profile stores the whole compiled HLO (HloProto) in xplane.pb; its directory also holds a <module>.hlo_proto.pb per module. Parse with XLA's hlo.proto to get textWhole-program counts: how many of a kind, in which loop (all of this section's counts)
XLA dump at compile timeRerun with XLA_FLAGS="--xla_dump_to=DIR --xla_dump_hlo_as_text"; *after_optimizations.txt is the final HLONo profile yet, or to see each compiler pass (recompile)
Too dense? Strip the noise (hundreds of characters a line, mostly layout and backend config)
  • Three kinds of noise make raw HLO hard to read: a memory layout after every value (e.g. {3,2,0,1:T(32,128)(4,1)}), a whole backend_config JSON blob, and long names stacked layer upon layer.
  • Keep only name = type[shape] opcode(operands), tagging each line with its op_name (JAX's name path, §4.5). The modified step's HLO text drops from 8.4M to 3.8M bytes, the original's from 9.6M to 6.8M; a loop body becomes a few hundred readable lines.
  • Locate in the stripped version; use the raw text for layouts or backend config. The HLO excerpts in §4.2 are stripped.
One HLO line, symbol by symbol (Figure 4.1: raw text in Details, ten notations)

Click a slice in Trace Viewer and its raw HLO appears under Start Stack Trace in Details; Graph Viewer nodes and Op Stats' HLO op text column show the same text. Below is the original recipe's gmm_megablox.16 line (same screenshot as recording stop 2 in §4.3):

Figure 4.1 · How to read one line of HLO (Original gmm_megablox.16, verbatim from Trace Viewer Details). ① name = result opcode(operands…), attributes…; ② type + shape; ③ {1,0: dimension order in memory; ④ :T(16,128)(2,1): tiling; ⑤ custom-call + target; ⑥ one operand: type %name; ⑦ weights: 192 experts; ⑧ suffix after the operand name; ⑨ per-operand layout constraints; ⑩ kernel-reported metadata
One HLO line: name, type and shape, layout, tiling, opcode, operands, attributes.
#WhatHow to read it (boxes follow the text on the page)
①name = result opcode(operands…), attributes…One instruction per line. Drop the % and the name is the slice name in Trace Viewer (leave the % out in Find Events)
②type + shapebf16[229376,1536]: an output of 229,376 rows × 1,536 columns. 229,376 = 7 sequences per device × 4,096 tokens × 8 experts per token (the §0.2 config); f8e4m3fn is FP8 (4 exponent bits, 3 mantissa bits)
③{1,0: dimension order in memoryminor_to_major: the first number is the dimension that varies fastest in memory. {1,0} is row-major (openxla.org/xla/shapes)
④:T(16,128)(2,1): tilingtiling: tile by 16×128 first, then (2,1) packs 2 adjacent bf16 values in a column into one 32-bit word (FP8 uses (4,1), 4 per word). The official tiled_layout page gives (8,128)(2,1) as typical for bf16; the HLO measured in this course has 16×128, and 32×128 for FP8. Go by the HLO
⑤custom-call + targetcustom-call = call an external kernel; custom_call_target="tpu_custom_call" = a Pallas kernel on TPU (JAX's public source, tpu_custom_call.py, uses exactly this name)
⑥one operand: type %nameEach operand in the parentheses is written “type name”, in the order of the 0, 1, 2… on Graph Viewer's edges: s32[193]{0:T(256)} %copy-done.141 is operand 1 (recording stop 8 ①)
⑦weights: 192 expertsOperand 7, f8e4m3fn[192,4096,1536]{2,1,0:T(32,128)(4,1)}: FP8, 192 experts, 32×128 tiles. This one number is what Section 4 is hunting for
⑧suffix after the operand name.args_refs_4_.qvalue: which kernel argument this data feeds (the quantized value of argument 4) [inferred from the name; we didn't check the source that builds it]
⑨per-operand layout constraintsoperand_layout_constraints={…}: the layout the kernel requires for each operand, in the same order as the parentheses; s32[193] and f8e4m3fn[192,4096,1536] appear here again
⑩kernel-reported metadatakernel_metadata carries this kernel's tile: tile_m 512, tile_k 2048, tile_n 1536, the tile set the original recipe injects (§0.2 fold: the tiles injected at runtime)

For a collective, read three more attributes (§4.2's all-gather.573; all-gather.580, Section 5's BF16 expert-weight gather):

AttributeExampleHow to read it
replica_groupsmesh[…'fsdp'=64…] {'fsdp'}Which devices form a group: only the fsdp axis is named, so 64 devices. The group size is the number of shards the tensor is split into
dimensions{1} (573), {2} (580)The dimension to concatenate along: 573 concatenates the expert dimension, [1,3,…] → [1,192,…]; 580 the 4096 dimension, [1,192,32,1536] → [1,192,4096,1536]
is_spmd_generated="true"580 has it, 573 doesn'tLiterally, generated during SPMD partitioning: the compiler inserted it to match the weights' sharding, so you won't find this all_gather in the code. 573 lacks it: it was written by hand inside a shard_map in the code (① in §4.5; its op_name also ends in shard_map/all_gather)
How to read an HLO lineOpcode first (compute, move, or wait) → result type and shape (what is moved, how big) → any *-done among the operands (which communication it waits on) → layout and tiling last (if you suspect padding or a relayout).

4.2 Counting collectives: the all-gathers of the 192 experts, 12 → 0

⭐ Counted line by line: the original has 12 all-gathers taking expert weights from this device's 3 to 192, and 6 reduce-scatters taking gradients from 192 back to 3. The modified run has none of these 18.

Count in the final HLOOriginalModifiedReading
all-gather/reduce-scatter/all-reduce (all)92/6/3682/0/36all-reduce unchanged, 10 fewer all-gathers, no reduce-scatter left
Expert-weight gathers: f8e4m3fn [1,3,…] → [1,192,…]1203 in the forward loop, 3 in the backward (regathered for recompute), 6 outside (the peeled-out layer)
Expert-gradient scatters: bf16 [192,…] → [3,…]603 in the backward loop, 3 outside; the loop's 3 are Section 2's 9–10 ms reduce-scatters on the SparseCore

Each MoE layer has three expert matrices (two up-projections, one down), hence the threes. The total (92 → 82) only says something is gone; counting by shape says what: expert weights and gradients. Here is the original's forward loop gathering gmm's first weight matrix (stripped; elided operands as …):

── Original · forward loop body
all-gather.574.cloned.1.call-start = ((f8e4m3fn[1,3,4096,1536], token[]), f8e4m3fn[1,192,4096,1536])
    async-start(fusion.1669, …)
    └ all-gather.573 = f8e4m3fn[1,192,4096,1536] all-gather(param_0.13420),
          replica_groups=mesh[…'fsdp'=64…] {'fsdp'}, dimensions={1}
all-gather.574.cloned.1.call-done = f8e4m3fn[1,192,4096,1536] async-done(all-gather.574.cloned.1.call-start)
bitcast.3183 = f8e4m3fn[192,4096,1536] bitcast(all-gather.574.cloned.1.call-done)
gmm_megablox.16 = bf16[229376,1536] custom-call(…, clamp_convert_fusion.165, …, bitcast.3183, …)  # weights: 192 experts
── Modified · forward loop body
fusion.1642 = (f8e4m3fn[3,4096,1536], f8e4m3fn[3,4096,1536]) fusion(…), kind=kLoop               # local 3 experts → FP8
gmm.27 = bf16[229376,1536] custom-call(…, clamp_convert_fusion.165, …, get-tuple-element.14549, …)
                                                                                      # weights: f8e4m3fn[3,4096,1536]
── Backward loop body: Original / Modified
tgmm_megablox.10 = bf16[192,4096,1536] custom-call(…)
    → reduce-scatter.13….call-start = ((bf16[192,4096,1536], …), bf16[3,4096,1536])
tgmm.8 = bf16[3,4096,1536] custom-call(…)               (no reduce-scatter after it)

192 ÷ 64 = 3 is right there in the first line: the async-start takes this device's [1,3,4096,1536] and outputs [1,192,4096,1536]. Inside, all-gather.573 does the work: it groups only along the 'fsdp'=64 axis (64 devices) and concatenates along dimension 1, 3 × 64 = 192. This is Section 2's all-gather on the SparseCore row; async_execution_thread="sparsecore" in the HLO sends it there. In the modified run, gmm.27's weights, [3,4096,1536], come straight from a fusion that only quantizes; nothing is gathered in between.

Section 1 said the original quantizes to FP8 before gathering. Did the modified run just move that step? Trace fusion.1669 (original) and fusion.1642 (modified) up to the parameters:

FP8 quantization is the same step in both: this device's 3 experts → f8e4m3fn; they differ only in whether they gather after it One device, one expert matrix, one layer. Cell color = number format; cells show HLO shape and size; bars at right, one scale: bytes gathered in this step f32 bf16 f8e4m3fn (FP8) Same in both programs 1 Master weights f32[3,79,4096,1536] 5,688 MiB Local 3 experts × 79 layers; optimizer updates it 2 To bf16 each step bf16[79,3,4096,1536] 2,844 MiB Outside the loop, once per step 3 Slice out this layer bf16[1,3,4096,1536] 36 MiB In the loop, dynamic-slice 4 Quantize to FP8 f8e4m3fn[1,3,4096,1536] 18 MiB ×2 → clamp ±448 → convert; same in both A Original Gathers after ④ B Modified No gather after ④ 5 all-gather of 64 shards [1,3,…] → [1,192,…] Gathers FP8; 2.38 ms on SparseCore 6 gmm_megablox.16 weights f8e4m3fn[192,4096,1536] 1,152 MiB 192 experts 5 No gather Quantized → straight to kernel 6 gmm.27 weights f8e4m3fn[3,4096,1536] 18 MiB Local 3 experts ⑦ Bytes to gather (per device, matrix, layer) FP8 1,152 MiB If gathered as bf16: 2,304 MiB 0: nothing to gather Already holds its 18 MiB Token activations are quantized before gmm too: same clamp_convert fusion in both, output f8e4m3fn[229376,4096]. Both kernel inputs are FP8. With quantization before the gather, the original gathers FP8, half the bytes of bf16; the modified run skips even that half The saving doesn't come from moving the quantization, which is identical in both. What's gone is the all-gather after it (and the backward reduce-scatter).
Top row identical; after ④, A gathers, B doesn't. Bars at right share a scale: gathering bf16 costs twice the bytes.
Sources and assumptions

📌 ① Expert parameters (fp8old param.391, fp8new param.619/620) f32[3,79,4096,1536], sharding devices=[64,1,1,1,2] … last_tile_dim_replicate; ② converted to bf16 before the loop: fp8old bitcast_convert_fusion.11 (the fusion body is just convert f32→bf16, in ENTRY, once per step), fp8new copy.2312/.2313 (an f32 → bf16 copy).

📌 ③④ In the loop body: fp8old fusion.1669, fp8new fusion.1642 (two matrices at once), both dynamic-slice one layer → multiply(×2) → clamp(−448, 448) → convert f8e4m3fn. 2 = 448 ÷ 224: the config sets weight_quantization_calibration_method=fixed,-224,224, and the largest finite value f8e4m3fn can represent is 448 [our derivation].

📌 ⑤⑥ fp8old: fusion.1669 → all-gather.574 (wrapping all-gather.573, {'fsdp'}, 64 shards, dimensions={1}) → bitcast → operand 7 of gmm_megablox.16; 2.380 ms on the SparseCore in layer 40's forward pass. fp8new: fusion.1642's output is operand 7 of gmm.27, [3,4096,1536]. What crosses the network is the other 63 shards, 1,152 × 63/64 ≈ 1,134 MiB.

📌 ⑦ “If gathered as bf16”: bf16[192,4096,1536] = 2,415,919,104 B. A ready-made example in the same program: the MTP layer's expert weights are gathered in bf16 by compiler-inserted all-gathers (fp8new all-gather.537/538/539). Token activations: both runs use clamp_convert_fusion.165 → f8e4m3fn[229376,4096] (gmm operand 5), config act_quantization_calibration_method=fixed,-224,224.

#StepOriginalModified
①f32 master (3 local experts × 79 layers)f32[3,79,4096,1536], 5,688 MiB, same in both
②To bf16 before the loop, once a stepbf16[79,3,4096,1536], 2,844 MiB
③This layer sliced out in the loopbf16[1,3,4096,1536], 36 MiB
④Quantize: ×2, clamp to ±448, to FP8f8e4m3fn[1,3,4096,1536], 18 MiB; the same four ops and two constants
⑤After quantizingGathers all 64 FP8 shards; 2.38 ms on SparseCore, layer 40 forwardNo gather
⑥Weights the kernel receivesf8e4m3fn[192,4096,1536], 1,152 MiBf8e4m3fn[3,4096,1536], 18 MiB
⑦Bytes gathered in this step1,152 MiB (2,304 MiB if gathered as bf16 first)0

Quantization is identical to the character; the 2 is 448 ÷ 224 (the config's weight quantization range is ±224; FP8's maximum is 448). The original does quantize first, so it gathers FP8, half the bytes of bf16; the modified recipe didn't move it, only dropped the all-gather after it.

The all-gathers left in the modified run (why not 92 → 80)
  • Of the modified run's 82 all-gathers, only two kinds have 192 in their shape: the router's small [4096,192] weight (gathered along the 4096 dimension), and bf16 versions of the three expert matrices of the MTP layer (multi-token prediction, the one layer beyond the 80-layer trunk), bf16[3,4096,1536] → [192,4096,1536].
  • The latter appear in both programs, with op_name convert_element_type rather than shard_map/all_gather: the compiler inserted them to match the weights' sharding, not the gathering lines in the code (all three of these all-gathers carry is_spmd_generated=true; the §4.1 fold explains the attribute). So in both programs the MTP layer gathers all 192 experts before computing (more in §4.5).
  • In the original, the forward pass of the layer outside the loop gathers its 3 FP8 weights in one all-gather with other weights (MTP layer, output layer): [3,1,4096,1536] → [192,1,4096,1536]. So the table counts 12 by shape but 10 by instruction.

4.3 Graph Viewer: center on the same kernel, trace its weight back

⭐ Original gmm_megablox.16’s weight, traced up: this device’s [1,3,…] → all-gather over 64 devices → [192,4096,1536]. Modified gmm.27’s weight: this device’s [3,4096,1536]. Both kernels get a 192-group token table.

Graph Viewer (an XProf page) draws one center instruction with its operands upstream and users downstream. Boxes are instructions, arrows data flow; a large titled box is a called sub-computation, a dashed box a node with undrawn inputs. The recording goes from a Trace Viewer slice into Graph Viewer, then up the edges:

Local open-source XProf 2.23.1. Stops 1–5, 12: real page screenshots (DPR2). Stops 6–11: Graph Viewer’s graphviz SVG, saved as is, zoomed locally; boxes follow its nodes and text; in-image zooms use a 2D transition. Autoplays by default; untick “Autoplay” to step; stops 1–12 match the table.
StopActionWhat you see
1Click gmm_megablox.16 in the original’s forward layer 40Wall Duration 3,425,196 ns; top of the Details right column: a Graph Viewer link
2Read Start Stack Trace in DetailsHLO text: operands f8e4m3fn[192,4096,1536] and s32[193] (read symbol by symbol in Figure 4.1)
3Click the linkXLA Op Name filled but not found: open source ignores the link’s module name; drop-down on jit_add
4Pick jit_train_step under XLA ModulesOp name cleared, shortcut buttons on the right; click gmm_megablox.16 (or retype the name and click Search Graph)
5The graph appearsCenter label too long, so the graph is tiny; Graph Width = levels drawn outward; Save SVG below
6Save the SVG and view the whole graphRed diamond = center; operands above; the large gray box below is the fusion consuming its output; dashed box = inputs not drawn (Graph Width limit)
7Zoom into the center nodecustom-call, tpu_custom_call, 3.43 ms; operand layouts: f8e4m3fn[192,4096,1536], s32[193]; tile 512 / 2048 / 1536; output bf16[229376,1536]
8Read the edge numbersOperand index: 1 is copy-done.141 (group offsets), 7 is bitcast.3183 (weight)
9Follow operand 7 upstreambitcast ← all-gather.574…call-done ← call-start, with async_execution_thread="sparsecore"
10Enter the sub-computationActual all-gather.573: this device’s [1,3,…], 'fsdp'=64, dimensions={1} → [1,192,…]
11Modified, centered on gmm.27Weight f8e4m3fn[3,4096,1536]: operand 7 ← quantization fusion.1642; offsets still s32[193]
12Back in Trace Viewer, click gmm.27Same HLO line in Details: f8e4m3fn[3,4096,1536], s32[193]; no Graph Viewer needed
How to read Graph ViewerSet the center (Details right-column link; open source: re-pick the module) → Read the direction (in above, out below; edge number = operand index in HLO parentheses) → Identify boxes (one per instruction; big titled box = inside a sub-computation or fusion; dashed = undrawn inputs) → Follow edges to *-done inputs (at the edge, raise Graph Width or re-center) → Check the HLO text (same line in Details, stop 12).

Full video: topic07-xrec-graph.en.mp4 (last frame: png). Figure 4.2: its zoomed views as one static comparison:

Figure 4.2 · Graph Viewer: the same expert matmul in both runs, and where its weight comes from. ① This device’s weight shard; ② gathered with whom, along which dimension; ③ after the gather; ④ dropping the extra dimension; ⑤ the weight the kernel receives; ⑥ the row where each token group starts; ⑦ the weight the Modified kernel receives; ⑧ where it comes from
Same expert matmul: ⑤ the original gets 192 experts, ⑦ the modified only 3; ⑥ both group tables have 192 groups.
#WhatWhat the figure shows (boxes follow nodes and text in the Graph Viewer SVG)
①This device’s shardParameter 0 = f8e4m3fn[1,3,4096,1536]: this device holds only 3 experts (192 ÷ 64), already FP8
②Gathered with whom, along which dimensionall-gather.573’s replica_groups name only one axis, 'fsdp'=64: groups of 64 devices. dimensions={1}: concatenate along dim 1 (experts)
③After the gatherOutput f8e4m3fn[1,192,4096,1536]: 3 × 64 = 192 experts. Wrapped in an async call (titled box = sub-computation), call-start → call-done outside. This is the all-gather block on the SC0 row
④Dropping the extra dimensionbitcast to f8e4m3fn[192,4096,1536]; the arrow goes into gmm_megablox.16
⑤The weight the kernel receivesgmm_megablox.16’s operand shapes list the weight as f8e4m3fn[192,4096,1536]: all 192 experts
⑥The row where each token group startss32[193]: group offsets for 192 experts (193 = 192 + 1 end). Identical in both: tokens still arrive in 192 groups
⑦The weight the Modified kernel receivesgmm.27’s weight is f8e4m3fn[3,4096,1536]: only this device’s 3
⑧Where it comes fromoperand 7 = output of fusion.1642, which quantizes this device’s 3 experts to FP8. No all-gather in between; no collective anywhere in the graph

Up the original kernel’s weight operand runs the whole gather; the modified kernel’s is just this device’s 3 experts, quantized. Watch ⑥ most: s32[193], offsets for 192 token groups, is the same on both sides. Tokens still arrive in 192 expert groups; the modified run has weights for only 3.

Open it yourself (same screen in local open-source XProf; first time, see “Opening it locally” in §2.1)
  • Tool graph_viewer (Graph Viewer under Tools on the home page), run fp8old. In the quick-start list at the bottom, click jit_train_step(…) - gmm_megablox.16 to center on it (modified run: gmm.27). Or type the name into XLA Op Name, set Graph Width to 1, click Search Graph.
  • The graph is tiny: the original’s center node is a very wide red diamond (label too long). Zoom in: the dashed box top right is the all-gather’s sub-computation. Save SVG below the graph saves it to zoom locally (Figure 4.2’s source; boxes at SVG node/text positions).
  • The page also has Check HLO Full Text (SHORT / LONG); our local 2.23.1 showed no full text, and we didn’t chase it. For the whole text, use the table’s route 2 or 3.

4.4 In motion: gather 192 experts, or compute only this device’s 3

⭐ Same device, same tokens: the original gathers all 192 experts, computes, then scatters the gradients back; the modified run holds 3 experts and computes only 3 groups.

4.3 leaves a question: the modified kernel multiplies 3 experts’ weights by tokens in 192 groups. Does it fail, or compute something? Both programs on device 0, slowed down:

Manim, black background. Left: one expert’s weights; right: this device’s token rows for that expert. Autoplays by default; untick “Autoplay” to step; stops 1–7 match the table.
StopWhat to watchWhat you see
1Original ① all-gatherThe other 189 cells come from the other 63 devices: [1,3,4096,1536] → [1,192,4096,1536] (Figure 4.2 ①–③)
2② gmmAll 192 groups have weights; the right side lights up: 3 gmms per forward layer, 10.1 ms
3③ tgmm + reduce-scatterGradients for 192 experts; summed across 64 devices, each keeps only its 3: [192,…] → [3,…]
4Modified ① no all-gatherStill holds 3: [3,4096,1536] (Figure 4.2 ⑦⑧)
5② gmmOnly 3 groups’ rows computed, the other 189 written as 0: 3 gmms per layer, 0.11 ms
6③ tgmmJust 3 experts’ gradients; nothing to scatter
7SummaryAll shapes legal, no error; 189/192 of the work gone; expert matmuls 3,668.8 → 98.3 ms per step

Full video: topic07-experts.en.mp4 (last frame: png, same as the first). It shows device 0, which holds exactly experts 0–2; the modified run also computes exactly groups 0–2 [inferred from the code], so on screen they line up one to one. Other devices: the 4.5 fold.

The animation is one device over time. Laid out statically below: how 192 experts split into 64 shards, how many each branch computes:

192 experts in 64 shards of 3: Original gathers all 192 first; Modified uses only its own 3 Top: the sharding. Bottom: device 0’s five steps on each branch, ①–⑤ aligned. One cell = 3 experts (or 3 token groups) Weights Token group × weight Zeros / none Gradients 192 experts 0 191 64 shards, in order Device 0 Exp. 0–2 Device 1 Exp. 3–5 Device 2 Exp. 6–8 …… Device 62 Exp. 186–188 Device 63 Exp. 189–191 3 per device = 192 ÷ 64 2 copies on the data axis, 128 devices in all A Original: gather all 192, compute, scatter back to 3 1 Local 3 + 189 gathered Forward all-gather from the other 63 [1,3,4096,1536] → [1,192,4096,1536] 2 192 token groups × 192 experts gmm: all 192 groups multiplied 3 gmms per forward layer: 10.1 ms 3 Grads for 192 experts Backward tgmm: grads for 192 experts bf16[192,4096,1536] 4 Sum over 64, keep 3 reduce-scatter: each keeps only its 3 [192,4096,1536] → [3,4096,1536] 5 + → Same 3 experts, 2 data copies all-reduce over 2 data shards bf16[3,4096,1536] B Modified: holds 3, computes 3 groups, zeros the rest 1 Only the local 3 No all-gather: just the local 3 f8e4m3fn[3,4096,1536] 2 3 groups weighted, 189 zeroed gmm: only the 3 weighted groups, no error 3 gmms per forward layer: 0.11 ms 3 Grads for only 3 experts tgmm: grads for the local 3 only bf16[3,4096,1536] 4 None No sum over 64 No reduce-scatter Grads not summed over 64 5 + → Same 3 experts, 2 data copies all-reduce over 2 data shards, as before bf16[3,4096,1536] Every shape is legal on both sides: the group table still has 192 groups (s32[193]); it still runs with weights for only 3 experts 189/192 of the work is gone, with no error. The only tells: “too good to be true”, and steps ① and ④ vanishing entirely from the SparseCore row.
Compare the columns step by step: ② has only one blue cell left, ④ is gone. Top: the 64-way split. Below: device 0 alone.
Sources and assumptions

📌 The split: config num_experts=192, ici_fsdp_parallelism=64, ici_data_parallelism=2 (64 chips × 2 TensorCores = 128 devices). Final HLO: this device’s expert parameter f32[3,79,4096,1536] (3 experts × 79 layers), sharding devices=[64,1,1,1,2] … last_tile_dim_replicate: leading dim 192 cut in order into 64 blocks, each on 2 devices. Device numbers in the figure count along the fsdp axis (0–63); the 2 devices on the data axis hold the same experts.

📌 ① Original all-gather.573: input this device’s [1,3,4096,1536], replica_groups=…{'fsdp'} (64 shards), dimensions={1}, output [1,192,4096,1536] (f8e4m3fn). ② Weight operand: gmm_megablox.16 [192,4096,1536], modified gmm.27 [3,4096,1536]; group offsets s32[193] in both. One forward layer’s 3 gmms: 10.107 → 0.109 ms (layer 40).

📌 ③④ Original tgmm_megablox.10 outputs bf16[192,4096,1536]; reduce-scatter over {'fsdp'} scatters back to [3,4096,1536]; modified tgmm.8 outputs [3,4096,1536] directly. ⑤ Expert-gradient all-reduce (original all-reduce.384, modified .394) runs over {'data'} in both.

📌 Modified, the other 189 groups get 0: public MaxText backend.py builds tables for only this device’s 3 groups, zero-filling the rest (L346–347, L555–561). Which 3 groups device 0 computes is read from code [inferred]; the figure doesn’t number them.

#OriginalModified
①The other 63 devices send 3 each, forming [1,192,4096,1536]No all-gather; it holds just [3,4096,1536]
②gmm: all 192 groups get weights; layer 40 forward, 3 gmms: 10.1 msgmm: only 3 groups have weights, other 189 output 0; 3 gmms: 0.11 ms
③Gradients for all 192: bf16[192,…]Only 3 gradients to begin with: bf16[3,…]
④reduce-scatter: sum over 64 shards, each keeps its 3None
⑤all-reduce: each expert lives on 2 data-axis devices, whose separate gradients are summed (both runs; ⑦ in the Section 2 communication figure)

This answers Section 2’s open question: the modified run sums expert gradients only across the 2 data-axis shards; ④, the sum across 64 shards, is gone. Every shape along the way is legal; nothing raises an error.

4.5 Back to the code: gather and scatter sit in one branch only

⭐ An HLO instruction’s op_name names the code branch that ran. Public MaxText has the weight all_gather and gradient psum_scatter only in the tokamax branch; the modified run’s branch calls the kernel directly, with neither.

Every HLO instruction carries an op_name: the name path JAX recorded while tracing, primitive last, enclosing scopes to its left. Their last segments, for the three key instructions:

InstructionLast segments of op_nameWhich branch
all-gather.573 (original)…/while/body/closed_call/shard_map/all_gatherA hand-written jax.lax.all_gather inside shard_map
gmm_megablox.16 (original)…/shard_map/tokamax:{"op":{"__type":"tokamax…PallasMosaicTpuRaggedDot", …}}Tokamax’s ragged_dot: the use_tokamax_gmm=True branch
gmm.27 (modified)…/shard_map/jit(gmm)/pallas_callMaxText’s own megablox backend.gmm: the other branch

Of Section 0’s 13 differing config lines, the only one besides the 12 tile values is use_tokamax_gmm: True → False. The two branches it selects (public repo, version notes in the fold; unrelated lines removed):

# src/maxtext/kernels/megablox/ops.py · _gmm_fwd (forward)
  # QAG is only supported for following conditions
  if use_tokamax_backend:                                                                   # L191
    if quantization_rule and quantization_rule.bwd_qtype:
      if quantization_rule.weight_calibration_method.startswith("fixed") and isinstance(rhs, qpl.QArray):
        if weight_gather_axes:
          for axis_name, axis_idx in weight_gather_axes:
            rhs_qvalue = jax.lax.all_gather(rhs.qvalue, axis_name, axis=axis_idx, tiled=True) # ① L196: gather all 192 experts
            rhs = dataclasses.replace(rhs, qvalue=rhs_qvalue)
  …
  elif use_tokamax_backend:
    out = tokamax.ragged_dot(lhs=lhs, rhs=rhs, group_sizes=group_sizes, …)                 # ② L223 → gmm_megablox.*
  else:
    out = backend.gmm(lhs, rhs, group_sizes, preferred_element_type, tiling[:3], …)        # ③ L235 → gmm.*: rhs still local 3

# same file · _gmm_bwd (backward)
  if use_tokamax_backend or use_gmm_v2:                                                     # L315
    …
    drhs = tokamax.ragged_dot_general(lhs=lhs, rhs=drhs_dout, …)                           # tgmm_megablox.*: [192,4096,1536]
    if quantization_rule and quantization_rule.bwd_qtype and weight_gather_axes:
      for axis_name, axis_idx in reversed(weight_gather_axes):
        drhs = jax.lax.psum_scatter(drhs, axis_name, scatter_dimension=axis_idx, tiled=True) # ④ L392: scatter back to 3 each
  else:
    dlhs = backend.gmm(dlhs_dout, rhs, group_sizes, …)                                      # ⑤ L394–416: no gather, no scatter
    drhs = backend.tgmm(lhs.swapaxes(0, 1), drhs_dout, group_sizes, …)                      #    tgmm.*: [3,4096,1536]

① and ④ are the 4.2 table’s 12 and 6 instructions: forward gather, backward scatter, paired, tokamax branch only. The 3.6 memory peak’s 3 blocks tgmm_megablox.10/11/12 are the drhs ④ scatters. Both branches get weight_gather_axes (in this config, with experts sharded on FSDP and a weight calibration starting with fixed, the MoE layer passes on “gather along fsdp”), but branch ③⑤ ignores it. Missing forward gather and backward scatter match the profile exactly: all-gather and reduce-scatter both vanish on the SparseCores.

⭐ Why it is silent: backend.gmm already supports devices that own only some expert groups. Seeing 3 weight groups and 192 table groups, it builds metadata and computes rows only for those 3; the rest get 0. Every shape is legal; nothing raises an error.

# src/maxtext/kernels/megablox/backend.py · gmm
  num_current_groups = rhs.shape[0]          # L346: 3
  num_total_groups = group_sizes.shape[0]    # L347: 192
  …                                          # L374: make_group_metadata(…, num_nonzero_groups=rhs.shape[0]), table for 3 only
  if existing_out is None and num_current_groups < num_total_groups:      # L555
    out = _zero_uninitialized_memory(out, start_group=group_offset[0],
                                     num_nonzero_groups=rhs.shape[0], …)  # zero the other rows: jnp.where(valid_mask[:, None], out, 0)
Checking version and line numbers (public repo, which version)
  • The public Hy3 tuning log puts the code on the hunyuan3 branch of github.com/yangwhale/maxtext (upstream main plus three commits). Line numbers above: its 1afaa0fb.
  • Those three commits leave ops.py and backend.py untouched: both are byte-identical to upstream AI-Hypercomputer/maxtext (formerly google/maxtext) at 188af874 (2026-07-28), same line numbers.
  • Checked against the profile: each HLO instruction also records its trace-time Python call stack. The modified run’s MTP-layer gmm stack has ops.py:235 _gmm_fwd (line ③) and backend.py:548 (call_gmm); the two runs’ MoE calls are at moe.py:1504 (tokamax branch) and moe.py:1534 (megablox branch), all matching this version’s line numbers. Call stacks sometimes point to the wrong place; here it agrees with op_name, so it serves as corroboration.
Which 3 groups are computed (read from code, not verified on hardware)
  • The group start group_offset the MoE layer passes to the kernel depends on device index only with ring-of-experts on. Here it is off (use_ring_of_experts=False), so the offset is 0 everywhere: every device computes groups 0–2’s tokens.
  • Device 0 holds exactly experts 0–2, so its 3 groups are right (the one animated). Device r holds experts 3r to 3r+2 yet multiplies them by groups 0–2’s tokens [inferred from the code, not verified].
  • The tuning log’s standalone kernel test: 3,582 of 229,376 rows covered (about 1.6%, the same order of magnitude as 3/192 ≈ 1.6%), 788 µs vs the full 5,154 µs (documented, not reproduced here).

4.6 Section 3’s two puzzles

⭐ Most of the extra loop-fusion time is that zero-fill. The 9 unshrunk expert matmuls outside the loop are the MTP layer, whose experts the compiler gathers.

Section 3’s puzzleWhat HLO and the trace showConclusion
Modified: loop fusion +276 msModified: two extra kinds of loop fusion per layer. Forward broadcast_select_fusion (op_name ends in jit(gmm)/jit(_where)/select_n, output bf16[229376,4096]), 79 per step, 95.1 ms; backward select_add_fusion, 79 per step, 225.6 ms: three selects (also from jit(gmm)/jit(_where)) plus an add_any summing two inputs’ gradients. In the original that add_any is a standalone elementwise op (135.5 ms per step, under non-fusion elementwise) backend.py’s zero-fill (jnp.where): each layer writes 0 to 189/192 of 229,376 rows, once forward, fused with the add backward. The two new fusions total 320.7 ms: ~135.5 ms is the old add, recategorized (meanwhile 3.4’s non-fusion elementwise drops 133.7 ms), and the remaining ~185 ms is the zero-fill itself. Loop fusion as a category grows by only 276 ms because other loop fusions shrink by ~45 ms in total (some up, some down; not traced one by one here)
Which layer: 9 unshrunk out-of-loop gmmsThe median step has 21 expert matmuls outside the loop. 3 before the forward loop and 9 after the backward loop have while/body in op_name and [3,…] weights and gradients, about 0.05–0.07 ms each (the loop’s peeled layer, shrunk too); the other 9 lack while/body, have bf16[192,…] gradients, 4.2–5.2 ms each (per kernel, 3 steps on TPU:0; 3.3’s Op Stats averages 8 cores, hence 4.2–5.0). Call stacks pass through multi_token_prediction.py; weight parameter mtp_block/…/MoeBlock_0/wi_0 The MTP layer (multi-token prediction, the one layer outside the backbone). The compiler gathers its experts to 192 in bf16 (4.2 fold), bypassing those two lines: nothing skipped, equally slow in both

4.7 After the fix

⭐ With the gather and scatter moved out of the tokamax branch so both branches share them, a step takes 12.501 s, 6% slower than the original’s 11.761 s. The “50% faster” was all skipped work.

The fix moves ①’s all_gather and ④’s psum_scatter out of the if: gather whenever weight_gather_axes is set; scatter backward whatever was gathered forward. Post-fix numbers (Hy3 tuning log, TUNING-v7 §3.4.10): 12.501 s per step, 637.0 TFLOP/s per chip; original 11.761 s, 677.0 TFLOP/s. documented No profile was kept; we can only quote it, not recompute it.

Why errors like this stay silent The 192-expert dimension is split 3 per device, and 3 experts is a fully legal grouped-matmul input: the kernel just sees 3 groups. Shard the contraction dimension (say, 4096) instead and shapes would clash, failing on the spot [inferred]. If a sharded-down dimension is still valid input downstream, no error is raised. The only alarm left: “too good to be true”.
Section 5

A control: what does a real speedup look like?

📝 Notes for this section

Section 4 confirmed FP8's 3.9 s was skipped work. Conversely, what does a truly faster change look like on the same pages? No new example needed: the same change (use_tokamax_gmm: True → False plus the same 12 tile values) was also made on the BF16 recipe.

5.1 The same change makes a BF16 step 1.05 s faster

⭐ The BF16 configs differ exactly as the FP8 pair do; one step goes 21,747.2 → 20,693.8 ms, 1,053.4 ms (4.8%) faster.

Same changeFP8 (§0–4)BF16 (§5)
Config diffuse_tokamax_gmm True → False + 12 w{i,o}_tile_* values 1024 → 2048/1536. Both diffs identical (each over the full MaxText config)
One step (TPU:0 median step)11,758.2 → 7,842.1 ms (−3,916.1)21,747.2 → 20,693.8 ms (−1,053.4)
SparseCore all-gather / reduce-scatter (ms/step)1,409.3 / 2,354.0 → 205.8 / 02,522.8 / 2,543.1 → 2,593.7 / 2,606.5
HLO: 192-expert all-gathers / expert-gradient reduce-scatters (count)12 / 6 → 0 / 012 / 6 → 12 / 6
Expert matmuls (gmm family, ms/step)3,668.8 → 98.3 (−97%)8,183.8 → 7,116.2 (−13%)
Op Profile: custom-call FLOPS utilization53.39% → 117.45%45.57% → 49.62%
Op Stats: rank of the first expert matmul#4 → #124 (out of the top 10)#4 → #4 (still 7 in the top 10)
Expert weights the kernel receives[192,4096,1536] → [3,4096,1536][192,4096,1536] → [192,4096,1536]
HBM: largest buffers at peak (Memory Viewer)This layer's 3 in-loop [192,…] expert-gradient buffers → none left; peak moves to the output headSame in both: logits + gathered weights of all 192 experts

In the right column every row is unchanged or barely changed; only the step time and the expert matmuls move. In the left, every row from “SparseCore all-gather” down is an alarm. The next two subsections check the right column in the profile, row by row.

5.2 Same layer, same collectives

⭐ Backward layer 40 side by side. On the SparseCore, both runs have the 3 all-gathers of the 192 experts and the expert-gradient reduce-scatters, at nearly equal widths. On the TensorCore all 9 expert matmuls are present, 79.11 → 66.89 ms in total.

Figure 5.1A · BF16 Original · Backward, same layer (MoE layer 40). ① Expert matmuls ×9; ② All-gather of the 192 experts; ③ Expert-gradient reduce-scatter
BF16 Original · backward layer 40.
#WhatHow to tell (names and times from the xplane)
①Expert matmuls ×9TC: this layer's 9 expert matmuls (gmm/tgmm) total 79.11 ms
6 boxes in the main figure: gmm_megablox.11+gmm_megablox.12 (at 35,667.665 ms, 8.699/9.627 ms); gmm_megablox_transpose_rhs.8 (at 35,687.634 ms, 8.405 ms); gmm_megablox.13 (at 35,698.274 ms, 7.202 ms); gmm_megablox_transpose_rhs.9+tgmm_megablox.10 (at 35,714.635 ms, 9.961/10.053 ms); gmm_megablox_transpose_rhs.10 (at 35,737.722 ms, 10.414 ms); tgmm_megablox.8+tgmm_megablox.9 (at 35,751.081 ms, 7.827/6.926 ms)
②All-gather of the 192 expertsSC0: all-gathers 527/529/531 of the 192 experts’ weights: 4.86/4.90/4.86 ms, shapes bf16[1,192,4096,1536] etc.
2 boxes in the main figure: all-gather.527 (at 35,631.171 ms, 4.860 ms); all-gather.529+all-gather.531 (at 35,667.669 ms, 4.899/4.865 ms)
③Expert-gradient reduce-scatterSC0: expert-gradient reduce-scatters 198/200/202: 8.86/8.58/8.16 ms, shapes bf16[192,32,1536] etc.
2 boxes in the main figure: reduce-scatter.198+reduce-scatter.200 (at 35,758.909 ms, 8.855/8.581 ms); reduce-scatter.202 (at 35,777.205 ms, 8.163 ms)
BracketThis layer: 194.25 msWindow = this layer plus 0.5 ms each side; the Modified shot, same width, starts at its own layer
Figure 5.1B · BF16 Modified · Backward, same layer (MoE layer 40). ① Expert matmuls ×9; ② All-gather of the 192 experts; ③ Expert-gradient reduce-scatter
BF16 Modified · backward layer 40: ② the expert gather and ③ the gradient scatter are still there; ① all 9 expert matmuls present, each a bit shorter.
#WhatHow to tell (names and times from the xplane)
①Expert matmuls ×9TC: this layer's 9 expert matmuls (gmm/tgmm) total 66.89 ms (original, same layer: 79.11 ms)
6 boxes in the main figure: gmm.18+gmm.19 (at 34,093.767 ms, 8.632/9.635 ms); gmm.20 (at 34,113.694 ms, 7.437 ms); gmm.21 (at 34,123.365 ms, 6.214 ms); gmm.22+tgmm.8 (at 34,138.796 ms, 6.305/6.834 ms); gmm.23 (at 34,155.005 ms, 7.000 ms); tgmm.6+tgmm.7 (at 34,164.978 ms, 7.867/6.971 ms)
②All-gather of the 192 expertsSC0: all-gathers 550/552/554 of the 192 experts’ weights: 4.86/4.95/4.87 ms, shapes bf16[1,192,4096,1536] etc.
2 boxes in the main figure: all-gather.550 (at 34,057.312 ms, 4.857 ms); all-gather.552+all-gather.554 (at 34,093.767 ms, 4.951/4.872 ms)
③Expert-gradient reduce-scatterSC0: expert-gradient reduce-scatters 198/200/202: 9.08/8.74/8.33 ms, shapes bf16[192,32,1536] etc.
2 boxes in the main figure: reduce-scatter.198+reduce-scatter.200 (at 34,172.845 ms, 9.081/8.737 ms); reduce-scatter.202 (at 34,191.669 ms, 8.329 ms)
BracketThis layer: 181.88 msGray dashed line: layer end; right of it, the next layer (A and B window = original recipe's layer width)

Compare Figures 2.5A/B in Section 2: there, ② and ③ in the FP8 modified run's layer are empty dashed boxes; here they are solid, almost as wide as in the original (all-gather 4.86–4.99 ms, reduce-scatter 8.2–9.1 ms). ① differs too: in the FP8 modified run the 9 kernels add up to just 0.48 ms; here each of the 9 still takes 6–10 ms, each still doing a full share of the work.

Same forward layer, whole-step skeleton (Figures 5.2, 5.3: same conclusion)
Figure 5.2A · BF16 Original · Forward, same layer (MoE layer 40). ① Expert matmuls ×3; ② All-gather of the 192 experts
BF16 Original · forward layer 40.
#WhatHow to tell (names and times from the xplane)
①Expert matmuls ×3TC: this layer's 3 expert matmuls (gmm) total 23.55 ms: 8.76/7.58/7.21 ms
2 boxes in the main figure: gmm_megablox.14+gmm_megablox.15 (at 25,086.325 ms, 8.761/7.583 ms); gmm_megablox.16 (at 25,105.379 ms, 7.210 ms)
②All-gather of the 192 expertsSC0: all-gathers 553/555/557 of the 192 experts’ weights: 4.86/4.92/4.88 ms, shapes bf16[1,192,4096,1536] etc.
2 boxes in the main figure: all-gather.553 (at 25,051.390 ms, 4.857 ms); all-gather.555+all-gather.557 (at 25,086.327 ms, 4.923/4.883 ms)
BracketThis layer: 68.86 msWindow = this layer plus 0.5 ms each side; the Modified shot, same width, starts at its own layer
Figure 5.2B · BF16 Modified · Forward, same layer (MoE layer 40). ① Expert matmuls ×3; ② All-gather of the 192 experts
BF16 Modified · forward layer 40: all 3 all-gathers present; the 3 gmm kernels go 23.55 → 22.48 ms.
#WhatHow to tell (names and times from the xplane)
①Expert matmuls ×3TC: this layer's 3 expert matmuls (gmm) total 22.48 ms: 8.69/7.57/6.22 ms (original, same layer: 23.55 ms)
2 boxes in the main figure: gmm.24+gmm.25 (at 24,029.011 ms, 8.692/7.570 ms); gmm.26 (at 24,047.985 ms, 6.221 ms)
②All-gather of the 192 expertsSC0: all-gathers 580/582/584 of the 192 experts’ weights: 4.87/4.99/4.89 ms, shapes bf16[1,192,4096,1536] etc.
2 boxes in the main figure: all-gather.580 (at 23,994.200 ms, 4.866 ms); all-gather.582+all-gather.584 (at 24,029.013 ms, 4.992/4.886 ms)
BracketThis layer: 67.97 msGray dashed line: layer end; right of it, the next layer (A and B window = original recipe's layer width)
Figure 5.3A · BF16 Original · Skeleton of one step. ① One step: jit_train_step; ② Forward scan (78 iterations); ③ Backward scan (78 iterations)
One step of the BF16 original recipe.
#WhatHow to tell (names and times from the xplane)
①One step: jit_train_stepjit_train_step on the XLA Modules row, 21,747.23 ms (median of 3 complete steps)
②Forward scan (78 iterations)while.365, 5,378.13 ms, 78 iterations (the compiler peels layer 0 out in front of the loop; see the fold below)
③Backward scan (78 iterations)while.361, 15,145.27 ms, 78 iterations, layer index counting down
Figure 5.3B · BF16 Modified · Skeleton of one step. ① One step: jit_train_step; ② Forward scan (78 iterations); ③ Backward scan (78 iterations)
One step of the BF16 modified recipe: same skeleton, both scans slightly shorter.
#WhatHow to tell (names and times from the xplane)
①One step: jit_train_stepjit_train_step on the XLA Modules row, 20,693.81 ms (median of 3 complete steps)
②Forward scan (78 iterations)while.415, 5,303.16 ms, 78 iterations (the compiler peels layer 0 out in front of the loop; see the fold below)
③Backward scan (78 iterations)while.412, 14,191.68 ms, 78 iterations, layer index counting down
Open it yourself (same screen in local open-source XProf; first time, see “Opening it locally” in §2.1)
  • Runs bf16old/bf16new. Set Trace Viewer's view_start/view_end to the windows below (backward layer 40, in ms); for the same forward layer, use the second pair.
  • Backward: bf16old 35630.027–35825.278, bf16new 34056.155–34251.406. Forward: bf16old 25050.132–25119.996, bf16new 23992.641–24062.505.
  • Trim to TPU:0 and its two SparseCores as in §2.2. Click the SparseCore slices to read names and shapes: the all-gather outputs bf16[1,192,4096,1536], the reduce-scatter outputs bf16[192,32,1536] (the third is bf16[32,294912]: 294,912 = 192 × 1,536, the same gradient in a different layout).

5.3 Where the 1.05 s went: a little off every expert matmul

⭐ The step loses 1,053.4 ms, the custom-call category 1,054.1 ms, its expert matmuls 1,067.6 ms. The books balance: every kernel remains, each a bit shorter.

BF16 · per step (TPU:0 median step)OriginalModifiedΔ
One step21,747.220,693.8−1,053.4
custom-call (Pallas-kernel category)13,121.312,067.2−1,054.1
of which expert matmuls (gmm/tgmm)8,183.8 (957 calls)7,116.2 (956 calls)−1,067.6
Backward layer 40: 9 expert matmuls, total79.1166.89−12.22
Op Stats, first expert matmul: avg per call (µs)10,411.3 (gmm_megablox_transpose_rhs.10)9,640.6 (gmm.19)
Same kernel's Model GFLOP/s (per-core peak ≈ 1,137,000)475,232513,223

Three cross-checks: step-time Δ ≈ custom-call Δ ≈ expert-matmul Δ; kernel counts match (off by 1 per step, not chased); each kernel still at a plausible speed (Model GFLOP/s a little over 40% of peak), unlike the modified FP8 gmm.27's reported 58 million GFLOP/s (25× FP8 peak). Why each call is about 10% faster lies in kernel and tiles, not covered here. The profile shows: the same work, done faster.

The saving also maps back layer by layer: the 9 kernels in backward layer 40 lose 12.22 ms, the 3 in the same forward layer 1.07 ms (Figures 5.2A/B); 13.29 ms per layer × 78 layers ≈ 1,037 ms, only a dozen or so ms short of the step's 1,053.4 ms.

The two summary pages show the same picture:

5.4 Why BF16 avoided the trap

⭐ BF16 shards expert weights along the 4096 dimension, not the expert one. The compiler inserts the gather itself (visible in the HLO), so on either branch the kernel gets all 192 experts.

The Hy3 tuning log (TUNING-v7 §3.4.10) gives three trigger conditions for the FP8 skipped work; all must hold. BF16, one by one:

Trigger conditionFP8BF16
① shard_exp_on_fsdp=True and a fixed… weight calibration → the code passes down “gather experts along fsdp”HoldsDoesn't hold: shard_exp_on_fsdp=False; BF16 isn't quantized either (quantization=NONE), so no calibration at all
② use_tokamax_gmm=False: the native branch runsHoldsHolds (same change)
③ Expert count divisible by the FSDP shard count, so the expert dimension really is splitHolds
192÷64=3
Doesn't hold: ici_fsdp_parallelism=-1 = 128 shards, 192 ÷ 128 isn't whole; ① already keeps the experts unsharded our derivation

The HLO shows the result: each BF16 device holds 192 experts, only 32 of 4096 rows each; the compiler inserts the all-gather itself:

── BF16 Modified · forward loop body (trimmed)
all-gather.580 = bf16[1,192,4096,1536] all-gather(param_0.11980),     # input param_0.11980 = bf16[1,192,32,1536]
    replica_groups=mesh['axis_0'=1,'axis_1'=128,'axis_2'=1] {'axis_1'}, dimensions={2},
    frontend_attributes={is_spmd_generated="true"}
gmm.24 = bf16[393216,1536] custom-call(…),
    operand_layout_constraints={…, bf16[393216,4096]{1,0}, bf16[192,4096,1536]{2,1,0}}

It joins 128 shards along dim 2 (from 0: the 4096 one): 32 × 128 = 4096. It carries is_spmd_generated="true"; no line of code has it (the three attributes in the §4.1 fold). gmm.24 gets weights [192,4096,1536]; 393,216 = 12 sequences per device × 4,096 tokens × 8 experts. The expert dimension stays whole, so there is no “3 experts is also a valid input” trap: the flip side of the §4.7 note.

5.5 Real speedup vs. skipped work, page by page

⭐ Real speedup: same work count and shapes, same kernels each a bit shorter, utilization still under 100%. Skipped work: a whole class of work changes in count or shape, and utilization exceeds the physical limit.

Where to lookReal speedup (BF16)Skipped work (FP8)
Estimate first (§1)+5%: expert matmuls are about 38% of a step (8,183.8 ÷ 21,747.2); 13% off them is right about 5% of the step+50%: a comparable MoE gains only +21% from FP8; this one matches the dense 405B's +53%
Trace Viewer, same layer (§2)Every row still there, slices narrowerTwo whole classes of SparseCore collectives gone; kernels shrink to 1/75–1/93
Op Profile / Op Stats (§3)Utilization 49.62%; rank and call count unchangedUtilization 117.45%; rank 4 → 124; a single kernel reports 25× the FP8 peak
HLO (§4)Same collective count, same kernel weight shapesExpert all-gathers 12 → 0; weights [192,…] → [3,…]
Memory (§3.6)Peaks 2 MiB apart; same largest buffers at the peakPeaks 169 MiB apart, but the 3 in-loop 192-expert gradient buffers are gone; peak shifts from the backward pass to the output head
Section 6

Wrap-up: how to investigate next time

📝 Notes for this section

Looking back, a number looked too good to be true. Neither doubting nor cheering, we read the pages in a fixed order, one question per step.

6.1 Diagnosis order: five steps

⭐ Overview characterizes → Trace Viewer locates → Op Profile / Op Stats reconcile → HLO finds the cause → code. Confirm each step before moving on; each one gives you a number to read.

Next time a number looks too good: five steps from characterizing it to one line of code Each step answers one question; confirm it before moving on. The bottom row of each column is what this case showed XProf page The program What we saw Start: estimate with the rulers first (MFU, gains of similar models) Here: an MoE got a dense model's FP8 gain, +53%. Not necessarily wrong, but worth opening the profile. ① Overview: characterize overview_page Ask TensorCore idle or busy? See Step time, TPU Duty Cycle, TensorCore idle Both Duty Cycles at 99.9%: the lost time isn't idle time ② Trace Viewer: locate trace_viewer Ask Which segment, which row? See Trim to one step, one layer; same layer, A above B SparseCore expert-gather and grad-scatter comms: all gone ③ Reconcile the books op_profile · hlo_stats Ask Which work lost the time? See Utilization by category; per-kernel count, time per call custom-call 117.45%; expert matmul: #4 → #124 ④ HLO: find the cause Graph Viewer · HloProto Ask What should be there? See Count collectives by shape; operands the kernel gets expert all-gather 12 → 0; weights [192,…] → [3,…] ⑤ Back to the code op_name → grep Ask Which branch? See op_name tail → the call; then read that branch Gather and scatter exist only in the tokamax branch Too good to be true: open the profile before you celebrate Real speedup: every page still shows the same things, only with narrower slices and utilization under 100% (BF16, same change). Skipped work: some page always shows a whole class of work gone, or a number past a physical limit (FP8: comms gone, 117.45%).
The diagnosis order: five steps, one question each. Red dashed boxes: what this case showed (§1–§5).
Sources and assumptions

📌 Bottom-row numbers: Duty Cycle 99.9% (§2.1 Overview); SparseCore comms gathering experts / scattering gradients (§2.4, same layer); 117.45%, rank #4 → #124 (§3.1–3.3 Op Profile / Op Stats); all-gather 12 → 0, weight shapes (§4.2, §4.3 HLO); the branch (§4.5, public MaxText code).

📌 Page names are short forms of the names in the Tools drop-down on the open-source XProf 2.23.1 home page (e.g. HLO Op Profile → Op Profile); HloProto is the compiled program stored in the profile (§4.1).

Where to look for common symptoms: new profile, new symptoms, same reading. Below are our twelve readings, each with this course's example; for over-long names see the §2.3 fold “Reading event names left to right”. Enter via this decision tree:

Symptom → where to look: walk down from a question to a number, then read that row of the symptom table Circled number = # column of the symptom table; box color = the layer you end on; gray text at the bottom of each box = this case's example Overview Trace Viewer Op Profile · Op Stats · Roofline · Memory HLO Code First: what kind of problem is it? Step time moved: where? Too good to be true Did a change land? Memory is tight Whose slice, from where? 1 TensorCore idle or busy? Overview: step time, Duty Cycle Both at 99.9% 2 Which segment differs? Same layer, A above B SparseCore comms all gone 3 TensorCore stalls on a done Wide done: match right edges all-gather.548 done: 1.18 ms 4 Absurd category utilization Op Profile, expand to a kernel gmm.27 group: 5,147.80% 8 How far from the roof? Roofline, filter by name gmm.27 plotted off the chart 11 Two pages disagree Measured, or booked by shape? 117.45% vs Wall Duration 5 Did it really apply? HLO in Details: dtypes, shapes weights [192,…] → [3,…] 6 Expected work missing Count comms, operands by shape all-gather 12 → 0 12 Which kernel parameters? kernel config in the HLO text 375 s run: all default tiles 9 Who owns peak memory? Memory Profile → Viewer 3 [192,…] grads in the loop 7 What are these slivers? Read a column: name path, line gmm_megablox.16 → ops.py:106 10 Which line of code? op_name, Source → the branch → two branches in ops.py Down a box = not answered yet; go on to the lower-level page The 12 rows are really one route: Overview characterizes → Trace locates → reconcile the books → HLO finds the cause → code Whichever entry you take, the box you stop at is always a station on this route; confirm one station before moving to the next.
Pick an entry, walk down to a number, read that row below. Numbers = the table's # column; color = the layer you end on.
Sources and assumptions

📌 Circled numbers match the symptom table's # column (rows 1–12); each box's “what to look at” and gray example abbreviate that row; every number is in the table and its section.

📌 Box color = the last page the row's “Confirm by” reads: 5 and 12 read HLO text in Details or the full HLO, so HLO; 7 reads Trace Viewer's name paths and source-line labels, so Trace Viewer; 11 reconciles two summary pages, so the summary layer.

#SymptomWhere to lookConfirm byThis case
1Step time shift: compute or waiting?Overview: avg step time, TPU Duty Cycle; Step-time Graph's TensorCore idleDuty Cycle near 100%, idle only a few ms → not idle; see what the TensorCore does in Trace Viewer§2.1: 99.9% in both
2Where two profiles differTrace Viewer: same layer, stacked (equal-width windows via view_start / view_end)Row by row: what narrowed, what vanished§2.4: SparseCore expert gather and grad scatter, gone
3TensorCore stalls on a doneTrace Viewer: click a wide *-done; below, find the SparseCore slice with the same right edgeEnding together = TensorCore waits for the SparseCore transfer; done width = exposed time§2.5: all-gather.548 done, 1.18 ms
4A category's utilization is absurdOp Profile: Group by Category → expand to one kernel → left panelA kernel's FLOP rate above hardware peak = work skipped or mis-booked; check its per-call time in the other profile, operand shapes in HLO§3.2: gmm.27 group 5,147.80%
5Did the change really apply?Trace Viewer: click the instruction; read HLO types, shapes in DetailsTypes and shapes, not names, not what the config file says§4.3 recording stops 2, 12: weights [192,…] and [3,…]
6Something expected is missingHLO: count collectives, kernel operands by shape (Graph Viewer, or the profile's full HLO)Compare counts by shape in both programs: the profile records only what ran; what didn't is countable only in the program§4.2: expert all-gather 12 → 0
7A layer of unknown sliversTrace Viewer: expand Framework Name Scope, Framework Ops, Source code; read a columnStacked = contained, count leaves only; name path segment 2: jvp() is forward, transpose(jvp()) is backwardFigure 2.3C: gmm_megablox.16 leads to ops.py:106
8How far a kernel is from its ceilingRoofline Section2: filter Operation by nameCan be real only if under the roof and below 100%; off the chart = booked ≠ done§3.2 recording: gmm_megablox.16 81.7%, gmm.27 off the chart
9Memory tight: who holds the peak?Memory Profile: peak time (Memory ID ≠ 0) → HLO's at time picks the program → Memory Viewer on it, hover bars in by Buffer SizeLeft-panel Framework Op → code; blocks without one are compiler-inserted, code can't remove them§3.6: original recipe's peak holds 3 loop-level [192,…] expert grads; modified: none
10Slice back to codeop_name (Framework Op), Source in Details; same instruction's op_name in HLOTrust name paths: …/shard_map/all_gather, tokamax, jit(gmm)/pallas_call point straight to the code branch§4.5: op_name → two branches in ops.py
11Two pages disagreeFirst check what each reads: measured time, or compile-time FLOP, byte, memory books by shapeTrust measured time; FLOPs, bytes are booked by shape, even for skipped work§3.2: 117.45% vs Wall Duration; §3.6: memory peaks 3.6 GiB apart (data-source figure below)
12Config changed: what did the kernel use?Kernel config in Details' HLO text, or counted by kernel name in the full HLOWhat was compiled, not the config file§3.5: 375 s run, all default tile 128×128×128

When two pages disagree, check which data in the file each reads

⭐ The pages are views of one xplane, not seven data sources. If they disagree, check what kind each reads: trust measured durations; compile-time flops, bytes, line numbers may be estimated by shape.

When two pages disagree, ask first: do they read the same kind of data? Counted live from the fp8old xplane (1.22 GiB), only TensorCore TPU:0 and its two SparseCores. Solid boxes: measured at runtime; dashed: written at compile time ① Device timeline ② Op metadata ③ HloProto ④ Allocation events 1 Device timeline runtime-measured TensorCore XLA Ops: 488,640 events Async XLA Ops 48,045 Sparse Core Ops 19,677 / 40,179 XLA Modules (module runs): 45 each event: start and duration plus a Time Scale Multiplier counters_0 counters: 94,039 2 Op metadata written at compile time TensorCore: 4,256, one per instruction category, flops, bytes_accessed memory_access_breakdown (bytes booked per memory space) shape 3,679 · op_name 3,030 source line 2,883 Pallas: self-reported cost_estimate 3 HloProto compiler output itself 14 modules in /host:metadata jit_train_step 13.3 MiB every instruction, shape, edge buffer assignment (compile time) the copy in the profile directory: *.hlo_proto.pb is that copy 4 Allocation events runtime-measured /host:CPU MemoryAllocation 672 MemoryDeallocation 584 each: in-use bytes, peak, fragmentation the runtime allocator's records Each tool page reads only some kinds (badges at top right = which) Trace Viewer 2 1 Slices are ①; HLO, op_name and source line in Details are ② Overview 1 Step time, Duty Cycle, TensorCore idle: all from ① Op Profile / Op Stats 2 1 ① durations summed by ② category; utilization = ② flops ÷ ① duration Roofline 2 1 ① durations paired with ② flops and two byte counts Graph Viewer 3 draws ③ as a graph Memory Viewer 3 ③ buffer assignment: compile time, one module Memory Profile 4 ④ records: runtime, per device HLO text 3 Graph Viewer's HLO and *.hlo_proto.pb both come from ③ Utilization Viewer ? No Data on all 7 profiles; which kind it reads: not checked Four times in this course two pages disagreed: each time the two sides read different data; neither page was wrong 5 117.45% vs Wall Duration 2 1 Op Profile utilization: numerator is the kernel's self-reported flops in ② (by shape, same in all 3), over ① time Wall Duration reads only ①: real time → trust durations; flops are estimates 6 5,674.9% vs 5,147.80% 2 1 Roofline and Op Profile both read ①②; Op Profile's peak is also multiplied by each ① event's Time Scale Multiplier (modified, time-weighted 1.1053) → same kernel, denominators 1.1× apart 7 93.23 GiB vs 91,766.56 MiB 3 4 Memory Profile reads ④: runtime, Memory ID 1 (stack 49.39 + heap 43.85 GiB) Memory Viewer reads ③: compile time, jit_train_step only (89.62 GiB) → both right, different questions 8 Left of ridge, labeled Compute 2 x position uses ② bytes_accessed (253.5 FLOP/Byte; ridge 279.1) Bound by uses another count in ②, memory_access_breakdown (386.2) → even one kind has two byte counts
One xplane, four kinds of data: each tool page reads a few. Top row first, then each page's badge; each of the four mismatches below traces to the kinds its sides read.
Sources and assumptions

📌 Counts: read live from the fp8old xplane with xplane_pb2 (TensorCore /device:TPU:0, SparseCore 0 / 1, /host:metadata, /host:CPU). The full xplane has 28 planes (8 TensorCore, 16 SparseCore); only the first TensorCore (/device:TPU:0) and its two SparseCores are counted here.

📌 Time Scale Multiplier: each TensorCore XLA Ops event carries one (1.15789 or 1.0 in the original run); weighted by duration it averages 1.10536, matching the 1.10537 atop the original's Roofline page; the modified run's page shows 1.10529. Both percentages come from the modified run: 5,674.9% ÷ 1.10529 = 5,134%, within the gmm.27 group's 5,134–5,158% in the expanded Op Profile; 5,147.80% is the group average.

📌 Memory: the Memory Profile peak is read at Memory ID 1 (capacity 94.74 GiB; IDs 1–7 are identical; the default ID 0 records 128 extra small allocations, its 93.28 GiB peak at the window's start). Memory Viewer shows jit_train_step (91,766.56 MiB; 94,009.15 MiB with fragmentation). openxla.org/xprof says Memory Viewer data is “purely static, obtained from the XLA compiler”.

📌 The Roofline's two byte counts: bytes_accessed ÷ duration = the table's Memory BW; HBM bytes in memory_access_breakdown ÷ duration = the table's HBM BW. Recomputed for splash, gmm_megablox.16 and the two dones, both match the page digit for digit.

📌 Utilization Viewer: No Data on all 7 profiles; this course didn't check what it reads or why it's empty.

The top row is what the fp8old xplane actually holds, counted live with xplane_pb2: solid boxes were recorded at runtime (event starts, durations; each allocation and free), dashed ones fixed at compile time (per-instruction category, flops, bytes, shape, op_name, source line; the compiled program). Middle-row badges: kinds each page reads. Bottom row, our four mismatches: ⑤ §3.2's 117%, ⑥ §3.2 recording stop 3 vs Op Profile's left panel, ⑦ §3.6 memory, ⑧ the Roofline Section1 point above. Each time, the two sides read different kinds; no page was wrong.

6.2 Three checks before celebrating

⭐ When a number looks great, first check three things: do the rulers agree, is the same work still there, does anything exceed a physical limit?

CheckHowReal speedup (BF16)Skipped work (FP8)
① Rulers agree?Convert to MFU; compare with similar models' gain from the same change+5%: matches expert matmuls 13% faster, scaled to the stepMoE gains as much as the dense model's +53%
② Same work still there?Same layer side by side; key-kernel call counts (Op Stats); HLO collective counts by shape; big buffers at memory peakEvery row there, just narrowerTwo whole classes of SparseCore collectives gone in the same layer (expert all-gathers, gradient reduce-scatters), 12 → 0 and 6 → 0 in HLO; in-loop [192,…] gradient buffers gone
③ Beyond a physical limit?Op Profile utilization; one kernel's FLOP rate, HBM bandwidth vs. hardware peak49.62%117.45%; one kernel at 25× peak

Only after all three pass comes the next question: does the loss curve still match the one before the change? That is numerical correctness, beyond what a profile can tell; compare over enough steps.

6.3 Capture and open your own

⭐ Set profiler=xplane in MaxText, capture a few steps, give the output's tensorboard/ to xprof --logdir: every screen here opens on your machine.

No need to capture this course's 7 profiles: get them from the GitHub Release. Each unpacks to <run>/plugins/profile/<timestamp>/<host>.xplane.pb, the same layout as your own.

The download, unpack and open commands are in the §2.1 fold “Opening it locally”.

MaxText configValue in these runsWhat it does
profilerxplaneEmpty by default (no capture); only xplane captures a TPU profile
skip_first_n_steps_for_profiler5 (4 for the 375 s run)Skips the first few steps, keeping compile/warm-up out of the profile
profiler_steps3Steps to capture. Take 3, read the median-length one (as throughout this course)
profile_cleanlyTrue (default)Waits at step end for the training state to finish, so profile step boundaries line up
base_output_directory, run_nameYour own bucket and nameThe profile goes to <base_output_directory>/<run_name>/tensorboard/
plugins/profile/<timestamp>/<hostname>.xplane.pb
# Add to the training command (other flags unchanged)
python3 -m maxtext.trainers.pre_train.train run_name=myrun base_output_directory=gs://<bucket>/out <your other flags> \
    profiler=xplane skip_first_n_steps_for_profiler=5 profiler_steps=3

# After the run, copy the tensorboard/ directory back and start open-source XProf
gcloud storage cp -r gs://<bucket>/out/myrun/tensorboard ~/prof/myrun
pip install xprof
xprof --logdir ~/prof --port 6311        # open http://127.0.0.1:6311/ in a browser; myrun is under Sessions in the left pane
Next: try it yourself Twenty-four exercises on the 7 Hy3 profiles, each solvable in open-source XProf alone. Download from the Release (commands in §2.1 “Opening it locally”); for your model, capture with the switches above and repeat: Exercises · find out yourself in open-source XProf.

← Course overview · Previous: Topic 6 · Inference · Instructor notes · Exercises