Download the 7 profiles (GitHub Release), unpack into one logdir, a run each, and open in open-source XProf (as in §2.1 “Opening it locally”).
The course's 7 profiles are public in GitHub Release topic07-hy3-profiles-v1 (Hy3 training on 64 TPU v7 chips; each is 3 steps from one host, 1.2–1.45 GB unpacked; only host names changed, to gke-node-N). Four copy-ready steps take you from download to an open profile:
# 1. Download (7 assets, 0.86 GB in total; if you only want one or two profiles, download just those)
mkdir -p ~/hy3-prof && cd ~/hy3-prof
B=https://github.com/yangwhale/gpu-tpu-pedia/releases/download/topic07-hy3-profiles-v1
for r in fp8old fp8new fp8nat r3anat fp8tk bf16old bf16new; do curl -fLO $B/hy3-$r.tar.zst; done
curl -fLO $B/SHA256SUMS && sha256sum -c SHA256SUMS
# If you downloaded only some, add --ignore-missing; on macOS use shasum -a 256 -c SHA256SUMS
# 2. Unpack (9.2 GB in total): each tar already contains <run>/plugins/profile/<timestamp>/<host>.xplane.pb
# No zstd command? Install it first: apt install zstd / brew install zstd
for f in hy3-*.tar.zst; do zstd -dc "$f" | tar -xf -; done
# 3. Install open-source XProf and point it at this directory
# ==2.23.1 is the version used for this course's screenshots; drop it to install the latest
pip install xprof==2.23.1
xprof --logdir ~/hy3-prof --port 6311
# You can also use TensorBoard: pip install tensorboard tensorboard-plugin-profile
# then tensorboard --logdir ~/hy3-prof --port 6311 and go to the PROFILE page
# 4. Open http://127.0.0.1:6311/ in a browser
# In the left panel, pick a run from the Sessions drop-down (e.g. fp8old/2026_08_15_03_57_28) and a tool under Tools<logdir>/<run>/plugins/profile/<timestamp>/<host>.xplane.pb; each subdirectory of the logdir is one run. Unpacked, ~/hy3-prof holds 7 run directories; lay out a profile of your own the same way to view it alongside.| run | What it is | Median step (ms) | Unpacked file |
|---|---|---|---|
fp8old | FP8 original recipe (tokamax, injected tiles 512/2048/1536) | 11,758.2 | fp8old/plugins/profile/2026_08_15_03_57_28/gke-node-1.xplane.pb |
fp8new | FP8 modified recipe (native + 12 new tile values): the one 50% faster | 7,842.1 | fp8new/plugins/profile/2026_08_15_04_11_46/gke-node-1.xplane.pb |
fp8nat | A rerun of fp8new | 7,846.1 | fp8nat/plugins/profile/2026_08_15_16_59_57/gke-node-4.xplane.pb |
r3anat | Another rerun of fp8new | 7,840.1 | r3anat/plugins/profile/2026_08_16_01_29_28/gke-node-5.xplane.pb |
fp8tk | Original recipe without injected tiles (default 128×128×128): the 375 s run | 375,160.0 | fp8tk/plugins/profile/2026_08_15_17_48_55/gke-node-4.xplane.pb |
bf16old | BF16 before the change (tokamax, injected tiles) | 21,747.2 | bf16old/plugins/profile/2026_08_15_09_13_02/gke-node-2.xplane.pb |
bf16new | BF16 after the same change (native + same 12 tile values) | 20,693.8 | bf16new/plugins/profile/2026_08_15_09_28_25/gke-node-3.xplane.pb |
view_start/view_end (ms from the profile's start); the “↗ Open” after a window opens that exact screen, the one after a tool opens that run's page. Links assume a local XProf on port 6311; if yours differs, change 6311 in the link./device:TPU:0 and its SparseCore 0/1 → Expand All. Start and Wall Duration in Details are in ns.view_start/view_end and shows the whole session; to open one layer, use the “↗ Open” after the exercise's window (it opens trace_viewer_index.html directly).jit_train_step once by hand (main text §4.3, recording stops 3–4).jit_add by default; switch to jit_train_step in the Hlo Modules drop-down. The Module Name row may then still show the old name; go by the numbers below it.gmm, click the = on its right to switch to Contains.utilization_viewer) gives a 404 on every profile in 2.23.1, error No Data. It's a bug in this version: the tool is routed to a processor that was never registered (the XProf log says No ProfileProcessor registered for tool: utilization_viewer); fixed in 2.23.2. The data is there: in fp8old, e.g., the /device:TPU:0–3 planes each have a counters_0 row of 94,039 hardware-counter events. To see this page, install 2.23.2 or later (the course's screenshots are all from 2.23.1).After this level you can read any profile's step time, box one step, and identify the communication in a layer. Goes with main-text Section 2.
Open fp8tk's Overview (tool overview_page) ↗ Open: its average step time is how many times the original recipe's (fp8old↗ Open)?
fp8tk 375,326.7 ms, fp8old 11,793.8 ms: about 31.8×.
How to tell: Average Step Time, left column of both Overviews. It splits steps by the host's Steps row and averages the 3, so it is 0.04% and 0.3% above, respectively, the median step from clicking jit_train_step in Trace Viewer (375,160.0/11,758.2, 31.9×). Different measures, same conclusion. Why so slow: Exercise 6.
Open fp8nat's Trace Viewer (no window: the whole session) ↗ Open, click the 3 jit_train_step blocks on the XLA Modules row and read each Wall Duration. How far is the median step from fp8new's 7,842.1 ms?
Steps: 7,846.7/7,846.1/7,845.4 ms; the median is the second block (from 8,239.931 ms), 7,846.1 ms, 4.0 ms (0.05%) above fp8new. The skipped-work result reproduces; it was not a one-off.
How to tell: 3 steps, 3 blocks; read XLA Modules, not Steps (main text §2.3 row table). f makes the selected step fill the screen.
Open bf16new, backward layer 40: view_start=34056.155&view_end=34251.406 ↗ Open. On SparseCore 0's Sparse Core Ops row, how many all-gathers gather the 192 experts, and how wide is each? And the reduce-scatters of the expert gradients? Click to read shapes.
all-gather.550/552/554: 4.86/4.95/4.87 ms, outputs bf16[1,192,4096,1536] ×2 and [1,192,1536,4096]; reduce-scatter.198/200/202: 9.08/8.74/8.33 ms, outputs bf16[192,32,1536] ×2 and bf16[32,294912]. Almost exactly as wide as the same layer in the original recipe, bf16old: not a single collective is missing (main-text Figure 5.1B).
How to tell: reduce-scatter.202's shape has no 192, but 294,912 = 192 × 1,536: it is the third expert matrix's gradient in another layout (trace its input up the HLO and you reach a tgmm's output). Also, all-gather.550 appears twice in this window (at the layer's start and end); whether that is two, or the end one belongs to the next layer, this course didn't chase it.
After this level you can say which SparseCore collective a done waits on and how much of it is hidden, tell a real speedup from skipped work on the two summary pages, and match the missing time to the work never done. Goes with main-text Sections 2, 3 and 5.
Open fp8old: view_start=31223.000&view_end=31227.000 ↗ Open (in backward layer 40). On the TensorCore's XLA Ops row, read the width of the all-gather.548…call-done slice; below, find the SparseCore slice with the same right edge. How long is the collective in total, and how much is hidden?
The done spans 31,225.002–31,226.183 ms (1.181 ms); it lines up with all-gather.547 on SparseCore 0 (31,223.777–31,226.183, 2.406 ms, f8e4m3fn[1,192,4096,1536]), right edges 0.1 µs apart. Hidden 2.406 − 1.181 = 1.225 ms; exposed 1.181 ms.
How to tell: the done's width is exactly the TensorCore's stall; the SparseCore's number is 1 lower (547 vs 548). Click the SparseCore slice first, then the done; the other way, Details grows and pushes the SparseCore row off screen. Main text §2.5, Figure 2.6.
Open bf16new's HLO Op Profile (tool op_profile) ↗ Open, set Group by to Category, and read the overall FLOPS utilization and the custom-call row's FLOPS. Compare with the same cells for fp8new↗ Open: which run is likely skipping work?
bf16new: 22.55% overall, custom-call 49.62% (bf16old: 21.44% and 45.57%). fp8new's custom-call is at 117.45%, above the peak this page itself uses: fp8new is the one skipping work.
How to tell: the BF16 cell rose by 4 points, matching the share of time saved; in FP8, time fell by more than half yet utilization doubled. “Over 100%” alone is not enough to conclude (collapsible block in main text §3.2): expand to one kernel and check whether its FLOP rate exceeds the hardware peak.
Open fp8tk's HLO Op Stats (tool hlo_stats) ↗ Open. Which op ranks first, and how long is each occurrence (read Avg. time (us))? Filter by name for gmm_megablox.16: how long is each occurrence, and how many times as long as the same-named kernel in fp8old?
First: tgmm_megablox.10, 418,120.7 µs per occurrence; gmm_megablox.16 takes 373,915.2 µs each against 3,432.0 µs for the same name in fp8old: 108.9× slower (Model 7,719 GFLOP/s). The top ten are all expert matmuls.
How to tell: both runs show 1,872 occurrences (78 iterations × 3 steps × 8 TensorCores): the same amount of work, with each kernel more than a hundred times slower. The cause: no injected tiles, so it falls back to the default 128×128×128 (main text §3.5). The Model GFLOP/s column is hidden by default; tick it in the Visible Columns drop-down above the table to see it.
Open fp8old's HLO Op Stats↗ Open, set the HLO op name box to Contains and type gmm. Sum the remaining rows' Total self time (us) (hidden by default: tick it under Visible Columns; with no child ops, a custom-call's equals the default-visible Total time (us)) and divide by 24 (8 TensorCores × 3 steps): expert-matmul ms per step per core? Repeat for fp8new↗ Open. Estimate from “only 3/192 was computed”: expected saving, actual saving, gap? Why slightly off the main text §3.4 table's 3,668.8/98.3 ms? Save the code below as opstats_sum.py to sum the thirty-odd rows via the table's data endpoint (the page's table reads from it; if XProf isn't on 6311, change the script's port):
import gzip, json, sys, urllib.parse, urllib.request
run, key = sys.argv[1], sys.argv[2] # run looks like fp8old/2026_08_15_03_57_28; key is a name fragment
host = sys.argv[3] if len(sys.argv) > 3 else "gke-node-1"
q = urllib.parse.urlencode(dict(run=run, tag="hlo_stats", host=host))
b = urllib.request.urlopen("http://127.0.0.1:6311/data/plugin/profile/data?" + q).read()
try: b = gzip.decompress(b)
except OSError: pass
d = json.loads(b); ids = [c["id"] for c in d["cols"]]
rows = [dict(zip(ids, [c.get("v") if c else None for c in r["c"]])) for r in d["rows"]]
m = [r for r in rows if key in r["hlo_op_name"]]
occ = sum(r["occurrences"] for r in m); us = sum(r["total_self_time"] for r in m)
print(len(m), "rows", int(occ), "occurrences", round(us / 1000, 1), "ms; ÷24 =", round(us / 24000, 1), "ms")
python3 opstats_sum.py fp8old/2026_08_15_03_57_28 gmm
python3 opstats_sum.py fp8new/2026_08_15_04_11_46 gmm
fp8old: 33 rows, #Occurrences totaling 22,968 (= 957 × 24, so ÷24 is right), Total self time totaling 87,860.6 ms: 3,660.9 ms per step per core. fp8new: 37 rows, 33 of them expert matmul kernels (again 22,968 occurrences, 99.1 ms per step per core); the other 4 are broadcasts whose names happen to contain gmm (such as jit_gmm_.688), about 0.09 ms per step in all; 99.2 ms with them. Expected saving 3,660.9 × 189/192 = 3,603.7 ms; actual 3,561.7 ms, 98.8% of the estimate, about 1% short: the two ends reconcile.
How to tell: both runs have the same counts (957 per step per core) while time fell 97%: the kernel is still launched every time, each doing only 3/192 of the work (main text §3.4, Section 4). The 0.2% (original recipe) and 0.9% (modified) gaps to the §3.4 table come from a different measure: Op Stats averages 8 TensorCores and 3 steps, §3.4 uses TPU:0's median step; each measure on its own gives “actual ÷ estimate” of about 99%. Contains matches only a name fragment, so other ops whose names happen to contain gmm get in too: the modified run's 4 extra rows have category broadcast, Framework op name jit(gmm), shapes s32[192] and s32[639], under 1 µs each. Before reconciling, scan the category column and drop anything not a custom-call; all 33 fp8old rows are custom-calls.
After this level you can read from Details' raw HLO, Graph Viewer and exported HLO text an instruction's inputs, how many such instructions the program has and a loop's hard-coded iteration count, and find matching numbers in Memory Profile. Goes with main-text Section 4, §2.3, §3.6.
Open bf16new, forward layer 40: view_start=23992.641&view_end=24062.505 ↗ Open. Click the first gmm.24 and read the raw HLO under Start Stack Trace in Details: what shape is the expert-weight operand? And the input-token operand?
Weights bf16[192,4096,1536]: all 192 experts are there. Input bf16[393216,4096], where 393,216 = 12 sequences per device × 4,096 tokens × 8 experts per token.
How to tell: quickest is the operand_layout_constraints part, which lists every operand's shape in order (main-text Figure 4.1 ⑨). Set it against fp8new's gmm.27 at Section 4's recording stop 12 (f8e4m3fn[3,4096,1536]) and you have Section 5's conclusion.
Open fp8old's Memory Profile (tool memory_profile) ↗ Open and read the in-window peak memory usage and the total capacity.
Peak 93.28 GiB, capacity 94.74 GiB: only about 1.5 GiB left. With Memory ID 1 it is 93.23 GiB, at 23,606.7 ms.
How to tell: read Memory Capacity and Profile-window Peak Memory Usage in the Memory Profile Summary at the top (the Free Memory line below says 1.46 GiB): almost full. Don't read the Lifetime Peak Heap Usage line above it (43.89 GiB): it counts only the heap, not the stack. The default is Memory ID 0, whose peak falls 1.4 ms into the window, before the training step starts; main text §3.6 switches to 1 and rereads so the peak lands inside the training step. The two differ by only 0.05 GiB. To see which buffers take the bulk, look by buffer in Memory Viewer (main text §3.6; BF16 run: Exercises 23 and 24).
Open fp8old's Graph Viewer (tool graph_viewer) ↗ Open, click gmm_megablox.16 in the quick-start list at the page bottom and draw the graph around it. Its incoming edges are numbered: which delivers s32[193]? Which node does edge 7 come from, what shape, and what is above it? In fp8new, click gmm.27: edge 1 and operand 7 (no such edge drawn? read the node box's operand lines)?
fp8old: edge 1 is copy-done.141, s32[193]; edge 7 is bitcast.3183, f8e4m3fn[192,4096,1536], fed by the call-done of all-gather.574. fp8new: edge 1 is copy-done.122, s32[193]; operand 7 has no edge, and the node box of gmm.27 says operand 7 = tuple-element 1 of fusion.1642, f8e4m3fn[3,4096,1536].
How to tell: in fp8old, the big titled box further up is the sub-computation all-gather.573; in fp8new, fusion.1642 is this device's 3 experts after quantization. An edge's number is the operand position, in the order of the operands in Details' raw HLO (Exercise 8's reading). An operand taken from a tuple (tuple-element) gets no edge; it is written in the node box's operand lines: operand 4 in the fp8old graph and operands 0, 4, 6, 7 and 8 in the fp8new graph. s32[193] is the group offsets for the 192 token groups, the same on both sides: tokens still arrive in 192 groups, but after the change the kernel holds weights for only 3 experts (main text §4.3, Figure 4.2). If the graph is too small, zoom with the scroll wheel, or click Save SVG and zoom the saved file; if you came from a link in Details, the drop-down sits on another module, so pick jit_train_step by hand first.
Graph Viewer draws only a small neighborhood around one instruction. Counting over the whole program needs the HLO text: once you've opened this run in Graph Viewer or Memory Viewer, the profile directory gains one *.hlo_proto.pb per module. Save the first code block below as hlo2txt.py; it converts the training step's module to text. Export fp8old and fp8new, and count by shape: how many all-gathers in each output f8e4m3fn[1,192,…] or [192,1,…]? How many reduce-scatters output bf16[3,…]?
# hlo2txt.py: convert the training step's HloProto in a profile directory to HLO text (needs only jax; the CPU build is enough)
import glob, sys
from jax._src.lib import xla_client as xc # internal module path; tested here with jax 0.11.0
d, out = sys.argv[1], sys.argv[2]
b = open(glob.glob(d + "/jit_train_step*.hlo_proto.pb")[0], "rb").read()
n, i, s = 0, 1, 0 # field 1 is hlo_module: read its length first
while True:
c = b[i]; i += 1; n |= (c & 0x7F) << s; s += 7
if c < 0x80: break
open(out, "w").write(xc.XlaComputation(b[i:i + n]).as_hlo_text())
pip install jax
python3 hlo2txt.py ~/hy3-prof/fp8old/plugins/profile/2026_08_15_03_57_28 fp8old.hlo.txt
grep -cE '= f8e4m3fn\[(1,192|192,1),\S* all-gather\(' fp8old.hlo.txt
grep -cE '= bf16\[3,\S* reduce-scatter\(' fp8old.hlo.txt
fp8old: 12 all-gathers and 6 reduce-scatters; fp8new: 0 and 0. The exported text is 9,599,857 bytes for fp8old and 8,403,650 bytes for fp8new: the two sizes before stripping in main text §4.1.
How to tell: the two greps count exactly what the main text §4.2 table counts. The all-gathers of the 192 experts' FP8 weights come in four shapes ([1,192,4096,1536]×6, [1,192,1536,4096]×3, [192,1,4096,1536]×2, [192,1,1536,4096]×1), and the reduce-scatters scattering this device's 3 experts' gradients back in two. Pages list only instructions that ran; “the modified recipe has none at all” takes a whole-program count. The script uses jax internals; with another jax version you may need to change the import line.
Main text §2.3 says the forward and backward loops each run 78 iterations, counted on the trace (the same block appears 78 times under while.457; in Op Stats, gmm_megablox.16's #Occurrences is 1,872 = 78 × 3 steps × 8 cores). Verify it in Exercise 11's fp8old.hlo.txt. A while's condition= points to a small computation holding “counter < some constant”; the parentheses of while(…) hold the initial-value tuple, whose first element is the counter's initial value. Start from this line:
grep -E '^\s*while\.457 = ' fp8old.hlo.txt | grep -oE 'while\([^)]+\)|condition=[^,]+'Follow the printed names: constant and initial value of forward while.457? Of backward while.453? Why 1 in one and 0 in the other?
while.457: the condition (wide.region_53.67_spmd.clone.clone.clone.clone) holds constant(79) and direction=LT; the initial-value tuple tuple.1968 starts with copy.2308 = copy(constant(1)); the body ROOT tuple starts with add.7090 = counter + constant(1). i runs 1 to 78: 78 iterations. while.453: constant(78), initial value constant(0): also 78 iterations.
How to tell: four greps in a row, each using names printed by earlier ones (the fourth's constant.4132.clone only shows up after grepping copy.2308 once; here the two are merged into one line):
grep -E '^\s*while\.457 = ' fp8old.hlo.txt | grep -oE 'while\([^)]+\)|condition=[^,]+'
grep -A4 '^wide.region_53.67_spmd.clone.clone.clone.clone {' fp8old.hlo.txt | grep -oE 'constant\([0-9]+\)|direction=\w+'
grep -E '^\s*tuple\.1968 = ' fp8old.hlo.txt | grep -oE '\) tuple\([^,]+'
grep -E '^\s*(copy\.2308|constant\.4132\.clone) = ' fp8old.hlo.txt | grep -oE '\S+ = \S+ (copy|constant)\([^)]*\)'Layer weights are stacked along a leading dimension of 79 (shapes like u32[79] and bf16[79,…] are all over the HLO). The forward pass peels layer 0 off before the loop, which takes layers 1–78, so the counter runs from 1 up to, but not including, 79; the backward pass takes layers 78–1 in reverse with layer 0 after the loop, so it runs from 0 up to, but not including, 78 (collapsible block in main text §2.2). The forward condition computation's parameter name contains loop_peel (one iteration peeled off), saying exactly this. The lesson: a count read off the trace should match a hard-coded constant in the HLO; if they disagree, you counted the wrong thing.
The main text reads just one done, one layer's total, two programs' instruction counts and two lines of code. These seven, easy to hard, move to another collective, every kernel in a layer, another profile's HLO and back to the code; answers only here. Then you can reconcile a layer by Ctrl multi-select, estimate a collective's duration from bytes and check against measurement, tell what a done waits on, and trace a slice to its public-code line.
Open bf16old, backward layer 40: view_start=35630.027&view_end=35825.278 ↗ Open. On the TensorCore's XLA Ops row, click the first expert matmul (name contains gmm), then Ctrl-click (Command on Mac) the other 8. Read “… items selected.” atop the panel and the Totals row's Wall Duration. Repeat on bf16new's same layer, view_start=34056.155&view_end=34251.406 ↗ Open. How much did its expert matmuls shrink in total? And Selection extent, under Totals? Finally, one by one (click, read Wall Duration): which kernels saved it?
“9 items selected.” Totals 79,113,086 → 66,893,085 ns (79.113 → 66.893 ms): this layer saves 12.220 ms. Selection extent, 98,189,886 / 86,068,731 ns, spans from the first kernel's start to the last one's end, not a sum. The three that gained most: gmm_megablox_transpose_rhs.9 → gmm.22 9.961 → 6.305 (−3.656); gmm_megablox_transpose_rhs.10 → gmm.23 10.414 → 7.000 (−3.414); tgmm_megablox.10 → tgmm.8 10.053 → 6.834 (−3.219). Together they save 10.289 ms, 84% of the total. gmm_megablox.12→gmm.19, tgmm_megablox.8→tgmm.6 and tgmm_megablox.9→tgmm.7 barely move (within ±0.05 ms).
How to tell: the two runs' 9 kernels pair up in order of appearance, output shapes matching one for one (click to confirm), so compare them pair by pair. A missing row means a click didn't register (clicks are ignored while the page spins). The same layer's forward pass tells the same story: its 3 kernels save 1.070 ms together, almost all in the third, gmm_megablox.16 → gmm.26 (0.988 ms faster); the first two differ by only tens of microseconds. One layer saves about 13.3 ms (backward 12.220 + forward 1.070); × 78 iterations ≈ 1,037 ms, which covers most of the 1,053.4 ms saved per step (the layer peeled out of the loop and the MTP layer not counted). Why exactly these kernels got faster, this course didn't chase.
Back to fp8old, backward layer 40, just right of Exercise 4's window: view_start=31254.000&view_end=31302.000 ↗ Open. On the TensorCore, click tgmm_megablox.10 and read its output shape. As it finishes, a reduce-scatter starts on the SparseCore: what is it called, how long does it run, what shape does it output? Where is its done (number one higher) on the TensorCore, and how wide?
tgmm_megablox.10 outputs bf16[192,4096,1536] (4.522 ms). At 31,260.076 the SparseCore starts reduce-scatter.12: 10.447 ms, output bf16[3,4096,1536]. The TensorCore's reduce-scatter.13…call-done (31,290.841) is zero-width: fully hidden.
How to tell: gradients for 192 experts go in and this device's 3 come out; this is the “scattering the expert gradients back” of main text §2.4. On the SparseCore, three reduce-scatters (12, 14, 16) run back to back, about 29.8 ms in total, while the TensorCore keeps computing other work, and all three dones (13, 15, 17; the last two at 31,298.349 and 31,300.427) have zero width. In the original recipe these collectives already cost the TensorCore no time, so removing them doesn't save it much either (main text §3.3, Figure 3.5).
Take Exercise 14's reduce-scatter.12. In the HLO exported in Exercise 11, look up its input and the devices taking part:
grep -B1 'ROOT reduce-scatter.12 = ' fp8old.hlo.txt | cut -c1-330The line above is the parameter(0) of its sub-computation. Estimate first: how many bytes is the input? At main text §2.5's measured all-gather speed (547 gathered 1.21 GB in 2.406 ms, about 0.50 TB/s), how long should moving it take? Then compare with Exercise 14's duration: what is the reduce-scatter's effective speed, and what fraction of the all-gather's? And the other two in the window (14 and 16)?
The input is bf16[192,4096,1536]: 192 × 4,096 × 1,536 × 2 bytes = 2,415,919,104 bytes ≈ 2.42 GB, twice what 547 gathers (FP8, 1.21 GB). At 0.50 TB/s that's 4.8 ms; the measured 10.447 ms is 2.2× the estimate. Effective speed: 2.42 GB ÷ 10.447 ms ≈ 0.23 TB/s, about 46% of the all-gather's. 14 and 16 take 10.246 / 9.134 ms, i.e. 0.24 / 0.26 TB/s, the same order of magnitude. replica_groups is {'fsdp'}, and the mesh has 'fsdp'=64 (plus 'data'=2): 64 chips are 128 devices (2 per chip), split into two groups of 64. Each group scatters once; each device keeps 192 ÷ 64 = 3 experts, matching the output bf16[3,4096,1536].
How to tell: use one ruler for both, full-block bytes (the 192-expert side) ÷ duration. In ring all-gather and ring reduce-scatter, each device sends and receives about 63/64 of the full block, so by bytes the two should be equally fast [inferred: we didn't check whether this build uses a ring algorithm]. The measured reduce-scatter is more than twice as slow; this course didn't chase why [inferred: each received chunk must be added to the local one before being sent on, adding reads, writes and additions]. Estimating first makes the comparison quantitative: one layer's three gradient scatters take about 29.8 ms, a step 2,354.7 ms (main text §3.3), and they cost nothing only because they overlap the TensorCore's compute (Exercise 14). When estimate and measurement differ by more than 2×, recheck bytes and shapes before suspecting the bandwidth.
Still in fp8old, backward layer 40: view_start=31218.000&view_end=31234.000 ↗ Open. Find the TensorCore's all-gather.546 call-start/call-done and SparseCore 0's all-gather.545 (number one lower). When is it issued, how long does it queue on the SparseCore, how long is the transfer? How wide is the done? Versus Exercise 4's 548, which is more exposed on the TensorCore, and why?
Start at 31,219.550 ms (issued 0.04 ms after 548). 545 starts moving only once 547 finishes, at 31,226.187: it waits 6.64 ms, transfers for 2.207 ms and finishes at 31,228.394. The done is at 31,232.781 with zero width: the TensorCore reaches it 4.39 ms after the transfer ends, so it is fully hidden. 548 exposed 1.181 ms; 546 exposed nothing.
How to tell: the transfer's wait doesn't matter; when the TensorCore needs the data does. 548's user is gmm_megablox.14, right behind it. 546 moves a different expert matrix, [1,192,1536,4096], whose user gmm_megablox_transpose_rhs.10 comes after gmm_megablox.14 and .13 and needs it more than 6 ms later. In the HLO, the last operand of gmm_megablox_transpose_rhs.10 is bitcast(all-gather.546…call-done) (finding it in the HLO: main text §4.1).
Back to bf16new, backward layer 40: view_start=34056.155&view_end=34251.406 ↗ Open. First, the 3 TensorCore dones of Exercise 3's 192-expert all-gathers (number one higher: all-gather.551/.553/.555): how wide? Click the widest done on the TensorCore XLA Ops row here, read name and width, and find the SparseCore slice below with the same right edge: what does it do, and what shape does it output?
All 3 dones are zero-width. The widest is gather_offload_async_done.3, 34,129.891–34,132.862 ms, 2.971 ms. Aligned with it is gather_offload_custom_fusion.70.cloned.1.call-done on the SparseCore's Sparse Core Ops row (the body, gather_offload_custom_fusion.69, is on the TEC 0–15 rows below), with output bf16[393216,4096]: it moves tokens to their experts, not a collective.
How to tell: this gather runs on both SparseCores, both from 34,129.579 ms; SparseCore 0 runs 3.277 ms, SparseCore 1 3.283 ms, and the done's right edge lines up with SparseCore 1. 393,216 = 12 sequences per device × 4,096 tokens × 8 experts per token, the same row count as Exercise 8's gmm input; in Details, this done's tf_op ends in sort_activations/gather. This layer has 10 dones wider than 50 µs, 5.86 ms in total. The lesson: the widest done is not necessarily a collective, so read the name before concluding; when both SparseCores run the same op, the done waits for the last to finish.
Export fp8tk's HLO with Exercise 11's script (directory ~/hy3-prof/fp8tk/plugins/profile/2026_08_15_17_48_55) and run the same two greps, then count all three collective kinds with grep -oE ' (all-gather|reduce-scatter|all-reduce)\(' fp8tk.hlo.txt | sort | uniq -c and compare with fp8old. Did the 375 s come from extra communication?
fp8tk: 12 and 6, as in fp8old. Totals for the three kinds: all-gather 92, reduce-scatter 6, all-reduce 36, identical to fp8old down to the last instruction (fp8new has 82 / 0 / 36). Not a single extra collective: the 375 s is spent in the kernels themselves (Exercise 6).
How to tell: fp8tk and fp8old share a recipe and differ only in tile injection (main text §3.5), so the program structure is identical and collectives match one for one. The lesson: one run fast, the other slow? First check whether the program structure (instruction counts) changed, then compare individual kernels.
Open fp8new, forward layer 40: view_start=9384.136&view_end=9419.813 ↗ Open. On the TensorCore's XLA Ops row, the third expert matmul, gmm.28, is directly followed by a broadcast_select_fusion… slice. Read both slices' Wall Duration and output shape, and the last segment of the second's tf_op in Details. Then open backend.py in the public MaxText (yangwhale/maxtext, hunyuan3 branch @1afaa0fb, as in main text §4.5). In gmm(), find the two lines at num_current_groups and the closing if, then jump to the function it calls. What does this slice do, and why only in the modified run? It is more than 30× slower than the kernel: estimate its duration from bytes. Finally, match it to the extra 276 ms of loop fusion in main text §3.4 and §4.6.
gmm.28: 0.035 ms, output bf16[229376,4096]. broadcast_select_fusion.99 starts right after: 1.202 ms (34×), same output bf16[229376,4096], tf_op ending in shard_map/jit(gmm)/jit(_where)/select_n. The code: L346 num_current_groups = rhs.shape[0] (after the change the weights have only 3 groups), L347 num_total_groups = group_sizes.shape[0] (192 groups); L555 if existing_out is None and num_current_groups < num_total_groups: holds, so L556 calls _zero_uninitialized_memory. That function (L268–281) builds a length-229,376 mask from out.shape[0], true only for the rows of this device's 3 groups (num_nonzero_groups groups from start_group, L280); finally L281 return jnp.where(valid_mask[:, None], out, 0) writes 0 into all other rows. Estimate: the output is 229,376 × 4,096 × 2 bytes = 1.88 GB; one read plus one write is 3.76 GB, 1.02 ms at 3,433 GiB/s of HBM bandwidth per core. The measured average, 1.204 ms, is about 85% of the bandwidth. Per step, the forward pass runs it 79 times for 95.1 ms, the backward pass runs select_add_fusion 79 times for 225.6 ms: the two categories in the §4.6 table.
How to tell: in the original recipe the gathered weights hold all 192 groups, so the condition num_current_groups < num_total_groups is 192 < 192, which fails, and this step never runs. After the change they hold only 3 groups; the kernel computes only those 3 groups' rows (hence only 0.035 ms), and this step zero-fills the other 189/192 of the rows in one sweep [inferred: the zero fill, one read plus one write, is HBM-bandwidth bound]. jnp.where is elementwise and compiles to a loop fusion, so the extra is booked under loop fusion (−275.9 ms in the main text §3.4 table). The backward copy is fused with the existing gradient addition (135.5 ms per step in the original recipe, booked under non-fusion elementwise), so of the two new fusions' roughly 321 ms, after taking out the 135.5 ms that only changed category, about 185 ms is the zero fill itself; loop fusion as a category grows by only 276 ms because other loop fusions shrink by about 45 ms in total (main text §4.6). The lesson: for an unfamiliar fusion in the trace, read its tf_op's last segment first; it is usually the function the line of code calls. Look up files and line numbers in the code version matching the profile (collapsible block in main text §4.5).
The main text reads the Roofline and memory pages on only one or two profiles. These five move to BF16 and the modified run, easy to hard; answers only here. Afterwards you can tell on the Roofline page whether a point can be plotted and whether a whole-step rate can be trusted, and use Memory Profile and Memory Viewer to say whether a change saved memory and who holds it at the peak.
Open bf16new's Roofline (tool roofline_model) ↗ Open, type gmm.19 in Section2's Operation box, then, in bf16old↗ Open, gmm_megablox_transpose_rhs.10. Read Model FLOP Rate, FLOP Rate / Peak and Source Info: can it be plotted?
bf16new gmm.19: 513,223 GFLOP/s, 49.9%, Source Info MaxText kernels/megablox/backend.py:548; bf16old: 475,232 GFLOP/s, 46.2% (layers/moe.py:1523). Both points sit under the roof and can be plotted.
How to tell: gmm.19 and fp8new's gmm.27 come from the same line of code (backend.py:548, the branch taken after the change). In FP8, gmm.27 reports 5,674.9% and its point shoots off the vertical axis, unplottable (collapsible block in main text §3.2); in BF16, gmm.19 reports 49.9%. Same code path, but the BF16 kernel receives all 192 experts (Exercise 8), so it really is doing the work. If you share a screenshot, mask the Program ID column.
Open the Roofline pages of bf16old↗ Open and bf16new↗ Open. In the first section's table (Section1), on the row whose Step is Average, read Total Time per core (us), Normalized FLOP Rate (GFLOP/s) and FLOP Rate / Peak. By what percentage did time and rate change? Multiply them: how many GFLOP per step per core are recorded? Then multiply the same row's values for fp8old and fp8new.
bf16old 21,767,912 µs, 474,422.33 GFLOP/s, 46.1%; bf16new 20,713,041 µs, 498,583.66 GFLOP/s, 48.5%. The time drops 4.85% and the rate rises 5.09%; the product is about 10,327,184 GFLOP per step per core for both, the same amount. FP8: fp8old 5,261,170 and fp8new 5,267,374 GFLOP, also nearly the same, yet the time dropped 33.4% and the rate rose 50.3%.
How to tell: Total Time per core is the Overview's average step time (21,767.9 / 20,713.0 ms). The booked FLOPs are the same in both runs because XProf books the FLOPs the program declares. The BF16 run did no less work (in Exercise 8 the kernel receives 192 experts) and its time really fell, so a 5% higher rate is plausible. In the FP8 run, the same booked FLOPs over a much shorter time push the rate up by half, while in fact less work was done (main text §3.2, Exercise 5). So compare the Average row's rate only between runs of the same program, and first confirm the product's numerator hasn't changed.
In Memory Viewer (tool memory_viewer) ↗ Open, switch bf16old, bf16new↗ Open and fp8new↗ Open to jit_train_step and read the top lines: Peak memory allocation, total argument size, stack simulated HLO temporary variables. fp8old (the original) shows 91,766.56 MiB peak, 44,533.22 MiB of entry parameters and 47,230.39 MiB of temporaries. How much did the same change move memory per precision?
BF16: 88,401.89 → 88,399.85 MiB (−2.04); entry parameters 22,271.03 in both; temporaries 66,127.62 → 66,125.64. FP8: fp8new 91,597.20 MiB, 169.36 less than fp8old; the same entry parameters; temporaries 47,061.12. In both precisions the change barely moves memory (FP8 saves 0.2%).
How to tell: after the change FP8 loses 12 all-gathers of the 192 experts' weights and 6 reduce-scatters scattering the gradients back, yet the peak drops only 169 MiB: the peak sees only the single highest moment. At the original recipe's peak, 5 bf16 [192,…] gradient buffers are live: 3 from the current layer in the backward loop, whose tgmm just finished and whose gradients aren't yet scattered back (the other 2: the all-reduce of the MTP layer's expert gradients, outside the loop). After the change those 3 in-loop buffers are gone, but the peak passes to the output head's logits (main text §3.6), so only this little is saved. The gathered FP8 expert weights were never among the largest buffers at either peak (hover over the first twenty bars to see what they are: parameters, optimizer state, the output head and these gradients). The lesson: to tell whether memory was saved, compare two lines: Peak memory allocation and the temporaries. After switching modules, the Module Name line may still show the default module; go by the numbers below.
Open bf16new's Memory Profile (tool memory_profile) ↗ Open. Memory ID defaults to 0: read Profile-window Peak Memory Usage and below it Timestamp, Stack Reservation, Heap Allocation and Free Memory. Switch to Memory ID 1 and reread. Enter Memory ID 1's peak time in the “HLO's at time (ms)” box above the HLO Modules table: which module is left? Finally compare Stack and Heap with fp8new's Memory ID 0.
Memory ID 0: 93.60 GiB at 2.0 ms; Stack 71.42, Heap 22.18, Free 1.13 GiB. Memory ID 1: 93.52 GiB at 41,419.7 ms; Stack 71.42, Heap 22.10, Free 1.21 GiB. After you enter 41419.7, 8 rows are left in the table, all of them the second jit_train_step (TPU:0–7, 21,109.4–41,803.3 ms). fp8new, Memory ID 0: Stack 47.72, Heap 43.89, Free 3.12 GiB.
How to tell: Memory ID is a device's index on this host (0–7 in the drop-down). Memory ID 0 peaks at 2.0 ms, the window's very start, while the HLO Modules table's first module starts only at 395 ms; at that moment it holds about 82 MiB more heap than Memory ID 1. The two devices share Stack, differing only by this bit of heap, so the conclusion stands, but HLO's at time needs a peak time inside a step. The whole timeline is almost flat; Stack stays at 71.42 GiB throughout. BF16's Heap is half of fp8new's and its Stack 23.7 GiB larger: the first matches Memory Viewer's entry parameters being only half as large (main text §3.6) [inferred: Heap is mostly resident parameters and optimizer state; the two differ by only about 0.4 GiB], the second agrees with the larger temporaries (Exercise 22).
Open bf16new's Memory Viewer (tool memory_viewer) ↗ Open, set Hlo Modules to jit_train_step and read Peak memory allocation at the top. Scroll down to the by Buffer Size chart and hover, left to right, over bars 0–4 and bar 11 (counting from 0); read name, Size, Shape and Framework Op in the left panel. What are the three largest buffers, and their combined share of the peak? What are bars 3, 4 and 11?
Peak 88,399.85 MiB (93,519.35 including fragmentation). Bars 0–2 are 11,328 MiB each, all shaped bf16[12,4096,120832] (multiply_convert_fusion.1, fusion.1218{1}, select_reduce_fusion.3{1}, with Framework Op ending in convert_element_type, dot_general and reduce_sum respectively); together 33,984 MiB, 38.4% of the peak. Bars 3 and 4 are gather_offload_custom_fusion.79/.81, 3,072 MiB each, bf16[393216,4096], Framework Op ending in sort_activations/gather. Bars 11–13 are 2,304 MiB each: all-gather.604/.605 (bf16[192,4096,1536]) and copy.1717… (bf16[192,1536,4096]). Bars of equal width can change order each time you open the page, so whichever of these three bar 11 shows on hover is a correct answer.
How to tell: 120,832 is the vocabulary size (vocab_size in the config) and 12 is the 12 sequences per device: 12 × 4,096 × 120,832 × 2 bytes = 11,328 MiB: output-head logits or similar [inferred: identified from the shape and Framework Op]. The 393,216 rows are the output of Exercise 17's token move to experts. The all-gather.604 group gathers all 192 experts' BF16 weights: 192 × 4,096 × 1,536 × 2 bytes = 2,304 MiB. It is still live at the peak: the BF16 program really does gather all 192 experts (Exercises 3 and 8). Bars 5–10 are parameters and Adam's nu, shapes like f32[192,79,32,1536]: all 192 experts present, sharded along the 4,096 dimension.
← Back to the Topic 7 course page · Instructor notes · Course overview