TPU vs GPU

This is not a tour of spec sheets.
Run one FlashAttention kernel on both chips; find the step where they diverge.

Two questions, one path. ① How do the two chips differ in hardware? ② Run one op end to end: where does each difference surface? The verdict: they differ everywhere, all because of one decision.

TPU side v7 (Ironwood) GPU side B200 teardown measured on GB300 NVL72 Sources public source code + own measurements ⏱ 109 min · 43 figures

Course author Chris Yang·Google Cloud AI Infra architect

This lecture has instructor notes.

They cover how to teach it: segment goals, pauses, common misconceptions, likely questions. The content is here.

Presenting: lines, figure cues and timing are written out. Self-study: give the script to an AI for a TA who knows the lecture and how to teach it.

⛔ Don't open before class: two warm-ups We do them together in class; peeking spoils them.

First, a question from last lecture

Last lecture we took apart DeepSeek V3: 671B total parameters, but each token activates only 37B—that’s MoE.
If only 37B are active, is it as cheap to serve as a 37B dense model?
What does MoE actually save?

It saves compute, not memory.
Each token takes only the 37B path, so the compute saving is real. But all 671B of weights stay resident, every byte: no one knows which experts the next token picks, so all sit in HBM. “37B active” ≠ “as cheap to serve as 37B.”

⭐ The idea behind the question: memory and compute are separate, and move independently. Last lecture kept the two accounts apart.
This lecture adds a third: bandwidth. What decides whether an op is fast is often neither of the first two. Section 1 works out the exchange rate between compute and data movement.

Two more numbers to guess (answers now, the “why” in Section 3)

We will take both chips apart, layer by layer. First, guess: how many pieces does each chip divide its compute into?
On B200 the unit is called a Tensor Core; on v7 it is the MXU. Both do matrix multiply-accumulate.
The question is the total per chip.

① How many Tensor Cores does a full B200 have?
② How many MXUs in a full TPU v7?

Answer the two independently; the answer appears once both are picked.

B200: 592. v7: 4.
592 = 148 SMs × 4 Tensor Cores per SM; 4 = 2 TensorCores × 2 MXUs per core. A 148× difference in count.

⭐ The surprise: The two chips differ by only 15% in MACs per cycle: 606,208 on B200 versus 524,288 on v7.
The same compute, cut into 592 small pieces vs. 4 large ones.

⭐ The “why” is an IOU. Keep the question in mind. How can the count differ by 148× while the total is nearly the same? What does each side pay for cutting fine vs. coarse? All of Section 3 answers these two. When you get there, you'll recognize which figure holds the answer.

Section 0

First, a premise: two chips, two origins

The figure below is the lecture's only premise. With it, you can predict each section's differences. Look, then move on: Section 1 is one division; Section 2 starts the teardown.

Think for thirty seconds first You must design a chip without knowing what customers will run. What do you do? Now flip it: you know from day one it runs just one kind of workload. The gap between the answers is the figure below.
One premise: different origins, so every layer differs GPU Origin: graphics processor Many unrelated tiny jobs Shape, access, control: unknown → most area: “handle any case” caches · warp schedulers · many threads Unknown workload → hedge at every step TPU Origin: matmul ASIC NN-only from day one Shape, access, control: known → buys simplicity, not compute No cache · no scheduler · same compute in 4 big blocks Known workload → hedge nothing vs
Two strategies, not better and worse: decide at runtime, or get it right at compile time. The one question: do you know in advance what you'll run? ⭐ The right side, “Known workload,” is the premise: every later difference traces back to it. Carry it forward; don't memorize parts yet.
Up front: we don't rank the chips Both origins were right at the time; each still carries its cost. You won't read “which is better,” only “who decides this, on this side.”
If you keep one sentence: they differ everywhere, all because of one decision. Section 2 names it.
⚠️ Section numbers skip (no Section 8); figure numbers follow the full edition (you missed nothing)

Section numbers match the full L300 edition. Upside: to dig deeper, open the same number in L300. Cost: numbers skip in this edition: 0 · 1 · 2 · 3 · 4 · 5 · 6–7 · 9. Section 8 exists only in L300, and 6 and 7 are merged into one section here.

⚠️ One more catch: figures carried over unchanged from L300 use L300 numbering. The major sections map as follows:

  • Figure’s §7 (measurements on both sides): §6.3
  • Figure’s §6 (what makes a comparison meaningful): §6.1 / §6.2
  • Figure’s §8 (how I verified this lecture's numbers): not in this edition; only in L300

Subsections too, and you'll hit these more often. Not in this edition; see L300: §1.4 (the 312 ridge point), §2.6 (how far constraints propagate), §2.8 (every op so far on one axis), §4.1 (interconnect limits), §5.5 (inference-engine porting pain points), §5.6 (predicting OOM with AOT, no chips needed).

This is deliberate. The figures belong to L300 and stay word-for-word; otherwise two versions would exist, and a fix to one would silently drift from the other.

Section 1

The dividing line: two chips, same appetite

This section computes one number: the compute-to-movement exchange rate, operations per byte moved. Every later op is checked against it: below the line, bandwidth rules; above it, compute. No conclusions here.

Think for thirty seconds first Only two spec-sheet numbers matter: operations per second and bytes moved per second.
How do you combine them into something useful?
Hint: not “which is bigger,” but something that can judge a particular op.

1.1 ⭐ Compute ÷ bandwidth: both vendors, one number

Divide the two spec numbers: operations per second by bytes per second. The ratio is operations done in the time one byte moves. A floor: each byte moved needs this many operations just to feed compute.

Two vendors, two designs, two philosophies: compute ÷ BW, one number All from 2 spec sheets; math shown TPU v7 (per chip) BF16 2,307 TFLOPS 7.37 TB/s 312.9 FLOP / byte FP8 4,614 TFLOPS 7.37 TB/s 625.8 FLOP / byte Source: Google Cloud TPU7x spec sheet, given per chip GB200 (in NVL72 · per GPU · dense) BF16 2,500 TFLOPS 8.0 TB/s 312.5 FLOP / byte FP8 5,000 TFLOPS 8.0 TB/s 625.0 FLOP / byte Source: NVIDIA GB200 NVL72 rack ÷ 72; sparse ÷ 2 = dense What it means · The hardware's “appetite”: in the time to move a byte from HBM, it does ~312 BF16 ops · So an algorithm needs 312 ops per byte to keep it fed; fewer and compute idles · Ops use the same unit: ops per byte moved — arithmetic intensity · One question per op: intensity vs 312? Above: compute-bound; below: memory-bound · Of Topic 1's 8 steps, ① embedding has intensity 0 — off this axis (full L300 §2.8); that's Section 2's job · ④ Dense MLP matmuls go far above 312: Section 3's turf. Same chip, opposite problems ⚠️ More than one line: it drops at each level inward Same division at each level. All three are GB200-only; derivation on each row HBM3e off-chip 2,500 TFLOPS ÷ 8.0 TB/s 312 L2 HW-managed 2,500 TFLOPS ÷ 21 TB/s 119 Shared / L1 SW-managed 8,192 FLOP ÷ 128 B (per SM per cycle) 64 FLOP / byte — each level's bar. Shorter = easier to turn compute-bound ⭐ HBM → shared memory: the line drops only ~5× — on-chip has limits Fusion cuts HBM-level traffic; data still lands on chip — the cost moves down a line. So after fusion compare to 64, not 312 — fuse too far and you hit the new wall. ⛔ GB200 only: TPU VMEM BW unpublished; line exists, value unknown 64 = 8,192 FLOP/cyc/SM (Fig. G-2) ÷ 128 B/cyc/SM; 3 public measurements agree (Hopper µbench 127.9 · SemiAnalysis · Chips and Cheese); 32 banks × 4 B. ⚠️ Bases differ: top two whole-chip (2,500 TFLOPS ÷ chip BW), third per SM per cycle. Same ratio: 148 and clock cancel; an identity, not an approximation. Check: 2,500 ÷ 64 = 39 TB/s/chip; ÷ (148 SM × 128 B) → clock ≈ 2.06 GHz, matches GB200. ⭐ The real gap isn't math but reachability: 8 TB/s HBM shared by 148 SMs; 128 B/cyc shared mem per SM.
The whole output: one number, one method. The method matters more: numbers change by generation; it doesn't.
⭐ Note the lower three lines: judging if fusion went too far is the same division, new denominator.

Everyone asks: both sides have eight 24 GB HBM3E stacks; why different bandwidth?
“HBM3E” names the generation, not the speed grade. Each stack: 1,024 data lines; bandwidth = 1,024 × pin rate. HBM3E spans 8.0–⁠9.6 Gbps/pin. Backing out: v7 runs 7.2 Gbps (7.37 TB/s ÷ 8, exact, though 7.2 is outside 8.0–9.6), GB200 about 7.8.
The buyer picks the grade (power, cooling, reliability, supply); it isn't the chip's ceiling. (Why not top grade? That also exposes a hole in our own numbers.)

⭐ Same eight 24 GB HBM3E stacks: why 7.37 vs. 8.0? Pin-level math finds a hole in our own numbers

The HBM configuration is identical: eight HBM3E stacks, 24 GB each, 192 GB total. (The v7 “eight stacks” comes from Google Cloud's Ironwood technical blog; the B200 figure from NVIDIA's Blackwell architecture page.)

Then why does the bandwidth differ? Because “HBM3E” is not a speed; it is a range of speeds. Every stack has 1,024 data lines, and bandwidth = 1,024 × per-pin data rate. Pin rates from 8.0 to 9.6 Gbps all count as HBM3E (Micron lists a pin speed above 9.2 Gbps and more than 1.2 TB/s; Rambus lists 9.6 Gb/s and 1,229 GB/s). “Same HBM3E” does not mean “same bandwidth.”

Working backwards to each side's grade:

  • TPU v7: official 7.37 TB/s ÷ 8 stacks = 921.6 GB/s per stack → 1,024 bit × 7.2 Gbps ÷ 8 = 921.6. Exactly 7.2 Gbps (1,024 × 7.2 ÷ 8 = 921.6, × 8 stacks = 7,372.8 GB/s).
    ⚠️ But 7.2 falls outside the 8.0–9.6 range, as does GB200's 7.81: neither back-calculated rate is a nominal grade, so each official figure hides an unstated down-clock or accounting convention. Both sides get the same treatment.
  • GB200: official 8 TB/s ÷ 8 stacks = 1,000 GB/s per stack → back-calculated pin rate 7.81 Gbps. ⚠️ That isn't any nominal grade.
⛔ So the “0.13% relative difference” line is no longer a selling point That 7.81 means “8 TB/s” hides a convention NVIDIA never states, with two plausible readings:
•  Decimal: the vendor down-clocks and reports exactly 8,000 GB/s (down-clocking HBM is routine)
•  Binary: it is actually the 8.0 Gbps grade, 8 × 1,024 GB/s = 8,192 GB/s, rounded to “8” in public specs
The two readings differ by 2.4%. Work it out: 2,500 ÷ 8.000 = 312.5, whereas 2,500 ÷ 8.192 = 305.2.
The third digit of “312.9 vs. 312.5” falls entirely inside that 2.4%.

⭐ The conclusion stands; the precision doesn't: the two exchange rates really are in the same range (within a few percent), so the section's result holds. But “0.13% apart” is false precision; don't sell it.
⭐⭐ The most useful part: “is this number binary or decimal?” is one of Section 6's four questions. Here it lands on this lecture's own headline number. Aim those four questions at your own material first, not just at others'.
⭐ So why not run at the top grade? Four reasons, hard to soft In one line: HBM's philosophy is “wide, not fast”. 1,024 lines per stack, so no line must run fast. So data rate is a tunable knob, not a spec to max out.

① Power. I/O dynamic power tracks data rate, and higher rates often need more voltage. In a 700–800 W+ package, HBM is no rounding error: top-grade bandwidth is paid from the compute budget.
② Heat, the worst one for HBM. In a 3D stack the bottom die is farthest from the heat sink; its heat can't escape. And hot DRAM must refresh more often. Refresh costs power and takes bus time. Counterintuitively, a hot, throttling top grade may deliver less than a steady lower one.
③ ⭐ Reliability, which answers Section 0's “is simpler more robust?” According to hyperscaler data, HBM failure is the leading cause of GPU failures, more often than any other datacenter chip. (Cited by SemiAnalysis, no specific numbers given.) In a 9,000+-chip pod, per-chip failure rate × scale = training interruption rate. Dropping one speed grade to buy cluster-wide uptime is an easy trade.
④ Supply and system balance. Top bins are costly and scarce; sourcing hundreds of thousands reliably is itself a constraint. And if compute can't be fed anyway, extra bandwidth wastes power.

⭐ So you were right: the grade is chosen by the buyer. JEDEC sets the generation's ceiling, DRAM makers bin by what parts sustain, and NVIDIA / Google each pick a point within their own power, thermal, reliability and supply constraints, then write it into the spec sheet.
So v7 at 7.2 and GB200 at around 7.8 are two trade-offs, not a capability gap. The cleanest case of the opening line, “the spec sheet is the first thing to see through”: the bandwidth on a spec sheet is the result of a trade-off, not a physical constant.
⭐ Second question: 8 TB/s each way, or shared read/write? Shared. HBM's data lines (DQ) are bidirectional: the same pins carry a read one cycle, a write the next (not full duplex), and each direction switch costs a bus turnaround. (HBM has always used a parallel bidirectional bus, per architecture surveys; industry calls it half-duplex.)

⭐ Two consequences for this lecture:
① Counting reads and writes together is right. The 6 bytes of a vector add = read a 2 + read b 2 + write y 2: one shared bandwidth. Numerator and denominator must match; here they do.
② It explains why fusion is worth so much. Writing an intermediate to HBM and reading it back uses the same bandwidth both ways: you pay twice. Fusion waives both: you save “two trips,” not “half.”

The figure continues below. The division works at every level; this course is about moving data between them. For now: the closer to compute, the lower the line. It pays off with fusion at section's end.

Using the line: one division, two ops, then what fusion moves The method: count bytes moved · count ops · divide — that's the op's arithmetic intensity; compare with §1.1's spec-sheet 312 to see where it's stuck. ⭐ The method beats the 312 itself: 312 changes each generation, the method doesn't. Three uses below. ① Vec add y = a + b, bf16 read a 2 B read b 2 B write y 2 B I/O 6 B op 1 FLOP Intensity 1 ÷ 6 = 0.17 Below 312 by over 1,800× At GB200's full 8 TB/s, only 1.3 TFLOPS of 2,500 peak — 0.05% used Would a pricier chip help? No. It isn't on the compute path. ⭐ This is memory-bound: huge denominator, tiny numerator. No compute can save it. ② Matmul N × N × N, bf16 op 2N³ FLOP I/O 6N² B (3 matrices in/out) So intensity = N ÷ 3 — only N matters N = 256 → 85 still BW-bound N ≈ 1,000 → 312 just at the line “Matmul = compute-bound” is a myth. Only if it's big enough. ⭐ Both columns together say: ignore the op's name; only its ratio counts. ③ The fix — only way: shrink the denominator Merge adjacent ops into one kernel so intermediates skip HBM — that is fusion. ⭐ It saves no FLOPs: numerator fixed, only the denominator changes. 312 memory-bound compute-bound fusion pushes it right ⚠️ But skipping HBM ≠ landing nowhere Lands on chip, which has its own line: 312 → 64 bar drops only 4.9× on-chip traffic can grow tens of × ⭐ So overfusing hits a wall on the new line — not a higher bar, a bigger denominator. All later sections sit on it §2 1,000× from fusion — the main example §3 One fusion on two chips §5 Fusion: who should decide it ⛔ Every number checkable: 6 = 2+2+2; 312 ÷ 0.167 = 1,872; 8e12 ÷ 6 = 1.33 TFLOPS, 0.053% of 2,500 2N³ ÷ 6N² = N/3, N/3 = 312 → N = 936, N = 256 → 85.3; 312 ÷ 64 = 4.9
⭐ One division, three uses: vector add 0.17, square matmul N ÷ 3; fusion only shrinks the denominator, pushing the point right: not one FLOP lost. Column ③'s short axis, the figure's only axis, shows “which way it moves,” not “who sits where.” Easy to miss, bottom right: against the on-chip 64 line, the bar drops only 4.9×, but on-chip traffic can grow tens of times; the wall is a bigger denominator, not a higher bar.
One vector add, end to end: one step computes, the rest move data ⭐ This is what 0.17 looks like. Matmul on this path: “one number, end to end” (Fig. T-6) — same path, different busy unit. a (2 B) b (2 B) y = a+b (2 B) the only op ① HBM off-chip a and b start here y is written back here ⭐ ratio at this level: 312 ② VMEM scratchpad, not cache a compiler-inserted DMA loads it ⛔ how much and when: fixed at compile time (T-6) ③ Vector reg VPU input 1 vector op takes 8 × 128 elements a, b: one vreg each ④ VPU vector unit does the add ⭐ the only red flash here ① ② ③ ④ DMA load feed + Gray = return: result → VMEM → HBM, same DMA, also compile-time — return uses BW too; y's 2 B count in the 6
Tally read a 2 B read b 2 B op 1 FLOP write y 2 B Total: 6 B, 1 op → intensity 0.17 Share of “compute” Four stops, red box flashes once. 6 bytes moved for 1 add — this machine could do 312 ops per byte moved, here only 0.17. ⭐ So “memory-bound” means idle, not slow — compute waits on data. Chain such ops, each round-tripping HBM: Section 2's target. ⚠️ Schematic: shows order, not real duration. True ratio (6 B moved vs. 1 op) won't fit to scale on screen — order, not speed. Every number checkable: bf16 = 2 B/element → read a 2 + read b 2 + write y 2 = 6; y = a+b is 1 add = 1 FLOP (⛔ not an FMA); 1 ÷ 6 = 0.167 ≈ 0.17; 312 ÷ 0.167 = 1,872×. ⭐ Format from How to Scale Your Model's pointwise-product animation (jax-ml.github.io/scaling-book, MIT); stops, colors per T-6; content: this section's 0.17 example.
Figure 1-7 One vector add, end to end. Four stops; the red box flashes once: 6 bytes moved for one add. That is 0.17. A matmul on the same path: “one number, end to end” (Figure T-6): same path, different busy unit.
⚠️ Timing is schematic: order, not speed.

Three uses of the line. Both examples skip attention on purpose: Section 2's first figure derives it; doing it now would spoil it.

1.2 Square is a special case: the smallest dimension sets intensity

The matmul above is square (all sides N), giving N/3. Real ones are M×K · K×N, three lengths. Which sets intensity?

Same division. The operation count is 2·M·N·K, and the bytes moved are 2·(MK + KN + MN). It simplifies cleanly: 1/intensity = 1/M + 1/N + 1/K.

Reciprocals add, so the smallest dominates. “Longer K, higher intensity” is true but capped: as K → ∞, intensity only reaches MN/(M+N). K can't rescue a small M or N.
⛔ The harshest case: batch = 1 decode, M = 1, so 1/intensity ≥ 1: intensity caps at 1, however large K and N. That one line of arithmetic pins LLM decode to bandwidth.

Attention: in QKT both outer dimensions are the sequence length S, and the contraction dimension is head_dim. Most models use 128, so intensity is ≈ 128: below v7's 312 line and v6e's 560.

Which dim sets intensity? The smallest. In attention, head_dim §1 did only squares (N/3). Real matmul: M×K · K×N — three dims, combined as “reciprocals add.” ops 2·M·N·K ÷ bytes 2·(MK+KN+MN) ⟹ 1/intensity = 1/M + 1/N + 1/K (M=N=K reduces to N/3, §1's example) ① Reciprocals add M×K · K×N, bf16, A/B/C once ops 2·M·N·K bytes: 2·(MK + KN + MN) ⟹ 1/I = 1/M + 1/N + 1/K Square, M=N=K: I = N/3 side 256 → I = 85.3 (§1's 85) Reciprocals add ⟹ smallest wins. Smallest dim always caps I. ② Longer K, higher I — capped M = N = 512 fixed, vary K: K = 64 → I = 51.2 K = 128 → I = 85.3 K = 512 → I = 170.7 K = 4,096 → I = 240.9 K → ∞ → I → 256 Cap = MN/(M+N) = 256, stuck ⛔ Corollary: batch=1 decode, M = 1, so I ≤ 1 — no K or N can save it. ③ Attention: K = head_dim Per head S = 75,600, d = 128 QKT: M=S, N=S, K=d=128   → I = 127.6 PV: M=S, K=S, N=d=128   → I = 127.6 (also capped by 128) Ridge: v7 312, v6e 560 128 < 312 < 560 — without Flash, both matmuls: memory-bound. ⚠️ Attention intensity: 2 right answers, 2 bases ≈ 128 here is one matmul alone (A/B/C once); §2.3's 64 is naive attention overall (S and P round-trip, 2× denominator). What Flash changes — and why it still can't fill the MXU ① Flash shrinks only the denominator: S×S stays off HBM; only Q/K/V/O cross it, 4·S·d·2 B, so I = 4S²d ÷ 8Sd = S/2 = 37,800 (S=75,600). Not one FLOP saved; recompute adds a bit. → §2 ② One 128 bites twice: contraction 128 → low I (here); MXU is 256×256, fed only 128 → half idle (§3.4 / §3.6). One is “wait for data?”, the other “full while computing?” — two issues, one number.
Figure 1-8 Three dimensions set intensity (reciprocals add); in attention the smallest is head_dim. The same 128 bites twice: intensity, and MXU fill.

So “attention is a matmul, hence compute-bound” is a myth. Unfused, it is memory-bound, held there by head_dim.
What does Flash change? Next section's first figure. The direction: it saves no FLOPs; it changes the denominator.
⭐ The same 128 bites again: the 256 × 256 MXU is fed only 128 (§3.4 / §3.6). Intensity asks “wait for data?”; filling asks “is the unit full?”

You now have a ruler. It answers one thing: bandwidth- or compute-bound?
Next, a real op, and the lecture's first big number: fusion changes intensity 1,000×.

Over inter-chip bandwidth, the same division gives 5,768 (tokens each chip needs per step; expand)

The ruler does more than judge ops. Put inter-chip bandwidth under compute, and the division answers a new question: the fewest tokens per chip per step that let cross-chip communication keep up with compute.

Worked for TPU v7 (both numbers on Figure T-1):
2,307 TFLOP/s ÷ 400 GB/s (one axis, bidirectional) ≈ 5,768
All three axes triple bandwidth, cutting the threshold to ≈ 1,922. A full 9,216-chip domain needs a global batch of about 17.7M tokens so communication doesn't drag. Hence huge training batches: not for convergence, but so the network keeps up.

⚠️ The denominator's basis is easy to get wrong. The number above uses one axis bidirectional, 400 GB/s (six links, 1,200 ÷ 3 axes), not the chip's full 1,200. Change the basis and the answer moves 3×; both versions read fine on paper.

⭐ So this section's real output is the division, not 312. Over HBM bandwidth it judges ops; over inter-chip bandwidth, parallelism strategy. The latter belongs to Topic 5; here we just show the ruler can measure something else.

What is that on-chip memory, and “why not keep it in registers?” (two follow-ups; expand)

The place it lands has a name: on GPU it's shared memory, up to 227 KiB; on TPU it's VMEM. HBM traffic does drop, but the data still lands on chip, where the exchange rate is 64, not 312 (derivation in the lower half of the figure in §1.1).

⛔ So why not just keep it in registers? An SM's register file totals 256 KiB, the same size as shared memory, and every extra register means fewer resident warps. Usually you can't. The numbers are in the SM breakdown figure in §3.2.

Why does 312's numerator count only matrix units? What about CUDA cores and the VPU?

312 is the ridge point of the matrix roofline: the numerator is Tensor Core / MXU peak; the vector units are left out.

Because the ops we judge are almost all matmul FLOPs (attention's two, the MLP's big ones); softmax, norms and other vector work are a FLOP rounding error.

How much smaller are the vector units? Work it out:
GB200: 18,944 CUDA Cores × 2 FLOP/cycle × ≈2.06 GHz ≈ 78 TFLOPS
Against 2,500 on the matrix side, a 32× gap. ⚠️ That clock is derived: NVIDIA hasn't published GB200's boost clock, so we use 2,500 TFLOPS ÷ (606,208 MACs/cycle × 2 FLOP) ≈ 2.06 GHz (606,208 = 148 SM × 4 Tensor Cores × 1,024 MACs/cycle; §3.7 takes this apart.) The lecture uses this number throughout (it retires an earlier 2.03 extrapolated from 1.83; 1.83 is a third-party HGX B200 figure, itself 1.4% off against 2,250).
TPU is the same: one TensorCore's two MXUs together do 262,144 MACs per cycle (131,072 per MXU; not the §3.1 figure's cell count), while the VPU processes 1,024 elements per cycle. Different units, but more than two orders of magnitude apart.

Strictly, element-wise ops should be judged against the vector roofline: its ridge point is 78 ÷ 8 ≈ 9.8, not 312.

⛔ The two rooflines are separate. Don't add the two peaks into one line. Adding them assumes the same FLOPs can feed both units at once. They can't: an element-wise kernel never touches the MXU, and a matmul doesn't occupy the vector unit. Draw both; judge each op by the unit it runs on.
Memory is the same: HBM, L2 and on-chip SRAM each have their own bandwidth, so a rigorous roofline is a family of lines, not one. We use only the HBM line: our one question is must this intermediate go out.
⚠️ Convention: one multiply-accumulate (MAC) counts as 2 FLOPs (multiply plus add), as in vendor peaks. So N×N×N is 2N³, not N³, which gives “intensity = N ÷ 3” above.

But the conclusion doesn't flip. The vector add's 0.17 is 58× below 9.8: still memory-bound. That 1.3 TFLOPS is 0.05% of matrix peak and 1.7% of vector peak. Both look bad; the latter is the fair comparison.

⭐ The real takeaway: weak vector units are not an oversight; they are a design choice both sides share. An op on the vector unit loses twice: once to bandwidth, and even without that limit, the usable compute is only a thirtieth.
This is the root reason fusion is mandatory: not to compute less, but to keep data off this narrow, weak path.

⚠️ On-chip SRAM looks like a few-percent near-tie. It isn't. An easy misjudgment

By total capacity it is GPU 231 MiB vs. TPU 134 MiB, 72% more. Same order of magnitude, so it looks safe to ignore. It isn't. The GPU's lead is almost entirely L2, and L2 is exactly the level the TPU lacks entirely. The gap isn't capacity; it's who controls that space. All three ways of counting are in §3.2b.

Full ten-row comparison table, for lookup (sources for footnotes * † ‡ §)
LevelTPU v7GB200 / B200The crux
Matrix unitMXU 256 × 256, 2 per core Tensor Core (5th gen), 4 per SMGrain: one big vs. many small
Scalar / addressPer core, only one scalar unit 128 CUDA cores per SM, 148 SMs*Who issues irregular addresses (§3)
On-chip · explicitVMEM 64 MiB / core · compiler-managed shared memory ≤ 228 KiB / SM · human-managedWho sets the tile size (§2)
On-chip · automaticCMEM = 0 (none) L1 + L2 126 MB⭐ Whole level, one side only
Program­mable co-processorSparseCore × 4 / chip
own programs: prefix sum · sort · count · gather/scatter
(none programmable)
TMA: fixed-function, not this class
⭐ A level, the other side only
Odd-job units XLU cross-lane units × 2†
transpose · cross-lane reduce · shuffle. Transcendentals (exp / tanh): own resource§; Pallas costs transcendentals beside flops. Name, count: not public
SFU × 4 / processing block‡
exp · rcp · rsqrt etc. Cross-lane: warp shuffle, unlisted
⚠️ Each side names one, silent on the other
HBM96 GiB / device (= 192 GiB ÷ 2); whole chip 7,372.8 GB/s (= 7.37 TB/s) 186 GB / GPU (SW-visible; 192 physical); 8,000 GB/sSame HBM gen and scale
Compute (chip / GPU)BF16 2,307 | FP8 4,614 TFLOPS BF16 2,500 | FP8 5,000 TFLOPS (dense, the GPU in NVL72)Same magnitude
Compute ÷ BW312.9 FLOP/byte 312.5 FLOP/byte⭐⭐ Nearly equal
Inter-chipICI 1,200 GB/s bidirectional · 3D torus · up to 9,216 chips NVLink 1.8 TB/s/GPU · NVL72 domain · 72 GPUs (we use 64) New protocol past the domain? (§4)

* 148 SMs is from a third-party teardown; NVIDIA's spec sheet doesn't list it. All other numbers: vendors' official docs.
† The XLU count comes from Google's public How to Scale Your Model (jax-ml.github.io/scaling-book): it says “a single scalar core controls a VPU […], 4 MXUs, 2 XLUs, and multiple DMA engines.” ⚠️ Its “4 MXUs” doesn't match v7 (v7 has 2 at 256×256), so the sentence isn't per-generation; we take the 2 only as “more than one.”
‡ NVIDIA states SFUs exist (Blackwell's SM lists Special Function Units), but the official diagram gives no count; ×4 follows Hopper's drawing.

§ This cell gets a note: rare three-tier evidence. “A neural-network chip must have hardware for exp / tanh / sin.” Right, but public evidence doesn't go that far:

① It existed once, on paper. TPU v1's diagram has a separate Activation unit, and the 2017 paper mentions on-chip “nonlinear function logic.” Not new on TPU.
② It is still a separate resource class today, with positive evidence. In open-source JAX, Pallas reports a TPU kernel's cost through CostEstimate(flops, transcendentals, bytes_accessed, …)—transcendentals is a peer column beside flops. Were they just “a few more instructions on an ordinary ALU,” they'd be counted in flops.
③ But the unit's name, count and implementation really aren't public. The TPU v4 ISCA paper's full description of a TensorCore: four 128×128 MXUs, a VPU with 128 lanes (16 ALUs per lane), and 16 MiB of VMEM. Whether any of the 16 ALUs does transcendentals, it doesn't say. Google Cloud's architecture docs, JAX's official Pallas hardware reference, and JAX's machine-readable chip spec table all list only MXU / vector unit / scalar unit / SparseCore. Nothing public describes a TPU counterpart to the SFUs' “table lookup + interpolation.”
⚠️ The last point matters most: “not listed” on a spec sheet does not mean “not there” on the chip. A counterexample sits in the same documents: the XLU isn't listed on any official spec sheet either, yet it exists (found in another public source). A spec sheet lists only what it wants you to buy on. So this cell says “a separate resource class, unit not public”, neither “none” nor any detail I can't back with a public source.

Section 2

⭐ Where they split: one load, 4 runtime decisions vs. 0

Same appetite, completely different eating habits.
The difference isn't a missing part. On one load path, one side makes four runtime-only decisions, the other none.
The cache accounts for two. The first: do a warp's 32 addresses coalesce? The last: what to evict on a miss. Every later difference grows from here.

Think for thirty seconds Loading a block from HBM onto the chip sets off a chain of events.
Which can be fixed at compile time? Which are known only at runtime?
Count the “only known at runtime” ones first; then we walk both paths.

2.1 Same load, two paths

Skip the specs; follow one load from HBM onto the chip. Count the decisions: who makes each, when.

One load, two paths: count the runtime decision points §0's two origins, made concrete GPU: adaptive HW decides for you; sloppy code still runs TPU: planned None at runtime; all planned ahead A warp's 32 threads: one address each Coalesce into how many cache lines? ◆ decide: coalesce ⇢ Same bank? branch: bank conflict ⚠️ banks: shared memory only, not global loads L1 hit? (256 KiB/SM incl. shared) ◆ decide: hit/miss L2 hit? (126 MB, GPU-wide) ◆ decide: hit/miss Evict which line? ◆ decide: evict HBM Compile time: XLA tiles each array 8 × 128 ☑ fixed Compile time: DMA timing & size ☑ fixed Compile time: checks VMEM fit ☑ fixed Runtime: DMA whole tiles HBM → VMEM (64 MiB) Runtime: VPU reads directly; shape already right (no such layer) HW cache = 0 HBM Main path: 4 runtime decisions (+1 via shared) You can only help HW guess: coalesce, dodge banks, hand-tile On path: 0 runtime decisions No cache → no hit/miss/eviction, and no safety net ⭐ Not about smarts: two routes, two bets GPU: runtime adaptation. Any shape runs; hit rate unknown, so tune by trial. TPU: compile-time plan. No cycle spent deciding; cost: no runtime rescue, a bad shape means changing code.
Same load, two paths. Red: known only at runtime; green: fixed at compile time. Main path: 4 vs. 0. Dashed: bank conflict, a branch on shared memory only.

Decision one is the only one unrelated to the cache.
Coalescing merges memory accesses. On a load, each of a warp's 32 threads computes an address; hardware merges them into the fewest transactions.
Merge quality swings bandwidth 8×, and it's known only at runtime: the compiler can't see how addresses are computed. The TPU skips this: arrays are tiled 8 × 128 at compile time and moved as whole tiles. No 32 addresses to merge.

Where the 8× comes from (sector arithmetic, two lines)

What counts is the sector. An NVIDIA cache line is 128 bytes, split into 4 sectors of 32 bytes, and the sector is the real unit of transfer.

Best case: 32 threads read consecutive 4-byte words = 128 bytes, exactly one cache line. 4 sectors, not a byte wasted.

Worst case: the 32 addresses are scattered (e.g., down a column of a row-major matrix), so each thread hits a different sector: 32 sectors, each fetching 32 bytes to use only 4.
32 ÷ 4 = 8×. You pay full bandwidth and discard seven-eighths of it.

⭐ Hence “access memory contiguously” is lesson one of GPU optimization. It optimizes not latency but how much bandwidth each access costs. Source: NVIDIA's cache organization (128 B line / 32 B sector). Nsight Compute reports this directly as sectors per request. A perfect access scores 4; anything above that is waste.

⭐ The course's central decision: this figure's “4 vs. 0” Red doesn't mean “slow”; it means “known only at runtime.” Hit or miss, page walk, queue position: none is known at compile time.
Green doesn't mean “fast”; it means “fixed when the code was written.”

So the labor splits:
The GPU leaves them to runtime hardware, in practice to the kernel writer.
The TPU has the compiler settle them at compile time, so you live with its quirks.

⭐ Section 0's “hedge at every step” is now literal. It's these four decision points, each a real circuit, asking on every access: coalesced? hit? “Hedge nothing” is literal too: the three compile-time givens.
Sections 3 and 5 all follow from this.

⚠️ Why can the TPU assume “the shape is already right”? Because a vector register and an array's memory layout share one shape. Data is used on arrival, no relayout. The cost isn't a non-contiguous row (that's the upside: each segment fills all 128 lanes); it's that the shape is nailed down. A new innermost dimension moves real data, uneven sizes need padding, outside handoffs need relayout. §3.3 has the numbers.

2.3 The main example: why FlashAttention

With the criterion and coordinates set, we run one op end to end. Why FlashAttention? Every large model runs it, and it sits on the wrong side of the 312 line, so both sides must act.

What makes Flash flash: the seq × seq matrix never touches HBM Main example: both load path and on-chip capacity Remember: 1 head, 128K seq, bf16 attn matrix = (2¹⁷)² × 2 B = 2³⁵ B = exactly 32 GiB. A v7 device's 94.74 GiB allocatable won't hold 3 heads; model has 128. ① Naive: intermediates round-trip HBM Q · Kᵀ → seq × seq score matrix S Write S → HBM 32 GiB write Read S back for softmax 32 GiB read Write softmax P to HBM 32 GiB write Read P × V 32 GiB read Per head: 4 × 32 GiB moved, for just 2 matmuls ② Flash: tiled; S never leaves chip Split Q/K/V into row tiles, one at a time S stays on-chip no HBM write online softmax: keep only running max & sum 2 scalars Accumulate, drop S no HBM write Next tile, repeat Full seq × seq matrix never hits HBM ③ Tile size? Same question, different decision-makers GPU: the kernel author. Budget: shared mem 228 KiB / SM; a Triton / CUTLASS knob; mistuned = slow, not broken. TPU: the compiler. Budget: VMEM 64 MiB; shapes must align to 8 × 128; misaligned → zero-pad; pads still take VMEM and MXU time.
①② do exactly the same MACs; only movement differs. ③ gives one question two owners: kernel author (GPU), compiler (TPU).

Panel ③ is what matters. “Who schedules the movement” is the previous figure's decision. On the 312 line, the bill is absurd:

Back to §1's 312 line: what Flash buys This axis returns in L300 §2.8 with five ops; only two here Same head (n = 128K, d = 128), same FLOPs; only the bytes moved differ 16 64 256 1024 4096 16384 65536 Arithmetic intensity (FLOP/byte, log scale) 312: ridge point ← memory-bound compute-bound → Naive attention 64 Flash 65,536 Naive: 64 FLOPs per byte, but the hardware could do 312: nearly 5× short, compute idles, memory-bound Flash: same FLOPs, 1,024× fewer bytes → compute-bound (where it should be) Derivation of both numbers FLOP, 2 matmuls (both): 2 × (2 · n² · d) = 2 × 2 × (2¹⁷)² × 128 = 8,796,093,022,208 FLOP Naive bytes: S, P each written + read = 4 × n² × 2 B = 137,438,953,472 B → intensity = 64 Flash bytes: Q / K / V / O once = 4 × n × d × 2 B = 134,217,728 B → intensity = 65,536 ⚠️ Ideal bound: real Flash re-reads K / V, so < 65,536. Still: it crosses 312, left → right.
Naive code needs 1/64 byte per FLOP; hardware gives 1/312, nearly 5× short. Flash didn't “optimize a kernel”; it moved the op across the ridge, from left to right.

So both sides must tile and rewrite. The rest of the course is that rewrite, side by side.

⭐ Side by side: same attention, identical FLOPs to the last MAC, 1,024× apart in intensity.
So intensity is set by the implementation, not just the math. That licenses every later optimization: if it were intrinsic, only new hardware could help. Because implementation sets it, rewriting pays.

⭐ Does it clear the on-chip bar too? Same division, new denominator (Answer: only if tiles have Br ≥ 64 rows. Expand for the math.)
“Crossing to the right” above means the HBM line (312).
But there is a second, on-chip line: shared memory at 64, the last step of Section 1's staircase.
⚠️ Not the 64 above. The axis's 64 is naive attention's intensity; this 64 is the shared-memory threshold. One is the op's, one the hardware's: pure coincidence.
Fusion moves bytes from HBM onto the chip, so divide again with the new denominator.

Let a tile have Br rows and Bc columns, with head_dim d.
Each step reads a K tile and a V tile from shared memory, Bc × d each; the Q tile stays in registers.
Compute: 2 matmuls, 4 × Br × Bc × d FLOPs.
Movement: 4 × Bc × d bytes (bf16).
Cancel terms, and on-chip intensity ≈ Br, the tile's row count.

Br = 128: 128 > 64, clears the bar.
Br = 32: 32 < 64, memory-bound again, this time on-chip.
That answers Section 1's “why still hit the wall?”: it depends on tile size.

⭐ External anchor: real FlashAttention kernels always use 64 or 128 rows per tile. No coincidence: this division forces it.

⚠️ Counts only shared memory, not softmax vector math or register traffic. Order of magnitude, not exact.

TPU: same method. Numerator 1,153.5 (per core: VMEM is 64 MiB/core, so the denominator is per core too; don't pair it with the per-chip 2,307); the denominator becomes VMEM bandwidth, which isn't published, so we won't guess it. Direction only: on-chip bandwidth is an order of magnitude above HBM's, so the threshold is an order lower, in the tens, like the GPU's 64. The TPU's workbench is 64 MiB (289× the GPU's 227 KiB), so tiles can be far larger; its limit isn't this line but §3.4's blade width.
⭐ Look back: that was operator fusion The last section's line, first put to real use.
Flash fuses “scores → softmax → × V” into one kernel, so the 32 GiB intermediate is never written out.
⭐ It saves not one MAC: panels ① and ② are identical. All the savings are in the denominator.
It's the last section's “push the op right,” a thousandfold.
So the question isn't “fuse?” but “who fuses?”
Here a person wrote Flash. Section 5 asks whether it must be a person.
Does fusion trade saved movement for extra FLOPs, and are they “useful”?

Fair question, with a standard answer. Extra compute comes in three very different kinds:

① Recomputation: the same operations, done again. Flash's forward has none. Panels ① and ② have equal MAC counts, a rarity. But the backward does: the unstored intermediate is rebuilt from Q, K. Saved memory is bought with recompute.

② Padding: the zeros added when shapes don't align. They take on-chip space and pass through the matrix unit but produce nothing. The worst of the three: it buys nothing at all. Section 3 shows how much.

③ Extra vector work: online softmax rescales the running result every tile. It isn't much by count, but it lands on the vector unit, which has under 1/30 of the matrix unit's compute (worked out at the end of Section 1). Counting FLOPs underestimates it; you only see it when you count time.

“Useful” has a formal answer: see Section 6 In short: count only the ops the model's math needs and you get MFU; count every op the hardware executes and you get HFU. Under MFU, recomputation and padding never count as useful.
Why it matters, how far apart they get, what to ask before comparing chips: all in Section 6, where they belong (it asks whether two numbers are comparable at all).

So is the trade worth it? Section 1's line earns its keep again, but first, one point made precise.

⭐ Arithmetic intensity's numerator is FLOPs executed, not FLOPs the model needs. Hardware can't tell recomputation apart. So which side of 312 you land on is set by the intensity including recomputation and padding.

But what you care about is useful FLOPs ÷ time. Together that gives two ceilings; the lower wins:

Useful throughput = min(peak × useful fraction, bandwidth × useful intensity) The first term is the compute side: if half the FLOPs are recomputation, it halves.
The second is the bandwidth side: recomputation doesn't change the bytes moved, so it can't help.
⭐ They cross exactly where intensity including recomputation = 312. So the ridge point stays at 312.

Example: intensity with recomputation 600; half the FLOPs are recomputation.
Useful fraction = 1/2 → compute ceiling = 50% of peak; useful intensity = 300 → bandwidth ceiling = 300 ÷ 312 = 96% of peak. The smaller wins: only half the roofline.

⚠️ Easy slip: bandwidth isn't saturated here. You're compute-bound, so movement waits on compute; bandwidth used is only 312 ÷ 600 ≈ 52%. The matrix unit is 100% busy, half on garbage, while bandwidth idles. The classic fingerprint of overdoing the trade.

⭐ So where should trading recompute for traffic stop? When memory-bound, extra compute is free: not cheap, free. It hides in the wait for data. Past 312, it cuts output proportionally, at once. No transition zone: a cliff, not a sliding rate.
The rule: trade until intensity including recomputation just reaches 312, then stop.
If useful intensity is already above 312 (a large matmul), you're compute-saturated: add no recomputation at all.

⛔ But don't remember it as “intensity shouldn't go much above 312.” Higher useful intensity is always better, no limit: a side-4096 matmul has intensity 1,365, zero waste, 100% of peak. Only the intensity you manufactured yourself should sit right at 312.

Finally, back to Flash: it is nowhere near that bad case. Forward is two matmuls, backward four; not storing the intermediate matrix costs backward one more, 7 versus 6 in total. That is HFU ÷ MFU = 1.167, or a useful fraction of 85.7%.

Data movement drops to 1/1,024. 17% more compute buys three orders of magnitude less traffic. Flash didn't just “optimize a kernel”; it changed the op's bound.

Section 3a

Taking it apart: one complexity, two homes

§2 counted decision points: 4 vs. 0. The first half drills inward: chip → core → floor plan → parallel levels → what one instruction consumes.
At each level: what does each side lack or add, and whose decision caused it?
The second half reassembles them and runs FlashAttention on both, side by side.

3.1 Whole chip first: one hides the seam, one hands it to you

Both chips are dual-die packages. One difference shows only at the package; skip to the core and you miss it.

One B200 — two dies, but one GPU to software · 148 SMs · 126 MB L2 · 8 TB/s HBM3e Gray: 3rd party Compute Disabled SM SRAM Off-chip Links Gray = 3rd-party, not NVIDIA One B200 (package) TSMC 4NP · 208B transistors · two reticle-size dies linked by NV-HBI as one CUDA device; coherent L2 ↔ TPU v7: also 2 chiplets, but inverted: 2 separate devices, each with own address space. Same package, opposite software view. die 0 80 SMs physical · 74 enabled HBM3e × 4 Chip: 8 stacks = raw 192 GB · 8.0 TB/s (186 / 180 GB visible to software, varies by SKU) HBM3e ≈24 GB HBM3e ≈24 GB HBM3e ≈24 GB HBM3e ≈24 GB die 0 L2 · 63 MB Measured 21 TB/s local (3rd-party Vulkan) Split into 2 partitions (4 per chip, 2× Hopper) Cross-die: 16.8 TB/s, higher latency SM grid Each cell = 1 SM. 6 red-boxed SMs fused off for yield: 148 = (80−6) × 2 GPCs: Blackwell Ultra official 8 GPC / 160 SM; 3rd parties report 10 GPC on B200. Unconfirmed, not drawn. off off off off off off die 1 80 SMs physical · 74 enabled HBM3e × 4 Chip: 8 stacks = raw 192 GB · 8.0 TB/s (186 / 180 GB visible to software, varies by SKU) HBM3e ≈24 GB HBM3e ≈24 GB HBM3e ≈24 GB HBM3e ≈24 GB die 1 L2 · 63 MB Measured 21 TB/s local (3rd-party Vulkan) Split into 2 partitions (4 per chip, 2× Hopper) Cross-die: 16.8 TB/s, higher latency SM grid Each cell = 1 SM. 6 red-boxed SMs fused off for yield: 148 = (80−6) × 2 GPCs: Blackwell Ultra official 8 GPC / 160 SM; 3rd parties report 10 GPC on B200. Unconfirmed, not drawn. off off off off off off N V - H B I 10 TB/s D2D coh. Totals SM 148 160 physical Tensor Core 592 148 × 4 CUDA Core 18,944 148 × 128 Reg file 37 MiB 256 KiB×148 L1/shared 37 MiB 256 KiB×148 TMEM 37 MiB 256 KiB×148 L2 126 MB 4 parts HBM3e 192 GB 8 stacks ↑ The three 37 MiB aren't a typo: register file, L1+shared and TMEM are each exactly 256 KiB/SM. Three equal SRAMs, three different jobs. Three paths out — a GPU is no island NVLink 5 · peers · 18 links × 100 GB/s bidir. = 1.8 TB/s / GPU · NVL72 rack: 72 GPUs all-to-all, 130 TB/s aggregate · ↔ TPU v7: 3D torus, 1.2 TB/s per chip, 6 neighbors only · Topology > bandwidth: crossbar vs torus NVLink-C2C · to Grace CPU · GB200 superchip = 1 Grace + 2 B200 · CPU mem GPU-addressable, no explicit copies · ↔ TPU: 4 chips / VM via host interface (HIB) · C2C bandwidth unverified PCIe Gen6 · to host & NIC · Mgmt + data load · Between NVLinked peers, runtime uses NVLink, not PCIe · P2P copy still needs cudaDeviceEnablePeerAccess · Gen6 ×16: 256 GB/s bidir.; vs NVLink 5, roughly 1/7
Figure G-1 One B200, whole chip. NV-HBI stitches two dies into “one GPU,” transparent to software. The seam remains (cross-die L2 drops from about 21 to 16.8 TB/s), and no API shows it. — from the GPU Microscope
One TPU v7 chip — two dies, two “independent” accelerators to software TensorCore (dense) SparseCore (sparse/comm) MXU (systolic) Mem ICI 1 package = 1 chip a.k.a. Ironwood / TPU v7 Peak bf16 2,307 TFLOP/s · fp8 4,614 TFLOP/s chiplet / die 0 Software: own device 0 TensorCore ×1 @ 2.2 GHz 2 MXUs: 131,072 cells total MXU 256 × 256 MXU 256 × 256 VPU (vector) 8 × 128 8 sublanes × 128 lanes/clk VMEM 64 MiB / core SW-managed scratchpad, not cache SMEM 1 MiB Scalars/descr. (≠ IMEM) SparseCore ×2 Each = 1 scalar sequencer + 16 vec tiles SC 0 SC 1 HBM3E 96 GiB Die 0 only — the two dies share no address space Pkg: 8 HBM3E stacks, 7.37 TB/s; ~3,433 GiB/s/die (split: no official source) = half the pkg's 192 GiB chiplet / die 1 Software: own device 1 TensorCore ×1 @ 2.2 GHz 2 MXUs: 131,072 cells total MXU 256 × 256 MXU 256 × 256 VPU (vector) 8 × 128 same VMEM 64 MiB / core same SMEM 1 MiB same SparseCore ×2 SC 2 SC 3 HBM3E 96 GiB Other half: die 1; separate address spaces = half the pkg's 192 GiB Die-to-die link Official: 6× an ICI link. It does not make one device; it only makes crossing dies cheaper than leaving the package. Still 2 address spaces. 6 × What one package contains TensorCore 2 = 2 JAX devices MXU 4 2 per core, 256×256 each SparseCore 4 2 per device SparseCore vector tiles 64 4 × 16 HBM size 192 GiB Official header: GiB; page's GB is a typo HBM BW 7.37 TB/s Also official: 7,380 GB/s, 0.1% off ICI BW 1,200 GB/s 6 links, bidir. sum Same dual-die package, opposite views NVIDIA B200 2 dies + coherent bus → SW sees one GPU. Seam stays (cross-die slower), but HW absorbs it. TPU v7 2 chiplets exposed as 2 devices, own address spaces; seam left to your sharding strategy. Beyond the package: chips per host Official: 4 chips per VM (= 8 devices), 224 vCPUs, 960 GB RAM, 2 NUMA nodes. Host PCIe ~119.2 GiB/s (no official source). Remember the 4: GKE machine types count devices, not chips. Six links off-chip — where TPU and GPU differ most Each chip: 6 physical ICI links, one per ± direction in 3D. Not an optional card-to-card add-on but built into the silicon from day one: a 3D torus. Above 64 chips, built from 4×4×4 cubes: copper inside a cube, fiber between cubes, rewired by optical circuit switches. X+ 200 GB/s X− 200 GB/s Y+ 200 GB/s Y− 200 GB/s Z+ 200 GB/s Z− 200 GB/s 6 links: 1,200 GB/s (bidir.) Trap: official text says “bi-directional bandwidth of 200 GBps per axis,” but 3 axes × 200 = 600, not the table's 1,200. Only “200 per link” (6 × 200 = 1,200) is self-consistent; we draw that. Compare: B200's NVLink 5 is 1,800 GB/s, bigger, but it covers 72 GPUs in a rack; beyond: InfiniBand/Ethernet. ICI's 1,200 GB/s stays one protocol up to 9,216 chips. Easiest misreading here Chip-to-device ratio is 1 : 2, and every framework log reports per device. So “1,153 TFLOP/s per device” hasn't lost anything: it's exactly half of 2,307. And tpu7x-128 isn't 128 chips, it's 64. For MFU the denominator is 2,307 (per chip) or 1,153.5 (per device); mix them up and you're off by 2×.
Figure T-1 One TPU v7, whole chip: also two dies, but exposed as 2 devices, seam in the open. Together: Section 2's decision at package level: one hides complexity, one hands it to you. — from the TPU Microscope
Same build, two presentations Hiding a cost doesn't remove it. You pay without seeing the bill.
This hide-or-expose split recurs at every level. Section 5 prices it.
⭐ Why hiding the seam needs an extra-wide die-to-die link (Bottom line: the 8× isn't a process gap but the price of the “behave like one chip” promise. Expand for the math.)
Two numbers, about 8× apart:
• B200 die-to-die link: 10 TB/s (NVIDIA public figure)
• TPU v7 D2D: 6 × one ICI link (200 GB/s bidirectional each), so ~1.2 TB/s (spec is a multiple of ICI; converted with the course's usual ICI figure)

⚠️ Don't read this as “whose process is better”—the 8× buys two different things.

The reason: the link carries not data but a promise: to everything above, the two dies behave as one chip.
Under it, any access to the other die crosses this link: cache-miss territory, unknowable at compile time. So it's built for an anytime worst case, nearly as wide as on-chip cache bandwidth.

The side that hands you the seam makes no such promise: crossing dies is a compile-time scheduled, avoidable action, not an accident—the link carries only planned ones. Lower demand, narrower link.
v7's link mainly carries the back-and-forth of one tensor-parallel dimension, schedulable in advance, not random bursts.

⭐ The 8× is not a process gap but the price of that promise. The cost is symmetric: one side never thinks about it (but still pays that 21 → 16.8, visible in no API); the other must (but can measure and route around it). ⚠️ Sourcing: 10 TB/s is NVIDIA's public figure; v7's “6 × one ICI link” comes from an internal architecture review; we found no official public source. Shipping the product didn't publish the number. Course rule: such a number gets a source or is marked unsourced. This one is marked.
⚠️ Don't multiply the figure's 131,072 (two MXUs' cell count) straight into peak FLOP/s (Skip “2 MACs per cell per cycle” and you get exactly half. Expand.)

It counts cells: two MXUs, 2 × 256² = 131,072 in all.
But each cell does 2 MACs per cycle; the figure omits this.
⚠️ “2 MACs” isn't “a multiply + an add.” One MAC is itself multiply + add: a × b + c, one cycle, 2 FLOP.
So “2 MACs per cell per cycle” = 4 FLOP, not 2. Miss it: off by 2×.
Common error, 1 MAC/cell/cycle: 131,072 × 2 core × 2 FLOP × 2.2 GHz = 1,153 TFLOP/s, half the header's 2,307. “2 per cell” is missing.
§3.4 walks the full chain.

3.2 Inside a core: five missing pieces, one decision

One level deeper. First what the GPU has, then what the TPU lacks. Reversed, the TPU looks like a “simplified” GPU.

One NVIDIA Blackwell SM, magnified (CC 10.0) · 4 SMSPs (processing blocks) · 128 CUDA Cores · 4 Tensor Cores · 256 KiB TMEM Gray: unofficial Control Vector: CUDA Core / reg Tensor Core / TMEM LD/ST SM-wide Count not public One SM (Streaming Multiprocessor) B200 total 148 SMs (160 physical, 80 per die, 74 enabled) · Blackwell Ultra: 160 · up to 64 warps, 32 thread blocks resident L1 I-cache · thread blocks dispatched by the GigaThread engine stay on this SM A CTA is pinned to one SM; each of its warps to one of the 4 processing blocks below, also for good. These two pins are why every capacity below is ÷ 4. SMSP 0 sub-core / partition L0 I-cache Warp sched ×1 + dispatch Per cycle: pick 1 ready warp, issue 1 instr Warp slots ×16 dark=used, white=free Reg file 16,384 × 32 bit = 64 KiB = 64K regs/SM ÷ 4 · ≤255/thread CUDA Core ×32 Unified FP32/INT32 1 instr → 32 lanes = 1 warp per cycle Tensor Core ×1 (5th gen) 1,024 MACs / cycle ← derived ↓ Inputs skip reg file, use TMEM Takes thin slices, not big squares How thin & how stacked: Fig. G-4 TMEM 32 lanes × 512 cols = 64 KiB ⚠️ Alloc by column, not split; only lanes 0–31 LD / ST ×8 SFU ×4 memory access exp · rcp · rsqrt SMSP 1 sub-core / partition L0 I-cache Warp sched ×1 + dispatch Warp slots ×16 Reg file 16,384 × 32 bit = 64 KiB CUDA Core ×32 Tensor Core ×1 (5th gen) 1,024 MACs / cycle TMEM 32 lanes × 512 cols = 64 KiB Only lanes 32–63 LD / ST ×8 SFU ×4 SMSP 2 sub-core / partition L0 I-cache Warp sched ×1 + dispatch Warp slots ×16 Reg file 16,384 × 32 bit = 64 KiB CUDA Core ×32 Tensor Core ×1 (5th gen) 1,024 MACs / cycle TMEM 32 lanes × 512 cols = 64 KiB Only lanes 64–95 LD / ST ×8 SFU ×4 SMSP 3 sub-core / partition L0 I-cache Warp sched ×1 + dispatch Warp slots ×16 Reg file 16,384 × 32 bit = 64 KiB CUDA Core ×32 Tensor Core ×1 (5th gen) 1,024 MACs / cycle TMEM 32 lanes × 512 cols = 64 KiB Only lanes 96–127 LD / ST ×8 SFU ×4 ↑ Same as SMSP 0, exactly; labels omitted, only TMEM lanes differ SM pair · cta_group::2 4 TCs here + neighbor SM's 4 do a single MMA, shared operands. Note the level: SM-to-SM pairing, not among the 4 SMSPs above, which don't even share TMEM. L1 + shared mem: one SRAM for all 4 SMSPs The explicitly managed level on GPU; ≈ TPU's VMEM Total 256 KiB /SM · shared mem ≤ 228 KiB (≤227 KiB/block, CUDA keeps 1 KiB) Carveout: 0 / 8 / 16 / 32 / 64 / 100 / 132 / 164 / 196 / 228 KiB · static ≤ 48 KiB; beyond needs opt-in 32 banks × 4 B/cyc; long-standing, not restated for Blackwell Other CTAs in the cluster can access it —distributed shared mem, Hopper+ L1 hit ≈ 39 cycles (3rd-party, unofficial) TMA: Tensor Mem. Accel. 1 instr moves an N-D tile: SW fills a descriptor (base, dims, strides, tile shape); HW generates addresses and moves data. The GPU's DMA engine, same class as TPU's: both take address generation off the compute units. Since Hopper, kept in Blackwell. Tex / sampler Shared atomics Measured 32/cycle/SM ⚠️ One SM, three 256 KiB pools, three owners: most-confused spot Reg file · 256 KiB 4 slices, 64 KiB each ⛔ Strictly private: own share only; cross-SMSP: no access L1 + shared · 256 KiB unsplit, all 4 SMSPs access it ⭐ The only path across SMSPs — what “thread block” is built on TMEM 256 KiB Alloc by column (spans all 128 lanes) In between: access split 4 ways (1 warp = 32 lanes), one owner Multiply back by 4: whole-SM totals CUDA Core 128 32 × 4 Tensor Core 4 1 × 4 Reg file 256 KiB 64 KiB × 4 TMEM 256 KiB 64 KiB × 4 L1/shared 256 KiB shared Max warps 64 16 × 4 Max threads 2,048 64 warp × 32 Issue/cycle 4 1 × 4 schedulers Source of “1,024 MACs/cycle” A100 whitepaper: 4 TCs, 1,024/cycle total FP16 MACs (256/core). H100 officially 2× A100; Blackwell 2× more → 1,024/core. B200: 1,024 × 2 FLOP × 4 TC × 148 SM × 1.83 GHz = 2.22 PFLOPS; official 2.25; the 1.4% gap is the unpublished exact clock. Full table in Microscope §2.
Figure G-2 One SM, taken apart. Warp schedulers, 256 KiB of registers, 228 KiB of L1/shared, scoreboards. One job: hiding latency and branches at runtime. — from the GPU Microscope
One TensorCore, taken apart — count what it lacks, not what it has MXU Vec/scalar SRAM DMA Dashed = unpublished One TensorCore (= one JAX device's compute) @ 2.2 GHz Scalar unit issues VLIW bundles One bundle/cycle, slots driving units below in parallel. Which unit fires when, when data lands: all set at compile time. Scalar Vector MXU Misc 9 slots Slot mix from v2/v3 papers; v7 unpublished MXU 256 × 256 65,536 cells × 2 MACs/cell/cycle = 131,072 MACs / cycle spec MXU 256 × 256 65,536 cells × 2 MACs/cell/cycle = 131,072 MACs / cycle spec VPU vector unit 8 sublane × 128 lane 1,024 elements/cycle Activation, norm, eltwise, reduce—all non-matmul work. The lanes and sublanes of §3 come from this 8×128 grid. Cross-lane ×2 Transpose, reduce, shuffle. The VPU's 128 lanes work alone; data must cross here. Count: no official source SRAM — sizes from JAX source, BW unpublished VMEM 64 MiB SW-managed scratchpad. Not a cache: no auto-fill, no eviction policy, no hits or misses SMEM 1 MiB Scalars + DMA descriptors Accumulators 128 Each (8, 256), 32 bit, on each MXU vregs unknown v2/v3 papers: 32 deep/sublane; v7 not public Off-core: DMA DMA moves HBM ↔ VMEM; descriptors come from the scalar unit, so when/how much to move sits in the instr stream, not HW-decided. DMA engines per TensorCore: unknown. So we draw one box, with no count. In a GPU SM, absent here What's gone, then where the silicon went ✕ 4 warp schedulers Who picks what, when — there is no picking here ✕ 64 warp slots (2,048 thread contexts) No threads, so no switch-threads escape hatch ✕ 256 KiB reg file Its size is mostly for dozens of resident contexts ✕ L1/shared auto-fill VMEM is pure scratch: who fills it owns it; no hit rate ✕ Scoreboard/OoO Moved to compile time; cost (Microscope §7): no safety net Saved silicon → SW scratchpad VMEM per TPU v7 chip: 128 MiB (64 MiB per core), and every single byte is compiler-placed. TPU v7 VMEM per chip 128 MiB B200 total shared memory ~34 MB 3rd-party Counts SW-managed scratch; excludes B200's 126 MB L2. One line to remember Most GPU SM transistors don't compute; they decide what to compute next — schedulers, contexts, scoreboards, cache control, all hiding latency and branches at runtime. TPU moves it all to compile time, so those parts go. Not “area for compute”: the machinery just isn't built — simpler circuits, lower power. It didn't become more compute or more SRAM (B200 has more of both). Not “TPU is simpler,” but complexity relocated: silicon → compiler.
Figure T-2 One TensorCore taken apart; see the right column. All five items are absent: not five independent trade-offs but consequences of one decision: no runtime task switching. — from the TPU Microscope
Pause here: the course's key contrast The TPU's five missing parts weren't saved; they became unnecessary.
With no runtime task switching, warp schedulers, scoreboards, dozens of warp slots, resident contexts, dynamic branching: all moot.

Where did the saved area go? Not into more compute or on-chip memory. Tables in §3.7 and §3.2b settle it: the GPU has 16% more total MACs and 72% more on-chip SRAM.
It went into absence itself: no cache or scheduler, and the same compute in 4 big blocks, not 592 small ones: 148× fewer control interfaces (592 vs. 4). ⚠️ Power and yield gains? We haven't measured either; official sources cover part (see fold-out).
Official source: the first TPU paper (ISCA 2017) argued this, and why it fails ten years on (quotes + four caveats)

The paper lists the omissions in one sentence (Jouppi et al., ISCA 2017, In-Datacenter Performance Analysis of a Tensor Processing Unit, arXiv:1704.04760, p. 8):

the single-threaded TPU has none of the sophisticated microarchitectural features that consume transistors and energy to improve the average case but not the 99th-percentile case: no caches, branch prediction, out-of-order execution, multiprocessing, speculative prefetching, address coalescing, multithreading, context switching, and so forth. Minimalism is a virtue of domain-specific processors.

What the savings became, per the abstract:

The lack of such features helps explain why, despite having myriad MACs and a big memory, the TPU is relatively small and low power.

⭐ The hardest number is in the Figure 2 floor plan: die area is data buffers 37% · compute 30% · I/O 10% · control logic just 2%. The next sentence:

Control is much larger (and much more difficult to design) in a CPU or GPU.

“Simple” on a floor plan: 2%.

On “stability,” the official word differs: not yield but deterministic execution. The paper argues CPU/GPU time-varying optimizations (caches, out-of-order execution, multithreading) help average throughput, not tail latency; TPU determinism better fits their 99th-percentile response time targets. ⚠️ No source found on yield/reliability, so no such claim.

⛔⛔ Four caveats for any citation of this paper ① It compares TPU v1 (2015) with the K80 (2014). Then the savings did become compute: a small die packed 25× the MACs, 3.5× the on-chip memory, at under half the K80's power.
⭐ But for v7 vs. B200, the conclusion flips: §3.7's accounting table puts total MACs at 606,208 vs. 524,288; the GPU has 16% more. Same mechanism (minimalism), different outcome a decade on.
The only defensible claim today: “what's saved is complexity,” not “the savings became compute.” Our opening once made that mistake, recalling the 2017 result. The constraint lifted a generation later; the conclusion stayed.

② Ironwood's official “2×” perf/watt is vs. the previous TPU (Trillium), not GPUs. (Google blog, 2025-04-09; peak FP8 flops per watt of per-chip TDP.) Using it against a GPU swaps the denominator.

③ ⛔ “So how much area was actually saved?” Public information can't answer this. NVIDIA publishes 208B transistors for B200 on two reticle-limit dies (~800 mm² each); Google never publishes TPU die size or transistor count. With only one side publishing, “whose die is smaller” has no common basis.
⭐ But the savings can be itemized, from this course's own two tables: TPU has no L2 at all (126 MB on B200), nor a register file for thread contexts (37 MiB on B200): 157 MiB of on-chip SRAM, spent on two things the TPU lacks entirely; plus 148× fewer control interfaces (592 Tensor Cores vs. 4 MXUs, a gap of 588: 148 is the ratio, not the gap). SRAM and control interfaces both take area. Only “whose chip is bigger” has no answer.

④ Yield and reliability: no citable source, so no claim. The official claim is “deterministic execution,” meaning predictable tail latency, not reliability. Full paper excerpts (all omitted features, the 2% floor plan, all v1-era results) are in L300 §3.2's fold-out.
Contrast notes: the five components, and the most-confused number
📖 Five parts above are for contrast only, not explained Topic 2 asks how the two differ, not how a GPU works inside. Five index entries are folded below.
Where to read about these five in the GPU Microscope (reference · skip when presenting)

① Warp scheduler: the part, GPU Microscope Section 2; at work, Section 7 (“Hiding latency: one swaps workers, one plans shifts”).
② Which silicon a thread / warp is pinned to: Section 3, each abstraction layer matched to silicon.
③ How an SM uses registers: Section 2. Preview: 256 KiB / SM split four ways, 64 KiB per processing block, ≤ 255 registers per thread; so large not for speed but to hold dozens of contexts at once.
④ How L2 is used: capacity and partitions, Section 1; a value's stops between HBM and the MAC units, Section 6.
⑤ How SMs cooperate: Section 3, clusters / distributed shared memory; it exists because the 148 SMs have no fast path between them but L2. The SM figure's cta_group::2 is a tighter tier: two SMs' Tensor Cores team up on one matmul.

⚠️ An easy number to mix up (the next table uses it) B200's L2 is 126 MB (63 MB per die, 4 partitions chip-wide). Not thirty-something MB.
That's the chip's total shared memory, 148 SM × 228 KiB = 33,744 KiB, about 33 MiB.
Nearly 4× apart, and they differ in kind by this course's very decision: L2 is managed by hardware; those 33 MiB, by software.

Next: on-chip real estate. §3.6's whole fusion payoff comes from intermediates staying there. Three questions: how big, how fast, how long you wait, a subsection each.

3.2b “But the GPU has more on-chip memory”: by what measure?

Fair challenge. Every on-chip SRAM on both chips, none skipped:

RoleWho decides GPU B200 (chip)TPU v7 (chip)
L2 cacheHardware 126 MB ≈ 120 MiBNo such level
Thread contexts (registers)Hardware / compile time 37 MiB (256 KiB × 148)None: nothing resident
Matrix operand stagingExplicit moves TMEM 37 MiBAccumulators 4 MiB
Software-managed scratchpadThe compiler / you L1+shared 37 MiB
≤ 227 KiB per thread block
VMEM 128 MiB + SMEM (scalar) 2 MiB
64 MiB block per core
Total— ≈ 231 MiB≈ 134 MiB
Excl. vector regs (unpublished)

Capacities from the two Microscopes' source notes (L2, shared memory: third-party tests/specs; VMEM 64 MiB / core, SMEM 1 MiB / core: JAX open-source code).
⚠️ TPU's SMEM is scalar memory (scalars, DMA descriptors), not CUDA's shared memory; in this table VMEM is the counterpart. Totals are our sums: 37+37+37+126 MB→120 MiB = 231 MiB; 64×2+1×2+1×2×2 = 134 MiB.

Same table, three yardsticks, three conflicting verdicts ① Total capacity: GPU wins, 231 vs. 134, 72% more. True; don't argue.
② Software data scratchpad (the “software-managed scratchpad” row only): TPU wins, 130 vs. 37, about 3.5×. The earlier “saved area went into on-chip scratchpad” meant this, unstated, so reading it as ① was fair.
⚠️ This excludes the “matrix operand staging” row. By “who decides,” TMEM's 37 and accumulators' 4 are software-managed too; with them, 134 vs. 74, 1.8×. §3.6 cites the first. Quote ratios with their rows.
③ Contiguous workbench per compute unit: TPU wins by 289×. 64 MiB ÷ 227 KiB ≈ 289. The GPU's 37 MiB isn't one block but 148 mutually invisible 256 KiB islands, L1 taking part of each: one thread block can explicitly manage at most 227 KiB.
⚠️ The 64 MiB is the total floor space, not a one-shot budget The table compares capacity. The 289× hides a caveat: one op's VMEM use has a separate cap.

XLA gives each instruction or fusion its own “scoped VMEM limit,” set by the target chip plus a tunable flag (xla_tpu_scoped_vmem_limit_kib; raising it when things don't fit is routine). The compiler tiles against this limit, not the 64 MiB.
Other tenants share the space: the pass that picks VMEM-resident tensors, and double buffering (prefetching the next tile).

⭐ 64 MiB caps how big a tile could be; the scoped limit sets this one's size. ⚠️ The default is unconfirmed and may vary by generation, so we name the gate, not a number. Measure on your target.
Two traps: the 289 is a cap divided by a cap, and ① misleads most (expand before quoting)
⚠️ The 289 is a cap over a cap: both “exclusives” are inflated The GPU's 227 KiB is not “one Tensor Core's workbench.” It's the cap one thread block can request; the SM has 4 Tensor Cores, maybe other resident blocks: all compete for that space. (Per-SM shared memory sizes: 0/8/16/32/64/100/132/164/196/228 KiB. 228 is the largest; a thread block can explicitly manage 227.)
Nor is the TPU's 64 MiB all yours. Officially, Ironwood has 64 MB of VMEM per TensorCore (two per chip, hence 128 MB/chip); the 64 is split between the current step and prefetching the next weights. How it's split is set by a flag (xla_tpu_scoped_vmem_limit_kib). ⚠️ The default is unpublished; official tutorials often set it to 65536 (= 64 MiB), implying a much lower default.

⭐ Read precisely, 289 is: the contiguous scratchpad one kernel instance can own, as a ratio of upper bounds. It still decides how big one fusion can be. But don't read it as “TPU has two orders of magnitude more on-chip memory”; that's yardstick ①'s question, and there the GPU has more.
⭐ Why ①, though correct, is the easiest one to be misled by Almost all the GPU's 97 MiB lead is the L2 row. L2 is also the only cell here you can't arrange: you can't “park” weights in L2, only load and hope they're still there.
Counting L2 as “my on-chip space” is the very mistake we've guarded against since Section 2. Two of Section 2's four runtime decision points (L2 hit, eviction) are about it.

⭐ The three yardsticks aren't interchangeable; pick the one matching your question: for “how much SRAM does this chip have,” ①; for “where can I keep intermediates without spilling to HBM”, only ③. Operator fusion faces that second question every time: in §3.6's side-by-side FlashAttention, this 289× sets each side's tile size.

3.2c Speed: L2 is big, but it isn't fast

§3.2b covered size and ownership. The other half is speed, with a result almost everyone guesses wrong.

LevelBandwidthvs. HBMSource
HBM3e (off-chip)8.0 TB/s1×Official
L2, local21 TB/s2.6×Third-party
L2, cross-die16.8 TB/s2.1×Third-party
L1 + shared (chip total)≈ 35 TB/s≈ 4.3× 128 B/cyc/SM
Measured
TPU VMEMUnpublished. Google says only: to the MXU, “significantly higher” than HBM. Order of magnitude: our estimate Magnitude only

⚠️ ≈ 35 TB/s is derived; 128 B/cycle is a ceiling, not a guarantee. The B200 clock is uncertain too, hence “≈ 35.” Derivation, backing, caveats below.

This row's source, and why we can state it now (provenance · skip in class)

Derivation: 32 banks × 4 B per bank per cycle = 128 B/cycle/SM, × 148 SM × 1.83 GHz = 34.7 TB/s. 128 B/cycle has third-party backing: a Hopper microbenchmark (arXiv 2402.13499 Table V) measured on H800 shared memory 127.9 and L1 125.8 B/cycle/SM—one ceiling, so one data path. Two more public sources independently give 128.
Cross-generation anchor: the formula gives H100 132 × 128 × 1.755 GHz ≈ 29.7 TB/s; a third party reports ~33 TB/s for H100 (implying ~1.95 GHz): magnitude matches.
⚠️ Two caveats: ① the B200 clock is uncertain (1.83 is a third-party boost figure; another measurement implies ~1.99), hence ≈ 35 TB/s; ② 128 B/cycle is a ceiling, not a guarantee: in the same table the RTX 4090 gets only 63.7 with scalar FP32, 121.2 with FP32.v4; the H800 gets 125.8 with scalar loads. Too-narrow accesses can't saturate it.

Why the TPU row gives a magnitude, not a number (note on figures)
⚠️ Why the TPU row gives a magnitude, not a number VMEM bandwidth is unpublished: the spec page gives capacity only. Officially it's just “significantly higher bandwidth to the MXU than HBM” (Google Developer forum's Ironwood tuning post); we estimate ~an order of magnitude from that, while the GPU's shared memory is 4.3×.
Certain whatever the numbers: each against its own HBM, the TPU's level pulls further ahead. Don't fill in a precise number or integer multiple: HBM bandwidth is public, so a multiple would reveal the number.
⭐ The counterintuitive one: L2 is big, but only 2.6× faster than HBM Most assume a cache is “an order of magnitude faster.” B200's L2 isn't.
L2's value isn't its own speed but skipping an HBM round trip. A hit gets 2.6×; a miss saves nothing and adds a detour.
So “just land it in L2” isn't a reliable strategy: the gain is small and not guaranteed. Two of Section 2's four decision points bet on this 2.6×, and it's only 2.6×.
⚠️ But “2.6×” assumes big transfers, which often fails That's peak bandwidth, reached only in HBM's best case: one large contiguous block at a time.
Each HBM access pays a fixed startup cost (hundreds of ns) and needs a very wide bus fetching in parallel to hit full speed. Small transfers do neither, so effective bandwidth falls 1–2 orders of magnitude below peak.
On-chip levels avoid this: close, fine-grained, nearly free to start.
⭐ The smaller the transfer, the more an L2 hit gains: 2.6× is its lower bound, not its typical value. This recasts §2.1's sector math: contiguous access also makes each transfer big enough, not just avoids waste. The drop varies with HBM generation, vendor and access pattern, so we give the direction, not the curve.
⭐ Why only 2.6×, and the question that flips it Why slow: L2 is shared chip-wide. Shared memory (≈ 4.3×) and VMEM (about an order of magnitude) both beat 2.6× because they sit beside the compute units, each SM reading its own; L2 data crosses the on-chip interconnect. (Architectural reasoning, not measured attribution. Dropping tags and replacement also saves area and power; distance is most of the gap.)

⭐ “2.6×” answers only a narrow question: how much faster is this byte from L2 than from HBM?
Ask instead about bandwidth per byte of capacity, and it flips: L2 gives 2.6× the bandwidth with 1/1524 the capacity. Multiply: L2's bandwidth density is about 4,000× HBM's.
Concretely: reading all of HBM takes 24 ms; all of L2, 6 µs (all L1 + shared memory: 1.1 µs).
⭐ Two numbers, two jobs; don't merge them • “2.6×” guides optimization decisions: L2 hits pay little; don't rely on them. That stands.
• “4,000×” explains the architecture: why the chip centers on on-chip memory.

⭐⭐ Their ratio is exactly the capacity ratio: on-chip memory's whole edge is being small. Fast for being small, close, short-wired, not for being SRAM.
Reversed, it's the course thread: fusable size and fusion payoff are one plot (§3.6's “stovetop”). ⚠️ No TPU ratio (no VMEM bandwidth data). Same trend, more extreme: VMEM > L2 (128 MiB vs. 126 MB); its HBM multiple (Google: “significantly higher”; est. an order up) dwarfs 2.6×.

Both truly fast levels are software-controlled.

Which GPU level is VMEM? None. By behavior, shared memory (software-moved, no tags, never misses); by capacity, L2-class (128 MiB vs 126 MB; the chip's shared memory totals 37 MiB). Precisely: the TPU built shared-memory behavior at L2 size, then deleted the L2 level.

⛔ Don't read this as “the TPU won three rounds” All three leads are real, and each has a price; remember both.
① A big, fast scratchpad, but no safety net. No cache, so nothing covers a wrong guess: a wrong GPU guess slows things a bit; a wrong TPU tiling leaves the chip stalled. This predictability is paid for, not free.
② The GPU's 37 MiB register file + 37 MiB TMEM look “not spent on data,” but they buy something: irregular shapes, varying lengths and unpredictable branches still run decently. On a TPU, the same work needs fixed shapes first. Section 5 does that math.

⭐ So the conclusion isn't “whose on-chip memory is better.”
Both handle the same thing, uncertainty, in different places.
The GPU spends silicon to absorb it: caches, fallbacks, schedulers.
The TPU eliminates it at compile time; the saved silicon became less circuitry, not more compute (§3.7 counts it). The price: you (or the compiler) commit up front.
Choosing hardware: regular, predictable work suits the TPU; irregular, dynamic-length work justifies the GPU's insurance premium. Section 5 prices that premium.
Two FAQs: can a die use the other die's L2? How do SMs swap data?

① L2 isn't “split in half per die”; it's 4 partitions, two per die (twice Hopper's count). A die uses the other die's L2 with no effort from you: NV-HBI (10 TB/s) joins the dies into one CUDA device with globally coherent L2. The cost is in the table: 21 drops to 16.8, and latency rises. §3.1's “you pay, you just don't see the bill,” made concrete. The TPU is the opposite: two halves, 2 devices; using the other half is an explicit step you write.

② SMs exchange data in three very different tiers: In a cluster (≤ 8 blocks, opt-in 16 since H100, same GPC): distributed shared memory, reading and writing other blocks' shared memory directly, bypassing L2. Blackwell adds a tighter tier, cta_group::2: two SMs' Tensor Cores on one matmul. Outside a cluster, only L2 and HBM: no other fast on-chip path, so sync goes through atomics, cooperative groups, or splitting the kernel.

⭐ Remember the direction of cause and effect: clusters were invented precisely because the 148 SMs have no path between them but L2. It's a layer wedged in between “one SM” and “the whole GPU.” The TPU doesn't need it: a chip has only 2 TensorCores, so core groups never come up. Few cores remove a whole abstraction layer: scale, not design skill.
More detail in GPU Microscope Section 3.

3.2d Latency: the same load, 30 cycles to 659

§3.2b asked “how big,” §3.2c “how fast.” One axis remains: how long you wait. It's the one the GPU itself can't answer.

The question: a warp needs data; not in L1 → check L2 → not there either → fetch from HBM. How long does the chain take?

Latency ladder: one load, 30–659 cycles §3.2b: capacity; §3.2c: bandwidth. Axis 3: latency—and the three disagree. NVIDIA GPU: where can one load land? Pointer-chase, unit: cycles. Top five: H100; last: consumer Blackwell. Register ≈ 1 cyc — right in hand Shared / L1 hit 30–40 cyc — H100 and Blackwell alike L2 hit 273 cyc — 7–8× slower than L1 L2, both partitions busy 508 cyc — no edge when congested All the way to HBM 659 cyc — full-miss total (Ref.) consumer GDDR7 877 cyc — new memory, new number ⛔ Don't add these. 659 cyc = one full miss, end to end, L1 and L2 misses included. Not the sum: one instruction swings from 30 to 659. ⚠️ No public B200 data. Scale and shape carry over; not B200 measurements. Source: arXiv 2507.10789 pointer-chase. TPU: constants only Not a hidden number: there is no “maybe hit, maybe miss” tier. VMEM load —compiler-run scratch. Not there by luck, the compiler's own DMA put it there HBM → VMEM DMA —latency too, but a scheduled transfer: known delay; planned waits ⭐ No cache misses on TPU: there's no HW cache. Caches guess; compiled schedules don't need to. How to plan? The GPU doesn't The 20× gap is invisible in source—same load. None of its 3 remedies tries to predict: ① Warps —dozens of warps per SM; stall → switch hidden behind others' work ② Scoreboard —issue when operands arrive ③ Cache —a guessing machine, bets on reuse ⚠️ No tuning knob is shared: GPU tunes “enough other work,” TPU “schedulability.” ⭐ In one line GPU hides latency behind other work; TPU removes the uncertainty. Statistical vs. deterministic. Section 1's theme, now in units of cycles. ⚠️ Not a divergence: peak CUDA kernels self-manage double buffering via async copy + barriers: a hand-written static schedule on GPU.
⭐ One load instruction, identical in source, varies 20× in latency. ⛔ Don't add the segments: 659 cycles is one full miss end to end, L1/L2 lookups included. ⭐ The point is the right half: on the TPU this axis has only constants (not secret numbers), no “maybe hit, maybe miss” tier. Numbers: pointer-chase microbenchmark, arXiv 2507.10789 (H100, consumer Blackwell). No public B200 data: borrow magnitude and shape only.
⛔ The easiest mistake: these segments don't add up “An L1 lookup takes time, L2 more, the HBM fetch more still”: this double-counts.
659 cycles is one full miss, issue to data, measured end to end. The L1 and L2 misses are inside those 659.
Remember the swing, not the sum: one instruction, 30 to 659 cycles.
⭐ So how do you plan? The GPU doesn't “Every case differs, so you can't schedule ahead”: right. The GPU's answer isn't “I can predict it” but “I don't need to.”

None of its three remedies predicts latency:
• Oversubscription: dozens of warps resident per SM; when one stalls, switch. Latency isn't eliminated; other work covers it.
• Scoreboard + dynamic issue: hardware issues each instruction at runtime once its operands arrive.
• The cache is a guessing machine: it bets you'll reuse the data.

The TPU deletes the uncertainty at the source. No hardware cache → no misses → no variable latency. Data sits in VMEM not by luck but because the compiler's DMA put it there; it knows which cycle to wait on.

⭐ In one line: the GPU hides uncertain latency behind other work; the TPU deletes the uncertainty. One answer is statistical, the other deterministic.
So lacking L2 isn't a missing feature: a machine scheduled to the cycle at compile time needn't guess.
⭐ Also why the two sides' “tuning” shares nothing On a GPU you tune for “enough other work”—occupancy, resident warps per SM, coalescing, trading registers for warps.
On a TPU you tune for “room for the compiler to schedule”—dimension alignment, tiling, enough double buffering, scoped vmem size.
Same word, no overlapping knobs. So experience barely transfers. Not because it's hard: your knobs don't exist on the other side.
⚠️ Not drifting apart; quite the opposite Modern CUDA kernels are moving toward the TPU. Async copies, TMA, explicit barriers: peak kernels like FlashAttention now issue their own async transfers, double-buffer and wait on their own.
That's a hand-written static schedule on the GPU. At peak, “guessing” isn't enough.
⭐ It returns in §3.5's further reading (Blackwell's matmul moves the accumulator out of registers, single-thread issue): same direction, two components.

3.3 Parallel hierarchy: which levels are set at runtime

First, two coordinate terms. This section and §3.4 rely on them, and both names mislead:

Lane and sublane are the shape of a register—not a memory concept, not a size Source: JAX public source, v7 branch: num_lanes = 128 / num_sublanes = 8 A vector register … … … … … … … … 128 cols 1 lane red column: its own ALU 1 sublane = one full row 128 cells long = 512 B 8 rows Whole register = 8 × 128 × 32 bit = 4,096 B = 4 KiB ⚠️ “sub” doesn't mean “smaller”: a sublane isn't a piece of a lane, it's a full row across all 128 lanes Two steps: one row, 128 cells long; stack 8 rows = one register. Mnemonic: innermost dim → lane, next → sublane. ⭐ XLA deliberately mirrors it in memory layout Array […, C, D] in memory: last dim D → lane (by 128), next C → sublane (by 8). So an [A, B, C, D] array is stored as A × B × ⌈C/8⌉ × ⌈D/128⌉ tiles of 4 KiB. ⚠️ So a “row” isn't contiguous: it's split across tiles. Next: what that costs.
One row is 128 cells; stack 8 rows: a register.
Lane = position in the row (0–127); sublane = which row (0–7).
⚠️ A sublane isn't part of a lane; it's a full row across all 128 lanes. “Sub” = second dimension, not smaller.
XLA mirrors register shape in memory on purpose; cost: a “row” isn't contiguous.

The die lost components. One level up, the programming model loses things too: same cause, again.

It comes down to one question.

⭐ Where everything starts Writing GPU code, you don't know which SM your work lands on, or when.
Hardware decides at runtime.

So you can't name anything. You can't write “SM 3, do this.”

Think for thirty seconds If you can't name the hardware, what can you still tell it? Hint: all you still know is how the pieces of work relate to each other.

All you can do: describe relationships. What must stay together, what can land anywhere.

The batch that “must stay together” is a block.

Why “must”? Because its pieces swap intermediate results,
in the SM's own small local memory.
Once a batch lands on an SM it can't move, or the intermediates are lost.

Work that never talks can land anywhere; all of it is the grid.

These two levels are yours. Your code says two things: these go together; this many in total.

Hardware has its own groupings, for another reason: saving circuitry.

Instruction fetch is costly. So 32 data elements share one.
That bundle of 32 is a warp. The number is fixed in silicon.

Four levels: bottom two fixed by hardware, top two set by you.

Two more came later, each forced by a specific problem:

warpgroup: newer matrix instructions are shaped for four warps issuing at once.
cluster: neighboring SMs want direct access to each other's local memory, no detour.

All six side by side, and what forced each.

Why six levels? Each one was forced by a concrete problem Nesting, who decides: folded. Silicon: Microscope G-3. Here: why each level exists Level Size Guarantee ⚡ Why this level exists grid grid All blocks of a kernel Only via L2/HBM; no on-chip path — a boundary, not forced cluster thread block cluster ≤ 8 blocks (16 opt-in, H100+) Same GPC: R/W peers' smem directly One SM's 227 KiB too small. Cost: neighbor's scratch via GPC network, slower Thread block CTA / block ≤ 1,024 threads (32 warps) On one SM: shares smem, __syncthreads “Who shares scratch, who syncs” needs a boundary: the CTA is that scope Warp group warpgroup 4 warps = 128 threads All 128 issue one wgmma Tensor Core outgrew one warp's registers; issue unit moved up a level warp warp 32 threads One instr stream; diverges only here Fetch/decode is costly: do it once for 32 elements, 32× less overhead Thread thread 1 Own regs; no own fate: follows warp — a boundary, not forced ⚠️ Three things people get wrong: 1,024 is the limit per block (= 32 warps), not per SM. An SM holds 64 warps = 2,048 threads at once (B200 / H100 / A100; consumer parts and Turing hold fewer). That is the pool of “other work to switch to.” Inside a warp, implicit lockstep ended with Volta: each thread has its own PC; sync needs __syncwarp(). ⭐ None of the six exists to make code easier Together: the organizational cost of runtime adaptation: runtime swapping and grouping need these levels in HW + schedulers, scoreboards, big reg file. ✕ TPU column: empty. No runtime “who runs next,” so all six levels lose their purpose; the two blanks in T-3 are this missing column.
Only grid and thread are boundaries. The other four each answer a hardware problem: costly fetch/decode, so 32 threads share one instruction stream; one warp can't feed a Tensor Core, so issue rose to 4 warps; shared scratch needs a scope, hence the CTA; one SM's 227 KiB is too little, hence clusters. Together: the organizational cost of runtime adaptation. The TPU skips that decision, so the column needn't exist.
⭐ The takeaway Trace those problems one step further back.
All six levels grow from the opening “you don't know.”
Runtime placement is why you need a language for who must be together, who sees whom.

The TPU fixes everything at compile time. No “you don't know.”

It needs not one level of that language.

The two sides' levels answer different questions. GPU levels describe who must be together; the TPU never needs to.
So the table below is ordered by question.

Hierarchy side by side — aligned by question, not by name NVIDIA B200 TPU v7 No such level on TPU Same question NVIDIA B200 TPU v7 What is the smallest unit? Smallest nameable executor 1 thread Own PC, own registers. Can take its own branch. 1 element One vector cell. No PC, no branch of its own: data, not an executor. Size of HW lockstep group? All run the same instr at once 1 warp = 32 lanes 32 in lockstep. Diverge: both paths run. 1 vector instruction = 8 × 128 cell = 8 lanes 8 sublanes × 128 lanes = 1,024 elements in step. No “divergence”: there are no branches. Who shares a scratchpad? Who sees whose on-chip data 1 thread block A block's warps share one smem (size set in kernel); count known only at runtime. ✕ No such level VMEM is the whole TensorCore's, not a group's. Visibility fixed at compile time; no “group” level. Runtime: who runs next? On a stall, can HW switch work? 4 warp schedulers / 64 warp slots Stalled warp instantly swapped for a ready one: all there is to GPU latency hiding. ✕ No such level Issue and data timing fixed at compile time. Stall = true idle; nothing steps in (Microscope §7). What's in one core? Smallest block w/ full control path 1 SM ×148 128 CUDA + 4 Tensor Cores + 228 KiB L1/shared (size: third-party) 1 TensorCore ×2 2 MXUs + 1 VPU + 64 MiB VMEM + 1 MiB SMEM Chip = how many devices? What devices() counts 1 2 dies, 1 GPU (coherent bus); API hides the slow seam. 2 2 chiplets, 2 devices. Visible seam: your sharding's job. How far on one protocol? Beyond: another interconnect 72 NVLink 5 1 NVLink domain; past it IB/Ethernet, new prog. model. 9,216 ICI 4.0 Pod-wide 3D torus, one ICI. Payoff: interconnect section. Look at the blank cells In two of seven rows the TPU cell is red, exactly the two the GPU uses to hide latency at runtime. Blank 1: no “group of threads” GPU block is a runtime concept: count and SM known only at launch. TPU has no middle layer; all ownership compiled into instructions. Blank 2: no “swap in another” Stalled GPU warp → next one: needs dozens of resident contexts: the 256 KiB reg file's main job. TPU keeps none, saving that area (see §3.2). So every TPU level describes “shape” lane, sublane, tile, slice: all say what data looks like, not who runs. TPU: tune sharding; CUDA: tune threads. Also answers a common question: why is irregular code awkward on TPU? Not compiler refusal: the hierarchy has no unit to carry irregularity. The table in a line The GPU hierarchy is a hierarchy of executors; the TPU's is a hierarchy of data shapes. thread, warp, block all say who runs; lane, sublane, tile, slice all say how the data is cut. In practice: GPU perf is mostly occupancy (enough resident warps to hide latency?); TPU perf is mostly shape (dims aligned, shards even?). Instincts don't transfer, which is why in the contraction-dim section head_dim=128 hurts TPU, not GPU.
Figure T-3 The TPU side is aligned by question, not name, on purpose. By name, a tidy glossary; by question, two TPU cells are empty: the GPU's block and warp scheduler. — from the TPU Microscope

CUDA's full nesting thread → warp → (warpgroup) → block → cluster → grid is folded below for reference. What to remember: the three “mismatches” after it.

Full six-level nesting table (reference · skip in class)
LevelSizeSet byMaps to
thread1— 1 lane in a sub-core
warp32 threads Fixed in hardware, not yours 1 of 4 sub-partitions (sub-cores)/SM
warpgroup4 warps = 128 threads Fixed by the ISA One warp from each of 4 sub-partitions
block (thread block)
(= CTA)
≤ 1,024 threads = ≤ 32 warps You, in code All on one SM; never migrates
cluster≤ 8 blocks
opt-in to 16 from H100
You, optional (Hopper+) Several SMs in one GPC
gridAll blocks of a launch YouAll SMs (B200: 148, two dies)
⭐ Skip the nesting; remember three mismatches ① An SM hosts several blocks. A block caps at 32 warps; an SM has 64 warp slots (= 2,048 threads), so blocks share an SM, blind to each other's shared memory. §5.1's “occupancy” counts filled slots of the 64.
② A warpgroup is an instruction unit, not a scheduling unit. Nothing “schedules a warpgroup”; it's the shape wgmma/TMEM need four consecutive warps to issue together. Hence §3.4's trend of matrix instructions mobilizing ever more threads, until Blackwell's tcgen05 reverses it: a single thread issues it.
③ The cluster came later, for §3.2c's reason. The 148 SMs connect only through L2; a cluster lets blocks in one GPC read and write each other's shared memory directly, bypassing L2. The TPU needs none: a chip has just two TensorCores.

⭐ Rule of thumb: hardware fixes the first three levels' boundaries; your code sets the last three's sizes. What each level shares, how it syncs: GPU Microscope, Figure G-3.
Origins of three names: warp / CTA / warpgroup (etymology; open only if the figure didn't land)

Warp comes from weaving: on a loom, the warp is the lengthwise threads pulled together, parallel strands that move as one. The threads are, literally, bound together.

Why 32? Wider amortizes fetch/decode better, but divergence costs more, coalescing coarsens. AMD long used 64 (the wavefront); newer designs went back to 32.

CTA stands for Cooperative Thread Array, and it is the thread block: CUDA calls the level a block; PTX and the hardware call it a CTA.

warpgroup does more than issue wgmma; it enabled warp specialization: one warpgroup only moves data from HBM to shared memory, the others only issue matrix instructions. Split movers from computers and the pipeline overlaps. FlashAttention-3 builds on this: the GPU side's shape in §3.6.

Deeper still. “An SM” has been a black box. Open it and the numbers above need no memorizing.

Inside one SM: not one block, but four quadrants Why warp = 32, how many fit, why switches are free: all here 1 SM (148 on full B200) SMSP 0 sub-core/quadrant 1 warp scheduler 32 CUDA Cores 1 Tensor Core 64 KiB regs·16 warp slots SMSP 1 sub-core/quadrant 1 warp scheduler 32 CUDA Cores 1 Tensor Core 64 KiB regs·16 warp slots SMSP 2 sub-core/quadrant 1 warp scheduler 32 CUDA Cores 1 Tensor Core 64 KiB regs·16 warp slots SMSP 3 sub-core/quadrant 1 warp scheduler 32 CUDA Cores 1 Tensor Core 64 KiB regs·16 warp slots Shared by all four: 228 KiB L1 + shared memory (one block can take up to 227 KiB) · TMA engine · port to L2 ⛔ A warp assigned to a quadrant stays there until it exits. This pinning underlies every later “÷ 4”. 4 quadrants = whole SM 128 CUDA Cores 4 Tensor Cores, not 1 65,536 32-bit regs = 256 KiB 64 warp slots = 2,048 threads max 32 blocks ⭐ Reg file 256 KiB: = L1 + shared memory (256 KiB, ≤ 228 shared). Half the priciest SRAM pays for standby threads. How many fit? Three gates; the tightest wins. That is what “occupancy” measures ① Registers: usually the first to bind 65,536 ÷ 2,048 = 32: only 32/thread fills the SM 32 per thread → 2,048 threads 100% occupancy 64 per thread → 1,024 threads 50% occupancy 128 per thread → 512 threads 25% occupancy 255 per thread (max) → 256 threads 12.5% occupancy ⛔ Unroll or cache in regs: fewer standby threads. ② Shared memory 228 KiB per SM; one block can take up to 227 KiB. Your block needs 100 KiB: the SM fits only 2 blocks, spare regs don't help. ⭐ “Can 100 threads hog it all?” This is how. A 227 KiB block is alone on the SM: ≤ 1,024 threads. ③ Slots At most 64 warps, at most 32 blocks. Tiny blocks lose too: at 32 threads per block, 32 blocks × 1 warp = 32 warps, only 32 of 64 filled. No limit hit; still unfilled. ⚡ So why not make a warp 128 wide and skip grouping four? Restated: 32 is one quadrant's width; 128 is one warp from each of four. So “warp = 128” means “merge four quadrants.” ① Merged: one warp scheduler. Today four quadrants run four unrelated instruction streams at once; that's lost. ② Most instructions (memory, integer, branch, transcendental) don't need 128 wide; only wgmma does. Widen, and all code pays for matmul. ③ 32 × 4 B = 128 B = one cache line. Divergence cost grows with width: one split if, whole group runs both paths. ⭐ Warp switches cost nothing, and this figure is the bill A CPU saves/restores regs; a GPU never moves them. All 2,048 threads' regs stay put; ready ones issue. The price: that 256 KiB. Biggest cost of “arrival known only at runtime”: not the scheduler, but regs for the standby crew.
Four quadrants, three answers: 32 is one quadrant's width, and 128 is one warp from each; occupancy is set by the tightest of three gates, usually registers: 65,536 ÷ 2,048 = 32 per thread to fill the SM; and warp switches are free because the 2,048 register sets never move. That last is the runtime-adaptation bill's priciest line: 256 KiB, more than the adjacent L1 + shared memory.
What a thread owns, what it shares (expand)
⭐ What a thread owns, what it shares It owns none of the compute units.
CUDA and Tensor Cores belong to no one; they're pipeline slots. This cycle's issuing warp uses them; next cycle, another.
Tensor Cores more so: a whole warp (four from Hopper on) issues one instruction; “a thread's Tensor Core” doesn't exist.

It truly owns its registers, plus its batch's share of shared memory.
A batch claims them on landing, returns them when done, never moves them between.

⭐ This explains two things.
Registers never move, so swapping warps is free: the old warp's state stays put.
Owned outright, registers always limit you first, not compute units.
Counting registers: units, limits, why multiples of 8 (reference · skip in class)

A register is 32 bits (4 bytes), the unit of the compiler's “registers per thread.” Check: 65,536 × 4 B = 256 KiB, the register file in the figure.

Per-thread cap: 255 on current GPUs (it has changed; early parts had 63), the figure's “255 per thread (max)” row. ⚠️ NVIDIA disagrees with itself: the Programming Guide says 255, CUDA's own cuda_occupancy.h says 256 from Volta on. I couldn't confirm the hardware truth (likely “256 allocatable, 255 usable,” my guess). Use 255.
Registers are allocated per warp, in units of 256, unchanged from Kepler to Blackwell: take “registers per thread × 32,” round up to a multiple of 256.
Per thread: round up to a multiple of 8. So shaving a register or two often changes nothing; you must cross a multiple of 8. Source: code, not docs; cuda_occupancy.h lists these per compute capability.

64-bit values take two registers: doubles, pointers. Extra pointers cut occupancy fast.

Last, the bottom row.

A warp is 32 threads. Each has an index in the warp, 0 to 31. That position is a lane.

So the GPU has lanes too. In hardware a lane is a SIMD channel, same as a TPU lane.

The only difference: GPU lanes have names; TPU lanes don't.

A GPU lane has a writable index. Give it its own if and loop, like a real thread.
A TPU lane is out of reach: you can't write “lane 37, do something else”.
The vector moves as one; the compiler maps elements to lanes.

So a CUDA “thread” isn't your usual thread. To pin it down, set it between its two neighbors.

Three “threads”: the middle one codes like the left, is built like the right CUDA threads borrow the OS thread's name and style, but are SIMD lanes underneath. Put both outer columns up and the oddity shows OS thread CUDA thread TPU counterpart Own fetch / issue? Yes, independent Own PC (since Volta), but issues as a warp so it can't choose its next instruction No. One slot of a vector instruction Has a “name”? Yes: pthread_self() Yes: threadIdx. The GPU's most unusual trait Write “thread 37, do something else,” and it does No. You can't write “lane 37, do X” —only whole-block operations Switch cost? Save/restore regs: microseconds Zero. Regs never move Price: 256 KiB always held; see “Inside one SM” No “switching”; order fixed at compile time Width per op? Your code One 32-bit lane. Loads vectorize to 128-bit; fp16 2/cycle Base unit is one cell, as expected One cell of the 8 sublane × 128 lane grid Who schedules it? OS scheduler, preemptible Warp scheduler, at runtime: whoever's operands arrive Compiler, static; no scheduler Count? 10s–100s 2,048 per SM; 148 SMs ≈ 300K One vector instruction: 1,024 cells ⭐ In one line: a CUDA thread is a “SIMD lane with a name” Written like an OS thread: scalar code, ifs, loops all work. But it has no fetch of its own; 32 obey one command. The name is the dividing line. GPU exposes lanes, paying with schedulers, scoreboards, regs for the standby crew; the TPU hides lanes behind the compiler and drops all that, but you can't command a single lane. “Why six levels?” is the other half of this story.
It has no instruction fetch of its own; 32 obey one command, like a TPU lane. But it has a name: “thread 37, do something else” works, like an OS thread. The name is the dividing line: the GPU exposes lanes, paying for schedulers, scoreboards and the last figure's standby register file; the TPU hides lanes behind the compiler, drops all that, and gives up commanding any single lane.

3.3a What one lane does in a real matmul

“A thread is a lane” is abstract. Ground it in a real matmul, one era at a time; mixing them confuses.

① Classic, no Tensor Cores. A thread owns one output cell (or a small tile). It runs its own K loop: loads A and B slices from shared memory into its own registers, multiply-accumulates, holds the sum, writes it back. It moves data and computes: most people's picture.

② Modern, with Tensor Cores: that picture is void. The matrix instruction is one instruction issued by the whole warp, on a fragment: one tile's elements, spread over 32 threads' registers in a fixed, PTX-defined layout.
In a typical m16n8k16, each thread holds 8 fp16 of A, 4 fp16 of B and 4 fp32 of accumulator C.
⭐ No thread computes a full output element alone. Results return as a scattered fragment.

A whole warp issues each Tensor Core instruction, so “which cell does this thread own?” is meaningless.
A thread is just a register holder and 1/32 of an issue. That's what “a thread is really a lane” means.

⭐ Is it SIMD? Two levels, two answers Hardware: yes. A warp's 32 lanes share fetch, decode and issue and run the same instruction each cycle. That's SIMD hardware.
Programming model: no. You write scalar code per thread, and since Volta each thread has its own program counter and branches.

⭐ So SIMT = scalar code on SIMD hardware. The cost: divergent branches take turns. They look independent but queue. ⚠️ Separate layer: a thread can also use vector instructions (128-bit loads). That's SIMD within a thread, not warp-level.

All this (you write scalar; hardware builds SIMD, queues divergence) is the price of runtime adaptation. The TPU skips that decision, so it lacks even the word “thread”: only lanes, never individually commanded.

⭐ Does a thread move data? Yes, but that role is retiring. Pre-Ampere, threads moved data (global → registers → shared), busy meanwhile; Ampere's cp.async goes straight to shared; with Hopper's TMA one thread issues one descriptor per tile; 31 sit out.
In three generations the thread went from “mover + computer” to “register holder.” §3.3b draws this path (three-generation figure); Blackwell then moves the accumulator out of registers into TMEM.

3.3b The runtime-adaptation bill: three circuits

That takeaway is about the programming model: the shape you see in code.

In hardware, the cost is three circuits, each answering one narrow question:

① Who can issue now?
② Who syncs with whom, and how?
③ How does data arrive with threads free?

On the first, we owe you a definition. “Scoreboard” came up several times (opening the SM, pricing exposed lanes) but was never defined.

In one sentence: a scoreboard is a table of whether each instruction's operands have arrived. Each cycle the scheduler scans it and issues a ready warp.
That's runtime-only knowledge of who can run, in silicon: something must record what isn't ready. The TPU has no such table.

How the scoreboard knows a warp is “ready” (mechanism; open if asked)
⭐ Scoreboard: how a warp counts as “ready” NVIDIA didn't coin the name; it dates to the 1964 CDC 6600.
It first let instructions complete out of order, needing a central table of busy units and pending register writes.
Its official name: Scoreboard.

On the GPU, NVIDIA splits instructions by one test: does the compiler know the latency?
The fixed-latency half needs no tracking: an FMA takes fixed cycles, so the compiler writes “stall N cycles” into the instruction; hardware obeys. Hence the control bits on every SASS instruction.
The variable-latency half is the scoreboard: memory, special functions, matrix ops (latency unknown at compile time). Each warp has six counters (SB0–SB5): a producer bumps one at issue, drops it at writeback; a consumer's 6-bit mask names which must hit zero.

⭐ Not a big table of dirty registers, but six tiny counters plus compiler-planted markers, answering: can this warp issue now? Latency hiding rests on it: without it, no warp swapping.

📌 A diagnostic: Nsight's long / short scoreboard stalls split on whether the awaited data leaves the SM: shared memory, special functions, matrix ops are short; global loads, long. Memory-bound kernels: long dominates.
⚠️ Six counters, control-bit format: from third-party microbenchmarks (Analyzing Modern NVIDIA GPU cores, 2025), not official.
Why “score”? And why this isn't the CDC's board (etymology, distinctions · skip in class)

Sourced vs. inferred. Sourced: the term comes from the CDC 6600, whose component was named Scoreboard. I found no source for why the word was chosen; what follows is speculation.

To keep score means not just tallying points but tracking the current state: a scoreboard at a ballpark lets you glance up once and know the situation, as this table does. So read it as “status board,” not points tally. On the GPU, though, the name fits literally: it counts, up on issue and down on writeback; the old board just raised a “busy” flag.

More importantly, the boards differ. The CDC one was a central table tracking hazards on all registers, even write-after-write (WAW: an earlier write landing late over a later one), all checked at runtime.
NVIDIA's has six counters per warp, asking only “is what I'm waiting for back?”; fixed-latency dependencies skip it; the compiler settles those in the control bits.
The name was inherited; the mechanism shrank. A footnote to the course's thread: what compile time can decide shouldn't wait for runtime, and even the GPU is moving that way.

The second: a few hardware barriers in the SM. Warps arrive, hardware counts heads; when full, all go.
Mechanism folded at the end; the practical point:

A warp waiting at a barrier keeps its slot. So all that waiting frees no slots: a sync-heavy kernel's occupancy doesn't improve.

The third gets a figure. Watch which column changes, which never do.

Into shared memory: GPU took four generations; TPU started at the finish Watch: does data pass through regs? And who computes addr/stride/bounds? As these change, thread goes from mover to dispatcher Staging: all the difference Who computes addresses Issuing thread then does Always there plain ld / st global regs data passes here shared Tensor Core thread computes, runs loop blocked till data lands Ampere+ cp.async (SASS: LDGSTS) global direct skips regs shared Tensor Core still the thread skips the reg hop, still runs addr math, loop Hopper+ TMA + copy descriptor global TMA unit by dims + coords shared Tensor Core HW: stride, offset and bounds 1 thread issues; engine reports to barrier Blackwell+ tcgen05 + matrix storage shared TMEM 256 KiB/SM operands + accum here TMEM Tensor Core still HW; and now results skip reg file one thread for whole CTA; explicit fetch-back TPU: from day one compiler-issued DMA HBM DMA unit same kind as row above VMEM MXU fixed at compile time N/A on TPU: there is no “thread” role ⭐ Top to bottom: movement goes from “a load's side effect” to “a descriptor-driven engine” GPU took four generations to get here; TPU began here. Not copying: one constraint, one answer: faster matrix units → feeding can't tie up compute. ⛔ Not “GPU moving to compile time”: descriptors are built at runtime. How data moves converged (tiles, descriptor, no thread); who decides didn't: runtime thread on GPU, compiled step on TPU. Main theme. “Load side effect or standalone DMA?” already appeared in GPU Microscope G-6; here is how it evolved step by step. ⚠️ No L2 in the figure, on purpose: it's transparent on this path Each global access passes L2 (126 MB, 4 partitions, outside SMs, shared), but is never a destination: every row ends in shared memory / VMEM. ⛔ Shared mem ≠ L2: shared mem is in the SM, same 228 KiB as L1; L2 is outside, ~550× bigger. TMA is in the SM too (1 per SM since Hopper), targeting shared mem. ⚠️ New producer, new sync method Row 3's engine isn't a thread: it never “arrives,” only “deposits bytes.” __syncthreads() counts heads: useless. So Hopper's barrier also counts bytes: it opens only when all have arrived and the bytes are in. Hence “async transaction barrier.” 📌 All official: cp.async direct path: CUDA Programming Guide; single-thread TMA issue, HW address gen and byte-counting async tx barriers: Hopper arch blog, verbatim. Blackwell: tcgen05 issued by one thread for the CTA; TMEM 256 KiB/SM, software-managed in MMA scope. Per PTX ISA + third-party microbenchmarks; no extra inference.
⭐ In rows 1–3 both ends stay put; only the staging column changes (rows 4–5, Blackwell and TPU, change ends too). Row 1: data via registers; the thread runs addresses and loop. cp.async drops the register hop; the thread still addresses. TMA takes addressing: one thread fills a descriptor, issues, moves on. ⭐ Now it's the TPU's compiler-issued DMA in kind; one gap left: issued at runtime by a thread, or fixed at compile time?
The other two answers: block sync, and how a cluster reads another SM (reference · skip in class)

1. Syncing within a block. Behind __syncthreads() is the SM's small pool of hardware barriers above. Three easily missed points: ① it counts warps, not threads; a warp moves as one; ② it doubles as a memory fence: shared-memory writes before it are visible to all after it, no separate fence; ③ a warp waiting at the barrier still holds its slot. (Already in the main text.)

Ampere added asynchronous barriers, splitting “arrive” from “wait.” A warp says “arrived” without blocking, works on, and waits only when it needs data. Early arrivals don't idle. Hopper lets waiters truly sleep; before, they spun on shared memory.

The two barriers also live in different places. Old bar / barrier: a small fixed SM pool, capped per CTA; mbarrier: a shared-memory object, as many as shared memory holds. Per the PTX docs.

2. How a cluster reads another SM's shared memory. Three things must hold: ① a cluster's blocks are guaranteed co-scheduled by hardware; ② each GPC has a dedicated SM-to-SM network; a cluster must fit in one GPC because that network stops at its edge; ③ cluster.map_shared_rank(ptr, target_rank) gives the other block's address (PTX mapa); plain loads/stores then use it, and the network routes.

⚠️ How near matters more H100 microbenchmarks: own shared memory, 29 cycles; own via the DSM interface, 33 (the extra 4 are the interface; nothing leaves the SM); another SM, 181, ~30% below L2; a global-memory round trip (store + load), 956.

So it does not “merge several SMs' shared memory into one big pool”: 181 cycles is over six times the local 29. A shortcut cheaper than L2: use it to save those 900-odd cycles, not as if it were local.
⭐ Three circuits, one bill Scoreboard: “who can issue now.” Barrier: “who syncs with whom, and how.” TMA: “how data arrives without tying up threads.” Together: the itemized bill for deciding at runtime.

The TPU needs none. Not because it's primitive: it answers all three at compile time. §3.2's five “missing components,” revisited: what's missing is this bill, not capability.

3.3c ⭐ In what order does hardware run a matmul?

Both sides are taken apart. Still unsaid, and more basic: in what order does hardware compute a matmul?

This may be the course's one spot where nearly everyone is wrong without noticing. The wrong version is too natural: it's how you do a matmul on paper.

The picture in your head: left row 1 times right column 1, summed pairwise, gives the top-left entry; that cell is done; next. Mathematically correct. Yet no real hardware, GPU or TPU, works this way.

One matmul, three ways onto silicon: same result, completely different dataflow ⭐ Most people picture a matmul as “finish one dot product first”; no hardware does that. The difference, on one small example: A is 2×3, B is 3×2, C is 2×2 (M=2, N=2, K=3). ① Naive (inner-product order): the one in your head. One accumulator, but almost no reuse In code: three nested loops, k innermost. Finish one cell of C, then the next. A (2×3) a₀₀ a₀₁ a₀₂ a₁₀ a₁₁ a₁₂ × B (3×2) b₀₀ b₀₁ b₁₀ b₁₁ b₂₀ b₂₁ = C (2×2) c₀₀ c₀₁ c₁₀ c₁₁ Step 1: A's whole row 0 (3 values) × B's whole column 0 (3 values), MAC: one cell c₀₀, written out. Step 2: same A row 0, now B column 1 → c₀₁. ⛔ A row 0's three values: read again. Steps 3–4 same: A row 1 also read twice. Count the cost 6 values of A read twice each = 12 reads; 6 of B twice each = 12. 24 reads in total, for only 12 MACs. ⛔ Reuse 0.5: two reads per MAC. No real hardware does this. (Just 1 accumulator; operands move over and over.) ⭐ What trips people up: “finish a cell, retire it” is so natural no one questions it. ② GPU (outer-product order): k moves outermost; all M×N cells grow together Each k step reads only A column k (M values) + B row k (N values), then pairs them all, updating all 4 cells at once. k = 0 b₀₀ b₀₁ B row 0 a₀₀ a₁₀ A col 0 +1 +1 +1 +1 accumulator all 4 update read 4 values → 4 MACs → 4 cells +1/3 each k = 1 b₁₀ b₁₁ B row 1 a₀₁ a₁₁ A col 1 +1 +1 +1 +1 accumulator all 4 update read 4 values → 4 MACs → 4 cells +1/3 each k = 2 b₂₀ b₂₁ B row 2 a₀₂ a₁₂ A col 2 +1 +1 +1 +1 accumulator all 4 update read 4 values → 4 MACs → 4 cells +1/3 each ⭐ Three steps, K done: all four cells finish together, drain at once. 12 reads, 12 MACs, reuse 1.0, doubled. ⭐ How often is each cell used? Exactly K times, top-left and bottom-right alike. Each k step updates every cell once: accumulator updates = M×N×K = MACs. ⚠️ This is the algorithm-level dataflow. Per instruction, one MMA already reduces a short K run (16/32/64) inside: same shape, batched. ③ TPU MXU (systolic array): the K sum needs no accumulator; it “flows down the array” B loads once, stays put (weight-stationary). A rows stream in from the left; partial sums add downward, exit as a finished C row. A rows (one at a time: time) a₀₀ a₀₁ a₀₂ a₁₀ a₁₁ a₁₂ t = 0 sends row 0, t = 1 sends row 1 MXU array: 256×256 on real HW K (3) = rows, N (2) = cols: both spatial b₀₀ b₀₁ b₁₀ b₁₁ b₂₀ b₂₁ a C row exits below Each PE's job ① take a from the left ② multiply by its b ③ add partial sum from above ④ pass a right, new partial sum down. That's all. ⭐ So the 3-term sum along K is done “in space”: three PEs each do one MAC, relaying partial sums down. No external accumulator. External accumulator only across K tiles: K often 1,000s; array 256; rest tiled ⭐ M runs in time: A rows in and out one by one. ⭐ Same reuse math: on TPU, B loaded once serves M rows; each A value serves the N PEs in its row. GPU gets it from its 2D accumulator: one equation, two physical forms. ⭐ Takeaway: same math; only which dims go to space and which to time ① Naive (inner) ② GPU (outer) ③ TPU (systolic) M (A rows) time space: accumulator rows time: rows stream in N (B cols) time space: accumulator cols space: PE cols K (summed) time time: K steps space: PE rows Accumulator site 1 reg, over and over M×N cells, each K times down the array, in flight Accumulator size 1 M×N (really one tile, not whole matrix) only cross-K-tile part Reuse 0.5 M×N ÷ (M+N): a 128×256 tile ≈ 85 B used M×, A N×; same math ⭐ In one line: GPU puts M, N in space, K in time; TPU puts K, N in space, M in time. That drives the rest: accumulator size, why tile is the key knob, why TPU keeps weights stationary. ⛔ All three give the identical C. Same result, different dataflow: the habit to build. Specs list results; speed is in the dataflow.
⭐ Exactly the same C, completely different dataflows. ① Naive (inner product): one cell at a time; 1 accumulator, but 24 reads for 12 units of work, reuse 0.5. ② GPU (outer product): k is the outer loop; each step reads a column of A + a row of B, all M×N cells advance together, each updated K times. ③ TPU (systolic array): B loads and stays; A streams through by rows, and the K sum happens in space, no external accumulator. ⭐ Final table: GPU puts M, N in space and K in time; TPU puts K, N in space and M in time.
⛔ Wrong dataflow, not wrong math; spec sheets list only math Inner- and outer-product order give identical results, to the last digit. No wrong answer ever exposes a wrong mental model.
They differ in how much data moves, the half that sets speed.
Inner: 24 reads, 12 MACs. Outer: 12 reads, 12 MACs. That's a 2×2 toy; big tiles make it tens to hundreds of times.
⭐ The equation to take away: accumulators buy reuse Why not just one row of accumulators? One clean equation answers.

For an M×N accumulator tile, each step along K reads one column of A (M values) + one row of B (N values): M + N values. That buys M × N MACs.

⭐ So the reuse factor = M × N ÷ (M + N).
ShapeReuseWork/read
1 × N (one row only)N ÷ (1+N)≈ 1, even at N = 10⁶
128 × 12816,384 ÷ 25664
128 × 25632,768 ÷ 384≈ 85
⭐ The harsh part: it's capped by the short side. Short side 1, long side a million: reuse is still 1. The long side doesn't help.
So an accumulator array doesn't “hold the result”; it buys reuse. Each extra accumulator pairs already-moved operands once more. ⭐ This is Section 1's arithmetic-intensity thread, first cast in silicon.
⭐ The circuit view also demands 2D The MXU is a 256×256 2D array that physically emits 256 partial sums every cycle. One row of accumulators would use 1/256 of the array.
A 2D array needs a 2D accumulator, shaped like the array, not an arbitrary choice.

Follow-up: isn't draining costly too? No. Any tiling drains exactly M×N. The real overpayment is B: row by row, you re-read all of B M times.
⚠️ Two accuracy caveats to keep when presenting ① The GPU column is algorithm-level dataflow. One MMA instruction already reduces a short run of K (16/32/64) inside: same shape, batched. Cross-k-tile accumulation is explicit: the first tile zeroes the accumulator; later tiles add.
② In the TPU column the K sum completes spatially inside the array, needing no external accumulator; external ones serve only across K tiles: real K is often thousands, and the array takes 256 at a time.

⭐ In one sentence: same math; only which dims go to space and which to time differs. It drives everything: accumulator size, the tile as core knob, why the TPU must keep weights still. §3.4 next compares how big a tile one instruction eats: this choice in numbers.

3.4 One instruction's bite: a 16× gap in contraction depth

§3.3 asked who issues the instruction. Now: how big a block does one instruction eat?

An internal detail that decides which head_dim is free.

Tensor Core bite per instruction—5 generations, true-scale overlay From the PTX ISA warp (32 threads) warpgroup (128 threads) SM pair (256 threads) TPU MXU, same ruler 5 gens: threads/MMA change; K=16 since Ampere Volta Gen 1 8 threads per MMA mma.sync.m8n8k4—one quad-pair (8 lanes) forms one MMA PTX note: m8n8k4 requires sm_70, optimized for sm_70, much slower elsewhere Operands in Regs 8 lanes Turing Gen 2 32 threads = 1 warp per MMA mma.sync.m16n8k8—warp shapes enter PTX here Same page: u8/s8 m8n8k16, u4/s4 m8n8k32, b1 m8n8k128 all need sm_75—integer and sub-byte precisions begin with Turing Operands in Regs 32 lanes Ampere Gen 3 32 threads = 1 warp per MMA mma.sync.aligned.m16n8k16—K 8→16; adds BF16, TF32 PTX note: m16n8k16 requires sm_80. Still in use today Operands in Regs 32 lanes Hopper Gen 4 128 threads = 1 warpgroup per MMA wgmma.mma_async.m64nNk16, N from 8 to 256 NVIDIA's words: Warp-group MMA across 128 threads. Hopper only (needs sm_90a) Operands in A: regs/shared mem; B: shared mem 128 lanes Blackwell Gen 5 SM pair · 1-thread issue per MMA tcgen05.mma · one SM: M = 64/128, N = 8,16,…256 (step 8) · SM pair: M = 128/256, N = 16,32,…256 (step 16) · K is still only 16 Paired: M, N steps both double—coarser grain, not a bigger op. ⚠️ Trend reverses: gens 1–4 grew the thread count; tcgen05: 1 thread issues (cta_group::2 = 2 CTAs' data, not 256 issuers); operands in TMEM—operand scope, not threads Operands in Operands + accumulator in TMEM, not regs 256 lanes One ruler—big square = one TPU MXU (256×256); inside: output tiles of three GPU instructions 1.17 px/element; tiles and square share one true scale. Shared top-left corner, so nested, not side by side. One TPU v7 MXU 256 × 256 65,536 MAC units, fed by one instr. tcgen05 m128 n256 wgmma m64 n256 tcgen05.mma M×N = 128 × 256 Output tile: 32,768 positions = 1/2 of TPU's 256×256 Blackwell's largest. Paired SMs take M to 256: then it is exactly half the square's height and already full width. wgmma M×N = 64 × 256 Output tile: 16,384 positions = 1/4 of TPU's 256×256 Hopper's warpgroup instruction. Same full width as above, half the height—the whole difference is M. Hopper only; Blackwell replaced it with tcgen05. mma.sync M×N = 16 × 8 Output tile: 128 positions = 1/512 of TPU's 256×256 Here it is just 9 × 19 px—the small circled blue dot top left. The warp-level instruction kept since Turing survives on Blackwell, and on the consumer die block-scaled quant only on it (Microscope G-5). Key dimension, contraction K: tiles above are outputs; K sets bite depth M and N set the output tile size; K sets the length of one MAC chain. On GPUs it went from 8 (Turing) to 16 (Ampere) and has not changed since: fp16/bf16 K=16, fp8 K=32, fp4 K=64—all 256 bits, i.e. the same eight 32-bit words. GPU Tensor Core K = 16 ← this narrow TPU MXU contraction side = 256—same ruler, 16× The 16× isn't “GPU can only go 16 deep”. Chained K=16 ops accumulate in TMEM, so depth adds up. The gap: per-op granularity, not a hard limit. Each granularity has a cost. Hence in attention head_dim=128 hurts TPU, not GPU: 128 hits the TPU's 256 contraction side, filling half; 128 = 8 × 16, so the GPU uses 8 instructions, zero waste.
Figure G-4 One Tensor Core instruction's bite. The GPU's contraction dimension is 16; the next figure needs it. — from the GPU Microscope
MXU—data crosses the 256×256 array diagonally MAC cell (holds a weight) Activations: left in, 1 col/cycle Partial sums: down to accumulator Gray = not yet reached One matmul in the array—snapshot, cycle 16 Shade = when a cell starts: top-left first, bottom-right last. One anti-diagonal fires in one cycle—that is what “systolic” means. ← cycle-16 front One MXU 256 × 256 1 square = 16 × 16 real cells Activations row-wise Weights: preloaded, stay for whole pass Accumulators 128 · (8, 256) · 32 bit Activations enter left row by row, 1 col/cycle; row r lags row 0 by r cycles. Partial sums flow down into the accumulators. Latency ≠ throughput. Entry to result takes 2N−1 cycles—for 256×256, 511 cycles. Once the array is full, every cycle a full column lands. So bigger matrices win: the 511 fill cycles amortize. Conversely, small matmuls are costly on TPU: for 256×256×256, fill + drain take two-thirds of the time (256 + 511 = 767 cycles, 511 not steady-state). One physical reason for Microscope §7's “compiler schedules ahead”. Peak chain is official 262,144 FLOP / cycle / MXU × 2.2 GHz × 4 MXUs = 2,307 TFLOP/s Google's engineering blog gives all three; product = official peak Per cell: 262,144 ÷ 2 ÷ (256×256) = 2, i.e. 2 MACs per cell per cycle. This is new in this architecture—the 128×128 gens (v3 / v4 / v5e / v5p) all give 1 per cell. How “2” is built in silicon (two multipliers per cell? or 256×256 is a logical view?)—public sources don't say; we won't guess. 256 is granularity, not a ceiling—often confused K > 256 costs nothing. K = 1024 = 4 passes, accumulator never spills—nearly as efficient as one. K < 256 is the real loss. It's 256 physical rows; feed 128 and half of them work—the other half idles, and nothing else can slot in (§3: TPU has no “run another” layer). Compare the GPU: its K is 16; 128 = 8 × 16, so 8 instructions, zero waste. Same model config, waste in completely different places. A real example: attention head_dim = 128 In MHA, the contraction dim of QKᵀ and output dim of PV both equal head_dim. At 128, each fills only half of 256—pure geometry; no inside info needed: cap is 50%. Measured: Qwen3-30B attention from 32 heads × 128 to 16 heads × 256 (params, FLOPs unchanged) raised MFU at 8K / 16K / 32K by 21% / 32% / 46%. Not “TPUs are bad at attention”: co-design model config and contraction width. On GPU, nearly zero gain. Fill: three K, one array K = 256 100% Fills K exactly. Ideal. K = 128 50% Half the rows work; half idle. head_dim=128 is this one. Gray half can't be reused—no second task can fill it. K = 512 100% Two passes, no spill. As full as the first. Pass 1 Pass 2 “Granularity, not a ceiling”: dims over 256 just more passes, no waste.
Figure T-4 The MXU is 256×256: 16× the GPU's contraction dimension. A real head_dim follows. — from the TPU Microscope

In both, look at the instruction's depth along the summed dimension.

GPU: 16. TPU: 256. A 16× gap.

That is the contraction dimension: the one a matmul sums away, i.e. terms accumulated per instruction.

One rule for the letters M/K/N: K alone appears twice and alone disappears (summed away). The other two shape the result.

⭐ The trouble with 128: it falls between two stools, too small for the array's edge (half of 256), too few streamed rows to amortize the weight load.

Why: the letters' origin, why N wastes half too, is transposing equivalent? (open if asked)
⭐ Getting M/K/N straight, once Write a matmul as A[M, K] × B[K, N] = C[M, N]. One rule covers all three letters:
⭐ K is the only one that appears twice, and the only one that disappears.
Twice: A's columns must equal B's rows, so it joins the two matrices. Gone: it is summed away, hence “contraction dimension”.

The other two are the result's shape: M = how many result rows, N = numbers per row. K = terms summed into each number.

⭐ With this you can read instruction names: m16n8k64: output 16×8, each number a sum of 64 terms. (The order follows BLAS: output shape M, N first, then K.)
⭐ Where the three letters come from: a triple loop The most naive matmul is three nested loops, by tradition named i, j, k, top to bottom:
for i (each result row) → for j (each number in it) → for k (add term k)

⭐ The three capitals are simply the bounds of the three lowercase loops. M, N, K = trip counts of i, j, k.
k is the innermost loop: it accumulates. So “K = contraction dimension” isn't convention: it names the accumulation loop.

⚠️ Why i/j/k/m/n: the notation comes from Fortran BLAS, and in Fortran, variables starting with I–N are integers by default, so indices and sizes use those six letters. (The usual explanation, not a standard's text; the Fortran rule itself is fact.) ⛔ Don't invent meanings for the letters (M≠Matrix, N≠Number); that makes them harder to remember.
⛔ The real snag isn't the letters, it's the word “rows” “Number of rows” measures a column's length; “number of columns”, a row's length. “M rows” is vertical. Genuinely counterintuitive.

Three reframings, from good to best:
① Not rows and columns: height and width. A is M tall, K wide; B is K tall, N wide. Multiplying needs “A's width = B's height”; result: “A's height × B's width”. No more “row count is column length”.
② Chain the shapes and watch them link. M K · K N → M N: adjacent letters must match, then both vanish. The rest is the answer.
③ ⭐ The most valuable one: M is time; K and N are space. On a systolic array the K×N block is loaded and held: K is the array's height, N its width, both fixed by hardware. M is just how many rows flow through, as long as you like.
⭐ Hence a pattern in this course: K=128 and N=128 each waste half, yet M never comes up: it takes no space, only time.
⚠️ Why does N=128 also keep only half? Same math, different reason Picture the array as a 256 × 256 grid of cells. For A[M,K] × B[K,N], the hardware lays B's K×N block onto the grid (K high, N wide), then streams A's rows in from the left.

A cell does useful work only if both hold:
① data flows into the row above it;
② someone needs the sum accumulated down its column.

⭐ Each kind of waste violates one:
· K=128 (half the height): the bottom 128 rows have no input above them. Fill them with something else? No: a column's cells add into one number, so anything else corrupts it.
· N=128 (half the width): the right 128 columns have no consumer below. They compute output channels nobody needs: your output has 128, no more.

The arithmetic: utilization = (K ÷ 256) × (N ÷ 256). One side at 128 (half), the other full: 50%. ⚠️ Derived from the array structure, not official, but it agrees with the official line: at most 50% at head_dim 128.

⭐⭐ Most important: the two wastes are not equally recoverable.
· N short: recoverable. If another matmul takes the same A, concatenate both B's along the width; one pass. Fusing the Q, K, V projections into one matmul does exactly this: all three take the same X, so their N dims concatenate.
· K short: not recoverable. K is summed: anything concatenated adds into the same result. The core difference from N.

⚠️ In attention, though, the PV step's N=128 is the unrecoverable kind: that needs matmuls sharing a P, and every head's P differs.
⭐ A false rescue, and two real ways out ⛔ First, the idea everyone has: “let all heads share one V?” MQA/GQA do exactly that. But it points the wrong way.
Concatenating N needs “same A, different B”; shared KV gives you “same B, different A”. That stacks only along M, and M is time, not space: stacking never fills that half.
(MQA/GQA save KV-cache memory and bandwidth, not array utilization here. Don't conflate.)

⭐ The two real ways out:
① Change head_dim. At 256, both wastes vanish. But that is a model architecture decision, not a tuning knob. A model's head_dim caps its attention's share of peak here.
② Compute the matmul transposed. PV hits N=128; compute the transpose Vᵀ · Pᵀ instead and all three letters move: M becomes 128; K and N become the two long dimensions.
⭐ This is “M is time, K and N are space” applied directly: move the 128 from space to time. 128 as array width is waste; as “rows flowing through” it costs nothing.
⚠️ Is the transpose still the same computation? Mathematically yes, but it has two costs
It is an identity: (P · V)ᵀ = Vᵀ · Pᵀ. Exact, not an approximation.
① The output is transposed. Either the consumer takes it, or you flip it back, which costs on a TPU (cross-lane moves, VPU work).
⭐ So you need a fully transposed pipeline, not one transposed step: the whole chain stays transposed, or flipping back and forth eats the savings. (Splash Attention's interface makes the Q/K/V layout a parameter, so the choice is yours by design, not hard-coded.)
② Same math ≠ same bits. Accumulation order changes; last bits may differ. Flag it in numerics alignment, or someone burns half a day on a “bug.”
⚠️ Others report it working (one report: efficiency 30-odd → 50-odd percent); we haven't reproduced it. Our three minimal repros were negative, on a baseline 8× off production, so we can neither confirm nor refute. ⛔ Not a conclusion.
Why does Vᵀ · Pᵀ equal (P · V)ᵀ? Step by step (shape → accumulation → numbers → cost)

Step 1: the shapes match.

  • P is [q, kv], V is [kv, d] → O = P·V is [q, d]
  • Vᵀ is [d, kv], Pᵀ is [kv, q] → Vᵀ·Pᵀ is [d, q]

[q, d] transposed is exactly [d, q]: shapes match; now check the numbers inside.

Step 2: cell by cell, the dot product uses the same two vectors.

Original cell: O[i, j] = P's row i (horizontal, length kv) dotted with V's column j (vertical, length kv).

Transposed cell: (Vᵀ·Pᵀ)[j, i] = Vᵀ's row j dotted with Pᵀ's column i.
By definition, row j of Vᵀ is column j of V, column i of Pᵀ is row i of P.

⭐ Both sides dot the same two vectors, just in swapped order.

Step 3 (the crux): why swapping the order changes nothing.

Matrix multiply isn't commutative (A·B ≠ B·A): swapping changes “whose rows pair with whose columns”. But here we transpose both at once; rows and columns swap too.

⭐ The two swaps cancel; the pairing is unchanged. Only the factor order in each term changes. They're scalars; scalars commute.

In one line:
O[i,j] = Σ P[i,k]·V[k,j] and (Vᵀ·Pᵀ)[j,i] = Σ V[k,j]·P[i,k]
Same products, same sum.

Step 4: numbers, matching term by term.

P = [ 1  2  3 ]        V = [  7   8 ]
    [ 4  5  6 ]            [  9  10 ]
                           [ 11  12 ]

O = P·V, top-left: 1×7 + 2×9 + 3×11 = 7 + 18 + 33 = 58

Vᵀ = [ 7  9 11 ]       Pᵀ = [ 1  4 ]
     [ 8 10 12 ]            [ 2  5 ]
                            [ 3  6 ]

Vᵀ·Pᵀ, top-left:   7×1 + 9×2 + 11×3 = 7 + 18 + 33 = 58

⭐ Not a coincidence of totals: the terms 7 + 18 + 33 match one by one. Overall, O = [[58,64],[139,154]], and Vᵀ·Pᵀ = [[58,139],[64,154]], exactly the transpose of O.

Step 5: the amount of compute is identical.

Original: q × d outputs, each kv MACs → q·d·kv.
Transposed: d × q outputs, each still kv MACs → d·q·kv.
Not one more or less.

⛔ Then why is it faster? What changes is not the number of MACs but how they are laid out on the 256×256 grid.

  • Original: M=q, K=kv, N=d=128 → only half the array's width is used
  • Transposed: M=d=128, K=kv, N=q → height and width fill; 128 moved to M

⭐ Same MACs, different layout, fewer idle cells. That is all “move the 128 from space to time” means.

⛔ But the transpose isn't free: it swaps costs “M can be as long as you like” covers only the space cost. A large M takes no array space, but a small M has its own cost.

A systolic array doesn't produce results instantly: data flows left to right, partial sums top to bottom; fill + drain take on the order of the array's side length.
Load weights once, stream just 128 rows: head and tail can't be amortized. Negligible over thousands of rows; over 128 it is the main cost. (⚠️ Steady state is full; only head and tail aren't. So: not “never full” but “full too briefly”.)

⚠️ A natural idea that doesn't work: “Then why not stream several heads' P back to back?”
After the transpose, what streams is Vᵀ, and what gets loaded is Pᵀ. But each head's Vᵀ must pair with its own Pᵀ: M can't be concatenated either, for the same reason N couldn't.

⭐ What the transpose really does is trade “half the array idle” for “poor pipeline amortization”. Whether it pays depends on which cost is larger, which paper can't settle.
⚠️ This may be why our three repro rounds came out negative, but that is speculation; this term wasn't measured separately. Not established.
One level deeper: how does the right matrix enter the array? Can both stream? (mechanism)

① Weights are shifted in; filling one tile takes 256 cycles.

Not stamped in at once: weights are pushed down row by row from a FIFO, rolling over the array like a shutter. The first-generation TPU paper: it takes 256 cycles to shift a tile in.

② Must the left matrix wait for the load? Usually not: double buffering.

Each cell has two sets of weight registers: one computes while the other loads the next tile. So the 256 cycles hide behind the previous tile's compute; the paper: “double-buffering (to hide the 256 cycles it takes to shift a tile in)”.

⛔ But hiding has a precondition, and 128 breaks it The condition: computing this tile must outlast loading the next. Loading ≈ 256 cycles; streaming M rows ≈ M cycles.
⚠️ Why does “compute” take only M cycles? Once the array is full, the left matrix advances one row per cycle and a column of results lands each cycle: M rows, M cycles, however many MACs. All 65,536 cells do their MACs simultaneously, at no extra time.

⭐ So at M = 128, compute takes 128 cycles, loading 256. Loading is slower; double buffering can't hide it. You stall on weights.
⚠️ This combines two public numbers (“256 cycles to load a tile” + “B rows take B pipeline cycles”); not an official conclusion, but it makes “a small M has its own cost” computable.

③ Once loaded, how is it “held”? Each cell has one weight register; a value stays put until the whole tile is done. The name: weight-stationary.

④ Can both matrices stream? Yes: one of the three classic dataflows.

They differ only in what stays put in the cells:

  • Weight-stationary (WS): cells hold a block of B, so the array holds K × N; M streams through. The TPU does this.
  • Output-stationary (OS): both matrices stream, A from left, B from top; results build up in place. The array holds M × N; K streams through. Your question.
  • Input-stationary (IS): cells hold a block of A, so the array holds M × K; N streams.

Output-stationary's advantage: no “load weights” step; both operands stream in together. The cost: results pile up in the cells and need a final pass to drain.

⭐⭐ Side by side, the three dataflows expose this section's real problem The three dataflows do one thing: choose which dimension is time.
· Weight-stationary: space is K and N, time is M
· Output-stationary: space is M and N, time is K
· Input-stationary: space is M and K, time is N

head_dim = 128 falls between two stools.
· As a space dimension: it is too small (only half of 256; the other half idles)
· As a time dimension: it is too short (128 cycles can't amortize a 256-cycle load)

⭐ So no dataflow saves it; only moving it does, hence “transpose the matmul”, not “use a different array”. Transposed, though, 128 lands on time and hits the “can't hide” problem above.
⚠️ Self-consistent but not measured: a way to understand it, not a verdict.

One figure, four bands: stationarity, tiling a real size, ping-pong loading, segments vs. reloads.

One matmul on the MXU—who stays, who flows, when weights load A real size: Q projection M = 128K, K = 7168, N = 1536, array 256 × 256 ① One difference: what stays put IS (A stays) Cells hold A A held Flows in: B from top Out: C out space = M × K time = N WS (B stays) ★ TPU Cells hold B B held Flows in: A from left Out: C sums ↓ space = K × N time = M OS (C stays) Cells hold C C held Flows in: A left + B top Out: then drain space = M × N time = K ⭐ All three just pick which dim is time: small space dims → idle cells; short time dim → unamortized loads. Below: WS (TPU's). ② Tiling: right → 168 tiles, left → 64 segments Right W = [7168, 1536] bf16, 21 MiB — all fits in VMEM K: 7168 ÷ 256 = 28 N: 1536 ÷ 256 = 6 168 tiles each 256×256 Left X = [131072, 7168] M: 2048-row segments, 64 total … 64 segments 2048 rows × 256 (K slice) Partial sums: VMEM 2048 × 256 × 4 B = 2 MiB ⛔ Why not stream M whole Per n-tile, sums need 131072 × 256 × 4 B = 128 MiB — more than VMEM (64 MiB). So: segment M. Then B must reload. ⚠️ Not bad code; inherent. ③ Ping-pong loading: 2 weight sets, one computes, one loads Compute Loading ⛔ Only tile 1 waits (256 cycles); rest hidden B0 idle B0 (2048 cycles) B1 256 cycles B1 (2048 cycles) B2 256 cycles B2 (2048 cycles) B3 256 cycles B3 (2048 cycles) B4 256 cycles B4 (2048 cycles) B5 256 cycles B5 (2048 cycles) …… ⭐ Weight-tile load: 256 cycles (TPU v1 paper); streaming 2048 rows takes 2048: the 256 hide inside the 2048. ⚠️ So the condition: rows/segment > 256. 2048 is 8×; safe. (Derived from two public numbers, not official.) ⛔ Transposed attention: M = 128, under 256, so ping-pong can't hide it. ④ Finer segments → more B reloads—length capped by the sum buffer Rows/segment Partial sums Total B loads Hidden? 256 0.25 MiB 86,016 ⛔ No (not > 256) 512 0.5 MiB 43,008 ✓ 2048 2 MiB 10,752 ✓ 8192 8 MiB 2,688 ✓ ⭐ Squeezed both ways longer → fewer reloads longer → bigger on-chip accumulator So on-chip space caps rows/segment. — same wall the GPU hit in 3.6.
⭐ From “what stays put” to “how many reloads”: ① schemes differ only in what stays put, fixing 2 spatial dims; ② right matrix → 28 × 6 = 168 tiles, left → 64 segments of 2048 rows; ③ ping-pong loading: a 256-cycle load hides in 2048 compute cycles, only tile 1 waits; ④ finer segments, more right-matrix reloads; on-chip accumulator caps segment length. ⛔ Bottom left: transposed attention's M=128 is under even the 256 bar.
A real size, start to finish: how the Q projection runs on the MXU (M=128K, K=7168, N=1536)

The three numbers. Left X: [131072, 7168]; right W: [7168, 1536]; result Q: [131072, 1536].

① Block size? No choice: 256 × 256. That is the array's size; the right matrix is cut to it.

② How many tiles in total?

  • K direction: 7168 ÷ 256 = 28
  • N direction: 1536 ÷ 256 = 6
  • In total 28 × 6 = 168 tiles.
⚠️ Two kinds of “moving”: the most common math error · HBM to on-chip: the whole W is only 7168 × 1536 × 2 B = 21 MiB, so it all fits in VMEM in one go (VMEM has 64 MiB). Once.
· On-chip, shifted into the array: 256 cycles every time a tile is used. Over ten thousand times.
When someone says “moving B”, ask which, or you'll be off by orders of magnitude.

③ The timeline. Feed the left matrix in 2048-row segments:

load B(k=0, n=0)       ← 256 cycles; 1st tile waits
      ↓
stream 2048 rows of X  ← 2048 cycles
  (meanwhile load B(k=1,n=0)) ← 256, hidden in 2048
      ↓
stream 2048 rows of X  ← using B(k=1,n=0)
  (meanwhile load B(k=2,n=0))
      ↓
      … after 28 passes …
      ↓
2048 rows × 256 out cols fully accumulated → write
      ↓
n=1: 28 more passes … all six n done → 2048 rows done
      ↓
next 2048-row segment → all 168 B tiles again

④ Does streaming wait for the load? Only on the first tile. Then double buffering overlaps loading the next tile with computing this one. Hiding needs “rows streamed > 256”: here 2048, 8× headroom.

⑤ Is each B tile used only once? No. It is reloaded once per left-matrix segment.

Segment rowsAccumulator B loadsHidden?
2560.25 MiB86,016No (256 ≤ 256)
5120.5 MiB43,008Yes
20482 MiB10,752Yes
81928 MiB2,688Yes

⭐ Why left-matrix segments can't be short: longer segments, fewer B reloads, but the accumulator stays on-chip and grows with them. So on-chip capacity caps segment length: the stovetop again.

⑥ Then stream all of M in one go, loading B only 168 times? No: the accumulator would need 131072 × 256 × 4 B = 128 MiB, more than all of VMEM. So M must be segmented, and segmenting reloads B. Not a weak implementation, just arithmetic.

⭐ This example also proves that a small M costs you Here M has 130K rows; any segment is far over 256, so loading hides. Projection-type matmuls never have this problem.
Transposed attention has M = 128, below even the 256 threshold. Same mechanism, opposite outcomes.

Plug in a real shape to feel the two numbers.

Attention's head_dim is commonly 128.

GPU: 128 ÷ 16 = 8: eight instructions, zero waste.
TPU: 128 against 256: half filled, half empty.

Why not fill the empty half with other work?

You can't. The contraction dimension is an accumulation chain: anything you add lands in the same partial sum.

Why doesn't the GPU pay this?

Not because it has a scheduler. Swapping warps fills time bubbles, not a half-empty contraction dimension.
Its K is only 16; nearly every shape is a multiple of 16.

Price of a K this narrow, aligned at runtime: the five parts §3.2 found the TPU lacks.

⚠️ Top trap: 16 and 256 are instruction shapes, not work per cycle. Divide them and the peak comes out absurd.

Reconciling the peaks; three math traps (to check the numbers)
⛔ The easiest trap: K=16 is one instruction's shape, not one cycle's work Common reasoning: “an instruction is 16 deep, an SM has one Tensor Core: 2,250 TFLOPS can't be.”
Both steps fail: wrong units; instruction shape taken for per-cycle throughput. Details below.
Check: 4,096 × 2 FLOP × 148 SM × 1.83 GHz = 2,219 TFLOPS vs. official 2,250; the 1.4% gap: clock uncertainty, not the formula.

⭐ Instruction shape sets work per instruction (control-overhead amortization); MACs per cycle is set by array size. Don't divide them.
⭐ The TPU peak chain, reconciled the same way The TPU's peak chain is stated verbatim by Google:
262,144 FLOP/cycle/MXU × 2.2 GHz × 4 MXUs = 2,307 TFLOPS
Official: 2,307, exact.
⭐ This also closes the hook from §3.1. The 131,072 in that figure is a cell count (two MXUs); the 262,144 here is FLOP per MXU per cycle. The units differ, so don't compare them directly.
Divide and it all fits: 262,144 FLOP ÷ 65,536 cells = 4 FLOP per cell per cycle, i.e. 2 MACs: one MAC = one multiply + one add = 2 FLOP, and two make 4.
That is the step §3.1 said “isn't in the figure”. Both sides multiply from MACs per cycle up to the published peak, so every number here stands.

⭐ How “592 small units” and “4 large ones” add up is §3.7's job; here, remember one pair: contraction dimension 16 vs. 256. Keep the two apart and §3.6's different tile sizes become obvious.
⚠️ A cross-generation warning Half of one TPU MXU's 131,072 MACs/cycle comes from “2 MACs per cell per cycle”, which only the 256×256 generations have; the older 128×128 ones (v3 / v4 / v5e / v5p) all do 1 per cell. Carry this constant across generations and you're off by 2×.
Three more plausible-sounding traps are folded below.
Where the wrong derivation fails, plus this model's three traps (reference · skip in class)
※ Common wrong derivation: two bad steps (moved from main text) ① It counts the wrong units. An SM has 4 Tensor Cores (one per sub-partition), not 1; CUDA cores number 128, not 4,096. The full chip has 592 Tensor Cores and 18,944 CUDA cores. 4,096 does exist, but as MACs per SM per cycle (4 Tensor Cores × 1,024), not a core count: familiar value, different units.
② It mistakes instruction shape for per-cycle throughput. mma.sync's m16n8k16 totals 16 × 8 × 16 = 2,048 MACs; one Tensor Core does 1,024 per cycle, so this instruction occupies it for 2 cycles. A package of work, not one cycle's. The largest, tcgen05.mma (128×256×16), is 524,288 MACs; at 4,096 per SM per cycle → 128 cycles; 64 with an SM pair.
⚠️ Three traps, each of which “sounds perfectly reasonable” ① That 16 is not the width of a GPU array; it is the instruction's contraction depth. A Tensor Core's 1,024 MACs/cycle is certain; their physical layout isn't public, so you can't square 16 into an “area”: that mixes two different units in one formula.
② Nor is 128 the square of 16 (that's 256). 128 = 131,072 ÷ 1,024 (MACs per cycle of one MXU vs. one Tensor Core); a ratio of MACs per cycle, unrelated to the array's side length.
③ The SM count is 148, not 128. They look alike, but one counts units, the other says how much bigger each unit is; their closeness makes the cancellation above work, but they differ.
⭐ The takeaway of this section One shape: a perfect fit on one side, half wasted on the other.
Neither design is worse; “whose granularity?” has two answers.

⭐ Back to fusion. Two numbers cap fusion size: stovetop size (§3.2b: 227 KiB vs. 64 MiB, 289×) and this granularity.
Small bench, fine grain: many small fusions. Big bench, coarse grain: one much larger fusion. §3.6 meets both at the tiling step.
⭐ Turning “289×” into “how big a tile can you actually use” Decisions hinge not on a ratio but on whether a tile fits. Same arithmetic, both sides: (head_dim = 128, bf16, K/V double-buffered, scores in fp32; real kernels keep Q and the accumulator in registers, so this is an order-of-magnitude estimate, not a byte count.)
Br × Bc2×(K+V)QScoresTotal
64 × 6464 KiB16 KiB16 KiB96 KiB
128 × 6464 KiB32 KiB32 KiB128 KiB
128 × 128128 KiB32 KiB64 KiB224 KiB ← at the cap
256 × 256256 KiB64 KiB256 KiB576 KiB ← over cap
512 × 10241.0 MiB128 KiB2.0 MiB3.1 MiB
2048 × 20482.0 MiB512 KiB16.0 MiB18.5 MiB
GPU thread-block cap, 227 KiB: 128 × 128's 224 KiB fits right at the ceiling; one step up is 2.5× the cap and overflows.
TPU core, 64 MiB: 2048 × 2048's 18.5 MiB fits with two-thirds to spare.

⭐ So “real FlashAttention kernels always use 64 or 128 block rows” isn't a tuning result; that 227 pins it down. Raise the cap to a few MiB and people would go bigger at once, as the TPU side does. ⚠️ §3.2b caveat: 64 MiB is the total; what one kernel may use at once is a separate budget. The last row fitting doesn't mean the compiler grants it all.

3.5 What each side added alone

So far: what one side has and the other lacks. Two pieces remain, one grown by each side.

The question isn't what they are, but: the product page says “supported”; can you use it?

Two examples, one per side.

GPU side: block-scaled quantization. The marketing line: “B200 supports FP4”.

True. But block-scaled instructions have two paths, on different dies; B200 has only one. (A warp-level instruction needs sm_120a: consumer dies only. tcgen05.mma's block_scale needs sm_100a: B200's only path. Figure: L300 G-5.)

⭐ Block scaling in hardware (NVFP4 vs. MXFP4, Blackwell's three additions, training-side 16×16): see Topic 8 · Precision and quantization.

TPU side: SparseCore, an embedding-lookup core. The marketing line: “the TPU has a core just for embedding lookups”.

Also true. But it pays off only when the same rows are looked up repeatedly. Recommenders, hammering a few hot rows, are its home turf.

Language models repeat too little (not zero). A 100K+-token sequence: only a few thousand distinct words, mostly the top 1–2K rows. Real repetition, but orders of magnitude below search/ads/recs, whose concentration sets SparseCore's break-even. This doesn't pay. The compiler never routes them there: zero table lookups in production.

⛔ “Can't use it” isn't “not using it”: you use this core daily Saying “language models can't use SparseCore” is wrong. What goes unused is only the table-lookup role.

Its real job in large-model training: offloading collectives, taking All-Gather and Reduce-Scatter off the TensorCore so it keeps computing while communication runs. One way the TPU overlaps compute and communication.

Sources are public: the Google Cloud blog “Training large models on Ironwood TPUs” lists it as its own tuning technique; the public MaxText repo's XLA flag library has SparseCore All Gather.

⭐ The deeper lesson: marketing doesn't just leave you unable to use what it advertises; it hides the part you already use. You search the profile for table lookups, find none, and conclude the core is idle.

So how does it handle irregular memory access? Take the core apart.

SparseCore dissected—what “irregular access” means Addresses from data, not known at compile time: 2 problems, 2 fixes TensorCore Covered in §1–6 MXU × 4 256×256 array VPU vreg 8 sublane × 128 lane VMEM — TensorCore only; SparseCore has no direct access ⛔ Static: addresses known ahead SparseCore 2 per device (4 physical/chip) · tiny die area Scalar × 1 · scalar · dynamic indexing · issues DMA/streams · own SMEM Vector × 16 (a.k.a. tile) Own VMEM + SMEM; own streams SIMD 16 (F32)/32 (BF16)—10-plus numbers, not 1,000 Shared VMEM OpenXLA calls it SPMEM tiny, fast, compiler-managed ⭐ Native data-dependent control flow & memory access: addresses can be freshly computed HBM — 192 GiB (v7). Tables, temps; both cores attach ⭐ So “VMEM or HBM?” is a false choice: it touches both, but the VMEM is its own, not the TensorCore's. Minimum granularity? First, untangle two conflated things TensorCore's (8,128) vreg shape (8 sublane × 128 lane = 1,024), and VMEM layout unit—4 KiB in fp32. ⛔ Not the minimum DMA size. I conflated them when teaching this. Still: for a few scattered rows, aligned fetches waste bytes. SparseCore DMA granularity 32 B — jax 0.11 code (docs say 64): pltpu.get_tpu_info().sparse_core on 7x reports num_cores=2, num_subcores=16, num_lanes=16, dma_granule_size_bytes=32 ⭐ So “can we avoid moving (8,128)?”: two machines, two units; it moves 32 B. One gather in this figure ① indices in SparseCore's VMEM → ② scalar subcore issues DMAs → ③ 16 subcores chase their own addresses, pull from HBM In Pallas, one line: sync_copy(data_ref.at[indices_ref], target_ref)— data in HBM, indices in VMEM. Scatter is the reverse. ⭐ Not only for embeddings: in the public Qwen 3.5 Ironwood tuning write-up, SparseCore gathers tokens from HBM by routing index, feeding the TensorCore's GMM, the same shape as fetching sparse attention's scattered top-k KV. ⚠️ Landing spot not public. Write-up: “contiguous virtual buffer”, no HBM intermediates—HBM bypass is stated, but no public doc says it lands in TensorCore VMEM; OpenXLA's embedding doc says SparseCore↔TensorCore “frequently involves HBM as an intermediary buffer”. The two disagree; no verdict here. ⭐ It hides latency with many fetches in flight, not faster ones The pain is waiting on each fetch, not bandwidth. 16 independent streams: fire in bulk, each returns alone. In short: TensorCore's dropped runtime-address ability became a small side core. Main core: static, full compute; dynamic → SparseCore. ⚠️ Specs: JAX Pallas + OpenXLA SparseCore docs; “why it suits offloading collectives” is our inference, not official.
⭐ Two hard parts, two fixes: data-computed addresses → native data-dependent control flow and memory access; scattered rows → finer granularity (v7 DMA granule: 32 bytes, from pltpu.get_tpu_info().sparse_core). ⚠️ “VMEM or HBM?” is a false choice: it uses both, but the VMEM is its own (one per subcore + one shared; OpenXLA: SPMEM), not the TensorCore's. ⭐ Latency hiding: not faster fetches, but 16 streams chasing their own addresses, many in flight.

Now what it is: lift the lid.

SparseCore opened—subcores, roles, wiring Previous figure: a black box; this one opens it. In shape it's a 16-way data mover, not a calculator. One SparseCore — v7: 4 physical cores/chip, 2 per device. Tiny area, yet this section's second star. ⭐ Key: orange dashed = dispatch (control, DMA requests); blue solid = data. Split because the commander moves no data. Scalar × 1 scalar subcore · scalar, 1 at a time · dynamic indexing:    addresses from live data · issues DMA/streams · own SMEM (scratch) ⭐ Dispatcher: no math; assigns moves Vector × 16 — docs call it a tile 16 identical, own memory, own streams ① tile 0 tile 1 tile 2 tile 3 tile 4 tile 5 tile 6 tile 7 tile 8 tile 9 tile 10 tile 11 tile 12 tile 13 tile 14 tile 15 ⛔ Not a “smaller TensorCore”: no MXU in here. Shared VMEM Docs: SPMEM · all 16 tiles use · tile exchange   via here · tiny/fast · software-managed,   not a cache ⛔ Not TensorCore's    VMEM ① Zoom into one tile 16 identical; learn one Vector ALU SIMD 16 lane (F32)/32 (BF16) Local VMEM Docs: TileSPMEM / local SPMEM Local SMEM tile's scalar scratch ⭐ 10-plus values, not 1,000 cmd data HBM — big tables, weights, temps live here. Min DMA 32 bytes (pltpu.get_tpu_info().sparse_core reports it) All three roles wire to HBM: DMAs go there; tiles fetch/write back; SPMEM fed from it. ⭐ Why this core exists: TensorCore fetches in vreg-shaped blocks (8 sublane × 128 lane); scattered rows pay for whole blocks. Here: 32 bytes. One gather along these lines 1. Indices — the row list is itself data, in SparseCore's own memory 2. Scalar issues DMAs — orange dashed: many requests, no waiting 3. Tiles chase addresses — blue solid: own HBM share, no queue 4. Cross-tile via SPMEM — scatter: same, reversed ⭐ Not faster fetches: many fetches in flight. Four job types (per the official docs) Vector math — short-vector +−× gather / scatter — indexed get/put sort·dedup·count·hist — next step depends on the data ragged ops — uneven row lengths ⭐ All four: computed addresses or ragged shapes, what static machines hate most. ⭐ Why the mover now does collectives All-Gather/Reduce-Scatter are also “fire DMAs, let each return, add en route”: gather's shape; the far end is another chip, not HBM. Hence: offload collectives from TensorCore; compute and comms overlap. Pallas exposes it: VectorSubcoreMesh lets you “apply the same programming model of TensorCores to write collectives on SparseCores” (JAX Pallas docs, verbatim). In XLA it's a set of flags. ⚠️ Collective-offload fit is inferred from shape; flags/gains are public, rationale isn't.
⭐ One SparseCore = 1 scalar subcore + 16 identical vector subcores (tiles) + one shared SPMEM, all wired directly to HBM. Watch the two line types: orange dashed is dispatch (scalar subcore issuing DMA requests), blue solid is data (tiles moving it themselves). The issuer moves no data, so many fetches stay in flight. ⛔ The tiles have no MXU; it is not a small TensorCore. Specs and ops: official JAX Pallas SparseCore docs.

In XLA, the “collectives” line is a set of flags. Here they are, so “offloading” is tangible, all from public sources:

flagTurns onKey point
① Main switches: collectives → SC
--xla_tpu_enable_sparse_core_collective_offload_all_gather All-Gather to SparseCore On Ironwood (v7) these three default to true.
So it already works for you, no flags set
…_offload_reduce_scatterReduce-Scatter
…_offload_all_reduceAll-Reduce
…_offload_2d_all_gather All-Gather on a two-level mesh Must be enabled explicitly
② Base: main switches need these
--xla_tpu_use_tc_device_shape_on_sc SC reuses the TensorCore's device shape MaxText groups these four as ENABLE_SPARSECORE_OFFLOADING_BASE_FLAGS,
set to true / false / false / true. The trap: the main switch alone often does nothing
--xla_sc_enable_instruction_fusionSC instruction fusion
--xla_sc_disjoint_spmem Disjoint per-tile blocks of shared SPMEM?
--xla_sc_disable_megacore_partitioning No megacore split
③ Visibility: without these, that stretch of the profile is dark
--xla_tpu_enable_all_gather_offload_tracing
…_reduce_scatter_offload_tracing
…_all_reduce_offload_tracing
Traces the offloaded stretch Only when tuning. “Read the trace, not the product page” needs the lights on first
④ One that isn't SparseCore but always gets tuned alongside
--xla_tpu_scoped_vmem_limit_kib Max VMEM per op; the rest goes to the next op's prefetch MaxText public recipe: dense models 98304, MoE 81920

⚠️ Two constraints. First, Continuation Fusion and SparseCore offload are mutually exclusive; MaxText's comment: “Either one of CF or SC can be enabled at a time”. Both overlap comms with compute by routes that don't stack; “both on is faster” is the commonest misuse. Second, 98304 KiB = 96 MiB, larger than v7's 64 MiB of VMEM per core. The recipe uses it as a cross-generation limit request; the target machine caps the real value. Reported as is, without guessing why.

⛔ First: the two “Tensor Cores” are two orders of magnitude apart The easiest cross-platform spec-sheet mistake; it makes every “compute per core” figure wrong too.
NVIDIA Tensor CoreTPU TensorCore
What it isA unit inside an SM The device's main compute core
ContainsJust the matmul array MXU + VPU + scalar unit + VMEM
Per chip100+ SMs, each with several On v7: 2
⭐ So a TPU TensorCore corresponds not to a GPU Tensor Core but to the level of a whole GPU die. §3.7's “592 Tensor Cores vs. 4 MXUs” table rests on this; get it wrong and you read it backwards.
⭐ Second: scalar vs. vector is not weak vs. strong “1 scalar subcore per 16 vector” invites the misreading “scalar compute is weak, so only one”. Wrong.

• A vector unit does the same thing to a row of numbers: a thousand numbers, one instruction, all plus two.
• A scalar unit makes one decision: how many loop iterations? where is the next address? skip this batch?

⭐ Kind, not quantity: a decision has one answer; you can't make it a thousand times. So 1:16 is division of labor.
Not just SparseCore: TensorCore and GPU SM share the trio. It's the machine's skeleton. 0-D/1-D/2-D is right; the error is reading it as strength.
⭐ Third: turning on both optimizations doesn't stack them The table's worst trap is the mutual exclusion, not a flag. Continuation Fusion (comms chopped into compute gaps; the TensorCore alternates computing and moving) and SparseCore offload (whole job outsourced next door) share one goal: two implementations of one thing.
The compiler rewrites that instruction one way only; both on = two passes fighting over one op.

⭐ The rule: before asking “is both on faster?”, ask if they are two ways of doing one thing. If so, both on doesn't stack; the compiler picks one arbitrarily. How CF works and the “picks one” source: Further reading 1.
⭐ Fourth: what MMA means, and why the TPU lacks it MMA = Matrix Multiply-Accumulate: one instruction computes D = A×B + C.
Remember: multiply and add are fused; the accumulator is far wider than the inputs: FP8/FP4 in, FP32 accumulation, no spill or rounding. ⭐ This makes 4-bit inputs usable: not that four bits suffice, but that they only feed the multiply; accumulation never loses precision.

Does the TPU have MMA? Mathematically yes, formally no. The difference isn't whether a matrix instruction exists but who decides when it runs. GPU: a thread issues it at runtime, hardware schedules, a scoreboard tracks. TPU: each cycle's issue is fixed at compile time.
⭐ The GPU made matmul an instruction; the TPU, a pipeline. §3.2d again, other unit. Warp → warpgroup → thread; sources: Further reading 3

Further reading: three topics the lecture may skip

Each came from a class question. Safe to skip; each pins down an easily blurred point.

Further reading 1: CF, and why it excludes SparseCore offload
CF = Continuation Fusion. Its goal is exactly the same as SparseCore offload's: overlap collectives with compute. Only the mechanism differs.

How CF works: it cuts one All-Gather into small chunks slipped into neighboring compute: the TensorCore computes, moves a chunk, computes again, alternating on the same core. Unfinished work passes to the next fusion as state, relay-style. That hand-off gives “continuation” its name.

How SparseCore offload works: the whole transfer goes next door; the TensorCore keeps computing.

Why only one? The MaxText config docs settle it: “The implementation (such as BC-offload or continuation fusion) is chosen based on other flag values.” Chosen. They're alternative implementations of one collective; the compiler picks one to rewrite that instruction. Turn both on and two passes fight over one op.

⭐ The takeaway: before asking if two optimizations stack, ask if they're two ways to do one thing. If so, they don't; the compiler picks one. Sources: public MaxText repo, src/maxtext/configs/README.md and benchmarks/xla_flags_library.py. Continuation-state passing between async collective fusions: public tech blogs; neither NVIDIA nor Google has published a formal CF design doc.
Further reading 2: Scalar, vector, matrix units (I always mix them up)
The intuition is right: sort by dimension. Scalar: one number at a time (0-D); vector: a row (1-D); matrix: two 2-D tiles (2-D). Shape-wise, that's it.

⛔ The common mistake is to read them as tiers of compute, as if a scalar unit were a weak vector unit. They do different kinds of work.

· A vector unit does the same thing to a row of numbers. 1,000 numbers, one instruction: all +2.
· A scalar unit makes a decision. How many loops? Next address? Skip this batch?

⭐ The difference is kind, not amount: a decision has one answer; you can't make it a thousand times. “Weak” isn't a flaw: the job needs doing once.

An analogy: the vector core is a hundred line workers turning the same screw; the scalar core is the one holding the order: “this batch, line three.” One is enough.

Back to the figure: SparseCore has 1 scalar subcore to 16 vector subcores. That ratio is the division of labor, not a scalar-compute shortfall. Orange dashed (commands) leave the scalar subcore; blue solid (data) are the 16 tiles each fetching their own. The commander doesn't haul data.

⭐ The most useful point: this trio isn't specific to SparseCore. The TensorCore has the same three layers: scalar unit + vector unit (VPU) + matrix unit (MXU). So does a GPU SM. The whole machine is built on this skeleton, so you won't have to sort it out again elsewhere.
Further reading 3: What is MMA? On TPU too?
MMA = Matrix Multiply-Accumulate. One instruction computes D = A×B + C.

In hardware it is special in three ways, each saving real cost:
  • ① One instruction = a whole small matmul, not one scalar multiply-add. Fetch/decode spread over hundreds of MACs: saves scheduling.
  • ② Fused multiply-add, wider accumulator. FP8/FP4 in, FP32 accumulate, no write-out or rounding between. ⭐ 4-bit inputs work not for accuracy: they only feed multiplies; adds stay 32-bit.
  • ③ Operands are reused inside the unit. A scalar loop re-reads each element of A for every column of B; an MMA unit reads it once, reuses it across the tile: the source of arithmetic intensity (Section 1's line).
⭐ Across generations, the instruction keeps getting bigger and the issuer smaller.
GPU genIssued byOperands in
Volta – AmpereOne 32-thread warp, in sync Private registers, one fragment each
HopperOne 128-thread warpgroup, async A straight from shared memory
BlackwellOne thread, fire and forget Shared + Tensor Memory (incl. accumulator)

The Blackwell row's “issued by a single thread” is from Colfax Research's Blackwell Tensor Memory tutorial (“can and must be issued by a single thread”; the PTX ISA officially says “single thread semantics”). The reason: none of its data sits in private registers; it all lives in CTA-shared memory, so the matrix unit is decoupled from the warp scheduler.

So does the TPU have MMA? Mathematically yes, formally no: that gap is this lecture's thread.

The MXU multiply-accumulates too. The question isn't whether a matrix instruction exists, but who decides when it runs. GPU: threads issue an MMA at runtime; hardware schedules it, the scoreboard tracks it. TPU: the machine is VLIW; each cycle's issue is fixed at compile time: no scoreboard, no out-of-order, not even threads.

⭐ The GPU turned matmul into an instruction; the TPU, into a pipeline. An instruction is issued, scheduled and tracked; a pipeline runs on schedule as data flows through.
⚠️ A long-standing confusion: “Tensor Core” means things two orders of magnitude apart · NVIDIA's Tensor Core is an execution unit inside an SM. A B200 has over a hundred SMs, each with several.
· The TPU's TensorCore is the device's main compute core, holding the MXUs, VPU, scalar unit and VMEM. A v7 chip has only 2.

⭐ So the TPU TensorCore corresponds not to the GPU Tensor Core but to an entire GPU die. The easiest cross-platform spec-sheet mistake; it makes every “compute per core” figure wrong too.

⭐ The takeaway: Blackwell's moves (accumulator out of registers, single-thread issue, operands in dedicated memory) point to the TPU. For matmul, it deliberately dropped “a fragment per thread” dynamism for bigger tiles, less scheduling overhead. Elsewhere, the GPU stays the do-anything dynamic machine. The paths haven't merged, but on the money-maker, they're a step closer.

Sources: NVIDIA PTX ISA docs (three gens: mma / wgmma / tcgen05.mma), CUTLASS Blackwell feature docs, Colfax Research's Blackwell Tensor Memory tutorial. “Blackwell is moving toward the TPU” is my judgment, not official from either side.
⭐ First half: five layers, one kind of missing thing Chip → core → floor plan → parallel hierarchy → instruction. At every layer the TPU lacks something.
Never a compute unit. Always cache, scoreboard and “decide at runtime” machinery: all there because the hardware doesn't know what you'll run.

⭐ Section 0's claim, now checked five times: everything differs, and each difference traces to “it already knows what you'll run.”
The second half reassembles all five.
Section 3b

⭐ Rebuilt: FlashAttention, side by side

All the first half's parts now work at once.
Two jobs. First, walk one op end to end (§3.6). Then pay off two earlier claims: §3.6's end settles Section 0's “different everywhere, but only one cause”; §3.7 explains Section 1's coincidence: same-range peaks.

3.6 ⭐ Climax: one FlashAttention, side by side on both chips

Two questions: how much fits in one pan (on-chip memory)? What holds each side back?

The op picked in Section 2 now runs once on each chip.

🍳 A kitchen analogy; every later number hangs on it Picture a three-step dish: chop, stir-fry, plate.
The clumsy way: chopped food goes to cold storage, out to be fried, back, out to be plated. Three steps, four trips.
The catch: the half-done food is huge. Ingredients: 100+ MB. Intermediate: 30+ GB, 256× bigger. Four trips: 100+ GB on the road, not one cut made.

⭐ FlashAttention in one line: all three steps on the stovetop at once; the intermediate never leaves it.

Three mappings to keep:
• Cold storage = HBM (off-chip main memory: big but far)
• Stovetop = on-chip scratchpad (GPU shared memory, TPU VMEM: small but close)
• Blade width = how much one instruction bites (§3.4's 16 vs. 256)
Both opening questions live here: pan size = stovetop size; the limit = stovetop or blade.

Now real numbers. First: why is tiling unavoidable? Section 2's “demand exceeds supply 5×” was a ratio; here are absolute sizes.

The problem: 128 GiB spent in transit, zero multiplies 3.6 intro HBM trips: real cost On-chip: saved Q / K / V / O: moved regardless One head/sample, seq 128K, head_dim 128, bf16 131,072 seq len n × 131,072 n × 2 B bf16 = 32 GiB attention matrix S alone Q / K / V / O together: just 128 MiB. Intermediate is 256× the I/O. ❌ Naive: four HBM passes of S Each step writes; the next re-reads ① S = QKᵀ write S → HBM 32 GiB ② softmax read S ← HBM 32 GiB (still step ②) write P → HBM 32 GiB ③ O = PV read P ← HBM 32 GiB Total: 128 GiB of HBM traffic, zero multiplies ✅ FlashAttention: S never hits HBM Tiles + online softmax, updated in place Scratchpad (shared/VMEM) ① load K/V tile + Q tile ② tile's S; update running max m ③ old × exp(old m − new m): same scale ④ add this tile; scale denom too Loop; one final divide = same as full row The cost K/V re-read per Q tile, so this term isn't zero. Gain depends on S: long seqs win, short may not. 128 GiB S term gone; math identical to naive 🔢 Only one cost moves FLOPs Forward unchanged Backward recomputes once: 7 vs 6 (see 2.3) HBM I/O O(n²) → O(n²/tile) The whole win Memory O(n²) → O(n) 32 GiB intermediate gone ⭐ Takeaway: the HBM-to-compute path is not an academic question The most important kernel of the past five years has nothing to do with compute. Same algorithm, same forward FLOPs, no new instructions — it just reroutes the data. Next, we walk the path stop by stop, side by side: one FlashAttention on GPU and TPU — where each stop is and who decides. ⚠️ Sources Computed here: 131,072² × 2 B = 32 GiB; ×4 = 128 GiB; Q/K/V/O = 4 × 131,072 × 128 × 2 B = 128 MiB. Verifiable from the figure. Published: forward FLOPs same, memory O(n²)→O(n) (FlashAttention paper). O(n²/tile) for HBM I/O is a simplification; paper: O(N²d²M⁻¹), M = on-chip size. “Short seqs may not pay” is public too; no cutoff given.
Figure P-38 128 GiB moved, zero multiplies. At 128K tokens the attention matrix S = 32 GiB; naive takes four passes (write/read S, write/read P). Q/K/V/O: just 128 MiB. FlashAttention only deletes that term; it saves no forward FLOPs (backward recomputes: 7 vs. 6). Four steps = online softmax: if a new tile raises the max, the old result is rescaled by exp(old m − new m), then added. Equivalence hinges on it. The yellow card's cost: K/V re-read per Q tile, not zero.
This is §1–2's operator fusion (expand to compare)
⭐ This figure is the operator fusion from Sections 1 and 2, laid out end to end 🍳 In the analogy, fusion means “all three steps on the stovetop in one go.” Red, left: a cold-storage trip per step; green, right: all on the stovetop.
Unfused, the 32 GiB S lands in HBM between steps; fused into one kernel, the S term disappears.

⭐ Every fusion has this cost profile, not just FlashAttention: zero FLOPs saved; all savings in the “intermediates never land” row. Hence Section 1 made “compute ÷ bandwidth” the course's master ratio: fusion is the only technique moving you right on the roofline via the data path, not the math.

⚠️ The math is unchanged, so hardware, not the algorithm, sets the ceiling: how much fuses depends on what fits on-chip. The paths below differ in how far the same fusion goes per chip.

How does each side delete it? We walk each path alone, then side by side. On purpose: a table up front makes everything look different; walked separately, most stops match.

From HBM to the MACs: each stop, and who moves the data Off-chip HW-managed: tags, can miss SW-managed: never misses, no safety net Only capacity published NVIDIA B200 Five stops. L2 and L1 are caches; hits are only known at runtime. HBM3e 192 GB 8.0 TB/s Off-chip · spec L2 cache 126 MB 4 partitions, 2 per die 21 TB/s chip (local partition), 16.8 cross-die · third-party Absent on TPU L1 Shared 256 KiB / SM Same SRAM, two roles: left HW, right SW Shared ≤ 227 KiB / block · 128 B/cycle/SM (3 public sources agree) · L1 hit ≈39 cycles Reg file 256 KiB / SM 64K × 32-bit ≤ 255/thread, compile-time CUDA Core 128 / SM FP32 / INT32 vector path HW auto miss → next HW auto no control explicit ld.shared direct zero latency TMA / tcgen05.cp async: whole tile → TMEM, skipping registers; SM keeps working meanwhile TMEM 256 KiB / SM 128 lane × 512 col New in Blackwell; Tensor Cores only Tensor Core 4 / SM 1,024 MAC/cycle ← derived; see Fig. G-2 direct skips regs Three things to remember 1.5 stops of cache. L2/L1 keep tags, evict lines, spend area and power guessing ahead; shared mem doesn't: a scratchpad like VMEM, two orders smaller. GPU has both. Guess wrong? Run another warp. GPU latency is hidden by other work, not removed, so it packs 64 warps. Tensor Core path is new. Before Blackwell, MMA operands went via registers; TMEM skips them. TPU v7 Four stops, no cache: every move is fixed at compile time. HBM3e 192 GiB 7.4 TB/s / chip Off-chip · spec · 96 GiB / device VMEM 64 MiB / core Scratchpad, sole MXU source No tags, no misses · size: JAX code; BW not public Vector regs Not public Shape: 8 × 128 2D tile Count not public; 8×128: Pallas docs VPU Vector unit All elementwise ops act/reduce/scale Writeback → VMEM → HBM Still DMA, compile-time vs GPU: writeback via L2 DMA engine compile-time vec load direct return Weights/data go straight in: never via vregs — GPU caught up only in Blackwell MXU 256 × 256 65,536 MACs, fed by one instruction Count/chip: docs conflict, see G-8 Accumulator 1 MiB / MXU 128 × (8×256) × 4 B Results stay, not in regs lands here Same 3 slots, on the TPU One fewer level. HBM → VMEM, no L2. Saves area and the “will it hit?” question itself. No warp to swap. No backup on a miss, so compiler must pin, per cycle where data sits—no guessing. On-chip size/BW unpublished: no numbers. Known: levels and data movement—the real point. Stack the chains — not speed, but who knows where data is Cache/scratch Same SRAM, opposite natures GPU L1/L2 is a cache: you issue loads, it handles hits. The cost: tags, eviction, coherence, unpredictable timing. VMEM is a scratchpad: the compiler DMAs data in. No tags, no misses, predictable. Cost: a bad plan is slow, no backup. Who starts it Same fetch, different issuer GPU: compute issues a load itself, arrival unknown; data moves as a side effect of a load. TPU: a separate DMA; descriptor gives src, dst, size, stride; a DMA engine runs it, not compute. “I want it, find it” vs. “stage it, I'll come get it.” Who hides the hundreds of cycles— next figure. GPU drifts this way TMEM is a tell Blackwell's TMEM: on-chip SRAM for Tensor Cores only, explicitly moved, outside the caches—essentially VMEM. On the matrix path, hardware guessing pays less; control goes to compiler and kernel authors.
Figure G-6 B200: five stops, two of them caches. Hit or miss is known only at runtime: the source of §2.1's 4 main-path red boxes. — from the GPU Microscope
Full trip: the middle stop is a scratchpad, not a cache HBM On-chip Math DMA (compiler-issued) Gray dash = not public Five stops, one question: who decides? Every TPU deck lists capacity and BW. What you hit in code is the “Who decides” row — no stop answers “hardware decides.” ① HBM Off-chip 96 GiB 3,433 GiB/s / core Who decides Compiler emits DMA; scalar unit issues it Size Per descriptor; min 32 B On a miss? No hit/miss concept: only “arrived” or “not yet” ② VMEM Scratchpad (not cache) 64 MiB / core Who decides Static, by compiler, like regs; fixed Size One tile = (8, 128) = 4,096 B On a miss? Overflow caught at compile time; auto-tile or fail ③ Vector regs VPU input Unknown v7 depth/count Who decides Compiler allocates; VLIW names them Size One op: 8 × 128 = 1,024 elements Why dashed v2/v3 papers: 32 deep/sublane; v7 unpublished, not reused. Do MXU operands pass here? Unpublished. ④ MXU 256×256, weights stay 65,536 MACs / MXU Who decides No “decision”: weights stay, activations stream (§3.4) Size Each cycle: 256 acts in, partials out On a miss? Underfed = idle, nothing fills in (§3.3) ⑤ Accumulator Results land 1 MiB / MXU (see below) Who decides HW accumulates; if K > 256, it stays put Size 128 × shape (8, 256) × 32 bit Why 1 MiB 128 × 8 × 256 × 4 B = 1,048,576 B — 1 MiB flat DMA load feed land Return: accumulator → VMEM → HBM, same DMA and descriptors — also compile-time, not a casual writeback Scratchpad vs. cache: the key cell in this figure Item GPU L1 / L2 (cache) TPU VMEM (scratch) Contents Guessed by HW Hard-coded Hit rate? Yes; key tuning metric No such thing Bad guess / plan Slower (extra trip) Stall (nothing to swap) Your control Indirect: reorder loads Direct: tile/shard shape Moves aren't “background” — they cost slots Every HBM ↔ VMEM move is a DMA, and the scalar unit writes its descriptor. So the when, how much and where, like the MAC ops, take slots in the same stream. This explains §3.2's odd design: why a matmul chip cares about its scalar unit — not to compute but to schedule moves. The real limit is the BW gap: scratchpad beats HBM by about an order of magnitude (magnitude only; unpublished). So “stay in scratch longer” isn't a style tip. One line to remember No step here is decided by hardware at runtime. What, how much, when, where: compiled into the instruction stream. TPU perf problems rarely look like cache misses; they look like “bad shape, no good schedule”. Hence Microscope §7's figure has to exist: give it all to compile time, then see what it can schedule.
Figure T-6 v7 path (to the accumulator): the middle stop is a scratchpad, not a cache. Each asks: who decided this step? Never “the hardware improvises.” §2.1's “zero red boxes on the TPU side,” stop by stop. — from the TPU Microscope

Now side by side: the main figure of the lecture.

FlashAttention on both paths, side by side: read down, then across 3.6 main GPU · B200 TPU v7 Real structural gaps What FlashAttention does here ① Off-chip Only Q / K / V / O live here. S never does. ② Cache Biggest structural gap is here. ③ Scratch Main stage: tiles here; capacity sets size. ④ Operands Before compute: stage in registers? ⑤ Compute Alternates two units: matmul, softmax, matmul GPU · B200 HBM3e 192 GiB · 8.0 TB/s Official. Chip-wide. (Sheet says 192 GB; it's binary: a stack = 8 layers × 3 GiB = 24 GiB; 8 stacks = 192 GiB = 206 GB, like-for-like.) 8.0 TB/s L2 cache 126 MB · 4 parts Automatic, with a hit rate. Code can't pick what stays or goes. L1 / shared 256 KiB / SM · shared ≤ 227 KiB / block Left: HW-run L1 Right: SW-run shared Tiles here; author loads them in. Once per tile Reg file + TMEM path 256 KiB/SM each Pre-Blackwell operands went via regs; now TMA/tcgen05.cp → TMEM, skipping regs. Units take turns Tensor Core 4 / SM QKᵀ / PV CUDA Core 128 / SM online softmax swap TPU · v7 HBM3e 192 GiB · 7.37 TB/s / chip Official. 96 GiB / device (v7: 2 devices / chip). No such stop HBM straight to VMEM. Saves area and the whole “will it hit?” question. One hop VMEM 64 MiB / core All SW-managed — no tags, no misses Pallas BlockSpec declares tile size; compiler does the rest, not the author. Once per tile Vector regs 8×128 tiles · count unpublished ⚠️ MXU operands: VMEM → MXU directly, or via vregs? Not public; no verdict. (Fig. T-6 draws it as MXU input: one reading.) Units take turns MXU 256 × 256 QKᵀ / PV VPU vector unit online softmax swap ⑥ Who schedules the swap — Section 5's theme in one real kernel GPU: warp scheduler Every cycle Runtime: issue a ready warp from dozens; if slow, swap — others' work hides latency. TPU: compiler + VLIW Static, cycle-exact Compile time fixes op, cycle, slot. No backup task — a bad plan really stalls. 📐 Stop ③ compared by area, not by number = GPU shared mem, 227 KiB (top-left) Green box = TPU VMEM 64 MiB / core 64 MiB ÷ 227 KiB ≈ 289×. Of 289 cells, GPU gets the blue one. ⚠️ This is per unit (one core vs one SM), not total — totals flip; see 3.2b. Tile size hinges on this per-unit cell; bigger tiles, fewer K/V passes. ⚠️ Reconciling counts We said “five” TPU stops once, “four” once — both right: MXU branch vs. VPU path. Two different paths, so no total here; just who's where. ⭐ Only two gaps are structural; the rest are sizes Row ②: GPU has a full HW cache tier; TPU has a hole. Row ⑥: the swap — picked per cycle vs. compiled in. Other four: same job, one part each, different size/name. These two make FlashAttention two different jobs; the next figure wraps it up. ⚠️ Sources Per Figs. G-6 / T-6, same caveats: L2 BW is third-party; TPU on-chip BW and vreg count unpublished. Computed here: 64 MiB ÷ 227 KiB = 288.7 ≈ 289×. The 17×17 grid is exactly 289 cells.
Figure P-39 One FlashAttention, stop by stop, one row per stop. Only two of six rows are structural: ② (the GPU has a whole hardware cache tier; the TPU doesn't) and ⑥ (who schedules switching). The yellow band settles the earlier “how many TPU stops?” mismatch: the counts follow different paths; no total.
⭐ In plain English Row ② (an extra automatic cache tier) was covered in §3.2–3.2c. Row ⑥ is new: who decides when to hand off. The dish hops between the matrix burner and the vector burner. The GPU picks on the fly, whoever is free; the TPU sets the roster in advance, to the minute. That's all of Section 5; just planted here.

⭐ Count with Section 1's test: of six stops, only these two exist on one side and not the other; the other four do the same job. Other differences exist, but they are only parameters such as capacity and naming (the largest is the 289× at stop ③), and they change next generation.
Fusion size is capped by two numbers together (from §3.4; expand)
⭐ From §3.4's close: two numbers jointly cap fusion size The first number, just drawn as area: stovetop size. A GPU thread block gets at most 227 KiB.
Three 128×128 bf16 tiles (Q/K/V) = 3 × 128 × 128 × 2 B = 96 KiB; add double buffering and you're near the limit.
On the GPU, tile size runs straight into capacity. The TPU's 64 MiB is 289× that; capacity isn't what holds it back.

The second: §3.4's granularity, how wide one instruction bites. Contraction dimension: GPU 16, TPU 256. head_dim=128 is exactly 8 × 16 on the GPU, every cut clean; on the TPU it fills only half the 256-wide edge.
How the two divide up, 🍳 in kitchen terms:
The GPU's stovetop is too small. You cook in small batches, refetching the same ingredients: K/V re-reads.
The TPU's stovetop is too big to fill; its knife is too wide. A 256-wide blade on 128-wide ingredients: half of it cuts air.
Same job, different walls. That's why §3.4 said to read the two numbers together; either alone misleads.
⭐ Same fusion, two chips: three differences seen only side by side ① Walls capping tile size: one on the GPU, three on the TPU.
GPU: only capacity, 227 KiB, so go as big as fits. Small tiles mean K/V reloads.
TPU: three. Upward, capacity (backward overflows first) and, sooner, parallelism (big tiles leave one KV block; the pipeline collapses: fits, but slower). Downward, per-tile overhead. So on the TPU, “bigger is better” is wrong.

② The GPU must decide who splits, who shares; the TPU never asks.
Hundreds of small GPU units share the work, so you pick a dimension to split; pick wrong and you need a reduction, sync and sum.
On the TPU, one core takes one head's Q tile start to finish. Parallelism: the VPU's 8×128, a core's MXUs, the chip's two cores; no “farm out to hundreds, then merge” step. (Which split avoids merging, and why the two FA versions swapped roles over it: Topic 3, Section 1.)

③ Identical MACs, different reachable peak.
Same math, same FLOPs. The GPU pays in data movement (K/V re-reads), the TPU in idle cycles.
The TPU idles on shape: head_dim = 128 meets a 256-wide MXU in both matmuls: as contraction dim in QKᵀ, output dim in PV. So MXU utilization caps at 50%, and no kernel parameter fixes it. That holds at the kernel-parameter level. The real ways out (§3.4): change head_dim (a model-architecture call, not tuning: 32 heads × 128 → 16 heads × 256 measured +21% / +32% / +46% MFU at 8K / 16K / 32K), or a fully transposed pipeline.

⭐ All three in one line: the array has two spatial dimensions and one temporal. Unfilled space idles; too little time can't amortize weight loads. head_dim = 128 falls in between: too small to fill space, too short to serve as time. The far smaller GPU array fits 128 any way: it hits capacity; 128 can't fill the TPU's: it hits shape. (Where this test comes from, and why all three dataflows fail it: the two folds in §3.4.)

⭐ The rest is FlashAttention's story: the inner/outer-loop figure, why versions swapped loops, measured tile sweeps, three-layer efficiency ceilings. All in Topic 3 · The evolution of attention, Section 3. Here it's just the vehicle.

The stovetop's second limit: delivery speed (analogy addendum)
🍳 The stovetop's second limit, after area: delivery speed The two walls govern “how much you can fuse”; Section 1's 64 governs whether fused is still fast. The trip from cold storage doesn't vanish; it moves to another road, with its own width. So after fusing, redo the math: compare the new intensity against 64, not 312.

Last figure. Ignore the stops; count what you must write, the coder's side.

One algorithm, two kinds of work — hand-moved vs. declared 3.6 close GPU: in the kernel TPU: declared Easy to overstate What's identical — only the last row differs Math online softmax: per-tile max, denom; rescale old to new max FLOPs (forward) Identical, same as naive · backward recomputes, 7 vs 6 (2.3) Result Numerically equal to untiled Differs: who moves data. GPU · hand-written CUDA Reuse made explicit in shared memory You write Tile size: Q / K / V tiles + temps fit in ≤ 227 KiB Issue copies: cp.async / TMA, next tile HBM → shared mem Double buffer: load next while computing current Warp roles: producer warps issue TMA, consumer warps wgmma — split so the pipeline overlaps Sync: Ampere uses __syncthreads(); Hopper on: mbarrier async barriers — tile k arrives, consumers start; k+1 loads L1 or not: cp.async .ca via L1+L2, .cg L2 only HW covers Warp scheduler picks a ready warp every cycle. Latency you left may hide behind other warps, so on GPU “ slow loads” often aren't fatal. TPU · Pallas / Splash Attention Declare shapes; compiler times moves You write BlockSpec: tile size and tile i's slice (index_map) grid: tile count, loop nest Compiler does DMA timing, double buffering, VMEM layout — all set at compile time, absent from source. Bonus: Splash Attention has block masks. Blocks fully masked by causal / sliding window are skipped; saves FLOPs, not moves. ⚖️ The shorter right column is the point: the missing four moved into the compiler. ⚠️ Half true: “FlashAttention bypasses the cache” L2 can't be bypassed — every HBM access, no instruction exempt. Only L1 is optional: cp.async .ca via L1+L2, .cg L2 only. It drops reliance on caches, not caches. With reuse explicit in shared mem, hit rate no longer rules — that's the point. ⭐ Takeaway: Section 5's theme in one real kernel On GPU, FlashAttention negotiates with HW guesses: caches and warp scheduling hedge, but this kernel needs no hedge and moves data itself. TPU has nothing to bypass: just declare shapes. Costs are symmetric: GPU kernels need retuning each generation; a bad TPU spec truly stalls, with no backup. Same algorithm, two kinds of engineering: runtime vs. compile time. ⚠️ Sources Looked up: cp.async .ca / .cg caching: PTX ISA; BlockSpec / index_map / grid: Pallas docs; Splash block masks: its public code. From earlier figures: shared mem ≤ 227 KiB / block (Fig. G-6). “FA puts reuse in shared memory”: common practice in public code, not a quote.
Figure P-40 One algorithm, two kinds of work. Math, FLOPs, results: identical. Only who arranges data movement differs.
GPU: the kernel author hand-writes 6 things: tile size, issuing cp.async/TMA, double buffering, warp roles, sync, L1 use.
TPU: just 2 declarations, BlockSpec and grid. The shorter column is the conclusion. Red band (most overstated): L2 isn't bypassed; reliance on it is.
⭐ Walk complete: Section 0's claim holds Four of six rows match. Evidence for “different everywhere, but only one cause”: no cherry-picked example, the whole path counted.
(The two differing rows are one question: auto-cache and hand-off scheduling both ask whether the decision waits for runtime.)
Section 2's 4 runtime decision points are now concrete: the left rows you type by hand.

⚠️ Don't read “shorter right column” as “TPU is less work.” One wrong character in those declarations and the compiler won't catch it. Left: “many decisions.” Right: “one chance to get it right.”

3.7 Section 1's coincidence: why the peaks are in the same range

Section 1 set the peaks side by side (2,307 vs. 2,500), same range, no why. Now we compute it. The first half's two most different numbers cancel.

🍳 The next figure, as one picture Nearly the same total blade, cut completely differently: many small knives versus a few big ones.

⭐ The question isn't “who has more compute” but “how many pieces, of what size”.

Watch the tally on the right. The knife has two sizes, both shown; don't merge them. Blade width (bite depth): §3.4's contraction dimension, 16 vs. 256. Cut per stroke (MACs/cycle): 1,024 vs. 131,072 (one Tensor Core vs. one MXU), 128× apart.

Areas to scale: totals nearly equal, granularity 128× apart B200: one Tensor Core TPU v7: one MXU Derived; chain and post-mortem in Microscope §2 / §8 NVIDIA B200 · chip 148 SMs × 4 Tensor Cores per SM 592 Tensor Cores × 1,024 MACs/cycle each = 606,208 1 Tensor Core 1,024 MACs / cycle (derived) Area ratio 1.16× TPU v7 · chip 2 TensorCores × 2 MXUs/core — official (bottom right) MXU 256×256 131,072 MACs/cycle MXU 256×256 131,072 MACs/cycle MXU 256×256 131,072 MACs/cycle MXU 256×256 131,072 MACs/cycle 4 MXUs Dots = density only; true 256×256 → 0.27 px per cell × 131,072 MACs/cycle each = 524,288 The math B200 TPU v7 chip Ratio MACs/cycle · total 606,208 524,288 1.16 × Independent units 592 Tensor Cores 4 MXUs 148 × Unit size 1,024 MACs/cycle 131,072 MACs/cycle 128 × ↑ why rows differ 148 ÷ 1.16 = units differ 128× ✓ Peak spec ˟ 2,250 TFLOPS FP16 2,307 TFLOPS BF16 1.03 × How derived 148 teardown, ×4 official; 1,024 each: spec + derived 4 MXU × 262,144 FLOP/ cycle × 2.2 GHz, all three official see left Official, not derived. Google blog, verbatim: 262,144 FLOP/cycle/MXU × 2.2 GHz × 4 MXUs = 2,307 TFLOPS. → 262,144 ÷ 2 ÷ (256×256) = 2 MACs per cell per cycle But 128×128 gens (v3/v4/v5e/v5p) all give 1 per cell So it's per-generation. Our last version reused “1 per cell” for 256×256, got 8 MXU — wrong; correction in Microscope §8. ˟ 2,250 = HGX B200; GB200 (NVL72): ~2,500 BF16 dense/GPU; use it in-domain, don't mix SKUs. What 128× means: three angles, no natural winner Why GPUs cut fine Fine grain buys generality A B200 runs many unrelated thread blocks; first done leaves first. For a general scheduler, units must be small enough for one warp each. Each of 592 has own datapath, accumulator, control: overhead ×592. Why TPUs cut big Coarse grain buys no overhead One MXU instruction fills 65,536 cells (131,072 MACs / cycle), so fetch/decode/control are spread over 60K+ multiplies: ~0. Only if a matrix that big exists — which the compiler must guarantee, or it idles. Which shapes fill? Recall Fig. G-4 GPU K is just 16; almost any shape is a multiple, easy to fill. The TPU MXU K is 256 — attention's usual head_dim=128 fills half. Same model config wastes in different spots on each side, so tuning doesn't port. A trade-off map, not a ranking Close totals and peaks: comparable density on one process node. The split is earlier: who knows where data is (G-6), who hides latency (G-7), how fine to cut (here). They interlock — change one, change all: fine cuts need a big RF for spare warps → no need for static schedules → caches must absorb latency. And vice versa.
Figure G-8 Nearly the same total, cut very differently. Area ∝ MACs/cycle, split by real unit count. Don't mix the two ratios: count 148×, unit size 128×; §3.4 means the latter. — from the GPU Microscope
⭐ Remember the right-hand 148 ÷ 128 Not approximate, exact: 148 ÷ 128 = 1.156, and 606,208 ÷ 524,288 = 1.156.
“148× more units” and “128× smaller units” nearly cancel: the real reason the peaks are close. The TPU recovers the last 1.16× with clock: 2.2 vs. 1.83 GHz is 20% higher: at 1.83 GHz the B200 does 606,208 × 2 × 1.83 GHz ≈ 2,219, so 2,307 leads by about 1.04×; the figure's bottom-right 1.03× uses the official 2,250, which implies about 1.86 GHz (NVIDIA doesn't publish the exact clock).
⚠️ This chain is for the HGX-board B200. Section 1 used 2,307 vs. 2,500: the NVL72 GB200 clocks about 2.06 GHz, so the TPU leads by under 10%, too little to offset 1.16×; hence Section 1's “GPU 8% higher.” Unit count and size match in both comparisons, so the peaks always share a range; who leads depends on the chip you pick.

⭐ Every ratio in this lecture measures piece size, not compute. Same order of silicon on both: 592 small pieces versus 4 big ones.
Section 4

Wiring 64 chips: two different answers

This is background for Section 6's measurements.
Those are 64 vs. 64, and the two sides wire them in structurally different ways.
Hardware only: link count, width, shape, worst-case hops. Which patterns suffer, and why expert parallelism (EP) is most topology-sensitive, is about sharding: Topic 5.

Wiring 64 chips: same 64, different structure The headline result is 64 vs. 64. First, see how each side wires its 64. ⛔ Structure only, not speed Which patterns lose, why EP is pickiest → Topic 5 GB300 · NVLink 5 + NVSwitch 18 links/GPU, one per switch = 18 rails 9 trays × 2 = 18 16 compute trays × 4 = 64 Switch 10 = rail 10 4 colors + the other 60 meet here Colors: 4 picked GPUs, each on all 18; other 60 alike (1,152 would blur) One rail, and how rails relate · 1 rail = 1 switch + links to all 64: a full star · No inter-switch links → 18 isolated rails, 1-tier non-blocking ⭐ You can count “any pair, 1 hop”: any 2 GPUs meet on every rail, so 1 switch between them, over 18 parallel paths—same for every pair. 64 GPUs, domain max 72: all 64 fit in one domain. TPU v7 · ICI 3D torus No switches: 6 neighbors per chip (±x ±y ±z) Red dash = wraparound (1 shown per axis; all 3 wrap) x z y (in) Its 6 links all direct x wrap y wrap z wrap 1D: 4 nodes Wrap: ≤2 steps (no wrap: 3) 4 per axis. No wrap: ≤3 steps; wrap: 2 steps 3 axes: 2 + 2 + 2 = 6 hops —worst pair; neighbors 1 ⭐ Here, cost depends on distance ⚠️ 6 hops needs wraparound: request “4×4×4,” not “64 chips” ⭐ Same ICI to 9,216 chips—no new protocol ⚠️ “1 hop” means different things—the most misread point GB300 hop = GPU → switch → GPU: 2 links + 1 switch. Location-independent, not zero. TPU v7 hop = a direct wire; 6 hops = 6 wires, not 6× slower. ⛔ 1 vs. 6 is structure, not a latency ratio. Link count, width, the spec-sheet mismatch: see the “chip to pod” figure.
⭐ Same 64 chips, wired differently. Left: 64 GPUs in one NVL72 domain (max 72); via a switch, any pair is one hop. Right: a 4×4×4 3D torus, no switches, 6 hops at worst; cost depends on distance.
The bottom band is the must-know caveat: “1 hop” differs by side: GB300's is GPU → switch → GPU; TPU's, one direct link. 1 vs. 6 is a structural difference, not a latency ratio.
⛔ Structure, not speed: this course has never measured collectives on a torus.

In one line: NVL72 is a switched domain; ICI is a direct-connect network.
The domain is capped at 72 chips, but any pair inside is one hop, wherever they sit. The network has no switches and links only neighbors, but one protocol scales to 9,216 chips.
64 sits where they differ most: under NVL72's cap, all one hop; on ICI, a 4×4×4 torus, 6 hops at worst.

⚠️ Two easily confused 900s Hopper's 900 is bidirectional; Blackwell's 900 is unidirectional. Same number, different quantity.
Blackwell: 1,800 bidirectional, 900 unidirectional. A GB200 board has two B200s, so the superchip totals 3,600 GB/s; each GPU is still 1,800. The general trap: one number, two generations, two conventions; ask which generation, which direction.
Chip to pod: where the programming model must change TPU v7: ICI NVIDIA B200: NVLink Protocol switch Gray = unpublished Why a torus, not a mesh Each chip: 6 ICI links, ± per axis. Join the end chips—1 more link halves the diameter. ● Red = wrap to far end; 3rd axis same, not drawn Wrap: chip 3 ↔ chip 0 ← one axis 6 ports per chip chip X− X+ Y+ Y− Z+ Z− 6 × 200 GB/s bidirectional = 1,200 Diameter: as a 4×4×4 mesh, 3+3+3 = 9 hops; as a torus, 2+2+2 = 6 hops, gap grows with size —this sets worst-case all-reduce latency. Torus isn't free: the slice must be a contiguous cube, so you request “a 4×4×4,” not “64 chips” —the shape is part of scheduling, as §3's table already hinted. Reach, on one log scale X: chips per interconnect domain (log). Watch the red line, not the ends: past it, comms code changes. 1 8 64 256 1,024 4,096 16,384 TPU v7 One ICI, 3D torus all the way DCN 1 chip 1 host 1 cube 4×4×4 1 pod 9,216 = 144 cubes B200 NVLink RoCE / RDMA 1 node (prev-gen HGX) 1 rack Protocol switch 128× = reach, not bandwidth. 9,216 ÷ 72 = 128; over that range, TPU collective code is unchanged (in-pod only; cross-pod DCN changes config). Past the red line, all-reduce drops from ~840 to ~325 GB/s (2.6×, not an order of magnitude); the library switches paths. Don't overstate: per-chip compute is about equal (NVL72 GB200 ~2,500 TFLOP/s dense BF16/GPU; TPU v7 chip 2,307—not HGX B200's 2,250, another SKU), and within 72 chips NVLink has more per-chip bandwidth (1.8 TB/s vs. 1.2 TB/s). Topology reach, not per-chip speed. Caveat: two official “pods” “Pod” officially means both the 9,216-chip system and a 256-chip unit sold. Both official, they conflict; none says which wins. When you see “a pod,” ask which one. Here, pod = 9,216, always with the number. Couldn't verify 1. Max schedulable slice. 9,216 in one torus is official; the max per job depends on the scheduler, and no citable limit is public. 2. When torus degrades. Do edge slices get wraps? Not public either. No guess. 1,200 GB/s: an official mix-up The text says “bi-directional bandwidth of 200 GBps per axis.” 3 × 200 = 600, not the table's 1,200. Only “200 per link” adds up: 6 × 200 = 1,200. We use that reading and note the wording—when docs conflict, flag it—don't quietly pick one. Both docs in one sentence GPU doc's last figure: B200 has 592 Tensor Cores, TPU v7 4 MXUs; count differs 148×, unit size 128×—the 128 we cite is the latter, not 592/4. GPU coordination is mostly on-chip. Here: 2 cores per TPU chip, but one protocol to 9,216 chips; GPUs switch at 72. TPU coordination is between chips. Neither is stronger; the complexity just lives elsewhere—the course's thesis.
Figure T-8 A 3D torus up to 9,216 chips, one ICI throughout. 9,216 ÷ 72 = 128 measures only topology reach, not speed (the top-right note blocks this misreading). — from the TPU Microscope

But reach is not what you actually get. Below are our own numbers: not vendor specs but the measured drop.

⚠️ Know what these are: different evidence from the two figures above. Those show structure; these are one side's measurements: GPU only, 8 to 32 GPUs in one NVL72 domain, not a comparison. We never ran the TPU counterpart (no torus collective measured). One point: “no distance in the domain” does not mean “no cost in the domain.” Which pattern suffers on which topology is again Topic 5.

Measured: all-to-all drops at scale—and a broken control Copy (source: intra-node) Reduce (also intra-node) Dispatch (inter-node) Combine (inter-node) Y-axis: % of 8 GPUs, not absolute bandwidth 8 GPUs = 100%; absolute scale flattens the other three 80% 85% 90% 95% 100% 105% 8 GPU (2 nodes) 16 GPU (4 nodes) 32 GPU (8 nodes) +2% -18% -9% -19% Bandwidth retained vs. 8 GPUs How to read: our error, then facts ① We picked Copy as the control: wrong path Logic: Copy flat → links fine → cause elsewhere. But source: Copy = “Intra-node GPU-to-GPU copy”—not the dispatch/ combine inter-node path. Intra-node data can't judge the off-node link. ② Certain: targets go 7 → 31 8 GPUs: tokens → 7 peers; 32: 31. Messages must shrink; small scattered writes use links worse than big contiguous ones. Source's “~100 / 22 GB/s per target” = total ÷ targets: derived, not measured; circular, so unused. ③ Combine falls faster, not from deeper reduction Each token adds just topk = 6 partials; fixed by config, not scale—6 at 8 or 32 GPUs. Likelier: peers to wait for: 7 → 31; it waits for all, so the straggler matters more. ④ Reduce fell 18% too—also “intra-node” Purple and red nearly overlap. Either the label is wrong or intra-node isn't immune to scale. Unresolved— flagged. Absolute values—GB/s; source: this repo, gpu/a4x/06-deepep-test/ Op 8 GPU/2 nodes 16 GPU/4 nodes 32 GPU/8 nodes Copy · labeled intra-node 5,600 5,500 5,700 Reduce · labeled intra-node 2,100 1,870 1,730 Dispatch · inter-node all-to-all 700 660 636 Combine · inter-node all-to-all + reduce 724 683 590 ⭐⭐ Takeaway, plus a lesson worth more Certain: at scale all-to-all drops with peer count. Unknown: share of smaller messages vs. inter-node contention. Can't separate here—the only control (Copy) is in-node; none for inter-node. Method (worth more than the data): ask a control not “is it stable?” but “same path as the test?” A wrong-path control is worse than none—it fakes an exclusion. | Source: 16-GPU drop = “Larger scale intensifies RDMA contention”; 32-GPU = “scatter fragmentation.” All in one NVL72 domain; inter-node via MNNVL or RDMA? The source says both. Unresolved here. | Why normalize: Copy 5,600 vs. Dispatch 700: on an absolute axis three lines go flat, and the drop is the whole point.
Figure P-6 Our runs: one NVL72 domain, 8 → 32 GPUs, Dispatch −9%, Combine −19%. The orange band below is as key: what we couldn't separate, and why.
⛔ The likeliest misreading 9,216 ÷ 72 = 128 measures only topology reach, not who runs faster. Reading it as “TPU clusters are 128× faster” is the course's easiest mistake, and the hardest to catch yourself.

⭐ Carry this into Section 6: its two numbers ran on two structurally different networks: one hop in a domain vs. up to six. We never split “faster compute” from “better wiring.” ⚠️ And we never measured collectives on a torus, so: structure, not speed.
The four comm primitives, NVL72 internals, cross-chip addressing: L300 §4; which sharding suits which topology: Topic 5.
Section 5

⭐ Runtime vs. compile time, unpacked

You've met this pair four times: §2.1's “4 vs. 0”; §3.3's runtime-only layers; §3.3b's three circuit sets; §3.6's two forking stops (of six).
Each hit this decision on a different part. Now the decision itself: who schedules, and how that flows from hardware to your code.

5.1 Hardware: who owns latency

The bottom layer is clearest. A load takes hundreds of cycles; who fills them? It's about responsibility, not speed.

Latency hiding: GPU rotates warps on one unit; TPU runs units side by side issuing or working waiting on data (dashed) bubble: nobody works, cycle lost not started / done B200 · one SM sub-partition (sub-core) One issue per cycle; any warp with data back may go. What counts is not one warp's speed but how many warps can swap in. cycles → (not to scale) warp 0 go go warp 1 go go warp 2 go go warp 3 go go warp 4 go warp 5 go Issue slot issued or idle idle idle Cycles 6–7: two bubbles: all six warps await data. 8 resident warps instead of 6 would fill both. Whether you get 8 depends on spare registers. TPU v7 · one TensorCore No warp swapping: the lanes are four separate physical units, already running at once; the compiler lines up their start/stop times. cycles → (not to scale) Scalar few cycles br W Vector (VPU) tens of cycles Matrix (MXU) hundreds DMA hundreds Unit busy any unit busy? One bundle 322 bits · v2/v3 Scalar ×2 Vector ×4 push/pop ×2 Misc ×1 Immediate ×6 Issue slots ×9: all 9 issue in one cycle Immediates ×6, no issue slot No bubbles. The push/pop matrix slot says it: the MXU isn't a function call but a pipeline: push in, pop later. Meanwhile the other slots keep issuing. Don't compare the two summary bars: they answer different questions. Left: “did the scheduler issue this cycle?” (one issue port for six warps, so it can idle). Right: “is any unit busy?” (four distinct units, so all-green comes easier). Simplification: each warp holds one instruction dependent on the load; real compilers fill these cycles with unrelated work, so bubbles are messier. What sets how many warps can swap in? Registers SM: 64K 32-bit registers, ≤ 64 resident warps (2,048 threads). A warp can return anytime: its state stays in registers. Zero-cost switching means every warp's registers are held at once. Hence: Registers per thread Resident warps per SM Share of 64 Covers the 8 cycles? 32 64 (at the cap) 100% Yes, plenty 64 32 50% OK 128 16 25% Tight 255 (max) 8 12.5% No, many bubbles CUDA tuning's core trade-off. Unrolling, holding intermediates in registers: speeding up one warp also shrinks its pool of stand-ins. How can the TPU be so precise? Constant latency No cache: no hit/miss variable. Fixed latencies (cycles): scalar single digits, vector tens, matrix hundreds. So the compiler can put WAIT on the exact cycle: 1 Matrix: push MXU starts; runs alone for hundreds of cycles 2 Vector, too next weights into the weight FIFO (background regs) 3 DMA, too copies tile after next HBM → VMEM 4 Misc: one WAIT at a set cycle: too early idles, too late stalls MXU 5 Scalar: checks sync MXU_BUSY clear → next instr., no stall The cost: no safety net. A GPU misguess only costs speed (other warps cover); on TPU an early WAIT idles, a late one leaves the MXU truly waiting. Uncertainty: two sides of one coin GPU: hardware absorbs it Caches: variable latency, no static schedule. So the SM needs a big bench of stand-ins: 256 KiB register file (context), 64 warp slots (bench), 4 independent schedulers (picking). None of it multiplies; its only job: cover stalls. TPU: compiler removes it No cache: fixed latency, each cycle filled at compile time. So no stand-ins, big register file, schedulers: the circuitry is saved, not swapped for more MACs. Fig. G-8 counts MACs/cycle, not area: totals comparable, split differs 148×. “Better-spent silicon” needs die area: unpublished; no claim. Why it outweighs bandwidth A bandwidth gap of tens of %: degree. Who tracks the data: kind. It shapes both compilers, how kernels are written, what tuning watches: the GPU watches occupancy, the TPU gaps in the schedule.
Figure G-7 How the GPU hides latency: a warp stalls, switch to another. It needs enough warps resident at once; that is all “occupancy” means. — from the GPU Microscope
Latency hiding: scheduled at compile time, frozen at runtime Scalar: DMA, addresses Vector: VPU Matrix: MXU Misc gray = slot empty this cycle (NOP) One VLIW bundle = one cycle's work, packed 322 bits total · TPU v2/v3, published No out-of-order, no scoreboard: what issues when, to which unit, is baked into the stream. Scalar Scalar Vector Vector Vector Vector MXU MXU Misc Immediate ×6 Slot mix: the published v2/v3 figure. v7's bundle width and slot mix are unpublished. Drawn because the structure (multi-slot cycles, compiler-filled) is unchanged across generations; counts vary. Compiler-scheduled chain: schematic, not a trace While DMA is in flight the matrix slot sits empty: fill it with other work or write it off 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 Scalar ×2 DMA next tile address next op Vector ×4 previous tile's norm / activation (independent) continues Matrix ×2 matmul: waits for tile Misc ×1 sync sync After cycle 14 all slots empty: no independent work. Nothing fills these four cycles: no thread to switch to. Cycles 1–6: vector slots hold the previous tile's post-processing, independent of this copy, so hoistable. Compiler's only latency trick: find independent work. One latency, two answers GPU: swap at runtime Warp stalls → pick a ready one. Cost: dozens of contexts resident at once; the 256 KiB register file exists for it. Pro: irregular shapes run tolerably. Con: same code, run times can vary a lot. TPU: hoist at compile Fills gaps with other work. No resident contexts: silicon goes to MXU, VMEM. Pro: predictable, same shape = same speed. Con: unfilled gaps idle for real; no runtime retry. This trade-off day to day: three symptoms, one cause ① Irregular shapes perform badly Not compiler laziness: fewer independent instructions to move. Small batch, ragged seqs, skewed routing: each step waits on the last, so cycles sit empty. ② Compiles are slow, for a reason XLA on v7: often 10–17 minutes. Not “translating”: doing the hardware's scheduling, which a GPU scheduler does live on every run, paid at runtime. ③ But rock-steady Same shape, repeated: step time barely jitters (no runtime decisions). Predictability is by design, not a side effect: capacity planning, regression checks get much simpler. Finishing §3.2's claim §3.2 said the TPU “saves the scheduler, register file and scoreboard silicon.” This is the other half: the saved work didn't vanish, it moved to compile time. Someone must schedule: hardware on every run, or the compiler once. “The TPU is simpler” is therefore a myth. It didn't delete complexity, it relocated it: from silicon to compiler, from runtime to when you write the config.
Figure T-7 How the TPU hides latency: the compiler schedules ahead. Gray cells are truly empty; nothing auto-fills them. Bundle layout: published v2/v3. Gantt: schematic, not measured. — from the TPU Microscope
When each scheme fails The failure conditions matter, not the mechanism. The GPU fails when it runs out of stand-ins.
Register pressure leaves fewer warps resident (that count is occupancy), exposing latency. Hence tuning often starts by cutting registers.
The TPU fails when it can't predict.
Dynamic shapes or runtime-only addresses leave the compiled schedule idle.
One fears tight resources, the other uncertainty. Remember it and you can derive, not memorize, L300 §5.5's porting costs.

5.2 Code: same job, two ways

The difference shows up verbatim in code.

Same job, two styles: one thread vs. whole array GPU / CUDA: SIMT, thread view TPU / XLA: SPMD, graph view gray = this side's price GPU · CUDA: describe what one thread does __global__ void add(float* a, float* b, float* c, int n) { int i = blockIdx.x * blockDim.x + threadIdx.x; if (i < n) c[i] = a[i] + b[i]; } In the code: blockIdx, threadIdx, and an if bounds check. “Who am I, which cell is mine” is in the code, so the partitioning is your call, and only yours. New shape or GPU: this code still runs, speed not guaranteed. TPU · JAX/XLA: describe how the whole array transforms c = a + b c = jax.lax.with_sharding_constraint( c, NamedSharding(mesh, P('data', 'model'))) This code has no indices. You never say who computes what, only which axis to split; the compiler generates partitioning, comms, overlap. New shape: full recompile. v7, measured: 10–17 min. Four rows, each a direct result of that choice Same question GPU / CUDA TPU / XLA When shape is set Variable at runtime. One kernel runs any shape Compile-time fixed; change → recompile Control flow Anything goes. if / while, branch at will Data-dependent branches costly; MoE routing awkward Who optimizes Hand kernels + libraries. Piece by piece Whole-graph view: global optimization Debugging print, step, profiler: every line visible Read compiler output, profiles. See results, not process The cost Shape change → recompile. 10–17 min on v7; interactive debugging is basically out. Ragged lengths need hacks. Bucket + pad: coarse wastes FLOPs, fine explodes compiles. Data-dependent ops are awkward. MoE routing, dynamic KV: rewrite to static shapes. The payoff The compiler sees the whole graph. Fusion, comm overlap, memory reuse: not your job, not in your code. Memory planning is deterministic. So you can predict OOM without a chip (no GPU equivalent); L300 §5.6 covers it. And all three are free. Yours by default; the price: can't turn them off when unwanted. The one-liner CUDA hands control to you; XLA, to the compiler. So the pains differ in kind: “I can't write a fast enough kernel” vs. “the compiler won't listen, and I can't step in”. Note: neither pain is lighter; they fall on different people: one needs a kernel writer, the other someone who reads compiler output.
Figure P-1 Same job, two ways. CUDA (left) has threadIdx; JAX (right), no index. It's in the code, not adjectives. Bottom columns equal on purpose: trade-off, not ranking.

Collectives, one level up, follow suit. GPU: you hand-write all-reduce as the training loop's second step. TPU: you never write it; the compiler inserts it from your declared parallelism. (Code pair: L300 §5.3.)

5.3 The compiler's limits

Natural reaction: “Why not leave it all to the compiler?” You can't; the line is clear.

What the compiler can and can't do: Flash Attention is the honest test Orchestrate: automatic Algorithm: human Both sides hand-write Compiler's turf · arranging existing ops Fusion Merge adjacent ops into one kernel; no HBM round trips. Comm overlap Whole-graph view shows which comms hide under compute. Reuse Buffers with disjoint lifetimes share an address. Layout, sharding Matrix layout and split axis, searched at compile time. Common thread: none changes what is computed, only how it's arranged. So the compiler may decide: worst case slow, never wrong. Limit Human turf · new algorithms Reorder the math Flash makes softmax mergeable piecewise; the attention matrix never fully exists. Not a better arrangement of old ops: a new algorithm. Invent a missing op A compiler only arranges what it knows. It won't dream up a kernel nobody has written. Past this line, the compiler can't help, on either side. Flash Attention: failing on both sides, it defines the boundary better than any success GPU CUTLASS / Triton Hand-written by researchers, not NVIDIA's compiler; libraries adopted it. TPU Pallas Hand-written too. Nor did XLA invent Flash; Splash was coded by hand. No compiler grew it on its own. Flash changes the order of the math: tiling, incremental softmax, attention never in HBM. Algorithm, not orchestration. Human work, always. Don't misread this: the compiler isn't useless. On GPUs, the left four are also automated by Triton / CUTLASS. The difference is degree of automation, not whether a boundary exists. Both have one, in about the same place. Past the boundary: Triton and Pallas play nearly the same role; the difference lies below Triton · GPU Python kernels → PTX. Mental unit: one program (thread group) per block; you handle shared memory and sync. Pallas · TPU Also Python kernels → Mosaic. Mental unit: a tile moving in/out of VMEM, via BlockSpec, not hand-written load/store. A limit on “let the compiler do it” Orchestration is the compiler's job; algorithms, always ours. Same line on both sides, so “TPUs need hand kernels” is no TPU flaw: GPUs need them too, with more authors and thicker libraries. The real gap is how much you can borrow past the line. GPUs lead today, but it's an ecosystem gap, not an architectural one; ecosystems converge over time. Don't conflate them.
Figure P-3 The compiler's limits. Left four: orchestration, its call. Right: changing the algorithm, beyond it. Flash Attention sits right; both sides hand-write it.
⭐ Where FlashAttention falls “Both sides hand-write it” completes §3.6's comparison table: the TPU spares you data movement, not algorithms. Full inference-engine porting costs, and what AOT can precompute: L300 §5.5 / §5.6.
⭐ Fusion, closed: two different things You may have tripped: the figure credits fusion to the compiler, yet the note says it can't do FlashAttention. Both hold: two kinds of fusion.

① Fusion that keeps the math: the compiler's job. Top-left cell: same math, no cold-storage trip. You write none; both compilers do it silently.
② Fusion that needs new math: always human.
FlashAttention fuses only because softmax was first rewritten to merge blockwise.
Plain softmax can't: its denominator sums the whole row, so nothing is output until the row is seen, nor can a small block stay on the stovetop. Online softmax fixes exactly that: running max and denominator update as it goes, one rescale at the end, identical results. The algorithm was swapped to enable fusion, not tweaked after. The compiler may decide how, not what, is computed.

⭐ “Who decides fusion?” Two halves:
• Whether and how much: the hardware (§3.6's stovetop and blade width jointly cap it)
• Who does it: whether the math changes (no: compiler; yes: you)
The paradox: fusion, the course's constant thread, is neither pure hardware nor pure software: on the seam, where all hardware study lands.

⚠️ Don't misread §3.6's “not a single FLOP is saved”: it counts the two matmuls. Online softmax's per-block rescale is O(n·d), negligible next to O(n²·d). “Keeping the math” means equivalent results, not zero extra instructions.

5.4 The objection: isn't CUDA Graph compile-time too?

Now the strongest objection, half right: record a whole step's graph once, run it straight through: the gains are real, often an order of magnitude.

What it overturns depends on why it's fast. The usual answer is wrong: it saves not “branches” but per-kernel launch CPU cost. So: which layer does it freeze?

Which layer freezes? CUDA Graph: the top. TPU compiler: all three static (fixed before running) live (chip decides at runtime) ① Sequencing Which kernel next, when sent ② Placement Which SM gets each block ③ Issue Which instruction next cycle GPU · no graph all three live Live CPU launches each live. Slow CPU, idle GPU. Live Hardware dispatcher picks an SM; not settable. Per cycle Scheduler picks a data-ready warp. GPU · CUDA Graph only top layer frozen At capture Whole DAG fixed at capture, then one replay call. This layer really is static now. Live Unchanged. Each node is a normal kernel launch; blocks still hardware-dispatched. Per cycle Unchanged. Schedulers, scoreboards, warp slots, contexts: all here, all working. TPU · XLA all frozen, to the cycle Compile time Whole step = one program. No per-kernel host launches, so no graph needed. Compile time No placement question: the unit is in the instruction. Compile time VLIW: instruction, cycle, issue slot all fixed. A stall is real idling. 🧳 Itinerary vs. inside the flight CUDA Graph is like printing your whole trip's itinerary in advance: which flight, then which train, all fixed, nothing looked up at the airport. What you save is the lookup at every leg. But inside each flight nothing changes: queues, boarding order, crew are all decided on the spot. The TPU compiler pre-plans exactly that layer: even who walks which aisle, minute by minute. 📏 Frozen grain: 3–4 orders of magnitude apart CUDA Graph's smallest frozen unit: “one kernel”, typically microseconds. The TPU compiler's: “one clock cycle”: nanoseconds. Microseconds ÷ nanoseconds: three to four orders of magnitude. (My estimate from the two magnitudes, not a published figure.) Calling both “static” hides this gap. ⭐ Takeaway: right on one layer, wrong on the two costly ones Right: sequencing really is frozen, and the gain is real (see L300 P-32). Wrong: placement and issue haven't changed at all. Yet those two layers are where the silicon goes. Unchanged, so none of their hardware can be cut. The next figure finishes the math. ⚠️ Sources The three-layer split is my framing, not a term; others might cut two or four. Who decides at each layer is sourced: HW block dispatch, per-cycle warp picks, compile-time VLIW slots. “Three to four orders” is derived: kernel µs ÷ clock-cycle ns. Neither is an exact spec, so only orders of magnitude, no ratio.
Figure P-33 Pinned down: which layer gets frozen?
Three layers: ① sequencing (which kernel next), ② placement (which SM gets a block), ③ issue (which instruction next cycle).
⭐ Answer, middle column: CUDA Graph freezes only ①; ② and ③ are untouched.
Each node is still a normal launch: schedulers, scoreboards, warp slots, resident contexts all run.
The TPU freezes all three, ③ down to the exact cycle and issue slot.
⚠️ Three layers: a teaching device, not a term.
What does CUDA Graph really produce, and who replays it? (four follow-ups)
① It doesn't produce code: not a single new line. Instantiation yields work descriptors: per node, which already compiled kernel entry to call, grid and block sizes, shared memory, where the parameters live, dependencies. The kernels are the same CUBINs as before, byte for byte. ⭐ NVIDIA named the API cudaGraphInstantiate: instantiate, not compile. Accurate.

② Operator fusion? None; not won't, can't. CUDA Graph can't see inside kernels: each node is an opaque entry address plus parameters. It doesn't know what you're computing.
⭐ It isn't on XLA's level at all: XLA works on HLO and sees the math, so it fuses ops, picks layouts, allocates buffers, schedules collectives, then emits new kernels; CUDA Graph emits nothing.
In one line: XLA, a compiler, changes what code runs; CUDA Graph, a submission list, changes only how it's submitted. The shared “graph” is chance.
📌 The two are complementary: XLA's GPU backend can wrap its kernels in a CUDA Graph to cut launch cost. Fuse first, then package.

③ Where does “replay” happen? ⚠️ First, a near-universal myth.
Kernel code lives on the GPU; a normal launch never uploads code. The CUBIN sits in device memory from module load; 10,000 launches don't move it.
Where do a launch's 5–10 µs go? The CPU builds a launch packet live: packing args, validating, writing the command buffer, ringing the doorbell. Small, but per kernel.
⭐ So CUDA Graph saves not “uploads” but the CPU rebuilding N launch packets every time. What moves to the GPU is launch metadata, not kernels.
⛔ Why it matters: if it saved uploads, bigger kernels would gain more; in fact smaller, more numerous ones do: it saves count, not size.

Two easily missed savings:
• Dependencies are enforced in hardware: NVIDIA: it “enforces the dependencies with hardware accelerations, instead of relying solely on CUDA streams and events.”
• The gap between nodes shrinks: purely device-side, not the CPU.

So who walks the nodes? The GPU. NVIDIA: the first launch “is responsible for uploading the work descriptions for a CUDA Graph to the GPU,” and a repeat launch means “launching a graph that has already been uploaded to the device.”
⭐ Stronger indirect evidence: replay's CPU cost barely depends on graph length (the NVIDIA post's title: “Constant Time Launch for Straight-Line CUDA Graphs”). If the CPU replayed node by node, cost would grow with node count. It's constant: the CPU isn't walking them.
📌 Since CUDA 12 a kernel can launch a graph: literal “replay inside the GPU”, but a different API and purpose; see below.

④ After replay, did the two lower layers change? Not at all. Each graph node is still a normal kernel launch, with a grid. The work distributor picks each block's SM at runtime, by free resources; the graph doesn't, and can't, record that. Same for clusters.
Inside the SM, same: which warp issues this cycle is still the scoreboard's call, hardware tracks operands, a load's hit is still luck (§3.2d: 30 vs. 659).

⭐⭐ Root cause: CUDA Graph works outside kernels; blind inside, it controls nothing there.
So the three-layer claim hardens: it doesn't just happen to freeze layer one; architecturally, it can't reach the other two. Submission can't rule execution.

📌 Appendix: host-side vs. device-side launch, which matters more? They solve different problems: host-side, “same work, many times” (volume); device-side, “next step depends on this step's result” (possibility), NVIDIA's positioning: dynamic control flow.
For today's large-model workloads, host-side dominates: steps are essentially static (shapes fixed ahead, routing padded to fixed capacity, precisely to keep the graph static). ⚠️ Inferred from design intent and common framework use; I have no measured comparison; measure to confirm. ③ quotes: NVIDIA Technical Blog, “Constant Time Launch for Straight-Line CUDA Graphs.”
⭐ Takeaway for §3–5: the objection is their best evidence Location settled, the point runs the other way:
① CUDA Graph itself needs static shapes: a new shape forces re-capture, the TPU's much-faulted static tax.
② The TPU doesn't need it: XLA already compiles the step as one program. CUDA Graph patches a problem of the GPU's own making.
③ Most important: hardware cost is set by the worst case it must support, not the case you run. Even with graphs on every workload, §3.2's five components stay: at tape-out nobody knew if you'd use them.

For this performance, the GPU must become more TPU-like, yet only the top layer can move earlier. If the others could, those transistors would be long gone.

5.4b Second objection: isn't TMA making data movement TPU-like?

It deserves more than CUDA Graph: it points the right way.

The trend down §3.3b's four generations:

Four GPU generations on: whole blocks, by descriptor, no threads held, into a software-managed scratchpad. Exactly the day-one TPU.

⭐ Only halfway; the other half is our thread How data moves converged: granularity (whole blocks), address generation (dedicated hardware, not 32 lanes), operand destination (a scratchpad, not registers). All three really do.
Who decides hasn't moved at all:
• TMA descriptors are built at runtime, not compiled in; shapes may change, you rebuild.
• L1/L2 remain; hits still known only at runtime: three of §2.1's four decisions live on.
• The warp scheduler still picks live, every cycle.

So not “the GPU is becoming a TPU,” but “the GPU made feeding TPU-like; who gives orders hasn't moved.” The sides never differed on the endpoint, only on who issues the DMA.
⚠️ Not imitation: the same constraint The faster the matrix unit, the less feeding may tie up compute. When one MMA eats tens of KB in a few cycles, 32 threads taking turns at addresses and register hops gatekeep their own compute.
Same physics, same answer; one vendor took four generations, the other shipped at the endpoint. The course's thesis, in time: not cleverness, one decision pushed through.
Sections 6–7

Can the two numbers be compared?

§3.6's six stops are done, but the two numbers below aren't from them. FlashAttention was structure and theoretical bounds (§2.3's figure notes “real Flash re-reads K/V; intensity never reaches 65,536”); the only like-for-like measurement: one pair of full-model Hunyuan3-295B runs.
So they pay off Section 1's line, not §3.6's side-by-side. First how to read them, then the numbers.

6.1 First, how to compare

Both usual comparisons fail. Peaks: what the chip can do, not your job's speed. MFU: pick a denominator, pick a conclusion. Only the figure's tier ③ stands.

Comparing: each tier costs more, proves more Free One run Weeks Four tiers · read cost with conclusion Reading only the right column, you'd jump straight to ④. But cost jumps with each tier. Most cases can stop at ②, as long as you admit what it proves. Tier Cost (log scale) Proves / don't forget ① Peak Free Proves almost nothing. Only the denominator; all traps live here: dense/sparse, SKU, chip vs. device. ② MFU A run per side How much of the hardware this stack uses. Both terms are choices (prev. 2 figs). High MFU ≠ fast: see right. ③ Same model & scale, each at its best Weeks + both chips Closest to the truth. ⭐ Our tier. Cost: tuner per side. One-sided = effort contest. ④ Per $ / per watt ③ ÷ $ or watts What procurement wants. Prices negotiated, not list; power: chip, system, or incl. cooling? ⚠️ High MFU ≠ fast Top misuse of tier ②. Not folklore: from one identity. Throughput = MFU × peak Example · bar = peak, fill = actual throughput Chip A 400 Peak 1,000 · MFU 40% Chip B 500 Peak 2,000 · MFU 25% Chip B: MFU 15 points lower, yet 25% faster. MFU = “is the HW maxed out?”: efficiency, not speed. Use it over time: same HW, this week vs. last. Across chips only throughput ranks: tok/s or s/step, 1 formula fewer than TFLOP/s (prev. fig. A). Simplest: report both: throughput = who's faster, MFU = headroom. ⭐ Why tier ③ is costly: tuning one side measures tuning effort Same model & scale: v7 tuned 445 → 666.6 TFLOP/s/chip, +50%. (⚠️ 6.3's measurement fig. has v7 at 662.3, not 666.6: different runs; 666.6 = tuning-log best, 662.3 = run beside GB300; 0.65% apart; always pair 662.3 with 854.0.) (8.6% is clock lock default→max: operating point, not tuning; without it ≈+38%; another doc starts at 457: +45.9%. Listing all three is the point.) That gap beats many “hardware comparison” verdicts. One-sided tuning only shows whose team knows its platform better. ⚠️ Our GPU log is one 7-config sweep (492 → 527, +7.1%), so next section's numbers sit at tier ③'s low edge too. Tier ③ data: next section · cost bars log, illustrative, unmeasured · v7/GPU tuning gains from repo tuning logs
Figure P-14 Your ticket into §6.3: it sets how to read the measurements there.

⚠️ Procurement wants the figure's ④ (per dollar / per watt), but no shortcut: ④'s numerator is ③. Without ③ measured, ④ is a ratio of two unknowns.

“Pick a denominator, pick a conclusion”: no scare tactic; we fell for it once (methodology)
⭐ “New denominator, new conclusion” is no scare tactic: we fell for it once Same run, same numerator 854.0, two MFUs reported:

• 854.0 ÷ 2,700 = 31.6% (committed 2026-08-05, stood for 26 days)
• 854.0 ÷ 2,500 = 34.16% (corrected 2026-08-31, the current figure)

A 2.6-point gap, no rerun; not one digit of the measurement changed.

⛔ What went wrong: that 2,700 was inferred, not looked up.
Its author took HGX B200's 2,250 × 1.2 = 2,700: “Ultra must be a bit faster.”
But NVIDIA has never published a 2,700. GB300 NVL72's dense BF16 peak equals GB200's.
Blackwell Ultra raises dense FP4 (1.5×) and attention (2×), not BF16.

⭐ The lesson isn't “be careful”; it's why it was hard to catch.
2,700 looks just like an official spec: right units, right magnitude, plausibly above GB200. Nothing throws an error. The downstream 31.6% hides better still: honestly computed, flawless arithmetic.
So “was this number derived?” asks about the number you cite, not you. One of our eight retractions; the full log (incl. which way each skewed) is in L300 Appendix A.

6.2 ⭐ The numerator: one run, two utilizations

That figure covered only the denominator. The numerator has two named flavors, both in use; unless you know which was reported, you can't compare.

One line: numerator = what the model's math needs, or what the hardware actually ran?

🍳 The knife again: cuts needed vs. cuts made Same denominator: the hardware's peak cuts per second.
MFU's numerator = cuts the recipe needs; HFU's = cuts you actually made (botched redos and air cuts included).
So HFU ≥ MFU always; the gap is wasted cuts.

Which to read depends on the question:
• “Is the knife idle?” → HFU
• “When is dinner ready, at what cost?” → MFU (wasted cuts won't speed dinner a second)
MFU is a fraction: both ends are choices; neither errors out Can more than double it Smaller, still must disclose Right column for this machine A · Numerator: what I compute Framework TFLOP/s: computed by a formula, not measured; formulas have versions. Formula missed a whitelist ~5× FLOP counter forks by family. Miss a name: no error, wrong expert width, shared skipped. Hit us porting. Count recompute? 4/3 HFU counts it, MFU doesn't. Full-recompute gap: 4/3 ≈ 1.33× (fwd twice), by def., unmeasured. All major report MFU. Halve causal attention's quadratic term? By seq len Negligible for short sequences; dominant at 128K. B · Same run, four MFUs 503 TFLOP/s/GPU measured, code unchanged. Below: only the divisor changes. 2,250 / 4,500 HGX B200 peak 2,500 / 5,000 GB200 · used here ÷ BF16 peak 22.4% ← doc's headline 20.1% ÷ FP8 peak 11.2% ← in a table note 10.1% 10.1% 22.4% 2.2× · from the denominator alone A's three, C's four: untouched, and they stack. 2.2× is a lower bound. C · Denominator: the peak I choose All four have official sources. Finding one is easy; saying which is hard. Dense or sparse 2× Footnote: sparse = 2× dense. Headlines default to sparse. Which SKU 1.11× See last figure. Per chip or per device 2× See last figure. Which precision's peak 2× FP8 run: BF16 or FP8 peak? Both used, both defensible. ↖ The middle table shows only two of these: SKU and precision. ⭐ So report MFU with this whole line: the one action to take from this section MFU 20.1% = 503 TFLOP/s ÷ 2,500 (GB200 · BF16 · dense · per GPU) · numerator: built-in formula, no recompute · formula version = that commit One line, copy it. Can't write this line? Don't report a percentage. Fall back to raw throughput (tokens/s, or seconds per step). It has a basis too, but at least you can feel fast vs. slow; a percentage can't. The next figure: when to fall back. Measured: A4X (GB200 NVL72) 2 nodes, 8 GPUs · Qwen3-style MoE · seq 16384 · FP8 training, 503 TFLOP/s/GPU measured · peak bases: previous figure
Figure P-13 The last figure tripped on “what unit?” This one, a layer down: right unit, wrong divisor.

The figure's 4/3 is an upper bound (full-layer recompute). Where does ours land? FlashAttention's backward recomputes the forward S once, wasting 1 of 7 matmuls (2 forward + 4 backward + 1 recompute): 7 ÷ 6 ≈ 1.167×, 85.7% useful. Both 1.167 and 1.33 are derived by definition, not measured. Overheads (16.7% vs. 33%) differ 2×; “recompute enabled” alone says nothing.

⭐ Third, easiest to get wrong: the roofline's numerator For “will I hit the roof?” use the HFU-side numerator, with recompute and padding.
Recompute really occupies matrix-unit cycles; hardware can't tell wasted work.

⚠️ Plot useful FLOPs and the error is one-sided: always an underestimate. Useful ≤ executed, so you think the ridge is far, add recompute, and crash through.
Example: an op that is 30% recompute: 312.5 × 70% = 218.8. Useful intensity is just 219; with recompute it's already 312. You think there's ample memory-bound headroom; you're already over.

One run, two questions, two numerators: HFU's to judge your side, MFU's to report output. Also how to use Section 1's line; derivation in Section 2's final collapsible.
⚠️ Trap: before comparing hardware, ask “same recompute?” Recompute fully off on one side, on at the other: HFU close, MFU far apart, or vice versa. Both read as “utilization”; they measure different things.

This explains item ③ of §6.3's closing figure (“what this ratio still lacks”): the recompute policies differ entirely.

6.3 Both sides, measured

Ticket in hand: skip the big numbers; read the two red segments first.

Measured on both sides: same model and scale, each tuned to its current level GB300 NVL72 · 64 GPU TPU v7 · 64 chips Light track = BF16 dense peak, dark = measured Two gaps, nearly equal length A · Two bars, one axis: track = peak, fill = measured One absolute-scale axis. The two red marks below are the point; numbers can wait. Axis: TFLOP/s (BF16 dense), full scale = 2,500 GB300 NVL72 64 GPU a4x-max · 16 nodes · 1 NVLink domain 854.0 Peak 2,500 MFU 34.2% · 6,242 tok/s per unit · step 21.0 s · GBS 2048 · TP1 PP2 VPP8 EP32 · MBS 1 ⚠️ Not maxed: EP 32→16 in same sweep gave 882.3 / 6,450 (+3.3%), not re-tested. So all ratios below are lower bounds. TPU v7 64 chips (4x4x4) 128 devices · one 3D torus slice 662.3 Peak 2,307 MFU 28.71% · 4,770 tok/s per unit · step 20.61 s · eff. GBS 1536 · DP1×FSDP128 · pdbs 12 · dvfs 7 Hit a wall: 8 runs (tile, XLA flags, SparseCore offload, bigger batch); none gained. Measured gap 191.7 = 29% of 662.3 Peak gap 193.0 = 8.4% of 2,307 Same baseline, compare → ⭐ The two segments are nearly equal: 193.0 vs 191.7 TFLOP/s. But one is 8.4% of 2,307, the other 29% of 662.3. Gaps must be divided, not subtracted. Subtracted, they're equal; divided, 3.5× apart. That's why the next panel's 1.084 × 1.190 is a product. B · Splitting the 1.29×: only half comes apart 854.0 ÷ 662.3 = 1.29 = 1.084 × 1.190 Per-chip tok/s: 6,242 ÷ 4,770 = 1.31, used hereafter 1.084 1.190 ↑ Log-scaled: the terms multiply; 0.084 : 0.190 would treat × as + ① Peak ratio: fully accounted for 2,500 ÷ 2,307 Official dense BF16, both on L300 P-12's basis. Fixed: it's what you buy. ② Beyond peak: residual, not “software” 34.160% ÷ 28.708% Not measured; the remainder. Software is in it, so is non-peak HW: HBM bandwidth, on-chip SRAM, MXU contraction shape, topology. ⚠️ This identity always holds; not news. MFU ≡ measured ÷ peak, so any two numbers split this way. Proves only: this 1.19× isn't from peak. Software's share? Not answered here; see next. ⚠️ Change the unit and the ratio changes; that 1.5% is exactly what §6 is about In TFLOP/s GB300 is 1.29× v7; in tok/s 1.31× (6,242 ÷ 4,770). Both right; the gap is the numerator formula: same model & tokens, yet the sides' analytic FLOP formulas differ by 1.5%. Size doesn't matter; the gap does. Tokens can be counted; TFLOP/s hides a formula. Reporting both? Put tok/s first. Not nitpicking: 1.5% is irrelevant here, but proves each side writes its own formula. Next time it may not be 1.5%; P-13 A's missed whitelist was 5×. Same Hunyuan3-295B-A21B (80 layers · 192 experts) · seq 4096 · synthetic data · no checkpoint writes either side (but recompute differs, see P-18 item ③) · both run by us
Figure P-15 The only figure with both sides' real numbers. Equal red segments; the big gap is denominator, not skill. Panel B's bottom-right red box: verdict, verbatim.

Hardest step next: not overstating this number. What does 1.31× still lack? List yours, then check below.
⚠️ Units first: the figure's 1.29 is per-chip TFLOP/s (854.0 ÷ 662.3); from here on we use per-chip token/s, 1.31 (6,242 ÷ 4,770). The gap is exactly the figure's 1.5%.

What it can't claim: six limits, each with a direction Pushes the ratio up (favors GB300) · 4 items Pushes the ratio down (favors TPU v7) · 2 items Bottom band has no direction: it's scope, not bias ① Different global batch → favors GB300 Q① GB300 2,048 seqs/step, v7 eff. 1,536; Q② 4,096 vs 1,408, 2.9×. Big batches amortize fixed cost. v7's batch is capped by 94.74 GiB: not a choice; that itself is a hardware difference. ② 64 is the GPU's home turf → favors GB300 All 64 GB300 GPUs in one NVL72 domain (no “distance”); v7: 4x4x4 torus, corners several hops. Model leans on all-to-all; §4: it alone forks by topology; TPU pays off only past 72 chips. ③ Recompute policy differs → favors GB300 “No checkpoint writes” = no disk writes, not no recompute. GB300: recompute off (230 / 288 GB); v7: remat_policy=custom + offload (91.94 / 94.74 GiB). Recompute off alone: +5.2%. ④ Software maturity gap → favors GB300 · will shift Key limit on this section's verdict; listed here, not just L300 P-16. If the gap is software, software changes: both stacks as of Aug 2026, not a hardware fact. Re-test next quarter: this moves first. ⑤ v7's 662.3 has a clock lock → favors v7 dvfs_p_state=7 locks max clock: +8.6% alone; §6: op. point, not tuning. GB300 had no equivalent knob. Hurts our case; it's our number. ⑥ “Per chip vs. GPU”: already generous → favors v7 One v7 chip = 2 devices; one GB300 GPU = 1. Per device = “one v7 die vs. a whole GB300”: worse for v7. Our basis is already v7's best case. See L300 P-12, panel B. ⚠️ A seventh item has no direction: one model, one scale. L300 §7 calls it “a hard constraint after three figures,” so here's how it would move Scale 64 → past 72, direction known from §4: GB300 must leave the NVLink domain, all-to-all goes “no distance” → “via switches and NICs”; v7's torus keeps extending linearly, no such step. Our 64 stops just before that fork. So “one scale” isn't modesty: the only limit with predictable direction but no data, pointing opposite to the six above. Cheapest way to overturn this section: don't dispute these 64 units' numbers; measure 128. This continues L300's P-17, which asks “what does this ratio still lack?”; this is the answer
Figure P-18 The answer: four red, two green. 1.31 is a point, not a bound: two opposite corrections, neither measured.
⚠️ The real output isn't the ratio The four red and two green point opposite ways: the number could move either way, and we've measured neither. Responsible work lists the unmeasured instead of rounding off.
Take away four questions: Was this number derived? Binary or decimal? Per chip, per device, or per core? Physical or usable? They work for any hardware pair. (The closing figure repeats them.) Tuning from 457 to 662.3, and what my checks missed: L300 §7 / §8.
Section 9

What is each good at?

Now the opening's open question. One sentence answers it; the rest qualifies.

9.1 Takeaway

From package to measurements, both chips are now taken apart. The opening promised “different everywhere, one cause”; in §3.6, four of six rows matched. Now, the cause.

Who gets control: every difference in this course follows from one decision Wrap-up · 3 GPU side: hardware at runtime, i.e. handed to people TPU side: compiler works it out at compile time; live with its quirks Right: four questions, any two chips ⚠️ Red/green mark sides, not better/worse; the course's only head-to-head was won by red The course asks one question Runtime-only facts (shapes, access targets, who's busy): who handles them? Hardware, at runtime i.e. handed to people The compiler, at compile time i.e. accept its quirks ① Decisions per memory access? §2.1 · §3.6 Main path: 4 runtime decisions; hit/miss unknown till run 0: all fixed at compile time; path is computed ② So when does the cost show up? L300 §5.5 · P-8 A bit per access, hidden at runtime; nothing to manage Paid once, by you. Cold start: 19 min compile; warm 7.5 min, 16 failures Recompiles: pure loss; needs Pallas authors ③ So what if the premise fails? See §2.1 Can reroute at runtime: HW works around it No runtime fix: back to code, recompile Four questions Nothing to do with these two chips; reuse on any pair. The course's truly portable part. 1. Was this number derived? If so, is the formula beside it? L300 P-19 2. Binary or decimal? Official HBM spec: GiB; 192 GiB = 206 GB, 7.4% apart: 192 GiB ≠ 192 GB L300 P-21 B① 3. Per chip, per device, or per core? v7: 2 devices/chip; logs report per device. Get it wrong: 2× off L300 P-21 B② 4. Physical or usable? 94.74 GiB is usable; 2,500 is dense, no sparsity L300 P-21 B③ ⭐ Not “which is stronger” but “who gets control” People: someone must write it. Compiler: accept its quirks. ⚠️ Take this sentence only together with the three below; on its own it's just a position ① Only head-to-head: GB300 1.31× faster Point, not bound. Two opposing, unmeasured fixes: EP16 (882.3) untested → up to 1.35?; 4/6 limits favor GB300 → less. Heaviest: SW maturity (P-18). ② Kernel gap: ecosystem, not architecture P-3's phrasing, in full: lead + kind + timing. Ecosystems converge; don't merge: architecture gaps don't close alone, ecosystem ones do. ③ TPU's premise was never backed with data Still owed §0 set a premise (“know early what you'll run”); §5 showed only its flip side. Calling it proven = “passing a name off as proof”. The takeaway alone is cheap; the three qualifiers below are what this course earned
Figure P-24 Center box: the course's skeleton. Right: four questions, the only takeaway for any hardware pair. Bottom band: three qualifiers, expanded below.
⭐ Keep one thing: operator fusion. The figure gives the principle; fusion is the course's one end-to-end example, and it walks all three layers:
Why fuse: the only way right of the ridge point without changing the math (§1 · §3.6)
How much: stovetop size + blade width; two hardware numbers jointly cap it (§3.2b · §3.4)
Who fuses: math-preserving, the compiler; math-changing, always people (§5.3)

Physics, hardware, control: each layer sets one limit. That's why studying hardware always lands on fusion.

That's the course. No buying verdict, but given “N× faster,” you'll know the four questions and where its denominator and numerator are. Recall §6.2's two utilizations: same denominator, all the gap in the numerator.

This is L200 · focused (core track, 109 min). What L300 · full adds (open to dig deeper)
This is L200 · focused (core track, 109 min) L300, the full edition (two hours, 72 figures), adds four deep dives:
· Chip package; full ten-row table
· Five figures, register file to bit line
· 4 comm primitives; domain layout
· Inference-engine porting cost; AOT
Plus a candid “how I checked the numbers” section: eight retractions logged, one slip made twice, twenty questions it can't answer.
All of it is in L300, the full edition
Section-number mapping between editions (L300 readers only)
§0–§5 and §9 share top-level numbers in both editions: match them section by section.
⚠️ Two mismatches: L300's §6 + §7 merge here as Sections 6–7 (measurements: §6.3); §8 is dropped.
Also, subsection numbers inside figures (§1.4, §2.6, §4.1, §5.5, §7, etc.) are L300's: those figures came over unchanged, so a figure's §7 (measurements) means §6.3 here. If a number doesn't match, look in L300.