Two questions, one path. ① How do the two chips differ in hardware? ② Run one op end to end: where does each difference surface? The verdict: they differ everywhere, all because of one decision.
This lecture has instructor notes.
They cover how to teach it: segment goals, pauses, common misconceptions, likely questions. The content is here.
Presenting: lines, figure cues and timing are written out. Self-study: give the script to an AI for a TA who knows the lecture and how to teach it.
Last lecture we took apart DeepSeek V3: 671B total parameters, but each token activates only 37B—that’s MoE.
If only 37B are active, is it as cheap to serve as a 37B dense model?
What does MoE actually save?
It saves compute, not memory.
Each token takes only the 37B path, so the compute saving is real. But all 671B of weights stay resident, every byte: no one knows which experts the next token picks, so all sit in HBM.
“37B active” ≠ “as cheap to serve as 37B.”
⭐ The idea behind the question:
memory and compute are separate, and move independently.
Last lecture kept the two accounts apart.
This lecture adds a third: bandwidth.
What decides whether an op is fast is often neither of the first two. Section 1 works out the exchange rate between compute and data movement.
We will take both chips apart, layer by layer. First, guess: how many pieces does each chip divide its compute into?
On B200 the unit is called a Tensor Core; on v7 it is the MXU. Both do matrix multiply-accumulate.
The question is the total per chip.
Answer the two independently; the answer appears once both are picked.
B200: 592. v7: 4.
592 = 148 SMs × 4 Tensor Cores per SM; 4 = 2 TensorCores × 2 MXUs per core.
A 148× difference in count.
⭐ The surprise:
The two chips differ by only 15% in MACs per cycle: 606,208 on B200 versus 524,288 on v7.
The same compute, cut into 592 small pieces vs. 4 large ones.
⭐ The “why” is an IOU. Keep the question in mind. How can the count differ by 148× while the total is nearly the same? What does each side pay for cutting fine vs. coarse? All of Section 3 answers these two. When you get there, you'll recognize which figure holds the answer.
The figure below is the lecture's only premise. With it, you can predict each section's differences. Look, then move on: Section 1 is one division; Section 2 starts the teardown.
Section numbers match the full L300 edition. Upside: to dig deeper, open the same number in L300. Cost: numbers skip in this edition: 0 · 1 · 2 · 3 · 4 · 5 · 6–7 · 9. Section 8 exists only in L300, and 6 and 7 are merged into one section here.
⚠️ One more catch: figures carried over unchanged from L300 use L300 numbering. The major sections map as follows:
Subsections too, and you'll hit these more often. Not in this edition; see L300: §1.4 (the 312 ridge point), §2.6 (how far constraints propagate), §2.8 (every op so far on one axis), §4.1 (interconnect limits), §5.5 (inference-engine porting pain points), §5.6 (predicting OOM with AOT, no chips needed).
This is deliberate. The figures belong to L300 and stay word-for-word; otherwise two versions would exist, and a fix to one would silently drift from the other.
This section computes one number: the compute-to-movement exchange rate, operations per byte moved. Every later op is checked against it: below the line, bandwidth rules; above it, compute. No conclusions here.
Divide the two spec numbers: operations per second by bytes per second. The ratio is operations done in the time one byte moves. A floor: each byte moved needs this many operations just to feed compute.
Everyone asks: both sides have eight 24 GB HBM3E stacks; why different bandwidth?
“HBM3E” names the generation, not the speed grade. Each stack: 1,024 data lines; bandwidth = 1,024 × pin rate. HBM3E spans 8.0–9.6 Gbps/pin.
Backing out: v7 runs 7.2 Gbps (7.37 TB/s ÷ 8, exact, though 7.2 is outside 8.0–9.6), GB200 about 7.8.
The buyer picks the grade (power, cooling, reliability, supply); it isn't the chip's ceiling.
(Why not top grade? That also exposes a hole in our own numbers.)
The HBM configuration is identical: eight HBM3E stacks, 24 GB each, 192 GB total. (The v7 “eight stacks” comes from Google Cloud's Ironwood technical blog; the B200 figure from NVIDIA's Blackwell architecture page.)
Then why does the bandwidth differ? Because “HBM3E” is not a speed; it is a range of speeds. Every stack has 1,024 data lines, and bandwidth = 1,024 × per-pin data rate. Pin rates from 8.0 to 9.6 Gbps all count as HBM3E (Micron lists a pin speed above 9.2 Gbps and more than 1.2 TB/s; Rambus lists 9.6 Gb/s and 1,229 GB/s). “Same HBM3E” does not mean “same bandwidth.”
Working backwards to each side's grade:
1,024 bit × 7.2 Gbps ÷ 8 = 921.6.
Exactly 7.2 Gbps (1,024 × 7.2 ÷ 8 = 921.6, × 8 stacks = 7,372.8 GB/s).
8 × 1,024 GB/s = 8,192 GB/s, rounded to “8” in public specs
2,500 ÷ 8.000 = 312.5, whereas 2,500 ÷ 8.192 = 305.2.
The figure continues below. The division works at every level; this course is about moving data between them. For now: the closer to compute, the lower the line. It pays off with fusion at section's end.
Three uses of the line. Both examples skip attention on purpose: Section 2's first figure derives it; doing it now would spoil it.
The matmul above is square (all sides N), giving N/3. Real ones are M×K · K×N, three lengths. Which sets intensity?
Same division. The operation count is 2·M·N·K, and the bytes moved are 2·(MK + KN + MN). It simplifies cleanly:
1/intensity = 1/M + 1/N + 1/K.
Reciprocals add, so the smallest dominates.
“Longer K, higher intensity” is true but capped: as K → ∞, intensity only reaches MN/(M+N).
K can't rescue a small M or N.
⛔ The harshest case: batch = 1 decode, M = 1, so 1/intensity ≥ 1: intensity caps at 1, however large K and N.
That one line of arithmetic pins LLM decode to bandwidth.
Attention: in QKT both outer dimensions are the sequence length S,
and the contraction dimension is head_dim. Most models use 128, so intensity is
≈ 128: below v7's 312 line and v6e's 560.
So “attention is a matmul, hence compute-bound” is a myth.
Unfused, it is memory-bound, held there by head_dim.
What does Flash change? Next section's first figure. The direction: it saves no FLOPs; it changes the denominator.
⭐ The same 128 bites again: the 256 × 256 MXU is fed only 128 (§3.4 / §3.6).
Intensity asks “wait for data?”; filling asks “is the unit full?”
You now have a ruler.
It answers one thing: bandwidth- or compute-bound?
Next, a real op, and the lecture's first big number: fusion changes intensity 1,000×.
The ruler does more than judge ops. Put inter-chip bandwidth under compute, and the division answers a new question: the fewest tokens per chip per step that let cross-chip communication keep up with compute.
Worked for TPU v7 (both numbers on Figure T-1):
2,307 TFLOP/s ÷ 400 GB/s (one axis, bidirectional) ≈ 5,768
All three axes triple bandwidth, cutting the threshold to ≈ 1,922. A full 9,216-chip domain needs a global batch of about 17.7M tokens
so communication doesn't drag. Hence huge training batches: not for convergence, but so the network keeps up.
⚠️ The denominator's basis is easy to get wrong. The number above uses one axis bidirectional, 400 GB/s (six links, 1,200 ÷ 3 axes), not the chip's full 1,200. Change the basis and the answer moves 3×; both versions read fine on paper.
⭐ So this section's real output is the division, not 312. Over HBM bandwidth it judges ops; over inter-chip bandwidth, parallelism strategy. The latter belongs to Topic 5; here we just show the ruler can measure something else.
The place it lands has a name: on GPU it's shared memory, up to 227 KiB; on TPU it's VMEM. HBM traffic does drop, but the data still lands on chip, where the exchange rate is 64, not 312 (derivation in the lower half of the figure in §1.1).
⛔ So why not just keep it in registers? An SM's register file totals 256 KiB, the same size as shared memory, and every extra register means fewer resident warps. Usually you can't. The numbers are in the SM breakdown figure in §3.2.
312 is the ridge point of the matrix roofline: the numerator is Tensor Core / MXU peak; the vector units are left out.
Because the ops we judge are almost all matmul FLOPs (attention's two, the MLP's big ones); softmax, norms and other vector work are a FLOP rounding error.
How much smaller are the vector units? Work it out:
GB200: 18,944 CUDA Cores × 2 FLOP/cycle × ≈2.06 GHz ≈ 78 TFLOPS
Against 2,500 on the matrix side, a 32× gap. ⚠️ That clock is derived: NVIDIA hasn't published GB200's boost clock, so we use 2,500 TFLOPS ÷ (606,208 MACs/cycle × 2 FLOP) ≈ 2.06 GHz
(606,208 = 148 SM × 4 Tensor Cores × 1,024 MACs/cycle; §3.7 takes this apart.) The lecture uses this number throughout (it retires an earlier 2.03 extrapolated from 1.83; 1.83 is a third-party HGX B200 figure, itself 1.4% off against 2,250).
TPU is the same: one TensorCore's two MXUs together do 262,144 MACs per cycle (131,072 per MXU; not the §3.1 figure's cell count), while the VPU processes 1,024 elements per cycle. Different units, but more than two orders of magnitude apart.
Strictly, element-wise ops should be judged against the vector roofline: its ridge point is 78 ÷ 8 ≈ 9.8, not 312.
But the conclusion doesn't flip. The vector add's 0.17 is 58× below 9.8: still memory-bound. That 1.3 TFLOPS is 0.05% of matrix peak and 1.7% of vector peak. Both look bad; the latter is the fair comparison.
⭐ The real takeaway: weak vector units are not an oversight; they are a design choice both sides share.
An op on the vector unit loses twice: once to bandwidth, and even without that limit, the usable compute is only a thirtieth.
This is the root reason fusion is mandatory: not to compute less, but to keep data off this narrow, weak path.
By total capacity it is GPU 231 MiB vs. TPU 134 MiB, 72% more. Same order of magnitude, so it looks safe to ignore. It isn't. The GPU's lead is almost entirely L2, and L2 is exactly the level the TPU lacks entirely. The gap isn't capacity; it's who controls that space. All three ways of counting are in §3.2b.
* † ‡ §)| Level | TPU v7 | GB200 / B200 | The crux |
|---|---|---|---|
| Matrix unit | MXU 256 × 256, 2 per core | Tensor Core (5th gen), 4 per SM | Grain: one big vs. many small |
| Scalar / address | Per core, only one scalar unit | 128 CUDA cores per SM, 148 SMs* | Who issues irregular addresses (§3) |
| On-chip · explicit | VMEM 64 MiB / core · compiler-managed | shared memory ≤ 228 KiB / SM · human-managed | Who sets the tile size (§2) |
| On-chip · automatic | CMEM = 0 (none) | L1 + L2 126 MB | ⭐ Whole level, one side only |
| Programmable co-processor | SparseCore × 4 / chip own programs: prefix sum · sort · count · gather/scatter |
(none programmable) TMA: fixed-function, not this class |
⭐ A level, the other side only |
| Odd-job units | XLU cross-lane units × 2† transpose · cross-lane reduce · shuffle. Transcendentals ( exp / tanh): own resource§; Pallas costs transcendentals beside flops. Name, count: not public |
SFU × 4 / processing block‡ exp · rcp · rsqrt etc. Cross-lane: warp shuffle, unlisted |
⚠️ Each side names one, silent on the other |
| HBM | 96 GiB / device (= 192 GiB ÷ 2); whole chip 7,372.8 GB/s (= 7.37 TB/s) | 186 GB / GPU (SW-visible; 192 physical); 8,000 GB/s | Same HBM gen and scale |
| Compute (chip / GPU) | BF16 2,307 | FP8 4,614 TFLOPS | BF16 2,500 | FP8 5,000 TFLOPS (dense, the GPU in NVL72) | Same magnitude |
| Compute ÷ BW | 312.9 FLOP/byte | 312.5 FLOP/byte | ⭐⭐ Nearly equal |
| Inter-chip | ICI 1,200 GB/s bidirectional · 3D torus · up to 9,216 chips | NVLink 1.8 TB/s/GPU · NVL72 domain · 72 GPUs (we use 64) | New protocol past the domain? (§4) |
* 148 SMs is from a third-party teardown; NVIDIA's spec sheet doesn't list it. All other numbers: vendors' official docs.
† The XLU count comes from Google's public How to Scale Your Model
(jax-ml.github.io/scaling-book): it says “a single scalar core controls a VPU […], 4 MXUs, 2 XLUs, and multiple DMA engines.” ⚠️ Its “4 MXUs” doesn't match v7
(v7 has 2 at 256×256), so the sentence isn't per-generation; we take the 2 only as “more than one.”
‡ NVIDIA states SFUs exist (Blackwell's SM lists Special Function Units), but the official diagram gives no count; ×4 follows Hopper's drawing.
§ This cell gets a note: rare three-tier evidence.
“A neural-network chip must have hardware for exp / tanh /
sin.” Right, but public evidence doesn't go that far:
① It existed once, on paper. TPU v1's diagram has a separate
Activation unit, and the 2017 paper mentions on-chip “nonlinear function logic.” Not new on TPU.
② It is still a separate resource class today, with positive evidence. In open-source JAX, Pallas reports a TPU kernel's cost through
CostEstimate(flops, transcendentals, bytes_accessed, …)—transcendentals is a peer column beside flops. Were they just “a few more instructions on an ordinary ALU,” they'd be counted in flops.
③ But the unit's name, count and implementation really aren't public.
The TPU v4 ISCA paper's full description of a TensorCore: four 128×128 MXUs, a VPU with 128 lanes (16 ALUs per lane), and 16 MiB of VMEM. Whether any of the 16 ALUs does transcendentals, it doesn't say. Google Cloud's architecture docs, JAX's official Pallas hardware reference, and JAX's machine-readable chip spec table all list only MXU / vector unit / scalar unit / SparseCore. Nothing public describes a TPU counterpart to the SFUs' “table lookup + interpolation.”
⚠️ The last point matters most: “not listed” on a spec sheet does not mean “not there” on the chip.
A counterexample sits in the same documents: the XLU isn't listed on any official spec sheet either, yet it exists (found in another public source). A spec sheet lists only what it wants you to buy on. So this cell says “a separate resource class, unit not public”,
neither “none” nor any detail I can't back with a public source.
Same appetite, completely different eating habits.
The difference isn't a missing part.
On one load path, one side makes four runtime-only decisions, the other none.
The cache accounts for two. The first: do a warp's 32 addresses coalesce?
The last: what to evict on a miss. Every later difference grows from here.
Skip the specs; follow one load from HBM onto the chip. Count the decisions: who makes each, when.
Decision one is the only one unrelated to the cache.
Coalescing merges memory accesses. On a load,
each of a warp's 32 threads computes an address;
hardware merges them into the fewest transactions.
Merge quality swings bandwidth 8×, and it's known only at runtime:
the compiler can't see how addresses are computed.
The TPU skips this: arrays are tiled 8 × 128 at compile time and
moved as whole tiles. No 32 addresses to merge.
What counts is the sector. An NVIDIA cache line is 128 bytes, split into 4 sectors of 32 bytes, and the sector is the real unit of transfer.
Best case: 32 threads read consecutive 4-byte words = 128 bytes, exactly one cache line.
4 sectors, not a byte wasted.
Worst case: the 32 addresses are scattered (e.g., down a column of a row-major matrix),
so each thread hits a different sector: 32 sectors,
each fetching 32 bytes to use only 4.
32 ÷ 4 = 8×. You pay full bandwidth and discard seven-eighths of it.
⭐ Hence “access memory contiguously” is lesson one of GPU optimization. It optimizes not latency but how much bandwidth each access costs. Source: NVIDIA's cache organization (128 B line / 32 B sector). Nsight Compute reports this directly as sectors per request. A perfect access scores 4; anything above that is waste.
⚠️ Why can the TPU assume “the shape is already right”? Because a vector register and an array's memory layout share one shape. Data is used on arrival, no relayout. The cost isn't a non-contiguous row (that's the upside: each segment fills all 128 lanes); it's that the shape is nailed down. A new innermost dimension moves real data, uneven sizes need padding, outside handoffs need relayout. §3.3 has the numbers.
With the criterion and coordinates set, we run one op end to end. Why FlashAttention? Every large model runs it, and it sits on the wrong side of the 312 line, so both sides must act.
Panel ③ is what matters. “Who schedules the movement” is the previous figure's decision. On the 312 line, the bill is absurd:
So both sides must tile and rewrite. The rest of the course is that rewrite, side by side.
⭐ Side by side: same attention, identical FLOPs
to the last MAC, 1,024× apart in intensity.
So intensity is set by the implementation, not just the math.
That licenses every later optimization:
if it were intrinsic, only new hardware could help.
Because implementation sets it, rewriting pays.
Br rows and Bc columns, with head_dim d.Bc × d each;
the Q tile stays in registers.4 × Br × Bc × d FLOPs.4 × Bc × d bytes (bf16).Br = 128: 128 > 64, clears the bar.Br = 32: 32 < 64, memory-bound again, this time on-chip.Fair question, with a standard answer. Extra compute comes in three very different kinds:
① Recomputation: the same operations, done again. Flash's forward has none. Panels ① and ② have equal MAC counts, a rarity. But the backward does: the unstored intermediate is rebuilt from Q, K. Saved memory is bought with recompute.
② Padding: the zeros added when shapes don't align. They take on-chip space and pass through the matrix unit but produce nothing. The worst of the three: it buys nothing at all. Section 3 shows how much.
③ Extra vector work: online softmax rescales the running result every tile. It isn't much by count, but it lands on the vector unit, which has under 1/30 of the matrix unit's compute (worked out at the end of Section 1). Counting FLOPs underestimates it; you only see it when you count time.
So is the trade worth it? Section 1's line earns its keep again, but first, one point made precise.
⭐ Arithmetic intensity's numerator is FLOPs executed, not FLOPs the model needs. Hardware can't tell recomputation apart. So which side of 312 you land on is set by the intensity including recomputation and padding.
But what you care about is useful FLOPs ÷ time. Together that gives two ceilings; the lower wins:
Example: intensity with recomputation 600; half the FLOPs are recomputation.
Useful fraction = 1/2 → compute ceiling = 50% of peak;
useful intensity = 300 → bandwidth ceiling = 300 ÷ 312 = 96% of peak.
The smaller wins: only half the roofline.
⚠️ Easy slip: bandwidth isn't saturated here. You're compute-bound, so movement waits on compute; bandwidth used is only 312 ÷ 600 ≈ 52%. The matrix unit is 100% busy, half on garbage, while bandwidth idles. The classic fingerprint of overdoing the trade.
Finally, back to Flash: it is nowhere near that bad case. Forward is two matmuls, backward four; not storing the intermediate matrix costs backward one more, 7 versus 6 in total. That is HFU ÷ MFU = 1.167, or a useful fraction of 85.7%.
Data movement drops to 1/1,024. 17% more compute buys three orders of magnitude less traffic. Flash didn't just “optimize a kernel”; it changed the op's bound.
§2 counted decision points: 4 vs. 0. The first half drills inward: chip → core → floor plan → parallel levels → what one instruction consumes.
At each level: what does each side lack or add, and whose decision caused it?
The second half reassembles them and runs FlashAttention on both, side by side.
Both chips are dual-die packages. One difference shows only at the package; skip to the core and you miss it.
It counts cells: two MXUs, 2 × 256² = 131,072 in all.
But each cell does 2 MACs per cycle; the figure omits this.
⚠️ “2 MACs” isn't “a multiply + an add.”
One MAC is itself multiply + add: a × b + c, one cycle, 2 FLOP.
So “2 MACs per cell per cycle” = 4 FLOP, not 2. Miss it: off by 2×.
Common error, 1 MAC/cell/cycle:
131,072 × 2 core × 2 FLOP × 2.2 GHz = 1,153 TFLOP/s, half the header's 2,307. “2 per cell” is missing.
§3.4 walks the full chain.
One level deeper. First what the GPU has, then what the TPU lacks. Reversed, the TPU looks like a “simplified” GPU.
The paper lists the omissions in one sentence (Jouppi et al., ISCA 2017, In-Datacenter Performance Analysis of a Tensor Processing Unit, arXiv:1704.04760, p. 8):
the single-threaded TPU has none of the sophisticated microarchitectural features that consume transistors and energy to improve the average case but not the 99th-percentile case: no caches, branch prediction, out-of-order execution, multiprocessing, speculative prefetching, address coalescing, multithreading, context switching, and so forth. Minimalism is a virtue of domain-specific processors.
What the savings became, per the abstract:
The lack of such features helps explain why, despite having myriad MACs and a big memory, the TPU is relatively small and low power.
⭐ The hardest number is in the Figure 2 floor plan: die area is data buffers 37% · compute 30% · I/O 10% · control logic just 2%. The next sentence:
Control is much larger (and much more difficult to design) in a CPU or GPU.
“Simple” on a floor plan: 2%.
On “stability,” the official word differs: not yield but deterministic execution. The paper argues CPU/GPU time-varying optimizations (caches, out-of-order execution, multithreading) help average throughput, not tail latency; TPU determinism better fits their 99th-percentile response time targets. ⚠️ No source found on yield/reliability, so no such claim.
① Warp scheduler: the part, GPU Microscope Section 2;
at work, Section 7 (“Hiding latency: one swaps workers, one plans shifts”).
② Which silicon a thread / warp is pinned to: Section 3, each abstraction layer matched to silicon.
③ How an SM uses registers: Section 2. Preview:
256 KiB / SM split four ways, 64 KiB per processing block, ≤ 255 registers per thread;
so large not for speed but to hold dozens of contexts at once.
④ How L2 is used: capacity and partitions, Section 1; a value's stops between HBM and the MAC units, Section 6.
⑤ How SMs cooperate: Section 3, clusters / distributed shared memory;
it exists because the 148 SMs have no fast path between them but L2. The SM figure's cta_group::2 is a tighter tier:
two SMs' Tensor Cores team up on one matmul.
148 SM × 228 KiB = 33,744 KiB, about 33 MiB. Next: on-chip real estate. §3.6's whole fusion payoff comes from intermediates staying there. Three questions: how big, how fast, how long you wait, a subsection each.
Fair challenge. Every on-chip SRAM on both chips, none skipped:
| Role | Who decides | GPU B200 (chip) | TPU v7 (chip) |
|---|---|---|---|
| L2 cache | Hardware | 126 MB ≈ 120 MiB | No such level |
| Thread contexts (registers) | Hardware / compile time | 37 MiB (256 KiB × 148) | None: nothing resident |
| Matrix operand staging | Explicit moves | TMEM 37 MiB | Accumulators 4 MiB |
| Software-managed scratchpad | The compiler / you | L1+shared 37 MiB ≤ 227 KiB per thread block |
VMEM 128 MiB + SMEM (scalar) 2 MiB 64 MiB block per core |
| Total | — | ≈ 231 MiB | ≈ 134 MiB Excl. vector regs (unpublished) |
Capacities from the two Microscopes' source notes (L2, shared memory: third-party tests/specs; VMEM 64 MiB / core, SMEM 1 MiB / core: JAX open-source code).
⚠️ TPU's SMEM is scalar memory (scalars, DMA descriptors), not CUDA's shared memory; in this table VMEM is the counterpart. Totals are our sums: 37+37+37+126 MB→120 MiB = 231 MiB;
64×2+1×2+1×2×2 = 134 MiB.
64 MiB ÷ 227 KiB ≈ 289.
The GPU's 37 MiB isn't one block but 148 mutually invisible 256 KiB islands, L1 taking part of each: one thread block can explicitly manage at most 227 KiB. xla_tpu_scoped_vmem_limit_kib; raising it when things don't fit is routine).
The compiler tiles against this limit, not the 64 MiB.
xla_tpu_scoped_vmem_limit_kib).
⚠️ The default is unpublished; official tutorials often set it to
65536 (= 64 MiB), implying a much lower default.
§3.2b covered size and ownership. The other half is speed, with a result almost everyone guesses wrong.
| Level | Bandwidth | vs. HBM | Source |
|---|---|---|---|
| HBM3e (off-chip) | 8.0 TB/s | 1× | Official |
| L2, local | 21 TB/s | 2.6× | Third-party |
| L2, cross-die | 16.8 TB/s | 2.1× | Third-party |
| L1 + shared (chip total) | ≈ 35 TB/s | ≈ 4.3× | 128 B/cyc/SM Measured |
| TPU VMEM | Unpublished. Google says only: to the MXU, “significantly higher” than HBM. Order of magnitude: our estimate | Magnitude only | |
⚠️ ≈ 35 TB/s is derived; 128 B/cycle is a ceiling, not a guarantee. The B200 clock is uncertain too, hence “≈ 35.” Derivation, backing, caveats below.
Derivation: 32 banks × 4 B per bank per cycle = 128 B/cycle/SM,
× 148 SM × 1.83 GHz = 34.7 TB/s.
128 B/cycle has third-party backing: a Hopper microbenchmark (arXiv 2402.13499 Table V) measured on H800 shared memory 127.9 and L1 125.8 B/cycle/SM—one ceiling, so one data path. Two more public sources independently give 128.
Cross-generation anchor: the formula gives H100 132 × 128 × 1.755 GHz ≈ 29.7 TB/s; a third party reports ~33 TB/s for H100 (implying ~1.95 GHz): magnitude matches.
⚠️ Two caveats: ① the B200 clock is uncertain (1.83 is a third-party boost figure; another measurement implies ~1.99), hence ≈ 35 TB/s;
② 128 B/cycle is a ceiling, not a guarantee: in the same table the RTX 4090 gets only 63.7 with scalar FP32, 121.2 with FP32.v4; the H800 gets 125.8 with scalar loads.
Too-narrow accesses can't saturate it.
Which GPU level is VMEM? None. By behavior, shared memory (software-moved, no tags, never misses); by capacity, L2-class (128 MiB vs 126 MB; the chip's shared memory totals 37 MiB). Precisely: the TPU built shared-memory behavior at L2 size, then deleted the L2 level.
① L2 isn't “split in half per die”; it's 4 partitions, two per die (twice Hopper's count). A die uses the other die's L2 with no effort from you: NV-HBI (10 TB/s) joins the dies into one CUDA device with globally coherent L2. The cost is in the table: 21 drops to 16.8, and latency rises. §3.1's “you pay, you just don't see the bill,” made concrete. The TPU is the opposite: two halves, 2 devices; using the other half is an explicit step you write.
② SMs exchange data in three very different tiers:
In a cluster (≤ 8 blocks, opt-in 16 since H100, same GPC):
distributed shared memory, reading and writing other blocks' shared memory directly, bypassing L2. Blackwell adds a tighter tier, cta_group::2: two SMs' Tensor Cores on one matmul.
Outside a cluster, only L2 and HBM: no other fast on-chip path, so sync goes through atomics, cooperative groups, or splitting the kernel.
⭐ Remember the direction of cause and effect: clusters were invented precisely because the 148 SMs have no path between them but L2.
It's a layer wedged in between “one SM” and “the whole GPU.” The TPU doesn't need it: a chip has only 2 TensorCores, so core groups never come up. Few cores remove a whole abstraction layer: scale, not design skill.
More detail in GPU Microscope Section 3.
§3.2b asked “how big,” §3.2c “how fast.” One axis remains: how long you wait. It's the one the GPU itself can't answer.
The question: a warp needs data; not in L1 → check L2 → not there either → fetch from HBM. How long does the chain take?
First, two coordinate terms. This section and §3.4 rely on them, and both names mislead:
The die lost components. One level up, the programming model loses things too: same cause, again.
It comes down to one question.
So you can't name anything. You can't write “SM 3, do this.”
All you can do: describe relationships. What must stay together, what can land anywhere.
The batch that “must stay together” is a block.
Why “must”? Because its pieces swap intermediate results,
in the SM's own small local memory.
Once a batch lands on an SM it can't move, or the intermediates are lost.
Work that never talks can land anywhere; all of it is the grid.
These two levels are yours. Your code says two things: these go together; this many in total.
Hardware has its own groupings, for another reason: saving circuitry.
Instruction fetch is costly. So 32 data elements share one.
That bundle of 32 is a warp. The number is fixed in silicon.
Four levels: bottom two fixed by hardware, top two set by you.
Two more came later, each forced by a specific problem:
warpgroup: newer matrix instructions are shaped for four warps issuing at once.
cluster: neighboring SMs want direct access to each other's local memory, no detour.
All six side by side, and what forced each.
The TPU fixes everything at compile time. No “you don't know.”
It needs not one level of that language.
The two sides' levels answer different questions.
GPU levels describe who must be together; the TPU never needs to.
So the table below is ordered by question.
CUDA's full nesting
thread → warp → (warpgroup) → block → cluster → grid
is folded below for reference.
What to remember: the three “mismatches” after it.
| Level | Size | Set by | Maps to |
|---|---|---|---|
| thread | 1 | — | 1 lane in a sub-core |
| warp | 32 threads | Fixed in hardware, not yours | 1 of 4 sub-partitions (sub-cores)/SM |
| warpgroup | 4 warps = 128 threads | Fixed by the ISA | One warp from each of 4 sub-partitions |
| block (thread block) (= CTA) |
≤ 1,024 threads = ≤ 32 warps | You, in code | All on one SM; never migrates |
| cluster | ≤ 8 blocks opt-in to 16 from H100 |
You, optional (Hopper+) | Several SMs in one GPC |
| grid | All blocks of a launch | You | All SMs (B200: 148, two dies) |
wgmma/TMEM need four consecutive warps to issue together.
Hence §3.4's trend of matrix instructions mobilizing ever more threads, until Blackwell's tcgen05 reverses it: a single thread issues it.
Warp comes from weaving: on a loom, the warp is the lengthwise threads pulled together, parallel strands that move as one. The threads are, literally, bound together.
Why 32? Wider amortizes fetch/decode better, but divergence costs more, coalescing coarsens. AMD long used 64 (the wavefront); newer designs went back to 32.
CTA stands for Cooperative Thread Array, and it is the thread block: CUDA calls the level a block; PTX and the hardware call it a CTA.
warpgroup does more than issue wgmma; it enabled
warp specialization: one warpgroup only moves data from HBM to
shared memory, the others only issue matrix instructions. Split movers from computers and the pipeline overlaps.
FlashAttention-3 builds on this: the GPU side's shape in §3.6.
Deeper still. “An SM” has been a black box. Open it and the numbers above need no memorizing.
A register is 32 bits (4 bytes), the unit of the compiler's “registers per thread.” Check: 65,536 × 4 B = 256 KiB, the register file in the figure.
Per-thread cap: 255 on current GPUs (it has changed; early parts had 63),
the figure's “255 per thread (max)” row.
⚠️ NVIDIA disagrees with itself: the Programming Guide says 255,
CUDA's own cuda_occupancy.h says 256 from Volta on.
I couldn't confirm the hardware truth (likely “256 allocatable, 255 usable,” my guess).
Use 255.
Registers are allocated per warp, in units of 256,
unchanged from Kepler to Blackwell: take “registers per thread × 32,” round up to a multiple of 256.
Per thread: round up to a multiple of 8.
So shaving a register or two often changes nothing; you must cross a multiple of 8.
Source: code, not docs; cuda_occupancy.h
lists these per compute capability.
64-bit values take two registers: doubles, pointers. Extra pointers cut occupancy fast.
Last, the bottom row.
A warp is 32 threads. Each has an index in the warp, 0 to 31. That position is a lane.
So the GPU has lanes too. In hardware a lane is a SIMD channel, same as a TPU lane.
The only difference: GPU lanes have names; TPU lanes don't.
A GPU lane has a writable index. Give it its own if and loop, like a real thread.
A TPU lane is out of reach: you can't write “lane 37, do something else”.
The vector moves as one; the compiler maps elements to lanes.
So a CUDA “thread” isn't your usual thread. To pin it down, set it between its two neighbors.
“A thread is a lane” is abstract. Ground it in a real matmul, one era at a time; mixing them confuses.
① Classic, no Tensor Cores. A thread owns one output cell (or a small tile). It runs its own K loop: loads A and B slices from shared memory into its own registers, multiply-accumulates, holds the sum, writes it back. It moves data and computes: most people's picture.
② Modern, with Tensor Cores: that picture is void.
The matrix instruction is one instruction issued by the whole warp,
on a fragment: one tile's elements, spread over 32 threads' registers in a fixed, PTX-defined layout.
In a typical m16n8k16, each thread holds 8 fp16 of A,
4 fp16 of B and 4 fp32 of accumulator C.
⭐ No thread computes a full output element alone.
Results return as a scattered fragment.
A whole warp issues each Tensor Core instruction, so “which cell does this thread own?”
is meaningless.
A thread is just a register holder and 1/32 of an issue.
That's what “a thread is really a lane” means.
⭐ Does a thread move data? Yes, but that role is retiring.
Pre-Ampere, threads moved data (global → registers → shared), busy meanwhile;
Ampere's cp.async goes straight to shared;
with Hopper's TMA one thread issues one descriptor per tile; 31 sit out.
In three generations the thread went from “mover + computer” to “register holder.”
§3.3b draws this path (three-generation figure);
Blackwell then moves the accumulator out of registers into TMEM.
That takeaway is about the programming model: the shape you see in code.
In hardware, the cost is three circuits, each answering one narrow question:
① Who can issue now?
② Who syncs with whom, and how?
③ How does data arrive with threads free?
On the first, we owe you a definition. “Scoreboard” came up several times (opening the SM, pricing exposed lanes) but was never defined.
In one sentence: a scoreboard is a table
of whether each instruction's operands have arrived.
Each cycle the scheduler scans it and issues a ready warp.
That's runtime-only knowledge of who can run, in silicon:
something must record what isn't ready. The TPU has no such table.
SB0–SB5):
a producer bumps one at issue, drops it at writeback; a consumer's 6-bit mask names which must hit zero.
Sourced vs. inferred. Sourced: the term comes from the CDC 6600, whose component was named Scoreboard. I found no source for why the word was chosen; what follows is speculation.
To keep score means not just tallying points but tracking the current state: a scoreboard at a ballpark lets you glance up once and know the situation, as this table does. So read it as “status board,” not points tally. On the GPU, though, the name fits literally: it counts, up on issue and down on writeback; the old board just raised a “busy” flag.
More importantly, the boards differ. The CDC one was a central table
tracking hazards on all registers, even write-after-write (WAW: an earlier write landing late over a later one),
all checked at runtime.
NVIDIA's has six counters per warp, asking only “is what I'm waiting for back?”;
fixed-latency dependencies skip it; the compiler settles those in the control bits.
The name was inherited; the mechanism shrank.
A footnote to the course's thread: what compile time can decide shouldn't wait for runtime,
and even the GPU is moving that way.
The second: a few hardware barriers in the SM. Warps arrive, hardware counts heads; when full, all go.
Mechanism folded at the end; the practical point:
A warp waiting at a barrier keeps its slot. So all that waiting frees no slots: a sync-heavy kernel's occupancy doesn't improve.
The third gets a figure. Watch which column changes, which never do.
cp.async drops the register hop; the thread still addresses.
TMA takes addressing: one thread fills a descriptor, issues, moves on.
⭐ Now it's the TPU's compiler-issued DMA in kind; one gap left: issued at runtime by a thread, or fixed at compile time?1. Syncing within a block.
Behind __syncthreads() is the SM's small pool of hardware barriers above. Three easily missed points:
① it counts warps, not threads; a warp moves as one;
② it doubles as a memory fence: shared-memory writes before it are visible to all after it,
no separate fence;
③ a warp waiting at the barrier still holds its slot.
(Already in the main text.)
Ampere added asynchronous barriers, splitting “arrive” from “wait.” A warp says “arrived” without blocking, works on, and waits only when it needs data. Early arrivals don't idle. Hopper lets waiters truly sleep; before, they spun on shared memory.
The two barriers also live in different places.
Old bar / barrier: a small fixed SM pool,
capped per CTA; mbarrier: a shared-memory object,
as many as shared memory holds.
Per the PTX docs.
2. How a cluster reads another SM's shared memory. Three things must hold:
① a cluster's blocks are guaranteed co-scheduled by hardware;
② each GPC has a dedicated SM-to-SM network;
a cluster must fit in one GPC because that network stops at its edge;
③ cluster.map_shared_rank(ptr, target_rank)
gives the other block's address (PTX mapa);
plain loads/stores then use it, and the network routes.
Both sides are taken apart. Still unsaid, and more basic: in what order does hardware compute a matmul?
This may be the course's one spot where nearly everyone is wrong without noticing. The wrong version is too natural: it's how you do a matmul on paper.
The picture in your head: left row 1 times right column 1, summed pairwise, gives the top-left entry; that cell is done; next. Mathematically correct. Yet no real hardware, GPU or TPU, works this way.
| Shape | Reuse | Work/read |
|---|---|---|
| 1 × N (one row only) | N ÷ (1+N) | ≈ 1, even at N = 10⁶ |
| 128 × 128 | 16,384 ÷ 256 | 64 |
| 128 × 256 | 32,768 ÷ 384 | ≈ 85 |
⭐ In one sentence: same math; only which dims go to space and which to time differs. It drives everything: accumulator size, the tile as core knob, why the TPU must keep weights still. §3.4 next compares how big a tile one instruction eats: this choice in numbers.
§3.3 asked who issues the instruction. Now: how big a block does one instruction eat?
An internal detail that decides which head_dim is free.
16; the next figure needs it. — from the GPU Microscopehead_dim follows. — from the TPU MicroscopeIn both, look at the instruction's depth along the summed dimension.
GPU: 16. TPU: 256. A 16× gap.
That is the contraction dimension: the one a matmul sums away, i.e. terms accumulated per instruction.
One rule for the letters M/K/N: K alone appears twice and alone disappears (summed away).
The other two shape the result.
⭐ The trouble with 128: it falls between two stools, too small for the array's edge (half of 256), too few streamed rows to amortize the weight load.
N wastes half too, is transposing equivalent? (open if asked)M/K/N straight, once
Write a matmul as A[M, K] × B[K, N] = C[M, N].
One rule covers all three letters:K is the only one that appears twice, and the only one that disappears.M = how many result rows, N = numbers per row.
K = terms summed into each number.
m16n8k64: output 16×8, each number a sum of 64 terms.
(The order follows BLAS: output shape M, N first, then K.)i, j, k, top to bottom:
for i (each result row) → for j (each number in it) → for k (add term k)
M, N, K = trip counts of
i, j, k.
k is the innermost loop: it accumulates.
So “K = contraction dimension” isn't convention: it names the accumulation loop.
M K · K N → M N: adjacent letters must match, then both vanish.
The rest is the answer.
M is time; K and N are space.
On a systolic array the K×N block is loaded and held: K is the array's height, N its width,
both fixed by hardware. M is just how many rows flow through,
as long as you like.
K=128 and N=128 each waste half,
yet M never comes up: it takes no space, only time.N=128 also keep only half? Same math, different reason
Picture the array as a 256 × 256 grid of cells.
For A[M,K] × B[K,N], the hardware
lays B's K×N block onto the grid (K high, N wide), then streams A's rows in from the left.
K=128 (half the height): the bottom 128 rows have no input above them.
Fill them with something else? No: a column's cells add into one number, so anything else corrupts it.
N=128 (half the width): the right 128 columns have no consumer below.
They compute output channels nobody needs: your output has 128, no more.
utilization = (K ÷ 256) × (N ÷ 256).
One side at 128 (half), the other full: 50%.
⚠️ Derived from the array structure, not official, but it agrees with the official line: at most 50% at head_dim 128.
N short: recoverable. If another matmul takes the same A, concatenate both B's along the width; one pass.
Fusing the Q, K, V projections into one matmul does exactly this: all three take the same X, so their N dims concatenate.
K short: not recoverable. K is summed: anything concatenated adds into the same result.
The core difference from N.
PV step's N=128
is the unrecoverable kind: that needs matmuls sharing a P,
and every head's P differs.N needs “same A, different B”; shared KV gives you “same B, different A”.
That stacks only along M, and M is time, not space: stacking never fills that half.
head_dim.
At 256, both wastes vanish.
But that is a model architecture decision, not a tuning knob. A model's head_dim caps its attention's share of peak here.
PV hits N=128; compute the transpose Vᵀ · Pᵀ instead and all three letters move:
M becomes 128; K and N become the two long dimensions.
(P · V)ᵀ = Vᵀ · Pᵀ.
Exact, not an approximation.
Vᵀ · Pᵀ equal (P · V)ᵀ? Step by step (shape → accumulation → numbers → cost)Step 1: the shapes match.
P is [q, kv], V is [kv, d]
→ O = P·V is [q, d]Vᵀ is [d, kv], Pᵀ is [kv, q]
→ Vᵀ·Pᵀ is [d, q][q, d] transposed is exactly [d, q]: shapes match; now check the numbers inside.
Step 2: cell by cell, the dot product uses the same two vectors.
Original cell: O[i, j] =
P's row i (horizontal, length kv) dotted with V's column j (vertical, length kv).
Transposed cell: (Vᵀ·Pᵀ)[j, i] =
Vᵀ's row j dotted with Pᵀ's column i.
By definition, row j of Vᵀ is column j of V,
column i of Pᵀ is row i of P.
⭐ Both sides dot the same two vectors, just in swapped order.
Step 3 (the crux): why swapping the order changes nothing.
Matrix multiply isn't commutative (A·B ≠ B·A): swapping changes “whose rows pair with whose columns”.
But here we transpose both at once; rows and columns swap too.
⭐ The two swaps cancel; the pairing is unchanged. Only the factor order in each term changes. They're scalars; scalars commute.
In one line:
O[i,j] = Σ P[i,k]·V[k,j] and (Vᵀ·Pᵀ)[j,i] = Σ V[k,j]·P[i,k]
Same products, same sum.
Step 4: numbers, matching term by term.
P = [ 1 2 3 ] V = [ 7 8 ]
[ 4 5 6 ] [ 9 10 ]
[ 11 12 ]
O = P·V, top-left: 1×7 + 2×9 + 3×11 = 7 + 18 + 33 = 58
Vᵀ = [ 7 9 11 ] Pᵀ = [ 1 4 ]
[ 8 10 12 ] [ 2 5 ]
[ 3 6 ]
Vᵀ·Pᵀ, top-left: 7×1 + 9×2 + 11×3 = 7 + 18 + 33 = 58
⭐ Not a coincidence of totals: the terms 7 + 18 + 33
match one by one.
Overall, O = [[58,64],[139,154]], and Vᵀ·Pᵀ = [[58,139],[64,154]], exactly the transpose of O.
Step 5: the amount of compute is identical.
Original: q × d outputs, each kv MACs → q·d·kv.
Transposed: d × q outputs, each still kv MACs → d·q·kv.
Not one more or less.
⛔ Then why is it faster? What changes is not the number of MACs but how they are laid out on the 256×256 grid.
M=q, K=kv, N=d=128
→ only half the array's width is usedM=d=128, K=kv, N=q
→ height and width fill; 128 moved to M⭐ Same MACs, different layout, fewer idle cells. That is all “move the 128 from space to time” means.
Vᵀ, and what gets loaded is Pᵀ.
But each head's Vᵀ must pair with its own Pᵀ: M can't be concatenated either, for the same reason N couldn't.
① Weights are shifted in; filling one tile takes 256 cycles.
Not stamped in at once: weights are pushed down row by row from a FIFO, rolling over the array like a shutter. The first-generation TPU paper: it takes 256 cycles to shift a tile in.
② Must the left matrix wait for the load? Usually not: double buffering.
Each cell has two sets of weight registers: one computes while the other loads the next tile. So the 256 cycles hide behind the previous tile's compute; the paper: “double-buffering (to hide the 256 cycles it takes to shift a tile in)”.
M rows ≈ M cycles.
M rows, M cycles, however many MACs. All 65,536 cells do their MACs simultaneously, at no extra time.
M = 128, compute takes 128 cycles, loading 256. Loading is slower; double buffering can't hide it.
You stall on weights.
③ Once loaded, how is it “held”? Each cell has one weight register; a value stays put until the whole tile is done. The name: weight-stationary.
④ Can both matrices stream? Yes: one of the three classic dataflows.
They differ only in what stays put in the cells:
B, so the array holds K × N; M streams through.
The TPU does this.M × N; K streams through.
Your question.A, so the array holds M × K; N streams. Output-stationary's advantage: no “load weights” step; both operands stream in together. The cost: results pile up in the cells and need a final pass to drain.
K and N, time is M
M and N, time is K
M and K, time is N
head_dim = 128 falls between two stools.
One figure, four bands: stationarity, tiling a real size, ping-pong loading, segments vs. reloads.
Q projection runs on the MXU
(M=128K, K=7168, N=1536)The three numbers.
Left X: [131072, 7168]; right W: [7168, 1536]; result Q: [131072, 1536].
① Block size? No choice: 256 × 256.
That is the array's size; the right matrix is cut to it.
② How many tiles in total?
K direction: 7168 ÷ 256 = 28N direction: 1536 ÷ 256 = 628 × 6 = 168 tiles.W is only 7168 × 1536 × 2 B = 21 MiB, so it all fits in VMEM in one go (VMEM has 64 MiB). Once.
③ The timeline. Feed the left matrix in 2048-row segments:
load B(k=0, n=0) ← 256 cycles; 1st tile waits
↓
stream 2048 rows of X ← 2048 cycles
(meanwhile load B(k=1,n=0)) ← 256, hidden in 2048
↓
stream 2048 rows of X ← using B(k=1,n=0)
(meanwhile load B(k=2,n=0))
↓
… after 28 passes …
↓
2048 rows × 256 out cols fully accumulated → write
↓
n=1: 28 more passes … all six n done → 2048 rows done
↓
next 2048-row segment → all 168 B tiles again
④ Does streaming wait for the load? Only on the first tile. Then double buffering overlaps loading the next tile with computing this one. Hiding needs “rows streamed > 256”: here 2048, 8× headroom.
⑤ Is each B tile used only once? No. It is reloaded once per left-matrix segment.
| Segment rows | Accumulator | B loads | Hidden? |
|---|---|---|---|
| 256 | 0.25 MiB | 86,016 | No (256 ≤ 256) |
| 512 | 0.5 MiB | 43,008 | Yes |
| 2048 | 2 MiB | 10,752 | Yes |
| 8192 | 8 MiB | 2,688 | Yes |
⭐ Why left-matrix segments can't be short: longer segments, fewer B reloads, but the accumulator stays on-chip and grows with them. So on-chip capacity caps segment length: the stovetop again.
⑥ Then stream all of M in one go, loading B only 168 times?
No: the accumulator would need
131072 × 256 × 4 B = 128 MiB,
more than all of VMEM.
So M must be segmented, and segmenting reloads B. Not a weak implementation, just arithmetic.
M has 130K rows; any segment is far over 256, so loading hides.
Projection-type matmuls never have this problem.
M = 128, below even the 256 threshold.
Same mechanism, opposite outcomes.Plug in a real shape to feel the two numbers.
Attention's head_dim is commonly 128.
GPU: 128 ÷ 16 = 8: eight instructions, zero waste.
TPU: 128 against 256: half filled, half empty.
Why not fill the empty half with other work?
You can't. The contraction dimension is an accumulation chain: anything you add lands in the same partial sum.
Why doesn't the GPU pay this?
Not because it has a scheduler.
Swapping warps fills time bubbles, not a half-empty contraction dimension.
Its K is only 16; nearly every shape is a multiple of 16.
Price of a K this narrow, aligned at runtime: the five parts §3.2 found the TPU lacks.
⚠️ Top trap:
16 and 256 are instruction shapes, not work per cycle. Divide them and the peak comes out absurd.
4,096 × 2 FLOP × 148 SM × 1.83 GHz = 2,219 TFLOPS vs. official 2,250; the 1.4% gap: clock uncertainty, not the formula.
262,144 FLOP/cycle/MXU × 2.2 GHz × 4 MXUs = 2,307 TFLOPS262,144 FLOP ÷ 65,536 cells = 4 FLOP per cell per cycle, i.e. 2 MACs: one MAC = one multiply + one add = 2 FLOP, and two make 4.mma.sync's m16n8k16 totals 16 × 8 × 16 = 2,048 MACs; one Tensor Core does 1,024 per cycle, so this instruction occupies it for 2 cycles. A package of work, not one cycle's.
The largest, tcgen05.mma (128×256×16), is 524,288 MACs; at 4,096 per SM per cycle → 128 cycles; 64 with an SM pair.head_dim = 128, bf16, K/V double-buffered, scores in fp32; real kernels keep Q and the accumulator in registers, so this is an order-of-magnitude estimate, not a byte count.)
| Br × Bc | 2×(K+V) | Q | Scores | Total |
|---|---|---|---|---|
| 64 × 64 | 64 KiB | 16 KiB | 16 KiB | 96 KiB |
| 128 × 64 | 64 KiB | 32 KiB | 32 KiB | 128 KiB |
| 128 × 128 | 128 KiB | 32 KiB | 64 KiB | 224 KiB ← at the cap |
| 256 × 256 | 256 KiB | 64 KiB | 256 KiB | 576 KiB ← over cap |
| 512 × 1024 | 1.0 MiB | 128 KiB | 2.0 MiB | 3.1 MiB |
| 2048 × 2048 | 2.0 MiB | 512 KiB | 16.0 MiB | 18.5 MiB |
128 × 128's 224 KiB
fits right at the ceiling; one step up is 2.5× the cap and overflows.
2048 × 2048's 18.5 MiB fits with two-thirds to spare.
So far: what one side has and the other lacks. Two pieces remain, one grown by each side.
The question isn't what they are, but: the product page says “supported”; can you use it?
Two examples, one per side.
GPU side: block-scaled quantization. The marketing line: “B200 supports FP4”.
True. But block-scaled instructions have two paths,
on different dies; B200 has only one.
(A warp-level instruction needs sm_120a: consumer dies only. tcgen05.mma's block_scale needs sm_100a: B200's only path. Figure: L300 G-5.)
⭐ Block scaling in hardware (NVFP4 vs. MXFP4, Blackwell's three additions, training-side 16×16): see Topic 8 · Precision and quantization.
TPU side: SparseCore, an embedding-lookup core. The marketing line: “the TPU has a core just for embedding lookups”.
Also true. But it pays off only when the same rows are looked up repeatedly. Recommenders, hammering a few hot rows, are its home turf.
Language models repeat too little (not zero). A 100K+-token sequence: only a few thousand distinct words, mostly the top 1–2K rows. Real repetition, but orders of magnitude below search/ads/recs, whose concentration sets SparseCore's break-even. This doesn't pay. The compiler never routes them there: zero table lookups in production.
SparseCore All Gather.
So how does it handle irregular memory access? Take the core apart.
pltpu.get_tpu_info().sparse_core). ⚠️ “VMEM or HBM?” is a false choice: it uses both, but the VMEM is its own (one per subcore + one shared; OpenXLA: SPMEM), not the TensorCore's. ⭐ Latency hiding: not faster fetches, but 16 streams chasing their own addresses, many in flight. Now what it is: lift the lid.
In XLA, the “collectives” line is a set of flags. Here they are, so “offloading” is tangible, all from public sources:
| flag | Turns on | Key point |
|---|---|---|
| ① Main switches: collectives → SC | ||
--xla_tpu_enable_sparse_core_collective_offload_all_gather |
All-Gather to SparseCore | On Ironwood (v7) these three default to true. So it already works for you, no flags set |
…_offload_reduce_scatter | Reduce-Scatter | |
…_offload_all_reduce | All-Reduce | |
…_offload_2d_all_gather |
All-Gather on a two-level mesh | Must be enabled explicitly |
| ② Base: main switches need these | ||
--xla_tpu_use_tc_device_shape_on_sc |
SC reuses the TensorCore's device shape | MaxText groups these four as
ENABLE_SPARSECORE_OFFLOADING_BASE_FLAGS, set to true / false / false / true.
The trap: the main switch alone often does nothing |
--xla_sc_enable_instruction_fusion | SC instruction fusion | |
--xla_sc_disjoint_spmem |
Disjoint per-tile blocks of shared SPMEM? | |
--xla_sc_disable_megacore_partitioning |
No megacore split | |
| ③ Visibility: without these, that stretch of the profile is dark | ||
--xla_tpu_enable_all_gather_offload_tracing…_reduce_scatter_offload_tracing…_all_reduce_offload_tracing |
Traces the offloaded stretch | Only when tuning. “Read the trace, not the product page” needs the lights on first |
| ④ One that isn't SparseCore but always gets tuned alongside | ||
--xla_tpu_scoped_vmem_limit_kib |
Max VMEM per op; the rest goes to the next op's prefetch | MaxText public recipe: dense models 98304, MoE 81920 |
⚠️ Two constraints.
First, Continuation Fusion and SparseCore offload are mutually exclusive; MaxText's comment: “Either one of CF or SC can be enabled at a time”. Both overlap comms with compute by routes that don't stack;
“both on is faster” is the commonest misuse.
Second, 98304 KiB = 96 MiB, larger than v7's 64 MiB of VMEM per core. The recipe uses it as a cross-generation limit request; the target machine caps the real value.
Reported as is, without guessing why.
| NVIDIA Tensor Core | TPU TensorCore | |
|---|---|---|
| What it is | A unit inside an SM | The device's main compute core |
| Contains | Just the matmul array | MXU + VPU + scalar unit + VMEM |
| Per chip | 100+ SMs, each with several | On v7: 2 |
D = A×B + C. Each came from a class question. Safe to skip; each pins down an easily blurred point.
src/maxtext/configs/README.md and
benchmarks/xla_flags_library.py.
Continuation-state passing between async collective fusions:
public tech blogs; neither NVIDIA nor Google has published a formal CF design doc.
D = A×B + C.
| GPU gen | Issued by | Operands in |
|---|---|---|
| Volta – Ampere | One 32-thread warp, in sync | Private registers, one fragment each |
| Hopper | One 128-thread warpgroup, async | A straight from shared memory |
| Blackwell | One thread, fire and forget | Shared + Tensor Memory (incl. accumulator) |
The Blackwell row's “issued by a single thread” is from Colfax Research's Blackwell Tensor Memory tutorial (“can and must be issued by a single thread”; the PTX ISA officially says “single thread semantics”). The reason: none of its data sits in private registers; it all lives in CTA-shared memory, so the matrix unit is decoupled from the warp scheduler.
So does the TPU have MMA? Mathematically yes, formally no: that gap is this lecture's thread.⭐ The takeaway: Blackwell's moves (accumulator out of registers, single-thread issue, operands in dedicated memory) point to the TPU. For matmul, it deliberately dropped “a fragment per thread” dynamism for bigger tiles, less scheduling overhead. Elsewhere, the GPU stays the do-anything dynamic machine. The paths haven't merged, but on the money-maker, they're a step closer.
Sources: NVIDIA PTX ISA docs (three gens:mma / wgmma /
tcgen05.mma), CUTLASS Blackwell feature docs,
Colfax Research's Blackwell Tensor Memory tutorial.
“Blackwell is moving toward the TPU” is my judgment, not official from either side.
All the first half's parts now work at once.
Two jobs. First, walk one op end to end (§3.6). Then pay off two earlier claims: §3.6's end settles Section 0's “different everywhere, but only one cause”; §3.7 explains Section 1's coincidence: same-range peaks.
Two questions: how much fits in one pan (on-chip memory)? What holds each side back?
The op picked in Section 2 now runs once on each chip.
Now real numbers. First: why is tiling unavoidable? Section 2's “demand exceeds supply 5×” was a ratio; here are absolute sizes.
How does each side delete it? We walk each path alone, then side by side. On purpose: a table up front makes everything look different; walked separately, most stops match.
Now side by side: the main figure of the lecture.
128×128 bf16 tiles (Q/K/V) = 3 × 128 × 128 × 2 B = 96 KiB; add double buffering and you're near the limit.head_dim=128 is exactly 8 × 16 on the GPU, every cut clean; on the TPU it fills only half the 256-wide edge.
head_dim = 128 meets a 256-wide MXU in both matmuls: as contraction dim in QKᵀ, output dim in PV.
So MXU utilization caps at 50%, and no kernel parameter fixes it.
That holds at the kernel-parameter level. The real ways out (§3.4): change head_dim (a model-architecture call, not tuning: 32 heads × 128 → 16 heads × 256 measured +21% / +32% / +46% MFU at 8K / 16K / 32K), or a fully transposed pipeline.
head_dim = 128 falls in between: too small to fill space, too short to serve as time.
The far smaller GPU array fits 128 any way: it hits capacity; 128 can't fill the TPU's: it hits shape.
(Where this test comes from, and why all three dataflows fail it: the two folds in §3.4.)⭐ The rest is FlashAttention's story: the inner/outer-loop figure, why versions swapped loops, measured tile sweeps, three-layer efficiency ceilings. All in Topic 3 · The evolution of attention, Section 3. Here it's just the vehicle.
Last figure. Ignore the stops; count what you must write, the coder's side.
cp.async/TMA, double buffering, warp roles, sync, L1 use.BlockSpec and grid. The shorter column is the conclusion. Red band (most overstated): L2 isn't bypassed; reliance on it is.Section 1 set the peaks side by side (2,307 vs. 2,500), same range, no why. Now we compute it. The first half's two most different numbers cancel.
Watch the tally on the right. The knife has two sizes, both shown; don't merge them. Blade width (bite depth): §3.4's contraction dimension, 16 vs. 256. Cut per stroke (MACs/cycle): 1,024 vs. 131,072 (one Tensor Core vs. one MXU), 128× apart.
148 ÷ 128 Not approximate, exact: 148 ÷ 128 = 1.156, and 606,208 ÷ 524,288 = 1.156.
This is background for Section 6's measurements.
Those are 64 vs. 64, and the two sides wire them
in structurally different ways.
Hardware only: link count, width, shape, worst-case hops.
Which patterns suffer, and why expert parallelism (EP) is most topology-sensitive, is about sharding: Topic 5.
In one line: NVL72 is a switched domain; ICI is a direct-connect network.
The domain is capped at 72 chips, but any pair inside is one hop, wherever they sit.
The network has no switches and links only neighbors, but one protocol scales to 9,216 chips.
64 sits where they differ most:
under NVL72's cap, all one hop;
on ICI, a 4×4×4 torus, 6 hops at worst.
But reach is not what you actually get. Below are our own numbers: not vendor specs but the measured drop.
⚠️ Know what these are: different evidence from the two figures above. Those show structure; these are one side's measurements: GPU only, 8 to 32 GPUs in one NVL72 domain, not a comparison. We never ran the TPU counterpart (no torus collective measured). One point: “no distance in the domain” does not mean “no cost in the domain.” Which pattern suffers on which topology is again Topic 5.
You've met this pair four times:
§2.1's “4 vs. 0”; §3.3's runtime-only layers;
§3.3b's three circuit sets; §3.6's two forking stops (of six).
Each hit this decision on a different part. Now the decision itself: who schedules, and how that flows from hardware to your code.
The bottom layer is clearest. A load takes hundreds of cycles; who fills them? It's about responsibility, not speed.
The difference shows up verbatim in code.
threadIdx; JAX (right), no index. It's in the code, not adjectives. Bottom columns equal on purpose: trade-off, not ranking.Collectives, one level up, follow suit.
GPU: you hand-write all-reduce as the training loop's second step.
TPU: you never write it; the compiler inserts it from your declared parallelism.
(Code pair: L300 §5.3.)
Natural reaction: “Why not leave it all to the compiler?” You can't; the line is clear.
O(n·d), negligible next to O(n²·d).
“Keeping the math” means equivalent results, not zero extra instructions.Now the strongest objection, half right: record a whole step's graph once, run it straight through: the gains are real, often an order of magnitude.
What it overturns depends on why it's fast. The usual answer is wrong: it saves not “branches” but per-kernel launch CPU cost. So: which layer does it freeze?
cudaGraphInstantiate:
instantiate, not compile. Accurate.
It deserves more than CUDA Graph: it points the right way.
The trend down §3.3b's four generations:
cp.async skips registers;
threads still compute addresses.tcgen05 goes furthest.
One thread issues MMAs for the whole CTA;
operands and accumulators live in a dedicated 256 KiB/SM matrix scratchpad;
results never return to registers.Four GPU generations on: whole blocks, by descriptor, no threads held, into a software-managed scratchpad. Exactly the day-one TPU.
§3.6's six stops are done, but the two numbers below aren't from them.
FlashAttention was structure and theoretical bounds (§2.3's figure notes “real Flash re-reads K/V; intensity never reaches 65,536”); the only like-for-like measurement: one pair of full-model Hunyuan3-295B runs.
So they pay off Section 1's line, not §3.6's side-by-side.
First how to read them, then the numbers.
Both usual comparisons fail. Peaks: what the chip can do, not your job's speed. MFU: pick a denominator, pick a conclusion. Only the figure's tier ③ stands.
⚠️ Procurement wants the figure's ④ (per dollar / per watt), but no shortcut: ④'s numerator is ③. Without ③ measured, ④ is a ratio of two unknowns.
854.0 ÷ 2,700 = 31.6%
(committed 2026-08-05, stood for 26 days)
854.0 ÷ 2,500 = 34.16%
(corrected 2026-08-31, the current figure)
That figure covered only the denominator. The numerator has two named flavors, both in use; unless you know which was reported, you can't compare.
One line: numerator = what the model's math needs, or what the hardware actually ran?
The figure's 4/3 is an upper bound (full-layer recompute).
Where does ours land? FlashAttention's backward recomputes the forward S once, wasting 1 of 7 matmuls (2 forward + 4 backward + 1 recompute):
7 ÷ 6 ≈ 1.167×, 85.7% useful.
Both 1.167 and 1.33 are derived by definition, not measured. Overheads (16.7% vs. 33%) differ 2×; “recompute enabled” alone says nothing.
312.5 × 70% = 218.8.
Useful intensity is just 219; with recompute it's already 312.
You think there's ample memory-bound headroom; you're already over.This explains item ③ of §6.3's closing figure (“what this ratio still lacks”): the recompute policies differ entirely.
Ticket in hand: skip the big numbers; read the two red segments first.
Hardest step next: not overstating this number.
What does 1.31× still lack? List yours, then check below.
⚠️ Units first: the figure's 1.29 is per-chip TFLOP/s (854.0 ÷ 662.3); from here on we use per-chip token/s,
1.31 (6,242 ÷ 4,770). The gap is exactly the figure's 1.5%.
Now the opening's open question. One sentence answers it; the rest qualifies.
From package to measurements, both chips are now taken apart. The opening promised “different everywhere, one cause”; in §3.6, four of six rows matched. Now, the cause.
That's the course. No buying verdict, but given “N× faster,” you'll know the four questions and where its denominator and numerator are. Recall §6.2's two utilizations: same denominator, all the gap in the numerator.