How to use these notes
These are the presenter's notes, not student material. Students read the
L200 course page; you read this.
Course author: Chris Yang · Google Cloud AI Infra architect.
| 🎯 | What this segment must leave behind The one sentence students should keep. Just one. Scattered means not landed. |
| 🗣 | Script Near-verbatim; read it as is. Yellow lines are said word for word; reword the rest. |
| 🖥 | Screen Where to scroll, and which cell of which figure to point at. |
| ❓ | Likely questions Real questions, real answers. No answer? Write “I'm not sure.” Invent one and you'll read it on stage. |
| ⚠️ | What not to say Where this segment drifts, and where the time goes. |
What the stage directions mean
These terms recur. Each is a concrete action, not an adjective. Do them as written; don't go by feel.
| Dead stop | After the last number, stop talking and count two beats. Not “pause a little.” Your default on stage is to keep going; without a count, you'll slide past. |
| Read it out | Verbatim on screen; say it in plain words, one number per line. Raw strings (errors, instruction names) can't be read aloud. |
| Read, don't explain | Then add a forward pointer: “Hold on to this; we'll use it when ××.” A setup with no payoff is noise. Name which section pays it off. |
| Slow down | Split the sentence in two, with one break. You can't control “slower.” You can control the break. |
| Really wait | Wait for a voice or hand, however long. None? Call on someone. Answering yourself wastes the 30 seconds. |
| Point at | Then read out the words in that cell. Don't make the room hunt. A gesture carries no information; your voice does. |
⛔ One general rule: any line explaining why it's taught this way (e.g. “the surprise survives intact, and the line holds up under questioning”) is for your judgment. Don't say it. It pulls the room off the content and onto your method.
⭐ The thread through the whole lecture: operator fusion
The surface topic is hardware differences. The real topic: should an op be merged with its neighbors into one kernel, and who decides? Learn this table. Don't give fusion its own segment. Name it once at each of these five points:
| Where | How to name it here |
|---|---|
| Section 1 | Line drawn, not yet used: name it. The only way to make a slow op fast is to shrink the denominator, best by keeping intermediates out of HBM. About 40 seconds |
| Section 2 | “Naive vs. Flash” is before and after fusion. Same FLOPs, 1,000× apart in intensity. The lecture's strongest fusion demo; nothing to add |
| Section 3.6 | The same fusion, landed on each side: tile size, and who tiles |
| Section 5 | ⭐ “Runtime or compile time” asks exactly who decides fusion. The lecture closes here. Everything before builds to this line |
| Section 9 | The close stays on this thread: no ranking, only who was handed the decision |
How the time is split
A plan, not a measurement. After each run, put real minutes in the generator script (it builds the table; one edit is enough).
| Minutes | Content | What this segment lands |
|---|---|---|
| 0–8 | Opening: two origins (roadmap figure gone) | Plant the claim “they differ everywhere, for one reason.” Still a claim, not a conclusion |
| 8–18 | Section 1 · The dividing line: compute ÷ bandwidth gives both the same number | First “sit up” moment: 312.9 vs. 312.5 exposes the spec sheet. At the end, name the thread: operator fusion, and say at once that it doesn't remove data movement, just moves it a level |
| 18–28 | ⭐ Section 2: the same load, 4 runtime decisions on one side, 0 on the other | The claim becomes visible. Also the strongest fusion demo: same attention, same FLOPs, 1,000× apart in intensity |
| 28–70 | ⭐ Section 3a: chip → core → core floor plan → parallel hierarchy → one instruction's bite | Parts, not conclusions. At each level, point at “what differs, and whose decision caused it.” Conclusions come in 3b |
| 70–80 | ⭐⭐ Section 3b: FlashAttention walked on both sides + paying off §1's peak coincidence | The climax is in these 10 minutes. Cut anything else first. Cut this and the lecture becomes a spec comparison |
| 80–84 | ⭐ Section 4: 64 chips each. One NVL72 domain (one hop) vs. 4×4×4 torus (≤ 6 hops) | Prerequisite for Section 6's measurements: those two numbers ran on different network structures, never measured apart here |
| 84–96 | ⭐ Section 5 · Unpacking the word: runtime, or compile time | The opening claim closes: four appearances, one thing. That is: who each side handed fusion to |
| 96–104 | Section 6 · After the run: two numbers; comparable? Denominator: entry price. Numerator: MFU / HFU | The output isn't the ratio but “this number could lean either way.” ⭐ MFU and HFU get proper treatment here: the numerator-side metric, premise for row ③ of the “boundaries” figure |
| 104–109 | Section 9 · Close: so what is each good at? | Back to the first line: no ranking, only who decides |
The full core track runs 109 minutes (no skippable detour:
§4 is back as the prerequisite for Section 6's measurements).
⭐ Not a one-hour lecture. On 2026-09-03 we decided not to cut content.
For an on-the-hour slot, round up. Truly only 60 minutes? Use the emergency list below.
What to cut first when you run out of time
Fixed order. Don't improvise:
① Section 4 to two sentences (never skipped whole; Section 6 rests on it;
a glance per figure plus “one hop vs. six”; −4);
② Section 6: just the two-measurements figure + one sentence on boundaries (−4).
⛔ 6.2 (MFU / HFU) stays: the numerator-side metric;
without it, row ③ of “boundaries” is empty;
③ cut one figure from each of Section 3's first three subsections (−4).
⛔ Section 3.6 (FlashAttention side by side) and Section 5 are never cut.
Without them it's “two spec sheets compared,”
what the opening promised not to do.
Opening: two warm-ups + two origins
⏱ 8 minScript
OK, let's start. This lecture is “TPU vs GPU,” but up front: I won't read you a spec sheet. The spec sheet is the first thing we'll take apart.
⭐ Scroll a bit to the yellow multiple-choice box. Don't click any of the four options; clicking reveals the answer.First, a warm-up from last lecture. We took apart DeepSeek V3: 671B parameters total, 37B active per token. So what does MoE actually save?
⭐ Really wait here. Hands or shout-outs both work; don't click till someone answers. Not a knowledge test: it switches the room from listening to answering. Answer it yourself and the 30 seconds are wasted.The answer: it saves compute, not memory. Each token only runs the 37B path, so the compute saving is real. But all 671B of weights stay resident, every byte. You don't know which experts the next token picks, so they all sit in HBM.
Reveal the answer; point at the last paragraph (the ⭐ line) and read it.But the answer isn't the point. The idea behind it is:
memory and compute are separate resources, and they move independently. Last lecture counted them separately.
This lecture adds a third: bandwidth.
And as we'll see, the one that decides whether an op is fast is often neither of the first two.
One more. This one's pure guessing. Later we'll take both chips apart, layer by layer. First, a guess. On B200 the matrix unit is a Tensor Core; on v7 it's an MXU. Same job. How many does each chip have?
⭐ People will get this wrong, and all in the same way. Most assume the same ballpark. Let them be wrong. That's the point.B200 has 592. v7 has 4. A 148× difference in count. But here's the real surprise: in MACs per cycle, the two chips differ by only 15%. Same compute: 592 small pieces on one side, 4 big ones on the other.
⭐ Read it and stop. Don't explain why. Point at the last line of the answer area and read it. The numbers are out; the “why” is an IOU. ⛔ Classic mistake: explaining it on the spot. The hook becomes a spoiler, and Section 3 loses its question. Let them carry “why hundreds vs. single digits?” through the first two sections. At the §3.7 count, someone will call out the answer.The whole lecture is two questions and one route. One: where exactly do these chips differ in hardware? Two: we run one op end to end and watch where each difference shows up. That op is FlashAttention.
Scroll one screen to the Section 0 figure. It has just two boxes. Nothing else to find. Make those two boxes clear, then move on.Here's the first figure. It's the lecture's only premise: these two boxes.
On the left, the GPU. It started as a graphics processor. From day one it had to serve tons of small, unrelated jobs. It doesn't know the shapes, the access patterns, or the control flow. So most of its die goes to handling any case: the cache hierarchy, the warp schedulers, the huge thread counts.
On the right, the TPU. It started as an ASIC built for matrix multiply, serving only neural networks from day one. Shapes are regular, access is predictable, control flow is static. So it needs none of that “handle any case” circuitry.
⛔ Easiest slip (I've made it myself): don't say “all the saved area went into compute.”The savings became neither compute nor on-chip storage. Section 3b rules out both with two tables: the GPU has 16% more total MACs and 73% more on-chip SRAM.
So what did it save? Complexity itself. No cache, no scheduler, layout fixed at compile time. The same compute in four big blocks, not 592 small ones. That's 148 times fewer sets of control circuitry.
⚠️ Someone will likely ask “So is it more power-efficient?” There's a solid official answer. Give it with its caveats.Yes, you can answer. Two things suffice. First, a quote from the first TPU paper. It lists everything left out: no caches, no branch prediction, no out-of-order execution, no multithreading, no context switching. The next sentence: “Minimalism is a virtue of domain-specific processors.”
⭐ Second, a number, which beats any adjective: “On that die, control logic takes just 2%.”Data buffers 37%, compute 30%, I/O 10%. Control: two percent. The paper adds that this part is much larger on CPUs and GPUs. ⛔ Then brake. All three caveats, every time:
One: the paper compares the 2015 TPU v1 with the K80. Back then the savings did become compute: a small die with 25× the MACs. For v7 vs. B200, Section 3b counts the GPU at 16% more. Same mechanism; ten years later, the opposite outcome.
Two: Ironwood's official 2× perf per watt is vs. the previous TPU, not GPUs.
Three: yield and reliability. No source, so don't say it. The official word is “deterministic execution”: tail latency, not reliability. Don't mix them. One beat. Next two lines: the takeaway.
One doesn't know what you'll run, so it hedges at every step. The other knows from day one, so it hedges nothing.
No more scrolling. Stay on the two boxes.So what is this lecture about? The two sides differ everywhere. But every difference follows from a single decision.
Which decision? Moving a block of data from HBM onto the chip: who schedules it. On the GPU, a cache guesses for you at runtime. The TPU has no cache; it's fixed at compile time.
Shapes padded to 8 by 128, how big a bite one instruction takes, who writes the kernels, even what wiring a whole machine costs: it all grows from that one decision.
Slow down. This is the lecture's claim; every later section is evidence for it. Rush it and it sounds like a slogan.One more thing, and I mean it: this lecture doesn't rank the chips. Both origins were the right call at the time, and each still carries its cost. You won't hear “which is better” today. Only “who decides this, on this side.”
That's the foundation. Straight into Section 1. It's one division, so it's quick. After that, Section 2 starts taking the hardware apart.
Likely questions
“Only one cause”: isn't that oversimplifying?
There are a dozen-odd symptoms. “One” means one cause. Caches, shape alignment, who writes kernels, interconnect wiring: all differ, but they're not a dozen parallel facts. They're a dozen consequences of one decision.
At the opening it's a claim, not a proven conclusion. Say so: “You don't have to believe it yet. Sections 3 and 5 each test it.” That convinces more than digging in, and gives them a reason to keep listening.
Does the TPU really have no cache at all? Then what is VMEM?
Precisely: the TPU has no hardware-managed cache tier. It has on-chip storage, plenty of it, but as a software-managed scratchpad: when data arrives, where it goes and when it leaves are all fixed by the compiler. Nothing decides anything at runtime.
That distinction is the main thread, so the question is a gift:
“Right. What's missing isn't capacity, it's who decides.”
⚠️ Don't go into VMEM size or partitioning at the opening. Section 3 has a figure for it.
Why make FlashAttention the star? Why not a GEMM, or a whole model?
It meets two conditions that rarely coincide.
First, it runs on one chip. No inter-chip traffic, so the story stays on one line (hence Section 4, on inter-chip links, never mentions it).
Second, it hits four things at once: memory access, shapes, on-chip capacity, kernel authorship. A pure GEMM is too clean to touch the last three. A whole model is too messy: the subject changes every step; nothing lands.
Section 4 has nothing to do with FlashAttention. Why is it in the core track?
It really is off the FlashAttention thread. The test is hard: FlashAttention appears 0 times in Section 4's main text (grepped). It finishes on one chip and never touches inter-chip links.
But it underpins Section 6's measurements: those numbers come from 64 vs. 64 chips, wired in structurally different ways. Without it, Section 6's ratio can't be read. So it stays in the core track; don't skip it whole. Short on time? Two sentences (emergency list, top).
What not to say
② No company history. The origins take two sentences (“doesn't know what you'll run” / “knows from day one”). Anything more is lost time.
③ Don't explain MXUs or warps here. Section 3 has thirteen microscope figures for that. An opening glossary is the costliest drift: ten minutes gone.
④ Don't say “which is better.” If someone steers toward chip selection, deflect with the course page's line: only “who decides this, on this side.”
If the audience is very mixed, cut origin detail, but always say “they differ everywhere, but every difference follows from one decision.” Without it, every later section is an isolated fact.
⚠️ This line isn't in the figure. It lives only in your mouth and the course page's green box. So don't drop it.
The dividing line: same appetite
⏱ 10 min⭐ By the end, also say what the line is for: judging whether fusion is worth it. The lecture's through-line gets its name here.
Script
I just said there's only one root cause. But before comparing anything, we need this machine's exchange rate between computing and moving data. That's all this section does: get that number, no conclusions.
Both spec sheets give two numbers: operations per second and bytes moved per second. Divide one by the other. The ratio is very concrete: in the time it takes to move one byte, how many operations can this hardware do?
This is easy to say backwards. Slow down. Not “312 operations need 1 byte of input,” but “while one byte moves, the machine can do 312 operations.” So it's a floor: for every byte moved, your algorithm must do 312 operations just to keep compute fed. Do fewer, and the spare compute sits idle. Stop on the two-panel figure (fig1-4). Don't scroll on. Let the numbers sit.On the left, TPU v7: 2,307 TFLOPS per chip, 7.37 terabytes per second (= 8 × 921.6 GB/s; this course uses 7.37 throughout; the 7.38 or 7.4 seen elsewhere is the same number rounded). That's 312.9.
On the right, GB200, the GPU in NVL72: 2,500 TFLOPS, 8.0 terabytes per second. That's 312.5.
Dead stop. Don't explain the numbers. Let the room notice they're the same. This is Section 1's only big reveal.⭐ How to stop (added after the 2026-09-10 run-through): after “312.5,” shut up and count two beats. “Dead stop” said stop, not for how long, and on stage the default is to keep talking. A cue with no concrete action is no cue. (Same lesson as the side track's XLA error: the notes said “Read it out,” but that English line can't be read aloud.)
Two companies, two architectures, two design philosophies. The ratios agree to the first decimal. FP8 too: 625.8 versus 625.0.
⛔ Never say “a relative difference of 0.13%.” That third digit is fake. GB200's “8 TB/s” has no stated basis, and the decimal and binary readings differ by 2.4%.Say this instead: “Two companies, two architectures, two philosophies. Compute divided by bandwidth lands in the same range.” Just as surprising, and it survives questioning.
⛔ That reasoning is for you; don't say it (I did, in the 2026-09-10 run-through). Rule: any “why I'm saying it this way” line belongs in the notes, not on stage. Said aloud, it pulls the room onto your method. Point at the blue line in the main text (“both sides have eight 24 GB HBM3E stacks; why different bandwidth?”). Raise it yourself; it's always asked. ⭐ One-line answer: “HBM3E” names the generation, not the speed grade.
Each stack has 1,024 data lines; bandwidth is that width times the per-pin rate. 8.0 to 9.6 Gbps all count as HBM3E in this generation. v7 runs 7.2, GB200 about 7.8. The whole difference is the pin rate.
Key point: the buyer picks the grade, for power, cooling and reliability. Not the chip's ceiling. ⭐ Bridge in words; don't jump cold (added 2026-09-10): lead in with “While we're here, there's a question I get every time, so I'll take it now.” Lead out with “And now the one thing I most want to say in this section.”
These were two adjacent stage cues with nothing sayable between: an abrupt jump.
⭐ This question turns the course's own test on itself.
⭐ Then the line that matters most: “And ‘binary or decimal?’ is one of the four questions Section 6 teaches. Here it hits my own headline number. Those four questions aren't for auditing other people. Aim them at yourself first.”
But the next sentence must follow: “both land on 312” is a fact. “Why both land on 312” is my inference. No vendor has explained it. And it depends on the SKU: the B200 on an HGX board gives 281.3, 11% off.
So from here on, 312 as a test means the qualified version. When you quote it, keep the qualifier, not just the number.
Scroll one screen: the figure continues with three unequal bars. Read, don't explain. 20 seconds.Quick aside: that division works at every level. HBM is 312, L2 is 119, shared memory is 64. The closer to compute, the lower the line.
Don't expand. The room doesn't know yet what it's for, so it won't stick. It returns with fusion at this section's end, and lands there.⚠️ But give the forward pointer: “Hold on to that. We'll use it at the end of this section.” (Added in the 2026-09-10 run-through.) A “Read, don't explain” with no signpost reads as a stray aside. A hook must say where it pays off, or it's noise. The side track hit the same rule: X1 planted a hook and named the wrong payoff section. Back to the two panels. Same figure.
Same ratio. Could numerator and denominator happen to be proportional? No. The panels give all four: compute 2,307 vs. 2,500, bandwidth 7.37 vs. 8.00. Both within 8%.
The section's real payoff: not just the ratio: numerator and denominator each nearly match. The “coincidentally proportional” escape route is closed.The spec table's ten rows look equally weighty. Only one kind is worth remembering: a level one side has and the other lacks entirely. The TPU has no hardware-managed on-chip cache level. The GPU has no programmable dedicated coprocessor level. The gap isn't capacity. It's who decides. Section 3 shows them; not a word more here. The rows are folded at the section's end, for lookup; don't open them now.
Scroll to the three-column figure “Using the line” (second figure); stop on column ①. The second time sink. Don't skim it as a wrap-up.We have the line. But 312 alone is useless. It's the hardware's number. You also need the op's number, so there's something to compare.
The op's number is arithmetic intensity, and the method is one sentence: count the bytes it moves, count the operations it does, divide.
The section's most practical sentence. Say it slowly. It's worth more than 312: 312 changes each hardware generation; the method doesn't. Scroll to the three-column figure “Using the line”; stop on column ①. The math is on it; read along.Let's measure two right now.
First, a vector add. y = a + b, in bf16.
Read a, two bytes. Read b, two bytes. Write y, two bytes. Six bytes moved.
Operations? One add. So the intensity is 1 over 6, or 0.17.
Against 312: over 1,800 times too low.
What does that mean? Run this line on GB200 with all 8 terabytes per second saturated, and you get 1.3 TFLOPS. The chip's peak is 2,500. That's 0.05% of it. Make compute ten times faster and it won't gain a second.
Pause and ask: “So would a more expensive chip make this line faster?” Let them answer “no.” Move to column ②.Second, a matmul. Square, N by N, bf16. The operation count is 2 N cubed. Moving three matrices in and out is 6 N squared bytes. Cancel, and the intensity is N over 3.
To reach 312? A side of nearly a thousand.
Flip it: a side-256 matmul has intensity of only 85, still bandwidth-bound. “Matmuls are compute-bound” is a myth. Only big ones are.
Heads usually come up here. It overturns something almost everyone assumes. Same figure, column ③. These 40 s begin the through-line. Don't rush.Now we can name the real protagonist of this lecture.
Both calculations said one thing: whether an op is fast depends on operations per byte moved. So there's one way to make a slow op fast: shrink the denominator. Move less.
And the best way to move less is to merge adjacent ops into one kernel, so intermediates never touch HBM. That's fusion.
Let it land: it doesn't save a single FLOP. Only the denominator changes. Many think fusion means computing less. It doesn't.So on this line, fusion pushes the op to the right, from the bandwidth side to the compute side.
Still column ③. Point at the yellow box below (312 → 64). Added because the last line gets misheard.I said intermediates never touch HBM. That's easy to hear as “the intermediates disappear.” They don't.
They're parked in on-chip scratch: the shared memory on the GPU, up to 227 KB; VMEM on the TPU. HBM traffic really drops, but the data still has to land on chip.
Pause: data movement didn't vanish; it moved a level. The 3 bars were for this.And the on-chip level has its own line. We read it earlier: 64, not 312. So after fusing, redo the math: compare the new intensity with 64.
Add three sentences, or the last one contradicts itself: 312 to 64 lowers the bar. It doesn't raise it.So why still hit a wall? The denominator moves too, not just the bar. Fusion turns HBM traffic into on-chip traffic. On-chip bytes can grow tens of times, while the bar drops only 5×. A bigger denominator, not a higher bar.
Someone usually asks: “So why not keep it all in registers?” Usually you can't. An SM's register file totals 256 KB, the same as shared memory. And every extra register you take means fewer resident warps. The numbers are in the SM breakdown figure in §3.2.
1.2 Square is a special case: the smallest dimension sets intensity (Figure 1-8)
Scroll to Figure 1-8. ⏱ 3 min here, already counted in §1's 10 minutes.⛔ All three are right, and they exposed a broken thread on the course page: §1 covered only square matrices, §2 jumped to Flash, §3.4 treated the contraction dimension alone. All about the same 128, never connected.
⚠️ So this part takes three minutes, but never skip it: it's the seam §1 → §2 → §3.4.
The matmuls in those examples were square: all sides N, giving N/3. But a real matmul is M×K · K×N. So which dimension sets the intensity?
Same division. Operations: 2 M N K. Bytes moved: 2 times MK plus KN plus MN. It simplifies cleanly: one over intensity equals one over M, plus one over N, plus one over K.
Say “reciprocals add” on its own, once. It's the key: reciprocals add, so the smallest one dominates.So “longer K, higher intensity” is right, and monotonic. But there's a ceiling. Column ②'s five rows show it: M and N fixed at 512, K from 64 up to a million, and intensity climbs from 51 to 256, then stops. The cap is MN over M plus N: exactly 256.
K can't rescue a small M or N. The harshest case: decode at batch size 1. M is 1, so intensity caps at 1, however big K and N get. That one line is why LLM decode is always bandwidth-bound.
Now attention. In Q times K transpose, both outer dimensions are the sequence length S, and the contraction dimension is head_dim. Most models use 128, so intensity is about 128. Below v7's 312 line, and v6e's 560.
⭐ Say it aloud; it overturns a common assumption: “Attention is a matmul, so of course it's compute-bound” is a myth. Unfused, it sits on the bandwidth side, pinned by head_dim.So what does Flash change? Direction now; Section 2's first figure does the math: it saves no FLOPs. It changes the denominator. The S×S matrix never touches HBM. Only Q, K, V and O cross it, so intensity becomes S over 2. With Wan's S of 75,600, that's 37,800.
⭐⭐ The payoff is the last line. Don't rush:The same 128 bites twice. Here, a contraction dim of 128 caps intensity. In §3.4 / §3.6, the 256 × 256 MXU gets only 128: half idle.
First: “wait for data?” Second: “is the unit full?” Two issues, one number.
Figure 1-8 puts attention's intensity at ≈128; the figure in §2.3 says 64. Both right; different denominators. 128 is one matmul in isolation (A/B/C each cross once). 64 is naive attention overall (S and P each round-trip, doubling the denominator).
⭐ Figure 1-8's yellow band says exactly this. Point at it; read it once, or a careful reader will think one figure is wrong.
OK: the line, its use, its purpose. Section 1 produces one thing: a number, and how to compute it. The number changes every hardware generation. The method doesn't.
⛔ Since 2026-09-05, the course page's “two outputs” box is gone. Its item ② said “only what one side has and the other lacks entirely is worth remembering,” but §1 never derives that test; where 3.5 / 3.6 use it context makes it self-evident, no advance definition needed. Don't add it back on stage.Every later section sits on this line. Section 2 measures before and after fusion: same attention, two implementations, identical FLOPs, 1,000× apart in intensity. Section 3, how it lands on each chip. Section 5 answers the real question: who should decide?
Likely questions
This 312 is for BF16. What about FP8? Does the test change?
Yes, it becomes 625. The figure has both rows: TPU 625.8, GB200 625.0.
Simple: compute doubles, bandwidth doesn't, so the ratio doubles. It doubles on both sides, so they still line up at FP8.
Use the line for the precision you actually run. Judge an FP8 op against BF16's 312 and you'll call compute-bound ops memory-bound.
Why does the TPU panel say “per chip”? My logs don't match.
A v7 chip holds two devices, and framework logs always report per device. So: per-chip = per-device from the log × 2.
Here both sides are per package: one TPU chip vs. one GB200 GPU, like with like. The basis is in the figure's small print; point at it.
⚠️ This 1:2 also affects batch: the “device” in per_device_batch_size
is half a chip on v7. But don't expand; that's another course.
Does compute ÷ bandwidth have an official name?
Yes: the x-coordinate of the roofline ridge point, or machine balance. The op-side counterpart is arithmetic intensity. Same unit, so they compare directly.
Name it, move on. Don't draw a roofline. It takes five minutes; Section 2's memory-access figure says it more concretely.
⭐ With 148 SMs, is shared memory's line of 64 an approximation summing all SMs?
Credit it first: the sharpest arithmetic question in the course. But the answer is firmer than they expect: nothing is summed. That number is per SM. Numerator 8,192 FLOP, denominator 128 bytes, both per SM per cycle.
Does whole-chip scale change it? No. Not an approximation: an identity. Chip compute = 148 × per-SM × clock; chip shared-memory bandwidth = 148 × 128 B × clock. 148 and the clock appear top and bottom, so they cancel. The course page's back-check line does exactly this: 2,500 ÷ 64 = 39 TB/s for the chip; divide by 148 × 128 and the clock is 2.06 GHz. That's GB200 (all of §1 uses it).
⛔ The course's back-check line used to say “matches B200.” Wrong label; it now says GB200
(caught re-checking §3 on 2026-09-10). 2.06 GHz is GB200:
2,500 ÷ (606,208 × 2) = 2.062.
The HGX B200 has different numbers: peak about 2,250 TFLOPS,
third-party boost 1.83 GHz (about 1.856 backed out from 8,192 FLOP/SM).
The same formula gives ≈ 35 TB/s, exactly the row in §3.2c.
⭐ So §1's 39 and §3.2c's 35 don't conflict: two different chips.
If asked, say that. And never say “2.06 matches B200” on stage:
the course's own GPU Microscope pins B200 at 1.83/1.856, and someone will catch it.
⚠️ But the other two lines (312 and 119) are whole-chip numbers. They sit side by side only thanks to that cancellation. Say so on the spot, or careful listeners will suspect the figure switched bases.
HBM's 8 TB/s is one resource that 148 SMs fight over. One SM can't saturate it; with all 148 reading, each gets a few tens of GB/s.
Shared memory's 128 B/cycle belongs to each SM. One SM or 148, each gets that much. The total scales linearly.
So fusion doesn't just drop the bar from 312 to 64. It trades a contested resource for one you own.
⛔ Also fix one direction: a lower bar is good. The smaller I*, the easier an op turns compute-bound. So why does overfusing hit a wall? A bigger denominator, not a higher bar: the bar drops only 5×, while fusion moves traffic on chip, where it can grow dozens of times.
Is arithmetic intensity a property of the op, or of the hardware?
Of the op. 312 is the hardware side: “while one byte moves, how many operations can I do?” Intensity is the op side: “per byte moved, how many do I actually do?” Same unit, so they compare directly.
One trap, which sets up the next section: intensity depends not just on the math but on the implementation.
Same attention, naive vs. Flash: identical FLOPs, 1,000× apart in intensity. The difference is all in bytes moved. That's Section 2. If asked, preview it in one line; don't finish the story here.
⭐ 312's numerator counts only the matrix units. Don't CUDA cores and the VPU count?
The section's best question; someone always asks. Credit it first. 312 is indeed the “matrix roofline” ridge point; vector units are left out.
First, why: the ops this course judges (attention's two matmuls, the MLP's big matmuls) put nearly all FLOPs on the matrix units. Softmax, normalization and such are rounding error.
Then the magnitude: GB200's 18,944 CUDA cores,
2 FLOP per cycle at about 2.06 GHz, total 78 TFLOPS.
The matrix side is 2,500, 32.5× more. The TPU is structurally the same:
two MXUs do 262,144 MACs per cycle (131,072 each); the VPU does 1,024 elements per cycle.
⚠️ That clock is derived. Say so.
NVIDIA hasn't published GB200's boost clock.
Then concede: strictly, element-wise ops belong on the vector roofline, whose ridge point is 9.6, not 312.
Then bring it back: the conclusion holds. The vector add's 0.17 is 58× below even 9.6: still bandwidth-bound. Bad either way.
⭐ Then turn it into a takeaway, the question's real value: weak vector units aren't an oversight; both sides chose them. An op that lands on the vector unit loses twice: once to bandwidth, again to having a thirtieth of the compute. That's why fusion is mandatory: not to compute less, but to keep data off this narrow, weak path.
Folded at the end of Section 1 on the course page. Don't raise it; open it only if asked. It's a denominator dispute; the conclusions stand.
Why are the two vendors' specs so similar? Did they copy each other?
I'm not sure; no vendor has explained it publicly. Just say so.
You may offer a labeled inference: both size HBM bandwidth to matrix compute on the same HBM generation, so matching ratios aren't surprising. But that's my back-calculation, not a source.
⚠️ Easy to slip into causation here. “The same workloads pinned both sides to the same ratio” sounds great, but it's reverse-engineered from two numbers, and another SKU breaks it (281.3).
What not to say
② Don't read the ten-row table line by line. It's folded on the course page for that reason. The main text needs one sentence: “only one kind is worth remembering: a level one side has and the other lacks entirely.” The rows are for lookup.
③ Don't linger on SparseCore or CMEM. Section 1 only needs “each lacks one of the other's levels.” Section 3 gives those levels a figure.
④ Don't turn 312 into “TPU and GPU are the same.” This section shows “the spec sheet can't tell them apart,” not “they're the same,” which the next section refutes.
⑤ ⭐ Don't explain how to fuse. This section only names it, in forty seconds. Which ops merge, how the compiler finds fusions, why fusion sometimes fails: not one word. It's the course's easiest twenty-minute tangent, because it truly is the key topic. But its evidence is in Sections 2, 3 and 5, not here. Covering it now spends three sections' time on a conclusion with no evidence yet.
One: the pause on 312.9 vs. 312.5. Read them, then count two beats in your head; let the room discover it. The course's first sit-up moment; rush it and it's gone.
Two: the two bits of arithmetic at the end. Skimming them as a wrap-up is the costliest mistake. Without them, 312 is just memorized; with them, students can measure any op on the spot. Keep both lines: “ten times the compute won't make this line faster” and “a matmul with side 256 is memory-bound.”
Three: the last forty seconds, naming fusion as the through-line. It must come right after the arithmetic. The room just worked out that the denominator decides it all, so “shrink the denominator” feels like their own conclusion. Anywhere else, it's a slogan.
This segment uses three figures (the 312 one, the three-panel one, Figure 1-8). On 2026-09-03 we deleted the old “three-panel takeaway figure” (two panels repeated earlier content, one said “nothing to see here”). The saved time goes to the last 40 seconds (naming fusion). End on the last blue line: “That's the end of this section. You now have a ruler. It answers one thing: bandwidth- or compute-bound?”
Then the hook, at once: “Next section, we use it on a real op, and get the lecture's first big number: fusion changes intensity 1,000×.”
⭐ Where they split: one load, 4 decisions on one path, 0 on the other
⏱ 10 minThe opening claim, “only one cause,” becomes visible for the first time.
Script
Last section: both chips have the same appetite. This section: they eat completely differently.
⭐ Decision one needs a sentence, not jargon.“Coalescing means merging. On a load, each of the warp's 32 threads computes an address; the hardware merges them into as few memory transactions as possible.”
Add the magnitude: “Good vs. bad merging: an 8× bandwidth swing, known only at runtime; the compiler can't see how addresses are computed.” ⛔ The surest way to wreck this segment: “the TPU has no cache.” That shrinks a whole mechanism to one part. And decision one isn't the cache at all; it's whether the warp's 32 addresses merge.
Deliver “4 vs. 0,” not “cache or no cache.”
No specs. We just follow one load: one block of data, from HBM onto the chip. How many decisions along the way? Who makes each one, and when?
Left half (GPU) only. Point at one decision point at a time; at each, say “This answer doesn't exist at compile time.”Left side, GPU. A warp has thirty-two threads, each with an address. Decision one: how many cache lines do they merge into? You find out at runtime.
On to L1. Decision two: hit or miss? Then L2. Decision three: hit or miss? On a miss, something gets evicted. Decision four: which line?
Four. And notice: I didn't call any of them “slow.” Red doesn't mean slow. It means “known only at runtime.”
Right half (TPU). Go fast: there's nothing to count.Right side, TPU. Compile time: XLA tiles each array into 8×128 blocks. Compile time: it decides when to move, and how much. Compile time: it works out what fits in VMEM. At runtime, two things remain: DMA moves whole tiles; the vector unit reads them directly.
Zero runtime decisions on the path. And the bottom cell is empty: no cache layer. So hit, miss and eviction don't even exist.
Dead stop. Read the figure's verdict: “Main path: 4 vs. 0.” This is the center of the course: Section 0's thesis lands for the first time. Don't explain it. Let it hang on screen.And green? Not “fast” either. It means “fixed when the code was written.”
That sets the division of labor. The GPU leaves these decisions to the hardware at runtime,
really to people, the hand writing the kernel. All you can do is help the hardware guess right:
contiguous access, no bank conflicts, hand tiling.
The TPU has the compiler settle them at compile time, so you live with its quirks.
The GPU bets on runtime adaptation. Any shape runs, right or wrong.
The cost: hit rate shows only at runtime; if bad, you can only retry.
The TPU bets on a compile-time plan. Shapes fixed up front; no cycle wasted deciding.
The cost: no runtime rescue; a bad shape means changing code.
⛔ Stop there. Don't go on to “so who got the saved area?” The course retracted that line: no source.
Everything in Sections 3 and 5 follows from this.
⛔ Lane / sublane is no longer taught here. It moved to the start of 3.3, where it's first used. Wrap up and scroll straight to 2.3. Scroll to 2.3; stop on the Flash comparison figure (fig2-3).We have the criterion and the coordinates. Now we run one op end to end: FlashAttention. The reason is practical. Every large model runs it today, and it sits on the wrong side of 312, so both sides have to act.
Remember: one head, 128K sequence, bf16, the score matrix is a full 32 GiB.
Give a comparison, then pause: a v7 device has only 94.74 GiB allocatable. Not even three heads. The model has 128.The naive version? Compute the score matrix S and write it to HBM. Read it back for softmax, write it out again. Read it again to multiply by V. Four trips of 32 GiB, for one head. And the real math is just two matmuls.
Flash? Split Q, K and V into row tiles; take one tile at a time. That tile's S lives only on-chip, never in HBM. Softmax goes online, keeping just two scalars: a running max and a running sum. Accumulate into the output tile, drop S on the spot, and on to the next tile.
The upshot: that seq-by-seq matrix never appears in HBM. Not once.
Point at panel ③. It's the bridge to Section 3.Panel ③: how big a tile? Same question on both sides; different people answer it.
GPU: the kernel author. Budget: 228 KiB of shared memory;
tile size is a tunable hyperparameter. Wrong still runs, just slower.
TPU: the compiler. Budget: 64 MiB of VMEM, but shapes must align to 8×128,
or get zero-padded; zeros still take VMEM, pass the MXU.
Now put that back on last section's line.
Naive: arithmetic intensity 64, 64 operations per byte moved; in that byte's time, the hardware could do 312. Nearly 5× short: ~80% of compute idles, pinned to bandwidth.
Flash: 65,536. Same FLOPs, 1,024 times fewer bytes. It jumps straight to the right, compute-bound. Exactly where it should be.
⚠️ Say right after: 65,536 is an ideal upper bound. Real Flash re-reads K and V, so it never gets there. The conclusion holds: it has crossed 312, left to right.
Don't scroll past. Below the axis is a new green box, “Does it clear the on-chip bar too?” Two minutes, then the orange box.I said Flash “crosses to the right.” It crosses the HBM line, 312.
But there's a second line on-chip: shared memory, at 64.
And fusion does exactly this: moves bytes from HBM onto the chip. So we redo the division with a new denominator.
Point at the green box's five-line derivation. Read every line.A tile has Br rows and Bc columns; head_dim is d.
Each step reads a K tile and a V tile from shared memory, each Bc by d. The Q tile stays in registers.
Compute: two matmuls, 4·Br·Bc·d FLOPs. Movement: 4·Bc·d bytes.
Br of 128: 128 against 64, it clears.
Br of 32: 32 against 64, memory-bound again, on-chip.
⭐ Then give the anchor: real FlashAttention kernels always use 64 or 128 rows per tile. Not a coincidence. This division forces it.
⚠️ Give the caveat: this counts only the shared-memory level, not softmax's vector math or the register level. It's an order of magnitude, not an exact value.
⚠️ TPU numerator: 1,153.5, not 2,307. VMEM is per core, so the denominator is per core, and so is the numerator. The third of the four questions: get it wrong, you're off 2×.
Small print on the last line (TPU side): read it verbatim. Add nothing.On the TPU the method is identical. Numerator 1,153.5 (per core); denominator: VMEM bandwidth.
That number isn't published, and we won't guess it.
Only the direction: on-chip bandwidth is an order above HBM, so the threshold is an order lower,
in the tens, the same order as the GPU's 64.
And the TPU's workbench is 64 MiB; what limits it isn't this line but 3.4's blade width.
So, look back. Flash fused the chain “compute scores → softmax → multiply by V” into one kernel. That's the name from last section: operator fusion.
Pause: not one MAC saved. Panels ①② are identical. All savings are in the denominator.So the premises are in place: the criterion is 312, naive is 64, so it must be rewritten.
The real question isn't “should we fuse?” It's “who does the fusing?” This time, people did: one paper and a pile of hand-written kernels. Section 5 asks: does it have to be people?
Likely questions
GPUs have shared memory too. Isn't that a programmable scratchpad?
Yes, and it's the best question here. Agree first. Shared memory is a software-managed on-chip scratchpad, the same kind as VMEM.
Two differences. Give both:
① The TPU has only this kind; the GPU has both. All four decision points sit on the global load path. Shared memory or not, data first comes from HBM along it. You can't route around it.
② Who manages it. Shared memory: the kernel author. VMEM: the compiler. That's panel ③'s “same question, different people,” and it's Section 5's topic.
⚠️ Don't get into shared-memory size or tiling. Section 3 has a figure for it. Say only “can't route around it” and “who manages it.”
Why is bank conflict a dashed branch, not one of the four?
Banks partition shared memory. They aren't on the global load path. Counting them would distort “4 vs. 0,” the center of the course. Don't blur it to add one more.
The figure says “+1 via shared.” If asked, point at that small print: we know about it and put it on the branch on purpose.
Where does 32 GiB come from?
The derivation is on the figure. Point at it; don't do mental math. 128K is 2 to the 17th. The score matrix is seq × seq; bf16 is 2 bytes. (2¹⁷)² × 2 = 2³⁵ bytes = exactly 32 GiB.
Stress “exactly”: it's not an estimate; it lands on a power of two.
Does Flash really reach 65,536?
No, and we must say so before anyone asks. Tiled Flash re-reads K and V, so real intensity is well below the bound.
65,536 assumes Q, K, V and O each pass once: the ideal where not one extra byte moves.
The conclusion is untouched. The criterion is 312. Even 1% of the bound is 655, still on the right. This section needs “it crossed,” not the exact number.
⭐ Is the saved movement paid for in extra FLOPs? Do those count as “useful”?
The section's second-best question, and we invited it. The orange box just said “not one FLOP saved,” which sounds free. Admit it: usually you pay.
Three forms of payment, different in nature. Take them separately:
① Recomputation. Flash's forward has none (panels ①② match);
backward does: the intermediate matrix wasn't stored.
② Padding. Alignment zeros take space, go through the MXU, produce nothing.
The worst: it buys nothing you can name.
③ Extra vector work. Online softmax rescales the accumulator every tile:
few ops, but on the vector unit, so FLOP counts understate it.
Then “is it useful?”: formally, MFU vs. HFU.
Numerator = FLOPs the model's math needs: MFU. FLOPs the hardware executes: HFU.
MFU never counts recomputation or padding as useful.
⛔ Say only that, then: “Section 6 covers these two terms properly.”
That section asks whether two numbers can be compared; these conventions are its core.
Expand here and you teach half of Section 6 early, and neither lands.
⭐ Finally, tie it back to the line; that's the payoff. First, one precise point:
Intensity's numerator is hardware-executed FLOPs;
hardware can't spot recomputation, so the 312 test includes it.
So there are two ceilings; the lower wins:
Useful throughput = min(peak × useful fraction, bandwidth × useful intensity)
Example: intensity with recomputation is 600, half of it recomputation.
Compute ceiling: 50% of peak. Bandwidth ceiling: 300 ÷ 312 = 96%. Take the smaller: half the roofline.
⚠️ Bandwidth isn't saturated either (312 ÷ 600 ≈ 52%). The matrix unit is 100% busy, half on garbage, while bandwidth idles. That's the fingerprint of overtrading.
The rule: while memory-bound, extra compute is free. Past 312, it cuts output proportionally. A cliff, not an exchange rate. Trade until intensity with recomputation reaches 312, then stop.
⛔ Don't let it become “intensity shouldn't go much above 312.” Useful intensity: higher is better, no limit (a 4096-side matmul: intensity 1,365, zero waste, 100% of peak). Only what you manufactured yourself should hug 312.
Flash is far from the bad case. Forward: 2 matmuls; backward: 4, plus 1 recomputed. 7 vs. 6: 85.7% useful. Pay 17% extra compute, save three orders of magnitude of movement.
Collapsed on the course page, under the orange box; don't open unprompted. Only if asked; full answer: 3 of Section 2's 10 minutes.
Does the TPU have FlashAttention, or not need it?
It does, and needs it as much. The 312 line applies to both: naive attention is memory-bound on the TPU too, and 32 GiB doesn't fit.
The difference is who sizes the tile: on the GPU, the kernel author tunes it; on the TPU, the compiler computes it, within 8 × 128.
That's all of §3.6, so just say “yes” and move on. ⛔ No Pallas in Section 2.
What not to say
② Don't explain how caches work. Set associativity, replacement, TLB: touch none of it. All we need: “there's a layer, and its answers don't exist at compile time.”
③ Don't teach lane / sublane here. It moved to the start of §3.3, where it's first actually used. Explain 8 × 128 fully here and Section 3 has nothing left.
④ No online-softmax math. “It keeps a running max and a running sum” is enough. The recurrence takes five minutes and has nothing to do with the hardware.
fig2-1: three minutes, with room for the “4 vs. 0” pause. It's the center of the course: Section 0's thesis gets its first evidence here. Section 3's fourteen figures and all of Section 5 are footnotes to it.
fig2-3 plus fig2-4: three minutes. Pause on 32 GiB, on 64 vs. 65,536, and on “not one FLOP saved.”
The green box under fig2-4: two minutes. The remaining two go to the closing orange box and transitions.
⚠️ Most likely overrun: fig2-3's panel ③. It's dense and slides easily into Section 3. Its only job here is to raise the question, not answer it.
⚠️ Second: the final collapsible (fusion's cost / MFU vs. HFU). It's collapsed; the main line skips it. But someone will almost surely ask, usually the most expert person in the room. The answer is under “Likely questions.” Give it, move on. Don't drift into accounting conventions.
Taking it apart: one complexity, two homes
⏱ 42 min① Section 2's “4 vs. 0” is not a slogan. From the whole chip down to one instruction, every level has a physical source you can point at.
② Every difference must point back to a specific component or instruction. No physical source, not in this section.
In one line This half hands out the parts. Reassembly and the bill come in the second half.
Script · 3.1 Whole chip first
Both chips have the same appetite, but eat completely differently. Now we actually take the chips apart to see where that difference lives.
Whole chip first. A B200 package holds two dies, joined by a 10 TB/s bridge. But CUDA shows you one device. The seam is hidden.
Switch to the TPU figure (ms-t1, Figure T-1). Point at both, back to back: a pair.TPU v7 is also two dies, but honestly shows up as two devices, each with its own 96 GB of HBM; neither sees the other's address space. Reaching the other half is a step you write.
Pause: “Seam in the open, or seam hidden.” That's Section 2's decision at the package level.Hiding the seam doesn't make the cost go away. You still pay; you just don't see the bill. Crossing dies, bandwidth drops from 21 to 16.8. The numbers are in the 3.2c table.
⚠️ Don't multiply the figure's 131,072 straight into peak.
It counts cells: two MXUs, 2 × 256² in all.
And each cell does two MACs per cycle. The figure doesn't show that step.
One MAC already is a multiply and an add: a times b plus c, in one cycle. That's two FLOPs.
So one cell per cycle is four FLOPs, not two. Mishear this and the whole chain is off by 2×. Whiteboard: 1 MAC = 1 multiply + 1 add = 2 FLOP; 2 MACs = 4 FLOP.
Read the wrong formula aloud, don't just say “it's wrong”: 131,072 × 2 cores × 2 FLOP × 2.2 GHz = 1,153, exactly half the header's 2,307. Missing: “two per cell.”
⭐ If asked “So what's the right chain?”, preview 3.4's bridging line: “The figure gives a cell count. The 262,144 in 3.4 is FLOP per MXU per cycle. Divide, and it's 4 FLOP per cell: two MACs.” Don't work the math in 3.1, or the pacing collapses.Script · 3.2 Taking one core apart
One level in. First what the GPU has, then what the TPU doesn't. Keep that order. Flip it and the TPU looks like a stripped-down version.
Inside one SM: warp schedulers, a 256 KB register file, 228 KB of L1 plus shared memory, scoreboards. Together they do one job: hide latency and branches dynamically, at runtime.
Switch to the TPU figure (ms-t2, Figure T-2). Point at the five empty cells.On the TPU, none of these five exist.
The point isn't “five things missing”: the five are really one thing. Do you switch tasks at runtime? If yes, you need dozens of resident contexts, something to pick what runs next, a way to know whose data arrived: all five. If no, all five lose their reason to exist.
The subsection's takeaway. Stop on it: “Having a cache and having a warp scheduler aren't two questions. They're two faces of one decision.”And Section 2's four red boxes are what you inevitably get once you say yes.
Script · 3.2b “But the GPU has more on-chip memory”
Meet an objection head-on: “You say the TPU spent its area on scratchpad. But the GPU has more on-chip SRAM.”
They're right. We list every on-chip SRAM on both chips: GPU 231 MiB, TPU 134 MiB. The GPU has 73% more. Don't argue this one.
Concede, then turn it over. Order matters: concede first, and the other two yardsticks get heard.But the same table gives three conflicting verdicts, depending on what you ask.
Yardstick one: total SRAM? The GPU wins.
Yardstick two: only the row software uses as data scratchpad?
TPU 130 vs. 37, about 3.5×.
Yardstick three, the one that matters: how big a contiguous workbench can one compute unit use?
TPU 64 MiB, GPU 227 KB: 289×.
Why such a gap? Because the GPU's 37 MiB isn't one block. It's 148 islands that can't see each other, 256 KB each, with L1 taking part of each. One thread block can explicitly manage at most 227 KB.
Dead stop. That 289× gets used directly in 3.6, when we size each side's tiles.One more must-say: yardstick one is correct, but the most misleading. Nearly all of the GPU's 97 MiB lead comes from the L2 row, the one cell in this table you can't arrange. You can't “park” weights in L2, only issue loads and hope they're still there.
Script · 3.2c “Speed”
That answered “how big” and “who controls it.” The other half: how fast. Add it and you hit a result almost everyone guesses wrong.
Most people assume a cache is an order of magnitude faster. B200's L2 isn't. HBM is 8 TB/s. L2 is only 21. 2.6×.
Pause. Let it land. 2.6×. ⛔ Add this at once, or “only 2.6×” gets heard backwards. The room will conclude “on-chip memory is no big deal.” It's the exact opposite.2.6× answers a narrow question: how much faster is a byte from L2 than HBM?
Ask another and it flips: bandwidth per byte of capacity?
L2 gets 2.6× the bandwidth from 1/1,500 the capacity:
4,000× HBM's bandwidth density.
⛔ Then pull back, lest it eclipse the main thread: “But when you optimize, 2.6× is still the only number you can use. You move bytes, not capacity.”
So L2's value isn't its own speed; it's skipping an HBM round trip. A hit gets you 2.6×. A miss saves nothing and adds a detour. That's why “just land it in L2” isn't a strategy you can rely on.
Why so slow? L2 is shared chip-wide; data crosses the on-chip interconnect to an SM. Shared memory is 4.3×, VMEM about an order higher; both sit next to the compute units.
Both subsections' takeaway: “Both truly fast levels are software-controlled.”⚠️ For the TPU row we give a magnitude, not a number. VMEM bandwidth is unpublished; only capacity is official. Go no further than: about an order of magnitude above HBM. No precise number, and no integer multiple: HBM bandwidth is public, so a multiple reveals the unpublished number.
Point at the last ⛔ box, “Don't read this as ‘the TPU won three rounds.’” Always read it aloud: it's the lecture's guardrail.All three leads are real, and each has a price.
① Big, fast scratchpad; the price is no safety net.
A GPU guessing wrong just slows a bit; a TPU compiler tiling wrong really stalls the chip.
② The GPU's two 37 MiB blocks look “not spent on data.”
but buy something else: irregular shapes, varying lengths, unpredictable branches still run decently.
So the real conclusion isn't “whose on-chip memory is better,” but: Both handle the same thing, uncertainty, just in different places. The GPU spends silicon to absorb it. The TPU eliminates it at compile time.
Script · 3.2d “Latency”
Capacity, done. Bandwidth, done. One axis left: how long you wait. And it's the only one of the three the GPU itself can't answer.
⭐ Ask the room first, no numbers yet: “A warp needs data. Not in L1, so check L2. Not there either: fetch from HBM. How long does that chain take?”Look at the figure. L1 hit: 30 to 40 cycles. L2 hit: 273. All the way to HBM: 659. Same load instruction. Identical in the source.
Pause. 30 cycles to 659. 20×.⛔ Head off a sum nearly everyone gets wrong: “One L1 lookup plus one L2 lookup plus one HBM fetch.” You can't add them. 659 is one full miss, end to end. The two misses before it are already included.
⭐ Then the real question: “Every case takes a different time. So how do you plan ahead?”The answer is blunt: you can't, so the GPU doesn't plan at all.
None of its three remedies tries to predict.
One, oversubscription. Dozens of warps live on an SM; one stalls, switch to the next.
Latency isn't eliminated; others' work covers it.
Two, the scoreboard. Hardware watches operands at runtime, issuing only once they arrive.
Three, the cache itself is a guessing machine, betting you'll reuse the data.
The TPU goes the other way: delete the uncertainty at the source. No hardware cache, no cache misses, no variable latency. Data is in VMEM not by luck, but because the compiler's own DMA put it there. And the compiler knows which cycle to wait on.
⭐⭐ First half lands here. Slowly: “The GPU hides uncertain latency behind other work; the TPU deletes the uncertainty. One answer is statistical, the other deterministic.”⭐ Follow with: “So lacking L2 isn't a missing feature: a machine scheduled to the cycle at compile time needn't guess.”
⭐ Give the room something practical: this is also why GPU and TPU “tuning” share nothing. On a GPU you tune for “enough other work”: occupancy, warp count, coalescing. On a TPU you tune for “room for the compiler to schedule”: alignment, tiling, double buffering. Same word, tuning. Not one knob in common.
⚠️ Don't let it become “the two keep drifting apart”: “Peak-chasing CUDA kernels today use async copies, explicit barriers, their own double buffering: a hand-written static schedule on the GPU. To really reach peak, guessing isn't enough.”⭐ A hook for the 3.5 boxes; pay it off there.
⚠️ Cite the source: measured on H100 and consumer Blackwell; B200 has no comparable public data. Borrow magnitude and shape; they aren't B200 data.
Script · 3.3 Parallel hierarchy
First, two terms. Next we ask what a thread really is, and 3.4 has that “8 × 128.” Both need the TPU's coordinate words, and both words mislead at face value.
Two steps build the picture.
Step one, one row. A sublane is one horizontal row,
128 cells long, 512 bytes. That's VMEM's architectural word width, not an arbitrary number.
Step two, stack 8 rows: one vector register, 8 × 128 = 1,024 cells.
Now each word has its place. Lane measures position along the row: cell 0 to 127.
Sublane measures which row: 0 to 7.
So “align to 8 × 128” just means: 8 rows down, 128 cells across.
⚠️ Here's how the name misleads: a sublane isn't a piece of a lane. It's a full row across all 128 lanes. “Sub” means second dimension, not smaller.
Lighten up here: “Almost everyone gets this backwards the first time. I did too.” Admitting it beats warning “don't get it backwards.”One line to remember: innermost dimension is lane, next in is sublane.
Then the figure's real point: XLA deliberately lays out memory in the register's shape. Load it and compute right away, with no relayout.
⛔ I used to get this wrong. Don't read the old version: “A row is no longer contiguous in memory.” That isn't the cost; it's the point. A run of 128 elements fills exactly 128 lanes: the goal.The real cost: the shape is locked in. Three: Changing the innermost dim moves data (tiling binds the last two dims; a transpose or some reshapes are copies, not stride changes). Uneven sizes get padded: zeros take VMEM, pass the MXU, count as FLOPs. Outside handoffs need relayout. §3.4 (transposes) and §3.6 (padding) pay for it.
⭐ Add a contrast; it ties back to the main thread: “GPU coalescing wants contiguity too, but as a soft constraint: misaligned still runs, just slower. Tiling here is a hard constraint: wrong shape means pad or relayout. One side fines you; the other won't let you through.”What just went missing were components on the silicon. One level up, the programming model loses things too. Same cause, second appearance.
I'll boil this section down to one question.
Pause. Slowly: “When you write GPU code, you don't know which SM your work lands on, or when. The hardware decides at runtime.” Whiteboard only: “decided at runtime.”Since you don't know, you can't name anything. You can't write “SM 3, do this.”
What's left: describe relationships: what must stay together, what can go anywhere.
A batch that “must stay together” is a block.
Why “must”? Because that work passes intermediate results around.
And they live in the SM's own small local memory, not outside.
So once a batch lands on an SM, it can't move. Move it, and the intermediates are lost.
Conversely, work that never talks can land anywhere. All of it together is the grid.
These two levels are yours. Writing the code really says two things: these go together, and this many in total.
The hardware groups too, for a different reason: saving circuitry.
Fetching an instruction is expensive, so 32 data elements share one instruction and compute together. That bundle of 32 is a warp. You can't change it; it's fixed in silicon.
Four levels: the bottom two fixed by hardware, the top two set by you.
The other two came later, each forced by a specific problem.
warpgroup: new matrix instructions need four warps issuing together to make the shape.
cluster: neighboring SMs want to reach each other's local memory directly, no detour.
The TPU, by contrast, fixed everything at compile time. There's no “you don't know.” So it needs not one level of this language.
⚠️ Head off a trap: the two sides' levels don't answer the same question. The GPU's describe “who must be together.” The TPU never needs to say that. So this table is ordered by question, not by name.
Blank one: no “group of threads.” GPU: thread blocks; TPU: no middle layer.
Blank two: no “swap in another.” GPU: 4 warp schedulers, 64 warp slots;
TPU: none.
Those two blank rows are 3.2's five missing pieces, projected onto the programming model. Parts missing from the silicon mean concepts missing from the code.
⭐ Of the three “mismatches,” cover only the first; save two for questions: “One SM hosts several blocks at once, side by side, blind to each other's local memory. Section 5's ‘occupancy’ measures how many more batches the SM can fit.” Scroll to “Inside one SM” (4 quadrants).One level deeper. So far, “an SM” has been a black box. Inside are four quadrants, each with its own scheduler and registers. 32 is one quadrant's width. 128 is one warp from each.
One question deserves its own moment: what does a thread own outright?
No compute units. CUDA cores and Tensor Cores aren't assigned to anyone. They're pipeline slots. Whichever warp issues this cycle uses them this cycle; next cycle, someone else.
All it owns outright is registers: claimed on landing, returned when done, never moved.
⭐ Two things follow. Say them back to back: “Because registers never move, swapping in another warp is free. Because they're owned outright, registers run out first, not compute units.”⛔ 32 bits, 4 bytes, a cap of 255, rounding to 8: all in the fold-out. Don't read any of it live.
Scroll to the last figure (“Three ‘threads’”).Finally, back to the bottom row. A warp is 32 threads, and each has an index in the warp, 0 to 31. That position is a lane.
So the GPU has lanes too: in hardware, one SIMD channel. The same thing as a TPU lane.
⭐ The section's last line. Say it, move on: “One difference: GPU lanes have names, TPU lanes don't. You can write an if for one GPU lane; on the TPU you can't write ‘lane 37, do something else.’ So a CUDA thread is a SIMD lane with a name.”Script · 3.3a What one lane does in a real matmul
That was right before Tensor Cores. Today it's wrong.
Start with the era when it was right. A thread was the owner of one output cell. It ran its own K loop, loaded slices of A and B from shared memory into its registers, multiply-accumulated, kept the sum, and wrote it back. It did both the movement and the compute.
⭐ Then overturn it. Slowly:“With Tensor Cores, the matrix instruction is one instruction issued by the whole warp. Its operands are spread across 32 threads' registers, a small handful each. No single thread can compute even one complete output element.”
If asked for specifics: in a typical instruction, each thread contributes 8 values of A, 4 of B and 4 of the accumulator. That's all. ⭐⭐ The takeaway: “So on a Tensor Core, ‘which cell does this thread own?’ stops meaning anything. A thread shrinks to two roles: a holder of registers, and one thirty-second of an instruction issue.”
That's the full sense of “a thread is really a lane.” Point at the box below, “Is it SIMD?”
The answer has two levels, and they disagree. In hardware, yes, it's SIMD: a warp's 32 lanes share fetch, decode and issue, running one instruction in the same cycle. In the programming model, no: you write scalar code; since Volta each thread has its own program counter and can branch alone.
⭐ Close with one line: “SIMT is scalar code on SIMD hardware. The cost is in branches: when they truly diverge, the paths take turns.”Back to the main thread: “All this is the price of relying on runtime adaptation. The TPU doesn't make that decision, so it lacks even the word ‘thread.’” ⚠️ Data movement only if asked (two minutes): Does a thread move data? Yes, but that half is retiring. Before Ampere, threads moved data themselves, tied up doing it. Ampere's cp.async bypasses registers. Hopper's TMA: one thread issues one descriptor, a whole tile moves, the other 31 sit out.
Script · 3.3b The apparatus (⚠️ skippable)
The fold below is the mechanism (four minutes, the lecture's likeliest overrun). If nobody asks, leave it shut; go to 3.4. Say just this: “A scoreboard is a table of whether each instruction's operands are in. Each cycle the scheduler scans it and issues whatever's ready.”
Then straight back: “On the TPU, this table doesn't exist.”
Twice now I've said “the GPU exposes lanes to you, and the price is a whole apparatus.” This subsection answers one thing: what's in it? Three things, each for one narrow problem.
Name all three in one breath, then each. Knowing the count and where you are keeps people with you:“One: who can issue now. Two: who syncs with whom. Three: how data gets into place without tying up threads.” Last: point at 3.3b's data-path figure (four GPU rows + one TPU row), first three rows only. It's the hook. Don't skip it.
Threads are retiring as data movers. Three generations, one layer peeled off each.
Before Ampere, threads moved data themselves: global memory to registers, then shared memory.
While moving, it can do nothing else.
From Ampere, a new instruction sends data past registers straight to shared memory. No middleman.
Hopper goes further: one thread issues one descriptor to move a whole tile.
The other thirty-one sit out.
With the thread hollowed out, what sets “how big a bite you take” isn't the thread. It's the instruction. There, the two sides differ by 16×. 3.4 measures it in front of you.
But don't jump to 3.4. This subsection first has to open the bill.All that: you write scalar code, hardware assembles it into SIMD, queues you when you diverge, and swaps you among dozens of batches of work. So what does “runtime adaptation” cost? Next: three concrete circuits, one by one.
First: the scoreboard. A hook that sticks: the name comes from the CDC 6600, in 1964. The first machine whose instructions could complete out of order. One of its parts was officially called the Scoreboard. (Why that word, and why today's is a different thing, is folded on the page. Don't open it live.)
Back to the GPU. Each cycle the scheduler issues from a “ready” warp.
Who decides “ready”? NVIDIA splits instructions by one test:
does the compiler know how long it takes?
If yes, no hardware: the compiler writes “stall N cycles” into the instruction.
If not (memory, special functions, matrix ops),
that's the scoreboard:
six counters per warp, +1 on issue, −1 on writeback.
The waiting instruction carries a mask of which must hit zero.
Second, one line: a few hardware barriers in the SM. Warps arrive one by one; once all are in, all go. Mechanism folded below, for reference.
⛔ You announced three; all three need a landing. “We'll skip the second” drops one in public. This one line is enough: “A warp waiting at a barrier keeps its slot. So in a sync-heavy kernel, everyone waiting doesn't make occupancy look better.”The third one gets a figure.
Data-path figure: rows 1–3, “Staging” column only; not row by row.The top three rows share both ends (global memory left; shared memory plus compute right).
Only the middle column changes.
At first, data went via registers; the thread computed every address and ran the loop.
From Ampere, cp.async cuts out the register hop; addresses are still the thread's job.
With Hopper's TMA, one thread fills a descriptor, sends it, moves on.
Address generation is all hardware.
Scoreboard: “who can issue”; barrier: “who syncs with whom”; TMA “how data arrives without tying up threads.” Together, the itemized bill for runtime decisions. The TPU needs none: not crude, it answers all three at compile time.
Script · 3.3c ⭐ A matmul's real order
We've taken both sides' parts apart. But I skipped something more basic: in what order does hardware actually compute a matmul?
⭐ No figure yet; the room says it first. Ask “How is a matmul computed?” All say: “Left row one, right column one, multiply pairwise, sum: the top-left number.”Get them to say it aloud. It matters.
Right. Mathematically, exactly correct. And real hardware, GPU or TPU, doesn't do it that way. Not one.
Let that hang for a beat.First, count the naive way's cost. Look at the first band.
Step one: A's row 0 times B's column 0 gives c₀₀. Write it out.
Step two: same row 0 of A, B's column 1.
Notice: you just read those three A values again.
After four steps: A's six values each read twice, B's six too. 24 reads
for just 12 multiply-accumulates.
Hardware does something else: k goes to the outermost loop.
Step one, k = 0: read just two things, A's column 0
and B's row 0: four values.
They pair up for four multiply-accumulates,
each into its own cell. All four move at once.
Steps two and three, same. After three, all four finish and drain together. 12 reads, 12 multiply-accumulates. Exactly twice as good.
⭐⭐ The line to remember. Slowly: “It's not one cell reused over and over. It's every cell filled in, slowly, at the same time.”And settle the obvious one: every cell is updated exactly K times, top-left and bottom-right alike.
⭐ Why not keep one row of accumulators? See the equation.
The accumulator is an M×N block. Each K step reads M + N values, buys M×N multiply-accumulates.
Reuse is M×N over M + N.
A 128 by 256 tile? Eighty-five.
The real conclusion: an accumulator array isn't for holding results but buying reuse. Each extra accumulator pairs already-moved operands once more. Section 1's arithmetic-intensity thread, first cast in silicon.
Scroll to the third band (TPU).The TPU flips it. B loads into the array, then stays put. A streams in from the left, row by row; partial sums add up on the way down. Out the bottom: a finished row of C.
⭐ The key line: “The sum along K happens in space. Three PEs each multiply-accumulate once and pass the partial sum down. No external accumulator.” Bottom comparison table: the landing.⭐ Three bands: same math, differing only in which dimensions are space and which time.
The GPU puts M and N in space, K in time.
The TPU puts K and N in space, M in time.
⚠️ Two caveats, in passing; don't expand: ① The GPU band is algorithm-level. At instruction level, one MMA already reduces a short run of K internally. Same shape, just batched. ② The TPU uses an external accumulator only across K tiles. Real K is often thousands; the array takes 256 at a time.
Script · 3.4 One instruction's bite
Last section: who issues the instruction. Now a different question. Once an instruction goes out, how big a block can it take in one bite?
Sounds internal; it decides something external: which head_dim costs nothing.
Scroll to Figure G-4, then T-4. In both, look at one spot: how deep the summed dimension is.GPU: over five instruction generations, more threads join, bites grow. But the summed dimension never changed: always sixteen.
On the TPU, the MXU is a 256 by 256 systolic array. That dimension is two fifty-six.
Dead stop: “Sixteen, against two fifty-six. A sixteen-fold gap.” The one number to remember.Its name: the contraction dimension. The one a matmul sums away: how many terms one instruction accumulates at once. The name doesn't matter; the numbers do.
What does 16× mean? Plug in a real shape: attention's head_dim is usually 128.
GPU: 128 over 16 is 8. Eight instructions, zero waste.
TPU: 128 against 256 fills only half. The other half sits empty.
So why not fill that empty half with other work?
You can't. The contraction dimension is an accumulation chain. Anything pushed in joins the same partial sum. Nothing else fits.
Flip it: why doesn't the GPU pay?
Not the scheduler. That swaps warps, filling time bubbles; it can't fill a contraction dimension's empty half. It's that its K is only 16, and nearly every shape is a multiple.
⛔ Read the line above as is. An old version said “the GPU has a scheduler, so it's shape-insensitive.” Wrong: any CUDA person refutes it in a sentence.⚠️ Neither side is badly designed. The GPU can make K this narrow and leave alignment to runtime, at the price of the five parts from 3.2 the TPU lacks. “Whose granularity to align to?” just has two answers.
⭐ Scroll to the big four-band figure at the end of 3.4 (caption: “From ‘what stays put’ to ‘how many reloads’”: stationarity / tiling / ping-pong / reloads).⛔ Don't weight the bands equally. Priority: ① Band three (ping-pong) first: what nobody else explains clearly.
“Loading a weight tile: 256 cycles. Each cell has two weight-register sets, one computing, one loading, so the 256 hide in the 2048 of compute. Only the first tile waits.”
⭐ Then the test at once: “It hides if a segment has more than 256 rows.”
② Back to band one (stationarity), one line: left-, right-, accumulator-stationary differ only in what stays put. The TPU is right-stationary. What stays put decides which two dimensions are space, which is time.
③ Bands two and four, for questions: where the 168 tiles come from, why M can't stream in one go (its accumulator would need 128 MiB, over VMEM), why finer segments mean more right-matrix reloads. Unasked, skip.
⭐ The figure's closing line; must say: “Longer segments: fewer right-matrix reloads, but a bigger accumulator. Turn this knob all the way and you still hit the on-chip stovetop, the GPU's wall in 3.6.” ⚠️ Someone will surely do the math aloud: “One instruction, only sixteen deep, can't add up to 2,250 TFLOPS.”Head it off: instruction shape is how much work you hand over at once; MACs per cycle is separate. To reconcile, quote the page: 4,096 × 2 × 148 SM × 1.83 GHz = 2,219, official 2,250, a 1.4% gap. ⛔ Don't follow their wrong path. Point at the closing peak chain. For the math-checkers: go fast, conclusion clear.
While we're here, multiply out the TPU's peak to show we're not making it up: one MXU, 262,144 floating-point operations per cycle, times 2.2 GHz, times four MXUs. That's 2,307 TFLOP/s. Official: 2,307. Exact.
⚠️ A cross-generation warning to take home: half that TPU number comes from “each cell does two multiply-accumulates per cycle.” Only the 256 by 256 generations; the older 128 by 128 ones do one. Carry the constant across and you're off by 2×.
Close with this. Must say: “How much you can fuse is capped by two numbers together. One is workbench size: 3.2b's 289×. The other is the granularity here. Tiling in 3.6, both show up at once.”Script · 3.5 What each side added alone
So far, the first half has counted what “one side has and the other doesn't.” Two pieces are left: each side grew one of its own.
But the question here isn't “what are they?” Slow down and ask: “The product page says it's supported. Can you actually use it?”Write just this on the whiteboard: “Can you use it?”
Just two examples, one per side.
Scroll to the block-scaling text. No paired figure; say them: one example, one threshold each.First, GPU side: block-scaled quantization. Marketing: “B200 supports FP4.”
That's true. The catch: block-scaled instructions have two paths, on different dies, and B200 has only one. So if you assume “it has both,” you're wrong.
Second, TPU side: SparseCore, a core dedicated to embedding-table lookups. Marketing: “the TPU has a core just for embedding lookups.”
Also true. The catch: it only pays off when the same rows get looked up over and over. The number that measures it is called the duplication factor.
Stop. For this room: “For language models, the duplication factor is very low, about 1.01: not zero, but nowhere near the payoff line. The compiler never sends work there; in production it has never looked up a single table.” ⛔ Follow up at once; don't stop there: “But ‘can't use it’ isn't ‘not using it.’ You use this core every day.”Its real job in large-model training: offloading collectives, taking All-Gather and Reduce-Scatter off the TensorCore so it keeps computing while communication runs alongside. One way the TPU overlaps compute and communication.
⭐ If asked “How does it handle irregular access?”: the page has a SparseCore dissection figure; don't scroll to it (three minutes).One line: “The TensorCore moves whole (8,128) blocks. SparseCore moves just 8 numbers, from freshly computed addresses: sixteen small units, each chasing its own addresses, hiding latency with many fetches in flight.”
Details: the figure, after class. ⛔ If pressed on granularity, get these two right:
“(8,128) is the vector register's shape, not the minimum DMA size.”
“SparseCore's DMA granularity on v7 is 32 bytes. Go by get_tpu_info in the JAX source; the docs page still shows an old 64. Don't read that.”
Trick question: “VMEM or HBM?” Answer: “Both, but its VMEM is its own, not the TensorCore's.” Sources, both public: the Google Cloud post “Training large models on Ironwood TPUs” lists it as its own tuning technique; the public MaxText repo's XLA flag library has SparseCore All Gather. ⛔ No internal measurements. ⭐ Only now is the lesson complete. Slowly: “Marketing doesn't just keep you from using what it advertises. It also hides what you already use. Because it's called a ‘lookup core,’ you search the profile for lookups, find none, and decide it's idle.”
One more thing shows: the two aren't even at the same level. Block-scaled quantization is an instruction variant; SparseCore is a physically separate core. “Each side added one piece” is itself a marketing framing.
Neither line lies: the error is reading “supported” as “fits my workload.”
Close with an action, not a sigh: “When you see ‘supports X,’ find the line where the threshold is written. If you can't, assume it doesn't apply to you.”“Five layers taken apart, and at every layer the TPU is missing something. But what's missing is never a compute unit. It's caches, scoreboards, the machinery for ‘wait until it runs, then decide.’ Each exists for ‘we don't know what you'll run.’”
Then land it: “The sentence from Section 0 has now been tested five times.” ⚠️ Must say. It's the lecture's only mid-point return to the opening thesis. Skip it, and the room leaves §3 not seeing they watched five pieces of evidence for one claim.