Topic 2 · TPU vs GPU L200

How to use these notes

These are the presenter's notes, not student material. Students read the L200 course page; you read this.
Course author: Chris Yang · Google Cloud AI Infra architect.

🎯What this segment must leave behind The one sentence students should keep. Just one. Scattered means not landed.
🗣Script Near-verbatim; read it as is. Yellow lines are said word for word; reword the rest.
🖥Screen Where to scroll, and which cell of which figure to point at.
❓Likely questions Real questions, real answers. No answer? Write “I'm not sure.” Invent one and you'll read it on stage.
⚠️What not to say Where this segment drifts, and where the time goes.

What the stage directions mean

These terms recur. Each is a concrete action, not an adjective. Do them as written; don't go by feel.

Dead stopAfter the last number, stop talking and count two beats. Not “pause a little.” Your default on stage is to keep going; without a count, you'll slide past.
Read it outVerbatim on screen; say it in plain words, one number per line. Raw strings (errors, instruction names) can't be read aloud.
Read, don't explainThen add a forward pointer: “Hold on to this; we'll use it when ××.” A setup with no payoff is noise. Name which section pays it off.
Slow downSplit the sentence in two, with one break. You can't control “slower.” You can control the break.
Really waitWait for a voice or hand, however long. None? Call on someone. Answering yourself wastes the 30 seconds.
Point atThen read out the words in that cell. Don't make the room hunt. A gesture carries no information; your voice does.

⛔ One general rule: any line explaining why it's taught this way (e.g. “the surprise survives intact, and the line holds up under questioning”) is for your judgment. Don't say it. It pulls the room off the content and onto your method.

⭐ The thread through the whole lecture: operator fusion

The surface topic is hardware differences. The real topic: should an op be merged with its neighbors into one kernel, and who decides? Learn this table. Don't give fusion its own segment. Name it once at each of these five points:

WhereHow to name it here
Section 1 Line drawn, not yet used: name it. The only way to make a slow op fast is to shrink the denominator, best by keeping intermediates out of HBM. About 40 seconds
Section 2 “Naive vs. Flash” is before and after fusion. Same FLOPs, 1,000× apart in intensity. The lecture's strongest fusion demo; nothing to add
Section 3.6 The same fusion, landed on each side: tile size, and who tiles
Section 5 ⭐ “Runtime or compile time” asks exactly who decides fusion. The lecture closes here. Everything before builds to this line
Section 9 The close stays on this thread: no ranking, only who was handed the decision

How the time is split

A plan, not a measurement. After each run, put real minutes in the generator script (it builds the table; one edit is enough).

MinutesContentWhat this segment lands
0–8Opening: two origins (roadmap figure gone)Plant the claim “they differ everywhere, for one reason.” Still a claim, not a conclusion
8–18Section 1 · The dividing line: compute ÷ bandwidth gives both the same numberFirst “sit up” moment: 312.9 vs. 312.5 exposes the spec sheet. At the end, name the thread: operator fusion, and say at once that it doesn't remove data movement, just moves it a level
18–28⭐ Section 2: the same load, 4 runtime decisions on one side, 0 on the otherThe claim becomes visible. Also the strongest fusion demo: same attention, same FLOPs, 1,000× apart in intensity
28–70⭐ Section 3a: chip → core → core floor plan → parallel hierarchy → one instruction's biteParts, not conclusions. At each level, point at “what differs, and whose decision caused it.” Conclusions come in 3b
70–80⭐⭐ Section 3b: FlashAttention walked on both sides + paying off §1's peak coincidenceThe climax is in these 10 minutes. Cut anything else first. Cut this and the lecture becomes a spec comparison
80–84⭐ Section 4: 64 chips each. One NVL72 domain (one hop) vs. 4×4×4 torus (≤ 6 hops)Prerequisite for Section 6's measurements: those two numbers ran on different network structures, never measured apart here
84–96⭐ Section 5 · Unpacking the word: runtime, or compile timeThe opening claim closes: four appearances, one thing. That is: who each side handed fusion to
96–104Section 6 · After the run: two numbers; comparable? Denominator: entry price. Numerator: MFU / HFUThe output isn't the ratio but “this number could lean either way.” ⭐ MFU and HFU get proper treatment here: the numerator-side metric, premise for row ③ of the “boundaries” figure
104–109Section 9 · Close: so what is each good at?Back to the first line: no ranking, only who decides

The full core track runs 109 minutes (no skippable detour: §4 is back as the prerequisite for Section 6's measurements).
⭐ Not a one-hour lecture. On 2026-09-03 we decided not to cut content. For an on-the-hour slot, round up. Truly only 60 minutes? Use the emergency list below.

What to cut first when you run out of time

Fixed order. Don't improvise: ① Section 4 to two sentences (never skipped whole; Section 6 rests on it; a glance per figure plus “one hop vs. six”; −4); ② Section 6: just the two-measurements figure + one sentence on boundaries (−4). ⛔ 6.2 (MFU / HFU) stays: the numerator-side metric; without it, row ③ of “boundaries” is empty; ③ cut one figure from each of Section 3's first three subsections (−4).
⛔ Section 3.6 (FlashAttention side by side) and Section 5 are never cut. Without them it's “two spec sheets compared,” what the opening promised not to do.

00

Opening: two warm-ups + two origins

⏱ 8 min
What this segment must leave behind The two sides differ everywhere, but every difference follows from one decision. The decision: who schedules moving data from HBM onto the chip: the runtime on one side, the compiler on the other. The rest of the lecture tests this.

Script

Stay on the first screen. Don't scroll yet.

OK, let's start. This lecture is “TPU vs GPU,” but up front: I won't read you a spec sheet. The spec sheet is the first thing we'll take apart.

⭐ Scroll a bit to the yellow multiple-choice box. Don't click any of the four options; clicking reveals the answer.

First, a warm-up from last lecture. We took apart DeepSeek V3: 671B parameters total, 37B active per token. So what does MoE actually save?

⭐ Really wait here. Hands or shout-outs both work; don't click till someone answers. Not a knowledge test: it switches the room from listening to answering. Answer it yourself and the 30 seconds are wasted.

The answer: it saves compute, not memory. Each token only runs the 37B path, so the compute saving is real. But all 671B of weights stay resident, every byte. You don't know which experts the next token picks, so they all sit in HBM.

Reveal the answer; point at the last paragraph (the ⭐ line) and read it.

But the answer isn't the point. The idea behind it is: memory and compute are separate resources, and they move independently. Last lecture counted them separately.
This lecture adds a third: bandwidth. And as we'll see, the one that decides whether an op is fast is often neither of the first two.

“The third is bandwidth” is the seam to the last lecture and the lead-in to Section 1. Say it and move on. Section 1 derives the exchange rate itself. ⭐ Scroll on to the second yellow box. Again, don't click.

One more. This one's pure guessing. Later we'll take both chips apart, layer by layer. First, a guess. On B200 the matrix unit is a Tensor Core; on v7 it's an MXU. Same job. How many does each chip have?

⭐ People will get this wrong, and all in the same way. Most assume the same ballpark. Let them be wrong. That's the point.

B200 has 592. v7 has 4. A 148× difference in count. But here's the real surprise: in MACs per cycle, the two chips differ by only 15%. Same compute: 592 small pieces on one side, 4 big ones on the other.

⭐ Read it and stop. Don't explain why. Point at the last line of the answer area and read it. The numbers are out; the “why” is an IOU. ⛔ Classic mistake: explaining it on the spot. The hook becomes a spoiler, and Section 3 loses its question. Let them carry “why hundreds vs. single digits?” through the first two sections. At the §3.7 count, someone will call out the answer.

The whole lecture is two questions and one route. One: where exactly do these chips differ in hardware? Two: we run one op end to end and watch where each difference shows up. That op is FlashAttention.

Scroll one screen to the Section 0 figure. It has just two boxes. Nothing else to find. Make those two boxes clear, then move on.

Here's the first figure. It's the lecture's only premise: these two boxes.

On the left, the GPU. It started as a graphics processor. From day one it had to serve tons of small, unrelated jobs. It doesn't know the shapes, the access patterns, or the control flow. So most of its die goes to handling any case: the cache hierarchy, the warp schedulers, the huge thread counts.

On the right, the TPU. It started as an ASIC built for matrix multiply, serving only neural networks from day one. Shapes are regular, access is predictable, control flow is static. So it needs none of that “handle any case” circuitry.

⛔ Easiest slip (I've made it myself): don't say “all the saved area went into compute.”
The savings became neither compute nor on-chip storage. Section 3b rules out both with two tables: the GPU has 16% more total MACs and 73% more on-chip SRAM.

So what did it save? Complexity itself. No cache, no scheduler, layout fixed at compile time. The same compute in four big blocks, not 592 small ones. That's 148 times fewer sets of control circuitry.

⚠️ Someone will likely ask “So is it more power-efficient?” There's a solid official answer. Give it with its caveats.

Yes, you can answer. Two things suffice. First, a quote from the first TPU paper. It lists everything left out: no caches, no branch prediction, no out-of-order execution, no multithreading, no context switching. The next sentence: “Minimalism is a virtue of domain-specific processors.”

⭐ Second, a number, which beats any adjective: “On that die, control logic takes just 2%.”
Data buffers 37%, compute 30%, I/O 10%. Control: two percent. The paper adds that this part is much larger on CPUs and GPUs.
⛔ Then brake. All three caveats, every time:
One: the paper compares the 2015 TPU v1 with the K80. Back then the savings did become compute: a small die with 25× the MACs. For v7 vs. B200, Section 3b counts the GPU at 16% more. Same mechanism; ten years later, the opposite outcome.
Two: Ironwood's official 2× perf per watt is vs. the previous TPU, not GPUs.
Three: yield and reliability. No source, so don't say it. The official word is “deterministic execution”: tail latency, not reliability. Don't mix them.
One beat. Next two lines: the takeaway.

One doesn't know what you'll run, so it hedges at every step. The other knows from day one, so it hedges nothing.

No more scrolling. Stay on the two boxes.

So what is this lecture about? The two sides differ everywhere. But every difference follows from a single decision.

Which decision? Moving a block of data from HBM onto the chip: who schedules it. On the GPU, a cache guesses for you at runtime. The TPU has no cache; it's fixed at compile time.

Shapes padded to 8 by 128, how big a bite one instruction takes, who writes the kernels, even what wiring a whole machine costs: it all grows from that one decision.

Slow down. This is the lecture's claim; every later section is evidence for it. Rush it and it sounds like a slogan.

One more thing, and I mean it: this lecture doesn't rank the chips. Both origins were the right call at the time, and each still carries its cost. You won't hear “which is better” today. Only “who decides this, on this side.”

That's the foundation. Straight into Section 1. It's one division, so it's quick. After that, Section 2 starts taking the hardware apart.

Likely questions

“Only one cause”: isn't that oversimplifying?

There are a dozen-odd symptoms. “One” means one cause. Caches, shape alignment, who writes kernels, interconnect wiring: all differ, but they're not a dozen parallel facts. They're a dozen consequences of one decision.

At the opening it's a claim, not a proven conclusion. Say so: “You don't have to believe it yet. Sections 3 and 5 each test it.” That convinces more than digging in, and gives them a reason to keep listening.

Does the TPU really have no cache at all? Then what is VMEM?

Precisely: the TPU has no hardware-managed cache tier. It has on-chip storage, plenty of it, but as a software-managed scratchpad: when data arrives, where it goes and when it leaves are all fixed by the compiler. Nothing decides anything at runtime.

That distinction is the main thread, so the question is a gift: “Right. What's missing isn't capacity, it's who decides.”
⚠️ Don't go into VMEM size or partitioning at the opening. Section 3 has a figure for it.

Why make FlashAttention the star? Why not a GEMM, or a whole model?

It meets two conditions that rarely coincide.

First, it runs on one chip. No inter-chip traffic, so the story stays on one line (hence Section 4, on inter-chip links, never mentions it).

Second, it hits four things at once: memory access, shapes, on-chip capacity, kernel authorship. A pure GEMM is too clean to touch the last three. A whole model is too messy: the subject changes every step; nothing lands.

Section 4 has nothing to do with FlashAttention. Why is it in the core track?

It really is off the FlashAttention thread. The test is hard: FlashAttention appears 0 times in Section 4's main text (grepped). It finishes on one chip and never touches inter-chip links.

But it underpins Section 6's measurements: those numbers come from 64 vs. 64 chips, wired in structurally different ways. Without it, Section 6's ratio can't be read. So it stays in the core track; don't skip it whole. Short on time? Two sentences (emergency list, top).

What not to say

Four likeliest drifts ① Don't compare specs at the opening. Compute, bandwidth and SRAM are Section 1's punchline. “The two are nearly identical” only lands once the line is drawn. Spoil it now and Section 1 is wasted.
② No company history. The origins take two sentences (“doesn't know what you'll run” / “knows from day one”). Anything more is lost time.
③ Don't explain MXUs or warps here. Section 3 has thirteen microscope figures for that. An opening glossary is the costliest drift: ten minutes gone.
④ Don't say “which is better.” If someone steers toward chip selection, deflect with the course page's line: only “who decides this, on this side.”
Pacing Eight minutes, max. This is the foundation, not the content. The sit-up moment is at the end of Section 1 (312.9 vs. 312.5). Get there early.
If the audience is very mixed, cut origin detail, but always say “they differ everywhere, but every difference follows from one decision.” Without it, every later section is an isolated fact.
⚠️ This line isn't in the figure. It lives only in your mouth and the course page's green box. So don't drop it.
01

The dividing line: same appetite

⏱ 10 min
What this segment must leave behind 312. In the time it takes to move 1 byte from HBM, the hardware can do about 312 BF16 operations. No conclusions here, just one number; every later op is checked against it.
⭐ By the end, also say what the line is for: judging whether fusion is worth it. The lecture's through-line gets its name here.

Script

Scroll to Section 1. Three figures (the 312 one, the three-panel one, Figure 1-8 in §1.2). Keep it brisk.

I just said there's only one root cause. But before comparing anything, we need this machine's exchange rate between computing and moving data. That's all this section does: get that number, no conclusions.

Both spec sheets give two numbers: operations per second and bytes moved per second. Divide one by the other. The ratio is very concrete: in the time it takes to move one byte, how many operations can this hardware do?

This is easy to say backwards. Slow down. Not “312 operations need 1 byte of input,” but “while one byte moves, the machine can do 312 operations.” So it's a floor: for every byte moved, your algorithm must do 312 operations just to keep compute fed. Do fewer, and the spare compute sits idle. Stop on the two-panel figure (fig1-4). Don't scroll on. Let the numbers sit.

On the left, TPU v7: 2,307 TFLOPS per chip, 7.37 terabytes per second (= 8 × 921.6 GB/s; this course uses 7.37 throughout; the 7.38 or 7.4 seen elsewhere is the same number rounded). That's 312.9.

On the right, GB200, the GPU in NVL72: 2,500 TFLOPS, 8.0 terabytes per second. That's 312.5.

Dead stop. Don't explain the numbers. Let the room notice they're the same. This is Section 1's only big reveal.
⭐ How to stop (added after the 2026-09-10 run-through): after “312.5,” shut up and count two beats. “Dead stop” said stop, not for how long, and on stage the default is to keep talking. A cue with no concrete action is no cue. (Same lesson as the side track's XLA error: the notes said “Read it out,” but that English line can't be read aloud.)

Two companies, two architectures, two design philosophies. The ratios agree to the first decimal. FP8 too: 625.8 versus 625.0.

⛔ Never say “a relative difference of 0.13%.” That third digit is fake. GB200's “8 TB/s” has no stated basis, and the decimal and binary readings differ by 2.4%.
Say this instead: “Two companies, two architectures, two philosophies. Compute divided by bandwidth lands in the same range.” Just as surprising, and it survives questioning.
⛔ That reasoning is for you; don't say it (I did, in the 2026-09-10 run-through). Rule: any “why I'm saying it this way” line belongs in the notes, not on stage. Said aloud, it pulls the room onto your method.
Point at the blue line in the main text (“both sides have eight 24 GB HBM3E stacks; why different bandwidth?”). Raise it yourself; it's always asked. ⭐ One-line answer: “HBM3E” names the generation, not the speed grade.
Each stack has 1,024 data lines; bandwidth is that width times the per-pin rate. 8.0 to 9.6 Gbps all count as HBM3E in this generation. v7 runs 7.2, GB200 about 7.8. The whole difference is the pin rate.
Key point: the buyer picks the grade, for power, cooling and reliability. Not the chip's ceiling.
⭐ Bridge in words; don't jump cold (added 2026-09-10): lead in with “While we're here, there's a question I get every time, so I'll take it now.” Lead out with “And now the one thing I most want to say in this section.”
These were two adjacent stage cues with nothing sayable between: an abrupt jump.
⭐ This question turns the course's own test on itself.
⭐ Then the line that matters most: “And ‘binary or decimal?’ is one of the four questions Section 6 teaches. Here it hits my own headline number. Those four questions aren't for auditing other people. Aim them at yourself first.”

But the next sentence must follow: “both land on 312” is a fact. “Why both land on 312” is my inference. No vendor has explained it. And it depends on the SKU: the B200 on an HGX board gives 281.3, 11% off.

So from here on, 312 as a test means the qualified version. When you quote it, keep the qualifier, not just the number.

Scroll one screen: the figure continues with three unequal bars. Read, don't explain. 20 seconds.

Quick aside: that division works at every level. HBM is 312, L2 is 119, shared memory is 64. The closer to compute, the lower the line.

Don't expand. The room doesn't know yet what it's for, so it won't stick. It returns with fusion at this section's end, and lands there.
⚠️ But give the forward pointer: “Hold on to that. We'll use it at the end of this section.” (Added in the 2026-09-10 run-through.) A “Read, don't explain” with no signpost reads as a stray aside. A hook must say where it pays off, or it's noise. The side track hit the same rule: X1 planted a hook and named the wrong payoff section.
Back to the two panels. Same figure.

Same ratio. Could numerator and denominator happen to be proportional? No. The panels give all four: compute 2,307 vs. 2,500, bandwidth 7.37 vs. 8.00. Both within 8%.

The section's real payoff: not just the ratio: numerator and denominator each nearly match. The “coincidentally proportional” escape route is closed.

The spec table's ten rows look equally weighty. Only one kind is worth remembering: a level one side has and the other lacks entirely. The TPU has no hardware-managed on-chip cache level. The GPU has no programmable dedicated coprocessor level. The gap isn't capacity. It's who decides. Section 3 shows them; not a word more here. The rows are folded at the section's end, for lookup; don't open them now.

Scroll to the three-column figure “Using the line” (second figure); stop on column ①. The second time sink. Don't skim it as a wrap-up.

We have the line. But 312 alone is useless. It's the hardware's number. You also need the op's number, so there's something to compare.

The op's number is arithmetic intensity, and the method is one sentence: count the bytes it moves, count the operations it does, divide.

The section's most practical sentence. Say it slowly. It's worth more than 312: 312 changes each hardware generation; the method doesn't. Scroll to the three-column figure “Using the line”; stop on column ①. The math is on it; read along.

Let's measure two right now.

First, a vector add. y = a + b, in bf16. Read a, two bytes. Read b, two bytes. Write y, two bytes. Six bytes moved. Operations? One add. So the intensity is 1 over 6, or 0.17.

Against 312: over 1,800 times too low.

What does that mean? Run this line on GB200 with all 8 terabytes per second saturated, and you get 1.3 TFLOPS. The chip's peak is 2,500. That's 0.05% of it. Make compute ten times faster and it won't gain a second.

Pause and ask: “So would a more expensive chip make this line faster?” Let them answer “no.” Move to column ②.

Second, a matmul. Square, N by N, bf16. The operation count is 2 N cubed. Moving three matrices in and out is 6 N squared bytes. Cancel, and the intensity is N over 3.

To reach 312? A side of nearly a thousand.

Flip it: a side-256 matmul has intensity of only 85, still bandwidth-bound. “Matmuls are compute-bound” is a myth. Only big ones are.

Heads usually come up here. It overturns something almost everyone assumes. Same figure, column ③. These 40 s begin the through-line. Don't rush.

Now we can name the real protagonist of this lecture.

Both calculations said one thing: whether an op is fast depends on operations per byte moved. So there's one way to make a slow op fast: shrink the denominator. Move less.

And the best way to move less is to merge adjacent ops into one kernel, so intermediates never touch HBM. That's fusion.

Let it land: it doesn't save a single FLOP. Only the denominator changes. Many think fusion means computing less. It doesn't.

So on this line, fusion pushes the op to the right, from the bandwidth side to the compute side.

Still column ③. Point at the yellow box below (312 → 64). Added because the last line gets misheard.

I said intermediates never touch HBM. That's easy to hear as “the intermediates disappear.” They don't.

They're parked in on-chip scratch: the shared memory on the GPU, up to 227 KB; VMEM on the TPU. HBM traffic really drops, but the data still has to land on chip.

Pause: data movement didn't vanish; it moved a level. The 3 bars were for this.

And the on-chip level has its own line. We read it earlier: 64, not 312. So after fusing, redo the math: compare the new intensity with 64.

Add three sentences, or the last one contradicts itself: 312 to 64 lowers the bar. It doesn't raise it.

So why still hit a wall? The denominator moves too, not just the bar. Fusion turns HBM traffic into on-chip traffic. On-chip bytes can grow tens of times, while the bar drops only 5×. A bigger denominator, not a higher bar.

Someone usually asks: “So why not keep it all in registers?” Usually you can't. An SM's register file totals 256 KB, the same as shared memory. And every extra register you take means fewer resident warps. The numbers are in the SM breakdown figure in §3.2.

1.2 Square is a special case: the smallest dimension sets intensity (Figure 1-8)

Scroll to Figure 1-8. ⏱ 3 min here, already counted in §1's 10 minutes.
⭐ Forced by live follow-ups on 2026-09-10; not in the original design. Verbatim: “What about M×K · K×N? Intensity should depend on the contraction dimension K; longer K, higher, right? And attention with head_dim=128 falls badly short. So what changes after Flash?”
⛔ All three are right, and they exposed a broken thread on the course page: §1 covered only square matrices, §2 jumped to Flash, §3.4 treated the contraction dimension alone. All about the same 128, never connected.
⚠️ So this part takes three minutes, but never skip it: it's the seam §1 → §2 → §3.4.

The matmuls in those examples were square: all sides N, giving N/3. But a real matmul is M×K · K×N. So which dimension sets the intensity?

Same division. Operations: 2 M N K. Bytes moved: 2 times MK plus KN plus MN. It simplifies cleanly: one over intensity equals one over M, plus one over N, plus one over K.

Say “reciprocals add” on its own, once. It's the key: reciprocals add, so the smallest one dominates.

So “longer K, higher intensity” is right, and monotonic. But there's a ceiling. Column ②'s five rows show it: M and N fixed at 512, K from 64 up to a million, and intensity climbs from 51 to 256, then stops. The cap is MN over M plus N: exactly 256.

K can't rescue a small M or N. The harshest case: decode at batch size 1. M is 1, so intensity caps at 1, however big K and N get. That one line is why LLM decode is always bandwidth-bound.

Now attention. In Q times K transpose, both outer dimensions are the sequence length S, and the contraction dimension is head_dim. Most models use 128, so intensity is about 128. Below v7's 312 line, and v6e's 560.

⭐ Say it aloud; it overturns a common assumption: “Attention is a matmul, so of course it's compute-bound” is a myth. Unfused, it sits on the bandwidth side, pinned by head_dim.

So what does Flash change? Direction now; Section 2's first figure does the math: it saves no FLOPs. It changes the denominator. The S×S matrix never touches HBM. Only Q, K, V and O cross it, so intensity becomes S over 2. With Wan's S of 75,600, that's 37,800.

⭐⭐ The payoff is the last line. Don't rush:
The same 128 bites twice. Here, a contraction dim of 128 caps intensity. In §3.4 / §3.6, the 256 × 256 MXU gets only 128: half idle.
First: “wait for data?” Second: “is the unit full?” Two issues, one number.
⛔ A basis trap. Flag it yourself, or it breaks at §2:
Figure 1-8 puts attention's intensity at ≈128; the figure in §2.3 says 64. Both right; different denominators. 128 is one matmul in isolation (A/B/C each cross once). 64 is naive attention overall (S and P each round-trip, doubling the denominator).
⭐ Figure 1-8's yellow band says exactly this. Point at it; read it once, or a careful reader will think one figure is wrong.

OK: the line, its use, its purpose. Section 1 produces one thing: a number, and how to compute it. The number changes every hardware generation. The method doesn't.

⛔ Since 2026-09-05, the course page's “two outputs” box is gone. Its item ② said “only what one side has and the other lacks entirely is worth remembering,” but §1 never derives that test; where 3.5 / 3.6 use it context makes it self-evident, no advance definition needed. Don't add it back on stage.

Every later section sits on this line. Section 2 measures before and after fusion: same attention, two implementations, identical FLOPs, 1,000× apart in intensity. Section 3, how it lands on each chip. Section 5 answers the real question: who should decide?

Likely questions

This 312 is for BF16. What about FP8? Does the test change?

Yes, it becomes 625. The figure has both rows: TPU 625.8, GB200 625.0.

Simple: compute doubles, bandwidth doesn't, so the ratio doubles. It doubles on both sides, so they still line up at FP8.

Use the line for the precision you actually run. Judge an FP8 op against BF16's 312 and you'll call compute-bound ops memory-bound.

Why does the TPU panel say “per chip”? My logs don't match.

A v7 chip holds two devices, and framework logs always report per device. So: per-chip = per-device from the log × 2.

Here both sides are per package: one TPU chip vs. one GB200 GPU, like with like. The basis is in the figure's small print; point at it.

⚠️ This 1:2 also affects batch: the “device” in per_device_batch_size is half a chip on v7. But don't expand; that's another course.

Does compute ÷ bandwidth have an official name?

Yes: the x-coordinate of the roofline ridge point, or machine balance. The op-side counterpart is arithmetic intensity. Same unit, so they compare directly.

Name it, move on. Don't draw a roofline. It takes five minutes; Section 2's memory-access figure says it more concretely.

⭐ With 148 SMs, is shared memory's line of 64 an approximation summing all SMs?

Credit it first: the sharpest arithmetic question in the course. But the answer is firmer than they expect: nothing is summed. That number is per SM. Numerator 8,192 FLOP, denominator 128 bytes, both per SM per cycle.

Does whole-chip scale change it? No. Not an approximation: an identity. Chip compute = 148 × per-SM × clock; chip shared-memory bandwidth = 148 × 128 B × clock. 148 and the clock appear top and bottom, so they cancel. The course page's back-check line does exactly this: 2,500 ÷ 64 = 39 TB/s for the chip; divide by 148 × 128 and the clock is 2.06 GHz. That's GB200 (all of §1 uses it).

⛔ The course's back-check line used to say “matches B200.” Wrong label; it now says GB200 (caught re-checking §3 on 2026-09-10). 2.06 GHz is GB200: 2,500 ÷ (606,208 × 2) = 2.062.
The HGX B200 has different numbers: peak about 2,250 TFLOPS, third-party boost 1.83 GHz (about 1.856 backed out from 8,192 FLOP/SM). The same formula gives ≈ 35 TB/s, exactly the row in §3.2c.
⭐ So §1's 39 and §3.2c's 35 don't conflict: two different chips. If asked, say that. And never say “2.06 matches B200” on stage: the course's own GPU Microscope pins B200 at 1.83/1.856, and someone will catch it.

⚠️ But the other two lines (312 and 119) are whole-chip numbers. They sit side by side only thanks to that cancellation. Say so on the spot, or careful listeners will suspect the figure switched bases.

⭐ The question's real value: it forces out “same ratio, completely different reachability.”
HBM's 8 TB/s is one resource that 148 SMs fight over. One SM can't saturate it; with all 148 reading, each gets a few tens of GB/s.
Shared memory's 128 B/cycle belongs to each SM. One SM or 148, each gets that much. The total scales linearly.
So fusion doesn't just drop the bar from 312 to 64. It trades a contested resource for one you own.

⛔ Also fix one direction: a lower bar is good. The smaller I*, the easier an op turns compute-bound. So why does overfusing hit a wall? A bigger denominator, not a higher bar: the bar drops only 5×, while fusion moves traffic on chip, where it can grow dozens of times.

Is arithmetic intensity a property of the op, or of the hardware?

Of the op. 312 is the hardware side: “while one byte moves, how many operations can I do?” Intensity is the op side: “per byte moved, how many do I actually do?” Same unit, so they compare directly.

One trap, which sets up the next section: intensity depends not just on the math but on the implementation.

Same attention, naive vs. Flash: identical FLOPs, 1,000× apart in intensity. The difference is all in bytes moved. That's Section 2. If asked, preview it in one line; don't finish the story here.

⭐ 312's numerator counts only the matrix units. Don't CUDA cores and the VPU count?

The section's best question; someone always asks. Credit it first. 312 is indeed the “matrix roofline” ridge point; vector units are left out.

First, why: the ops this course judges (attention's two matmuls, the MLP's big matmuls) put nearly all FLOPs on the matrix units. Softmax, normalization and such are rounding error.

Then the magnitude: GB200's 18,944 CUDA cores, 2 FLOP per cycle at about 2.06 GHz, total 78 TFLOPS. The matrix side is 2,500, 32.5× more. The TPU is structurally the same: two MXUs do 262,144 MACs per cycle (131,072 each); the VPU does 1,024 elements per cycle.
⚠️ That clock is derived. Say so. NVIDIA hasn't published GB200's boost clock.

Then concede: strictly, element-wise ops belong on the vector roofline, whose ridge point is 9.6, not 312.

Then bring it back: the conclusion holds. The vector add's 0.17 is 58× below even 9.6: still bandwidth-bound. Bad either way.

⭐ Then turn it into a takeaway, the question's real value: weak vector units aren't an oversight; both sides chose them. An op that lands on the vector unit loses twice: once to bandwidth, again to having a thirtieth of the compute. That's why fusion is mandatory: not to compute less, but to keep data off this narrow, weak path.

Folded at the end of Section 1 on the course page. Don't raise it; open it only if asked. It's a denominator dispute; the conclusions stand.

Why are the two vendors' specs so similar? Did they copy each other?

I'm not sure; no vendor has explained it publicly. Just say so.

You may offer a labeled inference: both size HBM bandwidth to matrix compute on the same HBM generation, so matching ratios aren't surprising. But that's my back-calculation, not a source.

⚠️ Easy to slip into causation here. “The same workloads pinned both sides to the same ratio” sounds great, but it's reverse-engineered from two numbers, and another SKU breaks it (281.3).

What not to say

Five likeliest drifts ① Don't expand on the roofline. If asked, name it; don't draw it. Five minutes minimum; Section 2 is more concrete.
② Don't read the ten-row table line by line. It's folded on the course page for that reason. The main text needs one sentence: “only one kind is worth remembering: a level one side has and the other lacks entirely.” The rows are for lookup.
③ Don't linger on SparseCore or CMEM. Section 1 only needs “each lacks one of the other's levels.” Section 3 gives those levels a figure.
④ Don't turn 312 into “TPU and GPU are the same.” This section shows “the spec sheet can't tell them apart,” not “they're the same,” which the next section refutes.
⑤ ⭐ Don't explain how to fuse. This section only names it, in forty seconds. Which ops merge, how the compiler finds fusions, why fusion sometimes fails: not one word. It's the course's easiest twenty-minute tangent, because it truly is the key topic. But its evidence is in Sections 2, 3 and 5, not here. Covering it now spends three sections' time on a conclusion with no evidence yet.
Pacing Ten minutes: §1.1 seven (two figures + two calculations + one name), §1.2's Figure 1-8 three. Three key moments, not one.
One: the pause on 312.9 vs. 312.5. Read them, then count two beats in your head; let the room discover it. The course's first sit-up moment; rush it and it's gone.
Two: the two bits of arithmetic at the end. Skimming them as a wrap-up is the costliest mistake. Without them, 312 is just memorized; with them, students can measure any op on the spot. Keep both lines: “ten times the compute won't make this line faster” and “a matmul with side 256 is memory-bound.”
Three: the last forty seconds, naming fusion as the through-line. It must come right after the arithmetic. The room just worked out that the denominator decides it all, so “shrink the denominator” feels like their own conclusion. Anywhere else, it's a slogan.
This segment uses three figures (the 312 one, the three-panel one, Figure 1-8). On 2026-09-03 we deleted the old “three-panel takeaway figure” (two panels repeated earlier content, one said “nothing to see here”). The saved time goes to the last 40 seconds (naming fusion). End on the last blue line: “That's the end of this section. You now have a ruler. It answers one thing: bandwidth- or compute-bound?”
Then the hook, at once: “Next section, we use it on a real op, and get the lecture's first big number: fusion changes intensity 1,000×.”
02

⭐ Where they split: one load, 4 decisions on one path, 0 on the other

⏱ 10 min
Takeaway 4 vs. 0. For the same “fetch a block from HBM,” the GPU path has four decision points answered only at runtime. The TPU path has none.
The opening claim, “only one cause,” becomes visible for the first time.

Script

Scroll to the start of Section 2; stop on the load-path figure (fig2-1). This is the center of the course. Leave it on screen longer.

Last section: both chips have the same appetite. This section: they eat completely differently.

⭐ Decision one needs a sentence, not jargon.
“Coalescing means merging. On a load, each of the warp's 32 threads computes an address; the hardware merges them into as few memory transactions as possible.”
Add the magnitude: “Good vs. bad merging: an 8× bandwidth swing, known only at runtime; the compiler can't see how addresses are computed.”
⛔ The surest way to wreck this segment: “the TPU has no cache.” That shrinks a whole mechanism to one part. And decision one isn't the cache at all; it's whether the warp's 32 addresses merge.
Deliver “4 vs. 0,” not “cache or no cache.”

No specs. We just follow one load: one block of data, from HBM onto the chip. How many decisions along the way? Who makes each one, and when?

Left half (GPU) only. Point at one decision point at a time; at each, say “This answer doesn't exist at compile time.”

Left side, GPU. A warp has thirty-two threads, each with an address. Decision one: how many cache lines do they merge into? You find out at runtime.

On to L1. Decision two: hit or miss? Then L2. Decision three: hit or miss? On a miss, something gets evicted. Decision four: which line?

Four. And notice: I didn't call any of them “slow.” Red doesn't mean slow. It means “known only at runtime.”

Right half (TPU). Go fast: there's nothing to count.

Right side, TPU. Compile time: XLA tiles each array into 8×128 blocks. Compile time: it decides when to move, and how much. Compile time: it works out what fits in VMEM. At runtime, two things remain: DMA moves whole tiles; the vector unit reads them directly.

Zero runtime decisions on the path. And the bottom cell is empty: no cache layer. So hit, miss and eviction don't even exist.

Dead stop. Read the figure's verdict: “Main path: 4 vs. 0.” This is the center of the course: Section 0's thesis lands for the first time. Don't explain it. Let it hang on screen.

And green? Not “fast” either. It means “fixed when the code was written.”

That sets the division of labor. The GPU leaves these decisions to the hardware at runtime, really to people, the hand writing the kernel. All you can do is help the hardware guess right: contiguous access, no bank conflicts, hand tiling.
The TPU has the compiler settle them at compile time, so you live with its quirks.

Flip back to Section 0's green box, “Up front: we don't rank the chips,” or just say: It's not who's smarter. It's two routes, each betting on one thing. Say it. It heads off every later “so which is better?”

The GPU bets on runtime adaptation. Any shape runs, right or wrong. The cost: hit rate shows only at runtime; if bad, you can only retry.
The TPU bets on a compile-time plan. Shapes fixed up front; no cycle wasted deciding. The cost: no runtime rescue; a bad shape means changing code.

⭐ Add one line; it closes the §0–§2 thread: “Only now is ‘hedge at every step’ more than an adjective. It's these four decision points. Each is a real circuit, waiting on every access to be asked: did it coalesce? Did it hit?” The opening's abstract line is now a hardware cost.
⛔ Stop there. Don't go on to “so who got the saved area?” The course retracted that line: no source.

Everything in Sections 3 and 5 follows from this.

⛔ Lane / sublane is no longer taught here. It moved to the start of 3.3, where it's first used. Wrap up and scroll straight to 2.3. Scroll to 2.3; stop on the Flash comparison figure (fig2-3).

We have the criterion and the coordinates. Now we run one op end to end: FlashAttention. The reason is practical. Every large model runs it today, and it sits on the wrong side of 312, so both sides have to act.

Remember: one head, 128K sequence, bf16, the score matrix is a full 32 GiB.

Give a comparison, then pause: a v7 device has only 94.74 GiB allocatable. Not even three heads. The model has 128.

The naive version? Compute the score matrix S and write it to HBM. Read it back for softmax, write it out again. Read it again to multiply by V. Four trips of 32 GiB, for one head. And the real math is just two matmuls.

Flash? Split Q, K and V into row tiles; take one tile at a time. That tile's S lives only on-chip, never in HBM. Softmax goes online, keeping just two scalars: a running max and a running sum. Accumulate into the output tile, drop S on the spot, and on to the next tile.

The upshot: that seq-by-seq matrix never appears in HBM. Not once.

Point at panel ③. It's the bridge to Section 3.

Panel ③: how big a tile? Same question on both sides; different people answer it.
GPU: the kernel author. Budget: 228 KiB of shared memory; tile size is a tunable hyperparameter. Wrong still runs, just slower.
TPU: the compiler. Budget: 64 MiB of VMEM, but shapes must align to 8×128, or get zero-padded; zeros still take VMEM, pass the MXU.

Scroll to the last figure, the 312 axis (fig2-4).

Now put that back on last section's line.

Naive: arithmetic intensity 64, 64 operations per byte moved; in that byte's time, the hardware could do 312. Nearly 5× short: ~80% of compute idles, pinned to bandwidth.

Flash: 65,536. Same FLOPs, 1,024 times fewer bytes. It jumps straight to the right, compute-bound. Exactly where it should be.

⚠️ Say right after: 65,536 is an ideal upper bound. Real Flash re-reads K and V, so it never gets there. The conclusion holds: it has crossed 312, left to right.

Don't scroll past. Below the axis is a new green box, “Does it clear the on-chip bar too?” Two minutes, then the orange box.

I said Flash “crosses to the right.” It crosses the HBM line, 312.
But there's a second line on-chip: shared memory, at 64.

⚠️ Call this out unprompted: two 64s are on screen. The axis one: naive attention's intensity; the new box's: the shared-memory threshold. One is the op's, one the hardware's; pure coincidence. Skip this and the room takes them as one number.

And fusion does exactly this: moves bytes from HBM onto the chip. So we redo the division with a new denominator.

Point at the green box's five-line derivation. Read every line.

A tile has Br rows and Bc columns; head_dim is d.
Each step reads a K tile and a V tile from shared memory, each Bc by d. The Q tile stays in registers.
Compute: two matmuls, 4·Br·Bc·d FLOPs. Movement: 4·Bc·d bytes.

After cancelling, wait a second before the answer: on-chip intensity equals Br, the tile's row count. Letting it surface beats announcing it.

Br of 128: 128 against 64, it clears.
Br of 32: 32 against 64, memory-bound again, on-chip.

⭐ Then give the anchor: real FlashAttention kernels always use 64 or 128 rows per tile. Not a coincidence. This division forces it.

⚠️ Give the caveat: this counts only the shared-memory level, not softmax's vector math or the register level. It's an order of magnitude, not an exact value.

⚠️ TPU numerator: 1,153.5, not 2,307. VMEM is per core, so the denominator is per core, and so is the numerator. The third of the four questions: get it wrong, you're off 2×.

Small print on the last line (TPU side): read it verbatim. Add nothing.

On the TPU the method is identical. Numerator 1,153.5 (per core); denominator: VMEM bandwidth. That number isn't published, and we won't guess it.
Only the direction: on-chip bandwidth is an order above HBM, so the threshold is an order lower, in the tens, the same order as the GPU's 64.
And the TPU's workbench is 64 MiB; what limits it isn't this line but 3.4's blade width.

Scroll to the closing orange box, “Look back: that was operator fusion.”

So, look back. Flash fused the chain “compute scores → softmax → multiply by V” into one kernel. That's the name from last section: operator fusion.

Pause: not one MAC saved. Panels ①② are identical. All savings are in the denominator.

So the premises are in place: the criterion is 312, naive is 64, so it must be rewritten.

The real question isn't “should we fuse?” It's “who does the fusing?” This time, people did: one paper and a pile of hand-written kernels. Section 5 asks: does it have to be people?

Likely questions

GPUs have shared memory too. Isn't that a programmable scratchpad?

Yes, and it's the best question here. Agree first. Shared memory is a software-managed on-chip scratchpad, the same kind as VMEM.

Two differences. Give both:

① The TPU has only this kind; the GPU has both. All four decision points sit on the global load path. Shared memory or not, data first comes from HBM along it. You can't route around it.

② Who manages it. Shared memory: the kernel author. VMEM: the compiler. That's panel ③'s “same question, different people,” and it's Section 5's topic.

⚠️ Don't get into shared-memory size or tiling. Section 3 has a figure for it. Say only “can't route around it” and “who manages it.”

Why is bank conflict a dashed branch, not one of the four?

Banks partition shared memory. They aren't on the global load path. Counting them would distort “4 vs. 0,” the center of the course. Don't blur it to add one more.

The figure says “+1 via shared.” If asked, point at that small print: we know about it and put it on the branch on purpose.

Where does 32 GiB come from?

The derivation is on the figure. Point at it; don't do mental math. 128K is 2 to the 17th. The score matrix is seq × seq; bf16 is 2 bytes. (2¹⁷)² × 2 = 2³⁵ bytes = exactly 32 GiB.

Stress “exactly”: it's not an estimate; it lands on a power of two.

Does Flash really reach 65,536?

No, and we must say so before anyone asks. Tiled Flash re-reads K and V, so real intensity is well below the bound.

65,536 assumes Q, K, V and O each pass once: the ideal where not one extra byte moves.

The conclusion is untouched. The criterion is 312. Even 1% of the bound is 655, still on the right. This section needs “it crossed,” not the exact number.

⭐ Is the saved movement paid for in extra FLOPs? Do those count as “useful”?

The section's second-best question, and we invited it. The orange box just said “not one FLOP saved,” which sounds free. Admit it: usually you pay.

Three forms of payment, different in nature. Take them separately:
① Recomputation. Flash's forward has none (panels ①② match); backward does: the intermediate matrix wasn't stored.
② Padding. Alignment zeros take space, go through the MXU, produce nothing. The worst: it buys nothing you can name.
③ Extra vector work. Online softmax rescales the accumulator every tile: few ops, but on the vector unit, so FLOP counts understate it.

Then “is it useful?”: formally, MFU vs. HFU. Numerator = FLOPs the model's math needs: MFU. FLOPs the hardware executes: HFU. MFU never counts recomputation or padding as useful.
⛔ Say only that, then: “Section 6 covers these two terms properly.” That section asks whether two numbers can be compared; these conventions are its core. Expand here and you teach half of Section 6 early, and neither lands.

⭐ Finally, tie it back to the line; that's the payoff. First, one precise point:
Intensity's numerator is hardware-executed FLOPs; hardware can't spot recomputation, so the 312 test includes it.

So there are two ceilings; the lower wins:
Useful throughput = min(peak × useful fraction, bandwidth × useful intensity)
Example: intensity with recomputation is 600, half of it recomputation. Compute ceiling: 50% of peak. Bandwidth ceiling: 300 ÷ 312 = 96%. Take the smaller: half the roofline.

⚠️ Bandwidth isn't saturated either (312 ÷ 600 ≈ 52%). The matrix unit is 100% busy, half on garbage, while bandwidth idles. That's the fingerprint of overtrading.

The rule: while memory-bound, extra compute is free. Past 312, it cuts output proportionally. A cliff, not an exchange rate. Trade until intensity with recomputation reaches 312, then stop.

⛔ Don't let it become “intensity shouldn't go much above 312.” Useful intensity: higher is better, no limit (a 4096-side matmul: intensity 1,365, zero waste, 100% of peak). Only what you manufactured yourself should hug 312.

Flash is far from the bad case. Forward: 2 matmuls; backward: 4, plus 1 recomputed. 7 vs. 6: 85.7% useful. Pay 17% extra compute, save three orders of magnitude of movement.

Collapsed on the course page, under the orange box; don't open unprompted. Only if asked; full answer: 3 of Section 2's 10 minutes.

Does the TPU have FlashAttention, or not need it?

It does, and needs it as much. The 312 line applies to both: naive attention is memory-bound on the TPU too, and 32 GiB doesn't fit.

The difference is who sizes the tile: on the GPU, the kernel author tunes it; on the TPU, the compiler computes it, within 8 × 128.

That's all of §3.6, so just say “yes” and move on. ⛔ No Pallas in Section 2.

What not to say

Four ways this segment drifts ① Don't call red “slow.” It's the most damaging mistake here. Say “the GPU path is slow” and the course becomes a performance shootout. Red means one thing: known only at runtime.
② Don't explain how caches work. Set associativity, replacement, TLB: touch none of it. All we need: “there's a layer, and its answers don't exist at compile time.”
③ Don't teach lane / sublane here. It moved to the start of §3.3, where it's first actually used. Explain 8 × 128 fully here and Section 3 has nothing left.
④ No online-softmax math. “It keeps a running max and a running sum” is enough. The recurrence takes five minutes and has nothing to do with the hardware.
Pacing Ten minutes, three figures + one green box, not split evenly: the first gets the most.
fig2-1: three minutes, with room for the “4 vs. 0” pause. It's the center of the course: Section 0's thesis gets its first evidence here. Section 3's fourteen figures and all of Section 5 are footnotes to it.
fig2-3 plus fig2-4: three minutes. Pause on 32 GiB, on 64 vs. 65,536, and on “not one FLOP saved.”
The green box under fig2-4: two minutes. The remaining two go to the closing orange box and transitions.
⚠️ Most likely overrun: fig2-3's panel ③. It's dense and slides easily into Section 3. Its only job here is to raise the question, not answer it.
⚠️ Second: the final collapsible (fusion's cost / MFU vs. HFU). It's collapsed; the main line skips it. But someone will almost surely ask, usually the most expert person in the room. The answer is under “Likely questions.” Give it, move on. Don't drift into accounting conventions.
03a

Taking it apart: one complexity, two homes

⏱ 42 min
Takeaway Two things, both required.
① Section 2's “4 vs. 0” is not a slogan. From the whole chip down to one instruction, every level has a physical source you can point at.
② Every difference must point back to a specific component or instruction. No physical source, not in this section.
In one line This half hands out the parts. Reassembly and the bill come in the second half.

Script · 3.1 Whole chip first

Scroll to Section 3; stop on the whole-chip GPU figure (ms-g1, Figure G-1). Fourteen figures. Set the pace now: fast early, spend on 3.6.

Both chips have the same appetite, but eat completely differently. Now we actually take the chips apart to see where that difference lives.

Whole chip first. A B200 package holds two dies, joined by a 10 TB/s bridge. But CUDA shows you one device. The seam is hidden.

Switch to the TPU figure (ms-t1, Figure T-1). Point at both, back to back: a pair.

TPU v7 is also two dies, but honestly shows up as two devices, each with its own 96 GB of HBM; neither sees the other's address space. Reaching the other half is a step you write.

Pause: “Seam in the open, or seam hidden.” That's Section 2's decision at the package level.

Hiding the seam doesn't make the cost go away. You still pay; you just don't see the bill. Crossing dies, bandwidth drops from 21 to 16.8. The numbers are in the 3.2c table.

⚠️ Don't multiply the figure's 131,072 straight into peak.
It counts cells: two MXUs, 2 × 256² in all.
And each cell does two MACs per cycle. The figure doesn't show that step.

⚠️ Say this unprompted: “Two MACs” does not mean “one multiply plus one add.”
One MAC already is a multiply and an add: a times b plus c, in one cycle. That's two FLOPs.
So one cell per cycle is four FLOPs, not two. Mishear this and the whole chain is off by 2×. Whiteboard: 1 MAC = 1 multiply + 1 add = 2 FLOP; 2 MACs = 4 FLOP.

Read the wrong formula aloud, don't just say “it's wrong”: 131,072 × 2 cores × 2 FLOP × 2.2 GHz = 1,153, exactly half the header's 2,307. Missing: “two per cell.”

⭐ If asked “So what's the right chain?”, preview 3.4's bridging line: “The figure gives a cell count. The 262,144 in 3.4 is FLOP per MXU per cycle. Divide, and it's 4 FLOP per cell: two MACs.” Don't work the math in 3.1, or the pacing collapses.

Script · 3.2 Taking one core apart

Scroll to the SM figure (ms-g2, Figure G-2).

One level in. First what the GPU has, then what the TPU doesn't. Keep that order. Flip it and the TPU looks like a stripped-down version.

Inside one SM: warp schedulers, a 256 KB register file, 228 KB of L1 plus shared memory, scoreboards. Together they do one job: hide latency and branches dynamically, at runtime.

Switch to the TPU figure (ms-t2, Figure T-2). Point at the five empty cells.

On the TPU, none of these five exist.

The point isn't “five things missing”: the five are really one thing. Do you switch tasks at runtime? If yes, you need dozens of resident contexts, something to pick what runs next, a way to know whose data arrived: all five. If no, all five lose their reason to exist.

The subsection's takeaway. Stop on it: “Having a cache and having a warp scheduler aren't two questions. They're two faces of one decision.”

And Section 2's four red boxes are what you inevitably get once you say yes.

Script · 3.2b “But the GPU has more on-chip memory”

Scroll to the ten-row SRAM table (no figure). Talk fast; stop on the three-yardsticks box.

Meet an objection head-on: “You say the TPU spent its area on scratchpad. But the GPU has more on-chip SRAM.”

They're right. We list every on-chip SRAM on both chips: GPU 231 MiB, TPU 134 MiB. The GPU has 73% more. Don't argue this one.

Concede, then turn it over. Order matters: concede first, and the other two yardsticks get heard.

But the same table gives three conflicting verdicts, depending on what you ask.

Yardstick one: total SRAM? The GPU wins.
Yardstick two: only the row software uses as data scratchpad? TPU 130 vs. 37, about 3.5×.
Yardstick three, the one that matters: how big a contiguous workbench can one compute unit use? TPU 64 MiB, GPU 227 KB: 289×.

Point at “The GPU's 37 MiB isn't one block.”

Why such a gap? Because the GPU's 37 MiB isn't one block. It's 148 islands that can't see each other, 256 KB each, with L1 taking part of each. One thread block can explicitly manage at most 227 KB.

Dead stop. That 289× gets used directly in 3.6, when we size each side's tiles.

One more must-say: yardstick one is correct, but the most misleading. Nearly all of the GPU's 97 MiB lead comes from the L2 row, the one cell in this table you can't arrange. You can't “park” weights in L2, only issue loads and hope they're still there.

Script · 3.2c “Speed”

Scroll to the bandwidth table. One counterintuitive fact. Two minutes.

That answered “how big” and “who controls it.” The other half: how fast. Add it and you hit a result almost everyone guesses wrong.

Most people assume a cache is an order of magnitude faster. B200's L2 isn't. HBM is 8 TB/s. L2 is only 21. 2.6×.

Pause. Let it land. 2.6×. ⛔ Add this at once, or “only 2.6×” gets heard backwards. The room will conclude “on-chip memory is no big deal.” It's the exact opposite.

2.6× answers a narrow question: how much faster is a byte from L2 than HBM?
Ask another and it flips: bandwidth per byte of capacity? L2 gets 2.6× the bandwidth from 1/1,500 the capacity: 4,000× HBM's bandwidth density.

⭐ Give a picture; it sticks better than a ratio: “Reading all of HBM once takes 24 milliseconds. All of L2, 6 microseconds.” ⭐⭐ Then the line to leave behind: “Those two times differ by exactly the capacity ratio. So on-chip memory's whole advantage is being small. Not fast because it's SRAM; fast because it's small, close and short-wired.”
⛔ Then pull back, lest it eclipse the main thread: “But when you optimize, 2.6× is still the only number you can use. You move bytes, not capacity.”

So L2's value isn't its own speed; it's skipping an HBM round trip. A hit gets you 2.6×. A miss saves nothing and adds a detour. That's why “just land it in L2” isn't a strategy you can rely on.

Why so slow? L2 is shared chip-wide; data crosses the on-chip interconnect to an SM. Shared memory is 4.3×, VMEM about an order higher; both sit next to the compute units.

Both subsections' takeaway: “Both truly fast levels are software-controlled.”

⚠️ For the TPU row we give a magnitude, not a number. VMEM bandwidth is unpublished; only capacity is official. Go no further than: about an order of magnitude above HBM. No precise number, and no integer multiple: HBM bandwidth is public, so a multiple reveals the unpublished number.

Point at the last ⛔ box, “Don't read this as ‘the TPU won three rounds.’” Always read it aloud: it's the lecture's guardrail.

All three leads are real, and each has a price.
① Big, fast scratchpad; the price is no safety net. A GPU guessing wrong just slows a bit; a TPU compiler tiling wrong really stalls the chip.
② The GPU's two 37 MiB blocks look “not spent on data.” but buy something else: irregular shapes, varying lengths, unpredictable branches still run decently.

So the real conclusion isn't “whose on-chip memory is better,” but: Both handle the same thing, uncertainty, just in different places. The GPU spends silicon to absorb it. The TPU eliminates it at compile time.

Script · 3.2d “Latency”

Scroll to the latency ladder. 3–4 minutes; the first half lands here. Don't rush.

Capacity, done. Bandwidth, done. One axis left: how long you wait. And it's the only one of the three the GPU itself can't answer.

⭐ Ask the room first, no numbers yet: “A warp needs data. Not in L1, so check L2. Not there either: fetch from HBM. How long does that chain take?”

Look at the figure. L1 hit: 30 to 40 cycles. L2 hit: 273. All the way to HBM: 659. Same load instruction. Identical in the source.

Pause. 30 cycles to 659. 20×.

⛔ Head off a sum nearly everyone gets wrong: “One L1 lookup plus one L2 lookup plus one HBM fetch.” You can't add them. 659 is one full miss, end to end. The two misses before it are already included.

⭐ Then the real question: “Every case takes a different time. So how do you plan ahead?”
The answer is blunt: you can't, so the GPU doesn't plan at all.

None of its three remedies tries to predict.
One, oversubscription. Dozens of warps live on an SM; one stalls, switch to the next. Latency isn't eliminated; others' work covers it.
Two, the scoreboard. Hardware watches operands at runtime, issuing only once they arrive.
Three, the cache itself is a guessing machine, betting you'll reuse the data.

The TPU goes the other way: delete the uncertainty at the source. No hardware cache, no cache misses, no variable latency. Data is in VMEM not by luck, but because the compiler's own DMA put it there. And the compiler knows which cycle to wait on.

⭐⭐ First half lands here. Slowly: “The GPU hides uncertain latency behind other work; the TPU deletes the uncertainty. One answer is statistical, the other deterministic.”
⭐ Follow with: “So lacking L2 isn't a missing feature: a machine scheduled to the cycle at compile time needn't guess.”

⭐ Give the room something practical: this is also why GPU and TPU “tuning” share nothing. On a GPU you tune for “enough other work”: occupancy, warp count, coalescing. On a TPU you tune for “room for the compiler to schedule”: alignment, tiling, double buffering. Same word, tuning. Not one knob in common.

⚠️ Don't let it become “the two keep drifting apart”: “Peak-chasing CUDA kernels today use async copies, explicit barriers, their own double buffering: a hand-written static schedule on the GPU. To really reach peak, guessing isn't enough.”
⭐ A hook for the 3.5 boxes; pay it off there.

⚠️ Cite the source: measured on H100 and consumer Blackwell; B200 has no comparable public data. Borrow magnitude and shape; they aren't B200 data.

Script · 3.3 Parallel hierarchy

First stop on 3.3's opening figure (fig2-2, “Lane and sublane are the shape of a register”). One minute: vocabulary, but 3.3 and 3.4 lean on it.

First, two terms. Next we ask what a thread really is, and 3.4 has that “8 × 128.” Both need the TPU's coordinate words, and both words mislead at face value.

Two steps build the picture.
Step one, one row. A sublane is one horizontal row, 128 cells long, 512 bytes. That's VMEM's architectural word width, not an arbitrary number.
Step two, stack 8 rows: one vector register, 8 × 128 = 1,024 cells.

Now each word has its place. Lane measures position along the row: cell 0 to 127. Sublane measures which row: 0 to 7.
So “align to 8 × 128” just means: 8 rows down, 128 cells across.

⚠️ Here's how the name misleads: a sublane isn't a piece of a lane. It's a full row across all 128 lanes. “Sub” means second dimension, not smaller.

Lighten up here: “Almost everyone gets this backwards the first time. I did too.” Admitting it beats warning “don't get it backwards.”

One line to remember: innermost dimension is lane, next in is sublane.

Then the figure's real point: XLA deliberately lays out memory in the register's shape. Load it and compute right away, with no relayout.

⛔ I used to get this wrong. Don't read the old version: “A row is no longer contiguous in memory.” That isn't the cost; it's the point. A run of 128 elements fills exactly 128 lanes: the goal.

The real cost: the shape is locked in. Three: Changing the innermost dim moves data (tiling binds the last two dims; a transpose or some reshapes are copies, not stride changes). Uneven sizes get padded: zeros take VMEM, pass the MXU, count as FLOPs. Outside handoffs need relayout. §3.4 (transposes) and §3.6 (padding) pay for it.

⭐ Add a contrast; it ties back to the main thread: “GPU coalescing wants contiguity too, but as a soft constraint: misaligned still runs, just slower. Tiling here is a hard constraint: wrong shape means pad or relayout. One side fines you; the other won't let you through.”
No figures yet. Talk for two minutes; show the first figure only at “six levels.”

What just went missing were components on the silicon. One level up, the programming model loses things too. Same cause, second appearance.

I'll boil this section down to one question.

Pause. Slowly: “When you write GPU code, you don't know which SM your work lands on, or when. The hardware decides at runtime.” Whiteboard only: “decided at runtime.”

Since you don't know, you can't name anything. You can't write “SM 3, do this.”

What's left: describe relationships: what must stay together, what can go anywhere.

A batch that “must stay together” is a block.

Why “must”? Because that work passes intermediate results around. And they live in the SM's own small local memory, not outside.
So once a batch lands on an SM, it can't move. Move it, and the intermediates are lost.

Conversely, work that never talks can land anywhere. All of it together is the grid.

These two levels are yours. Writing the code really says two things: these go together, and this many in total.

The hardware groups too, for a different reason: saving circuitry.

Fetching an instruction is expensive, so 32 data elements share one instruction and compute together. That bundle of 32 is a warp. You can't change it; it's fixed in silicon.

Four levels: the bottom two fixed by hardware, the top two set by you.

The other two came later, each forced by a specific problem.
warpgroup: new matrix instructions need four warps issuing together to make the shape.
cluster: neighboring SMs want to reach each other's local memory directly, no detour.

⭐ One more punch; it works: “The detour? Through that L2, only 2.6× faster than HBM.” 3.2c's odd number is now a whole level of the programming model. The room notices when an earlier number becomes structure. Now scroll to the first figure (“Why six levels?”). Point at “⚡ Why this level exists.” Skip “Size.” The takeaway. Pause two seconds after: “All six levels grow out of that one ‘you don't know.’” ⛔ The caption already ends “Together: the organizational cost of runtime adaptation.” Don't read it; that says it twice. Scroll to ms-t3 (Figure T-3). Point at the empty cells, not the text.

The TPU, by contrast, fixed everything at compile time. There's no “you don't know.” So it needs not one level of this language.

⚠️ Head off a trap: the two sides' levels don't answer the same question. The GPU's describe “who must be together.” The TPU never needs to say that. So this table is ordered by question, not by name.

Blank one: no “group of threads.” GPU: thread blocks; TPU: no middle layer.
Blank two: no “swap in another.” GPU: 4 warp schedulers, 64 warp slots; TPU: none.

Those two blank rows are 3.2's five missing pieces, projected onto the programming model. Parts missing from the silicon mean concepts missing from the code.

⭐ Of the three “mismatches,” cover only the first; save two for questions: “One SM hosts several blocks at once, side by side, blind to each other's local memory. Section 5's ‘occupancy’ measures how many more batches the SM can fit.” Scroll to “Inside one SM” (4 quadrants).

One level deeper. So far, “an SM” has been a black box. Inside are four quadrants, each with its own scheduler and registers. 32 is one quadrant's width. 128 is one warp from each.

One question deserves its own moment: what does a thread own outright?

No compute units. CUDA cores and Tensor Cores aren't assigned to anyone. They're pipeline slots. Whichever warp issues this cycle uses them this cycle; next cycle, someone else.

All it owns outright is registers: claimed on landing, returned when done, never moved.

⭐ Two things follow. Say them back to back: “Because registers never move, swapping in another warp is free. Because they're owned outright, registers run out first, not compute units.”

⛔ 32 bits, 4 bytes, a cap of 255, rounding to 8: all in the fold-out. Don't read any of it live.

Scroll to the last figure (“Three ‘threads’”).

Finally, back to the bottom row. A warp is 32 threads, and each has an index in the warp, 0 to 31. That position is a lane.

So the GPU has lanes too: in hardware, one SIMD channel. The same thing as a TPU lane.

⭐ The section's last line. Say it, move on: “One difference: GPU lanes have names, TPU lanes don't. You can write an if for one GPU lane; on the TPU you can't write ‘lane 37, do something else.’ So a CUDA thread is a SIMD lane with a name.”

Script · 3.3a What one lane does in a real matmul

Stay on the 3.3 figure “a thread is really a lane.” Don't scroll. ⛔ First, knock out a picture unprompted. Nine in ten people in the room hold it: “One thread is responsible for one cell of the output matrix.”
That was right before Tensor Cores. Today it's wrong.

Start with the era when it was right. A thread was the owner of one output cell. It ran its own K loop, loaded slices of A and B from shared memory into its registers, multiply-accumulated, kept the sum, and wrote it back. It did both the movement and the compute.

⭐ Then overturn it. Slowly:
“With Tensor Cores, the matrix instruction is one instruction issued by the whole warp. Its operands are spread across 32 threads' registers, a small handful each. No single thread can compute even one complete output element.”
If asked for specifics: in a typical instruction, each thread contributes 8 values of A, 4 of B and 4 of the accumulator. That's all.
⭐⭐ The takeaway: “So on a Tensor Core, ‘which cell does this thread own?’ stops meaning anything. A thread shrinks to two roles: a holder of registers, and one thirty-second of an instruction issue.”
That's the full sense of “a thread is really a lane.”
Point at the box below, “Is it SIMD?”

The answer has two levels, and they disagree. In hardware, yes, it's SIMD: a warp's 32 lanes share fetch, decode and issue, running one instruction in the same cycle. In the programming model, no: you write scalar code; since Volta each thread has its own program counter and can branch alone.

⭐ Close with one line: “SIMT is scalar code on SIMD hardware. The cost is in branches: when they truly diverge, the paths take turns.”
Back to the main thread: “All this is the price of relying on runtime adaptation. The TPU doesn't make that decision, so it lacks even the word ‘thread.’”
⚠️ Data movement only if asked (two minutes): Does a thread move data? Yes, but that half is retiring. Before Ampere, threads moved data themselves, tied up doing it. Ampere's cp.async bypasses registers. Hopper's TMA: one thread issues one descriptor, a whole tile moves, the other 31 sit out.

Script · 3.3b The apparatus (⚠️ skippable)

Scroll to 3.3b. Point at the one-line definition on the blue main-track line (“a scoreboard is a table…”). You must read it. Fifteen seconds.
The fold below is the mechanism (four minutes, the lecture's likeliest overrun). If nobody asks, leave it shut; go to 3.4.
Say just this: “A scoreboard is a table of whether each instruction's operands are in. Each cycle the scheduler scans it and issues whatever's ready.”
Then straight back: “On the TPU, this table doesn't exist.”

Twice now I've said “the GPU exposes lanes to you, and the price is a whole apparatus.” This subsection answers one thing: what's in it? Three things, each for one narrow problem.

Name all three in one breath, then each. Knowing the count and where you are keeps people with you:
“One: who can issue now. Two: who syncs with whom. Three: how data gets into place without tying up threads.”
Last: point at 3.3b's data-path figure (four GPU rows + one TPU row), first three rows only. It's the hook. Don't skip it.

Threads are retiring as data movers. Three generations, one layer peeled off each.

Before Ampere, threads moved data themselves: global memory to registers, then shared memory. While moving, it can do nothing else.
From Ampere, a new instruction sends data past registers straight to shared memory. No middleman.
Hopper goes further: one thread issues one descriptor to move a whole tile. The other thirty-one sit out.

⭐ Fold the three into one line, slowly: “In three generations, the thread went from ‘mover plus computer’ to a register holder.” The other half of “no thread computes a full output element”: one half stripped computing, the other moving. ⛔ Don't add “so the GPU is moving toward the TPU.” That trend is §5.4b's, with the full evidence and a key qualifier (what moved is “how data moves,” not “who decides”). Here, just the hook.

With the thread hollowed out, what sets “how big a bite you take” isn't the thread. It's the instruction. There, the two sides differ by 16×. 3.4 measures it in front of you.

But don't jump to 3.4. This subsection first has to open the bill.

All that: you write scalar code, hardware assembles it into SIMD, queues you when you diverge, and swaps you among dozens of batches of work. So what does “runtime adaptation” cost? Next: three concrete circuits, one by one.

First: the scoreboard. A hook that sticks: the name comes from the CDC 6600, in 1964. The first machine whose instructions could complete out of order. One of its parts was officially called the Scoreboard. (Why that word, and why today's is a different thing, is folded on the page. Don't open it live.)

Back to the GPU. Each cycle the scheduler issues from a “ready” warp. Who decides “ready”? NVIDIA splits instructions by one test: does the compiler know how long it takes?
If yes, no hardware: the compiler writes “stall N cycles” into the instruction.
If not (memory, special functions, matrix ops), that's the scoreboard: six counters per warp, +1 on issue, −1 on writeback. The waiting instruction carries a mask of which must hit zero.

“So it isn't a big table. It's six tiny counters. And it answers one question: can this warp issue right now?”

Second, one line: a few hardware barriers in the SM. Warps arrive one by one; once all are in, all go. Mechanism folded below, for reference.

⛔ You announced three; all three need a landing. “We'll skip the second” drops one in public. This one line is enough: “A warp waiting at a barrier keeps its slot. So in a sync-heavy kernel, everyone waiting doesn't make occupancy look better.”

The third one gets a figure.

Data-path figure: rows 1–3, “Staging” column only; not row by row.

The top three rows share both ends (global memory left; shared memory plus compute right). Only the middle column changes.
At first, data went via registers; the thread computed every address and ran the loop.
From Ampere, cp.async cuts out the register hop; addresses are still the thread's job.
With Hopper's TMA, one thread fills a descriptor, sends it, moves on. Address generation is all hardware.

“Now it's the same kind of thing as the TPU's compiler-issued DMA. One difference is left: does some thread issue this DMA at runtime, or is it a step fixed at compile time?”

Scoreboard: “who can issue”; barrier: “who syncs with whom”; TMA “how data arrives without tying up threads.” Together, the itemized bill for runtime decisions. The TPU needs none: not crude, it answers all three at compile time.

Script · 3.3c ⭐ A matmul's real order

Scroll to the big “three ways” figure. Stop on the top band (naive) only. Five or six minutes: the foundation of §3's second half. Don't rush.

We've taken both sides' parts apart. But I skipped something more basic: in what order does hardware actually compute a matmul?

⭐ No figure yet; the room says it first. Ask “How is a matmul computed?” All say: “Left row one, right column one, multiply pairwise, sum: the top-left number.”
Get them to say it aloud. It matters.

Right. Mathematically, exactly correct. And real hardware, GPU or TPU, doesn't do it that way. Not one.

Let that hang for a beat.

First, count the naive way's cost. Look at the first band.
Step one: A's row 0 times B's column 0 gives c₀₀. Write it out.
Step two: same row 0 of A, B's column 1. Notice: you just read those three A values again.
After four steps: A's six values each read twice, B's six too. 24 reads for just 12 multiply-accumulates.

⭐ An easy line: “Two reads for every piece of work.” It needs only one accumulator. If asked “Isn't that economical?”: it saves in the wrong place. Scroll to the second band (GPU).

Hardware does something else: k goes to the outermost loop.
Step one, k = 0: read just two things, A's column 0 and B's row 0: four values.
They pair up for four multiply-accumulates, each into its own cell. All four move at once.

Steps two and three, same. After three, all four finish and drain together. 12 reads, 12 multiply-accumulates. Exactly twice as good.

⭐⭐ The line to remember. Slowly: “It's not one cell reused over and over. It's every cell filled in, slowly, at the same time.”
And settle the obvious one: every cell is updated exactly K times, top-left and bottom-right alike.

⭐ Why not keep one row of accumulators? See the equation.
The accumulator is an M×N block. Each K step reads M + N values, buys M×N multiply-accumulates. Reuse is M×N over M + N.

⭐ The harsh reading: the short side caps it. “Keep one full row: the short side is 1. Make N a million; reuse stays 1. The long side doesn't help.”
A 128 by 256 tile? Eighty-five.

The real conclusion: an accumulator array isn't for holding results but buying reuse. Each extra accumulator pairs already-moved operands once more. Section 1's arithmetic-intensity thread, first cast in silicon.

Scroll to the third band (TPU).

The TPU flips it. B loads into the array, then stays put. A streams in from the left, row by row; partial sums add up on the way down. Out the bottom: a finished row of C.

⭐ The key line: “The sum along K happens in space. Three PEs each multiply-accumulate once and pass the partial sum down. No external accumulator.” Bottom comparison table: the landing.

⭐ Three bands: same math, differing only in which dimensions are space and which time.
The GPU puts M and N in space, K in time. The TPU puts K and N in space, M in time.

⭐⭐ Must say; it unlocks the next three sections: “This one choice drives everything after it: how big the accumulator is, why the tile is the main knob, and why the TPU must keep its weights still.”

⚠️ Two caveats, in passing; don't expand: ① The GPU band is algorithm-level. At instruction level, one MMA already reduces a short run of K internally. Same shape, just batched. ② The TPU uses an external accumulator only across K tiles. Real K is often thousands; the array takes 256 at a time.

Script · 3.4 One instruction's bite

No figure yet. Ask the question first, then scroll to Figure G-4.

Last section: who issues the instruction. Now a different question. Once an instruction goes out, how big a block can it take in one bite?

Sounds internal; it decides something external: which head_dim costs nothing.

Scroll to Figure G-4, then T-4. In both, look at one spot: how deep the summed dimension is.

GPU: over five instruction generations, more threads join, bites grow. But the summed dimension never changed: always sixteen.

On the TPU, the MXU is a 256 by 256 systolic array. That dimension is two fifty-six.

Dead stop: “Sixteen, against two fifty-six. A sixteen-fold gap.” The one number to remember.

Its name: the contraction dimension. The one a matmul sums away: how many terms one instruction accumulates at once. The name doesn't matter; the numbers do.

What does 16× mean? Plug in a real shape: attention's head_dim is usually 128.

GPU: 128 over 16 is 8. Eight instructions, zero waste.
TPU: 128 against 256 fills only half. The other half sits empty.

So why not fill that empty half with other work?

You can't. The contraction dimension is an accumulation chain. Anything pushed in joins the same partial sum. Nothing else fits.

Flip it: why doesn't the GPU pay?

Not the scheduler. That swaps warps, filling time bubbles; it can't fill a contraction dimension's empty half. It's that its K is only 16, and nearly every shape is a multiple.

⛔ Read the line above as is. An old version said “the GPU has a scheduler, so it's shape-insensitive.” Wrong: any CUDA person refutes it in a sentence.

⚠️ Neither side is badly designed. The GPU can make K this narrow and leave alignment to runtime, at the price of the five parts from 3.2 the TPU lacks. “Whose granularity to align to?” just has two answers.

⭐ Scroll to the big four-band figure at the end of 3.4 (caption: “From ‘what stays put’ to ‘how many reloads’”: stationarity / tiling / ping-pong / reloads).
⛔ Don't weight the bands equally. Priority:
① Band three (ping-pong) first: what nobody else explains clearly.
“Loading a weight tile: 256 cycles. Each cell has two weight-register sets, one computing, one loading, so the 256 hide in the 2048 of compute. Only the first tile waits.”
⭐ Then the test at once: “It hides if a segment has more than 256 rows.”

② Back to band one (stationarity), one line: left-, right-, accumulator-stationary differ only in what stays put. The TPU is right-stationary. What stays put decides which two dimensions are space, which is time.

③ Bands two and four, for questions: where the 168 tiles come from, why M can't stream in one go (its accumulator would need 128 MiB, over VMEM), why finer segments mean more right-matrix reloads. Unasked, skip.

⭐ The figure's closing line; must say: “Longer segments: fewer right-matrix reloads, but a bigger accumulator. Turn this knob all the way and you still hit the on-chip stovetop, the GPU's wall in 3.6.” ⚠️ Someone will surely do the math aloud: “One instruction, only sixteen deep, can't add up to 2,250 TFLOPS.”
Head it off: instruction shape is how much work you hand over at once; MACs per cycle is separate. To reconcile, quote the page: 4,096 × 2 × 148 SM × 1.83 GHz = 2,219, official 2,250, a 1.4% gap. ⛔ Don't follow their wrong path.
Point at the closing peak chain. For the math-checkers: go fast, conclusion clear.

While we're here, multiply out the TPU's peak to show we're not making it up: one MXU, 262,144 floating-point operations per cycle, times 2.2 GHz, times four MXUs. That's 2,307 TFLOP/s. Official: 2,307. Exact.

⚠️ A cross-generation warning to take home: half that TPU number comes from “each cell does two multiply-accumulates per cycle.” Only the 256 by 256 generations; the older 128 by 128 ones do one. Carry the constant across and you're off by 2×.

Close with this. Must say: “How much you can fuse is capped by two numbers together. One is workbench size: 3.2b's 289×. The other is the granularity here. Tiling in 3.6, both show up at once.”

Script · 3.5 What each side added alone

No figure yet. The question is one sentence: ask it, then scroll. Ninety seconds; don't expand.

So far, the first half has counted what “one side has and the other doesn't.” Two pieces are left: each side grew one of its own.

But the question here isn't “what are they?” Slow down and ask: “The product page says it's supported. Can you actually use it?”
Write just this on the whiteboard: “Can you use it?”

Just two examples, one per side.

Scroll to the block-scaling text. No paired figure; say them: one example, one threshold each.

First, GPU side: block-scaled quantization. Marketing: “B200 supports FP4.”

That's true. The catch: block-scaled instructions have two paths, on different dies, and B200 has only one. So if you assume “it has both,” you're wrong.

Second, TPU side: SparseCore, a core dedicated to embedding-table lookups. Marketing: “the TPU has a core just for embedding lookups.”

Also true. The catch: it only pays off when the same rows get looked up over and over. The number that measures it is called the duplication factor.

Stop. For this room: “For language models, the duplication factor is very low, about 1.01: not zero, but nowhere near the payoff line. The compiler never sends work there; in production it has never looked up a single table.” ⛔ Follow up at once; don't stop there: “But ‘can't use it’ isn't ‘not using it.’ You use this core every day.”

Its real job in large-model training: offloading collectives, taking All-Gather and Reduce-Scatter off the TensorCore so it keeps computing while communication runs alongside. One way the TPU overlaps compute and communication.

⭐ If asked “How does it handle irregular access?”: the page has a SparseCore dissection figure; don't scroll to it (three minutes).
One line: “The TensorCore moves whole (8,128) blocks. SparseCore moves just 8 numbers, from freshly computed addresses: sixteen small units, each chasing its own addresses, hiding latency with many fetches in flight.”
Details: the figure, after class.
⛔ If pressed on granularity, get these two right:
“(8,128) is the vector register's shape, not the minimum DMA size.”
“SparseCore's DMA granularity on v7 is 32 bytes. Go by get_tpu_info in the JAX source; the docs page still shows an old 64. Don't read that.”
Trick question: “VMEM or HBM?” Answer: “Both, but its VMEM is its own, not the TensorCore's.”
Sources, both public: the Google Cloud post “Training large models on Ironwood TPUs” lists it as its own tuning technique; the public MaxText repo's XLA flag library has SparseCore All Gather. ⛔ No internal measurements. ⭐ Only now is the lesson complete. Slowly: “Marketing doesn't just keep you from using what it advertises. It also hides what you already use. Because it's called a ‘lookup core,’ you search the profile for lookups, find none, and decide it's idle.”

One more thing shows: the two aren't even at the same level. Block-scaled quantization is an instruction variant; SparseCore is a physically separate core. “Each side added one piece” is itself a marketing framing.

Neither line lies: the error is reading “supported” as “fits my workload.”

Close with an action, not a sigh: “When you see ‘supports X,’ find the line where the threshold is written. If you can't, assume it doesn't apply to you.”
Scroll to the green box “First half: five layers, one kind of missing thing.” ⭐ All the first half delivers. Slowly:
“Five layers taken apart, and at every layer the TPU is missing something. But what's missing is never a compute unit. It's caches, scoreboards, the machinery for ‘wait until it runs, then decide.’ Each exists for ‘we don't know what you'll run.’”
Then land it: “The sentence from Section 0 has now been tested five times.”
⚠️ Must say. It's the lecture's only mid-point return to the opening thesis. Skip it, and the room leaves §3 not seeing they watched five pieces of evidence for one claim.
03b

⭐ Rebuilt: one FlashAttention, walked side by side on both chips

⏱ 10 min
What this segment must leave behind Two payoffs, each bigger than the last.
① End of 3.6: four of the six stops are the same on both sides. Section 0's “different everywhere, but one cause” goes from claim to evidence.
② 3.7: 148× more pieces × each 128× smaller nearly cancel out. Only now are Section 1's two same-range peaks explained.
One-line version The first half took the parts apart; this half reassembles them and pays off two old debts.

Script · 3.6 ⭐ Climax: walking it side by side

Stop on the 🍳 box under 3.6. No figure yet. Picture before any number; reverse it and everything after is wasted. Open with the two questions this section answers. Just two: “How much fits in one pan? What holds each side back?”
On the whiteboard: “How much · Which wall”.

The first half was all parts. Now we put them back together. The op we picked in Section 2 now actually runs, once on each chip.

Change tack. No numbers yet. Give them a kitchen. (Added after a session flopped: all jargon, no picture formed. Read as written; don't improvise.)

Think of this op as a dish with three steps: chop, stir-fry, plate.

The clumsy way: chopped, half-done food goes back to cold storage. To fry, fetch it. Fried, back it goes. Plate: fetch again. Three steps, four cold-storage trips.

The catch: the half-done food is huge. Ingredients: a hundred-odd megabytes. The intermediate: over thirty gigabytes, two hundred fifty-six times bigger. Four trips: over a hundred gigabytes on the road, not one cut made.

“FlashAttention in one line: all three steps on the stovetop at once; the intermediate never leaves it.”

The analogy has just three pieces; remember them and you'll never translate again:
Cold storage is HBM. Big but far.
The stovetop is the on-chip scratchpad: GPU shared memory, TPU VMEM. Small but close.
Blade width is how big a bite one instruction takes: 3.4's sixteen versus two fifty-six.

Back to our two questions, now with a picture: how much fits in one pan is how big the stovetop is. What holds you back is the stovetop or the blade.

Scroll to ms-p38. Analogy done. Now real numbers.

First, one we haven't faced head-on: why is tiling unavoidable? Section 2 said “demand is five times supply.” That's a ratio. Here are absolute sizes.

The naive version: write S, read S, write P, read P. Four passes, thirty-two GiB each: a hundred twenty-eight GiB of HBM traffic. And in that whole pile, not one multiply. Compare Q, K, V and O together: just a hundred twenty-eight MiB.

Read its three cost lines, one at a time, no summary: “Compute: unchanged. HBM traffic: one whole term gone. Memory footprint: quadratic down to linear.”

⭐ What you just saw is that word from Sections 1 and 2: operator fusion. This time laid out end to end.

🍳 In the kitchen: fusion is “all three steps on the stovetop at once; the intermediate stays.” Left, red: “a cold-storage trip per step.” Right, green: all on the stovetop.

Slowly: “So that three-line cost profile belongs to every fusion, not just FlashAttention. Not one FLOP saved. All the savings are in the ‘intermediates never land’ row.”

⚠️ Since it doesn't change the math, hardware sets its ceiling, not the algorithm. How much you can fuse depends on what fits on-chip. The two paths ahead: how far the same fusion gets on each chip.

Scroll to ms-g6, then ms-t6. One minute each. Not stop by stop: they only give the third figure its footing.

Each path alone first, then side by side. That order is deliberate. A table first makes them look different everywhere; walking each shows most stops match.

The GPU path: five stops, the middle two caches. Hit or miss is known only at runtime. The physical source of the four red boxes on 2.1's main path.

On the TPU, the middle stop is a scratchpad, not a cache. Ask each stop: who decided this step? None, end to end, answers “hardware improvised.”

Scroll to ms-p39. ⭐ The section's main figure, and one of the course's. Give it three minutes. Read across.

Now merge the paths. Each row is one stop: left, right.

Ask “who decided this step?” row by row; the result is clean: of the six rows, only two are structural.

Stop. Name them: “Row ②: the GPU has a hardware cache tier; the TPU doesn't. Row ⑥: who schedules hand-offs.”

Row ② got its full treatment in 3.2–3.2c. Row ⑥ we haven't unpacked.

This dish hops between two burners: the matrix burner and the vector burner. The GPU picks on the fly: whoever's free goes next. The TPU sets the roster in advance: who goes when, to the minute. That's all of Section 5. Here we just plant it.

⭐ Apply Section 1's test and count: of six stops, only these two “exist on one side and not the other.” The other four do the same job. They differ only in parameters like capacity and naming (largest: stop ③'s 289×), which change next generation.

Point at the green area mid-figure.

Recall 3.4's closing line: how much you can fuse is capped by two numbers together.

The first, just drawn as an area: workbench size. A GPU thread block gets at most two hundred twenty-seven KiB; three 128×128 bf16 tiles take ninety-six. With double buffering, you're about at the limit.
The TPU's sixty-four MiB is two hundred eighty-nine times that. There, capacity isn't what holds you back.

Second: 3.4's granularity. Contraction dimension: GPU sixteen, TPU two fifty-six. Same head_dim, 128: the GPU divides it evenly; the TPU fills half.

🍳 Both walls in kitchen terms. Say all of it: “The GPU's stovetop is too small. It cooks a bit at a time, so ingredients keep going back up: that's K and V re-read. The TPU's stovetop is too big to fill. Its problem is the knife: the blade's two fifty-six wide, your ingredients a hundred twenty-eight, so half the blade cuts air.”

Same job; each side hits a different wall. That's why 3.4 said to read the two numbers together. Either one alone gives the wrong conclusion.

Scroll to ms-p40, the last figure. Ninety seconds.

The last figure settles up. Ignore the stops; count only what you write. That's what matters to coders.

Same math, FLOPs, results. One row differs: who arranges data movement.
GPU: the kernel author hand-writes six: tile size, async copies, double buffering, warp roles, alignment, L1 use.
TPU: just two declarations.

“The shorter right column is the point.”

⭐ Walked through, Section 0's claim is paid off: four of six rows match on both sides. Not a friendly pick: we laid out the whole path and counted.

And the two rows that differ are really the same spot. Automatic caching or not, who schedules the hand-off: both ask “does this decision wait until runtime?”

⚠️ Must say; the easiest line to hear backwards: “Don't read ‘shorter right column’ as ‘TPU is less work.’ Get one character wrong in the two declarations; the compiler won't catch it. Left: ‘you decide a lot.’ Right: ‘one shot to get it right.’”

Script · 3.7 Adding up the totals

Last: ms-g8. Two minutes; right-hand tally only.

🍳 The one-picture version first: both chips have almost the same total blade, cut completely differently. One into lots of small knives, the other into a few big ones.

⚠️ Watch the tally on the right. One knife, two sizes; don't merge them:
Blade width (bite depth) is 3.4's contraction dimension, sixteen versus two fifty-six.
Cut per stroke (MACs per cycle) is 1,024 versus 131,072, 128 times apart.

⭐ Remember the 148 ÷ 128 on the right. Slowly: “148 times more pieces, each 128 times smaller; the two nearly cancel. The real reason the peaks are so close.”

Not roughly equal: exactly equal: 148 over 128 is 1.156, and 606,208 over 524,288 is also 1.156.

The remaining 1.16× the TPU makes up with clock: two point two GHz against one point eight three, about twenty percent higher. So 2,307 edges past 2,250.

⚠️ But that chain is the HGX B200's. Section 1 opened with 2,307 versus 2,500: NVL72's GB200. Its clock is about two point oh six GHz, so the TPU's lead is under ten percent, too little to beat that 1.16×. Hence Section 1's “the GPU is eight percent higher.”

Takeaway, then move on: “Every ratio here is about piece size, not compute amount. Both sides' silicon is in the same range. One cuts it into 592 small pieces, the other 4 big ones.”

Likely questions

⭐ You said the GPU has 73% more on-chip SRAM, then that the TPU has 3.5× more. Which is it?

Both, and it's a good sign: they're reading the table closely. Standard answer: ask back first, “What's the number for?”

“How much SRAM in total?” Use the total: GPU 231 versus 134.
“Where can I keep intermediates without spilling to HBM?” Only the third counts: one compute unit's contiguous workbench, 64 MiB versus 227 KiB, 289×.

⚠️ One more trap; flag it first. That “3.5×” counts only the “software-managed scratchpad” row. Read the whole “who decides” column: TMEM and accumulators are software-managed too. Count them all: 134 versus 74, only 1.8×.
So with such ratios, first say which rows you counted. That itself is part of this section's lesson.

L2 is 126 MB. Why isn't it an “on-chip workbench”?

A workbench is “space I can arrange,” and L2 is the one cell in this table you can't.

No instruction says “put these weights in L2 and keep them there.” You can only issue loads and hope it's still there. Two of Section 2's four runtime decision points (L2 hit, eviction) are exactly this.

⭐ Tie it back: “Counting L2 as ‘on-chip space I have’ is the very mistake this course has guarded against since Section 2.”

How fast is TPU VMEM? Why no number?

Unpublished: the official spec page gives capacity, not bandwidth. Say so; don't invent one.

You can say: about an order of magnitude above HBM; the GPU's shared memory is 4.3×. So one firm conclusion: each against its own HBM, the TPU's tier leads by more. Exact value irrelevant.

⛔ Don't estimate live or call it a round multiple. HBM bandwidth is public; state a multiple and you've disclosed the unpublished number.

The contraction dimension is only 16. Isn't the GPU worse off, looping so many more times?

Backwards. Narrow is exactly where it wins.

K of sixteen means almost every shape is a multiple of it. Easy to fill, almost no waste. On the TPU, K is 256, so head_dim 128 is halved at once.

More iterations isn't a loss: each is full. The real cost: a general scheduler running hundreds or thousands of fine tiles needs warp schedulers, a register file, a scoreboard: the five things the TPU drops in 3.2.

⭐ Land it: “Fine granularity buys ‘runs anything.’ Coarse granularity buys ‘almost no overhead.’”

So should head_dim become 256 to fill the TPU?

Fair thought, but don't present it as advice in class. head_dim is model architecture. Changing it means retraining. It's not a tuning knob.

The honest version: “Designing a new model for TPUs? Check this number at design time.” Telling someone with a trained model “wrong head_dim” helps no one.

⚠️ Say what you're unsure of: how 128 → 256 affects model quality, this course has no data on. No off-the-cuff conclusions.

S (3.6) isn't stored forward. And backward?

Recompute. When backward needs it, redo it on the spot from Q and K.

Cost: two matmuls forward, four backward, one more to recompute: seven versus six, 17% extra work. It buys three orders of magnitude less data movement.

⛔ Mention it and stop; no accounting. MFU and HFU get proper treatment in Section 6, where it belongs.

Doesn't the GPU have TMEM too? Isn't that like the TPU's accumulators?

Yes: Blackwell stepping toward the TPU. Credit the asker.

TMEM means matrix operands no longer land in the register file first. That whole leg is bypassed. Same kind of idea as the TPU's accumulators.

⭐ It's evidence for the main thread: “On the dense matrix path, hardware guessing pays off less and less; better to hand control back to the compiler and kernel author.” For that performance, the GPU becomes more TPU-like. Section 5.4 takes this on.

If asked “NVFP4 vs. MXFP4?” or “What did the B200 actually add in hardware?”

Thirty-second version, three points:

① The number format is identical: four bits, eight representable magnitudes. The difference is all in the scale. MXFP4 shares one per thirty-two values, a power of two only; NVFP4 shares one FP8 scale per sixteen: finer, more accurate.

② Hardware added three things, no coprocessor: four-bit multiplier inputs, a scale multiply in the MMA datapath, a few conversion instructions on the general cores. Hardware “applies” the scale; software “computes” it.

③ The multiply is compute-free, not free. Scales occupy TMEM (only 256 KiB; one accumulator eats half), need an extra path, and quantizing costs real time; watch only the MMA and you'll miss it.

⛔ Go no deeper on stage. The page's fold has the full pipeline. Send them there after class.

SparseCore sounds useful. Why not more on it?

It has a threshold, and most workloads don't reach it.

It shines when the same rows are looked up many times: recommender embedding tables are home turf. LLMs fall short of it.

⛔ But don't make it “LLMs have no use for this core.” Only table lookup is out of reach. In large-model training it runs daily doing collective offload (All-Gather / Reduce-Scatter). The main text covers that; this answer is only “why not more on lookups.”

⛔ Don't present the threshold formula or put it in any external material. Its source is internal. The page has only a plain-English paraphrase; stick to it.

What not to say

Six likeliest ways this segment drifts ① Don't make the first half a spec tour. Each subsection sets up 3.6; land each on “so 3.6 will use this.” Lose that thread and fourteen figures become fourteen loose facts.
② Don't read the 3.2b table cell by cell. Ten rows × two columns is four minutes aloud; you need only the three measures. The table is for after class; live, point at those three rows.
③ Don't walk CUDA's six nested layers in 3.3. The full containment table is collapsed. Leave it. The subsection's only job: “look at the gaps.”
④ Don't derive the wrong math in 3.4. The “4,096 Tensor Cores” mistake is folded below, for lookup. Live, it costs three minutes, changes no conclusion.
⑤ ⛔ Don't derive the online softmax recurrence in 3.6. The section's most expensive detour: five minutes, unrelated to how the chips differ. The one line you need is 5.3's: “The denominator sums the whole row; you can't compute any value without seeing all of it.”
⑥ Don't recite any “look-it-up” numbers. This segment's folds hold three batches, all for lookup: SB0–SB5 and the six-bit mask, the DSMEM 29 / 33 / 181 / 956 cycles, and the register facts: 32-bit, cap 255, rounded to multiples of 8. Say none of them live.
In 3.3, say just one line: “Only registers are private to a thread; execution units are shared in turns.” In 3.3b, just three things: the dividing line “does the compiler know how long it takes,” the middle column, and the closing takeaway.
⏱ Timing: §3 first half (03a) 42 min (re-measured 2026-09-06); second half (03b), same way, 10 min Method: count the spoken characters in the script (the <p> lines and “Pause” lines) and divide by 205 characters/min (a pace from the delivered §5 and §6: 201 and 213). Board cues aren't counted, nor is time pointing at figures, so it's a lower bound.
⚠️ History: 24 → 31 → 40 → 42 (the first three: all of §3, before the split). Every rise was added content: 3.2b / 3.2c, 3.3b, then on 2026-09-06 3.2d (the latency ladder) and 3.3c (three ways to lay out a matmul).
⛔ Adding a core-track subsection means updating the minute count in lec(). It's the only source of the course's total length, and no check catches it running low. ⭐ Decided: no cuts. The course is scheduled by the measured minutes Reason: no fat. Everything cuttable is something people really ask about (3.3b, 3.2d and 3.3c were all forced in by live questions). Retire the phrase “60-minute version.” If you really get a 60-minute slot: emergency list, fixed order, don't improvise A. Collapse 3.3 and 3.5 entirely (−5): one figure each, both “the Nth appearance of the same cause.” Dropping them doesn't break the chain.
B. In 3.2c keep only “L2 is only 2.6× faster”; skip the bandwidth-density branch (−4).
C. In 3.2d give only the three numbers (30 / 273 / 659) + the comparison line; fold the three remedies into one sentence (−3).
⛔ 3.3c, 3.4 and 3.6 can never be cut. 3.3c is 3.4's foundation; 3.6 is the course's climax. Cut them and the lecture falls back to “a spec comparison of two chips,” what the opening promised not to do.
04

Wiring 64 chips: the two sides wire them differently

⏱ 4 min
What this segment must leave behind Same 64 chips, wired two different ways.
GB300's 64 GPUs all fit in one NVL72 domain (max 72). Via a switch, any pair is one hop, wherever they sit.
TPU v7's 64 chips form a 4×4×4 3D torus. No switches, 6 hops at worst; cost depends on distance.
⚠️ Where this segment sits No longer a detour: it sets up Section 6's measurements. Skip it and 854 vs. 662.3 has nothing to stand on.

Script · Two ways to wire 64 chips

Scroll to the big Section 4 figure, “Wiring 64 chips: same 64, different structure.” Glance at each side; don't count cells. ⭐ Say where this sits first, or the room hears a spec dump: “Next section I'll put up both sides' numbers, 64 vs. 64. First you need to know: those 64 chips aren't wired the same way at all.”

On the left, GB300. It's a switched domain that tops out at 72 chips. We run 64, so it all fits. At this scale, any two GPUs are one switch hop apart. Every pair costs the same to talk.

On the right, TPU v7. No switches. Each chip only links to its neighbors, and the 64 form a 4×4×4 3D torus. Four per axis; with wraparound, the farthest is two steps. Three axes: six hops, worst case.

⭐⭐ The key line. Slow down: “On one side, location doesn't matter. On the other, location sets the cost.” Scroll to Figure T-8 (the “Chip to pod” one) and point at the “6 ports per chip” block.

First, the per-chip totals: ICI is six links at 200, 1,200 in all; NVLink is 1,800 per GPU. Both bidirectional. So within 72 chips, NVLink actually has more bandwidth per chip. This figure compares reach, not which single chip is faster.

⚠️ Say “200 per link” is the course's reading, not official wording: the official text says “200 GB/s bidirectional per axis,” but 3 axes × 200 is 600, not the 1,200 in that page's table; only “200 per link” is self-consistent.The course flags this next to the figure. If asked, point there; don't pick a reading for the vendor. ⛔ Brake here. Two things:
One: 9,216 ÷ 72 = 128. That measures reach, not speed. Reading it as “TPU clusters are 128× faster” is the easiest mistake in this course.
Two: this course never measured collectives on a torus. So this segment covers structure, not speed.
⚠️ Asked “which comms patterns lose?” Stop it in one line. “That's sharding, Topic 5. Here we only draw the hardware.” Opening it costs five minutes, and it doesn't belong here. ⛔ One more guardrail, more important than the link math. Always say it: “A hop isn't the same unit on both sides.” A GPU hop goes through a switch. A TPU hop is a direct wire to a neighbor. So 1 vs. 6 is structure, not a latency ratio.” Scroll to Figure P-6 (our own measurements). One minute.

⚠️ How far a topology reaches and what you actually get are two different things. This is the drop we measured ourselves, not a vendor spec. And it's GPU only: 8 to 32 GPUs in one NVL72 domain.

⭐ The one line from this figure: “No distance in the domain doesn't mean no cost in the domain.” ⛔ And say flatly: we never ran the TPU counterpart. So it can't compare the two sides. It only demystifies “one hop.” ⚠️ A common third question: what's that 900? “900 is Blackwell's unidirectional number; 1,800 is bidirectional. On Hopper, 900 was bidirectional. Two different 900s, often mixed up.” ⭐ Close and hand off: “Remember this. The two numbers I show next ran on two structurally different networks. How much is faster compute, how much is better wiring? This course never measured that split.”

Likely questions

This section has nothing to do with FlashAttention. Why is it in the core track?

It's off the FlashAttention storyline, by a hard test: FlashAttention appears zero times in it.

But it underpins Section 6's measurements: those two numbers are 64 vs. 64, and the two sides wire 64 chips in different structures. Skip it and they hang in the air.

So what does that 128× actually mean?

One thing only: how big you can go without changing protocols. NVLink domain: 72 chips. ICI: 9,216.

Past 72, GPUs can still connect, but over a different interconnect (the network), with different properties and bandwidth. That's what the number really says.

⛔ Don't stretch it to “so TPUs suit large clusters better.” No measurements at that scale here. The only like-for-like run, Section 6, is 64 vs. 64.

Where are full comms primitives and NVL72 internals?

L300 §4: four primitives, domain internals, cross-chip addressing. Here: one number, one warning.

What not to say

Two easy ways off track ① ⛔ Don't compare per-link bandwidth. Same ballpark, and the official ICI per-link wording is ambiguous; it proves nothing, burns time, and shifts the room from “how far” to “who's faster”: the very mistake this section prevents.
② Don't drift into topology. How the torus wraps, how NVL72 is cabled: that's ten minutes, against a four-minute budget.
Pacing Four minutes. Never skip it whole: it underpins Section 6.
Check the clock when Section 3b ends. Running over? Cut to two sentences per the emergency list at the top: a glance at each figure, then “one hop vs. six.” Give the time to Sections 5 and 6; they close the argument.
⚠️ Even when cutting, keep the opening line (“the next section's two numbers ran on two kinds of wiring”); without it, Section 6's ratio can't be read.
05

⭐ Unpacking the word for real: runtime or compile time

⏱ 12 min
What this segment must leave behind The two sides fear opposite things.
The GPU scheme fails when it runs out of stand-ins (register pressure, fewer tasks resident);
the TPU scheme fails when it can't predict (dynamic shapes, addresses known only at runtime).
One fears tight resources, the other uncertainty. One gap to close here “Operator fusion” is two things: fusion that keeps the math, the compiler does; fusion that needs new math is always human. Leave it open and the three-section fusion thread falls apart at the last step.

Script · 5.1 Who owns latency

Scroll to the start of Section 5; stop at Figure G-7.

Every section so far stopped at the same decision, just on a different part. This section looks at the decision itself. Who schedules the work? That decision flows from the hardware all the way to your hand on the keyboard.

Start at the bottom, where it's clearest. A load takes hundreds of cycles. What fills the gap? The answers differ not in speed but in who's responsible.

The GPU's answer: one warp stalls, run the next. In this Gantt chart, dashed boxes are “waiting for data,” while other warps keep issuing.

“This only hides the wait if enough warps are resident. That's all ‘occupancy’ means.”

And registers set how many fit. At 32 registers per thread, all 64 warps fit. At 128, only 16, and it starts to pinch. That's why tuning often starts by cutting registers.

Switch to Figure T-7. Another pair. Point at them back to back.

The TPU differs: the schedule is fixed at compile time, unchangeable at runtime.

A VLIW bundle is everything this cycle does, packed together: scalar slot fires a DMA, vector slot normalizes, matrix slot pushes a multiply, misc slot holds a wait. All in one cycle.

How can it be that precise? Latency is a constant. No cache, so latency is constant. Constant, so every cycle can be filled at compile time.

Say these back to back, two sides of one thing: “With a cache, latency isn't constant, so the compiler can't schedule, so you need a big bench of stand-ins.
No cache, latency is constant, so you fill every slot and need no stand-ins.”

⚠️ But look at the gray cells. Gray is truly empty, and nothing steps in automatically. The compiler either finds other work for it, or eats the loss.

Point at the box ending this subsection, “When each scheme fails.” It's where 5.1 lands. Read it out.

⭐ This pair's real value isn't the mechanism but the failure conditions.
GPU scheme fails when stand-ins run out. Tight registers, fewer warps: latency shows.
TPU scheme fails when it can't predict. Dynamic shapes, runtime-only addresses: the compiled schedule sits idle.

So the two sides fear opposite things: one tight resources, the other uncertainty.

Script · 5.2 The code level

Scroll to Figure P-1. One minute. Don't read code line by line.

This hardware difference shows up verbatim in what you type.

On the left, CUDA has threadIdx. On the right, JAX has no index at all. You declare the split axis; the compiler decides who runs which piece.

One level up, collectives follow the same pattern. On the GPU, all-reduce is a second step you write by hand. On the TPU, you never write it. The compiler inserts it, based on the parallelism you declared.

Script · 5.3 Where the compiler's limits are

Scroll to Figure P-3, the center. Two and a half minutes.

The natural reaction here: “Why not leave it all to the compiler?” You can't, and the line is pretty clear.

The left four are orchestration: operator fusion, overlapping comms with compute, memory reuse, layout and sharding. Those are the compiler's call.
The right is changing the algorithm. Past that line, the compiler can't help.

Point at the FlashAttention cell: “It's on the right, and both sides hand-write it.” One counterexample where both fail defines the boundary better than any success.

This completes §3.6's table: the TPU spares you data movement, not algorithms.

Scroll down to the ⭐ box “Fusion, closed: two different things.” ⛔ It closes the course's fusion thread; never skip it.

You may have just tripped: that figure's first left cell gives operator fusion to the compiler, yet the takeaway says the compiler can't do FlashAttention.

Pause: “Both hold: two kinds of fusion.”

Kind one: fusion that keeps the math. The compiler does it. Same expressions, just no trip to cold storage. You write zero lines. Both do it silently.

Kind two: fusion that only works with new math. Always human. FlashAttention fuses because softmax was first rewritten to merge block by block.

Plain softmax can't fuse: the denominator sums the whole row. No output before seeing the whole row, so no keeping a small block on the stovetop. Online softmax fixes it: update running max and denominator as you go, rescale once at the end; same results.

Slowly: “They didn't fuse then tweak the algorithm. They swapped it so fusion works. The compiler decides how work is scheduled, not what's computed.”

⭐ So “who decides fusion?” has two halves:
Whether and how much: the hardware (§3.6's stovetop and blade width jointly cap it);
who fuses: depends on whether the math changes (no: the compiler; yes: only you).

Hence an apparent paradox: fusion is the one topic running through the whole course, yet it's neither purely hardware nor purely software. It sits right on the boundary, which is why studying hardware always ends up there.

Script · 5.4 Catching the objection

Scroll to Figure P-33. Two minutes. Defend only.

Right about here, the strongest objection always comes up, and half of it is right: “Doesn't CUDA Graph make the GPU compile-time too?”

⭐ First, grant the right half in full. All the persuasion is here: “You record the whole step's kernel graph up front and run it straight through. Real gains: on many workloads, on vs. off is an order of magnitude.”

To judge what it overturns, be exact about why it's fast. The usual story is wrong.

It doesn't save “branches.” It saves the CPU launching kernels one by one. One launch is about 5–10 microseconds on the host; an MoE step has tens of thousands of kernels. Launching alone takes tens of milliseconds. A graph makes it all one replay.

Point at the figure's middle column. That's the answer.

It comes down to which layer. Split it in three: sequencing (which kernel next), placement (which SM gets a block), issue (which instruction next cycle).

CUDA Graph freezes only the first layer. The other two are untouched. Each graph node is still a normal kernel launch. Schedulers, scoreboards, warp slots, resident contexts: all still working.
The TPU freezes all three, the third down to the exact cycle and issue slot.

Three reasons. Stress the third: “Hardware cost is set by the worst case it must support, not the one you run. Even if 100% of workloads used graphs, none of §3.2's five components could go. At tape-out, no chip knows if you'll use graphs.”

So the real point runs the other way: for this performance, the GPU has to become more like a TPU. And it can only move the top layer earlier. The two below can't move. If they could, those transistors would be long gone.

📌 If pressed—“what does CUDA Graph actually compile, who replays it, is the scoreboard still there?”—don't improvise; a collapsed block on the page answers all four. Open it, read it. Memorize one line: CUDA Graph works on the outside of kernels. It doesn't know what's inside, so architecturally it can't reach the lower two layers.

Script · 5.4b Second objection: what about TMA?

Scroll to 5.4b. Two minutes. ⭐ More worth answering than CUDA Graph.

There's a second objection, far stronger, because it points the right way: “On data movement, isn't the GPU moving toward the TPU too?”

⭐ Concede it, gladly. The persuasion rests on conceding first. “Yes, and the trend is unmistakable. Read §3.3b's four-generation figure top to bottom.”

Always: plain loads through registers; threads compute addresses, strides and bounds; math waits for the copy.
From Ampere: no register hop, but threads still compute addresses.
From Hopper: TMA. One thread fills a descriptor, fires, moves on; hardware takes strides, offsets, bounds. 32 lanes each computing an address: deliberately abandoned here.
From Blackwell: the biggest step. One thread issues matrix instructions for the whole CTA. Operands and accumulators move to a dedicated matrix scratchpad; results never return to the register file.

Then name the endpoint: “Four generations on, it ends here: whole blocks, by descriptor, no threads busy, into a software-run dedicated scratchpad.”
“That's exactly the TPU on day one.”

⭐ But the trend only went halfway. The other half is this course's main thread.

How data moves has converged: granularity, who generates addresses, where operands land. Those three really are converging.
Who decides hasn't moved a step: a TMA descriptor is built at runtime, not baked into the binary. L1 and L2 remain; hits are still known only at runtime. The warp scheduler still picks live, every cycle.

⭐⭐ The landing. Slow down: “It's not ‘the GPU is becoming a TPU.’ It's ‘the GPU made feeding TPU-like, while the layer deciding who gives the orders hasn't moved.’ The split was never the endpoint. It's who issues the DMA.”

⚠️ Close with why the trend was inevitable, so nobody calls it copying: the faster the matrix unit, the less feeding can tie up the units doing math. One matrix instruction eats tens of KB in a few cycles. If 32 threads still take turns computing addresses, they gatekeep their own compute.
Same physics, same answer: one took four generations; the other began at the endpoint.

Likely questions

⭐ If the compiler can fuse, why can't it do the most famous fusion of all?

The question this section most deserves; the page has a box for it. Don't improvise: turn there and read.

“Fusion” covers two very different things.
① Keeps the math: same expressions, intermediates just never hit HBM. Compilers do this silently.
② Changes the math: FlashAttention. Plain softmax's denominator sums the whole row, so no output until you've seen the whole row. It can't keep just a small block on-chip. Only after switching to online softmax does fusion work.

One line: the compiler decides how work is scheduled, not what's computed.

Then what are Triton and Pallas? Aren't they compilers?

Both answer “what do you write past the line?” Nearly the same role. Shared trait: you don't write loads and stores.

Note where they sit: right of the line. Needing them proves the compiler can't do it automatically.

⛔ Don't debate which is nicer. Here that's an ecosystem gap, and ecosystems converge; architectures don't. That's point ② of Section 9's three closing caveats.

Does “not a single FLOP is saved” mean online softmax has zero overhead?

No. Head off this common misreading.

In §3.6, “not a single FLOP is saved” counts the two matmuls. Online softmax adds one rescale per block, but that's O(n·d), negligible next to O(n²·d).

“Keeping the math” means equivalent results, not zero extra instructions.

Is slow TPU compilation a big problem?

Yes, but be clear on its nature: the cost moved; it wasn't added.

On the GPU, the scheduler does it live every run, spreading the cost over execution; the TPU pays once, at compile time.

⭐ Mention what it buys: very stable runs. The same GPU code can vary a lot run to run; TPU runs barely do.

⚠️ Any shape change means a recompile. A real pain point. Full porting bill: L300 §5.5. Don't open it here.

If I turn CUDA Graph on everywhere, are the GPU's five components just dead weight?

Good question; the answer is this section's hardest point: dead weight you still carry.

Hardware cost is set by the worst case it must support, not the case you run. At tape-out, the chip doesn't know whether you'll use graphs.

Even with graphs, only the top layer freezes. Which SM gets a block, which instruction issues next: still decided at runtime. Schedulers and warp slots keep working.

What not to say

Four easy ways off track ① Don't walk the two Gantt charts cell by cell. They're schematics, not measured traces (labeled so). One line: one side swaps players, the other works a roster.
② Don't read 5.2's code line by line. The point: one has threadIdx, the other no index. Show both indices, move on.
③ ⛔ Don't turn 5.4 into nitpicking CUDA Graph. Its persuasion rests on granting the right half in full. Lead with faults and the CUDA expert in the room stops believing everything after.
④ Don't get into XLA passes. Figure P-3 is enough. Passes belong in a compilers course.
Pacing Twelve minutes, five figures, weight on 5.3.
5.1 three minutes (two figures; lands on “opposite failure conditions”), 5.2 one, 5.3 three and a half, 5.4 two, 5.4b two; the last half minute is transitions.
⭐ 5.3's “two kinds of fusion” box closes the course's fusion thread. §1 names it, §2 demos it, §3.6 grounds it, it finishes here. Never skip it, however tight. If you must cut, cut 5.4.
⚠️ 5.4's special risk: it draws out the room's best CUDA expert. The answers are under “Likely questions.” Use them, then move on. No debate.
06

After the runs: two numbers, but can you compare them?

⏱ 8 min
What this segment must leave behind ⛔ Not the 1.31×. If all they remember is “GB300 is 1.31× faster,” this segment failed.
Leave this line: the number could move either way, and we've measured neither.
Plus four questions to take out the door Was this number derived? · Binary or decimal? · Per chip or per device? · Physical or usable? They work for any two chips. That is the course's real output.

Script · 6.1 First, settle how to compare

Scroll to Sections 6–7, Figure P-14: the four tiers.

Six stops done, and each side has a number. Before we put them side by side, let's agree on what counts as a fair comparison.

The two usual comparisons are both wrong.
Peaks? That's what the chip can do, not how fast your job runs.
MFU? You pick the denominator. Different denominator, different conclusion.

The only one left standing is tier ③: the same workload, actually run on each side.

⚠️ There's also ④, per dollar and per watt. Procurement's real ask, but no shortcut: ④'s numerator is ③. Without ③ measured, ④ is one unknown over another.

Open the collapsible below (“…we fell for it once”). ⭐ All of 6.1's persuasion is here. Don't skip it; tell it calmly.

“Different denominator, different conclusion” isn't a scare tactic. We fell for it ourselves.

Same run, same numerator, 854.0, and we reported two MFUs:
854 over 2,700: 31.6%. Committed to the repo; stood for twenty-six days.
854 over 2,500: 34.16%. Corrected August 31.

Let it sit. “A 2.6-point gap. No rerun in between. The measurement didn't change by a digit.”

What went wrong? The 2,700 was inferred, not looked up. Its author took GB200's 2,500 times 1.2, because “Ultra must be a bit faster.” But NVIDIA never published a 2,700. GB300's dense BF16 peak is exactly GB200's. Ultra raises FP4 and attention, not BF16.

⭐ The landing point. Slow down: “The lesson isn't ‘be careful.’ It's why this was hard to catch. 2,700 looks exactly like an official spec. Right units, right magnitude, a bit above GB200 feels right. Nothing throws an error.”

The downstream 31.6% hides even better: computed honestly, flawless arithmetic.

So “was this number derived?” isn't about you. It's about the number you cite.

Script · 6.2 ⭐ The numerator side: MFU vs. HFU

Only one figure here (P-13): one box before, two after. Only your pacing gives breathing room; don't run four yellow screens together.

That ticket figure was all about which peak goes in the denominator. But the numerator has two flavors too, both named, both in use.

In one line: is the numerator what the model's math needs, or what the hardware actually ran?

🍳 Picture first. Same knife (set up in §3.6; just extend it from “how wide the blade” to “how many cuts.” No new analogy):

Same denominator: the most cuts this hardware can make per second.
MFU's numerator: the cuts the recipe needs.
HFU's numerator: the cuts you actually made, botched redos and air cuts included.

So HFU is always ≥ MFU. The gap is wasted cuts.

Which to read depends on your question:
“Is the knife idle?” → HFU.
“When is dinner ready, and at what cost?” → MFU. Wasted cuts never speed up dinner.

Scroll to Figure P-13, “MFU is a fraction.” ⭐ 6.2's main figure. Give it time. Start with panel A only.

Panel A's three rows are three numerator choices. The first is a hit we took while porting (the FLOP formula missed a whitelist entry, about 5×), not 6.1's 2,700 incident. The second is our topic, HFU vs. MFU. The third depends on sequence length.

How big is the second? Work it out for this lecture's kernel: FlashAttention's backward recomputes the forward S once, so one of the layer's seven matmuls is wasted. Seven over six: about 1.167×, 85.7% useful.

The figure's 4/3 is the upper bound: full-layer recompute, no intermediates kept, whole forward redone in backward.

Move to panel B, the middle: the figure's center.

Same run, 503 measured, no code changed, four legitimate percentages: 22.4, 20.1, 11.2 and 10.1. Along a row, the precision basis changes. Down a column, the SKU.

⭐ Say this out loud: “These four only change the denominator. Panel A's three numerator choices never moved. So 2.2× is a lower bound, not an upper bound.”

Panel C's four rows are denominator choices, each officially sourced on its own. The hard part was never finding one. It's saying which one.

⭐ The one action: the blue band at the bottom: “So report MFU with this whole line.” Can't write it? Report raw throughput, not a percentage. ⭐ The payoff line: “Both numbers come from the definition, not measurement. They differ by 2×: ‘recompute on’ alone tells you nothing.” Scroll to “Third, easiest to get wrong” (numerator to plot on the roofline).

⚠️ One more must-say. It ties back to Section 1: to ask “will I hit the roof?”, use the HFU-side numerator, the one including recompute and padding.

Why: recomputed ops really occupy matrix-unit cycles. The hardware can't tell what's recompute.

Rule: HFU's numerator to judge your side of the roof; MFU's to report output.

⭐ The page has a worked example. Read it; it grounds the rule: “Say 30% of an op's work is recompute. Useful intensity is just 219; with recompute it's already 312. By MFU's numerator you're far from the roof; by HFU's, touching it.”
⛔ Nail the direction: “The wrong numerator errs one way. It only underestimates.”
The last yellow box: one item, fifteen seconds.

Before comparing chips, ask: “Same recompute policy?” One side off, the other on: HFU can be close while MFU is far apart, or vice versa. That's item ③ in §6.3's closing “what this ratio still lacks” figure. Skip the setup here; it won't land there.

Script · 6.3 Both sides, measured

Scroll to Figure P-15. ⚠️ The trick: skip the two big numbers; point at the two red segments below first.

Ticket in hand. Now the numbers.

⭐ Never reverse this order. Red segments first: “These two are the same length.” Pause. Then the two big numbers.

GB300, 64 GPUs: 854.0. TPU v7, 64 chips: 662.3. Looks like a big gap. But the two red segments are the same length. Measured gap 191.7; peak gap 193.0.

The difference is the denominator, not skill.

Split that 1.29× and only half comes apart cleanly: the peak ratio, 1.084 (2,500 ÷ 2,307), fully accounted for. The other 1.190 is a residual, not “software”: contraction shape and interconnect topology are in it too.

Scroll to Figure P-18. ⭐ This section's real output.

The hardest step: not overstating the number. What does 1.31× still lack? Let the room list it first, then check the figure.

Four red, two green.
Four push up (favor GB300): different global batch, 64 is GPU home turf, different recompute policies, big software-maturity gap.
Two push down (favor v7): 662.3 is clock-locked; per-chip vs. per-GPU is already generous.

⭐ The landing. Read it verbatim: “Red and green point opposite ways. So the number could move either way, and we've measured neither. Responsible work doesn't round the number off here. It lists what wasn't measured.”

So take away the four questions: Was this number derived? · Binary or decimal? · Per chip or per device? · Physical or usable? They work for any two chips.

Likely questions

So which is faster, GB300 or TPU?

You'll get this; it's the section's acid test. Don't just play along.

Say exactly one thing: “On this model, at this scale, in this configuration, GB300 is 1.31× faster per chip.”

Then add at once: “Of six corrections to that number, four push it up, two push it down, and we've measured neither direction. It's a point, not a bound.”

⛔ Don't say “so the GPU is faster.” Don't argue “but the TPU is cheaper”: that's tier ④, and the course has no such numbers.

⭐ Why tell the class about your own 2,700 mistake?

The lecture's most convincing example, and it's ours.

It proves three things: ① the denominator really moves; ② enough to change the conclusion (2.6 points); ③ the failure wasn't arithmetic but treating an inferred number as official.

⛔ Don't tell it as “look how rigorous we are.” Land on the portable rule: a number that “looks like an official spec”? Ask for its source first.

MFU or HFU: which one do frameworks report?

Major frameworks report essentially MFU: the FLOP formula follows the model's math, excluding recompute.

⚠️ Plug that number straight into Section 1's line and you'll underestimate. To judge “will I hit the roof?”, use the one with recompute.

⭐ Practical tip: before comparing two chips, ask “same recompute policy?” One side off, the other on: HFU can be close while MFU is far apart. That's item ③ in P-18.

Why compare “per chip,” not “per card” or “per rack”?

It's already the TPU's most generous basis. Say so up front.

A v7 is two devices per chip. We chose per chip vs. per GPU: “two dies” against “one whole card.” That's one of P-18's two “favors v7” items.

Per device is worst for v7: 662.3 ÷ 2 = 331, and on TFLOP/s the ratio goes from 1.29 to 2.58.

⚠️ “Per chip, per device, or per core?” is the third of the four questions. If asked, that question just paid off live.

1.29 and 1.31: why two numbers?

Different units, and the gap itself is teaching material.

1.29 is per-chip TFLOP/s (854.0 ÷ 662.3); 1.31 is per-chip token/s (6,242 ÷ 4,770).

Change the unit and the ratio moves, by exactly that 1.5%. “Same thing, different reporting, different conclusion,” live.

What not to say

Four easy ways off track ① ⛔ Don't present 1.31 as the conclusion. It's the one fatal mistake here. If the ratio is what sticks, the previous five sections were wasted. It's a point, not a bound.
② Don't explain FLOP formulas in 6.2. How to count numerator and denominator, whether to include the whitelist: another course. Here: “two flavors, which question each answers.”
③ Don't get defensive. Told “the TPU is slow”? Don't hunt for excuses. Point them at P-18's six items; let the table speak.
④ Don't drift into price-performance. That's tier ④; the course has no such numbers. If it comes up, say: “④'s numerator is ③, and we only have ③.”
Pacing Eight minutes, four figures. But 6.2 is one figure among text boxes, where the section most easily slips away.
6.1 three minutes (one for the figure, two for the 2,700 red box; worth it), 6.2 two and a half, 6.3 two and a half.
⚠️ 6.2 runs box, figure, box, box. Text apart from the figure, no natural pause. Make your own pauses: once after the 🍳 knife analogy, again after 7/6 and 4/3. Otherwise you lose the room by the third box.
⭐ The 6.3 figure's order is fixed: red segments first, then the two big numbers. Reverse it and the figure loses its point: see 854 and 662 first, and everyone thinks “200 apart.” Nothing after pulls them back.
09

So what is each chip good at?

⏱ 5 min
What this segment must leave behind One sentence plus three qualifiers. All required.
“The difference isn't ‘which is stronger’ but ‘who gets control.’”
Take only that sentence home and it's just a stance. The three qualifiers are what the course earned. ⚠️ Most likely to be screenshotted alone So conclusion and limits share one frame. That's why each line on the page carries its source tag on the same line. ⛔ Check the course page first This section has one subsection, 9.1, and one figure (P-24). The “eight-row trade-off table” in pre-2026-09-06 drafts is gone. Don't present it.

Script · 9.1 Takeaway

Scroll to Section 9, Figure P-24. The course's last figure. Four minutes.

From package to measurements, we've taken both chips apart. At the start I said “different everywhere, but one cause.” §3.6 laid it all out: in four of six rows, the two sides matched. Now we can name that cause.

Start at the middle question box: the skeleton.

The whole course asks about one decision: things you only learn at runtime, like shapes, where an access lands, who's busy: who handles them?

GPU side: hardware handles them at runtime, which really means people.
TPU side: the compiler works them out at compile time, so you live with its quirks.

⚠️ At the red and green columns, pre-empt misreading: “Red and green just mark sides, not better or worse. And the only head-to-head run was won by red.” Read the three left rows across, one sentence each. Don't summarize.

① How many decisions does one memory access make? GPU main path: four. TPU: zero. GPU: hits known only at runtime. TPU: path computed at compile time.

② When does the cost show up? The GPU pays a little on every access, hidden at runtime; you never see or manage it. The TPU pays once, but you pay it.

⭐ Row ② has real numbers; say them. The course's only measure of the “static tax”: “Cold compile: nineteen minutes; warm, still seven and a half, with sixteen failed compiles. Every recompile is pure loss. And someone still must know Pallas.”

③ And when the premise fails? The GPU can reroute at runtime; the hardware works around it. The TPU has no runtime fix. Back to code; recompile.

Point at the right panel's four questions. The only thing they take home.

Four questions that work for any pair of chips. On the page each comes with a concrete trap. Read the traps too, or they're four slogans:

One: was this number derived? If so, is the formula beside it?
Two: binary or decimal? The official HBM spec is in GiB: 192 GiB is 206 GB, 7.4% apart.
Three: per chip, per device, or per core? v7 is 2 devices per chip, and logs report per device. Get it wrong and you're off by 2×.
Four: physical or usable? 94.74 GiB is usable; 2,500 is dense, no sparsity.

The whole course's landing. Slowly. Pause. Don't rush on. “The difference isn't ‘which is stronger’ but ‘who gets control.’ Hand it to people, and someone has to be able to write it. Hand it to the compiler, and you live with its quirks.” ⛔ Point at the three qualifiers at the bottom. Drop none. If time allows one thing, read these, not the conclusion; they've heard it for an hour and a half.

① The only like-for-like run says GB300 is 1.31× faster. It's a point, not a bound.
② “The TPU needs hand-written kernels” is an ecosystem gap, not an architectural one. Ecosystems converge; architecture doesn't. Conflating them claims a permanent disadvantage.
③ Mechanically, the TPU's premise holds on regular workloads, but this course never backed it with data. That one's on us.

⭐ If “who gets control” feels abstract, give a handle: “If you take back one thing, take operator fusion.” The figure states the principle. Fusion is the one concrete example running through the course, and it walks all three layers of it.

Why fuse: to pass the ridge point without changing the math, it's the only way.
How much: workbench size and blade width; two hardware numbers jointly cap it.
Who fuses: compilers do math-preserving fusion; math-changing is always human.

Physics, hardware, control: each layer sets one limit. Hence studying hardware always ends at fusion.

The last line. Finish and stop. No summary: “This course didn't tell you which to buy. It wants you, next time you see ‘A is N× faster than B,’ to know the four questions to ask, and which page holds its denominator and numerator.”
⭐ With ten seconds left, add a callback: “6.2's two ‘utilization’ numbers share a denominator. The whole gap is in the numerator.”