🌐 中文

Modern AI Accelerators: GPU / TPU Systems Course

Today's large models were shaped by the chips they run on.
And today's chips were shaped by the models they run.

Fourteen topics that take these two worlds apart and line them up, from the life of a single token to where the last 80% of the compute goes. Nothing to memorize: every step is worked out in front of you.

Course author Chris Yang·Google Cloud AI Infra architect

🎬 Video lectures for Topics 1–5 are out, in Mandarin (Bilibili / YouTube, 4K, with subtitles).

The 30-second version

Read this first, then decide whether to keep going.

What

The layer between large models and hardware: why models look the way they do, what machines they need, and why they need so many chips

Who

Anyone whose work touches large models, whether you write code, build products or make the calls

How long

14 topics, 60–80 minutes each, in three tiers: 7 core, the rest as needed

How to study

Each topic stands on its own. If it makes sense, keep going; if you get stuck, go back for the prerequisite

Not covered

Algorithm derivations · tuning recipes · step-by-step operations

Finish the core track and you can take a model and a set of chips
and say how to run it, and why.

What you'll be able to answer

Ask why large models are expensive and the usual answer is they have a lot of parameters. That's true, but push one level deeper and it stops being enough:

An optimization that works well on GPUs does nothing at all on TPUs. What's different?

Why does adding chips sometimes double your speed, and sometimes not help at all?

At inference time, how many users can one chip serve at once, and where does that number come from?

Why do tricks that worked last year stop working on this year's models?

None of these has a one-line answer. Each is a chain of causes, and walking that chain is what this course does.

The method is simple: don't memorize conclusions; do the math. We take apart a real model and follow one token all the way through, and at every step we work out the cost: how much HBM, how much compute, how much communication.

By the end, “it doesn't fit on any single chip” is no longer something you heard; it's a result you computed yourself. And every technique that follows (parallelism, quantization, sparse attention, expert routing) becomes a response to that result.

Who this is for

In one sentence: anyone whose work touches large models and who wants to understand what's really going on.

You'll get a lot out of it if:

  • Your work involves large models in any way, whether you write code, build products or make decisions
  • You want to know where large language models came from and why they look the way they do today
  • You need to read someone else's training or inference plan and judge whether it holds up
  • You talk to the teams that build these systems and want more than the vocabulary
  • You want to know what problems the new architectures (MLA, MoE, sparse attention) actually solve

What this course doesn't do:

  • It isn't a runbook to type along with. That lives in the repo, not in lecture time
  • It isn't an exhaustive survey. Everything we cover goes down to the root, but we don't try to list every alternative
  • It doesn't teach algorithms (how to design a model, how to tune hyperparameters). That's another course
No background required. Every lecture lists its prerequisites up front; if you get stuck, go back to that one section. The starting point is low, but the finish line isn't. The course's job is to hand you the map, and every main road on it runs all the way to the end: not a list of terms, but enough to keep reasoning on your own. If you want to go deeper still, let's talk one on one.

What we cover, and what we don't

This line sets the depth of the course: directions and orders of magnitude, not specific parameter values.

✅ Covered❌ Not covered
What a component costs, roughly (order of magnitude)Measurements to the decimal point
What a technique trades (compute for memory, …)How to set its parameters
Why a design came aboutIts mathematical derivation
The counterintuitive parts (looks right, is wrong)Full ablation tables
How the shape of the hardware shaped the shape of the modelHow to stand up a cluster or submit a job
Operational material (building clusters, submitting jobs, fault tolerance, compilation caches) stays out of the course. A step-by-step doc covers it; it isn't worth lecture time.

Course map

The core track is one causal chain: each lecture's conclusion is the next lecture's question. Skip one, and from there on you can only memorize conclusions.

1
The life of a tokenFollow one token all the way through, doing the math as you go
Conclusion: it doesn't fit on any single chip
2
TPU vs GPUSo what does a chip actually look like?
Naive attention sits at 64, below the ridge point of 312
3
The evolution of attentionSo how did everyone get around it?
The forward pass is now fully costed: what's still missing?
4
Backward pass and optimizersThe bill isn't finished, and the biggest item is here
Conclusion: it has to be split
5
Parallelism strategiesSo which dimension do you split along?
New setting: redo the whole bill
6
InferenceConstraints opposite to training's almost everywhere
It runs: so where is it slow?
7
Performance tuning and toolingFind the problem first, then fix it

Deep dives · as needed, any order

Hands-on · for engineers

These two are required only if you'll be writing code.
Everyone else can skip them.

Fourteen topics

Each topic spells out three things: what it covers · why it sits where it does · what it sets up for later.

Core trackRequired · in order
01The life of a tokenEnglish edition ✓
Not a generic tour of the Transformer. We take apart a real one, DeepSeek V3, and follow one token all the way through, doing the math at every step.
Covers
From tokenization through MLA, MoE and the 61-layer stack, all the way to the output. Seven steps, each asking the same four questions: how big are the weights, how big are the activations, how much compute, and does anything blow up here.
Why first
Every hardware requirement is set by the shape of the model. You need to know what's required before you can ask how a machine supplies it. And taking apart one real, current model teaches more than ten abstract concepts.
Sets up
A conclusion you compute: it doesn't fit on any single chip. Every later lecture answers the questions that conclusion raises.
02TPU vs GPUEnglish edition ✓ · L200
One grew out of gaming graphics cards; the other was built for matmul from day one. Yet on the spec sheet they're nearly identical, and that isn't where the difference lies.
Covers
Not a list of specs: the spec sheet is the first thing this lecture takes apart. Compute, HBM bandwidth and on-chip SRAM all agree within 10%, and the two chips' “compute ÷ bandwidth” ratios land on nearly the same number (312.9 vs 312.5). The real difference is that each lacks an entire layer the other has: the TPU has no hardware-managed on-chip cache, and the GPU has no programmable dedicated coprocessor. What's missing isn't capacity but who makes the decisions: one leaves them to runtime, the other to compile time. That one difference then plays out through memory access, compute units, interconnect and the whole software stack.
Why second
Lecture 1 has just shown that the model “doesn't fit.” This lecture opens by settling the number Lecture 1 left open (how much HBM a chip really gives you, and where the rest goes), then takes the chip apart.
Editions
L200 (about 109 minutes), 43 figures: for lectures and a first read. L300 (about two and a half hours), 74 figures: for self-study, lookups and digging one level deeper. Section numbers line up exactly (§0–§9): to dig into an L200 section, open the same section number in L300, no searching required. L200 isn't L300 with paragraphs cut; it's the same story distilled again. The English edition covers L200 for now.
Sets up
A ruler and a test. The ruler is the ridge point at arithmetic intensity 312: to its left, buying compute doesn't help; to its right, saving data movement doesn't help. The test is “does this optimization change who decides, or how much data moves?” Every later claim that “this optimization doesn't help on TPU” or “this needs static shapes” points back here.
03The evolution of attentionEnglish edition ✓ · core track
MLA, GQA, SWA, DSA, linear attention… so many names it looks like everyone is doing their own thing. But they all solve the same problem, and there are only three knobs to turn.
Covers
First, separate the two motivations: the hardware cost (the KV cache can outgrow the weights, and without a fix you can't ship) and the information argument (the attention matrix is extremely sparse to begin with, so most of what you compute is wasted). Then one table sorts every name onto three knobs: how much each token stores · how much each query looks at · different math altogether.
Why third
Lecture 1 computed a curve: at 4K context, attention is 12% of compute; at 128K, 82%; at 1M, 97%. Nothing about the model changes; the input just gets longer, and the bottleneck moves somewhere else entirely. Lecture 1 only shows why a change is unavoidable; this lecture shows how people made it. It comes right after, while the numbers are fresh.
Sets up
Every later lecture uses it: how to split the sequence dimension in parallelism, why prefill and decode have opposite bottlenecks in inference. All of it grows out of these three knobs.
Two editions
Core track (rewrite finished 2026-09-14): one story along a timeline. How RNNs remember → 2017 cuts the chain and replaces it with a table → long contexts make the table unaffordable → cut heads (MQA/GQA) → store a compressed form instead of K/V (MLA) → don't read everything (sliding window → DSA) → back to a fixed-size state → hybrid ratios. Many figures, few words; each chapter answers only “what did the last chapter leave unpaid, and how does this one pay it?”
L300 full edition: the earlier version, kept whole as an archive. Every derivation, ablation table and primary source, plus our own measurements on v7, 58 figures in all. Use it to look things up. The English edition covers the core track.
In L300
Every core section is drafted (2026-09-13). The focus isn't which schemes exist but how each was conceived and why it works: for example, how MLA can get away with compressing 32,768 down to 576, how DSA escapes the chicken-and-egg problem that judging what matters requires computing attention first, and why dropping four tokens from a sliding window sends perplexity from 5.40 to 5158.
Added 2026-09-16: the core track gained a full block on DeepSeek-V4. NSA / DSA / CSA / HCA are four steps along one road, and V4-Pro's KV looks nothing like what you'd guess (it isn't MLA but Shared K=V MQA; the 64-dim RoPE sits inside the 512, not outside).
Two threads also run through the whole lecture: compressing after the fact vs training compressed from scratch (it comes up four times), and hardware dictating the formula (from “dot products won because they can be written as matmuls” to “select by block, not by token”).
04Backward pass and optimizersPrereq: 01 · some sections need 03English edition ✓ · core track
Lecture 1 only costed the forward pass. The real bill is much larger, and its biggest item is neither the weights nor the activations.
Covers
How much compute the backward pass takes (answer: it triples the total). Why activations can't be thrown away once computed. And the surprise: the largest thing in HBM is the optimizer state, while the model itself is the smallest.
Why here
The forward-pass math is clean fact, good for an opening. The backward pass involves strategy choices, so it comes once readers can already do the math.
Sets up
A complete bill, which is exactly what parallelism strategies split up. And once you've seen it, there's no need to memorize ZeRO's staged design.
Order
⭐ This lecture is ordered by size, not by the training loop. The loop runs forward → backward → update, but that order would bring the biggest item on stage last, and it's the one that decides everything. Seven sections plus an appendix: one number and 671 billion parameters · how one number becomes 671 billion derivatives (how backpropagation works) · the cost: those intermediate results can't be thrown away · gradients in hand, how big a step to take (gradient descent and optimizers) · handing this bill to Topic 5 · putting all three on a machine you can touch · keeping training stable (what to do when the loss blows up) · appendix: how this lecture's numbers were checked.
⭐ Unlike Topic 3, this lecture can be taught out of order (it's a bill, not a storyline). Only one order is fixed: “why activations can't be discarded” must be established before the recomputation trade.
Four tests
Numbers go stale; these four don't. ① Whether a quantity needs high precision depends on its relative change per step versus the smallest step its mantissa can resolve (moving averages forget a little every step, so their updates register; master weights accumulate over a hundred thousand steps, each moving them by a few thousandths, and rounding swallows the update whole); ② whether to recompute depends on how many FLOPs you pay per byte saved: for a linear layer that number equals the input width, while for attention it grows linearly with sequence length, so the two lines must cross; ③ when two quantities grow at different rates in the same variable, and the faster one starts lower, their order must flip somewhere. Any “A is cheaper than B” must come with the range where it holds. This one appears three times in the lecture: sequence length, model size, and the same switch whose benefit changes sign with scale; ④ judge any stability technique on both “did it stabilize?” and “did quality drop?” Check only the first, and a learning rate of zero is optimal.
05Parallelism strategiesPrereq: 04English edition ✓ · core track
Spend one kind of communication to buy back memory or compute. Choosing a strategy means choosing which communication you're willing to pay for.
Covers
A map sorted by which dimension you split: batch, hidden dimension, sequence, layers, experts, across data centers. Each row fills four cells: communication primitive, volume, frequency, cost.
Why here
Earlier lectures showed that it “doesn't fit”; this one answers how to split it.
Sets up
Communication is the source of every later performance problem. And one rule keeps coming back: frequency decides how slow a link a dimension can live on.
06InferencePrereq: 03 / 05English edition ✓ · core track
Training builds the model; inference puts it to work. And their constraints are opposite almost everywhere.
Covers
Inference is split down the middle: swallowing the prompt is compute-bound, while emitting tokens one at a time is bandwidth-bound. The two phases have opposite bottlenecks, and all the complexity of inference systems comes from there. Plus the whole body of engineering built around the KV cache.
Why here
Most people deploy inference before they ever train. And it's the same math redone under different constraints, the best test of whether the earlier lectures stuck.
Sets up
Tuning, quantization and RL all come back to this lecture's two-phase model.
07Performance tuning and toolingPrereq: 02 / 05English edition ✓ · core track · planned 82′ · 34 screenshots + 10 diagrams + 6 step-through recordings + 2 animations · notes + 24 exercises · 7 real profiles to download
Change one set of parameters and a training step gets 50% faster. Hold the celebration: is that number even possible?
Covers
A real tuning run on Tencent Hunyuan3 in FP8. Estimate first, with the roofline and comparable yardsticks; then drill down with open-source XProf, one layer at a time: Overview → Trace Viewer → Op Profile / Op Stats → memory → HLO → code. The missing time turns out to be a large block of work that was never done. Then a control: the same change on BF16, where the speedup is real.
Why last
The first six lectures teach you to compute how fast it should be; this one teaches you to find out why it isn't, and what to check when a result looks too good to be true. Without that math, a profile is unreadable.
Hands-on
All 7 profiles used in the course can be downloaded. Install open-source XProf, open the same screens as the course page, and work through the 24 exercises.
Deep divesAs needed · any order
08GPU→TPU training numerics alignmentPrereq: 02 · 04English edition coming
TPU and GPU can never match bit for bit. What you can demand: the TPU–GPU gap is no bigger than the GPU's gap with itself.
The case
Hunyuan3 moved from GPU to TPU, and the TPU loss sat 1.5e-4 lower for 1,000 steps. Tracked down level by level, the TPU wasn't the one off: a 120k-long BF16 sum on the GPU was truncated toward zero at every step.
For now
The Chinese edition (courseware and lecture notes) is live.
09Precision and quantizationPrereq: 04 · 06English edition coming
Each step down in precision halves memory and bandwidth. The price is numerical safety margin; this lecture shows exactly where that boundary lies.
Done so far
Scaling itself (from per-tensor to groups of 16), the 16×16 and the chain rule in FP4 training, and what Blackwell actually added, all three with full source logs.
Also answers
A counterintuitive question: why the copy of the weights held by the optimizer must be high precision.
10Post-training and RLPrereq: 04 · 06
RL puts an inference engine inside the training loop.
Hard part
An RL system houses a training framework and an inference engine at once, sharing weights and competing for the same chips. Every difficulty follows from that one sentence.
11Diffusion and DiTPrereq: core track
Run a different family of models through the same framework, and you find it wants almost the opposite from the hardware.
Why bother
No KV cache, static shapes, compute-bound. It's both a control group and a test of whether the earlier framework holds up.
14Distillation and knowledge transferPrereq: 01 · 10
The same forward pass, the same 31.56 GiB of logits. Change the kind of label, and the supervision you get back differs by orders of magnitude.
Covers
Topic 1 computed how big that tensor is. This lecture asks the other half: how much information do we actually get back from it? Hinton's recipe (soft labels, dark knowledge, temperature) buys 33× data efficiency in experiments.
The twist
Almost no production distillation pipeline ships logits, including those that clearly have them. The second half does the math: storing logits costs 64,640× more than storing token IDs.
Hands-onFor engineers
12JAX and XLAPrereq: 02
Earlier lectures cover what to do; this one covers how to write it.
Core idea
A shift in mental model: from writing commands to writing descriptions. And the line that stops most people the first time: single-chip code plus a few annotations is multi-chip code.
13Migration: PyTorch → TPUPrereq: 12 · 09
The hard part isn't getting it to run. It's proving that what runs is still the same model.
Focus
Half is how to port it; the other half is a methodology for numerical validation, and that's where the real skill lies.

How to use this course

You don't need all of it. Pick a path for your role.

You are…Suggested path
After the big pictureCore 01–06 is enough
Making technical calls / reviewing designsCore + 07
Moving GPU training to TPUCore + 08, 09, 13
Focused on cost and capacity planning01, 03, 05 + 09
Going to write codeCore + 12, 13
Only deploying inference01, 02, 05, then 07, 09 as needed
Working on multimodal / generative modelsCore + 11
Making large models smaller / shipping them01, 06 + 09, 14
Every topic stands on its own. If you get stuck, check the prerequisites at the top and go back to that lecture.