Fourteen topics
Each topic spells out three things: what it covers · why it sits where it does · what it sets up for later.
Core trackRequired · in order
Not a generic tour of the Transformer. We take apart a real one, DeepSeek V3, and follow one token all the way through, doing the math at every step.
- Covers
- From tokenization through MLA, MoE and the 61-layer stack, all the way to the output. Seven steps, each asking the same four questions: how big are the weights, how big are the activations, how much compute, and does anything blow up here.
- Why first
- Every hardware requirement is set by the shape of the model. You need to know what's required before you can ask how a machine supplies it. And taking apart one real, current model teaches more than ten abstract concepts.
- Sets up
- A conclusion you compute: it doesn't fit on any single chip. Every later lecture answers the questions that conclusion raises.
One grew out of gaming graphics cards; the other was built for matmul from day one. Yet on the spec sheet they're nearly identical, and that isn't where the difference lies.
- Covers
- Not a list of specs: the spec sheet is the first thing this lecture takes apart. Compute, HBM bandwidth and on-chip SRAM all agree within 10%, and the two chips' “compute ÷ bandwidth” ratios land on nearly the same number (312.9 vs 312.5). The real difference is that each lacks an entire layer the other has: the TPU has no hardware-managed on-chip cache, and the GPU has no programmable dedicated coprocessor. What's missing isn't capacity but who makes the decisions: one leaves them to runtime, the other to compile time. That one difference then plays out through memory access, compute units, interconnect and the whole software stack.
- Why second
- Lecture 1 has just shown that the model “doesn't fit.” This lecture opens by settling the number Lecture 1 left open (how much HBM a chip really gives you, and where the rest goes), then takes the chip apart.
- Editions
- L200 (about 109 minutes), 43 figures: for lectures and a first read.
L300 (about two and a half hours), 74 figures: for self-study, lookups and digging one level deeper.
Section numbers line up exactly (§0–§9): to dig into an L200 section, open the same section number in L300,
no searching required. L200 isn't L300 with paragraphs cut; it's the same story distilled again. The English edition covers L200 for now.
- Sets up
- A ruler and a test. The ruler is the ridge point at arithmetic intensity 312: to its left, buying compute doesn't help; to its right, saving data movement doesn't help. The test is “does this optimization change who decides, or how much data moves?” Every later claim that “this optimization doesn't help on TPU” or “this needs static shapes” points back here.
MLA, GQA, SWA, DSA, linear attention… so many names it looks like everyone is doing their own thing. But they all solve the same problem, and there are only three knobs to turn.
- Covers
- First, separate the two motivations: the hardware cost (the KV cache can outgrow the weights, and without a fix you can't ship) and the information argument (the attention matrix is extremely sparse to begin with, so most of what you compute is wasted). Then one table sorts every name onto three knobs: how much each token stores · how much each query looks at · different math altogether.
- Why third
- Lecture 1 computed a curve: at 4K context, attention is 12% of compute; at 128K, 82%; at 1M, 97%. Nothing about the model changes; the input just gets longer, and the bottleneck moves somewhere else entirely. Lecture 1 only shows why a change is unavoidable; this lecture shows how people made it. It comes right after, while the numbers are fresh.
- Sets up
- Every later lecture uses it: how to split the sequence dimension in parallelism, why prefill and decode have opposite bottlenecks in inference. All of it grows out of these three knobs.
- Two editions
- Core track (rewrite finished 2026-09-14): one story along a timeline. How RNNs remember → 2017 cuts the chain and replaces it with a table → long contexts make the table unaffordable → cut heads (MQA/GQA) → store a compressed form instead of K/V (MLA) → don't read everything (sliding window → DSA) → back to a fixed-size state → hybrid ratios. Many figures, few words; each chapter answers only “what did the last chapter leave unpaid, and how does this one pay it?”
L300 full edition: the earlier version, kept whole as an archive. Every derivation, ablation table and primary source, plus our own measurements on v7, 58 figures in all. Use it to look things up. The English edition covers the core track.
- In L300
- Every core section is drafted (2026-09-13). The focus isn't which schemes exist but how each was conceived and why it works: for example, how MLA can get away with compressing 32,768 down to 576, how DSA escapes the chicken-and-egg problem that judging what matters requires computing attention first, and why dropping four tokens from a sliding window sends perplexity from 5.40 to 5158.
Added 2026-09-16: the core track gained a full block on DeepSeek-V4. NSA / DSA / CSA / HCA are four steps along one road, and V4-Pro's KV looks nothing like what you'd guess (it isn't MLA but Shared K=V MQA; the 64-dim RoPE sits inside the 512, not outside).
Two threads also run through the whole lecture: compressing after the fact vs training compressed from scratch (it comes up four times), and hardware dictating the formula (from “dot products won because they can be written as matmuls” to “select by block, not by token”).
Lecture 1 only costed the forward pass. The real bill is much larger, and its biggest item is neither the weights nor the activations.
- Covers
- How much compute the backward pass takes (answer: it triples the total). Why activations can't be thrown away once computed. And the surprise: the largest thing in HBM is the optimizer state, while the model itself is the smallest.
- Why here
- The forward-pass math is clean fact, good for an opening. The backward pass involves strategy choices, so it comes once readers can already do the math.
- Sets up
- A complete bill, which is exactly what parallelism strategies split up. And once you've seen it, there's no need to memorize ZeRO's staged design.
- Order
- ⭐ This lecture is ordered by size, not by the training loop. The loop runs forward → backward → update, but that order would bring the biggest item on stage last, and it's the one that decides everything. Seven sections plus an appendix: one number and 671 billion parameters · how one number becomes 671 billion derivatives (how backpropagation works) · the cost: those intermediate results can't be thrown away · gradients in hand, how big a step to take (gradient descent and optimizers) · handing this bill to Topic 5 · putting all three on a machine you can touch · keeping training stable (what to do when the loss blows up) · appendix: how this lecture's numbers were checked.
⭐ Unlike Topic 3, this lecture can be taught out of order (it's a bill, not a storyline). Only one order is fixed: “why activations can't be discarded” must be established before the recomputation trade.
- Four tests
- Numbers go stale; these four don't. ① Whether a quantity needs high precision depends on its relative change per step versus the smallest step its mantissa can resolve (moving averages forget a little every step, so their updates register; master weights accumulate over a hundred thousand steps, each moving them by a few thousandths, and rounding swallows the update whole); ② whether to recompute depends on how many FLOPs you pay per byte saved: for a linear layer that number equals the input width, while for attention it grows linearly with sequence length, so the two lines must cross; ③ when two quantities grow at different rates in the same variable, and the faster one starts lower, their order must flip somewhere. Any “A is cheaper than B” must come with the range where it holds. This one appears three times in the lecture: sequence length, model size, and the same switch whose benefit changes sign with scale; ④ judge any stability technique on both “did it stabilize?” and “did quality drop?” Check only the first, and a learning rate of zero is optimal.
Spend one kind of communication to buy back memory or compute. Choosing a strategy means choosing which communication you're willing to pay for.
- Covers
- A map sorted by which dimension you split: batch, hidden dimension, sequence, layers, experts, across data centers. Each row fills four cells: communication primitive, volume, frequency, cost.
- Why here
- Earlier lectures showed that it “doesn't fit”; this one answers how to split it.
- Sets up
- Communication is the source of every later performance problem. And one rule keeps coming back: frequency decides how slow a link a dimension can live on.
06InferencePrereq: 03 / 05English edition ✓ · core track
Training builds the model; inference puts it to work. And their constraints are opposite almost everywhere.
- Covers
- Inference is split down the middle: swallowing the prompt is compute-bound, while emitting tokens one at a time is bandwidth-bound. The two phases have opposite bottlenecks, and all the complexity of inference systems comes from there. Plus the whole body of engineering built around the KV cache.
- Why here
- Most people deploy inference before they ever train. And it's the same math redone under different constraints, the best test of whether the earlier lectures stuck.
- Sets up
- Tuning, quantization and RL all come back to this lecture's two-phase model.
07Performance tuning and toolingPrereq: 02 / 05English edition ✓ · core track · planned 82′ · 34 screenshots + 10 diagrams + 6 step-through recordings + 2 animations · notes + 24 exercises · 7 real profiles to download
Change one set of parameters and a training step gets 50% faster. Hold the celebration: is that number even possible?
- Covers
- A real tuning run on Tencent Hunyuan3 in FP8. Estimate first, with the roofline and comparable yardsticks; then drill down with open-source XProf, one layer at a time: Overview → Trace Viewer → Op Profile / Op Stats → memory → HLO → code. The missing time turns out to be a large block of work that was never done. Then a control: the same change on BF16, where the speedup is real.
- Why last
- The first six lectures teach you to compute how fast it should be; this one teaches you to find out why it isn't, and what to check when a result looks too good to be true. Without that math, a profile is unreadable.
- Hands-on
- All 7 profiles used in the course can be downloaded. Install open-source XProf, open the same screens as the course page, and work through the 24 exercises.
Deep divesAs needed · any order
08GPU→TPU training numerics alignmentPrereq: 02 · 04English edition coming
TPU and GPU can never match bit for bit. What you can demand: the TPU–GPU gap is no bigger than the GPU's gap with itself.
- The case
- Hunyuan3 moved from GPU to TPU, and the TPU loss sat 1.5e-4 lower for 1,000 steps. Tracked down level by level, the TPU wasn't the one off: a 120k-long BF16 sum on the GPU was truncated toward zero at every step.
- For now
- The Chinese edition (courseware and lecture notes) is live.
09Precision and quantizationPrereq: 04 · 06English edition coming
Each step down in precision halves memory and bandwidth. The price is numerical safety margin; this lecture shows exactly where that boundary lies.
- Done so far
- Scaling itself (from per-tensor to groups of 16), the 16×16 and the chain rule in FP4 training, and what Blackwell actually added, all three with full source logs.
- Also answers
- A counterintuitive question: why the copy of the weights held by the optimizer must be high precision.
10Post-training and RLPrereq: 04 · 06
RL puts an inference engine inside the training loop.
- Hard part
- An RL system houses a training framework and an inference engine at once, sharing weights and competing for the same chips. Every difficulty follows from that one sentence.
11Diffusion and DiTPrereq: core track
Run a different family of models through the same framework, and you find it wants almost the opposite from the hardware.
- Why bother
- No KV cache, static shapes, compute-bound. It's both a control group and a test of whether the earlier framework holds up.
14Distillation and knowledge transferPrereq: 01 · 10
The same forward pass, the same 31.56 GiB of logits. Change the kind of label, and the supervision you get back differs by orders of magnitude.
- Covers
- Topic 1 computed how big that tensor is. This lecture asks the other half: how much information do we actually get back from it? Hinton's recipe (soft labels, dark knowledge, temperature) buys 33× data efficiency in experiments.
- The twist
- Almost no production distillation pipeline ships logits, including those that clearly have them. The second half does the math: storing logits costs 64,640× more than storing token IDs.
Hands-onFor engineers
12JAX and XLAPrereq: 02
Earlier lectures cover what to do; this one covers how to write it.
- Core idea
- A shift in mental model: from writing commands to writing descriptions. And the line that stops most people the first time: single-chip code plus a few annotations is multi-chip code.
13Migration: PyTorch → TPUPrereq: 12 · 09
The hard part isn't getting it to run. It's proving that what runs is still the same model.
- Focus
- Half is how to port it; the other half is a methodology for numerical validation, and that's where the real skill lies.