How to use these notes
The presenter's notes, not student material. Students read the
course page; you read this.
Course author: Chris Yang · AI Infra architect, Google Cloud.
| 🎯 | What to leave behind The one line the room keeps. |
| 🗣 | Script Yellow: read as written; gray italics: the people's own words, read verbatim. |
| 🖥 | Screen Which course-page figure, and which panel to zoom into.
⭐ Every cue carries class="board"; the build flags a missing figure or section. |
| ⚠️ | What not to say Where this segment most easily drifts. |
⏱ Time: core 60′ / full 87′
⭐ Unlike the other lectures, this is one storyline: you can't skip around. Each section's first and last sentences are interfaces; skip one and the next has nothing to connect to.
| Segment | Core 60′ | Full 87′ | Notes |
|---|---|---|---|
| Opening (the gun) | 3 | 3 | ⛔ One number is enough. No survey |
| 1 · A chain | 5 | 7 | Only to set up the shape of “the box” |
| 2 · The 2017 cut | 8 | 10 | ⭐ The only scene with people (3′) + naming the lead, S |
| 3 · Bill due | 5 | 7 | The ruler for “what is saved / how much” |
| 4 · Cut heads | 6 | 8 | Test: volume vs. degrees of freedom |
| 5 · MLA | 10 | 12 | ⭐⭐ The hardest. Don't compress 5.4's five minutes |
| 6 · Read less | 9 | 12 | ⭐ Sliding-window whodunit + new DSA mechanism figure |
| 7 · Back to start | 8 | 10 | ⭐⭐⭐ The climax. Read all three quotes |
| 8 · Use both | 3 | 8 | Short on time: only 8.1 and 8.5 |
| 9 · Machine + close | 3 | 10 | ⛔ Closing two lines: never cut, however tight |
| Total | 60 | 87 |
⛔⛔ Above is the wish; below is the floor. “Floor” = all spoken words at 140 wpm (= Chinese 200 chars/min) + 15 s/figure switch, zero improvisation. Counted, not estimated.
| Section | Script (words) | Switches | Floor | 87′ budget |
|---|---|---|---|---|
| 1 · A chain | 1,887 | 4 | 14.5′ | 7′ |
| 2 · The 2017 cut | 931 | 3 | 7.4′ | 10′ |
| 3 · Bill due | 482 | 3 | 4.2′ | 7′ |
| 4 · Cut heads | 405 | 4 | 3.9′ | 8′ |
| 5 · MLA | 1,423 | 4 | 11.2′ | 12′ |
| 6 · Read less | 2,823 | 8 | 22.2′ | 12′ |
| 7 · Back to start | 2,586 | 5 | 19.7′ | 10′ |
| 8 · Use both | 1,347 | 6 | 11.1′ | 8′ |
| 9 · Machine + close | 801 | 5 | 7.0′ | 10′ |
| All nine sections | 101′ | 84′ | ||
⭐⭐ Cutting to 60 minutes: cut “expansions,” never “interfaces.”
- ⛔⛔ First, a correction. This used to say: “cut the three pains in 1, the three GQA reasons in 4, and 8.2–8.4, and you win back fourteen minutes.” That was miscounted. 1's three pains are already “read the headings; don't expand”: saves 0. 4's reasons are one spoken line (under 40 words): seconds. 8.4 was already “87′ version only.” Together: under a minute.
- ⭐ So the 60′ version means choosing whole sections, not trimming passages, by the floor table. The most expensive are 6, 7 and 1. But 6 holds the “saves no memory” interface, 7 is the climax, 1 builds the box the lecture returns to, so real time can only come from 8: cut to 8.1 and 8.5.
- ⛔ Cut these and it falls apart: 2.5 (“heads” are born) and 2.6 (naming S); the ruler in 3; 5.4; 7.5; the two closing lines in 9. The storyline's six joints.
- ⚠️ Say Section 6's “it doesn't save memory” in every version. Without it, 8's “so use both” comes from nowhere.
⛔⛔ The time math: the 60′ version can't be delivered as written. Decide cuts beforehand.
Counted, not estimated.
All spoken lines total 12,934 English words (including the quotes you must read; at 140 words/min
= 92 minutes) + 44 figure switches
(15 seconds per scroll-and-locate = 11.0 minutes).
That's 103 minutes, before any time for explaining figures, interaction or Q&A.
⛔ Hand-typed counts went stale (one by 600+ characters).
topic03-build-lecture.py recounts the Chinese on every build; recount these when the English changes.
⭐ 87′ version: leaves -16 minutes for figures and ad-libs.
⛔ 60′ version: only -43 minutes of buffer. One raised hand and it collapses.
⭐ So for the 60′ version, choose whole sections beforehand, not small trims: Section 1's three pains · Section 4's “why eight groups” · Section 8's 8.4: measured above, together under a minute. Real time exists only in Section 8: cut it to 8.1 and 8.5; the savings are in Section 8's floor-table row.
⛔ These six can't be touched in any version: 2.5 (“heads” are born) · 2.6 (naming S + the KV cache is born) · Section 3's two-question ruler · 5.4 · 7.5 · the two closing lines in Section 9.
🧵 Before class: three hidden threads
⭐ Thirty terms, but only three threads. Name the thread each time you touch one; the room remembers threads, not terms.
① Hardware ends up dictating the formula. Section 2 (dot product wins: it's a matmul) → Section 4 (eight groups, eight devices) → Section 6 (pick by block, not token) → Section 9 (even the block size changed to fit).
② One identity, parentheses moved twice.
Section 5 (absorption: nothing unpacked) → Section 7 (the big table is never built).
⛔ Only these two; Section 2 doesn't count: dot vs. additive changed the scoring formula, not associativity. Counting it breaks the thread at stop one.
⭐ Both times the enemy is alike: something wedged between two matrices, blocking the move.
Section 5: RoPE (a rotation with a position index, different per “who asks whom” pair).
Section 7: softmax. Move it aside first; then the parentheses move.
③ The box goes around and comes back. Set up in Section 1 → dropped in 2 → forced back once in 3 (decode) → invited back on purpose in 7.
⭐ If the room keeps only one sentence, make it Section 9's last line: for eight years we've been buying back the 2017 deal, without giving the parallelism up.🖥 Projecting: ⛔ never whole figures
⭐⭐ Zoom into one panel at a time. One panel filling the projection area is 2.8×: 11px becomes 31px, readable from the back row.
How: ① open the figure on its own and Ctrl/⌘ + scroll until one panel is left; ② then pan sideways to the next. Every “second panel” below assumes this.
⭐ Only three figures go up whole: Section 3's three-row comparison (Ⓐ Ⓑ Ⓒ together), Section 6's five masks (the skeleton emerges side by side), and the final chronicle: the closing frame; leave it up.Section 0 · Opening: the eight years at a glance (3′)
⭐ The opening's one job: hang the gun on the wall.
This lecture is about one number.
In 2020, GPT-3 could only remember 2,048 tokens at a time.
Hear “token” as roughly “a word”: in English a bit less than a word; in Chinese about one character.
I'll use “word” and “token” interchangeably.
Today's models remember a million.
That's 512×. This whole lecture is about how that 512× was bought.
⭐ And first, privately, guess a number. Of that 512×, how much came from simply buying better hardware? The last figure gives the answer.
🖥 Screen: “Attention chronicle” is collapsed. Open “First, the shape of the eight years,” show it whole for two seconds, then close it. ⛔ Read no names.
⭐⭐ Collapsed since the 2026-09-16 session. The words: “The chronicle further down, with the model names and years, is all I need.” ⭐ The test: when two things tell the same history, the weaker one shouldn't share the screen. Collapse, don't delete: ⛔ Section 9's callback points at it. ⚠️ This passage can be skipped; then point Section 9's callback at the model table.
Before we start, let's lay out everything that actually happened, from 2014 to today.
…There are over thirty names up there. Right now it's probably just noise to you.
That's normal, and it's on purpose.
⭐ Remember how it feels to look at it now. In two hours I'll ask you to look at this figure again.
By then it'll be a different figure.
🖥 Screen: “The story: six years spent making ‘remembering’ affordable” is collapsed. Open “Roadmap of the course (six stages) + two trend charts,” scroll to the two trend charts, point left and right, then close it. ⛔ Skip the six cards (for readers).
⭐⭐ Collapsed since the 2026-09-16 session. The words: “This part doesn't help the main line much. Collapse it. If there's time and someone wants to read it, they can read it on their own.” ⚠️ Skippable; Section 1 is unaffected. ⛔ But don't delete it: the trend charts are computed live from the course's 44-row model table, the lecture's only argument from our own data, and Section 9's close cites them.
⭐ Real data. The room needs one thing: one goes up, one goes down. No names. ⛔ The old “sweep your hand left to right” predates the lines. Skip it.
Left: each dot is how much context one model can take in. Right:
how much memory it pays at the same 128K length.
⭐ Don't read any dot. Watch only the two green lines: left goes up, right goes down.
What really happened in six years: the two lines moving opposite ways at once.
We hold ever more, and holding the same keeps getting cheaper. Agents are where they meet.
Today we'll take the six steps up there, left to right.
Each step fixes exactly one problem the last one left.
⭐ And the first step goes back before this line starts: the years before attention.
⚠️ Don't explain the KV cache in the opening. Save it for the end of Section 2, where it grows out of the story; now it's just a term to memorize.
⛔ Don't read a single name off the chronicle. It's there to show the mess; names wash that away.
⛔ Don't recite the roadmap's six stage names; they're for readers.
🖥 Main screen: course page Section 0 · The eight years at a glance · main figure “Attention chronicle”
Section 1 · A chain (7′)
🎯 Leave one shape in their heads: a fixed-size box. It comes back unchanged in Section 7, the lecture's climax.
Let's go back to 1990. The answer back then was almost crudely simple: carry a box. Read a word, mix it into the box, write it back. Read the next word, mix again. At the end, what's in the box is all the model knows about the passage.
⭐⭐ Spend thirty seconds unpacking the name. ⛔ The lecture says “RNN” from the first line but never spells it out. (Asked 2026-09-15.) ⭐ After the move, so the room recognizes the R from the picture. ⛔ Not in the opening: no picture yet.
While we're here, the three letters: RNN, Recurrent Neural Network.
⭐ The R refers to the move you just heard.
After the mix, the box doesn't leave. It loops back as the next step's input. That loop
is where the name comes from.
⭐ The word is plain, too. “Recurrent” is Latin: re-, back; currere, to run.
Literally, running back.
⭐⭐ Little known:
mathematicians didn't coin the word. It came from anatomy.
There, “recurrent” had long described looping structures.
Cajal saw them in the cerebellar cortex in 1901; Lorente de Nó found more in 1933.
⛔ Before that, the nervous system was treated as purely feed-forward.
Loops in the brain weren't seriously proposed until the 1940s.
⭐ This echoes in Section 7, where Albert Gu says “this branch is like a brain.” Borrowed from the brain, thirty years later borrowed back. ⛔ Not now.
⛔ And don't confuse it with a recursive neural network.
That's a different thing: it drills down a parse tree.
⭐ An RNN is recurrent: it loops along the time axis. Recursive drills down. One circles in place; the other digs.
🖥 Screen: in 1.1, scroll to “Opening the box”. Walk Ⓐ → Ⓑ, then read the takeaway band once.
⭐⭐ On 2026-09-16 this figure lost over half its height (3186 → 1300 px); 1.1b merged into 1.1. From the session: “Very simple things got far too many figures, and they overlap heavily.” ⛔ Panels Ⓒ and Ⓓ are gone: Ⓒ merged into Ⓑ (two sides of one move); Ⓓ became one line of the takeaway band.
⭐⭐⭐ Added 2026-09-16, after three questions in a row:
“What mechanism mixes it in?” “Why doesn't mixing scramble what's there?”
“Was KV invented here?” The course answered none.
⛔ Root cause: “mixing” suggests blending, which destroys.
⭐ A metaphor that plants a wrong physical intuition owes the room a figure.
A moment ago I said “mix.” That metaphor kept it simple,
but it misleads you.
⛔ “Mix” makes you think of blending. And blending destroys things.
⭐ So let's open the box and look.
⭐ Ⓐ. Inside one step, there are really just four moves.
① Rotate: multiply the previous box by a learned matrix. It turns to a new direction.
② Map: multiply the new word by another matrix, into the same space.
③ Add: add the two.
④ Squash: a nonlinearity pulls it back into range. Then write it back into the box.
⭐⭐ Step three matters most: it's an add, not a blend.
The new stacks on top of the old. Nothing old gets erased.
⛔ Ask the counter-question yourself.
⛔ The obvious question: after ten thousand adds, can you still pull them apart?
⭐⭐⭐ Ⓑ, left half first. This is the heart of the figure: not a claim, a number you can compute.
In two dimensions, two random directions are typically only about fifty degrees apart.
Stack them and they blur, beyond recovery.
But in 1,024 dimensions, two random directions are 88.6 degrees apart.
⭐ Almost perpendicular.
Two things stacked at nearly right angles can each be pulled back out.
⭐⭐ The line: superposition is recoverable because dimensions pay for it.
⚠️ Give the limit at once, or it sounds like “more dimensions fix all.”
⛔ But it's not unlimited: there are at most d directions that truly don't interfere.
Overfill it and it blurs anyway.
⭐⭐ Hold on to that. It comes back once more today,
to explain why another notepad fills up.
⭐ Ⓑ, right half. Why does it still fade with distance?
Because every step multiplies it again.
Step one to step one hundred: multiplied, and squashed, ninety-nine times.
⛔ If the matrix's “gain” is a bit under one, a hundred multiplies shrink it to zero exponentially;
a bit over, it blows up exponentially.
⭐⭐ That's the whole mechanism of fading: not washed out by time, but multiplied away.
⭐⭐⭐ The lecture's best early echo. Pause first.
⭐ A note for the end of today:
“multiply by less than one at every step” was this approach's fatal flaw in 1990.
Today's branch put it back in, on purpose.
One difference: back then the decay was uncontrolled. Today it's learned,
one per channel.
⭐⭐ Same property. Once it's the disease, once it's the cure.
⭐⭐⭐ The takeaway band. Last, an important question: was KV invented here?
⛔ No. The box holds neither.
The 1990 box holds a single vector. No keys, no content.
⭐ Keys and content as a pair came with attention.
The names Q, K and V had to wait until 2017.
The KV cache as “a bill to pay” came later still.
Only when autoregressive generation made recomputing the whole history every step cost real money
did anyone cache it.
⭐⭐ The next two lines land this section; then to “the box is always the same size.” ⛔ Don't skip: the ruler for the next eight sections; else each is “one more method.”
⭐⭐ Remember these two; they're the book's skeleton:
RNN state = all history squeezed into one blob.
KV cache = all history kept verbatim.
⭐⭐ The next three came from 2026-09-16 follow-ups. ⛔ The second answer is “the paper doesn't say”: present the silence as the answer. Don't supply a number.
⭐ Three questions you may have:
First, how long is that box, really? Smaller than you'd imagine.
In Elman's 1990 paper, the largest simulation used 150 numbers,
the two letter-prediction ones 20, the smallest, XOR, two.
⭐⭐ Today, one linear-attention notepad
holds tens of thousands of numbers per head, per layer.
With full attention, a long-context KV cache runs to gigabytes.
That's the jump from box to notepad.
Second, what precision? The paper doesn't say.
⭐ That's part of the answer: back then, precision wasn't something you traded off.
It was just the machine's ordinary floating point. Nobody treated it as a knob.
⭐⭐ Precision became worth discussing much later.
When a quantity starts showing up in papers is itself a signal:
that's when it became something to trade off.
And two you'll surely think of: where are the learned weights, and how big is the matrix that does the mixing?
⭐ Just three learned matrices: one maps the new word in, one rotates the previous state, one reads the answer out.
Plus biases.
⭐⭐ The sizes are in the figure. The sentence simulation: 31-dimensional input, 150-dimensional state.
The map-in matrix is 31 by 150: 4,650 numbers.
The rotate matrix is 150 by 150: 22,500 numbers.
The readout is another 4,650.
⭐⭐⭐ These two get an audible “oh.” Don't rush.
⛔ Surprise: the loop itself isn't trained.
The paper is blunt: “Recurrent connections are fixed at 1.0 and are not subject to adjustment.”
It just copies this step's state to the next, unchanged.
⭐ What's learned is the matrix that reads the copy back in.
And it holds seventy percent of all the parameters.
⭐⭐ For scale: the whole model has just over thirty thousand parameters.
Not thirty trillion, not three billion. Thirty thousand.
⭐⭐⭐ The most valuable line here: it explains what embeddings are for.
⭐ You'll ask: what did “word vectors” look like back then? How many bits?
⛔ Answer: there was no “embedding.”
Elman gave each word a 31-bit vector, one bit per word.
(Two bits were reserved for later experiments: the real vocabulary was 29 words.)
Assigned at random, never trained.
⭐⭐ The paper names two consequences:
First, the word vectors are mutually orthogonal.
Second, they carry no part of speech or meaning.
⭐⭐⭐ Those two sentences say exactly what embeddings were later for:
Then: perfectly orthogonal, but zero information.
Today: learned dense vectors, only nearly orthogonal, but carrying meaning.
⭐ Whether “nearly orthogonal” suffices is what the 88.6 degrees answered.
⭐⭐⭐ Added after four questions in a row on 2026-09-16. ⛔ Pause on the “8,000 numbers”: the cheapest scale comparison in the lecture.
⭐ Once usable, how big was the box?
Elman's 150 was a toy.
The first truly usable large RNN was the 2014 machine-translation paper:
four layers, 1,000 cells per layer, 1,000-dimensional word vectors, a 160,000-word input vocabulary, 80,000 on output.
⭐⭐ The very next sentence is the gem:
“The deep LSTM uses 8000 real numbers to represent a sentence.”
To this model, a whole sentence is eight thousand numbers.
⛔ Set that against the opening question:
a 488 GiB KV cache, stored in bf16, is 262 billion numbers.
⭐⭐⭐ Eight thousand to 262 billion: 32.75 million times.
That's the real gap between “squeezed into one blob” and “kept verbatim.”
⭐⭐⭐ The best of the four. Go slowly.
⭐ So what does each of the RNN's three matrices extract?
For attention you know it: Q is what I'm looking for, K is what I can answer, V is what I hand over.
⛔ The RNN's answer: no such division of labor.
⭐⭐ Not for lack of study: the architecture never separated the roles.
The input matrix translates the word into state space.
Forced into an analogy, it's K and V mixed:
what's written is both the content and its later lookup key, one thing.
The recurrent matrix does two jobs:
keeping (carrying the old state forward) and transforming (what grows, what decays).
So it's both keeper and retriever of memory.
⭐⭐⭐ This is exactly why attention was a breakthrough:
it split “what to remember” and “how to find it” into two independent parameterizations.
In an RNN, both jobs are one matrix moonlighting.
⭐ Q, K and V mean something clean
because retrieval was written into the architecture.
An RNN has no retrieval step; it only mixes.
⭐ So what are x and y?
x is the current word's vector. h is the state after all prior words.
y is the state through the output matrix: a vector as long as the vocabulary.
After softmax: each word's odds of coming next.
⭐⭐ So yes, it's next-token prediction.
Elman says so: “the task on each input cycle is to predict the next word in the sequence.”
1990. The same training objective, still used 35 years on.
🖥 Screen: point at the takeaway band's last item (starts with ⚠️).
⭐⭐ This passage had its own panel (Ⓐb, two loops) until 2026-09-16. ⛔ It covers Section 1's pain and Section 2's fix, not “what's in the box,” so the full version is in 1.2 and Section 2; here, one line. ⚠️ The session's intuition (“shouldn't y loop back as the next x?”) is right; state it as two loops, or you'll get teacher forcing wrong.
⭐ Briefly; details come next.
A sequence is actually tied together by two different loops.
One is the state loop: the box passed to the next step, in training and inference.
The 2017 cut undid it, which is why training went parallel.
⛔ The other is the autoregressive loop: the token emitted now is the next step's input.
It exists only in generation, and nobody has undone it yet.
⭐⭐⭐ So the line from the final section already makes sense:
every large model you use today, when it emits tokens, falls back to the 1990 chain.
⚠️ The passage below now lives only in a collapsed block on the course page (the “Eight live follow-ups” block). ⭐ Skippable: cover it if asked.
Last one: when did the state become two-dimensional? Much earlier than most think.
⛔ Matrix-valued state wasn't invented in 2020; it was rediscovered.
⭐ In the early nineties, summed outer-product matrices were already used as state.
In 2021 that line returned and proved it equivalent to linear attention.
⭐⭐ A line I love from the abstract: the outer products combine
“self-invented activation patterns (today called keys and values).”
Today's two words appeared, as outer products, thirty years ago.
⭐⭐⭐ One compressed as far as it can go, one not compressed at all.
They're the two extremes of the same thing.
And today's remaining eight sections cover all the space between them.
One detail here decided the next thirty years: the box is the same size, start to finish. Feed it ten words, it's this big. Feed it a hundred thousand, still this big.
⭐⭐ Pause. Add nothing. Someone will think “great, it saves space.” ⛔ Don't correct them. They're right, but only Section 7 shows it; correcting now spoils the biggest reversal.
🖥 Screen: go to “RNN: squeeze ‘the past’ into a fixed-size box” and zoom into the unrolled chain at right. Red row: “h3 needs h2, h2 needs h1.” Then the green/blue arrows: “Vertical arrows are independent; only the horizontal blocks parallelism.”
⭐ Take 1.2 slowly: the first time in the book the hardware gets the final say.
So what's wrong with it? First, it can only go one step at a time.
Ten thousand words means ten thousand rounds in a queue. Round two waits for round one: the previous word must update the box first.
⛔⛔ Say the words “can't run in parallel” here. The course page's Section 1 says “parallel” eight times; the old script, zero. Without it, Section 2's pitch, “training goes parallel, so it scales,” sounds like “a bit faster.”
“Ten thousand rounds” sounds merely slow. But the compute isn't the problem.
This machine does ten thousand words of compute in a blink. The ten thousand steps just can't start at once.
⭐ Slow and not parallel are different things. The next section depends on it.
On the real machine it's worse still: every round reloads all the weights from HBM, then does only a tiny bit of work with them.
That 313 isn't made up. It's the ridge point the previous lecture spent a whole section building:
for every byte moved from HBM, this machine can do 313 operations.
So this isn't “RNNs are a bit slow.” It's a very expensive machine spending ninety percent of its time waiting for data.
🖥 Screen: scroll to “Why RNNs are slow on accelerators” and point at the two trucks in the middle. ⛔ No arithmetic-intensity derivation; that's L300.
⭐ 1.3's three pains: point, read the headings, don't expand. Say only the bridge.
This section leaves two lines. Good: what you move per step doesn't grow with length.
Bad: strictly serial, and it can't hold enough: the farther it goes, the fainter.
The next cut targets the bad, but tosses the good too.
The remaining eight sections all pay for that.
⭐⭐⭐ Plant one line here. It pays off in Section 6. Section 6 uses it (good and bad are one property) to explain the attention-sink bug. ⛔ Heard first there, it won't land.
⭐ Last thing: in those two lines,
the good part and the bad part are one thing, both from “the box is always the same size.”
Nine times out of ten, a design's flaw is the same property as its strength.
We'll use that sentence three times today. This is the first.
⭐ The hook: Section 2 opens by answering it.
So if the attention add-on works so well,
why not drop the RNN and keep only the add-on?
…In the next section, somebody actually does.
⛔ For the persistent-RNN panel, one line (“people once tried pinning the weights on chip”). ⭐ But do say “this idea wins again six years later”: the fuse for Section 9.
🖥 Main screen: course page Section 1 · A chain · main figure “RNN: squeeze ‘the past’ into a fixed-size box”
Section 2 · The 2017 cut (10′)
🎯 Leave behind: the KV cache is born here.
⭐⭐ 2.1 is the lecture's only human story. Three minutes; read the original words.
The answer is: yes, and in 2017 someone actually did it.
⛔ But first, something many people get wrong: attention was not invented in 2017.
It already existed in 2014. And at first,
it was a small add-on bolted onto an RNN.
So the cut didn't invent attention.
It threw away the chain the add-on was bolted to, and kept only the add-on.
But before the cut, let me tell you where the add-on came from.
Nobody had a flash of inspiration. It was complained into existence, one version at a time.
⭐ Say the name once; after that, “the intern.” Foreign names are forgotten; “an intern” sticks, and who he was is half the story.
In 2014, Bengio's lab had an intern named Bahdanau.
He tried two designs, liking neither. The first was too complex to work; the second worked, but dumbly,
pairing words in order: third source word to third translated word.
Then one day…
⚠️ Before the quote, half a sentence of setup, or “decoder” is new to the room: “When he says ‘the decoder,’ he means the half that's writing out the translation.”
So one day I had this thought that it would be nice to enable the decoder RNN to learn to search where to put the cursor in the source sequence. This was sort of inspired by translation exercises that learning English in my middle school involved. Your gaze shifts back and forth between source and target sequence as you translate.
⭐ After the quote, pause a beat, then this line. It turns a term back into human experience.
So, that's where attention comes from. Not some grand theory. A student remembering how his eyes moved while he did his translation homework.
⭐⭐⭐ Softmax appears here first. Give it a plain line now: it's the root of the bug in Section 6. ⛔ Wait till then and the room meets a new mechanism and term at once.
He wrote this ‘soft search’ as something called softmax.
⭐ Think of softmax as one sentence: it turns a pile of scores into percentages that add up to exactly 100%.
Remember that: “adds up to exactly 100%.” Later today it causes real trouble.
It worked on the very first try.
He called it RNNSearch. Later he said: the name was not great.
The word “attention” was added by Bengio in one of the final passes.
⭐ Bengio's “why it has to be soft” is the foundation of the lecture. Read all of it. It explains every later softmax and plants the Section 6 bug.
This would be like first reading the whole book in French, then, having the story in mind, generating the English translation of the book. We would for sure forget some details. The solution was to allow looking back at any piece of detail in the French book while producing every additional word in the English translation. To make the whole thing compatible with traditional training with backpropagation, we had to make the attention ‘soft’: instead of picking a single location of focus, we have a total budget of attention that can be split across all the locations.
⭐ Notice his last line: “a total budget of attention
that can be split across all the locations.”
That's the softmax from a minute ago, in plain words.
⭐⭐ And it's a strength: because the total is fixed and can be split, the thing is trainable.
⛔ Hold on to this. In Section 6, it becomes the root of a bug.
⭐ Same line as in Section 1 (just a preview; its real second outing is Section 6):
Nine times out of ten, a design's flaw is the same property as its strength.
🖥 Screen: scroll to the 2.1 figure, “How attention came to be”, point at panel ②. Read the paper's reason, pause, and say: “Notice nothing in this reason is about which one is more accurate.” The hidden thread's first stop.
⛔⛔ First what the cut bought, then what it cost. Never reversed. The page's card says “Parallel, so it scales”; the old script never said “parallel.” ⭐ Connect to Section 1's “can't parallelize.”
In 2017 the cut came down: drop the recurrence, keep only the add-on.
⭐ Hence the paper's title: attention is all you need.
What did this cut buy? Parallelism.
Before: ten thousand steps, ten thousand rounds. Now all tokens go at once. No queue.
⭐⭐ Only this cut made scale possible;
without it, today's models couldn't be trained.
⭐ Move fast through 2.2–2.4: one panel per figure. The only stop: “the state is gone” on the 2.2 figure (“What it bought, what it owes” box). Payoff, then price.
Notice: not just the horizontal arrow was replaced. The box went with it. The advantage I told you about last section stopped being true the moment this cut landed.
⛔⛔ Finish the causal chain, every step. The old script jumped from “the box is gone” to 2.6's “S is the star,” skipping: it stops remembering → rereads everything → so the past must be kept. ⭐ They're the whole reason the KV cache has to exist; without them, 2.6's name is forced on the room.
So, with the box gone, how does it remember what was said before?
⭐ The answer is blunt: it doesn't.
Each time it emits a new token, it goes back and rereads every token so far.
But to go back and reread them, those earlier tokens have to be kept.
That's the first page of the bill.
But before we add it up, I need to show you something that multiplies it by eight.
🖥 Screen: in 2.5, scroll to “What multi-head buys”, point at panel Ⓐ.
⛔⛔ Go fast, but say the word “head” aloud. Section 4 is titled “cut heads.” If “head” is first heard as it's cut off, the cut doesn't hurt.
The left figure's one query wants three places at once:
the previous word, the subject, a name three paragraphs back.
⭐ Look at the red line. Averaged over three directions, it's short, hitting none. A single head's flaw: not too small, it averages away what should stay apart.
The fix, right: cut the 512 dimensions into eight slices of 64, each looking on its own, then concatenated.
⭐ These eight slices are the eight “heads.”
⭐ Remember two things. Section 4 needs them:
One: multi-head buys resolution, not compute. Same size, same parameters, just sliced.
Two: the cost: each head keeps its own K and V. Storage ×8, right there.
⛔ That eight is exactly what Section 4's first cut goes after.
⚠️ Add “eight is just what the 2017 paper chose.” Or Section 4's “dozens of heads” sounds misheard. DeepSeek-V3 has 128 heads (figure's bottom-right small print).
🖥 Screen: in 2.6, scroll to “Where the KV cache is born” (the no-symbols version), walk Ⓐ → Ⓑ → Ⓒ, and stay on Ⓒ. ⛔ The “Expert version” fold below it is collapsed by default. Don't open it in class.
⭐⭐ Live session, 2026-09-16: “That diagram is the Scaling Book's expert version. It's too dense;
I'm worried the students won't get it. Can you draw a kids' version?”
⭐ Hence the “no-symbols version”: nothing to decode.
⛔ The expert version stays: later sections redraw it, lighting what changed.
Not on this first pass; it undoes everything.
⛔ Don't make the room “find” the answer; Ⓒ shows it. ⭐ Give it, ask why: still interactive, and deeper.
Ⓑ has three piles.
Ⓒ already gives the answer: only the third keeps rising.
But take one second: why that one, and not the others?
…Because the others are fixed up front: weights, users, layers, layer width.
Only this strip grows on its own as the conversation goes on.
Name its length S. The star of this lecture is this S.
From here on, ask every trick one thing: does it shrink S's coefficient,
cut how much of S each step reads, or make S disappear from the shape altogether?
⭐⭐⭐ This is where the section lands. Say the term aloud. ⛔ Course 2.6 is titled “where the KV cache is born”; skip it and Section 3 opens with an unheard term.
S grows because every new token must save its own K and V:
the next question compares against them.
⭐ This growing stockpile has a name: the KV cache.
⭐⭐ So, this is where it's born: in the same second the 2017 cut landed.
All seven remaining sections ask one thing: this stockpile is too big. What now?
One last word on attention's inventor. When the Transformer paper came out in 2017,
Bahdanau's reaction: “I immediately declared to labmates that RNNs are dead.”
⭐ Half right. Next section, you'll see them alive and well elsewhere.
⛔ Don't derive √d_k. “It's just a standard deviation” is enough.
⚠️ The story invites rambling. At three minutes, move on: 2.6 is where it lands.
🖥 Main screen: course page Section 2 · The 2017 cut · main figure “How attention came to be”
Section 3 · The bill comes due (7′)
🎯 Leave one line: not if it fits, but how many it serves at once.
I just said he was only half right. This section makes good on that.
But first we turn last section's debt into something you can see.
⭐ This section introduces no new tricks. That's all it does.
🖥 Screen: in 3.1, scroll to “When generating text, the Transformer reverts to the 1990 chain”, all three rows at once (one of the few figures shown whole).
These three rows use the same ruler.
Ⓐ is the old chain from the opening.
Ⓑ is training: a whole passage fed in at once.
That's exactly where the 2017 cut won.
Ⓒ is when it writes back to you: the version you use every day.
⭐⭐ So here's what this section is really about:
It won at Ⓑ, but you use Ⓒ every day.
The row where it won and the row where you pay are not the same row.
And that's not a figure of speech. Look at the shape of row Ⓒ.
First token, second, third. Each waits for the one before it.
⭐ The previous token must join the stockpile before the next can see it.
The chain from Section 1 returns here, unchanged.
Only now it's forced: nobody wanted it.
⭐ Section 7 shows this scene again, and there someone invites it back on purpose.
🖥 Screen: in 3.2, scroll to “Why now: one model, two workloads, the lead changes”. Point at the two bars.
⛔ A line used to sit here: “same model, same weights, a different job, and the bottleneck moves.” The 2026-09-15 dry run skipped it, rightly: it repeated “where it won isn't where you pay” and the figure's own “No change. Only the workload moved.” One point, three times. ⭐ Instead read the figure's three numbers, never mentioned before. A bill needs numbers.
Why these last two years? The answer isn't in a paper. It's in the data center.
These two bars on screen are the same model.
On the left: one user, a short chat (4K context). On the right: sixty-four users, each with a long document (128K context).
⭐ Read three numbers with me.
The gray segment, weights: 625 GiB → 625 GiB. Not one byte changed.
The red segment, the stockpile: 0.27 GiB → 549 GiB. Up 2048×.
So devices needed: 7 → 13.
⭐⭐ Nothing changed. Same model, same weights, same technology. The workload changed.
⭐ Here's where ‘suddenly’ comes from.
In a few years, two things each grew tens of times. One: conversations got longer,
from thousands of tokens to hundreds of thousands. Two: more people served at once.
⛔ And both multiply the same term: that stockpile.
If one thing grows tenfold, you watch it get pricier bit by bit.
But if two things each grow tens of times and multiply,
you don't see it getting pricier. One morning it's suddenly the biggest piece.
🖥 Screen: same figure, box “A question order used all lecture”: that's the ruler.
Remember this ruler. From the next section on, every new term gets just two questions, in a fixed order:
One: what did it save? Memory, bandwidth, or compute?
Two: how much?
What it costs, we note along the way.
⛔ Never skip question one, and never ask it after question two.
A pretty multiple is best at making you forget what it actually saved.
⭐ Today you'll see a scheme that saves more than fifty times, and what it saves isn't what you think.
This section's last line is the lecture's key practical conclusion:
A machine holds one copy of the weights, and everyone shares it. But the KV needs a fresh copy for every user.
So what it limits isn't ‘can this machine hold the model,’
it's how many people this machine can serve at once.
⚠️ The three-row figure eats time. One sentence each for Ⓐ Ⓑ Ⓒ.
🖥 Main screen: course page Section 3 · The bill comes due · main figure “When generating text, the Transformer reverts to the 1990 chain”
Section 4 · The first saving: cut heads (8′)
🎯 Leave one test: volume, or degrees of freedom?
🖥 Screen: first, 4.1's “Four schemes”: one look at the whole, then move on.
⛔ Left to right it goes MHA → GQA → MQA → MLA, by “stores less and less,” not by date. We go by date (MQA 2019, then GQA 2023).
So for MQA, point at the third panel; don't just slide right.
There's the bill: one copy per user, and it grows as the chat gets longer.
So what's the most obvious way to save? This section tries it.
First the big picture: today, the first three; the rightmost, next section.
⭐ One reminder: this figure is ordered by who stores least, not who came first.
We'll go by date.
⛔⛔ Map the two vocabularies before going on. The old script jumped from “heads, K and V” to “eight readers, one address book,” with no link. The mapping is in the figure's header; the room only has ears. ⭐ Sixth time for this pattern: on the slide, never said aloud.
The next figure uses an everyday picture, so first the mapping:
a reader is a query head;
the address book in his hand is one copy of K and V.
Eight heads means eight readers and eight address books. The drawer is full.
The 2019 fix was blunt: all heads share one K and one V.
Dozens of heads used to mean dozens of copies. Now just one.
A clean saving. It's also where the model starts getting dumber.
⛔⛔ Two “eights” collide in this section. Separate them at the first “eight.”
The figure's eight readers are just for counting; 4.3's eight groups are a real hyperparameter.
⭐ At the first “eight,” say: the drawing's eight is for counting; the groups' is real. Or the two merge.
🖥 Screen: in 4.2, scroll to “Cutting heads—the book is costly, not the reader”, stop at panel ① and point at the three perplexities.
⭐ Point with the figure's own words. Its three options: “8 readers, a book each / 8 readers share 1 book / Send just 1 reader.”
Look at these three. The last two take exactly the same space: one address book.
The only difference: the second still has eight readers looking up; the third, just one.
That alone moves the number below by a full 1.0.
⚠️ “Perplexity” is new to the room; give it one plain line (it comes back four more times).
⭐ That number is perplexity. Think of it as: when the model reads the next word,
roughly how many candidates it's wavering between. Lower is better.
At around thirty, a gap of 1.0 is a big gap.
Later, some schemes change a whole lot to win back a few tenths.
⭐ So for any scheme, ask what it squeezed out:
a thinner address book (less volume), or fewer readers (fewer degrees of freedom)?
A thinner book is usually fine. Fewer readers is expensive.
We'll use this test four more times today.
⭐ Read the paper's line: MQA can lead to “quality degradation and training instability.” Not just worse: unstable. Data buys back lost quality. Instability means a run can crash anywhere.
🖥 Screen: stay on the same figure, panel ② “GQA—groups, one book each” (no new figure).
⛔⛔ Introduce GQA first, then ask “why eight groups.” The old script skipped the compromise: groups, not one book. ⭐ Same as Section 2's “the box is gone” and Section 3's “the chain returns”: conclusion present, middle step missing.
MQA went too far. Is there a middle path?
⭐ In 2023, someone offered one: don't merge them all; split into groups.
A few readers per group, one book per group.
That's GQA. However many groups, that's how many copies you store.
⭐⭐ Live session, 2026-09-16: “4.3 and 4.4 are a mess. They don't explain anything. Delete all that clutter.” ⛔ The page dropped the three “why exactly 8” arguments (shard replication, speed curve, Llama-2-70B config) and the GQA conversion cost. ⭐ One line stays, “8 groups ≈ 8 accelerators,” because Section 9's close points back to it (the thread “hardware set a hyperparameter”). Say only that line.
⭐ So why did everyone independently pick eight groups?
Because a server usually holds exactly eight accelerators: one group each.
⭐⭐ So that number wasn't derived from the model. It was counted off the chassis.
⛔⛔ 2026-09-16, second call-out: “This wrap-up is useless. Fold it away.” All of 4.3 is now collapsed by default.
⭐ Skippable: go from the ruler close straight to Section 5.
⚠️ If you cover it, open the fold: Section 5's opening “hard-coded matrix” refers to it. If you skip, open Section 5 with: “GQA's matrix is hard-coded, a few 1s in its cells.”
🖥 Screen: (optional) open the 4.3 fold, scroll to “‘Splitting and copying are simple linear transforms’: draw it”.
For the first time all four terms sit in one spot, compared on one thing only:
what's written in the same matrix.
GQA's is hard-coded, a few 1s in its cells. And the next section unlocks that matrix.
⭐ Before leaving, measure with Section 3's ruler: its first real use.
⭐ Let's close with the ruler:
What did it save? Memory. How much? The cache, divided by eight.
And the price? A little quality, and training can go unstable.
⚠️ The “eight accelerators” line: say it's this course's reading, not the paper's (the course page marks it “Aside”). The paper only says that sharding replicates, and GQA removes that waste.
🖥 Main screen: course page Section 4 · The first saving: cut heads · main figure “Cutting heads—the book is costly, not the reader”
Section 5 · MLA (12′, the hardest)
🎯 Leave one line: MLA is elegant because it satisfies two budgets at once. ⛔ Not “it saves 57×.” That number doesn't matter.
⛔⛔ Don't open with “unlock that matrix.” It's a jump. 2026-09-15, first question here: “this has nothing to do with anything.” ⛔ Fair: the room doesn't yet know all four share one shape. ⭐ Premise first, in four lines.
To follow this section, accept one premise: all four schemes share one shape.
Store one thing in a warehouse; from it, make K and V for all 128 heads.
⭐ All four do this. They differ in just two ways: how wide the stored thing is,
and what action “makes the 128 copies.”
Start with GQA. The warehouse holds eight cups of mixed juice.
At serving time, each cup is photocopied into sixteen identical ones: 128 in all.
⭐ Notice: those sixteen match to the drop.
The fifteen extras are pure redundancy, nothing new.
So why copy at all?
Because the next step needs a full copy per head for shapes to line up.
Same contents or not, you need 128.
⭐ GQA saves warehouse space: only eight cups.
The price: sixteen heads must share one cup. They can no longer look at different things.
The other two sit at either end.
MQA goes further: one cup, photocopied 128 times. Everyone drinks from it.
MHA is the other end: 128 different cups, stored honestly, none copied.
Costliest, but freest.
⭐⭐ Now MLA.
What it stores is not juice. It's a packet of concentrate.
Each of the 128 heads holds its own recipe,
and from the same packet they mix 128 cups, all different.
⭐ So “unlocking” frees the action itself:
from photocopying to mixing.
Copying only gives identical cups. Mixing can give different ones.
And the two cost about the same: either way you make
128 copies from the one thing in the warehouse.
⛔ So: if you must do this step anyway, why only copy, and never mix?
⚠️ A guardrail. ⛔ Not “MHA and GQA are special cases of MLA,” an online inflation. Su Jianlin: “MLA is regarded as a generalization of GQA.” ⭐ Say only: same position, same action; they differ in how much that action may do.
⛔⛔ Don't say “that matrix only has 0s and 1s in its cells.” 2026-09-15, verbatim: “What are the 0s and 1s? I still don't get it.” It restates the conclusion in symbols. ⭐ Photocopy and mix say it all. Symbols are L300.
⛔ An earlier error the room caught. Don't say “pour one bucket into sixteen cups.” That's splitting: a sixteenth each. The real action is copying: sixteen full copies. ⭐ That's the point: redundancy, not dilution.
⭐ Su Jianlin puts it most cleanly: splitting and copying are both simple linear transformations. MLA's first idea is to replace them with general linear transformations for a more capable model.
The last section ended on a matrix. Unlock it and let the model fill it in.
That's this section's star: MLA.
⭐ Multi-head Latent Attention.
The key is the middle word: latent, hidden.
What it stores isn't what you want. It's a “latent” that can turn into what you want.
⛔⛔ From here on, keep the “warehouse / stored copy” words apart from Section 1's “box.” ⭐ Spoken, “case” and “box” blur, and the box is Section 1's central image, back unchanged in Section 7. Say “the copy in the warehouse,” never “case.”
First, the most commonly misheard point: what MLA keeps in the warehouse is no longer K and V. It's a compressed copy. Each head's K and V are expanded from it on the fly, then thrown away.
🖥 Screen: scroll to “Knob ①: smaller copies”; point at the leftmost dot (MQA).
⛔ Don't let the room guess why it isn't the winner.
Headline: “The twist: MQA 2.25× smaller than MLA, yet worse”.
The leftmost dot is MQA. It stores the least, under half of MLA's.
⭐ But it isn't the winner. The last section showed why: what it cut
was the readers, not the address book.
The x-axis measures space taken, and less space doesn't mean better.
⭐⭐ 5.2, “absorption,” is this section's foundation. Slow down.
🖥 Screen: scroll to “What ‘absorption’ is”. ① and ② only.
So: if you store a compressed package, mustn't you unpack it to use it?
No. Same computation; only the parentheses move.
Move them, and none of the 130,000 packages is touched.
⛔⛔ Next, answer “so what does it save?” It isn't what most expect. 2026-09-15, verbatim: “Does this swap save compute? What does it save? Does it save memory?” ⭐ The old script said “no unpacking,” never what that saves.
First, what does not moving cost?
The warehouse holds 130K compressed copies. For the next token, you decompress each one
into each head's key, then compare it with the query.
⭐ That's 130K decompressions. The next token needs another 130K.
You can't keep the results: that's an uncompressed copy, and the compression is wasted.
And after the move?
First, “recipe transposed times query.” One query: one token per step.
⭐ So it runs once; dot the result straight against the 130K copies.
Decompressions: 130K down to zero.
So what does it save? Be precise.
⛔ Not cache memory. Still 576 numbers per token: not a byte less, moved or not.
⭐ It saves two other things. One is compute: decompression that “grows with history”
becomes a one-off step independent of history; the longer the history, the bigger the saving.
The other is bandwidth and scratch memory: the big decompressed blob had to be stored, written out, read back.
Now it never exists.
⭐⭐ One more layer, and it's this section's foundation: the two are tied together.
Without the move, you decompress 130K times per step; nobody would pay that.
You'd fall back to storing decompressed K and V, and the compression is wasted.
⛔ So: moving the parentheses saves no memory by itself, but it's what makes “store a compressed copy” viable at all.
⭐ “Moving the parentheses” appears just twice today. This is the first.
The other is Section 7: no attention matrix built at all, a bigger win.
⛔ Don't count Section 2's. Dot product beating additive attention changed the scoring formula, not associativity. It's another thread (“what shape can it be written in?” deciding the math). ⭐ The page reframed this 2026-09-15; match it.
⛔⛔⛔ 2026-09-16: the “free part vs. the gamble” segment was cut entirely,
and its low-rank comparison figure left L200. Don't teach that criterion.
Verbatim: “This subsection is nonsense. This low-rank compression, even in your left-hand example,
isn't free. The rank it can represent goes down.”
⭐ It's right. The math was fine (a true rank-2 matrix factored at rank 2 is
lossless); using it to vouch for MLA was the error. MLA compresses KV to 512
not because it was already low-rank: it's forced: the figure's right column.
⭐⭐ MLA's genuinely free part is absorption (an exact algebraic identity),
but it saves compute and bandwidth, not cache size.
⛔ Rule: the premise an identity rests on matters more than the identity.
🖥 Screen: 5.3, scroll to “That awkward narrow lane”. Point at ①, ②, ③ top to bottom, a sentence each. ⛔ Point at the middle yellow box and move on. For after class; expanding costs three minutes.
MLA has one ugly shape: each token stores 512 plus 64.
Why the 64? Three steps.
Panel one: the purple block could move from the warehouse side to the query side.
That's the parentheses move: no package gets unpacked.
Panel two: add RoPE, and a gate goes up mid-path. The purple block can't cross.
Panel three: so split the lane in two, and put the gate only on the narrow one.
⛔⛔ Don't let “RoPE puts up a gate” slide by. 2026-09-15, verbatim: “Why would a 64-dim rotary encoding affect the matrix math? Explain what's inside.” ⭐ “It carries an index, so it blocks” lands on nothing if the room doesn't know RoPE. ⛔ Thirty seconds on how it works first.
First, what RoPE does. The model on its own has no idea of order.
RoPE gives each position an angle. The fifth token's key is rotated five degrees,
the hundredth a hundred. The query rotates too, by whatever position it's at.
⭐ The clever part: a dot product only sees the angle difference.
You're turned a hundred degrees, it's turned five; the result depends only on ninety-five:
how far apart you are.
Position becomes distance, automatically.
That's the trouble. Recall the product: query, recipe, compressed copy.
RoPE puts a rotation in the middle.
⭐ Were it one fixed matrix, you could still move things:
premultiply it into the recipe for a new recipe.
So “something in the middle” isn't itself the problem.
⛔ But the angle = your position minus its position:
ninety-five degrees to the fifth token, ninety-four to the sixth.
Every “who asks whom” pair gets a different matrix.
⭐⭐ Plainly: you want to merge recipe and rotation ahead of time, but the rotation depends on whom you ask. Until you ask, you don't know how far to turn.
⭐ So the gate isn't “something in the middle”; a fixed one could be premultiplied.
⛔ It's the index. You want to premultiply “the one,” but there is no “one.”
⭐ Last, a detail many miss: those 64 dims
are one copy shared by all 128 heads.
So it costs only 64 extra, not 128 times 64. A very cheap compromise.
⚠️ Don't skip this line. Otherwise the room assumes a per-head lane, overcounts 128×, and 576 won't add up.
⛔⛔ 2026-09-16: the page now collapses this by default.
Verbatim: “This segment doesn't help the main thread, and we can't explain it properly yet. Collapse it.”
⭐ You may skip straight to 5.4.
⚠️ If presented, the figure's subtraction (“Two steps: −0.039; MLA beats the start by only −0.029”) is this course's, not the source's. The source only says “increasing head_dims helps most, partial RoPE helps somewhat.”
⛔ Rule: your own subtraction on others' data carries a double burden: right numbers, and saying “this step of reasoning is ours.” Can't carry both? Fold it away.
🖥 Screen: (skippable) open the fold, scroll to “The ‘reluctant’ narrow lane may be why MLA wins”.
At this point, many people see that narrow lane as a compromise.
But someone ran a controlled experiment, and it says the opposite.
Two source conclusions matter here, both on screen:
One, making each head wider helps the most.
Two, position info on just a small slice of dims also helps.
⭐ Now look back at MLA's “ugly” narrow lane.
It happens to do both at once.
So it isn't a compromise. MLA stumbled onto the right thing.
⭐⭐⭐ 5.4 is the climax: the lecture's five most important minutes.
Remember Section 3? One model, two different bottlenecks: reading the prompt, emitting tokens. One level deeper shows what MLA really solves.
The emitting side is bound by the cache. Given a cache budget, the theoretical best
is to give it all to one head, K and V shared too. MHA and GQA
both rewrite into that shape, so it's their superset.
⭐ Why a superset? “One big copy shared by all” can express
“128 small slices, one per head.” Not vice versa.
Same bytes, most capacity.
The prompt-reading side is bound by compute, roughly heads times dims per head. 128 to 512 dims per head
is four times the compute. This side wants plain MHA-128.
One side wants 512, the other wants 128. A deadlock.
⭐ MLA's big move breaks it. It projects in two steps:
compress to a 512-dim vector, then expand into many 128-dim heads.
Then, with the “move the parentheses” trick, it's an MHA-128 in training and an MQA-576 when emitting tokens.
576 = the 512 compressed copy + the 64-dim narrow lane.
Only with the NoPE in Section 8 does the 64 vanish, leaving a clean 512.
One set of weights, two algorithms, each at the optimum on its own side.
Su Jianlin calls this “Prefill and Decoding meeting halfway.”
⚠️ State the cost right after, or it's a sales pitch:
But it has a very real cost, grown from its most elegant feature.
GQA's KV splits by head: eight devices, an eighth each, a natural split.
MLA's compressed copy doesn't split by head at all. Across devices, two options.
Either every device stores a full copy, giving the savings back several times over.
Or attention alone splits differently (by request, not by head).
All production today takes the second, and both scheduling and framework must change.
⭐ This is the flip side of the lecture's hidden thread: math beautiful on one device
loses its natural split across many, precisely because it “doesn't split by head.”
⭐ The same ruler. What did it save? Memory, again.
How much? Another big chunk beyond the last section.
And the price? A bit more compute, and multi-device sharding got harder.
⛔ Don't explain RoPE itself (not this section's job). Do say “it carries an index, so it blocks the parentheses move”: panels ② and ③ of “That awkward narrow lane” in 5.3. Point and say three sentences. ⛔ But don't expand the yellow box (one fixed matrix vs. a stack of matrices). That's L300.
⚠️ This section overruns. 5.1–5.3 get seven minutes together, 5.4 gets five. Squeeze the front if needed, never 5.4.
🖥 Main screen: course page Section 5 · The second saving · main figure “What ‘absorption’ is”
Section 6 · Store as much, read less (12′)
🎯 Leave one line: this road saves not one byte of HBM. And because it saves nothing, Section 8 has to exist.
For two sections we've turned one knob: make each entry smaller.
This section goes a new way. Every entry stays the same size.
What changes is how many entries each step reads.
⛔ Up front, and hold this all section: this road saves not one byte of HBM.
⭐ And because it saves nothing, Section 8 has to exist.
🖥 Screen: 6.1, “Knob ②: read only part per step”, all five masks together.
⛔ Don't move on after “together.” Say the skeleton aloud, or the panel is wasted.
Not one at a time: all five together.
The skeleton only shows side by side.
⭐ Left to right, the black cells thin out: compute everything → neighbors only → neighbors plus the first few
→ pick some → compress, then pick.
Five pictures, one sentence: this whole branch only ever answers “at this step, which cells can I skip?”
⭐ 6.2, the sliding window, is the lecture's best detective story. Worth four minutes.
⛔⛔ Before the figure, say what a sliding window is, and get the room nodding.
The page has these two lines (“a fixed few hundred,” “Sensible”);
the script didn't, so the room
heard the window collapsed before learning what it was.
⭐ Ninth case of the pattern, reversed: not “on the figure, never said”
but “on the page, missing from the script.” A detective story can't skip the starting belief.
The obvious first move is a sliding window:
each token looks back only a fixed few hundred tokens, nothing earlier.
⭐ Sensible, right? Most of what we say depends on what's nearby.
⛔ But it has a famous failure, and it fails without warning.
🖥 Screen: “Sliding window: what justifies the cut, and why it breaks”. Point at ②’s three bars.
⚠️ First: these bars are log-scaled. 5158 versus 5.40 is three orders of magnitude, not 3×. Four tokens separate a working model from total gibberish.
⚠️ Someone may notice these don't match Section 4's three numbers (29.9 / 30.2 / 31.2). One line; don't expand.
⭐ Aside: this 5.40 and Section 4's 30 come from different models and benchmarks, not comparable. Perplexity compares only within one controlled set.
⛔⛔ Cover the third bar. It's the story's only decisive evidence. The page says “newlines” three times and “position, not content” twice; the script, never. So it was only asserted. ⭐ Without bar 3, “position, not content” is a claim; with it, the experiment forces it. (Seventh case of the pattern: on the figure, never in the presenter's mouth.)
So what's magic about these four tokens?
⭐ Someone ran a decisive experiment:
replace them with meaningless newlines.
If their content mattered, the model would still crash.
The result: 5.60.
That's almost the same as keeping the original four (5.40).
⭐⭐ So only one conclusion is left:
what matters is position, not content.
Those tokens don't mean anything. They're a discard bin.
⭐ How this bug was found is the template for a whole class:
nothing looks wrong in the formula.
Only drawing the full attention matrix showed
a huge pile-up at the start.
🖥 Screen: scroll to “Solved: what broke was the option to abstain”. Stay on panel Ⓐ. ⛔ Don't teach Ⓑ and Ⓒ, but say the line below, or “add a bin” sounds obvious.
Look at the left panel. Each row has 100 votes and must cast them all, and the only candidates
are the tokens before it. So that thick arrow floods onto token 0.
⭐ It isn't special. It's just the only one everybody can reach.
⭐ The right two panels are for after class. Conclusion:
an “I abstain” cell is the right idea,
but it must be learned.
Someone fixed the cell at zero: same shape,
yet perplexity 18 → 29,000.
⛔ Two identical-looking fixes, a thousand times apart.
⭐⭐ The next passage closes Sections 1 and 2 together. Say it in one go. ⛔ It was two back-to-back callbacks, Bengio's line twice.
⭐ Remember Bengio's line in Section 2? Attention had to be “soft,”
with a fixed total amount.
“Fixed total” means nobody may abstain.
Even with nothing worth seeing, a head must put its votes somewhere.
Window past the start, bin gone, votes stranded: instant collapse.
⭐⭐ That's Section 1's line coming true a second time:
nine times out of ten, a design's side effect and its strength share one property.
“Fixed total” is why it can be trained at all, and the root of this bug.
⭐ First was the box's fixed size; this is the second. The third is in Section 7.
🖥 Screen: 6.3, scroll to “The deadlock”. Cover only panel ① (the deadlock).
⭐ One bridging line before the figure. 6.2 landed on “windows collapse; keep the first few,” and the room expects something smarter. Let them feel clever, then drop them into the deadlock. ⛔ Opening with “there's a deadlock” spoils it.
So: windows collapse; keep the first few tokens.
Next, naturally: don't just look nearby; pick what matters.
⭐ Sounds much smarter than a sliding window, right?
… But there's a deadlock here.
To pick out what matters, you first have to know what matters.
And “what matters” is the attention score itself.
But if you've already computed attention before picking, there's no point.
⭐⭐ The step you wanted to save, you've already paid for.
⚠️ Don't unpack the three ways out (seven or eight more minutes). One line; the figure is for after class. ⭐ But give the bottom band one sentence: two different teams reached the third independently.
⭐ Three ways out, all drawn. I cover the first; the rest, after class.
But note the third now: shrink the pool 4× first,
then pick among the shortened cells. The deadlock remains, in a pool four times smaller.
⭐⭐ And two different teams reached it independently.
Twice, independently, weighs more than one lab's result.
Nobody's flash of genius: the deadlock itself forces that answer.
🖥 Screen: 6.4, first “Why 2048 is enough” (one minute), then “One DSA step” (four minutes).
DeepSeek's answer: teacher and student.
Real attention is the teacher; we train a separate, very cheap student.
⭐⭐ Keep the next Q&A. It's the only explanation of “cheap.” ⛔ “A very cheap student” alone sounds like patching with a small model. It's cheap not because it's small, but because its job is easier. Then point at panel ③ of “Why 2048 is enough” (“Why it's cheap: three cuts”), the math behind it.
What does the student learn? Not how to compute accurately.
It learns only the ranking of the teacher's distribution: who ranks first.
⭐⭐ That bar is much lower, so it can be very light.
⭐ You've seen this pattern elsewhere: speculative decoding, distillation, cache predictors of all kinds. Same shape: the expensive thing needn't go away. It just has to be the teacher once.
⭐⭐⭐ “One DSA step” was drawn for the four questions from class. Go panel by panel, ① ② ③.
① One step: a very light scorer sweeps all of history,
takes the 2048 top-scoring positions, and runs full attention on just those.
⛔ Note the red box: it must sweep all of history, skipping nothing. Until it has, it doesn't know what to skip.
② Stay here a little longer. This band is drawn to true scale:
the thin orange strip is what this step reads, 1.56 percent.
All that gray is still sitting in HBM.
③ Three columns: what's saved, what isn't saved at all, what costs extra.
⭐ Look at the middle column. It says one thing: HBM, not one byte less.
The logic is airtight: to pick from history, you must keep all of it.
⛔ “Not less” is polite. It's about 10% more.
The scorer keeps its own tiny marker per token, just for scoring.
⭐ That's close to 1 GiB, seven times the 2048 picked tokens.
And every step reads all of it: until then it can't know what to pick.
⭐⭐ So every later name in this branch shrinks that cost: several layers share one scorer instead of one each.
🖥 Screen: 6.4b, scroll to “DeepSeek's four steps”. Go Ⓐ → Ⓑ → Ⓒ; stay on this figure for the whole passage.
⭐⭐⭐ Added only on 2026-09-16. Before that, NSA / CSA were marked “side branch, skip.”
⛔ Not because “we had time,” but because
these four are four steps on one road; taught apart, each loses its control group.
Above all, explaining “why DSA is the most fragmented” needs NSA's block picking beside it.
⚠️ Said in the room: “These are DeepSeek's, so everyone's watching them.”
A real audience request, not our wish.
The DSA we just saw is neither the start of this road nor the end.
⭐ Here are the three steps around it. You'll see a very clean line.
⭐ Ⓐ, NSA first, February 2025, from DeepSeek.
It's not one sparse path but three branches side by side, merged by a gate:
① Compress: distant tokens, blockwise, into digests: a coarse view of all.
② Select: pick the few most relevant blocks and run full attention on them.
③ Sliding window: keep the nearby stretch in full, for local precision.
⭐⭐ Point out the colors. In this figure, color means one thing. Orange = known only at runtime; green = fixed at compile time. ⭐ In Ⓐ: only “select” is orange. The next three steps all move it.
⭐ Colors: only the middle branch, “select,” is orange.
Compression and window are static. Only “which blocks to pick” is known at runtime.
⛔ Remember that orange branch. The next three steps all move it.
⭐⭐ Underline one word in NSA: Native.
It means sparse during training, not sparsity bolted on afterward.
So this model grows up sparse.
⛔ Remember this, because the next panel is the opposite:
NSA picks blocks, not scattered single tokens.
A block is contiguous, so it loads cleanly.
The paper put it in the title: hardware-aligned.
⭐⭐⭐ Ⓑ is the center of the figure.
The four names on one chart, two axes:
across, how hard it compresses; up and down, whether it picks at runtime.
Left to right:
NSA picks blocks, plus a compression branch.
DSA: no compression, top-k single tokens; more precise, most fragmented.
CSA first compresses four tokens into one entry, then picks:
same deadlock, a pool four times smaller.
⭐⭐⭐ Slow down. This is where the figure, and the whole section, lands.
⛔⛔⛔ A real mistake in the room, 2026-09-16. Verbatim: “That 128× compression, does it really come from the 4× one compressed another 32×? Is it two-stage?” No.
⭐ Source-checked: HCA's compressor takes this layer's raw input, groups it by 128 once, and does one gated sum: straight to 128, no 4 in between.
⛔ And CSA and HCA are two layer types, interleaved. Each compresses directly from tokens. Neither continues the other.
⭐⭐ Test: being a multiple implies neither cause nor order. Never say “compressed another 32×”: that turns a ratio into a mechanism.
⭐ Then HCA. It compresses 128 tokens into one entry, far harder than CSA.
⛔ Note: not on top of CSA. A different layer, compressing straight from raw tokens.
⭐⭐ That hard, and look: it drops to the bottom row.
It stops picking.
⭐⭐⭐ The one line this figure is here to say:
compress hard enough, and you don't need to pick.
Picking exists to see a few out of a huge pile.
But shrink the pile, and you might as well read it all.
⛔ Say the cost right away, so it doesn't sound free.
⛔ The cost is the x-axis: at 128×, resolution is gone.
So HCA can't work alone. Beside it hangs a local sliding window.
Far away, heavily blurred. Nearby, nothing missed.
⭐⭐⭐ The next two were 2026-09-16 dry-run questions. Cover them unasked. ⛔ Not “too little detail.” Both assumed a premise the room lacked: ① “the compressor and the window are two parallel branches whose results are concatenated”: drawn, never said. ② “‘compression’ here and in Section 5's MLA are different axes”: same word, adjacent, easily mixed. Test: same word, different axes? Split them where it recurs.
⭐ Your question: does compression depend on distance, or is it uniform?
⭐⭐ It is uniform. The compressor doesn't know near from far. Every 128 tokens become one entry.
Nearby stays sharp not by going easy there, but thanks to the branch beside it:
a local window keeps the nearby stretch verbatim, uncompressed.
⛔⛔ It once said “the two results are added.” Wrong; fixed. NSA's gated merge, put on V4 unchecked; MaxText shows V4 concatenates. ⭐ The test: architecture relations that sound obvious are most dangerous: old ones break on new designs.
⛔ The two parts are not computed apart and merged; that's NSA.
⭐ V4 appends the compressed blocks after the uncompressed stretch: one longer KV sequence,
one attention pass.
⭐⭐ So “the farther, the coarser” comes from joining two branches.
Compression itself has no such idea.
You saw this in Ⓐ: NSA's ① compression covers far, ③ the window near.
⭐⭐⭐ HCA is what's left of NSA once you cut out the middle branch, “select.”
⛔ Slow down. This is the section's one collision with Section 5.
⭐ The second question is worth more: do you decompress this when you use it?
Or absorb the matrices, like MLA?
⭐⭐ No decompression, but for a reason unrelated to MLA's:
they compress different axes.
⭐ MLA compresses how wide each token is: the feature axis.
Compress that, and logically you must project back up, or dot-product widths mismatch.
⭐ That's what absorption is for: prefold the up-projection into Q and the output matrix,
so physically nothing is expanded.
⭐ CSA and HCA compress how many tokens there are, the sequence axis.
128 rows become one row.
⭐⭐ And that one row is itself the K and V that take part in attention.
It isn't a compressed copy of anything. It is the real thing.
⭐⭐⭐ So there's nothing to restore.
⭐ Side by side, the shape is clear:
MLA is “it needs decompressing, but we found a way not to”;
this branch is “there's no decompression step at all.”
⚠️ Add the cost right away, or the sequence axis sounds free.
⛔ The costs split along the same axes.
The feature axis is a low-rank approximation: lossy, but every token is still there.
The sequence axis merges 128 tokens into one; their identities are gone for good.
⭐⭐ That's the real reason the sliding window must be there.
Without it, even the sentence just said is a blur, and the model loses the thread.
🖥 Screen: switch to 6.4c, “DeepSeek-V4-Pro: what one KV entry is”; go Ⓐ → Ⓑ → Ⓒ → Ⓓ.
⭐⭐⭐ Bonus. All numbers and claims: paper and official docs. ⛔ Scrapped and rewritten once. Version one read only MaxText and said “V4 is plain MQA, nothing to do with MLA,” overruling a room that was right. ⭐ The test (say it aloud): code tells you “how,” never “why.” Names, conventions: from the paper; code only cross-checks.
⭐ Look at Ⓐ. First, a thought you're probably having:
“Isn't this just MLA? Don't all the heads end up as one compressed 512 entry too?”
⭐⭐ That intuition is half right. The paper's formal name is
Shared Key-Value Multi-Query Attention: just one KV head,
128 query heads all reading the same entry.
⛔ Don't rush to the difference; let the right half land for a beat. They got half of it right; don't answer with a correction. The lecture's seventh “let them feel right first.”
⭐ Where's the difference? An analogy:
MLA stores ingredients; V4 stores the finished dish.
Ingredients need prep: “project back up into each head's K and V.”
⭐⭐ Section 5's absorption prefolds that into other matrices.
V4 has no such step.
⭐⭐ This came up in the room and will again. Cover it unasked.
⭐ So where did the 64-dim RoPE go?
⭐⭐ It wasn't moved out. It lives inside the 512.
⭐ Look at the two color bands in Ⓐ:
V3.2: a 512 latent plus a separate 64 of RoPE outside, stored apart, 576 in total.
V4: 512, period. It uses partial RoPE,
rotating only each head's last 64 channels; the first 448 carry no position.
⭐⭐ In one line: V3.2's 64 is bolted on outside; V4's 64 lives inside.
⛔ Say the cost right away, or the trade sounds free.
⛔ Since K and V are one entry, there's a catch:
V gets rotated by RoPE too, and V shouldn't carry position.
⭐ Step 4 in Ⓓ: after attention,
the output's 64 rope dims rotate back by position minus i.
⭐⭐ Only then does each entry contribute by relative distance alone.
Paper section 2.3.3, Equation 26.
⭐ So of the five steps in Ⓓ, none is “decompress.”
The query dots straight into that 512 entry, softmax, then multiplies by the same entry.
Halving that memory costs one extra elementwise rotation.
⭐⭐⭐ Forced by room follow-ups; matters more than the above: Ⓐb is the spine. ⚠️ Say plainly it's our derivation: the paper never says “V4 is an MLA variant.”
⭐ Look at Ⓐb. Where does this 512 actually come from?
⭐⭐ From Section 5's MLA, four steps derive it.
① MLA: one 512 latent, two up-projections restoring each head's K and V,
and a separate 64-dim RoPE key. Cache: 576.
② First cut: drop the two up-projections. The 512 serves directly as K and as V.
③ Second cut: fold the 64 into the 512. Cache: 512.
④ Last: compress further along the sequence, atop this 512. 4× is CSA; 128× is HCA.
⭐⭐⭐ So “it looks like MLA” is no illusion at all.
MQA in form, MLA in effect.
⛔ This arithmetic is the section's crux. Work it live; don't just state the result.
⭐ Now: could you use just one of these two axes?
Suppose V4 did not build on that narrow entry, just standard multi-head:
128 heads, 128 dims each, K and V stored separately: 32,768 numbers per token per layer.
⭐ Compress 4 to 1 and you still have 8,192.
⛔ MLA is 576. That's 14× apart.
⭐⭐ So sequence compression doesn't just “get to” stack on width; it has to.
Compress count, not width, and you can't even beat the last generation.
⭐ Asked by name: Q's path. Don't skip.
⭐ Someone will ask: what about Q? One 1536 latent, or also just one 512?
⭐⭐ Q goes through a latent, but ends fully multi-head.
The hidden state goes down to 1536, is normalized, then up into 128 heads of 512 dims.
⭐ That 1536 matches V3 and V3.2.
V4 left the query side alone; every change is on the KV side.
⛔ And Q isn't cached, so this low-rank step saves compute, not memory.
⭐ This last one pins the two axes again.
⭐ Another question: the 128 sliding-window entries, are they MQA too?
⭐⭐ Yes, and the same entry, not a separate KV set.
The official docs call it the Shared sliding-window K=V branch.
⛔ Savor this. In the window, the sequence axis isn't compressed:
each token keeps its own entry;
but the width axis is still compressed: the same 512 entry.
⭐⭐⭐ The window decides only “don't merge this,” not each entry's width.
🖥 Screen: scroll to “DeepSeek-V4-Pro: what one KV entry is”, panel Ⓐc.
⭐⭐⭐ Ends 6.4c: the section's most valuable part. ⛔ Say first: the paper doesn't answer this directly; what follows is derived. ⚠️ The identity is public (DeepSeek-V2); “V4 = MLA's absorbed form, made native” is our judgment, not the paper's.
⭐ Last question, surely on your mind:
wasn't MQA judged “not capable enough” in 2019? When did it get good again?
⭐⭐ Let me be upfront: the paper doesn't answer this directly.
I went through sections 2.3.3 and 2.3.4:
no paragraph explains why shared KV doesn't cost quality.
So what follows is a derivation, not the paper's words.
⭐ Look at the identity atop Ⓐc.
In MLA, head h's score is q_h · k_h, with k_h = up-projection × the shared 512.
⭐⭐ Regroup the parentheses: transpose the up-projection and multiply it onto q.
Now the score is each head's own 512-dim query dotted with one shared 512-dim key.
⭐⭐⭐ Look at what that is.
It is literally MQA with head_dim equal to 512.
In other words: absorbed MLA already looks exactly like this.
V4 didn't bring MQA back. It took a form MLA always had
and made it the architecture.
⛔ The next line is where this lands. Slowly.
⭐ Why did 2019's version lose quality?
Not sharing, but deleting without replacing.
Its shared entry was one head wide; queries weren't widened.
Nothing took over the lost per-head views.
⭐⭐ On V4's side:
the shared entry is four times wider; queries are 512 per head;
the output has a grouped low-rank projection; each head gets a learnable attention sink.
⭐⭐⭐ The per-head views didn't disappear. They moved.
⚠️ Add the cost, or this sounds like a free lunch.
⛔ But don't hear “zero loss.”
Merging K and V into one entry is something even MLA didn't dare.
MLA at least keeps two matrices to separate them.
Two independent sets of degrees of freedom are now one. You can't save half for free.
⭐ Who absorbs that loss? Three things:
per-head query freedom, the grouped output projection, native training.
⭐⭐ Trained this way from day one, not converted from another model.
⭐⭐⭐ The last should sound familiar: the Native in NSA's name
means the same.
A structure grown with the model and one bolted on later are different things.
⛔ Worth noting: many old “method X doesn't work” verdicts
came from testing X only bolted on.
⭐ This one ties back to 6.4's DSA and closes the loop.
⭐ One last point, back to 6.4's DSA.
The 2048 it picks are picked on top of MLA.
V3.2's config spells it out: attention_type is mla, kv_lora_rank is 512,
and the indexer's topk is 2048.
⭐⭐ The order:
MLA's full latent vectors, all stored;
the indexer stores its own small markers separately;
scores all history with them, picks 2048, builds a mask;
then attention runs on MLA's full KV, under that mask.
⭐⭐⭐ And one line in the source makes this section's test concrete:
if the sequence length is at most topk, the indexer selects everything.
At short context, DSA is plain MLA.
⭐ Not my inference: you can point to it in the code.
⛔ Fold the three wrong turns into one shape. The lecture's method lands here.
⭐⭐ Today's assumptions all failed the same way:
an architecture relationship true in one generation, carried to the next unchecked.
“Three branches should be merged by a gate”: true for NSA, false for V4.
“DeepSeek equals MLA”: true for V3, false for V4.
⛔ So on architecture, don't go on impressions. Dig into configs and source.
⭐ Four-steps figure, panel Ⓒ: why this step matters so much on TPU.
⛔ The dynamic side (DSA, CSA): picks are known only at runtime, so the mask is built on the fly,
stored dense in HBM, tying up the compute units.
⭐ The static side (HCA): fixed window, fixed ratio, so
the mask is computed at compile time on the host, packed into a bitmap, loaded straight on-chip.
No dense mask ever lives in HBM.
⭐⭐⭐ So make this line clear:
HCA didn't make dynamic sparsity faster. It got rid of the dynamic part.
⭐⭐ Closing, tie this to the lecture's recurring test.
⭐ Put the four steps together:
NSA blocks → DSA single tokens → CSA compress, pick →
HCA compress only.
⭐⭐ Each step pushes runtime decisions toward compile time.
The same shape as the test we keep seeing today:
a problem's best ending isn't being solved. It's ceasing to exist.
⚠️ Two caveats, only if asked.
① No performance numbers here: the course measured no head-to-head of the four; any number would be invented.
② CSA / HCA structure and defaults come from MaxText's public implementation (Apache-2.0),
not the paper. Cite it.
⛔ Don't say “the middle column is empty.” It's full of text. Say “not one number in that column goes down.”
⭐ Of the extra costs, the last hurts most: the 2,000-odd picked tokens are scattered all over HBM,
not one block, so loads fragment.
Saving FLOPs is easy: skip 98% of cells on paper. But with those cells scattered,
the bytes moved don't shrink, so neither does the time.
⭐ Here the ruler reads very differently:
What did you save? Not memory. Compute and bandwidth.
How much? Each step reads about 2,000 tokens: one sixty-fourth.
And the price? Not one byte of HBM saved, and the loads got more fragmented.
⛔ Remember this column: Section 8 rests entirely on it.
⛔ Skip DSA's training recipe (KL warm-up, frozen main model). “Teacher and student” is enough.
⚠️ Say “it doesn't save HBM” out loud. It's why this section exists and the fuse for Section 8. Without it, Section 8 becomes an unmotivated “then use both.”
🖥 Main screen: course page Section 6 · Store as much, read less · main figure “One DSA step”
Section 7 · Back to the start (10′, the climax)
🎯 Leave one line: the KV cache is a kind of state, not the other way around.
Three roads so far: one changed entry size, one changed entries read per step.
This one goes furthest: it stores nothing.
It ends at Section 1's box.
⭐⭐⭐ The test lands here for the third and last time. Planted in Section 1, paid off once in Section 6. ⛔ Don't say it early. Wait until 7.3, “The notepad fills,” is done, then point back.
🖥 Screen: 7.1, scroll to “Knob ③'s basis: same product, parentheses moved”. Point at the shape of the lower cell, ② Drop softmax.
In 2020 someone made a provocative claim:
what really holds attention back is the softmax.
Why: softmax normalizes over every position, so the big “who attends to whom” table can't be avoided;
it must be built in full, and that table is the quadratic term.
Drop it, and three matrix products remain. And matrix products let you move the parentheses.
⭐ Look at that lower cell. The sequence-length axis is gone.
That alone backs this branch's “cost ignores length.”
Our second parenthesis move today. There are only two, and this one pays most.
Last time (Section 5), no compressed entry in the warehouse got unpacked. This time, the big table is never built.
🖥 Screen: 7.2, scroll to “Linear attention = a fixed-size notepad”.
⛔ No side-by-side windows: at 1748 px the notepad figure won't fit half a screen, and it breaks one-cell-at-a-time.
⭐ Scroll there and back: go to Section 1's “RNN: squeeze ‘the past’ into a fixed-size box”, point at the box for one sentence, return.
So what does the move leave? A fixed-size state. linear attention is an RNN at heart. Its state is just a matrix, not a vector.
⭐⭐⭐ Add one step here: how the “adding” actually works.
(Asked live, 2026-09-15.) ⛔ The lecture says “fold it in” and “add a stroke”
but never says what gets added; all three ideas in 7.3
(d orthogonal directions / smearing / the delta rule) rest on it.
⚠️ Worth two minutes.
First, a misreading: you don't add the token's vector straight onto the notepad.
⛔ Shapes don't match: the notepad is a d-by-d square; a token's vector, a line of length d.
The real move has two steps.
⭐ Step one: split it into a pair. Two learned matrices project the token into
a key k and a content v, each a line of length d.
⭐ Step two: weld them into a card. Take their outer product:
two length-d lines make a d-by-d square.
What goes onto the notepad is the card, not the parts. Now the shapes match.
⭐⭐ Once it's all one blob, how can you still pull out a single card?
The key: reading is linear too. Querying with a key = the notepad times that key.
The notepad is a sum of cards, so that product splits apart on its own:
each card's content, weighted by how similar its key is to yours.
⛔ So what comes out always has two parts.
Part one: the card you wanted, whose key matches yours exactly.
Part two: a little leakage from every other card, scaled by how similar their keys are to yours.
⭐⭐⭐ Part two is crosstalk. It's the “smearing” coming up shortly.
If all keys are mutually orthogonal, crosstalk is exactly zero: a clean read.
But a d-dimensional space has at most d mutually orthogonal directions.
⭐ That's the source of “the notepad only holds d entries.” Not a metaphor: you can check it.
⭐⭐ The rule below matters more than the passage. Say it.
⭐ Remember one thing: adding works only because reading is linear.
Write and read come as a pair.
Softmax reads nonlinearly: its denominator sums over every position.
So it can't squash history by adding; it must keep every entry as is.
⭐⭐ Now 7.1 clicks: dropping softmax wasn't only about parentheses.
Only once it's gone does adding become retrievable, for the first time.
⛔⛔ Someone will ask: “Won't constant adding blow up? Doesn't it need a norm?”
It exposes something this lecture glossed over. Own it.
(Asked live, 2026-09-15: 7.1's “drop the softmax”
is a simplification. ⛔ Never pass one off as truth.)
⭐ Cite: Eqs. (5) and (7) of arXiv 2006.16236.
Does constant adding blow up?
⭐ First, I owe you a correction: “drop the softmax”
was a simplification.
Look at the original paper's equation. The denominator is there.
It computes this entry's similarity divided by the sum of all similarities.
That's exactly the shape of softmax.
⭐⭐ So what got dropped isn't the normalization. It's the exp.
Softmax is two things: exp, then divide by the sum.
Linear attention drops the first half, the nonlinear shell
that locked the multiplication order. The second half stays.
⭐ The denominator is a running sum too, so the same parenthesis trick computes it, almost free.
⭐⭐⭐ Blow up? The numerator grows, and so does the denominator.
What comes out is a weighted average, not a growing sum.
That's the original norm, there from day one.
⭐ If asked about elu plus one: the paper says it keeps similarity positive; a denominator crossing zero would blow up. Not relu: gradient 0 on the negative half, so it won't train.
⭐⭐ But this branch's later mainstream doesn't rely on that denominator. It took a more aggressive route, one that fills all of 7.3.
The idea: each step, shrink the whole notepad a little, then add the new card.
“Keep adding” becomes keep adding while constantly forgetting.
⭐ The sum becomes a geometric series, not arithmetic: it converges by itself.
⭐ So your instinct was half right: something must keep it in check.
But the main force isn't the outer norm. It's the forgetting itself.
There's usually an outer norm too, but as a safety net, not the main reason.
⭐⭐ The unspoken question: why isn't it called an RNN?
⛔ Answer it unprompted; doubts distract.
(Asked at the 2026-09-15 trial run; script and page then said nothing.)
⭐ Source: arXiv 2006.16236, ICML 2020. Both points: the abstract's first sentence.
Some of you are wondering: so why isn't it called an RNN?
⭐ The paper that named it is titled “Transformers are RNNs.”
The authors knew from day one, and put it in the title.
“Linear attention” is the method in the subtitle.
So where's the “linear”? One word covers two things, and one causes the other.
⭐ First: the dot product becomes linear. The abstract says
a linear dot-product of kernel feature maps.
⛔ Put it in plain words at once.
Softmax is a nonlinear shell wrapped around the similarity.
Take it off and the similarity is just q and k, each lightly transformed,
then dotted.
⭐ No shell, so you don't need a whole row before computing.
⭐ Second: so the complexity becomes linear. A linear dot product lets
associativity apply: quadratic drops to O(N).
⛔ The first linear causes the second; not two parallel facts.
⭐⭐ So why not rename it an RNN? Because a name records where something came from, not what it is.
It was carved out of attention, not grown from RNNs.
In 2020, RNNs had lost to the Transformer just three years earlier. Back then, that name was a liability.
⭐ Call your new method an RNN and you've buried it yourself.
⭐ Then audience: they had to win over Transformer users,
so they named it in that crowd's terms.
“A kind of attention” keeps them reading. “A kind of RNN” makes them scroll past.
⭐⭐ Close Section 1's naming loop: put both names side by side. (1.1 explained the “R”, ⛔ deliberately without comparing.)
⭐ Last: side by side, the names show one more layer.
The “R” in RNN says what it looks like: an edge looping back to itself. A form.
The “linear” in linear attention says how it's computed:
the dot product is linear, so it can be reordered. A mathematical property.
⭐⭐ One is named for its form, the other for a property.
So they sound unrelated, yet describe the same thing.
Not a coincidence, but a flaw in naming itself:
a name records only the facet that mattered most when it was coined.
Now set this notepad beside Section 1's box. Same shape.
Fixed size, passed step by step, cost independent of length.
One difference: back then a vector, now a matrix.
And that difference is the biggest change in thirty years: the notepad got bigger, because now it can.
🖥 Screen: 7.3, scroll to “Read · erase · write: each model lacks one”.
⛔⛔ Let the room guess wrong first; stating “it only adds, never erases” outright falls flat. Third use of this pattern (6.2 sliding windows, 6.3 picking what matters): put up the reasonable wrong answer, then knock it down.
The notepad fills. Guess: written on and on, how does a fixed-size notepad fail?
⭐ First instinct: old content slowly fades, right?
Like reused tape: the oldest is blurriest.
⛔ No. Basic linear attention doesn't fail that way:
it only adds, never erases.
Each token read adds one stroke. Ten thousand tokens, ten thousand strokes.
No old stroke has gone anywhere. Nothing fades.
⭐⭐ So it doesn't fade. It smears.
Picture a whiteboard written on, never erased. By note one hundred,
all the ink is still there.
But you can't read a single note anymore.
⭐ Read this line out (in the figure, unreadable from the seats). ⛔ Name the speaker.
Neuroscientist David Eagleman has a line in his book “Livewired” that this branch quotes again and again:
The enemy of memory is not time; it's other memories.
So Section 1's “fades over distance” returns, changed.
It isn't washed out by time. It's crowded out by later memories.
Every new name in this branch answers one question: how do you erase?
⭐⭐⭐ The everyday version goes here: why storing more means more crowding.
⛔ After Eagleman, before the delta rule:
it catches “crowded out, not faded” and sets up the delta rule.
⚠️ Keep the closing boundary: the shape, not a claim about the brain.
There's an everyday version of this.
⭐ Adults often feel their memory is worse than a child's. Maybe not worse.
More likely, it's just full.
A child's head has few distractors. Something new arrives into empty space,
easy to tell apart and remember on its own.
Adults are different. You first have to find an empty spot, or a sensible place to slot it in,
so it doesn't clash with what you know. And that itself is hard.
⭐⭐ See it? It's exactly the “at most d directions in d dimensions” from earlier.
Empty notepad: a new entry goes anywhere, no clash. Full one:
it squeezes in beside others, smudging what it touches.
“Finding an empty spot” means finding a direction nobody has claimed yet.
⭐⭐⭐ And this picture also explains the trick coming up next.
“A sensible place to slot it in, without clashing” means, on the notepad:
look up what this key already holds, clear the old stroke, write the new one.
⭐ Tidy up before new things stick: that's the delta rule.
⚠️ One boundary: we borrow only the shape,
interference, not decay. The brain is another field's question.
What carries it is checkable: d dimensions hold at most d non-interfering directions.
⭐⭐ Before we get to erasing, one comparison. A residual network also adds. Why doesn't it smear?
A residual block computes output equals input plus F of input. Also adding in place.
Yet a hundred-plus layers don't smear. Why?
⛔ One difference, and it's decisive: it reads before it adds.
⭐ Basic linear attention adds blind: the card added at step t
comes from the current token alone; it never looks at the notepad, so can't tell when it overwrites.
⭐ A residual block reads, then adds. F's input is the current representation.
It can see what's already there, so it writes what's missing, not what it thinks should be there.
⭐ The 2015 ResNet paper by Kaiming He's team puts it precisely: learning residual functions with reference to the layer inputs. Learning against a reference. That “with reference to” is the whole reason it doesn't smear.
⭐⭐⭐ And the delta rule, coming up, does exactly this for linear attention.
It uses the key to pull out the old content at that spot first, then corrects by the gap.
In one line: the delta rule gives “add” the residual's “read first, then write.”
⚠️ If asked “why don't residuals pile up over many layers?”: a fixed tens to a hundred-odd layers, not tens of thousands of accumulations, and LayerNorm renormalizes between layers. ⛔ Skip the gradient story (the identity path keeps gradients from decaying, the paper's main motive).
⭐ Last, skip memorizing the family tree.
Meet a new name in this branch? Ask one thing: how do you erase?
How much, which parts, every step or not, learned or not.
The answer always lands in one of those four cells.
⭐⭐⭐ Now point back to the test, third and last time. The room has just watched the notepad fill.
⭐ Notice? We're back at today's first test.
This notepad is fixed-size. Upside: per-step traffic doesn't grow with length,
the one strength of Section 1's box, picked back up after a five-section detour.
Downside: it can't hold everything. It fills up.
⭐⭐ Same property again. First the box, then the discard bin,
now a third time. And this time we chose it back on purpose.
🖥 Screen: 7.4, scroll to “Squeezing parallelism out of a serial state”. One minute.
The state is back, and Section 1's “one step at a time” pain came with it. This branch stands only because it solved that again. Same formula, no symbol changed. Only the order of computation changed: not token by token, but batch by batch.
⭐⭐⭐ Added 2026-09-16, after a room question:
“How exactly does chunking work? How do GDN and KDA get parallel speedups on hardware?”
⛔ It gave only metaphors (service queue, handoff note),
nothing on the in-chunk math, the secret sauce.
⭐ Rule: metaphors keep people listening; mechanism makes them understand.
⭐ So, batch by batch: what happens inside a chunk?
⭐⭐ One line: the same sum, split in two, “past chunks” and “this chunk.”
Part one: outside the chunk.
All earlier chunks are already squeezed into one handoff note; each query here
just asks that note.
However many tokens came before, same cost.
⭐ The shape: a C-by-d Q times a d-by-d S. One matmul.
Part two: inside the chunk.
What's left: earlier tokens in this chunk. Unavoidable,
but only C of them.
⭐⭐ So it shrinks to a small C-by-C full attention.
Another matmul.
At the chunk's end, update the note once: add this chunk's K-transpose times V.
⭐ Again, a matmul.
⭐⭐⭐ The next line closes Section 1's loop.
⭐⭐ Count them: three operations, all matmuls.
And that's exactly the shape Tensor Cores and the MXU want.
⭐⭐⭐ So why is this change fast? The answer is back in Section 1.
The RNN's hardware problem wasn't compute: every step is
a matmul with one dimension equal to 1. Thousands of units, one fed at a time.
⭐ Chunkwise, in one line: it turns that 1 into C.
Serial steps drop from L to L over C, and every step becomes one fat matmul.
⭐ Quote the paper here.
⭐ Kimi's paper: “inter-block recurrent and intra-block parallel … to maximize matrix multiplication throughput, thereby fully utilizing the computational potential of Tensor Cores.”
⭐⭐⭐ GDN and KDA need one more step: what the “in-house kernels” really solve.
⛔ But that was the most basic version of linear attention.
⭐ It parallelizes inside a chunk because each token just adds one card.
Nobody looks at what anyone else wrote.
⛔⛔ The delta rule is different. Remember? It reads first, then writes.
What token r erases changes what token r plus one reads.
So the chunk can't just run in parallel.
⭐⭐ The fix is beautiful.
Fold the in-chunk chain into a C-by-C lower triangle; invert it once.
Then the whole chunk goes in one shot.
⭐ Paper: forward substitution, as in Gaussian elimination: cheap.
⭐⭐⭐ So “we built our own kernel”: built what, exactly?
Kimi's paper implements a “specialized” DPLR variant.
That's the point: the general triangular solve costs more;
the delta-rule case is cheaper.
⛔ One last line to take home:
“Linear attention is faster” comes with a premise.
It's fast because someone untangled the in-chunk dependencies for it.
Without that kernel, on hardware it's the same long, thin chain.
⚠️ Don't say “order doesn't matter within a batch.” In-batch order still holds; the bookkeeping just fills one small table in one pass instead of queuing. ⛔ Otherwise it sounds approximate, clashing with the figure's “equals the recurrence in math.”
⭐⭐⭐ 7.5 is the high point. Go slow; read the original words.
🖥 Screen: 7.5, scroll to “Full circle: KV cache is a kind of state”. Walk Ⓐ → Ⓑ → Ⓒ one cell at a time; stay on this figure. ⛔ Don't scroll away: the force is three cells in one figure.
⭐⭐⭐ This figure arrived only on 2026-09-15; before that, the high point had no figure.
⛔ The room had to flip “who contains whom” by ear.
⭐ Ⓑ is why the figure exists: the nesting flips. By ear that takes a mental turn; drawn, it's one glance.
⭐ Walk the figure in three lines (per cell: its line, then the quote):
⭐ Ⓐ This axis is the map of our first six sections.
Left end: kept verbatim. Right end: squeezed into one fixed-size thing.
The four points: full attention far left, GQA moves a bit, MLA a bit more,
and linear attention jumps straight to the right end.
⛔ Now the leftmost point, circled orange.
Sliding window and DSA sit at this stop too. They haven't moved an inch.
⭐ Because they cut how many entries each step reads, not how many are stored.
Section 6's “not one stored byte saved” is visible right here.
⭐⭐ Last, the bottom blocks. The left three keep growing; the right three stay one size.
That's all this section bought, and cost.
Equal blocks: what won't fit, won't fit.
⚠️ Ⓐ's axis has no scale. Don't say “compressed N times.” GQA cuts heads, MLA compresses dimensions, linear attention swaps the data structure: no common scalar. It shows only left, right, and who stays put.
⭐ Before reading, set up two things. The room doesn't know Mamba or the speaker (in round 4 the Su Jianlin quote fell flat for this).
⭐ Quick introduction: this branch's best-known public name is Mamba,
from 2023. It's the same notepad, the same “erase, then write.”
It just came from signal processing, in different terms, so it sounds like another school.
What follows is from Albert Gu, one of Mamba's authors, at Carnegie Mellon.
His framing is worth hearing in full: every autoregressive model
is some kind of state model. It holds a state and evolves it with every token emitted.
Only the state's form differs.
Attention's state caches every past element exactly.
This branch's state compresses all of history into one fixed-size thing.
⭐⭐⭐ Before the next line, move the laser to the two circles in Ⓑ. ⛔ Circles first, words second, and the line clicks. Then point at the green circle on the right again.
Then an interesting observation: many call this branch's state “a kind of KV cache.”
It's more accurate the other way around: the KV cache is a kind of state.
(On this axis it's the least compressed kind. That half is mine, not his words.)
⭐ The two circles: same two words; only who contains whom swapped.
The left view names the whole class after its least compressed member.
⭐⭐ Flip it and you gain one thing: siblings, not parent and child.
⭐ Now Section 1's box looks different. It's no obsolete old method. It's the prototype of this whole family.
His metaphor beats any metric: A Transformer is like a database.
Each new observation is filed as an important record, pulled out verbatim when needed.
This branch is like a brain: finite, always on, processing as it listens.
And we humans are the second kind.
None of us can memorize a phone book, let alone recite it.
We're terrible at exact recall and lookup.
⭐ Yet intelligence seems to show up anyway.
So “whose long context is better?” is the wrong question. Which is better, my own memory or my research notes? They're just different.
⭐ One last pass with the ruler. This branch answers most cleanly.
What did it save? All three at once: memory, bandwidth, compute.
How much? Cost ignores length entirely.
And the price? Exact retrieval. The notepad is only so big; no phone book fits.
⭐⭐ So, three roads walked: three roads, three bones too hard to chew.
And the whole industry's answer was remarkably uniform.
⚠️ Read the three quotes in 7.5; don't paraphrase. They're the literary backbone.
⚠️ Down to five minutes? Cut 7.3 and 7.4, keep 7.2 and 7.5.
🖥 Main screen: course page Section 7 · Back to the start · main figure “Linear attention = a fixed-size notepad”
Section 8 · Nobody wins, so use both (8′)
🎯 Leave one line: a hybrid isn't about saving money. It brings its own benefits.
⛔⛔ Voice the "just a compromise" thought for them, then knock it down.
⭐ The recurring shape, fourth time (6.2 sliding window / 6.3 pick what matters / 7.3 how the notepad breaks): reasonable wrong answer first.
⛔ Unnamed, "use both" sounds like all three failed, so patch them together;
we argue the opposite.
The last section left us stuck: three roads, three bones too hard to chew.
⭐ The most boring answer: use them all, several roads at once.
⚠️ Sounds like a compromise, right? "None works, so cobble them together."
⛔ This section is about why it isn't.
🖥 Screen: scroll to “Hybrids: a few juniors per senior”. Point at panel ①, left.
If cheap layers only saved money, all-expensive should score best. But the measurements say otherwise.
First-hand, from Albert Gu again.
He writes that swapping some, even most, of a pure Transformer's layers for state layers
improves efficiency and quality at once. The best ratio is roughly 3:1 to 10:1,
verified independently by dozens of research groups.
⭐ "Dozens" is his word; read out his names.
⭐ He names H3, Jamba, Zamba, Samba,
and a long list after them.
Labs starting from different places landed in the same range.
That outweighs any one lab saying "our ratio is best."
⭐⭐ Then stress "at once": it carries the sentence.
⭐ The sentence's most valuable word: "at once."
Not "a bit cheaper, a bit worse." Both sides move the right way.
⛔ Add two caveats at once, or it's a sales pitch: ① "better" claims more than the evidence shows; ② his range differs from the figure on screen.
⭐⭐⭐ Gu himself stitches Sections 7 and 8 together, in the same post. ⛔ 7.5's closing question ("my memory or my notes: which is better?") is answered here. Unlinked, Section 8 is just new material; linked, it answers Section 7.
⭐ Remember the question that ended the last section?
My own memory, or my research notes: which is better?
The answer then was: they're just different.
⭐⭐ And in the same post, Gu's very next line:
just as human intelligence is augmented by scratch pads and external references,
language models improve when you mix state layers and attention layers.
⛔ So Section 7's question was never either/or.
⭐⭐⭐ The answer is: you get both.
A brain, plus notes you can flip through any time. That's a hybrid architecture.
⚠️ Two caveats up front.
First, he means modeling quality, not cost.
The savings come free; they aren't the reason.
Second, discount "better": in our hardest controlled comparison
the whole spread is three percent. Strictly, the evidence says "no worse, and much cheaper."
"Better" is a direction, not settled.
⭐ One more thing, on screen shortly.
He says 3:1 to 10:1. The figure's band, 3:1 to 6:1, is what the ablation suggests.
The dots show where labs stop: none past 7:1.
No contradiction: he means the modeling optimum; the dots are where real labs stop today.
⛔ Everything built sits in the left half of his range.
🖥 Screen: 8.2, scroll to “‘The far past can be skimmed’: the rigorous version”. One-sentence conclusion only.
So why not go all-cheap? There's a law you can compute.
⭐ The figure's metaphor: the "book" is the whole input context;
the "notepad" is what the model carries.
Section 3 said "the far past can be skimmed." True, but incomplete:
not "the far past doesn't matter." The thin correlation between a book's two halves
keeps growing as the book thickens, just slowly. Slow is not zero.
So the notepad must grow with the book. Fix its size, and some book won't fit.
⭐⭐ That's why you can't go all-cheap. It's not that the engineering isn't good enough. The law doesn't allow it.
⭐⭐⭐ The next line lands 8.1 and 8.2 together; then 8.3. It sets up 8.4: the first time "ratio" carries weight.
Now the picture is clear: you can't go all-expensive, and you can't go all-cheap.
⭐⭐ The ratio is forced on you, not tuned.
🖥 Screen: 8.3, scroll to “Under ‘3:1 is best’ sits a five-row table”.
⛔⛔ Wrong answer first: the shape's fifth instance
(after 6.2 / 6.3 / 7.3 / 8.1).
⭐ It looks so reasonable.
How this table was made: a fixed sixteen layers, same compute budget,
the ratio swept end to end, each trained and compared.
⭐ Guess the ranking. If hybrids win: all-expensive below the middle,
all-cheap at the bottom. Right?
⛔ First surprise: pure full attention is not the worst.
⭐⭐ And give the denominator: only five ratios,
and it ranks fourth.
⛔ Fourth of five isn't mid-pack; it's second to last. The worst is the other extreme.
⭐ So the conclusion is stronger. Not "the more cheap layers the better,"
not "the fewer the safer." Both ends are bad; the good part is in the middle.
The benefit of mixing isn't linear.
⭐⭐⭐ For the second point, do the arithmetic live. ⛔ "Few ratios tile evenly" is an assertion; thirty seconds of division makes it checkable.
Second surprise: this table can't answer "why 3 and not 4."
⭐ Why not? Do the math with me.
A ratio of r to 1 means r cheap layers per expensive one,
so one cycle is r plus 1 layers. To tile evenly across sixteen layers,
r plus 1 has to divide sixteen.
⭐ So r plus 1 can only be 1, 2, 4, 8, or 16.
That means r can only be 0, 1, 3, 7, or 15.
⛔ Exactly the table's five rows.
Four to one needs cycles of five; five to one, cycles of six. Neither divides sixteen,
so neither ever made the table.
⭐⭐ To be clear: not tried and found worse. Never tried.
⭐ One detail shows how thin the margin is (in the figure; read it out).
Also: 3:1 and 7:1 are identical on training.
Both 9.23, to the last decimal.
It's validation that separates them, by 0.05.
⛔ So "3:1 is best" was squeezed out of a very thin gap.
⭐⭐⭐ One line to take with you:
whenever you read an ablation table, first ask which points it swept, and which it skipped.
Handed a nice optimum, it's easy to assume it was picked from every possibility.
⛔ Usually it wasn't. It was picked from the few that fit.
🖥 Screen: 8.4 is 87′ only. In the 60′ version scroll straight past, but say the line below. ⛔ Or it vanishes silently.
⭐ I'll skip a section for time. The conclusion:
as labs go deeper, what's conserved is the ratio, not the number of expensive layers.
So the deeper the model, the more expensive layers you pay for.
A hybrid saves a fixed percentage. Deeper isn't cheaper.
The full section is on the course page.
⭐ Full version, 87′ only (about 45 seconds): open with a concrete pair of numbers, not "two schools of thought."
⭐⭐⭐ State the stakes first, or it sounds like hair-splitting. ⛔ The two answers give opposite cost curves.
Sounds like a word game. It isn't.
It decides how costs grow as you scale.
⭐ If the count is conserved, great: however deep, a few expensive layers do.
The deeper, the better the deal.
⛔ If the ratio is conserved: the deeper the model, the more expensive layers you pay for,
always saving a fixed percentage.
⭐ This question isn't for arguing; it's for counting.
We pulled fourteen models' configs and counted.
⭐⭐ The cleanest refutation is Kimi itself: it published configs for two generations.
Same lab, same design, just deeper. Hard to argue with.
Kimi Linear: twenty-seven layers, seven expensive. Its successor K3: ninety-three, twenty-four expensive.
Depth more than tripled, and so did the expensive part.
⭐ So what's conserved isn't the count, it's the ratio.
Earlier: "you don't need many expensive layers." Fix one word:
not few in number, but low in share. Every lab lands between a quarter and a tenth.
⭐⭐ That's not derived; it's counted.
Fourteen configs: depth versus expensive-layer count is nearly a straight line.
⛔ Today's last test: whether a number is a hyperparameter shows in how much it spreads.
The count runs three to twenty-four, eightfold; the ratio, just over threefold.
What spreads was chosen. What doesn't is pinned down.
⭐⭐ You can use this test anywhere.
Want to know if a parameter was tuned or pinned down by physics?
Don't ask the designer. Look at how much it spreads.
🖥 Screen: 8.5, scroll to the “NoPE” figure. Point at ① versus ②.
⭐ Last, a free bonus. Remember Section 5's awkward sixty-four-dim narrow lane?
It had to exist because no layer in the model knows order by itself.
But state layers run step by step, so order is built into how they run.
So the cheap layers can take over position entirely.
In a hybrid, that narrow lane can go.
⛔⛔ Raise the objection for them; it was asked live right here: "But surely that MLA layer still needs RoPE?"
⭐ Test: if a conclusion invites an obvious objection, it belongs in the script.
Unasked, it drowns out the rest.
⭐ Cite: Kimi Linear tech report, arXiv 2510.26692,
a whole subsection, “No Position Encoding (NoPE) for MLA Layers.”
You're probably thinking: surely that full-attention layer still needs position?
⛔ No. None at all. Not my inference:
the paper has a whole subsection, “No Position Encoding (NoPE) for MLA Layers,” saying verbatim
"we apply NoPE to all full attention (MLA) layers."
⭐ So how does it know position? Position didn't vanish; its delivery route changed.
Before, every layer stamped its own: RoPE rotates that layer's keys by position,
so every layer is blocked by that rotation.
⭐⭐ Now lower layers mix it in; it arrives via the residual stream.
The vector the MLA layer reads already carries position; nothing left to do.
⭐⭐⭐ So who encodes "how far," exactly? Decay.
The notepad multiplies by a decay factor every step.
So how many decays an entry has seen tells how far back it is.
The decay itself is the ruler.
⭐ In the paper's words, KDA takes over "positional information and recency bias,"
as "the primary position-aware operator."
⭐ Recall our division of labor: "order" belongs to the causal mask;
"distance" used to belong to RoPE.
In a hybrid, "distance" moves from RoPE to the decay,
and RoPE loses its reason to exist entirely.
⚠️ Don't turn this into "hybrid always means NoPE": it's Kimi Linear's choice; not every hybrid does it. ⛔ Precondition: a layer type with built-in order, placed below the full-attention layers, so position is mixed in first.
⭐ A rare kind of good news in this lecture: a problem not solved, but gone, because the structure changed.
🖥 Screen: 8.6, scroll to the model table.
⛔ Don't fit it on one screen: forty-four rows; a screen holds about ten.
Show the header, scroll down once steadily, move on.
That's all the terms; this table puts them back in the real world. Every row cites a source; most check against public configs. Browse it after class.
⚠️ 8.4, “Is the ratio conserved, or the full-attention count?”, can be skipped whole if short on time. Before skipping, say: "Both camps exist; it's on the course page."
🖥 Main screen: course page Section 8 · Nobody wins, so use both · main figure “Hybrids: a few juniors per senior”
Section 9 · On the machine + closing (10′)
🎯 Leave one line: hardware picks the formula, and in the end even hyperparameters.
🖥 Screen: 9.1, scroll to “Cost: swap ↓↓↓ for bytes moved per decode step”. Point at the gray in panel ②.
⭐ That ends the lecture's "ideas" part.
Last section: the machine has the say.
⛔⛔ First say what the figure measures, and what gray and red are.
⛔ In the 2026-09-15 trial run I called gray "computing the weights, running the FFN." Wrong:
it's bytes one step moves out of HBM, not compute.
⭐ Test: read the legend first; don't invent meanings from colors.
"Gray = the other chores" sounds so natural nobody objects.
Multiples are relative. Milliseconds are absolute.
⛔ This figure measures bytes moved out of HBM per step.
Not computed. Moved.
Two colors.
⭐ Gray is the weights, shared by all; every step moves all of them.
DeepSeek-V3 activates thirty-seven billion parameters per token; in FP8, about 34.5 GiB.
⭐ Red is your own KV cache.
⭐⭐ Now the five bars. The gray is identical; all five reach one line.
Why? The weights don't care how long you talk.
⛔ The difference is all in the red.
⭐⭐⭐ Point back: the gray bar is Section 1's sentence. Section 1, on why RNNs are slow: every step moves all the weights out of HBM. That line never left. ⭐ The course comes full circle.
⭐ Remember Section 1's line?
Every step moves all the weights out of HBM.
Not a word of it has gone stale. It's this gray bar.
⭐⭐ "The KV cache is the bottleneck": we've said it since Section 3.
Here it finally gets drawn.
Not a claim: a red bar you can see. In one MHA step,
93 percent of the bytes are one user's KV.
⚠️ Point at the "diminishing returns" line too: sparsity on top of MLA is only 1.21× faster. The red is nearly gone, so the bottleneck is now all gray. ⭐ Another reason for Section 8's "use both": push one road harder and the returns run out.
🖥 Screen: 9.2, scroll to “On TPU: see which two things clash”. ⭐ Never skip for an infra audience.
The catch: most of these clever ideas grew up on GPUs. A machine's strengths are set at the factory; new machine, new premises.
⛔ After the figure, scroll on to the text. The next three points (286× · every block doubled · "could change training dynamics completely") are in the text below: the lecture's final blow. Go slowly.
In 2025 someone really ported NSA to TPU and kept a worklog.
Verdict: dynamic sparsity is hard at every layer of this stack.
⭐ Unpack "every layer": three, each deeper than the last.
Layer one, the compiler. It dislikes runtime-only variables and branches,
and "which blocks to pick" is known only at runtime.
Layer two, the kernel framework. It walks the grid cell by cell, in order,
but top-K's blocks jump around.
Layer three, the hardware itself. This one hurts most.
Fetching scattered blocks fell to a unit built for arithmetic,
not the hardware built to move data.
⭐ That unit applies one operation to hundreds of numbers at once.
Fetching items one by one, one slot of hundreds works; the rest sit idle.
⭐ Just making the indexing compiler-friendly gave 286×, algorithm untouched.
⛔ Sixth instance of the shape: seem trivial, then drop. (After 6.2 / 6.3 / 7.3 / 8.1 / 8.3.) ⭐ Wrong answer: "just a constant." So small-looking the reversal lands.
But what's worth remembering isn't the 286. It's the next point.
⭐ Fair warning: it sounds like a trivial tweak. Just a constant.
To fit the chip's matrix unit, he had to double the paper's blocks across the board: selection blocks from 64 to 128, compression blocks from 32 to 64, and the blocks picked per step from 16 down to 8. And he wrote: this could change training dynamics completely. Bigger blocks drag in irrelevant tokens, so fine work like needle-in-a-haystack may suffer.
⛔ Today's hidden thread is now complete.
Watch it creep toward "how smart the model is":
Section 2: hardware picked the formula.
Section 4: a hyperparameter (eight groups: one machine holds eight accelerators).
Section 5: a veto of beautiful math (the compressed copy isn't per-head, so no sharding).
Section 6: how to select (whole blocks, not scattered picks).
Now: the training dynamics,
via a number that affects how smart the model is.
⭐⭐ Look at its direction: it climbs all the way up.
At first it only picked the formula.
By the end, it reaches the model's IQ.
🖥 Screen: 9.5, scroll to “Closing: how was the cover's 512× bought?” Ask first, then explain.
I asked you to guess a number. Answer: of that 500-plus-fold gain,
the big stretch up front came from "making every token cheaper."
Only a small sliver shows at the end.
⛔ Be honest: don't call that sliver "the hardware's share." It's a remainder, never checked separately.
⭐ Precisely: this sliver is what's left after dividing out the first two.
Not checked separately.
It still mixes in at least two things we skipped:
one, storing KV at lower precision (every lab's default now; alone nearly halves it again);
two, prefix reuse.
⭐⭐ Pull them out, and the "buying more hardware" share only shrinks.
Today's conclusion gets stronger, not weaker.
⚠️ Always state the basis: "⛔ These two bars are separate bills. Don't add them. The top: what one token takes up; the bottom: what one step reads."
🖥 Screen: 9.6, scroll back to Section 0's “Attention chronicle”. Show it whole; leave it up to close.
⭐⭐⭐ The only figure shown twice, on purpose.
At the start, territory: eight real years, no name familiar.
Now, a report card: same figure, now readable.
⛔ It lands only if the opening line, "remember how this feels right now," was said.
Otherwise it's just an old figure again.
⭐⭐ Name each band; don't just say "you understand those branches now." "You've changed" makes them find the evidence; naming the bands lays it out, as the page's 9.6 does.
Back to the opening timeline.
Two hours ago it was noise. Read it top down.
The top gray band is the box, plus its 2014 add-on.
The blue band is Sections 4 and 5: make each copy smaller.
The orange band is Section 6: read only part each step.
The purple band is Section 7: back to a fixed-size notepad.
The bottom three company rows are Section 8: same knobs, each lab its own setting.
⭐ Not one pixel of that figure has changed. You have.
One last thing.
The 2017 cut was a deal: buy full parallelism with quadratic compute.
And ever since, the whole industry has been buying that deal back, without giving the parallelism up.
Fewer heads, a latent copy, partial reads: these buy back the quadratic's constant.
Linear attention goes after the quadratic itself, at the cost of going sequential.
So chunking returns to win parallelism back. A full circle.
Today was one story: for thirty years, people moved within one triangle: remember, compute fast, afford it. No corner can be dropped.
⭐⭐⭐ The next line is the real ending. Say it, stop, add nothing. What came full circle is Section 1's box.
⭐ But what really came full circle isn't the figure.
It's Section 1's box.
After all the buying back, what's on the table differs from the 1990 box in one way only:
it's a matrix, and it can erase.
That one difference took thirty years.
⚠️ Don't read the 9.7 list on stage. Just say: "A list at the end of the course page marks what each of today's sections still owes."
🖥 Main screen: course page Section 9 · On the machine · main figure “Closing: how was the cover's 512× bought?”
🚧 Where these notes are still thin
⭐ An honest record: this isn't finished.
- [ ] No pre-class or post-class questions. The L300 notes have a sixty-second one; this set doesn't. One fits at the end of Section 6: "Does sparse attention save memory?"
- [ ] The timings are estimates, not yet checked against one full delivery. After the next full run, update it to real times. ⛔ A stale schedule is worse than none: at minute forty you're only halfway.
- [ ] 8.4 still lacks a good telling. “Is the ratio conserved, or the full-attention count?” is a great question, hard to say in thirty seconds. The script says "skip it if short of time": avoidance, no fix.
📌 Settle one, strike it here, write the conclusion into its section. ⛔ Don't answer these on the course page: it shows conclusions, not our agonizing.