Vaswani et al. · 2017
The Transformer. It threw out twenty years of sequential processing and everything you now call AI is built on it.
What the lessons say
The Paper That Killed 20 Years of AI Orthodoxy in 12 Hours
Here's why this paper matters: it killed RNNs overnight by proving you don't need sequential processing to understand language.
The core insight is simple. Old models read sentences word-by-word, like ht depends on ht-1, which blocks parallelization. The Transformer ditches that entirely, using only attention to connect every word to every other word directly, regardless of distance. Look at Figure 1: it's encoder-decoder stacks, six layers each, but every sublayer wraps in a residual connection plus LayerNorm—that's x plus Sublayer(x), normalized. This lets them stack deep without gradients vanishing. Self-attention also solves the "distant dependency" problem that plagued ConvS2S and ByteNet, where relating far-apart words required more computation steps.
Next, you'll want to dig into section 3.2, multi-head attention—that's the real engine making this work.
They Deleted the Thing Everyone Thought Was Required
Here's why this paper reshaped every AI model you use today: it killed recurrence entirely.
Before this, translation models processed words one at a time, in order—like reading a sentence with your eyes closed except for one word. That meant you couldn't parallelize training, and long sequences ate memory alive.
The Transformer's move: replace that sequential loop with attention mechanisms alone. Specifically, scaled dot-product attention and multi-head attention—which let the model look at every word in a sentence simultaneously and weigh how relevant each one is to every other one, regardless of distance. No recurrence, no convolutions.
Result? 28.4 BLEU on English-to-German, beating previous best ensembles by 2 points, while training way faster.
Next, I'll show you exactly how multi-head attention splits that lookup into parallel perspectives.
The Footnote Nobody Reads That Rewrote AI History
Here's why that footnote matters: it tells you this paper had no single "lead author"—it's the rare case where credit is explicitly randomized because the work was truly collaborative.
Look at how it's structured: Jakob proposed ditching RNNs entirely. Noam invented scaled dot-product and multi-head attention—the actual mechanics you'll implement. Niki and Llion built and tested the codebase. Lukasz and Aidan created tensor2tensor, which became the backbone for training these models fast. That's the real story—six people solving different pieces of the same puzzle simultaneously, not one genius idea.
When you read the technical sections later, you'll recognize whose contribution you're studying—Noam's attention formulas, or Jakob's core architectural bet against recurrence.
The Paper From 2017 That's Secretly Still Being Edited in 2023
Here's why this paper matters: it's the reason ChatGPT, GPT-4, and basically every modern AI language model exists. So let's break down how to actually read it.
First, notice the abstract's structure—they ditch recurrence and convolutions entirely, betting everything on attention mechanisms alone. That's the radical move. Second, watch how they prove it: two translation benchmarks, WMT 2014 English-German and English-French, beating prior best models by over 2 BLEU points while training faster. Third, check the author contributions section—it's oddly detailed, showing how Ashish and Illia built the first prototype, while Noam invented the actual scaled dot-product attention mechanism you've heard about.
Next, you'll want to dig into section 3, where they actually define the architecture itself.
The Silent Killer Hiding Inside Every AI Model's Math
Here's why this matters: if you don't scale attention scores properly, your model's gradients basically vanish and training stalls silently.
So here's the fix Vaswani's team used. You take your query and key vectors, multiply them together with QK^T, but here's the key step—divide by the square root of dk before the softmax. Why? Because when dk gets large, those dot products blow up in magnitude, shoving softmax into flat regions with tiny gradients. Scaling keeps things stable.
Then, instead of doing this once, they split Q, K, V into multiple heads—say 8—each with smaller dimensions, run attention in parallel, and concatenate results through WO. Same compute cost, way more representational power.
Next time you read the formula, you'll know exactly why that square root is there.
