Capabilities · Intro
Transformers and Attention
This site set itself one rule before writing its first line: never redo, worse, what already exists in superb form. This page is the first to apply it.
The Transformer still needed explaining. It is the architecture behind nearly everything discussed here: ChatGPT, Claude, Gemini, image generators, and almost every system this site dissects story after story.
In September 2023, the FT’s visual storytelling team published “Generative AI exists because of the transformer”[1]. You scroll, and the architecture unfolds in order: words become tokens, tokens become vectors, the vectors are weighed, kneaded, compared and, through the magic of mathematical genius, meaning appears. Large Language Models can be born. Nearly three years on, it remains the finest way in we know. It is free to read, and worth every one of your minutes.
──────[ the FT story ]──────
“Generative AI exists because of the transformer”
Financial Times · visual storytelling · September 2023 · free to read
If you want to go back to the source of the source, the scientific paper that introduced the Transformers is called “Attention Is All You Need.” It dates from 2017, it is only 8 short pages, and it changed the world forever.
──────[ attention is all you need · 2017 ]──────
June 2017. Eight Google researchers publish “Attention Is All You Need”[2], the paper that replaces word-by-word reading with a single principle: let every word look at every other word, and learn which ones matter. The diagram above replays that gaze, on the canonical example popularized alongside the paper[3]. Almost all of modern AI descends from those eight pages.
The day someone does better than the FT, this page will be redone. Until then, it stays short on purpose: that is its way of being honest. And to understand why making these Transformers bigger made them so capable, the natural next step is our story on scaling laws.
Sources & further reading
- 1. Financial Times (Visual Storytelling), « Generative AI exists because of the transformer », septembre 2023
- 2. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, « Attention Is All You Need », NeurIPS 2017 (arXiv:1706.03762)
- 3. J. Uszkoreit, « Transformer: A Novel Neural Network Architecture for Language Understanding », Google Research Blog, août 2017