Attention in transformers, step-by-step | Deep Learning Chapter 6
Channel:3Blue1Brown · Length:26:09 · Published:2024-04-07 · Views:4966758
Summary
3Blue1Brown's visual walkthrough of the attention mechanism inside transformers, covering self-attention, multiple heads and cross-attention. It is chapter 6 of his deep learning series.
Why HN recommends it
HN treats it as the go-to visual explanation of attention, recommended to people who find text-based transformer explainers hard to follow.
what bade me finally understand the transformer architecture [0] and attention [1], are 3Blue1Brown's videos. — HN, 2024
Machine learning & deep learning · LLMs & generative AI
Subtitles (excerpt)
In the last chapter, you and I started to step through the internal workings of a transformer. This is one of the key pieces of technology inside large language models, and a lot of other tools in the modern wave of AI. It first hit the scene in a now-famous 2017 paper called Attention is All You Need, and in this chapter you and I will dig into what this attention mechanism is, visualizing how it processes data. As a quick recap, here's the important context I want you to have in mind. The goal of the model that you and I are studying is to take in a piece of text and predict what word comes next. The input text is broken up into little pieces that we call tokens, and these are very often words or pieces of words, but just to make the examples in this video easier for you and me to think about, let's simplify by pretending that tokens are always just words. The first step in a transformer is to associate each token with a high-dimensional vector, what we call its embedding. The most important idea I want you to have in mind is how directions in this high-dimensional space of all possible embeddings can correspond with semantic meaning. In the last chapter we saw an example for how direction can correspond to gender,
On Hacker News
Because it doesn't work like how you think at all. You're still thinking it works like Chain of Thought. It doesn't. And the difference is key! It works by introducing probabilistic noise, and exploring N paths fully (each with noise) in parallel (all compressed). It's reasoning at a much, much smaller (probabilistic) level than running everything through the expensive large model (deterministic)… — onlyrealcuzzo, 2026-05
This is good, but what bade me finally understand the transformer architecture [0] and attention [1], are 3Blue1Brown's videos. 0. [LINK] 1. [VIDEO] — tomashm, 2024-07
If you want the actual maths instead of handwaving, I recommend: [LINK] For something gentler, 3Blue1Brown: [VIDEO] (this is part of a series) — wren6991, 2026-06
> when an LLM tells us that the sun rises in the east, it's not because it truly understands astronomy or the solar system. Rather, it has seen the phrase "sun rises in the east" repeated so many times in its training data that this becomes the highest probability answer. It's less about understanding and more about pattern recognition. This is not too different from most human person's experienc… — superkuh, 2024-10
There are many others that are better. 1/ The Annotated Transformer Attention is All You Need [LINK] 2/ Transformers from Scratch [LINK] 3/ Andrej Karpathy has really good series of intros: [LINK] Let's build GPT: from scratch, in code, spelled out. [LINK] GPT with Andrej Karpathy: Part 1 [LINK] 4/ 3Blue1Brown: But what is a GPT? Visual intro to transformers | Chapter 5, Deep Learning [LINK] Atte… — MAXPOOL, 2024-07
Related videos
- Transformers, the tech behind LLMs | Deep Learning Chapter 5 · 3Blue1Brown · 27:14
- Are We There Yet - Rich Hickey · Zhang Jian · 1:10:05
- But what is a neural network? | Deep learning chapter 1 · 3Blue1Brown · 18:40
- But what is the Fourier Transform? A visual introduction. · 3Blue1Brown · 19:41
- Gradient descent, how neural networks learn | Deep Learning Chapter 2 · 3Blue1Brown · 20:33
- How does Computer Memory Work? 💻🛠 · Branch Education · 35:32
- AT&T Archives: Similiarities of Wave Behavior (Bonus Edition) · AT&T Tech Channel · 28:02
- The spelled-out intro to neural networks and backpropagation: building micrograd · Andrej Karpathy · 2:25:52
- AlphaGo - The Movie | Full award-winning documentary · Google DeepMind · 1:30:28
- USENIX Security '18-Q: Why Do Keynote Speakers Keep Suggesting That Improving Security Is Possible? · USENIX · 51:22
- Lecture 1 | Machine Learning (Stanford) · Stanford · 1:08:40
- MarI/O - Machine Learning for Video Games · SethBling · 5:57
- Deep Dive into LLMs like ChatGPT · Andrej Karpathy · 3:31:23
- Computer program that learns to play classic NES games · suckerpinch · 16:17
- RL Course by David Silver - Lecture 1: Introduction to Reinforcement Learning · Google DeepMind · 1:28:12
- Let's build GPT: from scratch, in code, spelled out. · Andrej Karpathy · 1:56:20