Attention in transformers, step-by-step | Deep Learning Chapter 6

Channel:3Blue1Brown · Length:26:09 · Published:2024-04-07 · Views:4966758

Watch on YouTube

Summary

3Blue1Brown's visual walkthrough of the attention mechanism inside transformers, covering self-attention, multiple heads and cross-attention. It is chapter 6 of his deep learning series.

Why HN recommends it

HN treats it as the go-to visual explanation of attention, recommended to people who find text-based transformer explainers hard to follow.

what bade me finally understand the transformer architecture [0] and attention [1], are 3Blue1Brown's videos. — HN, 2024

Machine learning & deep learning · LLMs & generative AI

Subtitles (excerpt)

In the last chapter, you and I started to step through the internal workings of a transformer. This is one of the key pieces of technology inside large language models, and a lot of other tools in the modern wave of AI. It first hit the scene in a now-famous 2017 paper called Attention is All You Need, and in this chapter you and I will dig into what this attention mechanism is, visualizing how it processes data. As a quick recap, here's the important context I want you to have in mind. The goal of the model that you and I are studying is to take in a piece of text and predict what word comes next. The input text is broken up into little pieces that we call tokens, and these are very often words or pieces of words, but just to make the examples in this video easier for you and me to think about, let's simplify by pretending that tokens are always just words. The first step in a transformer is to associate each token with a high-dimensional vector, what we call its embedding. The most important idea I want you to have in mind is how directions in this high-dimensional space of all possible embeddings can correspond with semantic meaning. In the last chapter we saw an example for how direction can correspond to gender,

On Hacker News

Because it doesn't work like how you think at all. You're still thinking it works like Chain of Thought. It doesn't. And the difference is key! It works by introducing probabilistic noise, and exploring N paths fully (each with noise) in parallel (all compressed). It's reasoning at a much, much smaller (probabilistic) level than running everything through the expensive large model (deterministic)… — onlyrealcuzzo, 2026-05
This is good, but what bade me finally understand the transformer architecture [0] and attention [1], are 3Blue1Brown's videos. 0. [LINK] 1. [VIDEO] — tomashm, 2024-07
If you want the actual maths instead of handwaving, I recommend: [LINK] For something gentler, 3Blue1Brown: [VIDEO] (this is part of a series) — wren6991, 2026-06
> when an LLM tells us that the sun rises in the east, it's not because it truly understands astronomy or the solar system. Rather, it has seen the phrase "sun rises in the east" repeated so many times in its training data that this becomes the highest probability answer. It's less about understanding and more about pattern recognition. This is not too different from most human person's experienc… — superkuh, 2024-10
There are many others that are better. 1/ The Annotated Transformer Attention is All You Need [LINK] 2/ Transformers from Scratch [LINK] 3/ Andrej Karpathy has really good series of intros: [LINK] Let's build GPT: from scratch, in code, spelled out. [LINK] GPT with Andrej Karpathy: Part 1 [LINK] 4/ 3Blue1Brown: But what is a GPT? Visual intro to transformers | Chapter 5, Deep Learning [LINK] Atte… — MAXPOOL, 2024-07

Related videos