Transformers, the tech behind LLMs | Deep Learning Chapter 5

Channel:3Blue1Brown · Length:27:14 · Published:2024-04-01 · Views:11724096

Watch on YouTube

Summary

3Blue1Brown's visual explanation of how transformers and GPT-style language models work, tracing data through tokens, embeddings and attention.

Why HN recommends it

HN treats it as the go-to visual introduction to transformers, and many say it finally made attention and embeddings click.

what bade me finally understand the transformer architecture [0] and attention [1], are 3Blue1Brown's videos. — HN, 2024

Machine learning & deep learning · LLMs & generative AI

Subtitles (excerpt)

The initials GPT stand for Generative Pretrained Transformer. So that first word is straightforward enough, these are bots that generate new text. Pretrained refers to how the model went through a process of learning from a massive amount of data, and the prefix insinuates that there's more room to fine-tune it on specific tasks with additional training. But the last word, that's the real key piece. A transformer is a specific kind of neural network, a machine learning model, and it's the core invention underlying the current boom in AI. What I want to do with this video and the following chapters is go through a visually-driven explanation for what actually happens inside a transformer. We're going to follow the data that flows through it and go step by step. There are many different kinds of models that you can build using transformers. Some models take in audio and produce a transcript. This sentence comes from a model going the other way around, producing synthetic speech just from text. All those tools that took the world by storm in 2022 like DALL-E and Midjourney that take in a text description and produce an image are based on transformers. Even if I can't quite get it to understand what a pi creature is supposed to be,

On Hacker News

3 blue 1 brown has a great visual introduction to transformers, the heart of LLMs. It's chapter 5. Start at chapter 1 if you want more background on neural nets and backprop. [VIDEO] — robrenaud, 2026-06
I really recommend watching this section of this video. Embeddings do encode plenty of "human knowledge" into the vector values and their relations to each other. [VIDEO] This even opens up a more data-based approach to linguistics, where it is also heavily used. — gf000, 2026-01
This video explains the direction-encodes-trait topic very well IMO: [VIDEO] It’s the first in a series of three that I can very highly recommend. > there's no way that there's a 1-to-1 correspondence between concepts and dimensions. I don’t know about that! Once you go very high dimensional, there is a lot of direction vectors that are almost perfectly perpendicular to each other (meaning they c… — manmal, 2025-05
This is good, but what bade me finally understand the transformer architecture [0] and attention [1], are 3Blue1Brown's videos. 0. [VIDEO] 1. [LINK] — tomashm, 2024-07
3Blue1Brown has a fantastic video on GPT that I'd highly recommend called "But what is a GPT? Visual intro to transformers" [1]. You'll need to have some background in mathematical reasoning, but not too much. He does a great job of drawing pictures to show the mathematical transformations that happen under the hood. [1] [VIDEO] — Wilduck, 2024-05

Related videos