Let's build GPT: from scratch, in code, spelled out.

Channel:Andrej Karpathy · Length:1:56:20 · Published:2023-01-17 · Views:8348985

Watch on YouTube

Summary

Andrej Karpathy builds a GPT from scratch in code, following Attention Is All You Need and GPT-2/3, in about two hours using PyTorch. He connects it to ChatGPT and assumes familiarity with his earlier makemore videos.

Why HN recommends it

It is the go-to recommendation for learning how transformers and LLMs actually work, praised for its clarity and Karpathy's teaching.

the best video I have seen on the topic — HN, 2023

Machine learning & deep learning · LLMs & generative AI

Subtitles (excerpt)

hi everyone so by now you have probably heard of chat GPT it has taken the world and AI Community by storm and it is a system that allows you to interact with an AI and give it text based tasks so for example we can ask chat GPT to write us a small Hau about how important it is that people understand Ai and then they can use it to improve the world and make it more prosperous so when we run this AI knowledge brings prosperity for all to see Embrace its power okay not bad and so you could see that chpt went from left to right and generated all these words SE sort of sequentially now I asked it already the exact same prompt a little bit earlier and it generated a slightly different outcome ai's power to grow ignorance holds us back learn Prosperity weights so uh pretty good in both cases and slightly different so you can see that chat GPT is a probabilistic system and for any one prompt it can give us multiple answers sort of uh replying to it now this is just one example of a problem people have come up with many many examples and there are entire websites that index interactions with chpt and so many of them are quite humorous explain HTML to me like I'm a dog uh write release notes for chess 2 write a note about Elon

On Hacker News

+1, Andrey is an amazing educator! I'd also recommend his [VIDEO] and checking out his github repos. MinGPT, for example, implements a small gpt model that's compatible with HF API, whereas more modern nanoGPT shows how to use newer features such as flash attention. The quality of every video, every blog post is just so high. — villedespommes, 2024-01
Andrej Karpathy, the director of AI at tesla, published an AMAZING video about how the current models are built which I think everyone in the tech space should watch. It is intuitive, easy to follow and quite frankly the best video I have seen on the topic. Here is the link: [VIDEO] If you conclude, after watching this, that this is all intelligence is - namely cascading optimization and data str… — wesammikhail, 2023-05
Very impressive. Recently I watched this really amazing lecture on building GPT from scratch from Karpathy, I was blown away: [VIDEO] — durdn, 2023-02
Great compilation, would be great to see Vision Transformer (ViT) included. Andrej Karpathy's GPT video is a must have companion for this [VIDEO] I was going nuts trying to grok Key, Query,Position and Value until Andrej broke it down for me. — yonz, 2023-01
Attention is all you need and GPT-2 were well known at that point. Many might doubt whether this approach leads to "general" intelligence – depends on the definition. BTW, Karpathy has a nice video tutorial about building an LLM: [VIDEO] — yreg, 2024-12

Related videos