Let's build GPT: from scratch, in code, spelled out.
Channel:Andrej Karpathy · Length:1:56:20 · Published:2023-01-17 · Views:8348985
Summary
Andrej Karpathy builds a GPT from scratch in code, following Attention Is All You Need and GPT-2/3, in about two hours using PyTorch. He connects it to ChatGPT and assumes familiarity with his earlier makemore videos.
Why HN recommends it
It is the go-to recommendation for learning how transformers and LLMs actually work, praised for its clarity and Karpathy's teaching.
the best video I have seen on the topic — HN, 2023
Machine learning & deep learning · LLMs & generative AI
Subtitles (excerpt)
hi everyone so by now you have probably heard of chat GPT it has taken the world and AI Community by storm and it is a system that allows you to interact with an AI and give it text based tasks so for example we can ask chat GPT to write us a small Hau about how important it is that people understand Ai and then they can use it to improve the world and make it more prosperous so when we run this AI knowledge brings prosperity for all to see Embrace its power okay not bad and so you could see that chpt went from left to right and generated all these words SE sort of sequentially now I asked it already the exact same prompt a little bit earlier and it generated a slightly different outcome ai's power to grow ignorance holds us back learn Prosperity weights so uh pretty good in both cases and slightly different so you can see that chat GPT is a probabilistic system and for any one prompt it can give us multiple answers sort of uh replying to it now this is just one example of a problem people have come up with many many examples and there are entire websites that index interactions with chpt and so many of them are quite humorous explain HTML to me like I'm a dog uh write release notes for chess 2 write a note about Elon
On Hacker News
+1, Andrey is an amazing educator! I'd also recommend his [VIDEO] and checking out his github repos. MinGPT, for example, implements a small gpt model that's compatible with HF API, whereas more modern nanoGPT shows how to use newer features such as flash attention. The quality of every video, every blog post is just so high. — villedespommes, 2024-01
Andrej Karpathy, the director of AI at tesla, published an AMAZING video about how the current models are built which I think everyone in the tech space should watch. It is intuitive, easy to follow and quite frankly the best video I have seen on the topic. Here is the link: [VIDEO] If you conclude, after watching this, that this is all intelligence is - namely cascading optimization and data str… — wesammikhail, 2023-05
Very impressive. Recently I watched this really amazing lecture on building GPT from scratch from Karpathy, I was blown away: [VIDEO] — durdn, 2023-02
Great compilation, would be great to see Vision Transformer (ViT) included. Andrej Karpathy's GPT video is a must have companion for this [VIDEO] I was going nuts trying to grok Key, Query,Position and Value until Andrej broke it down for me. — yonz, 2023-01
Attention is all you need and GPT-2 were well known at that point. Many might doubt whether this approach leads to "general" intelligence – depends on the definition. BTW, Karpathy has a nice video tutorial about building an LLM: [VIDEO] — yreg, 2024-12
Related videos
- Civilizations at the End of Time: Iron Stars · Isaac Arthur · 43:56
- A Million Dollars vs A Billion Dollars, Visualized: A Road Trip · Tom Scott · 1:19:08
- Why Heat Pumps are Immensely Important Right Now · Technology Connections · 21:03
- Traffic Has a Perfect Solution. Humans Are the Problem. · CGP Grey · 4:30
- Tom Scott vs Irving Finkel: The Royal Game of Ur | PLAYTHROUGH | International Tabletop Day 2017 · The British Museum · 25:33
- Did We Just Change Animation Forever? · Corridor Crew · 23:02
- ANIME ROCK, PAPER, SCISSORS · Corridor Digital · 7:21
- AI Language Models & Transformers - Computerphile · Computerphile · 20:39
- Lecture 6: Version Control (git) (2020) · Missing Semester · 1:24:59
- The spelled-out intro to neural networks and backpropagation: building micrograd · Andrej Karpathy · 2:25:52
- But what is a neural network? | Deep learning chapter 1 · 3Blue1Brown · 18:40
- AlphaGo - The Movie | Full award-winning documentary · Google DeepMind · 1:30:28
- Transformers, the tech behind LLMs | Deep Learning Chapter 5 · 3Blue1Brown · 27:14
- USENIX Security '18-Q: Why Do Keynote Speakers Keep Suggesting That Improving Security Is Possible? · USENIX · 51:22
- Lecture 1 | Machine Learning (Stanford) · Stanford · 1:08:40
- MarI/O - Machine Learning for Video Games · SethBling · 5:57