The spelled-out intro to language modeling: building makemore
Channel:Andrej Karpathy · Length:1:57:45 · Published:2022-09-07 · Views:1336231
Summary
Andrej Karpathy builds a bigram character-level language model from scratch, introducing torch.Tensor and the framework of training, sampling and loss evaluation. It is the first step toward a GPT-style Transformer in his makemore series.
Why HN recommends it
HN recommends it as a long but worthwhile way to demystify how language models work technically.
It's long (2h), but well worth the time if anyone is curious about the technical workings of LLMs. — HN, 2024
Machine learning & deep learning · LLMs & generative AI
Subtitles (excerpt)
Hi everyone. Hope you're well. And uh next up what I'd like to do is I'd like to build out make more. Like micrograd before it, make more is a repository that I have on my GitHub webpage. Uh you can look at it. Uh but just like with micrograd, I'm going to build it out step-by-step and I'm going to spell everything out. So, we're going to build it out slowly and together. Now, what is make more? Make more, uh as the name suggests, uh makes more of things that you give it. So, here's an example. names.txt is an example data set to make more. And when you look at names.txt, you'll find that it's a very large data set of names. So, here's lots of different types of names. In fact, I believe there are 32,000 names that I've sort of found randomly on a government website. And if you train make more on this data set, it will learn to make more of things like this. Um and in particular in this case, that will mean more things that sound name-like, but are actually unique names. And maybe if you have a baby and you're trying to assign name, maybe you're looking for a cool new sounding unique name, make more might help you. So, here are some example generations from the neural network once we train it on
On Hacker News
You've made something interesting. Many people have already answered this here [1], so there's no point in me explaining it again. Dropping some pointers that will be helpful for you: - [LINK] - Highly recommend this one [VIDEO] - [LINK] - [LINK] 1: ex: [LINK] — georgehill, 2024-04
I resource I found invaluable in demystifying LLM's was Andrej Karpathy's Youtube series. In particular, his "Makemore" video built a basic LLM in the same vein as OP's. It's long (2h), but well worth the time if anyone is curious about the technical workings of LLMs. [VIDEO] — islewis, 2024-02
For reference, this project is from Eureka Labs, which is the new education startup from Andrej Karpathy. * [LINK] It appears to be a generalized version of the bi-gram project, which he links to here: * [VIDEO] It's cool to see him putting in even more effort on the edu work. It's a real gift he has, and was visible all the way back in his grad student projects. — caprock, 2024-07
This is beautiful. I'd actually been going down this same rabbit hole with sqlite, I hadn't gotten far enough to bring a neural net into it. I'd been inspired by the makemore lecture series[0]. At the 1hr mark or so, he switches from counting to using a nn, which is about as far as I've gotten. Breaking it down into a relational model is actually a really great exercise. [0] [VIDEO] — chrsig, 2024-02
Related videos
- The spelled-out intro to neural networks and backpropagation: building micrograd · Andrej Karpathy · 2:25:52
- But what is a neural network? | Deep learning chapter 1 · 3Blue1Brown · 18:40
- AlphaGo - The Movie | Full award-winning documentary · Google DeepMind · 1:30:28
- Transformers, the tech behind LLMs | Deep Learning Chapter 5 · 3Blue1Brown · 27:14
- USENIX Security '18-Q: Why Do Keynote Speakers Keep Suggesting That Improving Security Is Possible? · USENIX · 51:22
- Lecture 1 | Machine Learning (Stanford) · Stanford · 1:08:40
- MarI/O - Machine Learning for Video Games · SethBling · 5:57
- Deep Dive into LLMs like ChatGPT · Andrej Karpathy · 3:31:23
- Computer program that learns to play classic NES games · suckerpinch · 16:17
- RL Course by David Silver - Lecture 1: Introduction to Reinforcement Learning · Google DeepMind · 1:28:12
- Let's build GPT: from scratch, in code, spelled out. · Andrej Karpathy · 1:56:20
- Bay Area Vision Meeting: Unsupervised Feature Learning and Deep Learning · Google TechTalks · 48:10
- A Hackers' Guide to Language Models · Jeremy Howard · 1:31:13
- Andrej Karpathy - AI for Full-Self Driving at Tesla · Matroid · 30:08
- [1hr Talk] Intro to Large Language Models · Andrej Karpathy · 59:48
- Joscha Bach: Artificial Consciousness and the Nature of Reality | Lex Fridman Podcast #101 · Lex Fridman · 3:00:18