The spelled-out intro to language modeling: building makemore

Channel:Andrej Karpathy · Length:1:57:45 · Published:2022-09-07 · Views:1336231

Watch on YouTube

Summary

Andrej Karpathy builds a bigram character-level language model from scratch, introducing torch.Tensor and the framework of training, sampling and loss evaluation. It is the first step toward a GPT-style Transformer in his makemore series.

Why HN recommends it

HN recommends it as a long but worthwhile way to demystify how language models work technically.

It's long (2h), but well worth the time if anyone is curious about the technical workings of LLMs. — HN, 2024

Machine learning & deep learning · LLMs & generative AI

Subtitles (excerpt)

Hi everyone. Hope you're well. And uh next up what I'd like to do is I'd like to build out make more. Like micrograd before it, make more is a repository that I have on my GitHub webpage. Uh you can look at it. Uh but just like with micrograd, I'm going to build it out step-by-step and I'm going to spell everything out. So, we're going to build it out slowly and together. Now, what is make more? Make more, uh as the name suggests, uh makes more of things that you give it. So, here's an example. names.txt is an example data set to make more. And when you look at names.txt, you'll find that it's a very large data set of names. So, here's lots of different types of names. In fact, I believe there are 32,000 names that I've sort of found randomly on a government website. And if you train make more on this data set, it will learn to make more of things like this. Um and in particular in this case, that will mean more things that sound name-like, but are actually unique names. And maybe if you have a baby and you're trying to assign name, maybe you're looking for a cool new sounding unique name, make more might help you. So, here are some example generations from the neural network once we train it on

On Hacker News

You've made something interesting. Many people have already answered this here [1], so there's no point in me explaining it again. Dropping some pointers that will be helpful for you: - [LINK] - Highly recommend this one [VIDEO] - [LINK] - [LINK] 1: ex: [LINK] — georgehill, 2024-04
I resource I found invaluable in demystifying LLM's was Andrej Karpathy's Youtube series. In particular, his "Makemore" video built a basic LLM in the same vein as OP's. It's long (2h), but well worth the time if anyone is curious about the technical workings of LLMs. [VIDEO] — islewis, 2024-02
For reference, this project is from Eureka Labs, which is the new education startup from Andrej Karpathy. * [LINK] It appears to be a generalized version of the bi-gram project, which he links to here: * [VIDEO] It's cool to see him putting in even more effort on the edu work. It's a real gift he has, and was visible all the way back in his grad student projects. — caprock, 2024-07
This is beautiful. I'd actually been going down this same rabbit hole with sqlite, I hadn't gotten far enough to bring a neural net into it. I'd been inspired by the makemore lecture series[0]. At the 1hr mark or so, he switches from counting to using a nn, which is about as far as I've gotten. Breaking it down into a relational model is actually a really great exercise. [0] [VIDEO] — chrsig, 2024-02

Related videos