[1hr Talk] Intro to Large Language Models
Channel:Andrej Karpathy · Length:59:48 · Published:2023-11-23 · Views:4396545
Summary
Andrej Karpathy gives a one-hour general-audience introduction to large language models, covering what they are, how they are trained, comparisons to operating systems, and security challenges such as jailbreaks.
Why HN recommends it
HN recommends it as the dense, approachable starting point for understanding LLMs and often cites it for specific claims about scaling and training.
Start with his short intro to LLMs, only 1 hour long but is dense with valuable stuff — HN, 2024
Machine learning & deep learning · LLMs & generative AI
Subtitles (excerpt)
hi everyone so recently I gave a 30-minute talk on large language models just kind of like an intro talk um unfortunately that talk was not recorded but a lot of people came to me after the talk and they told me that uh they really liked the talk so I would just I thought I would just re-record it and basically put it up on YouTube so here we go the busy person's intro to large language models director Scott okay so let's begin first of all what is a large language model really well a large language model is just two files right um there will be two files in this hypothetical directory so for example working with a specific example of the Llama 270b model this is a large language model released by meta Ai and this is basically the Llama series of language models the second iteration of it and this is the 70 billion parameter model of uh of this series so there's multiple models uh belonging to the Llama 2 Series uh 7 billion um 13 billion 34 billion and 70 billion is the biggest one now many people like this model specifically because it is probably today the most powerful open weights model so basically the weights and the architecture and a paper was all released by meta so anyone can work with this model
On Hacker News
All of Andrej Karpathy's stuff on YouTube is absolutely top tier. He's a rare example of a brilliant contributor in his field who is an equally brilliant educator. Start with his short intro to LLMs, only 1 hour long but is dense with valuable stuff: [VIDEO] — mindwok, 2024-06
Short answer is No. I highly recommended watching Andrej Karpathy's Intro to LLMs talk, particularly the section on System 1 vs System 2 thinking. Long story short, what you are describing, using more processing to prepare a better response, is something that is an area of interest, but is not currently part of ChatGPT (or any other LLM that I am aware of). See: [VIDEO] — FlyingAvatar, 2024-02
Andrej Karpathy's "Zero to Hero" series on YouTube is the ultimate guide to building LLMs. Extremely information-dense but as complete as it gets: [LINK] Also, an amazing high-level overview of LLMs, including extensive discussion about attack vectors, that he published a couple days ago: [VIDEO] — shrimpx, 2023-11
The weights are just numbers. I don't what technical background you have in other areas of computing, but I think that this is a good, short introduction that doesn't assume too much: [LINK] To quote part of it, Training a model can be thought of as tuning the dials on a really big machine. The way that a language model behaves is entirely determined by these many different continuous values, usu… — philipkglass, 2026-06
Look into the training process - [VIDEO] The goal of the initial pretraining phase is to make it good at predicting the next word. The rest of the training process is aimed at making it (1) helpful and (2) as correct as possible. I think some people oversimplify things by calling LLMs "next token predictors" and they leave out the tuning towards helpfulness and correctness. — sk11001, 2024-04
Related videos
- Obsidian for Beginners: Start HERE — How to Use the Obsidian App for Notes · Linking Your Thinking with Nick Milo · 12:27
- Deep Dive into LLMs like ChatGPT · Andrej Karpathy · 3:31:23
- State of GPT | BRK216HFS · Microsoft Developer · 42:40
- A Hackers' Guide to Language Models · Jeremy Howard · 1:31:13
- Gimbal UFO - A New Analysis · Mick West · 21:25
- The spelled-out intro to neural networks and backpropagation: building micrograd · Andrej Karpathy · 2:25:52
- But what is a neural network? | Deep learning chapter 1 · 3Blue1Brown · 18:40
- AlphaGo - The Movie | Full award-winning documentary · Google DeepMind · 1:30:28
- Transformers, the tech behind LLMs | Deep Learning Chapter 5 · 3Blue1Brown · 27:14
- USENIX Security '18-Q: Why Do Keynote Speakers Keep Suggesting That Improving Security Is Possible? · USENIX · 51:22
- Lecture 1 | Machine Learning (Stanford) · Stanford · 1:08:40
- MarI/O - Machine Learning for Video Games · SethBling · 5:57
- Computer program that learns to play classic NES games · suckerpinch · 16:17
- RL Course by David Silver - Lecture 1: Introduction to Reinforcement Learning · Google DeepMind · 1:28:12
- Let's build GPT: from scratch, in code, spelled out. · Andrej Karpathy · 1:56:20
- Bay Area Vision Meeting: Unsupervised Feature Learning and Deep Learning · Google TechTalks · 48:10