Large Language Models from scratch
Channel:Graphics in 5 Minutes · Length:8:25 · Published:2022-07-17 · Views:375793
Summary
A short cartoon explaining how language models like ChatGPT work, from autocomplete and conditional probabilities to neural networks, gradient descent and backpropagation. It shows how networks act as universal function approximators.
Why HN recommends it
HN recommends it as a quick, non-dumbed-down way to get up to speed on how LLMs are built.
I highly recommend this video by Steve Seitz — HN, 2023
Machine learning & deep learning · LLMs & generative AI
Subtitles (excerpt)
hello everyone in this video you'll learn all about large language models let's start with that autocomplete feature on your mobile phone did you ever wonder how it works the suggested word here is the it's the most used word in the english language let's type y next there are a number of words that start with t y if you took the shortest you get t y e this graph plots the frequency of this word over time it was pretty popular in 1806 but today you'd have to read 20 million words to find your first occurrence of tye we can look up these frequencies for every word starting with t y and if we sort by word frequency we have a clear winner the same approach works for phrases and sentences you see it every time you start typing into your internet search engine the search engine scores each query by calculating its frequency in other words how many other people have used that query so is that all there is to language modeling indeed the goal is to assign a probability to every sentence and frequencies are one way to multiple probabilities the problem is that this frequency approach doesn't allow you to score new sentences this is a perfectly valid sentence but one that may not have appeared previously you may ask how many
On Hacker News
Consider the following frame/time from the video: [VIDEO] At this point in time in the video, we see the following 3 things: 1) Fourier Series 2) Taylor Series 3) Neural Network The remainder of the video goes on to show how Neural Networks can approximate functions -- much like the Fourier Series and Taylor Series can... Now, I don't hold an advanced Ph.D. degree in Math, and prior to watching t… — peter_d_sherman, 2023-05
If anyone is looking for a quick overview of how LLMs are built, I highly recommend this video by Steve Seitz: [VIDEO] . It does an excellent job of taking you from 0 to a decent understanding without dumbing down the content or abusing analogies. — sainez, 2023-04
Not sure I can give an ELI5 explanation, but for me this the closest to what you are after in two short videos on the topic. At least they helped me get up to speed fast. Hope they will do the same for you ;-) Large Language Models from scratch [VIDEO] Large Language Models: Part 2 [LINK] — runetech, 2023-05
(Note: I am a different Human than the one you are replying to - apologies for interjecting.) > "Physically preventable" is a moving target. So are rockets and artillery shells, but check this out: [LINK] Humans can be very clever and generous ($60,000 USD per interceptor missile, hoo baby!! Is "democracy" not the greatest thing ever!!!???) when they set their minds to something. Unfortunately, d… — mistermann, 2024-03
> What is so unserious, disingenuous, and illegitimate about the “system”? Some examples (I could write indefinitely): [LINK] (section "Bank Error In Our Favor" is particularly offensive, to me) [LINK] (this sort of thing (which is fairly common) is offensive to me, as is our culture's complacency towards it) How it is (in part) perpetuated, in my estimation: - I’m sure whatever the system is it … — mistermann, 2023-01
Related videos
- AT&T Archives: Similiarities of Wave Behavior (Bonus Edition) · AT&T Tech Channel · 28:02
- The Original Double Slit Experiment · Veritasium · 7:39
- The spelled-out intro to neural networks and backpropagation: building micrograd · Andrej Karpathy · 2:25:52
- But what is a neural network? | Deep learning chapter 1 · 3Blue1Brown · 18:40
- AlphaGo - The Movie | Full award-winning documentary · Google DeepMind · 1:30:28
- Transformers, the tech behind LLMs | Deep Learning Chapter 5 · 3Blue1Brown · 27:14
- USENIX Security '18-Q: Why Do Keynote Speakers Keep Suggesting That Improving Security Is Possible? · USENIX · 51:22
- Lecture 1 | Machine Learning (Stanford) · Stanford · 1:08:40
- MarI/O - Machine Learning for Video Games · SethBling · 5:57
- Deep Dive into LLMs like ChatGPT · Andrej Karpathy · 3:31:23
- Computer program that learns to play classic NES games · suckerpinch · 16:17
- RL Course by David Silver - Lecture 1: Introduction to Reinforcement Learning · Google DeepMind · 1:28:12
- Let's build GPT: from scratch, in code, spelled out. · Andrej Karpathy · 1:56:20
- Bay Area Vision Meeting: Unsupervised Feature Learning and Deep Learning · Google TechTalks · 48:10
- A Hackers' Guide to Language Models · Jeremy Howard · 1:31:13
- Andrej Karpathy - AI for Full-Self Driving at Tesla · Matroid · 30:08