An Observation on Generalization
Channel:Simons Institute for the Theory of Computing · Length:57:21 · Published:2023-08-14 · Views:211694
Summary
Ilya Sutskever gives a Simons Institute talk on why neural networks generalize, framing learning as finding short descriptions, like compression. It was part of the Large Language Models and Transformers program.
Why HN recommends it
Cited in discussions of learning as compression and of why neural networks generalize.
Ilya argues that the reason why neural networks generalize -- why they work at all -- is because they are effectively finding a simple description — HN, 2026
Machine learning & deep learning · LLMs & generative AI · Information theory & signal processing
Subtitles (excerpt)
YEJIN CHOI: It has a citation count that's six digits. It's more than 139,000, and then many more. So super excited to hear what Ilya has to say about LLMs and the future. Take it away. ILYA SUTSKEVER: OK. Hi. [APPLAUSE] Hi. Hi, everyone. Thank you for the introduction. When Umesh invited me to attend this event, I got really excited. I saw all the speaker list. And I thought, great, I'll go and I'll talk about something. And then I ran into a problem, which is that a lot of the technical work that we're doing at OpenAI, I actually can't talk about. [LAUGHTER] So I was really wracking my head what could I talk about. Right now, so as of not too long ago, I switched all my focus to working on AI alignment, all my research focus. And we'll have very cool results to show there, but not just yet. So I'm like, so this would have to be for the next talk. But what would be good for this talk? And I came up with something. I'll tell you about some very old results that we had at OpenAI many years ago, back in 2016 even, which really affected the way I think about unsupervised learning. And I thought I'd share them with you. It is possible that at this point, you will find them obvious. But maybe not all of them.
On Hacker News
Noting the theory papers on Kolmorogov complexity. For those not familiar, Ilya argues that the reason why neural networks generalize -- why they work at all -- is because they are effectively finding a simple description of their training data , converging down onto the limit of the Kolmorogov complexity. [1] [1] [VIDEO] — jawarner, 2026-07
> We are now getting an equivalent definition of what neural nets are being trained for! LLMs are trained to compress the internet as much as possible! Nice payoff. Others have also called out the relationship to compression ( [VIDEO] ). — JoshCole, 2025-09
I am curious why isn't the network always in an "interference" state, but sometimes it collapses into a "restricted" ONB? > the neural networks we observe in practice are in some sense noisily simulating larger, highly sparse networks This seems somewhat related to a point made by Ilya Sutskever here [1]: NNs can be though of an approximation to the Kolmogorov compressor. Speculating, one could s… — DigitalNoumena, 2023-08
Related videos
- The spelled-out intro to neural networks and backpropagation: building micrograd · Andrej Karpathy · 2:25:52
- But what is the Fourier Transform? A visual introduction. · 3Blue1Brown · 19:41
- But what is a neural network? | Deep learning chapter 1 · 3Blue1Brown · 18:40
- D/A and A/D | Digital Show and Tell (Monty Montgomery @ xiph.org) · FL Studio · 23:52
- AlphaGo - The Movie | Full award-winning documentary · Google DeepMind · 1:30:28
- Transformers, the tech behind LLMs | Deep Learning Chapter 5 · 3Blue1Brown · 27:14
- USENIX Security '18-Q: Why Do Keynote Speakers Keep Suggesting That Improving Security Is Possible? · USENIX · 51:22
- Lecture 1 | Machine Learning (Stanford) · Stanford · 1:08:40
- MarI/O - Machine Learning for Video Games · SethBling · 5:57
- Deep Dive into LLMs like ChatGPT · Andrej Karpathy · 3:31:23
- Computer program that learns to play classic NES games · suckerpinch · 16:17
- The Fast Fourier Transform (FFT): Most Ingenious Algorithm Ever? · Reducible · 28:23
- RL Course by David Silver - Lecture 1: Introduction to Reinforcement Learning · Google DeepMind · 1:28:12
- Let's build GPT: from scratch, in code, spelled out. · Andrej Karpathy · 1:56:20
- Bay Area Vision Meeting: Unsupervised Feature Learning and Deep Learning · Google TechTalks · 48:10
- A Hackers' Guide to Language Models · Jeremy Howard · 1:31:13