John Schulman - Reinforcement Learning from Human Feedback: Progress and Challenges
Channel:UC Berkeley EECS · Length:1:03:32 · Published:2023-04-20 · Views:88511
Summary
UC Berkeley EECS colloquium by OpenAI's John Schulman on reinforcement learning from human feedback. He covers why RL helps with truthfulness, hallucination, and the limits of behavior cloning.
Why HN recommends it
Recommended in LLM threads as an authoritative explanation from a ChatGPT architect of why models hallucinate and why imitation is not enough.
There's a very good talk by John Schulman, chief architect of ChatGPT — HN, 2023
Machine learning & deep learning · LLMs & generative AI
Subtitles (excerpt)
PETER: Hello, everyone. Let's get things started. [APPLAUSE] Welcome to, I think, Ken, is this the fifth in the series? Yes, the fifth seminar in the Berkeley AI series. Thank you, Ken, for hosting the whole series and setting this up. It's an honor today to have with us here John Schulman. John is actually a Berkeley graduate, a graduate from Berkeley's PhD in 2016, is that right? From there he co-founded OpenAI and most people say rest of history, but not only that, he also is the chief architect of ChatGPT. He is the inventor of the modern deep learning-based policy gradient algorithms, including transition policy optimization, which he did at Berkeley together with Mike and me actually, then proximal policy optimization, the most widely used algorithm today in that space and used as part of ChatGPT training. So it's a real pleasure to have John back here with us. I'll tell you one quick story of my own first encounter with John. My own first encounter was not directly John, it was Professor Jose Carmena comes to me and he works in neurosciences, there's this new student that I really want to recruit. He's absolutely the best, this is the person I want to recruit. He wants to work on prosthetics and robotics is going to play a part in that.
On Hacker News
There is this idea that the goal of RLHF is to make ChatGPT woke or as you put it to lobotomize it. I suspect that this is a conspiracy theory. There's a very good talk by John Schulman, chief architect of ChatGPT [0], where he explains that if you don't include a RL component in your training, you're essentially doing imitation learning. It's well known that imitation learning fails miserably wh… — KKKKkkkk1, 2023-05
Sure, here you go: [VIDEO] Summarizing: 1. Obviously you can't completely "solve" truthfulness because people disagree on what is and is not true. But you can go a long way. 2. The models do know what they don't know. Their level of uncertainty is not only expressed in the final token logprobs but also seems to be reified somehow, such that they can express their own level of certainty in words. … — mike_hearn, 2023-04
How is that "as good as an intern?" Interns, for the most part, do not hallucinate and confidently lie on topics they know nothing about. Good interns tend to ask a lot of questions and quickly learn the cultural cues around the office, allowing them to fit in and become productive. LLMs are incapable of any of this. John Schulman from OpenAI recently gave a talk about the hallucination and uncer… — chongli, 2023-07
This is exactly the reason OpenAI isn't afraid of the open-source community, like many kneejerk opponents of regulatory capture assume (they are probably still afraid of Google). Also why they still do the expensive and cumbersome RLHF training, instead of those deceptively cheap and fast finetunes. They understand their own tech and why there isn't free lunch. Recently, John Schulman explained t… — airgapstopgap, 2023-05
Do yourself a favor and skip right through to the Twitter link to another link to this excellent post [1] on the actual reason that training new models on ChatGPT output in the manner of supervised learning (in contrast to reinforcement learning) will not produce a model as good as ChatGPT >For this type of interaction, we must use RL training, as supervised training teaches the model to lie. The… — lysozyme, 2023-04
Related videos
- The spelled-out intro to neural networks and backpropagation: building micrograd · Andrej Karpathy · 2:25:52
- But what is a neural network? | Deep learning chapter 1 · 3Blue1Brown · 18:40
- AlphaGo - The Movie | Full award-winning documentary · Google DeepMind · 1:30:28
- Transformers, the tech behind LLMs | Deep Learning Chapter 5 · 3Blue1Brown · 27:14
- USENIX Security '18-Q: Why Do Keynote Speakers Keep Suggesting That Improving Security Is Possible? · USENIX · 51:22
- Lecture 1 | Machine Learning (Stanford) · Stanford · 1:08:40
- MarI/O - Machine Learning for Video Games · SethBling · 5:57
- Deep Dive into LLMs like ChatGPT · Andrej Karpathy · 3:31:23
- Computer program that learns to play classic NES games · suckerpinch · 16:17
- RL Course by David Silver - Lecture 1: Introduction to Reinforcement Learning · Google DeepMind · 1:28:12
- Let's build GPT: from scratch, in code, spelled out. · Andrej Karpathy · 1:56:20
- Bay Area Vision Meeting: Unsupervised Feature Learning and Deep Learning · Google TechTalks · 48:10
- A Hackers' Guide to Language Models · Jeremy Howard · 1:31:13
- Andrej Karpathy - AI for Full-Self Driving at Tesla · Matroid · 30:08
- [1hr Talk] Intro to Large Language Models · Andrej Karpathy · 59:48
- Joscha Bach: Artificial Consciousness and the Nature of Reality | Lex Fridman Podcast #101 · Lex Fridman · 3:00:18