Training AI Without Writing A Reward Function, with Reward Modelling
Channel:Robert Miles AI Safety · Length:17:51 · Published:2019-12-13 · Views:259736
Summary
Robert Miles explains reward modelling, in which an AI learns what humans want from their preference comparisons instead of a hand-written reward function. He walks through the 2017 paper on deep reinforcement learning from human preferences.
Why HN recommends it
Seen as the clearest introduction to the idea behind RLHF, and also borrowed by commenters as a metaphor for human motivation.
I find this video by Rob Miles still the best presentation of the ingenious original 2017(!) paper — HN, 2023
Machine learning & deep learning · AI safety, AGI & impact
Subtitles (excerpt)
Hi. What is technology? Don't skip ahead, I promise I'm going somewhere with this. So you could have some kind of definition from a dictionary that's like, technology is machinery and equipment made using scientific knowledge, something like that. But where are the boundaries of the category? What counts? For example, pair of scissors. Technology? I think most people would say no, although it does meet the definition. Perhaps scissors used to be technology, but now I think they're too simple, they're too well understood. I think once we've really nailed something down and figured out all of the details, people stop thinking of it as technology. I think in order to be technology, something has to be complex and unpredictable, maybe even unreliable. YouTube, for example, is definitely technology, as is the device you're watching this on. Okay, why does this matter? I guess part of my point is, exact definitions are really difficult. And this generally isn't much of a problem, because language doesn't really work by exact definitions. Maybe it's hard to specify exactly what we mean when we use a word like technology. But to paraphrase something from the US Supreme Court, you know it when you see it. And that's good enough
On Hacker News
Specifically about RLHF, I find this video by Rob Miles still the best presentation of the ingenious original 2017(!) paper: [VIDEO] RLHF is actually older than GPT-1, which came out in 2018. It didn't get applied to language models until 2022 with InstructGPT, an approach which combined supervised instruction fine-tuning with RLHF. — cubefox, 2023-07
I would highly recommend [VIDEO] as a way of framing the problem. It seems like our options for ML are currently: self-play, dataset fitting, or human feedback. It's possible to solve any problem that can fit into one of those molds. (Unfortunately the opposite is true: if a problem doesn't fit into those categories, it seems very hard to solve with any form of ML.) But, now that I think about it… — sillysaurusx, 2019-12
ADHDer here. Please steal some of the motivation-techniques I've learnt over the years: 1. Find a friend who is a software engineer or data analyst and cares about you as a person. Talk about this with them. Humans evolved being able to talk things over with their aunt or best friend. When you are perplexed by something, a rubber duck probably won't cut it -- you should talk with a human. 2. Make… — afarrell, 2020-07
I think your URL is a fantastic choice and hints at what a lot of "accountability coaching" really is: Camaraderie An easily-imaginable person who acts as a reward model[1] you can imagine saying "yea! Make progress on the thing! You're on the right path[2]! Keep going." When you anticipate [social] rewards for an action, that action is motivating. If you watch Dr. Andrew Huberman's talks about t… — afarrell, 2021-01
A. Meet your basic biological needs. Your brain is a part of your body. A person with an iron deficiency can feel compelled to chew ice[1]. Ice does not cures anemia, brains are weird parts of the body and do weird things when the body is not getting what it needs. So, get better sleep, food, and exercise. Prof Andrew Huberman has some good tips[2]. B. Decide what few things you really value and … — afarrell, 2021-01
Related videos
- Is Most Published Research Wrong? · Veritasium · 12:22
- The Battle of Midway 1942: Told from the Japanese Perspective (1/3) · Montemayor · 41:46
- Why Meat is the Best Worst Thing in the World 🍔 · Kurzgesagt – In a Nutshell · 7:44
- Ben Marking's Fake BBC Nuclear Attack Broadcast (Full Resolution, Uncut, Unedited) · Tooth · 55:42
- The Zipf Mystery · Vsauce · 21:05
- Antonin Scalia - On American Exceptionalism · AmericanRhetoric.com · 7:21
- Self: The Movie; · Jecel Assumpção Jr · 21:01
- 30 Weird Chess Algorithms: Elo World · suckerpinch · 42:35
- 4.5 Billion Years in 1 Hour · Kurzgesagt – In a Nutshell · 1:03:31
- Try to Rule Fairly and You'll Be Overthrown · CGP Grey · 18:13
- The Agricultural Revolution: Crash Course World History #1 · CrashCourse · 11:10
- Master Your Sleep & Be More Alert When Awake · Andrew Huberman · 1:22:05
- You Need a Theme, Not a Goal · CGP Grey · 6:23
- But how does bitcoin actually work? · 3Blue1Brown · 25:15
- Big Plane vs Little Plane (The Economics of Long-Haul Flights) · Wendover Productions · 10:08
- Essence of linear algebra preview · 3Blue1Brown · 5:09
- The Solar Gravitational Lens will Map Exoplanets. Seriously. · Launch Pad Astronomy · 23:32
- How I implemented MegaTextures on real Nintendo 64 hardware · James Lambert · 8:51