Training AI Without Writing A Reward Function, with Reward Modelling

Channel:Robert Miles AI Safety · Length:17:51 · Published:2019-12-13 · Views:259736

Watch on YouTube

Summary

Robert Miles explains reward modelling, in which an AI learns what humans want from their preference comparisons instead of a hand-written reward function. He walks through the 2017 paper on deep reinforcement learning from human preferences.

Why HN recommends it

Seen as the clearest introduction to the idea behind RLHF, and also borrowed by commenters as a metaphor for human motivation.

I find this video by Rob Miles still the best presentation of the ingenious original 2017(!) paper — HN, 2023

Machine learning & deep learning · AI safety, AGI & impact

Subtitles (excerpt)

Hi. What is technology? Don't skip ahead, I promise I'm going somewhere with this. So you could have some kind of definition from a dictionary that's like, technology is machinery and equipment made using scientific knowledge, something like that. But where are the boundaries of the category? What counts? For example, pair of scissors. Technology? I think most people would say no, although it does meet the definition. Perhaps scissors used to be technology, but now I think they're too simple, they're too well understood. I think once we've really nailed something down and figured out all of the details, people stop thinking of it as technology. I think in order to be technology, something has to be complex and unpredictable, maybe even unreliable. YouTube, for example, is definitely technology, as is the device you're watching this on. Okay, why does this matter? I guess part of my point is, exact definitions are really difficult. And this generally isn't much of a problem, because language doesn't really work by exact definitions. Maybe it's hard to specify exactly what we mean when we use a word like technology. But to paraphrase something from the US Supreme Court, you know it when you see it. And that's good enough

On Hacker News

Specifically about RLHF, I find this video by Rob Miles still the best presentation of the ingenious original 2017(!) paper: [VIDEO] RLHF is actually older than GPT-1, which came out in 2018. It didn't get applied to language models until 2022 with InstructGPT, an approach which combined supervised instruction fine-tuning with RLHF. — cubefox, 2023-07
I would highly recommend [VIDEO] as a way of framing the problem. It seems like our options for ML are currently: self-play, dataset fitting, or human feedback. It's possible to solve any problem that can fit into one of those molds. (Unfortunately the opposite is true: if a problem doesn't fit into those categories, it seems very hard to solve with any form of ML.) But, now that I think about it… — sillysaurusx, 2019-12
ADHDer here. Please steal some of the motivation-techniques I've learnt over the years: 1. Find a friend who is a software engineer or data analyst and cares about you as a person. Talk about this with them. Humans evolved being able to talk things over with their aunt or best friend. When you are perplexed by something, a rubber duck probably won't cut it -- you should talk with a human. 2. Make… — afarrell, 2020-07
I think your URL is a fantastic choice and hints at what a lot of "accountability coaching" really is: Camaraderie An easily-imaginable person who acts as a reward model[1] you can imagine saying "yea! Make progress on the thing! You're on the right path[2]! Keep going." When you anticipate [social] rewards for an action, that action is motivating. If you watch Dr. Andrew Huberman's talks about t… — afarrell, 2021-01
A. Meet your basic biological needs. Your brain is a part of your body. A person with an iron deficiency can feel compelled to chew ice[1]. Ice does not cures anemia, brains are weird parts of the body and do weird things when the body is not getting what it needs. So, get better sleep, food, and exercise. Prof Andrew Huberman has some good tips[2]. B. Decide what few things you really value and … — afarrell, 2021-01

Related videos