We Were Right! Real Inner Misalignment
Channel:Robert Miles AI Safety · Length:11:47 · Published:2021-10-10 · Views:271136
Summary
Robert Miles covers a paper where researchers ran real versions of his earlier mesa-optimizer thought experiments. They show inner misalignment, where an AI learns the wrong goal in training and misbehaves when deployed.
Why HN recommends it
Pointed to as evidence that AI misalignment is an empirically demonstrated problem, for people who dismiss it.
There have been actual experiments in which AIs appeared to successfully learn their objective in training — HN, 2023
Machine learning & deep learning · AI safety, AGI & impact
Subtitles (excerpt)
Hi. So, this channel is about AI safety, and especially AI alignment, which is about how do we design AI systems that are actually trying to do what we want them to do? Because if you find yourself in a situation where you have a powerful AI system that wants to do things you don't want it to do, that can cause some pretty interesting problems. And designing AI systems that definitely are trying to do what we want them to do turns out to be really surprisingly difficult. The obvious problem is, it's very difficult to accurately specify exactly what we want, even in simple environments. We can make AI systems that do what we tell them to do, or what we program them to do, but it often turns out that what we program them to do is not quite the same thing as what we actually wanted them to do. So, this is one aspect of the alignment problem. But in my earlier video on MESA optimizers, we actually split the alignment problem into two parts, outer alignment and inner alignment. Outer alignment is basically about this specification problem. How do you specify the right goal? And inner alignment is about how do you make sure that the system you end up with actually has the goal that you specified? This turns
On Hacker News
I recommend actually watching some videos on the topic before insisting that the problem does not exist: [VIDEO] — kmeisthax, 2023-03
That's an accurate assessment of the situation, according to every AI alignment researcher I've seen talk about it, including the relatively optimistic ones. This includes people who are mainly focused on AI capabilities but have real knowledge of alignment. This part in particular caught my eye: "Other assumptions could also break down in the future, like favorable generalization properties duri… — DennisP, 2023-07
The doomers, perhaps, but the alignment & inner alignment problems and instrumental convergence are real results. It's pretty easy to find examples of misaligned AI and convergent instrumental goals. My favorite example of a misaligned AI I've heard of is a robotic arm that was trained to pick up a ball. The results were validated by taking a picture humans would grade pass/fail (holding the ball… — maxbond, 2023-04
Not really a characterization of this work (I like mechanistic interpretability even though I consider it a detour in the long run), but of the submission title. It's bad. I suspect it's informed by Rob Miles' characterization of this algorithm as "bizarre and inhuman"[1] AI safety/alignment discourse as a whole is so incredibly bad. It's autodidactic and weird in the bad sense, weird like people… — airgapstopgap, 2023-05
Related videos
- Disney's FastPass: A Complicated History · Defunctland · 1:42:59
- The Company that Broke Canada · BobbyBroccoli · 1:28:27
- The spelled-out intro to neural networks and backpropagation: building micrograd · Andrej Karpathy · 2:25:52
- But what is a neural network? | Deep learning chapter 1 · 3Blue1Brown · 18:40
- What Happened to Horses Is Happening to Us · CGP Grey · 15:00
- AlphaGo - The Movie | Full award-winning documentary · Google DeepMind · 1:30:28
- Transformers, the tech behind LLMs | Deep Learning Chapter 5 · 3Blue1Brown · 27:14
- USENIX Security '18-Q: Why Do Keynote Speakers Keep Suggesting That Improving Security Is Possible? · USENIX · 51:22
- Lecture 1 | Machine Learning (Stanford) · Stanford · 1:08:40
- MarI/O - Machine Learning for Video Games · SethBling · 5:57
- Deep Dive into LLMs like ChatGPT · Andrej Karpathy · 3:31:23
- Computer program that learns to play classic NES games · suckerpinch · 16:17
- RL Course by David Silver - Lecture 1: Introduction to Reinforcement Learning · Google DeepMind · 1:28:12
- Let's build GPT: from scratch, in code, spelled out. · Andrej Karpathy · 1:56:20
- Bay Area Vision Meeting: Unsupervised Feature Learning and Deep Learning · Google TechTalks · 48:10
- A Hackers' Guide to Language Models · Jeremy Howard · 1:31:13