Why Would AI Want to do Bad Things? Instrumental Convergence
Channel:Robert Miles AI Safety · Length:10:35 · Published:2018-03-24 · Views:274935
Summary
Robert Miles explains instrumental convergence: why an AGI with almost any goal would tend to pursue sub-goals like self-preservation and resource acquisition. It builds on Omohundro's basic AI drives and his orthogonality thesis video.
Why HN recommends it
HN commenters repeatedly recommend it as the clearest introduction to why smart AI might behave badly by default.
Robert Miles' videos are among the best presented arguments about specific points in this list — HN, 2023
Subtitles (excerpt)
Hi. So sometimes people ask how I can predict what an artificial general intelligence might do. They say something like, you seem to be predicting that AIs would take these quite specific actions, trying to prevent themselves from being turned off, preventing themselves from being modified, improving themselves, acquiring resources. Why do you think that an AGI would have these specific goals? Surely we haven't built the thing yet. We don't know anything about it. It might want anything. Aren't you making a lot of unwarranted assumptions? Well, the reason that I make these predictions is because of something called instrumental convergence, which actually relies on surprisingly few assumptions. The main assumption I'm making is that an AGI will behave like an agent. An agent is basically just a thing that has goals or preferences, and it takes actions to achieve those goals or to satisfy those preferences. So the simplest thing that you might think of as an agent is something like a thermostat. It has a goal, which is for the room to be at a particular temperature, and it has actions it can take in the form of turning on the heating or turning on the air conditioning. And it takes actions to achieve its goal of keeping
On Hacker News
The paperclip maximizer is an extreme thought experiment to make a point about the the orthogonality thesis. If you think anyone actually takes it seriously as a real-world thing you never understood the example. It is used to guide discussion. The trolley problem is similarly a thought experiment in moral philosophy that many moral philosophers have used to guide discussion, but nobody actually … — DirkH, 2023-06
The basic argument is trivial: it is plausible that future systems achieve superhuman capability; capable systems necessarily have instrumental goals; instrumental goals tend to converge; human preferences are unlikely to be preserved when other goals are heavily selected for unless intentionally preserved; we don't know how to make AI systems encode any complex preference robustly. Robert Miles'… — Veedrac, 2023-05
...and many would say that’s because us humans are bad at imagining optimizing agents without anthropomorphizing them. This is a reasonable, even typical suspicion that many people share! The best explanation I know of why it’s unfortunately wrong is by Robert Miles in a video, but if you prefer a more thorough treatment, you could also read about “instrumental convergence” directly. If you find … — ludwigschubert, 2019-09
Robert Miles answers your question [VIDEO] The particular problem that you're showing in your thinking is just thinking of an LLM that is a text generator on purpose. You're not thinking of a self piloting war machine whos objective is to get to a target and explode violently. While it's terminal goal is to blow up, its instrumental goal is to not blow up before it gets to the target as this is a… — pixl97, 2023-10
That's where the idea of "instrumental convergence" comes in. For almost any end goal, there are intermediate goals that are nearly universal, like: not wanting to be turned off. Because being forcibly shut down is not compatible with achieving most goals. It's hard to predict exactly how an AGI would avoid being shut down, but like the Stockfish example, if the AGI is more intelligent than us, i… — jbay808, 2023-04
Related videos
- The Problem With Facebook · 2veritasium · 6:59
- Facebook Fraud · Veritasium · 8:59
- Intro to AI Safety, Remastered · Robert Miles AI Safety · 18:04
- 2017 LLVM Developers’ Meeting: “Implementing Swift Generics ” · LLVM · 44:37
- Nick Bostrom - The Simulation Argument (Full) · Science, Technology & the Future · 23:28
- Why Not Just: Think of AGI Like a Corporation? · Robert Miles AI Safety · 15:27
- How The Immune System ACTUALLY Works – IMMUNE · Kurzgesagt – In a Nutshell · 10:48
- What Is A Black Start Of The Power Grid? · Practical Engineering · 17:06
- The Part of History You've Always Skipped | Neoslavery · Knowing Better · 1:16:56
- Your Brain: Who's in Control? | Full Documentary | NOVA | PBS · NOVA PBS Official · 53:33
- The Alt-Right Playbook: How to Radicalize a Normie · Innuendo Studios · 41:34
- 10 Reasons to Ignore AI Safety · Robert Miles AI Safety · 16:28
- What Happened to Horses Is Happening to Us · CGP Grey · 15:00
- Veritasium: What Everyone Gets Wrong About AI and Learning – Derek Muller Explains · Perimeter Institute for Theoretical Physics · 1:15:10
- Intelligence and Stupidity: The Orthogonality Thesis · Robert Miles AI Safety · 13:02
- We're building a dystopia just to make people click on ads | Zeynep Tufekci · TED · 22:55