Training an unbeatable AI in Trackmania
Channel:Yosh · Length:20:41 · Published:2023-09-30 · Views:17725360
Summary
Yosh trains a reinforcement learning agent to drive Trackmania until he can no longer beat it. The video explains how the training works and the design choices involved.
Why HN recommends it
Recommended as an approachable explanation of how reinforcement learning works in practice, and cited for the constraints needed to stop exploits.
be sure to check out the videos by Yosh. They're amazing, and do a great job explaining how reinforcement learning works in practice — HN, 2024
Machine learning & deep learning · Gaming culture & analysis
Subtitles (excerpt)
Each of these cars is controlled by an Artificial Intelligence (AI) in the racing game Trackmania. And this AI is designed to improve over time through trial and error. The longer it trains, the better it gets. With enough training, it should be able to find the best lines, to drift perfectly, it might even become unbeatable. At least that's the theory. Actually, I've already tried to build such an AI, several times, and it could drive but I've played this game for years and it wasn't enough to beat me. Still I think it has some potential. So about six months ago, I decided to give this project one last chance, and to offer the AI an opportunity for revenge. This video is the conclusion of a three-year journey to make an AI that could beat me in Trackmania. And this time it got much better than I had expected. But first let's start with this very simple track, to better visualize what this AI does. The AI uses something called an artificial neural network. Basically, it's some kind of mathematical tool which roughly models how a brain works. Every tenth of a second, this neural network receives a few numbers which describe what's happening in the game. In response, it outputs new numbers
On Hacker News
For those who only read the comments, be sure to check out the videos by Yosh. They're amazing, and do a great job explaining how reinforcement learning works in practice: [VIDEO] [LINK] — smokel, 2024-07
There’s a few very cool examples where someone recently used RL to solve trackmania, and ends up having to add all sorts of constraints/penalties to prevent extremely strange exploits/glitches that are discovered IIRC… been a while since I watched. [VIDEO] [LINK] — szvsw, 2025-02
Depends what you consider fun, and how far you take it. Some people enjoy programming more than repetitive clicking in a GUI. For a clicker game, writing a bot lets you iterate on strategies easier - is it faster to get to level 2 if I buy the upgrade for A or B first? For Trackmania, it lets you get a world record and a YouTube video with 14M views. [VIDEO] — fragmede, 2024-10
>The entire point of this is to not encode the board state! I am not sure about this. From the article "The 50M parameter model played at 1300 ELO with 99.8% of its moves being legal within one day of training." I thought that the experiment was how well the model will perform, given that it's reward function is to predict text, rather than checkmate. Leela, Alpha0 their reward function is to win… — emporas, 2024-01
Related videos
- Classic Game Postmortem: Alone in the Dark · GDC Festival of Gaming · 57:43
- The spelled-out intro to neural networks and backpropagation: building micrograd · Andrej Karpathy · 2:25:52
- In Search Of A Flat Earth · Folding Ideas · 1:16:16
- But what is a neural network? | Deep learning chapter 1 · 3Blue1Brown · 18:40
- This Is Phil Fish · Innuendo Studios · 19:05
- AlphaGo - The Movie | Full award-winning documentary · Google DeepMind · 1:30:28
- Transformers, the tech behind LLMs | Deep Learning Chapter 5 · 3Blue1Brown · 27:14
- USENIX Security '18-Q: Why Do Keynote Speakers Keep Suggesting That Improving Security Is Possible? · USENIX · 51:22
- Lecture 1 | Machine Learning (Stanford) · Stanford · 1:08:40
- MarI/O - Machine Learning for Video Games · SethBling · 5:57
- Deep Dive into LLMs like ChatGPT · Andrej Karpathy · 3:31:23
- The Engoodening of No Man's Sky · Internet Historian · 53:51
- Computer program that learns to play classic NES games · suckerpinch · 16:17
- RL Course by David Silver - Lecture 1: Introduction to Reinforcement Learning · Google DeepMind · 1:28:12
- Let's build GPT: from scratch, in code, spelled out. · Andrej Karpathy · 1:56:20
- Bay Area Vision Meeting: Unsupervised Feature Learning and Deep Learning · Google TechTalks · 48:10