Enhancing Photorealism Enhancement
Channel:ISL and Collaborators · Length:8:33 · Published:2021-05-10 · Views:6276362
Summary
A video presenting an Intel research paper on enhancing the photorealism of synthetic images. A convolutional network uses rendering intermediates, shown on GTA V footage that comes out looking like real street scenes.
Why HN recommends it
Used as a go-to example of neural networks enhancing game rendering, referenced in later AI-graphics discussions.
The video linked of applying a neural net to GTAV for photorealism is really impressive if you haven't seen it. — HN, 2021
Machine learning & deep learning · LLMs & generative AI · Computer graphics & rendering
Subtitles (excerpt)
In this work, we are enhancing photo-realism enhancement. To understand what that means, let us have a look at some footage from the game Grand Theft Auto V. The game looks great, but it is not quite photo-realistic. us now turn on our method. method modifies the images from the game to look more realistic. It is a convolutional network which produces images frame by frame and can be run at interactive rates. This model was trained to translate GTA V to the Cityscapes dataset. Cityscapes contains mostly images of German cities recorded with an automotive-grade camera. Here are a few examples from the dataset. With a model of reality induced by Cityscapes, our method paves the roads with smoother asphalt, increases the glossiness of car paint, and greens the sun-dried hills of GTA's California. The results are geometrically and semantically consistent with the input images and temporally stable. Let us have a look at the state of the art in comparison to our work. Here is contrastive unpaired image-to-image translation or cut for short. Cut is less temporally stable and hallucinates objects such as trees in the sky or Mercedes stars on the hood of the car. Here's another state-of-the-art approach.
On Hacker News
DLSS & ideas like it outside an idea are really interesting: [LINK] Ex. run the base game at an insanely crummy/low resolution. On low(er) it’s possible to run a model to kinda “upscale” in each frame in a performant way. Or even this CNN initiative from Intel where they took Grand Theft Auto & “painted” it with dashcam footage from rural Germany: [VIDEO] The results are pretty interesting— all “… — valgaze, 2022-04
The video linked of applying a neural net to GTAV for photorealism is really impressive if you haven't seen it. [VIDEO] — arduinomancer, 2021-05
Researchers trained a deep learning network on photos to make GTA5 more realistic-looking: [VIDEO] I was surprised that most of it is is just tone mapping. It looks too green-ish to me, needing some white balancing, but realistic. I think Video games are intentionally over-saturated and contrasty to make them look better, just as paintings are[1]. I took many screenshots in Horizon Zero Dawn that… — meinersbur, 2025-07
I think for the near future we’ll see something like this: [VIDEO] From a few years ago, where the game is rendered traditionally and used as a ground truth, with a model on top of it that enhances the graphics. After maybe 10-15 years we will be past the point where the entire game can be generated without obvious mistakes in consistency. Realtime AI dialogue is already possible but still a bit … — joegibbs, 2024-02
> AI will be doing graphics programming? I doubt that. I think the bigger concern (if you could call it that) is that certain AI techniques may be able to replace entire stacks once they get advanced/stable enough. Something along the lines of this perhaps, where you render simplified geometry and let the AI fill in the blanks: [VIDEO] Even without experimental technologies, you can get a glimpse… — FoodWThrow, 2023-11
Related videos
- Let's port Diablo to SerenityOS! · Andreas Kling · 1:19:39
- State of GPT | BRK216HFS · Microsoft Developer · 42:40
- - Startups - David H. Hansson, Partner at 37signals -TWiST #E337 · This Week in Startups · 1:38:37
- Lecture 1 | Machine Learning (Stanford) · Stanford · 1:08:40
- The spelled-out intro to neural networks and backpropagation: building micrograd · Andrej Karpathy · 2:25:52
- But what is a neural network? | Deep learning chapter 1 · 3Blue1Brown · 18:40
- The Beauty of Bézier Curves · Freya Holmér · 24:25
- The Continuity of Splines · Freya Holmér · 1:13:50
- elevated by Rgba & TBC | 4k intro (FullHD 1080p demoscene demo) · Demoscene High-Quality Videos (Annikras) · 3:36
- AlphaGo - The Movie | Full award-winning documentary · Google DeepMind · 1:30:28
- Transformers, the tech behind LLMs | Deep Learning Chapter 5 · 3Blue1Brown · 27:14
- Painting a Character with Maths · Inigo Quilez · 24:46
- USENIX Security '18-Q: Why Do Keynote Speakers Keep Suggesting That Improving Security Is Possible? · USENIX · 51:22
- The Real Story Behind Wayland and X - Daniel Stone (linux.conf.au 2013) · Mladen Mijatov · 45:35
- A Deep Dive into Nanite Virtualized Geometry · SIGGRAPH Advances in Real-Time Rendering · 1:09:59
- MarI/O - Machine Learning for Video Games · SethBling · 5:57