Face2Face: Real-time Face Capture and Reenactment of RGB Videos (CVPR 2016 Oral)

Channel:Matthias Niessner · Length:6:36 · Published:2016-03-17 · Views:4960144

Watch on YouTube

Summary

The CVPR 2016 oral demo of Face2Face, a system that captures a person's face and reenacts another video's face in real time using only RGB video. The authors frame it as research on modern vision and graphics.

Why HN recommends it

HN uses it as a reference point in deepfake and fake-news discussions, showing that convincing real-time video manipulation was already possible in 2016.

The face2face[1] tech looks pretty convincing to me and it already works in real-time — HN, 2016

LLMs & generative AI · Computer vision, speech & NLP · Computer graphics & rendering

Subtitles (excerpt)

We present a novel real-time facial reenactment method that works with any commodity webcam. Since our method only uses RGB data for both the source and target actor, we are able to manipulate YouTube videos in real time. Here we demonstrate our method in a live setup. On the right, a source actor is captured with a standard webcam. This input drives the animation of the face in the video shown on the monitor. to the left. A significant difference to previous methods is the rerendering of the mouth interior. To this end, we reynthesize the mouth interior of the target actor using video footage from the training sequence based on temporal and phototric similarity. As we can see, we are able to generate a realistic and convincing reenactment result. Here we show a close-up of the footage from the previous live reenactment. The input video stream is shown on the left. Note that the target actor is rerendered in a neutral pose. On the right, we can see the final output of our method. Our system reconstructs and tracks both source and target actors using a dense phototric energy minimization. Using a novel subspace deformation transfer technique, we transfer the expressions from the source to the target actor.

On Hacker News

Founder here Maybe – one of my co-founders is Prof Matthias Niessner who's been behind a large chunk of the seminal and widespread research in this space. [1] [VIDEO] [2] [LINK] — 0xx, 2021-05
Founder here. AMA :) To answer a few recurring questions in the thread ---> Use case. Video is a way more effective way to communicate than text. Not for the HN crowd, but if you're a blue collar worker a 2 minute video in your native language is much preferred to a 5 page pdf for training. Anyone who has tried to record a simple corporate video know the pain of cameras, film crews, 25 takes to g… — 0xx, 2021-05
[VIDEO] Face2Face would defeat this: anyone with enough images of the victim (and copy of ID) could impersonate them in this scheme. — cwkoss, 2018-08
Photo, audio, and video evidence should already be dismissed until one is able to verify the integrity and source. All of these can already be believably faked - it's just a matter of educating people that a layperson can easily create fake things by using tools developed by research teams. Fake text is the easiest to fake if you can identify the font used - any image editor will work. HN uses 9p… — Nadya, 2018-06
Here's another scary GAN proof of concept [0]. In this case, researchers transferred someone's face in real time to facial expression and mouth movements of public figures. Combined with DeepMind's new tech that seems to be able to produce human voice with believable candor and inflection [1], you could make some very convincing fake footage. [0] [VIDEO] [1] [LINK] — opportune, 2018-03

Related videos