Facebook's Prototype VR Face Tracking Got Even Better

Channel:UploadVR · Length:3:57 · Published:2019-06-28 · Views:69595

Watch on YouTube

Summary

UploadVR covers a Facebook Reality Labs prototype that tracks facial expressions using cameras mounted in a VR headset. The updated version captures subtle micro-expressions for realistic avatars.

Why HN recommends it

Pointed to in metaverse and VR threads as proof that realtime photorealistic face tracking from headset cameras is coming.

seriously jawdropping demo of facial tracking using a few cameras in a headset — HN, 2021

Computer vision, speech & NLP · VR, AR & spatial computing

Subtitles (excerpt)

our system runs live in real-time and it works for a wide range of expressions including puffins chicks biting lips moving tongues and details like wrinkles that are hard to be precisely animated for previous methods our goal is to train an encoder to convert input images to a facial expression code in real time this code will be later decoded - and 3d avatar however there is no trivial way to get precise correspondence between input and output our training data to solve this problem we build another headset with six additional cameras at a more accommodating positions to help establish correspondence once we correspond the nine view images with the avatar we can simply drop additional views to get the training data we need we use the idea of analysis by synthetase approach estimating facial expression and have the position from input images if our estimation is correct the rendered avatar should align with input images in every view however because of the large domain gap we can not directly compare them to cross domain gap we train generative at the virtual networks to perform image style translation a key requirement here is that the facial expressions should be unmodified and consistent across views after inputs are style transfers to ever

On Hacker News

> The author mentions lip syncing and hand tracking. That goes a long way, but it doesn't recreate the video experience to me. The oculus team put together a seriously jawdropping demo of facial tracking using a few cameras in a headset, combined with a lot of AI. The prototype was from a couple years ago. I can't wait until it makes it into commercial headsets. I wonder what the hold up is? Well… — josephg, 2021-09
Title should be amended to something like 'Face-to-face, rather than shoulder-to-shoulder, enhances learning, innovation.' For a high quality face-to-face remote experience, I expect we'll start to see really impressive consumer ready technology announced soon. See what Facebook have been doing with cameras mounted in VR headsets [1] or Google's Project Starline [2]. That said, when we're doing t… — microjim, 2022-03
There are algos that reconstruct photorealistic 3D facial expressions from headset cameras in realtime. [VIDEO] — roveo, 2021-11
I don't think they don't understand it, they understand incremental improvements. Whatever you're talking about is coming: [VIDEO] — baby, 2021-08
It will ( [VIDEO] ), and VR already conveys MUCH more of the micro-interactions of being with people that we are unknowingly blind to in Zoom meetings, which is one reason Zoom meetings are so draining. — jp555, 2020-08

Related videos