Facebook's Prototype VR Face Tracking Got Even Better
Channel:UploadVR · Length:3:57 · Published:2019-06-28 · Views:69595
Summary
UploadVR covers a Facebook Reality Labs prototype that tracks facial expressions using cameras mounted in a VR headset. The updated version captures subtle micro-expressions for realistic avatars.
Why HN recommends it
Pointed to in metaverse and VR threads as proof that realtime photorealistic face tracking from headset cameras is coming.
seriously jawdropping demo of facial tracking using a few cameras in a headset — HN, 2021
Computer vision, speech & NLP · VR, AR & spatial computing
Subtitles (excerpt)
our system runs live in real-time and it works for a wide range of expressions including puffins chicks biting lips moving tongues and details like wrinkles that are hard to be precisely animated for previous methods our goal is to train an encoder to convert input images to a facial expression code in real time this code will be later decoded - and 3d avatar however there is no trivial way to get precise correspondence between input and output our training data to solve this problem we build another headset with six additional cameras at a more accommodating positions to help establish correspondence once we correspond the nine view images with the avatar we can simply drop additional views to get the training data we need we use the idea of analysis by synthetase approach estimating facial expression and have the position from input images if our estimation is correct the rendered avatar should align with input images in every view however because of the large domain gap we can not directly compare them to cross domain gap we train generative at the virtual networks to perform image style translation a key requirement here is that the facial expressions should be unmodified and consistent across views after inputs are style transfers to ever
On Hacker News
> The author mentions lip syncing and hand tracking. That goes a long way, but it doesn't recreate the video experience to me. The oculus team put together a seriously jawdropping demo of facial tracking using a few cameras in a headset, combined with a lot of AI. The prototype was from a couple years ago. I can't wait until it makes it into commercial headsets. I wonder what the hold up is? Well… — josephg, 2021-09
Title should be amended to something like 'Face-to-face, rather than shoulder-to-shoulder, enhances learning, innovation.' For a high quality face-to-face remote experience, I expect we'll start to see really impressive consumer ready technology announced soon. See what Facebook have been doing with cameras mounted in VR headsets [1] or Google's Project Starline [2]. That said, when we're doing t… — microjim, 2022-03
There are algos that reconstruct photorealistic 3D facial expressions from headset cameras in realtime. [VIDEO] — roveo, 2021-11
I don't think they don't understand it, they understand incremental improvements. Whatever you're talking about is coming: [VIDEO] — baby, 2021-08
It will ( [VIDEO] ), and VR already conveys MUCH more of the micro-interactions of being with people that we are unknowingly blind to in Zoom meetings, which is one reason Zoom meetings are so draining. — jp555, 2020-08
Related videos
- Jordan Peterson debate on the gender pay gap, campus protests and postmodernism · Channel 4 News · 29:56
- Hjernevask - Brainwashing (Eng Sub) Part 1 - The Gender Equality Paradox · The Bookman · 38:52
- How one tweet can ruin your life | Jon Ronson · TED · 17:11
- Scale By The Bay 2018: Bryan Cantrill, Rust and Other Interesting Things · FunctionalTV · 37:06
- Harder Drive: Hard drives we didn't want or need · suckerpinch · 36:46
- Old Geeks Yell At Cloud (with Andrew Clay Shafer & Bryan Cantrill) · Arrested DevOps · 52:53
- The Value of Values with Rich Hickey · InfoQ · 31:43
- Badness 0 (Apostrophe‛s version) · suckerpinch · 22:15
- "The Economics of Programming Languages" by Evan Czaplicki (Strange Loop 2023) · Strange Loop Conference · 43:57
- Canceling | ContraPoints · ContraPoints · 1:40:27
- Roblox Pressured Us to Delete Our Video. So We Dug Deeper. · People Make Games · 35:21
- Bay Area Vision Meeting: Unsupervised Feature Learning and Deep Learning · Google TechTalks · 48:10
- QuakeCon 2012 - John Carmack Keynote · QuakeCon Official · 3:36:38
- "Voice Driven Development: Who needs a keyboard anyway?" by Emily Shea · Strange Loop Conference · 41:17
- Andrej Karpathy - AI for Full-Self Driving at Tesla · Matroid · 30:08
- RiftSketch: Live coding in VR with the Oculus Rift, Firefox WebVR, JavaScript and Three.js · Brian Peiris · 10:36