Movie reconstruction from human brain activity

Channel:Jack Gallant · Length:0:29 · Published:2011-09-22 · Views:6867281

Watch on YouTube

Summary

Jack Gallant's lab demonstrates reconstructing movie trailer clips a subject watched from fMRI brain activity, using regression models trained on hours of viewing data. The video shows the original clip beside the reconstruction.

Why HN recommends it

HN cites it as the landmark brain-decoding demo and compares newer reconstruction work against it.

Reconstructing what a person is seeing out of their fMRI scan (2011) — HN, 2023

Computer vision, speech & NLP · Neuroscience & the brain

On Hacker News

This is a much cooler example [VIDEO] If I'm understanding the description correctly, they are just training it to recognize what the image is closest to and taking an slice of a youtube video that most closely matches it. I imagine if they used a more efficient method or trained it more, they could do way better. It seems like most of the data to build an accurate picture of what they are seeing… — Houshalter, 2013-08
I imagine the best case outcome would look something like Jack Gallant's 2011 work on visual mind reading, where they trained a model on brain activity watching hundreds of hours of YouTube and then attempted to reconstruct a view of video not in the training set by correlating the new brain activity with frames from the input... Sorry I'm not explaining very clearly but there's a YouTube video o… — jazzyjackson, 2024-12
I mean in 2011, they were already able to do some basic reconstruction: see [VIDEO] I could imagine improvements since then,especially with advances in image networks. — SubiculumCode, 2023-03
There was a video many years ago (early 2010s?) demoing a similar technology, which would overlay and blend many images on top of each other to make a fuzzy image approximating what was actually being viewed. Edit: found it! [VIDEO] — andai, 2023-03
Reminds me of this paper [1] from 2011. See it in action in [2] 1. [LINK] 2. [VIDEO] Edit: Just realized the paper above is also from Shinji Nishimoto — Lutzb, 2023-03

Related videos