The mind’s eye revealed
New technology uses brain scans to see what a person is watching
Researchers have just wrapped production on a special movie of the mind that stars a brain scanner, a sophisticated computer program and millions of YouTube videos. By monitoring people’s brains as they watched movies and then re-creating what they saw, the new release has tiptoed closer to technology that can read minds by decoding mental activities, researchers report in an upcoming Current Biology paper.
“It’s very dramatic. It really is like Minority Report,” says neuroscientist James Haxby of Dartmouth College, referring to the 2002 Tom Cruise vehicle in which decoded visions from psychic brains help identify criminals before any crime is committed.
In the study, researchers led by Jack Gallant of the University of California, Berkeley used a type of brain scan called fMRI to record brain activity in three people (who were all coauthors on the paper) as they watched hours of Hollywood movie trailers. Brain signals were fed into a computer program that learned how each person’s visual system responded to scenes in the movies. Once the computer program had a good handle on the brains’ responses, the researchers went backwards and attempted to re-create what people were watching solely on the basis of brain signals.
It worked. The technique roughly reproduced movie clips that showed a red bird swooping across the scene, elephants marching across the desert and Steve Martin’s hilarious antics, the team reports.
The reproductions reflected only visual details such as the red of a feather, shape of a face and motion of an airplane — not the viewers’ full emotional reactions to the films. One of the main reasons the method worked was that scientists have a deep understanding of the visual system, Gallant says, an understanding that’s lacking for most other mental tasks. As researchers learn more about how the brain dreams, thinks or even feels emotions, other cognitive processes could be revealed by reading brain signals, too, he says. “Eventually I think we’ll be able to decode other kinds of things.”
Until now, such technology had been successful at decoding only stationary objects — such as an image of a chair — from brain activity. fMRI detects sluggish blood flow changes in the brain — signals that are usually too slow to capture the activity that accompanies watching a fast-paced movie. To get around this, the researchers fed the slow fMRI signals through a complex program that filled the gaps in the data to approximate the rapid chain of events as nerve cells in the visual system detect a moving scene. Estimating these quick brain changes from slow data was a key to the technique’s success, Gallant says. “Here, we’re modeling in way, way more detail than anyone has ever done before,” he says.
Guided by these lightning-quick predictions, the computer program then cobbled together scenes from a library made of millions of one-second snippets from YouTube videos. When Gallant and his team averaged the program’s top 100 predictions, the result was a blurry clip eerily similar to what had been watched.
Scientists will probably attempt other brain decoding feats soon, Haxby says. “I’m sure people are going to try to do imagination,” he says. “We’ll calibrate someone’s brain somehow, and say, ‘OK, I’d like you to imagine an ocean beach scene or imagine looking at your mother’s face.’ And then we’ll say, ‘Imagine whatever you want to, and we’re going to try to guess what you’re looking at.’”
REEL MIND-READING
People watched the movie clip on the left while an fMRI machine measured visual system responses. Based on the brain activity pattern alone, a computer program produced the movie on the right from a library of 18 million seconds of YouTube videos.
Credit: Gallant Lab/UC Berkeley
VISUAL MATCH
As three participants (rows) watched the movie clips at the top, a computer program used the participants’ brain activity to re-create a movie (made up of one-second video clips found on YouTube) that’s an average of the top 100 movie predictions (left column), a single best movie guess (second column from the left) and other good guesses.
Credit: Gallant Lab/UC Berkeley
