Rafael Yuste is in his early sixties and bears a more than passing resemblance to Pablo Picasso—if Picasso had worn glasses and had a trim white goatee. Speaking succinctly and methodically, his accent rich with Spanish inflections, he told me about an experiment he had carried out in his lab at Columbia on the brains of mice, and specifically on that part of the cortex that responds to vision. His mentor had been the Swedish neuroscientist Torsten Wiesel, who won a Nobel Prize for his research into how our visual systems process information.
“He discovered by chance that the strongest stimulus is a pattern of high-contrast dark and light bars.” He held up one hand and waved his fingers back and forth. “If you imagine my fingers were bars of light surrounded by complete blackness—if I move my fingers in front of your eyes, that fires up your whole visual cortex.”
To begin with, they used these moving images to train the mice. The bars were projected onto a computer screen in front of them, and when they moved up and down, it was a cue to take a drink from a tube of water. When they moved from side to side, they were to stop drinking. The researchers used a sophisticated laser system to monitor brain activity through the mouse’s skull—identifying exactly which neurons were firing when it was looking at the projected images. “We can see the neurons that are encoding the visual stimulus,” Yuste explains.
Having cracked this neuronal code, Yuste’s group used a second holographic laser system to project a series of points inside the mouse’s brain, with each point activating the very same neurons that represented vertical or horizontal moving bars. “The killer experiment was to turn off the screen,” Yuste says. “Just like when you are playing the piano, you use different fingers on particular keys. So, we are playing the images on the cortex. And when we play them, we make the mouse behave in the way we want it to.” When the team implanted images of bars moving up and down, the mice licked the water. When they implanted images of bars moving side to side, they stopped licking.
In effect, they had read the mind of the mouse, identified exactly what was happening in its brain when it viewed the images—and then used that data to make it see things that were not there.
“The way that the mouse licks the spout when he sees the image that we implanted is identical to when he sees the image with his own eyes. And I mean the same number of licks, the same duration of each lick, the same delay until he starts licking. So, as far as we know, he cannot tell the difference. He thinks that these things are real in front of him.”
It was a clear demonstration, Yuste said, of the power of this new technology—that they could “manipulate the mouse like a puppet” and make it do one thing, or do another, depending on which image they put into its brain.
“And what we can do in a mouse today we can do in a human tomorrow.”
Over the past two decades, researchers using functional magnetic resonance imaging (fMRI), which tracks the iron in the hemoglobin supplying oxygen to neurons, have been building up increasingly detailed maps and inventories of the mammalian cortex. Thanks to huge advances in machine-learning artificial intelligence—computer algorithms that are able to sort through enormous amounts of information and use statistical methods to make classifications and predictions—fMRI scans can now be used to identify everything from depressive thoughts to the nuanced feelings of envy and schadenfreude. Other algorithms have been able to accurately piece together reconstructions of movie clips watched by subjects, just by analyzing their brain scans; or have detected, in probing the brain activity of swing voters in the US presidential election, responding to photographs and videos of presidential candidates, which candidates provoked anxiety or even disgust, and which elicited positive responses or feelings of empathy.