Seeing and Imagining Are Handled by the Same Brain Cells

In one of the first single-neuron studies of imagination in the human brain, researchers found a striking match between how the brain sees objects and how it imagines them. The findings make important inroads into two previously indecipherable aspects of neuroscience: memory and the origins of creativity.

The research focused on a region of the brain called the ventral temporal cortex, which is responsible for object recognition. Measuring neurons in that region, researchers with the Simons Collaboration on the Global Brain (SCGB) found that many of the same neurons fire in response to a specific object, whether it is seen or imagined. In other words, the brain draws on overlapping groups of cells to perceive an apple and to conjure a mental picture of one. The finding suggests that the brain implements a ‘generative model’: a model that can take a high-level concept like ‘flower’ and synthesize a detailed picture of one, complete with colors, shading, shape and textures.

“Generative models are one of the most fundamental ways of thinking about how the brain is organized,” says Ueli Rutishauser, a senior author of the study, an SCGB investigator and a researcher at Cedars-Sinai Medical Center in Los Angeles, where the data were recorded. “Evidence for a generative model in the brain is a really important advance because it constrains how we think our brains compute.”

The results, published in Science, are an important first step toward understanding how creative processes — like making art and music — arise in the brain and how memories are retrieved. The findings could also help scientists better understand diseases that affect memory, such as Alzheimer’s, and diseases that are characterized by uncontrolled imagination, such as post-traumatic stress disorder.

Getting Inside the Mind’s Eye

The ability to conjure mental imagery is a key aspect of being human. The mind’s eye allows us to bring memories back to life, create art, imagine future scenarios, and visualize how we’d look in a new pair of shoes. But despite this, mental imagery is one of the least understood cognitive processes. While other perceptions, such as seeing, smelling and tasting, can be tested with animal models, it’s nearly impossible to confirm what an animal is thinking or imagining. As a result, research on mental imagery is limited to human studies, which are much harder to conduct.

In the new study, researchers enlisted the help of patients with electrodes implanted in their temporal lobe to diagnose their epilepsy. The wires protruding from the electrodes, each the width of a human hair, could monitor multiple neurons individually, giving the researchers a high-resolution neural map of the part of the brain responsible for representing visual objects. With 16 patients, the researchers were able to probe nearly 1,000 neurons in the ventral temporal cortex.

To see if individual neurons responded to unique objects, the researchers showed the patients 500 different images of everyday objects like planes, animals, trees and musical instruments, and measured which neurons responded. The researchers then asked the patients to imagine a subset of the specific objects they had seen. Nearly half of the same neurons reactivated.

Knowing that many of the same neurons were active in both vision and imagination, the researchers investigated whether the neurons were active in the same way. For each neuron, the researchers determined a mapping between visual features in the presented image and its neural response. They then tested whether they could use this mapping or ‘visual code’ to predict which object a patient was imagining from the bank of images. Remarkably, the imagined images the researchers guessed from the neural activity mapping were identical or close, such as two different images of headphones, or an image of a pot instead of a teacup. This suggests a shared neural code for perception and imagination.

“We now understand the way that arbitrary objects are translated into network activity,” says Varun Wadia, an SCGB researcher at Cedars-Sinai Medical Center and lead author of the study. “So we can use this ‘visual code’ to look at neural activity and predict the visual features of the image the patient is looking at or imagining.”

Researchers have previously tried this type of ‘mind reading’ — scientifically known as image reconstruction — but with limited success. This time, however, the SCGB researchers had a detailed understanding of the visual code used by the neurons, which led to more accurate results.

“We can actually reconstruct fine visual details of the images that people imagined, which to my knowledge is a first,” Wadia says. “But what struck us was how even in the misidentified cases, the features are quite similar, such as having the same basic shape.”

AI and Beyond

Generative models are a hot topic — and not just in neuroscience. These models are also the backbone of generative AI, which was inspired by this theory of how the brain works.

“When you write a prompt for AI, and it generates an image, that’s the same computational process as what we found happens in the human brain,” says Doris Tsao, an SCGB investigator, a researcher at the University of California, Berkeley, and one of the senior authors of the study.

The findings are a significant advance for neuroscience as the study represents a key first step toward an understanding of mental imagery. However, more work is needed. The researchers only showed the patients static, two-dimensional images. Adding dimensions, colors and rotation would increase the complexity.

The results are giving scientists a clearer view of an under-studied phenomenon in the brain. Hopefully, Wadia says, these findings will open new avenues for research into essential processes in the brain and help researchers chart a path toward treating disorders associated with mental imagery, such as post-traumatic stress disorder and visual hallucinations.

“I think this project is one of the first rigorous steps in that direction,” Wadia says.