When Less is More: Sparse Networks Can Outperform Dense Ones
Granule cells of the cerebellum, which comprise an astonishing 50 percent of the brain, are the most numerous of all our neurons. But these cells have a curious property. Whereas many neurons, such as cortical cells, receive thousands of inputs from other cells, granule cells get relatively few — roughly four inputs per neuron. That sparseness has long intrigued Larry Abbott, a theoretical neuroscientist at Columbia University and an investigator with the Simons Collaboration on the Global Brain (SCGB). “We would expect that a denser network would convey more information, so why is the connectivity so sparse?” Abbott says.
In new research published in Neuron in March, Abbott and his collaborators showed that sparsely connected networks are sometimes superior to densely connected ones. They used a model based on the biological structure to show that, despite a dearth of connections, these networks can richly represent information by encoding data in many different dimensions.
The findings support the idea that the brain employs two general computational strategies: “The cortical strategy of many connections, with plasticity spread all over the circuit, and the cerebellar form of computation, a sparsely connected layer of cells where plasticity associated with learning is much more localized,” Abbott says. The two approaches are roughly analogous to two different types of machine learning — the cerebellum is similar to extreme learning, whereas the cortex mirrors deep learning.
When sparseness rules
The fly has in its head a mushroom-shaped structure involved in olfactory learning. This ‘mushroom body’ shares a similar architecture with the cerebellar cortex. Numerous Kenyon cells in the mushroom body each get inputs from about six projection neurons, a configuration that resembles that of cerebellar granule cells. In both cases, the ratio of cell number to inputs, which Abbott dubs the expansion ratio, is about 20.
Abbott and his collaborators had previously shown that connections onto Kenyon cells in the mushroom body are random. This finding supports theories suggesting that sparse, randomly connected networks are good for classifying stimuli into categories, such as safe or dangerous, as noted previously by Stefano Fusi, another theorist at Columbia and an SCGB investigator. In the new study, Abbott, postdoctoral researcher Ashok Litwin-Kumar and their collaborators showed that a small number of random inputs most effectively maximizes dimension — meaning it’s easier for the network to detect features in input patterns — which in turn enhances the ability to learn.
In research presented at the Cosyne 2017 Workshops in Snowbird, Utah, in February, Abbott and his collaborators used a combination of computation and simulation to create a network in which each neuron in the granule layer has a unique set of connections. In other words, each cell is maximally different. In building the network, each Kenyon cell first receives inputs from two projection neurons, repeating the process until all unique options are exhausted. The cycle then repeats with Kenyon cells receiving three unique connections, and so on. The researchers found that a model with six synapses per Kenyon cell — the same number as in the mushroom body — created a network in which 90 to 95 percent of cells are uniquely connected. “The point is to make a diverse set of cells with the hope that at least some contain the information you need,” Abbott says.
The result looks remarkably like real data collected using electron microscopy images from larval flies, Abbott says. “As we got to know more about the mushroom body, all the numbers fell into place,” he says. “I was really pleased that it fits the data so well.”
Random connections
The findings help explain connectivity patterns in the mushroom body and cerebellum. But they also highlight two potential problems. “If the activity patterns, in addition to the connectivity, were completely random, every odor — even very similar odors — would activate completely different neurons,” says Evan Schaffer, a postdoctoral researcher in Richard Axel’s lab at Columbia and an SCGB fellow. If that was the case, then two different smells — say, coffee and coffee with hazelnuts — would activate different populations, despite the fact that the two odors are similar.
Moreover, if brains are wired randomly, then the left and right hemisphere of the brain — or the brains of two different flies, mice or people — have two different wiring maps. One might expect that two different animals, with two different olfactory systems, would behave differently or that conflicts would arise between the two brain hemispheres. But the two hemispheres presumably generate similar decisions about the same odor, as do two flies or two mice for the most part.
In research presented at the Cosyne 2017 Workshops, Abbott, Schaffer and their collaborators identified a solution to these issues. The system can generalize — meaning the animal will know that hazelnut coffee smells similar to plain coffee — if the random structure is large enough, Schaffer says.
The researchers also found that if two networks get similar instruction early on — say, an animal learns to associate a specific smell with a shock — then the network outputs will respond similarly to novel smells in the future. “Even though behavior is driven by a randomly wired part of the brain, if that part is big enough, and the animals have had some shared experience, they will agree on which other stimuli are good and bad,” Schaffer says.
So why does the brain employ two very different systems — sparse, cerebellar-like connectivity and dense cortical connectivity? According to Abbott, the key difference is where learning takes place. In the cerebellar system, learning is localized to specific synapses. That system is likely optimized for rapid learning. If you lift your arm to grab something and miss, the cerebellum instantly gets an error signal and implements a rapid correction.
Plasticity in the cortical network, however, is much more distributed. That creates a potentially more powerful network capable of lengthy, complex tasks, such as learning a foreign language. But the cortical system brings its own challenges. Most notable in this context is the credit assignment problem, Abbott says. When learning a complex task, it’s hard to figure out whether a specific synapse in the middle of the cortex should be stronger or weaker. “Somehow, the cortex has solved that problem,” Abbott says. “But we don’t know how.”
