From lazy to rich to exclusive task representations in neural networks and neural codes

Modular representations emerge in neural networks trained to perform context-dependent tasks

The brain has large-scale modular structure in the form of brain regions, which are thought to arise from constraints on connectivity and the physical geometry of the cortical sheet. In contrast, experimental and theoretical work has argued both for and against the existence of specialized sub-populations of neurons (modules) within single brain regions. By studying artificial neural networks, we show that this local modularity emerges to support context-dependent behavior, but only when the input is low-dimensional. No anatomical constraints are required. We also show when modular specialization emerges at the population level (different modules correspond to orthogonal subspaces). Modularity yields abstract representations, allows for rapid learning and generalization on novel tasks, and facilitates the rapid learning of related contexts. Non-modular representations facilitate the rapid learning of unrelated contexts. Our findings reconcile conflicting experimental results and make predictions for future experiments.