Visual prototypes in the ventral stream are attuned to complexity and gaze behavior

Early theories of efficient coding suggested the visual system could compress the world by learning to represent features where information was concentrated, such as contours. This view was validated by the discovery that neurons in posterior visual cortex respond to edges and curvature. Still, it remains unclear what other information-rich features are encoded by neurons in more anterior cortical regions (e.g., inferotemporal cortex). Here, we use a generative deep neural network to synthesize images guided by neuronal responses from across the visuocortical hierarchy, using floating microelectrode arrays in areas V1, V4 and inferotemporal cortex of two macaque monkeys. We hypothesize these images ("prototypes") represent such predicted information-rich features. Prototypes vary across areas, show moderate complexity, and resemble salient visual attributes and semantic content of natural images, as indicated by the animals' gaze behavior. This suggests the code for object recognition represents compressed features of behavioral relevance, an underexplored aspect of efficient coding.

Evolving Images for Visual Neurons Using a Deep Generative Network Reveals Coding Principles and Neuronal Preferences

What specific features should visual neurons encode, given the infinity of real-world images and the limited number of neurons available to represent them? We investigated neuronal selectivity in monkey inferotemporal cortex via the vast hypothesis space of a generative deep neural network, avoiding assumptions about features or semantic categories. A genetic algorithm searched this space for stimuli that maximized neuronal firing. This led to the evolution of rich synthetic images of objects with complex combinations of shapes, colors, and textures, sometimes resembling animals or familiar people, other times revealing novel patterns that did not map to any clear semantic category. These results expand our conception of the dictionary of features encoded in the cortex, and the approach can potentially reveal the internal representations of any system whose input can be captured by a generative model.

Seeing faces is necessary for face-domain formation

Here we report that monkeys raised without exposure to faces did not develop face domains, but did develop domains for other categories and did show normal retinotopic organization, indicating that early face deprivation leads to a highly selective cortical processing deficit. Therefore, experience must be necessary for the formation (or maintenance) of face domains. Gaze tracking revealed that control monkeys looked preferentially at faces, even at ages prior to the emergence of face domains, but face-deprived monkeys did not, indicating that face looking is not innate. A retinotopic organization is present throughout the visual system at birth, so selective early viewing behavior could bias category-specific visual responses toward particular retinotopic representations, thereby leading to domain formation in stereotyped locations in inferotemporal cortex, without requiring category-specific templates or biases. Thus, we propose that environmental importance influences viewing behavior, viewing behavior drives neuronal activity, and neuronal activity sculpts domain formation.

Integrating motion and depth via parallel pathways

Processing of visual information is both parallel and hierarchical, with each visual area richly interconnected with other visual areas. An example of the parallel architecture of the primate visual system is the existence of two principal pathways providing input to the middle temporal visual area (MT): namely, a direct projection from striate cortex (V1), and a set of indirect projections that also originate in V1 but then relay through V2 and V3. Here we have reversibly inactivated the indirect pathways while recording from MT neurons and measuring eye movements in alert monkeys, a procedure that has enabled us to assess whether the two different input pathways are redundant or whether they carry different kinds of information. We find that this inactivation causes a disproportionate degradation of binocular disparity tuning relative to direction tuning in MT neurons, suggesting that the indirect pathways are important in the recovery of depth in three-dimensional scenes.

Temporal evolution of 2-dimensional direction signals used to guide eye movements

The smooth pursuit system must integrate many local motion measurements into a coherent estimate of target velocity. Several laboratories have studied this integration process using eye movements elicited by targets, such as tilted bars, containing conflicts between local motion signals measured along contours [one dimensional (1D)] and those measured at the bar's endpoints, or terminators [two dimensional (2D)]. The general finding is that 1D signals dominate early responses, whereas later components of the behavior are determined by 2D signals. We studied the dynamics of the integration process in macaque monkeys by systematically varying the relative proportions of 1D and 2D signals and the retinal eccentricities at which they appeared. Predictably, longer bars produced greater and longer-lasting contour-induced deviations. The evolution of the 2D response occurred over a period of 50-400 ms, depending on the relative proportions of 1D and 2D signals. As contours were displaced from the fovea the deviation decreased but much less so for early (1st 40 ms) than for late (subsequent 40 ms) pursuit initiation. These bottom-up effects could be overcome to a limited extent by the top-down influence of predictability. Finally, we observed that when animals were free to track any part of the bar, they spontaneously made short-latency saccades to the terminators on most trials, especially when the bars were tilted. This suggests an increased saliency of moving terminators, particularly when discrepancies exist among local motion signals.