On the relationship between maps and domains in inferotemporal cortex

A computational deep learning investigation of animacy perception in the human brain

The functional organization of the human object vision pathway distinguishes between animate and inanimate objects. To understand animacy perception, we explore the case of zoomorphic objects resembling animals. While the perception of these objects as animal-like seems obvious to humans, such "Animal bias" is a striking discrepancy between the human brain and deep neural networks (DNNs). We computationally investigated the potential origins of this bias. We successfully induced this bias in DNNs trained explicitly with zoomorphic objects. Alternative training schedules failed to cause an Animal bias. We considered the superordinate distinction between animate and inanimate classes, the sensitivity for faces and bodies, the bias for shape over texture, the role of ecologically valid categories, recurrent connections, and language-informed visual processing. These findings provide computational support that the Animal bias for zoomorphic objects is a unique property of human perception yet can be explained by human learning history.

Cross-sectional and longitudinal changes in category-selectivity in visual cortex following pediatric cortical resection

The topographic organization of category-selective responses in human ventral occipitotemporal cortex (VOTC) and its relationship to regions subserving language functions is remarkably uniform across individuals. This arrangement is thought to result from the clustering of neurons responding to similar inputs, constrained by intrinsic architecture and tuned by experience. We examined the malleability of this organization in individuals with unilateral resection of VOTC during childhood for the management of drug-resistant epilepsy. In cross-sectional and longitudinal functional imaging studies, we compared the topography and neural representations of 17 category-selective regions in individuals with a VOTC resection, a 'control patient' with resection outside VOTC, and typically developing matched controls. We demonstrated both adherence to and deviation from the standard topography and uncovered fine-grained competitive dynamics between word- and face-selectivity over time in the single, preserved VOTC. The findings elucidate the nature and extent of cortical plasticity and highlight the potential for remodeling of extrastriate architecture and function.

Parallel development of object recognition in newborn chicks and deep neural networks

How do newborns learn to see? We propose that visual systems are space-time fitters, meaning visual development can be understood as a blind fitting process (akin to evolution) in which visual systems gradually adapt to the spatiotemporal data distributions in the newborn's environment. To test whether space-time fitting is a viable theory for learning how to see, we performed parallel controlled-rearing experiments on newborn chicks and deep neural networks (DNNs), including CNNs and transformers. First, we raised newborn chicks in impoverished environments containing a single object, then simulated those environments in a video game engine. Second, we recorded first-person images from agents moving through the virtual animal chambers and used those images to train DNNs. Third, we compared the viewpoint-invariant object recognition performance of the chicks and DNNs. When DNNs received the same visual diet (training data) as chicks, the models developed common object recognition skills as chicks. DNNs that used time as a teaching signal-space-time fitters-also showed common patterns of successes and failures across the test viewpoints as chicks. Thus, DNNs can learn object recognition in the same impoverished environments as newborn animals. We argue that space-time fitters can serve as formal scientific models of newborn visual systems, providing image-computable models for studying how newborns learn to see from raw visual experiences.

Spatiotemporal hierarchies of face representation in the human ventral temporal cortex

In this study, we examined the relatively unexplored realm of face perception, investigating the activities within human brain face-selective regions during the observation of faces at both subordinate and superordinate levels. We recorded intracranial EEG signals from the ventral temporal cortex in neurosurgical patients implanted with subdural electrodes during viewing of face subcategories (human, mammal, bird, and marine faces) as well as various non-face control stimuli. The results revealed a noteworthy correlation in response patterns across all face-selective areas in the ventral temporal cortex, not only within the same face category but also extending to different face categories. Intriguingly, we observed a systematic decrease in response correlation coupled with an increased response onset time from human face to mammalian face, bird face and marine faces. Our result aligns with the notion that distinctions at the basic level category (e.g., human face versus non-human face) emerges earlier than those at the superordinate level (e.g., animate versus inanimate). This indicates response gradient in the representation of facial images within human face-sensitive regions, transitioning progressively from human faces to non-face stimuli. Our findings provide insights into spatiotemporal dynamic of face representations which varies spatially and at different timescales depending on the face subcategory represented.

Distributed representations of behaviour-derived object dimensions in the human visual system

Object vision is commonly thought to involve a hierarchy of brain regions processing increasingly complex image features, with high-level visual cortex supporting object recognition and categorization. However, object vision supports diverse behavioural goals, suggesting basic limitations of this category-centric framework. To address these limitations, we mapped a series of dimensions derived from a large-scale analysis of human similarity judgements directly onto the brain. Our results reveal broadly distributed representations of behaviourally relevant information, demonstrating selectivity to a wide variety of novel dimensions while capturing known selectivities for visual features and categories. Behaviour-derived dimensions were superior to categories at predicting brain responses, yielding mixed selectivity in much of visual cortex and sparse selectivity in category-selective clusters. This framework reconciles seemingly disparate findings regarding regional specialization, explaining category selectivity as a special case of sparse response profiles among representational dimensions, suggesting a more expansive view on visual processing in the human brain.

Development of Higher-Level Vision: A Network Perspective

Most studies on the development of the visual system have focused on the mechanisms shaping early visual stages up to the level of primary visual cortex (V1). Much less is known about the development of the stages after V1 that handle the higher visual functions fundamental to everyday life. The standard model for the maturation of these areas is that it occurs sequentially, according to the positions of areas in the adult hierarchy. Yet, the existing literature reviewed here paints a different picture, one in which the adult configuration emerges through a sequence of unique network configurations that are not mere partial versions of the adult hierarchy. In addition to studying higher visual development per se to fill major gaps in knowledge, it will be crucial to adopt a network-level perspective in future investigations to unravel normal developmental mechanisms, identify vulnerabilities to developmental disorders, and eventually devise treatments for these disorders.

Neuronal Mechanisms Underlying Face Recognition in Non-human Primates

Humans and primates rely on visual face recognition for social interactions. Damage to specific brain areas causes prosopagnosia, a condition characterized by the inability to recognize familiar faces, indicating the presence of specialized brain areas for face processing. A breakthrough finding came from a non-human primate (NHP) study conducted in the early 2000s; it was the first to identify multiple face processing areas in the temporal lobe, termed face patches. Subsequent studies have demonstrated the unique role of each face patch in the structural analysis of faces. More recent studies have expanded these findings by exploring the role of face patch networks in social and memory functions and the importance of early face exposure in the development of the system. In this review, we discuss the neuronal mechanisms responsible for analyzing facial features, categorizing faces, and associating faces with memory and social contexts within both the cerebral cortex and subcortical areas. Use of NHPs in neuropsychological and neurophysiological studies can highlight the mechanistic aspects of the neuronal circuit underlying face recognition at both the single-neuron and whole-brain network levels.

Contrastive learning explains the emergence and function of visual category-selective regions

Modular and distributed coding theories of category selectivity along the human ventral visual stream have long existed in tension. Here, we present a reconciling framework-contrastive coding-based on a series of analyses relating category selectivity within biological and artificial neural networks. We discover that, in models trained with contrastive self-supervised objectives over a rich natural image diet, category-selective tuning naturally emerges for faces, bodies, scenes, and words. Further, lesions of these model units lead to selective, dissociable recognition deficits, highlighting their distinct functional roles in information processing. Finally, these pre-identified units can predict neural responses in all corresponding face-, scene-, body-, and word-selective regions of human visual cortex, under a highly constrained sparse positive encoding procedure. The success of this single model indicates that brain-like functional specialization can emerge without category-specific learning pressures, as the system learns to untangle rich image content. Contrastive coding, therefore, provides a unifying account of object category emergence and representation in the human brain.

Horizontal cortical connections shape intrinsic traveling waves into feature-selective motifs that regulate perceptual sensitivity

Intrinsic cortical activity forms traveling waves that modulate sensory-evoked responses and perceptual sensitivity. These intrinsic traveling waves (iTWs) may arise from the coordination of synaptic activity through long-range feature-dependent horizontal connectivity within cortical areas. In a spiking network model that incorporates feature-selective patchy connections, we observe iTW motifs that result from shifts in excitatory/inhibitory balance as action potentials traverse these patchy connections. To test whether feature-selective motifs occur in vivo, we examined data recorded in the middle temporal visual area (Area MT) of marmosets performing a visual detection task. We find that some iTWs form motifs that are feature selective, exhibiting direction-selective modulations in spiking activity. Further, motifs modulate the gain of target-evoked responses and perceptual sensitivity if the target matches the preference of the motif. These results suggest that iTWs are shaped by the patchy horizontal fiber projections in the cortex and can regulate neural and perceptual sensitivity in a feature-selective manner.

A Whole-Brain Topographic Ontology

It is a common view that the intricate array of specialized domains in the ventral visual pathway is innately prespecified. What this review postulates is that it is not. We explore the origins of domain specificity, hypothesizing that the adult brain emerges from an interplay between a domain-general map-based architecture, shaped by intrinsic mechanisms, and experience. We argue that the most fundamental innate organization of cortex in general, and not just the visual pathway, is a map-based topography that governs how the environment maps onto the brain, how brain areas interconnect, and ultimately, how the brain processes information.

White matter connections of human ventral temporal cortex are organized by cytoarchitecture, eccentricity, and category-selectivity from birth

Category-selective regions in ventral temporal cortex (VTC) have a consistent anatomical organization, which is hypothesized to be scaffolded by white matter connections. However, it is unknown how white matter connections are organized from birth. Here, we scanned newborn to 6-month-old infants and adults and used a data-driven approach to determine the organization of the white matter connections of VTC. We find that white matter connections are organized by cytoarchitecture, eccentricity, and category from birth. Connectivity profiles of functional regions in the same cytoarchitectonic area are similar from birth and develop in parallel, with decreases in endpoint connectivity to lateral occipital, and parietal, and somatosensory cortex, and increases to lateral prefrontal cortex. Additionally, connections between VTC and early visual cortex are organized topographically by eccentricity bands and predict eccentricity biases in VTC. These data have important implications for theories of cortical functional development and open new possibilities for understanding typical and atypical white matter development.

Distributed representations of behavior-derived object dimensions in the human visual system

Object vision is commonly thought to involve a hierarchy of brain regions processing increasingly complex image features, with high-level visual cortex supporting object recognition and categorization. However, object vision supports diverse behavioral goals, suggesting basic limitations of this category-centric framework. To address these limitations, we mapped a series of dimensions derived from a large-scale analysis of human similarity judgments directly onto the brain. Our results reveal broadly distributed representations of behaviorally-relevant information, demonstrating selectivity to a wide variety of novel dimensions while capturing known selectivities for visual features and categories. Behavior-derived dimensions were superior to categories at predicting brain responses, yielding mixed selectivity in much of visual cortex and sparse selectivity in category-selective clusters. This framework reconciles seemingly disparate findings regarding regional specialization, explaining category selectivity as a special case of sparse response profiles among representational dimensions, suggesting a more expansive view on visual processing in the human brain.

A spatial map: a propitious choice for constraining the binding problem

Many studies have shown that the human visual system has two major functionally distinct cortical visual pathways: a ventral pathway, thought to be important for object recognition, and a dorsal pathway, thought to be important for spatial cognition. According to our and others previous studies, artificial neural networks with two segregated pathways can determine objects' identities and locations more accurately and efficiently than one-pathway artificial neural networks. In addition, we showed that these two segregated artificial cortical visual pathways can each process identity and spatial information of visual objects independently and differently. However, when using such networks to process multiple objects' identities and locations, a binding problem arises because the networks may not associate each object's identity with its location correctly. In a previous study, we constrained the binding problem by training the artificial identity pathway to retain relative location information of objects. This design uses a location map to constrain the binding problem. One limitation of that study was that we only considered two attributes of our objects (identity and location) and only one possible map (location) for binding. However, typically the brain needs to process and bind many attributes of an object, and any of these attributes could be used to constrain the binding problem. In our current study, using visual objects with multiple attributes (identity, luminance, orientation, and location) that need to be recognized, we tried to find the best map (among an identity map, a luminance map, an orientation map, or a location map) to constrain the binding problem. We found that in our experimental simulations, when visual attributes are independent of each other, a location map is always a better choice than the other kinds of maps examined for constraining the binding problem. Our findings agree with previous neurophysiological findings that show that the organization or map in many visual cortical areas is primarily retinotopic or spatial.

Face detection mechanisms: Nature vs. nurture

For many animals, faces are a vitally important visual stimulus. Hence, it is not surprising that face perception has become a very popular research topic in neuroscience, with ca. 2000 papers published every year. As a result, significant progress has been made in understanding the intricate mechanisms underlying this phenomenon. However, the ontogeny of face perception, in particular the role of innate predispositions, remains largely unexplored at the neural level. Several influential studies in monkeys have suggested that seeing faces is necessary for the development of the face-selective brain domains. At the same time, behavioural experiments with newborn human babies and newly-hatched domestic chicks demonstrate that a spontaneous preference towards faces emerges early in life without pre-existing experience. Moreover, we were recently able to record face-selective neural responses in the brain of young, face-naïve chicks, thus demonstrating the existence of an innate face detection mechanism. In this review, we discuss these seemingly contradictory results and propose potential experimental approaches to resolve some of the open questions.

Mechanisms underlying category learning in the human ventral occipito-temporal cortex

The human ventral occipito-temporal cortex (VOTC) has evolved into specialized regions that process specific categories, such as words, tools, and animals. The formation of these areas is driven by bottom-up visual and top-down nonvisual experiences. However, the specific mechanisms through which top-down nonvisual experiences modulate category-specific regions in the VOTC are still unknown. To address this question, we conducted a study in which participants were trained for approximately 13 h to associate three sets of novel meaningless figures with different top-down nonvisual features: the wordlike category with word features, the non-wordlike category with nonword features, and the visual familiarity condition with no nonvisual features. Pre- and post-training functional MRI (fMRI) experiments were used to measure brain activity during stimulus presentation. Our results revealed that training induced a categorical preference for the two training categories within the VOTC. Moreover, the locations of two training category-specific regions exhibited a notable overlap. Remarkably, within the overlapping category-specific region, training resulted in a dissociation in activation intensity and pattern between the two training categories. These findings provide important insights into how different nonvisual categorical information is encoded in the human VOTC.

Horizontal cortical connections shape intrinsic traveling waves into feature-selective motifs that regulate perceptual sensitivity

Intrinsic, ongoing fluctuations of cortical activity form traveling waves that modulate the gain of sensory-evoked responses and perceptual sensitivity. Several lines of evidence suggest that intrinsic traveling waves (iTWs) may arise, in part, from the coordination of synaptic activity through the recurrent horizontal connectivity within cortical areas, which include long range patchy connections that link neurons with shared feature preferences. In a spiking network model with anatomical topology that incorporates feature-selective patchy connections, which we call the Balanced Patchy Network (BPN), we observe repeated iTWs, which we refer to as motifs. In the model, motifs stem from fluctuations in the excitability of like-tuned neurons that result from shifts in E/I balance as action potentials traverse these patchy connections. To test if feature-selective motifs occur in vivo, we examined data previously recorded using multielectrode arrays in Area MT of marmosets trained to perform a threshold visual detection task. Using a newly developed method for comparing the similarity of wave patterns we found that some iTWs can be grouped into motifs. As predicted by the BPN, many of these motifs are feature selective, exhibiting direction-selective modulations in ongoing spiking activity. Further, motifs modulate the gain of the response evoked by a target and perceptual sensitivity to the target if the target matches the preference of the motif. These results provide evidence that iTWs are shaped by the patterns of horizontal fiber projections in the cortex and that patchy connections enable iTWs to regulate neural and perceptual sensitivity in a feature selective manner.

Dissecting neural computations in the human auditory pathway using deep neural networks for speech

The human auditory system extracts rich linguistic abstractions from speech signals. Traditional approaches to understanding this complex process have used linear feature-encoding models, with limited success. Artificial neural networks excel in speech recognition tasks and offer promising computational models of speech processing. We used speech representations in state-of-the-art deep neural network (DNN) models to investigate neural coding from the auditory nerve to the speech cortex. Representations in hierarchical layers of the DNN correlated well with the neural activity throughout the ascending auditory system. Unsupervised speech models performed at least as well as other purely supervised or fine-tuned models. Deeper DNN layers were better correlated with the neural activity in the higher-order auditory cortex, with computations aligned with phonemic and syllabic structures in speech. Accordingly, DNN models trained on either English or Mandarin predicted cortical responses in native speakers of each language. These results reveal convergence between DNN model representations and the biological auditory pathway, offering new approaches for modeling neural coding in the auditory cortex.

The development of human cortical scene processing

Decades of research have uncovered the neural basis of place (or "scene") processing in adulthood, revealing a set of three regions that respond selectively to visual scene information, each hypothesized to support distinct functions within scene processing (e.g., recognizing a particular kind of place versus navigating through it). Despite this considerable progress, surprisingly little is known about how these cortical regions develop. Here we review the limited evidence to date, highlighting the first few studies exploring the origins of cortical scene processing in infancy, and the several studies addressing when the scene regions reach full maturity, unfortunately with inconsistent findings. This inconsistency likely stems from common pitfalls in pediatric functional magnetic resonance imaging, and accordingly, we discuss how these pitfalls may be avoided. Furthermore, we point out that almost all studies to date have focused only on general scene selectivity and argue that greater insight could be gleaned by instead exploring the more distinct functions of each region, as well as their connectivity. Finally, with this last point in mind, we offer a novel hypothesis that scene regions supporting navigation (including the occipital place area and retrosplenial complex) mature later than those supporting scene categorization (including the parahippocampal place area).

High-dimensional topographic organization of visual features in the primate temporal lobe

The inferotemporal cortex supports our supreme object recognition ability. Numerous studies have been conducted to elucidate the functional organization of this brain area, but there are still important questions that remain unanswered, including how this organization differs between humans and non-human primates. Here, we use deep neural networks trained on object categorization to construct a 25-dimensional space of visual features, and systematically measure the spatial organization of feature preference in both male monkey brains and human brains using fMRI. These feature maps allow us to predict the selectivity of a previously unknown region in monkey brains, which is corroborated by additional fMRI and electrophysiology experiments. These maps also enable quantitative analyses of the topographic organization of the temporal lobe, demonstrating the existence of a pair of orthogonal gradients that differ in spatial scale and revealing significant differences in the functional organization of high-level visual areas between monkey and human brains.

The neural code for "face cells" is not face-specific

Face cells are neurons that respond more to faces than to non-face objects. They are found in clusters in the inferotemporal cortex, thought to process faces specifically, and, hence, studied using faces almost exclusively. Analyzing neural responses in and around macaque face patches to hundreds of objects, we found graded response profiles for non-face objects that predicted the degree of face selectivity and provided information on face-cell tuning beyond that from actual faces. This relationship between non-face and face responses was not predicted by color and simple shape properties but by information encoded in deep neural networks trained on general objects rather than face classification. These findings contradict the long-standing assumption that face versus non-face selectivity emerges from face-specific features and challenge the practice of focusing on only the most effective stimulus. They provide evidence instead that category-selective neurons are best understood by their tuning directions in a domain-general object space.

A large-scale fMRI dataset for the visual processing of naturalistic scenes

One ultimate goal of visual neuroscience is to understand how the brain processes visual stimuli encountered in the natural environment. Achieving this goal requires records of brain responses under massive amounts of naturalistic stimuli. Although the scientific community has put a lot of effort into collecting large-scale functional magnetic resonance imaging (fMRI) data under naturalistic stimuli, more naturalistic fMRI datasets are still urgently needed. We present here the Natural Object Dataset (NOD), a large-scale fMRI dataset containing responses to 57,120 naturalistic images from 30 participants. NOD strives for a balance between sampling variation between individuals and sampling variation between stimuli. This enables NOD to be utilized not only for determining whether an observation is generalizable across many individuals, but also for testing whether a response pattern is generalized to a variety of naturalistic stimuli. We anticipate that the NOD together with existing naturalistic neuroimaging datasets will serve as a new impetus for our understanding of the visual processing of naturalistic stimuli.

Specialized Networks for Social Cognition in the Primate Brain

Primates have evolved diverse cognitive capabilities to navigate their complex social world. To understand how the brain implements critical social cognitive abilities, we describe functional specialization in the domains of face processing, social interaction understanding, and mental state attribution. Systems for face processing are specialized from the level of single cells to populations of neurons within brain regions to hierarchically organized networks that extract and represent abstract social information. Such functional specialization is not confined to the sensorimotor periphery but appears to be a pervasive theme of primate brain organization all the way to the apex regions of cortical hierarchies. Circuits processing social information are juxtaposed with parallel systems involved in processing nonsocial information, suggesting common computations applied to different domains. The emerging picture of the neural basis of social cognition is a set of distinct but interacting subnetworks involved in component processes such as face perception and social reasoning, traversing large parts of the primate brain.

Hippocampal spatial view cells for memory and navigation, and their underlying connectivity in humans

Hippocampal and parahippocampal gyrus spatial view neurons in primates respond to the spatial location being looked at. The representation is allocentric, in that the responses are to locations "out there" in the world, and are relatively invariant with respect to retinal position, eye position, head direction, and the place where the individual is located. The underlying connectivity in humans is from ventromedial visual cortical regions to the parahippocampal scene area, leading to the theory that spatial view cells are formed by combinations of overlapping feature inputs self-organized based on their closeness in space. Thus, although spatial view cells represent "where" for episodic memory and navigation, they are formed by ventral visual stream feature inputs in the parahippocampal gyrus in what is the parahippocampal scene area. A second "where" driver of spatial view cells are parietal inputs, which it is proposed provide the idiothetic update for spatial view cells, used for memory recall and navigation when the spatial view details are obscured. Inferior temporal object "what" inputs and orbitofrontal cortex reward inputs connect to the human hippocampal system, and in macaques can be associated in the hippocampus with spatial view cell "where" representations to implement episodic memory. Hippocampal spatial view cells also provide a basis for navigation to a series of viewed landmarks, with the orbitofrontal cortex reward inputs to the hippocampus providing the goals for navigation, which can then be implemented by hippocampal connectivity in humans to parietal cortex regions involved in visuomotor actions in space. The presence of foveate vision and the highly developed temporal lobe for object and scene processing in primates including humans provide a basis for hippocampal spatial view cells to be key to understanding episodic memory in the primate and human hippocampus, and the roles of this system in primate including human navigation.

Neural mechanisms for emotional contagion and spontaneous mimicry of live facial expressions

Viewing a live facial expression typically elicits a similar expression by the observer (facial mimicry) that is associated with a concordant emotional experience (emotional contagion). The model of embodied emotion proposes that emotional contagion and facial mimicry are functionally linked although the neural underpinnings are not known. To address this knowledge gap, we employed a live two-person paradigm (n = 20 dyads) using functional near-infrared spectroscopy during live emotive face-processing while also measuring eye-tracking, facial classifications and ratings of emotion. One dyadic partner, 'Movie Watcher', was instructed to emote natural facial expressions while viewing evocative short movie clips. The other dyadic partner, 'Face Watcher', viewed the Movie Watcher's face. Task and rest blocks were implemented by timed epochs of clear and opaque glass that separated partners. Dyadic roles were alternated during the experiment. Mean cross-partner correlations of facial expressions (r = 0.36 ± 0.11 s.e.m.) and mean cross-partner affect ratings (r = 0.67 ± 0.04) were consistent with facial mimicry and emotional contagion, respectively. Neural correlates of emotional contagion based on covariates of partner affect ratings included angular and supramarginal gyri, whereas neural correlates of the live facial action units included motor cortex and ventral face-processing areas. Findings suggest distinct neural components for facial mimicry and emotional contagion. This article is part of a discussion meeting issue 'Face2face: advancing the science of social interaction'.

Theoretical considerations and supporting evidence for the primary role of source geometry on field potential amplitude and spatial extent

Field potential (FP) recording is an accessible means to capture the shifts in the activity of neuron populations. However, the spatial and composite nature of these signals has largely been ignored, at least until it became technically possible to separate activities from co-activated sources in different structures or those that overlap in a volume. The pathway-specificity of mesoscopic sources has provided an anatomical reference that facilitates transcending from theoretical analysis to the exploration of real brain structures. We review computational and experimental findings that indicate how prioritizing the spatial geometry and density of sources, as opposed to the distance to the recording site, better defines the amplitudes and spatial reach of FPs. The role of geometry is enhanced by considering that zones of the active populations that act as sources or sinks of current may arrange differently with respect to each other, and have different geometry and densities. Thus, observations that seem counterintuitive in the scheme of distance-based logic alone can now be explained. For example, geometric factors explain why some structures produce FPs and others do not, why different FP motifs generated in the same structure extend far while others remain local, why factors like the size of an active population or the strong synchronicity of its neurons may fail to affect FPs, or why the rate of FP decay varies in different directions. These considerations are exemplified in large structures like the cortex and hippocampus, in which the role of geometrical elements and regional activation in shaping well-known FP oscillations generally go unnoticed. Discovering the geometry of the sources in play will decrease the risk of population or pathway misassignments based solely on the FP amplitude or temporal pattern.

A modality-independent proto-organization of human multisensory areas

The processing of multisensory information is based upon the capacity of brain regions, such as the superior temporal cortex, to combine information across modalities. However, it is still unclear whether the representation of coherent auditory and visual events requires any prior audiovisual experience to develop and function. Here we measured brain synchronization during the presentation of an audiovisual, audio-only or video-only version of the same narrative in distinct groups of sensory-deprived (congenitally blind and deaf) and typically developed individuals. Intersubject correlation analysis revealed that the superior temporal cortex was synchronized across auditory and visual conditions, even in sensory-deprived individuals who lack any audiovisual experience. This synchronization was primarily mediated by low-level perceptual features, and relied on a similar modality-independent topographical organization of slow temporal dynamics. The human superior temporal cortex is naturally endowed with a functional scaffolding to yield a common representation across multisensory events.

The human posterior cingulate, retrosplenial, and medial parietal cortex effective connectome, and implications for memory and navigation

The human posterior cingulate, retrosplenial, and medial parietal cortex are involved in memory and navigation. The functional anatomy underlying these cognitive functions was investigated by measuring the effective connectivity of these Posterior Cingulate Division (PCD) regions in the Human Connectome Project-MMP1 atlas in 171 HCP participants, and complemented with functional connectivity and diffusion tractography. First, the postero-ventral parts of the PCD (31pd, 31pv, 7m, d23ab, and v23ab) have effective connectivity with the temporal pole, inferior temporal visual cortex, cortex in the superior temporal sulcus implicated in auditory and semantic processing, with the reward-related vmPFC and pregenual anterior cingulate cortex, with the inferior parietal cortex, and with the hippocampal system. This connectivity implicates it in hippocampal episodic memory, providing routes for "what," reward and semantic schema-related information to access the hippocampus. Second, the antero-dorsal parts of the PCD (especially 31a and 23d, PCV, and also RSC) have connectivity with early visual cortical areas including those that represent spatial scenes, with the superior parietal cortex, with the pregenual anterior cingulate cortex, and with the hippocampal system. This connectivity implicates it in the "where" component for hippocampal episodic memory and for spatial navigation. The dorsal-transitional-visual (DVT) and ProStriate regions where the retrosplenial scene area is located have connectivity from early visual cortical areas to the parahippocampal scene area, providing a ventromedial route for spatial scene information to reach the hippocampus. These connectivities provide important routes for "what," reward, and "where" scene-related information for human hippocampal episodic memory and navigation. The midcingulate cortex provides a route from the anterior dorsal parts of the PCD and the supracallosal part of the anterior cingulate cortex to premotor regions.

Understanding Human Object Vision: A Picture Is Worth a Thousand Representations

Objects are the core meaningful elements in our visual environment. Classic theories of object vision focus upon object recognition and are elegant and simple. Some of their proposals still stand, yet the simplicity is gone. Recent evolutions in behavioral paradigms, neuroscientific methods, and computational modeling have allowed vision scientists to uncover the complexity of the multidimensional representational space that underlies object vision. We review these findings and propose that the key to understanding this complexity is to relate object vision to the full repertoire of behavioral goals that underlie human behavior, running far beyond object recognition. There might be no such thing as core object recognition, and if it exists, then its importance is more limited than traditionally thought.

Face processing in the infant brain after pandemic lockdown

The role of visual experience in the development of face processing has long been debated. We present a new angle on this question through a serendipitous study that cannot easily be repeated. Infants viewed short blocks of faces during fMRI in a repetition suppression task. The same identity was presented multiple times in half of the blocks (repeat condition) and different identities were presented once each in the other half (novel condition). In adults, the fusiform face area (FFA) tends to show greater neural activity for novel versus repeat blocks in such designs, suggesting that it can distinguish same versus different face identities. As part of an ongoing study, we collected data before the COVID-19 pandemic and after an initial local lockdown was lifted. The resulting sample of 12 infants (9-24 months) divided equally into pre- and post-lockdown groups with matching ages and data quantity/quality. The groups had strikingly different FFA responses: pre-lockdown infants showed repetition suppression (novel > repeat), whereas post-lockdown infants showed the opposite (repeat > novel), often referred to as repetition enhancement. These findings provide speculative evidence that altered visual experience during the lockdown, or other correlated environmental changes, may have affected face processing in the infant brain.

Anatomical correlates of category-selective visual regions have distinctive signatures of connectivity in neonates

The ventral visual stream is shaped during development by innate proto-organization within the visual system, such as the strong input from the fovea to the fusiform face area. In adults, category-selective regions have distinct signatures of connectivity to brain regions beyond the visual system, likely reflecting cross-modal and motoric associations. We tested if this long-range connectivity is part of the innate proto-organization, or if it develops with postnatal experience, by using diffusion-weighted imaging to characterize the connectivity of anatomical correlates of category-selective regions in neonates (N = 445), 1-9 month old infants (N = 11), and adults (N = 14). Using the HCP data we identified face- and place- selective regions and a third intermediate region with a distinct profile of selectivity. Using linear classifiers, these regions were found to have distinctive connectivity at birth, to other regions in the visual system and to those outside of it. The results support an extended proto-organization that includes long-range connectivity that shapes, and is shaped by, experience-dependent development.

Development of visual cortex in human neonates is selectively modified by postnatal experience

Experience-dependent cortical plasticity is a pivotal process of human brain development and essential for the formation of most cognitive functions. Although studies found that early visual experience could influence the endogenous development of visual cortex in animals, little is known about such impact on human infants. Using the multimodal MRI data from the developing human connectome project, we characterized the early structural and functional maps in the ventral visual cortex and their development during neonatal period. Particularly, we found that postnatal time selectively modulated the cortical thickness in the ventral visual cortex and the functional circuit between bilateral primary visual cortices. But the cortical myelination and functional connections of the high-order visual cortex developed without significant influence of postnatal time in such an early period. The structure-function analysis further revealed that the postnatal time had a direct influence on the development of homotopic connection in area V1, while gestational time had an indirect effect on it through cortical myelination. These findings were further validated in preterm-born infants who had longer postnatal time but shorter gestational time at birth. In short, these data suggested in human newborns that early postnatal time shaped the structural and functional development of the visual cortex in selective and organized patterns.

A highly selective response to food in human visual cortex revealed by hypothesis-free voxel decomposition

Prior work has identified cortical regions selectively responsive to specific categories of visual stimuli. However, this hypothesis-driven work cannot reveal how prominent these category selectivities are in the overall functional organization of the visual cortex, or what others might exist that scientists have not thought to look for. Furthermore, standard voxel-wise tests cannot detect distinct neural selectivities that coexist within voxels. To overcome these limitations, we used data-driven voxel decomposition methods to identify the main components underlying fMRI responses to thousands of complex photographic images. Our hypothesis-neutral analysis rediscovered components selective for faces, places, bodies, and words, validating our method and showing that these selectivities are dominant features of the ventral visual pathway. The analysis also revealed an unexpected component with a distinct anatomical distribution that responded highly selectively to images of food. Alternative accounts based on low- to mid-level visual features, such as color, shape, or texture, failed to account for the food selectivity of this component. High-throughput testing and control experiments with matched stimuli on a highly accurate computational model of this component confirm its selectivity for food. We registered our methods and hypotheses before replicating them on held-out participants and in a novel dataset. These findings demonstrate the power of data-driven methods and show that the dominant neural responses of the ventral visual pathway include not only selectivities for faces, scenes, bodies, and words but also the visually heterogeneous category of food, thus constraining accounts of when and why functional specialization arises in the cortex.

The hippocampus, ventromedial prefrontal cortex, and episodic and semantic memory

The human ventromedial prefrontal cortex (vmPFC)/anterior cingulate cortex is implicated in reward and emotion, but also in memory. It is shown how the human orbitofrontal cortex connecting with the vmPFC and anterior cingulate cortex provide a route to the hippocampus for reward and emotional value to be incorporated into episodic memory, enabling memory of where a reward was seen. It is proposed that this value component results in primarily episodic memories with some value component to be repeatedly recalled from the hippocampus so that they are more likely to become incorporated into neocortical semantic and autobiographical memories. The same orbitofrontal and anterior cingulate regions also connect in humans to the septal and basal forebrain cholinergic nuclei, thereby helping to consolidate memory, and helping to account for why damage to the vMPFC impairs memory. The human hippocampus and vmPFC thus contribute in complementary ways to forming episodic and semantic memories.

Multiple cortical visual streams in humans

The effective connectivity between 55 visual cortical regions and 360 cortical regions was measured in 171 HCP participants using the HCP-MMP atlas, and complemented with functional connectivity and diffusion tractography. A Ventrolateral Visual "What" Stream for object and face recognition projects hierarchically to the inferior temporal visual cortex, which projects to the orbitofrontal cortex for reward value and emotion, and to the hippocampal memory system. A Ventromedial Visual "Where" Stream for scene representations connects to the parahippocampal gyrus and hippocampus. An Inferior STS (superior temporal sulcus) cortex Semantic Stream receives from the Ventrolateral Visual Stream, from visual inferior parietal PGi, and from the ventromedial-prefrontal reward system and connects to language systems. A Dorsal Visual Stream connects via V2 and V3A to MT+ Complex regions (including MT and MST), which connect to intraparietal regions (including LIP, VIP and MIP) involved in visual motion and actions in space. It performs coordinate transforms for idiothetic update of Ventromedial Stream scene representations. A Superior STS cortex Semantic Stream receives visual inputs from the Inferior STS Visual Stream, PGi, and STV, and auditory inputs from A5, is activated by face expression, motion and vocalization, and is important in social behaviour, and connects to language systems.

The human language effective connectome

To advance understanding of brain networks involved in language, the effective connectivity between 26 cortical regions implicated in language by a community analysis and 360 cortical regions was measured in 171 humans from the Human Connectome Project, and complemented with functional connectivity and diffusion tractography, all using the HCP multimodal parcellation atlas. A (semantic) network (Group 1) involving inferior cortical regions of the superior temporal sulcus cortex (STS) with the adjacent inferior temporal visual cortex TE1a and temporal pole TG, and the connected parietal PGi region, has effective connectivity with inferior temporal visual cortex (TE) regions; with parietal PFm which also has visual connectivity; with posterior cingulate cortex memory-related regions; with the frontal pole, orbitofrontal cortex, and medial prefrontal cortex; with the dorsolateral prefrontal cortex; and with 44 and 45 for output regions. It is proposed that this system can build in its temporal lobe (STS and TG) and parietal parts (PGi and PGs) semantic representations of objects incorporating especially their visual and reward properties. Another (semantic) network (Group 3) involving superior regions of the superior temporal sulcus cortex and more superior temporal lobe regions including STGa, auditory A5, TPOJ1, the STV and the Peri-Sylvian Language area (PSL) has effective connectivity with auditory areas (A1, A4, A5, Pbelt); with relatively early visual areas involved in motion, e.g., MT and MST, and faces/words (FFC); with somatosensory regions (frontal opercular FOP, insula and parietal PF); with other TPOJ regions; and with the inferior frontal gyrus regions (IFJa and IFSp). It is proposed that this system builds semantic representations specialising in auditory and related facial motion information useful in theory of mind and somatosensory / body image information, with outputs directed not only to regions 44 and 45, but also to premotor 55b and midcingulate premotor cortex. Both semantic networks (Groups 1 and 3) have access to the hippocampal episodic memory system via parahippocampal TF. A third largely frontal network (Group 2) (44, 45, 47l; 55b; the Superior Frontal Language region SFL; and including temporal pole TGv) receives effective connectivity from the two semantic systems, and is implicated in syntax and speech output.

More Than the Face: Representations of Bodies in the Inferior Temporal Cortex

Visual representations of bodies, in addition to those of faces, contribute to the recognition of con- and heterospecifics, to action recognition, and to nonverbal communication. Despite its importance, the neural basis of the visual analysis of bodies has been less studied than that of faces. In this article, I review what is known about the neural processing of bodies, focusing on the macaque temporal visual cortex. Early single-unit recording work suggested that the temporal visual cortex contains representations of body parts and bodies, with the dorsal bank of the superior temporal sulcus representing bodily actions. Subsequent functional magnetic resonance imaging studies in both humans and monkeys showed several temporal cortical regions that are strongly activated by bodies. Single-unit recordings in the macaque body patches suggest that these represent mainly body shape features. More anterior patches show a greater viewpoint-tolerant selectivity for body features, which may reflect a processing principle shared with other object categories, including faces. Expected final online publication date for the Annual Review of Vision Science, Volume 8 is September 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

What and Where: Location-Dependent Feature Sensitivity as a Canonical Organizing Principle of the Visual System

Traditionally, functional representations in early visual areas are conceived as retinotopic maps preserving ego-centric spatial location information while ensuring that other stimulus features are uniformly represented for all locations in space. Recent results challenge this framework of relatively independent encoding of location and features in the early visual system, emphasizing location-dependent feature sensitivities that reflect specialization of cortical circuits for different locations in visual space. Here we review the evidence for such location-specific encoding including: (1) systematic variation of functional properties within conventional retinotopic maps in the cortex; (2) novel periodic retinotopic transforms that dramatically illustrate the tight linkage of feature sensitivity, spatial location, and cortical circuitry; and (3) retinotopic biases in cortical areas, and groups of areas, that have been defined by their functional specializations. We propose that location-dependent feature sensitivity is a fundamental organizing principle of the visual system that achieves efficient representation of positional regularities in visual experience, and reflects the evolutionary selection of sensory and motor circuits to optimally represent behaviorally relevant information. Future studies are necessary to discover mechanisms underlying joint encoding of location and functional information, how this relates to behavior, emerges during development, and varies across species.

A neural population selective for song in human auditory cortex

How is music represented in the brain? While neuroimaging has revealed some spatial segregation between responses to music versus other sounds, little is known about the neural code for music itself. To address this question, we developed a method to infer canonical response components of human auditory cortex using intracranial responses to natural sounds, and further used the superior coverage of fMRI to map their spatial distribution. The inferred components replicated many prior findings, including distinct neural selectivity for speech and music, but also revealed a novel component that responded nearly exclusively to music with singing. Song selectivity was not explainable by standard acoustic features, was located near speech- and music-selective responses, and was also evident in individual electrodes. These results suggest that representations of music are fractionated into subpopulations selective for different types of music, one of which is specialized for the analysis of song.

Face perception: computational insights from phylogeny

Studies of face perception in primates elucidate the psychological and neural mechanisms that support this critical and complex ability. Recent progress in characterizing face perception across species, for example in insects and reptiles, has highlighted the ubiquity over phylogeny of this key ability for social interactions and survival. Here, we review the competence in face perception across species and the types of computation that support this behavior. We conclude that the computational complexity of face perception evinced by a species is not related to phylogenetic status and is, instead, largely a product of environmental context and social and adaptive pressures. Integrating findings across evolutionary data permits the derivation of computational principles that shed further light on primate face perception.

A self-supervised domain-general learning framework for human ventral stream representation

Anterior regions of the ventral visual stream encode substantial information about object categories. Are top-down category-level forces critical for arriving at this representation, or can this representation be formed purely through domain-general learning of natural image structure? Here we present a fully self-supervised model which learns to represent individual images, rather than categories, such that views of the same image are embedded nearby in a low-dimensional feature space, distinctly from other recently encountered views. We find that category information implicitly emerges in the local similarity structure of this feature space. Further, these models learn hierarchical features which capture the structure of brain responses across the human ventral visual stream, on par with category-supervised models. These results provide computational support for a domain-general framework guiding the formation of visual representation, where the proximate goal is not explicitly about category information, but is instead to learn unique, compressed descriptions of the visual world.

Selective responses to faces, scenes, and bodies in the ventral visual pathway of infants

Three of the most robust functional landmarks in the human brain are the selective responses to faces in the fusiform face area (FFA), scenes in the parahippocampal place area (PPA), and bodies in the extrastriate body area (EBA). Are the selective responses of these regions present early in development or do they require many years to develop? Prior evidence leaves this question unresolved. We designed a new 32-channel infant magnetic resonance imaging (MRI) coil and collected high-quality functional MRI (fMRI) data from infants (2-9 months of age) while they viewed stimuli from four conditions-faces, bodies, objects, and scenes. We find that infants have face-, scene-, and body-selective responses in the location of the adult FFA, PPA, and EBA, respectively, powerfully constraining accounts of cortical development.

A connectivity-constrained computational account of topographic organization in primate high-level visual cortex

Inferotemporal (IT) cortex in humans and other primates is topographically organized, containing multiple hierarchically organized areas selective for particular domains, such as faces and scenes. This organization is commonly viewed in terms of evolved domain-specific visual mechanisms. Here, we develop an alternative, domain-general and developmental account of IT cortical organization. The account is instantiated in interactive topographic networks (ITNs), a class of computational models in which a hierarchy of model IT areas, subject to biologically plausible connectivity-based constraints, learns high-level visual representations optimized for multiple domains. We find that minimizing a wiring cost on spatially organized feedforward and lateral connections, alongside realistic constraints on the sign of neuronal connectivity within model IT, results in a hierarchical, topographic organization. This organization replicates a number of key properties of primate IT cortex, including the presence of domain-selective spatial clusters preferentially involved in the representation of faces, objects, and scenes; columnar responses across separate excitatory and inhibitory units; and generic spatial organization whereby the response correlation of pairs of units falls off with their distance. We thus argue that topographic domain selectivity is an emergent property of a visual system optimized to maximize behavioral performance under generic connectivity-based constraints.

A connectivity-constrained computational account of topographic organization in primate high-level visual cortex

Inferotemporal (IT) cortex in humans and other primates is topographically organized, containing multiple hierarchically organized areas selective for particular domains, such as faces and scenes. This organization is commonly viewed in terms of evolved domain-specific visual mechanisms. Here, we develop an alternative, domain-general and developmental account of IT cortical organization. The account is instantiated in interactive topographic networks (ITNs), a class of computational models in which a hierarchy of model IT areas, subject to biologically plausible connectivity-based constraints, learns high-level visual representations optimized for multiple domains. We find that minimizing a wiring cost on spatially organized feedforward and lateral connections, alongside realistic constraints on the sign of neuronal connectivity within model IT, results in a hierarchical, topographic organization. This organization replicates a number of key properties of primate IT cortex, including the presence of domain-selective spatial clusters preferentially involved in the representation of faces, objects, and scenes; columnar responses across separate excitatory and inhibitory units; and generic spatial organization whereby the response correlation of pairs of units falls off with their distance. We thus argue that topographic domain selectivity is an emergent property of a visual system optimized to maximize behavioral performance under generic connectivity-based constraints.

Visuospatial coding as ubiquitous scaffolding for human cognition

For more than 100 years we have known that the visual field is mapped onto the surface of visual cortex, imposing an inherently spatial reference frame on visual information processing. Recent studies highlight visuospatial coding not only throughout visual cortex, but also brain areas not typically considered visual. Such widespread access to visuospatial coding raises important questions about its role in wider cognitive functioning. Here, we synthesise these recent developments and propose that visuospatial coding scaffolds human cognition by providing a reference frame through which neural computations interface with environmental statistics and task demands via perception-action loops.

What Is Social about Autism? The Role of Allostasis-Driven Learning

Scientific research on neuro-cognitive mechanisms of autism often focuses on circuits that support social functioning. However, autism is a heterogeneous developmental variation in multiple domains, including social communication, but also language, cognition, and sensory-motor control. This suggests that the underlying mechanisms of autism share a domain-general foundation that impacts all of these processes. In this Perspective Review, we propose that autism is not a social deficit that results from an atypical "social brain". Instead, typical social development relies on learning. In social animals, infants depend on their caregivers for survival, which makes social information vitally salient. The infant must learn to socially interact in order to survive and develop, and the most prominent learning in early life is crafted by social interactions. Therefore, the most prominent outcome of a learning variation is atypical social development. To support the hypothesis that autism results from a variation in learning, we first review evidence from neuroscience and developmental science, demonstrating that typical social development depends on two domain-general processes that determine learning: (a) motivation, guided by allostatic regulation of the internal milieu; and (b) multi-modal associations, determined by the statistical regularities of the external milieu. These two processes are basic ingredients of typical development because they determine allostasis-driven learning of the social environment. We then review evidence showing that allostasis and learning are affected among individuals with autism, both neurally and behaviorally. We conclude by proposing a novel domain-general framework that emphasizes allostasis-driven learning as a key process underlying autism. Guided by allostasis, humans learn to become social, therefore, the atypical social profile seen in autism can reflect a domain-general variation in allostasis-driven learning. This domain-general view raises novel research questions in both basic and clinical research and points to targets for clinical intervention that can lower the age of diagnosis and improve the well-being of individuals with autism.