Category-specific organization in the human brain does not require visual experience

Learning Warps Object Representations in the Ventral Temporal Cortex

The human ventral temporal cortex (VTC) plays a critical role in object recognition. Although it is well established that visual experience shapes VTC object representations, the impact of semantic and contextual learning is unclear. In this study, we tracked changes in representations of novel visual objects that emerged after learning meaningful information about each object. Over multiple training sessions, participants learned to associate semantic features (e.g., "made of wood," "floats") and spatial contextual associations (e.g., "found in gardens") with novel objects. fMRI was used to examine VTC activity for objects before and after learning. Multivariate pattern similarity analyses revealed that, after learning, VTC activity patterns carried information about the learned contextual associations of the objects, such that objects with contextual associations exhibited higher pattern similarity after learning. Furthermore, these learning-induced increases in pattern information about contextual associations were correlated with reductions in pattern information about the object's visual features. In a second experiment, we validated that these contextual effects translated to real-life objects. Our findings demonstrate that visual object representations in VTC are shaped by the knowledge we have about objects and show that object representations can flexibly adapt as a consequence of learning with the changes related to the specific kind of newly acquired information.

Object Domain and Modality in the Ventral Visual Pathway

The nature of domain-specific organization in higher-order visual cortex (ventral occipital temporal cortex, VOTC) has been investigated both in the case of visual experience deprivation and of modality of stimulation in sighted individuals. Object domain interacts in an intriguing and revelatory way with visual experience and modality of stimulation: selectivity for artifacts and scene domains is largely immune to visual deprivation and is multi-modal, whereas selectivity for animate items in lateral posterior fusiform gyrus is present only with visual stimulation. This domain-by-modality interaction is not readily accommodated by existing theories of VOTC representation. We conjecture that these effects reflect a distinction between the visual features that characterize different object domains and their interaction with different types of downstream computational systems.

Semantics of the Visual Environment Encoded in Parahippocampal Cortex

Semantic representations capture the statistics of experience and store this information in memory. A fundamental component of this memory system is knowledge of the visual environment, including knowledge of objects and their associations. Visual semantic information underlies a range of behaviors, from perceptual categorization to cognitive processes such as language and reasoning. Here we examine the neuroanatomic system that encodes visual semantics. Across three experiments, we found converging evidence indicating that knowledge of verbally mediated visual concepts relies on information encoded in a region of the ventral-medial temporal lobe centered on parahippocampal cortex. In an fMRI study, this region was strongly engaged by the processing of concepts relying on visual knowledge but not by concepts relying on other sensory modalities. In a study of patients with the semantic variant of primary progressive aphasia (semantic dementia), atrophy that encompassed this region was associated with a specific impairment in verbally mediated visual semantic knowledge. Finally, in a structural study of healthy adults from the fMRI experiment, gray matter density in this region related to individual variability in the processing of visual concepts. The anatomic location of these findings aligns with recent work linking the ventral-medial temporal lobe with high-level visual representation, contextual associations, and reasoning through imagination. Together, this work suggests a critical role for parahippocampal cortex in linking the visual environment with knowledge systems in the human brain.

The origins of metamodality in visual object area LO: Bodily topographical biases and increased functional connectivity to S1

Recent evidence from blind participants suggests that visual areas are task-oriented and sensory modality input independent rather than sensory-specific to vision. Specifically, visual areas are thought to retain their functional selectivity when using non-visual inputs (touch or sound) even without having any visual experience. However, this theory is still controversial since it is not clear whether this also characterizes the sighted brain, and whether the reported results in the sighted reflect basic fundamental a-modal processes or are an epiphenomenon to a large extent. In the current study, we addressed these questions using a series of fMRI experiments aimed to explore visual cortex responses to passive touch on various body parts and the coupling between the parietal and visual cortices as manifested by functional connectivity. We show that passive touch robustly activated the object selective parts of the lateral–occipital (LO) cortex while deactivating almost all other occipital–retinotopic-areas. Furthermore, passive touch responses in the visual cortex were specific to hand and upper trunk stimulations. Psychophysiological interaction (PPI) analysis suggests that LO is functionally connected to the hand area in the primary somatosensory homunculus (S1), during hand and shoulder stimulations but not to any of the other body parts. We suggest that LO is a fundamental hub that serves as a node between visual-object selective areas and S1 hand representation, probably due to the critical evolutionary role of touch in object recognition and manipulation. These results might also point to a more general principle suggesting that recruitment or deactivation of the visual cortex by other sensory input depends on the ecological relevance of the information conveyed by this input to the task/computations carried out by each area or network. This is likely to rely on the unique and differential pattern of connectivity for each visual area with the rest of the brain.

•

We studied cross-modal effects of passive somatosensory inputs on the visual cortex.

•

Passive touch on the body evoked massive deactivation in the visual cortex.

•

Passive hand stimulation evoked unique activation in visual object area LO.

•

This area was also uniquely connected to the hand area in Penfield's homunculus — S1.

Dense sampling reveals behavioral oscillations in rapid visual categorization

Perceptual systems must create discrete objects and events out of a continuous flow of sensory information. Previous studies have demonstrated oscillatory effects in the behavioral outcome of low-level visual tasks, suggesting a cyclic nature of visual processing as the solution. To investigate whether these effects extend to more complex tasks, a stream of "neutral" photographic images (not containing targets) was rapidly presented (20 ms/image). Embedded were one or two presentations of a randomly selected target image (vehicles and animals). Subjects reported the perceived target category. On dual-presentation trials, the ISI varied systematically from 0 to 600 ms. At randomized timing before first target presentation, the screen was flashed with the intent of creating a phase reset in the visual system. Sorting trials by temporal distance between flash and first target presentation revealed strong oscillations in behavioral performance, peaking at 5 Hz. On dual-target trials, longer ISIs led to reduced performance, implying a temporal integration window for object category discrimination. The "animal" trials exhibited a significant oscillatory component around 5 Hz. Our results indicate that oscillatory effects are not mere fringe effects relevant only with simple stimuli, but are resultant from the core mechanisms of visual processing and may well extend into real-life scenarios.

Understanding What We See: How We Derive Meaning From Vision

Recognising objects goes beyond vision, and requires models that incorporate different aspects of meaning. Most models focus on superordinate categories (e.g., animals, tools) which do not capture the richness of conceptual knowledge. We argue that object recognition must be seen as a dynamic process of transformation from low-level visual input through categorical organisation to specific conceptual representations. Cognitive models based on large normative datasets are well-suited to capture statistical regularities within and between concepts, providing both category structure and basic-level individuation. We highlight recent research showing how such models capture important properties of the ventral visual pathway. This research demonstrates that significant advances in understanding conceptual representations can be made by shifting the focus from studying superordinate categories to basic-level concepts.

A Model of Emergent Category-specific Activation in the Posterior Fusiform Gyrus of Sighted and Congenitally Blind Populations

Theories about the neural bases of semantic knowledge tend between two poles, one proposing that distinct brain regions are innately dedicated to different conceptual domains and the other suggesting that all concepts are encoded within a single network. Category-sensitive functional activations in the fusiform cortex of the congenitally blind have been taken to support the former view but also raise several puzzles. We use neural network models to assess a hypothesis that spans the two poles: The interesting functional activation patterns reflect the base connectivity of a domain-general semantic network. Both similarities and differences between sighted and congenitally blind groups can emerge through learning in a neural network, but only in architectures adopting real anatomical constraints. Surprisingly, the same constraints suggest a novel account of a quite different phenomenon: the dyspraxia observed in patients with semantic impairments from anterior temporal pathology. From this work, we suggest that the cortical semantic network is wired not to encode knowledge of distinct conceptual domains but to promote learning about both conceptual and affordance structure in the environment.

The contributions of vision and haptics to reaching and grasping

This review aims to provide a comprehensive outlook on the sensory (visual and haptic) contributions to reaching and grasping. The focus is on studies in developing children, normal, and neuropsychological populations, and in sensory-deprived individuals. Studies have suggested a right-hand/left-hemisphere specialization for visually guided grasping and a left-hand/right-hemisphere specialization for haptically guided object recognition. This poses the interesting possibility that when vision is not available and grasping relies heavily on the haptic system, there is an advantage to use the left hand. We review the evidence for this possibility and dissect the unique contributions of the visual and haptic systems to grasping. We ultimately discuss how the integration of these two sensory modalities shape hand preference.

How Visual Is the Visual Cortex? Comparing Connectional and Functional Fingerprints between Congenitally Blind and Sighted Individuals

Classical animal visual deprivation studies and human neuroimaging studies have shown that visual experience plays a critical role in shaping the functionality and connectivity of the visual cortex. Interestingly, recent studies have additionally reported circumscribed regions in the visual cortex in which functional selectivity was remarkably similar in individuals with and without visual experience. Here, by directly comparing resting-state and task-based fMRI data in congenitally blind and sighted human subjects, we obtained large-scale continuous maps of the degree to which connectional and functional "fingerprints" of ventral visual cortex depend on visual experience. We found a close agreement between connectional and functional maps, pointing to a strong interdependence of connectivity and function. Visual experience (or the absence thereof) had a pronounced effect on the resting-state connectivity and functional response profile of occipital cortex and the posterior lateral fusiform gyrus. By contrast, connectional and functional fingerprints in the anterior medial and posterior lateral parts of the ventral visual cortex were statistically indistinguishable between blind and sighted individuals. These results provide a large-scale mapping of the influence of visual experience on the development of both functional and connectivity properties of visual cortex, which serves as a basis for the formulation of new hypotheses regarding the functionality and plasticity of specific subregions. Significance statement: How is the functionality and connectivity of the visual cortex shaped by visual experience? By directly comparing resting-state and task-based fMRI data in congenitally blind and sighted subjects, we obtained large-scale continuous maps of the degree to which connectional and functional "fingerprints" of ventral visual cortex depend on visual experience. In addition to revealing regions that are strongly dependent on visual experience (early visual cortex and posterior fusiform gyrus), our results showed regions in which connectional and functional patterns are highly similar in blind and sighted individuals (anterior medial and posterior lateral ventral occipital temporal cortex). These results serve as a basis for the formulation of new hypotheses regarding the functionality and plasticity of specific subregions of the visual cortex.

Development of Tool Representations in the Dorsal and Ventral Visual Object Processing Pathways

Tools represent a special class of objects, because they are processed across both the dorsal and ventral visual object processing pathways. Three core regions are known to be involved in tool processing: the left posterior middle temporal gyrus, the medial fusiform gyrus (bilaterally), and the left inferior parietal lobule. A critical and relatively unexplored issue concerns whether, in development, tool preferences emerge at the same time and to a similar degree across all regions of the tool-processing network. To test this issue, we used functional magnetic resonance imaging to measure the neural amplitude, peak location, and the dispersion of tool-related neural responses in the youngest sample of children tested to date in this domain (ages 4-8 years). We show that children recruit overlapping regions of the adult tool-processing network and also exhibit similar patterns of co-activation across the network to adults. The amplitude and co-activation data show that the core components of the tool-processing network are established by age 4. Our findings on the distributions of peak location and dispersion of activation indicate that the tool network undergoes refinement between ages 4 and 8 years.

Functional connectivity of visual cortex in the blind follows retinotopic organization principles

Although early visual experience is essential for the proper development of visual cortex, Striem-Amit et al. show that the underlying connectivity structure of retinotopic mapping is retained even in congenitally blind individuals. This basic organisational principle emerges independently of visual input and persists despite lifelong experience-dependent plasticity.

Is visual input during critical periods of development crucial for the emergence of the fundamental topographical mapping of the visual cortex? And would this structure be retained throughout life-long blindness or would it fade as a result of plastic, use-based reorganization? We used functional connectivity magnetic resonance imaging based on intrinsic blood oxygen level-dependent fluctuations to investigate whether significant traces of topographical mapping of the visual scene in the form of retinotopic organization, could be found in congenitally blind adults. A group of 11 fully and congenitally blind subjects and 18 sighted controls were studied. The blind demonstrated an intact functional connectivity network structural organization of the three main retinotopic mapping axes: eccentricity (centre-periphery), laterality (left-right), and elevation (upper-lower) throughout the retinotopic cortex extending to high-level ventral and dorsal streams, including characteristic eccentricity biases in face- and house-selective areas. Functional connectivity-based topographic organization in the visual cortex was indistinguishable from the normally sighted retinotopic functional connectivity structure as indicated by clustering analysis, and was found even in participants who did not have a typical retinal development in utero (microphthalmics). While the internal structural organization of the visual cortex was strikingly similar, the blind exhibited profound differences in functional connectivity to other (non-visual) brain regions as compared to the sighted, which were specific to portions of V1. Central V1 was more connected to language areas but peripheral V1 to spatial attention and control networks. These findings suggest that current accounts of critical periods and experience-dependent development should be revisited even for primary sensory areas, in that the connectivity basis for visual cortex large-scale topographical organization can develop without any visual experience and be retained through life-long experience-dependent plasticity. Furthermore, retinotopic divisions of labour, such as that between the visual cortex regions normally representing the fovea and periphery, also form the basis for topographically-unique plastic changes in the blind.

Spontaneous object and movement representations in 4-month-old human infants and albino Swiss mice

Can young infants decompose visual events into independent representations of objects and movements? Previous studies suggest that human infants may be born with the notion of objects but there is little evidence for movement representations during the first months of life. We devised a novel Rapid Visual Recognition Procedure to test whether the nervous system is innately disposed for the conceptual decomposition of visual events. We show that 4-month-old infants can spontaneously build object and movement representations and recognize these in partially matching test events. Also albino Swiss mice that were tested on a comparable procedure could spontaneously build detailed mental representations of moving objects. Our results dissociate the ability to conceptually decompose physical events into objects and spatio-temporal relations from various types of human and non-human specific experience, and suggest that the nervous system is genetically predisposed to anticipate the representation of objects and movements in both humans and non-human species.

The Animacy Continuum in the Human Ventral Vision Pathway

Major theories for explaining the organization of semantic memory in the human brain are premised on the often-observed dichotomous dissociation between living and nonliving objects. Evidence from neuroimaging has been interpreted to suggest that this distinction is reflected in the functional topography of the ventral vision pathway as lateral-to-medial activation gradients. Recently, we observed that similar activation gradients also reflect differences among living stimuli consistent with the semantic dimension of graded animacy. Here, we address whether the salient dichotomous distinction between living and nonliving objects is actually reflected in observable measured brain activity or whether previous observations of a dichotomous dissociation were the illusory result of stimulus sampling biases. Using fMRI, we measured neural responses while participants viewed 10 animal species with high to low animacy and two inanimate categories. Representational similarity analysis of the activity in ventral vision cortex revealed a main axis of variation with high-animacy species maximally different from artifacts and with the least animate species closest to artifacts. Although the associated functional topography mirrored activation gradients observed for animate–inanimate contrasts, we found no evidence for a dichotomous dissociation. We conclude that a central organizing principle of human object vision corresponds to the graded psychological property of animacy with no clear distinction between living and nonliving stimuli. The lack of evidence for a dichotomous dissociation in the measured brain activity challenges theories based on this premise.

A lack of experience-dependent plasticity after more than a decade of recovered sight

In 2000, monocular vision was restored to M. M., who had been blind between the ages of 3 and 46 years. Tests carried out over 2 years following the surgery revealed impairments of 3-D form, object, and face processing and an absence of object- and face-selective blood-oxygen-level-dependent responses in ventral visual cortex. In the present research, we reexamined M. M. to test for experience-dependent recovery of visual function. Behaviorally, M. M. remains impaired in 3-D form, object, and face processing. Accordingly, we found little to no evidence of the category-selective organization within ventral visual cortex typically associated with face, body, scene, or object processing. We did observe remarkably normal object selectivity within lateral occipital cortex, consistent with M. M.'s previously reported shape-discrimination performance. Together, these findings provide little evidence for recovery of high-level visual function after more than a decade of visual experience in adulthood.

A number-form area in the blind

Distinct preference for visual number symbols was recently discovered in the human right inferior temporal gyrus (rITG). It remains unclear how this preference emerges, what is the contribution of shape biases to its formation and whether visual processing underlies it. Here we use congenital blindness as a model for brain development without visual experience. During fMRI, we present blind subjects with shapes encoded using a novel visual-to-music sensory-substitution device (The EyeMusic). Greater activation is observed in the rITG when subjects process symbols as numbers compared with control tasks on the same symbols. Using resting-state fMRI in the blind and sighted, we further show that the areas with preference for numerals and letters exhibit distinct patterns of functional connectivity with quantity and language-processing areas, respectively. Our findings suggest that specificity in the ventral ‘visual’ stream can emerge independently of sensory modality and visual experience, under the influence of distinct connectivity patterns.

The human visual cortex includes areas with preference for various object categories. Here, Abboud et al. demonstrate using visual-to-music substitution, that the congenitally blind show a similar preference for numerals in the right inferior temporal cortex as sighted individuals, despite having no visual experience.

Inborn and experience-dependent models of categorical brain organization. A position paper

The present review aims to summarize the debate in contemporary neuroscience between inborn and experience-dependent models of conceptual representations that goes back to the description of category-specific semantic disorders for biological and artifact categories. Experience-dependent models suggest that categorical disorders are the by-product of the differential weighting of different sources of knowledge in the representation of biological and artifact categories. These models maintain that semantic disorders are not really category-specific, because they do not respect the boundaries between different categories. They also argue that the brain structures which are disrupted in a given type of category-specific semantic disorder should correspond to the areas of convergence of the sensory-motor information which play a major role in the construction of that category. Furthermore, they provide a simple interpretation of gender-related categorical effects and are supported by studies assessing the importance of prior experience in the cortical representation of objects On the other hand, inborn models maintain that category-specific semantic disorders reflect the disruption of innate brain networks, which are shaped by natural selection to allow rapid identification of objects that are very relevant for survival. From the empirical point of view, these models are mainly supported by observations of blind subjects, which suggest that visual experience is not necessary for the emergence of category-specificity in the ventral stream of visual processing. The weight of the data supporting experience-dependent and inborn models is thoroughly discussed, stressing the fact observations made in blind subjects are still the subject of intense debate. It is concluded that at the present state of knowledge it is not possible to choose between experience-dependent and inborn models of conceptual representations.

The Representation of Object-Directed Action and Function Knowledge in the Human Brain

The appropriate use of everyday objects requires the integration of action and function knowledge. Previous research suggests that action knowledge is represented in frontoparietal areas while function knowledge is represented in temporal lobe regions. Here we used multivoxel pattern analysis to investigate the representation of object-directed action and function knowledge while participants executed pantomimes of familiar tool actions. A novel approach for decoding object knowledge was used in which classifiers were trained on one pair of objects and then tested on a distinct pair; this permitted a measurement of classification accuracy over and above object-specific information. Region of interest (ROI) analyses showed that object-directed actions could be decoded in tool-preferring regions of both parietal and temporal cortex, while no independently defined tool-preferring ROI showed successful decoding of object function. However, a whole-brain searchlight analysis revealed that while frontoparietal motor and peri-motor regions are engaged in the representation of object-directed actions, medial temporal lobe areas in the left hemisphere are involved in the representation of function knowledge. These results indicate that both action and function knowledge are represented in a topographically coherent manner that is amenable to study with multivariate approaches, and that the left medial temporal cortex represents knowledge of object function.

Tracking the evolution of crossmodal plasticity and visual functions before and after sight restoration

Visual deprivation leads to massive reorganization in both the structure and function of the occipital cortex, raising crucial challenges for sight restoration. We tracked the behavioral, structural, and neurofunctional changes occurring in an early and severely visually impaired patient before and 1.5 and 7 mo after sight restoration with magnetic resonance imaging. Robust presurgical auditory responses were found in occipital cortex despite residual preoperative vision. In primary visual cortex, crossmodal auditory responses overlapped with visual responses and remained elevated even 7 mo after surgery. However, these crossmodal responses decreased in extrastriate occipital regions after surgery, together with improved behavioral vision and with increases in both gray matter density and neural activation in low-level visual regions. Selective responses in high-level visual regions involved in motion and face processing were observable even before surgery and did not evolve after surgery. Taken together, these findings demonstrate that structural and functional reorganization of occipital regions are present in an individual with a long-standing history of severe visual impairment and that such reorganizations can be partially reversed by visual restoration in adulthood.

Anatomical and functional plasticity in early blind individuals and the mixture of experts architecture

As described elsewhere in this special issue, recent advances in neuroimaging over the last decade have led to a rapid expansion in our knowledge of anatomical and functional correlations within the normal and abnormal human brain. Here, we review how early blindness has been used as a model system for examining the role of visual experience in the development of anatomical connections and functional responses. We discuss how lack of power in group comparisons may provide a potential explanation for why extensive anatomical changes in cortico-cortical connectivity are not observed. Finally we suggest a framework-cortical specialization via hierarchical mixtures of experts-which offers some promise in reconciling a wide range of functional and anatomical data.

Picturing words? Sensorimotor cortex activation for printed words in child and adult readers

Learning to read involves associating abstract visual shapes with familiar meanings. Embodiment theories suggest that word meaning is at least partially represented in distributed sensorimotor networks in the brain (Barsalou, 2008; Pulvermueller, 2013). We explored how reading comprehension develops by tracking when and how printed words start activating these "semantic" sensorimotor representations as children learn to read. Adults and children aged 7-10 years showed clear category-specific cortical specialization for tool versus animal pictures during a one-back categorisation task. Thus, sensorimotor representations for these categories were in place at all ages. However, co-activation of these same brain regions by the visual objects' written names was only present in adults, even though all children could read and comprehend all presented words, showed adult-like task performance, and older children were proficient readers. It thus takes years of training and expert reading skill before spontaneous processing of printed words' sensorimotor meanings develops in childhood.

Auditory motion processing after early blindness

Studies showing that occipital cortex responds to auditory and tactile stimuli after early blindness are often interpreted as demonstrating that early blind subjects "see" auditory and tactile stimuli. However, it is not clear whether these occipital responses directly mediate the perception of auditory/tactile stimuli, or simply modulate or augment responses within other sensory areas. We used fMRI pattern classification to categorize the perceived direction of motion for both coherent and ambiguous auditory motion stimuli. In sighted individuals, perceived motion direction was accurately categorized based on neural responses within the planum temporale (PT) and right lateral occipital cortex (LOC). Within early blind individuals, auditory motion decisions for both stimuli were successfully categorized from responses within the human middle temporal complex (hMT+), but not the PT or right LOC. These findings suggest that early blind responses within hMT+ are associated with the perception of auditory motion, and that these responses in hMT+ may usurp some of the functions of nondeprived PT. Thus, our results provide further evidence that blind individuals do indeed "see" auditory motion.

Neural basis of language switching in the brain: fMRI evidence from Korean-Chinese early bilinguals

Using fMRI, we conducted two types of property generation task that involved language switching, with early bilingual speakers of Korean and Chinese. The first is a more conventional task in which a single language (L1 or L2) was used within each trial, but switched randomly from trial to trial. The other consists of a novel experimental design where language switching happens within each trial, alternating in the direction of the L1/L2 translation required. Our findings support a recently introduced cognitive model, the 'hodological' view of language switching proposed by Moritz-Gasser and Duffau. The nodes of a distributed neural network that this model proposes are consistent with the informative regions that we extracted in this study, using both GLM methods and Multivariate Pattern Analyses: the supplementary motor area, caudate, supramarginal gyrus and fusiform gyrus and other cortical areas.

The brain network underlying the recognition of hand gestures in the blind: the supramodal role of the extrastriate body area

The visual perception of others' body parts is critical for understanding and imitating their behavior. The visual cortex in humans includes the extrastriate body area (EBA), which is a large portion of the occipitotemporal cortex that is selectively responsive to visually perceived body parts. Previous neuroimaging studies showed that the EBA not only receives sensory inputs regarding others' body information but also receives kinesthetic feedback regarding one's own actions. This finding raised the possibility that the EBA could be formed via nonvisual sensory modalities. However, the effect of visual deprivation on the formation of the EBA has remained largely unknown. Here, we used fMRI to investigate the effect of vision loss on the development of the EBA. Blind and sighted human subjects performed equally well in a haptic-identification task involving three categories of objects (hand shapes, toy cars, and teapots). The superior part (i.e., the middle temporal gyrus and angular gyrus) of the EBA and the supramarginal gyrus showed greater sensitivity to recognized hand shapes than to inanimate objects, regardless of the sensory modality and visual experience. Unlike the superior part of the EBA, the sensitivity of the inferior part (i.e., the inferior temporal sulcus and middle occipital gyrus) depended on visual experience. However, this vision-dependent sensitivity explained minor individual differences in hand-recognition performance. These results indicate that nonvisual modalities drive the development of the cortical network underlying the recognition of hand gestures with a node in the visual cortex.

Noun and knowledge retrieval for biological and non-biological entities following right occipitotemporal lesions

We investigated the critical contribution of right ventral occipitotemporal cortex to knowledge of visual and functional-associative attributes of biological and non-biological entities and how this relates to category-specificity during confrontation naming. In a consecutive series of 7 patients with lesions confined to right ventral occipitotemporal cortex, we conducted an extensive assessment of oral generation of visual-sensory and functional-associative features in response to the names of biological and nonbiological entities. Subjects also performed a confrontation naming task for these categories. Our main novel finding related to a unique case with a small lesion confined to right medial fusiform gyrus who showed disproportionate naming impairment for nonbiological versus biological entities, specifically for tools. Generation of visual and functional-associative features was preserved for biological and non-biological entities. In two other cases, who had a relatively small posterior lesion restricted to primary visual and posterior fusiform cortex, retrieval of visual attributes was disproportionately impaired compared to functional-associative attributes, in particular for biological entities. However, these cases did not show a category-specific naming deficit. Two final cases with the largest lesions showed a classical dissociation between biological versus nonbiological entities during naming, with normal feature generation performance. This is the first lesion-based evidence of a critical contribution of the right medial fusiform cortex to tool naming. Second, dissociations along the dimension of attribute type during feature generation do not co-occur with category-specificity during naming in the current patient sample.

The functional architecture of the ventral temporal cortex and its role in categorization

Visual categorization is thought to occur in the human ventral temporal cortex (VTC), but how this categorization is achieved is still largely unknown. In this Review, we consider the computations and representations that are necessary for categorization and examine how the microanatomical and macroanatomical layout of the VTC might optimize them to achieve rapid and flexible visual categorization. We propose that efficient categorization is achieved by organizing representations in a nested spatial hierarchy in the VTC. This spatial hierarchy serves as a neural infrastructure for the representational hierarchy of visual information in the VTC and thereby enables flexible access to category information at several levels of abstraction.

Decoding neural representational spaces using multivariate pattern analysis

A major challenge for systems neuroscience is to break the neural code. Computational algorithms for encoding information into neural activity and extracting information from measured activity afford understanding of how percepts, memories, thought, and knowledge are represented in patterns of brain activity. The past decade and a half has seen significant advances in the development of methods for decoding human neural activity, such as multivariate pattern classification, representational similarity analysis, hyperalignment, and stimulus-model-based encoding and decoding. This article reviews these advances and integrates neural decoding methods into a common framework organized around the concept of high-dimensional representational spaces.

The blind brain: How (lack of) vision shapes the morphological and functional architecture of the human brain

Since the early days, how we represent the world around us has been a matter of philosophical speculation. Over the last few decades, modern neuroscience, and specifically the development of methodologies for the structural and the functional exploration of the brain have made it possible to investigate old questions with an innovative approach. In this brief review, we discuss the main findings from a series of brain anatomical and functional studies conducted in sighted and congenitally blind individuals by our’s and others' laboratories. Historically, research on the ‘blind brain’ has focused mainly on the cross-modal plastic changes that follow sensory deprivation. More recently, a novel line of research has been developed to determine to what extent visual experience is truly required to achieve a representation of the surrounding environment. Overall, the results of these studies indicate that most of the brain fine morphological and functional architecture is programmed to develop and function independently from any visual experience. Distinct cortical areas are able to process information in a supramodal fashion, that is, independently from the sensory modality that carries that information to the brain. These observations strongly support the hypothesis of a modality-independent, i.e. more abstract, cortical organization, and may contribute to explain how congenitally blind individuals may interact efficiently with an external world that they have never seen.

Domain-specific genetic influence on visual-ambiguity resolution

The visual world is flooded with ambiguity. Generally, people can resolve the ambiguity almost instantaneously, as when they distinguish at a glance whether a maiden in a portrait by Picasso is in profile or facing front. However, perception of the same reality, though relatively stable at the individual level, can vary dramatically from person to person, manifesting idiosyncratic perceptual biases. What drives the heterogeneity of human vision as reflected in the resolution of visual ambiguity? Using the twin method, we demonstrated a significant genetic contribution to individual differences in the visual disambiguation of bistable biological-motion stimuli but not inanimate motion stimuli. These findings challenge the prevailing view that the way the human brain makes sense of visual input is largely shaped by a person's perceptual history. Rather, the visual perception of biologically salient information can be guided by adaptive mental "priors" that are genetically transmitted.

Object-specific semantic coding in human perirhinal cortex

Category-specificity has been demonstrated in the human posterior ventral temporal cortex for a variety of object categories. Although object representations within the ventral visual pathway must be sufficiently rich and complex to support the recognition of individual objects, little is known about how specific objects are represented. Here, we used representational similarity analysis to determine what different kinds of object information are reflected in fMRI activation patterns and uncover the relationship between categorical and object-specific semantic representations. Our results show a gradient of informational specificity along the ventral stream from representations of image-based visual properties in early visual cortex, to categorical representations in the posterior ventral stream. A key finding showed that object-specific semantic information is uniquely represented in the perirhinal cortex, which was also increasingly engaged for objects that are more semantically confusable. These findings suggest a key role for the perirhinal cortex in representing and processing object-specific semantic information that is more critical for highly confusable objects. Our findings extend current distributed models by showing coarse dissociations between objects in posterior ventral cortex, and fine-grained distinctions between objects supported by the anterior medial temporal lobes, including the perirhinal cortex, which serve to integrate complex object information.

Empirically grounding grounded cognition: the case of color

Grounded cognition theories hold that the neural states involved in experiencing objects play a direct functional role in representing and accessing object knowledge from memory. However, extant data marshaled to support this view are also consistent with an opposing view that perceptuo-motor activations occur only following access to knowledge from amodal memory systems. We provide novel discriminating evidence for the functional involvement of visuo-perceptual states specifically in accessing knowledge about an object's color. We recorded event-related brain potentials (ERPs) while manipulating the visual contrast of monochromatic words ("lime") in a semantic decision task: responses were made for valid colors ("green") and locations ("kitchen") and withheld for invalid colors and locations. Low contrast delayed perceptual processing for both color and location. Critically, low contrast slowed access to color knowledge only. This finding reveals that the visual system plays a functional role in accessing object knowledge and uniquely supports grounded views of cognition.

A bimodal tuning curve for spatial frequency across left and right human orbital frontal cortex during object recognition

Orbital frontal cortex (OFC) is known to play a role in object recognition by generating "first-pass" hypotheses about the identity of naturalistic images based on low spatial frequency (SF) information. These hypotheses are evaluated by more detailed (and slower) ventral visual pathway processes. While it has been suggested on theoretical grounds, it remains unknown whether OFC also receives postrecognition feedback about stimulus identity. We used a novel paradigm in the context of functional magnetic resonance imaging that permits the first few hundred milliseconds of object recognition to be spread out over 120 s. OFC shows a robust response to low and relatively high SFs, whereas ventral stream regions display unimodal response distributions shifted toward high SFs. These findings in OFC were modulated by hemisphere, with right OFC differentially responding to low SFs and left OFC differentially responding to high SFs. Psychophysical experiments confirmed that the same ranges of SFs preferred by ventral stream regions are critical for determining the accuracy and speed of object recognition. Our findings indicate that OFC accesses global form (low SF information, right OFC) and object identity (high SF information, left OFC), and suggest that OFC receives feedback about the accuracy of its initial hypothesis regarding stimulus identity.

The origin and function of mirror neurons: the missing link

We argue, by analogy to the neural organization of the object recognition system, that demonstration of modulation of mirror neurons by associative learning does not imply absence of genetic adaptation. Innate connectivity defines the types of processes mirror neurons can participate in while allowing for extensive local plasticity. However, the proper function of these neurons remains to be worked out.

Functional Connectivity Density in Congenitally and Late Blind Subjects

Visual deprivation during different developmental periods leads to different structural and functional alterations in the brain; however, the effects of visual deprivation on the spontaneous functional organization of the brain remain largely unknown. In this study, we used voxel-based functional connectivity density (FCD) analyses to investigate the effects of visual deprivation during different developmental periods on the spontaneous functional organization of the brain. Compared with the sighted controls (SC), both the congenitally blind (CB) and the late blind (LB) exhibited decreased short- and long-range FCDs in the primary visual cortex (V1) and decreased long-range FCDs in the primary somatosensory and auditory cortices. Although both the CB and LB exhibited increased short-range FCD in the dorsal visual stream, the CB exhibited greater increases in the short- and long-range FCDs in the ventral visual stream and hippocampal complex compared with the LB. Moreover, the short-range FCD of the left V1 exhibited a significant positive correlation with the duration of blindness in the LB. Our findings suggest that visual deprivation before the developmental sensitive period can induce more extensive brain functional reorganization than does visual deprivation after the sensitive period, which may underlie an enhanced capacity for processing nonvisual information in the CB.

Brain systems mediating voice identity processing in blind humans

Blind people rely more on vocal cues when they recognize a person's identity than sighted people. Indeed, a number of studies have reported better voice recognition skills in blind than in sighted adults. The present functional magnetic resonance imaging study investigated changes in the functional organization of neural systems involved in voice identity processing following congenital blindness. A group of congenitally blind individuals and matched sighted control participants were tested in a priming paradigm, in which two voice stimuli (S1, S2) were subsequently presented. The prime (S1) and the target (S2) were either from the same speaker (person-congruent voices) or from two different speakers (person-incongruent voices). Participants had to classify the S2 as either a old or a young person. Person-incongruent voices (S2) compared with person-congruent voices elicited an increased activation in the right anterior fusiform gyrus in congenitally blind individuals but not in matched sighted control participants. In contrast, only matched sighted controls showed a higher activation in response to person-incongruent compared with person-congruent voices (S2) in the right posterior superior temporal sulcus. These results provide evidence for crossmodal plastic changes of the person identification system in the brain after visual deprivation.

Visual cortex extrastriate body-selective area activation in congenitally blind people "seeing" by using sounds

Vision is by far the most prevalent sense for experiencing others' body shapes, postures, actions, and intentions, and its congenital absence may dramatically hamper body-shape representation in the brain. We investigated whether the absence of visual experience and limited exposure to others' body shapes could still lead to body-shape selectivity. We taught congenitally fully-blind adults to perceive full-body shapes conveyed through a sensory-substitution algorithm topographically translating images into soundscapes [1]. Despite the limited experience of the congenitally blind with external body shapes (via touch of close-by bodies and for ~10 hr via soundscapes), once the blind could retrieve body shapes via soundscapes, they robustly activated the visual cortex, specifically the extrastriate body area (EBA; [2]). Furthermore, body selectivity versus textures, objects, and faces in both the blind and sighted control groups was not found in the temporal (auditory) or parietal (somatosensory) cortex but only in the visual EBA. Finally, resting-state data showed that the blind EBA is functionally connected to the temporal cortex temporal-parietal junction/superior temporal sulcus Theory-of-Mind areas [3]. Thus, the EBA preference is present without visual experience and with little exposure to external body-shape information, supporting the view that the brain has a sensory-independent, task-selective supramodal organization rather than a sensory-specific organization.

Body and object effectors: the organization of object representations in high-level visual cortex reflects body-object interactions

The principles driving the functional organization of object representations in high-level visual cortex are not yet fully understood. In four human fMRI experiments, we provide evidence that the organization of high-level visual cortex partly reflects the degree to which objects are typically controlled by the body to interact with the world, thereby extending the body's boundaries. Univariate whole-brain analysis showed an overlap between responses to body effectors (e.g., hands, feet, and limbs) and object effectors (e.g., hammers, combs, and tennis rackets) in lateral occipitotemporal cortex (LOTC) and parietal cortex. Region of interest analyses showed that a hand-selective region in left LOTC responded selectively to object effectors relative to a range of noneffector object control conditions (e.g., graspable objects, "act-on" objects, musical instruments). Object ratings showed that the strong response to object effectors in hand-selective LOTC was not due to general action-related object properties shared with these control conditions, such as hand priming, hand grasping, and hand-action centrality. Finally, whole-brain representational similarity analysis revealed that the similarity of multivoxel object response patterns in left lateral occipitotemporal cortex selectively predicted the degree to which objects were rated as being controlled by and extending the body. Together, these results reveal a clustering of body and object effector representations, indicating that the organization of object representations in high-level visual cortex partly reflects how objects relate to the body.

Nonvisual and visual object shape representations in occipitotemporal cortex: evidence from congenitally blind and sighted adults

Knowledge of object shape is primarily acquired through the visual modality but can also be acquired through other sensory modalities. In the present study, we investigated the representation of object shape in humans without visual experience. Congenitally blind and sighted participants rated the shape similarity of pairs of 33 familiar objects, referred to by their names. The resulting shape similarity matrices were highly similar for the two groups, indicating that knowledge of the objects' shapes was largely independent of visual experience. Using fMRI, we tested for brain regions that represented object shape knowledge in blind and sighted participants. Multivoxel activity patterns were established for each of the 33 aurally presented object names. Sighted participants additionally viewed pictures of these objects. Using representational similarity analysis, neural similarity matrices were related to the behavioral shape similarity matrices. Results showed that activity patterns in occipitotemporal cortex (OTC) regions, including inferior temporal (IT) cortex and functionally defined object-selective cortex (OSC), reflected the behavioral shape similarity ratings in both blind and sighted groups, also when controlling for the objects' tactile and semantic similarity. Furthermore, neural similarity matrices of IT and OSC showed similarities across blind and sighted groups (within the auditory modality) and across modality (within the sighted group), but not across both modality and group (blind auditory-sighted visual). Together, these findings provide evidence that OTC not only represents objects visually (requiring visual experience) but also represents objects nonvisually, reflecting knowledge of object shape independently of the modality through which this knowledge was acquired.

Sensory substitution: closing the gap between basic research and widespread practical visual rehabilitation

Sensory substitution devices (SSDs) have come a long way since first developed for visual rehabilitation. They have produced exciting experimental results, and have furthered our understanding of the human brain. Unfortunately, they are still not used for practical visual rehabilitation, and are currently considered as reserved primarily for experiments in controlled settings. Over the past decade, our understanding of the neural mechanisms behind visual restoration has changed as a result of converging evidence, much of which was gathered with SSDs. This evidence suggests that the brain is more than a pure sensory-machine but rather is a highly flexible task-machine, i.e., brain regions can maintain or regain their function in vision even with input from other senses. This complements a recent set of more promising behavioral achievements using SSDs and new promising technologies and tools. All these changes strongly suggest that the time has come to revive the focus on practical visual rehabilitation with SSDs and we chart several key steps in this direction such as training protocols and self-train tools.

Controversies over the mechanisms underlying the crucial role of the left fronto-parietal areas in the representation of tools

Anatomo-clinical and neuroimaging data show that the left fronto-parietal areas play an important role in representing tools. As manipulation is an important source of knowledge about tools, it has been assumed that motor activity explains the link between tool knowledge and the left fronto-parietal areas. However, controversies exist over the exact mechanisms underlying this relationship. According to a strong version of the “embodied cognition theory,” activation of a tool concept necessarily involves re-enactment of the corresponding kind of action. Impairment of the ability to use tools should, therefore, lead to impairment of tool knowledge. Both the “domains of knowledge hypothesis” and the “sensory-motor model of conceptual knowledge” refute the strong version of the “embodied cognition hypothesis” but acknowledge that manipulation and other action schemata play an important role in our knowledge of tools. The basic difference between these two models is that the former is based on an innate model and the latter holds that the brain’s organization of categories is experience dependent. Data supporting and arguing against each of these models are briefly reviewed. In particular, the following lines of research, which argue against the innate nature of the brain’s categorical organization, are discussed: (1) the observation that in patients with category-specific disorders the semantic impairment does not respect the boundaries between biological entities and artifact items; (2) data showing that experience-driven neuroplasticity in musicians is not confined to alterations of perceptual and motor maps but also leads to the establishment of higher-level semantic representations for musical instruments; (3) results of experiments using previously unfamiliar materials showing that the history of our sensory-motor experience with an object significantly affects its neural representation.

Sensitive and critical periods in visual sensory deprivation

While the demonstration of crossmodal plasticity is well established in congenital and early blind individuals, great debate still surrounds whether those who acquire blindness later in life can also benefit from such compensatory changes. No proper consensus has been reached despite the fact that a proper understanding of the developmental time course of these changes, and whether their occurrence is limited to—or within—specific time windows, is crucial to our understanding of the crossmodal phenomena. An extensive review of the literature reveals that while the majority of investigations to date have examined the crossmodal plasticity available to late blind individuals in quantitative terms, recent findings rather suggest that this reorganization also likely changes qualitatively compared to what is observed in early blindness. This obviously could have significant repercussions not only for the training and rehabilitation of blind individuals, but for the development of appropriate neuroprostheses designed to aid and potentially restore vision. Important parallels will also be drawn with the current state of research on deafness, which is particularly relevant given in the development of successful neuroprostheses (e.g., cochlear implants) for providing auditory input to the central nervous system otherwise aurally deafferented. Lastly, this paper will address important inconsistencies across the literature concerning the definition of distinct blind groups based on the age of blindness onset, and propose several alternatives to using such a categorization.

The mid-fusiform sulcus: a landmark identifying both cytoarchitectonic and functional divisions of human ventral temporal cortex

Human ventral temporal cortex (VTC) plays a pivotal role in high-level vision. An under-studied macroanatomical feature of VTC is the mid-fusiform sulcus (MFS), a shallow longitudinal sulcus separating the lateral and medial fusiform gyrus (FG). Here, we quantified the morphological features of the MFS in 69 subjects (ages 7-40), and investigated its relationship to both cytoarchitectonic and functional divisions of VTC with four main findings. First, despite being a minor sulcus, we found that the MFS is a stable macroanatomical structure present in all 138 hemispheres with morphological characteristics developed by age 7. Second, the MFS is the locus of a lateral-medial cytoarchitectonic transition within the posterior FG serving as the boundary between cytoarchitectonic regions FG1 and FG2. Third, the MFS predicts a lateral-medial functional transition in eccentricity bias representations in children, adolescents, and adults. Fourth, the anterior tip of the MFS predicts the location of a face-selective region, mFus-faces/FFA-2. These findings are the first to illustrate that a macroanatomical landmark identifies both cytoarchitectonic and functional divisions of high-level sensory cortex in humans and have important implications for understanding functional and structural organization in the human brain.

Tripartite organization of the ventral stream by animacy and object size

Occipito-temporal cortex is known to house visual object representations, but the organization of the neural activation patterns along this cortex is still being discovered. Here we found a systematic, large-scale structure in the neural responses related to the interaction between two major cognitive dimensions of object representation: animacy and real-world size. Neural responses were measured with functional magnetic resonance imaging while human observers viewed images of big and small animals and big and small objects. We found that real-world size drives differential responses only in the object domain, not the animate domain, yielding a tripartite distinction in the space of object representation. Specifically, cortical zones with distinct response preferences for big objects, all animals, and small objects, are arranged in a spoked organization around the occipital pole, along a single ventromedial, to lateral, to dorsomedial axis. The preference zones are duplicated on the ventral and lateral surface of the brain. Such a duplication indicates that a yet unknown higher-order division of labor separates object processing into two substreams of the ventral visual pathway. Broadly, we suggest that these large-scale neural divisions reflect the major joints in the representational structure of objects and thus place informative constraints on the nature of the underlying cognitive architecture.

Right occipital cortex activation correlates with superior odor processing performance in the early blind

Using functional magnetic resonance imaging (fMRI) in ten early blind humans, we found robust occipital activation during two odor-processing tasks (discrimination or categorization of fruit and flower odors), as well as during control auditory-verbal conditions (discrimination or categorization of fruit and flower names). We also found evidence for reorganization and specialization of the ventral part of the occipital cortex, with dissociation according to stimulus modality: the right fusiform gyrus was most activated during olfactory conditions while part of the left ventral lateral occipital complex showed a preference for auditory-verbal processing. Only little occipital activation was found in sighted subjects, but the same right-olfactory/left-auditory-verbal hemispheric lateralization was found overall in their brain. This difference between the groups was mirrored by superior performance of the blind in various odor-processing tasks. Moreover, the level of right fusiform gyrus activation during the olfactory conditions was highly correlated with individual scores in a variety of odor recognition tests, indicating that the additional occipital activation may play a functional role in odor processing.

The emergence of semantic meaning in the ventral temporal pathway

In the ventral visual pathway, early visual areas encode light patterns on the retina in terms of image properties, for example, edges and color, whereas higher areas encode visual information in terms of objects and categories. At what point does semantic knowledge, as instantiated in human language, emerge? We examined this question by studying whether semantic similarity in language relates to the brain's organization of object representations in inferior temporal cortex (ITC), an area of the brain at the crux of several proposals describing how the brain might represent conceptual knowledge. Semantic relationships among words can be viewed as a geometrical structure with some pairs of words close in their meaning (e.g., man and boy) and other pairs more distant (e.g., man and tomato). ITC's representation of objects similarly can be viewed as a complex structure with some pairs of stimuli evoking similar patterns of activation (e.g., man and boy) and other pairs evoking very different patterns (e.g., man and tomato). In this study, we examined whether the geometry of visual object representations in ITC bears a correspondence to the geometry of semantic relationships between word labels used to describe the objects. We compared ITC's representation to semantic structure, evaluated by explicit ratings of semantic similarity and by five computational measures of semantic similarity. We show that the representational geometry of ITC-but not of earlier visual areas (V1)-is reflected both in explicit behavioral ratings of semantic similarity and also in measures of semantic similarity derived from word usage patterns in natural language. Our findings show that patterns of brain activity in ITC not only reflect the organization of visual information into objects but also represent objects in a format compatible with conceptual thought and language.