Tripartite organization of the ventral stream by animacy and object size

Semantic and emotional content of imagined representations in human occipitotemporal cortex

Mental imagery is a critical cognitive function, clinically important, but poorly understood. When visual objects are perceived, many of their sensory, semantic and emotional properties are represented in occipitotemporal cortex. Visual imagery has been found to activate some of the same brain regions, but it was not known what properties are re-created in these regions during imagery. We therefore examined the representation during imagery for two stimuli in depth, by comparing the pattern of fMRI response to the patterns evoked by the perception of 200 diverse objects chosen to de-correlate their properties. Real-time, adaptive stimulus selection allowed efficient sampling of this broad stimulus space. Our experiments show that occipitotemporal cortex, which encoded sensory, semantic and emotional properties during perception, can robustly represent semantic and emotional properties during imagery, but that these representations depend on the object being imagined and on individual differences in style and reported vividness of imagery.

Neural Decoding and "Inner" Psychophysics: A Distance-to-Bound Approach for Linking Mind, Brain, and Behavior

A fundamental challenge for cognitive neuroscience is characterizing how the primitives of psychological theory are neurally implemented. Attempts to meet this challenge are a manifestation of what Fechner called "inner" psychophysics: the theory of the precise mapping between mental quantities and the brain. In his own time, inner psychophysics remained an unrealized ambition for Fechner. We suggest that, today, multivariate pattern analysis (MVPA), or neural "decoding," methods provide a promising starting point for developing an inner psychophysics. A cornerstone of these methods are simple linear classifiers applied to neural activity in high-dimensional activation spaces. We describe an approach to inner psychophysics based on the shared architecture of linear classifiers and observers under decision boundary models such as signal detection theory. Under this approach, distance from a decision boundary through activation space, as estimated by linear classifiers, can be used to predict reaction time in accordance with signal detection theory, and distance-to-bound models of reaction time. Our "neural distance-to-bound" approach is potentially quite general, and simple to implement. Furthermore, our recent work on visual object recognition suggests it is empirically viable. We believe the approach constitutes an important step along the path to an inner psychophysics that links mind, brain, and behavior.

Conceptual size representation in ventral visual cortex

Recent findings suggest that visual objects may be mapped along the ventral occipitotemporal cortex according to their real-world size (Konkle and Oliva, 2012). It has been argued that such mapping does not reflect an abstract, conceptual size representation, but rather the visual or functional properties associated with small versus big real-world objects. To determine whether a more abstract conceptual size representation may affect visual cortical activation we used meaningless geometrical shapes, devoid of semantic or functional associations, which were associated with specific size representations by virtue of extensive training. Following training, participants underwent functional magnetic resonance imaging (fMRI) scanning while performing a conceptual size comparison task on the geometrical shapes. In addition, a size comparison task was conducted for numeral digits denoting small and big numbers. A region-of-interest analysis revealed larger blood oxygenation level dependent (BOLD) responses for conceptually 'big' than for conceptually 'small' shapes, as well as for big versus small numbers, within medial (parahippocampal place area, PPA) and lateral (occipital place area, OPA) place-selective regions. Processing of the 'big' visual shapes further elicited enhanced activation in early visual cortex, possibly reflecting top-down projections from PPA. By using arbitrary shapes and numbers we minimized visual, categorical, or functional influences on fMRI measurement, providing evidence for a possible neural mechanism underlying the representation of abstract conceptual size within the ventral visual stream.

Neural evidence that three dimensions organize mental state representation: Rationality, social impact, and valence

How do people understand the minds of others? Existing psychological theories have suggested a number of dimensions that perceivers could use to make sense of others' internal mental states. However, it remains unclear which of these dimensions, if any, the brain spontaneously uses when we think about others. The present study used multivoxel pattern analysis (MVPA) of neuroimaging data to identify the primary organizing principles of social cognition. We derived four unique dimensions of mental state representation from existing psychological theories and used functional magnetic resonance imaging to test whether these dimensions organize the neural encoding of others' mental states. MVPA revealed that three such dimensions could predict neural patterns within the medial prefrontal and parietal cortices, temporoparietal junction, and anterior temporal lobes during social thought: rationality, social impact, and valence. These results suggest that these dimensions serve as organizing principles for our understanding of other people.

Spatial properties of objects predict patterns of neural response in the ventral visual pathway

Neuroimaging studies have revealed topographically organised patterns of response to different objects in the ventral visual pathway. These patterns are thought to be based on the form of the object. However, it is not clear what dimensions of object form are important. Here, we determined the extent to which spatial properties (energy across the image) could explain patterns of response in these regions. We compared patterns of fMRI response to images from different object categories presented at different retinal sizes. Although distinct neural patterns were evident to different object categories, changing the size (and thus the spatial properties) of the images had a significant effect on these patterns. Next, we used a computational approach to determine whether more fine-grained differences in the spatial properties can explain the patterns of neural response to different objects. We found that the spatial properties of the image were able to predict patterns of neural response, even when categorical factors were removed from the analysis. We also found that the effect of spatial properties on the patterns of response varies across the ventral visual pathway. These results show how spatial properties can be an important organising principle in the topography of the ventral visual pathway.

Visual Awareness Is Limited by the Representational Architecture of the Visual System

Visual perception and awareness have strict limitations. We suggest that one source of these limitations is the representational architecture of the visual system. Under this view, the extent to which items activate the same neural channels constrains the amount of information that can be processed by the visual system and ultimately reach awareness. Here, we measured how well stimuli from different categories (e.g., faces and cars) blocked one another from reaching awareness using two distinct paradigms that render stimuli invisible: visual masking and continuous flash suppression. Next, we used fMRI to measure the similarity of the neural responses elicited by these categories across the entire visual hierarchy. Overall, we found strong brain–behavior correlations within the ventral pathway, weaker correlations in the dorsal pathway, and no correlations in early visual cortex (V1–V3). These results suggest that the organization of higher level visual cortex constrains visual awareness and the overall processing capacity of visual cognition.

How Visual Is the Visual Cortex? Comparing Connectional and Functional Fingerprints between Congenitally Blind and Sighted Individuals

Classical animal visual deprivation studies and human neuroimaging studies have shown that visual experience plays a critical role in shaping the functionality and connectivity of the visual cortex. Interestingly, recent studies have additionally reported circumscribed regions in the visual cortex in which functional selectivity was remarkably similar in individuals with and without visual experience. Here, by directly comparing resting-state and task-based fMRI data in congenitally blind and sighted human subjects, we obtained large-scale continuous maps of the degree to which connectional and functional "fingerprints" of ventral visual cortex depend on visual experience. We found a close agreement between connectional and functional maps, pointing to a strong interdependence of connectivity and function. Visual experience (or the absence thereof) had a pronounced effect on the resting-state connectivity and functional response profile of occipital cortex and the posterior lateral fusiform gyrus. By contrast, connectional and functional fingerprints in the anterior medial and posterior lateral parts of the ventral visual cortex were statistically indistinguishable between blind and sighted individuals. These results provide a large-scale mapping of the influence of visual experience on the development of both functional and connectivity properties of visual cortex, which serves as a basis for the formulation of new hypotheses regarding the functionality and plasticity of specific subregions. Significance statement: How is the functionality and connectivity of the visual cortex shaped by visual experience? By directly comparing resting-state and task-based fMRI data in congenitally blind and sighted subjects, we obtained large-scale continuous maps of the degree to which connectional and functional "fingerprints" of ventral visual cortex depend on visual experience. In addition to revealing regions that are strongly dependent on visual experience (early visual cortex and posterior fusiform gyrus), our results showed regions in which connectional and functional patterns are highly similar in blind and sighted individuals (anterior medial and posterior lateral ventral occipital temporal cortex). These results serve as a basis for the formulation of new hypotheses regarding the functionality and plasticity of specific subregions of the visual cortex.

A Retinotopic Basis for the Division of High-Level Scene Processing between Lateral and Ventral Human Occipitotemporal Cortex

In humans, there is a repeated category-selective organization across the lateral and ventral surfaces of the occipitotemporal cortex. This apparent redundancy is often explained as a feedforward hierarchy, with processing within lateral areas preceding the processing within ventral areas. Here, we tested the alternative hypothesis that this structure better reflects distinct high-level representations of the upper (ventral surface) and lower (lateral surface) contralateral quadrants of the visual field, consistent with anatomical projections from early visual areas to these surfaces in monkey. Using complex natural scenes, we provide converging evidence from three independent functional imaging and behavioral studies. First, population receptive field mapping revealed strong biases for the contralateral upper and lower quadrant within the ventral and lateral scene-selective regions, respectively. Second, these same biases were observed in the position information available both in the magnitude and multivoxel response across these areas. Third, behavioral judgments of a scene property strongly represented within the ventral scene-selective area (open/closed), but not another equally salient property (manmade/natural), were more accurate in the upper than the lower field. Such differential representation of visual space poses a substantial challenge to the idea of a strictly hierarchical organization between lateral and ventral scene-selective regions. Moreover, such retinotopic biases seem to extend beyond these regions throughout both surfaces. Thus, the large-scale organization of high-level extrastriate cortex likely reflects the need for both specialized representations of particular categories and constraints from the structure of early vision.

SIGNIFICANCE STATEMENT

One of the most striking findings in fMRI has been the presence of matched category-selective regions on the lateral and ventral surfaces of human occipitotemporal cortex. Here, we focus on scene-selective regions and provide converging evidence for a retinotopic explanation of this organization. Specifically, we demonstrate that scene-selective regions exhibit strong biases for different portions of the visual field, with the lateral region representing the contralateral lower visual field and the ventral region the contralateral upper visual field. These biases are consistent with the retinotopy found in the early visual areas that lie directly antecedent to category-selective areas on both surfaces. Furthermore, these biases extend beyond scene-selective cortex and provide a retinotopic basis for the large-scale organization of occipitotemporal cortex.

Corresponding ECoG and fMRI category-selective signals in human ventral temporal cortex

Functional magnetic resonance imaging (fMRI) and electrocorticography (ECoG) research have been influential in revealing the functional characteristics of category-selective responses in human ventral temporal cortex (VTC). One important, but unanswered, question is how these two types of measurements might be related with respect to the VTC. Here we examined which components of the ECoG signal correspond to the fMRI response by using a rare opportunity to measure both fMRI and ECoG responses from the same individuals to images of exemplars of various categories including faces, limbs, cars and houses. Our data reveal three key findings. First, we discovered that the coupling between fMRI and ECoG responses is frequency and time dependent. The strongest and most sustained correlation is observed between fMRI and high frequency broadband (HFB) ECoG responses (30-160 hz). In contrast, the correlation between fMRI and ECoG signals in lower frequency bands is temporally transient, where the correlation is initially positive, but then tapers off or becomes negative. Second, we find that the strong and positive correlation between fMRI and ECoG signals in all frequency bands emerges rapidly around 100 ms after stimulus onset, together with the onset of the first stimulus-driven neural signals in VTC. Third, we find that the spatial topology and representational structure of category-selectivity in VTC reflected in ECoG HFB responses mirrors the topology and structure observed with fMRI. These findings of a strong and rapid coupling between fMRI and HFB responses validate fMRI measurements of functional selectivity with recordings of direct neural activity and suggest that fMRI category-selective signals in VTC are associated with feed-forward neural processing.

Emerging Object Representations in the Visual System Predict Reaction Times for Categorization

Recognizing an object takes just a fraction of a second, less than the blink of an eye. Applying multivariate pattern analysis, or “brain decoding”, methods to magnetoencephalography (MEG) data has allowed researchers to characterize, in high temporal resolution, the emerging representation of object categories that underlie our capacity for rapid recognition. Shortly after stimulus onset, object exemplars cluster by category in a high-dimensional activation space in the brain. In this emerging activation space, the decodability of exemplar category varies over time, reflecting the brain’s transformation of visual inputs into coherent category representations. How do these emerging representations relate to categorization behavior? Recently it has been proposed that the distance of an exemplar representation from a categorical boundary in an activation space is critical for perceptual decision-making, and that reaction times should therefore correlate with distance from the boundary. The predictions of this distance hypothesis have been born out in human inferior temporal cortex (IT), an area of the brain crucial for the representation of object categories. When viewed in the context of a time varying neural signal, the optimal time to “read out” category information is when category representations in the brain are most decodable. Here, we show that the distance from a decision boundary through activation space, as measured using MEG decoding methods, correlates with reaction times for visual categorization during the period of peak decodability. Our results suggest that the brain begins to read out information about exemplar category at the optimal time for use in choice behaviour, and support the hypothesis that the structure of the representation for objects in the visual system is partially constitutive of the decision process in recognition.

Recognizing an object in the world (e.g. a cat) takes just a fraction of a second. Recent advances in neuroscience have allowed researchers to measure the emergence and dynamics of time-varying neural signals that allow us to quickly recognize objects visually. At each moment in time, these neural signals can be characterized as patterns of neural activity that cluster categorically in a high-dimensional activation space. Within this space, a boundary can be drawn between the clusters of activity patterns, which can then be used to discriminate object categories (e.g. cats vs. cars). Based on which side of the boundary a pattern falls, researchers can guess (or “decode”) the category membership of an object that an observer is viewing. In the present study, we provide evidence that at the time when category clusters are best separated in activation space—that is, the time when category information is best suited to be “read out” from the brain’s signals—the structure of activation space can be used to predict behaviour. Our results provide insights into both when and how the brain’s representational architecture supports rapid object recognition.

What is special about expertise? Visual expertise reveals the interactive nature of real-world object recognition

Ever since Diamond and Carey (1986. J. Exp. Psychol.: Gen., vol. 115, pp. 107-117) seminal work, the main model for studying expertise in visual object recognition ("visual expertise") has been face perception. The underlying assumption was that since faces may be considered the ultimate domain of visual expertise, any face-processing signature might actually be a general characteristic of visual expertise. However, while humans are clearly experts in face recognition, visual expertise is not restricted to faces and can be observed in a variety of domains. This raises the question of whether face recognition is in fact the right model to study visual expertise, and if not, what are the common cognitive and neural characteristics of visual expertise. The current perspective article addresses this question by revisiting past and recent neuroimaging and behavioural works on visual expertise. The view of visual expertise that emerges from these works is that expertise is a unique phenomenon, with distinctive neural and cognitive characteristics. Specifically, visual expertise is a controlled, interactive process that develops from the reciprocal interactions between the visual system and multiple top-down factors, including semantic knowledge, top-down attentional control, and task relevance. These interactions enable the ability to flexibly access domain-specific information at multiple scales and levels guided by multiple recognition goals. Extensive visual experience with a given object category culminates in the recruitment of these multiple systems, and is reflected in widespread neural activity, extending well beyond visual cortex, to include higher-level cortical areas.

Body selectivity in occipitotemporal cortex: Causal evidence

Perception of others' bodies provides information that is useful for a number of important social-cognitive processes. Evidence from neuroimaging methods has identified focal cortical regions that are highly selective for perceiving bodies and body parts, including the extrastriate body area (EBA) and fusiform body area (FBA). Our understanding of the functional properties of these regions, and their causal contributions to behavior, has benefitted from the study of neuropsychological patients and particularly from investigations using transcranial magnetic stimulation (TMS). We review this evidence, focusing on TMS studies that are revealing of how (and when) activity in EBA contributes to detecting people in natural scenes; to resolving their body shape, movements, actions, individual parts, and identities; and to guiding goal-directed behavior. These findings are considered in reference to a framework for body perception in which the patterns of neural activity in EBA and FBA jointly serve to make explicit the elements of the visual scene that correspond to the body and its parts. These representations are modulated by other sources of information such as prior knowledge, and are shared with wider brain networks involved in many aspects of social cognition.

The lateral occipitotemporal cortex in action

Understanding and responding to other people's actions is fundamental for social interactions. Whereas many studies emphasize the importance of parietal and frontal regions for these abilities, several lines of recent research show that the human lateral occipitotemporal cortex (LOTC) represents varied aspects of action, ranging from perception of tools and bodies and the way they typically move, to understanding the meaning of actions, to performing overt actions. Here, we highlight common themes across these lines of work, which have informed theories related to high-level vision, concepts, social cognition, and apraxia. We propose that patterns of activity in LOTC form representational spaces, the dimensions of which capture perceptual, semantic, and motor knowledge of how actions change the state of the world.

Basic level category structure emerges gradually across human ventral visual cortex

Objects can be simultaneously categorized at multiple levels of specificity ranging from very broad ("natural object") to very distinct ("Mr. Woof"), with a mid-level of generality (basic level: "dog") often providing the most cognitively useful distinction between categories. It is unknown, however, how this hierarchical representation is achieved in the brain. Using multivoxel pattern analyses, we examined how well each taxonomic level (superordinate, basic, and subordinate) of real-world object categories is represented across occipitotemporal cortex. We found that, although in early visual cortex objects are best represented at the subordinate level (an effect mostly driven by low-level feature overlap between objects in the same category), this advantage diminishes compared to the basic level as we move up the visual hierarchy, disappearing in object-selective regions of occipitotemporal cortex. This pattern stems from a combined increase in within-category similarity (category cohesion) and between-category dissimilarity (category distinctiveness) of neural activity patterns at the basic level, relative to both subordinate and superordinate levels, suggesting that successive visual areas may be optimizing basic level representations.

Fine-grained stimulus representations in body selective areas of human occipito-temporal cortex

Neurophysiological and functional imaging studies have investigated the representation of animate and inanimate stimulus classes in monkey inferior temporal (IT) and human occipito-temporal cortex (OTC). These studies proposed a distributed representation of stimulus categories across IT and OTC and at the same time highlighted category specific modules for the processing of bodies, faces and objects. Here, we investigated whether the stimulus representation within the extrastriate (EBA) and the fusiform (FBA) body areas differed from the representation across OTC. To address this question, we performed an event-related fMRI experiment, evaluating the pattern of activation elicited by 200 individual stimuli that had already been extensively tested in our earlier monkey imaging and single cell studies (Popivanov et al., 2012, 2014). The set contained achromatic images of headless monkey and human bodies, two sets of man-made objects, monkey and human faces, four-legged mammals, birds, fruits, and sculptures. The fMRI response patterns within EBA and FBA primarily distinguished bodies from non-body stimuli, with subtle differences between the areas. However, despite responding on average stronger to bodies than to other categories, classification performance for preferred and non-preferred categories was comparable. OTC primarily distinguished animate from inanimate stimuli. However, cluster analysis revealed a much more fine-grained representation with several homogeneous clusters consisting entirely of stimuli of individual categories. Overall, our data suggest that category representation varies with location within OTC. Nevertheless, body modules contain information to discriminate also non-preferred stimuli and show an increasing specificity in a posterior to anterior gradient.

Left occipitotemporal cortex contributes to the discrimination of tool-associated hand actions: fMRI and TMS evidence

Functional neuroimaging studies have implicated the left lateral occipitotemporal cortex (LOTC) in both tool and hand perception but the functional role of this region is not fully known. Here, by using a task manipulation, we tested whether tool-/hand-selective LOTC contributes to the discrimination of tool-associated hand actions. Participants viewed briefly presented pictures of kitchen and garage tools while they performed one of two tasks: in the action task, they judged whether the tool is associated with a hand rotation action (e.g., screwdriver) or a hand squeeze action (e.g., garlic press), while in the location task they judged whether the tool is typically found in the kitchen (e.g., garlic press) or in the garage (e.g., screwdriver). Both tasks were performed on the same stimulus set and were matched for difficulty. Contrasting fMRI responses between these tasks showed stronger activity during the action task than the location task in both tool- and hand-selective LOTC regions, which closely overlapped. No differences were found in nearby object- and motion-selective control regions. Importantly, these findings were confirmed by a TMS study, which showed that effective TMS over the tool-/hand-selective LOTC region significantly slowed responses for tool action discriminations relative to tool location discriminations, with no such difference during sham TMS. We conclude that left LOTC contributes to the discrimination of tool-associated hand actions.