Object-specific semantic coding in human perirhinal cortex

Semantic representation in the white matter pathway

Object conceptual processing has been localized to distributed cortical regions that represent specific attributes. A challenging question is how object semantic space is formed. We tested a novel framework of representing semantic space in the pattern of white matter (WM) connections by extending the representational similarity analysis (RSA) to structural lesion pattern and behavioral data in 80 brain-damaged patients. For each WM connection, a neural representational dissimilarity matrix (RDM) was computed by first building machine-learning models with the voxel-wise WM lesion patterns as features to predict naming performance of a particular item and then computing the correlation between the predicted naming score and the actual naming score of another item in the testing patients. This correlation was used to build the neural RDM based on the assumption that if the connection pattern contains certain aspects of information shared by the naming processes of these two items, models trained with one item should also predict naming accuracy of the other. Correlating the neural RDM with various cognitive RDMs revealed that neural patterns in several WM connections that connect left occipital/middle temporal regions and anterior temporal regions associated with the object semantic space. Such associations were not attributable to modality-specific attributes (shape, manipulation, color, and motion), to peripheral picture-naming processes (picture visual similarity, phonological similarity), to broad semantic categories, or to the properties of the cortical regions that they connected, which tended to represent multiple modality-specific attributes. That is, the semantic space could be represented through WM connection patterns across cortical regions representing modality-specific attributes.

One of the most challenging questions in cognitive neuroscience is how semantic knowledge, for example, that “scissors” and “knives” are related in meaning, can emerge from primary sensory dimensions such as visual forms. It is often assumed that in the human brain, semantics are stored in regions of the brain cortex, where distinct types of modality-specific information are transferred to and bind together. We tested an alternative hypothesis—“representation by connection”—in which higher-order semantic information could be coded by means of connection patterns between cortical regions. Combining data from behavior and brain imaging of 80 patients with brain lesions, we applied machine learning to construct the mapping models between the lesion patterns on axonal tracts (white matter) and item-specific object-naming performances. We found that specific white matter lesions produced deficits in object naming associated with the object’s semantic space, but not relevant to its primary dimension. The naming performances of semantically related objects were better predicted from white matter lesion-pattern models. That is, the higher-order semantic space could be coded in patterns of brain connections by linking cortical areas that do not necessarily contain such information.

Doctor, Teacher, and Stethoscope: Neural Representation of Different Types of Semantic Relations

Concepts can be related in many ways. They can belong to the same taxonomic category (e.g., "doctor" and "teacher," both in the category of people) or be associated with the same event context (e.g., "doctor" and "stethoscope," both associated with medical scenarios). How are these two major types of semantic relations coded in the brain? We constructed stimuli from three taxonomic categories (people, manmade objects, and locations) and three thematic categories (school, medicine, and sports) and investigated the neural representations of these two dimensions using representational similarity analyses in human participants (10 men and nine women). In specific regions of interest, the left anterior temporal lobe (ATL) and the left temporoparietal junction (TPJ), we found that, whereas both areas had significant effects of taxonomic information, the taxonomic relations had stronger effects in the ATL than in the TPJ ("doctor" and "teacher" closer in ATL neural activity), with the reverse being true for thematic relations ("doctor" and "stethoscope" closer in TPJ neural activity). A whole-brain searchlight analysis revealed that widely distributed regions, mainly in the left hemisphere, represented the taxonomic dimension. Interestingly, the significant effects of the thematic relations were only observed after the taxonomic differences were controlled for in the left TPJ, the right superior lateral occipital cortex, and other frontal, temporal, and parietal regions. In summary, taxonomic grouping is a primary organizational dimension across distributed brain regions, with thematic grouping further embedded within such taxonomic structures.SIGNIFICANCE STATEMENT How are concepts organized in the brain? It is well established that concepts belonging to the same taxonomic categories (e.g., "doctor" and "teacher") share neural representations in specific brain regions. How concepts are associated in other manners (e.g., "doctor" and "stethoscope," which are thematically related) remains poorly understood. We used representational similarity analyses to unravel the neural representations of these different types of semantic relations by testing the same set of words that could be differently grouped by taxonomic categories or by thematic categories. We found that widely distributed brain areas primarily represented taxonomic categories, with the thematic categories further embedded within the taxonomic structure.

The effect of age on the FCSRT-IR and temporary visual memory binding

ABSTRACTBackground:Cognitive markers of early Alzheimer's disease (AD) should be sensitive and specific to memory impairments that are not associated with healthy cognitive aging. In the present study, we investigated the effect of healthy cognitive aging on two proposed cognitive markers of AD: the Free and Cued Selective Reminding Task with Immediate Recall (FCSRT-IR) and a temporary visual memory binding (TMB) task.

METHOD

Free recall and the cost of holding bound information in visual memory were compared between 24 younger and 24 older participants in a mixed, fully counterbalanced experiment.

RESULTS

A significant effect of age was observed on free recall in the FCSRT-IR only and not on the cost of binding in the TMB task.

CONCLUSIONS

Of these two cognitive markers, the TMB task is more likely to be specific to memory impairments that are independent of age.

In sight, in mind

A region of the brain called the perirhinal cortex represents both what things look like and what they mean.

Deep Residual Network Predicts Cortical Representation and Organization of Visual Features for Rapid Categorization

The brain represents visual objects with topographic cortical patterns. To address how distributed visual representations enable object categorization, we established predictive encoding models based on a deep residual network, and trained them to predict cortical responses to natural movies. Using this predictive model, we mapped human cortical representations to 64,000 visual objects from 80 categories with high throughput and accuracy. Such representations covered both the ventral and dorsal pathways, reflected multiple levels of object features, and preserved semantic relationships between categories. In the entire visual cortex, object representations were organized into three clusters of categories: biological objects, non-biological objects, and background scenes. In a finer scale specific to each cluster, object representations revealed sub-clusters for further categorization. Such hierarchical clustering of category representations was mostly contributed by cortical representations of object features from middle to high levels. In summary, this study demonstrates a useful computational strategy to characterize the cortical organization and representations of visual features for rapid categorization.

Integrative and distinctive coding of visual and conceptual object features in the ventral visual stream

A significant body of research in cognitive neuroscience is aimed at understanding how object concepts are represented in the human brain. However, it remains unknown whether and where the visual and abstract conceptual features that define an object concept are integrated. We addressed this issue by comparing the neural pattern similarities among object-evoked fMRI responses with behavior-based models that independently captured the visual and conceptual similarities among these stimuli. Our results revealed evidence for distinctive coding of visual features in lateral occipital cortex, and conceptual features in the temporal pole and parahippocampal cortex. By contrast, we found evidence for integrative coding of visual and conceptual object features in perirhinal cortex. The neuroanatomical specificity of this effect was highlighted by results from a searchlight analysis. Taken together, our findings suggest that perirhinal cortex uniquely supports the representation of fully specified object concepts through the integration of their visual and conceptual features.

Neural Representations of Belief Concepts: A Representational Similarity Approach to Social Semantics

The present experiment identified neural regions that represent a class of concepts that are independent of perceptual or sensory attributes. During functional magnetic resonance imaging scanning, participants viewed names of social groups (e.g. Atheists, Evangelicals, and Economists) and performed a one-back similarity judgment according to 1 of 2 dimensions of belief attributes: political orientation (Liberal to Conservative) or spiritualism (Spiritualist to Materialist). By generalizing across a wide variety of social groups that possess these beliefs, these attribute concepts did not coincide with any specific sensory quality, allowing us to target conceptual, rather than perceptual, representations. Multi-voxel pattern searchlight analysis was used to identify regions in which activation patterns distinguished the 2 ends of both dimensions: Conservative from Liberal social groups when participants focused on the political orientation dimension, and spiritual from Materialist groups when participants focused on the spiritualism dimension. A cluster in right precuneus exhibited such a pattern, indicating that it carries information about belief-attribute concepts and forms part of semantic memory—perhaps a component particularly concerned with psychological traits. This region did not overlap with the theory of mind network, which engaged nearby, but distinct, parts of precuneus. These findings have implications for the neural organization of conceptual knowledge, especially the understanding of social groups.

On the partnership between neural representations of object categories and visual features in the ventral visual pathway

A dominant view in the cognitive neuroscience of object vision is that regions of the ventral visual pathway exhibit some degree of category selectivity. However, recent findings obtained with multivariate pattern analyses (MVPA) suggest that apparent category selectivity in these regions is dependent on more basic visual features of stimuli. In which case a rethinking of the function and organization of the ventral pathway may be in order. We suggest that addressing this issue of functional specificity requires clear coding hypotheses, about object category and visual features, which make contrasting predictions about neuroimaging results in ventral pathway regions. One way to differentiate between categorical and featural coding hypotheses is to test for residual categorical effects: effects of category selectivity that cannot be accounted for by visual features of stimuli. A strong method for testing these effects, we argue, is to make object category and target visual features orthogonal in stimulus design. Recent studies that adopt this approach support a feature-based categorical coding hypothesis according to which regions of the ventral stream do indeed code for object category, but in a format at least partially based on the visual features of stimuli.

Representation of Semantic Similarity in the Left Intraparietal Sulcus: Functional Magnetic Resonance Imaging Evidence

According to a recent study, semantic similarity between concrete entities correlates with the similarity of activity patterns in left middle IPS during category naming. We examined the replicability of this effect under passive viewing conditions, the potential role of visuoperceptual similarity, where the effect is situated compared to regions that have been previously implicated in visuospatial attention, and how it compares to effects of object identity and location. Forty-six subjects participated. Subjects passively viewed pictures from two categories, musical instruments and vehicles. Semantic similarity between entities was estimated based on a concept-feature matrix obtained in more than 1,000 subjects. Visuoperceptual similarity was modeled based on the HMAX model, the AlexNet deep convolutional learning model, and thirdly, based on subjective visuoperceptual similarity ratings. Among the IPS regions examined, only left middle IPS showed a semantic similarity effect. The effect was significant in hIP1, hIP2, and hIP3. Visuoperceptual similarity did not correlate with similarity of activity patterns in left middle IPS. The semantic similarity effect in left middle IPS was significantly stronger than in the right middle IPS and also stronger than in the left or right posterior IPS. The semantic similarity effect was similar to that seen in the angular gyrus. Object identity effects were much more widespread across nearly all parietal areas examined. Location effects were relatively specific for posterior IPS and area 7 bilaterally. To conclude, the current findings replicate the semantic similarity effect in left middle IPS under passive viewing conditions, and demonstrate its anatomical specificity within a cytoarchitectonic reference frame. We propose that the semantic similarity effect in left middle IPS reflects the transient uploading of semantic representations in working memory.

The influence of low-level stimulus features on the representation of contexts, items, and their mnemonic associations

Since the earliest attempts to characterize the "receptive fields" of neurons, a central aim of many neuroscience experiments is to elucidate the information that is represented in various regions of the brain. Recent studies suggest that, in the service of memory, information is represented in the medial temporal lobe in a conjunctive or associative form with the contextual aspects of the experience being the primary factor or highest level of the conjunctive hierarchy. A critical question is whether the information that has been observed in these studies reflects notions such as a cognitive representation of context or whether the information reflects the low-level sensory differences between stimuli. We performed two functional magnetic resonance imaging experiments to address this question and we found that associative representations observed between context and item (and order) in the human brain can be highly influenced by low-level sensory differences between stimuli. Our results place clear constraints on the experimental design of studies that aim to investigate the representation of contexts and items during performance of associative memory tasks. Moreover, our results raise interesting theoretical questions regarding the disambiguation of memory-related representations from processing-related representations.

Dynamics of scene representations in the human brain revealed by magnetoencephalography and deep neural networks

Human scene recognition is a rapid multistep process evolving over time from single scene image to spatial layout processing. We used multivariate pattern analyses on magnetoencephalography (MEG) data to unravel the time course of this cortical process. Following an early signal for lower-level visual analysis of single scenes at ~100ms, we found a marker of real-world scene size, i.e. spatial layout processing, at ~250ms indexing neural representations robust to changes in unrelated scene properties and viewing conditions. For a quantitative model of how scene size representations may arise in the brain, we compared MEG data to a deep neural network model trained on scene classification. Representations of scene size emerged intrinsically in the model, and resolved emerging neural scene size representation. Together our data provide a first description of an electrophysiological signal for layout processing in humans, and suggest that deep neural networks are a promising framework to investigate how spatial layout representations emerge in the human brain.

Functional organization of the medial temporal lobe memory system following neonatal hippocampal lesion in rhesus monkeys

Hippocampal damage in adult humans impairs episodic and semantic memory, whereas hippocampal damage early in life impairs episodic memory but leaves semantic learning relatively preserved. We have previously shown a similar behavioral dissociation in nonhuman primates. Hippocampal lesion in adult monkeys prevents allocentric spatial relational learning, whereas spatial learning persists following neonatal lesion. Here, we quantified the number of cells expressing the immediate-early gene c-fos, a marker of neuronal activity, to characterize the functional organization of the medial temporal lobe memory system following neonatal hippocampal lesion. Ninety minutes before brain collection, three control and four adult monkeys with bilateral neonatal hippocampal lesions explored a novel environment to activate brain structures involved in spatial learning. Three other adult monkeys with neonatal hippocampal lesions remained in their housing quarters. In unlesioned monkeys, we found high levels of c-fos expression in the intermediate and caudal regions of the entorhinal cortex, and in the perirhinal, parahippocampal, and retrosplenial cortices. In lesioned monkeys, spatial exploration induced an increase in c-fos expression in the intermediate field of the entorhinal cortex, the perirhinal, parahippocampal, and retrosplenial cortices, but not in the caudal entorhinal cortex. These findings suggest that different regions of the medial temporal lobe memory system may require different types of interaction with the hippocampus in support of memory. The caudal perirhinal cortex, the parahippocampal cortex, and the retrosplenial cortex may contribute to spatial learning in the absence of functional hippocampal circuits, whereas the caudal entorhinal cortex may require hippocampal output to support spatial learning.

Evidence for similar patterns of neural activity elicited by picture- and word-based representations of natural scenes

A long-standing core question in cognitive science is whether different modalities and representation types (pictures, words, sounds, etc.) access a common store of semantic information. Although different input types have been shown to activate a shared network of brain regions, this does not necessitate that there is a common representation, as the neurons in these regions could still differentially process the different modalities. However, multi-voxel pattern analysis can be used to assess whether, e.g., pictures and words evoke a similar pattern of activity, such that the patterns that separate categories in one modality transfer to the other. Prior work using this method has found support for a common code, but has two limitations: they have either only examined disparate categories (e.g. animals vs. tools) that are known to activate different brain regions, raising the possibility that the pattern separation and inferred similarity reflects only large scale differences between the categories or they have been limited to individual object representations. By using natural scene categories, we not only extend the current literature on cross-modal representations beyond objects, but also, because natural scene categories activate a common set of brain regions, we identify a more fine-grained (i.e. higher spatial resolution) common representation. Specifically, we studied picture- and word-based representations of natural scene stimuli from four different categories: beaches, cities, highways, and mountains. Participants passively viewed blocks of either phrases (e.g. "sandy beach") describing scenes or photographs from those same scene categories. To determine whether the phrases and pictures evoke a common code, we asked whether a classifier trained on one stimulus type (e.g. phrase stimuli) would transfer (i.e. cross-decode) to the other stimulus type (e.g. picture stimuli). The analysis revealed cross-decoding in the occipitotemporal, posterior parietal and frontal cortices. This similarity of neural activity patterns across the two input types, for categories that co-activate local brain regions, provides strong evidence of a common semantic code for pictures and words in the brain.

Decoding the time-course of object recognition in the human brain: From visual features to categorical decisions

Visual object recognition is a complex, dynamic process. Multivariate pattern analysis methods, such as decoding, have begun to reveal how the brain processes complex visual information. Recently, temporal decoding methods for EEG and MEG have offered the potential to evaluate the temporal dynamics of object recognition. Here we review the contribution of M/EEG time-series decoding methods to understanding visual object recognition in the human brain. Consistent with the current understanding of the visual processing hierarchy, low-level visual features dominate decodable object representations early in the time-course, with more abstract representations related to object category emerging later. A key finding is that the time-course of object processing is highly dynamic and rapidly evolving, with limited temporal generalisation of decodable information. Several studies have examined the emergence of object category structure, and we consider to what degree category decoding can be explained by sensitivity to low-level visual features. Finally, we evaluate recent work attempting to link human behaviour to the neural time-course of object processing.

Perirhinal cortex tracks degree of recent as well as cumulative lifetime experience with object concepts

Evidence from numerous sources indicates that recognition of the prior occurrence of objects requires computations of perirhinal cortex (PrC) in the medial temporal lobe (MTL). Extant research has primarily probed recognition memory based on item exposure in a recent experimental study episode. Outside the laboratory, however, familiarity for objects typically accrues gradually with learning across many different episodic contexts, which can be distributed over a lifetime of experience. It is currently unknown whether PrC also tracks this cumulative lifetime experience with object concepts. To address this issue, we conducted a functional magnetic resonance imaging (fMRI) experiment in healthy individuals in which we compared judgments of the perceived lifetime familiarity with object concepts, a task that has previously been employed in many normative studies on concept knowledge, with frequency judgments for recent laboratory exposure in a study phase. Guided by neurophysiological data showing that neurons in primate PrC signal prior object exposure at multiple time scales, we predicted that PrC responses would track perceived prior experience in both types of judgments. Left PrC and a number of cortical regions that are often co-activated as part of the default-mode network showed an increase in Blood-Oxygen-Level Dependent (BOLD) response in relation to increases in the perceived cumulative lifetime familiarity of object concepts. These regions included the left hippocampus, left mid-lateral temporal cortex, as well as anterior and posterior cortical midline structures. Critically, left PrC was found to be the only region that showed this response in combination with the typically observed decrease in signal for perceived recent exposure in the experimental study phase. These findings provide, to our knowledge, the first evidence that ties signals in human PrC to variations in cumulative lifetime experience with object concepts. They offer a new link between the role of PrC in recognition memory and its broader role in conceptual processing.

Representational Similarity Mapping of Distributional Semantics in Left Inferior Frontal, Middle Temporal, and Motor Cortex

Language comprehension engages a distributed network of frontotemporal, parietal, and sensorimotor regions, but it is still unclear how meaning of words and their semantic relationships are represented and processed within these regions and to which degrees lexico-semantic representations differ between regions and semantic types. We used fMRI and representational similarity analysis to relate word-elicited multivoxel patterns to semantic similarity between action and object words. In left inferior frontal (BA 44-45-47), left posterior middle temporal and left precentral cortex, the similarity of brain response patterns reflected semantic similarity among action-related verbs, as well as across lexical classes-between action verbs and tool-related nouns and, to a degree, between action verbs and food nouns, but not between action verbs and animal nouns. Instead, posterior inferior temporal cortex exhibited a reverse response pattern, which reflected the semantic similarity among object-related nouns, but not action-related words. These results show that semantic similarity is encoded by a range of cortical areas, including multimodal association (e.g., anterior inferior frontal, posterior middle temporal) and modality-preferential (premotor) cortex and that the representational geometries in these regions are partly dependent on semantic type, with semantic similarity among action-related words crossing lexical-semantic category boundaries.

The Role of Visual and Semantic Properties in the Emergence of Category-Specific Patterns of Neural Response in the Human Brain

Brain-imaging studies have found distinct spatial and temporal patterns of response to different object categories across the brain. However, the extent to which these categorical patterns of response reflect higher-level semantic or lower-level visual properties of the stimulus remains unclear. To address this question, we measured patterns of EEG response to intact and scrambled images in the human brain. Our rationale for using scrambled images is that they have many of the visual properties found in intact images, but do not convey any semantic information. Images from different object categories (bottle, face, house) were briefly presented (400 ms) in an event-related design. A multivariate pattern analysis revealed categorical patterns of response to intact images emerged ∼80-100 ms after stimulus onset and were still evident when the stimulus was no longer present (∼800 ms). Next, we measured the patterns of response to scrambled images. Categorical patterns of response to scrambled images also emerged ∼80-100 ms after stimulus onset. However, in contrast to the intact images, distinct patterns of response to scrambled images were mostly evident while the stimulus was present (∼400 ms). Moreover, scrambled images were able to account only for all the variance in the intact images at early stages of processing. This direct manipulation of visual and semantic content provides new insights into the temporal dynamics of object perception and the extent to which different stages of processing are dependent on lower-level or higher-level properties of the image.

Coding of Visual, Auditory, Rule, and Response Information in the Brain: 10 Years of Multivoxel Pattern Analysis

How is the processing of task information organized in the brain? Many views of brain function emphasize modularity, with different regions specialized for processing different types of information. However, recent accounts also highlight flexibility, pointing especially to the highly consistent pattern of frontoparietal activation across many tasks. Although early insights from functional imaging were based on overall activation levels during different cognitive operations, in the last decade many researchers have used multivoxel pattern analyses to interrogate the representational content of activations, mapping out the brain regions that make particular stimulus, rule, or response distinctions. Here, we drew on 100 searchlight decoding analyses from 57 published papers to characterize the information coded in different brain networks. The outcome was highly structured. Visual, auditory, and motor networks predominantly (but not exclusively) coded visual, auditory, and motor information, respectively. By contrast, the frontoparietal multiple-demand network was characterized by domain generality, coding visual, auditory, motor, and rule information. The contribution of the default mode network and voxels elsewhere was minor. The data suggest a balanced picture of brain organization in which sensory and motor networks are relatively specialized for information in their own domain, whereas a specific frontoparietal network acts as a domain-general "core" with the capacity to code many different aspects of a task.

The Left Fusiform Gyrus is a Critical Region Contributing to the Core Behavioral Profile of Semantic Dementia

Given that extensive cerebral regions are co-atrophic in semantic dementia (SD), it is not yet known which critical regions (SD-semantic-critical regions) are really responsible for the semantic deficits of SD. To identify the SD-semantic-critical regions, we explored the relationship between the degree of cerebral atrophy in the whole brain and the severity of semantic deficits in 19 individuals with SD. We found that the gray matter volumes (GMVs) of two regions [left fusiform gyrus (lFFG) and left parahippocampal gyrus (lPHG)] significantly correlated with the semantic scores of patients with SD. Importantly, the effects of the lFFG remained significant after controlling for the GMVs of the lPHG. Moreover, the effects of the region could not be accounted for by the total GMV, general cognitive ability, laterality of brain atrophy, or control task performance. We further observed that each atrophic portion of the lFFG along the anterior–posterior axis might dedicate to the loss of semantic functions in SD. These results reveal that the lFFG could be a critical region contributing to the semantic deficits of SD.

De Novo Emergence of Odor Category Representations in the Human Brain

Categorization allows organisms to efficiently extract relevant information from a diverse environment. Because of the multidimensional nature of odor space, this ability is particularly important for the olfactory system. However, categorization relies on experience, and the processes by which the human brain forms categorical representations about new odor percepts are currently unclear. Here we used olfactory psychophysics and multivariate fMRI techniques, in the context of a paired-associates learning task, to examine the emergence of novel odor category representations in the human brain. We found that learning between novel odors and visual category information induces a perceptual reorganization of those odors, in parallel with the emergence of odor category-specific ensemble patterns in perirhinal, orbitofrontal, piriform, and insular cortices. Critically, the learning-induced pattern effects in orbitofrontal and perirhinal cortex predicted the magnitude of categorical learning and perceptual plasticity. The formation of de novo category-specific representations in olfactory and limbic brain regions suggests that such ensemble patterns subserve the development of perceptual classes of information, and imply that these patterns are instrumental to the brain's capacity for odor categorization.

SIGNIFICANCE STATEMENT

How the human brain assigns novel odors to perceptual classes and categories is poorly understood. We combined an olfactory-visual paired-associates task with multivariate pattern-based fMRI approaches to investigate the de novo formation of odor category representations within the human brain. The identification of emergent odor category codes within the perirhinal, piriform, orbitofrontal, and insular cortices suggests that these regions can integrate multimodal sensory input to shape category-specific olfactory representations for novel odors, and may ultimately play an important role in assembling each individual's semantic knowledge base of the olfactory world.

Typicality sharpens category representations in object-selective cortex

The purpose of categorization is to identify generalizable classes of objects whose members can be treated equivalently. Within a category, however, some exemplars are more representative of that concept than others. Despite long-standing behavioral effects, little is known about how typicality influences the neural representation of real-world objects from the same category. Using fMRI, we showed participants 64 subordinate object categories (exemplars) grouped into 8 basic categories. Typicality for each exemplar was assessed behaviorally and we used several multi-voxel pattern analyses to characterize how typicality affects the pattern of responses elicited in early visual and object-selective areas: V1, V2, V3v, hV4, LOC. We found that in LOC, but not in early areas, typical exemplars elicited activity more similar to the central category tendency and created sharper category boundaries than less typical exemplars, suggesting that typicality enhances within-category similarity and between-category dissimilarity. Additionally, we uncovered a brain region (cIPL) where category boundaries favor less typical categories. Our results suggest that typicality may constitute a previously unexplored principle of organization for intra-category neural structure and, furthermore, that this representation is not directly reflected in image features describing natural input, but rather built by the visual system at an intermediate processing stage.

Tactile Object Familiarity in the Blind Brain Reveals the Supramodal Perceptual-Mnemonic Nature of the Perirhinal Cortex

This study is the first to investigate the neural underpinnings of tactile object familiarity in the blind during both perception and memory. In the sighted, the perirhinal cortex (PRC) has been implicated in the assessment of visual object familiarity-a crucial everyday task-as evidenced by reduced activation when an object becomes familiar. Here, to examine the PRC's role in tactile object familiarity in the absence of vision, we trained blind participants on a unique memory-guided drawing technique and measured brain activity while they perceptually explored raised-line drawings, drew them from tactile memory, and scribbled (control). Functional magnetic resonance imaging (fMRI) before and after a week of training revealed a significant decrease in PRC activation from pre- to post-training (i.e., from unfamiliar to familiar) during perceptual exploration as well as memory-guided drawing, but not scribbling. This familiarity-based reduction is the first evidence that the PRC represents tactile object familiarity in the blind. Furthermore, the finding of this effect during both tactile perception and tactile memory provides the critical link in establishing the PRC as a structure whose representations are supramodal for both perception and memory.

Evidence for Integrated Visual Face and Body Representations in the Anterior Temporal Lobes

Research on visual face perception has revealed a region in the ventral anterior temporal lobes, often referred to as the anterior temporal face patch (ATFP), which responds strongly to images of faces. To date, the selectivity of the ATFP has been examined by contrasting responses to faces against a small selection of categories. Here, we assess the selectivity of the ATFP in humans with a broad range of visual control stimuli to provide a stronger test of face selectivity in this region. In Experiment 1, participants viewed images from 20 stimulus categories in an event-related fMRI design. Faces evoked more activity than all other 19 categories in the left ATFP. In the right ATFP, equally strong responses were observed for both faces and headless bodies. To pursue this unexpected finding, in Experiment 2, we used multivoxel pattern analysis to examine whether the strong response to face and body stimuli reflects a common coding of both classes or instead overlapping but distinct representations. On a voxel-by-voxel basis, face and whole-body responses were significantly positively correlated in the right ATFP, but face and body-part responses were not. This finding suggests that there is shared neural coding of faces and whole bodies in the right ATFP that does not extend to individual body parts. In contrast, the same approach revealed distinct face and body representations in the right fusiform gyrus. These results are indicative of an increasing convergence of distinct sources of person-related perceptual information proceeding from the posterior to the anterior temporal cortex.

Perirhinal cortex supports tactual discrimination tasks with increasing levels of complexity: Retrograde effect

Recent studies have suggested that the perirhinal cortex (Prh) supports representations of feature conjunctions in the visual modality during the acquisition/encoding of complex discriminations. To extend this idea to other sensory modalities and to another stage of the discrimination process, we studied the effect of Prh lesions on the expression of a series of tactual discrimination tasks learned preoperatively. These tasks differed from one another in the degree of feature overlap of the stimuli and in the difficulty of the task. During pre- and post-operative testing phases, rats had to discriminate among 3 stimuli simultaneously exposed in 3 arms of a 4-arm plus-shaped maze. Prh-damaged rats showed a profound impairment in the expression of tactual discrimination tasks when the stimuli had a high or intermediate degree of feature ambiguity, but not when they had a low degree of ambiguity (experiments 1a-1c). In order to experimentally dissociate between subregions within the medial temporal lobe, experiment 2 was conducted to show that hippocampal lesions did not cause any impairment in task expression even when the stimuli had a high degree of feature ambiguity. When the tactual discrimination tasks used simple/individual nonoverlapping features of the stimuli (size), Prh lesions did not affect the expression of these discriminations despite the high level of difficulty of these tasks (experiments 3a and 3b). These findings suggest that, in the somatosensory modality, the Prh plays an essential role in the processing of complex stimuli with overlapping features but not in simple tactual discriminations. Furthermore, the Prh is necessary not just during acquisition but also during expression/performance of the discrimination task.

The 'when' and 'where' of semantic coding in the anterior temporal lobe: Temporal representational similarity analysis of electrocorticogram data

Electrocorticograms (ECoG) provide a unique opportunity to monitor neural activity directly at the cortical surface. Ten patients with subdural electrodes covering ventral and lateral anterior temporal regions (ATL) performed a picture naming task. Temporal representational similarity analysis (RSA) was used, for the first time, to compare spatio-temporal neural patterns from the ATL surface with pre-defined theoretical models. The results indicate that the neural activity in the ventral subregion of the ATL codes semantic representations from 250 msec after picture onset. The observed activation similarity was not related to the visual similarity of the pictures or the phonological similarity of their names. In keeping with convergent evidence for the importance of the ATL in semantic processing, these results provide the first direct evidence of semantic coding from the surface of the ventral ATL and its time-course.

Learning Warps Object Representations in the Ventral Temporal Cortex

The human ventral temporal cortex (VTC) plays a critical role in object recognition. Although it is well established that visual experience shapes VTC object representations, the impact of semantic and contextual learning is unclear. In this study, we tracked changes in representations of novel visual objects that emerged after learning meaningful information about each object. Over multiple training sessions, participants learned to associate semantic features (e.g., "made of wood," "floats") and spatial contextual associations (e.g., "found in gardens") with novel objects. fMRI was used to examine VTC activity for objects before and after learning. Multivariate pattern similarity analyses revealed that, after learning, VTC activity patterns carried information about the learned contextual associations of the objects, such that objects with contextual associations exhibited higher pattern similarity after learning. Furthermore, these learning-induced increases in pattern information about contextual associations were correlated with reductions in pattern information about the object's visual features. In a second experiment, we validated that these contextual effects translated to real-life objects. Our findings demonstrate that visual object representations in VTC are shaped by the knowledge we have about objects and show that object representations can flexibly adapt as a consequence of learning with the changes related to the specific kind of newly acquired information.

Neural representation for object recognition in inferotemporal cortex

We suggest that population representation of objects in inferotemporal cortex lie on a continuum between a purely structural, parts-based description and a purely holistic description. The intrinsic dimensionality of object representation is estimated to be around 100, perhaps with lower dimensionalities for object representations more toward the holistic end of the spectrum. Cognitive knowledge in the form of semantic information and task information feed back to inferotemporal cortex from perirhinal and prefrontal cortex respectively, providing high-level multimodal-based expectations that assist in the interpretation of object stimuli. Integration of object information across eye movements may also contribute to object recognition through a process of active vision.

Spatial properties of objects predict patterns of neural response in the ventral visual pathway

Neuroimaging studies have revealed topographically organised patterns of response to different objects in the ventral visual pathway. These patterns are thought to be based on the form of the object. However, it is not clear what dimensions of object form are important. Here, we determined the extent to which spatial properties (energy across the image) could explain patterns of response in these regions. We compared patterns of fMRI response to images from different object categories presented at different retinal sizes. Although distinct neural patterns were evident to different object categories, changing the size (and thus the spatial properties) of the images had a significant effect on these patterns. Next, we used a computational approach to determine whether more fine-grained differences in the spatial properties can explain the patterns of neural response to different objects. We found that the spatial properties of the image were able to predict patterns of neural response, even when categorical factors were removed from the analysis. We also found that the effect of spatial properties on the patterns of response varies across the ventral visual pathway. These results show how spatial properties can be an important organising principle in the topography of the ventral visual pathway.

Understanding What We See: How We Derive Meaning From Vision

Recognising objects goes beyond vision, and requires models that incorporate different aspects of meaning. Most models focus on superordinate categories (e.g., animals, tools) which do not capture the richness of conceptual knowledge. We argue that object recognition must be seen as a dynamic process of transformation from low-level visual input through categorical organisation to specific conceptual representations. Cognitive models based on large normative datasets are well-suited to capture statistical regularities within and between concepts, providing both category structure and basic-level individuation. We highlight recent research showing how such models capture important properties of the ventral visual pathway. This research demonstrates that significant advances in understanding conceptual representations can be made by shifting the focus from studying superordinate categories to basic-level concepts.

Compositionality and the angular gyrus: A multi-voxel similarity analysis of the semantic composition of nouns and verbs

The cognitive and neural systems that enable conceptual processing must support the ability to combine (and recombine) concepts to form an infinite number of ideas. Two candidate neural systems for conceptual combination-the left anterior temporal lobe (ATL) and the left angular gyrus (AG)-have been characterized as "semantic hubs" due to both functional and anatomical properties; however, these two regions likely support different aspects of composition. Here we consider two hypotheses for the role of AG in conceptual combination, both of which differ from a putative role for the ATL in "feature-based" combinatorics (i.e., meaning derived by combining concepts' features). Firstly, we examine whether AG is more sensitive to function-argument relations of the sort that arise when a predicate is combined with its arguments. Secondly, we examine the non-mutually exclusive possibility that AG represents information carried on a verb in particular, whether this be information about event composition or about thematic relations denoted uniquely by verbs. We identified voxels that respond differentially to two-word versus one-word stimuli, and we measured the similarity of the patterns in these voxels evoked by (1) pairs of two-word phrases that shared a noun that was an argument, thus sharing function-argument composition (e.g. eats meat and with meat), in comparison with two-word phrases that shared only a noun, not an argument (e.g., eats meat and tasty meat); and (2) stimulus pairs that shared only an event (operationalized here as sharing a verb; e.g. eats meat and eats quickly), in comparison to both of the above. We found that activity patterns in left AG tracked information relating to the presence of an event-denoting verb in a pair of two-word phrases. We also found that the neural similarity in AG voxel patterns between two phrases sharing a verb correlated with subjects' ratings of how similar the meanings of those two verb phrases were. These findings indicate that AG represents information specific to verbs, perhaps event structure or thematic relations mediated by verbs, as opposed to argument structure in general.

Predicting the Time Course of Individual Objects with MEG

To respond appropriately to objects, we must process visual inputs rapidly and assign them meaning. This involves highly dynamic, interactive neural processes through which information accumulates and cognitive operations are resolved across multiple time scales. However, there is currently no model of object recognition which provides an integrated account of how visual and semantic information emerge over time; therefore, it remains unknown how and when semantic representations are evoked from visual inputs. Here, we test whether a model of individual objects--based on combining the HMax computational model of vision with semantic-feature information--can account for and predict time-varying neural activity recorded with magnetoencephalography. We show that combining HMax and semantic properties provides a better account of neural object representations compared with the HMax alone, both through model fit and classification performance. Our results show that modeling and classifying individual objects is significantly improved by adding semantic-feature information beyond ∼200 ms. These results provide important insights into the functional properties of visual processing across time.

Time-varying effective connectivity during visual object naming as a function of semantic demands

Accumulating evidence suggests that visual object understanding involves a rapid feedforward sweep, after which subsequent recurrent interactions are necessary. The extent to which recurrence plays a critical role in object processing remains to be determined. Recent studies have demonstrated that recurrent processing is modulated by increasing semantic demands. Differentially from previous studies, we used dynamic causal modeling to model neural activity recorded with magnetoencephalography while 14 healthy humans named two sets of visual objects that differed in the degree of semantic accessing demands, operationalized in terms of the values of basic psycholinguistic variables associated with the presented objects (age of acquisition, frequency, and familiarity). This approach allowed us to estimate the directionality of the causal interactions among brain regions and their associated connectivity strengths. Furthermore, to understand the dynamic nature of connectivity (i.e., the chronnectome; Calhoun et al., 2014) we explored the time-dependent changes of effective connectivity during a period (200-400 ms) where adding semantic-feature information improves modeling and classifying visual objects, at 50 ms increments. First, we observed a graded involvement of backward connections, that became active beyond 200 ms. Second, we found that semantic demands caused a suppressive effect in the backward connection from inferior frontal cortex (IFC) to occipitotemporal cortex over time. These results complement those from previous studies underscoring the role of IFC as a common source of top-down modulation, which drives recurrent interactions with more posterior regions during visual object recognition. Crucially, our study revealed the inhibitory modulation of this interaction in situations that place greater demands on the conceptual system.

Low-level properties of natural images predict topographic patterns of neural response in the ventral visual pathway

Neuroimaging research over the past 20 years has begun to reveal a picture of how the human visual system is organized. A key distinction that has arisen from these studies is the difference in the organization of low-level and high-level visual regions. Low-level regions contain topographic maps that are tightly linked to properties of the image. In contrast, high-level visual areas are thought to be arranged in modules that are tightly linked to categorical or semantic information in the image. To date, an unresolved question has been how the strong functional selectivity for object categories in high-level visual regions might arise from the image-based representations found in low-level visual regions. Here, we review recent evidence suggesting that patterns of response in high-level visual areas may be better explained by response to image properties that are characteristic of different object categories.

Multisensory integration and cross-modal learning in synaesthesia: A unifying model

Recent research into synaesthesia has highlighted the role of learning, yet synaesthesia is clearly a genetic condition. Here we ask how can the idea that synaesthesia reflects innate, genetic differences be reconciled with models that suggest it is driven by learning. A number of lines of evidence suggest that synaesthesia relies on, or at least interacts with, processes of multisensory integration that are common across all people. These include multisensory activations that arise in early regions of the brain as well as feedback from longer-term cross-modal associations generated in memory. These cognitive processes may interact independently to influence the phenomenology of the synaesthetic experience, as well as the individual differences within particular types of synaesthesia. The theoretical framework presented here is consistent with both an innate difference as the fundamental driver of the condition of synaesthesia, and with experiential and semantic influences on the eventual phenotype that emerges. In particular, it proposes that the internally generated synaesthetic percepts are treated similarly to other sensory information as the brain is learning the multisensory attributes of objects and developing cross-modal associations that merge in the concept of the object.

Separate Brain Circuits Support Integrative and Semantic Priming in the Human Language System

Semantic priming is a crucial phenomenon to study the organization of semantic memory. A novel type of priming effect, integrative priming, has been identified behaviorally, whereby a prime word facilitates recognition of a target word when the 2 concepts can be combined to form a unitary representation. We used both functional and anatomical imaging approaches to investigate the neural substrates supporting such integrative priming, and compare them with those in semantic priming. Similar behavioral priming effects for both semantic (Bread–Cake) and integrative conditions (Cherry–Cake) were observed when compared with an unrelated condition. However, a clearly dissociated brain response was observed between these 2 types of priming. The semantic-priming effect was localized to the posterior superior temporal and middle temporal gyrus. In contrast, the integrative-priming effect localized to the left anterior inferior frontal gyrus and left anterior temporal cortices. Furthermore, fiber tractography showed that the integrative-priming regions were connected via uncinate fasciculus fiber bundle forming an integrative circuit, whereas the semantic-priming regions connected to the posterior frontal cortex via separated pathways. The results point to dissociable neural pathways underlying the 2 distinct types of priming, illuminating the neural circuitry organization of semantic representation and integration.

Hemispheric lateralization of semantic feature distinctiveness

Recent models of semantic memory propose that the semantic representation of concepts is based, in part, on a network of features. In this view, a feature that is distinctive for an object (a zebra has stripes) is processed differently from a feature that is shared across many objects (a zebra has four legs). The goal of this paper is to determine whether there are hemispheric differences in such processing. In a feature verification task, participants responded 'yes' or 'no' following concepts which were presented to a single visual field (left or right) paired with a shared or distinctive feature. Both hemispheres showed faster reaction times to shared features than to distinctive features, although right hemisphere responses were significantly slower overall and particularly in the processing of distinctive features. These findings support models of semantic processing in which the dominant left hemisphere more efficiently performs highly discriminating 'fine' encoding, in contrast to the right hemisphere which performs less discriminating 'coarse' encoding.

Significance of objects in the perirhinal cortex

The perirhinal cortex is known to play a role in recognition memory and visual perception of objects. A recent single-unit recording study adds to our understanding of perirhinal cortex function, suggesting that it may also play a role in evaluating the significance of objects in a context-dependent manner.

Dynamic information processing states revealed through neurocognitive models of object semantics

Recognising objects relies on highly dynamic, interactive brain networks to process multiple aspects of object information. To fully understand how different forms of information about objects are represented and processed in the brain requires a neurocognitive account of visual object recognition that combines a detailed cognitive model of semantic knowledge with a neurobiological model of visual object processing. Here we ask how specific cognitive factors are instantiated in our mental processes and how they dynamically evolve over time. We suggest that coarse semantic information, based on generic shared semantic knowledge, is rapidly extracted from visual inputs and is sufficient to drive rapid category decisions. Subsequent recurrent neural activity between the anterior temporal lobe and posterior fusiform supports the formation of object-specific semantic representations - a conjunctive process primarily driven by the perirhinal cortex. These object-specific representations require the integration of shared and distinguishing object properties and support the unique recognition of objects. We conclude that a valuable way of understanding the cognitive activity of the brain is though testing the relationship between specific cognitive measures and dynamic neural activity. This kind of approach allows us to move towards uncovering the information processing states of the brain and how they evolve over time.

Basic level category structure emerges gradually across human ventral visual cortex

Objects can be simultaneously categorized at multiple levels of specificity ranging from very broad ("natural object") to very distinct ("Mr. Woof"), with a mid-level of generality (basic level: "dog") often providing the most cognitively useful distinction between categories. It is unknown, however, how this hierarchical representation is achieved in the brain. Using multivoxel pattern analyses, we examined how well each taxonomic level (superordinate, basic, and subordinate) of real-world object categories is represented across occipitotemporal cortex. We found that, although in early visual cortex objects are best represented at the subordinate level (an effect mostly driven by low-level feature overlap between objects in the same category), this advantage diminishes compared to the basic level as we move up the visual hierarchy, disappearing in object-selective regions of occipitotemporal cortex. This pattern stems from a combined increase in within-category similarity (category cohesion) and between-category dissimilarity (category distinctiveness) of neural activity patterns at the basic level, relative to both subordinate and superordinate levels, suggesting that successive visual areas may be optimizing basic level representations.

The perirhinal cortex and conceptual processing: Effects of feature-based statistics following damage to the anterior temporal lobes

The anterior temporal lobe (ATL) plays a prominent role in models of semantic knowledge, although it remains unclear how the specific subregions within the ATL contribute to semantic memory. Patients with neurodegenerative diseases, like semantic dementia, have widespread damage to the ATL thus making inferences about the relationship between anatomy and cognition problematic. Here we take a detailed anatomical approach to ask which substructures within the ATL contribute to conceptual processing, with the prediction that the perirhinal cortex (PRc) will play a critical role for concepts that are more semantically confusable. We tested two patient groups, those with and without damage to the PRc, across two behavioural experiments - picture naming and word-picture matching. For both tasks, we manipulated the degree of semantic confusability of the concepts. By contrasting the performance of the two groups, along with healthy controls, we show that damage to the PRc results in worse performance in processing concepts with higher semantic confusability across both experiments. Further by correlating the degree of damage across anatomically defined regions of interest with performance, we find that PRc damage is related to performance for concepts with increased semantic confusability. Our results show that the PRc supports a necessary and crucial neurocognitve function that enables fine-grained conceptual processes to take place through the resolution of semantic confusability.

The Representation of Object-Directed Action and Function Knowledge in the Human Brain

The appropriate use of everyday objects requires the integration of action and function knowledge. Previous research suggests that action knowledge is represented in frontoparietal areas while function knowledge is represented in temporal lobe regions. Here we used multivoxel pattern analysis to investigate the representation of object-directed action and function knowledge while participants executed pantomimes of familiar tool actions. A novel approach for decoding object knowledge was used in which classifiers were trained on one pair of objects and then tested on a distinct pair; this permitted a measurement of classification accuracy over and above object-specific information. Region of interest (ROI) analyses showed that object-directed actions could be decoded in tool-preferring regions of both parietal and temporal cortex, while no independently defined tool-preferring ROI showed successful decoding of object function. However, a whole-brain searchlight analysis revealed that while frontoparietal motor and peri-motor regions are engaged in the representation of object-directed actions, medial temporal lobe areas in the left hemisphere are involved in the representation of function knowledge. These results indicate that both action and function knowledge are represented in a topographically coherent manner that is amenable to study with multivariate approaches, and that the left medial temporal cortex represents knowledge of object function.

Delay-dependent contributions of medial temporal lobe regions to episodic memory retrieval

The medial temporal lobes play an important role in episodic memory, but over time, hippocampal contributions to retrieval may be diminished. However, it is unclear whether such changes are related to the ability to retrieve contextual information, and whether they are common across all medial temporal regions. Here, we used functional neuroimaging to compare neural responses during immediate and delayed recognition. Results showed that recollection-related activity in the posterior hippocampus declined after a 1-day delay. In contrast, activity was relatively stable in the anterior hippocampus and in neocortical areas. Multi-voxel pattern similarity analyses also revealed that anterior hippocampal patterns contained information about context during item recognition, and after a delay, context coding in this region was related to successful retention of context information. Together, these findings suggest that the anterior and posterior hippocampus have different contributions to memory over time and that neurobiological models of memory must account for these differences.

DOI:http://dx.doi.org/10.7554/eLife.05025.001

In 1953, an American man called Henry Molaison underwent surgery to remove the medial temporal lobes of his brain in an effort to cure him of severe epilepsy. After the surgery, his epilepsy was indeed improved, but he was left without the ability to form new memories. His case is now seen as one of the first demonstrations of the medial temporal lobes being involved in memory.

Beneath the surface of each medial temporal lobe is a structure called the hippocampus, which is essential for the formation of new memories. However, memories are not stored permanently within the hippocampus: instead they are transferred ultimately to the neocortex, which is the outer layer of the brain.

Some neuroscientists believe that the content of memories remains unchanged during this transfer, whereas others argue that memories are stripped of their context—that is, details of when and where they were acquired—before they reach the neocortex.

In a brain imaging experiment, Ritchey et al. have now attempted to distinguish between these two possibilities. Volunteers were asked to memorize sentences linking an object to a room, such as ‘the apple is in the bedroom’, on two occasions 24 hr apart. Immediately after the second session, the volunteers were asked to complete memory tests while lying in the brain scanner. In one test the volunteer was shown a list of objects and asked to identify those objects they could recall seeing in either of the training sessions, and to identify objects they recognised as familiar, even if they could not specifically remember seeing these objects during training sessions.

Analysis of the brain imaging data revealed that regions of the medial temporal lobes were more active when the volunteers recalled objects than when they recognised them as familiar. Moreover, for the ‘recall’ responses—in which the volunteers could still retrieve contextual information—the activity of the hippocampus depended on the age of the memories. The anterior (front) part of the hippocampus was active when subjects recalled either new memories or memories from 24 hr previously, whereas the posterior (rear) hippocampus was active only during the recall of new memories. In addition, patterns of activity observed in the anterior hippocampus could be used to determine which room was previously associated with the object. This suggests that contextual information is retained in the anterior hippocampus, but lost from the posterior hippocampus over time.

Overall the results of Ritchey et al. highlight the fact that individual components of the medial temporal lobes, including hippocampal subregions, have distinct roles in the storage of memories, with these roles also changing over time. Moreover, the data lend support to the idea that contextual information may be lost from memories before they reach the neocortex.

DOI:http://dx.doi.org/10.7554/eLife.05025.002