Integrative and distinctive coding of visual and conceptual object features in the ventral visual stream

Semantic structures facilitate threat memory integration throughout the medial temporal lobe and medial prefrontal cortex

Emotional experiences can profoundly impact our conceptual model of the world, modifying how we represent and remember a host of information even indirectly associated with that experienced in the past. Yet, how a new emotional experience infiltrates and spreads across pre-existing semantic knowledge structures (e.g., categories) is unknown. We used a modified aversive sensory preconditioning paradigm in fMRI (n = 35) to investigate whether threat memories integrate with a pre-established category to alter the representation of the entire category. We observed selective but transient changes in the representation of conceptually related items in the amygdala, medial prefrontal cortex, and occipitotemporal cortex following threat conditioning to a simple cue (geometric shape) pre-associated with a different, but related, set of category exemplars. These representational changes persisted beyond 24 h in the hippocampus and perirhinal cortex. Reactivation of the semantic category during threat conditioning, combined with activation of the hippocampus or medial prefrontal cortex, was predictive of subsequent amygdala reactivity toward novel category members at test. This provides evidence for online integration of emotional experiences into semantic categories, which then promotes threat generalization. Behaviorally, threat conditioning by proxy selectively and retroactively enhanced recognition memory and increased the perceived typicality of the semantic category indirectly associated with threat. These findings detail a complex route through which new emotional learning generalizes by modifying semantic structures built up over time and stored in memory as conceptual knowledge.

Distributed representations of behavior-derived object dimensions in the human visual system

Object vision is commonly thought to involve a hierarchy of brain regions processing increasingly complex image features, with high-level visual cortex supporting object recognition and categorization. However, object vision supports diverse behavioral goals, suggesting basic limitations of this category-centric framework. To address these limitations, we mapped a series of dimensions derived from a large-scale analysis of human similarity judgments directly onto the brain. Our results reveal broadly distributed representations of behaviorally-relevant information, demonstrating selectivity to a wide variety of novel dimensions while capturing known selectivities for visual features and categories. Behavior-derived dimensions were superior to categories at predicting brain responses, yielding mixed selectivity in much of visual cortex and sparse selectivity in category-selective clusters. This framework reconciles seemingly disparate findings regarding regional specialization, explaining category selectivity as a special case of sparse response profiles among representational dimensions, suggesting a more expansive view on visual processing in the human brain.

Brain-wide representation of social knowledge

Understanding how the human brain maps different dimensions of social conceptualizations remains a key unresolved issue. We performed a functional magnetic resonance imaging (MRI) study in which participants were exposed to audio definitions of personality traits and asked to simulate experiences associated with the concepts. Half of the concepts were affective (e.g. empathetic), and the other half were non-affective (e.g. intelligent). Orthogonally, half of the concepts were highly likable (e.g. sincere) and half were socially undesirable (e.g. liar). Behaviourally, we observed that the dimension of social desirability reflected the participant's subjective ratings better than affect. FMRI decoding results showed that both social desirability and affect could be decoded in local patterns of activity through distributed brain regions including the superior temporal, inferior frontal, precuneus and key nodes of the default mode network in posterior/anterior cingulate and ventromedial prefrontal cortex. Decoding accuracy was better for social desirability than affect. A representational similarity analysis further demonstrated that a deep language model significantly predicted brain activity associated with the concepts in bilateral regions of superior and anterior temporal lobes. The results demonstrate a brain-wide representation of social knowledge, involving default model network systems that support the multimodal simulation of social experience, with a further reliance on language-related preprocessing.

How the intrinsic functional connectivity patterns of the semantic network support semantic processing

Semantic processing, a core of language comprehension, involves the activation of brain regions dispersed extensively across the frontal, temporal, and parietal cortices that compose the semantic network. To comprehend the functional structure of this semantic network and how it prepares for semantic processing, we investigated its intrinsic functional connectivity (FC) and the relation between this pattern and semantic processing ability in a large sample from the Human Connectome Project (HCP) dataset. We first defined a well-studied brain network for semantic processing, and then we characterized the within-network connectivity (WNC) and the between-network connectivity (BNC) within this network using a voxel-based global brain connectivity (GBC) method based on resting-state functional magnetic resonance imaging (fMRI). The results showed that 97.73% of the voxels in the semantic network displayed considerably greater WNC than BNC, demonstrating that the semantic network is a fairly encapsulated network. Moreover, multiple connector hubs in the semantic network were identified after applying the criterion of WNC > 1 SD above the mean WNC of the semantic network. More importantly, three of these connector hubs (i.e., the left anterior temporal lobe, angular gyrus, and orbital part of the inferior frontal gyrus) were reliably associated with semantic processing ability. Our findings suggest that the three identified regions use WNC as the central mechanism for supporting semantic processing and that task-independent spontaneous connectivity in the semantic network is essential for semantic processing.

Experience transforms crossmodal object representations in the anterior temporal lobes

Combining information from multiple senses is essential to object recognition, core to the ability to learn concepts, make new inferences, and generalize across distinct entities. Yet how the mind combines sensory input into coherent crossmodal representations - the crossmodal binding problem - remains poorly understood. Here, we applied multi-echo fMRI across a 4-day paradigm, in which participants learned three-dimensional crossmodal representations created from well-characterized unimodal visual shape and sound features. Our novel paradigm decoupled the learned crossmodal object representations from their baseline unimodal shapes and sounds, thus allowing us to track the emergence of crossmodal object representations as they were learned by healthy adults. Critically, we found that two anterior temporal lobe structures - temporal pole and perirhinal cortex - differentiated learned from non-learned crossmodal objects, even when controlling for the unimodal features that composed those objects. These results provide evidence for integrated crossmodal object representations in the anterior temporal lobes that were different from the representations for the unimodal features. Furthermore, we found that perirhinal cortex representations were by default biased toward visual shape, but this initial visual bias was attenuated by crossmodal learning. Thus, crossmodal learning transformed perirhinal representations such that they were no longer predominantly grounded in the visual modality, which may be a mechanism by which object concepts gain their abstraction.

Representational formats of human memory traces

Neural representations are internal brain states that constitute the brain's model of the external world or some of its features. In the presence of sensory input, a representation may reflect various properties of this input. When perceptual information is no longer available, the brain can still activate representations of previously experienced episodes due to the formation of memory traces. In this review, we aim at characterizing the nature of neural memory representations and how they can be assessed with cognitive neuroscience methods, mainly focusing on neuroimaging. We discuss how multivariate analysis techniques such as representational similarity analysis (RSA) and deep neural networks (DNNs) can be leveraged to gain insights into the structure of neural representations and their different representational formats. We provide several examples of recent studies which demonstrate that we are able to not only measure memory representations using RSA but are also able to investigate their multiple formats using DNNs. We demonstrate that in addition to slow generalization during consolidation, memory representations are subject to semantization already during short-term memory, by revealing a shift from visual to semantic format. In addition to perceptual and conceptual formats, we describe the impact of affective evaluations as an additional dimension of episodic memories. Overall, these studies illustrate how the analysis of neural representations may help us gain a deeper understanding of the nature of human memory.

Mechanisms underlying category learning in the human ventral occipito-temporal cortex

The human ventral occipito-temporal cortex (VOTC) has evolved into specialized regions that process specific categories, such as words, tools, and animals. The formation of these areas is driven by bottom-up visual and top-down nonvisual experiences. However, the specific mechanisms through which top-down nonvisual experiences modulate category-specific regions in the VOTC are still unknown. To address this question, we conducted a study in which participants were trained for approximately 13 h to associate three sets of novel meaningless figures with different top-down nonvisual features: the wordlike category with word features, the non-wordlike category with nonword features, and the visual familiarity condition with no nonvisual features. Pre- and post-training functional MRI (fMRI) experiments were used to measure brain activity during stimulus presentation. Our results revealed that training induced a categorical preference for the two training categories within the VOTC. Moreover, the locations of two training category-specific regions exhibited a notable overlap. Remarkably, within the overlapping category-specific region, training resulted in a dissociation in activation intensity and pattern between the two training categories. These findings provide important insights into how different nonvisual categorical information is encoded in the human VOTC.

Harmonic memory signals in the human cerebral cortex induced by semantic relatedness of words

When we memorize multiple words simultaneously, semantic relatedness among those words assists memory. For example, the information about "apple", "banana," and "orange" will be connected via a common concept of "fruits" and become easy to retain and recall. Neural mechanisms underlying this semantic integration in verbal working memory remain unclear. Here I used electroencephalography (EEG) and investigated neural signals when healthy human participants memorized five nouns semantically related (Sem trial) or not (NonSem trial). The regularity of oscillatory signals (8-30 Hz) during the retention period was found to be lower in NonSem than Sem trials, indicating that memorizing words unrelated to each other induced a non-harmonic (irregular) waveform in the temporal cortex. These results suggest that (i) semantic features of a word are retained as a set of neural oscillations at specific frequencies and (ii) memorizing words sharing a common semantic feature produces harmonic brain responses through a resonance or integration (sharing) of the oscillatory signals.

Using deep neural networks to disentangle visual and semantic information in human perception and memory

Mental representations of familiar categories are composed of visual and semantic information. Disentangling the contributions of visual and semantic information in humans is challenging because they are intermixed in mental representations. Deep neural networks that are trained either on images or on text or by pairing images and text enable us now to disentangle human mental representations into their visual, visual-semantic and semantic components. Here we used these deep neural networks to uncover the content of human mental representations of familiar faces and objects when they are viewed or recalled from memory. The results show a larger visual than semantic contribution when images are viewed and a reversed pattern when they are recalled. We further reveal a previously unknown unique contribution of an integrated visual-semantic representation in both perception and memory. We propose a new framework in which visual and semantic information contribute independently and interactively to mental representations in perception and memory.

Semantic representations in inferior frontal and lateral temporal cortex during picture naming, reading, and repetition

Reading, naming, and repetition are classical neuropsychological tasks widely used in the clinic and psycholinguistic research. While reading and repetition can be accomplished by following a direct or an indirect route, pictures can be named only by means of semantic mediation. By means of fMRI multivariate pattern analysis, we evaluated whether this well-established fundamental difference at the cognitive level is associated at the brain level with a difference in the degree to which semantic representations are activated during these tasks. Semantic similarity between words was estimated based on a word association model. Twenty subjects participated in an event-related fMRI study where the three tasks were presented in pseudo-random order. Linear discriminant analysis of fMRI patterns identified a set of regions that allow to discriminate between words at a high level of word-specificity across tasks. Representational similarity analysis was used to determine whether semantic similarity was represented in these regions and whether this depended on the task performed. The similarity between neural patterns of the left Brodmann area 45 (BA45) and of the superior portion of the left supramarginal gyrus correlated with the similarity in meaning between entities during picture naming. In both regions, no significant effects were seen for repetition or reading. The semantic similarity effect during picture naming was significantly larger than the similarity effect during the two other tasks. In contrast, several regions including left anterior superior temporal gyrus and left ventral BA44/frontal operculum, among others, coded for semantic similarity in a task-independent manner. These findings provide new evidence for the dynamic, task-dependent nature of semantic representations in the left BA45 and a more task-independent nature of the representational activation in the lateral temporal cortex and ventral BA44/frontal operculum.

Parallel cognitive maps for multiple knowledge structures in the hippocampal formation

The hippocampal-entorhinal system uses cognitive maps to represent spatial knowledge and other types of relational information. However, objects can often be characterized by different types of relations simultaneously. How does the hippocampal formation handle the embedding of stimuli in multiple relational structures that differ vastly in their mode and timescale of acquisition? Does the hippocampal formation integrate different stimulus dimensions into one conjunctive map or is each dimension represented in a parallel map? Here, we reanalyzed human functional magnetic resonance imaging data from Garvert et al. (2017) that had previously revealed a map in the hippocampal formation coding for a newly learnt transition structure. Using functional magnetic resonance imaging adaptation analysis, we found that the degree of representational similarity in the bilateral hippocampus also decreased as a function of the semantic distance between presented objects. Importantly, while both map-like structures localized to the hippocampal formation, the semantic map was located in more posterior regions of the hippocampal formation than the transition structure and thus anatomically distinct. This finding supports the idea that the hippocampal-entorhinal system forms parallel cognitive maps that reflect the embedding of objects in diverse relational structures.

Representation of event and object concepts in ventral anterior temporal lobe and angular gyrus

Semantic knowledge includes understanding of objects and their features and also understanding of the characteristics of events. The hub-and-spoke theory holds that these conceptual representations rely on multiple information sources that are integrated in a central hub in the ventral anterior temporal lobes. The dual-hub theory expands this framework with the claim that the ventral anterior temporal lobe hub is specialized for object representation, while a second hub in angular gyrus is specialized for event representation. To test these ideas, we used representational similarity analysis, univariate and psychophysiological interaction analyses of fMRI data collected while participants processed object and event concepts (e.g. "an apple," "a wedding") presented as images and written words. Representational similarity analysis showed that angular gyrus encoded event concept similarity more than object similarity, although the left angular gyrus also encoded object similarity. Bilateral ventral anterior temporal lobes encoded both object and event concept structure, and left ventral anterior temporal lobe exhibited stronger coding for events. Psychophysiological interaction analysis revealed greater connectivity between left ventral anterior temporal lobe and right pMTG, and between right angular gyrus and bilateral ITG and middle occipital gyrus, for event concepts compared to object concepts. These findings support the specialization of angular gyrus for event semantics, though with some involvement in object coding, but do not support ventral anterior temporal lobe specialization for object concepts.

An exploration of neural predictors of treatment compliance in cognitive-behavioral group therapy for hoarding disorder

A persistent and influential barrier to effective cognitive-behavioral therapy (CBT) for patients with hoarding disorder (HD) is treatment retention and compliance. Recent research has suggested that HD patients have abnormal brain activity identified by functional magnetic resonance (fMRI) in regions often engaged for executive functioning (e.g., right superior frontal gyrus, anterior insula, and anterior cingulate), which raises questions about whether these abnormalities could relate to patients' ability to attend, understand, and engage in HD treatment. We examined data from 74 HD-diagnosed adults who completed fMRI-measured brain activity during a discarding task designed to elicit symptom-related brain dysfunction, exploring which regions' activity might predict treatment compliance variables, including treatment engagement (within-session compliance), homework completion (between-session compliance), and treatment attendance. Brain activity that was significantly related to within- and between-session compliance was found largely in insula, parietal, and premotor areas. No brain regions were associated with treatment attendance. The results add to findings from prior research that have found prefrontal, cingulate, and insula activity abnormalities in HD by suggesting that some aspects of HD brain dysfunction might play a role in preventing the engagement needed for therapeutic benefit.

Hippocampal Functions Modulate Transfer-Appropriate Cortical Representations Supporting Subsequent Memory

The hippocampus plays a central role as a coordinate system or index of information stored in neocortical loci. Nonetheless, it remains unclear how hippocampal processes integrate with cortical information to facilitate successful memory encoding. Thus, the goal of the current study was to identify specific hippocampal-cortical interactions that support object encoding. We collected fMRI data while 19 human participants (7 female and 12 male) encoded images of real-world objects and tested their memory for object concepts and image exemplars (i.e., conceptual and perceptual memory). Representational similarity analysis revealed robust representations of visual and semantic information in canonical visual (e.g., occipital cortex) and semantic (e.g., angular gyrus) regions in the cortex, but not in the hippocampus. Critically, hippocampal functions modulated the mnemonic impact of cortical representations that are most pertinent to future memory demands, or transfer-appropriate representations Subsequent perceptual memory was best predicted by the strength of visual representations in ventromedial occipital cortex in coordination with hippocampal activity and pattern information during encoding. In parallel, subsequent conceptual memory was best predicted by the strength of semantic representations in left inferior frontal gyrus and angular gyrus in coordination with either hippocampal activity or semantic representational strength during encoding. We found no evidence for transfer-incongruent hippocampal-cortical interactions supporting subsequent memory (i.e., no hippocampal interactions with cortical visual/semantic representations supported conceptual/perceptual memory). Collectively, these results suggest that diverse hippocampal functions flexibly modulate cortical representations of object properties to satisfy distinct future memory demands.Significance Statement The hippocampus is theorized to index pieces of information stored throughout the cortex to support episodic memory. Yet how hippocampal processes integrate with cortical representation of stimulus information remains unclear. Using fMRI, we examined various forms of hippocampal-cortical interactions during object encoding in relation to subsequent performance on conceptual and perceptual memory tests. Our results revealed novel hippocampal-cortical interactions that utilize semantic and visual representations in transfer-appropriate manners: conceptual memory supported by hippocampal modulation of frontoparietal semantic representations, and perceptual memory supported by hippocampal modulation of occipital visual representations. These findings provide important insights into the neural mechanisms underlying the formation of information-rich episodic memory and underscore the value of studying the flexible interplay between brain regions for complex cognition.

Recurrent connectivity supports higher-level visual and semantic object representations in the brain

Visual object recognition has been traditionally conceptualised as a predominantly feedforward process through the ventral visual pathway. While feedforward artificial neural networks (ANNs) can achieve human-level classification on some image-labelling tasks, it's unclear whether computational models of vision alone can accurately capture the evolving spatiotemporal neural dynamics. Here, we probe these dynamics using a combination of representational similarity and connectivity analyses of fMRI and MEG data recorded during the recognition of familiar, unambiguous objects. Modelling the visual and semantic properties of our stimuli using an artificial neural network as well as a semantic feature model, we find that unique aspects of the neural architecture and connectivity dynamics relate to visual and semantic object properties. Critically, we show that recurrent processing between the anterior and posterior ventral temporal cortex relates to higher-level visual properties prior to semantic object properties, in addition to semantic-related feedback from the frontal lobe to the ventral temporal lobe between 250 and 500 ms after stimulus onset. These results demonstrate the distinct contributions made by semantic object properties in explaining neural activity and connectivity, highlighting it as a core part of object recognition not fully accounted for by current biologically inspired neural networks.

The good, the bad, and the ambivalent: Extrapolating affective values for 38,000+ Chinese words via a computational model

Word affective ratings are important tools in psycholinguistic research, natural language processing, and many other fields. However, even for well-studied languages, such norms are usually limited in scale. To extrapolate affective (i.e., valence and arousal) values for words in the SUBTLEX-CH database (Cai & Brysbaert, 2010, PLoS ONE, 5(6):e10729), we implemented a computational neural network which captured how words' vector-based semantic representations corresponded to the probability densities of their valence and arousal. Based on these probability density functions, we predicted not only a word's affective values, but also their respective degrees of variability that could characterize individual differences in human affective ratings. The resulting estimates of affective values largely converged with human ratings for both valence and arousal, and the estimated degrees of variability also captured important features of the variability in human ratings. We released the extrapolated affective values, together with their corresponding degrees of variability, for over 38,000 Chinese words in the Open Science Framework ( https://osf.io/s9zmd/ ). We also discussed how the view of embodied cognition could be illuminated by this computational model.

Effects of Aging on Successful Object Encoding: Enhanced Semantic Representations Compensate for Impaired Visual Representations

Although episodic memory and visual processing decline substantially with healthy aging, semantic knowledge is generally spared. There is evidence that older adults' spared semantic knowledge can support episodic memory. Here, we used functional magnetic resonance imaging (fMRI) combined with representational similarity analyses (RSAs) to examine how novel visual and preexisting semantic representations at encoding predict subjective memory vividness at retrieval. Eighteen young and seventeen older adults (female and male participants) encoded images of objects during fMRI scanning and recalled these images while rating the vividness of their memories. After scanning, participants discriminated between studied images and similar lures. RSA based on a deep convolutional neural network and normative concept feature data were used to link patterns of neural activity during encoding to visual and semantic representations. Relative to young adults, the specificity of activation patterns for visual features was reduced in older adults, consistent with dedifferentiation. However, the specificity of activation patterns for semantic features was enhanced in older adults, consistent with hyperdifferentiation. Despite dedifferentiation, visual representations in early visual cortex (EVC) predicted high memory vividness in both age groups. In contrast, semantic representations in lingual gyrus (LG) and fusiform gyrus (FG) were associated with high memory vividness only in the older adults. Intriguingly, data suggests that older adults with lower specificity of visual representations in combination with higher specificity of semantic representations tended to rate their memories as more vivid. Our findings suggest that memory vividness in aging relies more on semantic representations over anterior regions, potentially compensating for age-related dedifferentiation of visual information in posterior regions.SIGNIFICANCE STATEMENT Normal aging is associated with impaired memory for events while semantic knowledge might even improve. We investigated the effects of aging on the specificity of visual and semantic information in the brain when viewing common objects and how this information enables subsequent memory vividness for these objects. Using functional magnetic resonance imaging (fMRI) combined with modeling of the stimuli we found that visual information was represented with less specificity in older than young adults while still supporting memory vividness. In contrast semantic information supported memory vividness only in older adults and especially in those individuals that had the lowest specificity of visual information. These findings provide evidence for a spared semantic memory system increasingly recruited to compensate for degraded visual representations in older age.

Anterior Hippocampal Engagement during Memory Formation Predicts Subsequent False Recognition of Similar Experiences

People better remember experiences when they orient to meaning over surface-level perceptual features. Such an orientation-related memory boost has been associated with engagement of both hippocampus (HPC) and neocortex during encoding. However, less is known about the neural mechanisms by which a cognitive orientation toward meaning might also promote memory errors, with one open question being whether the HPC-a region traditionally implicated in precise memory formation-also contributes to behavioral imprecision. We used fMRI to characterize encoding-phase signatures as people oriented toward the meaning (story) versus perceptual style (artist) of storybook-style illustrations and then linked them to subsequent true and false memories. We found that story and artist orientation tasks were each associated with both unique univariate profiles and consistent neural states defined using multivoxel patterns. Linking these neural signatures to behavior, we found that greater medial pFC activation and alignment of neural patterns to the story (but not artist) state was related to subsequent memory success on a trial-by-trial basis. Moreover, among successfully remembered experiences, greater anterior HPC engagement at encoding was associated with a higher likelihood of related false recognitions, consistent with the encoding of broad traces in this region. Interestingly, these effects did not reliably vary by cued orientation. These results suggest that, irrespective of the cued encoding orientation, neocortical and hippocampal mechanisms associated with orienting to meaning (story) over perceptual (artist) features may support memory, with the formation of generalizable memories being a specialty of anterior HPC.

Distinct Patterns of Connectivity between Brain Regions Underlie the Intra-Modal and Cross-Modal Value-Driven Modulations of the Visual Cortex

Past reward associations may be signaled from different sensory modalities; however, it remains unclear how different types of reward-associated stimuli modulate sensory perception. In this human fMRI study (female and male participants), a visual target was simultaneously presented with either an intra- (visual) or a cross-modal (auditory) cue that was previously associated with rewards. We hypothesized that, depending on the sensory modality of the cues, distinct neural mechanisms underlie the value-driven modulation of visual processing. Using a multivariate approach, we confirmed that reward-associated cues enhanced the target representation in early visual areas and identified the brain valuation regions. Then, using an effective connectivity analysis, we tested three possible patterns of connectivity that could underlie the modulation of the visual cortex: a direct pathway from the frontal valuation areas to the visual areas, a mediated pathway through the attention-related areas, and a mediated pathway that additionally involved sensory association areas. We found evidence for the third model demonstrating that the reward-related information in both sensory modalities is communicated across the valuation and attention-related brain regions. Additionally, the superior temporal areas were recruited when reward was cued cross-modally. The strongest dissociation between the intra- and cross-modal reward-driven effects was observed at the level of the feedforward and feedback connections of the visual cortex estimated from the winning model. These results suggest that, in the presence of previously rewarded stimuli from different sensory modalities, a combination of domain-general and domain-specific mechanisms are recruited across the brain to adjust the visual perception.SIGNIFICANCE STATEMENT Reward has a profound effect on perception, but it is not known whether shared or disparate mechanisms underlie the reward-driven effects across sensory modalities. In this human fMRI study, we examined the reward-driven modulation of the visual cortex by visual (intra-modal) and auditory (cross-modal) reward-associated cues. Using a model-based approach to identify the most plausible pattern of inter-regional effective connectivity, we found that higher-order areas involved in the valuation and attentional processing were recruited by both types of rewards. However, the pattern of connectivity between these areas and the early visual cortex was distinct between the intra- and cross-modal rewards. This evidence suggests that, to effectively adapt to the environment, reward signals may recruit both domain-general and domain-specific mechanisms.

Probing Lexical Ambiguity in Chinese Characters via Their Word Formations: Convergence of Perceived and Computed Metrics

Lexical ambiguity is pervasive in language, and the nature of the representations of an ambiguous word's multiple meanings is yet to be fully understood. With a special focus on Chinese characters, the present study first established that native speaker's perception about a character's number of meanings was heavily influenced by the availability of its distinct word formations, while whether these meanings would be perceived to be closely related was driven by further conceptual analysis. These notions were operationalized as two computed metrics, which assessed the degree of dispersion across individual word formations and the degree of propinquity across clusters of word formations, respectively, in a distributional semantic space. The observed correlations between the computed and the perceived metrics indicated that the utility of word formations to tap into meaning representations of Chinese characters was indeed cognitively plausible. The results have demonstrated the extent to which distributional semantics could inform about meaning representations of Chinese characters, which has theoretical implications for the representation of ambiguous words more generally.

Complex visual discrimination is impaired after right, but not left, anterior temporal lobectomy

The prevailing view in human cognitive neuroscience associates the medial temporal lobes (MTLs) with declarative memory. Compelling experimental evidence has, however, demonstrated that these regions are specialized according to the representations processed, irrespective of the cognitive domain assessed. This account was supported by the study of patients with bilateral medial temporal amnesia, who exhibit impairments in perceptual tasks involving complex visual stimuli. Yet, little is known regarding the impact of unilateral MTL damage on complex visual abilities. To address this issue, we administered a visual matching task to 20 patients who underwent left (N = 12) or right (N = 8) anterior temporal lobectomy for drug-resistant epilepsy and to 38 healthy controls. Presentation viewpoint was manipulated to increase feature ambiguity, as this is critical to reveal impairments in perceptual tasks. Similar to control participants, patients with left-sided damage succeeded in all task conditions. In contrast, patients with right-sided damage had decreased accuracy compared with that of the other two groups, as well as increased response time. Notably, the accuracy of those with right-sided damage did not exceed chance level when feature ambiguity was high (i.e., when stimuli were presented from different viewpoints) for the most complex classes of stimuli (i.e., scenes and buildings, compared with single objects). The pattern reported in bilateral patients in previous studies was therefore reproduced in patients with right, but not left, resection. These results suggest that the complex visual-representation functions supported by the MTL are right-lateralized, and raise the question as to how the representational account of these regions applies to representations supported by left MTL regions.

Perirhinal cortex is associated with fine-grained discrimination of conceptually confusable objects in Alzheimer's disease

The perirhinal cortex (PrC) stands among the first brain areas to deteriorate in Alzheimer's disease (AD). This study tests to what extent the PrC is involved in representing and discriminating confusable objects based on the conjunction of their perceptual and conceptual features. To this aim, AD patients and control counterparts performed 3 tasks: a naming, a recognition memory, and a conceptual matching task, where we manipulated conceptual and perceptual confusability. A structural MRI of the antero-lateral parahippocampal subregions was obtained for each participant. We found that the sensitivity to conceptual confusability was associated with the left PrC volume in both AD patients and control participants for the recognition memory task, while it was specifically associated with the volume of the left PrC in AD patients for the conceptual matching task. This suggests that a decreased volume of the PrC is related to the ability to disambiguate conceptually confusable items. Therefore, testing recognition memory or conceptual matching of easily conceptually confusable items can provide a potential cognitive marker of PrC atrophy.

Adding Meaning to Memories: How Parietal Cortex Combines Semantic Content with Episodic Experience

Neuroimaging studies of human memory have consistently found that univariate responses in parietal cortex track episodic experience with stimuli (whether stimuli are 'old' or 'new'). More recently, pattern-based fMRI studies have shown that parietal cortex also carries information about the semantic content of remembered experiences. However, it is not well understood how memory-based and content-based signals are integrated within parietal cortex. Here, in humans (males and females), we used voxel-wise encoding models and a recognition memory task to predict the fMRI activity patterns evoked by complex natural scene images based on (1) the episodic history and (2) the semantic content of each image. Models were generated and compared across distinct subregions of parietal cortex and for occipitotemporal cortex. We show that parietal and occipitotemporal regions each encode memory and content information, but they differ in how they combine this information. Among parietal subregions, angular gyrus was characterized by robust and overlapping effects of memory and content. Moreover, subject-specific semantic tuning functions revealed that successful recognition shifted the amplitude of tuning functions in angular gyrus but did not change the selectivity of tuning. In other words, effects of memory and content were additive in angular gyrus. This pattern of data contrasted with occipitotemporal cortex where memory and content effects were interactive: memory effects were preferentially expressed by voxels tuned to the content of a remembered image. Collectively, these findings provide unique insight into how parietal cortex combines information about episodic memory and semantic content.SIGNIFICANCE STATEMENT Neuroimaging studies of human memory have identified multiple brain regions that not only carry information about "whether" a visual stimulus is successfully recognized but also "what" the content of that stimulus includes. However, a fundamental and open question concerns how the brain integrates these two types of information (memory and content). Here, using a powerful combination of fMRI analysis methods, we show that parietal cortex, particularly the angular gyrus, robustly combines memory- and content-related information, but these two forms of information are represented via additive, independent signals. In contrast, memory effects in high-level visual cortex critically depend on (and interact with) content representations. Together, these findings reveal multiple and distinct ways in which the brain combines memory- and content-related information.

Perception and Memory in the Ventral Visual Stream and Medial Temporal Lobe

Perception and memory are traditionally thought of as separate cognitive functions, supported by distinct brain regions. The canonical perspective is that perceptual processing of visual information is supported by the ventral visual stream, whereas long-term declarative memory is supported by the medial temporal lobe. However, this modular framework cannot account for the increasingly large body of evidence that reveals a role for early visual areas in long-term recognition memory and a role for medial temporal lobe structures in high-level perceptual processing. In this article, we review relevant research conducted in humans, nonhuman primates, and rodents. We conclude that the evidence is largely inconsistent with theoretical proposals that draw sharp functional boundaries between perceptual and memory systems in the brain. Instead, the weight of the empirical findings is best captured by a representational-hierarchical model that emphasizes differences in content, rather than in cognitive processes within the ventral visual stream and medial temporal lobe.

Perirhinal cortex automatically tracks multiple types of familiarity regardless of task-relevance

Perirhinal cortex (PrC) has long been implicated in familiarity assessment for objects and corresponding concepts. However, extant studies have focused mainly on changes in familiarity induced by recent exposure in laboratory settings. There is an increasing appreciation of other types of familiarity signals, in particular graded familiarity accumulated throughout one's lifetime. In prior work (Duke et al., 2017, Cortex, 89, 61-70), PrC has been shown to track lifetime familiarity ratings when participants make related judgements. A theoretically important characteristic of familiarity is its proposed automaticity. Support for automaticity comes from a documented impact of recent stimulus exposure on behavioral performance, and on PrC signals, under conditions in which this exposure is not task relevant. In the current fMRI study, we tested whether PrC also tracks lifetime familiarity of object concepts automatically, and whether this type of familiarity influences behavior even when it is not task relevant. During scanning, neurotypical participants (N = 30, age range 18-40, 7 males) provided animacy judgements about concrete object concepts presented at differing frequencies in an initial study phase. In a subsequent test phase, they made graded judgements of recent or lifetime familiarity. Behavioral performance showed sensitivity to lifetime familiarity even when it was not relevant for the task at hand. Across five sets of fMRI analyses, we found that PrC consistently tracked recent and lifetime familiarity of object concepts regardless of the task performed. Critically, while several other temporal-lobe regions also showed isolated familiarity effects, none of them tracked familiarity with the same consistency. These findings demonstrate that PrC automatically tracks multiple types of familiarity. They support models that assign a broad role in the representation of information about object concepts to this structure.

Memory reactivation and suppression modulate integration of the semantic features of related memories in hippocampus

Encoding an event that overlaps with a previous experience may involve reactivating an existing memory and integrating it with new information or suppressing the existing memory to promote formation of a distinct, new representation. We used fMRI during overlapping event encoding to track reactivation and suppression of individual, related memories. We further used a model of semantic knowledge based on Wikipedia to quantify both reactivation of semantic knowledge related to a previous event and formation of integrated memories containing semantic features of both events. Representational similarity analysis revealed that reactivation of semantic knowledge related to a prior event in posterior medial prefrontal cortex (pmPFC) supported memory integration during new learning. Moreover, anterior hippocampus (aHPC) formed integrated representations combining the semantic features of overlapping events. We further found evidence that aHPC integration may be modulated on a trial-by-trial basis by interactions between ventrolateral PFC and anterior mPFC, with suppression of item-specific memory representations in anterior mPFC inhibiting hippocampal integration. These results suggest that PFC-mediated control processes determine the availability of specific relevant memories during new learning, thus impacting hippocampal memory integration.

Similar object shape representation encoded in the inferolateral occipitotemporal cortex of sighted and early blind people

We can sense an object's shape by vision or touch. Previous studies suggested that the inferolateral occipitotemporal cortex (ILOTC) implements supramodal shape representations as it responds more to seeing or touching objects than shapeless textures. However, such activation in the anterior portion of the ventral visual pathway could be due to the conceptual representation of an object or visual imagery triggered by touching an object. We addressed these possibilities by directly comparing shape and conceptual representations of objects in early blind (who lack visual experience/imagery) and sighted participants. We found that bilateral ILOTC in both groups showed stronger activation during a shape verification task than during a conceptual verification task made on the names of the same manmade objects. Moreover, the distributed activity in the ILOTC encoded shape similarity but not conceptual association among objects. Besides the ILOTC, we also found shape representation in both groups' bilateral ventral premotor cortices and intraparietal sulcus (IPS), a frontoparietal circuit relating to object grasping and haptic processing. In contrast, the conceptual verification task activated both groups' left perisylvian brain network relating to language processing and, interestingly, the cuneus in early blind participants only. The ILOTC had stronger functional connectivity to the frontoparietal circuit than to the left perisylvian network, forming a modular structure specialized in shape representation. Our results conclusively support that the ILOTC selectively implements shape representation independently of visual experience, and this unique functionality likely comes from its privileged connection to the frontoparietal haptic circuit.

The effect of memory load on object reconstruction: Insights from an online mouse-tracking task

Why can't we remember everything that we experience? Previous work in the domain of object memory has suggested that our ability to resolve interference between relevant and irrelevant object features may limit how much we can remember at any given moment. Here, we developed an online mouse-tracking task to study how memory load influences object reconstruction, testing participants synchronously over virtual conference calls. We first tested up to 18 participants concurrently, replicating memory findings from a condition where participants were tested individually. Next, we examined how memory load influenced mouse trajectories as participants reconstructed target objects. We found interference between the contents of working memory and what was perceived during object reconstruction, an effect that interacted with visual similarity and memory load. Furthermore, we found interference from previously studied but currently irrelevant objects, providing evidence of object-to-location binding errors. At the greatest memory load, participants were nearly three times more likely to move their mouse cursor over previously studied nontarget objects, an effect observed primarily during object reconstruction rather than in the period before the final response. As evidence of the dynamic interplay between working memory and perception, these results show that object reconstruction behavior may be altered by (i) interference between what is represented in mind and what is currently being viewed, and (ii) interference from previously studied but currently irrelevant information. Finally, we discuss how mouse tracking can provide a rich characterization of participant behavior at millisecond temporal resolution, enormously increasing power in cognitive psychology experiments.

Targeting the function of the transentorhinal cortex to identify early cognitive markers of Alzheimer's disease

Initial neuropathology of early Alzheimer's disease accumulates in the transentorhinal cortex. We review empirical data suggesting that tasks assessing cognitive functions supported by the transenthorinal cortex are impaired as early as the preclinical stages of Alzheimer's disease. These tasks span across various domains, including episodic memory, semantic memory, language, and perception. We propose that all tasks sensitive to Alzheimer-related transentorhinal neuropathology commonly rely on representations of entities supporting the processing and discrimination of items having perceptually and conceptually overlapping features. In the future, we suggest a screening tool that is sensitive and specific to very early Alzheimer's disease to probe memory and perceptual discrimination of highly similar entities.

Identifying causal subsequent memory effects

Over 40 y of accumulated research has detailed associations between neuroimaging signals measured during a memory encoding task and later memory performance, across a variety of brain regions, measurement tools, statistical approaches, and behavioral tasks. But the interpretation of these subsequent memory effects (SMEs) remains unclear: if the identified signals reflect cognitive and neural mechanisms of memory encoding, then the underlying neural activity must be causally related to future memory. However, almost all previous SME analyses do not control for potential confounders of this causal interpretation, such as serial position and item effects. We collect a large fMRI dataset and use an experimental design and analysis approach that allows us to statistically adjust for nearly all known exogenous confounding variables. We find that, using standard approaches without adjustment, we replicate several univariate and multivariate subsequent memory effects and are able to predict memory performance across people. However, we are unable to identify any signal that reliably predicts subsequent memory after adjusting for confounding variables, bringing into doubt the causal status of these effects. We apply the same approach to subjects' judgments of learning collected following an encoding period and show that these behavioral measures of mnemonic status do predict memory after adjustments, suggesting that it is possible to measure signals near the time of encoding that reflect causal mechanisms but that existing neuroimaging measures, at least in our data, may not have the precision and specificity to do so.

Extra-hippocampal contributions to pattern separation

Pattern separation, or the process by which highly similar stimuli or experiences in memory are represented by non-overlapping neural ensembles, has typically been ascribed to processes supported by the hippocampus. Converging evidence from a wide range of studies, however, suggests that pattern separation is a multistage process supported by a network of brain regions. Based on this evidence, considered together with related findings from the interference resolution literature, we propose the 'cortico-hippocampal pattern separation' (CHiPS) framework, which asserts that brain regions involved in cognitive control play a significant role in pattern separation. Particularly, these regions may contribute to pattern separation by (1) resolving interference in sensory regions that project to the hippocampus, thus regulating its cortical input, or (2) directly modulating hippocampal processes in accordance with task demands. Considering recent interest in how hippocampal operations are modulated by goal states likely represented and regulated by extra-hippocampal regions, we argue that pattern separation is similarly supported by neocortical-hippocampal interactions.

Semantic cognition in healthy ageing: Neural signatures of representation and control mechanisms in naming typical and atypical objects

Effective use of conceptual knowledge engages semantic representation and control processes to access information in a goal-driven manner. Neuropsychological findings of patients presenting either degraded knowledge (e.g., semantic dementia) or disrupted control (e.g., semantic aphasia) converge with neuroimaging evidence from young adults, and delineate the neural segregation of representation and control mechanisms. However, there is still scarce research on the neurofunctional underpinnings of such mechanisms in healthy ageing. To address this, we conducted an fMRI study, wherein young and older adults performed a covert naming task of typical and atypical objects. Three main age-related differences were found. As shown by age group and typicality interactions, older adults exhibited overactivation during naming of atypical (e.g., avocado) relative to typical concepts in brain regions associated to semantic representation, including anterior and medial portions of left temporal lobe (respectively, ATL and MTG). This provides evidence for the reorganization of neural activity in these brain regions contingent to the enrichment of semantic repositories in older ages. The medial orbitofrontal gyrus was also overactivated, indicating that the processing of atypical concepts (relative to typical items) taxes additional control resources in the elderly. Increased activation in the inferior frontal gyrus (IFG) was observed in naming typical items (relative to atypical ones), but only for young adults. This suggests that naming typical items (e.g., strawberry) taxes more on control processes in younger ages, presumably due to the semantic competition set by other items that share multiple features with the target (e.g., raspberry, blackberry, cherry). Together, these results reveal the dynamic nature of semantic control interplaying with conceptual representations as people grow older, by indicating that distinct neural bases uphold semantic performance from young to older ages. These findings may be explained by neural compensation mechanisms coming into play to support neurocognitive changes in healthy ageing.

Typicality in the brain during semantic and episodic memory decisions

Concept typicality is a key semantic dimension supporting the categorical organization of items based on their features, such that typical items share more features with other members of their category than atypical items, which are more distinctive. Typicality effects manifest in better accuracy and faster response times during categorization tasks, but higher performance for atypical items in episodic memory tasks, due to their distinctiveness. At a neural level, typicality has been linked to the anterior temporal lobe (ATL) and the inferior frontal gyrus (IFG) in semantic decision tasks, but patterns of brain activity during episodic memory tasks remain to be understood. We investigated the neural correlates of typicality in semantic and episodic memory to determine the brain regions associated with semantic typicality and uncover effects arising when items are reinstated during retrieval. In an fMRI study, 26 healthy young subjects first performed a category verification task on words representing typical and atypical concepts (encoding), and then completed a recognition memory task (retrieval). In line with previous literature, we observed higher accuracy and faster response times for typical items in the category verification task, while atypical items were better recognized in the episodic memory task. During category verification, univariate analyses revealed a greater involvement of the angular gyrus for typical items and the inferior frontal gyrus for atypical items. During the correct recognition of old items, regions belonging to the core recollection network were activated. We then compared the similarity of the representations from encoding to retrieval (ERS) using Representation Similarity Analyses. Results showed that typical items were reinstated more than atypical ones in several regions including the left precuneus and left anterior temporal lobe (ATL). This suggests that the correct retrieval of typical items requires finer-grained processing, evidenced by greater item-specific reinstatement, which is needed to resolve their confusability with other members of the category due to their higher feature similarity. Our findings confirm the centrality of the ATL in the processing of typicality while extending it to memory retrieval.

Decoding semantic representations in mind and brain

A key goal for cognitive neuroscience is to understand the neurocognitive systems that support semantic memory. Recent multivariate analyses of neuroimaging data have contributed greatly to this effort, but the rapid development of these novel approaches has made it difficult to track the diversity of findings and to understand how and why they sometimes lead to contradictory conclusions. We address this challenge by reviewing cognitive theories of semantic representation and their neural instantiation. We then consider contemporary approaches to neural decoding and assess which types of representation each can possibly detect. The analysis suggests why the results are heterogeneous and identifies crucial links between cognitive theory, data collection, and analysis that can help to better connect neuroimaging to mechanistic theories of semantic cognition.

From scanner to court: A neuroscientifically informed "reasonable person" test of trademark infringement

Many legal decisions center on the thoughts or perceptions of some idealized group of individuals, referred to variously as the "average person," "the typical consumer," or the "reasonable person." Substantial concerns exist, however, regarding the subjectivity and vulnerability to biases inherent in conventional means of assessing such responses, particularly the use of self-report evidence. Here, we addressed these concerns by complementing self-report evidence with neural data to inform the mental representations in question. Using an example from intellectual property law, we demonstrate that it is possible to construct a parsimonious neural index of visual similarity that can inform the reasonable person test of trademark infringement. Moreover, when aggregated across multiple participants, this index was able to detect experimenter-induced biases in self-report surveys in a sensitive and replicable fashion. Together, these findings potentially broaden the possibilities for neuroscientific data to inform legal decision-making across a range of settings.

Different computational relations in language are captured by distinct brain systems

A critical way for humans to acquire information is through language, yet whether and how language experience drives specific neural semantic representations is still poorly understood. We considered statistical properties captured by 3 different computational principles of language (simple co-occurrence, network-(graph)-topological relations, and neural-network-vector-embedding relations) and tested the extent to which they can explain the neural patterns of semantic representations, measured by 2 functional magnetic resonance imaging experiments that shared common semantic processes. Distinct graph-topological word relations, and not simple co-occurrence or neural-network-vector-embedding relations, had unique explanatory power for the neural patterns in the anterior temporal lobe (capturing graph-common-neighbors), inferior frontal gyrus, and posterior middle/inferior temporal gyrus (capturing graph-shortest-path). These results were relatively specific to language: they were not explained by sensory-motor similarities and the same computational relations of visual objects (based on visual image database) showed effects in the visual cortex in the picture naming experiment. That is, different topological properties within language and the same topological computations (common-neighbors) for language and visual inputs are captured by different brain regions. These findings reveal the specific neural semantic representations along graph-topological properties of language, highlighting the information type-specific and statistical property-specific manner of semantic representations in the human brain.

On the Application of Developmental Cognitive Neuroscience in Educational Environments

The paper overviews components of neurologic processing efficiencies to develop innovative methodologies and thinking to school-based applications and changes in educational leadership based on sound findings in the cognitive neurosciences applied to schools and learners. Systems science can allow us to better manage classroom-based learning and instruction on the basis of relatively easily evaluated efficiencies or inefficiencies and optimization instead of simply examining achievement. "Medicalizing" the learning process with concepts such as "learning disability" or employing grading methods such as pass-fail does little to aid in understanding the processes that learners employ to acquire, integrate, remember, and apply information learned. The paper endeavors to overview and provided reference to tools that can be employed that allow a better focus on nervous system-based strategic approaches to classroom learning.

Visual and semantic similarity norms for a photographic image stimulus set containing recognizable objects, animals and scenes

We collected visual and semantic similarity norms for a set of photographic images comprising 120 recognizable objects/animals and 120 indoor/outdoor scenes. Human observers rated the similarity of pairs of images within four categories of stimuli-inanimate objects, animals, indoor scenes and outdoor scenes-via Amazon's Mechanical Turk. We performed multidimensional scaling (MDS) on the collected similarity ratings to visualize the perceived similarity for each image category, for both visual and semantic ratings. The MDS solutions revealed the expected similarity relationships between images within each category, along with intuitively sensible differences between visual and semantic similarity relationships for each category. Stress tests performed on the MDS solutions indicated that the MDS analyses captured meaningful levels of variance in the similarity data. These stimuli, associated norms and naming data are made available to all researchers, and should provide a useful resource for researchers of vision, memory and conceptual knowledge wishing to run experiments using well-parameterized stimulus sets.

A Distributed Network for Multimodal Experiential Representation of Concepts

Neuroimaging, neuropsychological, and psychophysical evidence indicate that concept retrieval selectively engages specific sensory and motor brain systems involved in the acquisition of the retrieved concept. However, it remains unclear which supramodal cortical regions contribute to this process and what kind of information they represent. Here, we used representational similarity analysis of two large fMRI datasets with a searchlight approach to generate a detailed map of human brain regions where the semantic similarity structure across individual lexical concepts can be reliably detected. We hypothesized that heteromodal cortical areas typically associated with the default mode network encode multimodal experiential information about concepts, consistent with their proposed role as cortical integration hubs. In two studies involving different sets of concepts and different participants (both sexes), we found a distributed, bihemispheric network engaged in concept representation, composed of high-level association areas in the anterior, lateral, and ventral temporal lobe; inferior parietal lobule; posterior cingulate gyrus and precuneus; and medial, dorsal, ventrolateral, and orbital prefrontal cortex. In both studies, a multimodal model combining sensory, motor, affective, and other types of experiential information explained significant variance in the neural similarity structure observed in these regions that was not explained by unimodal experiential models or by distributional semantics (i.e., word2vec similarity). These results indicate that during concept retrieval, lexical concepts are represented across a vast expanse of high-level cortical regions, especially in the areas that make up the default mode network, and that these regions encode multimodal experiential information.SIGNIFICANCE STATEMENT Conceptual knowledge includes information acquired through various modalities of experience, such as visual, auditory, tactile, and emotional information. We investigated which brain regions encode mental representations that combine information from multiple modalities when participants think about the meaning of a word. We found that such representations are encoded across a widely distributed network of cortical areas in both hemispheres, including temporal, parietal, limbic, and prefrontal association areas. Several areas not traditionally associated with semantic cognition were also implicated. Our results indicate that the retrieval of conceptual knowledge during word comprehension relies on a much larger portion of the cerebral cortex than previously thought and that multimodal experiential information is represented throughout the entire network.

Representations of Temporal Community Structure in Hippocampus and Precuneus Predict Inductive Reasoning Decisions

Our understanding of the world is shaped by inferences about underlying structure. For example, at the gym, you might notice that the same people tend to arrive around the same time and infer that they are friends that work out together. Consistent with this idea, after participants are presented with a temporal sequence of objects that follows an underlying community structure, they are biased to infer that objects from the same community share the same properties. Here, we used fMRI to measure neural representations of objects after temporal community structure learning and examine how these representations support inference about object relationships. We found that community structure learning affected inferred object similarity: When asked to spatially group items based on their experience, participants tended to group together objects from the same community. Neural representations in perirhinal cortex predicted individual differences in object grouping, suggesting that high-level object representations are affected by temporal community learning. Furthermore, participants were biased to infer that objects from the same community would share the same properties. Using computational modeling of temporal learning and inference decisions, we found that inductive reasoning is influenced by both detailed knowledge of temporal statistics and abstract knowledge of the temporal communities. The fidelity of temporal community representations in hippocampus and precuneus predicted the degree to which temporal community membership biased reasoning decisions. Our results suggest that temporal knowledge is represented at multiple levels of abstraction, and that perirhinal cortex, hippocampus, and precuneus may support inference based on this knowledge.

Damage to the human dentate gyrus impairs the perceptual discrimination of complex, novel objects

The hippocampus (HPC), and the dentate gyrus (DG) subregion in particular, is purported to be a pattern separator, orthogonally representing similar information so that distinct memories may be formed. The HPC may also be involved in complex perceptual discrimination. It is unclear if this role is limited to spatial/scene stimuli or extends to the discrimination of objects. Also unclear is whether the DG itself contributes to pattern separation beyond memory. BL, an individual with bilateral DG lesions, was previously shown to have poor discrimination of similar, everyday objects in memory. Here, we demonstrate that BL's deficit extends to complex perceptual discrimination of novel objects. Specifically, BL was presented with closely matched possible and impossible objects, which give rise to fundamentally different 3D perceptual representations despite being visually similar. BL performed significantly worse than controls when asked to select an odd object (e.g., impossible) amongst three identical counterpart objects (e.g., possible) presented at different rotations. His deficit was also evident in an atypical eye fixation pattern during this task. In contrast, BL's performance was indistinguishable from that of controls on other tasks involving the same objects, indicating that he could visually differentiate the object pairs, that he perceived the objects holistically in 3D, and that he has only a mild weakness in categorizing object possibility. Furthermore, his performance on standardized neuropsychological measures indicated intact mental rotation, visual-spatial attention, and working memory (visual and auditory). Collectively, these results provide evidence that the DG is necessary for complex perceptual discrimination of novel objects, indicating that the DG might function as a generic pattern separator of a wide range of stimuli within high-level perception, and that its role is not limited to memory.

Semantic Expectation Effects on Object Detection: Using Figure Assignment to Elucidate Mechanisms

Recent evidence suggesting that object detection is improved following valid rather than invalid labels implies that semantics influence object detection. It is not clear, however, whether the results index object detection or feature detection. Further, because control conditions were absent and labels and objects were repeated multiple times, the mechanisms are unknown. We assessed object detection via figure assignment, whereby objects are segmented from backgrounds. Masked bipartite displays depicting a portion of a mono-oriented object (a familiar configuration) on one side of a central border were shown once only for 90 or 100 ms. Familiar configuration is a figural prior. Accurate detection was indexed by reports of an object on the familiar configuration side of the border. Compared to control experiments without labels, valid labels improved accuracy and reduced response times (RTs) more for upright than inverted objects (Studies 1 and 2). Invalid labels denoting different superordinate-level objects (DSC; Study 1) or same superordinate-level objects (SSC; Study 2) reduced accuracy for upright displays only. Orientation dependency indicates that effects are mediated by activated object representations rather than features which are invariant over orientation. Following invalid SSC labels (Study 2), accurate detection RTs were longer than control for both orientations, implicating conflict between semantic representations that had to be resolved before object detection. These results demonstrate that object detection is not just affected by semantics, it entails semantics.

Activity in perirhinal and entorhinal cortex predicts perceived visual similarities among category exemplars with highest precision

Vision neuroscience has made great strides in understanding the hierarchical organization of object representations along the ventral visual stream (VVS). How VVS representations capture fine-grained visual similarities between objects that observers subjectively perceive has received limited examination so far. In the current study, we addressed this question by focussing on perceived visual similarities among subordinate exemplars of real-world categories. We hypothesized that these perceived similarities are reflected with highest fidelity in neural activity patterns downstream from inferotemporal regions, namely in perirhinal (PrC) and anterolateral entorhinal cortex (alErC) in the medial temporal lobe. To address this issue with functional magnetic resonance imaging (fMRI), we administered a modified 1-back task that required discrimination between category exemplars as well as categorization. Further, we obtained observer-specific ratings of perceived visual similarities, which predicted behavioural discrimination performance during scanning. As anticipated, we found that activity patterns in PrC and alErC predicted the structure of perceived visual similarity relationships among category exemplars, including its observer-specific component, with higher precision than any other VVS region. Our findings provide new evidence that subjective aspects of object perception that rely on fine-grained visual differentiation are reflected with highest fidelity in the medial temporal lobe.

Context free and context-dependent conceptual representation in the brain

How concepts are coded in the brain is a core issue in cognitive neuroscience. Studies have focused on how individual concepts are processed, but the way in which conceptual representation changes to suit the context is unclear. We parametrically manipulated the association strength between words, presented in pairs one word at a time using a slow event-related fMRI design. We combined representational similarity analysis and computational linguistics to probe the neurocomputational content of these trials. Individual word meaning was maintained in supramarginal gyrus (associated with verbal short-term memory) when items were judged to be unrelated, but not when a linking context was retrieved. Context-dependent meaning was instead represented in left lateral prefrontal gyrus (associated with controlled retrieval), angular gyrus, and ventral temporal lobe (regions associated with integrative aspects of memory). Analyses of informational connectivity, examining the similarity of activation patterns across trials between sites, showed that control network regions had more similar multivariate responses across trials when association strength was weak, reflecting a common controlled retrieval state when the task required more unusual associations. These findings indicate that semantic control and representational sites amplify contextually relevant meanings in trials judged to be related.

Modelling brain representations of abstract concepts

Abstract conceptual representations are critical for human cognition. Despite their importance, key properties of these representations remain poorly understood. Here, we used computational models of distributional semantics to predict multivariate fMRI activity patterns during the activation and contextualization of abstract concepts. We devised a task in which participants had to embed abstract nouns into a story that they developed around a given background context. We found that representations in inferior parietal cortex were predicted by concept similarities emerging in models of distributional semantics. By constructing different model families, we reveal the models’ learning trajectories and delineate how abstract and concrete training materials contribute to the formation of brain-like representations. These results inform theories about the format and emergence of abstract conceptual representations in the human brain.

How do we conceive abstract concepts, like love, peace, or truth? In this study, we investigate how our brains support the activation and contextualization of such abstract concepts. We asked participants to embed abstract nouns into a coherent story while we recorded functional MRI. Using multivariate analysis techniques, we computed how similar different abstract concepts were represented during this task. We then modelled these neural similarities among concepts with computational models of distributional semantics which capture the words’ co-occurance statistics in large natural language corpora. Our results reveal a correspondence between the computational models and brain representations in the inferior parietal cortex. This correspondence held when the computational models were only trained on subsets of the corpora that contained as few as 100,000 sentences and only abstract or concrete words. Our findings establish a neural correlate of abstract concept representation in the inferior parietal cortex, and they provide a first characterization of the format of these representations.

Distinct medial-tempora lobe mechanisms of encoding and amygdala-mediated memory reinstatement for disgust and fear

Current models of episodic memory posit that retrieval involves the reenactment of encoding processes. Recent evidence has shown that this reinstatement process - indexed by subsequent encoding-retrieval similarity of brain activity patterns - is related to the activity in the hippocampus during encoding. However, we tend to re-experience emotional events in memory more richly than dull events. The role of amygdala - a critical hub of emotion processing - in reinstatement of emotional events was poorly understood. To investigate it, we leveraged a previously overlooked divergence in the role of amygdala in memory modulation by distinct emotions - disgust and fear. Here we used a novel paradigm in which participants encoded complex events (word pairs) and their memory was tested after 3 weeks, both phases during fMRI scanning. Using representational similarity analysis and univariate analyses, we show that the strength of amygdala activation during encoding was correlated with memory reinstatement of individual event representations in emotion-specific regions. Critically, amygdala modulated reinstatement more for disgust than fear. This was in line with other differences observed at the level of memory performance and neural mechanisms of encoding. Specifically, amygdala and perirhinal cortex were more involved during encoding of disgust-related events, whereas hippocampus and parahippocampal gyrus during encoding of fear-related events. Together, these findings shed a new light on the role of the amygdala and medial temporal lobe regions in encoding and reinstatement of specific emotional memories.

Unique, Shared, and Dominant Brain Activation in Visual Word Form Area and Lateral Occipital Complex during Reading and Picture Naming

Identifying printed words and pictures concurrently is ubiquitous in daily tasks, and so it is important to consider the extent to which reading words and naming pictures may share a cognitive-neurophysiological functional architecture. Two functional magnetic resonance imaging (fMRI) experiments examined whether reading along the left ventral occipitotemporal region (vOT; often referred to as a visual word form area, VWFA) has activation that is overlapping with referent pictures (i.e., both conditions significant and shared, or with one significantly more dominant) or unique (i.e., one condition significant, the other not), and whether picture naming along the right lateral occipital complex (LOC) has overlapping or unique activation relative to referent words. Experiment 1 used familiar regular and exception words (to force lexical reading) and their corresponding pictures in separate naming blocks, and showed dominant activation for pictures in the LOC, and shared activation in the VWFA for exception words and their corresponding pictures (regular words did not elicit significant VWFA activation). Experiment 2 controlled for visual complexity by superimposing the words and pictures and instructing participants to either name the word or the picture, and showed primarily shared activation in the VWFA and LOC regions for both word reading and picture naming, with some dominant activation for pictures in the LOC. Overall, these results highlight the importance of including exception words to force lexical reading when comparing to picture naming, and the significant shared activation in VWFA and LOC serves to challenge specialized models of reading or picture naming.

Activation of CB1 pathway in the perirhinal cortex is necessary but not sufficient for destabilization of contextual fear memory in rats

According to the reconsolidation theory, memories can be modified through the destabilization-reconsolidation process. The rodent perirhinal cortex (PER; Brodmann areas 35 and 36) critically participates in the process of fear conditioning. Previous studies showed that some of the parahippocampal regions are critical for contextual fear memory reconsolidation. In our research, through a three-day paradigm of CFC, we showed that protein synthesis in PER of rats is required for memory reconsolidation, and activation of CB1 pathway is necessary but not sufficient in inducing memory destabilization. This result underlines parahippocampal regions in destabilization and reconsolidation process of fear memory besides amygdala and hippocampus.

Increasing stimulus similarity drives nonmonotonic representational change in hippocampus

Studies of hippocampal learning have obtained seemingly contradictory results, with manipulations that increase coactivation of memories sometimes leading to differentiation of these memories, but sometimes not. These results could potentially be reconciled using the nonmonotonic plasticity hypothesis, which posits that representational change (memories moving apart or together) is a U-shaped function of the coactivation of these memories during learning. Testing this hypothesis requires manipulating coactivation over a wide enough range to reveal the full U-shape. To accomplish this, we used a novel neural network image synthesis procedure to create pairs of stimuli that varied parametrically in their similarity in high-level visual regions that provide input to the hippocampus. Sequences of these pairs were shown to human participants during high-resolution fMRI. As predicted, learning changed the representations of paired images in the dentate gyrus as a U-shaped function of image similarity, with neural differentiation occurring only for moderately similar images.