Representational similarity analysis reveals task-dependent semantic influence of the visual word form area

Beyond the Visual Word Form Area - a cognitive characterization of the left ventral occipitotemporal cortex

The left ventral occipitotemporal cortex has been traditionally viewed as a pathway for visual object recognition including written letters and words. Its crucial role in reading was strengthened by the studies on the functionally localized "Visual Word Form Area" responsible for processing word-like information. However, in the past 20 years, empirical studies have challenged the assumptions of this brain region as processing exclusively visual or even orthographic stimuli. In this review, we aimed to present the development of understanding of the left ventral occipitotemporal cortex from the visually based letter area to the modality-independent symbolic language related region. We discuss theoretical and empirical research that includes orthographic, phonological, and semantic properties of language. Existing results showed that involvement of the left ventral occipitotemporal cortex is not limited to unimodal activity but also includes multimodal processes. The idea of the integrative nature of this region is supported by the broad functional and structural connectivity with language-related and attentional brain networks. We conclude that although the function of the area is not yet fully understood in human cognition, its role goes beyond visual word form processing. The left ventral occipitotemporal cortex seems to be crucial for combining higher-level language information with abstract forms that convey meaning independently of modality.

The depth of semantic processing modulates cross-language pattern similarity in Chinese-English bilinguals

Previous studies have investigated factors related to the degree of cross-language overlap in brain activations in bilinguals/multilinguals. However, it is still unclear whether and how the depth of semantic processing (a critical task-related factor) affects the neural pattern similarity between native and second languages. To address this question, 26 Chinese-English bilinguals were scanned with fMRI while performing a word naming task (i.e., a task with shallow semantic processing) and a semantic judgment task (i.e., a task with deep semantic processing) in both native and second languages. Based on three sets of representational similarity analysis (whole brain, ROI-based, and within-language vs. cross-language semantic representation), we found that select regions in the reading brain network showed higher cross-language pattern similarity and higher cross-language semantic representations during deep semantic processing than during shallow semantic processing. These results suggest that compared to shallow semantic processing, deep semantic processing may lead to greater language-independent processing (i.e., cross-language semantic representation) and cross-language pattern similarity, and provide direct quantitative neuroimaging evidence for cognitive models of bilingual lexical memory.

Machine-learning as a validated tool to characterize individual differences in free recall of naturalistic events

The use of naturalistic stimuli, such as narrative movies, is gaining popularity in many fields, characterizing memory, affect, and decision-making. Narrative recall paradigms are often used to capture the complexity and richness of memory for naturalistic events. However, scoring narrative recalls is time-consuming and prone to human biases. Here, we show the validity and reliability of using a natural language processing tool, the Universal Sentence Encoder (USE), to automatically score narrative recalls. We compared the reliability in scoring made between two independent raters (i.e., hand scored) and between our automated algorithm and individual raters (i.e., automated) on trial-unique video clips of magic tricks. Study 1 showed that our automated segmentation approaches yielded high reliability and reflected measures yielded by hand scoring. Study 1 further showed that the results using USE outperformed another popular natural language processing tool, GloVe. In Study 2, we tested whether our automated approach remained valid when testing individuals varying on clinically relevant dimensions that influence episodic memory, age, and anxiety. We found that our automated approach was equally reliable across both age groups and anxiety groups, which shows the efficacy of our approach to assess narrative recall in large-scale individual difference analysis. In sum, these findings suggested that machine learning approach implementing USE is a promising tool for scoring large-scale narrative recalls and perform individual difference analysis for research using naturalistic stimuli.

Intersecting distributed networks support convergent linguistic functioning across different languages in bilinguals

How bilingual brains accomplish the processing of more than one language has been widely investigated by neuroimaging studies. The assimilation-accommodation hypothesis holds that both the same brain neural networks supporting the native language and additional new neural networks are utilized to implement second language processing. However, whether and how this hypothesis applies at the finer-grained levels of both brain anatomical organization and linguistic functions remains unknown. To address this issue, we scanned Chinese-English bilinguals during an implicit reading task involving Chinese words, English words and Chinese pinyin. We observed broad brain cortical regions wherein interdigitated distributed neural populations supported the same cognitive components of different languages. Although spatially separate, regions including the opercular and triangular parts of the inferior frontal gyrus, temporal pole, superior and middle temporal gyrus, precentral gyrus and supplementary motor areas were found to perform the same linguistic functions across languages, indicating regional-level functional assimilation supported by voxel-wise anatomical accommodation. Taken together, the findings not only verify the functional independence of neural representations of different languages, but show co-representation organization of both languages in most language regions, revealing linguistic-feature specific accommodation and assimilation between first and second languages.

The Brain Connectome for Chinese Reading

Chinese, as a logographic language, fundamentally differs from alphabetic languages like English. Previous neuroimaging studies have mainly focused on alphabetic languages, while the exploration of Chinese reading is still an emerging and fast-growing research field. Recently, a growing number of neuroimaging studies have explored the neural circuit of Chinese reading. Here, we summarize previous research on Chinese reading from a connectomic perspective. Converging evidence indicates that the left middle frontal gyrus is a specialized hub region that connects the ventral with dorsal pathways for Chinese reading. Notably, the orthography-to-phonology and orthography-to-semantics mapping, mainly processed in the ventral pathway, are more specific during Chinese reading. Besides, in addition to the left-lateralized language-related regions, reading pathways in the right hemisphere also play an important role in Chinese reading. Throughout, we comprehensively review prior findings and emphasize several challenging issues to be explored in future work.

Semantic flow and its relation to controlled semantic retrieval deficits in the narrative production of people with aphasia

Aphasia has had a profound influence on our understanding of how language is instantiated within the human brain. Historically, aphasia has yielded an in vivo model for elucidating the effects of impaired lexical-semantic access on language comprehension and production. Aphasiology has focused intensively on single word dissociations. Yet, less is known about the integrity of combinatorial semantic processes required to construct well-formed narratives. Here we addressed the question of how controlled lexical-semantic retrieval deficits (a hallmark of aphasia) might compound over the course of longer narratives. We specifically examined word-by-word flow of taxonomic vs. thematic semantic distance in the storytelling narratives of individuals with chronic post-stroke aphasia (n = 259) relative to age-matched controls (n = 203). We first parsed raw transcribed narratives into content words and computed inter-word semantic distances for every running pair of words in each narrative (N = 232,490 word transitions). The narratives of people with aphasia showed significant reductions in taxonomic and thematic semantic distance relative to controls. Both distance metrics were strongly predictive of offline measures of semantic impairment and aphasia severity. Since individuals with aphasia often exhibit perseverative language output (i.e., repetitions), we performed additional analyses with repetitions excluded. When repetitions were excluded, group differences in semantic distances persisted and thematic distance was still predictive of semantic impairment, although some findings changed. These results demonstrate the cumulative impact of deficits in controlled word retrieval over the course of narrative production. We discuss the nature of semantic flow between words as a novel metric of characterizing discourse and elucidating the nature of lexical-semantic access impairment in aphasia at multiword levels.

Modelling brain representations of abstract concepts

Abstract conceptual representations are critical for human cognition. Despite their importance, key properties of these representations remain poorly understood. Here, we used computational models of distributional semantics to predict multivariate fMRI activity patterns during the activation and contextualization of abstract concepts. We devised a task in which participants had to embed abstract nouns into a story that they developed around a given background context. We found that representations in inferior parietal cortex were predicted by concept similarities emerging in models of distributional semantics. By constructing different model families, we reveal the models’ learning trajectories and delineate how abstract and concrete training materials contribute to the formation of brain-like representations. These results inform theories about the format and emergence of abstract conceptual representations in the human brain.

How do we conceive abstract concepts, like love, peace, or truth? In this study, we investigate how our brains support the activation and contextualization of such abstract concepts. We asked participants to embed abstract nouns into a coherent story while we recorded functional MRI. Using multivariate analysis techniques, we computed how similar different abstract concepts were represented during this task. We then modelled these neural similarities among concepts with computational models of distributional semantics which capture the words’ co-occurance statistics in large natural language corpora. Our results reveal a correspondence between the computational models and brain representations in the inferior parietal cortex. This correspondence held when the computational models were only trained on subsets of the corpora that contained as few as 100,000 sentences and only abstract or concrete words. Our findings establish a neural correlate of abstract concept representation in the inferior parietal cortex, and they provide a first characterization of the format of these representations.

A data-driven framework for mapping domains of human neurobiology

Functional neuroimaging has been a mainstay of human neuroscience for the past 25 years. Interpretation of fMRI data has often occurred within knowledge frameworks crafted by experts, which have the potential to amplify biases that limit the replicability of findings. Here, we employ a computational approach to derive a data-driven framework for neurobiological domains that synthesizes the texts and data of nearly 20,000 human neuroimaging articles. Across multiple levels of domain specificity, the structure-function links within domains better replicate in held-out articles than those mapped from dominant frameworks in neuroscience and psychiatry. We further show that the data-driven framework partitions the literature into modular subfields, for which domains serve as generalizable prototypes of structure-function patterns in single articles. The approach to computational ontology we present here is the most comprehensive characterization of human brain circuits quantifiable with fMRI and may be extended to synthesize other scientific literatures.

Scene context shapes category representational geometry during processing of tools

Tools are ubiquitous in human environments and to think about them we use concepts. Increasingly, conceptual representation is thought to be dynamic and sensitive to the goals of the observer. Indeed, observer goals can reshape representational geometry within cortical networks supporting concepts. In the present study, we investigated the novel hypothesis that task-irrelevant scene context may implicitly alter the representational geometry of regions within the tool network. Participants performed conceptual judgments on images of tools embedded in scenes that either suggested their use (i.e., a kitchen timer sitting on a kitchen counter with vegetables in a frying pan) or that they would simply be moved (i.e., a kitchen timer sitting in an open drawer with other miscellaneous kitchen items around). We investigated whether representations in the tool network reflect category, grip, and shape information using a representational similarity analysis (RSA). We show that a) a number of regions of the tool network reflect category information about tools and b) category information predicts patterns in supramarginal gyrus more strongly in use contexts than in move contexts. Together, these results show that information about tool category is distributed across different regions of the tool network and that scene context helps shape the representational geometry of the tool network.

How usefulness shapes neural representations during goal-directed behavior

Value is often associated with reward, emphasizing its hedonic aspects. However, when circumstances change, value must also change (a compass outvalues gold, if you are lost). How are value representations in the brain reshaped under different behavioral goals? To answer this question, we devised a new task that decouples usefulness from its hedonic attributes, allowing us to study flexible goal-dependent mapping. Here, we show that, unlike sensory cortices, regions in the prefrontal cortex (PFC)-usually associated with value computation-remap their representation of perceptually identical items according to how useful the item has been to achieve a specific goal. Furthermore, we identify a coding scheme in the PFC that represents value regardless of the goal, thus supporting generalization across contexts. Our work questions the dominant view that equates value with reward, showing how a change in goals triggers a reorganization of the neural representation of value, enabling flexible behavior.

NeuroRA: A Python Toolbox of Representational Analysis From Multi-Modal Neural Data

In studies of cognitive neuroscience, multivariate pattern analysis (MVPA) is widely used as it offers richer information than traditional univariate analysis. Representational similarity analysis (RSA), as one method of MVPA, has become an effective decoding method based on neural data by calculating the similarity between different representations in the brain under different conditions. Moreover, RSA is suitable for researchers to compare data from different modalities and even bridge data from different species. However, previous toolboxes have been made to fit specific datasets. Here, we develop NeuroRA, a novel and easy-to-use toolbox for representational analysis. Our toolbox aims at conducting cross-modal data analysis from multi-modal neural data (e.g., EEG, MEG, fNIRS, fMRI, and other sources of neruroelectrophysiological data), behavioral data, and computer-simulated data. Compared with previous software packages, our toolbox is more comprehensive and powerful. Using NeuroRA, users can not only calculate the representational dissimilarity matrix (RDM), which reflects the representational similarity among different task conditions and conduct a representational analysis among different RDMs to achieve a cross-modal comparison. Besides, users can calculate neural pattern similarity (NPS), spatiotemporal pattern similarity (STPS), and inter-subject correlation (ISC) with this toolbox. NeuroRA also provides users with functions performing statistical analysis, storage, and visualization of results. We introduce the structure, modules, features, and algorithms of NeuroRA in this paper, as well as examples applying the toolbox in published datasets.

Knowing what you need to know in advance: The neural processes underpinning flexible semantic retrieval of thematic and taxonomic relations

Semantic retrieval is flexible, allowing us to focus on subsets of features and associations that are relevant to the current task or context: for example, we use taxonomic relations to locate items in the supermarket (carrots are a vegetable), but thematic associations to decide which tools we need when cooking (carrot goes with peeler). We used fMRI to investigate the neural basis of this form of semantic flexibility; in particular, we asked how retrieval unfolds differently when participants have advanced knowledge of the type of link to retrieve between concepts (taxonomic or thematic). Participants performed a semantic relatedness judgement task: on half the trials, they were cued to search for a taxonomic or thematic link, while on the remaining trials, they judged relatedness without knowing which type of semantic relationship would be relevant. Left inferior frontal gyrus showed greater activation when participants knew the trial type in advance. An overlapping region showed a stronger response when the semantic relationship between the items was weaker, suggesting this structure supports both top-down and bottom-up forms of semantic control. Multivariate pattern analysis further revealed that the neural response in left inferior frontal gyrus reflects goal information related to different conceptual relationships. Top-down control specifically modulated the response in visual cortex: when the goal was unknown, there was greater deactivation to the first word, and greater activation to the second word. We conclude that top-down control of semantic retrieval is primarily achieved through the gating of task-relevant 'spoke' regions.

Neural Components of Reading Revealed by Distributed and Symbolic Computational Models

Determining how the cognitive components of reading - orthographic, phonological, and semantic representations - are instantiated in the brain has been a longstanding goal of psychology and human cognitive neuroscience. The two most prominent computational models of reading instantiate different cognitive processes, implying different neural processes. Artificial neural network (ANN) models of reading posit non-symbolic, distributed representations. The dual-route cascaded (DRC) model instead suggests two routes of processing, one representing symbolic rules of spelling-sound correspondence, the other representing orthographic and phonological lexicons. These models are not adjudicated by behavioral data and have never before been directly compared in terms of neural plausibility. We used representational similarity analysis to compare the predictions of these models to neural data from participants reading aloud. Both the ANN and DRC model representations corresponded with neural activity. However, ANN model representations correlated to more reading-relevant areas of cortex. When contributions from the DRC model were statistically controlled, partial correlations revealed that the ANN model accounted for significant variance in the neural data. The opposite analysis, examining the variance explained by the DRC model with contributions from the ANN model factored out, revealed no correspondence to neural activity. Our results suggest that ANNs trained using distributed representations provide a better correspondence between cognitive and neural coding. Additionally, this framework provides a principled approach for comparing computational models of cognitive function to gain insight into neural representations.

Neural representational similarity between L1 and L2 in spoken and written language processing

Despite substantial research on the brain mechanisms of L1 and L2 processing in bilinguals, it is still unknown whether language modality (i.e., visual vs. auditory) plays a role in determining whether L1 and L2 are processed similarly. Therefore, we examined the neural representational similarity in neural networks between L1 and L2 in spoken and written word processing in Korean-English-Chinese trilinguals. Participants performed both visual and auditory rhyming judgments in the three languages: Korean, English, and Chinese. The results showed greater similarity among the three languages in the auditory modality than in the visual modality, suggesting more differentiated networks for written word processing in the three languages than spoken word processing. In addition, there was less similarity between spoken and written word processing in L1 than the L2s, suggesting a more specialized network for each modality in L1 than L2s. Finally, the similarity between the two L2s (i.e., Chinese and English) was greater than that between each L2 and L1 after task performance was regressed out, especially in the visual modality, suggesting that L2s are processed similarly. These findings provide important insights about spoken and written language processing in the bilingual brain.

Dissociating the functions of superior and inferior parts of the left ventral occipito-temporal cortex during visual word and object processing

During word and object recognition, extensive activation has consistently been observed in the left ventral occipito-temporal cortex (vOT), focused around the occipito-temporal sulcus (OTs). Previous studies have shown that there is a hierarchy of responses from posterior to anterior vOT regions (along the y-axis) that corresponds with increasing levels of recognition - from perceptual to semantic processing, respectively. In contrast, the functional differences between superior and inferior vOT responses (i.e. along the z-axis) have not yet been elucidated. To investigate, we conducted an extensive review of the literature and found that peak activation for reading varies by more than 1 cm in the z-axis. In addition, we investigated functional differences between superior and inferior parts of left vOT by analysing functional MRI data from 58 neurologically normal skilled readers performing 8 different visual processing tasks. We found that group activation in superior vOT was significantly more sensitive than inferior vOT to the type of task, with more superior vOT activation when participants were matching visual stimuli for their semantic or perceptual content than producing speech to the same stimuli. This functional difference along the z-axis was compared to existing boundaries between cytoarchitectonic areas around the OTs. In addition, using dynamic causal modelling, we show that connectivity from superior vOT to anterior vOT increased with semantic content during matching tasks but not during speaking tasks whereas connectivity from inferior vOT to anterior vOT was sensitive to semantic content for matching and speaking tasks. The finding of a functional dissociation between superior and inferior parts of vOT has implications for predicting deficits and response to rehabilitation for patients with partial damage to vOT following stroke or neurosurgery.