Semantic representation in the white matter pathway

Top-down generation of low-resolution representations improves visual perception and imagination

Perception or imagination requires top-down signals from high-level cortex to primary visual cortex (V1) to reconstruct or simulate the representations bottom-up stimulated by the seen images. Interestingly, top-down signals in V1 have lower spatial resolution than bottom-up representations. It is unclear why the brain uses low-resolution signals to reconstruct or simulate high-resolution representations. By modeling the top-down pathway of the visual system using the decoder of a variational auto-encoder (VAE), we reveal that low-resolution top-down signals can better reconstruct or simulate the information contained in the sparse activities of V1 simple cells, which facilitates perception and imagination. This advantage of low-resolution generation is related to facilitating high-level cortex to form geometry-respecting representations observed in experiments. Furthermore, we present two findings regarding this phenomenon in the context of AI-generated sketches, a style of drawings made of lines. First, we found that the quality of the generated sketches critically depends on the thickness of the lines in the sketches: thin-line sketches are harder to generate than thick-line sketches. Second, we propose a technique to generate high-quality thin-line sketches: instead of directly using original thin-line sketches, we use blurred sketches to train VAE or GAN (generative adversarial network), and then infer the thin-line sketches from the VAE- or GAN-generated blurred sketches. Collectively, our work suggests that low-resolution top-down generation is a strategy the brain uses to improve visual perception and imagination, which inspires new sketch-generation AI techniques.

Different computational relations in language are captured by distinct brain systems

A critical way for humans to acquire information is through language, yet whether and how language experience drives specific neural semantic representations is still poorly understood. We considered statistical properties captured by 3 different computational principles of language (simple co-occurrence, network-(graph)-topological relations, and neural-network-vector-embedding relations) and tested the extent to which they can explain the neural patterns of semantic representations, measured by 2 functional magnetic resonance imaging experiments that shared common semantic processes. Distinct graph-topological word relations, and not simple co-occurrence or neural-network-vector-embedding relations, had unique explanatory power for the neural patterns in the anterior temporal lobe (capturing graph-common-neighbors), inferior frontal gyrus, and posterior middle/inferior temporal gyrus (capturing graph-shortest-path). These results were relatively specific to language: they were not explained by sensory-motor similarities and the same computational relations of visual objects (based on visual image database) showed effects in the visual cortex in the picture naming experiment. That is, different topological properties within language and the same topological computations (common-neighbors) for language and visual inputs are captured by different brain regions. These findings reveal the specific neural semantic representations along graph-topological properties of language, highlighting the information type-specific and statistical property-specific manner of semantic representations in the human brain.

Mapping lesion, structural disconnection, and functional disconnection to symptoms in semantic aphasia

Patients with semantic aphasia have impaired control of semantic retrieval, often accompanied by executive dysfunction following left hemisphere stroke. Many but not all of these patients have damage to the left inferior frontal gyrus, important for semantic and cognitive control. Yet semantic and cognitive control networks are highly distributed, including posterior as well as anterior components. Accordingly, semantic aphasia might not only reflect local damage but also white matter structural and functional disconnection. Here, we characterise the lesions and predicted patterns of structural and functional disconnection in individuals with semantic aphasia and relate these effects to semantic and executive impairment. Impaired semantic cognition was associated with infarction in distributed left-hemisphere regions, including in the left anterior inferior frontal and posterior temporal cortex. Lesions were associated with executive dysfunction within a set of adjacent but distinct left frontoparietal clusters. Performance on executive tasks was also associated with interhemispheric structural disconnection across the corpus callosum. In contrast, poor semantic cognition was associated with small left-lateralized structurally disconnected clusters, including in the left posterior temporal cortex. Little insight was gained from functional disconnection symptom mapping. These results demonstrate that while left-lateralized semantic and executive control regions are often damaged together in stroke aphasia, these deficits are associated with distinct patterns of structural disconnection, consistent with the bilateral nature of executive control and the left-lateralized yet distributed semantic control network.

Anatomical correlates of category-selective visual regions have distinctive signatures of connectivity in neonates

The ventral visual stream is shaped during development by innate proto-organization within the visual system, such as the strong input from the fovea to the fusiform face area. In adults, category-selective regions have distinct signatures of connectivity to brain regions beyond the visual system, likely reflecting cross-modal and motoric associations. We tested if this long-range connectivity is part of the innate proto-organization, or if it develops with postnatal experience, by using diffusion-weighted imaging to characterize the connectivity of anatomical correlates of category-selective regions in neonates (N = 445), 1-9 month old infants (N = 11), and adults (N = 14). Using the HCP data we identified face- and place- selective regions and a third intermediate region with a distinct profile of selectivity. Using linear classifiers, these regions were found to have distinctive connectivity at birth, to other regions in the visual system and to those outside of it. The results support an extended proto-organization that includes long-range connectivity that shapes, and is shaped by, experience-dependent development.

Characterizing the semantic and form-based similarity spaces of the mental lexicon by means of the multi-arrangement method

Collecting human similarity judgments is instrumental to measuring and modeling neurocognitive representations (e.g., through representational similarity analysis) and has been made more efficient by the multi-arrangement task. While this task has been tested for collecting semantic similarity judgments, it is unclear whether it also lends itself to phonological and orthographic similarity judgments of words. We have extended the task to include these lexical modalities and compared the results between modalities and against computational models. We find that similarity judgments can be collected for all three modalities, although word forms were considered more difficult to sort and resulted in less consistent inter- and intra-rater agreement than semantics. For all three modalities we can construct stable group-level representational similarity matrices. However, these do not capture significant idiosyncratic similarity information unique to each participant. We discuss the potential underlying causes for differences between modalities and their effect on the application of the multi-arrangement task.

Edges in brain networks: Contributions to models of structure and function

Network models describe the brain as sets of nodes and edges that represent its distributed organization. So far, most discoveries in network neuroscience have prioritized insights that highlight distinct groupings and specialized functional contributions of network nodes. Importantly, these functional contributions are determined and expressed by the web of their interrelationships, formed by network edges. Here, we underscore the important contributions made by brain network edges for understanding distributed brain organization. Different types of edges represent different types of relationships, including connectivity and similarity among nodes. Adopting a specific definition of edges can fundamentally alter how we analyze and interpret a brain network. Furthermore, edges can associate into collectives and higher order arrangements, describe time series, and form edge communities that provide insights into brain network topology complementary to the traditional node-centric perspective. Focusing on the edges, and the higher order or dynamic information they can provide, discloses previously underappreciated aspects of structural and functional network organization.

Social and emotion dimensional organizations in the abstract semantic space: the neuropsychological evidence

An essential aspect of human cognition is supported by a rich reservoir of abstract concepts without tangible external referents (e.g., “honor”, “relationship”, “direction”). While decades of research showed that the neural organization of conceptual knowledge referring to concrete words respects domains of evolutionary salience and sensorimotor attributes, the organization principles of abstract word meanings are poorly understood. Here, we provide neuropsychological evidence for a domain (sociality) and attribute (emotion) structure in abstract word processing. Testing 34 brain-damaged patients on a word-semantic judgment task, we observed double dissociations between social and nonsocial words and a single dissociation of sparing of emotional (relative to non-emotional) words. The lesion profiles of patients with specific dissociations suggest potential neural correlates positively or negatively associated with each dimension. These results unravel a general domain-attribute architecture of word meanings and highlight the roles of the social domain and the emotional attribute in the non-object semantic space.

Dual coding of knowledge in the human brain

How does the human brain code knowledge about the world? While disciplines such as artificial intelligence represent world knowledge based on human language, neurocognitive models of knowledge have been dominated by sensory embodiment, in which knowledge is derived from sensory/motor experience and supported by high-level sensory/motor and association cortices. The neural correlates of an alternative disembodied symbolic system had previously been difficult to establish. A recent line of studies exploring knowledge about visual properties, such as color, in visually deprived individuals converge to provide positive, compelling evidence for non-sensory, language-derived, knowledge representation in dorsal anterior temporal lobe and extended language network, in addition to the sensory-derived representations, leading to a sketch of a dual-coding knowledge neural framework.

White matter network of oral word reading identified by network-based lesion-symptom mapping

Oral word reading is supported by a neural subnetwork that includes gray matter regions and white matter tracts connected by the regions. Traditional methods typically determine the reading-relevant focal gray matter regions or white matter tracts rather than the reading-relevant global subnetwork. The present study developed a network-based lesion-symptom mapping (NLSM) method to identify the reading-relevant global white matter subnetwork in 84 brain-damaged patients. The global subnetwork was selected among all possible subnetworks because its global efficiency exhibited the best explanatory power for patients' reading scores. This reading subnetwork was left lateralized and included 7 gray matter regions and 15 white matter tracts. Moreover, the reading subnetwork had additional explanatory power for the patients' reading performance after eliminating the effects of reading-related local regions and tracts. These findings refine the reading neuroanatomical architecture and indicate that the NLSM can be a better method for revealing behavior-specific subnetworks.

Cognitive impairment after focal brain lesions is better predicted by damage to structural than functional network hubs

Hubs are highly connected brain regions important for coordinating processing in brain networks. It is unclear, however, which measures of network "hubness" are most useful in identifying brain regions critical to human cognition. We tested how closely two measures of hubness-edge density and participation coefficient, derived from white and gray matter, respectively-were associated with general cognitive impairment after brain damage in two large cohorts of patients with focal brain lesions (N = 402 and 102, respectively) using cognitive tests spanning multiple cognitive domains. Lesions disrupting white matter regions with high edge density were associated with cognitive impairment, whereas lesions damaging gray matter regions with high participation coefficient had a weaker, less consistent association with cognitive outcomes. Similar results were observed with six other gray matter hubness measures. This suggests that damage to densely connected white matter regions is more cognitively impairing than similar damage to gray matter hubs, helping to explain interindividual differences in cognitive outcomes after brain damage.

Rapidly Decoding Image Categories From MEG Data Using a Multivariate Short-Time FC Pattern Analysis Approach

Recent advances in the development of multivariate analysis methods have led to the application of multivariate pattern analysis (MVPA) to investigate the interactions between brain regions using graph theory (functional connectivity, FC) and decode visual categories from functional magnetic resonance imaging (fMRI) data from a continuous multicategory paradigm. To estimate stable FC patterns from fMRI data, previous studies required long periods in the order of several minutes, in comparison to the human brain that categories visual stimuli within hundreds of milliseconds. Constructing short-time dynamic FC patterns in the order of milliseconds and decoding visual categories is a relatively novel concept. In this study, we developed a multivariate decoding algorithm based on FC patterns and applied it to magnetoencephalography (MEG) data. MEG data were recorded from participants presented with image stimuli in four categories (faces, scenes, animals and tools). MEG data from 17 participants demonstrate that short-time dynamic FC patterns yield brain activity patterns that can be used to decode visual categories with high accuracy. Our results show that FC patterns change over the time window, and FC patterns extracted in the time window of 0∼200 ms after the stimulus onset were most stable. Further, the categorizing accuracy peaked (the mean binary accuracy is above 78.6% at individual level) in the FC patterns estimated within the 0∼200 ms interval. These findings elucidate the underlying connectivity information during visual category processing on a relatively smaller time scale and demonstrate that the contribution of FC patterns to categorization fluctuates over time.

Multimodal mapping of the face connectome

Face processing supports our ability to recognize friend from foe, form tribes and understand the emotional implications of changes in facial musculature. This skill relies on a distributed network of brain regions, but how these regions interact is poorly understood. Here we integrate anatomical and functional connectivity measurements with behavioural assays to create a global model of the face connectome. We dissect key features, such as the network topology and fibre composition. We propose a neurocognitive model with three core streams; face processing along these streams occurs in a parallel and reciprocal manner. Although long-range fibre paths are important, the face network is dominated by short-range fibres. Finally, we provide evidence that the well-known right lateralization of face processing arises from imbalanced intra- and interhemispheric connections. In summary, the face network relies on dynamic communication across highly structured fibre tracts, enabling coherent face processing that underpins behaviour and cognition.

A Guide to Representational Similarity Analysis for Social Neuroscience

Representational similarity analysis (RSA) is a computational technique that uses pairwise comparisons of stimuli to reveal their representation in higher-order space. In the context of neuroimaging, mass-univariate analyses and other multivariate analyses can provide information on what and where information is represented but have limitations in their ability to address how information is represented. Social neuroscience is a field that can particularly benefit from incorporating RSA techniques to explore hypotheses regarding the representation of multidimensional data, how representations can predict behavior, how representations differ between groups and how multimodal data can be compared to inform theories. The goal of this paper is to provide a practical as well as theoretical guide to implementing RSA in social neuroscience studies.

Decoding disparity categories in 3-dimensional images from fMRI data using functional connectivity patterns

Humans use binocular disparity to extract depth information from two-dimensional retinal images in a process called stereopsis. Previous studies usually introduce the standard univariate analysis to describe the correlation between disparity level and brain activity within a given brain region based on functional magnetic resonance imaging (fMRI) data. Recently, multivariate pattern analysis has been developed to extract activity patterns across multiple voxels for deciphering categories of binocular disparity. However, the functional connectivity (FC) of patterns based on regions of interest or voxels and their mapping onto disparity category perception remain unknown. The present study extracted functional connectivity patterns for three disparity conditions (crossed disparity, uncrossed disparity, and zero disparity) at distinct spatial scales to decode the binocular disparity. Results of 27 subjects' fMRI data demonstrate that FC features are more discriminatory than traditional voxel activity features in binocular disparity classification. The average binary classification of the whole brain and visual areas are respectively 87% and 79% at single subject level, and thus above the chance level (50%). Our research highlights the importance of exploring functional connectivity patterns to achieve a novel understanding of 3D image processing.

Human white matter and knowledge representation

Understanding how knowledge is represented in the human brain is a fundamental challenge in neuroscience. To date, most of the work on this topic has focused on knowledge representation in cortical areas and debated whether knowledge is represented in a distributed or localized fashion. Fang and colleagues provide evidence that brain connections and the white matter supporting such connections might play a significant role. The work opens new avenues of investigation, breaking through disciplinary boundaries across network neuroscience, computational neuroscience, cognitive science, and classical lesion studies.