Decoding the meaning of unconsciously processed words using fMRI-based MVPA

Aberrant functional hubs and related networks attributed to cognitive impairment in patients with anti‑N‑methyl‑D‑aspartate receptor encephalitis

Anti-N-methyl-D-aspartate receptor (anti-NMDAR) encephalitis results in severe neuropsychiatric symptoms and persistent cognitive impairment; however, the underlying mechanism is still not fully understood. The present study utilized the degree centrality (DC), functional connectivity (FC) and multivariate pattern analysis (MVPA) to further explore neurofunctional symptoms in patients with anti-NMDAR encephalitis. A total of 29 patients with anti-NMDAR encephalitis and 26 healthy controls (HCs) were enrolled for neuropsychological assessment and resting-state functional MRI (rs-fMRI) scans. DC, FC and MVPA were examined to investigate cerebral functional activity and distinguish neuroimaging characteristics between the patient and HC groups based on the rs-fMRI data. Compared with the HCs, the patients exhibited cognitive deficits, anxiety and depression. In the DC analysis, the patients exhibited significantly decreased DC strength in the left rectus gyrus, left caudate nucleus (LCN) and bilateral superior medial frontal gyrus, as well as increased DC strength in the cerebellar anterior lobe, compared with the HCs. In the subsequent FC analysis, the LCN showed decreased FC strength in the bilateral middle frontal gyrus and right precuneus. Furthermore, correlation analysis indicated that disrupted cerebral functional activity was significantly correlated with the alerting effect and Hamilton Depression Scale score. Using DC maps and receiver operating characteristic curve analysis, the MVPA classifier exhibited an area under curve of 0.79, and the accuracy classification rate was 76.36%, with a sensitivity of 79.31% and a specificity of 78.18%. The present study revealed that the disrupted functional activity of hub and related networks in the cerebellum, including the default mode network and executive control network, contributed to deficits in cognition and emotion in patients with anti-NMDAR encephalitis. In conclusion, the present study provided imaging evidence and primary diagnostic markers for pathological and compensatory mechanisms of anti-NMDAR encephalitis, with the aim of improving the understanding of this disease.

Decoding N400m Evoked Component: A Tutorial on Multivariate Pattern Analysis for OP-MEG Data

Multivariate pattern analysis (MVPA) has played an extensive role in interpreting brain activity, which has been applied in studies with modalities such as functional Magnetic Resonance Imaging (fMRI), Magnetoencephalography (MEG) and Electroencephalography (EEG). The advent of wearable MEG systems based on optically pumped magnetometers (OPMs), i.e., OP-MEG, has broadened the application of bio-magnetism in the realm of neuroscience. Nonetheless, it also raises challenges in temporal decoding analysis due to the unique attributes of OP-MEG itself. The efficacy of decoding performance utilizing multimodal fusion, such as MEG-EEG, also remains to be elucidated. In this regard, we investigated the impact of several factors, such as processing methods, models and modalities, on the decoding outcomes of OP-MEG. Our findings indicate that the number of averaged trials, dimensionality reduction (DR) methods, and the number of cross-validation folds significantly affect the decoding performance of OP-MEG data. Additionally, decoding results vary across modalities and fusion strategy. In contrast, decoder type, resampling frequency, and sliding window length exert marginal effects. Furthermore, we introduced mutual information (MI) to investigate how information loss due to OP-MEG data processing affect decoding accuracy. Our study offers insights for linear decoding research using OP-MEG and expand its application in the fields of cognitive neuroscience.

Artificial intelligence based multimodal language decoding from brain activity: A review

Decoding brain activity is conducive to the breakthrough of brain-computer interface (BCI) technology. The development of artificial intelligence (AI) continually promotes the progress of brain language decoding technology. Existent research has mainly focused on a single modality and paid insufficient attention to AI methods. Therefore, our objective is to provide an overview of relevant decoding research from the perspective of different modalities and methodologies. The modalities involve text, speech, image, and video, whereas the core method is using AI-built decoders to translate brain signals induced by multimodal stimuli into text or vocal language. The semantic information of brain activity can be successfully decoded into a language at various levels, ranging from words through sentences to discourses. However, the decoding effect is affected by various factors, such as the decoding model, vector representation model, and brain regions. Challenges and future directions are also discussed. The advances in brain language decoding and BCI technology will potentially assist patients with clinical aphasia in regaining the ability to communicate.

EEG-based BCI Dataset of Semantic Concepts for Imagination and Perception Tasks

Electroencephalography (EEG) is a widely-used neuroimaging technique in Brain Computer Interfaces (BCIs) due to its non-invasive nature, accessibility and high temporal resolution. A range of input representations has been explored for BCIs. The same semantic meaning can be conveyed in different representations, such as visual (orthographic and pictorial) and auditory (spoken words). These stimuli representations can be either imagined or perceived by the BCI user. In particular, there is a scarcity of existing open source EEG datasets for imagined visual content, and to our knowledge there are no open source EEG datasets for semantics captured through multiple sensory modalities for both perceived and imagined content. Here we present an open source multisensory imagination and perception dataset, with twelve participants, acquired with a 124 EEG channel system. The aim is for the dataset to be open for purposes such as BCI related decoding and for better understanding the neural mechanisms behind perception, imagination and across the sensory modalities when the semantic category is held constant.

The depth of semantic processing modulates cross-language pattern similarity in Chinese-English bilinguals

Previous studies have investigated factors related to the degree of cross-language overlap in brain activations in bilinguals/multilinguals. However, it is still unclear whether and how the depth of semantic processing (a critical task-related factor) affects the neural pattern similarity between native and second languages. To address this question, 26 Chinese-English bilinguals were scanned with fMRI while performing a word naming task (i.e., a task with shallow semantic processing) and a semantic judgment task (i.e., a task with deep semantic processing) in both native and second languages. Based on three sets of representational similarity analysis (whole brain, ROI-based, and within-language vs. cross-language semantic representation), we found that select regions in the reading brain network showed higher cross-language pattern similarity and higher cross-language semantic representations during deep semantic processing than during shallow semantic processing. These results suggest that compared to shallow semantic processing, deep semantic processing may lead to greater language-independent processing (i.e., cross-language semantic representation) and cross-language pattern similarity, and provide direct quantitative neuroimaging evidence for cognitive models of bilingual lexical memory.

Emulating future neurotechnology using magic

Recent developments in neuroscience and artificial intelligence have allowed machines to decode mental processes with growing accuracy. Neuroethicists have speculated that perfecting these technologies may result in reactions ranging from an invasion of privacy to an increase in self-understanding. Yet, evaluating these predictions is difficult given that people are poor at forecasting their reactions. To address this, we developed a paradigm using elements of performance magic to emulate future neurotechnologies. We led 59 participants to believe that a (sham) neurotechnological machine could infer their preferences, detect their errors, and reveal their deep-seated attitudes. The machine gave participants randomly assigned positive or negative feedback about their brain's supposed attitudes towards charity. Around 80% of participants in both groups provided rationalisations for this feedback, which shifted their attitudes in the manipulated direction but did not influence donation behaviour. Our paradigm reveals how people may respond to prospective neurotechnologies, which may inform neuroethical frameworks.

Do sparse brain activity patterns underlie human cognition?

Accumulating multivariate pattern analysis (MVPA) results from fMRI studies suggest that information is represented in fingerprint patterns of activations and deactivations during perception, emotions, and cognition. We postulate that these fingerprint patterns might reflect neuronal-population level sparse code documented in two-photon calcium imaging studies in animal models, i.e., information represented in specific and reproducible ensembles of a few percent of active neurons amidst widespread inhibition in neural populations. We suggest that such representations constitute a fundamental organizational principle via interacting across multiple levels of brain hierarchy, thus giving rise to perception, emotions, and cognition.

Different Neural Activities for Actions and Language within the Shared Brain Regions: Evidence from Action and Verb Generation

The Inferior Frontal Gyrus, Premotor Cortex and Inferior Parietal Lobe were suggested to be involved in action and language processing. However, the patterns of neural activities in the shared neural regions are still unclear. This study designed an fMRI experiment to analyze the neural activity associations between action and verb generation for object nouns. Using noun reading as a control task, we compared the differences and similarities of brain regions activated by action and verb generation. The results showed that the action generation task activated more in the dorsal Premotor Cortex (PMC), parts of the midline of PMC and the left Inferior Parietal Lobe (IPL) than the verb generation task. Subregions in the bilateral Supplementary Motor Area (SMA) and the left Inferior Frontal Gyrus (IFG) were found to be shared by action and verb generation. Then, mean activation level analysis and multi-voxel pattern analysis (MVPA) were performed in the overlapping activation regions of two generation tasks in the shared regions. The bilateral SMA and the left IFG were found to have overlapping activations with action and verb generation. All the shared regions were found to have different activation patterns, and the mean activation levels of the shared regions in the bilateral of SMA were significantly higher in the action generation. Based on the function of these brain regions, it can be inferred that the shared regions in the bilateral SMA and the left IFG process action and language generation in a task-specific and intention-specific manner, respectively.

Neuroscientific Evidence for Processing Without Awareness

The extent to which we are affected by perceptual input of which we are unaware is widely debated. By measuring neural responses to sensory stimulation, neuroscientific data could complement behavioral results with valuable evidence. Here we review neuroscientific findings of processing of high-level information, as well as interactions with attention and memory. Although the results are mixed, we find initial support for processing object categories and words, possibly to the semantic level, as well as emotional expressions. Robust neural evidence for face individuation and integration of sentences or scenes is lacking. Attention affects the processing of stimuli that are not consciously perceived, and such stimuli may exogenously but not endogenously capture attention when relevant, and be maintained in memory over time. Sources of inconsistency in the literature include variability in control for awareness as well as individual differences, calling for future studies that adopt stricter measures of awareness and probe multiple processes within subjects.

Seeing food fast and slow: Arousing pictures and words have reverse priorities in accessing awareness

Previous studies have shown that stimuli triggering higher arousal (e.g., attractiveness) can access awareness faster than those triggering lower arousal, yet no studies have examined the effect of food calories. Since food brings us energy, satiety, and positive emotions, food stimuli bringing higher arousal would likely have higher priority in accessing awareness over those with lower arousal. We used high-calorie and low-calorie food stimuli as representatives for high and low arousal stimuli, respectively, based on the tight relationship between calorie and arousal. By adopting the breaking continuous flash suppression (b-CFS) paradigm, we had high-calorie and low-calorie food pictures or words presented dichoptically with dynamic Mondrian masks and measured the time for food stimuli to be released from suppression. Our results showed that high-calorie food pictures could access visual awareness faster than low-calorie food pictures (Experiment 1), and the reverse pattern was observed for food words (Experiment 2). We ruled out the possibility of the difference in low-level features (Experiment 3) and post-perceptual response bias (Experiment 4) as the causes for the observed b-CFS time differences. This study revealed the dissociation of the unconscious processing of pictures and words, which may rely on mechanisms related to attentional capture. High-arousing stimuli do not always enjoy priority in accessing visual awareness.

Different Neural Information Flows Affected by Activity Patterns for Action and Verb Generation

Shared brain regions have been found for processing action and language, including the left inferior frontal gyrus (IFG), the premotor cortex (PMC), and the inferior parietal lobule (IPL). However, in the context of action and language generation that shares the same action semantics, it is unclear whether the activity patterns within the overlapping brain regions would be the same. The changes in effective connectivity affected by these activity patterns are also unclear. In this fMRI study, participants were asked to perform hand action and verb generation tasks toward object pictures. We identified shared and specific brain regions for the two tasks in the left PMC, IFG, and IPL. The mean activation level and multi-voxel pattern analysis revealed that the activity patterns in the shared sub-regions were distinct for the two tasks. The dynamic causal modeling results demonstrated that the information flows for the two tasks were different across the shared sub-regions. These results provided the first neuroimaging evidence that the action and verb generation were task context driven in the shared regions, and the distinct patterns of neural information flow across the PMC-IFG-IPL neural network were affected by the polymodal processing in the shared regions.

The Challenge of Inferring Unconscious Mental Processes

Studies of unconscious mental processes often compare a performance measure (e.g., some assessment of perception or memory) with a measure of awareness (e.g., a verbal report or forced-choice response) of the critical cue or contingency taken either concurrently or separately. The resulting patterns of bivariate data across participants lend themselves to several analytic approaches for inferring the existence of unconscious mental processes, but it is rare for researchers to consider the underlying generative processes that might cause these patterns. We show that bivariate data are generally insufficient to discriminate single-process models, with a unitary latent process determining both performance and awareness, from dual-process models, comprising distinct latent processes for performance and awareness. Future research attempting to isolate and investigate unconscious processes will need to employ richer types of data and analyses.

Infective Messages: Definitions, Processes, and Implications for Trauma, Identities, Internal Conflicts, Psychotherapy, and Research

BACKGROUND

Throughout life, individuals are bombarded by countless emotion-generating messages. Certain of these messages, for example, some insults, admonitions, rejections, challenges, or insightful declarations, can be viewed as "infective." Infective messages shock, puncture, adhere, disturb, and generate discernable host responses that assimilate, accommodate, or repel the intruding messages. Messages originating in one's own mind can stimulate psychological equivalents of autoimmune responses. Sometimes, these messages produce enduring psychological changes. Infective messages may traumatize, organize, or therapize. The aims of this article are to consider how infective messages a) relate to their messengers (vectors), structural characteristics, and recipient (host) factors; b) might contribute to trauma, personal identity formation, psychological conflicts, and emotional self-regulatory and cognitive heuristics; c) might inform the conduct of psychotherapy; and d) suggest future research.

METHODS

Clinical observations were augmented with selective literature reviews. These sources contributed to perspectives regarding how certain messages might become infective; contribute to trauma, complex aspects of identity formation, and inner conflict; and inform the conduct of psychotherapy.

RESULTS

Infective messages are commonly characterized by short, cogent communications emanating from credible sources that fall on vulnerable recipients' receptive, dispositional feeling states. These infective stimuli trigger reactive and defensive emotions and associated behaviors reflecting responses to significant threats or benefits relative to the individual's deepest needs, motivations, and values. Defensive overreactions may occur in response to messages to which individuals have been previously sensitized. Infective message contents appear to assimilate into constantly evolving, dynamic autobiographical brain maps consisting of highly specific neuronal networks that range from the brainstem through limbic structures to multiple cortical areas. Autobiographical maps built from message-injected contents may transform, akin to the incorporation of retroviruses or prions, to encode personal identities as aspects of the self. Contrasting maps may exist semi-independently of one another, continuously evolving, interacting, combining, dividing, conflicting, and contending with one another for attention. Achieving attentional awareness, such maps help structure how individuals conceptualize and communicate about their encoded traumas, diverse identities, and the conflicts among them.

CONCLUSIONS AND IMPLICATIONS

During psychotherapy, aftereffects of traumatizing infective messages might be detoxified by deconstructing, desensitizing, and processing messages' precise words and emotional envelopes in relation to contexts in which they were delivered, and the individual's inner dispositions at moments of impact. Carefully crafted, timely interpretations can be therapeutically infective, generating enduring positive impacts. Future studies using an assortment of approaches can test these perspectives.

Brain decoding in multiple languages: Can cross-language brain decoding work?

The approach of cross-language brain decoding is to use models of brain decoding from one language to decode stimuli of another language. It has the potential to provide new insights into how our brain represents multiple languages. While it is possible to decode semantic information across different languages from neuroimaging data, the approach's overall success remains to be tested and depends on a number of factors such as cross-language similarity, age of acquisition/proficiency levels, and depth of language processing. We expect to see continued progress in this domain, from a traditional focus on words and concrete concepts toward the use of naturalistic experimental tasks involving higher-level language processing (e.g., discourse processing). The approach can also be applied to understand how cross-modal, cross-cultural, and other nonlinguistic factors may influence neural representations of different languages. This article provides an overview of cross-language brain decoding with suggestions for future research directions.

Neurocognitive mechanisms supporting the generalization of concepts across languages

The neurocognitive mechanisms that support the generalization of semantic representations across different languages remain to be determined. Current psycholinguistic models propose that semantic representations are likely to overlap across languages, although there is evidence also to the contrary. Neuroimaging studies observed that brain activity patterns associated with the meaning of words may be similar across languages. However, the factors that mediate cross-language generalization of semantic representations are not known. We here identify a key factor: the depth of processing. Human participants were asked to process visual words as they underwent functional MRI. We found that, during shallow processing, multivariate pattern classifiers could decode the word semantic category within each language in putative substrates of the semantic network, but there was no evidence of cross-language generalization in the shallow processing context. By contrast, when the depth of processing was higher, significant cross-language generalization was observed in several regions, including inferior parietal, ventromedial, lateral temporal, and inferior frontal cortex. These results are in keeping with distributed-only views of semantic processing and favour models based on multiple semantic hubs. The results also have ramifications for existing psycholinguistic models of word processing such as the BIA+, which by default assumes non-selective access to both native and second languages.

A Novel Framework for Unconscious Processing

Understanding the distinction between conscious and unconscious cognition remains a priority in psychology and neuroscience. A comprehensive neurocognitive account of conscious awareness will not be possible without a sound framework to isolate and understand unconscious information processing. Here, we provide a brain-based framework that allows the identification of unconscious processes, even with null effects on behaviour.