Advancing Clinical Practice: The Potential of Multimodal Technology in Modern Medicine

Multimodal technology is poised to revolutionize clinical practice by integrating artificial intelligence with traditional diagnostic modalities. This evolution traces its roots from Hippocrates' humoral theory to the use of sophisticated AI-driven platforms that synthesize data across multiple sensory channels. The interplay between historical medical practices and modern technology challenges conventional patient-clinician interactions and redefines diagnostic accuracy. Highlighting applications from neurology to radiology, the potential of multimodal technology emerges, suggesting a future where AI not only supports but enhances human sensory inputs in medical diagnostics. This shift invites the medical community to navigate the ethical, practical, and technological changes reshaping the landscape of clinical medicine.

MGA-Net: A novel mask-guided attention neural network for precision neonatal brain imaging

In this study, we introduce MGA-Net, a novel mask-guided attention neural network, which extends the U-net model for precision neonatal brain imaging. MGA-Net is designed to extract the brain from other structures and reconstruct high-quality brain images. The network employs a common encoder and two decoders: one for brain mask extraction and the other for brain region reconstruction. A key feature of MGA-Net is its high-level mask-guided attention module, which leverages features from the brain mask decoder to enhance image reconstruction. To enable the same encoder and decoder to process both MRI and ultrasound (US) images, MGA-Net integrates sinusoidal positional encoding. This encoding assigns distinct positional values to MRI and US images, allowing the model to effectively learn from both modalities. Consequently, features learned from a single modality can aid in learning a modality with less available data, such as US. We extensively validated the proposed MGA-Net on diverse and independent datasets from varied clinical settings and neonatal age groups. The metrics used for assessment included the DICE similarity coefficient, recall, and accuracy for image segmentation; structural similarity for image reconstruction; and root mean squared error for total brain volume estimation from 3D ultrasound images. Our results demonstrate that MGA-Net significantly outperforms traditional methods, offering superior performance in brain extraction and segmentation while achieving high precision in image reconstruction and volumetric analysis. Thus, MGA-Net represents a robust and effective preprocessing tool for MRI and 3D ultrasound images, marking a significant advance in neuroimaging that enhances both research and clinical diagnostics in the neonatal period and beyond.

A Review of EEG-based Localization of Epileptic Seizure Foci: Common Points with Multimodal Fusion of Brain Data

Unexpected seizures significantly decrease the quality of life in epileptic patients. Seizure attacks are caused by hyperexcitability and anatomical lesions of special regions of the brain, and cognitive impairments and memory deficits are their most common concomitant effects. In addition to seizure reduction treatments, medical rehabilitation involving brain-computer interfaces and neurofeedback can improve cognition and quality of life in patients with focal epilepsy in most cases, in particular when resective epilepsy surgery has been considered treatment in drug-resistant epilepsy. Source estimation and precise localization of epileptic foci can improve such rehabilitation and treatment. Electroencephalography (EEG) monitoring and multimodal noninvasive neuroimaging techniques such as ictal/interictal single-photon emission computerized tomography (SPECT) imaging and structural magnetic resonance imaging are common practices for the localization of epileptic foci and have been studied in several kinds of researches. In this article, we review the most recent research on EEG-based localization of seizure foci and discuss various methods, their advantages, limitations, and challenges with a focus on model-based data processing and machine learning algorithms. In addition, we survey whether combined analysis of EEG monitoring and neuroimaging techniques, which is known as multimodal brain data fusion, can potentially increase the precision of the seizure foci localization. To this end, we further review and summarize the key parameters and challenges of processing, fusion, and analysis of multiple source data, in the framework of model-based signal processing, for the development of a multimodal brain data analyzing system. This article has the potential to be used as a valuable resource for neuroscience researchers for the development of EEG-based rehabilitation systems based on multimodal data analysis related to focal epilepsy.

Two is better? combining EEG and fMRI for BCI and neurofeedback: a systematic review

Electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) are two commonly used non-invasive techniques for measuring brain activity in neuroscience and brain-computer interfaces (BCI).Objective. In this review, we focus on the use of EEG and fMRI in neurofeedback (NF) and discuss the challenges of combining the two modalities to improve understanding of brain activity and achieve more effective clinical outcomes. Advanced technologies have been developed to simultaneously record EEG and fMRI signals to provide a better understanding of the relationship between the two modalities. However, the complexity of brain processes and the heterogeneous nature of EEG and fMRI present challenges in extracting useful information from the combined data.Approach. We will survey existing EEG-fMRI combinations and recent studies that exploit EEG-fMRI in NF, highlighting the experimental and technical challenges.Main results. We made a classification of the different combination of EEG-fMRI for NF, we provide a review of multimodal analysis methods for EEG-fMRI features. We also survey the current state of research on EEG-fMRI in the different existing NF paradigms. Finally, we also identify some of the remaining challenges in this field.Significance. By exploring EEG-fMRI combinations in NF, we are advancing our knowledge of brain function and its applications in clinical settings. As such, this review serves as a valuable resource for researchers, clinicians, and engineers working in the field of neural engineering and rehabilitation, highlighting the promising future of EEG-fMRI-based NF.

Age-dependent neurovascular coupling characteristics in children and adults during general anesthesia

General anesthesia is an indispensable procedure in clinical practice. Anesthetic drugs induce dramatic changes in neuronal activity and cerebral metabolism. However, the age-related changes in neurophysiology and hemodynamics during general anesthesia remain unclear. Therefore, the objective of this study was to explore the neurovascular coupling between neurophysiology and hemodynamics in children and adults during general anesthesia. We analyzed frontal electroencephalogram (EEG) and functional near-infrared spectroscopy (fNIRS) signals recorded from children (6-12 years old, n = 17) and adults (18-60 years old, n = 25) during propofol-induced and sevoflurane-maintained general anesthesia. The neurovascular coupling was evaluated in wakefulness, maintenance of a surgical state of anesthesia (MOSSA), and recovery by using correlation, coherence and Granger-causality (GC) between the EEG indices [EEG power in different bands and permutation entropy (PE)], and hemodynamic responses the oxyhemoglobin (Δ[HbO]) and deoxy-hemoglobin (Δ[Hb]) from fNIRS in the frequency band in 0.01-0.1 Hz. The PE and Δ[Hb] performed well in distinguishing the anesthesia state (p > 0.001). The correlation between PE and Δ[Hb] was higher than those of other indices in the two age groups. The coherence significantly increased during MOSSA (p < 0.05) compared with wakefulness, and the coherences between theta, alpha and gamma, and hemodynamic activities of children are significantly stronger than that of adults' bands. The GC from neuronal activities to hemodynamic responses decreased during MOSSA, and can better distinguish anesthesia state in adults. Propofol-induced and sevoflurane-maintained combination exhibited age-dependent neuronal activities, hemodynamics, and neurovascular coupling, which suggests the need for separate rules for children's and adults' brain states monitoring during general anesthesia.

Haptic shared control improves neural efficiency during myoelectric prosthesis use

Clinical myoelectric prostheses lack the sensory feedback and sufficient dexterity required to complete activities of daily living efficiently and accurately. Providing haptic feedback of relevant environmental cues to the user or imbuing the prosthesis with autonomous control authority have been separately shown to improve prosthesis utility. Few studies, however, have investigated the effect of combining these two approaches in a shared control paradigm, and none have evaluated such an approach from the perspective of neural efficiency (the relationship between task performance and mental effort measured directly from the brain). In this work, we analyzed the neural efficiency of 30 non-amputee participants in a grasp-and-lift task of a brittle object. Here, a myoelectric prosthesis featuring vibrotactile feedback of grip force and autonomous control of grasping was compared with a standard myoelectric prosthesis with and without vibrotactile feedback. As a measure of mental effort, we captured the prefrontal cortex activity changes using functional near infrared spectroscopy during the experiment. It was expected that the prosthesis with haptic shared control would improve both task performance and mental effort compared to the standard prosthesis. Results showed that only the haptic shared control system enabled users to achieve high neural efficiency, and that vibrotactile feedback was important for grasping with the appropriate grip force. These results indicate that the haptic shared control system synergistically combines the benefits of haptic feedback and autonomous controllers, and is well-poised to inform such hybrid advancements in myoelectric prosthesis technology.

Attention Classification Based on Biosignals during Standard Cognitive Tasks for Occupational Domains

Occupational disorders considerably impact workers’ quality of life and organizational productivity, and even affect mortality worldwide. Such health issues are related to mental health and ergonomics risk factors. In particular, mental health may be affected by cognitive strain caused by unexpected interruptions and other attention compromising factors. Risk factors assessment associated with cognitive strain in office environments, namely related to attention states, still suffers from the lack of scientifically validated tools. In this work, we aim to develop a series of classification models that can classify attention during pre-defined cognitive tasks based on the acquisition of biosignals to create a ground truth of attention. Biosignals, such as electrocardiography, electroencephalography, and functional near-infrared spectroscopy, were acquired from eight subjects during standard cognitive tasks inducing attention. Individually tuned machine learning models trained with those biosignals allowed us to successfully detect attention on the individual level, with results in the range of 70–80%. The electroencephalogram and electrocardiogram were revealed to be the most appropriate sensors in this context, and the combination of multiple sensors demonstrated the importance of using multiple sources. These models prove to be relevant for the development of attention identification tools by providing ground truth to determine which human–computer interaction variables have strong associations with attention.

Neuroimaging PheWAS (Phenome-Wide Association Study): A Free Cloud-Computing Platform for Big-Data, Brain-Wide Imaging Association Studies

Large-scale, case-control genome-wide association studies (GWASs) have revealed genetic variations associated with diverse neurological and psychiatric disorders. Recent advances in neuroimaging and genomic databases of large healthy and diseased cohorts have empowered studies to characterize effects of the discovered genetic factors on brain structure and function, implicating neural pathways and genetic mechanisms in the underlying biology. However, the unprecedented scale and complexity of the imaging and genomic data requires new advanced biomedical data science tools to manage, process and analyze the data. In this work, we introduce Neuroimaging PheWAS (phenome-wide association study): a web-based system for searching over a wide variety of brain-wide imaging phenotypes to discover true system-level gene-brain relationships using a unified genotype-to-phenotype strategy. This design features a user-friendly graphical user interface (GUI) for anonymous data uploading, study definition and management, and interactive result visualizations as well as a cloud-based computational infrastructure and multiple state-of-art methods for statistical association analysis and multiple comparison correction. We demonstrated the potential of Neuroimaging PheWAS with a case study analyzing the influences of the apolipoprotein E (APOE) gene on various brain morphological properties across the brain in the Alzheimer's Disease Neuroimaging Initiative (ADNI) cohort. Benchmark tests were performed to evaluate the system's performance using data from UK Biobank. The Neuroimaging PheWAS system is freely available. It simplifies the execution of PheWAS on neuroimaging data and provides an opportunity for imaging genetics studies to elucidate routes at play for specific genetic variants on diseases in the context of detailed imaging phenotypic data.

Bimodal Data Fusion of Simultaneous Measurements of EEG and fNIRS during Lower Limb Movements

Electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) have temporal and spatial characteristics that may complement each other and, therefore, pose an intriguing approach for brain-computer interaction (BCI). In this work, the relationship between the hemodynamic response and brain oscillation activity was investigated using the concurrent recording of fNIRS and EEG during ankle joint movements. Twenty subjects participated in this experiment. The EEG was recorded using 20 electrodes and hemodynamic responses were recorded using 32 optodes positioned over the motor cortex areas. The event-related desynchronization (ERD) feature was extracted from the EEG signal in the alpha band (8-11) Hz, and the concentration change of the oxy-hemoglobin (oxyHb) was evaluated from the hemodynamics response. During the motor execution of the ankle joint movements, a decrease in the alpha (8-11) Hz amplitude (desynchronization) was found to be correlated with an increase of the oxyHb (r = -0.64061, p < 0.00001) observed on the Cz electrode and the average of the fNIRS channels (ch28, ch25, ch32, ch35) close to the foot area representation. Then, the correlated channels in both modalities were used for ankle joint movement classification. The result demonstrates that the integrated modality based on the correlated channels provides a substantial enhancement in ankle joint classification accuracy of 93.01 ± 5.60% (p < 0.01) compared with single modality. These results highlight the potential of the bimodal fNIR-EEG approach for the development of future BCI for lower limb rehabilitation.

A Review of Cerebral Hemodynamics During Sleep Using Near-Infrared Spectroscopy

Investigating cerebral hemodynamic changes during regular sleep cycles and sleep disorders is fundamental to understanding the nature of physiological and pathological mechanisms in the regulation of cerebral oxygenation during sleep. Although sleep neuroimaging methods have been studied and have been well-reviewed, they have limitations in terms of technique and experimental design. Neurologists are convinced that Near-infrared spectroscopy (NIRS) provides essential information and can be used to assist the assessment of cerebral hemodynamics, and numerous studies regarding sleep have been carried out based on NIRS. Thus, a brief historical overview of the sleep studies using NIRS will be helpful for the biomedical students, academicians, and engineers to better understand NIRS from various perspectives. In this study, the existing literature on sleep studies is reviewed, and an overview of the NIRS applications is synthesized and provided. The paper first reviews the application scenarios, as well as the patterns of fluctuation of NIRS, which includes the investigation in regular sleep and sleep-disordered breathing. Various factors such as different sleep stages, populations, and degrees of severity were considered. Furthermore, the experimental design and signal processing, as well as the regulation mechanisms involved in regular and pathological sleep, are investigated and discussed. The strengths and weaknesses of the existing NIRS applications are addressed and presented, which can direct further NIRS analysis and utilization.

Inter-subject pattern analysis for multivariate group analysis of functional neuroimaging. A unifying formalization

BACKGROUND AND OBJECTIVE

In medical imaging, population studies have to overcome the differences that exist between individuals to identify invariant image features that can be used for diagnosis purposes. In functional neuroimaging, an appealing solution to identify neural coding principles that hold at the population level is inter-subject pattern analysis, i.e. to learn a predictive model on data from multiple subjects and evaluate its generalization performance on new subjects. Although it has gained popularity in recent years, its widespread adoption is still hampered by the blatant lack of a formal definition in the literature. In this paper, we precisely introduce the first principled formalization of inter-subject pattern analysis targeted at multivariate group analysis of functional neuroimaging.

METHODS

We propose to frame inter-subject pattern analysis as a multi-source transductive transfer question, thus grounding it within several well defined machine learning settings and broadening the spectrum of usable algorithms. We describe two sets of inter-subject brain decoding experiments that use several open datasets: a magneto-encephalography study with 16 subjects and a functional magnetic resonance imaging paradigm with 100 subjects. We assess the relevance of our framework by performing model comparisons, where one brain decoding model exploits our formalization while others do not.

RESULTS

The first set of experiments demonstrates the superiority of a brain decoder that uses subject-by-subject standardization compared to state of the art models that use other standardization schemes, making the case for the interest of the transductive and the multi-source components of our formalization The second set of experiments quantitatively shows that, even after such transformation, it is more difficult for a brain decoder to generalize to new participants rather than to new data from participants available in the training phase, thus highlighting the transfer gap that needs to be overcome.

CONCLUSION

This paper describes the first formalization of inter-subject pattern analysis as a multi-source transductive transfer learning problem. We demonstrate the added value of this formalization using proof-of-concept experiments on several complementary functional neuroimaging datasets. This work should contribute to popularize inter-subject pattern analysis for functional neuroimaging population studies and pave the road for future methodological innovations.

A Survey on Deep Learning for Multimodal Data Fusion

With the wide deployments of heterogeneous networks, huge amounts of data with characteristics of high volume, high variety, high velocity, and high veracity are generated. These data, referred to multimodal big data, contain abundant intermodality and cross-modality information and pose vast challenges on traditional data fusion methods. In this review, we present some pioneering deep learning models to fuse these multimodal big data. With the increasing exploration of the multimodal big data, there are still some challenges to be addressed. Thus, this review presents a survey on deep learning for multimodal data fusion to provide readers, regardless of their original community, with the fundamentals of multimodal deep learning fusion method and to motivate new multimodal data fusion techniques of deep learning. Specifically, representative architectures that are widely used are summarized as fundamental to the understanding of multimodal deep learning. Then the current pioneering multimodal data fusion deep learning models are summarized. Finally, some challenges and future topics of multimodal data fusion deep learning models are described.

Improved physiological noise regression in fNIRS: A multimodal extension of the General Linear Model using temporally embedded Canonical Correlation Analysis

For the robust estimation of evoked brain activity from functional Near-Infrared Spectroscopy (fNIRS) signals, it is crucial to reduce nuisance signals from systemic physiology and motion. The current best practice incorporates short-separation (SS) fNIRS measurements as regressors in a General Linear Model (GLM). However, several challenging signal characteristics such as non-instantaneous and non-constant coupling are not yet addressed by this approach and additional auxiliary signals are not optimally exploited. We have recently introduced a new methodological framework for the unsupervised multivariate analysis of fNIRS signals using Blind Source Separation (BSS) methods. Building onto the framework, in this manuscript we show how to incorporate the advantages of regularized temporally embedded Canonical Correlation Analysis (tCCA) into the supervised GLM. This approach allows flexible integration of any number of auxiliary modalities and signals. We provide guidance for the selection of optimal parameters and auxiliary signals for the proposed GLM extension. Its performance in the recovery of evoked HRFs is then evaluated using both simulated ground truth data and real experimental data and compared with the GLM with short-separation regression. Our results show that the GLM with tCCA significantly improves upon the current best practice, yielding significantly better results across all applied metrics: Correlation (HbO max. +45%), Root Mean Squared Error (HbO max. −55%), F-Score (HbO up to 3.25-fold) and p-value as well as power spectral density of the noise floor. The proposed method can be incorporated into the GLM in an easily applicable way that flexibly combines any available auxiliary signals into optimal nuisance regressors. This work has potential significance both for conventional neuroscientific fNIRS experiments as well as for emerging applications of fNIRS in everyday environments, medicine and BCI, where high Contrast to Noise Ratio is of importance for single trial analysis.

A Newcomer's Guide to Functional Near Infrared Spectroscopy Experiments

This review presents a practical primer for functional near-infrared spectroscopy (fNIRS) with respect to technology, experimentation, and analysis software. Its purpose is to jump-start interested practitioners considering utilizing a non-invasive, versatile, nevertheless challenging window into the brain using optical methods. We briefly recapitulate relevant anatomical and optical foundations and give a short historical overview. We describe competing types of illumination (trans-illumination, reflectance, and differential reflectance) and data collection methods (continuous wave, time domain and frequency domain). Basic components (light sources, detection, and recording components) of fNIRS systems are presented. Advantages and limitations of fNIRS techniques are offered, followed by a list of very practical recommendations for its use. A variety of experimental and clinical studies with fNIRS are sampled, shedding light on many brain-related ailments. Finally, we describe and discuss a number of freely available analysis and presentation packages suited for data analysis. In conclusion, we recommend fNIRS due to its ever-growing body of clinical applications, state-of-the-art neuroimaging technique and manageable hardware requirements. It can be safely concluded that fNIRS adds a new arrow to the quiver of neuro-medical examinations due to both its great versatility and limited costs.

Simultaneous EEG-NIRS Measurement of the Inferior Parietal Lobule During a Reaching Task With Delayed Visual Feedback

We investigated whether the inferior parietal lobule (IPL) responds in real-time to multisensory inconsistency during movement. The IPL is thought to be involved in both the detection of inconsistencies in multisensory information obtained during movement and that obtained during self-other discrimination. However, because of the limited temporal resolution of conventional neuroimaging techniques, it is difficult to distinguish IPL activity during movement from that during self-other discrimination. We simultaneously conducted electroencephalography (EEG) and near-infrared spectroscopy (NIRS) with the goal of examining IPL activity with a high spatiotemporal resolution during single reaching movements. Under a visual feedback-delay condition, gamma event-related synchronization (γ-ERS), i.e., an increase in gamma (31–47 Hz) EEG power occurred during reaching movements. This γ-ERS is considered to reflect processing of information about prediction errors. To integrate this temporal information with spatial information from the NIRS signals, we developed a new analysis technique that enabled estimation of the regions that show a hemodynamic response characterized by EEG fluctuation present in the visual feedback-delay condition. As a result, IPL activity was explained by γ-ERS specific to visual feedback delay during movements. Thus, we succeeded in demonstrating real-time activation of the IPL in response to multisensory inconsistency. However, we did not find any correlation between either IPL activity or γ-ERS with the sense of agency. Therefore, our results suggest that while the IPL is influenced by prediction error signals, it does not engage in direct processing underlying the conscious experience of making a movement, which is the foundation of self-other discrimination.

Canonical maximization of coherence: A novel tool for investigation of neuronal interactions between two datasets

Synchronization between oscillatory signals is considered to be one of the main mechanisms through which neuronal populations interact with each other. It is conventionally studied with mass-bivariate measures utilizing either sensor-to-sensor or voxel-to-voxel signals. However, none of these approaches aims at maximizing synchronization, especially when two multichannel datasets are present. Examples include cortico-muscular coherence (CMC), cortico-subcortical interactions or hyperscanning (where electroencephalographic EEG/magnetoencephalographic MEG activity is recorded simultaneously from two or more subjects). For all of these cases, a method which could find two spatial projections maximizing the strength of synchronization would be desirable. Here we present such method for the maximization of coherence between two sets of EEG/MEG/EMG (electromyographic)/LFP (local field potential) recordings. We refer to it as canonical Coherence (caCOH). caCOH maximizes the absolute value of the coherence between the two multivariate spaces in the frequency domain. This allows very fast optimization for many frequency bins. Apart from presenting details of the caCOH algorithm, we test its efficacy with simulations using realistic head modelling and focus on the application of caCOH to the detection of cortico-muscular coherence. For this, we used diverse multichannel EEG and EMG recordings and demonstrate the ability of caCOH to extract complex patterns of CMC distributed across spatial and frequency domains. Finally, we indicate other scenarios where caCOH can be used for the extraction of neuronal interactions.

Disrupted core-periphery structure of multimodal brain networks in Alzheimer's disease

In Alzheimer's disease (AD), the progressive atrophy leads to aberrant network reconfigurations both at structural and functional levels. In such network reorganization, the core and peripheral nodes appear to be crucial for the prediction of clinical outcome because of their ability to influence large-scale functional integration. However, the role of the different types of brain connectivity in such prediction still remains unclear. Using a multiplex network approach we integrated information from DWI, fMRI, and MEG brain connectivity to extract an enriched description of the core-periphery structure in a group of AD patients and age-matched controls. Globally, the regional coreness-that is, the probability of a region to be in the multiplex core-significantly decreased in AD patients as result of a random disconnection process initiated by the neurodegeneration. Locally, the most impacted areas were in the core of the network-including temporal, parietal, and occipital areas-while we reported compensatory increments for the peripheral regions in the sensorimotor system. Furthermore, these network changes significantly predicted the cognitive and memory impairment of patients. Taken together these results indicate that a more accurate description of neurodegenerative diseases can be obtained from the multimodal integration of neuroimaging-derived network data.

Multimodal Neuroimaging: Basic Concepts and Classification of Neuropsychiatric Diseases

Neuroimaging techniques are widely used in neuroscience to visualize neural activity, to improve our understanding of brain mechanisms, and to identify biomarkers-especially for psychiatric diseases; however, each neuroimaging technique has several limitations. These limitations led to the development of multimodal neuroimaging (MN), which combines data obtained from multiple neuroimaging techniques, such as electroencephalography, functional magnetic resonance imaging, and yields more detailed information about brain dynamics. There are several types of MN, including visual inspection, data integration, and data fusion. This literature review aimed to provide a brief summary and basic information about MN techniques (data fusion approaches in particular) and classification approaches. Data fusion approaches are generally categorized as asymmetric and symmetric. The present review focused exclusively on studies based on symmetric data fusion methods (data-driven methods), such as independent component analysis and principal component analysis. Machine learning techniques have recently been introduced for use in identifying diseases and biomarkers of disease. The machine learning technique most widely used by neuroscientists is classification-especially support vector machine classification. Several studies differentiated patients with psychiatric diseases and healthy controls with using combined datasets. The common conclusion among these studies is that the prediction of diseases increases when combining data via MN techniques; however, there remain a few challenges associated with MN, such as sample size. Perhaps in the future N-way fusion can be used to combine multiple neuroimaging techniques or nonimaging predictors (eg, cognitive ability) to overcome the limitations of MN.

Longitudinal brain functional and structural connectivity changes after hemispherotomy in two pediatric patients with drug-resistant epilepsy

The main focus of the present study was to explore the longitudinal changes in the brain executive control system and default mode network after hemispherotomy. Resting-state functional magnetic resonance imaging and diffusion tensor imaging were collected in two children with drug-resistnt epilepsy underwent hemispherotomy. Two patients with different curative effects showed different trajectories of brain connectivity after surgery. The failed hemispherotomy might be due to the fact that the synchrony of epileptic neurons in both hemispheres is preserved by residual neural pathways. Loss of interhemispheric correlations with increased intrahemispheric correlations can be considered as neural marker for evaluating the success of hemispherotomy.

Symbolic time series analysis of fNIRS signals in brain development assessment

OBJECTIVE

Assessing an infant's brain development remains a challenge for neuroscientists and pediatricians despite great technological advances. As a non-invasive neuroimaging tool, functional near-infrared spectroscopy (fNIRS) has great advantages in monitoring an infant's brain activity. To explore the dynamic features of hemodynamic changes in infants, in-pattern exponent (IPE), anti-pattern exponent (APE), as well as permutation cross-mutual information (PCMI) based on symbolic dynamics are proposed to measure the phase differences and coupling strength in oxyhemoglobin (HbO) and deoxyhemoglobin (Hb) signals from fNIRS.

APPROACH

First, simulated sinusoidal oscillation signals and four coupled nonlinear systems were employed for performance assessments. Hilbert transform based measurements of hemoglobin phase oxygenation and deoxygenation (hPod) and phase-locking index of hPod (hPodL) were calculated for comparison. Then, the IPE, APE and PCMI indices from resting state fNIRS data of preterm, term infants and adults were calculated to estimate the phase difference and coupling of HbO and Hb. All indices' performance was assessed by the degree of monotonicity (DoM). The box plots and coefficients of variation (CV) were employed to assess the measurements and robustness in the results.

MAIN RESULTS

In the simulation analysis, IPE and APE can distinguish the phase difference of two sinusoidal oscillation signals. Both hPodL and PCMI can track the strength of two coupled nonlinear systems. Compared to hPodL, the PCMI had higher DoM indices in measuring the coupling of two nonlinear systems. In the fNIRS data analysis, similar to hPod, the IPE and APE can distinguish preterm, term infants, and adults in 0.01-0.05 Hz, 0.05-0.1 Hz, and 0.01-0.1 Hz frequency bands, respectively. PCMI more effectively distinguished the term and preterm infants than hPodL in the 0.05-0.1 Hz frequency band. As symbolic time series measures, the IPE and APE were able to detect the brain developmental changes in subjects of different ages. PCMI can assess the resting-state HbO and Hb coupling changes across different developmental ages, which may reflect the metabolic and neurovascular development.

SIGNIFICANCE

The symbolic-based methodologies are promising measures for fNIRS in estimating the brain development, especially in assessing newborns' brain developmental status.

Neuroscience Information Toolbox: An Open Source Toolbox for EEG-fMRI Multimodal Fusion Analysis

Recently, scalp electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) multimodal fusion has been pursued in an effort to study human brain function and dysfunction to obtain more comprehensive information on brain activity in which the spatial and temporal resolutions are both satisfactory. However, a more flexible and easy-to-use toolbox for EEG–fMRI multimodal fusion is still lacking. In this study, we therefore developed a freely available and open-source MATLAB graphical user interface toolbox, known as the Neuroscience Information Toolbox (NIT), for EEG–fMRI multimodal fusion analysis. The NIT consists of three modules: (1) the fMRI module, which has batch fMRI preprocessing, nuisance signal removal, bandpass filtering, and calculation of resting-state measures; (2) the EEG module, which includes artifact removal, extracting EEG features (event onset, power, and amplitude), and marking interesting events; and (3) the fusion module, in which fMRI-informed EEG analysis and EEG-informed fMRI analysis are included. The NIT was designed to provide a convenient and easy-to-use toolbox for researchers, especially for novice users. The NIT can be downloaded for free at http://www.neuro.uestc.edu.cn/NIT.html, and detailed information, including the introduction of NIT, user’s manual and example data sets, can also be observed on this website. We hope that the NIT is a promising toolbox for exploring brain information in various EEG and fMRI studies.

Decoding human mental states by whole-head EEG+fNIRS during category fluency task performance

OBJECTIVE

Concurrent scalp electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS), which we refer to as EEG+fNIRS, promises greater accuracy than the individual modalities while remaining nearly as convenient as EEG. We sought to quantify the hybrid system's ability to decode mental states and compare it with its unimodal components.

APPROACH

We recorded from healthy volunteers taking the category fluency test and applied machine learning techniques to the data.

MAIN RESULTS

EEG+fNIRS's decoding accuracy was greater than that of its subsystems, partly due to the new type of neurovascular features made available by hybrid data.

SIGNIFICANCE

Availability of an accurate and practical decoding method has potential implications for medical diagnosis, brain-computer interface design, and neuroergonomics.

Enhanced Brain Network Activity in Complex Movement Rhythms: A Simultaneous Functional Magnetic Resonance Imaging and Electroencephalography Study

Generating movement rhythms is known to involve a network of distributed brain regions associated with motor planning, control, execution, and perception of timing for the repertoire of motor actions. What brain areas are bound in the network and how the network activity is modulated by rhythmic complexity have not been completely explored. To contribute to answering these questions, we designed a study in which nine healthy participants performed simple to complex rhythmic finger movement tasks while undergoing simultaneous functional magnetic resonance imaging and electroencephalography (fMRI-EEG) recordings of their brain activity during the tasks and rest. From fMRI blood oxygenation-level-dependent (BOLD) measurements, we found that the complexity of rhythms was associated with brain activations in the primary motor cortex (PMC), supplementary motor area (SMA), and cerebellum (Cb), and with network interactions from these cortical regions to the cerebellum. The spectral analysis of single-trial EEG source waveforms at the cortical regions further showed that there were bidirectional interactions between PMC and SMA, and the complexity of rhythms was associated with power spectra and Granger causality spectra in the beta (13-30 Hz) frequency band, not in the alpha (8-12 Hz) and gamma (30-58 Hz) bands. These results provide us new insights into the mechanisms for movement rhythm complexity.

Schizophrenia Shows Disrupted Links between Brain Volume and Dynamic Functional Connectivity

Studies featuring multimodal neuroimaging data fusion for understanding brain function and structure, or disease characterization, leverage the partial information available in each of the modalities to reveal data variations not exhibited through the independent analyses. Similar to other complex syndromes, the characteristic brain abnormalities in schizophrenia may be better understood with the help of the additional information conveyed by leveraging an advanced modeling method involving multiple modalities. In this study, we propose a novel framework to fuse feature spaces corresponding to functional magnetic resonance imaging (functional) and gray matter (structural) data from 151 schizophrenia patients and 163 healthy controls. In particular, the features for the functional and structural modalities include dynamic (i.e., time-varying) functional network connectivity (dFNC) maps and the intensities of the gray matter (GM) maps, respectively. The dFNC maps are estimated from group independent component analysis (ICA) network time-courses by first computing windowed functional correlations using a sliding window approach, and then estimating subject specific states from this windowed data using temporal ICA followed by spatio-temporal regression. For each subject, the functional data features are horizontally concatenated with the corresponding GM features to form a combined feature space that is subsequently decomposed through a symmetric multimodal fusion approach involving a combination of multiset canonical correlation analysis (mCCA) and joint ICA (jICA). Our novel combined analyses successfully linked changes in the two modalities and revealed significantly disrupted links between GM volumes and time-varying functional connectivity in schizophrenia. Consistent with prior research, we found significant group differences in GM comprising regions in the superior parietal lobule, precuneus, postcentral gyrus, medial/superior frontal gyrus, superior/middle temporal gyrus, insula and fusiform gyrus, and several significant aberrations in the inter-regional functional connectivity strength as well. Importantly, structural and dFNC measures have independently shown changes associated with schizophrenia, and in this work we begin the process of evaluating the links between the two, which could shed light on the illness beyond what we can learn from a single imaging modality. In future work, we plan to evaluate replication of the inferred structure-function relationships in independent partitions of larger multi-modal schizophrenia datasets.

Multimodal neural correlates of cognitive control in the Human Connectome Project

Cognitive control is a construct that refers to the set of functions that enable decision-making and task performance through the representation of task states, goals, and rules. The neural correlates of cognitive control have been studied in humans using a wide variety of neuroimaging modalities, including structural MRI, resting-state fMRI, and task-based fMRI. The results from each of these modalities independently have implicated the involvement of a number of brain regions in cognitive control, including dorsal prefrontal cortex, and frontal parietal and cingulo-opercular brain networks. However, it is not clear how the results from a single modality relate to results in other modalities. Recent developments in multimodal image analysis methods provide an avenue for answering such questions and could yield more integrated models of the neural correlates of cognitive control. In this study, we used multiset canonical correlation analysis with joint independent component analysis (mCCA + jICA) to identify multimodal patterns of variation related to cognitive control. We used two independent cohorts of participants from the Human Connectome Project, each of which had data from four imaging modalities. We replicated the findings from the first cohort in the second cohort using both independent and predictive analyses. The independent analyses identified a component in each cohort that was highly similar to the other and significantly correlated with cognitive control performance. The replication by prediction analyses identified two independent components that were significantly correlated with cognitive control performance in the first cohort and significantly predictive of performance in the second cohort. These components identified positive relationships across the modalities in neural regions related to both dynamic and stable aspects of task control, including regions in both the frontal-parietal and cingulo-opercular networks, as well as regions hypothesized to be modulated by cognitive control signaling, such as visual cortex. Taken together, these results illustrate the potential utility of multi-modal analyses in identifying the neural correlates of cognitive control across different indicators of brain structure and function.

Measuring Mental Workload with EEG+fNIRS

We studied the capability of a Hybrid functional neuroimaging technique to quantify human mental workload (MWL). We have used electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) as imaging modalities with 17 healthy subjects performing the letter n-back task, a standard experimental paradigm related to working memory (WM). The level of MWL was parametrically changed by variation of n from 0 to 3. Nineteen EEG channels were covering the whole-head and 19 fNIRS channels were located on the forehead to cover the most dominant brain region involved in WM. Grand block averaging of recorded signals revealed specific behaviors of oxygenated-hemoglobin level during changes in the level of MWL. A machine learning approach has been utilized for detection of the level of MWL. We extracted different features from EEG, fNIRS, and EEG+fNIRS signals as the biomarkers of MWL and fed them to a linear support vector machine (SVM) as train and test sets. These features were selected based on their sensitivity to the changes in the level of MWL according to the literature. We introduced a new category of features within fNIRS and EEG+fNIRS systems. In addition, the performance level of each feature category was systematically assessed. We also assessed the effect of number of features and window size in classification performance. SVM classifier used in order to discriminate between different combinations of cognitive states from binary- and multi-class states. In addition to the cross-validated performance level of the classifier other metrics such as sensitivity, specificity, and predictive values were calculated for a comprehensive assessment of the classification system. The Hybrid (EEG+fNIRS) system had an accuracy that was significantly higher than that of either EEG or fNIRS. Our results suggest that EEG+fNIRS features combined with a classifier are capable of robustly discriminating among various levels of MWL. Results suggest that EEG+fNIRS should be preferred to only EEG or fNIRS, in developing passive BCIs and other applications which need to monitor users' MWL.

How to Build a Hybrid Neurofeedback Platform Combining EEG and fMRI

Multimodal neurofeedback estimates brain activity using information acquired with more than one neurosignal measurement technology. In this paper we describe how to set up and use a hybrid platform based on simultaneous electroencephalography (EEG) and functional magnetic resonance imaging (fMRI), then we illustrate how to use it for conducting bimodal neurofeedback experiments. The paper is intended for those willing to build a multimodal neurofeedback system, to guide them through the different steps of the design, setup, and experimental applications, and help them choose a suitable hardware and software configuration. Furthermore, it reports practical information from bimodal neurofeedback experiments conducted in our lab. The platform presented here has a modular parallel processing architecture that promotes real-time signal processing performance and simple future addition and/or replacement of processing modules. Various unimodal and bimodal neurofeedback experiments conducted in our lab showed high performance and accuracy. Currently, the platform is able to provide neurofeedback based on electroencephalography and functional magnetic resonance imaging, but the architecture and the working principles described here are valid for any other combination of two or more real-time brain activity measurement technologies.

A Hitchhiker's Guide to Functional Magnetic Resonance Imaging

Functional Magnetic Resonance Imaging (fMRI) studies have become increasingly popular both with clinicians and researchers as they are capable of providing unique insights into brain functions. However, multiple technical considerations (ranging from specifics of paradigm design to imaging artifacts, complex protocol definition, and multitude of processing and methods of analysis, as well as intrinsic methodological limitations) must be considered and addressed in order to optimize fMRI analysis and to arrive at the most accurate and grounded interpretation of the data. In practice, the researcher/clinician must choose, from many available options, the most suitable software tool for each stage of the fMRI analysis pipeline. Herein we provide a straightforward guide designed to address, for each of the major stages, the techniques, and tools involved in the process. We have developed this guide both to help those new to the technique to overcome the most critical difficulties in its use, as well as to serve as a resource for the neuroimaging community.

Functional Near-Infrared Spectroscopy (fNIRS) for Assessing Cerebral Cortex Function During Human Behavior in Natural/Social Situations: A Concise Review

Upon adequate stimulation, real-time maps of cortical hemodynamic responses can be obtained by functional near-infrared spectroscopy (fNIRS), which noninvasively measures changes in oxygenated and deoxygenated hemoglobin after positioning multiple sources and detectors over the human scalp. This review is aimed at giving a concise and simple overview of the basic principles of fNIRS including features, strengths, advantages, limitations, and utility for evaluating human behavior. The transportable/wireless commercially available fNIRS systems have a time resolution of 1 to 10 Hz, a depth sensitivity of about 1.5 cm, and a spatial resolution up to 1 cm. fNIRS has been found suitable for many applications on human beings, either adults or infants/children, in the field of social sciences, neuroimaging basic research, and medicine. Some examples of present and future prospects of fNIRS for assessing cerebral cortex function during human behavior in different situations (in natural and social situations) will be provided. Moreover, the most recent fNIRS studies for investigating interpersonal interactions by adopting the hyperscanning approach, which consists of the measurement of brain activity simultaneously on two or more people, will be reported.

Hemodynamic correlates of spontaneous neural activity measured by human whole-head resting state EEG+fNIRS

The brains of awake, resting human subjects display spontaneously occurring neural activity patterns whose magnitude is typically many times greater than those triggered by cognitive or perceptual performance. Evoked and resting state activations affect local cerebral hemodynamic properties through processes collectively referred to as neurovascular coupling. Its investigation calls for an ability to track both the neural and vascular aspects of brain function. We used scalp electroencephalography (EEG), which provided a measure of the electrical potentials generated by cortical postsynaptic currents. Simultaneously we utilized functional near-infrared spectroscopy (NIRS) to continuously monitor hemoglobin concentration changes in superficial cortical layers. The multi-modal signal from 18 healthy adult subjects allowed us to investigate the association of neural activity in a range of frequencies over the whole-head to local changes in hemoglobin concentrations. Our results verified the delayed alpha (8-16Hz) modulation of hemodynamics in posterior areas known from the literature. They also indicated strong beta (16-32Hz) modulation of hemodynamics. Analysis revealed, however, that beta modulation was likely generated by the alpha-beta coupling in EEG. Signals from the inferior electrode sites were dominated by scalp muscle related activity. Our study aimed to characterize the phenomena related to neurovascular coupling observable by practical, cost-effective, and non-invasive multi-modal techniques.

A Wearable EEG-HEG-HRV Multimodal System With Simultaneous Monitoring of tES for Mental Health Management

A multimodal mental management system in the shape of the wearable headband and earplugs is proposed to monitor electroencephalography (EEG), hemoencephalography (HEG) and heart rate variability (HRV) for accurate mental health monitoring. It enables simultaneous transcranial electrical stimulation (tES) together with real-time monitoring. The total weight of the proposed system is less than 200 g. The multi-loop low-noise amplifier (MLLNA) achieves over 130 dB CMRR for EEG sensing and the capacitive correlated-double sampling transimpedance amplifier (CCTIA) has low-noise characteristics for HEG and HRV sensing. Measured three-physiology domains such as neural, vascular and autonomic domain signals are combined with canonical correlation analysis (CCA) and temporal kernel canonical correlation analysis (tkCCA) algorithm to find the neural-vascular-autonomic coupling. It supports highly accurate classification with the 19% maximum improvement with multimodal monitoring. For the multi-channel stimulation functionality, after-effects maximization monitoring and sympathetic nerve disorder monitoring, the stimulator is designed as reconfigurable. The 3.37 × 2.25 mm(2) chip has 2-channel EEG sensor front-end, 2-channel NIRS sensor front-end, NIRS current driver to drive dual-wavelength VCSEL and 6-b DAC current source for tES mode. It dissipates 24 mW with 2 mA stimulation current and 5 mA NIRS driver current.

Simultaneous measurement of electroencephalography and near-infrared spectroscopy during voluntary motor preparation

We investigated the relationship between electrophysiological activity and haemodynamic response during motor preparation by simultaneous recording of electroencephalography (EEG) and near-infrared spectroscopy (NIRS). It is still unknown how exactly EEG signals correlate with the haemodynamic response, although the activation in the premotor area during motor preparation has been captured by EEG and haemodynamic approaches separately. We conducted EEG-NIRS simultaneous recordings over the sensorimotor area with a self-paced button press task. Participants were instructed to press a button at their own pace after a cue was shown. The result showed that the readiness potential (RP), a negative slow potential shift occurring during motor preparation, on C3 in the extended 10–20 system occurred about 1000 ms before the movement onset. An increase in concentration of oxyhaemoglobin (oxyHb) in the premotor cortex during motor preparation was also confirmed by NIRS, which resulted in a significant correlation between the amplitude of the RP and the change in oxyHb concentration (Pearson’s correlation r² = 0.235, p = 0.03). We show that EEG-NIRS simultaneous recording can demonstrate the correlation between the RP and haemodynamic response in the premotor cortex contralateral to the performing hand.

Toward a Wireless Open Source Instrument: Functional Near-infrared Spectroscopy in Mobile Neuroergonomics and BCI Applications

Brain-Computer Interfaces (BCIs) and neuroergonomics research have high requirements regarding robustness and mobility. Additionally, fast applicability and customization are desired. Functional Near-Infrared Spectroscopy (fNIRS) is an increasingly established technology with a potential to satisfy these conditions. EEG acquisition technology, currently one of the main modalities used for mobile brain activity assessment, is widely spread and open for access and thus easily customizable. fNIRS technology on the other hand has either to be bought as a predefined commercial solution or developed from scratch using published literature. To help reducing time and effort of future custom designs for research purposes, we present our approach toward an open source multichannel stand-alone fNIRS instrument for mobile NIRS-based neuroimaging, neuroergonomics and BCI/BMI applications. The instrument is low-cost, miniaturized, wireless and modular and openly documented on www.opennirs.org. It provides features such as scalable channel number, configurable regulated light intensities, programmable gain and lock-in amplification. In this paper, the system concept, hardware, software and mechanical implementation of the lightweight stand-alone instrument are presented and the evaluation and verification results of the instrument's hardware and physiological fNIRS functionality are described. Its capability to measure brain activity is demonstrated by qualitative signal assessments and a quantitative mental arithmetic based BCI study with 12 subjects.

Bayesian Estimation of Conditional Independence Graphs Improves Functional Connectivity Estimates

Functional connectivity concerns the correlated activity between neuronal populations in spatially segregated regions of the brain, which may be studied using functional magnetic resonance imaging (fMRI). This coupled activity is conveniently expressed using covariance, but this measure fails to distinguish between direct and indirect effects. A popular alternative that addresses this issue is partial correlation, which regresses out the signal of potentially confounding variables, resulting in a measure that reveals only direct connections. Importantly, provided the data are normally distributed, if two variables are conditionally independent given all other variables, their respective partial correlation is zero. In this paper, we propose a probabilistic generative model that allows us to estimate functional connectivity in terms of both partial correlations and a graph representing conditional independencies. Simulation results show that this methodology is able to outperform the graphical LASSO, which is the de facto standard for estimating partial correlations. Furthermore, we apply the model to estimate functional connectivity for twenty subjects using resting-state fMRI data. Results show that our model provides a richer representation of functional connectivity as compared to considering partial correlations alone. Finally, we demonstrate how our approach can be extended in several ways, for instance to achieve data fusion by informing the conditional independence graph with data from probabilistic tractography. As our Bayesian formulation of functional connectivity provides access to the posterior distribution instead of only to point estimates, we are able to quantify the uncertainty associated with our results. This reveals that while we are able to infer a clear backbone of connectivity in our empirical results, the data are not accurately described by simply looking at the mode of the distribution over connectivity. The implication of this is that deterministic alternatives may misjudge connectivity results by drawing conclusions from noisy and limited data.

Significant neuroscientific effort is devoted to elucidating functional connectivity between spatially segregated brain regions. This requires that we are able to quantify the degree of dependence between the signals of different areas. Yet how this must be accomplished—using which measures, each with their own limitations and interpretations—is far from a trivial task. One frequently advocated metric for functional connectivity is partial correlation, which is related to conditional independence: if two regions are independent, conditioned on all other regions, then their partial correlation is zero, assuming Gaussian data. Here, we use a probabilistic generative model to describe the relationship between functional connectivity and conditional independence. We apply this Bayesian approach to reveal functional connectivity between subcortical areas, and in addition we propose different variants of the generative model for connectivity. In the first, we address how a Bayesian formulation of connectivity allows for integration with other imaging modalities, resulting in data fusion. Secondly, we show how prior constraints can be incorporated in our estimates of connectivity.

Estimating multivariate similarity between neuroimaging datasets with sparse canonical correlation analysis: an application to perfusion imaging

An increasing number of neuroimaging studies are based on either combining more than one data modality (inter-modal) or combining more than one measurement from the same modality (intra-modal). To date, most intra-modal studies using multivariate statistics have focused on differences between datasets, for instance relying on classifiers to differentiate between effects in the data. However, to fully characterize these effects, multivariate methods able to measure similarities between datasets are needed. One classical technique for estimating the relationship between two datasets is canonical correlation analysis (CCA). However, in the context of high-dimensional data the application of CCA is extremely challenging. A recent extension of CCA, sparse CCA (SCCA), overcomes this limitation, by regularizing the model parameters while yielding a sparse solution. In this work, we modify SCCA with the aim of facilitating its application to high-dimensional neuroimaging data and finding meaningful multivariate image-to-image correspondences in intra-modal studies. In particular, we show how the optimal subset of variables can be estimated independently and we look at the information encoded in more than one set of SCCA transformations. We illustrate our framework using Arterial Spin Labeling data to investigate multivariate similarities between the effects of two antipsychotic drugs on cerebral blood flow.

A lower limb exoskeleton control system based on steady state visual evoked potentials

OBJECTIVE

We have developed an asynchronous brain-machine interface (BMI)-based lower limb exoskeleton control system based on steady-state visual evoked potentials (SSVEPs).

APPROACH

By decoding electroencephalography signals in real-time, users are able to walk forward, turn right, turn left, sit, and stand while wearing the exoskeleton. SSVEP stimulation is implemented with a visual stimulation unit, consisting of five light emitting diodes fixed to the exoskeleton. A canonical correlation analysis (CCA) method for the extraction of frequency information associated with the SSVEP was used in combination with k-nearest neighbors.

MAIN RESULTS

Overall, 11 healthy subjects participated in the experiment to evaluate performance. To achieve the best classification, CCA was first calibrated in an offline experiment. In the subsequent online experiment, our results exhibit accuracies of 91.3 ± 5.73%, a response time of 3.28 ± 1.82 s, an information transfer rate of 32.9 ± 9.13 bits/min, and a completion time of 1100 ± 154.92 s for the experimental parcour studied.

SIGNIFICANCE

The ability to achieve such high quality BMI control indicates that an SSVEP-based lower limb exoskeleton for gait assistance is becoming feasible.

A multimodal stress monitoring system with canonical correlation analysis

The multimodal stress monitoring headband is proposed for mobile stress management system. It is composed of headband and earplugs. Electroencephalography (EEG), hemoencephalography (HEG) and heart-rate variability (HRV) can be achieved simultaneously in the proposed system for user status estimation. With canonical correlation analysis (CCA) and temporal-kernel CCA (tkCCA) algorithm, those different signals can be combined for maximum correlation. Thanks to the proposed combination algorithm, the accuracy of the proposed system increased up to 19 percentage points than unimodal monitoring system in n-back task.

Three-way analysis of spectrospatial electromyography data: classification and interpretation

Classifying multivariate electromyography (EMG) data is an important problem in prosthesis control as well as in neurophysiological studies and diagnosis. With modern high-density EMG sensor technology, it is possible to capture the rich spectrospatial structure of the myoelectric activity. We hypothesize that multi-way machine learning methods can efficiently utilize this structure in classification as well as reveal interesting patterns in it. To this end, we investigate the suitability of existing three-way classification methods to EMG-based hand movement classification in spectrospatial domain, as well as extend these methods by sparsification and regularization. We propose to use Fourier-domain independent component analysis as preprocessing to improve classification and interpretability of the results. In high-density EMG experiments on hand movements across 10 subjects, three-way classification yielded higher average performance compared with state-of-the art classification based on temporal features, suggesting that the three-way analysis approach can efficiently utilize detailed spectrospatial information of high-density EMG. Phase and amplitude patterns of features selected by the classifier in finger-movement data were found to be consistent with known physiology. Thus, our approach can accurately resolve hand and finger movements on the basis of detailed spectrospatial information, and at the same time allows for physiological interpretation of the results.

Characterizing nonlinear relationships in functional imaging data using eigenspace maximal information canonical correlation analysis (emiCCA)

Many important problems in the analysis of neuroimages can be formulated as discovering the relationship between two sets of variables, a task for which linear techniques such as canonical correlation analysis (CCA) have been commonly used. However, to further explore potential nonlinear processes that might co-exist with linear ones in brain function, a more flexible method is required. Here, we propose a new unsupervised and data-driven method, termed the eigenspace maximal information canonical correlation analysis (emiCCA), which is capable of automatically capturing the linear and/or nonlinear relationships between various data sets. A simulation confirmed the superior performance of emiCCA in comparison with linear CCA and kernel CCA (a nonlinear version of CCA). An emiCCA framework for functional magnetic resonance imaging (fMRI) data processing was designed and applied to data from a real motor execution fMRI experiment. This analysis uncovered one linear (in primary motor cortex) and a few nonlinear networks (e.g., in the supplementary motor area, bilateral insula, and cerebellum). This suggests that these various task-related brain areas are part of networks that also contribute to the execution of movements of the hand. These results suggest that emiCCA is a promising technique for exploring various data.

Multimodal imaging in autism: an early review of comprehensive neural circuit characterization

There is accumulating evidence that the neurobiology of autism spectrum disorders (ASD) is linked to atypical neural communication and connectivity. This body of work emphasizes the need to characterize the function of multiple regions that comprise neural circuits rather than focusing on singular regions as contributing to deficits in ASD. Multimodal neuroimaging - the formal combination of multiple functional and structural measures of the brain - is extremely promising as an approach to understanding neural deficits in ASD. This review provides an overview of the multimodal imaging approach, and then provides a snapshot of how multimodal imaging has been applied in the study of ASD to date. This body of work is separated into two categories: one concerning whole brain connectomics and the other focused on characterizing neural circuits implicated as altered in ASD. We end this review by highlighting emerging themes from the existing body of literature, and new resources that will likely influence future multimodal imaging studies.

Structural Connectivity Changes Underlying Altered Working Memory Networks in Mild Cognitive Impairment: A Three-Way Image Fusion Analysis

BACKGROUND AND PURPOSE

Working memory impairment is among the earliest signs of cognitive decline in Alzheimer's disease (AD) and mild cognitive impairment (MCI). We aimed to study the functional and structural substrate of working memory impairment in early AD dementia and MCI.

METHODS

We studied a group of 12 MCI and AD subjects compared to 12 age- and gender-matched healthy elderly controls using diffusion tensor imaging (DTI), and functional magnetic resonance imaging (fMRI) during a 2-back versus 1-back letter recognition task. We performed a three-way image fusion analysis with joint independent component analysis of cortical activation during working memory, and DTI derived measures of fractional anisotropy (FA) and the mode of anisotropy.

RESULTS

We found significant hypoactivation in posterior brain areas and relative hyperactivation in anterior brain areas during working memory in AD/MCI subjects compared to controls. Corresponding independent components from DTI data revealed reduced FA and reduced mode of anisotropy in intracortical projecting fiber tracts with posterior predominance and increased FA and increased mode along the corticospinal tract in AD/MCI compared to controls.

CONCLUSIONS

Our findings suggest that impairments of structural fiber tract integrity accompany breakdown of posterior and relatively preserved anterior cortical activation during working memory performance in MCI/AD subjects.

Simultaneous EEG-fMRI: trial level spatio-temporal fusion for hierarchically reliable information discovery

Simultaneous electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) have been pursued in an effort to integrate complementary noninvasive information on brain activity. The primary goal involves better information discovery of the event-related neural activations at a spatial region of the BOLD fluctuation with the temporal resolution of the electrical signal. Many techniques and algorithms have been developed to integrate EEGs and fMRIs; however, the relative reliability of the integrated information is unclear. In this work, we propose a hierarchical framework to ensure the relative reliability of the integrated results and attempt to understand brain activation using this hierarchical ideal. First, spatial Independent Component Analysis (ICA) of fMRI and temporal ICA of EEG were performed to extract features at the trial level. Second, the maximal information coefficient (MIC) was adopted to temporally match them across the modalities for both linear and non-linear associations. Third, fMRI-constrained EEG source imaging was utilized to spatially match components across modalities. The simultaneously occurring events in the above two match steps provided EEG-fMRI spatial-temporal reliable integrated information, resulting in the most reliable components with high spatial and temporal resolution information. The other components discovered in the second or third steps provided second-level complementary information for flexible and cautious explanations. This paper contains two simulations and an example of real data, and the results indicate that the framework is a feasible approach to reveal cognitive processing in the human brain.