Canonical Correlation Analysis for Data Fusion and Group Inferences: Examining applications of medical imaging data

Neuroimage Analysis Methods and Artificial Intelligence Techniques for Reliable Biomarkers and Accurate Diagnosis of Schizophrenia: Achievements Made by Chinese Scholars Around the Past Decade

BACKGROUND AND HYPOTHESIS

Schizophrenia (SZ) is characterized by significant cognitive and behavioral disruptions. Neuroimaging techniques, particularly magnetic resonance imaging (MRI), have been widely utilized to investigate biomarkers of SZ, distinguish SZ from healthy conditions or other mental disorders, and explore biotypes within SZ or across SZ and other mental disorders, which aim to promote the accurate diagnosis of SZ. In China, research on SZ using MRI has grown considerably in recent years.

STUDY DESIGN

The article reviews advanced neuroimaging and artificial intelligence (AI) methods using single-modal or multimodal MRI to reveal the mechanism of SZ and promote accurate diagnosis of SZ, with a particular emphasis on the achievements made by Chinese scholars around the past decade.

STUDY RESULTS

Our article focuses on the methods for capturing subtle brain functional and structural properties from the high-dimensional MRI data, the multimodal fusion and feature selection methods for obtaining important and sparse neuroimaging features, the supervised statistical analysis and classification for distinguishing disorders, and the unsupervised clustering and semi-supervised learning methods for identifying neuroimage-based biotypes. Crucially, our article highlights the characteristics of each method and underscores the interconnections among various approaches regarding biomarker extraction and neuroimage-based diagnosis, which is beneficial not only for comprehending SZ but also for exploring other mental disorders.

CONCLUSIONS

We offer a valuable review of advanced neuroimage analysis and AI methods primarily focused on SZ research by Chinese scholars, aiming to promote the diagnosis, treatment, and prevention of SZ, as well as other mental disorders, both within China and internationally.

A novel temporal-frequency combination pattern optimization approach based on information fusion for motor imagery BCIs

Motor imagery (MI) stands as a powerful paradigm within Brain-Computer Interface (BCI) research due to its ability to induce changes in brain rhythms detectable through common spatial patterns (CSP). However, the raw feature sets captured often contain redundant and invalid information, potentially hindering CSP performance. Methodology-wise, we propose the Information Fusion for Optimizing Temporal-Frequency Combination Pattern (IFTFCP) algorithm to enhance raw feature optimization. Initially, preprocessed data undergoes simultaneous processing in both time and frequency domains via sliding overlapping time windows and filter banks. Subsequently, we introduce the Pearson-Fisher combinational method along with Discriminant Correlation Analysis (DCA) for joint feature selection and fusion. These steps aim to refine raw electroencephalogram (EEG) features. For precise classification of binary MI problems, an Radial Basis Function (RBF)-kernel Support Vector Machine classifier is trained. To validate the efficacy of IFTFCP and evaluate it against other techniques, we conducted experimental investigations using two EEG datasets. Results indicate a notably superior classification performance, boasting an average accuracy of 78.14% and 85.98% on dataset 1 and dataset 2, which is better than other methods outlined in this article. The study's findings suggest potential benefits for the advancement of MI-based BCI strategies, particularly in the domain of feature fusion.

Multi-Server Multi-Function Distributed Computation

The work here studies the communication cost for a multi-server multi-task distributed computation framework, as well as for a broad class of functions and data statistics. Considering the framework where a user seeks the computation of multiple complex (conceivably non-linear) tasks from a set of distributed servers, we establish the communication cost upper bounds for a variety of data statistics, function classes, and data placements across the servers. To do so, we proceed to apply, for the first time here, Körner's characteristic graph approach-which is known to capture the structural properties of data and functions-to the promising framework of multi-server multi-task distributed computing. Going beyond the general expressions, and in order to offer clearer insight, we also consider the well-known scenario of cyclic dataset placement and linearly separable functions over the binary field, in which case, our approach exhibits considerable gains over the state of the art. Similar gains are identified for the case of multi-linear functions.

A novel biomarker selection method using multimodal neuroimaging data

Identifying biomarkers is essential to obtain the optimal therapeutic benefit while treating patients with late-life depression (LLD). We compare LLD patients with healthy controls (HC) using resting-state functional magnetic resonance and diffusion tensor imaging data to identify neuroimaging biomarkers that may be potentially associated with the underlying pathophysiology of LLD. We implement a Bayesian multimodal local false discovery rate approach for functional connectivity, borrowing strength from structural connectivity to identify disrupted functional connectivity of LLD compared to HC. In the Bayesian framework, we develop an algorithm to control the overall false discovery rate of our findings. We compare our findings with the literature and show that our approach can better detect some regions never discovered before for LLD patients. The Hub of our discovery related to various neurobehavioral disorders can be used to develop behavioral interventions to treat LLD patients who do not respond to antidepressants.

Modelling-based joint embedding of histology and genomics using canonical correlation analysis for breast cancer survival prediction

Traditional approaches to predicting breast cancer patients' survival outcomes were based on clinical subgroups, the PAM50 genes, or the histological tissue's evaluation. With the growth of multi-modality datasets capturing diverse information (such as genomics, histology, radiology and clinical data) about the same cancer, information can be integrated using advanced tools and have improved survival prediction. These methods implicitly exploit the key observation that different modalities originate from the same cancer source and jointly provide a complete picture of the cancer. In this work, we investigate the benefits of explicitly modelling multi-modality data as originating from the same cancer under a probabilistic framework. Specifically, we consider histology and genomics as two modalities originating from the same breast cancer under a probabilistic graphical model (PGM). We construct maximum likelihood estimates of the PGM parameters based on canonical correlation analysis (CCA) and then infer the underlying properties of the cancer patient, such as survival. Equivalently, we construct CCA-based joint embeddings of the two modalities and input them to a learnable predictor. Real-world properties of sparsity and graph-structures are captured in the penalized variants of CCA (pCCA) and are better suited for cancer applications. For generating richer multi-dimensional embeddings with pCCA, we introduce two novel embedding schemes that encourage orthogonality to generate more informative embeddings. The efficacy of our proposed prediction pipeline is first demonstrated via low prediction errors of the hidden variable and the generation of informative embeddings on simulated data. When applied to breast cancer histology and RNA-sequencing expression data from The Cancer Genome Atlas (TCGA), our model can provide survival predictions with average concordance-indices of up to 68.32% along with interpretability. We also illustrate how the pCCA embeddings can be used for survival analysis through Kaplan-Meier curves.

Apathy scores in Parkinson's disease relate to EEG components in an incentivized motor task

Apathy is one of the most prevalent non-motor symptoms of Parkinson's disease and is characterized by decreased goal-directed behaviour due to a lack of motivation and/or impaired emotional reactivity. Despite its high prevalence, the neurophysiological mechanisms underlying apathy in Parkinson's disease, which may guide neuromodulation interventions, are poorly understood. Here, we investigated the neural oscillatory characteristics of apathy in Parkinson's disease using EEG data recorded during an incentivized motor task. Thirteen Parkinson's disease patients with apathy and 13 Parkinson's disease patients without apathy as well as 12 healthy controls were instructed to squeeze a hand grip device to earn a monetary reward proportional to the grip force they used. Event-related spectral perturbations during the presentation of a reward cue and squeezing were analysed using multiset canonical correlation analysis to detect different orthogonal components of temporally consistent event-related spectral perturbations across trials and participants. The first component, predominantly located over parietal regions, demonstrated suppression of low-beta (12-20 Hz) power (i.e. beta desynchronization) during reward cue presentation that was significantly smaller in Parkinson's disease patients with apathy compared with healthy controls. Unlike traditional event-related spectral perturbation analysis, the beta desynchronization in this component was significantly correlated with clinical apathy scores. Higher monetary rewards resulted in larger beta desynchronization in healthy controls but not Parkinson's disease patients. The second component contained gamma and theta frequencies and demonstrated exaggerated theta (4-8 Hz) power in Parkinson's disease patients with apathy during the reward cue and squeezing compared with healthy controls (HCs), and this was positively correlated with Montreal Cognitive Assessment scores. The third component, over central regions, demonstrated significantly different beta power across groups, with apathetic groups having the lowest beta power. Our results emphasize that altered low-beta and low-theta oscillations are critical for reward processing and motor planning in Parkinson's disease patients with apathy and these may provide a target for non-invasive neuromodulation.

Dysconnection and cognition in schizophrenia: A spectral dynamic causal modeling study

Schizophrenia (SZ) is a severe mental disorder characterized by failure of functional integration (aka dysconnection) across the brain. Recent functional connectivity (FC) studies have adopted functional parcellations to define subnetworks of large-scale networks, and to characterize the (dys)connection between them, in normal and clinical populations. While FC examines statistical dependencies between observations, model-based effective connectivity (EC) can disclose the causal influences that underwrite the observed dependencies. In this study, we investigated resting state EC within seven large-scale networks, in 66 SZ and 74 healthy subjects from a public dataset. The results showed that a remarkable 33% of the effective connections (among subnetworks) of the cognitive control network had been pathologically modulated in SZ. Further dysconnection was identified within the visual, default mode and sensorimotor networks of SZ subjects, with 24%, 20%, and 11% aberrant couplings. Overall, the proportion of discriminative connections was remarkably larger in EC (24%) than FC (1%) analysis. Subsequently, to study the neural correlates of impaired cognition in SZ, we conducted a canonical correlation analysis between the EC parameters and the cognitive scores of the patients. As such, the self-inhibitions of supplementary motor area and paracentral lobule (in the sensorimotor network) and the excitatory connection from parahippocampal gyrus to inferior temporal gyrus (in the cognitive control network) were significantly correlated with the social cognition, reasoning/problem solving and working memory capabilities of the patients. Future research can investigate the potential of whole-brain EC as a biomarker for diagnosis of brain disorders and for neuroimaging-based cognitive assessment.

Inter-individual single-trial classification of MEG data using M-CCA

Neuroscientific studies often involve some form of group analysis over multiple participants. This requires alignment of recordings across participants. A naive solution is to assume that participants' recordings can be aligned anatomically in sensor space. However, this assumption is likely violated due to anatomical and functional differences between individual brains. In magnetoencephalography (MEG) recordings the problem of inter-subject alignment is exacerbated by the susceptibility of MEG to individual cortical folding patterns as well as the inter-subject variability of sensor locations over the brain due to the use of a fixed helmet. Hence, an approach to combine MEG data over individual brains should relax the assumptions that a) brain anatomy and function are tightly linked and b) that the same sensors capture functionally comparable brain activation across individuals. Here we use multiset canonical correlation analysis (M-CCA) to find a common representation of MEG activations recorded from 15 participants performing a grasping task. The M-CCA algorithm was applied to transform the data of a set of multiple participants into a common space with maximum correlation between participants. Importantly, we derive a method to transform data from a new, previously unseen participant into this common representation. This makes it useful for applications that require transfer of models derived from a group of individuals to new individuals. We demonstrate the usefulness and superiority of the approach with respect to previously used approaches. Finally, we show that our approach requires only a small number of labeled data from the new participant. The proposed method demonstrates that functionally motivated common spaces have potential applications in reducing training time of online brain-computer interfaces, where models can be pre-trained on previous participants/sessions. Moreover, inter-subject alignment via M-CCA has the potential for combining data of different participants and could become helpful in future endeavors on large open datasets.

A Discriminant Information Theoretic Learning Framework for Multi-modal Feature Representation

As sensory and computing technology advances, multi-modal features have been playing a central role in ubiquitously representing patterns and phenomenons for effective information analysis and recognition. As a result, multi-modal feature representation is becoming a progressively significant direction of academic research and real applications. Nevertheless, numerous challenges remain ahead, especially in the joint utilization of discriminatory representations and complementary representations from multi-modal features. In this paper, a discriminant information theoretic learning (DITL) framework is proposed to address these challenges. By employing this proposed framework, the discrimination and complementation within the given multi-modal features are exploited jointly, resulting a high quality feature representation. According to characteristics of the DITL framework, the newly generated feature representation is further optimized, leading to lower computational complexity and improved system performance. To demonstrate the effectiveness and generality of DITL, we conducted experiments on several recognition examples, including both static cases such as handwritten digit recognition, face recognition and object recognition, and dynamic cases such as video based human emotion recognition and action recognition. The results show that the proposed framework outperforms state-of-the-art algorithms.

Finding Explanations in AI Fusion of Electro-Optical/Passive Radio-Frequency Data

In the Information Age, the widespread usage of blackbox algorithms makes it difficult to understand how data is used. The practice of sensor fusion to achieve results is widespread, as there are many tools to further improve the robustness and performance of a model. In this study, we demonstrate the utilization of a Long Short-Term Memory (LSTM-CCA) model for the fusion of Passive RF (P-RF) and Electro-Optical (EO) data in order to gain insights into how P-RF data are utilized. The P-RF data are constructed from the in-phase and quadrature component (I/Q) data processed via histograms, and are combined with enhanced EO data via dense optical flow (DOF). The preprocessed data are then used as training data with an LSTM-CCA model in order to achieve object detection and tracking. In order to determine the impact of the different data inputs, a greedy algorithm (explainX.ai) is implemented to determine the weight and impact of the canonical variates provided to the fusion model on a scenario-by-scenario basis. This research introduces an explainable LSTM-CCA framework for P-RF and EO sensor fusion, providing novel insights into the sensor fusion process that can assist in the detection and differentiation of targets and help decision-makers to determine the weights for each input.

Modern views of machine learning for precision psychiatry

In light of the National Institute of Mental Health (NIMH)'s Research Domain Criteria (RDoC), the advent of functional neuroimaging, novel technologies and methods provide new opportunities to develop precise and personalized prognosis and diagnosis of mental disorders. Machine learning (ML) and artificial intelligence (AI) technologies are playing an increasingly critical role in the new era of precision psychiatry. Combining ML/AI with neuromodulation technologies can potentially provide explainable solutions in clinical practice and effective therapeutic treatment. Advanced wearable and mobile technologies also call for the new role of ML/AI for digital phenotyping in mobile mental health. In this review, we provide a comprehensive review of ML methodologies and applications by combining neuroimaging, neuromodulation, and advanced mobile technologies in psychiatry practice. We further review the role of ML in molecular phenotyping and cross-species biomarker identification in precision psychiatry. We also discuss explainable AI (XAI) and neuromodulation in a closed human-in-the-loop manner and highlight the ML potential in multi-media information extraction and multi-modal data fusion. Finally, we discuss conceptual and practical challenges in precision psychiatry and highlight ML opportunities in future research.

Early soft and flexible fusion of electroencephalography and functional magnetic resonance imaging via double coupled matrix tensor factorization for multisubject group analysis

Data fusion refers to the joint analysis of multiple datasets that provide different (e.g., complementary) views of the same task. In general, it can extract more information than separate analyses can. Jointly analyzing electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) measurements has been proved to be highly beneficial to the study of the brain function, mainly because these neuroimaging modalities have complementary spatiotemporal resolution: EEG offers good temporal resolution while fMRI is better in its spatial resolution. The EEG–fMRI fusion methods that have been reported so far ignore the underlying multiway nature of the data in at least one of the modalities and/or rely on very strong assumptions concerning the relation of the respective datasets. For example, in multisubject analysis, it is commonly assumed that the hemodynamic response function is a priori known for all subjects and/or the coupling across corresponding modes is assumed to be exact (hard). In this article, these two limitations are overcome by adopting tensor models for both modalities and by following soft and flexible coupling approaches to implement the multimodal fusion. The obtained results are compared against those of parallel independent component analysis and hard coupling alternatives, with both synthetic and real data (epilepsy and visual oddball paradigm). Our results demonstrate the clear advantage of using soft and flexible coupled tensor decompositions in scenarios that do not conform with the hard coupling assumption.

The integration of social and neural synchrony: a case for ecologically valid research using MEG neuroimaging

The recent decade has seen a shift from artificial and environmentally deprived experiments in neuroscience to real-life studies on multiple brains in interaction, coordination and synchrony. In these new interpersonal synchrony experiments, there has been a growing trend to employ naturalistic social interactions to evaluate mechanisms underlying synchronous neuronal communication. Here, we emphasize the importance of integrating the assessment of neural synchrony with measurement of nonverbal behavioral synchrony as expressed in various social contexts: relaxed social interactions, planning a joint pleasurable activity, conflict discussion, invocation of trauma, or support giving and assess the integration of neural and behavioral synchrony across developmental stages and psychopathological conditions. We also showcase the advantages of magnetoencephalography neuroimaging as a promising tool for studying interactive neural synchrony and consider the challenge of ecological validity at the expense of experimental rigor. We review recent evidence of rhythmic information flow between brains in interaction and conclude with addressing state-of-the-art developments that may contribute to advance research on brain-to-brain coordination to the next level.

Multiuser gesture recognition using sEMG signals via canonical correlation analysis and optimal transport

Myoelectric interfaces have received much attention in the field of prosthesis control, neuro-rehabilitation systems and human-computer interaction. However, when different users perform the same gesture, the electromyography (EMG) signals can vary greatly. It is essential to design a multiuser myoelectric interface that can be simply used by novel users while maintaining good gesture classification performance. To cope with this problem, canonical correlation analysis (CCA) has been used to extract the inherent user-independent properties of EMG signals generated from the same gestures from multiple users and demonstrated superior performance. In this paper, we move forward to propose a novel framework based on CCA and optimal transport (OT), termed as CCA-OT. By optimal transport, the discrepancies in data distribution between the transformed feature matrix from the training and the testing sets can be further reduced. Experimental results on the defined 13 Chinese sign language gestures performed by 10 intact-limbed subjects demonstrated that the classification rate of our proposed CCA-OT framework is significantly higher than that of the CCA-only framework with an 8.49% promotion, which shows the necessity to reduce the drift in probability distribution functions (PDFs) of the different domains. The CCA-OT framework provides a promising method for the multiuser myoelectric interface which can be easily adapted to new users. This improvement will further facilitate the widespread implementation of myoelectric control systems using pattern recognition techniques.

A technical review of canonical correlation analysis for neuroscience applications

Collecting comprehensive data sets of the same subject has become a standard in neuroscience research and uncovering multivariate relationships among collected data sets have gained significant attentions in recent years. Canonical correlation analysis (CCA) is one of the powerful multivariate tools to jointly investigate relationships among multiple data sets, which can uncover disease or environmental effects in various modalities simultaneously and characterize changes during development, aging, and disease progressions comprehensively. In the past 10 years, despite an increasing number of studies have utilized CCA in multivariate analysis, simple conventional CCA dominates these applications. Multiple CCA-variant techniques have been proposed to improve the model performance; however, the complicated multivariate formulations and not well-known capabilities have delayed their wide applications. Therefore, in this study, a comprehensive review of CCA and its variant techniques is provided. Detailed technical formulation with analytical and numerical solutions, current applications in neuroscience research, and advantages and limitations of each CCA-related technique are discussed. Finally, a general guideline in how to select the most appropriate CCA-related technique based on the properties of available data sets and particularly targeted neuroscience questions is provided.

Meta-Modal Information Flow: A Method for Capturing Multimodal Modular Disconnectivity in Schizophrenia

OBJECTIVE

Multimodal measurements of the same phenomena provide complementary information and highlight different perspectives, albeit each with their own limitations. A focus on a single modality may lead to incorrect inferences, which is especially important when a studied phenomenon is a disease. In this paper, we introduce a method that takes advantage of multimodal data in addressing the hypotheses of disconnectivity and dysfunction within schizophrenia (SZ).

METHODS

We start with estimating and visualizing links within and among extracted multimodal data features using a Gaussian graphical model (GGM). We then propose a modularity-based method that can be applied to the GGM to identify links that are associated with mental illness across a multimodal data set. Through simulation and real data, we show our approach reveals important information about disease-related network disruptions that are missed with a focus on a single modality. We use functional MRI (fMRI), diffusion MRI (dMRI), and structural MRI (sMRI) to compute the fractional amplitude of low frequency fluctuations (fALFF), fractional anisotropy (FA), and gray matter (GM) concentration maps. These three modalities are analyzed using our modularity method.

RESULTS

Our results show missing links that are only captured by the cross-modal information that may play an important role in disconnectivity between the components.

CONCLUSION

We identified multimodal (fALFF, FA and GM) disconnectivity in the default mode network area in patients with SZ, which would not have been detectable in a single modality.

SIGNIFICANCE

The proposed approach provides an important new tool for capturing information that is distributed among multiple imaging modalities.

Resolving the Connectome, Spectrally-Specific Functional Connectivity Networks and Their Distinct Contributions to Behavior

The resting human brain exhibits spontaneous patterns of activity that reflect features of the underlying neural substrate. Examination of interareal coupling of resting-state oscillatory activity has revealed that the brain’s resting activity is composed of functional networks, whose topographies differ depending on oscillatory frequency, suggesting a role for carrier frequency as a means of creating multiplexed, or functionally segregated, communication channels between brain areas. Using canonical correlation analysis (CCA), we examined spectrally resolved resting-state connectivity patterns derived from magnetoencephalography (MEG) recordings to determine the relationship between connectivity intrinsic to different frequency channels and a battery of over a hundred behavioral and demographic indicators, in a group of 89 young healthy participants. We demonstrate that each of the classical frequency bands in the range 1–40 Hz (δ, θ, α, β, and γ) delineates a subnetwork that is behaviorally relevant, spatially distinct, and whose expression is either negatively or positively predictive of individual traits, with the strongest link in the α-band being negative and networks oscillating at different frequencies, such as θ, β, and γ carrying positive function.

Finding the needle in a high-dimensional haystack: Canonical correlation analysis for neuroscientists

The 21st century marks the emergence of "big data" with a rapid increase in the availability of datasets with multiple measurements. In neuroscience, brain-imaging datasets are more commonly accompanied by dozens or hundreds of phenotypic subject descriptors on the behavioral, neural, and genomic level. The complexity of such "big data" repositories offer new opportunities and pose new challenges for systems neuroscience. Canonical correlation analysis (CCA) is a prototypical family of methods that is useful in identifying the links between variable sets from different modalities. Importantly, CCA is well suited to describing relationships across multiple sets of data, such as in recently available big biomedical datasets. Our primer discusses the rationale, promises, and pitfalls of CCA.

Roy's largest root under rank-one perturbations: the complex valued case and applications

The largest eigenvalue of a single or a double Wishart matrix, both known as Roy's largest root, plays an important role in a variety of applications. Recently, via a small noise perturbation approach with fixed dimension and degrees of freedom, Johnstone and Nadler derived simple yet accurate approximations to its distribution in the real valued case, under a rank-one alternative. In this paper, we extend their results to the complex valued case for five common single matrix and double matrix settings. In addition, we study the finite sample distribution of the leading eigenvector. We present the utility of our results in several signal detection and communication applications, and illustrate their accuracy via simulations.

In search of multimodal brain alterations in Alzheimer's and Binswanger's disease

Structural and functional brain abnormalities have been widely identified in dementia, but with variable replicability and significant overlap. Alzheimer's disease (AD) and Binswanger's disease (BD) share similar symptoms and common brain changes that can confound diagnosis. In this study, we aimed to investigate correlated structural and functional brain changes in AD and BD by combining resting-state functional magnetic resonance imaging (fMRI) and diffusion MRI. A group independent component analysis was first performed on the fMRI data to extract 49 intrinsic connectivity networks (ICNs). Then we conducted a multi-set canonical correlation analysis on three features, functional network connectivity (FNC) between ICNs, fractional anisotropy (FA) and mean diffusivity (MD). Two inter-correlated components show significant group differences. The first component demonstrates distinct brain changes between AD and BD. AD shows increased cerebellar FNC but decreased thalamic and hippocampal FNC. Such FNC alterations are linked to the decreased corpus callosum FA. AD also has increased MD in the frontal and temporal cortex, but BD shows opposite alterations. The second component demonstrates specific brain changes in BD. Increased FNC is mainly between default mode and sensory regions, while decreased FNC is mainly within the default mode domain and related to auditory regions. The FNC changes are associated with FA changes in posterior/middle cingulum cortex and visual cortex and increased MD in thalamus and hippocampus. Our findings provide evidence of linked functional and structural deficits in dementia and suggest that AD and BD have both common and distinct changes in white matter integrity and functional connectivity.

•

This is the first study to explore multi-modalities changes in different dementia.

•

A multimodal fusion method is applied to identify joint components.

•

Brain abnormalities in different modalities are highly correlated.

•

Alzheimer's and Binswanger's disease share similar brain changes.

•

Alzheimer's and Binswanger's disease also have distinct brain changes.

Performing Sparse Regularization and Dimension Reduction Simultaneously in Multimodal Data Fusion

Collecting multiple modalities of neuroimaging data on the same subject is increasingly becoming the norm in clinical practice and research. Fusing multiple modalities to find related patterns is a challenge in neuroimaging analysis. Canonical correlation analysis (CCA) is commonly used as a symmetric data fusion technique to find related patterns among multiple modalities. In CCA-based data fusion, principal component analysis (PCA) is frequently applied as a preprocessing step to reduce data dimension followed by CCA on dimension-reduced data. PCA, however, does not differentiate between informative voxels from non-informative voxels in the dimension reduction step. Sparse PCA (sPCA) extends traditional PCA by adding sparse regularization that assigns zero weights to non-informative voxels. In this study, sPCA is incorporated into CCA-based fusion analysis and applied on neuroimaging data. A cross-validation method is developed and validated to optimize the parameters in sPCA. Different simulations are carried out to evaluate the improvement by introducing sparsity constraint to PCA. Four fusion methods including sPCA+CCA, PCA+CCA, parallel ICA and sparse CCA were applied on structural and functional magnetic resonance imaging data of mild cognitive impairment subjects and normal controls. Our results indicate that sPCA significantly can reduce the impact of non-informative voxels and lead to improved statistical power in uncovering disease-related patterns by a fusion analysis.

Aims, Study Design, and Enrollment Results From the Assessing Predictors of Infant Respiratory Syncytial Virus Effects and Severity Study

Background

The majority of infants hospitalized with primary respiratory syncytial virus (RSV) infection have no obvious risk factors for severe disease.

Objective

The aim of this study (Assessing Predictors of Infant RSV Effects and Severity, AsPIRES) was to identify factors associated with severe disease in full-term healthy infants younger than 10 months with primary RSV infection.

Methods

RSV infected infants were enrolled from 3 cohorts during consecutive winters from August 2012 to April 2016 in Rochester, New York. A birth cohort was prospectively enrolled and followed through their first winter for development of RSV infection. An outpatient supplemental cohort was enrolled in the emergency department or pediatric offices, and a hospital cohort was enrolled on admission with RSV infection. RSV was diagnosed by reverse transcriptase-polymerase chain reaction. Demographic and clinical data were recorded and samples collected for assays: buccal swab (cytomegalovirus polymerase chain reaction, PCR), nasal swab (RSV qualitative PCR, complete viral gene sequence, 16S ribosomal ribonucleic acid [RNA] amplicon microbiota analysis), nasal wash (chemokine and cytokine assays), nasal brush (nasal respiratory epithelial cell gene expression using RNA sequencing [RNAseq]), and 2 to 3 ml of heparinized blood (flow cytometry, RNAseq analysis of purified cluster of differentiation [CD]4+, CD8+, B cells and natural killer cells, and RSV-specific antibody). Cord blood (RSV-specific antibody) was also collected for the birth cohort. Univariate and multivariate logistic regression will be used for analysis of data using a continuous Global Respiratory Severity Score (GRSS) as the outcome variable. Novel statistical methods will be developed for integration of the large complex datasets.

Results

A total of 453 infants were enrolled into the 3 cohorts; 226 in the birth cohort, 60 in the supplemental cohort, and 78 in the hospital cohort. A total of 126 birth cohort infants remained in the study and were evaluated for 150 respiratory illnesses. Of the 60 RSV positive infants in the supplemental cohort, 42 completed the study, whereas all 78 of the RSV positive hospital cohort infants completed the study. A GRSS was calculated for each RSV-infected infant and is being used to analyze each of the complex datasets by correlation with disease severity in univariate and multivariate methods.

Conclusions

The AsPIRES study will provide insights into the complex pathogenesis of RSV infection in healthy full-term infants with primary RSV infection. The analysis will allow assessment of multiple factors potentially influencing the severity of RSV infection including the level of RSV specific antibodies, the innate immune response of nasal epithelial cells, the adaptive response by various lymphocyte subsets, the resident airway microbiota, and viral factors. Results of this study will inform disease interventions such as vaccines and antiviral therapies.

Exercise-Induced Changes of Multimodal Interactions Within the Autonomic Nervous Network

Physical exercise has been shown to modulate activity within the autonomic nervous system (ANS). Considering physical exercise as a holistic stimulus on the nervous system and specifically the ANS, uni- and multimodal analysis tools were applied to characterize centrally driven interactions and control of ANS functions. Nineteen young and physically active participants performed treadmill tests at individually determined moderate and high intensities. Continuous electrodermal activity (EDA), heart rate (HR), and skin temperature at wrist (Temp) were recorded by wireless multisensor devices (Empatica^® E4, Milan, Italy) before and 30 min after exercise. Artifact-free continuous 3 min intervals were analyzed. For unimodal analysis, mean values were calculated, for bimodal and multimodal analysis canonical correlation analysis (CCA) was performed. Unimodal results indicate that physical exercise affects ANS activity. More specifically, Temp increased due to physical activity (moderate intensity: from 34.15°C to 35.34°C and high intensity: from 34.11°C to 35.09°C). HR increased more for the high (from 60.76 bpm to 79.89 bpm) than for the moderate (from 64.81 bpm to 70.83 bpm) intensity. EDA was higher for the high (pre: 8.06 μS and post: 9.37 μS) than for the moderate (pre: 4.31 μS and post: 3.91 μS) intensity. Bimodal analyses revealed high variations in correlations before exercise. The overall correlation coefficient showed varying correlations in pretest measures for all modality pairs (EDA-HR, HR-Temp, Temp-EDA at moderate: 0.831, 0.998, 0.921 and high: 0.706, 0, 0.578). After exercising at moderate intensity coefficients changed little (0.828, 0.744, 0.994), but increased substantially for all modality pairs after exercising at high intensity (0.976, 0.898, 0.926). Multimodal analysis confirmed bimodal results. Exercise-induced changes in ANS activity can be found in multiple ANS modalities as well as in their interactions. Those changes are intensity-specific: with higher intensity the interactions increase. Canonical correlations between different ANS modalities may therefore offer a feasible approach to determine exercise induced modulations of ANS activity.

Multivariate group-level analysis for task fMRI data with canonical correlation analysis

Task-based functional Magnetic Resonance Imaging (fMRI) has been widely used to determine population-based brain activations for cognitive tasks. Popular group-level analysis in fMRI is based on the general linear model and constitutes a univariate method. However, univariate methods are known to suffer from low sensitivity for a given specificity because the spatial covariance structure at each voxel is not taken entirely into account. In this study, a spatially constrained local multivariate model is introduced for group-level analysis to improve sensitivity at a given specificity for activation detection. The proposed model is formulated in terms of a multivariate constrained optimization problem based on the maximum log likelihood method and solved efficiently with numerical optimization techniques. Both simulated data mimicking real fMRI time series at multiple noise fractions and real fMRI episodic memory data have been used to evaluate the performance of the proposed method. For simulated data, the area under the receiver operating characteristic curves in detecting group activations increases for the subject and group level multivariate method by 20%, as compared to the univariate method. Results from real fMRI data indicate a significant increase in group-level activation detection, particularly in hippocampus, para-hippocampal area and nearby medial temporal lobe regions with the proposed method.

A Projection CCA Method for Effective fMRI Data Analysis

OBJECTIVE

Canonical correlation analysis (CCA) is a data-driven method that has been successfully used in functional magnetic resonance imaging (fMRI) data analysis. Standard CCA extracts meaningful information from a pair of data sets by seeking pairs of linear combinations from two sets of variables with maximum pairwise correlation. So far, however, this method has been used without incorporating prior information available for fMRI data. In this paper, we address this issue by proposing a new CCA method named pCCA (for projection CCA).

METHODS

The proposed method is obtained by projection onto a set of basis vectors that better characterize temporal information in the fMRI data set. A methodology is presented to describe the basis selection process using discrete cosine transform (DCT) basis functions. Employing DCT guides the estimated canonical variates, yielding a more computationally efficient CCA procedure.

RESULTS

The performance gain of the proposed pCCA algorithm over standard and regularized CCA is illustrated on both simulated and real fMRI datasets from resting state, block paradigm task-related and event-related experiments. The results have shown that the proposed pCCA successfully extracts latent components from the task as well as resting-state datasets with increased specificity of the activated voxels.

CONCLUSION

In addition to offering a new CCA approach, when applied on fMRI data, the proposed algorithm adapts to variations of brain activity patterns and reveals the functionally connected brain regions.

SIGNIFICANCE

The proposed method can be seen as a regularized CCA method where regularization is introduced via basis expansion, which has the advantage of enforcing smoothness on canonical components.

Linked 4-Way Multimodal Brain Differences in Schizophrenia in a Large Chinese Han Population

Multimodal fusion has been regarded as a promising tool to discover covarying patterns of multiple imaging types impaired in brain diseases, such as schizophrenia (SZ). In this article, we aim to investigate the covarying abnormalities underlying SZ in a large Chinese Han population (307 SZs, 298 healthy controls [HCs]). Four types of magnetic resonance imaging (MRI) features, including regional homogeneity (ReHo) from resting-state functional MRI, gray matter volume (GM) from structural MRI, fractional anisotropy (FA) from diffusion MRI, and functional network connectivity (FNC) resulted from group independent component analysis, were jointly analyzed by a data-driven multivariate fusion method. Results suggest that a widely distributed network disruption appears in SZ patients, with synchronous changes in both functional and structural regions, especially the basal ganglia network, salience network (SAN), and the frontoparietal network. Such a multimodal coalteration was also replicated in another independent Chinese sample (40 SZs, 66 HCs). Our results on auditory verbal hallucination (AVH) also provide evidence for the hypothesis that prefrontal hypoactivation and temporal hyperactivation in SZ may lead to failure of executive control and inhibition, which is relevant to AVH. In addition, impaired working memory performance was found associated with GM reduction and FA decrease in SZ in prefrontal and superior temporal area, in both discovery and replication datasets. In summary, by leveraging multiple imaging and clinical information into one framework to observe brain in multiple views, we can integrate multiple inferences about SZ from large-scale population and offer unique perspectives regarding the missing links between the brain function and structure that may not be achieved by separate unimodal analyses.

Canonical correlation analysis of brain prefrontal activity measured by functional near infra-red spectroscopy (fNIRS) during a moral judgment task

Individuals differ in the extent to which they make decisions in different moral dilemmas. In this study, we investigated the relationship between functional brain activities during moral decision making and psychopathic personality traits in a healthy population. We measured the hemodynamic activities of the brain by functional near-infrared spectroscopy (fNIRS). FNIRS is an evolving non-invasive neuroimaging modality which is relatively inexpensive, patient friendly and robust to subject movement. Psychopathic traits were evaluated through a self-report questionnaire called the Psychopathic Personality Inventory Revised (PPI-R). We recorded functional brain activities of 30 healthy subjects while they performed a moral judgment (MJ) task. Regularized canonical correlation analysis (R-CCA) was applied to find the relationships between activation in different regions of prefrontal cortex (PFC) and the core psychopathic traits. Our results showed a significant canonical correlation between PFC activation and PPI-R content scale (PPI-R-CS). Specifically, coldheartedness and carefree non-planfulness were the only PPI-R-CS factors that were highly correlated with PFC activation during personal (emotionally salient) MJ, while Machiavellian egocentricity, rebellious nonconformity, coldheartedness, and carefree non-planfulness were the core traits that exhibited the same dynamics as PFC activation during impersonal (more logical) MJ. Furthermore, ventromedial prefrontal cortex (vmPFC) and left lateral PFC were the most positively correlated regions with PPI-R-CS traits during personal MJ, and the right vmPFC and right lateral PFC in impersonal MJ.

Multimodal Neuroimaging: Basic Concepts and Classification of Neuropsychiatric Diseases

Neuroimaging techniques are widely used in neuroscience to visualize neural activity, to improve our understanding of brain mechanisms, and to identify biomarkers-especially for psychiatric diseases; however, each neuroimaging technique has several limitations. These limitations led to the development of multimodal neuroimaging (MN), which combines data obtained from multiple neuroimaging techniques, such as electroencephalography, functional magnetic resonance imaging, and yields more detailed information about brain dynamics. There are several types of MN, including visual inspection, data integration, and data fusion. This literature review aimed to provide a brief summary and basic information about MN techniques (data fusion approaches in particular) and classification approaches. Data fusion approaches are generally categorized as asymmetric and symmetric. The present review focused exclusively on studies based on symmetric data fusion methods (data-driven methods), such as independent component analysis and principal component analysis. Machine learning techniques have recently been introduced for use in identifying diseases and biomarkers of disease. The machine learning technique most widely used by neuroscientists is classification-especially support vector machine classification. Several studies differentiated patients with psychiatric diseases and healthy controls with using combined datasets. The common conclusion among these studies is that the prediction of diseases increases when combining data via MN techniques; however, there remain a few challenges associated with MN, such as sample size. Perhaps in the future N-way fusion can be used to combine multiple neuroimaging techniques or nonimaging predictors (eg, cognitive ability) to overcome the limitations of MN.

Multiway canonical correlation analysis of brain data

Brain data recorded with electroencephalography (EEG), magnetoencephalography (MEG) and related techniques often have poor signal-to-noise ratios due to the presence of multiple competing sources and artifacts. A common remedy is to average responses over repeats of the same stimulus, but this is not applicable for temporally extended stimuli that are presented only once (speech, music, movies, natural sound). An alternative is to average responses over multiple subjects that were presented with identical stimuli, but differences in geometry of brain sources and sensors reduce the effectiveness of this solution. Multiway canonical correlation analysis (MCCA) brings a solution to this problem by allowing data from multiple subjects to be fused in such a way as to extract components common to all. This paper reviews the method, offers application examples that illustrate its effectiveness, and outlines the caveats and risks entailed by the method.

Constrained temporal parallel decomposition for EEG-fMRI fusion

OBJECTIVE

Multimodal neuroimaging has become a common practice in neuroscience research. Simultaneous EEG-fMRI is a popular multimodal recording approach due to the complementary spatiotemporal relationship between the two modalities. Several data fusion techniques have been proposed in the literature for EEG-fMRI fusion, including joint-ICA and parallel-ICA frameworks. Previous EEG-fMRI fusion approaches have used sensor-level EEG features. Recently, we introduced source-space ICA for EEG-MEG source reconstruction and component identification, which was shown to be a superior alternative to sensor-space ICA.

APPROACH

Here, we extend source-space ICA to the fusion of EEG-fMRI data. Additionally, we incorporate the use of a paradigm signal (constrained) and a lag-based signal decomposition approach to accommodate recent findings demonstrating the potentially variable lag structure between electrophysiological and BOLD signals. We evaluated this method on simulated concurrent EEG-fMRI during a boxcar task design, as well as real concurrent EEG-fMRI data from three participants performing an N-Back working memory task. The block diagram of the algorithm and corresponding source codes are provided.

MAIN RESULTS

Based on the results of the real working memory task, for all three subjects, one frontal theta component, and one right posterior alpha component had the highest contribution coefficients (~0.5) to the paradigm-related fused component. There were also two more alpha band components with contribution coefficients of 0.3. The highest contributing fMRI component (~0.8) was one known in the literature to be related to the attention network. The second fMRI component was related to the well-known default mode network, with a contribution coefficient of 0.3.

SIGNIFICANCE

The proposed EEG-fMRI fusion approach, is capable of estimating the brain maps of the EEG and fMRI for the fused components and account for the variable lag structure between electrophysiological and BOLD signals.

Position-independent gesture recognition using sEMG signals via canonical correlation analysis

Gesture recognition based on surface electromyogram (sEMG) signals has drawn significant attention and obtained satisfactory achievement in the field of human-computer interaction. However, the same gesture performed with different arm positions tends not to generate the same sEMG signals. Traditional solutions can be divided into two types. One type treats the same gesture with different arm positions as the same type, leading to a relatively low classification rate. The other type adopts a gesture classifier followed by the position classifier, which will achieve a satisfactory classification accuracy but at the expenses of high training burdens. To address these issues, we propose a novel framework to explore the intrinsic position independent (PI) characteristics of sEMG signals generated from the same gesture with different arm positions by canonical correlation analysis (CCA), termed as PICCA. In this framework, with the bridge link of the predefined expert set, both the training set and the testing set can be mapped into a unified-style with CCA, and hence, the classification accuracy can be improved in both user-dependent and user-independent manners. Experimental results on 13 gestures with 3 arm positions indicate that the proposed PICCA can achieve higher classification rates than those without CCA (with 28.52% and 44.19% promotions during user-dependent and user-independent manners respectively) while maintaining acceptable training burdens. These improvements will facilitate the practical implementation of myoelectric interfaces.

Movement Symmetry Assessment by Bilateral Motion Data Fusion

OBJECTIVE

A new approach, named bilateral motion data fusion, was proposed for the analysis of movement symmetry, which takes advantage of cross-information between both sides of the body and processes the unilateral motion data at the same time.

METHODS

This was accomplished using canonical correlation analysis and joint independent component analysis. It should be noted that human movements include many categories, which cannot be enumerated one by one. Therefore, the gait rhythm fluctuations of the healthy subjects and patients with neurodegenerative diseases were employed as an example for method illustration. In addition, our model explains the movement data by latent parameters in the time and frequency domains, respectively, which were both based on bilateral motion data fusion.

RESULTS

They show that our method not only reflects the physiological correlates of movement but also obtains the differential signatures of movement asymmetry in diverse neurodegenerative diseases. Furthermore, the latent variables also exhibit the potentials for sharper disease distinctions.

CONCLUSION

We have provided a new perspective on movement analysis, which may prove to be a promising approach.

SIGNIFICANCE

This method exhibits the potentials for effective movement feature extractions, which might contribute to many research fields such as rehabilitation, neuroscience, biomechanics, and kinesiology.

Discriminative Multiple Canonical Correlation Analysis for Information Fusion

In this paper, we propose the discriminative multiple canonical correlation analysis (DMCCA) for multimodal information analysis and fusion. DMCCA is capable of extracting more discriminative characteristics from multimodal information representations. Specifically, it finds the projected directions, which simultaneously maximize the within-class correlation and minimize the between-class correlation, leading to better utilization of the multimodal information. In the process, we analytically demonstrate that the optimally projected dimension by DMCCA can be quite accurately predicted, leading to both superior performance and substantial reduction in computational cost. We further verify that canonical correlation analysis (CCA), multiple canonical correlation analysis (MCCA) and discriminative canonical correlation analysis (DCCA) are special cases of DMCCA, thus establishing a unified framework for canonical correlation analysis. We implement a prototype of DMCCA to demonstrate its performance in handwritten digit recognition and human emotion recognition. Extensive experiments show that DMCCA outperforms the traditional methods of serial fusion, CCA, MCCA, and DCCA.

Combining multi-modality data for searching biomarkers in schizophrenia

Identification of imaging biomarkers for schizophrenia is an important but still challenging problem. Even though considerable efforts have been made over the past decades, quantitative alterations between patients and healthy subjects have not yet provided a diagnostic measure with sufficient high sensitivity and specificity. One of the most important reasons is the lack of consistent findings, which is in part due to single-mode study, which only detects single dimensional information by each modality, and thus misses the most crucial differences between groups. Here, we hypothesize that multimodal integration of functional MRI (fMRI), structural MRI (sMRI), and diffusion tensor imaging (DTI) might yield more power for the diagnosis of schizophrenia. A novel multivariate data fusion method for combining these modalities is introduced without reducing the dimension or using the priors from 161 schizophrenia patients and 168 matched healthy controls. The multi-index feature for each ROI is constructed and summarized with Wilk's lambda by performing multivariate analysis of variance to calculate the significant difference between different groups. Our results show that, among these modalities, fMRI has the most significant featureby calculating the Jaccard similarity coefficient (0.7416) and Kappa index (0.4833). Furthermore, fusion of these modalities provides the most plentiful information and the highest predictive accuracy of 86.52%. This work indicates that multimodal integration can improve the ability of distinguishing differences between groups and might be assisting in further diagnosis of schizophrenia.

Optimizing Within-Subject Experimental Designs for jICA of Multi-Channel ERP and fMRI

Joint independent component analysis (jICA) can be applied within subject for fusion of multi-channel event-related potentials (ERP) and functional magnetic resonance imaging (fMRI), to measure brain function at high spatiotemporal resolution (Mangalathu-Arumana et al., 2012). However, the impact of experimental design choices on jICA performance has not been systematically studied. Here, the sensitivity of jICA for recovering neural sources in individual data was evaluated as a function of imaging SNR, number of independent representations of the ERP/fMRI data, relationship between instantiations of the joint ERP/fMRI activity (linear, non-linear, uncoupled), and type of sources (varying parametrically and non-parametrically across representations of the data), using computer simulations. Neural sources were simulated with spatiotemporal and noise attributes derived from experimental data. The best performance, maximizing both cross-modal data fusion and the separation of brain sources, occurred with a moderate number of representations of the ERP/fMRI data (10-30), as in a mixed block/event related experimental design. Importantly, the type of relationship between instantiations of the ERP/fMRI activity, whether linear, non-linear or uncoupled, did not in itself impact jICA performance, and was accurately recovered in the common profiles (i.e., mixing coefficients). Thus, jICA provides an unbiased way to characterize the relationship between ERP and fMRI activity across brain regions, in individual data, rendering it potentially useful for characterizing pathological conditions in which neurovascular coupling is adversely affected.

Schizophrenia Shows Disrupted Links between Brain Volume and Dynamic Functional Connectivity

Studies featuring multimodal neuroimaging data fusion for understanding brain function and structure, or disease characterization, leverage the partial information available in each of the modalities to reveal data variations not exhibited through the independent analyses. Similar to other complex syndromes, the characteristic brain abnormalities in schizophrenia may be better understood with the help of the additional information conveyed by leveraging an advanced modeling method involving multiple modalities. In this study, we propose a novel framework to fuse feature spaces corresponding to functional magnetic resonance imaging (functional) and gray matter (structural) data from 151 schizophrenia patients and 163 healthy controls. In particular, the features for the functional and structural modalities include dynamic (i.e., time-varying) functional network connectivity (dFNC) maps and the intensities of the gray matter (GM) maps, respectively. The dFNC maps are estimated from group independent component analysis (ICA) network time-courses by first computing windowed functional correlations using a sliding window approach, and then estimating subject specific states from this windowed data using temporal ICA followed by spatio-temporal regression. For each subject, the functional data features are horizontally concatenated with the corresponding GM features to form a combined feature space that is subsequently decomposed through a symmetric multimodal fusion approach involving a combination of multiset canonical correlation analysis (mCCA) and joint ICA (jICA). Our novel combined analyses successfully linked changes in the two modalities and revealed significantly disrupted links between GM volumes and time-varying functional connectivity in schizophrenia. Consistent with prior research, we found significant group differences in GM comprising regions in the superior parietal lobule, precuneus, postcentral gyrus, medial/superior frontal gyrus, superior/middle temporal gyrus, insula and fusiform gyrus, and several significant aberrations in the inter-regional functional connectivity strength as well. Importantly, structural and dFNC measures have independently shown changes associated with schizophrenia, and in this work we begin the process of evaluating the links between the two, which could shed light on the illness beyond what we can learn from a single imaging modality. In future work, we plan to evaluate replication of the inferred structure-function relationships in independent partitions of larger multi-modal schizophrenia datasets.

Multimodal neural correlates of cognitive control in the Human Connectome Project

Cognitive control is a construct that refers to the set of functions that enable decision-making and task performance through the representation of task states, goals, and rules. The neural correlates of cognitive control have been studied in humans using a wide variety of neuroimaging modalities, including structural MRI, resting-state fMRI, and task-based fMRI. The results from each of these modalities independently have implicated the involvement of a number of brain regions in cognitive control, including dorsal prefrontal cortex, and frontal parietal and cingulo-opercular brain networks. However, it is not clear how the results from a single modality relate to results in other modalities. Recent developments in multimodal image analysis methods provide an avenue for answering such questions and could yield more integrated models of the neural correlates of cognitive control. In this study, we used multiset canonical correlation analysis with joint independent component analysis (mCCA + jICA) to identify multimodal patterns of variation related to cognitive control. We used two independent cohorts of participants from the Human Connectome Project, each of which had data from four imaging modalities. We replicated the findings from the first cohort in the second cohort using both independent and predictive analyses. The independent analyses identified a component in each cohort that was highly similar to the other and significantly correlated with cognitive control performance. The replication by prediction analyses identified two independent components that were significantly correlated with cognitive control performance in the first cohort and significantly predictive of performance in the second cohort. These components identified positive relationships across the modalities in neural regions related to both dynamic and stable aspects of task control, including regions in both the frontal-parietal and cingulo-opercular networks, as well as regions hypothesized to be modulated by cognitive control signaling, such as visual cortex. Taken together, these results illustrate the potential utility of multi-modal analyses in identifying the neural correlates of cognitive control across different indicators of brain structure and function.

Quantifying the Interaction and Contribution of Multiple Datasets in Fusion: Application to the Detection of Schizophrenia

The extraction of information from multiple sets of data is a problem inherent to many disciplines. This is possible by either analyzing the data sets jointly as in data fusion or separately and then combining as in data integration. However, selecting the optimal method to combine and analyze multiset data is an ever-present challenge. The primary reason for this is the difficulty in determining the optimal contribution of each data set to an analysis as well as the amount of potentially exploitable complementary information among data sets. In this paper, we propose a novel classification rate-based technique to unambiguously quantify the contribution of each data set to a fusion result as well as facilitate direct comparisons of fusion methods on real data and apply a new method, independent vector analysis (IVA), to multiset fusion. This classification rate-based technique is used on functional magnetic resonance imaging data collected from 121 patients with schizophrenia and 150 healthy controls during the performance of three tasks. Through this application, we find that though optimal performance is achieved by exploiting all tasks, each task does not contribute equally to the result and this framework enables effective quantification of the value added by each task. Our results also demonstrate that data fusion methods are more powerful than data integration methods, with the former achieving a classification rate of 73.5 % and the latter achieving one of 70.9 %, a difference which we show is significant when all three tasks are analyzed together. Finally, we show that IVA, due to its flexibility, has equivalent or superior performance compared with the popular data fusion method, joint independent component analysis.

Structured and Sparse Canonical Correlation Analysis as a Brain-Wide Multi-Modal Data Fusion Approach

Multi-modal data fusion has recently emerged as a comprehensive neuroimaging analysis approach, which usually uses canonical correlation analysis (CCA). However, the current CCA-based fusion approaches face problems like high-dimensionality, multi-collinearity, unimodal feature selection, asymmetry, and loss of spatial information in reshaping the imaging data into vectors. This paper proposes a structured and sparse CCA (ssCCA) technique as a novel CCA method to overcome the above problems. To investigate the performance of the proposed algorithm, we have compared three data fusion techniques: standard CCA, regularized CCA, and ssCCA, and evaluated their ability to detect multi-modal data associations. We have used simulations to compare the performance of these approaches and probe the effects of non-negativity constraint, the dimensionality of features, sample size, and noise power. The results demonstrate that ssCCA outperforms the existing standard and regularized CCA-based fusion approaches. We have also applied the methods to real functional magnetic resonance imaging (fMRI) and structural MRI data of Alzheimer's disease (AD) patients (n = 34) and healthy control (HC) subjects (n = 42) from the ADNI database. The results illustrate that the proposed unsupervised technique differentiates the transition pattern between the subject-course of AD patients and HC subjects with a p-value of less than 1×10^-6 . Furthermore, we have depicted the brain mapping of functional areas that are most correlated with the anatomical changes in AD patients relative to HC subjects.

Inter-subject alignment of MEG datasets in a common representational space

Pooling neural imaging data across subjects requires aligning recordings from different subjects. In magnetoencephalography (MEG) recordings, sensors across subjects are poorly correlated both because of differences in the exact location of the sensors, and structural and functional differences in the brains. It is possible to achieve alignment by assuming that the same regions of different brains correspond across subjects. However, this relies on both the assumption that brain anatomy and function are well correlated, and the strong assumptions that go into solving the under-determined inverse problem given the high-dimensional source space. In this article, we investigated an alternative method that bypasses source-localization. Instead, it analyzes the sensor recordings themselves and aligns their temporal signatures across subjects. We used a multivariate approach, multiset canonical correlation analysis (M-CCA), to transform individual subject data to a low-dimensional common representational space. We evaluated the robustness of this approach over a synthetic dataset, by examining the effect of different factors that add to the noise and individual differences in the data. On an MEG dataset, we demonstrated that M-CCA performs better than a method that assumes perfect sensor correspondence and a method that applies source localization. Last, we described how the standard M-CCA algorithm could be further improved with a regularization term that incorporates spatial sensor information. Hum Brain Mapp 38:4287-4301, 2017. © 2017 Wiley Periodicals, Inc.

Assessment of mental stress effects on prefrontal cortical activities using canonical correlation analysis: an fNIRS-EEG study

This paper presents an investigation about the effects of mental stress on prefrontal cortex (PFC) subregions using simultaneous measurement of functional Near-Infrared Spectroscopy (fNIRS) and Electroencephalography (EEG) signals. The aim is to explore canonical correlation analysis (CCA) technique to study the relationship among the bi-modality signals in mental stress assessment, and how we could fuse the signals for better accuracy in stress detection. Twenty-five male healthy subjects participated in the study while performing mental arithmetic task under control and stress (under time pressure with negative feedback) conditions. The fusion of brain signals acquired by fNIRS-EEG was performed at feature-level using CCA by maximizing the inter-subject covariance across modalities. The CCA result discovered the associations across the modalities and estimated the components responsible for these associations. The experiment results showed that mental stress experienced by this cohort of subjects is subregion specific and localized to the right ventrolateral PFC subregion. These suggest the right ventrolateral PFC as a suitable candidate region to extract biomarkers as performance indicators of neurofeedback training in stress coping.