Using deep learning to investigate the neuroimaging correlates of psychiatric and neurological disorders: Methods and applications

A Long Short-Term Memory neural network for the detection of epileptiform spikes and high frequency oscillations

Over the last two decades, the evidence has been growing that in addition to epileptic spikes high frequency oscillations (HFOs) are important biomarkers of epileptogenic tissue. New methods of artificial intelligence such as deep learning neural networks can provide additional tools for automated analysis of EEG. Here we present a Long Short-Term Memory neural network for detection of spikes, ripples and ripples-on-spikes (RonS). We used intracranial EEG (iEEG) from two independent datasets. First dataset (7 patients) was used for network training and testing. The second dataset (5 patients) was used for cross-institutional validation. 1000 events of each class (spike, RonS, ripple and baseline) were selected from the candidates initially found using a novel threshold method. Network training was performed using random selections of 50–500 events (per class) from all patients from the 1^st dataset. This ‘global’ network was then tested on other events for each patient from both datasets. The network was able to detect events with a good generalisability namely, with total accuracy and specificity for each class exceeding 90% in all cases, and sensitivity less than 86% in only two cases (82.5% for spikes in one patient and 81.9% for ripples in another patient). The deep learning networks can significantly accelerate the analysis of iEEG data and increase their diagnostic value which may improve surgical outcome in patients with localization-related intractable epilepsy.

Using Deep Learning and Resting-State fMRI to Classify Chronic Pain Conditions

Chronic pain is known as a complex disease due to its comorbidities with other symptoms and the lack of effective treatments. As a consequence, chronic pain seems to be under-diagnosed in more than 75% of patients. At the same time, the advance in brain imaging, the popularization of machine learning techniques and the development of new diagnostic tools based on these technologies have shown that these tools could be an option in supporting decision-making of healthcare professionals. In this study, we computed functional brain connectivity using resting-state fMRI data from one hundred and fifty participants to assess the performance of different machine learning models, including deep learning (DL) neural networks in classifying chronic pain patients and pain-free controls. The best result was obtained by training a convolutional neural network fed with data preprocessed using the MSDL probabilistic atlas and using the dynamic time warping (DTW) as connectivity measure. DL models had a better performance compared to other less costly models such as support vector machine (SVM) and RFC, with balanced accuracy ranged from 69 to 86%, while the area under the curve (ROC) ranged from 0.84 to 0.93. Also, DTW overperformed correlation as connectivity measure. These findings support the notion that resting-state fMRI data could be used as a potential biomarker of chronic pain conditions.

Classification and Visualization of Alzheimer's Disease using Volumetric Convolutional Neural Network and Transfer Learning

Recently, deep-learning-based approaches have been proposed for the classification of neuroimaging data related to Alzheimer's disease (AD), and significant progress has been made. However, end-to-end learning that is capable of maximizing the impact of deep learning has yet to receive much attention due to the endemic challenge of neuroimaging caused by the scarcity of data. Thus, this study presents an approach meant to encourage the end-to-end learning of a volumetric convolutional neural network (CNN) model for four binary classification tasks (AD vs. normal control (NC), progressive mild cognitive impairment (pMCI) vs. NC, stable mild cognitive impairment (sMCI) vs. NC and pMCI vs. sMCI) based on magnetic resonance imaging (MRI) and visualizes its outcomes in terms of the decision of the CNNs without any human intervention. In the proposed approach, we use convolutional autoencoder (CAE)-based unsupervised learning for the AD vs. NC classification task, and supervised transfer learning is applied to solve the pMCI vs. sMCI classification task. To detect the most important biomarkers related to AD and pMCI, a gradient-based visualization method that approximates the spatial influence of the CNN model's decision was applied. To validate the contributions of this study, we conducted experiments on the ADNI database, and the results demonstrated that the proposed approach achieved the accuracies of 86.60% and 73.95% for the AD and pMCI classification tasks respectively, outperforming other network models. In the visualization results, the temporal and parietal lobes were identified as key regions for classification.

Machine Learning With Neuroimaging: Evaluating Its Applications in Psychiatry

Psychiatric disorders are complex, involving heterogeneous symptomatology and neurobiology that rarely involves the disruption of single, isolated brain structures. In an attempt to better describe and understand the complexities of psychiatric disorders, investigators have increasingly applied multivariate pattern classification approaches to neuroimaging data and in particular supervised machine learning methods. However, supervised machine learning approaches also come with unique challenges and trade-offs, requiring additional study design and interpretation considerations. The goal of this review is to provide a set of best practices for evaluating machine learning applications to psychiatric disorders. We discuss how to evaluate two common efforts: 1) making predictions that have the potential to aid in diagnosis, prognosis, and treatment and 2) interrogating the complex neurophysiological mechanisms underlying psychopathology. We focus here on machine learning as applied to functional connectivity with magnetic resonance imaging, as an example to ground discussion. We argue that for machine learning classification to have translational utility for individual-level predictions, investigators must ensure that the classification is clinically informative, independent of confounding variables, and appropriately assessed for both performance and generalizability. We contend that shedding light on the complex mechanisms underlying psychiatric disorders will require consideration of the unique utility, interpretability, and reliability of the neuroimaging features (e.g., regions, networks, connections) identified from machine learning approaches. Finally, we discuss how the rise of large, multisite, publicly available datasets may contribute to the utility of machine learning approaches in psychiatry.

Integrating machining learning and multimodal neuroimaging to detect schizophrenia at the level of the individual

Schizophrenia is a severe psychiatric disorder associated with both structural and functional brain abnormalities. In the past few years, there has been growing interest in the application of machine learning techniques to neuroimaging data for the diagnostic and prognostic assessment of this disorder. However, the vast majority of studies published so far have used either structural or functional neuroimaging data, without accounting for the multimodal nature of the disorder. Structural MRI and resting‐state functional MRI data were acquired from a total of 295 patients with schizophrenia and 452 healthy controls at five research centers. We extracted features from the data including gray matter volume, white matter volume, amplitude of low‐frequency fluctuation, regional homogeneity and two connectome‐wide based metrics: structural covariance matrices and functional connectivity matrices. A support vector machine classifier was trained on each dataset separately to distinguish the subjects at individual level using each of the single feature as well as their combination, and 10‐fold cross‐validation was used to assess the performance of the model. Functional data allow higher accuracy of classification than structural data (mean 82.75% vs. 75.84%). Within each modality, the combination of images and matrices improves performance, resulting in mean accuracies of 81.63% for structural data and 87.59% for functional data. The use of all combined structural and functional measures allows the highest accuracy of classification (90.83%). We conclude that combining multimodal measures within a single model is a promising direction for developing biologically informed diagnostic tools in schizophrenia.

The Neurodevelopment of Autism from Infancy Through Toddlerhood

Autism spectrum disorder (ASD) emerges during early childhood and is marked by a relatively narrow window in which infants transition from exhibiting normative behavioral profiles to displaying the defining features of the ASD phenotype in toddlerhood. Prospective brain imaging studies in infants at high familial risk for autism have revealed important insights into the neurobiology and developmental unfolding of ASD. In this article, we review neuroimaging studies of brain development in ASD from birth through toddlerhood, relate these findings to candidate neurobiological mechanisms, and discuss implications for future research and translation to clinical practice.

Deep neural networks in psychiatry

Machine and deep learning methods, today's core of artificial intelligence, have been applied with increasing success and impact in many commercial and research settings. They are powerful tools for large scale data analysis, prediction and classification, especially in very data-rich environments ("big data"), and have started to find their way into medical applications. Here we will first give an overview of machine learning methods, with a focus on deep and recurrent neural networks, their relation to statistics, and the core principles behind them. We will then discuss and review directions along which (deep) neural networks can be, or already have been, applied in the context of psychiatry, and will try to delineate their future potential in this area. We will also comment on an emerging area that so far has been much less well explored: by embedding semantically interpretable computational models of brain dynamics or behavior into a statistical machine learning context, insights into dysfunction beyond mere prediction and classification may be gained. Especially this marriage of computational models with statistical inference may offer insights into neural and behavioral mechanisms that could open completely novel avenues for psychiatric treatment.

Clinical-learning versus machine-learning for transdiagnostic prediction of psychosis onset in individuals at-risk

Predicting the onset of psychosis in individuals at-risk is based on robust prognostic model building methods including a priori clinical knowledge (also termed clinical-learning) to preselect predictors or machine-learning methods to select predictors automatically. To date, there is no empirical research comparing the prognostic accuracy of these two methods for the prediction of psychosis onset. In a first experiment, no improved performance was observed when machine-learning methods (LASSO and RIDGE) were applied-using the same predictors-to an individualised, transdiagnostic, clinically based, risk calculator previously developed on the basis of clinical-learning (predictors: age, gender, age by gender, ethnicity, ICD-10 diagnostic spectrum), and externally validated twice. In a second experiment, two refined versions of the published model which expanded the granularity of the ICD-10 diagnosis were introduced: ICD-10 diagnostic categories and ICD-10 diagnostic subdivisions. Although these refined versions showed an increase in apparent performance, their external performance was similar to the original model. In a third experiment, the three refined models were analysed under machine-learning and clinical-learning with a variable event per variable ratio (EPV). The best performing model under low EPVs was obtained through machine-learning approaches. The development of prognostic models on the basis of a priori clinical knowledge, large samples and adequate events per variable is a robust clinical prediction method to forecast psychosis onset in patients at-risk, and is comparable to machine-learning methods, which are more difficult to interpret and implement. Machine-learning methods should be preferred for high dimensional data when no a priori knowledge is available.

Novel relative relevance score for estimating brain connectivity from fMRI data using an explainable neural network approach

BACKGROUND

Functional integration or connectivity in brain is directional, non-linear as well as variable in time-lagged dependence. Deep neural networks (DNN) have become an indispensable tool everywhere, by learning higher levels of abstract and complex patterns from raw data. However, in neuroscientific community they generally work as black-boxes, leading to the explanation of results difficult and less intuitive. We aim to propose a brain-connectivity measure based on an explainable NN (xNN) approach.

NEW METHOD

We build a NN-based predictor for regression problem. Since we aim to determine the contribution/relevance of past data-point from one region i in the prediction of current data-point from another region j, i.e. the higher-order connectivity between two brain-regions, we employ layer-wise relevance propagation (Bach et al., 2015) (LRP, a method for explaining DNN predictions), which has not been done before to the best of our knowledge. Specifically, we propose a novel score depending on weights as a quantitative measure of connectivity, called as relative relevance score (xNN-RRS). The RRS is an intuitive and transparent score. We provide an interpretation of the trained NN weights with-respect-to the brain-connectivity.

RESULTS

Face validity of our approach is demonstrated with experiments on simulated data, over existing methods. We also demonstrate construct validity of xNN-RRS in a resting-state fMRI experiment.

COMPARISON

Our approach shows superior performance, in terms of accuracy and computational complexity, over existing state-of-the-art methods for brain-connectivity estimation.

CONCLUSION

The proposed method is promising to serve as a first post-hoc explainable NN-approach for brain-connectivity analysis in clinical applications.

Three dimensional convolutional neural network-based classification of conduct disorder with structural MRI

Conduct disorder (CD) is a common child and adolescent psychiatric disorder with various representative symptoms, and may cause long-term burden to patients and society. Recently, an increasing number of studies have used deep learning-based approaches, such as convolutional neural network (CNN), to analyze neuroimaging data and to identify biomarkers. In this study, we applied an optimized 3D AlexNet CNN model to automatically extract multi-layer high dimensional features of structural magnetic resonance imaging (sMRI), and to classify CD from healthy controls (HCs). We acquired high-resolution sMRI from 60 CD and 60 age- and gender-matched HCs. All subjects were male, and the age (mean ± std. dev) of participants in the CD and HC groups was 15.3 ± 1.0 and 15.5 ± 0.7, respectively. Five-fold cross validation (CV) was used to train and test this model. The receiver operating characteristic (ROC) curve for this model and that for support vector machine (SVM) model were compared. Feature visualization was performed to obtain intuition about the sMRI features learned by our AlexNet model. Our proposed AlexNet model achieved high classification performance with accuracy of 0.85, specificity of 0.82 and sensitivity of 0.87. The area under the ROC curve (AUC) of AlexNet was 0.86, significantly higher than that of SVM (AUC = 0.78; p = 0.046). The saliency maps for each convolutional layer highlighted the different brain regions in sMRI of CD, mainly including the frontal lobe, superior temporal gyrus, parietal lobe and occipital lobe. The classification results indicated that deep learning-based method is able to explore the hidden features from the sMRI of CD and might assist clinicians in the diagnosis of CD.

Uncovering convolutional neural network decisions for diagnosing multiple sclerosis on conventional MRI using layer-wise relevance propagation

Machine learning-based imaging diagnostics has recently reached or even surpassed the level of clinical experts in several clinical domains. However, classification decisions of a trained machine learning system are typically non-transparent, a major hindrance for clinical integration, error tracking or knowledge discovery. In this study, we present a transparent deep learning framework relying on 3D convolutional neural networks (CNNs) and layer-wise relevance propagation (LRP) for diagnosing multiple sclerosis (MS), the most widespread autoimmune neuroinflammatory disease. MS is commonly diagnosed utilizing a combination of clinical presentation and conventional magnetic resonance imaging (MRI), specifically the occurrence and presentation of white matter lesions in T2-weighted images. We hypothesized that using LRP in a naive predictive model would enable us to uncover relevant image features that a trained CNN uses for decision-making. Since imaging markers in MS are well-established this would enable us to validate the respective CNN model. First, we pre-trained a CNN on MRI data from the Alzheimer's Disease Neuroimaging Initiative (n = 921), afterwards specializing the CNN to discriminate between MS patients (n = 76) and healthy controls (n = 71). Using LRP, we then produced a heatmap for each subject in the holdout set depicting the voxel-wise relevance for a particular classification decision. The resulting CNN model resulted in a balanced accuracy of 87.04% and an area under the curve of 96.08% in a receiver operating characteristic curve. The subsequent LRP visualization revealed that the CNN model focuses indeed on individual lesions, but also incorporates additional information such as lesion location, non-lesional white matter or gray matter areas such as the thalamus, which are established conventional and advanced MRI markers in MS. We conclude that LRP and the proposed framework have the capability to make diagnostic decisions of CNN models transparent, which could serve to justify classification decisions for clinical review, verify diagnosis-relevant features and potentially gather new disease knowledge.

Progress in psychoradiology, the clinical application of psychiatric neuroimaging

Psychoradiology is an emerging field that applies radiological imaging technologies to psychiatric conditions. In the past three decades, brain imaging techniques have rapidly advanced understanding of illness and treatment effects in psychiatry. Based on these advances, radiologists have become increasingly interested in applying these advances for differential diagnosis and individualized patient care selection for common psychiatric illnesses. This shift from research to clinical practice represents the beginning evolution of psychoradiology. In this review, we provide a summary of recent progress relevant to this field based on their clinical functions, namely the (1) classification and subtyping; (2) prediction and monitoring of treatment outcomes; and (3) treatment selection. In addition, we provide guidelines for the practice of psychoradiology in clinical settings and suggestions for future research to validate broader clinical applications. Given the high prevalence of psychiatric disorders and the importance of increased participation of radiologists in this field, a guide regarding advances in this field and a description of relevant clinical work flow patterns help radiologists contribute to this fast-evolving field.

Convolutional Neural Network for Automated FLAIR Lesion Segmentation on Clinical Brain MR Imaging

BACKGROUND AND PURPOSE

Most brain lesions are characterized by hyperintense signal on FLAIR. We sought to develop an automated deep learning-based method for segmentation of abnormalities on FLAIR and volumetric quantification on clinical brain MRIs across many pathologic entities and scanning parameters. We evaluated the performance of the algorithm compared with manual segmentation and existing automated methods.

MATERIALS AND METHODS

We adapted a U-Net convolutional neural network architecture for brain MRIs using 3D volumes. This network was retrospectively trained on 295 brain MRIs to perform automated FLAIR lesion segmentation. Performance was evaluated on 92 validation cases using Dice scores and voxelwise sensitivity and specificity, compared with radiologists' manual segmentations. The algorithm was also evaluated on measuring total lesion volume.

RESULTS

Our model demonstrated accurate FLAIR lesion segmentation performance (median Dice score, 0.79) on the validation dataset across a large range of lesion characteristics. Across 19 neurologic diseases, performance was significantly higher than existing methods (Dice, 0.56 and 0.41) and approached human performance (Dice, 0.81). There was a strong correlation between the predictions of lesion volume of the algorithm compared with true lesion volume (ρ = 0.99). Lesion segmentations were accurate across a large range of image-acquisition parameters on >30 different MR imaging scanners.

CONCLUSIONS

A 3D convolutional neural network adapted from a U-Net architecture can achieve high automated FLAIR segmentation performance on clinical brain MR imaging across a variety of underlying pathologies and image acquisition parameters. The method provides accurate volumetric lesion data that can be incorporated into assessments of disease burden or into radiologic reports.

A Comparison of Shallow and Deep Learning Methods for Predicting Cognitive Performance of Stroke Patients From MRI Lesion Images

Stroke causes behavioral deficits in multiple cognitive domains and there is a growing interest in predicting patient performance from neuroimaging data using machine learning techniques. Here, we investigated a deep learning approach based on convolutional neural networks (CNNs) for predicting the severity of language disorder from 3D lesion images from magnetic resonance imaging (MRI) in a heterogeneous sample of stroke patients. CNN performance was compared to that of conventional (shallow) machine learning methods, including ridge regression (RR) on the images' principal components and support vector regression. We also devised a hybrid method based on re-using CNN's high-level features as additional input to the RR model. Predictive accuracy of the four different methods was further investigated in relation to the size of the training set and the level of redundancy across lesion images in the dataset, which was evaluated in terms of location and topological properties of the lesions. The Hybrid model achieved the best performance in most cases, thereby suggesting that the high-level features extracted by CNNs are complementary to principal component analysis features and improve the model's predictive accuracy. Moreover, our analyses indicate that both the size of training data and image redundancy are critical factors in determining the accuracy of a computational model in predicting behavioral outcome from the structural brain imaging data of stroke patients.

Layer-Wise Relevance Propagation for Explaining Deep Neural Network Decisions in MRI-Based Alzheimer's Disease Classification

Deep neural networks have led to state-of-the-art results in many medical imaging tasks including Alzheimer's disease (AD) detection based on structural magnetic resonance imaging (MRI) data. However, the network decisions are often perceived as being highly non-transparent, making it difficult to apply these algorithms in clinical routine. In this study, we propose using layer-wise relevance propagation (LRP) to visualize convolutional neural network decisions for AD based on MRI data. Similarly to other visualization methods, LRP produces a heatmap in the input space indicating the importance/relevance of each voxel contributing to the final classification outcome. In contrast to susceptibility maps produced by guided backpropagation ("Which change in voxels would change the outcome most?"), the LRP method is able to directly highlight positive contributions to the network classification in the input space. In particular, we show that (1) the LRP method is very specific for individuals ("Why does this person have AD?") with high inter-patient variability, (2) there is very little relevance for AD in healthy controls and (3) areas that exhibit a lot of relevance correlate well with what is known from literature. To quantify the latter, we compute size-corrected metrics of the summed relevance per brain area, e.g., relevance density or relevance gain. Although these metrics produce very individual "fingerprints" of relevance patterns for AD patients, a lot of importance is put on areas in the temporal lobe including the hippocampus. After discussing several limitations such as sensitivity toward the underlying model and computation parameters, we conclude that LRP might have a high potential to assist clinicians in explaining neural network decisions for diagnosing AD (and potentially other diseases) based on structural MRI data.

Artificial intelligence for precision education in radiology

In the era of personalized medicine, the emphasis of health care is shifting from populations to individuals. Artificial intelligence (AI) is capable of learning without explicit instruction and has emerging applications in medicine, particularly radiology. Whereas much attention has focused on teaching radiology trainees about AI, here our goal is to instead focus on how AI might be developed to better teach radiology trainees. While the idea of using AI to improve education is not new, the application of AI to medical and radiological education remains very limited. Based on the current educational foundation, we highlight an AI-integrated framework to augment radiology education and provide use case examples informed by our own institution's practice. The coming age of "AI-augmented radiology" may enable not only "precision medicine" but also what we describe as "precision medical education," where instruction is tailored to individual trainees based on their learning styles and needs.

Cortical graph neural network for AD and MCI diagnosis and transfer learning across populations

Combining machine learning with neuroimaging data has a great potential for early diagnosis of mild cognitive impairment (MCI) and Alzheimer's disease (AD). However, it remains unclear how well the classifiers built on one population can predict MCI/AD diagnosis of other populations. This study aimed to employ a spectral graph convolutional neural network (graph-CNN), that incorporated cortical thickness and geometry, to identify MCI and AD based on 3089 T1-weighted MRI data of the ADNI-2 cohort, and to evaluate its feasibility to predict AD in the ADNI-1 cohort (n = 3602) and an Asian cohort (n = 347). For the ADNI-2 cohort, the graph-CNN showed classification accuracy of controls (CN) vs. AD at 85.8% and early MCI (EMCI) vs. AD at 79.2%, followed by CN vs. late MCI (LMCI) (69.3%), LMCI vs. AD (65.2%), EMCI vs. LMCI (60.9%), and CN vs. EMCI (51.8%). We demonstrated the robustness of the graph-CNN among the existing deep learning approaches, such as Euclidean-domain-based multilayer network and 1D CNN on cortical thickness, and 2D and 3D CNNs on T1-weighted MR images of the ADNI-2 cohort. The graph-CNN also achieved the prediction on the conversion of EMCI to AD at 75% and that of LMCI to AD at 92%. The find-tuned graph-CNN further provided a promising CN vs. AD classification accuracy of 89.4% on the ADNI-1 cohort and >90% on the Asian cohort. Our study demonstrated the feasibility to transfer AD/MCI classifiers learned from one population to the other. Notably, incorporating cortical geometry in CNN has the potential to improve classification performance.

Studying the Manifold Structure of Alzheimer's Disease: A Deep Learning Approach Using Convolutional Autoencoders

Many classical machine learning techniques have been used to explore Alzheimer's disease (AD), evolving from image decomposition techniques such as principal component analysis toward higher complexity, non-linear decomposition algorithms. With the arrival of the deep learning paradigm, it has become possible to extract high-level abstract features directly from MRI images that internally describe the distribution of data in low-dimensional manifolds. In this work, we try a new exploratory data analysis of AD based on deep convolutional autoencoders. We aim at finding links between cognitive symptoms and the underlying neurodegeneration process by fusing the information of neuropsychological test outcomes, diagnoses, and other clinical data with the imaging features extracted solely via a data-driven decomposition of MRI. The distribution of the extracted features in different combinations is then analyzed and visualized using regression and classification analysis, and the influence of each coordinate of the autoencoder manifold over the brain is estimated. The imaging-derived markers could then predict clinical variables with correlations above 0.6 in the case of neuropsychological evaluation variables such as the MMSE or the ADAS11 scores, achieving a classification accuracy over 80% for the diagnosis of AD.

Detection of mild cognitive impairment in a community-dwelling population using quantitative, multiparametric MRI-based classification

Early and accurate mild cognitive impairment (MCI) detection within a heterogeneous, nonclinical population is needed to improve care for persons at risk of developing dementia. Magnetic resonance imaging (MRI)-based classification may aid early diagnosis of MCI, but has only been applied within clinical cohorts. We aimed to determine the generalizability of MRI-based classification probability scores to detect MCI on an individual basis within a general population. To determine classification probability scores, an AD, mild-AD, and moderate-AD detection model were created with anatomical and diffusion MRI measures calculated from a clinical Alzheimer's Disease (AD) cohort and subsequently applied to a population-based cohort with 48 MCI and 617 normal aging subjects. Each model's ability to detect MCI was quantified using area under the receiver operating characteristic curve (AUC) and compared with an MCI detection model trained and applied to the population-based cohort. The AD-model and mild-AD identified MCI from controls better than chance level (AUC = 0.600, p = 0.025; AUC = 0.619, p = 0.008). In contrast, the moderate-AD-model was not able to separate MCI from normal aging (AUC = 0.567, p = 0.147). The MCI-model was able to separate MCI from controls better than chance (p = 0.014) with mean AUC values comparable with the AD-model (AUC = 0.611, p = 1.0). Within our population-based cohort, classification models detected MCI better than chance. Nevertheless, classification performance rates were moderate and may be insufficient to facilitate robust MRI-based MCI detection on an individual basis. Our data indicate that multiparametric MRI-based classification algorithms, that are effective in clinical cohorts, may not straightforwardly translate to applications in a general population.

Localized Connectivity in Obsessive-Compulsive Disorder: An Investigation Combining Univariate and Multivariate Pattern Analyses

Recent developments in psychoradiological researches have highlighted the disrupted organization of large-scale functional brain networks in obsessive-compulsive disorder (OCD). However, whether abnormal activation of localized brain areas would affect network dysfunction remains to be fully characterized. We applied both univariate analysis and multivariate pattern analysis (MVPA) approaches to investigate the abnormalities of regional homogeneity (ReHo), an index to measure the localized connectivity, in 88 medication-free patients with OCD and 88 healthy control subjects (HCS). Resting-state functional magnetic resonance imaging (RS-fMRI) data of all the participants were acquired in a 3.0-T scanner. First, we adopted a traditional univariate analysis to explore ReHo alterations between the patient group and the control group. Subsequently, we utilized a support vector machine (SVM) to examine whether ReHo could be further used to differentiate patients with OCD from HCS at the individual level. Relative to HCS, OCD patients showed lower ReHo in the bilateral cerebellum and higher ReHo in the bilateral superior frontal gyri (SFG), right inferior parietal gyrus (IPG), and precuneus [P < 0.05, family-wise error (FWE) correction]. ReHo value in the left SFG positively correlated with Yale-Brown Obsessive Compulsive Scale (Y-BOCS) total score (r = 0 0.241, P = 0.024) and obsessive subscale (r = 0.224, P = 0.036). The SVM classification regarding ReHo yielded an accuracy of 78.98% (sensitivity = 78.41%, specificity = 79.55%) with P < 0.001 after permutation testing. The most discriminative regions contributing to the SVM classification were mainly located in the frontal, temporal, and parietal regions as well as in the cerebellum while the right orbital frontal cortex was identified with the highest discriminative power. Our findings not only suggested that the localized activation disequilibrium between the prefrontal cortex (PFC) and the cerebellum appeared to be associated with the pathophysiology of OCD but also indicated the translational role of the localized connectivity as a potential discriminative pattern to detect OCD at the individual level.

ASA: A framework for Arabic sentiment analysis

Sentiment analysis (SA), also known as opinion mining, is a growing important research area. Generally, it helps to automatically determine if a text expresses a positive, negative or neutral sentiment. It enables to mine the huge increasing resources of shared opinions such as social networks, review sites and blogs. In fact, SA is used by many fields and for various languages such as English and Arabic. However, since Arabic is a highly inflectional and derivational language, it raises many challenges. In fact, SA of Arabic text should handle such complex morphology. To better handle these challenges, we decided to provide the research community and Arabic users with a new efficient framework for Arabic Sentiment Analysis (ASA). Our primary goal is to improve the performance of ASA by exploiting deep learning while varying the preprocessing techniques. For that, we implement and evaluate two deep learning models namely convolutional neural network (CNN) and long short-term memory (LSTM) models. The framework offers various preprocessing techniques for ASA (including stemming, normalisation, tokenization and stop words). As a result of this work, we first provide a new rich and publicly available Arabic corpus called Moroccan Sentiment Analysis Corpus (MSAC). Second, the proposed framework demonstrates improvement in ASA. In fact, the experimental results prove that deep learning models have a better performance for ASA than classical approaches (support vector machines, naive Bayes classifiers and maximum entropy). They also show the key role of morphological features in Arabic Natural Language Processing (NLP).

Translational machine learning for psychiatric neuroimaging

Despite its initial promise, neuroimaging has not been widely translated into clinical psychiatry to assist in the prediction of diagnoses, prognoses, and optimal therapeutic strategies. Machine learning approaches may enhance the translational potential of neuroimaging because they specifically focus on overcoming biases by optimizing the generalizability of pipelines that measure complex brain patterns to predict targets at a single-subject level. This article introduces some fundamentals of a translational machine learning approach before selectively reviewing literature to-date. Promising initial results are then balanced by the description of limitations that should be considered in order to interpret existing research and maximize the possibility of future translation. Future directions are then presented in order to inspire further research and progress the field towards clinical translation.

The roles of supervised machine learning in systems neuroscience

Over the last several years, the use of machine learning (ML) in neuroscience has been rapidly increasing. Here, we review ML's contributions, both realized and potential, across several areas of systems neuroscience. We describe four primary roles of ML within neuroscience: (1) creating solutions to engineering problems, (2) identifying predictive variables, (3) setting benchmarks for simple models of the brain, and (4) serving itself as a model for the brain. The breadth and ease of its applicability suggests that machine learning should be in the toolbox of most systems neuroscientists.

White matter connectomes at birth accurately predict cognitive abilities at age 2

Cognitive ability is an important predictor of mental health outcomes that is influenced by neurodevelopment. Evidence suggests that the foundational wiring of the human brain is in place by birth, and that the white matter (WM) connectome supports developing brain function. It is unknown, however, how the WM connectome at birth supports emergent cognition. In this study, a deep learning model was trained using cross-validation to classify full-term infants (n = 75) as scoring above or below the median at age 2 using WM connectomes generated from diffusion weighted magnetic resonance images at birth. Results from this model were used to predict individual cognitive scores. We additionally identified WM connections important for classification. The model was also evaluated in a separate set of preterm infants (n = 37) scanned at term-age equivalent. Findings revealed that WM connectomes at birth predicted 2-year cognitive score group with high accuracy in both full-term (89.5%) and preterm (83.8%) infants. Scores predicted by the model were strongly correlated with actual scores (r = 0.98 for full-term and r = 0.96 for preterm). Connections within the frontal lobe, and between the frontal lobe and other brain areas were found to be important for classification. This work suggests that WM connectomes at birth can accurately predict a child's 2-year cognitive group and individual score in full-term and preterm infants. The WM connectome at birth appears to be a useful neuroimaging biomarker of subsequent cognitive development that deserves further study.

Alternative Diagnosis of Epilepsy in Children Without Epileptiform Discharges Using Deep Convolutional Neural Networks

Numerous nonepileptic paroxysmal events, such as syncope and psychogenic nonepileptic seizures, may imitate seizures and impede diagnosis. Misdiagnosis can lead to mistreatment, affecting patients’ lives considerably. Electroencephalography is commonly used for diagnosing epilepsy. Although on electroencephalograms (EEGs), epileptiform discharges (ED) specifically indicate epilepsy, only approximately 50% of patients with epilepsy have ED in their first EEG. In this study, we developed a deep convolutional neural network (ConvNet)-based classifier to distinguish EEG between patients with epilepsy without ED and controls. Overall, 25 patients with epilepsy without ED in their EEGs and 25 age-matched patients with Tourette syndrome or syncope were enrolled. Their EEGs were classified using the deep ConvNet. When the EEG data without overlapping were used, the accuracy, sensitivity, and specificity were 65.00%, 48.00%, and 82.00%, respectively. The performance measures improved when the input EEG data were augmented through overlapping. With 95% EEG data overlapping, the accuracy, sensitivity, and specificity increased to 80.00%, 70.00%, and 90.00%, respectively. The proposed method could be regarded as a pilot study to demonstrate a proof of concept of a potential diagnostic value of deep ConvNet in patients with epilepsy without ED. Further studies are needed to assist neurologists in distinguishing nonepileptic paroxysmal events from epilepsy.

Connectome-Based Biomarkers Predict Subclinical Depression and Identify Abnormal Brain Connections With the Lateral Habenula and Thalamus

Subclinical depression (SD) has been considered as the precursor to major depressive disorder. Accurate prediction of SD and identification of its etiological origin are urgent. Bursts within the lateral habenula (LHb) drive depression in rats, but whether dysfunctional LHb is associated with SD in human is unknown. Here we develop connectome-based biomarkers which predict SD and identify dysfunctional brain regions and connections. T1 weighted images and resting-state functional MRI (fMRI) data were collected from 34 subjects with SD and 40 healthy controls (HCs). After the brain is parcellated into 48 brain regions (246 subregions) using the human Brainnetome Atlas, the functional network of each participant is constructed by calculating the correlation coefficient for the time series of fMRI signals of each pair of subregions. Initial candidates of abnormal connections are identified by the two-sample t-test and input into Support Vector Machine models as features. A total of 24 anatomical-region-based models, 231 sliding-window-based models, and 100 random-selection-based models are built. The performance of these models is estimated through leave-one-out cross-validation and evaluated by measures of accuracy, sensitivity, confusion matrix, receiver operating characteristic curve, and the area under the curve (AUC). After confirming the region with the highest accuracy, subregions within the thalamus and connections associated with subregions of LHb are compared. It is found that five prediction models using connections of the thalamus, posterior superior temporal sulcus, cingulate gyrus, superior parietal lobule, and superior frontal gyrus achieve an accuracy >0.9 and an AUC >0.93. Among 90 abnormal connections associated with the thalamus, the subregion of the right posterior parietal thalamus where LHb is located has the most connections (n = 18), the left subregion only has 3 connections. In SD group, 10 subregions in the thalamus have significantly different node degrees with those in the HC group, while 8 subregions have lower degrees ( p < 0.01), including the one with LHb. These results implicate abnormal brain connections associated with the thalamus and LHb to be associated with SD. Integration of these connections by machine learning can provide connectome-based biomarkers to accurately diagnose SD.

Multimodal Neuroimaging: Basic Concepts and Classification of Neuropsychiatric Diseases

Neuroimaging techniques are widely used in neuroscience to visualize neural activity, to improve our understanding of brain mechanisms, and to identify biomarkers-especially for psychiatric diseases; however, each neuroimaging technique has several limitations. These limitations led to the development of multimodal neuroimaging (MN), which combines data obtained from multiple neuroimaging techniques, such as electroencephalography, functional magnetic resonance imaging, and yields more detailed information about brain dynamics. There are several types of MN, including visual inspection, data integration, and data fusion. This literature review aimed to provide a brief summary and basic information about MN techniques (data fusion approaches in particular) and classification approaches. Data fusion approaches are generally categorized as asymmetric and symmetric. The present review focused exclusively on studies based on symmetric data fusion methods (data-driven methods), such as independent component analysis and principal component analysis. Machine learning techniques have recently been introduced for use in identifying diseases and biomarkers of disease. The machine learning technique most widely used by neuroscientists is classification-especially support vector machine classification. Several studies differentiated patients with psychiatric diseases and healthy controls with using combined datasets. The common conclusion among these studies is that the prediction of diseases increases when combining data via MN techniques; however, there remain a few challenges associated with MN, such as sample size. Perhaps in the future N-way fusion can be used to combine multiple neuroimaging techniques or nonimaging predictors (eg, cognitive ability) to overcome the limitations of MN.

Psychiatric manifestations of neurologic diseases: Etiology, phenomenology, and treatment

Understanding the etiology and meaning of behavioral symptomatology in the context of neurologic disease, and choosing the most effective intervention is a vexing task. This introduction summarizes the history of our understanding of the relationship between behavioral symptoms and primary neurologic conditions, and considers the ways in which both psychiatric and neurologic disorders occurring simultaneously may inform both knowledge of etiology and treatment decisions.

Development of Neuroimaging-Based Biomarkers in Psychiatry

This chapter presents an overview of accumulating neuroimaging data with emphasis on translational potential. The subject will be described in the context of three disease states, i.e., schizophrenia, bipolar disorder, and major depressive disorder, and for three clinical goals, i.e., disease risk assessment, subtyping, and treatment decision.

Automated classification of Alzheimer's disease and mild cognitive impairment using a single MRI and deep neural networks

We built and validated a deep learning algorithm predicting the individual diagnosis of Alzheimer's disease (AD) and mild cognitive impairment who will convert to AD (c-MCI) based on a single cross-sectional brain structural MRI scan. Convolutional neural networks (CNNs) were applied on 3D T1-weighted images from ADNI and subjects recruited at our Institute (407 healthy controls [HC], 418 AD, 280 c-MCI, 533 stable MCI [s-MCI]). CNN performance was tested in distinguishing AD, c-MCI and s-MCI. High levels of accuracy were achieved in all the classifications, with the highest rates achieved in the AD vs HC classification tests using both the ADNI dataset only (99%) and the combined ADNI + non-ADNI dataset (98%). CNNs discriminated c-MCI from s-MCI patients with an accuracy up to 75% and no difference between ADNI and non-ADNI images. CNNs provide a powerful tool for the automatic individual patient diagnosis along the AD continuum. Our method performed well without any prior feature engineering and regardless the variability of imaging protocols and scanners, demonstrating that it is exploitable by not-trained operators and likely to be generalizable to unseen patient data. CNNs may accelerate the adoption of structural MRI in routine practice to help assessment and management of patients.

An overview of deep learning in medical imaging focusing on MRI

What has happened in machine learning lately, and what does it mean for the future of medical image analysis? Machine learning has witnessed a tremendous amount of attention over the last few years. The current boom started around 2009 when so-called deep artificial neural networks began outperforming other established models on a number of important benchmarks. Deep neural networks are now the state-of-the-art machine learning models across a variety of areas, from image analysis to natural language processing, and widely deployed in academia and industry. These developments have a huge potential for medical imaging technology, medical data analysis, medical diagnostics and healthcare in general, slowly being realized. We provide a short overview of recent advances and some associated challenges in machine learning applied to medical image processing and image analysis. As this has become a very broad and fast expanding field we will not survey the entire landscape of applications, but put particular focus on deep learning in MRI. Our aim is threefold: (i) give a brief introduction to deep learning with pointers to core references; (ii) indicate how deep learning has been applied to the entire MRI processing chain, from acquisition to image retrieval, from segmentation to disease prediction; (iii) provide a starting point for people interested in experimenting and perhaps contributing to the field of deep learning for medical imaging by pointing out good educational resources, state-of-the-art open-source code, and interesting sources of data and problems related medical imaging.

Big data and biomedical informatics: Preparing for the modernization of clinical neuropsychology

OBJECTIVE

Neuropsychology is poised for a fundamental shift as we modernize the ways in which behavior is measured. The amount and complexity of data generated by these new methods will be several orders of magnitude greater than what is currently created by analog measures and will quickly adopt characteristics of "Big Data." Adequate preparation for managing the influx of data will be critical for technology integration and modernization to be successful. Drawing from information technology, mathematics, statistics, computer science, and engineering, as well as, biology, genetics, and medicine, the field of biomedical informatics has rapidly evolved from its early days in computational biology to a burgeoning independent discipline that has much to offer neuropsychology.

METHOD

Following a critical review of the relevant literature, the present article (1) provides an introductory overview of biomedical informatics and how these concepts are relevant to neuropsychology; (2) describes how biomedical informatics applications can be utilized to leverage existing data sources more effectively; and (3) discusses ideas for future developments designed to facilitate integration of new data derived from novel, technologically driven measurement tools. Within this context, applications intended for use by both the individual neuropsychologist to increase clinical efficiencies, as well as, larger field-wide initiatives intended to generate new information and derive new knowledge are discussed.

CONCLUSIONS

By no means a comprehensive review of biomedical informatics, the present paper highlights that our approach to data needs to become a multidisciplinary endeavor in order to develop applications capable of effectively utilizing modern data sources.

Transfer learning improves resting-state functional connectivity pattern analysis using convolutional neural networks

Background

Deep learning is gaining importance in the prediction of cognitive states and brain pathology based on neuroimaging data. Including multiple hidden layers in artificial neural networks enables unprecedented predictive power; however, the proper training of deep neural networks requires thousands of exemplars. Collecting this amount of data is not feasible in typical neuroimaging experiments. A handy solution to this problem, which has largely fallen outside the scope of deep learning applications in neuroimaging, is to repurpose deep networks that have already been trained on large datasets by fine-tuning them to target datasets/tasks with fewer exemplars. Here, we investigated how this method, called transfer learning, can aid age category classification and regression based on brain functional connectivity patterns derived from resting-state functional magnetic resonance imaging. We trained a connectome-convolutional neural network on a larger public dataset and then examined how the knowledge learned can be used effectively to perform these tasks on smaller target datasets collected with a different type of scanner and/or imaging protocol and pre-processing pipeline.

Results

Age classification on the target datasets benefitted from transfer learning. Significant improvement (∼9%–13% increase in accuracy) was observed when the convolutional layers’ weights were initialized based on the values learned on the public dataset and then fine-tuned to the target datasets. Transfer learning also appeared promising in improving the otherwise poor prediction of chronological age.

Conclusions

Transfer learning is a plausible solution to adapt convolutional neural networks to neuroimaging data with few exemplars and different data acquisition and pre-processing protocols.

A Comparison of Regularization Techniques in Deep Neural Networks

Artificial neural networks (ANN) have attracted significant attention from researchers because many complex problems can be solved by training them. If enough data are provided during the training process, ANNs are capable of achieving good performance results. However, if training data are not enough, the predefined neural network model suffers from overfitting and underfitting problems. To solve these problems, several regularization techniques have been devised and widely applied to applications and data analysis. However, it is difficult for developers to choose the most suitable scheme for a developing application because there is no information regarding the performance of each scheme. This paper describes comparative research on regularization techniques by evaluating the training and validation errors in a deep neural network model, using a weather dataset. For comparisons, each algorithm was implemented using a recent neural network library of TensorFlow. The experiment results showed that an autoencoder had the worst performance among schemes. When the prediction accuracy was compared, data augmentation and the batch normalization scheme showed better performance than the others.

Using deep autoencoders to identify abnormal brain structural patterns in neuropsychiatric disorders: A large-scale multi-sample study

Machine learning is becoming an increasingly popular approach for investigating spatially distributed and subtle neuroanatomical alterations in brain-based disorders. However, some machine learning models have been criticized for requiring a large number of cases in each experimental group, and for resembling a "black box" that provides little or no insight into the nature of the data. In this article, we propose an alternative conceptual and practical approach for investigating brain-based disorders which aim to overcome these limitations. We used an artificial neural network known as "deep autoencoder" to create a normative model using structural magnetic resonance imaging data from 1,113 healthy people. We then used this model to estimate total and regional neuroanatomical deviation in individual patients with schizophrenia and autism spectrum disorder using two independent data sets (n = 263). We report that the model was able to generate different values of total neuroanatomical deviation for each disease under investigation relative to their control group (p < .005). Furthermore, the model revealed distinct patterns of neuroanatomical deviations for the two diseases, consistent with the existing neuroimaging literature. We conclude that the deep autoencoder provides a flexible and promising framework for assessing total and regional neuroanatomical deviations in neuropsychiatric populations.

Perspectives on Machine Learning for Classification of Schizotypy Using fMRI Data

Functional magnetic resonance imaging is capable of estimating functional activation and connectivity in the human brain, and lately there has been increased interest in the use of these functional modalities combined with machine learning for identification of psychiatric traits. While these methods bear great potential for early diagnosis and better understanding of disease processes, there are wide ranges of processing choices and pitfalls that may severely hamper interpretation and generalization performance unless carefully considered. In this perspective article, we aim to motivate the use of machine learning schizotypy research. To this end, we describe common data processing steps while commenting on best practices and procedures. First, we introduce the important role of schizotypy to motivate the importance of reliable classification, and summarize existing machine learning literature on schizotypy. Then, we describe procedures for extraction of features based on fMRI data, including statistical parametric mapping, parcellation, complex network analysis, and decomposition methods, as well as classification with a special focus on support vector classification and deep learning. We provide more detailed descriptions and software as supplementary material. Finally, we present current challenges in machine learning for classification of schizotypy and comment on future trends and perspectives.

A computational framework for the detection of subcortical brain dysmaturation in neonatal MRI using 3D Convolutional Neural Networks

Deep neural networks are increasingly being used in both supervised learning for classification tasks and unsupervised learning to derive complex patterns from the input data. However, the successful implementation of deep neural networks using neuroimaging datasets requires adequate sample size for training and well-defined signal intensity based structural differentiation. There is a lack of effective automated diagnostic tools for the reliable detection of brain dysmaturation in the neonatal period, related to small sample size and complex undifferentiated brain structures, despite both translational research and clinical importance. Volumetric information alone is insufficient for diagnosis. In this study, we developed a computational framework for the automated classification of brain dysmaturation from neonatal MRI, by combining a specific deep neural network implementation with neonatal structural brain segmentation as a method for both clinical pattern recognition and data-driven inference into the underlying structural morphology. We implemented three-dimensional convolution neural networks (3D-CNNs) to specifically classify dysplastic cerebelli, a subset of surface-based subcortical brain dysmaturation, in term infants born with congenital heart disease. We obtained a 0.985 ± 0. 0241-classification accuracy of subtle cerebellar dysplasia in CHD using 10-fold cross-validation. Furthermore, the hidden layer activations and class activation maps depicted regional vulnerability of the superior surface of the cerebellum, (composed of mostly the posterior lobe and the midline vermis), in regards to differentiating the dysplastic process from normal tissue. The posterior lobe and the midline vermis provide regional differentiation that is relevant to not only to the clinical diagnosis of cerebellar dysplasia, but also genetic mechanisms and neurodevelopmental outcome correlates. These findings not only contribute to the detection and classification of a subset of neonatal brain dysmaturation, but also provide insight to the pathogenesis of cerebellar dysplasia in CHD. In addition, this is one of the first examples of the application of deep learning to a neuroimaging dataset, in which the hidden layer activation revealed diagnostically and biologically relevant features about the clinical pathogenesis. The code developed for this project is open source, published under the BSD License, and designed to be generalizable to applications both within and beyond neonatal brain imaging.

Textural differences between renal cell carcinoma subtypes: Machine learning-based quantitative computed tomography texture analysis with independent external validation

OBJECTIVE

To develop externally validated, reproducible, and generalizable models for distinguishing three major subtypes of renal cell carcinomas (RCCs) using machine learning-based quantitative computed tomography (CT) texture analysis (qCT-TA).

MATERIALS AND METHODS

Sixty-eight RCCs were included in this retrospective study for model development and internal validation. Another 26 RCCs were included from public databases (The Cancer Genome Atlas-TCGA) for independent external validation. Following image preparation steps (reconstruction, resampling, normalization, and discretization), 275 texture features were extracted from unenhanced and corticomedullary phase CT images. Feature selection was firstly done with reproducibility analysis by three radiologists, and; then, with a wrapper-based classifier-specific algorithm. A nested cross-validation was performed for feature selection and model optimization. Base classifiers were the artificial neural network (ANN) and support vector machine (SVM). Base classifiers were also combined with three additional algorithms to improve generalizability performance. Classifications were done with the following groups: (i), non-clear cell RCC (non-cc-RCC) versus clear cell RCC (cc-RCC) and (ii), cc-RCC versus papillary cell RCC (pc-RCC) versus chromophobe cell RCC (chc-RCC). Main performance metric for comparisons was the Matthews correlation coefficient (MCC).

RESULTS

Number of the reproducible features is smaller for the unenhanced images (93 out of 275) compared to the corticomedullary phase images (232 out of 275). Overall performance metrics of the machine learning-based qCT-TA derived from corticomedullary phase images were better than those of unenhanced images. Using corticomedullary phase images, ANN with adaptive boosting algorithm performed best for discrimination of non-cc-RCCs from cc-RCCs (MCC = 0.728) with an external validation accuracy, sensitivity, and specificity of 84.6%, 69.2%, and 100%, respectively. On the other hand, the performance of the machine learning-based qCT-TA is rather poor for distinguishing three major subtypes. The SVM with bagging algorithm performed best for discrimination of pc-RCC from other RCC subtypes (MCC = 0.804) with an external validation accuracy, sensitivity, and specificity of 69.2%, 71.4%, and 100%, respectively.

CONCLUSIONS

Machine learning-based qCT-TA can distinguish non-cc-RCCs from cc-RCCs with a satisfying performance. On the other hand, the performance of the method for distinguishing three major subtypes is rather poor. Corticomedullary phase CT images provide much more valuable texture parameters than unenhanced images.

Classification and Prediction of Brain Disorders Using Functional Connectivity: Promising but Challenging

Brain functional imaging data, especially functional magnetic resonance imaging (fMRI) data, have been employed to reflect functional integration of the brain. Alteration in brain functional connectivity (FC) is expected to provide potential biomarkers for classifying or predicting brain disorders. In this paper, we present a comprehensive review in order to provide guidance about the available brain FC measures and typical classification strategies. We survey the state-of-the-art FC analysis methods including widely used static functional connectivity (SFC) and more recently proposed dynamic functional connectivity (DFC). Temporal correlations among regions of interest (ROIs), data-driven spatial network and functional network connectivity (FNC) are often computed to reflect SFC from different angles. SFC can be extended to DFC using a sliding-window framework, and intrinsic connectivity states along the time-varying connectivity patterns are typically extracted using clustering or decomposition approaches. We also briefly summarize window-less DFC approaches. Subsequently, we highlight various strategies for feature selection including the filter, wrapper and embedded methods. In terms of model building, we include traditional classifiers as well as more recently applied deep learning methods. Moreover, we review representative applications with remarkable classification accuracy for psychosis and mood disorders, neurodevelopmental disorder, and neurological disorders using fMRI data. Schizophrenia, bipolar disorder, autism spectrum disorder (ASD), attention deficit hyperactivity disorder (ADHD), Alzheimer's disease and mild cognitive impairment (MCI) are discussed. Finally, challenges in the field are pointed out with respect to the inaccurate diagnosis labeling, the abundant number of possible features and the difficulty in validation. Some suggestions for future work are also provided.

Reading the (functional) writing on the (structural) wall: Multimodal fusion of brain structure and function via a deep neural network based translation approach reveals novel impairments in schizophrenia

This work presents a novel approach to finding linkage/association between multimodal brain imaging data, such as structural MRI (sMRI) and functional MRI (fMRI). Motivated by the machine translation domain, we employ a deep learning model, and consider two different imaging views of the same brain like two different languages conveying some common facts. That analogy enables finding linkages between two modalities. The proposed translation-based fusion model contains a computing layer that learns "alignments" (or links) between dynamic connectivity features from fMRI data and static gray matter patterns from sMRI data. The approach is evaluated on a multi-site dataset consisting of eyes-closed resting state imaging data collected from 298 subjects (age- and gender matched 154 healthy controls and 144 patients with schizophrenia). Results are further confirmed on an independent dataset consisting of eyes-open resting state imaging data from 189 subjects (age- and gender matched 91 healthy controls and 98 patients with schizophrenia). We used dynamic functional connectivity (dFNC) states as the functional features and ICA-based sources from gray matter densities as the structural features. The dFNC states characterized by weakly correlated intrinsic connectivity networks (ICNs) were found to have stronger association with putamen and insular gray matter pattern, while the dFNC states of profuse strongly correlated ICNs exhibited stronger links with the gray matter pattern in precuneus, posterior cingulate cortex (PCC), and temporal cortex. Further investigation with the estimated link strength (or alignment score) showed significant group differences between healthy controls and patients with schizophrenia in several key regions including temporal lobe, and linked these to connectivity states showing less occupancy in healthy controls. Moreover, this novel approach revealed significant correlation between a cognitive score (attention/vigilance) and the function/structure alignment score that was not detected when data modalities were considered separately.