Multimodal Integration of Brain Images for MRI-Based Diagnosis in Schizophrenia

Recent deep learning-based brain tumor segmentation models using multi-modality magnetic resonance imaging: a prospective survey

Radiologists encounter significant challenges when segmenting and determining brain tumors in patients because this information assists in treatment planning. The utilization of artificial intelligence (AI), especially deep learning (DL), has emerged as a useful tool in healthcare, aiding radiologists in their diagnostic processes. This empowers radiologists to understand the biology of tumors better and provide personalized care to patients with brain tumors. The segmentation of brain tumors using multi-modal magnetic resonance imaging (MRI) images has received considerable attention. In this survey, we first discuss multi-modal and available magnetic resonance imaging modalities and their properties. Subsequently, we discuss the most recent DL-based models for brain tumor segmentation using multi-modal MRI. We divide this section into three parts based on the architecture: the first is for models that use the backbone of convolutional neural networks (CNN), the second is for vision transformer-based models, and the third is for hybrid models that use both convolutional neural networks and transformer in the architecture. In addition, in-depth statistical analysis is performed of the recent publication, frequently used datasets, and evaluation metrics for segmentation tasks. Finally, open research challenges are identified and suggested promising future directions for brain tumor segmentation to improve diagnostic accuracy and treatment outcomes for patients with brain tumors. This aligns with public health goals to use health technologies for better healthcare delivery and population health management.

A review of deep learning-based information fusion techniques for multimodal medical image classification

Multimodal medical imaging plays a pivotal role in clinical diagnosis and research, as it combines information from various imaging modalities to provide a more comprehensive understanding of the underlying pathology. Recently, deep learning-based multimodal fusion techniques have emerged as powerful tools for improving medical image classification. This review offers a thorough analysis of the developments in deep learning-based multimodal fusion for medical classification tasks. We explore the complementary relationships among prevalent clinical modalities and outline three main fusion schemes for multimodal classification networks: input fusion, intermediate fusion (encompassing single-level fusion, hierarchical fusion, and attention-based fusion), and output fusion. By evaluating the performance of these fusion techniques, we provide insight into the suitability of different network architectures for various multimodal fusion scenarios and application domains. Furthermore, we delve into challenges related to network architecture selection, handling incomplete multimodal data management, and the potential limitations of multimodal fusion. Finally, we spotlight the promising future of Transformer-based multimodal fusion techniques and give recommendations for future research in this rapidly evolving field.

DeepGAMI: deep biologically guided auxiliary learning for multimodal integration and imputation to improve genotype-phenotype prediction

BACKGROUND

Genotypes are strongly associated with disease phenotypes, particularly in brain disorders. However, the molecular and cellular mechanisms behind this association remain elusive. With emerging multimodal data for these mechanisms, machine learning methods can be applied for phenotype prediction at different scales, but due to the black-box nature of machine learning, integrating these modalities and interpreting biological mechanisms can be challenging. Additionally, the partial availability of these multimodal data presents a challenge in developing these predictive models.

METHOD

To address these challenges, we developed DeepGAMI, an interpretable neural network model to improve genotype-phenotype prediction from multimodal data. DeepGAMI leverages functional genomic information, such as eQTLs and gene regulation, to guide neural network connections. Additionally, it includes an auxiliary learning layer for cross-modal imputation allowing the imputation of latent features of missing modalities and thus predicting phenotypes from a single modality. Finally, DeepGAMI uses integrated gradient to prioritize multimodal features for various phenotypes.

RESULTS

We applied DeepGAMI to several multimodal datasets including genotype and bulk and cell-type gene expression data in brain diseases, and gene expression and electrophysiology data of mouse neuronal cells. Using cross-validation and independent validation, DeepGAMI outperformed existing methods for classifying disease types, and cellular and clinical phenotypes, even using single modalities (e.g., AUC score of 0.79 for Schizophrenia and 0.73 for cognitive impairment in Alzheimer's disease).

CONCLUSION

We demonstrated that DeepGAMI improves phenotype prediction and prioritizes phenotypic features and networks in multiple multimodal datasets in complex brains and brain diseases. Also, it prioritized disease-associated variants, genes, and regulatory networks linked to different phenotypes, providing novel insights into the interpretation of gene regulatory mechanisms. DeepGAMI is open-source and available for general use.

Supervised machine learning classification of psychosis biotypes based on brain structure: findings from the Bipolar-Schizophrenia network for intermediate phenotypes (B-SNIP)

Traditional diagnostic formulations of psychotic disorders have low correspondence with underlying disease neurobiology. This has led to a growing interest in using brain-based biomarkers to capture biologically-informed psychosis constructs. Building upon our prior work on the B-SNIP Psychosis Biotypes, we aimed to examine whether structural MRI (an independent biomarker not used in the Biotype development) can effectively classify the Biotypes. Whole brain voxel-wise grey matter density (GMD) maps from T1-weighted images were used to train and test (using repeated randomized train/test splits) binary L2-penalized logistic regression models to discriminate psychosis cases (n = 557) from healthy controls (CON, n = 251). A total of six models were evaluated across two psychosis categorization schemes: (i) three Biotypes (B1, B2, B3) and (ii) three DSM diagnoses (schizophrenia (SZ), schizoaffective (SAD) and bipolar (BD) disorders). Above-chance classification accuracies were observed in all Biotype (B1 = 0.70, B2 = 0.65, and B3 = 0.56) and diagnosis (SZ = 0.64, SAD = 0.64, and BD = 0.59) models. However, the only model that showed evidence of specificity was B1, i.e., the model was able to discriminate B1 vs. CON and did not misclassify other psychosis cases (B2 or B3) as B1 at rates above nominal chance. The GMD-based classifier evidence for B1 showed a negative association with an estimate of premorbid general intellectual ability, regardless of group membership, i.e. psychosis or CON. Our findings indicate that, complimentary to clinical diagnoses, the B-SNIP Psychosis Biotypes may offer a promising approach to capture specific aspects of psychosis neurobiology.

Reinforcement learning deficits exhibited by postnatal PCP-treated rats enable deep neural network classification

The ability to appropriately update the value of a given action is a critical component of flexible decision making. Several psychiatric disorders, including schizophrenia, are associated with impairments in flexible decision making that can be evaluated using the probabilistic reversal learning (PRL) task. The PRL task has been reverse-translated for use in rodents. Disrupting glutamate neurotransmission during early postnatal neurodevelopment in rodents has induced behavioral, cognitive, and neuropathophysiological abnormalities relevant to schizophrenia. Here, we tested the hypothesis that using the NMDA receptor antagonist phencyclidine (PCP) to disrupt postnatal glutamatergic transmission in rats would lead to impaired decision making in the PRL. Consistent with this hypothesis, compared to controls the postnatal PCP-treated rats completed fewer reversals and exhibited disruptions in reward and punishment sensitivity (i.e., win-stay and lose-shift responding, respectively). Moreover, computational analysis of behavior revealed that postnatal PCP-treatment resulted in a pronounced impairment in the learning rate throughout PRL testing. Finally, a deep neural network (DNN) trained on the rodent behavior could accurately predict the treatment group of subjects. These data demonstrate that disrupting early postnatal glutamatergic neurotransmission impairs flexible decision making and provides evidence that DNNs can be trained on behavioral datasets to accurately predict the treatment group of new subjects, highlighting the potential for DNNs to aid in the diagnosis of schizophrenia.

Sampling inequalities affect generalization of neuroimaging-based diagnostic classifiers in psychiatry

BACKGROUND

The development of machine learning models for aiding in the diagnosis of mental disorder is recognized as a significant breakthrough in the field of psychiatry. However, clinical practice of such models remains a challenge, with poor generalizability being a major limitation.

METHODS

Here, we conducted a pre-registered meta-research assessment on neuroimaging-based models in the psychiatric literature, quantitatively examining global and regional sampling issues over recent decades, from a view that has been relatively underexplored. A total of 476 studies (n = 118,137) were included in the current assessment. Based on these findings, we built a comprehensive 5-star rating system to quantitatively evaluate the quality of existing machine learning models for psychiatric diagnoses.

RESULTS

A global sampling inequality in these models was revealed quantitatively (sampling Gini coefficient (G) = 0.81, p < .01), varying across different countries (regions) (e.g., China, G = 0.47; the USA, G = 0.58; Germany, G = 0.78; the UK, G = 0.87). Furthermore, the severity of this sampling inequality was significantly predicted by national economic levels (β =  - 2.75, p < .001, R²_adj = 0.40; r =  - .84, 95% CI: - .41 to - .97), and was plausibly predictable for model performance, with higher sampling inequality for reporting higher classification accuracy. Further analyses showed that lack of independent testing (84.24% of models, 95% CI: 81.0-87.5%), improper cross-validation (51.68% of models, 95% CI: 47.2-56.2%), and poor technical transparency (87.8% of models, 95% CI: 84.9-90.8%)/availability (80.88% of models, 95% CI: 77.3-84.4%) are prevailing in current diagnostic classifiers despite improvements over time. Relating to these observations, model performances were found decreased in studies with independent cross-country sampling validations (all p < .001, BF₁₀ > 15). In light of this, we proposed a purpose-built quantitative assessment checklist, which demonstrated that the overall ratings of these models increased by publication year but were negatively associated with model performance.

CONCLUSIONS

Together, improving sampling economic equality and hence the quality of machine learning models may be a crucial facet to plausibly translating neuroimaging-based diagnostic classifiers into clinical practice.

Structural and functional differences in the brains of patients with MCI with and without depressive symptoms and their relations with Alzheimer's disease: an MRI study

Background

The mild cognitive impairment (MCI) stage among elderly individuals is very complex, and the level of diagnostic accuracy is far from ideal. Some studies have tried to improve the 'MCI due to Alzheimer's disease (AD)' classification by further stratifying these patients into subgroups. Depression-related symptoms may play an important role in helping to better define the MCI stage in elderly individuals.

Objective

In this work, we explored functional and structural differences in the brains of patients with nondepressed MCI (nDMCI) and patients with MCI with depressive symptoms (DMCI), and we examined how these groups relate to AD atrophy patterns and cognitive functioning.

Methods

Sixty-five participants underwent MRI exams and were divided into four groups: cognitively normal, nDMCI, DMCI, and AD. We compared the regional brain volumes, cortical thickness, and white matter microstructure measures using diffusion tensor imaging among groups. Additionally, we evaluated changes in functional connectivity using fMRI data.

Results

In comparison to the nDMCI group, the DMCI patients had more pronounced atrophy in the hippocampus and amygdala. Additionally, DMCI patients had asymmetric damage in the limbic-frontal white matter connection. Furthermore, two medial posterior regions, the isthmus of cingulate gyrus and especially the lingual gyrus, had high importance in the structural and functional differentiation between the two groups.

Conclusion

It is possible to differentiate nDMCI from DMCI patients using MRI techniques, which may contribute to a better characterization of subtypes of the MCI stage.

Machine learning approaches linking brain function to behavior in the ABCD STOP task

The stop-signal task (SST) is one of the most common fMRI tasks of response inhibition, and its performance measure, the stop-signal reaction-time (SSRT), is broadly used as a measure of cognitive control processes. The neurobiology underlying individual or clinical differences in response inhibition remain unclear, consistent with the general pattern of quite modest brain-behavior associations that have been recently reported in well-powered large-sample studies. Here, we investigated the potential of multivariate, machine learning (ML) methods to improve the estimation of individual differences in SSRT with multimodal structural and functional region of interest-level neuroimaging data from 9- to 11-year-olds children in the ABCD Study. Six ML algorithms were assessed across modalities and fMRI tasks. We verified that SST activation performed best in predicting SSRT among multiple modalities including morphological MRI (cortical surface area/thickness), diffusion tensor imaging, and fMRI task activations, and then showed that SST activation explained 12% of the variance in SSRT using cross-validation and out-of-sample lockbox data sets (n = 7298). Brain regions that were more active during the task and that showed more interindividual variation in activation were better at capturing individual differences in performance on the task, but this was only true for activations when successfully inhibiting. Cortical regions outperformed subcortical areas in explaining individual differences but the two hemispheres performed equally well. These results demonstrate that the detection of reproducible links between brain function and performance can be improved with multivariate approaches and give insight into a number of brain systems contributing to individual differences in this fundamental cognitive control process.

Evaluation of Risk of Bias in Neuroimaging-Based Artificial Intelligence Models for Psychiatric Diagnosis: A Systematic Review

IMPORTANCE

Neuroimaging-based artificial intelligence (AI) diagnostic models have proliferated in psychiatry. However, their clinical applicability and reporting quality (ie, feasibility) for clinical practice have not been systematically evaluated.

OBJECTIVE

To systematically assess the risk of bias (ROB) and reporting quality of neuroimaging-based AI models for psychiatric diagnosis.

EVIDENCE REVIEW

PubMed was searched for peer-reviewed, full-length articles published between January 1, 1990, and March 16, 2022. Studies aimed at developing or validating neuroimaging-based AI models for clinical diagnosis of psychiatric disorders were included. Reference lists were further searched for suitable original studies. Data extraction followed the CHARMS (Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modeling Studies) and PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analyses) guidelines. A closed-loop cross-sequential design was used for quality control. The PROBAST (Prediction Model Risk of Bias Assessment Tool) and modified CLEAR (Checklist for Evaluation of Image-Based Artificial Intelligence Reports) benchmarks were used to systematically evaluate ROB and reporting quality.

FINDINGS

A total of 517 studies presenting 555 AI models were included and evaluated. Of these models, 461 (83.1%; 95% CI, 80.0%-86.2%) were rated as having a high overall ROB based on the PROBAST. The ROB was particular high in the analysis domain, including inadequate sample size (398 of 555 models [71.7%; 95% CI, 68.0%-75.6%]), poor model performance examination (with 100% of models lacking calibration examination), and lack of handling data complexity (550 of 555 models [99.1%; 95% CI, 98.3%-99.9%]). None of the AI models was perceived to be applicable to clinical practices. Overall reporting completeness (ie, number of reported items/number of total items) for the AI models was 61.2% (95% CI, 60.6%-61.8%), and the completeness was poorest for the technical assessment domain with 39.9% (95% CI, 38.8%-41.1%).

CONCLUSIONS AND RELEVANCE

This systematic review found that the clinical applicability and feasibility of neuroimaging-based AI models for psychiatric diagnosis were challenged by a high ROB and poor reporting quality. Particularly in the analysis domain, ROB in AI diagnostic models should be addressed before clinical application.

Application of artificial intelligence tools in diagnosis and treatmentof mental disorders

Introduction: Artificial intelligence research is increasing its application in mental health services. Machine learning, deep learning, semantic analysis in the form of transcriptions of patients' statements enable early diagnosis of psychotic disorders, ADHD, anorexia nervosa. Of great importance are the so-called digital therapists. This paper aims to show the use of AI tools in diagnosing, treating, the benefits and limitations associated with mental disorders. Material and methodS: This literature review was conducted by searching scientific articles from 2015 to 2022. The basis were PubMED, OpenKnowledge, Web of Science, using the following keywords: artificial intelligence, digital therapy, psychiatry, machine learning. Results: A review indicates the widespread use of AI tools in screening for mental disorders. These tools advance the clinical diagnosis medical specialists make up for several years. They impact solving medical staff shortages, lack of access to medical facilities and leveling patient resistance to treatment. The benefits are ultra-fast analysis of large sets of information, effective screening of people in need of specialized psychiatric care, reduction of doctors' duties and maximization of their work efficiency. During the current COVID 19 pandemic, robots in the form of digital psychotherapists are playing a special role. Conclusions: The need for further research, testing and clarification of regulations related to the use of AI tools is indicated. Ethical and social problems need to be resolved. The tools should not form the basis of autonomous therapy without the supervision of highly trained professionals. Human beings should be at the center of analysis just as their health and well-being. Keywords: artificial intelligence, digital therapy, psychiatry, machine learning

Application of machine learning methods in predicting schizophrenia and bipolar disorders: A systematic review

Background and Aim

Schizophrenia and bipolar disorder (BD) are critical and high-risk inherited mental disorders with debilitating symptoms. Worldwide, 3% of the population suffers from these disorders. The mortality rate of these patients is higher compared to other people. Current procedures cannot effectively diagnose these disorders because it takes an average of 10 years from the onset of the first symptoms to the definitive diagnosis of the disease. Machine learning (ML) techniques are used to meet this need. This study aimed to summarize information on the use of ML techniques for predicting schizophrenia and BD to help early and timely diagnosis of the disease.

Methods

A systematic literature search included articles published until January 19, 2020 in 3 databases. Two reviewers independently assessed original papers to determine eligibility for inclusion in this review. PRISMA guidelines were followed to conduct the study, and the Prediction Model Risk of Bias Assessment Tool (PROBAST) to assess included papers.

Results

In this review, 1243 papers were retrieved through database searches, of which 15 papers were included based on full-text assessment. ML techniques were used to predict schizophrenia and BDs. The main algorithms applied were support vector machine (SVM) (10 studies), random forests (RF) (5 studies), and gradient boosting (GB) (3 studies). Input and output characteristics were very diverse and have been kept to enable future research. RFs algorithms demonstrated significantly higher accuracy and sensitivity than SVM and GB. GB demonstrated significantly higher specificity than SVM and RF. We found no significant difference between RF and SVM in terms of specificity.

Conclusion

ML can precisely predict results and assist in making clinical decisions-concerning schizophrenia and BD. RF often performed better than other algorithms in supervised learning tasks. This study identified gaps in the literature and opportunities for future psychological ML research.

Deep multimodal predictome for studying mental disorders

Characterizing neuropsychiatric disorders is challenging due to heterogeneity in the population. We propose combining structural and functional neuroimaging and genomic data in a multimodal classification framework to leverage their complementary information. Our objectives are two-fold (i) to improve the classification of disorders and (ii) to introspect the concepts learned to explore underlying neural and biological mechanisms linked to mental disorders. Previous multimodal studies have focused on naïve neural networks, mostly perceptron, to learn modality-wise features and often assume equal contribution from each modality. Our focus is on the development of neural networks for feature learning and implementing an adaptive control unit for the fusion phase. Our mid fusion with attention model includes a multilayer feed-forward network, an autoencoder, a bi-directional long short-term memory unit with attention as the features extractor, and a linear attention module for controlling modality-specific influence. The proposed model acquired 92% (p < .0001) accuracy in schizophrenia prediction, outperforming several other state-of-the-art models applied to unimodal or multimodal data. Post hoc feature analyses uncovered critical neural features and genes/biological pathways associated with schizophrenia. The proposed model effectively combines multimodal neuroimaging and genomics data for predicting mental disorders. Interpreting salient features identified by the model may advance our understanding of their underlying etiological mechanisms.

Going deep into schizophrenia with artificial intelligence

Despite years of research, the mechanisms governing the onset, relapse, symptomatology, and treatment of schizophrenia (SZ) remain elusive. The lack of appropriate analytic tools to deal with the heterogeneity and complexity of SZ may be one of the reasons behind this situation. Deep learning, a subfield of artificial intelligence (AI) inspired by the nervous system, has recently provided an accessible way of modeling and analyzing complex, high-dimensional, nonlinear systems. The unprecedented accuracy of deep learning algorithms in classification and prediction tasks has revolutionized a wide range of scientific fields and is rapidly permeating SZ research. Deep learning has the potential of becoming a valuable aid for clinicians in the prediction, diagnosis, and treatment of SZ, especially in combination with principles from Bayesian statistics. Furthermore, deep learning could become a powerful tool for uncovering the mechanisms underlying SZ thanks to a growing number of techniques designed for improving model interpretability and causal reasoning. The purpose of this article is to introduce SZ researchers to the field of deep learning and review its latest applications in SZ research. In general, existing studies have yielded impressive results in classification and outcome prediction tasks. However, methodological concerns related to the assessment of model performance in several studies, the widespread use of small training datasets, and the little clinical value of some models suggest that some of these results should be taken with caution.

Analyzing Hierarchical Multi-View MRI Data With StaPLR: An Application to Alzheimer's Disease Classification

Multi-view data refers to a setting where features are divided into feature sets, for example because they correspond to different sources. Stacked penalized logistic regression (StaPLR) is a recently introduced method that can be used for classification and automatically selecting the views that are most important for prediction. We introduce an extension of this method to a setting where the data has a hierarchical multi-view structure. We also introduce a new view importance measure for StaPLR, which allows us to compare the importance of views at any level of the hierarchy. We apply our extended StaPLR algorithm to Alzheimer's disease classification where different MRI measures have been calculated from three scan types: structural MRI, diffusion-weighted MRI, and resting-state fMRI. StaPLR can identify which scan types and which derived MRI measures are most important for classification, and it outperforms elastic net regression in classification performance.

Structural and diffusion MRI based schizophrenia classification using 2D pretrained and 3D naive Convolutional Neural Networks

The ability of automatic feature learning makes Convolutional Neural Network (CNN) potentially suitable to uncover the complex and widespread brain changes in schizophrenia. Despite that, limited studies have been done on schizophrenia identification using interpretable deep learning approaches on multimodal neuroimaging data. Here, we developed a deep feature approach based on pre-trained 2D CNN and naive 3D CNN models trained from scratch for schizophrenia classification by integrating 3D structural and diffusion magnetic resonance imaging (MRI) data. We found that the naive 3D CNN models outperformed the pretrained 2D CNN models and the handcrafted feature-based machine learning approach using support vector machine during both cross-validation and testing on an independent dataset. Multimodal neuroimaging-based models accomplished performance superior to models based on a single modality. Furthermore, we identified brain grey matter and white matter regions critical for illness classification at the individual- and group-level which supported the salience network and striatal dysfunction hypotheses in schizophrenia. Our findings underscore the potential of CNN not only to automatically uncover and integrate multimodal 3D brain imaging features for schizophrenia identification, but also to provide relevant neurobiological interpretations which are crucial for developing objective and interpretable imaging-based probes for prognosis and diagnosis in psychiatric disorders.

Four-modality imaging of unmedicated subjects with schizophrenia: 18F-fluorodeoxyglucose and 18F-fallypride PET, diffusion tensor imaging, and MRI

Diminished prefrontal function, dopaminergic abnormalities in the striatum and thalamus, reductions in white matter integrity and frontotemporal gray matter deficits are the most replicated findings in schizophrenia. We used four imaging modalities (¹⁸F-fluorodeoxyglucose and ¹⁸F-fallypride PET, diffusion tensor imaging, structural MRI) in 19 healthy and 25 schizophrenia subjects to assess the relationship between functional (dopamine D₂/D₃ receptor binding potential, glucose metabolic rate) and structural (fractional anisotropy, MRI) correlates of schizophrenia and their additive diagnostic prediction potential. Multivariate ANOVA was used to compare structural and functional image sets for identification of schizophrenia. Integration of data from all four modalities yielded better predictive power than less inclusive combinations, specifically in the thalamus, left dorsolateral prefrontal and temporal regions. Among the modalities, fractional anisotropy showed highest discrimination in white matter whereas ¹⁸F-fallypride binding showed highest discrimination in gray matter. Structural and functional modalities displayed comparable discriminative power but different topography, with higher sensitivity of structural modalities in the left prefrontal region. Combination of functional and structural imaging modalities with inclusion of both gray and white matter appears most effective in diagnostic discrimination. The highest sensitivity of ¹⁸F-fallypride PET to gray matter changes in schizophrenia supports the primacy of dopaminergic abnormalities in its pathophysiology.

Searching for Imaging Biomarkers of Psychotic Dysconnectivity

BACKGROUND

Progress in precision psychiatry is predicated on identifying reliable individual-level diagnostic biomarkers. For psychosis, measures of structural and functional connectivity could be promising biomarkers given consistent reports of dysconnectivity across psychotic disorders using magnetic resonance imaging.

METHODS

We leveraged data from four independent cohorts of patients with psychosis and control subjects with observations from approximately 800 individuals. We used group-level analyses and two supervised machine learning algorithms (support vector machines and ridge regression) to test within-, between-, and across-sample classification performance of white matter and resting-state connectivity metrics.

RESULTS

Although we replicated group-level differences in brain connectivity, individual-level classification was suboptimal. Classification performance within samples was variable across folds (highest area under the curve [AUC] range = 0.30) and across datasets (average support vector machine AUC range = 0.50; average ridge regression AUC range = 0.18). Classification performance between samples was similarly variable or resulted in AUC values of approximately 0.65, indicating a lack of model generalizability. Furthermore, collapsing across samples (resting-state functional magnetic resonance imaging, N = 888; diffusion tensor imaging, N = 860) did not improve model performance (maximal AUC = 0.67). Ridge regression models generally outperformed support vector machine models, although classification performance was still suboptimal in terms of clinical relevance. Adjusting for demographic covariates did not greatly affect results.

CONCLUSIONS

Connectivity measures were not suitable as diagnostic biomarkers for psychosis as assessed in this study. Our results do not negate that other approaches may be more successful, although it is clear that a systematic approach to individual-level classification with large independent validation samples is necessary to properly vet neuroimaging features as diagnostic biomarkers.

Multi-modal deep learning of functional and structural neuroimaging and genomic data to predict mental illness

Neuropsychiatric disorders such as schizophrenia are very heterogeneous in nature and typically diagnosed using self-reported symptoms. This makes it difficult to pose a confident prediction on the cases and does not provide insight into the underlying neural and biological mechanisms of these disorders. Combining neuroimaging and genomic data with a multi-modal 'predictome' paves the way for biologically informed markers and may improve prediction reliability. With that, we develop a multi-modal deep learning framework by fusing data from different modalities to capture the interaction between the latent features and evaluate their complementary information in characterizing schizophrenia. Our deep model uses structural MRI, functional MRI, and genome-wide polymorphism data to perform the classification task. It includes a multi-layer feed-forward network, an encoder, and a long short-term memory (LSTM) unit with attention to learn the latent features and adopt a joint training scheme capturing synergies between the modalities. The hybrid network also uses different regularizers for addressing the inherent overfitting and modality-specific bias in the multi-modal setup. Next, we run the network through a saliency model to analyze the learned features. Integrating modalities enhances the performance of the classifier, and our framework acquired 88% (P < 0.0001) accuracy on a dataset of 437 subjects. The trimodal accuracy is comparable to the state-of-the-art performance on a data collection of this size and outperforms the unimodal and bimodal baselines we compared. Model introspection was used to expose the salient neural features and genes/biological pathways associated with schizophrenia. To our best knowledge, this is the first approach that fuses genomic information with structural and functional MRI biomarkers for predicting schizophrenia. We believe this type of modality blending can better explain the disorder's dynamics by adding cross-modal prospects.Clinical Relevance- This study combinedly learns imaging and genomic features for the classification of schizophrenia. The data fusion scheme extracts modality interactions, and the saliency experiments report multiple functional and structural networks closely connected to the disorder.

A systematic review of the neural correlates of multisensory integration in schizophrenia

Multisensory integration (MSI), in which sensory signals from different modalities are unified, is necessary for our comprehensive perception of and effective adaptation to the objects and events around us. However, individuals with schizophrenia suffer from impairments in MSI, which could explain typical symptoms like hallucination and reality distortion. Because the neural correlates of aberrant MSI in schizophrenia help us understand the physiognomy of this psychiatric disorder, we performed a systematic review of the current research on this subject. The literature search concerned investigated MSI in diagnosed schizophrenia patients compared to healthy controls using brain imaging. Seventeen of 317 identified studies were finally included. To assess risk of bias, the Newcastle-Ottawa quality assessment was used, and the review was written according to the Preferred Reporting Items for Systematic Reviews and Meta-analysis (PRISMA). The results indicated that multisensory processes in schizophrenia are associated with aberrant, mainly reduced, neural activity in several brain regions, as measured by event-related potentials, oscillations, activity and connectivity. The conclusion is that a fronto-temporal region, comprising the frontal inferior gyrus, middle temporal gyrus and superior temporal gyrus/sulcus, along with the fusiform gyrus and dorsal visual stream in the occipital-parietal lobe are possible key regions of deficient MSI in schizophrenia.

Machine Learning of Schizophrenia Detection with Structural and Functional Neuroimaging

Schizophrenia (SZ) is a severe psychiatric illness, and it affects around 1% of the general population; however, its reliable diagnosis is challenging. Functional MRI (fMRI) and structural MRI (sMRI) are useful techniques for investigating the functional and structural abnormalities of the human brain, and a growing number of studies have reported that multimodal brain data can improve diagnostic accuracy. Machine learning (ML) is widely used in the diagnosis of neuroscience and neuropsychiatry diseases, and it can obtain high accuracy. However, the conventional ML which concatenated the features into a longer feature vector could not be sufficiently effective to combine different features from different modalities. There are considerable controversies over the use of global signal regression (GSR), and few studies have explored the role of GSR in ML in diagnosing neurological diseases. The current study utilized fMRI and sMRI data to implement a new method named multimodal imaging and multilevel characterization with multiclassifier (M3) to classify SZs and healthy controls (HCs) and investigate the influence of GSR in SZ classification. We found that when we used Brainnetome 246 atlas and without performed GSR, our method obtained a classification accuracy of 83.49%, with a sensitivity of 68.69%, a specificity of 93.75%, and an AUC of 0.8491, respectively. We also got great classification performances with different processing methods (with/without GSR and different brain parcellation schemes). We found that the accuracy and specificity of the models without GSR were higher than that of the models with GSR. Our findings indicate that the M3 method is an effective tool to distinguish SZs from HCs, and it can identify discriminative regions to detect SZ to explore the neural mechanisms underlying SZ. The global signal may contain important neuronal information; it can improve the accuracy and specificity of SZ detection.

A deep learning fusion model for brain disorder classification: Application to distinguishing schizophrenia and autism spectrum disorder

Deep learning has shown a great promise in classifying brain disorders due to its powerful ability in learning optimal features by nonlinear transformation. However, given the high-dimension property of neuroimaging data, how to jointly exploit complementary information from multimodal neuroimaging data in deep learning is difficult. In this paper, we propose a novel multilevel convolutional neural network (CNN) fusion method that can effectively combine different types of neuroimage-derived features. Importantly, we incorporate a sequential feature selection into the CNN model to increase the feature interpretability. To evaluate our method, we classified two symptom-related brain disorders using large-sample multi-site data from 335 schizophrenia (SZ) patients and 380 autism spectrum disorder (ASD) patients within a cross-validation procedure. Brain functional networks, functional network connectivity, and brain structural morphology were employed to provide possible features. As expected, our fusion method outperformed the CNN model using only single type of features, as our method yielded higher classification accuracy (with mean accuracy >85%) and was more reliable across multiple runs in differentiating the two groups. We found that the default mode, cognitive control, and subcortical regions contributed more in their distinction. Taken together, our method provides an effective means to fuse multimodal features for the diagnosis of different psychiatric and neurological disorders.