Neuroimaging modality fusion in Alzheimer's classification using convolutional neural networks

A review of deep learning-based information fusion techniques for multimodal medical image classification

Multimodal medical imaging plays a pivotal role in clinical diagnosis and research, as it combines information from various imaging modalities to provide a more comprehensive understanding of the underlying pathology. Recently, deep learning-based multimodal fusion techniques have emerged as powerful tools for improving medical image classification. This review offers a thorough analysis of the developments in deep learning-based multimodal fusion for medical classification tasks. We explore the complementary relationships among prevalent clinical modalities and outline three main fusion schemes for multimodal classification networks: input fusion, intermediate fusion (encompassing single-level fusion, hierarchical fusion, and attention-based fusion), and output fusion. By evaluating the performance of these fusion techniques, we provide insight into the suitability of different network architectures for various multimodal fusion scenarios and application domains. Furthermore, we delve into challenges related to network architecture selection, handling incomplete multimodal data management, and the potential limitations of multimodal fusion. Finally, we spotlight the promising future of Transformer-based multimodal fusion techniques and give recommendations for future research in this rapidly evolving field.

Longitudinal MRI analysis using a hybrid DenseNet-BiLSTM method for Alzheimer's disease prediction

Alzheimer's disease is a progressive neurological disorder characterized by brain atrophy and cell death, leading to cognitive decline and impaired functioning. Previous research has primarily focused on using cross-sectional data for Alzheimer's disease identification, but analyzing longitudinal sequential MR images is crucial for improved diagnostic accuracy and understanding disease progression. However, existing deep learning models face challenges in learning spatial and temporal features from such data. To address these challenges, this study presents a novel hybrid DenseNet-BiLSTM method for Alzheimer's disease prediction using longitudinal MRI analysis. The proposed framework combines Convolutional DenseNet for spatial information extraction and joined BiLSTM layers for capturing temporal characteristics and relationships between longitudinal images at different time points. This approach overcomes issues like overfitting, vanishing gradients, and incomplete patient data. We evaluated the model on 684 longitudinal MRI images from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database, including normal controls, individuals with mild cognitive impairment, and Alzheimer's disease patients. The results demonstrate high classification accuracy, with 95.28% for AD/CN, 88.19% for NC/MCI, 83.51% for sMCI/pMCI, and 92.14% for MCI/AD. These findings highlight the substantial improvement in Alzheimer's disease diagnosis achieved through the utilization of longitudinal MRI images. The contributions of this study lie in both the deep learning and medical domains. In the deep learning domain, our hybrid framework effectively learns spatial and temporal features from longitudinal data, addressing the challenges associated with multi-dimensional and sequential time series data. In the medical domain, our study emphasizes the importance of analyzing baseline and longitudinal MR images for accurate diagnosis and understanding disease progression.

Alzheimer's disease detection using data fusion with a deep supervised encoder

Alzheimer's disease (AD) is affecting a growing number of individuals. As a result, there is a pressing need for accurate and early diagnosis methods. This study aims to achieve this goal by developing an optimal data analysis strategy to enhance computational diagnosis. Although various modalities of AD diagnostic data are collected, past research on computational methods of AD diagnosis has mainly focused on using single-modal inputs. We hypothesize that integrating, or "fusing," various data modalities as inputs to prediction models could enhance diagnostic accuracy by offering a more comprehensive view of an individual's health profile. However, a potential challenge arises as this fusion of multiple modalities may result in significantly higher dimensional data. We hypothesize that employing suitable dimensionality reduction methods across heterogeneous modalities would not only help diagnosis models extract latent information but also enhance accuracy. Therefore, it is imperative to identify optimal strategies for both data fusion and dimensionality reduction. In this paper, we have conducted a comprehensive comparison of over 80 statistical machine learning methods, considering various classifiers, dimensionality reduction techniques, and data fusion strategies to assess our hypotheses. Specifically, we have explored three primary strategies: (1) Simple data fusion, which involves straightforward concatenation (fusion) of datasets before inputting them into a classifier; (2) Early data fusion, in which datasets are concatenated first, and then a dimensionality reduction technique is applied before feeding the resulting data into a classifier; and (3) Intermediate data fusion, in which dimensionality reduction methods are applied individually to each dataset before concatenating them to construct a classifier. For dimensionality reduction, we have explored several commonly-used techniques such as principal component analysis (PCA), autoencoder (AE), and LASSO. Additionally, we have implemented a new dimensionality-reduction method called the supervised encoder (SE), which involves slight modifications to standard deep neural networks. Our results show that SE substantially improves prediction accuracy compared to PCA, AE, and LASSO, especially in combination with intermediate fusion for multiclass diagnosis prediction.

Evaluation of MRI-based machine learning approaches for computer-aided diagnosis of dementia in a clinical data warehouse

A variety of algorithms have been proposed for computer-aided diagnosis of dementia from anatomical brain MRI. These approaches achieve high accuracy when applied to research data sets but their performance on real-life clinical routine data has not been evaluated yet. The aim of this work was to study the performance of such approaches on clinical routine data, based on a hospital data warehouse, and to compare the results to those obtained on a research data set. The clinical data set was extracted from the hospital data warehouse of the Greater Paris area, which includes 39 different hospitals. The research set was composed of data from the Alzheimer's Disease Neuroimaging Initiative data set. In the clinical set, the population of interest was identified by exploiting the diagnostic codes from the 10th revision of the International Classification of Diseases that are assigned to each patient. We studied how the imbalance of the training sets, in terms of contrast agent injection and image quality, may bias the results. We demonstrated that computer-aided diagnosis performance was strongly biased upwards (over 17 percent points of balanced accuracy) by the confounders of image quality and contrast agent injection, a phenomenon known as the Clever Hans effect or shortcut learning. When these biases were removed, the performance was very poor. In any case, the performance was considerably lower than on the research data set. Our study highlights that there are still considerable challenges for translating dementia computer-aided diagnosis systems to clinical routine.

Mapping the association between tau-PET and Aβ-amyloid-PET using deep learning

In Alzheimer’s disease, the molecular pathogenesis of the extracellular Aβ-amyloid (Aβ) instigation of intracellular tau accumulation is poorly understood. We employed a high-resolution PET scanner, with low detection thresholds, to examine the Aβ-tau association using a convolutional neural network (CNN), and compared results to a standard voxel-wise linear analysis. The full range of Aβ Centiloid values was highly predicted by the tau topography using the CNN (training R² = 0.86, validation R² = 0.75, testing R² = 0.72). Linear models based on tau-SUVR identified widespread positive correlations between tau accumulation and Aβ burden throughout the brain. In contrast, CNN analysis identified focal clusters in the bilateral medial temporal lobes, frontal lobes, precuneus, postcentral gyrus and middle cingulate. At low Aβ levels, information from the middle cingulate, frontal lobe and precuneus regions was more predictive of Aβ burden, while at high Aβ levels, the medial temporal regions were more predictive of Aβ burden. The data-driven CNN approach revealed new associations between tau topography and Aβ burden.

Cross-Modal Transfer Learning Methods for Alzheimer's Disease Diagnosis

In this paper we propose cross-modal transfer learning for Alzheimer's disease detection. We use positron emission tomography (PET) and magnetic resonance imaging (MRI) brain scans from ADNI to train convolutional neural networks (CNNs) on one modality and fine-tune it on the other modality. We start by showing that cross-modal transfer learning approaches outperform CNNs trained from scratch on a single modality. We then show that cross-modal transfer-learning also outperforms multimodal approaches using the same data.

An Exploration: Alzheimer's Disease Classification Based on Convolutional Neural Network

Alzheimer's disease (AD) is the most generally known neurodegenerative disorder, leading to a steady deterioration in cognitive ability. Deep learning models have shown outstanding performance in the diagnosis of AD, and these models do not need any handcrafted feature extraction over conventional machine learning algorithms. Since the 2012 AlexNet accomplishment, the convolutional neural network (CNN) has been progressively utilized by the medical community to assist practitioners to early diagnose AD. This paper explores the current cutting edge applications of CNN on single and multimodality (combination of two or more modalities) neuroimaging data for the classification of AD. An exhaustive systematic search is conducted on four notable databases: Google Scholar, IEEE Xplore, ACM Digital Library, and PubMed in June 2021. The objective of this study is to examine the effectiveness of classification approaches on AD to analyze different kinds of datasets, neuroimaging modalities, preprocessing techniques, and data handling methods. However, CNN has achieved great success in the classification of AD; still, there are a lot of challenges particularly due to scarcity of medical imaging data and its possible scope in this field.

Classification of Alzheimer's Disease Using Gaussian-Based Bayesian Parameter Optimization for Deep Convolutional LSTM Network

Alzheimer's disease (AD) is one of the most important causes of mortality in elderly people, and it is often challenging to use traditional manual procedures when diagnosing a disease in the early stages. The successful implementation of machine learning (ML) techniques has also shown their effectiveness and its reliability as one of the better options for an early diagnosis of AD. But the heterogeneous dimensions and composition of the disease data have undoubtedly made diagnostics more difficult, needing a sufficient model choice to overcome the difficulty. Therefore, in this paper, four different 2D and 3D convolutional neural network (CNN) frameworks based on Bayesian search optimization are proposed to develop an optimized deep learning model to predict the early onset of AD binary and ternary classification on magnetic resonance imaging (MRI) scans. Moreover, certain hyperparameters such as learning rate, optimizers, and hidden units are to be set and adjusted for the performance boosting of the deep learning model. Bayesian optimization enables to leverage advantage throughout the experiments: A persistent hyperparameter space testing provides not only the output but also about the nearest conclusions. In this way, the series of experiments needed to explore space can be substantially reduced. Finally, alongside the use of Bayesian approaches, long short-term memory (LSTM) through the process of augmentation has resulted in finding the better settings of the model that too in less iterations with an relative improvement (RI) of 7.03%, 12.19%, 10.80%, and 11.99% over the four systems optimized with manual hyperparameters tuning such that hyperparameters that look more appealing from past data as well as the conventional techniques of manual selection.

Automatic quality control of brain T1-weighted magnetic resonance images for a clinical data warehouse

Many studies on machine learning (ML) for computer-aided diagnosis have so far been mostly restricted to high-quality research data. Clinical data warehouses, gathering routine examinations from hospitals, offer great promises for training and validation of ML models in a realistic setting. However, the use of such clinical data warehouses requires quality control (QC) tools. Visual QC by experts is time-consuming and does not scale to large datasets. In this paper, we propose a convolutional neural network (CNN) for the automatic QC of 3D T1-weighted brain MRI for a large heterogeneous clinical data warehouse. To that purpose, we used the data warehouse of the hospitals of the Greater Paris area (Assistance Publique-Hôpitaux de Paris [AP-HP]). Specifically, the objectives were: 1) to identify images which are not proper T1-weighted brain MRIs; 2) to identify acquisitions for which gadolinium was injected; 3) to rate the overall image quality. We used 5000 images for training and validation and a separate set of 500 images for testing. In order to train/validate the CNN, the data were annotated by two trained raters according to a visual QC protocol that we specifically designed for application in the setting of a data warehouse. For objectives 1 and 2, our approach achieved excellent accuracy (balanced accuracy and F1-score >90%), similar to the human raters. For objective 3, the performance was good but substantially lower than that of human raters. Nevertheless, the automatic approach accurately identified (balanced accuracy and F1-score >80%) low quality images, which would typically need to be excluded. Overall, our approach shall be useful for exploiting hospital data warehouses in medical image computing.

Artificial intelligence for molecular neuroimaging

In recent years, artificial intelligence (AI) or the study of how computers and machines can gain intelligence, has been increasingly applied to problems in medical imaging, and in particular to molecular imaging of the central nervous system. Many AI innovations in medical imaging include improving image quality, segmentation, and automating classification of disease. These advances have led to an increased availability of supportive AI tools to assist physicians in interpreting images and making decisions affecting patient care. This review focuses on the role of AI in molecular neuroimaging, primarily applied to positron emission tomography (PET) and single photon emission computed tomography (SPECT). We emphasize technical innovations such as AI in computed tomography (CT) generation for the purposes of attenuation correction and disease localization, as well as applications in neuro-oncology and neurodegenerative diseases. Limitations and future prospects for AI in molecular brain imaging are also discussed. Just as new equipment such as SPECT and PET revolutionized the field of medical imaging a few decades ago, AI and its related technologies are now poised to bring on further disruptive changes. An understanding of these new technologies and how they work will help physicians adapt their practices and succeed with these new tools.

The role of the deep convolutional neural network as an aid to interpreting brain [18F]DOPA PET/CT in the diagnosis of Parkinson's disease

OBJECTIVES

To test the performance of a 3D convolutional neural network (CNN) in analysing brain [¹⁸F]DOPA PET/CT in order to identify patients with nigro-striatal neurodegeneration. We evaluated the robustness of the 3D CNN by testing it against a manual regional analysis of the striata by using a striatal-to-occipital ratio (SOR).

METHODS

We analyzed patients who had undergone [¹⁸F]DOPA PET/CT from 2016 to 2018. Two examiners interpreted PET/CT images as positive or negative. Only patients with at least 2 years of follow-up and an ascertained neurological diagnosis were included. A 3D CNN was developed to evaluate [¹⁸F]DOPA PET/CT and refine the diagnosis of movement disorder. This system required training and testing, which were carried out on 2/3 and 1/3 of patients, respectively. A regional analysis was also conducted by drawing region of interest on T1-weighted 3D MRI scans, on which the [¹⁸F]DOPA PET images were first co-registered.

RESULTS

Ninety-eight patients were enrolled: 43 presented nigro-striatal degeneration and 55 negative cases used as controls. After training on 69 patients, the diagnostic performance of the 3D CNN was then calculated in 29 patients. Sensitivity, specificity, negative predictive value, positive predictive value and accuracy were 100%, 89%, 100%, 85% and 93%, respectively. When we compared the 3D CNN results with the SOR analysis, we found that the two patients falsely classified as positive by the 3D CNN procedure showed SOR values ≤ 5^th percentile of the negative cases' distribution.

CONCLUSIONS

3D CNNs are able to interpret [¹⁸F]DOPA PET/CT properly, revealing patients affected by Parkinson's disease.

KEY POINTS

• [¹⁸F]DOPA PET/CT is a sensitive diagnostic tool to identify patients with nigro-striatal neurodegeneration. • A semiquantitative evaluation of the images allows a more confident interpretation of the PET findings. • 3D convolutional neural network allows an accurate interpretation of 18F-DOPA PET/CT images, revealing patients affected by Parkinson's disease.

Application and Construction of Deep Learning Networks in Medical Imaging

Deep learning (DL) approaches are part of the machine learning (ML) subfield concerned with the development of computational models to train artificial intelligence systems. DL models are characterized by automatically extracting high-level features from the input data to learn the relationship between matching datasets. Thus, its implementation offers an advantage over common ML methods that often require the practitioner to have some domain knowledge of the input data to select the best latent representation. As a result of this advantage, DL has been successfully applied within the medical imaging field to address problems, such as disease classification and tumor segmentation for which it is difficult or impossible to determine which image features are relevant. Therefore, taking into consideration the positive impact of DL on the medical imaging field, this article reviews the key concepts associated with its evolution and implementation. The sections of this review summarize the milestones related to the development of the DL field, followed by a description of the elements of deep neural network and an overview of its application within the medical imaging field. Subsequently, the key steps necessary to implement a supervised DL application are defined, and associated limitations are discussed.

An Effective Multimodal Image Fusion Method Using MRI and PET for Alzheimer's Disease Diagnosis

Alzheimer's disease (AD) is an irreversible brain disease that severely damages human thinking and memory. Early diagnosis plays an important part in the prevention and treatment of AD. Neuroimaging-based computer-aided diagnosis (CAD) has shown that deep learning methods using multimodal images are beneficial to guide AD detection. In recent years, many methods based on multimodal feature learning have been proposed to extract and fuse latent representation information from different neuroimaging modalities including magnetic resonance imaging (MRI) and 18-fluorodeoxyglucose positron emission tomography (FDG-PET). However, these methods lack the interpretability required to clearly explain the specific meaning of the extracted information. To make the multimodal fusion process more persuasive, we propose an image fusion method to aid AD diagnosis. Specifically, we fuse the gray matter (GM) tissue area of brain MRI and FDG-PET images by registration and mask coding to obtain a new fused modality called "GM-PET." The resulting single composite image emphasizes the GM area that is critical for AD diagnosis, while retaining both the contour and metabolic characteristics of the subject's brain tissue. In addition, we use the three-dimensional simple convolutional neural network (3D Simple CNN) and 3D Multi-Scale CNN to evaluate the effectiveness of our image fusion method in binary classification and multi-classification tasks. Experiments on the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset indicate that the proposed image fusion method achieves better overall performance than unimodal and feature fusion methods, and that it outperforms state-of-the-art methods for AD diagnosis.

Brain [F-18]FDG PET for Clinical Dementia Workup: Differential Diagnosis of Alzheimer's Disease and Other Types of Dementing Disorders

PET imaging with [F-18]FDG has been used extensively for research and clinical applications in dementia. In the brain, [F-18]FDG accumulates around synapses and represents local neuronal activity. Patterns of altered [F-18]FDG uptake reflecting local neuronal dysfunction provide differential diagnostic clues for various dementing disorders. Image interpretation can be accomplished by employing statistical brain mapping techniques. Various guidelines have been published to support the appropriate use of [F-18]FDG PET for clinical dementia workup. PET images with [F-18]FDG demonstrate distinct patterns of decreased uptake for Alzheimer's disease (AD), Dementia with Lewy bodies (DLB), and frontotemporal dementia (FTD) as well as its multiple subtypes such as behavioral variant FTD, primary progressive aphasia (PPA), progressive supranuclear palsy, and corticobasal degeneration to aid in the differential diagnoses. Mixed dementia, not only AD + Vascular Dementia, but also AD + other neurodegenerative disorders, should also be considered when interpreting [F-18]FDG PET images. Brain PET imaging with [F-18]FDG remains a valuable component of dementia workup owing to its relatively low cost, differential diagnostic performance, widespread availability, and physicians' experience over more than 40 years since the initial development.

Deep learning for brain disorders: from data processing to disease treatment

In order to reach precision medicine and improve patients' quality of life, machine learning is increasingly used in medicine. Brain disorders are often complex and heterogeneous, and several modalities such as demographic, clinical, imaging, genetics and environmental data have been studied to improve their understanding. Deep learning, a subpart of machine learning, provides complex algorithms that can learn from such various data. It has become state of the art in numerous fields, including computer vision and natural language processing, and is also growingly applied in medicine. In this article, we review the use of deep learning for brain disorders. More specifically, we identify the main applications, the concerned disorders and the types of architectures and data used. Finally, we provide guidelines to bridge the gap between research studies and clinical routine.

Ensemble of ROI-based convolutional neural network classifiers for staging the Alzheimer disease spectrum from magnetic resonance imaging

Patches from three orthogonal views of selected cerebral regions can be utilized to learn convolutional neural network (CNN) models for staging the Alzheimer disease (AD) spectrum including preclinical AD, mild cognitive impairment due to AD, and dementia due to AD and normal controls. Hippocampi, amygdalae and insulae were selected from the volumetric analysis of structured magnetic resonance images (MRIs). Three-view patches (TVPs) from these regions were fed to the CNN for training. MRIs were classified with the SoftMax-normalized scores of individual model predictions on TVPs. The significance of each region of interest (ROI) for staging the AD spectrum was evaluated and reported. The results of the ensemble classifier are compared with state-of-the-art methods using the same evaluation metrics. Patch-based ROI ensembles provide comparable diagnostic performance for AD staging. In this work, TVP-based ROI analysis using a CNN provides informative landmarks in cerebral MRIs and may have significance in clinical studies and computer-aided diagnosis system design.

A 5-min Cognitive Task With Deep Learning Accurately Detects Early Alzheimer's Disease

Introduction: The goal of this study was to investigate and compare the classification performance of machine learning with behavioral data from standard neuropsychological tests, a cognitive task, or both.

Methods: A neuropsychological battery and a simple 5-min cognitive task were administered to eight individuals with mild cognitive impairment (MCI), eight individuals with mild Alzheimer's disease (AD), and 41 demographically match controls (CN). A fully connected multilayer perceptron (MLP) network and four supervised traditional machine learning algorithms were used.

Results: Traditional machine learning algorithms achieved similar classification performances with neuropsychological or cognitive data. MLP outperformed traditional algorithms with the cognitive data (either alone or together with neuropsychological data), but not neuropsychological data. In particularly, MLP with a combination of summarized scores from neuropsychological tests and the cognitive task achieved ~90% sensitivity and ~90% specificity. Applying the models to an independent dataset, in which the participants were demographically different from the ones in the main dataset, a high specificity was maintained (100%), but the sensitivity was dropped to 66.67%.

Discussion: Deep learning with data from specific cognitive task(s) holds promise for assisting in the early diagnosis of Alzheimer's disease, but future work with a large and diverse sample is necessary to validate and to improve this approach.

Artificial intelligence for brain diseases: A systematic review

Artificial intelligence (AI) is a major branch of computer science that is fruitfully used for analyzing complex medical data and extracting meaningful relationships in datasets, for several clinical aims. Specifically, in the brain care domain, several innovative approaches have achieved remarkable results and open new perspectives in terms of diagnosis, planning, and outcome prediction. In this work, we present an overview of different artificial intelligent techniques used in the brain care domain, along with a review of important clinical applications. A systematic and careful literature search in major databases such as Pubmed, Scopus, and Web of Science was carried out using "artificial intelligence" and "brain" as main keywords. Further references were integrated by cross-referencing from key articles. 155 studies out of 2696 were identified, which actually made use of AI algorithms for different purposes (diagnosis, surgical treatment, intra-operative assistance, and postoperative assessment). Artificial neural networks have risen to prominent positions among the most widely used analytical tools. Classic machine learning approaches such as support vector machine and random forest are still widely used. Task-specific algorithms are designed for solving specific problems. Brain images are one of the most used data types. AI has the possibility to improve clinicians' decision-making ability in neuroscience applications. However, major issues still need to be addressed for a better practical use of AI in the brain. To this aim, it is important to both gather comprehensive data and build explainable AI algorithms.

Interpretation of Brain Morphology in Association to Alzheimer's Disease Dementia Classification Using Graph Convolutional Networks on Triangulated Meshes

We propose a mesh-based technique to aid in the classification of Alzheimer's disease dementia (ADD) using mesh representations of the cortex and subcortical structures. Deep learning methods for classification tasks that utilize structural neuroimaging often require extensive learning parameters to optimize. Frequently, these approaches for automated medical diagnosis also lack visual interpretability for areas in the brain involved in making a diagnosis. This work: (a) analyzes brain shape using surface information of the cortex and subcortical structures, (b) proposes a residual learning framework for state-of-the-art graph convolutional networks which offer a significant reduction in learnable parameters, and (c) offers visual interpretability of the network via class-specific gradient information that localizes important regions of interest in our inputs. With our proposed method leveraging the use of cortical and subcortical surface information, we outperform other machine learning methods with a 96.35% testing accuracy for the ADD vs. healthy control problem. We confirm the validity of our model by observing its performance in a 25-trial Monte Carlo cross-validation. The generated visualization maps in our study show correspondences with current knowledge regarding the structural localization of pathological changes in the brain associated to dementia of the Alzheimer's type.

Multi-class diagnosis of Alzheimer's disease using cascaded three dimensional-convolutional neural network

Dementia is a social problem in the aging society of advanced countries. Presently, 46.8 million people affected with dementia worldwide, and it may increase to 130 million by 2050. Alzheimer's disease (AD) is the most common form of dementia. The cost of care for AD patients in 2015 was 818 billion US dollars and is expected to increase intensely due to the increasing number of patients due to the aging society. It isn't easy to cure AD, but early detection is crucial. This paper proposes a multi-class classification of AD, mild cognitive impairment (MCI), and normal control (NC) subjects using three dimensional-convolutional neural network with Support Vector Machine classifier. A cross-sectional study on structural MRI data of 465 subjects, including 132 AD patients, 181 MCI, and 152 NC, is performed in this paper. The highly complex and spatial atrophy patterns of the brain related to Alzheimer's Disease and MCI are extracted from structural MRI images using cascaded layers of the three dimensional convolutional neural network. The hectic process of segmentation and further extraction of handcrafted features is eliminated. The complete image is considered for the processing, thus incorporating every region of the brain for the classification. The features extracted using four cascaded layers of three dimensional-convolutional neural network are fed into the Support Vector Machine classifier. The proposed method achieved 97.77% accuracy which outperforms state of the art, and this algorithm is a promising indicator for the diagnosis of AD.