Joint Classification and Regression via Deep Multi-Task Multi-Channel Learning for Alzheimer's Disease Diagnosis

M3T-LM: A multi-modal multi-task learning model for jointly predicting patient length of stay and mortality

Ensuring accurate predictions of inpatient length of stay (LoS) and mortality rates is essential for enhancing hospital service efficiency, particularly in light of the constraints posed by limited healthcare resources. Integrative analysis of heterogeneous clinic record data from different sources can hold great promise for improving the prognosis and diagnosis level of LoS and mortality. Currently, most existing studies solely focus on single data modality or tend to single-task learning, i.e., training LoS and mortality tasks separately. This limits the utilization of available multi-modal data and prevents the sharing of feature representations that could capture correlations between different tasks, ultimately hindering the model's performance. To address the challenge, this study proposes a novel Multi-Modal Multi-Task learning model, termed as M3T-LM, to integrate clinic records to predict inpatients' LoS and mortality simultaneously. The M3T-LM framework incorporates multiple data modalities by constructing sub-models tailored to each modality. Specifically, a novel attention-embedded one-dimensional (1D) convolutional neural network (CNN) is designed to handle numerical data. For clinical notes, they are converted into sequence data, and then two long short-term memory (LSTM) networks are exploited to model on textual sequence data. A two-dimensional (2D) CNN architecture, noted as CRXMDL, is designed to extract high-level features from chest X-ray (CXR) images. Subsequently, multiple sub-models are integrated to formulate the M3T-LM to capture the correlations between patient LoS and modality prediction tasks. The efficiency of the proposed method is validated on the MIMIC-IV dataset. The proposed method attained a test MAE of 5.54 for LoS prediction and a test F1 of 0.876 for mortality prediction. The experimental results demonstrate that our approach outperforms state-of-the-art (SOTA) methods in tackling mixed regression and classification tasks.

Bilinear Perceptual Fusion Algorithm Based on Brain Functional and Structural Data for ASD Diagnosis and Regions of Interest Identification

Autism spectrum disorder (ASD) is a serious mental disorder with a complex pathogenesis mechanism and variable presentation among individuals. Although many deep learning algorithms have been used to diagnose ASD, most of them focus on a single modality of data, resulting in limited information extraction and poor stability. In this paper, we propose a bilinear perceptual fusion (BPF) algorithm that leverages data from multiple modalities. In our algorithm, different schemes are used to extract features according to the characteristics of functional and structural data. Through bilinear operations, the associations between the functional and structural features of each region of interest (ROI) are captured. Then the associations are used to integrate the feature representation. Graph convolutional neural networks (GCNs) can effectively utilize topology and node features in brain network analysis. Therefore, we design a deep learning framework called BPF-GCN and conduct experiments on publicly available ASD dataset. The results show that the classification accuracy of BPF-GCN reached 82.35%, surpassing existing methods. This demonstrates the superiority of its classification performance, and the framework can extract ROIs related to ASD. Our work provides a valuable reference for the timely diagnosis and treatment of ASD.

A transformer-based unified multimodal framework for Alzheimer's disease assessment

In Alzheimer's disease (AD) assessment, traditional deep learning approaches have often employed separate methodologies to handle the diverse modalities of input data. Recognizing the critical need for a cohesive and interconnected analytical framework, we propose the AD-Transformer, a novel transformer-based unified deep learning model. This innovative framework seamlessly integrates structural magnetic resonance imaging (sMRI), clinical, and genetic data from the extensive Alzheimer's Disease Neuroimaging Initiative (ADNI) database, encompassing 1651 subjects. By employing a Patch-CNN block, the AD-Transformer efficiently transforms image data into image tokens, while a linear projection layer adeptly converts non-image data into corresponding tokens. As the core, a transformer block learns comprehensive representations of the input data, capturing the intricate interplay between modalities. The AD-Transformer sets a new benchmark in AD diagnosis and Mild Cognitive Impairment (MCI) conversion prediction, achieving remarkable average area under curve (AUC) values of 0.993 and 0.845, respectively, surpassing those of traditional image-only models and non-unified multimodal models. Our experimental results confirmed the potential of the AD-Transformer as a potent tool in AD diagnosis and MCI conversion prediction. By providing a unified framework that jointly learns holistic representations of both image and non-image data, the AD-Transformer paves the way for more effective and precise clinical assessments, offering a clinically adaptable strategy for leveraging diverse data modalities in the battle against AD.

Deep multimodal saliency parcellation of cerebellar pathways: Linking microstructure and individual function through explainable multitask learning

Parcellation of human cerebellar pathways is essential for advancing our understanding of the human brain. Existing diffusion magnetic resonance imaging tractography parcellation methods have been successful in defining major cerebellar fibre tracts, while relying solely on fibre tract structure. However, each fibre tract may relay information related to multiple cognitive and motor functions of the cerebellum. Hence, it may be beneficial for parcellation to consider the potential importance of the fibre tracts for individual motor and cognitive functional performance measures. In this work, we propose a multimodal data-driven method for cerebellar pathway parcellation, which incorporates both measures of microstructure and connectivity, and measures of individual functional performance. Our method involves first training a multitask deep network to predict various cognitive and motor measures from a set of fibre tract structural features. The importance of each structural feature for predicting each functional measure is then computed, resulting in a set of structure-function saliency values that are clustered to parcellate cerebellar pathways. We refer to our method as Deep Multimodal Saliency Parcellation (DeepMSP), as it computes the saliency of structural measures for predicting cognitive and motor functional performance, with these saliencies being applied to the task of parcellation. Applying DeepMSP to a large-scale dataset from the Human Connectome Project Young Adult study (n = 1065), we found that it was feasible to identify multiple cerebellar pathway parcels with unique structure-function saliency patterns that were stable across training folds. We thoroughly experimented with all stages of the DeepMSP pipeline, including network selection, structure-function saliency representation, clustering algorithm, and cluster count. We found that a 1D convolutional neural network architecture and a transformer network architecture both performed comparably for the multitask prediction of endurance, strength, reading decoding, and vocabulary comprehension, with both architectures outperforming a fully connected network architecture. Quantitative experiments demonstrated that a proposed low-dimensional saliency representation with an explicit measure of motor versus cognitive category bias achieved the best parcellation results, while a parcel count of four was most successful according to standard cluster quality metrics. Our results suggested that motor and cognitive saliencies are distributed across the cerebellar white matter pathways. Inspection of the final k = 4 parcellation revealed that the highest-saliency parcel was most salient for the prediction of both motor and cognitive performance scores and included parts of the middle and superior cerebellar peduncles. Our proposed saliency-based parcellation framework, DeepMSP, enables multimodal, data-driven tractography parcellation. Through utilising both structural features and functional performance measures, this parcellation strategy may have the potential to enhance the study of structure-function relationships of the cerebellar pathways.

Knowledge‐based deep learning system for classifying Alzheimer's disease for multi‐task learning

Deep learning has recently become a viable approach for classifying Alzheimer's disease (AD) in medical imaging. However, existing models struggle to efficiently extract features from medical images and may squander additional information resources for illness classification. To address these issues, a deep three‐dimensional convolutional neural network incorporating multi‐task learning and attention mechanisms is proposed. An upgraded primary C3D network is utilised to create rougher low‐level feature maps. It introduces a new convolution block that focuses on the structural aspects of the magnetic resonance imaging image and another block that extracts attention weights unique to certain pixel positions in the feature map and multiplies them with the feature map output. Then, several fully connected layers are used to achieve multi‐task learning, generating three outputs, including the primary classification task. The other two outputs employ backpropagation during training to improve the primary classification job. Experimental findings show that the authors’ proposed method outperforms current approaches for classifying AD, achieving enhanced classification accuracy and other indicators on the Alzheimer's disease Neuroimaging Initiative dataset. The authors demonstrate promise for future disease classification studies.

A systematic literature review on the significance of deep learning and machine learning in predicting Alzheimer's disease

BACKGROUND

Alzheimer's disease (AD) is the most prevalent cause of dementia, characterized by a steady decline in mental, behavioral, and social abilities and impairs a person's capacity for independent functioning. It is a fatal neurodegenerative disease primarily affecting older adults.

OBJECTIVES

The purpose of this literature review is to investigate various AD detection techniques, datasets, input modalities, algorithms, libraries, and performance evaluation metrics used to determine which model or strategy may provide superior performance.

METHOD

The initial search yielded 807 papers, but only 100 research articles were chosen after applying the inclusion-exclusion criteria. This SLR analyzed research items published between January 2019 and December 2022. The ACM, Elsevier, IEEE Xplore Digital Library, PubMed, Springer and Taylor & Francis were systematically searched. The current study considers articles that used Magnetic Resonance Imaging (MRI), Positron Emission Tomography (PET), APOe4 genotype, Diffusion Tensor Imaging (DTI) and Cerebrospinal Fluid (CSF) biomarkers. The study was performed following Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) guidelines.

CONCLUSION

According to the literature survey, most studies (n = 76) used the DL strategy. The datasets used by studies were primarily derived from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database. The majority of studies (n = 73) used single-modality neuroimaging data, while the remaining used multi-modal input data. In a multi-modality approach, the combination of MRI and PET scans is commonly preferred. Also, Regarding the algorithm used, Convolution Neural Network (CNN) showed the highest accuracy, 100 %, in classifying AD vs. CN subjects whereas the SVM was the most common ML algorithm, with a maximum accuracy of 99.82 %.

Predicting recovery following stroke: Deep learning, multimodal data and feature selection using explainable AI

Machine learning offers great potential for automated prediction of post-stroke symptoms and their response to rehabilitation. Major challenges for this endeavour include the very high dimensionality of neuroimaging data, the relatively small size of the datasets available for learning and interpreting the predictive features, as well as, how to effectively combine neuroimaging and tabular data (e.g. demographic information and clinical characteristics). This paper evaluates several solutions based on two strategies. The first is to use 2D images that summarise MRI scans. The second is to select key features that improve classification accuracy. Additionally, we introduce the novel approach of training a convolutional neural network (CNN) on images that combine regions-of-interests (ROIs) extracted from MRIs, with symbolic representations of tabular data. We evaluate a series of CNN architectures (both 2D and a 3D) that are trained on different representations of MRI and tabular data, to predict whether a composite measure of post-stroke spoken picture description ability is in the aphasic or non-aphasic range. MRI and tabular data were acquired from 758 English speaking stroke survivors who participated in the PLORAS study. Each participant was assigned to one of five different groups that were matched for initial severity of symptoms, recovery time, left lesion size and the months or years post-stroke that spoken description scores were collected. Training and validation were carried out on the first four groups. The fifth (lock-box/test set) group was used to test how well model accuracy generalises to new (unseen) data. The classification accuracy for a baseline logistic regression was 0.678 based on lesion size alone, rising to 0.757 and 0.813 when initial symptom severity and recovery time were successively added. The highest classification accuracy (0.854), area under the curve (0.899) and F1 score (0.901) were observed when 8 regions of interest were extracted from each MRI scan and combined with lesion size, initial severity and recovery time in a 2D Residual Neural Network (ResNet). This was also the best model when data were limited to the 286 participants with moderate or severe initial aphasia (with area under curve = 0.865), a group that would be considered more difficult to classify. Our findings demonstrate how imaging and tabular data can be combined to achieve high post-stroke classification accuracy, even when the dataset is small in machine learning terms. We conclude by proposing how the current models could be improved to achieve even higher levels of accuracy using images from hospital scanners.

Web-based artificial intelligence to predict cognitive impairment following stroke: A multicenter study

BACKGROUND AND PURPOSE

Post-stroke cognitive impairment (PSCI) is highly prevalent in modern society. However, there is limited study implying an accurate and explainable machine learning model to predict PSCI. The aim of this study is to develop and validate a web-based artificial intelligence (AI) tool for predicting PSCI.

METHODS

The retrospective cohort study design was conducted to develop and validate a web-based prediction model. Adults who experienced a stroke between January 1, 2004, and September 30, 2017, were enrolled, and patients with PSCI were followed up from the stroke index date until their last follow-up. The model's performance metrics, including accuracy, area under the curve (AUC), recall, precision, and F1 score, were compared.

RESULTS

A total of 3209 stroke patients were included in the study. The model demonstrated an accuracy of 0.8793, AUC of 0.9200, recall of 0.6332, precision of 0.9664, and F1 score of 0.7651. In the external validation phase, the accuracy improved to 0.9039, AUC to 0.9094, recall to 0.7457, precision to 0.9168, and F1 score to 0.8224. The final model can be accessed at https://psci-calculator.my.id/.

CONCLUSION

Our results are able to produce a user-friendly interface that is useful for health practitioners to perform early prediction on PSCI. These findings also suggest that the provided AI model is reliable and can serve as a roadmap for future studies using AI models in a clinical setting.

Deep Learning for Alzheimer's Disease Prediction: A Comprehensive Review

Alzheimer's disease (AD) is a neurological disorder that significantly impairs cognitive function, leading to memory loss and eventually death. AD progresses through three stages: early stage, mild cognitive impairment (MCI) (middle stage), and dementia. Early diagnosis of Alzheimer's disease is crucial and can improve survival rates among patients. Traditional methods for diagnosing AD through regular checkups and manual examinations are challenging. Advances in computer-aided diagnosis systems (CADs) have led to the development of various artificial intelligence and deep learning-based methods for rapid AD detection. This survey aims to explore the different modalities, feature extraction methods, datasets, machine learning techniques, and validation methods used in AD detection. We reviewed 116 relevant papers from repositories including Elsevier (45), IEEE (25), Springer (19), Wiley (6), PLOS One (5), MDPI (3), World Scientific (3), Frontiers (3), PeerJ (2), Hindawi (2), IO Press (1), and other multiple sources (2). The review is presented in tables for ease of reference, allowing readers to quickly grasp the key findings of each study. Additionally, this review addresses the challenges in the current literature and emphasizes the importance of interpretability and explainability in understanding deep learning model predictions. The primary goal is to assess existing techniques for AD identification and highlight obstacles to guide future research.

Utilizing Siamese 4D-AlzNet and Transfer Learning to Identify Stages of Alzheimer's Disease

Alzheimer's disease (AD) is the general form of dementia, leading to a progressive neurological disorder characterized by memory loss due to brain cell damage. Artificial Intelligence (AI) assists in the early identification and prediction of AD patients, determining future risks and benefits for radiologists and doctors to save time and cost. Since deep learning (DL) approaches work well with massive datasets and have recently become helpful for AD detection, there remains an area for improvement in automating detection performance. Present approaches somehow addressed the challenges of limited annotated data samples for binary classification. This contrasts with prior state-of-the-art techniques, which were constrained by their incapacity to capture abstract-level information. In this paper, we proposed a Siamese 4D-AlzNet model comprised of four parallel convolutional neural network (CNN) streams (Five CNN layer blocks) and customized transfer learning models (Frozen VGG-19, Frozen VGG-16, and customized AlexNet). Siamese 4D-AlzNet was vertically and horizontally stored, and the spatial features were passed to the final layer for classification. For experiments, T1-weighted MRI images comprised of four distinct subject classes, normal control (NC), mild cognitive impairment (MCI), late mild cognitive impairment (LMCI), and AD, have been employed. Our proposed models achieved outstanding accuracy, with a remarkable 95.05% accuracy distinguishing between normal and AD subjects. The performance across remaining binary class pairs consistently exceeded 90%. We thoroughly compared our model with the latest methods using the same dataset as our reference. Our proposed model improved NC-AD and MCI-AD classification accuracy by 2% 7%.

Multi-Template Meta-Information Regularized Network for Alzheimer's Disease Diagnosis Using Structural MRI

Structural magnetic resonance imaging (sMRI) has been widely applied in computer-aided Alzheimer's disease (AD) diagnosis, owing to its capabilities in providing detailed brain morphometric patterns and anatomical features in vivo. Although previous works have validated the effectiveness of incorporating metadata (e.g., age, gender, and educational years) for sMRI-based AD diagnosis, existing methods solely paid attention to metadata-associated correlation to AD (e.g., gender bias in AD prevalence) or confounding effects (e.g., the issue of normal aging and metadata-related heterogeneity). Hence, it is difficult to fully excavate the influence of metadata on AD diagnosis. To address these issues, we constructed a novel Multi-template Meta-information Regularized Network (MMRN) for AD diagnosis. Specifically, considering diagnostic variation resulting from different spatial transformations onto different brain templates, we first regarded different transformations as data augmentation for self-supervised learning after template selection. Since the confounding effects may arise from excessive attention to meta-information owing to its correlation with AD, we then designed the modules of weakly supervised meta-information learning and mutual information minimization to learn and disentangle meta-information from learned class-related representations, which accounts for meta-information regularization for disease diagnosis. We have evaluated our proposed MMRN on two public multi-center cohorts, including the Alzheimer's Disease Neuroimaging Initiative (ADNI) with 1,950 subjects and the National Alzheimer's Coordinating Center (NACC) with 1,163 subjects. The experimental results have shown that our proposed method outperformed the state-of-the-art approaches in both tasks of AD diagnosis, mild cognitive impairment (MCI) conversion prediction, and normal control (NC) vs. MCI vs. AD classification.

Longitudinal MRI analysis using a hybrid DenseNet-BiLSTM method for Alzheimer's disease prediction

Alzheimer's disease is a progressive neurological disorder characterized by brain atrophy and cell death, leading to cognitive decline and impaired functioning. Previous research has primarily focused on using cross-sectional data for Alzheimer's disease identification, but analyzing longitudinal sequential MR images is crucial for improved diagnostic accuracy and understanding disease progression. However, existing deep learning models face challenges in learning spatial and temporal features from such data. To address these challenges, this study presents a novel hybrid DenseNet-BiLSTM method for Alzheimer's disease prediction using longitudinal MRI analysis. The proposed framework combines Convolutional DenseNet for spatial information extraction and joined BiLSTM layers for capturing temporal characteristics and relationships between longitudinal images at different time points. This approach overcomes issues like overfitting, vanishing gradients, and incomplete patient data. We evaluated the model on 684 longitudinal MRI images from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database, including normal controls, individuals with mild cognitive impairment, and Alzheimer's disease patients. The results demonstrate high classification accuracy, with 95.28% for AD/CN, 88.19% for NC/MCI, 83.51% for sMCI/pMCI, and 92.14% for MCI/AD. These findings highlight the substantial improvement in Alzheimer's disease diagnosis achieved through the utilization of longitudinal MRI images. The contributions of this study lie in both the deep learning and medical domains. In the deep learning domain, our hybrid framework effectively learns spatial and temporal features from longitudinal data, addressing the challenges associated with multi-dimensional and sequential time series data. In the medical domain, our study emphasizes the importance of analyzing baseline and longitudinal MR images for accurate diagnosis and understanding disease progression.

Shared Manifold Regularized Joint Feature Selection for Joint Classification and Regression in Alzheimer's Disease Diagnosis

In Alzheimer's disease (AD) diagnosis, joint feature selection for predicting disease labels (classification) and estimating cognitive scores (regression) with neuroimaging data has received increasing attention. In this paper, we propose a model named Shared Manifold regularized Joint Feature Selection (SMJFS) that performs classification and regression in a unified framework for AD diagnosis. For classification, unlike the existing works that build least squares regression models which are insufficient in the ability of extracting discriminative information for classification, we design an objective function that integrates linear discriminant analysis and subspace sparsity regularization for acquiring an informative feature subset. Furthermore, the local data relationships are learned according to the samples' transformed distances to exploit the local data structure adaptively. For regression, in contrast to previous works that overlook the correlations among cognitive scores, we learn a latent score space to capture the correlations and employ the latent space to design a regression model with l2,1 -norm regularization, facilitating the feature selection in regression task. Moreover, the missing cognitive scores can be recovered in the latent space for increasing the number of available training samples. Meanwhile, to capture the correlations between the two tasks and describe the local relationships between samples, we construct an adaptive shared graph to guide the subspace learning in classification and the latent cognitive score learning in regression simultaneously. An efficient iterative optimization algorithm is proposed to solve the optimization problem. Extensive experiments on three datasets validate the discriminability of the features selected by SMJFS.

SVFR: A novel slice-to-volume feature representation framework using deep neural networks and a clustering model for the diagnosis of Alzheimer's disease

Deep neural networks (DNNs) have been effective in classifying structural magnetic resonance imaging (sMRI) images for Alzheimer's disease (AD) diagnosis. In this study, we propose a novel two-phase slice-to-volume feature representation (SVFR) framework for AD diagnosis. Specifically, we design a slice-level feature extractor to automatically select informative slice images and extract their slice-level features, by combining DNN and clustering models. Furthermore, we propose a joint volume-level feature generator and classifier to hierarchically aggregate the slice-level features into volume-level features and to classify images, by devising a spatial pyramid set pooling module and a fusion module. Experimental results demonstrate the superior performance of the proposed SVFR, surpassing the majority of the state-of-the-art methods and achieving comparable results to the best-performing approach. Experimental results also showcase the efficacy of the slice-level feature extractor in the selection of informative slice images, as well as the effectiveness of the volume-level feature generator and classifier in the integration of slice-level features for image classification. The source code for this study is publicly available at https://github.com/gll89/SVFR.

Disentangling Accelerated Cognitive Decline from the Normal Aging Process and Unraveling Its Genetic Components: A Neuroimaging-Based Deep Learning Approach

Background

The progressive cognitive decline, an integral component of Alzheimer's disease (AD), unfolds in tandem with the natural aging process. Neuroimaging features have demonstrated the capacity to distinguish cognitive decline changes stemming from typical brain aging and AD between different chronological points.

Objective

To disentangle the normal aging effect from the AD-related accelerated cognitive decline and unravel its genetic components using a neuroimaging-based deep learning approach.

Methods

We developed a deep-learning framework based on a dual-loss Siamese ResNet network to extract fine-grained information from the longitudinal structural magnetic resonance imaging (MRI) data from the Alzheimer's Disease Neuroimaging Initiative (ADNI) study. We then conducted genome-wide association studies (GWAS) and post-GWAS analyses to reveal the genetic basis of AD-related accelerated cognitive decline.

Results

We used our model to process data from 1,313 individuals, training it on 414 cognitively normal people and predicting cognitive assessment for all participants. In our analysis of accelerated cognitive decline GWAS, we identified two genome-wide significant loci: APOE locus (chromosome 19 p13.32) and rs144614292 (chromosome 11 p15.1). Variant rs144614292 (G > T) has not been reported in previous AD GWA studies. It is within the intronic region of NELL1, which is expressed in neurons and plays a role in controlling cell growth and differentiation. The cell-type-specific enrichment analysis and functional enrichment of GWAS signals highlighted the microglia and immune-response pathways.

Conclusions

Our deep learning model effectively extracted relevant neuroimaging features and predicted individual cognitive decline. We reported a novel variant (rs144614292) within the NELL1 gene.

Gaussian Process-based prediction of memory performance and biomarker status in ageing and Alzheimer's disease-A systematic model evaluation

Neuroimaging markers based on Magnetic Resonance Imaging (MRI) combined with various other measures (such as genetic covariates, biomarkers, vascular risk factors, neuropsychological tests etc.) might provide useful predictions of clinical outcomes during the progression towards Alzheimer's disease (AD). The use of multiple features in predictive frameworks for clinical outcomes has become increasingly prevalent in AD research. However, many studies do not focus on systematically and accurately evaluating combinations of multiple input features. Hence, the aim of the present work is to explore and assess optimal combinations of various features for MR-based prediction of (1) cognitive status and (2) biomarker positivity with a multi-kernel learning Gaussian process framework. The explored features and parameters included (A) combinations of brain tissues, modulation, smoothing, and image resolution; (B) incorporating demographics & clinical covariates; (C) the impact of the size of the training data set; (D) the influence of dimensionality reduction and the choice of kernel types. The approach was tested in a large German cohort including 959 subjects from the multicentric longitudinal study of cognitive impairment and dementia (DELCODE). Our evaluation suggests the best prediction of memory performance was obtained for a combination of neuroimaging markers, demographics, genetic information (ApoE4) and CSF biomarkers explaining 57% of outcome variance in out-of-sample predictions. The highest performance for Aβ42/40 status classification was achieved for a combination of demographics, ApoE4, and a memory score while usage of structural MRI further improved the classification of individual patient's pTau status.

BabyNet＋＋: Fetal birth weight prediction using biometry multimodal data acquired less than 24 hours before delivery

Accurate prediction of fetal weight at birth is essential for effective perinatal care, particularly in the context of antenatal management, which involves determining the timing and mode of delivery. The current standard of care involves performing a prenatal ultrasound 24 hours prior to delivery. However, this task presents challenges as it requires acquiring high-quality images, which becomes difficult during advanced pregnancy due to the lack of amniotic fluid. In this paper, we present a novel method that automatically predicts fetal birth weight by using fetal ultrasound video scans and clinical data. Our proposed method is based on a Transformer-based approach that combines a Residual Transformer Module with a Dynamic Affine Feature Map Transform. This method leverages tabular clinical data to evaluate 2D+t spatio-temporal features in fetal ultrasound video scans. Development and evaluation were carried out on a clinical set comprising 582 2D fetal ultrasound videos and clinical records of pregnancies from 194 patients performed less than 24 hours before delivery. Our results show that our method outperforms several state-of-the-art automatic methods and estimates fetal birth weight with an accuracy comparable to human experts. Hence, automatic measurements obtained by our method can reduce the risk of errors inherent in manual measurements. Observer studies suggest that our approach may be used as an aid for less experienced clinicians to predict fetal birth weight before delivery, optimizing perinatal care regardless of the available expertise.

A Model-Agnostic Approach to Mitigate Gradient Interference for Multi-Task Learning

Multitask learning (MTL) is a powerful technique for jointly learning multiple tasks. However, it is difficult to achieve a tradeoff between tasks during iterative training, as some tasks may compete with each other. Existing methods manually design specific network models to mitigate task conflicts, but they require considerable manual effort and prior knowledge about task relationships to tune the model so as to obtain the best performance for each task. Moreover, few works have offered formal descriptions of task conflicts and theoretical explanations for the cause of task conflict problems. In this article, we provide a formal description of task conflicts that are caused by the gradient interference problem of tasks. To alleviate this issue, we propose a novel model-agnostic approach to mitigate gradient interference (MAMG) by designing a gradient clipping rule that directly modifies the interfering components on the gradient interfering direction. Specifically, MAMG is model-agnostic and thus it can be applied to a large number of multitask models. We also theoretically prove the convergence of MAMG and its superiority to existing MTL methods. We evaluate our method on a variety of real-world large datasets, and extensive experimental results confirm that MAMG can outperform some state-of-the-art algorithms on different types of tasks and can be easily applied to various methods.

Structural and functional connectome relationships in early childhood

There is strong evidence that the functional connectome is highly related to the white matter connectome in older children and adults, though little is known about structure-function relationships in early childhood. We investigated the development of cortical structure-function coupling in children longitudinally scanned at 1, 2, 4, and 6 years of age (N = 360) and in a comparison sample of adults (N = 89). We also applied a novel graph convolutional neural network-based deep learning model with a new loss function to better capture inter-subject heterogeneity and predict an individual's functional connectivity from the corresponding structural connectivity. We found regional patterns of structure-function coupling in early childhood that were consistent with adult patterns. In addition, our deep learning model improved the prediction of individual functional connectivity from its structural counterpart compared to existing models.

A common spectrum underlying brain disorders across lifespan revealed by deep learning on brain networks

Brain disorders in the early and late life of humans potentially share pathological alterations in brain functions. However, the key neuroimaging evidence remains unrevealed for elucidating such commonness and the relationships among these disorders. To explore this puzzle, we build a restricted single-branch deep learning model, using multi-site functional magnetic resonance imaging data (N = 4,410, 6 sites), for classifying 5 different early- and late-life brain disorders from healthy controls (cognitively unimpaired). Our model achieves 62.6 ± 1.9% overall classification accuracy and thus supports us in detecting a set of commonly affected functional subnetworks, including default mode, executive control, visual, and limbic networks. In the deep-layer representation of data, we observe young and aging patients with disorders are continuously distributed, which is in line with the clinical concept of the "spectrum of disorders." The relationships among brain disorders from the revealed spectrum promote the understanding of disorder comorbidities and time associations in the lifespan.

Advancements in computer-assisted diagnosis of Alzheimer's disease: A comprehensive survey of neuroimaging methods and AI techniques for early detection

Alzheimer's Disease (AD) is a brain disorder that causes the brain to shrink and eventually causes brain cells to die. This neurological condition progressively hampers cognitive and memory functions, along with the ability to carry out fundamental tasks over time. From the symptoms it is very difficult to detect during its early stage. It has become necessary to develop a computer assisted diagnostic models for the early AD detection. This survey work, discussed about a review of 110 published AD detection methods and techniques from the year 2011 to till-date. This study lies in its comprehensive exploration of AD detection methods using a range of artificial intelligence (AI) techniques and neuroimaging modalities. By collecting and analysing 50 papers related to AD diagnosis datasets, the study provides a comprehensive understanding of the diversity of input types, subjects, and classes used in AD research. Summarizing 60 papers on methodologies gives researchers a succinct overview of various approaches that contribute to enhancing detection accuracy. From the review, data are acquired and pre-processed form multiple modalities of neuroimaging. This paper mainly focused on review of different datasets used, various feature extraction methods, parameters used in neuro images. To diagnosis the Alzheimer's disease, the existing methods utilized three most common artificial intelligence techniques such as machine learning, deep learning, and transfer learning. We conclude this survey work by providing future perspectives for AD diagnosis at early stage.

Diagnosis of Alzheimer's disease by joining dual attention CNN and MLP based on structural MRIs, clinical and genetic data

Alzheimer's disease (AD) is an irreversible central nervous degenerative disease, while mild cognitive impairment (MCI) is a precursor state of AD. Accurate early diagnosis of AD is conducive to the prevention and early intervention treatment of AD. Although some computational methods have been developed for AD diagnosis, most employ only neuroimaging, ignoring other data (e.g., genetic, clinical) that may have potential disease information. In addition, the results of some methods lack interpretability. In this work, we proposed a novel method (called DANMLP) of joining dual attention convolutional neural network (CNN) and multilayer perceptron (MLP) for computer-aided AD diagnosis by integrating multi-modality data of the structural magnetic resonance imaging (sMRI), clinical data (i.e., demographics, neuropsychology), and APOE genetic data. Our DANMLP consists of four primary components: (1) the Patch-CNN for extracting the image characteristics from each local patch, (2) the position self-attention block for capturing the dependencies between features within a patch, (3) the channel self-attention block for capturing dependencies of inter-patch features, (4) two MLP networks for extracting the clinical features and outputting the AD classification results, respectively. Compared with other state-of-the-art methods in the 5CV test, DANMLP achieves 93% and 82.4% classification accuracy for the AD vs. MCI and MCI vs. NC tasks on the ADNI database, which is 0.2%∼15.2% and 3.4%∼26.8% higher than that of other five methods, respectively. The individualized visualization of focal areas can also help clinicians in the early diagnosis of AD. These results indicate that DANMLP can be effectively used for diagnosing AD and MCI patients.

Disentangling accelerated cognitive decline from the normal aging process and unraveling its genetic components: A neuroimaging-based deep learning approach

Background

The progressive cognitive decline that is an integral component of AD unfolds in tandem with the natural aging process. Neuroimaging features have demonstrated the capacity to distinguish cognitive decline changes stemming from typical brain aging and Alzheimer's disease between different chronological points.

Methods

We developed a deep-learning framework based on dual-loss Siamese ResNet network to extract fine-grained information from the longitudinal structural magnetic resonance imaging (MRI) data from the Alzheimer's Disease Neuroimaging Initiative (ADNI) study. We then conducted genome-wide association studies (GWAS) and post-GWAS analyses to reveal the genetic basis of AD-related accelerated cognitive decline.

Results

We used our model to process data from 1,313 individuals, training it on 414 cognitively normal people and predicting cognitive assessment for all participants. In our analysis of accelerated cognitive decline GWAS, we identified two genome-wide significant loci: APOE locus (chromosome 19 p13.32) and rs144614292 (chromosome 11 p15.1). Variant rs144614292 (G>T) has not been reported in previous AD GWA studies. It is within the intronic region of NELL1, which is expressed in neuron and plays a role in controlling cell growth and differentiation. In addition, MUC7 and PROL1/OPRPNon chromosome 4 were significant at the gene level. The cell-type-specific enrichment analysis and functional enrichment of GWAS signals highlighted the microglia and immune-response pathways. Furthermore, we found that the cognitive decline slope GWAS was positively correlated with previous AD GWAS.

Conclusion

Our deep learning model was demonstrated effective on extracting relevant neuroimaging features and predicting individual cognitive decline. We reported a novel variant (rs144614292) within the NELL1 gene. Our approach has the potential to disentangle accelerated cognitive decline from the normal aging process and to determine its related genetic factors, leveraging opportunities for early intervention.

End-to-end automatic pathology localization for Alzheimer's disease diagnosis using structural MRI

Structural magnetic resonance imaging (sMRI) is an essential part of the clinical assessment of patients at risk of Alzheimer dementia. One key challenge in sMRI-based computer-aided dementia diagnosis is to localize local pathological regions for discriminative feature learning. Existing solutions predominantly depend on generating saliency maps for pathology localization and handle the localization task independently of the dementia diagnosis task, leading to a complex multi-stage training pipeline that is hard to optimize with weakly-supervised sMRI-level annotations. In this work, we aim to simplify the pathology localization task and construct an end-to-end automatic localization framework (AutoLoc) for Alzheimer's disease diagnosis. To this end, we first present an efficient pathology localization paradigm that directly predicts the coordinate of the most disease-related region in each sMRI slice. Then, we approximate the non-differentiable patch-cropping operation with the bilinear interpolation technique, which eliminates the barrier to gradient backpropagation and thus enables the joint optimization of localization and diagnosis tasks. Extensive experiments on commonly used ADNI and AIBL datasets demonstrate the superiority of our method. Especially, we achieve 93.38% and 81.12% accuracy on Alzheimer's disease classification and mild cognitive impairment conversion prediction tasks, respectively. Several important brain regions, such as rostral hippocampus and globus pallidus, are identified to be highly associated with Alzheimer's disease.

Fine-Grained and Multiple Classification for Alzheimer's Disease With Wavelet Convolution Unit Network

In this article, we propose a novel wavelet convolution unit for the image-oriented neural network to integrate wavelet analysis with a vanilla convolution operator to extract deep abstract features more efficiently. On one hand, in order to acquire non-local receptive fields and avoid information loss, we define a new convolution operation by composing a traditional convolution function and approximate and detailed representations after single-scale wavelet decomposition of source images. On the other hand, multi-scale wavelet decomposition is introduced to obtain more comprehensive multi-scale feature information. Then, we fuse all these cross-scale features to improve the problem of inaccurate localization of singular points. Given the novel wavelet convolution unit, we further design a network based on it for fine-grained Alzheimer's disease classifications (i.e., Alzheimer's disease, Normal controls, early mild cognitive impairment, late mild cognitive impairment). Up to now, only a few methods have studied one or several fine-grained classifications, and even fewer methods can achieve both fine-grained and multi-class classifications. We adopt the novel network and diffuse tensor images to achieve fine-grained classifications, which achieved state-of-the-art accuracy for all eight kinds of fine-grained classifications, up to 97.30%, 95.78%, 95.00%, 94.00%, 97.89%, 95.71%, 95.07%, 93.79%. In order to build a reference standard for Alzheimer's disease classifications, we actually implemented all twelve coarse-grained and fine-grained classifications. The results show that the proposed method achieves solidly high accuracy for them. Its classification ability greatly exceeds any kind of existing Alzheimer's disease classification method.

Comprehensive assessment of the anterior segment in refraction corrected OCT based on multitask learning

Anterior segment diseases are among the leading causes of irreversible blindness. However, a method capable of recognizing all important anterior segment structures for clinical diagnosis is lacking. By sharing the knowledge learned from each task, we proposed a fully automated multitask deep learning method that allows for simultaneous segmentation and quantification of all major anterior segment structures, including the iris, lens, cornea, as well as implantable collamer lens (ICL) and intraocular lens (IOL), and meanwhile for landmark detection of scleral spur and iris root in anterior segment OCT (AS-OCT) images. In addition, we proposed a refraction correction method to correct for the true geometry of the anterior segment distorted by light refraction during OCT imaging. 1251 AS-OCT images from 180 patients were collected and were used to train and test the model. Experiments demonstrated that our proposed network was superior to state-of-the-art segmentation and landmark detection methods, and close agreement was achieved between manually and automatically computed clinical parameters associated with anterior chamber, pupil, iris, ICL, and IOL. Finally, as an example, we demonstrated how our proposed method can be applied to facilitate the clinical evaluation of cataract surgery.

Multi-modal cross-attention network for Alzheimer's disease diagnosis with multi-modality data

Alzheimer's disease (AD) is a neurodegenerative disorder, the most common cause of dementia, so the accurate diagnosis of AD and its prodromal stage mild cognitive impairment (MCI) is significant. Recent studies have demonstrated that multiple neuroimaging and biological measures contain complementary information for diagnosis. Many existing multi-modal models based on deep learning simply concatenate each modality's features despite substantial differences in representation spaces. In this paper, we propose a novel multi-modal cross-attention AD diagnosis (MCAD) framework to learn the interaction between modalities for better playing their complementary roles for AD diagnosis with multi-modal data including structural magnetic resonance imaging (sMRI), fluorodeoxyglucose-positron emission tomography (FDG-PET) and cerebrospinal fluid (CSF) biomarkers. Specifically, the imaging and non-imaging representations are learned by the image encoder based on cascaded dilated convolutions and CSF encoder, respectively. Then, a multi-modal interaction module is introduced, which takes advantage of cross-modal attention to integrate imaging and non-imaging information and reinforce relationships between these modalities. Moreover, an extensive objective function is designed to reduce the discrepancy between modalities for effectively fusing the features of multi-modal data, which could further improve the diagnosis performance. We evaluate the effectiveness of our proposed method on the ADNI dataset, and the extensive experiments demonstrate that our MCAD achieves superior performance for multiple AD-related classification tasks, compared to several competing methods. Also, we investigate the importance of cross-attention and the contribution of each modality to the diagnostics performance. The experimental results demonstrate that combining multi-modality data via cross-attention is helpful for accurate AD diagnosis.

[Research on mild cognitive impairment diagnosis based on Bayesian optimized long-short-term neural network model]

The recurrent neural network architecture improves the processing ability of time-series data. However, issues such as exploding gradients and poor feature extraction limit its application in the automatic diagnosis of mild cognitive impairment (MCI). This paper proposed a research approach for building an MCI diagnostic model using a Bayesian-optimized bidirectional long short-term memory network (BO-BiLSTM) to address this problem. The diagnostic model was based on a Bayesian algorithm and combined prior distribution and posterior probability results to optimize the BO-BiLSTM network hyperparameters. It also used multiple feature quantities that fully reflected the cognitive state of the MCI brain, such as power spectral density, fuzzy entropy, and multifractal spectrum, as the input of the diagnostic model to achieve automatic MCI diagnosis. The results showed that the feature-fused Bayesian-optimized BiLSTM network model achieved an MCI diagnostic accuracy of 98.64% and effectively completed the diagnostic assessment of MCI. In conclusion, based on this optimization, the long short-term neural network model has achieved automatic diagnostic assessment of MCI, providing a new diagnostic model for intelligent diagnosis of MCI.

Predictive Models for Health Deterioration: Understanding Disease Pathways for Personalized Medicine

Artificial intelligence (AI) and machine learning (ML) methods are currently widely employed in medicine and healthcare. A PubMed search returns more than 100,000 articles on these topics published between 2018 and 2022 alone. Notwithstanding several recent reviews in various subfields of AI and ML in medicine, we have yet to see a comprehensive review around the methods' use in longitudinal analysis and prediction of an individual patient's health status within a personalized disease pathway. This review seeks to fill that gap. After an overview of the AI and ML methods employed in this field and of specific medical applications of models of this type, the review discusses the strengths and limitations of current studies and looks ahead to future strands of research in this field. We aim to enable interested readers to gain a detailed impression of the research currently available and accordingly plan future work around predictive models for deterioration in health status.

A unique color-coded visualization system with multimodal information fusion and deep learning in a longitudinal study of Alzheimer's disease

PURPOSE

Automated diagnosis and prognosis of Alzheimer's Disease remain a challenging problem that machine learning (ML) techniques have attempted to resolve in the last decade. This study introduces a first-of-its-kind color-coded visualization mechanism driven by an integrated ML model to predict disease trajectory in a 2-year longitudinal study. The main aim of this study is to help capture visually in 2D and 3D renderings the diagnosis and prognosis of AD, therefore augmenting our understanding of the processes of multiclass classification and regression analysis.

METHOD

The proposed method, Machine Learning for Visualizing AD (ML4VisAD), is designed to predict disease progression through a visual output. This newly developed model takes baseline measurements as input to generate a color-coded visual image that reflects disease progression at different time points. The architecture of the network relies on convolutional neural networks. With 1123 subjects selected from the ADNI QT-PAD dataset, we use a 10-fold cross-validation process to evaluate the method. Multimodal inputs* include neuroimaging data (MRI, PET), neuropsychological test scores (excluding MMSE, CDR-SB, and ADAS to avoid bias), cerebrospinal fluid (CSF) biomarkers with measures of amyloid beta (ABETA), phosphorylated tau protein (PTAU), total tau protein (TAU), and risk factors that include age, gender, years of education, and ApoE4 gene.

FINDINGS/RESULTS

Based on subjective scores reached by three raters, the results showed an accuracy of 0.82 ± 0.03 for a 3-way classification and 0.68 ± 0.05 for a 5-way classification. The visual renderings were generated in 0.08 msec for a 23 × 23 output image and in 0.17 ms for a 45 × 45 output image. Through visualization, this study (1) demonstrates that the ML visual output augments the prospects for a more accurate diagnosis and (2) highlights why multiclass classification and regression analysis are incredibly challenging. An online survey was conducted to gauge this visualization platform's merits and obtain valuable feedback from users. All implementation codes are shared online on GitHub.

CONCLUSION

This approach makes it possible to visualize the many nuances that lead to a specific classification or prediction in the disease trajectory, all in context to multimodal measurements taken at baseline. This ML model can serve as a multiclass classification and prediction model while reinforcing the diagnosis and prognosis capabilities by including a visualization platform.

Vision transformers for the prediction of mild cognitive impairment to Alzheimer's disease progression using mid-sagittal sMRI

Background

Alzheimer's disease (AD) is one of the most common causes of neurodegenerative disease affecting over 50 million people worldwide. However, most AD diagnosis occurs in the moderate to late stage, which means that the optimal time for treatment has already passed. Mild cognitive impairment (MCI) is an intermediate state between cognitively normal people and AD patients. Therefore, the accurate prediction in the conversion process of MCI to AD may allow patients to start preventive intervention to slow the progression of the disease. Nowadays, neuroimaging techniques have been developed and are used to determine AD-related structural biomarkers. Deep learning approaches have rapidly become a key methodology applied to these techniques to find biomarkers.

Methods

In this study, we aimed to investigate an MCI-to-AD prediction method using Vision Transformers (ViT) to structural magnetic resonance images (sMRI). The Alzheimer's Disease Neuroimaging Initiative (ADNI) database containing 598 MCI subjects was used to predict MCI subjects' progression to AD. There are three main objectives in our study: (i) to propose an MRI-based Vision Transformers approach for MCI to AD progression classification, (ii) to evaluate the performance of different ViT architectures to obtain the most advisable one, and (iii) to visualize the brain region mostly affect the prediction of deep learning approach to MCI progression.

Results

Our method achieved state-of-the-art classification performance in terms of accuracy (83.27%), specificity (85.07%), and sensitivity (81.48%) compared with a set of conventional methods. Next, we visualized the brain regions that mostly contribute to the prediction of MCI progression for interpretability of the proposed model. The discriminative pathological locations include the thalamus, medial frontal, and occipital-corroborating the reliability of our model.

Conclusion

In conclusion, our methods provide an effective and accurate technique for the prediction of MCI conversion to AD. The results obtained in this study outperform previous reports using the ADNI collection, and it suggests that sMRI-based ViT could be efficiently applied with a considerable potential benefit for AD patient management. The brain regions mostly contributing to prediction, in conjunction with the identified anatomical features, will support the building of a robust solution for other neurodegenerative diseases in future.

Machine learning-based brief version of the Caregiver-Teacher Report Form for preschoolers

BACKGROUND

The Caregiver-Teacher Report Form of the Child Behavior Checklist for Ages 1½-5 (C-TRF) is a widely used checklist to identify emotional and behavioral problems in preschoolers. However, the 100-item C-TRF restricts its utility.

AIMS

This study aimed to develop a machine learning-based short-form of the C-TRF (C-TRF-ML).

METHODS AND PROCEDURES

Three steps were executed. First, we split the data into three datasets in a ratio of 3:1:1 for training, validation, and cross-validation, respectively. Second, we selected a shortened item set and trained a scoring algorithm using joint learning for classification and regression using the training dataset. Then, we evaluated the similarity of scores between the C-TRF-ML and the C-TRF by r-squared and weighted kappa values using the validation dataset. Third, we cross-validated the C-TRF-ML by calculating the r-squared and weighted kappa values using the cross-validation dataset.

OUTCOMES AND RESULTS

Data of 363 children were analyzed. Thirty-six items of the C-TRF were retained. The r-squared values of C-TRF-ML scores were 0.86-0.96 in the cross-validation dataset. Weighted kappa values of the syndrome/problem grading were 0.72-0.94 in the cross-validation dataset.

CONCLUSIONS AND IMPLICATIONS

The C-TRF-ML had about 60 % fewer items than the C-TRF but yielded comparable scores with the C-TRF.

Multi-task deep learning for medical image computing and analysis: A review

The renaissance of deep learning has provided promising solutions to various tasks. While conventional deep learning models are constructed for a single specific task, multi-task deep learning (MTDL) that is capable to simultaneously accomplish at least two tasks has attracted research attention. MTDL is a joint learning paradigm that harnesses the inherent correlation of multiple related tasks to achieve reciprocal benefits in improving performance, enhancing generalizability, and reducing the overall computational cost. This review focuses on the advanced applications of MTDL for medical image computing and analysis. We first summarize four popular MTDL network architectures (i.e., cascaded, parallel, interacted, and hybrid). Then, we review the representative MTDL-based networks for eight application areas, including the brain, eye, chest, cardiac, abdomen, musculoskeletal, pathology, and other human body regions. While MTDL-based medical image processing has been flourishing and demonstrating outstanding performance in many tasks, in the meanwhile, there are performance gaps in some tasks, and accordingly we perceive the open challenges and the perspective trends. For instance, in the 2018 Ischemic Stroke Lesion Segmentation challenge, the reported top dice score of 0.51 and top recall of 0.55 achieved by the cascaded MTDL model indicate further research efforts in high demand to escalate the performance of current models.

Structural Attention Graph Neural Network for Diagnosis and Prediction of COVID-19 Severity

With rapid worldwide spread of Coronavirus Disease 2019 (COVID-19), jointly identifying severe COVID-19 cases from mild ones and predicting the conversion time (from mild to severe) is essential to optimize the workflow and reduce the clinician's workload. In this study, we propose a novel framework for COVID-19 diagnosis, termed as Structural Attention Graph Neural Network (SAGNN), which can combine the multi-source information including features extracted from chest CT, latent lung structural distribution, and non-imaging patient information to conduct diagnosis of COVID-19 severity and predict the conversion time from mild to severe. Specifically, we first construct a graph to incorporate structural information of the lung and adopt graph attention network to iteratively update representations of lung segments. To distinguish different infection degrees of left and right lungs, we further introduce a structural attention mechanism. Finally, we introduce demographic information and develop a multi-task learning framework to jointly perform both tasks of classification and regression. Experiments are conducted on a real dataset with 1687 chest CT scans, which includes 1328 mild cases and 359 severe cases. Experimental results show that our method achieves the best classification (e.g., 86.86% in terms of Area Under Curve) and regression (e.g., 0.58 in terms of Correlation Coefficient) performance, compared with other comparison methods.

Multi-tasking deep network for tinnitus classification and severity prediction from multimodal structural MR images

Objective:Subjective tinnitus is an auditory phantom perceptual disorder without an objective biomarker. Fast and efficient diagnostic tools will advance clinical practice by detecting or confirming the condition, tracking change in severity, and monitoring treatment response. Motivated by evidence of subtle anatomical, morphological, or functional information in magnetic resonance images of the brain, we examine data-driven machine learning methods for joint tinnitus classification (tinnitus or no tinnitus) and tinnitus severity prediction.Approach:We propose a deep multi-task multimodal framework for tinnitus classification and severity prediction using structural MRI (sMRI) data. To leverage complementary information multimodal neuroimaging data, we integrate two modalities of three-dimensional sMRI-T1 weighted (T1w) and T2 weighted (T2w) images. To explore the key components in the MR images that drove task performance, we segment both T1w and T2w images into three different components-cerebrospinal fluid, grey matter and white matter, and evaluate performance of each segmented image.Main results:Results demonstrate that our multimodal framework capitalizes on the information across both modalities (T1w and T2w) for the joint task of tinnitus classification and severity prediction.Significance:Our model outperforms existing learning-based and conventional methods in terms of accuracy, sensitivity, specificity, and negative predictive value.

Retinal Structure Detection in OCTA Image via Voting-Based Multitask Learning

Automated detection of retinal structures, such as retinal vessels (RV), the foveal avascular zone (FAZ), and retinal vascular junctions (RVJ), are of great importance for understanding diseases of the eye and clinical decision-making. In this paper, we propose a novel Voting-based Adaptive Feature Fusion multi-task network (VAFF-Net) for joint segmentation, detection, and classification of RV, FAZ, and RVJ in optical coherence tomography angiography (OCTA). A task-specific voting gate module is proposed to adaptively extract and fuse different features for specific tasks at two levels: features at different spatial positions from a single encoder, and features from multiple encoders. In particular, since the complexity of the microvasculature in OCTA images makes simultaneous precise localization and classification of retinal vascular junctions into bifurcation/crossing a challenging task, we specifically design a task head by combining the heatmap regression and grid classification. We take advantage of three different en face angiograms from various retinal layers, rather than following existing methods that use only a single en face. We carry out extensive experiments on three OCTA datasets acquired using different imaging devices, and the results demonstrate that the proposed method performs on the whole better than either the state-of-the-art single-purpose methods or existing multi-task learning solutions. We also demonstrate that our multi-task learning method generalizes across other imaging modalities, such as color fundus photography, and may potentially be used as a general multi-task learning tool. We also construct three datasets for multiple structure detection, and part of these datasets with the source code and evaluation benchmark have been released for public access.

Single-shot T2 mapping via multi-echo-train multiple overlapping-echo detachment planar imaging and multitask deep learning

BACKGROUND

Quantitative magnetic resonance imaging provides robust biomarkers in clinics. Nevertheless, the lengthy scan time reduces imaging throughput and increases the susceptibility of imaging results to motion. In this context, a single-shot T₂ mapping method based on multiple overlapping-echo detachment (MOLED) planar imaging was presented, but the relatively small echo time range limits its accuracy, especially in tissues with large T₂ .

PURPOSE

In this work we proposed a novel single-shot method, Multi-Echo-Train Multiple OverLapping-Echo Detachment (METMOLED) planar imaging, to accommodate a large range of T₂ quantification without additional measurements to rectify signal degeneration arisen from refocusing pulse imperfection.

METHODS

Multiple echo-train techniques were integrated into the MOLED sequence to capture larger TE information. Maps of T₂ , B₁ , and spin density were reconstructed synchronously from acquired METMOLED data via multitask deep learning. A typical U-Net was trained with 3000/600 synthetic data with geometric/brain patterns to learn the mapping relationship between METMOLED signals and quantitative maps. The refocusing pulse imperfection was settled through the inherent information of METMOLED data and auxiliary tasks.

RESULTS

Experimental results on the digital brain (structural similarity (SSIM) index = 0.975/0.991/0.988 for MOLED/METMOLED-2/METMOLED-3, hyphenated number denotes the number of echo-trains), physical phantom (the slope of linear fitting with reference T₂ map = 1.047/1.017/1.006 for MOLED/METMOLED-2/METMOLED-3), and human brain (Pearson's correlation coefficient (PCC) = 0.9581/0.9760/0.9900 for MOLED/METMOLED-2/METMOLED-3) demonstrated that the METMOLED improved the quantitative accuracy and the tissue details in contrast to the MOLED. These improvements were more pronounced in tissues with large T₂ and in application scenarios with high temporal resolution (PCC = 0.8692/0.9465/0.9743 for MOLED/METMOLED-2/METMOLED-3). Moreover, the METMOLED could rectify the signal deviations induced by the non-ideal slice profiles of refocusing pulses without additional measurements. A preliminary measurement also demonstrated that the METMOLED is highly repeatable (mean coefficient of variation (CV) = 1.65%).

CONCLUSIONS

METMOLED breaks the restriction of echo-train length to TE and implements unbiased T₂ estimates in an extensive range. Furthermore, it corrects the effect of refocusing pulse inaccuracy without additional measurements or signal post-processing, thus retaining its single-shot characteristic. This technique would be beneficial for accurate T₂ quantification.

A survey of data element perspective: Application of artificial intelligence in health big data

Artificial intelligence (AI) based on the perspective of data elements is widely used in the healthcare informatics domain. Large amounts of clinical data from electronic medical records (EMRs), electronic health records (EHRs), and electroencephalography records (EEGs) have been generated and collected at an unprecedented speed and scale. For instance, the new generation of wearable technologies enables easy-collecting peoples’ daily health data such as blood pressure, blood glucose, and physiological data, as well as the application of EHRs documenting large amounts of patient data. The cost of acquiring and processing health big data is expected to reduce dramatically with the help of AI technologies and open-source big data platforms such as Hadoop and Spark. The application of AI technologies in health big data presents new opportunities to discover the relationship among living habits, sports, inheritances, diseases, symptoms, and drugs. Meanwhile, with the development of fast-growing AI technologies, many promising methodologies are proposed in the healthcare field recently. In this paper, we review and discuss the application of machine learning (ML) methods in health big data in two major aspects: (1) Special features of health big data including multimodal, incompletion, time validation, redundancy, and privacy. (2) ML methodologies in the healthcare field including classification, regression, clustering, and association. Furthermore, we review the recent progress and breakthroughs of automatic diagnosis in health big data and summarize the challenges, gaps, and opportunities to improve and advance automatic diagnosis in the health big data field.

Diffusion Kernel Attention Network for Brain Disorder Classification

Constructing and analyzing functional brain networks (FBN) has become a promising approach to brain disorder classification. However, the conventional successive construct-and-analyze process would limit the performance due to the lack of interactions and adaptivity among the subtasks in the process. Recently, Transformer has demonstrated remarkable performance in various tasks, attributing to its effective attention mechanism in modeling complex feature relationships. In this paper, for the first time, we develop Transformer for integrated FBN modeling, analysis and brain disorder classification with rs-fMRI data by proposing a Diffusion Kernel Attention Network to address the specific challenges. Specifically, directly applying Transformer does not necessarily admit optimal performance in this task due to its extensive parameters in the attention module against the limited training samples usually available. Looking into this issue, we propose to use kernel attention to replace the original dot-product attention module in Transformer. This significantly reduces the number of parameters to train and thus alleviates the issue of small sample while introducing a non-linear attention mechanism to model complex functional connections. Another limit of Transformer for FBN applications is that it only considers pair-wise interactions between directly connected brain regions but ignores the important indirect connections. Therefore, we further explore diffusion process over the kernel attention to incorporate wider interactions among indirectly connected brain regions. Extensive experimental study is conducted on ADHD-200 data set for ADHD classification and on ADNI data set for Alzheimer's disease classification, and the results demonstrate the superior performance of the proposed method over the competing methods.

Environmental and Geographical (EG) Image Classification Using FLIM and CNN Algorithms

Intelligent machines have grown in importance in recent years in object recognition in terms of their ability to envision, comprehend, and reach decisions. There are a lot of complicated algorithms that accomplish AI utilities. In addition to their use in the medical industry, these methods of object recognition have a wide range of other fields, most notably industries, in which they can be applied. In contrast to the proposed calculation, the proposed calculation is less complex and more accurate under certain SNR conditions. In the deep nervous tissue fine-tuning discriminator, phantom highlights and binding highlights are separated as sources; modified direct components are used as neuronal activation abilities; and cross entropy is used as unfortunate abilities. Optimized recognition of profound nervous tissue builds profound and periodic nervous tissue for regulatory confirmation of the corresponding signal.

Multi-Task Weakly-Supervised Attention Network for Dementia Status Estimation With Structural MRI

Accurate prediction of clinical scores (of neuropsychological tests) based on noninvasive structural magnetic resonance imaging (MRI) helps understand the pathological stage of dementia (e.g., Alzheimer's disease (AD)) and forecast its progression. Existing machine/deep learning approaches typically preselect dementia-sensitive brain locations for MRI feature extraction and model construction, potentially leading to undesired heterogeneity between different stages and degraded prediction performance. Besides, these methods usually rely on prior anatomical knowledge (e.g., brain atlas) and time-consuming nonlinear registration for the preselection of brain locations, thereby ignoring individual-specific structural changes during dementia progression because all subjects share the same preselected brain regions. In this article, we propose a multi-task weakly-supervised attention network (MWAN) for the joint regression of multiple clinical scores from baseline MRI scans. Three sequential components are included in MWAN: 1) a backbone fully convolutional network for extracting MRI features; 2) a weakly supervised dementia attention block for automatically identifying subject-specific discriminative brain locations; and 3) an attention-aware multitask regression block for jointly predicting multiple clinical scores. The proposed MWAN is an end-to-end and fully trainable deep learning model in which dementia-aware holistic feature learning and multitask regression model construction are integrated into a unified framework. Our MWAN method was evaluated on two public AD data sets for estimating clinical scores of mini-mental state examination (MMSE), clinical dementia rating sum of boxes (CDRSB), and AD assessment scale cognitive subscale (ADAS-Cog). Quantitative experimental results demonstrate that our method produces superior regression performance compared with state-of-the-art methods. Importantly, qualitative results indicate that the dementia-sensitive brain locations automatically identified by our MWAN method well retain individual specificities and are biologically meaningful.

A robust framework to investigate the reliability and stability of explainable artificial intelligence markers of Mild Cognitive Impairment and Alzheimer's Disease

In clinical practice, several standardized neuropsychological tests have been designed to assess and monitor the neurocognitive status of patients with neurodegenerative diseases such as Alzheimer's disease. Important research efforts have been devoted so far to the development of multivariate machine learning models that combine the different test indexes to predict the diagnosis and prognosis of cognitive decline with remarkable results. However, less attention has been devoted to the explainability of these models. In this work, we present a robust framework to (i) perform a threefold classification between healthy control subjects, individuals with cognitive impairment, and subjects with dementia using different cognitive indexes and (ii) analyze the variability of the explainability SHAP values associated with the decisions taken by the predictive models. We demonstrate that the SHAP values can accurately characterize how each index affects a patient's cognitive status. Furthermore, we show that a longitudinal analysis of SHAP values can provide effective information on Alzheimer's disease progression.

DAFT: A universal module to interweave tabular data and 3D images in CNNs

Prior work on Alzheimer's Disease (AD) has demonstrated that convolutional neural networks (CNNs) can leverage the high-dimensional image information for diagnosing patients. Beside such data-driven approaches, many established biomarkers exist and are typically represented as tabular data, such as demographics, genetic alterations, or laboratory measurements from cerebrospinal fluid. However, little research has focused on the effective integration of tabular data into existing CNN architectures to improve patient diagnosis. We introduce the Dynamic Affine Feature Map Transform (DAFT), a general-purpose module for CNNs that incites or represses high-level concepts learned from a 3D image by conditioning feature maps of a convolutional layer on both a patient's image and tabular clinical information. This is achieved by using an auxiliary neural network that outputs a scaling factor and offset to dynamically apply an affine transformation to the feature maps of a convolutional layer. In our experiments on AD diagnosis and time-to-dementia prediction, we show that the DAFT is highly effective in combining 3D image and tabular information by achieving a mean balanced accuracy of 0.622 for diagnosis, and mean c-index of 0.748 for time-to-dementia prediction, thus outperforming all baseline methods. Finally, our extensive ablation study and empirical experiments reveal that the performance improvement due to the DAFT is robust with respect to many design choices.

Alzheimer's disease diagnosis via multimodal feature fusion

Alzheimer's disease (AD) is the most common neurodegenerative disorder in the elderly. Early diagnosis of AD plays a vital role in slowing down the progress of AD because there is no effective drug to treat the disease. Some deep learning models have recently been presented for AD diagnosis and have more satisfactory performance than classic machine learning methods. Nevertheless, most of the existing computer-aided diagnostic models used neuroimaging features for diagnosis, ignoring patients' clinical and biological information. This makes the AD diagnosis inaccurate. In this study, we propose a novel multimodal feature transformation and fusion model for AD diagnosis. The feature transformation aims to avoid the difference in feature dimensions between different modal data and further mine the significant features for AD diagnosis. A geometric algebra-based feature extension method is proposed to obtain different levels of high-dimensional features from patients' clinical and personal biological data. Then, an influence degree-based feature filtration algorithm is proposed to filtrate those features that have no apparent guiding significance for AD diagnosis. Finally, an ANN (Artificial Neural Network)-based framework is designed to fuse transformed features with neuroimaging features extracted by CNN (Convolutional Neural Network) for AD diagnosis. The more in-depth feature mining of patients' clinical information and biological information can significantly improve the performance of computer-aided AD diagnosis. The experiments are obtained on the ADNI dataset. Our proposed model can converge faster and achieves 96.2% accuracy in AD diagnostic task and 87.4% accuracy in MCI (Mild Cognitive Impairment) diagnostic task. Compared with other methods, our proposed approach has an excellent performance in AD diagnosis and surpasses SOTA (state-of-the-art) methods. Therefore, our model can provide more reasonable suggestions for clinicians to diagnose and treat disease.

Hierarchical deep learning model to simulate phytoplankton at phylum/class and genus levels and zooplankton at the genus level

Harmful algal blooms (HABs) have become a global issue, affecting public health and water industries in numerous countries. Because funds for monitoring HABs are limited, model development may be an alternative approach for understanding and managing HABs. Continuous monitoring based on grab sampling is time-consuming, costly, and labor-intensive. However, improving simulation performance remains a major challenge in modeling, and current methods are limited to simulating phytoplankton (e.g., Microcystis sp., Anabaena sp., Aulacoseira sp., Cyclotella sp., Pediastrum sp., and Eudorina sp.) and zooplankton (e.g., Cyclotella sp., Pediastrum sp., and Eudorina sp.) at the genus level. The traditional modeling approach is limited for evaluating the interactions between phytoplankton and zooplankton. Recently, deep learning (DL) models have been proposed for solving modeling problems because of their large data handling capabilities and model structure flexibilities. In this study, we evaluated the applicability of DL for simulating phytoplankton at the phylum/class and genus levels and zooplankton at the genus level. Our work was an explicit representation of the taxonomic and ecological hierarchy of the DL model structure. The prerequisite for this model design was the data collection at two taxonomic and hierarchical levels. Our model consisted of hierarchical DL with classification transformer (TF) and regression TF models. These DL models were hierarchically connected; the output of the phylum/class level model was transferred to the genus level simulation model, and the output of the genus level model was fed into the zooplankton simulation model. The classification TF model determined the phytoplankton occurrence initiation date, whereas the regression TF model quantified the cell concentration of plankton. The hierarchical DL showed potential to simulate phytoplankton at the phylum/class and genus levels by producing average R^2, and root mean standard error values of 0.42 and 0.83 [log(cells mL^-1)], respectively. All simulated plankton results closely matched the measured concentrations. Particularly, the simulated cyanobacteria showed good agreement with the measured cell concentration, with an R² value of 0.72. In addition, our simulated result showed good agreement in peak concentration compared to observations. However, a limitation remained in following the temporal variation of Tintinnopsis sp. and Bosmia sp. Using an importance map from the TF model, water temperature, total phosphorus, and total nitrogen were identified as significant variables influencing phytoplankton and zooplankton blooms. Overall, our study demonstrated that DL can be used for modeling HABs at the phylum/class and genus levels.