Bald eagle-optimized transformer networks with temporal-spatial mid-level features for pancreatic tumor classification

The classification and diagnosis of pancreatic tumors present significant challenges due to their inherent complexity and variability. Traditional methods often struggle to capture the dynamic nature of these tumors, highlighting the need for advanced techniques that improve precision and robustness. This study introduces a novel approach that combines temporal-spatial mid-level features (CTSF) with bald eagle search (BES) optimized transformer networks to enhance pancreatic tumor classification. By leveraging temporal-spatial features that encompass both spatial structure and temporal evolution, we employ the BES algorithm to optimize the vision transformer (ViT) and swin transformer (ST) models, significantly enhancing their capacity to process complex datasets. The study underscores the critical role of temporal features in pancreatic tumor classification, enabling the capture of changes over time to improve our understanding of tumor progression and treatment responses. Among the models evaluated-GRU, LSTM, and ViT-the ViTachieved superior performance, with accuracy rates of 94.44%, 89.44%, and 87.22% on the TCIA-Pancreas-CT, Decathlon Pancreas, and NIH-Pancreas-CT datasets, respectively. Spatial features extracted from ResNet50, VGG16, and ST were also essential, with the ST model attaining the highest accuracy of 95.00%, 95.56%, and 93.33% on the same datasets. The integration of temporal and spatial features within the CTSF model resulted in accuracy rates of 96.02%, 97.21%, and 95.06% for the TCIA-Pancreas-CT, Decathlon Pancreas, and NIH-Pancreas-CT datasets, respectively. Furthermore, optimization techniques, particularly hyperparameter tuning, further enhanced performance, with the BES-optimized model achieving the highest accuracy of 98.02%, 98.92%, and 98.89%. The superiority of the CTSF-BES approach was confirmed through the Friedman test and Bonferroni-Dunn test, while execution time analysis demonstrated a favourable balance between performance and efficiency.

Self-supervised graph contrastive learning with diffusion augmentation for functional MRI analysis and brain disorder detection

Resting-state functional magnetic resonance imaging (rs-fMRI) provides a non-invasive imaging technique to study patterns of brain activity, and is increasingly used to facilitate automated brain disorder analysis. Existing fMRI-based learning methods often rely on labeled data to construct learning models, while the data annotation process typically requires significant time and resource investment. Graph contrastive learning offers a promising solution to address the small labeled data issue, by augmenting fMRI time series for self-supervised learning. However, data augmentation strategies employed in these approaches may damage the original blood-oxygen-level-dependent (BOLD) signals, thus hindering subsequent fMRI feature extraction. In this paper, we propose a self-supervised graph contrastive learning framework with diffusion augmentation (GCDA) for functional MRI analysis. The GCDA consists of a pretext model and a task-specific model. In the pretext model, we first augment each brain functional connectivity network derived from fMRI through a graph diffusion augmentation (GDA) module, and then use two graph isomorphism networks with shared parameters to extract features in a self-supervised contrastive learning manner. The pretext model can be optimized without the need for labeled training data, while the GDA focuses on perturbing graph edges and nodes, thus preserving the integrity of original BOLD signals. The task-specific model involves fine-tuning the trained pretext model to adapt to downstream tasks. Experimental results on two rs-fMRI cohorts with a total of 1230 subjects demonstrate the effectiveness of our method compared with several state-of-the-arts.

Evaluating Cognitive Function and Brain Activity Patterns via Blood Oxygen Level-Dependent Transformer in N-Back Working Memory Tasks

(1) Background: Working memory, which involves temporary storage, information processing, and regulating attention resources, is a fundamental cognitive process and constitutes a significant component of neuroscience research. This study aimed to evaluate brain activation patterns by analyzing functional magnetic resonance imaging (fMRI) time-series data collected during a designed N-back working memory task with varying cognitive demands. (2) Methods: We utilized a novel transformer model, blood oxygen level-dependent transformer (BolT), to extract the activation level features of brain regions in the cognitive process, thereby obtaining the influence weights of regions of interest (ROIs) on the corresponding tasks. (3) Results: Compared with previous studies, our work reached similar conclusions in major brain region performance and provides a more precise analysis for identifying brain activation patterns. For each type of working memory task, we selected the top 5 percent of the most influential ROIs and conducted a comprehensive analysis and discussion. Additionally, we explored the effect of prior knowledge conditions on the performance of different tasks in the same period and the same tasks at different times. (4) Conclusions: The comparison results reflect the brain's adaptive strategies and dependencies in coping with different levels of cognitive demands and the stability optimization of the brain's cognitive processing. This study introduces innovative methodologies for understanding brain function and cognitive processes, highlighting the potential of transformer in cognitive neuroscience. Its findings offer new insights into brain activity patterns associated with working memory, contributing to the broader landscape of neuroscience research.

DEAF-Net: Detail-Enhanced Attention Feature Fusion Network for Retinal Vessel Segmentation

Retinal vessel segmentation is crucial for the diagnosis of ophthalmic and cardiovascular diseases. However, retinal vessels are densely and irregularly distributed, with many capillaries blending into the background, and exhibit low contrast. Moreover, the encoder-decoder-based network for retinal vessel segmentation suffers from irreversible loss of detailed features due to multiple encoding and decoding, leading to incorrect segmentation of the vessels. Meanwhile, the single-dimensional attention mechanisms possess limitations, neglecting the importance of multidimensional features. To solve these issues, in this paper, we propose a detail-enhanced attention feature fusion network (DEAF-Net) for retinal vessel segmentation. First, the detail-enhanced residual block (DERB) module is proposed to strengthen the capacity for detailed representation, ensuring that intricate features are efficiently maintained during the segmentation of delicate vessels. Second, the multidimensional collaborative attention encoder (MCAE) module is proposed to optimize the extraction of multidimensional information. Then, the dynamic decoder (DYD) module is introduced to preserve spatial information during the decoding process and reduce the information loss caused by upsampling operations. Finally, the proposed detail-enhanced feature fusion (DEFF) module composed of DERB, MCAE and DYD modules fuses feature maps from both encoding and decoding and achieves effective aggregation of multi-scale contextual information. The experiments conducted on the datasets of DRIVE, CHASEDB1, and STARE, achieving Sen of 0.8305, 0.8784, and 0.8654, and AUC of 0.9886, 0.9913, and 0.9911 on DRIVE, CHASEDB1, and STARE, respectively, demonstrate the performance of our proposed network, particularly in the segmentation of fine retinal vessels.

Multimodal multiview bilinear graph convolutional network for mild cognitive impairment diagnosis

Mild cognitive impairment (MCI) is a significant predictor of the early progression of Alzheimer's disease (AD) and can serve as an important indicator of disease progression. However, many existing methods focus mainly on the image when processing brain imaging data, ignoring other non-imaging data (e.g., genetic, clinical information, etc.) that may have underlying disease information. In addition, imaging data acquired from different devices may exhibit varying degrees of heterogeneity, potentially resulting in numerous noisy connections during network construction. To address these challenges, this study proposes a Multimodal Multiview Bilinear Graph Convolution (MMBGCN) framework for disease risk prediction. Firstly, grey matter (GM), white matter (WM) and cerebrospinal fluid (CSF) features are extracted from magnetic resonance imaging (MRI), and non-imaging information is combined with the features extracted from MRI to construct a multimodal shared adjacency matrix. The shared adjacency matrix is then used to construct the multiview network so that the effect of potential disease information in the non-imaging information on the model can be considered. Finally, the MRI features extracted by the multiview network are weighted to reduce noise, and then the spatial pattern is restored by bilinear convolution. The features of the recovered spatial patterns are then combined with the genetic information for disease prediction. The proposed method is tested on the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset. Extensive experiments demonstrate the superior performance of the proposed framework and its ability to outperform other related algorithms. The average classification accuracy in the binary classification task in this study is 89.6%. The experimental results demonstrate that the method proposed in this study facilitates research on MCI diagnosis using multimodal data.

Prob-sparse self-attention extraction of time-aligned dynamic functional connectivity for ASD diagnosis

Dynamic functional connectivity (DFC) has shown promise in the diagnosis of Autism Spectrum Disorder (ASD). However, extracting highly discriminative information from the complex DFC matrix remains a challenging task. In this paper, we propose an ASD classification framework PSA-FCN which is based on time-aligned DFC and Prob-Sparse Self-Attention to address this problem. Specifically, we introduce Prob-Sparse Self-Attention to selectively extract global features, and use self-attention distillation as a transition at each layer to capture local patterns and reduce dimensionality. Additionally, we construct a time-aligned DFC matrix to mitigate the time sensitivity of DFC and extend the dataset, thereby alleviating model overfitting. Our model is evaluated on fMRI data from the ABIDE NYU site, and the experimental results demonstrate that the model outperforms other methods in the paper with a classification accuracy of 81.8 %. Additionally, our research findings reveal significant variability in the DFC connections of brain regions of ASD patients, including Cuneus (CUN), Lingual gyrus (LING), Superior occipital gyrus (SOG), Posterior cingulate gyrus (PCG), and Precuneus (PCUN), which is consistent with prior research. In summary, our proposed PSA framework shows potential in ASD diagnosis as well as automatic discovery of critical ASD-related biomarkers.

Improving fMRI-Based Autism Severity Identification via Brain Network Distance and Adaptive Label Distribution Learning

Machine learning methodologies have been profoundly researched in the realm of autism spectrum disorder (ASD) diagnosis. Nonetheless, owing to the ambiguity of ASD severity labels and individual differences in ASD severity, current fMRI-based methods for identifying ASD severity still do not achieve satisfactory performance. Besides, the potential association between brain functional networks(BFN) and ASD symptom severity remains under investigation. To address these problems, we propose a low&high-level BFN distance method and an adaptive multi-label distribution(HBFND-AMLD) technique for ASD severity identification. First, a low-level and high-level BFN distance(HBFND) is proposed to construct BFN that reflects differences in ASD severity. This method can measure the distance between the ASD and the health control(HC) on the low-order and high-order BFN respectively, which can distinguish the severity of ASD. After that, a multi-task network is proposed for ASD severity identification which considers the individual differences of ASD severity in communication and society, which considers the individual differences in language and social skills of ASD patients. Finally, a novel adaptive label distribution(ALD) technique is employed to train the ASD severity identification model, effectively preventing network overfitting by restricting label probability distribution. We evaluate the proposed framework on the public ABIDE I dataset. The promising results obtained by our framework outperform the state-of-the-art methods with an increase in identification performance, indicating that it has a potential clinical prospect for practical ASD severity diagnosis.

Multi-Modal Diagnosis of Alzheimer's Disease Using Interpretable Graph Convolutional Networks

The interconnection between brain regions in neurological disease encodes vital information for the advancement of biomarkers and diagnostics. Although graph convolutional networks are widely applied for discovering brain connection patterns that point to disease conditions, the potential of connection patterns that arise from multiple imaging modalities has yet to be fully realized. In this paper, we propose a multi-modal sparse interpretable GCN framework (SGCN) for the detection of Alzheimer's disease (AD) and its prodromal stage, known as mild cognitive impairment (MCI). In our experimentation, SGCN learned the sparse regional importance probability to find signature regions of interest (ROIs), and the connective importance probability to reveal disease-specific brain network connections. We evaluated SGCN on the Alzheimer's Disease Neuroimaging Initiative database with multi-modal brain images and demonstrated that the ROI features learned by SGCN were effective for enhancing AD status identification. The identified abnormalities were significantly correlated with AD-related clinical symptoms. We further interpreted the identified brain dysfunctions at the level of large-scale neural systems and sex-related connectivity abnormalities in AD/MCI. The salient ROIs and the prominent brain connectivity abnormalities interpreted by SGCN are considerably important for developing novel biomarkers. These findings contribute to a better understanding of the network-based disorder via multi-modal diagnosis and offer the potential for precision diagnostics. The source code is available at https://github.com/Houliang-Zhou/SGCN.

Multi-Scale Spatial-Temporal Attention Networks for Functional Connectome Classification

Many neuropsychiatric disorders are considered to be associated with abnormalities in the functional connectivity networks of the brain. The research on the classification of functional connectivity can therefore provide new perspectives for understanding the pathology of disorders and contribute to early diagnosis and treatment. Functional connectivity exhibits a nature of dynamically changing over time, however, the majority of existing methods are unable to collectively reveal the spatial topology and time-varying characteristics. Furthermore, despite the efforts of limited spatial-temporal studies to capture rich information across different spatial scales, they have not delved into the temporal characteristics among different scales. To address above issues, we propose a novel Multi-Scale Spatial-Temporal Attention Networks (MSSTAN) to exploit the multi-scale spatial-temporal information provided by functional connectome for classification. To fully extract spatial features of brain regions, we propose a Topology Enhanced Graph Transformer module to guide the attention calculations in the learning of spatial features by incorporating topology priors. A Multi-Scale Pooling Strategy is introduced to obtain representations of brain connectome at various scales. Considering the temporal dynamic characteristics between dynamic functional connectome, we employ Locality Sensitive Hashing attention to further capture long-term dependencies in time dynamics across multiple scales and reduce the computational complexity of the original attention mechanism. Experiments on three brain fMRI datasets of MDD and ASD demonstrate the superiority of our proposed approach. In addition, benefiting from the attention mechanism in Transformer, our results are interpretable, which can contribute to the discovery of biomarkers. The code is available at https://github.com/LIST-KONG/MSSTAN.

Lightweight skin cancer detection IP hardware implementation using cycle expansion and optimal computation arrays methods

Skin cancer is recognized as one of the most perilous diseases globally. In the field of medical image classification, precise identification of early-stage skin lesions is imperative for accurate diagnosis. However, deploying these algorithms on low-cost devices and attaining high-efficiency operation with minimal energy consumption poses a formidable challenge due to their intricate computational demands. This study proposes a lightweight hardware design based on a convolutional neural network (CNN) for real-time processing of skin disease classifiers on portable devices. Our skin cancer recognition processor utilizes an optimally parallel designed processing engine (PE) for global computation, which greatly reduces hardware resource utilization by multiplexing of computational unit circuits. In addition, a design approach that provides loop unrolling effectively reduces the number of data accesses, thereby reducing computational complexity and logic resource requirements. The hardware circuits in this design perform data inference in convolutional, pooling, and fully connected layers based on 16-bit floating-point numbers. Evaluation of the HAM10000 database dataset shows that the architecture achieves an average classification accuracy of 97.8 %. We are the first to implement an all-hardware FPGA-based skin cancer detection platform that offers a 3.5x speedup in recognition compared to existing skin cancer accelerators at 50 MHz while consuming only 0.48 W of power. The implementation of this hardware architecture meets the major constraints of portable devices, featuring low resource utilization, low power consumption, and cost-effectiveness, while still providing efficient classification and high accuracy results.

Constructing High-Order Functional Connectivity Networks With Temporal Information From fMRI Data

Conducting functional connectivity analysis on functional magnetic resonance imaging (fMRI) data presents a significant and intricate challenge. Contemporary studies typically analyze fMRI data by constructing high-order functional connectivity networks (FCNs) due to their strong interpretability. However, these approaches often overlook temporal information, resulting in suboptimal accuracy. Temporal information plays a vital role in reflecting changes in blood oxygenation level-dependent signals. To address this shortcoming, we have devised a framework for extracting temporal dependencies from fMRI data and inferring high-order functional connectivity among regions of interest (ROIs). Our approach postulates that the current state can be determined by the FCN and the state at the previous time, effectively capturing temporal dependencies. Furthermore, we enhance FCN by incorporating high-order features through hypergraph-based manifold regularization. Our algorithm involves causal modeling of the dynamic brain system, and the obtained directed FC reveals differences in the flow of information under different patterns. We have validated the significance of integrating temporal information into FCN using four real-world fMRI datasets. On average, our framework achieves 12% higher accuracy than non-temporal hypergraph-based and low-order FCNs, all while maintaining a short processing time. Notably, our framework successfully identifies the most discriminative ROIs, aligning with previous research, and thereby facilitating cognitive and behavioral studies.

Temporal Dynamic Synchronous Functional Brain Network for Schizophrenia Classification and Lateralization Analysis

Available evidence suggests that dynamic functional connectivity can capture time-varying abnormalities in brain activity in resting-state cerebral functional magnetic resonance imaging (rs-fMRI) data and has a natural advantage in uncovering mechanisms of abnormal brain activity in schizophrenia (SZ) patients. Hence, an advanced dynamic brain network analysis model called the temporal brain category graph convolutional network (Temporal-BCGCN) was employed. Firstly, a unique dynamic brain network analysis module, DSF-BrainNet, was designed to construct dynamic synchronization features. Subsequently, a revolutionary graph convolution method, TemporalConv, was proposed based on the synchronous temporal properties of features. Finally, the first modular test tool for abnormal hemispherical lateralization in deep learning based on rs-fMRI data, named CategoryPool, was proposed. This study was validated on COBRE and UCLA datasets and achieved 83.62% and 89.71% average accuracies, respectively, outperforming the baseline model and other state-of-the-art methods. The ablation results also demonstrate the advantages of TemporalConv over the traditional edge feature graph convolution approach and the improvement of CategoryPool over the classical graph pooling approach. Interestingly, this study showed that the lower-order perceptual system and higher-order network regions in the left hemisphere are more severely dysfunctional than in the right hemisphere in SZ, reaffirmings the importance of the left medial superior frontal gyrus in SZ. Our code was available at: https://github.com/swfen/Temporal-BCGCN.

Enhancing Autism Spectrum Disorder identification in multi-site MRI imaging: A multi-head cross-attention and multi-context approach for addressing variability in un-harmonized data

Multi-site MRI imaging poses a significant challenge due to the potential variations in images across different scanners at different sites. This variability can introduce ambiguity in further image analysis. Consequently, the image analysis techniques become site-dependent and scanner-dependent, implying that adjustments in the analysis methodologies are necessary for each scanner configuration. Further, implementing real-time modifications becomes intricate, particularly when incorporating a new type of scanner, as it requires adapting the analysis methods accordingly. Taking into account the aforementioned challenge, we have considered its implications for an Autism spectrum disorder (ASD) application. Our objective is to minimize the impact of site and scanner variability in the analysis, aiming to develop a model that remains effective across different scanners and sites. This entails devising a methodology that allows the same model to function seamlessly across multiple scanner configurations and sites. ASD, a behavioral disorder affecting child development, requires early detection. Clinical observation is time-consuming, prompting the use of fMRI with machine/deep learning for expedited diagnosis. Previous methods leverage fMRI's functional connectivity but often rely on less generalized feature extractors and classifiers. Hence, there is significant room for improvement in the generalizability of detection methods across multi-site data, which is acquired from multiple scanners with different settings. In this study, we propose a Cross-Combination Multi-Scale Multi-Context Framework (CCMSMCF) capable of performing neuroimaging-based diagnostic classification of mental disorders for a multi-site dataset. Thus, this framework attains a degree of internal data harmonization, rendering it to some extent site and scanner-agnostic. Our proposed network, CCMSMCF, is constructed by integrating two sub-modules: the Multi-Head Attention Cross-Scale Module (MHACSM) and the Residual Multi-Context Module (RMCN). We also employ multiple loss functions in a novel manner for training the model, which includes Binary Cross Entropy, Dice loss, and Embedding Coupling loss. The model is validated on the Autism Brain Imaging Data Exchange I (ABIDE-I) dataset, which includes data from multiple scanners across different sites, and achieves promising results.

Dynamic functional connections analysis with spectral learning for brain disorder detection

Dynamic functional connections (dFCs), can reveal neural activities, which provides an insightful way of mining the temporal patterns within the human brain and further detecting brain disorders. However, most existing studies focus on the dFCs estimation to identify brain disorders by shallow temporal features and methods, which cannot capture the inherent temporal patterns of dFCs effectively. To address this problem, this study proposes a novel method, named dynamic functional connections analysis with spectral learning (dCSL), to explore inherently temporal patterns of dFCs and further detect the brain disorders. Concretely, dCSL includes two components, dFCs estimation module and dFCs analysis module. In the former, dFCs are estimated via the sliding window technique. In the latter, the spectral kernel mapping is first constructed by combining the Fourier transform with the non-stationary kernel. Subsequently, the spectral kernel mapping is stacked into a deep kernel network to explore higher-order temporal patterns of dFCs through spectral learning. The proposed dCSL, sharing the benefits of deep architecture and non-stationary kernel, can not only calculate the long-range relationship but also explore the higher-order temporal patterns of dFCs. To evaluate the proposed method, a set of brain disorder classification tasks are conducted on several public datasets. As a result, the proposed dCSL achieves 5% accuracy improvement compared with the widely used approaches for analyzing sequence data, 1.3% accuracy improvement compared with the state-of-the-art methods for dFCs. In addition, the discriminative brain regions are explored in the ASD detection task. The findings in this study are consistent with the clinical performance in ASD.

A Novel Method to Identify Mild Cognitive Impairment Using Dynamic Spatio-Temporal Graph Neural Network

Resting-state functional magnetic resonance imaging (rs-fMRI) has been widely used in the identification of mild cognitive impairment (MCI) research, MCI patients are relatively at a higher risk of progression to Alzheimer's disease (AD). However, almost machine learning and deep learning methods are rarely analyzed from the perspective of spatial structure and temporal dimension. In order to make full use of rs-fMRI data, this study constructed a dynamic spatiotemporal graph neural network model, which mainly includes three modules: temporal block, spatial block, and graph pooling block. Our proposed model can extract the BOLD signal of the subject's fMRI data and the spatial structure of functional connections between different brain regions, and improve the decision-making results of the model. In the study of AD, MCI and NC, the classification accuracy reached 83.78% outperforming previously reported, which manifested that our model could effectively learn spatiotemporal, and dynamic spatio-temporal method plays an important role in identifying different groups of subjects. In summary, this paper proposed an end-to-end dynamic spatio-temporal graph neural network model, which uses the information of the temporal dimension and spatial structure in rs-fMRI data, and achieves the improvement of the three classification performance among AD, MCI and NC.

Mild cognitive impairment prediction based on multi-stream convolutional neural networks

BACKGROUND

Mild cognitive impairment (MCI) is the transition stage between the cognitive decline expected in normal aging and more severe cognitive decline such as dementia. The early diagnosis of MCI plays an important role in human healthcare. Current methods of MCI detection include cognitive tests to screen for executive function impairments, possibly followed by neuroimaging tests. However, these methods are expensive and time-consuming. Several studies have demonstrated that MCI and dementia can be detected by machine learning technologies from different modality data. This study proposes a multi-stream convolutional neural network (MCNN) model to predict MCI from face videos.

RESULTS

The total effective data are 48 facial videos from 45 participants, including 35 videos from normal cognitive participants and 13 videos from MCI participants. The videos are divided into several segments. Then, the MCNN captures the latent facial spatial features and facial dynamic features of each segment and classifies the segment as MCI or normal. Finally, the aggregation stage produces the final detection results of the input video. We evaluate 27 MCNN model combinations including three ResNet architectures, three optimizers, and three activation functions. The experimental results showed that the ResNet-50 backbone with Swish activation function and Ranger optimizer produces the best results with an F1-score of 89% at the segment level. However, the ResNet-18 backbone with Swish and Ranger achieves the F1-score of 100% at the participant level.

CONCLUSIONS

This study presents an efficient new method for predicting MCI from facial videos. Studies have shown that MCI can be detected from facial videos, and facial data can be used as a biomarker for MCI. This approach is very promising for developing accurate models for screening MCI through facial data. It demonstrates that automated, non-invasive, and inexpensive MCI screening methods are feasible and do not require highly subjective paper-and-pencil questionnaires. Evaluation of 27 model combinations also found that ResNet-50 with Swish is more stable for different optimizers. Such results provide directions for hyperparameter tuning to further improve MCI predictions.

Explainable spatio-temporal graph evolution learning with applications to dynamic brain network analysis during development

Modeling dynamic interactions among network components is crucial to uncovering the evolution mechanisms of complex networks. Recently, spatio-temporal graph learning methods have achieved noteworthy results in characterizing the dynamic changes of inter-node relations (INRs). However, challenges remain: The spatial neighborhood of an INR is underexploited, and the spatio-temporal dependencies in INRs' dynamic changes are overlooked, ignoring the influence of historical states and local information. In addition, the model's explainability has been understudied. To address these issues, we propose an explainable spatio-temporal graph evolution learning (ESTGEL) model to model the dynamic evolution of INRs. Specifically, an edge attention module is proposed to utilize the spatial neighborhood of an INR at multi-level, i.e., a hierarchy of nested subgraphs derived from decomposing the initial node-relation graph. Subsequently, a dynamic relation learning module is proposed to capture the spatio-temporal dependencies of INRs. The INRs are then used as adjacent information to improve the node representation, resulting in comprehensive delineation of dynamic evolution of the network. Finally, the approach is validated with real data on brain development study. Experimental results on dynamic brain networks analysis reveal that brain functional networks transition from dispersed to more convergent and modular structures throughout development. Significant changes are observed in the dynamic functional connectivity (dFC) associated with functions including emotional control, decision-making, and language processing.

Learnable Brain Connectivity Structures for Identifying Neurological Disorders

Brain networks/graphs have been widely recognized as powerful and efficient tools for identifying neurological disorders. In recent years, various graph neural network models have been developed to automatically extract features from brain networks. However, a key limitation of these models is that the inputs, namely brain networks/graphs, are constructed using predefined statistical metrics (e.g., Pearson correlation) and are not learnable. The lack of learnability restricts the flexibility of these approaches. While statistically-specific brain networks can be highly effective in recognizing certain diseases, their performance may not exhibit robustness when applied to other types of brain disorders. To address this issue, we propose a novel module called Brain Structure Inference (termed BSI), which can be seamlessly integrated with multiple downstream tasks within a unified framework, enabling end-to-end training. It is highly flexible to learn the most beneficial underlying graph structures directly for specific downstream tasks. The proposed method achieves classification accuracies of 74.83% and 79.18% on two publicly available datasets, respectively. This suggests an improvement of at least 3% over the best-performing existing methods for both tasks. In addition to its excellent performance, the proposed method is highly interpretable, and the results are generally consistent with previous findings.

A Multiview Brain Network Transformer Fusing Individualized Information for Autism Spectrum Disorder Diagnosis

Functional connectivity (FC) networks, built from analyses of resting-state magnetic resonance imaging (rs-fMRI), serve as efficacious biomarkers for identifying Autism Spectrum Disorders (ASD) patients. Given the neurobiological heterogeneity across individuals and the unique presentation of ASD symptoms, the fusion of individualized information into diagnosis becomes essential. However, this aspect is overlooked in most methods. Furthermore, the existing methods typically focus on studying direct pairwise connections between brain ROIs, while disregarding interactions between indirectly connected neighbors. To overcome above challenges, we build common FC and individualized FC by tangent pearson embedding (TP) and common orthogonal basis extraction (COBE) respectively, and present a novel multiview brain transformer (MBT) aimed at effectively fusing common and indivinformation of subjects. MBT is mainly constructed by transformer layers with diffusion kernel (DK), fusion quality-inspired weighting module (FQW), similarity loss and orthonormal clustering fusion readout module (OCFRead). DK transformer can incorporate higher-order random walk methods to capture wider interactions among indirectly connected brain regions. FQW promotes adaptive fusion of features between views, and similarity loss and OCFRead are placed on the last layer to accomplish the ultimate integration of information. In our method, TP, DK and FQW modules all help to model wider connectivity in the brain that make up for the shortcomings of traditional methods. We conducted experiments on the public ABIDE dataset based on AAL and CC200 respectively. Our framework has shown promising results, outperforming state-of-the-art methods on both templates. This suggests its potential as a valuable approach for clinical ASD diagnosis.

Spatial-Temporal Co-Attention Learning for Diagnosis of Mental Disorders From Resting-State fMRI Data

Neuroimaging techniques have been widely adopted to detect the neurological brain structures and functions of the nervous system. As an effective noninvasive neuroimaging technique, functional magnetic resonance imaging (fMRI) has been extensively used in computer-aided diagnosis (CAD) of mental disorders, e.g., autism spectrum disorder (ASD) and attention deficit/hyperactivity disorder (ADHD). In this study, we propose a spatial-temporal co-attention learning (STCAL) model for diagnosing ASD and ADHD from fMRI data. In particular, a guided co-attention (GCA) module is developed to model the intermodal interactions of spatial and temporal signal patterns. A novel sliding cluster attention module is designed to address global feature dependency of self-attention mechanism in fMRI time series. Comprehensive experimental results demonstrate that our STCAL model can achieve competitive accuracies of 73.0 ± 4.5%, 72.0 ± 3.8%, and 72.5 ± 4.2% on the ABIDE I, ABIDE II, and ADHD-200 datasets, respectively. Moreover, the potential for feature pruning based on the co-attention scores is validated by the simulation experiment. The clinical interpretation analysis of STCAL can allow medical professionals to concentrate on the discriminative regions of interest and key time frames from fMRI data.

BrainDAS: Structure-aware domain adaptation network for multi-site brain network analysis

In the medical field, datasets are mostly integrated across sites due to difficult data acquisition and insufficient data at a single site. The domain shift problem caused by the heterogeneous distribution among multi-site data makes autism spectrum disorder (ASD) hard to identify. Recently, domain adaptation has received considerable attention as a promising solution. However, domain adaptation on graph data like brain networks has not been fully studied. It faces two major challenges: (1) complex graph structure; and (2) multiple source domains. To overcome the issues, we propose an end-to-end structure-aware domain adaptation framework for brain network analysis (BrainDAS) using resting-state functional magnetic resonance imaging (rs-fMRI). The proposed approach contains two stages: supervision-guided multi-site graph domain adaptation with dynamic kernel generation and graph classification with attention-based graph pooling. We evaluate our BrainDAS on a public dataset provided by Autism Brain Imaging Data Exchange (ABIDE) which includes 871 subjects from 17 different sites, surpassing state-of-the-art algorithms in several different evaluation settings. Furthermore, our promising results demonstrate the interpretability and generalization of the proposed method. Our code is available at https://github.com/songruoxian/BrainDAS.

STDCformer: Spatial-temporal dual-path cross-attention model for fMRI-based autism spectrum disorder identification

Resting-state functional magnetic resonance imaging (rs-fMRI) is a non-invasive neuroimaging technique widely utilized in the research of Autism Spectrum Disorder (ASD), providing preliminary insights into the potential biological mechanisms underlying ASD. Deep learning techniques have demonstrated significant potential in the analysis of rs-fMRI. However, accurately distinguishing between healthy control group and ASD has been a longstanding challenge. In this regard, this work proposes a model featuring a dual-path cross-attention framework for spatial and temporal patterns, named STDCformer, aiming to enhance the accuracy of ASD identification. STDCformer can preserve both temporal-specific patterns and spatial-specific patterns while explicitly interacting spatiotemporal information in depth. The embedding layer of the STDCformer embeds temporal and spatial patterns in dual paths. For the temporal path, we introduce a perturbation positional encoding to improve the issue of signal misalignment caused by individual differences. For the spatial path, we propose a correlation metric based on Gramian angular field similarity to establish a more specific whole-brain functional network. Subsequently, we interleave the query and key vectors of dual paths to interact spatial and temporal information. We further propose integrating the dual-path attention into a tensor that retains spatiotemporal dimensions and utilizing 2D convolution for feed-forward processing. Our attention layer allows the model to represent spatiotemporal correlations of signals at multiple scales to alleviate issues of information distortion and loss. Our STDCformer demonstrates competitive results compared to state-of-the-art methods on the ABIDE dataset. Additionally, we conducted interpretative analyses of the model to preliminarily discuss the potential physiological mechanisms of ASD. This work once again demonstrates the potential of deep learning technology in identifying ASD and developing neuroimaging biomarkers for ASD.

Identification of autism spectrum disorder using multiple functional connectivity-based graph convolutional network

Presently, the combination of graph convolutional networks (GCN) with resting-state functional magnetic resonance imaging (rs-fMRI) data is a promising approach for early diagnosis of autism spectrum disorder (ASD). However, the prevalent approach involves exclusively full-brain functional connectivity data for disease classification using GCN, while overlooking the prior information related to the functional connectivity of brain subnetworks associated with ASD. Therefore, in this study, the multiple functional connectivity-based graph convolutional network (MFC-GCN) framework is proposed, using not only full brain functional connectivity data but also the established functional connectivity data from networks of key brain subnetworks associated with ASD, and the GCN is adopted to acquire complementary feature information for the final classification task. Given the heterogeneity within the Autism Brain Imaging Data Exchange (ABIDE) dataset, a novel External Attention Network Readout (EANReadout) is introduced. This design enables the exploration of potential subject associations, effectively addressing the dataset's heterogeneity. Experiments were conducted on the ABIDE dataset using the proposed framework, involving 714 subjects, and the average accuracy of the framework was 70.31%. The experimental results show that the proposed EANReadout outperforms the best traditional readout layer and improves the average accuracy of the framework by 4.32%.

Deep Self-Reconstruction Fusion Similarity Hashing for the Diagnosis of Alzheimer's Disease on Multi-Modal Data

The pathogenesis of Alzheimer's disease (AD) is extremely intricate, which makes AD patients almost incurable. Recent studies have demonstrated that analyzing multi-modal data can offer a comprehensive perspective on the different stages of AD progression, which is beneficial for early diagnosis of AD. In this paper, we propose a deep self-reconstruction fusion similarity hashing (DS-FSH) method to effectively capture the AD-related biomarkers from the multi-modal data and leverage them to diagnose AD. Given that most existing methods ignore the topological structure of the data, a deep self-reconstruction model based on random walk graph regularization is designed to reconstruct the multi-modal data, thereby learning the nonlinear relationship between samples. Additionally, a fused similarity hash based on anchor graph is proposed to generate discriminative binary hash codes for multi-modal reconstructed data. This allows sample fused similarity to be effectively modeled by a fusion similarity matrix based on anchor graph while modal correlation can be approximated by Hamming distance. Especially, extracted features from the multi-modal data are classified using deep sparse autoencoders classifier. Finally, experiments conduct on the AD Neuroimaging Initiative database show that DS-FSH outperforms comparable methods of AD classification. To conclude, DS-FSH identifies multi-modal features closely associated with AD, which are expected to contribute significantly to understanding of the pathogenesis of AD.

Altered dynamic amplitude of low-frequency fluctuation in individuals at high risk for Alzheimer's disease

The brain's dynamic spontaneous neural activity is significant in supporting cognition; however, how brain dynamics go awry in subjective cognitive decline (SCD) and mild cognitive impairment (MCI) remains unclear. Thus, the current study aimed to investigate the dynamic amplitude of low-frequency fluctuation (dALFF) alterations in patients at high risk for Alzheimer's disease and to explore its correlation with clinical cognitive assessment scales, to identify an early imaging sign for these special populations. A total of 152 participants, including 72 SCD patients, 44 MCI patients and 36 healthy controls (HCs), underwent a resting-state functional magnetic resonance imaging and were assessed with various neuropsychological tests. The dALFF was measured using sliding-window analysis. We employed canonical correlation analysis (CCA) to examine the bi-multivariate correlations between neuropsychological scales and altered dALFF among multiple regions in SCD and MCI patients. Compared to those in the HC group, both the MCI and SCD groups showed higher dALFF values in the right opercular inferior frontal gyrus (voxel P < .001, cluster P < .05, correction). Moreover, the CCA models revealed that behavioural tests relevant to inattention correlated with the dALFF of the right middle frontal gyrus and right opercular inferior frontal gyrus, which are involved in frontoparietal networks (R = .43, P = .024). In conclusion, the brain dynamics of neural activity in frontal areas provide insights into the shared neural basis underlying SCD and MCI.

Inferring brain causal and temporal-lag networks for recognizing abnormal patterns of dementia

Brain functional network analysis has become a popular method to explore the laws of brain organization and identify biomarkers of neurological diseases. However, it is still a challenging task to construct an ideal brain network due to the limited understanding of the human brain. Existing methods often ignore the impact of temporal-lag on the results of brain network modeling, which may lead to some unreliable conclusions. To overcome this issue, we propose a novel brain functional network estimation method, which can simultaneously infer the causal mechanisms and temporal-lag values among brain regions. Specifically, our method converts the lag learning into an instantaneous effect estimation problem, and further embeds the search objectives into a deep neural network model as parameters to be learned. To verify the effectiveness of the proposed estimation method, we perform experiments on the Alzheimer's Disease Neuroimaging Initiative (ADNI) database by comparing the proposed model with several existing methods, including correlation-based and causality-based methods. The experimental results show that our brain networks constructed by the proposed estimation method can not only achieve promising classification performance, but also exhibit some characteristics of physiological mechanisms. Our approach provides a new perspective for understanding the pathogenesis of brain diseases. The source code is released at https://github.com/NJUSTxiazw/CTLN.

Predicting rectal cancer prognosis from histopathological images and clinical information using multi-modal deep learning

Objective

The objective of this study was to provide a multi-modal deep learning framework for forecasting the survival of rectal cancer patients by utilizing both digital pathological images data and non-imaging clinical data.

Materials and methods

The research included patients diagnosed with rectal cancer by pathological confirmation from January 2015 to December 2016. Patients were allocated to training and testing sets in a randomized manner, with a ratio of 4:1. The tissue microarrays (TMAs) and clinical indicators were obtained. Subsequently, we selected distinct deep learning models to individually forecast patient survival. We conducted a scanning procedure on the TMAs in order to transform them into digital pathology pictures. Additionally, we performed pre-processing on the clinical data of the patients. Subsequently, we selected distinct deep learning algorithms to conduct survival prediction analysis using patients' pathological images and clinical data, respectively.

Results

A total of 292 patients with rectal cancer were randomly allocated into two groups: a training set consisting of 234 cases, and a testing set consisting of 58 instances. Initially, we make direct predictions about the survival status by using pre-processed Hematoxylin and Eosin (H&E) pathological images of rectal cancer. We utilized the ResNest model to extract data from histopathological images of patients, resulting in a survival status prediction with an AUC (Area Under the Curve) of 0.797. Furthermore, we employ a multi-head attention fusion (MHAF) model to combine image features and clinical features in order to accurately forecast the survival rate of rectal cancer patients. The findings of our experiment show that the multi-modal structure works better than directly predicting from histopathological images. It achieves an AUC of 0.837 in predicting overall survival (OS).

Conclusions

Our study highlights the potential of multi-modal deep learning models in predicting survival status from histopathological images and clinical information, thus offering valuable insights for clinical applications.

Integration of temporal & spatial properties of dynamic functional connectivity based on two-directional two-dimensional principal component analysis for disease analysis

Dynamic functional connectivity, derived from resting-state functional magnetic resonance imaging (rs-fMRI), has emerged as a crucial instrument for investigating and supporting the diagnosis of neurological disorders. However, prevalent features of dynamic functional connectivity predominantly capture either temporal or spatial properties, such as mean and global efficiency, neglecting the significant information embedded in the fusion of spatial and temporal attributes. In addition, dynamic functional connectivity suffers from the problem of temporal mismatch, i.e., the functional connectivity of different subjects at the same time point cannot be matched. To address these problems, this article introduces a novel feature extraction framework grounded in two-directional two-dimensional principal component analysis. This framework is designed to extract features that integrate both spatial and temporal properties of dynamic functional connectivity. Additionally, we propose to use Fourier transform to extract temporal-invariance properties contained in dynamic functional connectivity. Experimental findings underscore the superior performance of features extracted by this framework in classification experiments compared to features capturing individual properties.

A Learnable Counter-Condition Analysis Framework for Functional Connectivity-Based Neurological Disorder Diagnosis

To understand the biological characteristics of neurological disorders with functional connectivity (FC), recent studies have widely utilized deep learning-based models to identify the disease and conducted post-hoc analyses via explainable models to discover disease-related biomarkers. Most existing frameworks consist of three stages, namely, feature selection, feature extraction for classification, and analysis, where each stage is implemented separately. However, if the results at each stage lack reliability, it can cause misdiagnosis and incorrect analysis in afterward stages. In this study, we propose a novel unified framework that systemically integrates diagnoses (i.e., feature selection and feature extraction) and explanations. Notably, we devised an adaptive attention network as a feature selection approach to identify individual-specific disease-related connections. We also propose a functional network relational encoder that summarizes the global topological properties of FC by learning the inter-network relations without pre-defined edges between functional networks. Last but not least, our framework provides a novel explanatory power for neuroscientific interpretation, also termed counter-condition analysis. We simulated the FC that reverses the diagnostic information (i.e., counter-condition FC): converting a normal brain to be abnormal and vice versa. We validated the effectiveness of our framework by using two large resting-state functional magnetic resonance imaging (fMRI) datasets, Autism Brain Imaging Data Exchange (ABIDE) and REST-meta-MDD, and demonstrated that our framework outperforms other competing methods for disease identification. Furthermore, we analyzed the disease-related neurological patterns based on counter-condition analysis.

A Multi-Graph Cross-Attention-Based Region-Aware Feature Fusion Network Using Multi-Template for Brain Disorder Diagnosis

Functional connectivity (FC) networks based on resting-state functional magnetic imaging (rs-fMRI) are reliable and sensitive for brain disorder diagnosis. However, most existing methods are limited by using a single template, which may be insufficient to reveal complex brain connectivities. Furthermore, these methods usually neglect the complementary information between static and dynamic brain networks, and the functional divergence among different brain regions, leading to suboptimal diagnosis performance. To address these limitations, we propose a novel multi-graph cross-attention based region-aware feature fusion network (MGCA-RAFFNet) by using multi-template for brain disorder diagnosis. Specifically, we first employ multi-template to parcellate the brain space into different regions of interest (ROIs). Then, a multi-graph cross-attention network (MGCAN), including static and dynamic graph convolutions, is developed to explore the deep features contained in multi-template data, which can effectively analyze complex interaction patterns of brain networks for each template, and further adopt a dual-view cross-attention (DVCA) to acquire complementary information. Finally, to efficiently fuse multiple static-dynamic features, we design a region-aware feature fusion network (RAFFNet), which is beneficial to improve the feature discrimination by considering the underlying relations among static-dynamic features in different brain regions. Our proposed method is evaluated on both public ADNI-2 and ABIDE-I datasets for diagnosing mild cognitive impairment (MCI) and autism spectrum disorder (ASD). Extensive experiments demonstrate that the proposed method outperforms the state-of-the-art methods. Our source code is available at https://github.com/mylbuaa/MGCA-RAFFNet.

Effective hyper-connectivity network construction and learning: Application to major depressive disorder identification

Functional connectivity (FC) derived from resting-state fMRI (rs-fMRI) is a primary approach for identifying brain diseases, but it is limited to capturing the pairwise correlation between regions-of-interest (ROIs) in the brain. Thus, hyper-connectivity which describes the higher-order relationship among multiple ROIs is receiving increasing attention. However, most hyper-connectivity methods overlook the directionality of connections. The direction of information flow constitutes a pivotal factor in shaping brain activity and cognitive processes. Neglecting this directional aspect can lead to an incomplete understanding of high-order interactions within the brain. To this end, we propose a novel effective hyper-connectivity (EHC) network that integrates direction detection and hyper-connectivity modeling. It characterizes the high-order directional information flow among multiple ROIs, providing a more comprehensive understanding of brain activity. Then, we develop a directed hypergraph convolutional network (DHGCN) to acquire deep representations from EHC network and functional indicators of ROIs. In contrast to conventional hypergraph convolutional networks designed for undirected hypergraphs, DHGCN is specifically tailored to handle directed hypergraph data structures. Moreover, unlike existing methods that primarily focus on fMRI time series, our proposed DHGCN model also incorporates multiple functional indicators, providing a robust framework for feature learning. Finally, deep representations generated via DHGCN, combined with demographic factors, are used for major depressive disorder (MDD) identification. Experimental results demonstrate that the proposed framework outperforms both FC and undirected hyper-connectivity models, as well as surpassing other state-of-the-art methods. The identification of EHC abnormalities through our framework can enhance the analysis of brain function in individuals with MDD.

Dynamic functional network connectivity analysis in schizophrenia based on a spatiotemporal CPD framework

Objective.Dynamic functional network connectivity (dFNC), based on data-driven group independent component (IC) analysis, is an important avenue for investigating underlying patterns of certain brain diseases such as schizophrenia. Canonical polyadic decomposition (CPD) of a higher-way dynamic functional connectivity tensor, can offer an innovative spatiotemporal framework to accurately characterize potential dynamic spatial and temporal fluctuations. Since multi-subject dFNC data from sliding-window analysis are also naturally a higher-order tensor, we propose an innovative sparse and low-rank CPD (SLRCPD) for the three-way dFNC tensor to excavate significant dynamic spatiotemporal aberrant changes in schizophrenia.Approach.The proposed SLRCPD approach imposes two constraints. First, the L1regularization on spatial modules is applied to extract sparse but significant dynamic connectivity and avoid overfitting the model. Second, low-rank constraint is added on time-varying weights to enhance the temporal state clustering quality. Shared dynamic spatial modules, group-specific dynamic spatial modules and time-varying weights can be extracted by SLRCPD. The strength of connections within- and between-IC networks and connection contribution are proposed to inspect the spatial modules. K-means clustering and classification are further conducted to explore temporal group difference.Main results.82 subject resting-state functional magnetic resonance imaging (fMRI) dataset and opening Center for Biomedical Research Excellence (COBRE) schizophrenia dataset both containing schizophrenia patients (SZs) and healthy controls (HCs) were utilized in our work. Three typical dFNC patterns between different brain functional regions were obtained. Compared to the spatial modules of HCs, the aberrant connections among auditory network, somatomotor, visual, cognitive control and cerebellar networks in 82 subject dataset and COBRE dataset were detected. Four temporal states reveal significant differences between SZs and HCs for these two datasets. Additionally, the accuracy values for SZs and HCs classification based on time-varying weights are larger than 0.96.Significance.This study significantly excavates spatio-temporal patterns for schizophrenia disease.

Deep Fusion of Multi-Template Using Spatio-Temporal Weighted Multi-Hypergraph Convolutional Networks for Brain Disease Analysis

Conventional functional connectivity network (FCN) based on resting-state fMRI (rs-fMRI) can only reflect the relationship between pairwise brain regions. Thus, the hyper-connectivity network (HCN) has been widely used to reveal high-order interactions among multiple brain regions. However, existing HCN models are essentially spatial HCN, which reflect the spatial relevance of multiple brain regions, but ignore the temporal correlation among multiple time points. Furthermore, the majority of HCN construction and learning frameworks are limited to using a single template, while the multi-template carries richer information. To address these issues, we first employ multiple templates to parcellate the rs-fMRI into different brain regions. Then, based on the multi-template data, we propose a spatio-temporal weighted HCN (STW-HCN) to capture more comprehensive high-order temporal and spatial properties of brain activity. Next, a novel deep fusion model of multi-template called spatio-temporal weighted multi-hypergraph convolutional network (STW-MHGCN) is proposed to fuse the STW-HCN of multiple templates, which extracts the deep interrelation information between different templates. Finally, we evaluate our method on the ADNI-2 and ABIDE-I datasets for mild cognitive impairment (MCI) and autism spectrum disorder (ASD) analysis. Experimental results demonstrate that the proposed method is superior to the state-of-the-art approaches in MCI and ASD classification, and the abnormal spatio-temporal hyper-edges discovered by our method have significant significance for the brain abnormalities analysis of MCI and ASD.

Enhancing EEG and sEMG Fusion Decoding Using a Multi-Scale Parallel Convolutional Network With Attention Mechanism

Electroencephalography (EEG) and surface electromyography (sEMG) have been widely used in the rehabilitation training of motor function. However, EEG signals have poor user adaptability and low classification accuracy in practical applications, and sEMG signals are susceptible to abnormalities such as muscle fatigue and weakness, resulting in reduced stability. To improve the accuracy and stability of interactive training recognition systems, we propose a novel approach called the Attention Mechanism-based Multi-Scale Parallel Convolutional Network (AM-PCNet) for recognizing and decoding fused EEG and sEMG signals. Firstly, we design an experimental scheme for the synchronous collection of EEG and sEMG signals and propose an ERP-WTC analysis method for channel screening of EEG signals. Then, the AM-PCNet network is designed to extract the time-domain, frequency-domain, and mixed-domain information of the EEG and sEMG fusion spectrogram images, and the attention mechanism is introduced to extract more fine-grained multi-scale feature information of the EEG and sEMG signals. Experiments on datasets obtained in the laboratory have shown that the average accuracy of EEG and sEMG fusion decoding is 96.62%. The accuracy is significantly improved compared with the classification performance of single-mode signals. When the muscle fatigue level reaches 50% and 90%, the accuracy is 92.84% and 85.29%, respectively. This study indicates that using this model to fuse EEG and sEMG signals can improve the accuracy and stability of hand rehabilitation training for patients.

Adversarially Trained Persistent Homology Based Graph Convolutional Network for Disease Identification Using Brain Connectivity

Brain disease propagation is associated with characteristic alterations in the structural and functional connectivity networks of the brain. To identify disease-specific network representations, graph convolutional networks (GCNs) have been used because of their powerful graph embedding ability to characterize the non-Euclidean structure of brain networks. However, existing GCNs generally focus on learning the discriminative region of interest (ROI) features, often ignoring important topological information that enables the integration of connectome patterns of brain activity. In addition, most methods fail to consider the vulnerability of GCNs to perturbations in network properties of the brain, which considerably degrades the reliability of diagnosis results. In this study, we propose an adversarially trained persistent homology-based graph convolutional network (ATPGCN) to capture disease-specific brain connectome patterns and classify brain diseases. First, the brain functional/structural connectivity is constructed using different neuroimaging modalities. Then, we develop a novel strategy that concatenates the persistent homology features from a brain algebraic topology analysis with readout features of the global pooling layer of a GCN model to collaboratively learn the individual-level representation. Finally, we simulate the adversarial perturbations by targeting the risk ROIs from clinical prior, and incorporate them into a training loop to evaluate the robustness of the model. The experimental results on three independent datasets demonstrate that ATPGCN outperforms existing classification methods in disease identification and is robust to minor perturbations in network architecture. Our code is available at https://github.com/CYB08/ATPGCN.

Mapping Multi-Modal Brain Connectome for Brain Disorder Diagnosis via Cross-Modal Mutual Learning

Recently, the study of multi-modal brain connectome has recorded a tremendous increase and facilitated the diagnosis of brain disorders. In this paradigm, functional and structural networks, e.g., functional and structural connectivity derived from fMRI and DTI, are in some manner interacted but are not necessarily linearly related. Accordingly, there remains a great challenge to leverage complementary information for brain connectome analysis. Recently, Graph Convolutional Networks (GNN) have been widely applied to the fusion of multi-modal brain connectome. However, most existing GNN methods fail to couple inter-modal relationships. In this regard, we propose a Cross-modal Graph Neural Network (Cross-GNN) that captures inter-modal dependencies through dynamic graph learning and mutual learning. Specifically, the inter-modal representations are attentively coupled into a compositional space for reasoning inter-modal dependencies. Additionally, we investigate mutual learning in explicit and implicit ways: (1) Cross-modal representations are obtained by cross-embedding explicitly based on the inter-modal correspondence matrix. (2) We propose a cross-modal distillation method to implicitly regularize latent representations with cross-modal semantic contexts. We carry out statistical analysis on the attentively learned correspondence matrices to evaluate inter-modal relationships for associating disease biomarkers. Our extensive experiments on three datasets demonstrate the superiority of our proposed method for disease diagnosis with promising prediction performance and multi-modal connectome biomarker location.

Dynamic functional connectivity analysis with temporal convolutional network for attention deficit/hyperactivity disorder identification

Introduction

Dynamic functional connectivity (dFC), which can capture the abnormality of brain activity over time in resting-state functional magnetic resonance imaging (rs-fMRI) data, has a natural advantage in revealing the abnormal mechanism of brain activity in patients with Attention Deficit/Hyperactivity Disorder (ADHD). Several deep learning methods have been proposed to learn dynamic changes from rs-fMRI for FC analysis, and achieved superior performance than those using static FC. However, most existing methods only consider dependencies of two adjacent timestamps, which is limited when the change is related to the course of many timestamps.

Methods

In this paper, we propose a novel Temporal Dependence neural Network (TDNet) for FC representation learning and temporal-dependence relationship tracking from rs-fMRI time series for automated ADHD identification. Specifically, we first partition rs-fMRI time series into a sequence of consecutive and non-overlapping segments. For each segment, we design an FC generation module to learn more discriminative representations to construct dynamic FCs. Then, we employ the Temporal Convolutional Network (TCN) to efficiently capture long-range temporal patterns with dilated convolutions, followed by three fully connected layers for disease prediction.

Results

As the results, we found that considering the dynamic characteristics of rs-fMRI time series data is beneficial to obtain better diagnostic performance. In addition, dynamic FC networks generated in a data-driven manner are more informative than those constructed by Pearson correlation coefficients.

Discussion

We validate the effectiveness of the proposed approach through extensive experiments on the public ADHD-200 database, and the results demonstrate the superiority of the proposed model over state-of-the-art methods in ADHD identification.

Artificial intelligence for diagnostic and prognostic neuroimaging in dementia: A systematic review

INTRODUCTION

Artificial intelligence (AI) and neuroimaging offer new opportunities for diagnosis and prognosis of dementia.

METHODS

We systematically reviewed studies reporting AI for neuroimaging in diagnosis and/or prognosis of cognitive neurodegenerative diseases.

RESULTS

A total of 255 studies were identified. Most studies relied on the Alzheimer's Disease Neuroimaging Initiative dataset. Algorithmic classifiers were the most commonly used AI method (48%) and discriminative models performed best for differentiating Alzheimer's disease from controls. The accuracy of algorithms varied with the patient cohort, imaging modalities, and stratifiers used. Few studies performed validation in an independent cohort.

DISCUSSION

The literature has several methodological limitations including lack of sufficient algorithm development descriptions and standard definitions. We make recommendations to improve model validation including addressing key clinical questions, providing sufficient description of AI methods and validating findings in independent datasets. Collaborative approaches between experts in AI and medicine will help achieve the promising potential of AI tools in practice.

HIGHLIGHTS

There has been a rapid expansion in the use of machine learning for diagnosis and prognosis in neurodegenerative disease Most studies (71%) relied on the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset with no other individual dataset used more than five times There has been a recent rise in the use of more complex discriminative models (e.g., neural networks) that performed better than other classifiers for classification of AD vs healthy controls We make recommendations to address methodological considerations, addressing key clinical questions, and validation We also make recommendations for the field more broadly to standardize outcome measures, address gaps in the literature, and monitor sources of bias.

Alzheimer's Disease Prediction via Brain Structural-Functional Deep Fusing Network

Fusing structural-functional images of the brain has shown great potential to analyze the deterioration of Alzheimer's disease (AD). However, it is a big challenge to effectively fuse the correlated and complementary information from multimodal neuroimages. In this work, a novel model termed cross-modal transformer generative adversarial network (CT-GAN) is proposed to effectively fuse the functional and structural information contained in functional magnetic resonance imaging (fMRI) and diffusion tensor imaging (DTI). The CT-GAN can learn topological features and generate multimodal connectivity from multimodal imaging data in an efficient end-to-end manner. Moreover, the swapping bi-attention mechanism is designed to gradually align common features and effectively enhance the complementary features between modalities. By analyzing the generated connectivity features, the proposed model can identify AD-related brain connections. Evaluations on the public ADNI dataset show that the proposed CT-GAN can dramatically improve prediction performance and detect AD-related brain regions effectively. The proposed model also provides new insights into detecting AD-related abnormal neural circuits.

Data-driven multimodal fusion: approaches and applications in psychiatric research

In the era of big data, where vast amounts of information are being generated and collected at an unprecedented rate, there is a pressing demand for innovative data-driven multi-modal fusion methods. These methods aim to integrate diverse neuroimaging perspectives to extract meaningful insights and attain a more comprehensive understanding of complex psychiatric disorders. However, analyzing each modality separately may only reveal partial insights or miss out on important correlations between different types of data. This is where data-driven multi-modal fusion techniques come into play. By combining information from multiple modalities in a synergistic manner, these methods enable us to uncover hidden patterns and relationships that would otherwise remain unnoticed. In this paper, we present an extensive overview of data-driven multimodal fusion approaches with or without prior information, with specific emphasis on canonical correlation analysis and independent component analysis. The applications of such fusion methods are wide-ranging and allow us to incorporate multiple factors such as genetics, environment, cognition, and treatment outcomes across various brain disorders. After summarizing the diverse neuropsychiatric magnetic resonance imaging fusion applications, we further discuss the emerging neuroimaging analyzing trends in big data, such as N-way multimodal fusion, deep learning approaches, and clinical translation. Overall, multimodal fusion emerges as an imperative approach providing valuable insights into the underlying neural basis of mental disorders, which can uncover subtle abnormalities or potential biomarkers that may benefit targeted treatments and personalized medical interventions.

A comprehensive survey of complex brain network representation

Recent years have shown great merits in utilizing neuroimaging data to understand brain structural and functional changes, as well as its relationship to different neurodegenerative diseases and other clinical phenotypes. Brain networks, derived from different neuroimaging modalities, have attracted increasing attention due to their potential to gain system-level insights to characterize brain dynamics and abnormalities in neurological conditions. Traditional methods aim to pre-define multiple topological features of brain networks and relate these features to different clinical measures or demographical variables. With the enormous successes in deep learning techniques, graph learning methods have played significant roles in brain network analysis. In this survey, we first provide a brief overview of neuroimaging-derived brain networks. Then, we focus on presenting a comprehensive overview of both traditional methods and state-of-the-art deep-learning methods for brain network mining. Major models, and objectives of these methods are reviewed within this paper. Finally, we discuss several promising research directions in this field.

Autism spectrum disorder diagnosis based on deep unrolling-based spatial constraint representation

Accurate diagnosis of autism spectrum disorder (ASD) is crucial for effective treatment and prognosis. Functional brain networks (FBNs) constructed from functional magnetic resonance imaging (fMRI) have become a popular tool for ASD diagnosis. However, existing model-driven approaches used to construct FBNs lack the ability to capture potential non-linear relationships between data and labels. Moreover, most existing studies treat the FBNs construction and disease classification as separate steps, leading to large inter-subject variability in the estimated FBNs and reducing the statistical power of subsequent group comparison. To address these limitations, we propose a new approach to FBNs construction called the deep unrolling-based spatial constraint representation (DUSCR) model and integrate it with a convolutional classifier to create an end-to-end framework for ASD recognition. Specifically, the model spatial constraint representation (SCR) is solved using a proximal gradient descent algorithm, and we unroll it into deep networks using the deep unrolling algorithm. Classification is then performed using a convolutional prototype learning model. We evaluated the effectiveness of the proposed method on the ABIDE I dataset and observed a significant improvement in model performance and classification accuracy. The resting state fMRI images are preprocessed into time series data and 3D coordinates of each region of interest. The data are fed into the DUSCR model, a model for building functional brain networks using deep learning instead of traditional models, that we propose, and then the outputs are fed into the convolutional classifier with prototype learning to determine whether the patient has ASD disease.

Global Transformer and Dual Local Attention Network via Deep-Shallow Hierarchical Feature Fusion for Retinal Vessel Segmentation

Clinically, retinal vessel segmentation is a significant step in the diagnosis of fundus diseases. However, recent methods generally neglect the difference of semantic information between deep and shallow features, which fail to capture the global and local characterizations in fundus images simultaneously, resulting in the limited segmentation performance for fine vessels. In this article, a global transformer (GT) and dual local attention (DLA) network via deep-shallow hierarchical feature fusion (GT-DLA-dsHFF) are investigated to solve the above limitations. First, the GT is developed to integrate the global information in the retinal image, which effectively captures the long-distance dependence between pixels, alleviating the discontinuity of blood vessels in the segmentation results. Second, DLA, which is constructed using dilated convolutions with varied dilation rates, unsupervised edge detection, and squeeze-excitation block, is proposed to extract local vessel information, consolidating the edge details in the segmentation result. Finally, a novel deep-shallow hierarchical feature fusion (dsHFF) algorithm is studied to fuse the features in different scales in the deep learning framework, respectively, which can mitigate the attenuation of valid information in the process of feature fusion. We verified the GT-DLA-dsHFF on four typical fundus image datasets. The experimental results demonstrate our GT-DLA-dsHFF achieves superior performance against the current methods and detailed discussions verify the efficacy of the proposed three modules. Segmentation results of diseased images show the robustness of our proposed GT-DLA-dsHFF. Implementation codes will be available on https://github.com/YangLibuaa/GT-DLA-dsHFF.

Correlation between cerebral hemodynamic functional near-infrared spectroscopy and positron emission tomography for assessing mild cognitive impairment and Alzheimer's disease: An exploratory study

This study was performed to investigate the usefulness of functional near-infrared spectroscopy (fNIRS) by conducting a comparative analysis of hemodynamic activation detected by fNIRS and positron emission tomography (PET) and magnetic resonance imaging (MRI) in patients with mild cognitive impairment (MCI) and Alzheimer's disease (AD). Participants were divided into four groups: the subjective memory impairment (SMI), amnestic MCI (aMCI), non-amnestic MCI (naMCI), and AD groups. We recorded the hemodynamic response during the semantic verbal fluency task (SVFT) using a commercial wireless continuous-wave NIRS system. The correlation between the parameters of the neuroimaging assessments among the groups was analyzed. Region of interest-based comparisons showed that the four groups had significantly different hemodynamic responses during SVFT in the bilateral dorsolateral prefrontal cortex (DLPFC). The linear mixed effect model result indicates that the mean ΔHbO2 from the bilateral DLPFC regions showed a significant positive correlation to the overall FDG-PET after controlling for age and group differences in the fNIRS signals. Amyloid PET signals tended to better differentiate the AD group from other groups, and fNIRS signals tended to better differentiate the SMI group from other groups. In addition, a comparison between the group pairs revealed a mirrored pattern between the hippocampal volume and hemodynamic response in the DLPFC. The hemodynamic response detected by fNIRS showed a significant correlation with metabolic and anatomical changes associated with disease progression. Therefore, fNIRS may be considered as a screening tool to predict the hemodynamic and metabolic statuses of the brain in patients with MCI and AD.

Learning to Fuse Multiple Brain Functional Networks for Automated Autism Identification

Functional connectivity network (FCN) has become a popular tool to identify potential biomarkers for brain dysfunction, such as autism spectrum disorder (ASD). Due to its importance, researchers have proposed many methods to estimate FCNs from resting-state functional MRI (rs-fMRI) data. However, the existing FCN estimation methods usually only capture a single relationship between brain regions of interest (ROIs), e.g., linear correlation, nonlinear correlation, or higher-order correlation, thus failing to model the complex interaction among ROIs in the brain. Additionally, such traditional methods estimate FCNs in an unsupervised way, and the estimation process is independent of the downstream tasks, which makes it difficult to guarantee the optimal performance for ASD identification. To address these issues, in this paper, we propose a multi-FCN fusion framework for rs-fMRI-based ASD classification. Specifically, for each subject, we first estimate multiple FCNs using different methods to encode rich interactions among ROIs from different perspectives. Then, we use the label information (ASD vs. healthy control (HC)) to learn a set of fusion weights for measuring the importance/discrimination of those estimated FCNs. Finally, we apply the adaptively weighted fused FCN on the ABIDE dataset to identify subjects with ASD from HCs. The proposed FCN fusion framework is straightforward to implement and can significantly improve diagnostic accuracy compared to traditional and state-of-the-art methods.

BolT: Fused window transformers for fMRI time series analysis

Deep-learning models have enabled performance leaps in analysis of high-dimensional functional MRI (fMRI) data. Yet, many previous methods are suboptimally sensitive for contextual representations across diverse time scales. Here, we present BolT, a blood-oxygen-level-dependent transformer model, for analyzing multi-variate fMRI time series. BolT leverages a cascade of transformer encoders equipped with a novel fused window attention mechanism. Encoding is performed on temporally-overlapped windows within the time series to capture local representations. To integrate information temporally, cross-window attention is computed between base tokens in each window and fringe tokens from neighboring windows. To gradually transition from local to global representations, the extent of window overlap and thereby number of fringe tokens are progressively increased across the cascade. Finally, a novel cross-window regularization is employed to align high-level classification features across the time series. Comprehensive experiments on large-scale public datasets demonstrate the superior performance of BolT against state-of-the-art methods. Furthermore, explanatory analyses to identify landmark time points and regions that contribute most significantly to model decisions corroborate prominent neuroscientific findings in the literature.

Personalized Functional Connectivity Based Spatio-Temporal Aggregated Attention Network for MCI Identification

Functional connectivity (FC) networks deri- ved from resting-state magnetic resonance image (rs-fMRI) are effective biomarkers for identifying mild cognitive impairment (MCI) patients. However, most FC identification methods simply extract features from group-averaged brain templates, and neglect inter-subject functional variations. Furthermore, the existing methods generally concentrate on spatial correlation among brain regions, resulting in the inefficient capture of the fMRI temporal features. To address these limitations, we propose a novel personalized functional connectivity based dual-branch graph neural network with spatio-temporal aggregated attention (PFC-DBGNN-STAA) for MCI identification. Specifically, a personalized functional connectivity (PFC) template is firstly constructed to align 213 functional regions across samples and generate discriminative individualized FC features. Secondly, a dual-branch graph neural network (DBGNN) is conducted by aggregating features from the individual- and group-level templates with the cross-template FC, which is beneficial to improve the feature discrimination by considering dependency between templates. Finally, a spatio-temporal aggregated attention (STAA) module is investigated to capture the spatial and dynamic relationships between functional regions, which solves the limitation of insufficient temporal information utilization. We evaluate our proposed method on 442 samples from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database, and achieve the accuracies of 90.1%, 90.3%, 83.3% for normal control (NC) vs. early MCI (EMCI), EMCI vs. late MCI (LMCI), and NC vs. EMCI vs. LMCI classification tasks, respectively, indicating that our method boosts MCI identification performance and outperforms state-of-the-art methods.

Dynamic multi-site graph convolutional network for autism spectrum disorder identification

Multi-site learning has attracted increasing interests in autism spectrum disorder (ASD) identification tasks by its efficacy on capturing data heterogeneity of neuroimaging taken from different medical sites. However, existing multi-site graph convolutional network (MSGCN) often ignores the correlations between different sites, and may obtain suboptimal identification results. Moreover, current feature extraction methods characterizing temporal variations of functional magnetic resonance imaging (fMRI) signals require the time series to be of the same length and cannot be directly applied to multi-site fMRI datasets. To address these problems, we propose a dual graph based dynamic multi-site graph convolutional network (DG-DMSGCN) for multi-site ASD identification. First, a sliding-window dual-graph convolutional network (SW-DGCN) is introduced for feature extraction, simultaneously capturing temporal and spatial features of fMRI data with different series lengths. Then we aggregate the features extracted from multiple medical sites through a novel dynamic multi-site graph convolutional network (DMSGCN), which effectively considers the correlations between different sites and is beneficial to improve identification performance. We evaluate the proposed DG-DMSGCN on public ABIDE I dataset containing data from 17 medical sites. The promising results obtained by our framework outperforms the state-of-the-art methods with increase in identification accuracy, indicating that it has a potential clinical prospect for practical ASD diagnosis. Our codes are available on https://github.com/Junling-Du/DG-DMSGCN.

MCFSA-Net: A multi-scale channel fusion and spatial activation network for retinal vessel segmentation

As the only vascular tissue that can be directly viewed in vivo, retinal vessels are medically important in assisting the diagnosis of ocular and cardiovascular diseases. They generally appear as different morphologies and uneven thickness in fundus images. Therefore, the single-scale segmentation method may fail to capture abundant morphological features, suffering from the deterioration in vessel segmentation, especially for tiny vessels. To alleviate this issue, we propose a multi-scale channel fusion and spatial activation network (MCFSA-Net) for retinal vessel segmentation with emphasis on tiny ones. Specifically, the Hybrid Convolution-DropBlock (HC-Drop) is first used to extract deep features of vessels and construct multi-scale feature maps by progressive down-sampling. Then, the Channel Cooperative Attention Fusion (CCAF) module is designed to handle different morphological vessels in a multi-scale manner. Finally, the Global Spatial Activation (GSA) module is introduced to aggregate global feature information for improving the attention on tiny vessels in the spatial domain and realizing effective segmentation for them. Experiments are carried out on three datasets including DRIVE, CHASE_DB1, and STARE. Our retinal vessel segmentation method achieves Accuracy of 96.95%, 97.57%, and 97.83%, and F1 score of 82.67%, 81.82%, and 82.95% in the above datasets, respectively. Qualitative and quantitative analysis show that the proposed method outperforms current advanced vessel segmentation methods, especially for tiny vessels.

Multicenter and Multichannel Pooling GCN for Early AD Diagnosis Based on Dual-Modality Fused Brain Network

For significant memory concern (SMC) and mild cognitive impairment (MCI), their classification performance is limited by confounding features, diverse imaging protocols, and limited sample size. To address the above limitations, we introduce a dual-modality fused brain connectivity network combining resting-state functional magnetic resonance imaging (fMRI) and diffusion tensor imaging (DTI), and propose three mechanisms in the current graph convolutional network (GCN) to improve classifier performance. First, we introduce a DTI-strength penalty term for constructing functional connectivity networks. Stronger structural connectivity and bigger structural strength diversity between groups provide a higher opportunity for retaining connectivity information. Second, a multi-center attention graph with each node representing a subject is proposed to consider the influence of data source, gender, acquisition equipment, and disease status of those training samples in GCN. The attention mechanism captures their different impacts on edge weights. Third, we propose a multi-channel mechanism to improve filter performance, assigning different filters to features based on feature statistics. Applying those nodes with low-quality features to perform convolution would also deteriorate filter performance. Therefore, we further propose a pooling mechanism, which introduces the disease status information of those training samples to evaluate the quality of nodes. Finally, we obtain the final classification results by inputting the multi-center attention graph into the multi-channel pooling GCN. The proposed method is tested on three datasets (i.e., an ADNI 2 dataset, an ADNI 3 dataset, and an in-house dataset). Experimental results indicate that the proposed method is effective and superior to other related algorithms, with a mean classification accuracy of 93.05% in our binary classification tasks. Our code is available at: https://github.com/Xuegang-S.