AGGN: Attention-based glioma grading network with multi-scale feature extraction and multi-modal information fusion

A review of deep learning-based information fusion techniques for multimodal medical image classification

Multimodal medical imaging plays a pivotal role in clinical diagnosis and research, as it combines information from various imaging modalities to provide a more comprehensive understanding of the underlying pathology. Recently, deep learning-based multimodal fusion techniques have emerged as powerful tools for improving medical image classification. This review offers a thorough analysis of the developments in deep learning-based multimodal fusion for medical classification tasks. We explore the complementary relationships among prevalent clinical modalities and outline three main fusion schemes for multimodal classification networks: input fusion, intermediate fusion (encompassing single-level fusion, hierarchical fusion, and attention-based fusion), and output fusion. By evaluating the performance of these fusion techniques, we provide insight into the suitability of different network architectures for various multimodal fusion scenarios and application domains. Furthermore, we delve into challenges related to network architecture selection, handling incomplete multimodal data management, and the potential limitations of multimodal fusion. Finally, we spotlight the promising future of Transformer-based multimodal fusion techniques and give recommendations for future research in this rapidly evolving field.

Parallel Dual-Branch Fusion Network for Epileptic Seizure Prediction

Epilepsy is a prevalent chronic disorder of the central nervous system. The timely and accurate seizure prediction using the scalp Electroencephalography (EEG) signal can make patients adopt reasonable preventive measures before seizures occur and thus reduce harm to patients. In recent years, deep learning-based methods have made significant progress in solving the problem of epileptic seizure prediction. However, most current methods mainly focus on modeling short- or long-term dependence in EEG, while neglecting to consider both. In this study, we propose a Parallel Dual-Branch Fusion Network (PDBFusNet) which aims to combine the complementary advantages of Convolutional Neural Network (CNN) and Transformer. Specifically, the features of the EEG signal are first extracted using Mel Frequency Cepstral Coefficients (MFCC). Then, the extracted features are delivered into the parallel dual-branches to simultaneously capture the short- and long-term dependencies of EEG signal. Further, regarding the Transformer branch, a novel feature fusion module is developed to enhance the ability of utilizing time, frequency, and channel information. To evaluate our proposal, we perform sufficient experiments on the public epileptic EEG dataset CHB-MIT, where the accuracy, sensitivity, specificity and precision are 95.76%, 95.81%, 95.71% and 95.71%, respectively. PDBFusNet shows superior performance compared to state-of-the-art competitors, which confirms the effectiveness of our proposal.

Joint learning of multi-level dynamic brain networks for autism spectrum disorder diagnosis

Graph convolutional networks (GCNs), with their powerful ability to model non-Euclidean graph data, have shown advantages in learning representations of brain networks. However, considering the complexity, multilayeredness, and spatio-temporal dynamics of brain activities, we have identified two limitations in current GCN-based research on brain networks: 1) Most studies have focused on unidirectional information transmission across brain network levels, neglecting joint learning or bidirectional information exchange among networks. 2) Most of the existing models determine node neighborhoods by thresholding or simply binarizing the brain network, which leads to the loss of edge weight information and weakens the model's sensitivity to important information in the brain network. To address the above issues, we propose a multi-level dynamic brain network joint learning architecture based on GCN for autism spectrum disorder (ASD) diagnosis. Specifically, firstly, constructing different-level dynamic brain networks. Then, utilizing joint learning based on GCN for interactive information exchange among these multi-level brain networks. Finally, designing an edge self-attention mechanism to assign different edge weights to inter-node connections, which allows us to pick out the crucial features for ASD diagnosis. Our proposed method achieves an accuracy of 81.5 %. The results demonstrate that our method enables bidirectional transfer of high-order and low-order information, facilitating information complementarity between different levels of brain networks. Additionally, the use of edge weights enhances the representation capability of ASD-related features.

Enhanced multi-label cardiology diagnosis with channel-wise recurrent fusion

The timely detection of abnormal electrocardiogram (ECG) signals is vital for preventing heart disease. However, traditional automated cardiology diagnostic methods have the limitation of being unable to simultaneously identify multiple diseases in a segment of ECG signals, and do not consider the potential correlations between the 12-lead ECG signals. To address these issues, this paper presents a novel network architecture, denoted as Branched Convolution and Channel Fusion Network (BCCF-Net), designed for the multi-label diagnosis of ECG cardiology to achieve simultaneous identification of multiple diseases. Among them, the BCCF-Net incorporates the Channel-wise Recurrent Fusion (CRF) network, which is designed to enhance the ability to explore potential correlation information between 12 leads. Furthermore, the utilization of the squeeze and excitation (SE) attention mechanism maximizes the potential of the convolutional neural network (CNN). In order to efficiently capture complex patterns in space and time across various scales, the multi branch convolution (MBC) module has been developed. Through extensive experiments on two public datasets with seven subtasks, the efficacy and robustness of the proposed ECG multi-label classification framework have been comprehensively evaluated. The results demonstrate the superior performance of the BCCF-Net compared to other state-of-the-art algorithms. The developed framework holds practical application in clinical settings, allowing for the refined diagnosis of cardiac arrhythmias through ECG signal analysis.

ELTS-Net: An enhanced liver tumor segmentation network with augmented receptive field and global contextual information

The liver is one of the organs with the highest incidence rate in the human body, and late-stage liver cancer is basically incurable. Therefore, early diagnosis and lesion location of liver cancer are of important clinical value. This study proposes an enhanced network architecture ELTS-Net based on the 3D U-Net model, to address the limitations of conventional image segmentation methods and the underutilization of image spatial features by the 2D U-Net network structure. ELTS-Net expands upon the original network by incorporating dilated convolutions to increase the receptive field of the convolutional kernel. Additionally, an attention residual module, comprising an attention mechanism and residual connections, replaces the original convolutional module, serving as the primary components of the encoder and decoder. This design enables the network to capture contextual information globally in both channel and spatial dimensions. Furthermore, deep supervision modules are integrated between different levels of the decoder network, providing additional feedback from deeper intermediate layers. This constrains the network weights to the target regions and optimizing segmentation results. Evaluation on the LiTS2017 dataset shows improvements in evaluation metrics for liver and tumor segmentation tasks compared to the baseline 3D U-Net model, achieving 95.2% liver segmentation accuracy and 71.9% tumor segmentation accuracy, with accuracy improvements of 0.9% and 3.1% respectively. The experimental results validate the superior segmentation performance of ELTS-Net compared to other comparison models, offering valuable guidance for clinical diagnosis and treatment.

Advances in the Use of Deep Learning for the Analysis of Magnetic Resonance Image in Neuro-Oncology

Machine Learning is entering a phase of maturity, but its medical applications still lag behind in terms of practical use. The field of oncological radiology (and neuro-oncology in particular) is at the forefront of these developments, now boosted by the success of Deep-Learning methods for the analysis of medical images. This paper reviews in detail some of the most recent advances in the use of Deep Learning in this field, from the broader topic of the development of Machine-Learning-based analytical pipelines to specific instantiations of the use of Deep Learning in neuro-oncology; the latter including its use in the groundbreaking field of ultra-low field magnetic resonance imaging.

Deep wavelet scattering orthogonal fusion network for glioma IDH mutation status prediction

Accurately predicting the isocitrate dehydrogenase (IDH) mutation status of gliomas is greatly significant for formulating appropriate treatment plans and evaluating the prognoses of gliomas. Although existing studies can accurately predict the IDH mutation status of gliomas based on multimodal magnetic resonance (MR) images and machine learning methods, most of these methods cannot fully explore multimodal information and effectively predict IDH status for datasets acquired from multiple centers. To address this issue, a novel wavelet scattering (WS)-based orthogonal fusion network (WSOFNet) was proposed in this work to predict the IDH mutation status of gliomas from multiple centers. First, transformation-invariant features were extracted from multimodal MR images with a WS network, and then the multimodal WS features were used instead of the original images as the inputs of WSOFNet and were fully fused through an adaptive multimodal feature fusion module (AMF2M) and an orthogonal projection module (OPM). Finally, the fused features were input into a fully connected classifier to predict IDH mutation status. In addition, to achieve improved prediction accuracy, four auxiliary losses were also used in the feature extraction modules. The comparison results showed that the prediction area under the curve (AUC) of WSOFNet on a single-center dataset was 0.9966 and that on a multicenter dataset was approximately 0.9655, which was at least 3.9% higher than that of state-of-the-art methods. Moreover, the ablation experimental results also proved that the adaptive multimodal feature fusion strategy based on orthogonal projection could effectively improve the prediction performance of the model, especially for an external validation dataset.

The interrelationship between lower limb movement, muscle activity, and joint moment during half squat and gait

Joint moment is the resultant force of limb movements. However, estimation methods for joint moments using surface electromyography frequently use joint angles instead of limb angles. The limb angle in joint moment estimation using electromyography could clarify the effects of muscle activity on the limbs: acceleration, deceleration, or stabilization. No study has quantified the comprehensive relationship between limb movement, muscle activity, and joint moment. This study aimed to determine the influencing factors for ankle-joint moment and knee-joint moment in the sagittal plane among muscle activities and parameters related to limb movements during half squat and gait. This study included 29 healthy adults (16 female participants, 21.1 ± 2.09 years). Using inertial measurement units, thigh, shank, and foot inclination angles and angular accelerations were calculated as the parameters of limb movements. Muscle activations of the biceps femoris long head, rectus femoris, gastrocnemius, and tibialis anterior were measured. Ankle joint moment and knee-joint moment were measured using a three-dimensional motion capture system and two force plates. Regression models showed high accuracy in measuring ankle-joint moment during a half squat and gait (R2f = 0.92, 0.97, respectively) and knee-joint moment during a half squat (R2f = 0.98), but not knee-joint moment during gait (R2f = 0.63). However, only a maximum of five parameters were selected from muscle activities and limb angular information. Tibialis anterior and gastrocnemius activity were the largest contributors to ankle-joint moment during a half squat and gait, respectively, while muscle activities were not directly reflected in the knee-joint moment during either movement. Consideration of the interrelationships among limb movement, muscle activity, and joint moment is required when adjusting joint movements according to the target and aim of the therapeutic interventions.

A Global and Local Feature fused CNN architecture for the sEMG-based hand gesture recognition

Deep learning methods have been widely used for the classification of hand gestures using sEMG signals. Existing deep learning architectures only captures local spatial information and has limitations in extracting global temporal dependency to enhance the model's performance. In this paper, we propose a Global and Local Feature fused CNN (GLF-CNN) model that extracts features both globally and locally from sEMG signals to enhance the performance of hand gestures classification. The model contains two independent branches extracting local and global features each and fuses them to learn more diversified features and effectively improve the stability of gesture recognition. Besides, it also exhibits lower computational cost compared to the present approaches. We conduct experiments on five benchmark databases, including the NinaPro DB4, NinaPro DB5, BioPatRec DB1-DB3, and the Mendeley Data. The proposed model achieved the highest average accuracy of 88.34% on these databases, with a 9.96% average accuracy improvement and a 50% reduction in variance compared to the models with the same number of parameters. Moreover, the classification accuracies for the BioPatRec DB1, BioPatRec DB3 and Mendeley Data are 91.4%, 91.0% and 88.6% respectively, corresponding to an improvement of 13.2%, 41.5% and 12.2% over the respective state-of-the-art models. The experimental results demonstrate that the proposed model effectively enhances robustness, with improved gesture recognition performance and generalization ability. It contributes a new way for prosthetic control and human-machine interaction.

MDF-Net for abnormality detection by fusing X-rays with clinical data

This study investigates the effects of including patients' clinical information on the performance of deep learning (DL) classifiers for disease location in chest X-ray images. Although current classifiers achieve high performance using chest X-ray images alone, consultations with practicing radiologists indicate that clinical data is highly informative and essential for interpreting medical images and making proper diagnoses. In this work, we propose a novel architecture consisting of two fusion methods that enable the model to simultaneously process patients' clinical data (structured data) and chest X-rays (image data). Since these data modalities are in different dimensional spaces, we propose a spatial arrangement strategy, spatialization, to facilitate the multimodal learning process in a Mask R-CNN model. We performed an extensive experimental evaluation using MIMIC-Eye, a dataset comprising different modalities: MIMIC-CXR (chest X-ray images), MIMIC IV-ED (patients' clinical data), and REFLACX (annotations of disease locations in chest X-rays). Results show that incorporating patients' clinical data in a DL model together with the proposed fusion methods improves the disease localization in chest X-rays by 12% in terms of Average Precision compared to a standard Mask R-CNN using chest X-rays alone. Further ablation studies also emphasize the importance of multimodal DL architectures and the incorporation of patients' clinical data in disease localization. In the interest of fostering scientific reproducibility, the architecture proposed within this investigation has been made publicly accessible( https://github.com/ChihchengHsieh/multimodal-abnormalities-detection ).

Research on obstacle avoidance optimization and path planning of autonomous vehicles based on attention mechanism combined with multimodal information decision-making thoughts of robots

With the development of machine perception and multimodal information decision-making techniques, autonomous driving technology has become a crucial area of advancement in the transportation industry. The optimization of vehicle navigation, path planning, and obstacle avoidance tasks is of paramount importance. In this study, we explore the use of attention mechanisms in a end-to-end architecture for optimizing obstacle avoidance and path planning in autonomous driving vehicles. We position our research within the broader context of robotics, emphasizing the fusion of information and decision-making capabilities. The introduction of attention mechanisms enables vehicles to perceive the environment more accurately by focusing on important information and making informed decisions in complex scenarios. By inputting multimodal information, such as images and LiDAR data, into the attention mechanism module, the system can automatically learn and weigh crucial environmental features, thereby placing greater emphasis on key information during obstacle avoidance decisions. Additionally, we leverage the end-to-end architecture and draw from classical theories and algorithms in the field of robotics to enhance the perception and decision-making abilities of autonomous driving vehicles. Furthermore, we address the optimization of path planning using attention mechanisms. We transform the vehicle's navigation task into a sequential decision-making problem and employ LSTM (Long Short-Term Memory) models to handle dynamic navigation in varying environments. By applying attention mechanisms to weigh key points along the navigation path, the vehicle can flexibly select the optimal route and dynamically adjust it based on real-time conditions. Finally, we conducted extensive experimental evaluations and software experiments on the proposed end-to-end architecture on real road datasets. The method effectively avoids obstacles, adheres to traffic rules, and achieves stable, safe, and efficient autonomous driving in diverse road scenarios. This research provides an effective solution for optimizing obstacle avoidance and path planning in the field of autonomous driving. Moreover, it contributes to the advancement and practical applications of multimodal information fusion in navigation, localization, and human-robot interaction.

Deep residual-dense network based on bidirectional recurrent neural network for atrial fibrillation detection

Atrial fibrillation easily leads to stroke, cerebral infarction and other complications, which will seriously harm the life and health of patients. Traditional deep learning methods have weak anti-interference and generalization ability. Therefore, we propose a new-fashioned deep residual-dense network via bidirectional recurrent neural network (RNN) model for atrial fibrillation detection. The combination of one-dimensional dense residual network and bidirectional RNN for atrial fibrillation detection simplifies the tedious feature extraction steps, and constructs the end-to-end neural network to achieve atrial fibrillation detection through data feature learning. Meanwhile, the attention mechanism is utilized to fuse the different features and extract the high-value information. The accuracy of the experimental results is 97.72%, the sensitivity and specificity are 93.09% and 98.71%, respectively compared with other methods.

Generative adversarial networks based skin lesion segmentation

Skin cancer is a serious condition that requires accurate diagnosis and treatment. One way to assist clinicians in this task is using computer-aided diagnosis tools that automatically segment skin lesions from dermoscopic images. We propose a novel adversarial learning-based framework called Efficient-GAN (EGAN) that uses an unsupervised generative network to generate accurate lesion masks. It consists of a generator module with a top-down squeeze excitation-based compound scaled path, an asymmetric lateral connection-based bottom-up path, and a discriminator module that distinguishes between original and synthetic masks. A morphology-based smoothing loss is also implemented to encourage the network to create smooth semantic boundaries of lesions. The framework is evaluated on the International Skin Imaging Collaboration Lesion Dataset. It outperforms the current state-of-the-art skin lesion segmentation approaches with a Dice coefficient, Jaccard similarity, and accuracy of 90.1%, 83.6%, and 94.5%, respectively. We also design a lightweight segmentation framework called Mobile-GAN (MGAN) that achieves comparable performance as EGAN but with an order of magnitude lower number of training parameters, thus resulting in faster inference times for low compute resource settings.

Detection and localization of multi-scale and oriented objects using an enhanced feature refinement algorithm

Object detection is a fundamental aspect of computer vision, with numerous generic object detectors proposed by various researchers. The proposed work presents a novel single-stage rotation detector that can detect oriented and multi-scale objects accurately from diverse scenarios. This detector addresses the challenges faced by current rotation detectors, such as the detection of arbitrary orientations, objects that are densely arranged, and the issue of loss discontinuity. First, the detector also adopts a progressive regression form (coarse-to-fine-grained approach) that uses both horizontal anchors (speed and higher recall) and rotating anchors (oriented objects) in cluttered backgrounds. Second, the proposed detector includes a feature refinement module that helps minimize the problems related to feature angulation and reduces the number of bounding boxes generated. Finally, to address the issue of loss discontinuity, the proposed detector utilizes a newly formulated adjustable loss function that can be extended to both single-stage and two-stage detectors. The proposed detector shows outstanding performance on benchmark datasets and significantly outperforms other state-of-the-art methods in terms of speed and accuracy.

Image-guided prostate biopsy robots: A review

At present, the incidence of prostate cancer (PCa) in men is increasing year by year. So, the early diagnosis of PCa is of great significance. Transrectal ultrasonography (TRUS)-guided biopsy is a common method for diagnosing PCa. The biopsy process is performed manually by urologists but the diagnostic rate is only 20%-30% and its reliability and accuracy can no longer meet clinical needs. The image-guided prostate biopsy robot has the advantages of a high degree of automation, does not rely on the skills and experience of operators, reduces the work intensity and operation time of urologists and so on. Capable of delivering biopsy needles to pre-defined biopsy locations with minimal needle placement errors, it makes up for the shortcomings of traditional free-hand biopsy and improves the reliability and accuracy of biopsy. The integration of medical imaging technology and the robotic system is an important means for accurate tumor location, biopsy puncture path planning and visualization. This paper mainly reviews image-guided prostate biopsy robots. According to the existing literature, guidance modalities are divided into magnetic resonance imaging (MRI), ultrasound (US) and fusion image. First, the robot structure research by different guided methods is the main line and the actuators and material research of these guided modalities is the auxiliary line to introduce and compare. Second, the robot image-guided localization technology is discussed. Finally, the image-guided prostate biopsy robot is summarized and suggestions for future development are provided.

Prehospital stroke-scale machine-learning model predicts the need for surgical intervention

While the development of prehospital diagnosis scales has been reported in various regions, we have also developed a scale to predict stroke type using machine learning. In the present study, we aimed to assess for the first time a scale that predicts the need for surgical intervention across stroke types, including subarachnoid haemorrhage and intracerebral haemorrhage. A multicentre retrospective study was conducted within a secondary medical care area. Twenty-three items, including vitals and neurological symptoms, were analysed in adult patients suspected of having a stroke by paramedics. The primary outcome was a binary classification model for predicting surgical intervention based on eXtreme Gradient Boosting (XGBoost). Of the 1143 patients enrolled, 765 (70%) were used as the training cohort, and 378 (30%) were used as the test cohort. The XGBoost model predicted stroke requiring surgical intervention with high accuracy in the test cohort, with an area under the receiver operating characteristic curve of 0.802 (sensitivity 0.748, specificity 0.853). We found that simple survey items, such as the level of consciousness, vital signs, sudden headache, and speech abnormalities were the most significant variables for accurate prediction. This algorithm can be useful for prehospital stroke management, which is crucial for better patient outcomes.

CVCL: Context-aware Voxel-wise Contrastive Learning for label-efficient multi-organ segmentation

Despite the significant performance improvement on multi-organ segmentation with supervised deep learning-based methods, the label-hungry nature hinders their applications in practical disease diagnosis and treatment planning. Due to the challenges in obtaining expert-level accurate, densely annotated multi-organ datasets, label-efficient segmentation, such as partially supervised segmentation trained on partially labeled datasets or semi-supervised medical image segmentation, has attracted increasing attention recently. However, most of these methods suffer from the limitation that they neglect or underestimate the challenging unlabeled regions during model training. To this end, we propose a novel Context-aware Voxel-wise Contrastive Learning method, referred as CVCL, to take full advantage of both labeled and unlabeled information in label-scarce datasets for a performance improvement on multi-organ segmentation. Experimental results demonstrate that our proposed method achieves superior performance than other state-of-the-art methods.

Optimizing the input feature sets and machine learning algorithms for reliable and accurate estimation of continuous, cuffless blood pressure

The advent of mobile devices, wearables and digital healthcare has unleashed a demand for accurate, reliable, and non-interventional ways to measure continuous blood pressure (BP). Many consumer products claim to measure BP with a cuffless device, but their lack of accuracy and reliability limit clinical adoption. Here, we demonstrate how multimodal feature datasets, comprising: (i) pulse arrival time (PAT); (ii) pulse wave morphology (PWM), and (iii) demographic data, can be combined with optimized Machine Learning (ML) algorithms to estimate Systolic BP (SBP), Diastolic BP (DBP) and Mean Arterial Pressure (MAP) within a 5 mmHg bias of the gold standard Intra-Arterial BP, well within the acceptable limits of the IEC/ANSI 80601-2-30 (2018) standard. Furthermore, DBP's calculated using 126 datasets collected from 31 hemodynamically compromised patients had a standard deviation within 8 mmHg, while SBP's and MAP's exceeded these limits. Using ANOVA and Levene's test for error means and standard deviations, we found significant differences in the various ML algorithms but found no significant differences amongst the multimodal feature datasets. Optimized ML algorithms and key multimodal features obtained from larger real-world data (RWD) sets could enable more reliable and accurate estimation of continuous BP in cuffless devices, accelerating wider clinical adoption.

Preparing pathological data to develop an artificial intelligence model in the nonclinical study

Artificial intelligence (AI)-based analysis has recently been adopted in the examination of histological slides via the digitization of glass slides using a digital scanner. In this study, we examined the effect of varying the staining color tone and magnification level of a dataset on the result of AI model prediction in hematoxylin and eosin stained whole slide images (WSIs). The WSIs of liver tissues with fibrosis were used as an example, and three different datasets (N20, B20, and B10) were prepared with different color tones and magnifications. Using these datasets, we built five models trained Mask R-CNN algorithm by a single or mixed dataset of N20, B20, and B10. We evaluated their model performance using the test dataset of three datasets. It was found that the models that were trained with mixed datasets (models B20/N20 and B10/B20), which consist of different color tones or magnifications, performed better than the single dataset trained models. Consequently, superior performance of the mixed models was obtained from the actual prediction results of the test images. We suggest that training the algorithm with various staining color tones and multi-scaled image datasets would be more optimized for consistent remarkable performance in predicting pathological lesions of interest.

AA-WGAN: Attention augmented Wasserstein generative adversarial network with application to fundus retinal vessel segmentation

In this paper, a novel attention augmented Wasserstein generative adversarial network (AA-WGAN) is proposed for fundus retinal vessel segmentation, where a U-shaped network with attention augmented convolution and squeeze-excitation module is designed to serve as the generator. In particular, the complex vascular structures make some tiny vessels hard to segment, while the proposed AA-WGAN can effectively handle such imperfect data property, which is competent in capturing the dependency among pixels in the whole image to highlight the regions of interests via the applied attention augmented convolution. By applying the squeeze-excitation module, the generator is able to pay attention to the important channels of the feature maps, and the useless information can be suppressed as well. In addition, gradient penalty method is adopted in the WGAN backbone to alleviate the phenomenon of generating large amounts of repeated images due to excessive concentration on accuracy. The proposed model is comprehensively evaluated on three datasets DRIVE, STARE, and CHASE_DB1, and the results show that the proposed AA-WGAN is a competitive vessel segmentation model as compared with several other advanced models, which obtains the accuracy of 96.51%, 97.19% and 96.94% on each dataset, respectively. The effectiveness of the applied important components is validated by ablation study, which also endows the proposed AA-WGAN with considerable generalization ability.