Deep Learning in Medical Image Analysis

Multi-modal representation learning in retinal imaging using self-supervised learning for enhanced clinical predictions

Self-supervised learning has become the cornerstone of building generalizable and transferable artificial intelligence systems in medical imaging. In particular, contrastive representation learning techniques trained on large multi-modal datasets have demonstrated impressive capabilities of producing highly transferable representations for different downstream tasks. In ophthalmology, large multi-modal datasets are abundantly available and conveniently accessible as modern retinal imaging scanners acquire both 2D fundus images and 3D optical coherence tomography (OCT) scans to assess the eye. In this context, we introduce a novel multi-modal contrastive learning-based pipeline to facilitate learning joint representations for the two retinal imaging modalities. After self-supervised pre-training on 153,306 scan pairs, we show that such a pre-training framework can provide both a retrieval system and encoders that produce comprehensive OCT and fundus image representations that generalize well for various downstream tasks on three independent external datasets, explicitly focusing on clinically pertinent prediction tasks. In addition, we show that interchanging OCT with lower-cost fundus imaging can preserve the predictive power of the trained models.

Application and performance enhancement of FAIMS spectral data for deep learning analysis using generative adversarial network reinforcement

When using High-field asymmetric ion mobility spectrometry (FAIMS) to process complex mixtures for deep learning analysis, there is a problem of poor recognition performance due to the lack of high-quality data and low sample diversity. In this paper, a Generative Adversarial Network (GAN) method is introduced to simulate and generate highly realistic and diverse spectral for expanding the dataset using real mixture spectral data of 15 classes collected by FAIMS. The mixed datasets were put into VGG and ResNeXt for testing respectively, and the experimental results proved that the best recognition effect was achieved when the ratio of real data to generated data was 1:4: where accuracy improved by 24.19 % and 6.43 %; precision improved by 23.71 % and 6.97 %; recall improved by 21.08 % and 7.09 %; and F1-score improved by 24.50 % and 8.23 %. The above results strongly demonstrate that GAN can effectively expand the data volume and increase the sample diversity without increasing the additional experimental cost, which significantly enhances the experimental effect of FAIMS spectral for the analysis of complex mixtures.

A medical image classification method based on self-regularized adversarial learning

BACKGROUND

Deep learning (DL) techniques have been extensively applied in medical image classification. The unique characteristics of medical imaging data present challenges, including small labeled datasets, severely imbalanced class distribution, and significant variations in imaging quality. Recently, generative adversarial network (GAN)-based classification methods have gained attention for their ability to enhance classification accuracy by incorporating realistic GAN-generated images as data augmentation. However, the performance of these GAN-based methods often relies on high-quality generated images, while large amounts of training data are required to train GAN models to achieve optimal performance.

PURPOSE

In this study, we propose an adversarial learning-based classification framework to achieve better classification performance. Innovatively, GAN models are employed as supplementary regularization terms to support classification, aiming to address the challenges described above.

METHODS

The proposed classification framework, GAN-DL, consists of a feature extraction network (F-Net), a classifier, and two adversarial networks, specifically a reconstruction network (R-Net) and a discriminator network (D-Net). The F-Net extracts features from input images, and the classifier uses these features for classification tasks. R-Net and D-Net have been designed following the GAN architecture. R-Net employs the extracted feature to reconstruct the original images, while D-Net is tasked with the discrimination between the reconstructed image and the original images. An iterative adversarial learning strategy is designed to guide model training by incorporating multiple network-specific loss functions. These loss functions, serving as supplementary regularization, are automatically derived during the reconstruction process and require no additional data annotation.

RESULTS

To verify the model's effectiveness, we performed experiments on two datasets, including a COVID-19 dataset with 13 958 chest x-ray images and an oropharyngeal squamous cell carcinoma (OPSCC) dataset with 3255 positron emission tomography images. Thirteen classic DL-based classification methods were implemented on the same datasets for comparison. Performance metrics included precision, sensitivity, specificity, andF 1 $F_1$ -score. In addition, we conducted ablation studies to assess the effects of various factors on model performance, including the network depth of F-Net, training image size, training dataset size, and loss function design. Our method achieved superior performance than all comparative methods. On the COVID-19 dataset, our method achieved95.4 % ± 0.6 % $95.4\%\pm 0.6\%$ ,95.3 % ± 0.9 % $95.3\%\pm 0.9\%$ ,97.7 % ± 0.4 % $97.7\%\pm 0.4\%$ , and95.3 % ± 0.9 % $95.3\%\pm 0.9\%$ in terms of precision, sensitivity, specificity, andF 1 $F_1$ -score, respectively. It achieved96.2 % ± 0.7 % $96.2\%\pm 0.7\%$ across all these metrics on the OPSCC dataset. The study to investigate the effects of two adversarial networks highlights the crucial role of D-Net in improving model performance. Ablation studies further provide an in-depth understanding of our methodology.

CONCLUSION

Our adversarial-based classification framework leverages GAN-based adversarial networks and an iterative adversarial learning strategy to harness supplementary regularization during training. This design significantly enhances classification accuracy and mitigates overfitting issues in medical image datasets. Moreover, its modular design not only demonstrates flexibility but also indicates its potential applicability to various clinical contexts and medical imaging applications.

Exploring cortical morphology biomarkers of amnesic mild cognitive impairment using novel fractal dimension-based structural MRI analysis

Amnestic mild cognitive impairment (aMCI) is considered as an intermediate stage of Alzheimer's disease, but no MRI biomarkers currently distinguish aMCI from healthy individuals effectively. Fractal dimension, a quantitative parameter, provides superior morphological information compared to conventional cortical thickness methods. Few studies have used cortical fractal dimension values to differentiate aMCI from healthy controls. In this study, we aim to build an automated discriminator for accurately distinguishing aMCI using fractal dimension measures of the cerebral cortex. Thirty aMCI patients and 30 health controls underwent structural MRI of the brain. First, the atrophy of participants' cortical sub-regions of Desikan-Killiany cortical atlas was assessed using fractal dimension and cortical thickness. The fractal dimension is more sensitive than cortical thickness in reducing dimensional effects and may accurately reflect morphological changes of the cortex in aMCI. The aMCI group had significantly lower fractal dimension values in the bilateral temporal lobes, right limbic lobe and right parietal lobe, whereas they showed significantly lower cortical thickness values only in the bilateral temporal lobes. Fractal dimension analysis was able to depict most of the significantly different focal regions detected by cortical thickness, but additionally with more regions. Second, applying the measured fractal dimensions (and cortical thickness) of both cerebral hemispheres, an unsupervised discriminator was built for the aMCI and healthy controls. The proposed fractal dimension-based method achieves 80.54% accuracy in discriminating aMCI from healthy controls. The fractal dimension appears to be a promising biomarker for cortical morphology changes that can discriminate patients with aMCI from healthy controls.

Advancements in automated classification of chronic obstructive pulmonary disease based on computed tomography imaging features through deep learning approaches

Chronic Obstructive Pulmonary Disease (COPD) represents a global public health issue that significantly impairs patients' quality of life and overall health. As one of the primary causes of chronic respiratory diseases and global mortality, effective diagnosis and classification of COPD are crucial for clinical management. Pulmonary function tests (PFTs) are standard for diagnosing COPD, yet their accuracy is influenced by patient compliance and other factors, and they struggle to detect early disease pathologies. Furthermore, the complexity of COPD pathological changes poses additional challenges for clinical diagnosis, increasing the difficulty for physicians in practice. Recently, deep learning (DL) technologies have demonstrated significant potential in medical image analysis, particularly for the diagnosis and classification of COPD. By analyzing key radiological features such as airway alterations, emphysema, and vascular characteristics in Computed Tomography (CT) scan images, DL enhances diagnostic accuracy and efficiency, providing more precise treatment plans for COPD patients. This article reviews the latest research advancements in DL methods based on principal radiological features of COPD for its classification and discusses the advantages, challenges, and future research directions of DL in this field, aiming to provide new perspectives for the personalized management and treatment of COPD.

A Review of Application of Deep Learning in Endoscopic Image Processing

Deep learning, particularly convolutional neural networks (CNNs), has revolutionized endoscopic image processing, significantly enhancing the efficiency and accuracy of disease diagnosis through its exceptional ability to extract features and classify complex patterns. This technology automates medical image analysis, alleviating the workload of physicians and enabling a more focused and personalized approach to patient care. However, despite these remarkable achievements, there are still opportunities to further optimize deep learning models for endoscopic image analysis, including addressing limitations such as the requirement for large annotated datasets and the challenge of achieving higher diagnostic precision, particularly for rare or subtle pathologies. This review comprehensively examines the profound impact of deep learning on endoscopic image processing, highlighting its current strengths and limitations. It also explores potential future directions for research and development, outlining strategies to overcome existing challenges and facilitate the integration of deep learning into clinical practice. Ultimately, the goal is to contribute to the ongoing advancement of medical imaging technologies, leading to more accurate, personalized, and optimized medical care for patients.

Application of machine learning to predict in-hospital mortality after transcatheter mitral valve repair

INTRODUCTION

Transcatheter mitral valve repair offers a minimally invasive treatment option for patients at high risk for traditional open repair. We sought to develop dynamic machine-learning risk prediction models for in-hospital mortality after transcatheter mitral valve repair using a national cohort.

METHODS

All adult hospitalization records involving transcatheter mitral valve repair were identified in the 2016-2020 Nationwide Readmissions Database. As a result of initial class imbalance, undersampling of the majority class and subsequent oversampling of the minority class using Synthetic Minority Oversampling TEchnique were employed in each cross-validation training fold. Machine-learning models were trained to predict patient mortality after transcatheter mitral valve repair and compared with traditional logistic regression. Shapley additive explanations plots were also developed to understand the relative impact of each feature used for training.

RESULTS

Among 2,450 patients included for analysis, the in-hospital mortality rate was 1.8%. Naïve Bayes and random forest models were the best at predicting transcatheter mitral valve repair postoperative mortality, with an area under the receiver operating characteristic curve of 0.83 ± 0.05 and 0.82 ± 0.04, respectively. Both models demonstrated superior ability to predict mortality relative to logistic regression (P < .001 for both). Medicare insurance coverage, comorbid liver disease, congestive heart failure, renal failure, and previous coronary artery bypass grafting were associated with greater predicted likelihood of in-hospital mortality, whereas elective surgery and private insurance coverage were linked with lower odds of mortality.

CONCLUSION

Machine-learning models significantly outperformed traditional regression methods in predicting in-hospital mortality after transcatheter mitral valve repair. Furthermore, we identified key patient factors and comorbidities linked with greater postoperative mortality. Future work and clinical validation are warranted to continue improving risk assessment in transcatheter mitral valve repair .

SDS-Net: A Synchronized Dual-Stage Network for Predicting Patients Within 4.5-h Thrombolytic Treatment Window Using MRI

Timely and precise identification of acute ischemic stroke (AIS) within 4.5 h is imperative for effective treatment decision-making. This study aims to construct a novel network that utilizes limited datasets to recognize AIS patients within this critical window. We conducted a retrospective analysis of 265 AIS patients who underwent both fluid attenuation inversion recovery (FLAIR) and diffusion-weighted imaging (DWI) within 24 h of symptom onset. Patients were categorized based on the time since stroke onset (TSS) into two groups: TSS ≤ 4.5 h and TSS > 4.5 h. The TSS was calculated as the time from stroke onset to MRI completion. We proposed a synchronized dual-stage network (SDS-Net) and a sequential dual-stage network (Dual-stage Net), which were comprised of infarct voxel identification and TSS classification stages. The models were trained on 181 patients and validated on an independent external cohort of 84 patients using metrics of area under the curve (AUC), sensitivity, specificity, and accuracy. A DeLong test was used to statistically compare the performance of the two models. SDS-Net achieved an accuracy of 0.844 with an AUC of 0.914 in the validation dataset, outperforming the Dual-stage Net, which had an accuracy of 0.822 and an AUC of 0.846. In the external test dataset, SDS-Net further demonstrated superior performance with an accuracy of 0.800 and an AUC of 0.879, compared to the accuracy of 0.694 and AUC of 0.744 of Dual-stage Net (P = 0.049). SDS-Net is a robust and reliable tool for identifying AIS patients within a 4.5-h treatment window using MRI. This model can assist clinicians in making timely treatment decisions, potentially improving patient outcomes.

LLM-driven multimodal target volume contouring in radiation oncology

Target volume contouring for radiation therapy is considered significantly more challenging than the normal organ segmentation tasks as it necessitates the utilization of both image and text-based clinical information. Inspired by the recent advancement of large language models (LLMs) that can facilitate the integration of the textural information and images, here we present an LLM-driven multimodal artificial intelligence (AI), namely LLMSeg, that utilizes the clinical information and is applicable to the challenging task of 3-dimensional context-aware target volume delineation for radiation oncology. We validate our proposed LLMSeg within the context of breast cancer radiotherapy using external validation and data-insufficient environments, which attributes highly conducive to real-world applications. We demonstrate that the proposed multimodal LLMSeg exhibits markedly improved performance compared to conventional unimodal AI models, particularly exhibiting robust generalization performance and data-efficiency.

Automated volumetric analysis of the inner ear fluid space from hydrops magnetic resonance imaging using 3D neural networks

Due to the development of magnetic resonance (MR) imaging processing technology, image-based identification of endolymphatic hydrops (EH) has played an important role in understanding inner ear illnesses, such as Meniere's disease or fluctuating sensorineural hearing loss. We segmented the inner ear, consisting of the cochlea, vestibule, and semicircular canals, using a 3D-based deep neural network model for accurate and automated EH volume ratio calculations. We built a dataset of MR cisternography (MRC) and HYDROPS-Mi2 stacks labeled with the segmentation of the perilymph fluid space and endolymph fluid space of the inner ear to devise a 3D segmentation deep neural network model. End-to-end learning was used to segment the perilymph fluid and the endolymph fluid spaces simultaneously using aligned pair data of the MRC and HYDROPS-Mi2 stacks. Consequently, the segmentation performance of the total fluid space and endolymph fluid space had Dice similarity coefficients of 0.9574 and 0.9186, respectively. In addition, the EH volume ratio calculated by experienced otologists and the EH volume ratio value predicted by the proposed deep learning model showed high agreement according to the interclass correlation coefficient (ICC) and Bland-Altman plot analysis.

Image Captioning Based on Semantic Scenes

With the development of artificial intelligence and deep learning technologies, image captioning has become an important research direction at the intersection of computer vision and natural language processing. The purpose of image captioning is to generate corresponding natural language descriptions by understanding the content of images. This technology has broad application prospects in fields such as image retrieval, autonomous driving, and visual question answering. Currently, many researchers have proposed region-based image captioning methods. These methods generate captions by extracting features from different regions of an image. However, they often rely on local features of the image and overlook the understanding of the overall scene, leading to captions that lack coherence and accuracy when dealing with complex scenes. Additionally, image captioning methods are unable to extract complete semantic information from visual data, which may lead to captions with biases and deficiencies. Due to these reasons, existing methods struggle to generate comprehensive and accurate captions. To fill this gap, we propose the Semantic Scenes Encoder (SSE) for image captioning. It first extracts a scene graph from the image and integrates it into the encoding of the image information. Then, it extracts a semantic graph from the captions and preserves semantic information through a learnable attention mechanism, which we refer to as the dictionary. During the generation of captions, it combines the encoded information of the image and the learned semantic information to generate complete and accurate captions. To verify the effectiveness of the SSE, we tested the model on the MSCOCO dataset. The experimental results show that the SSE improves the overall quality of the captions. The improvement in scores across multiple evaluation metrics further demonstrates that the SSE possesses significant advantages when processing identical images.

Cross-modality image translation of 3 Tesla Magnetic Resonance Imaging to 7 Tesla using Generative Adversarial Networks

The rapid advancements in magnetic resonance imaging (MRI) technology have precipitated a new paradigm wherein cross-modality data translation across diverse imaging platforms, field strengths, and different sites is increasingly challenging. This issue is particularly accentuated when transitioning from 3 Tesla (3T) to 7 Tesla (7T) MRI systems. This study proposes a novel solution to these challenges using generative adversarial networks (GANs)-specifically, the CycleGAN architecture-to create synthetic 7T images from 3T data. Employing a dataset of 1112 and 490 unpaired 3T and 7T MR images, respectively, we trained a 2-dimensional (2D) CycleGAN model, evaluating its performance on a paired dataset of 22 participants scanned at 3T and 7T. Independent testing on 22 distinct participants affirmed the model's proficiency in accurately predicting various tissue types, encompassing cerebral spinal fluid, gray matter, and white matter. Our approach provides a reliable and efficient methodology for synthesizing 7T images, achieving a median Dice of 6.82%,7,63%, and 4.85% for Cerebral Spinal Fluid (CSF), Gray Matter (GM), and White Matter (WM), respectively, in the testing dataset, thereby significantly aiding in harmonizing heterogeneous datasets. Furthermore, it delineates the potential of GANs in amplifying the contrast-to-noise ratio (CNR) from 3T, potentially enhancing the diagnostic capability of the images. While acknowledging the risk of model overfitting, our research underscores a promising progression towards harnessing the benefits of 7T MR systems in research investigations while preserving compatibility with existent 3T MR data. This work was previously presented at the ISMRM 2021 conference (Diniz, Helmet, Santini, Aizenstein, & Ibrahim, 2021).

fMRI-based Alzheimer's disease detection via functional connectivity analysis: a systematic review

Alzheimer's disease is a common brain disorder affecting many people worldwide. It is the primary cause of dementia and memory loss. The early diagnosis of Alzheimer's disease is essential to provide timely care to AD patients and prevent the development of symptoms of this disease. Various non-invasive techniques can be utilized to diagnose Alzheimer's in its early stages. These techniques include functional magnetic resonance imaging, electroencephalography, positron emission tomography, and diffusion tensor imaging. They are mainly used to explore functional and structural connectivity of human brains. Functional connectivity is essential for understanding the co-activation of certain brain regions co-activation. This systematic review scrutinizes various works of Alzheimer's disease detection by analyzing the learning from functional connectivity of fMRI datasets that were published between 2018 and 2024. This work investigates the whole learning pipeline including data analysis, standard preprocessing phases of fMRI, feature computation, extraction and selection, and the various machine learning and deep learning algorithms that are used to predict the occurrence of Alzheimer's disease. Ultimately, the paper analyzed results on AD and highlighted future research directions in medical imaging. There is a need for an efficient and accurate way to detect AD to overcome the problems faced by patients in the early stages.

MIDC: Medical image dataset cleaning framework based on deep learning

Deep learning technology is widely used in the field of medical imaging. Among them, Convolutional Neural Networks (CNNs) are the most widely used, and the quality of the dataset is crucial for the training of CNN diagnostic models, as mislabeled data can easily affect the accuracy of the diagnostic models. However, due to medical specialization, it is difficult for non-professional physicians to judge mislabeled medical image data. In this paper, we proposed a new framework named medical image dataset cleaning (MIDC), whose main contribution is to improve the quality of public datasets by automatically cleaning up mislabeled data. The main innovations of MIDC are: firstly, the framework innovatively utilizes multiple public datasets of the same disease, relying on different CNNs to automatically recognize images and remove mislabeled data to complete the data cleaning process. This process does not rely on annotations from professional physicians and does not require additional datasets with more reliable labels; Secondly, a novel grading rule is designed to divide the datasets into high-accuracy datasets and low-accuracy datasets, based on which the data cleaning process can be performed; Thirdly, a novel data cleaning module based on CNN is designed to identify and clean low-accuracy datasets by using high-accuracy datasets. In the experiments, the validity of the proposed framework was verified by using four kinds of datasets diabetic retinal, viral pneumonia, breast tumor, and skin cancer, with results showing an increase in the average diagnostic accuracy from 71.18 % to 85.13 %, 82.50 %to 93.79 %, 85.59 %to 93.45 %, and 84.55 %to 94.21 %, respectively. The proposed data cleaning framework MIDC could better help physicians diagnose diseases based on the dataset with mislabeled data.

Towards a general computed tomography image segmentation model for anatomical structures and lesions

Numerous deep-learning models have been developed using task-specific data, but they ignore the inherent connections among different tasks. By jointly learning a wide range of segmentation tasks, we prove that a general medical image segmentation model can improve segmentation performance for computerized tomography (CT) volumes. The proposed general CT image segmentation (gCIS) model utilizes a common transformer-based encoder for all tasks and incorporates automatic pathway modules for task prompt-based decoding. It is trained on one of the largest datasets, comprising 36,419 CT scans and 83 tasks. gCIS can automatically perform various segmentation tasks using automatic pathway modules of decoding networks through text prompt inputs, achieving an average Dice coefficient of 82.84%. Furthermore, the proposed automatic pathway routing mechanism allows for parameter pruning of the network during deployment, and gCIS can also be quickly adapted to unseen tasks with minimal training samples while maintaining great performance.

Adding the third dimension: 3D convolutional neural network diagnosis of temporal lobe epilepsy

Convolutional neural networks (CNN) show great promise for translating decades of research on structural abnormalities in temporal lobe epilepsy into clinical practice. Three-dimensional CNNs typically outperform two-dimensional CNNs in medical imaging. Here we explore for the first time whether a three-dimensional CNN outperforms a two-dimensional CNN for identifying temporal lobe epilepsy-specific features on MRI. Using 1178 T1-weighted images (589 temporal lobe epilepsy, 589 healthy controls) from 12 surgical centres, we trained 3D and 2D CNNs for temporal lobe epilepsy versus healthy control classification, using feature visualization to identify important regions. The 3D CNN was compared to the 2D model and to a randomized model (comparison to chance). Further, we explored the effect of sample size with subsampling, examined model performance based on single-subject clinical characteristics, and tested the impact of image harmonization on model performance. Across 50 datapoints (10 runs with 5-folds each) the 3D CNN median accuracy was 86.4% (35.3% above chance) and the median F1-score was 86.1% (33.3% above chance). The 3D model yielded higher accuracy compared to the 2D model on 84% of datapoints (median 2D accuracy, 83.0%), a significant outperformance for the 3D model (binomial test: P < 0.001). This advantage of the 3D model was only apparent at the highest sample size. Saliency maps exhibited the importance of medial-ventral temporal, cerebellar, and midline subcortical regions across both models for classification. However, the 3D model had higher salience in the most important regions, the ventral-medial temporal and midline subcortical regions. Importantly, the model achieved high accuracy (82% accuracy) even in patients without MRI-identifiable hippocampal sclerosis. Finally, applying ComBat for harmonization did not improve performance. These findings highlight the value of 3D CNNs for identifying subtle structural abnormalities on MRI, especially in patients without clinically identified temporal lobe epilepsy lesions. Our findings also reveal that the advantage of 3D CNNs relies on large sample sizes for model training.

Early screening of miliary tuberculosis with tuberculous meningitis based on few-shot learning with multiple windows and feature granularities

Tuberculous meningitis (TBM) is a fatal tuberculosis caused by a large number of Mycobacterium tuberculosis (M. tuberculosis) spread by blood flow, with a case fatality rate of more than 50%. It is one of the most serious complications of miliary tuberculosis (MT), whose incidence is closely related to MT. If doctors can provide early diagnosis and active treatment for TBM, the case fatality rate will be significantly reduced. At present, there is a lack of methods to predict the progression of MT to TBM in clinic. To explore whether MT cases will experience TBM, we propose an early screening model of miliary tuberculosis with tuberculous meningitis (MT-TBM) based on few-shot learning with multiple windows and feature granularities (MWFG). This model aims to screen potential TBM cases through chest computerized tomography (CT) images of MT cases. Chest CT is a routine examination for MT cases. The MWFG module can extract more comprehensive features from a set of CT images of each MT case. The softmax classifier with adaptive regularization is trained on the cooperation of support set and query set, which can effectively prevent overfitting. Experiments on a dataset of 40 MT cases with chest CT images established by the medical records demonstrate that our proposed model achieves state-of-the-art performance in the early screening of MT-TBM. It can establish the connection between MT and MT-TBM through chest CT images of MT cases. The early screening model of MT-TBM based on few-shot learning with MWFG fills the research gap in computer-aided predicting TBM and has certain clinical effects. This research can provide some reference for clinicians in early diagnosis of MT-TBM and help clinicians in the early prevention and treatment of TBM for MT patients.

Mahmoud T, Rajasekaran A, Bhatia B. Evaluating Explainable Artificial Intelligence (XAI) techniques in chest radiology imaging through a human-centered Lens

The field of radiology imaging has experienced a remarkable increase in using of deep learning (DL) algorithms to support diagnostic and treatment decisions. This rise has led to the development of Explainable AI (XAI) system to improve the transparency and trust of complex DL methods. However, XAI systems face challenges in gaining acceptance within the healthcare sector, mainly due to technical hurdles in utilizing these systems in practice and the lack of human-centered evaluation/validation. In this study, we focus on visual XAI systems applied to DL-enabled diagnostic system in chest radiography. In particular, we conduct a user study to evaluate two prominent visual XAI techniques from the human perspective. To this end, we created two clinical scenarios for diagnosing pneumonia and COVID-19 using DL techniques applied to chest X-ray and CT scans. The achieved accuracy rates were 90% for pneumonia and 98% for COVID-19. Subsequently, we employed two well-known XAI methods, Grad-CAM (Gradient-weighted Class Activation Mapping) and LIME (Local Interpretable Model-agnostic Explanations), to generate visual explanations elucidating the AI decision-making process. The visual explainability results were shared through a user study, undergoing evaluation by medical professionals in terms of clinical relevance, coherency, and user trust. In general, participants expressed a positive perception of the use of XAI systems in chest radiography. However, there was a noticeable lack of awareness regarding their value and practical aspects. Regarding preferences, Grad-CAM showed superior performance over LIME in terms of coherency and trust, although concerns were raised about its clinical usability. Our findings highlight key user-driven explainability requirements, emphasizing the importance of multi-modal explainability and the necessity to increase awareness of XAI systems among medical practitioners. Inclusive design was also identified as a crucial need to ensure better alignment of these systems with user needs.

Fully Interpretable Deep Learning Model Using IR Thermal Images for Possible Breast Cancer Cases

Breast cancer remains a global health problem requiring effective diagnostic methods for early detection, in order to achieve the World Health Organization's ultimate goal of breast self-examination. A literature review indicates the urgency of improving diagnostic methods and identifies thermography as a promising, cost-effective, non-invasive, adjunctive, and complementary detection method. This research explores the potential of using machine learning techniques, specifically Bayesian networks combined with convolutional neural networks, to improve possible breast cancer diagnosis at early stages. Explainable artificial intelligence aims to clarify the reasoning behind any output of artificial neural network-based models. The proposed integration adds interpretability of the diagnosis, which is particularly significant for a medical diagnosis. We constructed two diagnostic expert models: Model A and Model B. In this research, Model A, combining thermal images after the explainable artificial intelligence process together with medical records, achieved an accuracy of 84.07%, while model B, which also includes a convolutional neural network prediction, achieved an accuracy of 90.93%. These results demonstrate the potential of explainable artificial intelligence to improve possible breast cancer diagnosis, with very high accuracy.

Prediction of benign and malignant ground glass pulmonary nodules based on multi-feature fusion of attention mechanism

Objectives

The purpose of this study was to develop and validate a new feature fusion algorithm to improve the classification performance of benign and malignant ground-glass nodules (GGNs) based on deep learning.

Methods

We retrospectively collected 385 cases of GGNs confirmed by surgical pathology from three hospitals. We utilized 239 GGNs from Hospital 1 as the training and internal validation set, and 115 and 31 GGNs from Hospital 2 and Hospital 3, respectively, as external test sets 1 and 2. Among these GGNs, 172 were benign and 203 were malignant. First, we evaluated clinical and morphological features of GGNs at baseline chest CT and simultaneously extracted whole-lung radiomics features. Then, deep convolutional neural networks (CNNs) and backpropagation neural networks (BPNNs) were applied to extract deep features from whole-lung CT images, clinical, morphological features, and whole-lung radiomics features separately. Finally, we integrated these four types of deep features using an attention mechanism. Multiple metrics were employed to evaluate the predictive performance of the model.

Results

The deep learning model integrating clinical, morphological, radiomics and whole lung CT image features with attention mechanism (CMRI-AM) achieved the best performance, with area under the curve (AUC) values of 0.941 (95% CI: 0.898-0.972), 0.861 (95% CI: 0.823-0.882), and 0.906 (95% CI: 0.878-0.932) on the internal validation set, external test set 1, and external test set 2, respectively. The AUC differences between the CMRI-AM model and other feature combination models were statistically significant in all three groups (all p<0.05).

Conclusion

Our experimental results demonstrated that (1) applying attention mechanism to fuse whole-lung CT images, radiomics features, clinical, and morphological features is feasible, (2) clinical, morphological, and radiomics features provide supplementary information for the classification of benign and malignant GGNs based on CT images, and (3) utilizing baseline whole-lung CT features to predict the benign and malignant of GGNs is an effective method. Therefore, optimizing the fusion of baseline whole-lung CT features can effectively improve the classification performance of GGNs.

Truncated M13 phage for smart detection of E. coli under dark field

BACKGROUND

The urgent need for affordable and rapid detection methodologies for foodborne pathogens, particularly Escherichia coli (E. coli), highlights the importance of developing efficient and widely accessible diagnostic systems. Dark field microscopy, although effective, requires specific isolation of the target bacteria which can be hindered by the high cost of producing specialized antibodies. Alternatively, M13 bacteriophage, which naturally targets E. coli, offers a cost-efficient option with well-established techniques for its display and modification. Nevertheless, its filamentous structure with a large length-diameter ratio contributes to nonspecific binding and low separation efficiency, posing significant challenges. Consequently, refining M13 phage methodologies and their integration with advanced microscopy techniques stands as a critical pathway to improve detection specificity and efficiency in food safety diagnostics.

METHODS

We employed a dual-plasmid strategy to generate a truncated M13 phage (tM13). This engineered tM13 incorporates two key genetic modifications: a partial mutation at the N-terminus of pIII and biotinylation at the hydrophobic end of pVIII. These alterations enable efficient attachment of tM13 to diverse E. coli strains, facilitating rapid magnetic separation. For detection, we additionally implemented a convolutional neural network (CNN)-based algorithm for precise identification and quantification of bacterial cells using dark field microscopy.

RESULTS

The results obtained from spike-in and clinical sample analyses demonstrated the accuracy, high sensitivity (with a detection limit of 10 CFU/μL), and time-saving nature (30 min) of our tM13-based immunomagnetic enrichment approach combined with AI-enabled analytics, thereby supporting its potential to facilitate the identification of diverse E. coli strains in complex samples.

CONCLUSION

The study established a rapid and accurate detection strategy for E. coli utilizing truncated M13 phages as capture probes, along with a dark field microscopy detection platform that integrates an image processing model and convolutional neural network.

Automated Neural Architecture Search for Cardiac Amyloidosis Classification from [18F]-Florbetaben PET Images

Medical image classification using convolutional neural networks (CNNs) is promising but often requires extensive manual tuning for optimal model definition. Neural architecture search (NAS) automates this process, reducing human intervention significantly. This study applies NAS to [18F]-Florbetaben PET cardiac images for classifying cardiac amyloidosis (CA) sub-types (amyloid light chain (AL) and transthyretin amyloid (ATTR)) and controls. Following data preprocessing and augmentation, an evolutionary cell-based NAS approach with a fixed network macro-structure is employed, automatically deriving cells' micro-structure. The algorithm is executed five times, evaluating 100 mutating architectures per run on an augmented dataset of 4048 images (originally 597), totaling 5000 architectures evaluated. The best network (NAS-Net) achieves 76.95% overall accuracy. K-fold analysis yields mean ± SD percentages of sensitivity, specificity, and accuracy on the test dataset: AL subjects (98.7 ± 2.9, 99.3 ± 1.1, 99.7 ± 0.7), ATTR-CA subjects (93.3 ± 7.8, 78.0 ± 2.9, 70.9 ± 3.7), and controls (35.8 ± 14.6, 77.1 ± 2.0, 96.7 ± 4.4). NAS-derived network performance rivals manually determined networks in the literature while using fewer parameters, validating its automatic approach's efficacy.

Deep learning to assess bone quality from panoramic radiographs: the feasibility of clinical application through comparison with an implant surgeon and cone-beam computed tomography

PURPOSE

Bone quality is one of the most important clinical factors for the primary stability and successful osseointegration of dental implants. This preliminary pilot study aimed to evaluate the clinical applicability of deep learning (DL) for assessing bone quality using panoramic (PA) radiographs compared with an implant surgeon's subjective tactile sense and cone-beam computed tomography (CBCT) values.

METHODS

In total, PA images of 2,270 edentulous sites for implant placement were selected, and the corresponding CBCT relative gray value measurements and bone quality classification were performed using 3-dimensional dental image analysis software. Based on the pre-trained and fine-tuned ResNet-50 architecture, the bone quality classification of PA images was classified into 4 levels, from D1 to D4, and Spearman correlation analyses were performed with the implant surgeon's tactile sense and CBCT values.

RESULTS

The classification accuracy of DL was evaluated using a test dataset comprising 454 cropped PA images, and it achieved an area under the receiving characteristic curve of 0.762 (95% confidence interval [CI], 0.714-0.810). Spearman correlation analysis of bone quality showed significant positive correlations with the CBCT classification (r=0.702; 95% CI, 0.651-0.747; P<0.001) and the surgeon's tactile sense (r=0.658; 95% CI, 0.600-0.708, P<0.001) versus the DL classification.

CONCLUSIONS

DL classification using PA images showed a significant and consistent correlation with CBCT classification and the surgeon's tactile sense in classifying the bone quality at the implant placement site. Further research based on high-quality quantitative datasets is essential to increase the reliability and validity of this method for actual clinical applications.

Accurate segmentation of intracellular organelle networks using low-level features and topological self-similarity

MOTIVATION

Intracellular organelle networks (IONs) such as the endoplasmic reticulum (ER) network and the mitochondrial (MITO) network serve crucial physiological functions. The morphology of these networks plays a critical role in mediating their functions. Accurate image segmentation is required for analyzing the morphology and topology of these networks for applications such as molecular mechanism analysis and drug target screening. So far, however, progress has been hindered by their structural complexity and density.

RESULTS

In this study, we first establish a rigorous performance baseline for accurate segmentation of these organelle networks from fluorescence microscopy images by optimizing a baseline U-Net model. We then develop the multi-resolution encoder (MRE) and the hierarchical fusion loss (Lhf) based on two inductive components, namely low-level features and topological self-similarity, to assist the model in better adapting to the task of segmenting IONs. Empowered by MRE and Lhf, both U-Net and Pyramid Vision Transformer (PVT) outperform competing state-of-the-art models such as U-Net++, HR-Net, nnU-Net, and TransUNet on custom datasets of the ER network and the MITO network, as well as on public datasets of another biological network, the retinal blood vessel network. In addition, integrating MRE and Lhf with models such as HR-Net and TransUNet also enhances their segmentation performance. These experimental results confirm the generalization capability and potential of our approach. Furthermore, accurate segmentation of the ER network enables analysis that provides novel insights into its dynamic morphological and topological properties.

AVAILABILITY AND IMPLEMENTATION

Code and data are openly accessible at https://github.com/cbmi-group/MRE.

The Integration of Radiomics and Artificial Intelligence in Modern Medicine

With profound effects on patient care, the role of artificial intelligence (AI) in radiomics has become a disruptive force in contemporary medicine. Radiomics, the quantitative feature extraction and analysis from medical images, offers useful imaging biomarkers that can reveal important information about the nature of diseases, how well patients respond to treatment and patient outcomes. The use of AI techniques in radiomics, such as machine learning and deep learning, has made it possible to create sophisticated computer-aided diagnostic systems, predictive models, and decision support tools. The many uses of AI in radiomics are examined in this review, encompassing its involvement of quantitative feature extraction from medical images, the machine learning, deep learning and computer-aided diagnostic (CAD) systems approaches in radiomics, and the effect of radiomics and AI on improving workflow automation and efficiency, optimize clinical trials and patient stratification. This review also covers the predictive modeling improvement by machine learning in radiomics, the multimodal integration and enhanced deep learning architectures, and the regulatory and clinical adoption considerations for radiomics-based CAD. Particular emphasis is given to the enormous potential for enhancing diagnosis precision, treatment personalization, and overall patient outcomes.

Dose-Incorporated Deep Ensemble Learning for Improving Brain Metastasis Stereotactic Radiosurgery Outcome Prediction

PURPOSE

To develop a novel deep ensemble learning model for accurate prediction of brain metastasis (BM) local control outcomes after stereotactic radiosurgery (SRS).

METHODS AND MATERIALS

A total of 114 brain metastases (BMs) from 82 patients were evaluated, including 26 BMs that developed biopsy-confirmed local failure post-SRS. The SRS spatial dose distribution (Dmap) of each BM was registered to the planning contrast-enhanced T1 (T1-CE) magnetic resonance imaging (MRI). Axial slices of the Dmap, T1-CE, and planning target volume (PTV) segmentation (PTVseg) intersecting the BM center were extracted within a fixed field of view determined by the 60% isodose volume in Dmap. A spherical projection was implemented to transform planar image content onto a spherical surface using multiple projection centers, and the resultant T1-CE/Dmap/PTVseg projections were stacked as a 3-channel variable. Four Visual Geometry Group (VGG-19) deep encoders were used in an ensemble design, with each submodel using a different spherical projection formula as input for BM outcome prediction. In each submodel, clinical features after positional encoding were fused with VGG-19 deep features to generate logit results. The ensemble's outcome was synthesized from the 4 submodel results via logistic regression. In total, 10 model versions with random validation sample assignments were trained to study model robustness. Performance was compared with (1) a single VGG-19 encoder, (2) an ensemble with a T1-CE MRI as the sole image input after projections, and (3) an ensemble with the same image input design without clinical feature inclusion.

RESULTS

The ensemble model achieved an excellent area under the receiver operating characteristic curve (AUCROC: 0.89 ± 0.02) with high sensitivity (0.82 ± 0.05), specificity (0.84 ± 0.11), and accuracy (0.84 ± 0.08) results. This outperformed the MRI-only VGG-19 encoder (sensitivity: 0.35 ± 0.01, AUCROC: 0.64 ± 0.08), the MRI-only deep ensemble (sensitivity: 0.60 ± 0.09, AUCROC: 0.68 ± 0.06), and the 3-channel ensemble without clinical feature fusion (sensitivity: 0.78 ± 0.08, AUCROC: 0.84 ± 0.03).

CONCLUSIONS

Facilitated by the spherical image projection method, a deep ensemble model incorporating Dmap and clinical variables demonstrated excellent performance in predicting BM post-SRS local failure. Our novel approach could improve other radiation therapy outcome models and warrants further evaluation.

Review of Machine Learning Techniques in Soft Tissue Biomechanics and Biomaterials

BACKGROUND AND OBJECTIVE

Advanced material models and material characterization of soft biological tissues play an essential role in pre-surgical planning for vascular surgeries and transcatheter interventions. Recent advances in heart valve engineering, medical device and patch design are built upon these models. Furthermore, understanding vascular growth and remodeling in native and tissue-engineered vascular biomaterials, as well as designing and testing drugs on soft tissue, are crucial aspects of predictive regenerative medicine. Traditional nonlinear optimization methods and finite element (FE) simulations have served as biomaterial characterization tools combined with soft tissue mechanics and tensile testing for decades. However, results obtained through nonlinear optimization methods are reliable only to a certain extent due to mathematical limitations, and FE simulations may require substantial computing time and resources, which might not be justified for patient-specific simulations. To a significant extent, machine learning (ML) techniques have gained increasing prominence in the field of soft tissue mechanics in recent years, offering notable advantages over conventional methods. This review article presents an in-depth examination of emerging ML algorithms utilized for estimating the mechanical characteristics of soft biological tissues and biomaterials. These algorithms are employed to analyze crucial properties such as stress-strain curves and pressure-volume loops. The focus of the review is on applications in cardiovascular engineering, and the fundamental mathematical basis of each approach is also discussed.

METHODS

The review effort employed two strategies. First, the recent studies of major research groups actively engaged in cardiovascular soft tissue mechanics are compiled, and research papers utilizing ML and deep learning (DL) techniques were included in our review. The second strategy involved a standard keyword search across major databases. This approach provided 11 relevant ML articles, meticulously selected from reputable sources including ScienceDirect, Springer, PubMed, and Google Scholar. The selection process involved using specific keywords such as "machine learning" or "deep learning" in conjunction with "soft biological tissues", "cardiovascular", "patient-specific," "strain energy", "vascular" or "biomaterials". Initially, a total of 25 articles were selected. However, 14 of these articles were excluded as they did not align with the criteria of focusing on biomaterials specifically employed for soft tissue repair and regeneration. As a result, the remaining 11 articles were categorized based on the ML techniques employed and the training data utilized.

RESULTS

ML techniques utilized for assessing the mechanical characteristics of soft biological tissues and biomaterials are broadly classified into two categories: standard ML algorithms and physics-informed ML algorithms. The standard ML models are then organized based on their tasks, being grouped into Regression and Classification subcategories. Within these categories, studies employ various supervised learning models, including support vector machines (SVMs), bagged decision trees (BDTs), artificial neural networks (ANNs) or deep neural networks (DNNs), and convolutional neural networks (CNNs). Additionally, the utilization of unsupervised learning approaches, such as autoencoders incorporating principal component analysis (PCA) and/or low-rank approximation (LRA), is based on the specific characteristics of the training data. The training data predominantly consists of three types: experimental mechanical data, including uniaxial or biaxial stress-strain data; synthetic mechanical data generated through non-linear fitting and/or FE simulations; and image data such as 3D second harmonic generation (SHG) images or computed tomography (CT) images. The evaluation of performance for physics-informed ML models primarily relies on the coefficient of determinationR 2 . In contrast, various metrics and error measures are utilized to assess the performance of standard ML models. Furthermore, our review includes an extensive examination of prevalent biomaterial models that can serve as physical laws for physics-informed ML models.

CONCLUSION

ML models offer an accurate, fast, and reliable approach for evaluating the mechanical characteristics of diseased soft tissue segments and selecting optimal biomaterials for time-critical soft tissue surgeries. Among the various ML models examined in this review, physics-informed neural network models exhibit the capability to forecast the mechanical response of soft biological tissues accurately, even with limited training samples. These models achieve highR 2 values ranging from 0.90 to 1.00. This is particularly significant considering the challenges associated with obtaining a large number of living tissue samples for experimental purposes, which can be time-consuming and impractical. Additionally, the review not only discusses the advantages identified in the current literature but also sheds light on the limitations and offers insights into future perspectives.

A multi-slice attention fusion and multi-view personalized fusion lightweight network for Alzheimer's disease diagnosis

OBJECTIVE

Alzheimer's disease (AD) is a type of neurological illness that significantly impacts individuals' daily lives. In the intelligent diagnosis of AD, 3D networks require larger computational resources and storage space for training the models, leading to increased model complexity and training time. On the other hand, 2D slices analysis may overlook the 3D structural information of MRI and can result in information loss.

APPROACH

We propose a multi-slice attention fusion and multi-view personalized fusion lightweight network for automated AD diagnosis. It incorporates a multi-branch lightweight backbone to extract features from sagittal, axial, and coronal view of MRI, respectively. In addition, we introduce a novel multi-slice attention fusion module, which utilizes a combination of global and local channel attention mechanism to ensure consistent classification across multiple slices. Additionally, a multi-view personalized fusion module is tailored to assign appropriate weights to the three views, taking into account the varying significance of each view in achieving accurate classification results. To enhance the performance of the multi-view personalized fusion module, we utilize a label consistency loss to guide the model's learning process. This encourages the acquisition of more consistent and stable representations across all three views.

MAIN RESULTS

The suggested strategy is efficient in lowering the number of parameters and FLOPs, with only 3.75M and 4.45G respectively, and accuracy improved by 10.5% to 14% in three tasks. Moreover, in the classification tasks of AD vs. CN, AD vs. MCI and MCI vs. CN, the accuracy of the proposed method is 95.63%, 86.88% and 85.00%, respectively, which is superior to the existing methods.

CONCLUSIONS

The results show that the proposed approach not only excels in resource utilization, but also significantly outperforms the four comparison methods in terms of accuracy and sensitivity, particularly in detecting early-stage AD lesions. It can precisely capture and accurately identify subtle brain lesions, providing crucial technical support for early intervention and treatment.

Deep learning based detection and classification of fetal lip in ultrasound images

OBJECTIVES

Fetal cleft lip is a common congenital defect. Considering the delicacy and difficulty of observing fetal lips, we have utilized deep learning technology to develop a new model aimed at quickly and accurately assessing the development of fetal lips during prenatal examinations. This model can detect ultrasound images of the fetal lips and classify them, aiming to provide a more objective prediction for the development of fetal lips.

METHODS

This study included 632 pregnant women in their mid-pregnancy stage, who underwent ultrasound examinations of the fetal lips, collecting both normal and abnormal fetal lip ultrasound images. To improve the accuracy of the detection and classification of fetal lips, we proposed and validated the Yolov5-ECA model.

RESULTS

The experimental results show that, compared with the currently popular 10 models, our model achieved the best results in the detection and classification of fetal lips. In terms of the detection of fetal lips, the mean average precision (mAP) at 0.5 and mAP at 0.5:0.95 were 0.920 and 0.630, respectively. In the classification of fetal lip ultrasound images, the accuracy reached 0.925.

CONCLUSIONS

The deep learning algorithm has accuracy consistent with manual evaluation in the detection and classification process of fetal lips. This automated recognition technology can provide a powerful tool for inexperienced young doctors, helping them to accurately conduct examinations and diagnoses of fetal lips.

Enhancing brain tumor detection: a novel CNN approach with advanced activation functions for accurate medical imaging analysis

Introduction

Brain tumors are characterized by abnormal cell growth within or around the brain, posing severe health risks often associated with high mortality rates. Various imaging techniques, including magnetic resonance imaging (MRI), are commonly employed to visualize the brain and identify malignant growths. Computer-aided diagnosis tools (CAD) utilizing Convolutional Neural Networks (CNNs) have proven effective in feature extraction and predictive analysis across diverse medical imaging modalities.

Methods

This study explores a CNN trained and evaluated with nine activation functions, encompassing eight established ones from the literature and a modified version of the soft sign activation function.

Results

The latter demonstrates notable efficacy in discriminating between four types of brain tumors in MR images, achieving an accuracy of 97.6%. The sensitivity for glioma is 93.7%; for meningioma, it is 97.4%; for cases with no tumor, it is 98.8%; and for pituitary tumors, it reaches 100%.

Discussion

In this manuscript, we propose an advanced CNN architecture that integrates a newly developed activation function. Our extensive experimentation and analysis showcase the model's remarkable ability to precisely distinguish between different types of brain tumors within a substantial and diverse dataset. The findings from our study suggest that this model could serve as an invaluable supplementary tool for healthcare practitioners, including specialized medical professionals and resident physicians, in the accurate diagnosis of brain tumors.

Advancing precise diagnosis of nasopharyngeal carcinoma through endoscopy-based radiomics analysis

Nasopharyngeal carcinoma (NPC) has high metastatic potential and is hard to detect early. This study aims to develop a deep learning model for NPC diagnosis using optical imagery. From April 2008 to May 2021, we analyzed 12,087 nasopharyngeal endoscopic images and 309 videos from 1,108 patients. The pretrained model was fine-tuned with stochastic gradient descent on the final layers. Data augmentation was applied during training. Videos were converted to images for malignancy scoring. Performance metrics like AUC, accuracy, and sensitivity were calculated based on the malignancy score. The deep learning model demonstrated high performance in identifying NPC, with AUC values of 0.981 (95% confidence of interval [CI] 0.965-0.996) for the Fujian Cancer Hospital dataset and 0.937 (0.905-0.970) for the Jiangxi Cancer Hospital dataset. The proposed model effectively diagnoses NPC with high accuracy, sensitivity, and specificity across multiple datasets. It shows promise for early NPC detection, especially in identifying latent lesions.

Systematic bibliometric and visualized analysis of research hotspots and trends on the application of artificial intelligence in glaucoma from 2013 to 2022

AIM

To conduct a bibliometric analysis of research on artificial intelligence (AI) in the field of glaucoma to gain a comprehensive understanding of the current state of research and identify potential new directions for future studies.

METHODS

Relevant articles on the application of AI in the field of glaucoma from the Web of Science Core Collection were retrieved, covering the period from January 1, 2013, to December 31, 2022. In order to assess the contributions and co-occurrence relationships among different countries/regions, institutions, authors, and journals, CiteSpace and VOSviewer software were employed and the research hotspots and future trends within the field were identified.

RESULTS

A total of 750 English articles published between 2013 and 2022 were collected, and the number of publications exhibited an overall increasing trend. The majority of the articles were from China, followed by the United States and India. National University of Singapore, Chinese Academy of Sciences, and Sun Yat-sen University made significant contributions to the published works. Weinreb RN and Fu HZ ranked first among authors and cited authors. American Journal of Ophthalmology is the most impactful academic journal in the field of AI application in glaucoma. The disciplinary scope of this field includes ophthalmology, computer science, mathematics, molecular biology, genetics, and other related disciplines. The clustering and identification of keyword nodes in the co-occurrence network reveal the evolving landscape of AI application in the field of glaucoma. Initially, the hot topics in this field were primarily "segmentation", "classification" and "diagnosis". However, in recent years, the focus has shifted to "deep learning", "convolutional neural network" and "artificial intelligence".

CONCLUSION

With the rapid development of AI technology, scholars have shown increasing interest in its application in the field of glaucoma. Moreover, the application of AI in assisting treatment and predicting prognosis in glaucoma may become a future research hotspot. However, the reliability and interpretability of AI data remain pressing issues that require resolution.

Interactive computer-aided diagnosis on medical image using large language models

Computer-aided diagnosis (CAD) has advanced medical image analysis, while large language models (LLMs) have shown potential in clinical applications. However, LLMs struggle to interpret medical images, which are critical for decision-making. Here we show a strategy integrating LLMs with CAD networks. The framework uses LLMs' medical knowledge and reasoning to enhance CAD network outputs, such as diagnosis, lesion segmentation, and report generation, by summarizing information in natural language. The generated reports are of higher quality and can improve the performance of vision-based CAD models. In chest X-rays, an LLM using ChatGPT improved diagnosis performance by 16.42 percentage points compared to state-of-the-art models, while GPT-3 provided a 15.00 percentage point F1-score improvement. Our strategy allows accurate report generation and creates a patient-friendly interactive system, unlike conventional CAD systems only understood by professionals. This approach has the potential to revolutionize clinical decision-making and patient communication.

MyThisYourThat for interpretable identification of systematic bias in federated learning for biomedical images

Distributed collaborative learning is a promising approach for building predictive models for privacy-sensitive biomedical images. Here, several data owners (clients) train a joint model without sharing their original data. However, concealed systematic biases can compromise model performance and fairness. This study presents MyThisYourThat (MyTH) approach, which adapts an interpretable prototypical part learning network to a distributed setting, enabling each client to visualize feature differences learned by others on their own image: comparing one client's 'This' with others' 'That'. Our setting demonstrates four clients collaboratively training two diagnostic classifiers on a benchmark X-ray dataset. Without data bias, the global model reaches 74.14% balanced accuracy for cardiomegaly and 74.08% for pleural effusion. We show that with systematic visual bias in one client, the performance of global models drops to near-random. We demonstrate how differences between local and global prototypes reveal biases and allow their visualization on each client's data without compromising privacy.

ChoroidSeg-ViT: A Transformer Model for Choroid Layer Segmentation Based on a Mixed Attention Feature Enhancement Mechanism

Purpose

To develop a Vision Transformer (ViT) model based on the mixed attention feature enhancement mechanism, ChoroidSeg-ViT, for choroid layer segmentation in optical coherence tomography (OCT) images.

Methods

This study included a dataset of 100 OCT B-scans images. Ground truths were carefully labeled by experienced ophthalmologists. An end-to-end local-enhanced Transformer model, ChoroidSeg-ViT, was designed to segment the choroid layer by integrating the local enhanced feature extraction and semantic feature fusion paths. Standard segmentation metrics were selected to evaluate ChoroidSeg-ViT.

Results

Experimental results demonstrate that ChoroidSeg-ViT exhibited superior segmentation performance (mDice: 98.31, mIoU: 96.62, mAcc: 98.29) compared to other deep learning approaches, thus indicating the effectiveness and superiority of this proposed model for the choroid layer segmentation task. Furthermore, ablation and generalization experiments validated the reasonableness of the module design.

Conclusions

We developed a novel Transformer model to precisely and automatically segment the choroid layer and achieved the state-of-the-art performance.

Translational Relevance

ChoroidSeg-ViT could segment precise and smooth choroid layers and form the basis of an automatic choroid analysis system that would facilitate future choroidal research in ophthalmology.

Learning co-plane attention across MRI sequences for diagnosing twelve types of knee abnormalities

Multi-sequence magnetic resonance imaging is crucial in accurately identifying knee abnormalities but requires substantial expertise from radiologists to interpret. Here, we introduce a deep learning model incorporating co-plane attention across image sequences to classify knee abnormalities. To assess the effectiveness of our model, we collected the largest multi-sequence knee magnetic resonance imaging dataset involving the most comprehensive range of abnormalities, comprising 1748 subjects and 12 types of abnormalities. Our model achieved an overall area under the receiver operating characteristic curve score of 0.812. It achieved an average accuracy of 0.78, outperforming junior radiologists (accuracy 0.65) and remains competitive with senior radiologists (accuracy 0.80). Notably, with the assistance of model output, the diagnosis accuracy of all radiologists was improved significantly (p < 0.001), elevating from 0.73 to 0.79 on average. The interpretability analysis demonstrated that the model decision-making process is consistent with the clinical knowledge, enhancing its credibility and reliability in clinical practice.

Doctor simulator: Delta-Age-Sex-AdaIn enhancing bone age assessment through AdaIn style transfer

BACKGROUND

Bone age assessment assists physicians in evaluating the growth and development of children. However, deep learning methods for bone age estimation do not currently incorporate differential features obtained through comparisons with other bone atlases.

OBJECTIVE

To propose a more accurate method, Delta-Age-Sex-AdaIn (DASA-net), for bone age assessment, this paper combines age and sex distribution through adaptive instance normalization (AdaIN) and style transfer, simulating the process of visually comparing hand images with a standard bone atlas to determine bone age.

MATERIALS AND METHODS

The proposed Delta-Age-Sex-AdaIn (DASA-net) consists of four modules: BoneEncoder, Binary code distribution, Delta-Age-Sex-AdaIn, and AgeDecoder. It is compared with state-of-the-art methods on both a public Radiological Society of North America (RSNA) pediatric bone age prediction dataset (14,236 hand radiographs, ranging from 1 to 228 months) and a private bone age prediction dataset from Zigong Fourth People's Hospital (474 hand radiographs, ranging from 12 to 218 months, 268 male). Ablation experiments were designed to demonstrate the necessity of incorporating age distribution and sex distribution.

RESULTS

The DASA-net model achieved a lower mean absolute deviation (MAD) of 3.52 months on the RSNA dataset, outperforming other methods such as BoneXpert, Deeplasia, BoNet, and other deep learning based methods. On the private dataset, the DASA-net model obtained a MAD of 3.82 months, which is also superior to other methods.

CONCLUSION

The proposed DASA-net model aided the model's learning of the distinctive characteristics of hand bones of various ages and both sexes by integrating age and sex distribution into style transfer.

Immune Cell-Based Microrobots for Remote Magnetic Actuation, Antitumor Activity, and Medical Imaging

Translating medical microrobots into clinics requires tracking, localization, and performing assigned medical tasks at target locations, which can only happen when appropriate design, actuation mechanisms, and medical imaging systems are integrated into a single microrobot. Despite this, these parameters are not fully considered when designing macrophage-based microrobots. This study presents living macrophage-based microrobots that combine macrophages with magnetic Janus particles coated with FePt nanofilm for magnetic steering and medical imaging and bacterial lipopolysaccharides for stimulating macrophages in a tumor-killing state. The macrophage-based microrobots combine wireless magnetic actuation, tracking with medical imaging techniques, and antitumor abilities. These microrobots are imaged under magnetic resonance imaging and optoacoustic imaging in soft-tissue-mimicking phantoms and ex vivo conditions. Magnetic actuation and real-time imaging of microrobots are demonstrated under static and physiologically relevant flow conditions using optoacoustic imaging. Further, macrophage-based microrobots are magnetically steered toward urinary bladder tumor spheroids and imaged with a handheld optoacoustic device, where the microrobots significantly reduce the viability of tumor spheroids. The proposed approach demonstrates the proof-of-concept feasibility of integrating macrophage-based microrobots into clinic imaging modalities for cancer targeting and intervention, and can also be implemented for various other medical applications.

Generating Synthetic Data for Medical Imaging

Artificial intelligence (AI) models for medical imaging tasks, such as classification or segmentation, require large and diverse datasets of images. However, due to privacy and ethical issues, as well as data sharing infrastructure barriers, these datasets are scarce and difficult to assemble. Synthetic medical imaging data generated by AI from existing data could address this challenge by augmenting and anonymizing real imaging data. In addition, synthetic data enable new applications, including modality translation, contrast synthesis, and professional training for radiologists. However, the use of synthetic data also poses technical and ethical challenges. These challenges include ensuring the realism and diversity of the synthesized images while keeping data unidentifiable, evaluating the performance and generalizability of models trained on synthetic data, and high computational costs. Since existing regulations are not sufficient to guarantee the safe and ethical use of synthetic images, it becomes evident that updated laws and more rigorous oversight are needed. Regulatory bodies, physicians, and AI developers should collaborate to develop, maintain, and continually refine best practices for synthetic data. This review aims to provide an overview of the current knowledge of synthetic data in medical imaging and highlights current key challenges in the field to guide future research and development.

Artificial Intelligence in Metabolomics: A Current Review

Metabolomics and artificial intelligence (AI) form a synergistic partnership. Metabolomics generates large datasets comprising hundreds to thousands of metabolites with complex relationships. AI, aiming to mimic human intelligence through computational modeling, possesses extraordinary capabilities for big data analysis. In this review, we provide a recent overview of the methodologies and applications of AI in metabolomics studies in the context of systems biology and human health. We first introduce the AI concept, history, and key algorithms for machine learning and deep learning, summarizing their strengths and weaknesses. We then discuss studies that have successfully used AI across different aspects of metabolomic analysis, including analytical detection, data preprocessing, biomarker discovery, predictive modeling, and multi-omics data integration. Lastly, we discuss the existing challenges and future perspectives in this rapidly evolving field. Despite limitations and challenges, the combination of metabolomics and AI holds great promises for revolutionary advancements in enhancing human health.

Preliminary experiments on interpretable ChatGPT-assisted diagnosis for breast ultrasound radiologists

Background

Ultrasound is essential for detecting breast lesions. The American College of Radiology's Breast Imaging Reporting and Data System (BI-RADS) classification system is widely used, but its subjectivity can lead to inconsistency in diagnostic outcomes. Artificial intelligence (AI) models, such as ChatGPT-3.5, may potentially enhance diagnostic accuracy and efficiency in medical settings. This study aimed to assess the utility of the ChatGPT-3.5 model in generating BI-RADS classifications for breast ultrasound reports and its ability to replicate the "chain of thought" (CoT) in clinical decision-making to improve model interpretability.

Methods

Breast ultrasound reports were collected, and ChatGPT-3.5 was used to generate diagnoses and treatment plans. We evaluated GPT-4's performance by comparing its generated reports to those from doctors with various levels of experience. We also conducted a Turing test and a consistency analysis. To enhance the interpretability of the model, we applied the CoT method to deconstruct the decision-making chain of the GPT model.

Results

A total of 131 patients were evaluated, with 57 doctors participating in the experiment. ChatGPT-3.5 showed promising performance in structure and organization (S&O), professional terminology and expression (PTE), treatment recommendations (TR), and clarity and comprehensibility (C&C). However, improvements are needed in BI-RADS classification, malignancy diagnosis (MD), likelihood of being written by a physician (LWBP), and ultrasound doctor artificial intelligence acceptance (UDAIA). Turing test results indicated that AI-generated reports convincingly resembled human-authored reports. Reproducibility experiments displayed consistent performance. Erroneous report analysis revealed issues related to incorrect diagnosis, inconsistencies, and overdiagnosis. The CoT investigation supports the potential of ChatGPT to replicate the clinical decision-making process and offers insights into AI interpretability.

Conclusions

The ChatGPT-3.5 model holds potential as a valuable tool for assisting in the efficient determination of BI-RADS classifications and enhancing diagnostic performance.

SymTC: A symbiotic Transformer-CNN net for instance segmentation of lumbar spine MRI

Intervertebral disc disease, a prevalent ailment, frequently leads to intermittent or persistent low back pain, and diagnosing and assessing of this disease rely on accurate measurement of vertebral bone and intervertebral disc geometries from lumbar MR images. Deep neural network (DNN) models may assist clinicians with more efficient image segmentation of individual instances (discs and vertebrae) of the lumbar spine in an automated way, which is termed as instance image segmentation. In this work, we proposed SymTC, an innovative lumbar spine MR image segmentation model that combines the strengths of Transformer and Convolutional Neural Network (CNN). Specifically, we designed a parallel dual-path architecture to merge CNN layers and Transformer layers, and we integrated a novel position embedding into the self-attention module of Transformer, enhancing the utilization of positional information for more accurate segmentation. To further improve model performance, we introduced a new data synthesis technique to create synthetic yet realistic MR image dataset, named SSMSpine, which is made publicly available. We evaluated our SymTC and the other 16 representative image segmentation models on our private in-house dataset and public SSMSpine dataset, using two metrics, Dice Similarity Coefficient and the 95th percentile Hausdorff Distance. The results indicate that SymTC surpasses the other 16 methods, achieving the highest dice score of 96.169 % for segmenting vertebral bones and intervertebral discs on the SSMSpine dataset. The SymTC code and SSMSpine dataset are publicly available at https://github.com/jiasongchen/SymTC.

An emerging network for COVID-19 CT-scan classification using an ensemble deep transfer learning model

Over the past few years, the widespread outbreak of COVID-19 has caused the death of millions of people worldwide. Early diagnosis of the virus is essential to control its spread and provide timely treatment. Artificial intelligence methods are often used as powerful tools to reach a COVID-19 diagnosis via computed tomography (CT) samples. In this paper, artificial intelligence-based methods are introduced to diagnose COVID-19. At first, a network called CT6-CNN is designed, and then two ensemble deep transfer learning models are developed based on Xception, ResNet-101, DenseNet-169, and CT6-CNN to reach a COVID-19 diagnosis by CT samples. The publicly available SARS-CoV-2 CT dataset is utilized for our implementation, including 2481 CT scans. The dataset is separated into 2108, 248, and 125 images for training, validation, and testing, respectively. Based on experimental results, the CT6-CNN model achieved 94.66% accuracy, 94.67% precision, 94.67% sensitivity, and 94.65% F1-score rate. Moreover, the ensemble learning models reached 99.2% accuracy. Experimental results affirm the effectiveness of designed models, especially the ensemble deep learning models, to reach a diagnosis of COVID-19.

Deep learning-based computer-aided diagnosis system for the automatic detection and classification of lateral cervical lymph nodes on original ultrasound images of papillary thyroid carcinoma: a prospective diagnostic study

PURPOSE

This study aims to develop a deep learning-based computer-aided diagnosis (CAD) system for the automatic detection and classification of lateral cervical lymph nodes (LNs) on original ultrasound images of papillary thyroid carcinoma (PTC) patients.

METHODS

A retrospective data set of 1801 cervical LN ultrasound images from 1675 patients with PTC and a prospective test set including 185 images from 160 patients were collected. Four different deep leaning models were trained and validated in the retrospective data set. The best model was selected for CAD system development and compared with three sonographers in the retrospective and prospective test sets.

RESULTS

The Deformable Detection Transformer (DETR) model showed the highest diagnostic efficacy, with a mean average precision score of 86.3% in the retrospective test set, and was therefore used in constructing the CAD system. The detection performance of the CAD system was superior to the junior sonographer and intermediate sonographer with accuracies of 86.3% and 92.4% in the retrospective and prospective test sets, respectively. The classification performance of the CAD system was better than all sonographers with the areas under the curve (AUCs) of 94.4% and 95.2% in the retrospective and prospective test sets, respectively.

CONCLUSIONS

This study developed a Deformable DETR model-based CAD system for automatically detecting and classifying lateral cervical LNs on original ultrasound images, which showed excellent diagnostic efficacy and clinical utility. It can be an important tool for assisting sonographers in the diagnosis process.

TG-Net: Using text prompts for improved skin lesion segmentation

Automatic skin segmentation is an efficient method for the early diagnosis of skin cancer, which can minimize the missed detection rate and treat early skin cancer in time. However, significant variations in texture, size, shape, the position of lesions, and obscure boundaries in dermoscopy images make it extremely challenging to accurately locate and segment lesions. To address these challenges, we propose a novel framework named TG-Net, which exploits textual diagnostic information to guide the segmentation of dermoscopic images. Specifically, TG-Net adopts a dual-stream encoder-decoder architecture. The dual-stream encoder comprises Res2Net for extracting image features and our proposed text attention (TA) block for extracting textual features. Through hierarchical guidance, textual features are embedded into the process of image feature extraction. Additionally, we devise a multi-level fusion (MLF) module to merge higher-level features and generate a global feature map as guidance for subsequent steps. In the decoding stage of the network, local features and the global feature map are utilized in three multi-scale reverse attention modules (MSRA) to produce the final segmentation results. We conduct extensive experiments on three publicly accessible datasets, namely ISIC 2017, HAM10000, and PH2. Experimental results demonstrate that TG-Net outperforms state-of-the-art methods, validating the reliability of our method. Source code is available at https://github.com/ukeLin/TG-Net.

Anatomy-specific Progression Classification in Chest Radiographs via Weakly Supervised Learning

Purpose To develop a machine learning approach for classifying disease progression in chest radiographs using weak labels automatically derived from radiology reports. Materials and Methods In this retrospective study, a twin neural network was developed to classify anatomy-specific disease progression into four categories: improved, unchanged, worsened, and new. A two-step weakly supervised learning approach was employed, pretraining the model on 243 008 frontal chest radiographs from 63 877 patients (mean age, 51.7 years ± 17.0 [SD]; 34 813 [55%] female) included in the MIMIC-CXR database and fine-tuning it on the subset with progression labels derived from consecutive studies. Model performance was evaluated for six pathologic observations on test datasets of unseen patients from the MIMIC-CXR database. Area under the receiver operating characteristic (AUC) analysis was used to evaluate classification performance. The algorithm is also capable of generating bounding-box predictions to localize areas of new progression. Recall, precision, and mean average precision were used to evaluate the new progression localization. One-tailed paired t tests were used to assess statistical significance. Results The model outperformed most baselines in progression classification, achieving macro AUC scores of 0.72 ± 0.004 for atelectasis, 0.75 ± 0.007 for consolidation, 0.76 ± 0.017 for edema, 0.81 ± 0.006 for effusion, 0.7 ± 0.032 for pneumonia, and 0.69 ± 0.01 for pneumothorax. For new observation localization, the model achieved mean average precision scores of 0.25 ± 0.03 for atelectasis, 0.34 ± 0.03 for consolidation, 0.33 ± 0.03 for edema, and 0.31 ± 0.03 for pneumothorax. Conclusion Disease progression classification models were developed on a large chest radiograph dataset, which can be used to monitor interval changes and detect new pathologic conditions on chest radiographs. Keywords: Prognosis, Unsupervised Learning, Transfer Learning, Convolutional Neural Network (CNN), Emergency Radiology, Named Entity Recognition Supplemental material is available for this article. © RSNA, 2024 See also commentary by Alves and Venkadesh in this issue.

A systematic review of deep learning-based spinal bone lesion detection in medical images

Spinal bone lesions encompass a wide array of pathologies, spanning from benign abnormalities to aggressive malignancies, such as diffusely localized metastases. Early detection and accurate differentiation of the underlying diseases is crucial for every patient's clinical treatment and outcome, with radiological imaging being a core element in the diagnostic pathway. Across numerous pathologies and imaging techniques, deep learning (DL) models are progressively considered a valuable resource in the clinical setting. This review describes not only the diagnostic performance of these models and the differing approaches in the field of spinal bone malignancy recognition, but also the lack of standardized methodology and reporting that we believe is currently hampering this newly founded area of research. In line with their established and reliable role in lesion detection, this publication focuses on both computed tomography and magnetic resonance imaging, as well as various derivative modalities (i.e. SPECT). After conducting a systematic literature search and subsequent analysis for applicability and quality using a modified QUADAS-2 scoring system, we confirmed that most of the 14 identified studies were plagued by major limitations, such as insufficient reporting of model statistics and data acquisition, a lacking external validation dataset, and potentially biased annotation. Although we experienced these limitations, we nonetheless conclude that the potential of these methods shines through in the presented results. These findings underline the need for more stringent quality controls in DL studies, as well as model development to afford increased insight and progress in this promising novel field.

Sparse and Expandable Network for Google's Pathways

Introduction

Recently, Google introduced Pathways as its next-generation AI architecture. Pathways must address three critical challenges: learning one general model for several continuous tasks, ensuring tasks can leverage each other without forgetting old tasks, and learning from multi-modal data such as images and audio. Additionally, Pathways must maintain sparsity in both learning and deployment. Current lifelong multi-task learning approaches are inadequate in addressing these challenges.

Methods

To address these challenges, we propose SEN, a Sparse and Expandable Network. SEN is designed to handle multiple tasks concurrently by maintaining sparsity and enabling expansion when new tasks are introduced. The network leverages multi-modal data, integrating information from different sources while preventing interference between tasks.

Results

The proposed SEN model demonstrates significant improvements in multi-task learning, successfully managing task interference and forgetting. It effectively integrates data from various modalities and maintains efficiency through sparsity during both the learning and deployment phases.

Discussion

SEN offers a straightforward yet effective solution to the limitations of current lifelong multi-task learning methods. By addressing the challenges identified in the Pathways architecture, SEN provides a promising approach for developing AI systems capable of learning and adapting over time without sacrificing performance or efficiency.

The diagnostic value of artificial intelligence-assisted imaging for developmental dysplasia of the hip: a systematic review and meta-analysis

OBJECTIVE

To clarify the efficacy of artificial intelligence (AI)-assisted imaging in the diagnosis of developmental dysplasia of the hip (DDH) through a meta-analysis.

METHODS

Relevant literature on AI for early DDH diagnosis was searched in PubMed, Web of Science, Embase, and The Cochrane Library databases until April 4, 2024. The Quality Assessment of Diagnostic Accuracy Studies tool was used to assess the quality of included studies. Revman5.4 and StataSE-64 software were used to calculate the combined sensitivity, specificity, AUC value, and DOC value of AI-assisted imaging for DDH diagnosis.

RESULTS

The meta-analysis included 13 studies (6 prospective and 7 retrospective) with 28 AI models and a total of 10,673 samples. The summary sensitivity, specificity, AUC value, and DOC value were 99.0% (95% CI: 97.0-100.0%), 94.0% (95% CI: 89.0-96.0%), 99.0% (95% CI: 98.0-100.0%), and 1342 (95% CI: 469-3842), respectively.

CONCLUSION

AI-assisted imaging demonstrates high diagnostic efficacy for DDH detection, improving the accuracy of early DDH imaging examination. More prospective studies are needed to further confirm the value of AI-assisted imaging for early DDH diagnosis.

AI-Driven Diagnostic Assistance in Medical Inquiry: Reinforcement Learning Algorithm Development and Validation

BACKGROUND

For medical diagnosis, clinicians typically begin with a patient's chief concerns, followed by questions about symptoms and medical history, physical examinations, and requests for necessary auxiliary examinations to gather comprehensive medical information. This complex medical investigation process has yet to be modeled by existing artificial intelligence (AI) methodologies.

OBJECTIVE

The aim of this study was to develop an AI-driven medical inquiry assistant for clinical diagnosis that provides inquiry recommendations by simulating clinicians' medical investigating logic via reinforcement learning.

METHODS

We compiled multicenter, deidentified outpatient electronic health records from 76 hospitals in Shenzhen, China, spanning the period from July to November 2021. These records consisted of both unstructured textual information and structured laboratory test results. We first performed feature extraction and standardization using natural language processing techniques and then used a reinforcement learning actor-critic framework to explore the rational and effective inquiry logic. To align the inquiry process with actual clinical practice, we segmented the inquiry into 4 stages: inquiring about symptoms and medical history, conducting physical examinations, requesting auxiliary examinations, and terminating the inquiry with a diagnosis. External validation was conducted to validate the inquiry logic of the AI model.

RESULTS

This study focused on 2 retrospective inquiry-and-diagnosis tasks in the emergency and pediatrics departments. The emergency departments provided records of 339,020 consultations including mainly children (median age 5.2, IQR 2.6-26.1 years) with various types of upper respiratory tract infections (250,638/339,020, 73.93%). The pediatrics department provided records of 561,659 consultations, mainly of children (median age 3.8, IQR 2.0-5.7 years) with various types of upper respiratory tract infections (498,408/561,659, 88.73%). When conducting its own inquiries in both scenarios, the AI model demonstrated high diagnostic performance, with areas under the receiver operating characteristic curve of 0.955 (95% CI 0.953-0.956) and 0.943 (95% CI 0.941-0.944), respectively. When the AI model was used in a simulated collaboration with physicians, it notably reduced the average number of physicians' inquiries to 46% (6.037/13.26; 95% CI 6.009-6.064) and 43% (6.245/14.364; 95% CI 6.225-6.269) while achieving areas under the receiver operating characteristic curve of 0.972 (95% CI 0.970-0.973) and 0.968 (95% CI 0.967-0.969) in the scenarios. External validation revealed a normalized Kendall τ distance of 0.323 (95% CI 0.301-0.346), indicating the inquiry consistency of the AI model with physicians.

CONCLUSIONS

This retrospective analysis of predominantly respiratory pediatric presentations in emergency and pediatrics departments demonstrated that an AI-driven diagnostic assistant had high diagnostic performance both in stand-alone use and in simulated collaboration with clinicians. Its investigation process was found to be consistent with the clinicians' medical investigation logic. These findings highlight the diagnostic assistant's promise in assisting the decision-making processes of health care professionals.