3D Deep Learning on Medical Images: A Review

A survey on advancements in image-text multimodal models: From general techniques to biomedical implementations

With the significant advancements of Large Language Models (LLMs) in the field of Natural Language Processing (NLP), the development of image-text multimodal models has garnered widespread attention. Current surveys on image-text multimodal models mainly focus on representative models or application domains, but lack a review on how general technical models influence the development of domain-specific models, which is crucial for domain researchers. Based on this, this paper first reviews the technological evolution of image-text multimodal models, from early explorations of feature space to visual language encoding structures, and then to the latest large model architectures. Next, from the perspective of technological evolution, we explain how the development of general image-text multimodal technologies promotes the progress of multimodal technologies in the biomedical field, as well as the importance and complexity of specific datasets in the biomedical domain. Then, centered on the tasks of image-text multimodal models, we analyze their common components and challenges. After that, we summarize the architecture, components, and data of general image-text multimodal models, and introduce the applications and improvements of image-text multimodal models in the biomedical field. Finally, we categorize the challenges faced in the development and application of general models into external factors and intrinsic factors, further refining them into 2 external factors and 5 intrinsic factors, and propose targeted solutions, providing guidance for future research directions. For more details and data, please visit our GitHub page: https://github.com/i2vec/A-survey-on-image-text-multimodal-models.

Deep magnetic resonance fingerprinting based on Local and Global Vision Transformer

To mitigate systematic errors in magnetic resonance fingerprinting (MRF), the precomputed dictionary is usually computed with minimal granularity across the entire range of tissue parameters. However, the dictionary grows exponentially with the number of parameters increase, posing significant challenges to the computational efficiency and matching accuracy of pattern-matching algorithms. Existing works, primarily based on convolutional neural networks (CNN), focus solely on local information to reconstruct multiple parameter maps, lacking in-depth investigations on the MRF mechanism. These methods may not exploit long-distance redundancies and the contextual information within voxel fingerprints introduced by the Bloch equation dynamics, leading to limited reconstruction speed and accuracy. To overcome these limitations, we propose a novel end-to-end neural network called the Local and Global Vision Transformer (LG-ViT) for MRF parameter reconstruction. Our proposed LG-ViT employs a multi-stage architecture that effectively reduces the computational overhead associated with the high-dimensional MRF data and the transformer model. Specifically, a local Transformer encoder is proposed to capture contextual information embedded within voxel fingerprints and local correlations introduced by the interconnected human tissues. Additionally, a global Transformer encoder is proposed to leverage long-distance dependencies arising from shared characteristics among different tissues across various spatial regions. By incorporating MRF physics-based data priors and effectively capturing local and global correlations, our proposed LG-ViT can achieve fast and accurate MRF parameter reconstruction. Experiments on both simulation and in vivo data demonstrate that the proposed method enables faster and more accurate MRF parameter reconstruction compared to state-of-the-art deep learning-based methods.

Greater benefits of deep learning-based computer-aided detection systems for finding small signals in 3D volumetric medical images

Purpose

Radiologists are tasked with visually scrutinizing large amounts of data produced by 3D volumetric imaging modalities. Small signals can go unnoticed during the 3D search because they are hard to detect in the visual periphery. Recent advances in machine learning and computer vision have led to effective computer-aided detection (CADe) support systems with the potential to mitigate perceptual errors.

Approach

Sixteen nonexpert observers searched through digital breast tomosynthesis (DBT) phantoms and single cross-sectional slices of the DBT phantoms. The 3D/2D searches occurred with and without a convolutional neural network (CNN)-based CADe support system. The model provided observers with bounding boxes superimposed on the image stimuli while they looked for a small microcalcification signal and a large mass signal. Eye gaze positions were recorded and correlated with changes in the area under the ROC curve (AUC).

Results

The CNN-CADe improved the 3D search for the small microcalcification signal ( Δ   AUC = 0.098 , p = 0.0002 ) and the 2D search for the large mass signal ( Δ   AUC = 0.076 , p = 0.002 ). The CNN-CADe benefit in 3D for the small signal was markedly greater than in 2D ( Δ Δ   AUC = 0.066 , p = 0.035 ). Analysis of individual differences suggests that those who explored the least with eye movements benefited the most from the CNN-CADe ( r = - 0.528 , p = 0.036 ). However, for the large signal, the 2D benefit was not significantly greater than the 3D benefit ( Δ Δ   AUC = 0.033 , p = 0.133 ).

Conclusion

The CNN-CADe brings unique performance benefits to the 3D (versus 2D) search of small signals by reducing errors caused by the underexploration of the volumetric data.

The effect of head motion on brain age prediction using deep convolutional neural networks

Deep learning can be used effectively to predict participants' age from brain magnetic resonance imaging (MRI) data, and a growing body of evidence suggests that the difference between predicted and chronological age-referred to as brain-predicted age difference (brain-PAD)-is related to various neurological and neuropsychiatric disease states. A crucial aspect of the applicability of brain-PAD as a biomarker of individual brain health is whether and how brain-predicted age is affected by MR image artifacts commonly encountered in clinical settings. To investigate this issue, we trained and validated two different 3D convolutional neural network architectures (CNNs) from scratch and tested the models on a separate dataset consisting of motion-free and motion-corrupted T1-weighted MRI scans from the same participants, the quality of which were rated by neuroradiologists from a clinical diagnostic point of view. Our results revealed a systematic increase in brain-PAD with worsening image quality for both models. This effect was also observed for images that were deemed usable from a clinical perspective, with brains appearing older in medium than in good quality images. These findings were also supported by significant associations found between the brain-PAD and standard image quality metrics indicating larger brain-PAD for lower-quality images. Our results demonstrate a spurious effect of advanced brain aging as a result of head motion and underline the importance of controlling for image quality when using brain-predicted age based on structural neuroimaging data as a proxy measure for brain health.

Identification of Calculous Pyonephrosis by CT-Based Radiomics and Deep Learning

Urgent detection of calculous pyonephrosis is crucial for surgical planning and preventing severe outcomes. This study aims to evaluate the performance of computed tomography (CT)-based radiomics and a three-dimensional convolutional neural network (3D-CNN) model, integrated with independent clinical factors, to identify patients with calculous pyonephrosis. We recruited 182 patients receiving either percutaneous nephrostomy tube placement or percutaneous nephrolithotomy for calculous hydronephrosis or pyonephrosis. The regions of interest were manually delineated on plain CT images and the CT attenuation value (HU) was measured. Radiomics analysis was performed using least absolute shrinkage and selection operator (LASSO). A 3D-CNN model was also developed. The better-performing machine-learning model was combined with independent clinical factors to build a comprehensive clinical machine-learning model. The performance of these models was assessed using receiver operating characteristic analysis and decision curve analysis. Fever, blood neutrophils, and urine leukocytes were independent risk factors for pyonephrosis. The radiomics model showed higher area under the curve (AUC) than the 3D-CNN model and HU (0.876 vs. 0.599, 0.578; p = 0.003, 0.002) in the testing cohort. The clinical machine-learning model surpassed the clinical model in both the training (0.975 vs. 0.904, p = 0.019) and testing (0.967 vs. 0.889, p = 0.045) cohorts.

CAManim: Animating end-to-end network activation maps

Deep neural networks have been widely adopted in numerous domains due to their high performance and accessibility to developers and application-specific end-users. Fundamental to image-based applications is the development of Convolutional Neural Networks (CNNs), which possess the ability to automatically extract features from data. However, comprehending these complex models and their learned representations, which typically comprise millions of parameters and numerous layers, remains a challenge for both developers and end-users. This challenge arises due to the absence of interpretable and transparent tools to make sense of black-box models. There exists a growing body of Explainable Artificial Intelligence (XAI) literature, including a collection of methods denoted Class Activation Maps (CAMs), that seek to demystify what representations the model learns from the data, how it informs a given prediction, and why it, at times, performs poorly in certain tasks. We propose a novel XAI visualization method denoted CAManim that seeks to simultaneously broaden and focus end-user understanding of CNN predictions by animating the CAM-based network activation maps through all layers, effectively depicting from end-to-end how a model progressively arrives at the final layer activation. Herein, we demonstrate that CAManim works with any CAM-based method and various CNN architectures. Beyond qualitative model assessments, we additionally propose a novel quantitative assessment that expands upon the Remove and Debias (ROAD) metric, pairing the qualitative end-to-end network visual explanations assessment with our novel quantitative "yellow brick ROAD" assessment (ybROAD). This builds upon prior research to address the increasing demand for interpretable, robust, and transparent model assessment methodology, ultimately improving an end-user's trust in a given model's predictions. Examples and source code can be found at: https://omni-ml.github.io/pytorch-grad-cam-anim/.

Generative AI unlocks PET insights: brain amyloid dynamics and quantification

Introduction

Studying the spatiotemporal patterns of amyloid accumulation in the brain over time is crucial in understanding Alzheimer's disease (AD). Positron Emission Tomography (PET) imaging plays a pivotal role because it allows for the visualization and quantification of abnormal amyloid beta (Aβ) load in the living brain, providing a powerful tool for tracking disease progression and evaluating the efficacy of anti-amyloid therapies. Generative artificial intelligence (AI) can learn complex data distributions and generate realistic synthetic images. In this study, we demonstrate for the first time the potential of Generative Adversarial Networks (GANs) to build a low-dimensional representation space that effectively describes brain amyloid load and its dynamics.

Methods

Using a cohort of 1,259 subjects with AV45 PET images from the Alzheimer's Disease Neuroimaging Initiative (ADNI), we develop a 3D GAN model to project images into a latent representation space and generate back synthetic images. Then, we build a progression model on the representation space based on non-parametric ordinary differential equations to study brain amyloid evolution.

Results

We found that global SUVR can be accurately predicted with a linear regression model only from the latent representation space (RMSE = 0.08 ± 0.01). We generated synthetic PET trajectories and illustrated predicted Aβ change in four years compared with actual progression.

Discussion

Generative AI can generate rich representations for statistical prediction and progression modeling and simulate evolution in synthetic patients, providing an invaluable tool for understanding AD, assisting in diagnosis, and designing clinical trials. The aim of this study was to illustrate the huge potential that generative AI has in brain amyloid imaging and to encourage its advancement by providing use cases and ideas for future research tracks.

Review of robotic systems for thoracoabdominal puncture interventional surgery

Cancer, with high morbidity and high mortality, is one of the major burdens threatening human health globally. Intervention procedures via percutaneous puncture have been widely used by physicians due to its minimally invasive surgical approach. However, traditional manual puncture intervention depends on personal experience and faces challenges in terms of precisely puncture, learning-curve, safety and efficacy. The development of puncture interventional surgery robotic (PISR) systems could alleviate the aforementioned problems to a certain extent. This paper attempts to review the current status and prospective of PISR systems for thoracic and abdominal application. In this review, the key technologies related to the robotics, including spatial registration, positioning navigation, puncture guidance feedback, respiratory motion compensation, and motion control, are discussed in detail.

Clinical prediction models for knee pain in patients with knee osteoarthritis: a systematic review

OBJECTIVE

To identify and describe existing models for predicting knee pain in patients with knee osteoarthritis.

METHODS

The electronic databases PubMed, EMBASE, CINAHL, Web of Science, and Cochrane Library were searched from their inception to May 2023 for any studies to develop and validate a prediction model for predicting knee pain in patients with knee osteoarthritis. Two reviewers independently screened titles, abstracts, and full-text qualifications, and extracted data. Risk of bias was assessed using the PROBAST. Data extraction of eligible articles was extracted by a data extraction form based on CHARMS. The quality of evidence was graded according to GRADE. The results were summarized with descriptive statistics.

RESULTS

The search identified 2693 records. Sixteen articles reporting on 26 prediction models were included targeting occurrence (n = 9), others (n = 7), progression (n = 5), persistent (n = 2), incident (n = 1), frequent (n = 1), and flares (n = 1) of knee pain. Most of the studies (94%) were at high risk of bias. Model discrimination was assessed by the AUROC ranging from 0.62 to 0.81. The most common predictors were age, BMI, gender, baseline pain, and joint space width. Only frequent knee pain had a moderate quality of evidence; all other types of knee pain had a low quality of evidence.

CONCLUSION

There are many prediction models for knee pain in patients with knee osteoarthritis that do show promise. However, the clinical extensibility, applicability, and interpretability of predictive tools should be considered during model development.

Staging of Liver Fibrosis Based on Energy Valley Optimization Multiple Stacking (EVO-MS) Model

Currently, staging the degree of liver fibrosis predominantly relies on liver biopsy, a method fraught with potential risks, such as bleeding and infection. With the rapid development of medical imaging devices, quantification of liver fibrosis through image processing technology has become feasible. Stacking technology is one of the effective ensemble techniques for potential usage, but precise tuning to find the optimal configuration manually is challenging. Therefore, this paper proposes a novel EVO-MS model-a multiple stacking ensemble learning model optimized by the energy valley optimization (EVO) algorithm to select most informatic features for fibrosis quantification. Liver contours are profiled from 415 biopsied proven CT cases, from which 10 shape features are calculated and inputted into a Support Vector Machine (SVM) classifier to generate the accurate predictions, then the EVO algorithm is applied to find the optimal parameter combination to fuse six base models: K-Nearest Neighbors (KNNs), Decision Tree (DT), Naive Bayes (NB), Extreme Gradient Boosting (XGB), Gradient Boosting Decision Tree (GBDT), and Random Forest (RF), to create a well-performing ensemble model. Experimental results indicate that selecting 3-5 feature parameters yields satisfactory results in classification, with features such as the contour roundness non-uniformity (Rmax), maximum peak height of contour (Rp), and maximum valley depth of contour (Rm) significantly influencing classification accuracy. The improved EVO algorithm, combined with a multiple stacking model, achieves an accuracy of 0.864, a precision of 0.813, a sensitivity of 0.912, a specificity of 0.824, and an F1-score of 0.860, which demonstrates the effectiveness of our EVO-MS model in staging the degree of liver fibrosis.

Analysis of 3D pathology samples using weakly supervised AI

Human tissue, which is inherently three-dimensional (3D), is traditionally examined through standard-of-care histopathology as limited two-dimensional (2D) cross-sections that can insufficiently represent the tissue due to sampling bias. To holistically characterize histomorphology, 3D imaging modalities have been developed, but clinical translation is hampered by complex manual evaluation and lack of computational platforms to distill clinical insights from large, high-resolution datasets. We present TriPath, a deep-learning platform for processing tissue volumes and efficiently predicting clinical outcomes based on 3D morphological features. Recurrence risk-stratification models were trained on prostate cancer specimens imaged with open-top light-sheet microscopy or microcomputed tomography. By comprehensively capturing 3D morphologies, 3D volume-based prognostication achieves superior performance to traditional 2D slice-based approaches, including clinical/histopathological baselines from six certified genitourinary pathologists. Incorporating greater tissue volume improves prognostic performance and mitigates risk prediction variability from sampling bias, further emphasizing the value of capturing larger extents of heterogeneous morphology.

Progression or Aging? A Deep Learning Approach for Distinguishing Glaucoma Progression From Age-Related Changes in OCT Scans

PURPOSE

To develop deep learning (DL) algorithm to detect glaucoma progression using optical coherence tomography (OCT) images, in the absence of a reference standard.

DESIGN

Retrospective cohort study.

METHODS

Glaucomatous and healthy eyes with ≥5 reliable peripapillary OCT (Spectralis, Heidelberg Engineering) circle scans were included. A weakly supervised time-series learning model, called noise positive-unlabeled (Noise-PU) DL was developed to classify whether sequences of OCT B-scans showed glaucoma progression. The model used 2 learning schemes, one to identify age-related changes by differentiating test sequences from glaucoma vs healthy eyes, and the other to identify test-retest variability based on scrambled OCTs of glaucoma eyes. Both models' bases were convolutional neural networks (CNN) and long short-term memory (LSTM) networks which were combined to form a CNN-LSTM model. Model features were combined and jointly trained to identify glaucoma progression, accounting for age-related loss. The DL model's outcomes were compared with ordinary least squares (OLS) regression of retinal nerve fiber layer (RNFL) thickness over time, matched for specificity. The hit ratio was used as a proxy for sensitivity.

RESULTS

Eight thousand seven hundred eighty-five follow-up sequences of 5 consecutive OCT tests from 3253 eyes (1859 subjects) were included in the study. The mean follow-up time was 3.5 ± 1.6 years. In the test sample, the hit ratios of the DL and OLS methods were 0.498 (95%CI: 0.470-0.526) and 0.284 (95%CI: 0.258-0.309) respectively (P < .001) when the specificities were equalized to 95%.

CONCLUSION

A DL model was able to identify longitudinal glaucomatous structural changes in OCT B-scans using a surrogate reference standard for progression.

Point Cloud Attacks in Graph Spectral Domain: When 3D Geometry Meets Graph Signal Processing

With the increasing attention in various 3D safety-critical applications, point cloud learning models have been shown to be vulnerable to adversarial attacks. Although existing 3D attack methods achieve high success rates, they delve into the data space with point-wise perturbation, which may neglect the geometric characteristics. Instead, we propose point cloud attacks from a new perspective-the graph spectral domain attack, aiming to perturb graph transform coefficients in the spectral domain that correspond to varying certain geometric structures. Specifically, leveraging on graph signal processing, we first adaptively transform the coordinates of points onto the spectral domain via graph Fourier transform (GFT) for compact representation. Then, we analyze the influence of different spectral bands on the geometric structure, based on which we propose to perturb the GFT coefficients via a learnable graph spectral filter. Considering the low-frequency components mainly contribute to the rough shape of the 3D object, we further introduce a low-frequency constraint to limit perturbations within imperceptible high-frequency components. Finally, the adversarial point cloud is generated by transforming the perturbed spectral representation back to the data domain via the inverse GFT. Experimental results demonstrate the effectiveness of the proposed attack in terms of both the imperceptibility and attack success rates.

Registration of multimodal bone images based on edge similarity metaheuristic

OBJECTIVE

Blurry medical images affect the accuracy and efficiency of multimodal image registration, whose existing methods require further improvement.

METHODS

We propose an edge-based similarity registration method optimised for multimodal medical images, especially bone images, by a balance optimiser. First, we use a GPU (graphics processing unit) rendering simulation to convert computed tomography data into digitally reconstructed radiographs. Second, we introduce the improved cascaded edge network (ICENet), a convolutional neural network that extracts edge information of blurry medical images. Then, the bilateral Gaussian-weighted similarity of pairs of X-ray images and digitally reconstructed radiographs is measured. The a balanced optimiser is iteratively applied to finally estimate the best pose to perform image registration.

RESULTS

Experimental results show that, on average, the proposed method with ICENet outperforms other edge detection networks by 20%, 12%, 18.83%, and 11.93% in the overall Dice similarity, overall intersection over union, peak signal-to-noise ratio, and structural similarity index, respectively, with a registration success rate up to 90% and average reduction of 220% in registration time.

CONCLUSION

The proposed method with ICENet can achieve a high registration success rate even for blurry medical images, and its efficiency and robustness are higher than those of existing methods.

SIGNIFICANCE

Our proposal may be suitable for supporting medical diagnosis, radiation therapy, image-guided surgery, and other clinical applications.

DepthCrackNet: A Deep Learning Model for Automatic Pavement Crack Detection

Detecting cracks in the pavement is a vital component of ensuring road safety. Since manual identification of these cracks can be time-consuming, an automated method is needed to speed up this process. However, creating such a system is challenging due to factors including crack variability, variations in pavement materials, and the occurrence of miscellaneous objects and anomalies on the pavement. Motivated by the latest progress in deep learning applied to computer vision, we propose an effective U-Net-shaped model named DepthCrackNet. Our model employs the Double Convolution Encoder (DCE), composed of a sequence of convolution layers, for robust feature extraction while keeping parameters optimally efficient. We have incorporated the TriInput Multi-Head Spatial Attention (TMSA) module into our model; in this module, each head operates independently, capturing various spatial relationships and boosting the extraction of rich contextual information. Furthermore, DepthCrackNet employs the Spatial Depth Enhancer (SDE) module, specifically designed to augment the feature extraction capabilities of our segmentation model. The performance of the DepthCrackNet was evaluated on two public crack datasets: Crack500 and DeepCrack. In our experimental studies, the network achieved mIoU scores of 77.0% and 83.9% with the Crack500 and DeepCrack datasets, respectively.

Semi or fully automatic tooth segmentation in CBCT images: a review

Cone beam computed tomography (CBCT) is widely employed in modern dentistry, and tooth segmentation constitutes an integral part of the digital workflow based on these imaging data. Previous methodologies rely heavily on manual segmentation and are time-consuming and labor-intensive in clinical practice. Recently, with advancements in computer vision technology, scholars have conducted in-depth research, proposing various fast and accurate tooth segmentation methods. In this review, we review 55 articles in this field and discuss the effectiveness, advantages, and disadvantages of each approach. In addition to simple classification and discussion, this review aims to reveal how tooth segmentation methods can be improved by the application and refinement of existing image segmentation algorithms to solve problems such as irregular morphology and fuzzy boundaries of teeth. It is assumed that with the optimization of these methods, manual operation will be reduced, and greater accuracy and robustness in tooth segmentation will be achieved. Finally, we highlight the challenges that still exist in this field and provide prospects for future directions.

Deep Learning for Automated Measurement of Total Cardiac Volume for Heart Transplantation Size Matching

Total Cardiac Volume (TCV)-based size matching using Computed Tomography (CT) is a novel technique to compare donor and recipient heart size in pediatric heart transplant that may increase overall utilization of available grafts. TCV requires manual segmentation, which limits its widespread use due to time and specialized software and training needed for segmentation. This study aims to determine the accuracy of a Deep Learning (DL) approach using 3-dimensional Convolutional Neural Networks (3D-CNN) to calculate TCV, with the clinical aim of enabling fast and accurate TCV use at all transplant centers. Ground truth TCV was segmented on CT scans of subjects aged 0-30 years, identified retrospectively. Ground truth segmentation masks were used to train and test a custom 3D-CNN model consisting of a DenseNet architecture in combination with residual blocks of ResNet architecture. The model was trained on a cohort of 270 subjects and a validation cohort of 44 subjects (36 normal, 8 heart disease retained for model testing). The average Dice similarity coefficient of the validation cohort was 0.94 ± 0.03 (range 0.84-0.97). The mean absolute percent error of TCV estimation was 5.5%. There is no significant association between model accuracy and subject age, weight, or height. DL-TCV was on average more accurate for normal hearts than those listed for transplant (mean absolute percent error 4.5 ± 3.9 vs. 10.5 ± 8.5, p = 0.08). A deep learning-based 3D-CNN model can provide accurate automatic measurement of TCV from CT images. This initial study is limited as a single-center study, though future multicenter studies may enable generalizable and more accurate TCV measurement by inclusion of more diverse cardiac pathology and increasing the training data.

Artificial intelligence serving pre-surgical digital implant planning: A scoping review

OBJECTIVES

To conduct a scoping review focusing on artificial intelligence (AI) applications in presurgical dental implant planning. Additionally, to assess the automation degree of clinically available pre-surgical implant planning software.

DATA AND SOURCES

A systematic electronic literature search was performed in five databases (PubMed, Embase, Web of Science, Cochrane Library, and Scopus), along with exploring gray literature web-based resources until November 2023. English-language studies on AI-driven tools for digital implant planning were included based on an independent evaluation by two reviewers. An assessment of automation steps in dental implant planning software available on the market up to November 2023 was also performed.

STUDY SELECTION AND RESULTS

From an initial 1,732 studies, 47 met eligibility criteria. Within this subset, 39 studies focused on AI networks for anatomical landmark-based segmentation, creating virtual patients. Eight studies were dedicated to AI networks for virtual implant placement. Additionally, a total of 12 commonly available implant planning software applications were identified and assessed for their level of automation in pre-surgical digital implant workflows. Notably, only six of these featured at least one fully automated step in the planning software, with none possessing a fully automated implant planning protocol.

CONCLUSIONS

AI plays a crucial role in achieving accurate, time-efficient, and consistent segmentation of anatomical landmarks, serving the process of virtual patient creation. Additionally, currently available systems for virtual implant placement demonstrate different degrees of automation. It is important to highlight that, as of now, full automation of this process has not been documented nor scientifically validated.

CLINICAL SIGNIFICANCE

Scientific and clinical validation of AI applications for presurgical dental implant planning is currently scarce. The present review allows the clinician to identify AI-based automation in presurgical dental implant planning and assess the potential underlying scientific validation.

Preliminary data on artificial intelligence tool in magnetic resonance imaging assessment of degenerative pathologies of lumbar spine

PURPOSE

To evaluate the ability of an artificial intelligence (AI) tool in magnetic resonance imaging (MRI) assessment of degenerative pathologies of lumbar spine using radiologist evaluation as a gold standard.

METHODS

Patients with degenerative pathologies of lumbar spine, evaluated with MRI study, were enrolled in a retrospective study approved by local ethical committee. A comprehensive software solution (CoLumbo; SmartSoft Ltd., Varna, Bulgaria) designed to label the segments of the lumbar spine and to detect a broad spectrum of degenerative pathologies based on a convolutional neural network (CNN) was employed, utilizing an automatic segmentation. The AI tool efficacy was compared to data obtained by a senior neuroradiologist that employed a semiquantitative score. Chi-square test was used to assess the differences among groups, and Spearman's rank correlation coefficient was calculated between the grading assigned by radiologist and the grading obtained by software. Moreover, agreement was assessed between the value assigned by radiologist and software.

RESULTS

Ninety patients (58 men; 32 women) affected with degenerative pathologies of lumbar spine and aged from 60 to 81 years (mean 66 years) were analyzed. Significant correlations were observed between grading assigned by radiologist and the grading obtained by software for each localization. However, only when the localization was L2-L3, there was a good correlation with a coefficient value of 0.72. The best agreements were obtained in case of L1-L2 and L2-L3 localizations and were, respectively, of 81.1% and 72.2%. The lowest agreement of 51.1% was detected in case of L4-L5 locations. With regard canal stenosis and compression, the highest agreement was obtained for identification of in L5-S1 localization.

CONCLUSIONS

AI solution represents an efficacy and useful toll degenerative pathologies of lumbar spine to improve radiologist workflow.

3D CNN-based Deep Learning Model-based Explanatory Prognostication in Patients with Multiple Myeloma using Whole-body MRI

Although magnetic resonance imaging (MRI) data of patients with multiple myeloma (MM) are used to predict prognosis, few reports have applied artificial intelligence (AI) techniques for this purpose. We aimed to analyze whole-body diffusion-weighted MRI data using three-dimensional (3D) convolutional neural networks (CNNs) and Gradient-weighted Class Activation Mapping (Grad-CAM), an explainable AI, to predict prognosis and explore the factors involved in prediction. We retrospectively analyzed the MRI data of a total of 142 patients with MM obtained from two medical centers. We defined the occurrence of progressive disease after MRI evaluation within 12 months as a poor prognosis and constructed a 3D CNN-based deep learning model to predict prognosis. Images from 111 cases were used as the training and internal validation data; images from 31 cases were used as the external validation data. Internal validation of the AI model with stratified 5-fold cross-validation resulted in a significant difference in progression-free survival (PFS) between good and poor prognostic cases (2-year PFS, 91.2% versus [vs.] 61.1%, P = 0.0002). The AI model clearly stratified good and poor prognostic cases in the external validation cohort (2-year PFS, 92.9% vs. 55.6%, P = 0.004), with an area under the receiver operating characteristic curve of 0.804. According to Grad-CAM, the MRI signals of the spleen and bones of the vertebrae and pelvis contributed to prognosis prediction. This study is the first to show that image analysis of whole-body MRI using a 3D CNN without any other clinical data is effective in predicting the prognosis of patients with MM.

Added prognostic value of 3D deep learning-derived features from preoperative MRI for adult-type diffuse gliomas

BACKGROUND

To investigate the prognostic value of spatial features from whole-brain MRI using a three-dimensional (3D) convolutional neural network for adult-type diffuse gliomas.

METHODS

In a retrospective, multicenter study, 1925 diffuse glioma patients were enrolled from 5 datasets: SNUH (n = 708), UPenn (n = 425), UCSF (n = 500), TCGA (n = 160), and Severance (n = 132). The SNUH and Severance datasets served as external test sets. Precontrast and postcontrast 3D T1-weighted, T2-weighted, and T2-FLAIR images were processed as multichannel 3D images. A 3D-adapted SE-ResNeXt model was trained to predict overall survival. The prognostic value of the deep learning-based prognostic index (DPI), a spatial feature-derived quantitative score, and established prognostic markers were evaluated using Cox regression. Model evaluation was performed using the concordance index (C-index) and Brier score.

RESULTS

The MRI-only median DPI survival prediction model achieved C-indices of 0.709 and 0.677 (BS = 0.142 and 0.215) and survival differences (P < 0.001 and P = 0.002; log-rank test) for the SNUH and Severance datasets, respectively. Multivariate Cox analysis revealed DPI as a significant prognostic factor, independent of clinical and molecular genetic variables: hazard ratio = 0.032 and 0.036 (P < 0.001 and P = 0.004) for the SNUH and Severance datasets, respectively. Multimodal prediction models achieved higher C-indices than models using only clinical and molecular genetic variables: 0.783 vs. 0.774, P = 0.001, SNUH; 0.766 vs. 0.748, P = 0.023, Severance.

CONCLUSIONS

The global morphologic feature derived from 3D CNN models using whole-brain MRI has independent prognostic value for diffuse gliomas. Combining clinical, molecular genetic, and imaging data yields the best performance.

AI-organoid integrated systems for biomedical studies and applications

In this review, we explore the growing role of artificial intelligence (AI) in advancing the biomedical applications of human pluripotent stem cell (hPSC)-derived organoids. Stem cell-derived organoids, these miniature organ replicas, have become essential tools for disease modeling, drug discovery, and regenerative medicine. However, analyzing the vast and intricate datasets generated from these organoids can be inefficient and error-prone. AI techniques offer a promising solution to efficiently extract insights and make predictions from diverse data types generated from microscopy images, transcriptomics, metabolomics, and proteomics. This review offers a brief overview of organoid characterization and fundamental concepts in AI while focusing on a comprehensive exploration of AI applications in organoid-based disease modeling and drug evaluation. It provides insights into the future possibilities of AI in enhancing the quality control of organoid fabrication, label-free organoid recognition, and three-dimensional image reconstruction of complex organoid structures. This review presents the challenges and potential solutions in AI-organoid integration, focusing on the establishment of reliable AI model decision-making processes and the standardization of organoid research.

A novel radiological software prototype for automatically detecting the inner ear and classifying normal from malformed anatomy

BACKGROUND

To develop an effective radiological software prototype that could read Digital Imaging and Communications in Medicine (DICOM) files, crop the inner ear automatically based on head computed tomography (CT), and classify normal and inner ear malformation (IEM).

METHODS

A retrospective analysis was conducted on 2053 patients from 3 hospitals. We extracted 1200 inner ear CTs for importing, cropping, and training, testing, and validating an artificial intelligence (AI) model. Automated cropping algorithms based on CTs were developed to precisely isolate the inner ear volume. Additionally, a simple graphical user interface (GUI) was implemented for user interaction. Using cropped CTs as input, a deep learning convolutional neural network (DL CNN) with 5-fold cross-validation was used to classify inner ear anatomy as normal or abnormal. Five specific IEM types (cochlear hypoplasia, ossification, incomplete partition types I and III, and common cavity) were included, with data equally distributed between classes. Both the cropping tool and the AI model were extensively validated.

RESULTS

The newly developed DICOM viewer/software successfully achieved its objectives: reading CT files, automatically cropping inner ear volumes, and classifying them as normal or malformed. The cropping tool demonstrated an average accuracy of 92.25%. The DL CNN model achieved an area under the curve (AUC) of 0.86 (95% confidence interval: 0.81-0.91). Performance metrics for the AI model were: accuracy (0.812), precision (0.791), recall (0.8), and F1-score (0.766).

CONCLUSION

This study successfully developed and validated a fully automated workflow for classifying normal versus abnormal inner ear anatomy using a combination of advanced image processing and deep learning techniques. The tool exhibited good diagnostic accuracy, suggesting its potential application in risk stratification. However, it is crucial to emphasize the need for supervision by qualified medical professionals when utilizing this tool for clinical decision-making.

A hybrid multimodal machine learning model for Detecting Alzheimer's disease

Alzheimer's disease (AD) diagnosis utilizing single modality neuroimaging data has limitations. Multimodal fusion of complementary biomarkers may improve diagnostic performance. This study proposes a multimodal machine learning framework integrating magnetic resonance imaging (MRI), positron emission tomography (PET) and cerebrospinal fluid (CSF) assays for enhanced AD characterization. The model incorporates a hybrid algorithm combining enhanced Harris Hawks Optimization (HHO) algorithm referred to as ILHHO, with Kernel Extreme Learning Machine (KELM) classifier for simultaneous feature selection and classification. ILHHO enhances HHO's search efficiency by integrating iterative mapping (IM) to improve population diversity and local escaping operator (LEO) to balance exploration-exploitation. Comparative analysis with other improved HHO algorithms, classic meta-heuristic algorithms (MHAs), and state-of-the-art MHAs on IEEE CEC2014 benchmark functions indicates that ILHHO achieves superior optimization performance compared to other comparative algorithms. The synergistic ILHHO-KELM model is evaluated on 202 AD Neuroimaging Initiative (ADNI) subjects. Results demonstrate superior multimodal classification accuracy over single modalities, validating the importance of fusing heterogeneous biomarkers. MRI + PET + CSF achieves 99.2 % accuracy for AD vs. normal control (NC), outperforming conventional and proposed methods. Discriminative feature analysis provides further insights into differential AD-related neurodegeneration patterns detected by MRI and PET. The differential PET and MRI features demonstrate how the two modalities provide complementary biomarkers. The neuroanatomical relevance of selected features supports ILHHO-KELM's potential for extracting sensitive AD imaging signatures. Overall, the study showcases the advantages of capitalizing on complementary multimodal data through advanced feature learning techniques for improving AD diagnosis.

Radiomics for residual tumour detection and prognosis in newly diagnosed glioblastoma based on postoperative [11C] methionine PET and T1c-w MRI

Personalized treatment strategies based on non-invasive biomarkers have potential to improve patient management in patients with newly diagnosed glioblastoma (GBM). The residual tumour burden after surgery in GBM patients is a prognostic imaging biomarker. However, in clinical patient management, its assessment is a manual and time-consuming process that is at risk of inter-rater variability. Furthermore, the prediction of patient outcome prior to radiotherapy may identify patient subgroups that could benefit from escalated radiotherapy doses. Therefore, in this study, we investigate the capabilities of traditional radiomics and 3D convolutional neural networks for automatic detection of the residual tumour status and to prognosticate time-to-recurrence (TTR) and overall survival (OS) in GBM using postoperative [11C] methionine positron emission tomography (MET-PET) and gadolinium-enhanced T1-w magnetic resonance imaging (MRI). On the independent test data, the 3D-DenseNet model based on MET-PET achieved the best performance for residual tumour detection, while the logistic regression model with conventional radiomics features performed best for T1c-w MRI (AUC: MET-PET 0.95, T1c-w MRI 0.78). For the prognosis of TTR and OS, the 3D-DenseNet model based on MET-PET integrated with age and MGMT status achieved the best performance (Concordance-Index: TTR 0.68, OS 0.65). In conclusion, we showed that both deep-learning and conventional radiomics have potential value for supporting image-based assessment and prognosis in GBM. After prospective validation, these models may be considered for treatment personalization.

Deep learning-enabled detection of hypoxic-ischemic encephalopathy after cardiac arrest in CT scans: a comparative study of 2D and 3D approaches

Objective

To establish a deep learning model for the detection of hypoxic-ischemic encephalopathy (HIE) features on CT scans and to compare various networks to determine the best input data format.

Methods

168 head CT scans of patients after cardiac arrest were retrospectively identified and classified into two categories: 88 (52.4%) with radiological evidence of severe HIE and 80 (47.6%) without signs of HIE. These images were randomly divided into a training and a test set, and five deep learning models based on based on Densely Connected Convolutional Networks (DenseNet121) were trained and validated using different image input formats (2D and 3D images).

Results

All optimized stacked 2D and 3D networks could detect signs of HIE. The networks based on the data as 2D image data stacks provided the best results (S100: AUC: 94%, ACC: 79%, S50: AUC: 93%, ACC: 79%). We provide visual explainability data for the decision making of our AI model using Gradient-weighted Class Activation Mapping.

Conclusion

Our proof-of-concept deep learning model can accurately identify signs of HIE on CT images. Comparing different 2D- and 3D-based approaches, most promising results were achieved by 2D image stack models. After further clinical validation, a deep learning model of HIE detection based on CT images could be implemented in clinical routine and thus aid clinicians in characterizing imaging data and predicting outcome.

Deep learning framework for three-dimensional surface reconstruction of object of interest in photoacoustic tomography

Photoacoustic tomography (PAT) is a non-ionizing hybrid imaging technology of clinical importance that combines the high contrast of optical imaging with the high penetration of ultrasonic imaging. Two-dimensional (2D) tomographic images can only provide the cross-sectional structure of the imaging target rather than its overall spatial morphology. This work proposes a deep learning framework for reconstructing three-dimensional (3D) surface of an object of interest from a series of 2D images. It achieves end-to-end mapping from a series of 2D images to a 3D image, visually displaying the overall morphology of the object. The framework consists of four modules: segmentation module, point cloud generation module, point cloud completion module, and mesh conversion module, which respectively implement the tasks of segmenting a region of interest, generating a sparse point cloud, completing sparse point cloud and reconstructing 3D surface. The network model is trained on simulation data sets and verified on simulation, phantom, and in vivo data sets. The results showed superior 3D reconstruction performance both visually and on the basis of quantitative evaluation metrics compared to the state-of-the-art non-learning and learning approaches. This method potentially enables high-precision 3D surface reconstruction from the tomographic images output by the preclinical PAT system without changing the imaging system. It provides a general deep learning scheme for 3D reconstruction from tomographic scanning data.

The geometric evolution of aortic dissections: Predicting surgical success using fluctuations in integrated Gaussian curvature

Clinical imaging modalities are a mainstay of modern disease management, but the full utilization of imaging-based data remains elusive. Aortic disease is defined by anatomic scalars quantifying aortic size, even though aortic disease progression initiates complex shape changes. We present an imaging-based geometric descriptor, inspired by fundamental ideas from topology and soft-matter physics that captures dynamic shape evolution. The aorta is reduced to a two-dimensional mathematical surface in space whose geometry is fully characterized by the local principal curvatures. Disease causes deviation from the smooth bent cylindrical shape of normal aortas, leading to a family of highly heterogeneous surfaces of varying shapes and sizes. To deconvolute changes in shape from size, the shape is characterized using integrated Gaussian curvature or total curvature. The fluctuation in total curvature (δK) across aortic surfaces captures heterogeneous morphologic evolution by characterizing local shape changes. We discover that aortic morphology evolves with a power-law defined behavior with rapidly increasing δK forming the hallmark of aortic disease. Divergent δK is seen for highly diseased aortas indicative of impending topologic catastrophe or aortic rupture. We also show that aortic size (surface area or enclosed aortic volume) scales as a generalized cylinder for all shapes. Classification accuracy for predicting aortic disease state (normal, diseased with successful surgery, and diseased with failed surgical outcomes) is 92.8±1.7%. The analysis of δK can be applied on any three-dimensional geometric structure and thus may be extended to other clinical problems of characterizing disease through captured anatomic changes.

Comparing three-dimensional and two-dimensional deep-learning, radiomics, and fusion models for predicting occult lymph node metastasis in laryngeal squamous cell carcinoma based on CT imaging: a multicentre, retrospective, diagnostic study

Background

The occult lymph node metastasis (LNM) of laryngeal squamous cell carcinoma (LSCC) affects the treatment and prognosis of patients. This study aimed to comprehensively compare the performance of the three-dimensional and two-dimensional deep learning models, radiomics model, and the fusion models for predicting occult LNM in LSCC.

Methods

In this retrospective diagnostic study, a total of 553 patients with clinical N0 stage LSCC, who underwent surgical treatment without distant metastasis and multiple primary cancers, were consecutively enrolled from four Chinese medical centres between January 01, 2016 and December 30, 2020. The participant data were manually retrieved from medical records, imaging databases, and pathology reports. The study cohort was divided into a training set (n = 300), an internal test set (n = 89), and two external test sets (n = 120 and 44, respectively). The three-dimensional deep learning (3D DL), two-dimensional deep learning (2D DL), and radiomics model were developed using CT images of the primary tumor. The clinical model was constructed based on clinical and radiological features. Two fusion strategies were utilized to develop the fusion model: the feature-based DLRad_FB model and the decision-based DLRad_DB model. The discriminative ability and correlation of 3D DL, 2D DL and radiomics features were analysed comprehensively. The performances of the predictive models were evaluated based on the pathological diagnosis.

Findings

The 3D DL features had superior discriminative ability and lower internal redundancy compared to 2D DL and radiomics features. The DLRad_DB model achieved the highest AUC (0.89-0.90) among all the study sets, significantly outperforming the clinical model (AUC = 0.73-0.78, P = 0.0001-0.042, Delong test). Compared to the DLRad_DB model, the AUC values for the DLRad_FB, 3D DL, 2D DL, and radiomics models were 0.82-0.84 (P = 0.025-0.46), 0.86-0.89 (P = 0.75-0.97), 0.83-0.86 (P = 0.029-0.66), and 0.79-0.82 (P = 0.0072-0.10), respectively in the study sets. Additionally, the DLRad_DB model exhibited the best sensitivity (82-88%) and specificity (79-85%) in the test sets.

Interpretation

The decision-based fusion model DLRad_DB, which combines 3D DL, 2D DL, radiomics, and clinical data, can be utilized to predict occult LNM in LSCC. This has the potential to minimize unnecessary lymph node dissection and prophylactic radiotherapy in patients with cN0 disease.

Funding

National Natural Science Foundation of China, Natural Science Foundation of Shandong Province.

MGMT promoter methylation prediction based on multiparametric MRI via vision graph neural network

Purpose

Glioblastoma (GBM) is aggressive and malignant. The methylation status of the O 6 -methylguanine-DNA methyltransferase (MGMT) promoter in GBM tissue is considered an important biomarker for developing the most effective treatment plan. Although the standard method for assessing the MGMT promoter methylation status is via bisulfite modification and deoxyribonucleic acid (DNA) sequencing of biopsy or surgical specimens, a secondary automated method based on medical imaging may improve the efficiency and accuracy of those tests.

Approach

We propose a deep vision graph neural network (ViG) using multiparametric magnetic resonance imaging (MRI) to predict the MGMT promoter methylation status noninvasively. Our model was compared to the RSNA radiogenomic classification winners. The dataset includes 583 usable patient cases. Combinations of MRI sequences were compared. Our multi-sequence fusion strategy was compared with those using single MR sequences.

Results

Our best model [Fluid Attenuated Inversion Recovery (FLAIR), T1-weighted pre-contrast (T1w), T2-weighted (T2)] outperformed the winning models with a test area under the curve (AUC) of 0.628, an accuracy of 0.632, a precision of 0.646, a recall of 0.677, a specificity of 0.581, and an F1 score of 0.661. Compared to the winning models with single MR sequences, our ViG utilizing fused-MRI showed a significant improvement statistically in AUC scores, which are FLAIR (p = 0.042 ), T1w (p = 0.017 ), T1wCE (p = 0.001 ), and T2 (p = 0.018 ).

Conclusions

Our model is superior to challenge champions. A graph representation of the medical images enabled good handling of complexity and irregularity. Our work provides an automatic secondary check pipeline to ensure the correctness of MGMT methylation status prediction.

Brain age prediction using combined deep convolutional neural network and multi-layer perceptron algorithms

The clinical applications of brain age prediction have expanded, particularly in anticipating the onset and prognosis of various neurodegenerative diseases. In the current study, we proposed a deep learning algorithm that leverages brain structural imaging data and enhances prediction accuracy by integrating biological sex information. Our model for brain age prediction, built on deep neural networks, employed a dataset of 3004 healthy subjects aged 18 and above. The T1-weighted images were minimally preprocessed and analyzed using the convolutional neural network (CNN) algorithm. The categorical sex information was then incorporated using the multi-layer perceptron (MLP) algorithm. We trained and validated both a CNN-only algorithm (utilizing only brain structural imaging data), and a combined CNN-MLP algorithm (using both structural brain imaging data and sex information) for age prediction. By integrating sex information with T1-weighted imaging data, our proposed CNN-MLP algorithm outperformed not only the CNN-only algorithm but also established algorithms, such as brainageR, in prediction accuracy. Notably, this hybrid CNN-MLP algorithm effectively distinguished between mild cognitive impairment and Alzheimer's disease groups by identifying variances in brain age gaps between them, highlighting the algorithm's potential for clinical application. Overall, these results underscore the enhanced precision of the CNN-MLP algorithm in brain age prediction, achieved through the integration of sex information.

Voxel-wise Fusion of Resting fMRI Networks and Gray Matter Volume for Alzheimer's Disease Classification using Deep Multimodal Learning

Alzheimer's disease (AD) is a prevalent neurodegenerative disorder requiring accurate and early diagnosis for effective treatment. Resting-state functional magnetic resonance imaging (rs-fMRI) and gray matter volume analysis from structural MRI have emerged as valuable tools for investigating AD-related brain alterations. However, the potential benefits of integrating these modalities using deep learning techniques remain unexplored. In this study, we propose a novel framework that fuses composite images of multiple rs-fMRI networks (called voxelwise intensity projection) and gray matter segmentation images through a deep learning approach for improved AD classification. We demonstrate the superiority of fMRI networks over commonly used metrics such as amplitude of low-frequency fluctuations (ALFF) and fractional ALFF in capturing spatial maps critical for AD classification. We use a multi-channel convolutional neural network incorporating the AlexNet dropout architecture to effectively model spatial and temporal dependencies in the integrated data. Extensive experiments on the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset of AD patients and cognitively normal (CN) validate the efficacy of our approach, showcasing improved classification performance of 94.12% test accuracy and an area under the curve (AUC) score of 97.79 compared to existing methods. Our results show that the fusion results generally outperformed the unimodal results. The saliency visualizations also show significant differences in the hippocampus, amygdala, putamen, caudate nucleus, and regions of basal ganglia which are in line with the previous neurobiological literature. Our research offers a novel method to enhance our grasp of AD pathology. By integrating data from various functional networks with structural MRI insights, we significantly improve diagnostic accuracy. This accuracy is further boosted by the effective visualization of this combined information. This lays the groundwork for further studies focused on providing a more accurate and personalized approach to AD diagnosis. The proposed framework and insights gained from fMRI networks provide a promising avenue for future research in deep multimodal fusion and neuroimaging analysis.

A Systematic Literature Review of 3D Deep Learning Techniques in Computed Tomography Reconstruction

Computed tomography (CT) is used in a wide range of medical imaging diagnoses. However, the reconstruction of CT images from raw projection data is inherently complex and is subject to artifacts and noise, which compromises image quality and accuracy. In order to address these challenges, deep learning developments have the potential to improve the reconstruction of computed tomography images. In this regard, our research aim is to determine the techniques that are used for 3D deep learning in CT reconstruction and to identify the training and validation datasets that are accessible. This research was performed on five databases. After a careful assessment of each record based on the objective and scope of the study, we selected 60 research articles for this review. This systematic literature review revealed that convolutional neural networks (CNNs), 3D convolutional neural networks (3D CNNs), and deep learning reconstruction (DLR) were the most suitable deep learning algorithms for CT reconstruction. Additionally, two major datasets appropriate for training and developing deep learning systems were identified: 2016 NIH-AAPM-Mayo and MSCT. These datasets are important resources for the creation and assessment of CT reconstruction models. According to the results, 3D deep learning may increase the effectiveness of CT image reconstruction, boost image quality, and lower radiation exposure. By using these deep learning approaches, CT image reconstruction may be made more precise and effective, improving patient outcomes, diagnostic accuracy, and healthcare system productivity.

Diagnostic test accuracy of machine learning algorithms for the detection intracranial hemorrhage: a systematic review and meta-analysis study

BACKGROUND

This systematic review and meta-analysis were conducted to objectively evaluate the evidence of machine learning (ML) in the patient diagnosis of Intracranial Hemorrhage (ICH) on computed tomography (CT) scans.

METHODS

Until May 2023, systematic searches were conducted in ISI Web of Science, PubMed, Scopus, Cochrane Library, IEEE Xplore Digital Library, CINAHL, Science Direct, PROSPERO, and EMBASE for studies that evaluated the diagnostic precision of ML model-assisted ICH detection. Patients with and without ICH as the target condition who were receiving CT-Scan were eligible for the research, which used ML algorithms based on radiologists' reports as the gold reference standard. For meta-analysis, pooled sensitivities, specificities, and a summary receiver operating characteristics curve (SROC) were used.

RESULTS

At last, after screening the title, abstract, and full paper, twenty-six retrospective and three prospective, and two retrospective/prospective studies were included. The overall (Diagnostic Test Accuracy) DTA of retrospective studies with a pooled sensitivity was 0.917 (95% CI 0.88-0.943, I2 = 99%). The pooled specificity was 0.945 (95% CI 0.918-0.964, I2 = 100%). The pooled diagnostic odds ratio (DOR) was 219.47 (95% CI 104.78-459.66, I2 = 100%). These results were significant for the specificity of the different network architecture models (p-value = 0.0289). However, the results for sensitivity (p-value = 0.6417) and DOR (p-value = 0.2187) were not significant. The ResNet algorithm has higher pooled specificity than other algorithms with 0.935 (95% CI 0.854-0.973, I2 = 93%).

CONCLUSION

This meta-analysis on DTA of ML algorithms for detecting ICH by assessing non-contrast CT-Scans shows the ML has an acceptable performance in diagnosing ICH. Using ResNet in ICH detection remains promising prediction was improved via training in an Architecture Learning Network (ALN).

A framework for immunofluorescence image augmentation and classification based on unsupervised attention mechanism

Autoimmune encephalitis (AE) is a common neurological disorder. As a standard method for neuroautoantibody detection, pathologists use tissue matrix assays (TBA) for initial disease screening. In this study, microscopic fluorescence imaging was combined with deep learning to improve AE diagnostic accuracy. Due to the inter-class imbalance of medical data, we propose an innovative generative adversarial network supplemented with attention mechanisms to highlight key regions in images to synthesize high-quality fluorescence images. However, securing annotated medical data is both time-consuming and costly. To circumvent this problem, we employ a self-supervised learning approach that utilizes unlabeled fluorescence data to support downstream classification tasks. To better understand the fluorescence properties in the data, we introduce a multichannel input convolutional neural network that adds additional channels of fluorescence intensity. This study builds an AE immunofluorescence dataset and obtains the classification accuracy of 88.5% using our method, thus confirming the effectiveness of the proposed method.

Prognosis prediction of patients with malignant pleural mesothelioma using conditional variational autoencoder on 3D PET images and clinical data

BACKGROUND

Deep learning (DL) has been widely used for diagnosis and prognosis prediction of numerous frequently occurring diseases. Generally, DL models require large datasets to perform accurate and reliable prognosis prediction and avoid overlearning. However, prognosis prediction of rare diseases is still limited owing to the small number of cases, resulting in small datasets.

PURPOSE

This paper proposes a multimodal DL method to predict the prognosis of patients with malignant pleural mesothelioma (MPM) with a small number of 3D positron emission tomography-computed tomography (PET/CT) images and clinical data.

METHODS

A 3D convolutional conditional variational autoencoder (3D-CCVAE), which adds a 3D-convolutional layer and conditional VAE to process 3D images, was used for dimensionality reduction of PET images. We developed a two-step model that performs dimensionality reduction using the 3D-CCVAE, which is resistant to overlearning. In the first step, clinical data were input to condition the model and perform dimensionality reduction of PET images, resulting in more efficient dimension reduction. In the second step, a subset of the dimensionally reduced features and clinical data were combined to predict 1-year survival of patients using the random forest classifier. To demonstrate the usefulness of the 3D-CCVAE, we created a model without the conditional mechanism (3D-CVAE), one without the variational mechanism (3D-CCAE), and one without an autoencoder (without AE), and compared their prediction results. We used PET images and clinical data of 520 patients with histologically proven MPM. The data were randomly split in a 2:1 ratio (train : test) and three-fold cross-validation was performed. The models were trained on the training set and evaluated based on the test set results. The area under the receiver operating characteristic curve (AUC) for all models was calculated using their 1-year survival predictions, and the results were compared.

RESULTS

We obtained AUC values of 0.76 (95% confidence interval [CI], 0.72-0.80) for the 3D-CCVAE model, 0.72 (95% CI, 0.68-0.77) for the 3D-CVAE model, 0.70 (95% CI, 0.66-0.75) for the 3D-CCAE model, and 0.69 (95% CI 0.65-0.74) for the without AE model. The 3D-CCVAE model performed better than the other models (3D-CVAE, p = 0.039; 3D-CCAE, p = 0.0032; and without AE, p = 0.0011).

CONCLUSIONS

This study demonstrates the usefulness of the 3D-CCVAE in multimodal DL models learned using a small number of datasets. Additionally, it shows that dimensionality reduction via AE can be used to learn a DL model without increasing the overlearning risk. Moreover, the VAE mechanism can overcome the uncertainty of the model parameters that commonly occurs for small datasets, thereby eliminating the risk of overlearning. Additionally, more efficient dimensionality reduction of PET images can be performed by providing clinical data as conditions and ignoring clinical data-related features.

Computer Vision in Osteoporotic Vertebral Fracture Risk Prediction: A Systematic Review

Osteoporotic vertebral fractures (OVFs) are a significant health concern linked to increased morbidity, mortality, and diminished quality of life. Traditional OVF risk assessment tools like bone mineral density (BMD) only capture a fraction of the risk profile. Artificial intelligence, specifically computer vision, has revolutionized other fields of medicine through analysis of videos, histopathology slides and radiological scans. In this review, we provide an overview of computer vision algorithms and current computer vision models used in predicting OVF risk. We highlight the clinical applications, future directions and limitations of computer vision in OVF risk prediction.

Comparison of 2D, 2.5D, and 3D segmentation networks for maxillary sinuses and lesions in CBCT images

BACKGROUND

The purpose of this study was to compare the segmentation performances of the 2D, 2.5D, and 3D networks for maxillary sinuses (MSs) and lesions inside the maxillary sinus (MSL) with variations in sizes, shapes, and locations in cone beam CT (CBCT) images under the same constraint of memory capacity.

METHODS

The 2D, 2.5D, and 3D networks were compared comprehensively for the segmentation of the MS and MSL in CBCT images under the same constraint of memory capacity. MSLs were obtained by subtracting the prediction of the air region of the maxillary sinus (MSA) from that of the MS.

RESULTS

The 2.5D network showed the highest segmentation performances for the MS and MSA compared to the 2D and 3D networks. The performances of the Jaccard coefficient, Dice similarity coefficient, precision, and recall by the 2.5D network of U-net +  + reached 0.947, 0.973, 0.974, and 0.971 for the MS, respectively, and 0.787, 0.875, 0.897, and 0.858 for the MSL, respectively.

CONCLUSIONS

The 2.5D segmentation network demonstrated superior segmentation performance for various MSLs with an ensemble learning approach of combining the predictions from three orthogonal planes.

The Use of Artificial Intelligence for the Classification of Craniofacial Deformities

Positional cranial deformities are a common finding in toddlers, yet differentiation from craniosynostosis can be challenging. The aim of this study was to train convolutional neural networks (CNNs) to classify craniofacial deformities based on 2D images generated using photogrammetry as a radiation-free imaging technique. A total of 487 patients with photogrammetry scans were included in this retrospective cohort study: children with craniosynostosis (n = 227), positional deformities (n = 206), and healthy children (n = 54). Three two-dimensional images were extracted from each photogrammetry scan. The datasets were divided into training, validation, and test sets. During the training, fine-tuned ResNet-152s were utilized. The performance was quantified using tenfold cross-validation. For the detection of craniosynostosis, sensitivity was at 0.94 with a specificity of 0.85. Regarding the differentiation of the five existing classes (trigonocephaly, scaphocephaly, positional plagiocephaly left, positional plagiocephaly right, and healthy), sensitivity ranged from 0.45 (positional plagiocephaly left) to 0.95 (scaphocephaly) and specificity ranged from 0.87 (positional plagiocephaly right) to 0.97 (scaphocephaly). We present a CNN-based approach to classify craniofacial deformities on two-dimensional images with promising results. A larger dataset would be required to identify rarer forms of craniosynostosis as well. The chosen 2D approach enables future applications for digital cameras or smartphones.

Enhancing Fetal Anomaly Detection in Ultrasonography Images: A Review of Machine Learning-Based Approaches

Fetal development is a critical phase in prenatal care, demanding the timely identification of anomalies in ultrasound images to safeguard the well-being of both the unborn child and the mother. Medical imaging has played a pivotal role in detecting fetal abnormalities and malformations. However, despite significant advances in ultrasound technology, the accurate identification of irregularities in prenatal images continues to pose considerable challenges, often necessitating substantial time and expertise from medical professionals. In this review, we go through recent developments in machine learning (ML) methods applied to fetal ultrasound images. Specifically, we focus on a range of ML algorithms employed in the context of fetal ultrasound, encompassing tasks such as image classification, object recognition, and segmentation. We highlight how these innovative approaches can enhance ultrasound-based fetal anomaly detection and provide insights for future research and clinical implementations. Furthermore, we emphasize the need for further research in this domain where future investigations can contribute to more effective ultrasound-based fetal anomaly detection.

An interpretable machine learning approach to study the relationship beetwen retrognathia and skull anatomy

Mandibular retrognathia (C2Rm) is one of the most common oral pathologies. Acquiring a better understanding of the points of impact of C2Rm on the entire skull is of major interest in the diagnosis, treatment, and management of this dysmorphism, but also permits us to contribute to the debate on the changes undergone by the shape of the skull during human evolution. However, conventional methods have some limits in meeting these challenges, insofar as they require defining in advance the structures to be studied, and identifying them using landmarks. In this context, our work aims to answer these questions using AI tools and, in particular, machine learning, with the objective of relaying these treatments automatically. We propose an innovative methodology coupling convolutional neural networks (CNNs) and interpretability algorithms. Applied to a set of radiographs classified into physiological versus pathological categories, our methodology made it possible to: discuss the structures impacted by retrognathia and already identified in literature; identify new structures of potential interest in medical terms; highlight the dynamic evolution of impacted structures according to the level of gravity of C2Rm; provide for insights into the evolution of human anatomy. Results were discussed in terms of the major interest of this approach in the field of orthodontics and, more generally, in the field of automated processing of medical images.

Binding affinity predictions with hybrid quantum-classical convolutional neural networks

Central in drug design is the identification of biomolecules that uniquely and robustly bind to a target protein, while minimizing their interactions with others. Accordingly, precise binding affinity prediction, enabling the accurate selection of suitable candidates from an extensive pool of potential compounds, can greatly reduce the expenses associated to practical experimental protocols. In this respect, recent advances revealed that deep learning methods show superior performance compared to other traditional computational methods, especially with the advent of large datasets. These methods, however, are complex and very time-intensive, thus representing an important clear bottleneck for their development and practical application. In this context, the emerging realm of quantum machine learning holds promise for enhancing numerous classical machine learning algorithms. In this work, we take one step forward and present a hybrid quantum-classical convolutional neural network, which is able to reduce by 20% the complexity of the classical counterpart while still maintaining optimal performance in the predictions. Additionally, this results in a significant cost and time savings of up to 40% in the training stage, which means a substantial speed-up of the drug design process.

Stacked ensemble deep learning for pancreas cancer classification using extreme gradient boosting

Ensemble learning aims to improve prediction performance by combining several models or forecasts. However, how much and which ensemble learning techniques are useful in deep learning-based pipelines for pancreas computed tomography (CT) image classification is a challenge. Ensemble approaches are the most advanced solution to many machine learning problems. These techniques entail training multiple models and combining their predictions to improve the predictive performance of a single model. This article introduces the idea of Stacked Ensemble Deep Learning (SEDL), a pipeline for classifying pancreas CT medical images. The weak learners are Inception V3, VGG16, and ResNet34, and we employed a stacking ensemble. By combining the first-level predictions, an input train set for XGBoost, the ensemble model at the second level of prediction, is created. Extreme Gradient Boosting (XGBoost), employed as a strong learner, will make the final classification. Our findings showed that SEDL performed better, with a 98.8% ensemble accuracy, after some adjustments to the hyperparameters. The Cancer Imaging Archive (TCIA) public access dataset consists of 80 pancreas CT scans with a resolution of 512 * 512 pixels, from 53 male and 27 female subjects. A sample of two hundred and twenty-two images was used for training and testing data. We concluded that implementing the SEDL technique is an effective way to strengthen the robustness and increase the performance of the pipeline for classifying pancreas CT medical images. Interestingly, grouping like-minded or talented learners does not make a difference.

Deep Learning-Based Feature Extraction from Whole-Body PET/CT Employing Maximum Intensity Projection Images: Preliminary Results of Lung Cancer Data

Purpose

Deep learning (DL) has been widely used in various medical imaging analyses. Because of the difficulty in processing volume data, it is difficult to train a DL model as an end-to-end approach using PET volume as an input for various purposes including diagnostic classification. We suggest an approach employing two maximum intensity projection (MIP) images generated by whole-body FDG PET volume to employ pre-trained models based on 2-D images.

Methods

As a retrospective, proof-of-concept study, 562 [18F]FDG PET/CT images and clinicopathological factors of lung cancer patients were collected. MIP images of anterior and lateral views were used as inputs, and image features were extracted by a pre-trained convolutional neural network (CNN) model, ResNet-50. The relationship between the images was depicted on a parametric 2-D axes map using t-distributed stochastic neighborhood embedding (t-SNE), with clinicopathological factors.

Results

A DL-based feature map extracted by two MIP images was embedded by t-SNE. According to the visualization of the t-SNE map, PET images were clustered by clinicopathological features. The representative difference between the clusters of PET patterns according to the posture of a patient was visually identified. This map showed a pattern of clustering according to various clinicopathological factors including sex as well as tumor staging.

Conclusion

A 2-D image-based pre-trained model could extract image patterns of whole-body FDG PET volume by using anterior and lateral views of MIP images bypassing the direct use of 3-D PET volume that requires large datasets and resources. We suggest that this approach could be implemented as a backbone model for various applications for whole-body PET image analyses.

Using deep learning for an automatic detection and classification of the vascular bifurcations along the Circle of Willis

Most of the intracranial aneurysms (ICA) occur on a specific portion of the cerebral vascular tree named the Circle of Willis (CoW). More particularly, they mainly arise onto fifteen of the major arterial bifurcations constituting this circular structure. Hence, for an efficient and timely diagnosis it is critical to develop some methods being able to accurately recognize each Bifurcation of Interest (BoI). Indeed, an automatic extraction of the bifurcations presenting the higher risk of developing an ICA would offer the neuroradiologists a quick glance at the most alarming areas. Due to the recent efforts on Artificial Intelligence, Deep Learning turned out to be the best performing technology for many pattern recognition tasks. Moreover, various methods have been particularly designed for medical image analysis purposes. This study intends to assist the neuroradiologists to promptly locate any bifurcation presenting a high risk of ICA occurrence. It can be seen as a Computer Aided Diagnosis scheme, where the Artificial Intelligence facilitates the access to the regions of interest within the MRI. In this work, we propose a method for a fully automatic detection and recognition of the bifurcations of interest forming the Circle of Willis. Several neural networks architectures have been tested, and we thoroughly evaluate the bifurcation recognition rate.

Automated Computer-Aided Detection and Classification of Intracranial Hemorrhage Using Ensemble Deep Learning Techniques

Diagnosing Intracranial Hemorrhage (ICH) at an early stage is difficult since it affects the blood vessels in the brain, often resulting in death. We propose an ensemble of Convolutional Neural Networks (CNNs) combining Squeeze and Excitation-based Residual Networks with the next dimension (SE-ResNeXT) and Long Short-Term Memory (LSTM) Networks in order to address this issue. This research work primarily used data from the Radiological Society of North America (RSNA) brain CT hemorrhage challenge dataset and the CQ500 dataset. Preprocessing and data augmentation are performed using the windowing technique in the proposed work. The ICH is then classified using ensembled CNN techniques after being preprocessed, followed by feature extraction in an automatic manner. ICH is classified into the following five types: epidural, intraventricular, subarachnoid, intra-parenchymal, and subdural. A gradient-weighted Class Activation Mapping method (Grad-CAM) is used for identifying the region of interest in an ICH image. A number of performance measures are used to compare the experimental results with various state-of-the-art algorithms. By achieving 99.79% accuracy with an F-score of 0.97, the proposed model proved its efficacy in detecting ICH compared to other deep learning models. The proposed ensembled model can classify epidural, intraventricular, subarachnoid, intra-parenchymal, and subdural hemorrhages with an accuracy of 99.89%, 99.65%, 98%, 99.75%, and 99.88%. Simulation results indicate that the suggested approach can categorize a variety of intracranial bleeding types. By implementing the ensemble deep learning technique using the SE-ResNeXT and LSTM models, we achieved significant classification accuracy and AUC scores.

An appraisal of the performance of AI tools for chronic stroke lesion segmentation

Automated demarcation of stoke lesions from monospectral magnetic resonance imaging scans is extremely useful for diverse research and clinical applications, including lesion-symptom mapping to explain deficits and predict recovery. There is a significant surge of interest in the development of supervised artificial intelligence (AI) methods for that purpose, including deep learning, with a performance comparable to trained experts. Such AI-based methods, however, require copious amounts of data. Thanks to the availability of large datasets, the development of AI-based methods for lesion segmentation has immensely accelerated in the last decade. One of these datasets is the Anatomical Tracings of Lesions After Stroke (ATLAS) dataset which includes T1-weighted images from hundreds of chronic stroke survivors with their manually traced lesions. This systematic review offers an appraisal of the impact of the ATLAS dataset in promoting the development of AI-based segmentation of stroke lesions. An examination of all published studies, that used the ATLAS dataset to both train and test their methods, highlighted an overall moderate performance (median Dice index = 59.40%) and a huge variability across studies in terms of data preprocessing, data augmentation, AI architecture, and the mode of operation (two-dimensional versus three-dimensional methods). Perhaps most importantly, almost all AI tools were borrowed from existing AI architectures in computer vision, as 90% of all selected studies relied on conventional convolutional neural network-based architectures. Overall, current research has not led to the development of robust AI architectures than can handle spatially heterogenous lesion patterns. This review also highlights the difficulty of gauging the performance of AI tools in the presence of uncertainties in the definition of the ground truth.

Use of Artificial Intelligence in Coronary Artery Calcium Scoring

Coronary artery calcium (CAC) scoring improves traditional risk factor-based coronary heart disease (CHD) risk stratification. Here, the contribution of CAC scoring to a traditional 10-year CHD risk prediction scores and new artificial intelligence methods used to automate CAC scoring were reviewed. Research shows that traditional risk factors tend to overestimate or underestimate the actual risk of CHD, meaning that including CAC score in the risk stratification has potential to reduce over- and undertreatment. The automated CAC scoring methods are shown to be accurate and significantly more time-effective than the commonly used semi-automated method.

Surface Defect Detection of Bearing Rings Based on an Improved YOLOv5 Network

Considering the characteristics of complex texture backgrounds, uneven brightness, varying defect sizes, and multiple defect types of the bearing surface images, a surface defect detection method for bearing rings is proposed based on improved YOLOv5. First, replacing the C3 module in the backbone network with a C2f module can effectively reduce the number of network parameters and computational complexity, thereby improving the speed and accuracy of the backbone network. Second, adding the SPD module into the backbone and neck networks enhances their ability to process low-resolution and small-object images. Next, replacing the nearest-neighbor upsampling with the lightweight and universal CARAFE operator fully utilizes feature semantic information, enriches contextual information, and reduces information loss during transmission, thereby effectively improving the model's diversity and robustness. Finally, we constructed a dataset of bearing ring surface images collected from industrial sites and conducted numerous experiments based on this dataset. Experimental results show that the mean average precision (mAP) of the network is 97.3%, especially for dents and black spot defects, improved by 2.2% and 3.9%, respectively, and that the detection speed can reach 100 frames per second (FPS). Compared with mainstream surface defect detection algorithms, the proposed method shows significant improvements in both accuracy and detection time and can meet the requirements of industrial defect detection.

Predicting glioblastoma molecular subtypes and prognosis with a multimodal model integrating convolutional neural network, radiomics, and semantics

OBJECTIVE

The aim of this study was to build a convolutional neural network (CNN)-based prediction model of glioblastoma (GBM) molecular subtype diagnosis and prognosis with multimodal features.

METHODS

In total, 222 GBM patients were included in the training set from Sun Yat-sen University Cancer Center (SYSUCC) and 107 GBM patients were included in the validation set from SYSUCC, Xuanwu Hospital Capital Medical University, and the First Hospital of Jilin University. The multimodal model was trained with MR images (pre- and postcontrast T1-weighted images and T2-weighted images), corresponding MRI impression, and clinical patient information. First, the original images were segmented using the Multimodal Brain Tumor Image Segmentation Benchmark toolkit. Convolutional features were extracted using 3D residual deep neural network (ResNet50) and convolutional 3D (C3D). Radiomic features were extracted using pyradiomics. Report texts were converted to word embedding using word2vec. These three types of features were then integrated to train neural networks. Accuracy, precision, recall, and F1-score were used to evaluate the model performance.

RESULTS

The C3D-based model yielded the highest accuracy of 91.11% in the prediction of IDH1 mutation status. Importantly, the addition of semantics improved precision by 11.21% and recall in MGMT promoter methylation status prediction by 14.28%. The areas under the receiver operating characteristic curves of the C3D-based model in the IDH1, ATRX, MGMT, and 1-year prognosis groups were 0.976, 0.953, 0.955, and 0.976, respectively. In external validation, the C3D-based model showed significant improvement in accuracy in the IDH1, ATRX, MGMT, and 1-year prognosis groups, which were 88.30%, 76.67%, 85.71%, and 85.71%, respectively (compared with 3D ResNet50: 83.51%, 66.67%, 82.14%, and 70.79%, respectively).

CONCLUSIONS

The authors propose a novel multimodal model integrating C3D, radiomics, and semantics, which had a great performance in predicting IDH1, ATRX, and MGMT molecular subtypes and the 1-year prognosis of GBM.