Deep learning model calibration for improving performance in class-imbalanced medical image classification tasks

Advancing Glaucoma Diagnosis: Employing Confidence-Calibrated Label Smoothing Loss for Model Calibration

Objective

The aim of our research is to enhance the calibration of machine learning models for glaucoma classification through a specialized loss function named Confidence-Calibrated Label Smoothing (CC-LS) loss. This approach is specifically designed to refine model calibration without compromising accuracy by integrating label smoothing and confidence penalty techniques, tailored to the specifics of glaucoma detection.

Design

This study focuses on the development and evaluation of a calibrated deep learning model.

Participants

The study employs fundus images from both external datasets-the Online Retinal Fundus Image Database for Glaucoma Analysis and Research (482 normal, 168 glaucoma) and the Retinal Fundus Glaucoma Challenge (720 normal, 80 glaucoma)-and an extensive internal dataset (4639 images per category), aiming to bolster the model's generalizability. The model's clinical performance is validated using a comprehensive test set (47 913 normal, 1629 glaucoma) from the internal dataset.

Methods

The CC-LS loss function seamlessly integrates label smoothing, which tempers extreme predictions to avoid overfitting, with confidence-based penalties. These penalties deter the model from expressing undue confidence in incorrect classifications. Our study aims at training models using the CC-LS and comparing their performance with those trained using conventional loss functions.

Main Outcome Measures

The model's precision is evaluated using metrics like the Brier score, sensitivity, specificity, and the false positive rate, alongside qualitative heatmap analyses for a holistic accuracy assessment.

Results

Preliminary findings reveal that models employing the CC-LS mechanism exhibit superior calibration metrics, as evidenced by a Brier score of 0.098, along with notable accuracy measures: sensitivity of 81%, specificity of 80%, and weighted accuracy of 80%. Importantly, these enhancements in calibration are achieved without sacrificing classification accuracy.

Conclusions

The CC-LS loss function presents a significant advancement in the pursuit of deploying machine learning models for glaucoma diagnosis. By improving calibration, the CC-LS ensures that clinicians can interpret and trust the predictive probabilities, making artificial intelligence-driven diagnostic tools more clinically viable. From a clinical standpoint, this heightened trust and interpretability can potentially lead to more timely and appropriate interventions, thereby optimizing patient outcomes and safety.

Financial Disclosures

Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.

Deep learning model based on endoscopic images predicting treatment response in locally advanced rectal cancer undergo neoadjuvant chemoradiotherapy: a multicenter study

PURPOSE

Neoadjuvant chemoradiotherapy has been the standard practice for patients with locally advanced rectal cancer. However, the treatment response varies greatly among individuals, how to select the optimal candidates for neoadjuvant chemoradiotherapy is crucial. This study aimed to develop an endoscopic image-based deep learning model for predicting the response to neoadjuvant chemoradiotherapy in locally advanced rectal cancer.

METHODS

In this multicenter observational study, pre-treatment endoscopic images of patients from two Chinese medical centers were retrospectively obtained and a deep learning-based tumor regression model was constructed. Treatment response was evaluated based on the tumor regression grade and was defined as good response and non-good response. The prediction performance of the deep learning model was evaluated in the internal and external test sets. The main outcome was the accuracy of the treatment prediction model, measured by the AUC and accuracy.

RESULTS

This deep learning model achieved favorable prediction performance. In the internal test set, the AUC and accuracy were 0.867 (95% CI: 0.847-0.941) and 0.836 (95% CI: 0.818-0.896), respectively. The prediction performance was fully validated in the external test set, and the model had an AUC of 0.758 (95% CI: 0.724-0.834) and an accuracy of 0.807 (95% CI: 0.774-0.843).

CONCLUSION

The deep learning model based on endoscopic images demonstrated exceptional predictive power for neoadjuvant treatment response, highlighting its potential for guiding personalized therapy.

ELRL-MD: a deep learning approach for myocarditis diagnosis using cardiac magnetic resonance images with ensemble and reinforcement learning integration

Objective.Myocarditis poses a significant health risk, often precipitated by viral infections like coronavirus disease, and can lead to fatal cardiac complications. As a less invasive alternative to the standard diagnostic practice of endomyocardial biopsy, which is highly invasive and thus limited to severe cases, cardiac magnetic resonance (CMR) imaging offers a promising solution for detecting myocardial abnormalities.Approach.This study introduces a deep model called ELRL-MD that combines ensemble learning and reinforcement learning (RL) for effective myocarditis diagnosis from CMR images. The model begins with pre-training via the artificial bee colony (ABC) algorithm to enhance the starting point for learning. An array of convolutional neural networks (CNNs) then works in concert to extract and integrate features from CMR images for accurate diagnosis. Leveraging the Z-Alizadeh Sani myocarditis CMR dataset, the model employs RL to navigate the dataset's imbalance by conceptualizing diagnosis as a decision-making process.Main results.ELRL-DM demonstrates remarkable efficacy, surpassing other deep learning, conventional machine learning, and transfer learning models, achieving an F-measure of 88.2% and a geometric mean of 90.6%. Extensive experimentation helped pinpoint the optimal reward function settings and the perfect count of CNNs.Significance.The study addresses the primary technical challenge of inherent data imbalance in CMR imaging datasets and the risk of models converging on local optima due to suboptimal initial weight settings. Further analysis, leaving out ABC and RL components, confirmed their contributions to the model's overall performance, underscoring the effectiveness of addressing these critical technical challenges.

Mpox-AISM: AI-mediated super monitoring for mpox and like-mpox

Swift and accurate diagnosis for earlier-stage monkeypox (mpox) patients is crucial to avoiding its spread. However, the similarities between common skin disorders and mpox and the need for professional diagnosis unavoidably impaired the diagnosis of earlier-stage mpox patients and contributed to mpox outbreak. To address the challenge, we proposed "Super Monitoring", a real-time visualization technique employing artificial intelligence (AI) and Internet technology to diagnose earlier-stage mpox cheaply, conveniently, and quickly. Concretely, AI-mediated "Super Monitoring" (mpox-AISM) integrates deep learning models, data augmentation, self-supervised learning, and cloud services. According to publicly accessible datasets, mpox-AISM's Precision, Recall, Specificity, and F1-score in diagnosing mpox reach 99.3%, 94.1%, 99.9%, and 96.6%, respectively, and it achieves 94.51% accuracy in diagnosing mpox, six like-mpox skin disorders, and normal skin. With the Internet and communication terminal, mpox-AISM has the potential to perform real-time and accurate diagnosis for earlier-stage mpox in real-world scenarios, thereby preventing mpox outbreak.

Empirical data drift detection experiments on real-world medical imaging data

While it is common to monitor deployed clinical artificial intelligence (AI) models for performance degradation, it is less common for the input data to be monitored for data drift - systemic changes to input distributions. However, when real-time evaluation may not be practical (eg., labeling costs) or when gold-labels are automatically generated, we argue that tracking data drift becomes a vital addition for AI deployments. In this work, we perform empirical experiments on real-world medical imaging to evaluate three data drift detection methods' ability to detect data drift caused (a) naturally (emergence of COVID-19 in X-rays) and (b) synthetically. We find that monitoring performance alone is not a good proxy for detecting data drift and that drift-detection heavily depends on sample size and patient features. Our work discusses the need and utility of data drift detection in various scenarios and highlights gaps in knowledge for the practical application of existing methods.

AggBERT: Best in Class Prediction of Hexapeptide Amyloidogenesis with a Semi-Supervised ProtBERT Model

The prediction of peptide amyloidogenesis is a challenging problem in the field of protein folding. Large language models, such as the ProtBERT model, have recently emerged as powerful tools in analyzing protein sequences for applications, such as predicting protein structure and function. In this article, we describe the use of a semisupervised and fine-tuned ProtBERT model to predict peptide amyloidogenesis from sequences alone. Our approach, which we call AggBERT, achieved state-of-the-art performance, demonstrating the potential for large language models to improve the accuracy and speed of amyloid fibril prediction over simple heuristics or structure-based approaches. This work highlights the transformative potential of machine learning and large language models in the fields of chemical biology and biomedicine.

A comparative study of model-centric and data-centric approaches in the development of cardiovascular disease risk prediction models in the UK Biobank

Aims

A diverse set of factors influence cardiovascular diseases (CVDs), but a systematic investigation of the interplay between these determinants and the contribution of each to CVD incidence prediction is largely missing from the literature. In this study, we leverage one of the most comprehensive biobanks worldwide, the UK Biobank, to investigate the contribution of different risk factor categories to more accurate incidence predictions in the overall population, by sex, different age groups, and ethnicity.

Methods and results

The investigated categories include the history of medical events, behavioural factors, socioeconomic factors, environmental factors, and measurements. We included data from a cohort of 405 257 participants aged 37-73 years and trained various machine learning and deep learning models on different subsets of risk factors to predict CVD incidence. Each of the models was trained on the complete set of predictors and subsets where each category was excluded. The results were benchmarked against QRISK3. The findings highlight that (i) leveraging a more comprehensive medical history substantially improves model performance. Relative to QRISK3, the best performing models improved the discrimination by 3.78% and improved precision by 1.80%. (ii) Both model- and data-centric approaches are necessary to improve predictive performance. The benefits of using a comprehensive history of diseases were far more pronounced when a neural sequence model, BEHRT, was used. This highlights the importance of the temporality of medical events that existing clinical risk models fail to capture. (iii) Besides the history of diseases, socioeconomic factors and measurements had small but significant independent contributions to the predictive performance.

Conclusion

These findings emphasize the need for considering broad determinants and novel modelling approaches to enhance CVD incidence prediction.

ExtRanFS: An Automated Lung Cancer Malignancy Detection System Using Extremely Randomized Feature Selector

Lung cancer is an abnormality where the body's cells multiply uncontrollably. The disease can be deadly if not detected in the initial stage. To address this issue, an automated lung cancer malignancy detection (ExtRanFS) framework is developed using transfer learning. We used the IQ-OTH/NCCD dataset gathered from the Iraq Hospital in 2019, encompassing CT scans of patients suffering from various lung cancers and healthy subjects. The annotated dataset consists of CT slices from 110 patients, of which 40 were diagnosed with malignant tumors and 15 with benign tumors. Fifty-five patients were determined to be in good health. All CT images are in DICOM format with a 1mm slice thickness, consisting of 80 to 200 slices at various sides and angles. The proposed system utilized a convolution-based pre-trained VGG16 model as the feature extractor and an Extremely Randomized Tree Classifier as the feature selector. The selected features are fed to the Multi-Layer Perceptron (MLP) Classifier for detecting whether the lung cancer is benign, malignant, or normal. The accuracy, sensitivity, and F1-Score of the proposed framework are 99.09%, 98.33%, and 98.33%, respectively. To evaluate the proposed model, a comparison is performed with other pre-trained models as feature extractors and also with the existing state-of-the-art methodologies as classifiers. From the experimental results, it is evident that the proposed framework outperformed other existing methodologies. This work would be beneficial to both the practitioners and the patients in identifying whether the tumor is benign, malignant, or normal.

Automatic classification of the vertebral endplate lesions in magnetic resonance imaging by deep learning model

Introduction

A novel classification scheme for endplate lesions, based on T2-weighted images from magnetic resonance imaging (MRI) scan, has been recently introduced and validated. The scheme categorizes intervertebral spaces as "normal," "wavy/irregular," "notched," and "Schmorl's node." These lesions have been associated with spinal pathologies, including disc degeneration and low back pain. The exploitation of an automatic tool for the detection of the lesions would facilitate clinical practice by reducing the workload and the diagnosis time. The present work exploits a deep learning application based on convolutional neural networks to automatically classify the type of lesion.

Methods

T2-weighted MRI scans of the sagittal lumbosacral spine of consecutive patients were retrospectively collected. The middle slice of each scan was manually processed to identify the intervertebral spaces from L1L2 to L5S1, and the corresponding lesion type was labeled. A total of 1,559 gradable discs were obtained, with the following types of distribution: "normal" (567 discs), "wavy/irregular" (485), "notched" (362), and "Schmorl's node" (145). The dataset was divided randomly into a training set and a validation set while preserving the original distribution of lesion types in each set. A pretrained network for image classification was utilized, and fine-tuning was performed using the training set. The retrained net was then applied to the validation set to evaluate the overall accuracy and accuracy for each specific lesion type.

Results

The overall rate of accuracy was found equal to 88%. The accuracy for the specific lesion type was found as follows: 91% (normal), 82% (wavy/irregular), 93% (notched), and 83% (Schmorl's node).

Discussion

The results indicate that the deep learning approach achieved high accuracy for both overall classification and individual lesion types. In clinical applications, this implementation could be employed as part of an automatic detection tool for pathological conditions characterized by the presence of endplate lesions, such as spinal osteochondrosis.

A Strategic Research Framework for Defeating Diabetes in India: A 21st-Century Agenda

Indian people are at high risk for type 2 diabetes (T2DM) even at younger ages and lower body weights. Already 74 million people in India have the disease, and the proportion of those with T2DM is increasing across all strata of society. Unique aspects, related to lower insulin secretion or function, and higher hepatic fat deposition, accompanied by the rise in overweight (related to lifestyle changes) may all be responsible for this unrelenting epidemic of T2DM. Yet, research to understand the causes, pathophysiology, phenotypes, prevention, treatment, and healthcare delivery of T2DM in India seriously lags behind. There are major opportunities for scientific discovery and technological innovation, which if tapped can generate solutions for T2DM relevant to the country's context and make leading contributions to global science. We analyze the situation of T2DM in India, and present a four-pillar (etiology, precision medicine, implementation research, and health policy) strategic research framework to tackle the challenge. We offer key research questions for each pillar, and identify infrastructure needs. India offers a fertile environment for shifting the paradigm from imprecise late-stage diabetes treatment toward early-stage precision prevention and care. Investing in and leveraging academic and technological infrastructures, across the disciplines of science, engineering, and medicine, can accelerate progress toward a diabetes-free nation.

Deep-learning-based prognostic modeling for incident heart failure in patients with diabetes using electronic health records: A retrospective cohort study

Patients with type 2 diabetes mellitus (T2DM) have more than twice the risk of developing heart failure (HF) compared to patients without diabetes. The present study is aimed to build an artificial intelligence (AI) prognostic model that takes in account a large and heterogeneous set of clinical factors and investigates the risk of developing HF in diabetic patients. We carried out an electronic health records- (EHR-) based retrospective cohort study that included patients with cardiological clinical evaluation and no previous diagnosis of HF. Information consists of features extracted from clinical and administrative data obtained as part of routine medical care. The primary endpoint was diagnosis of HF (during out-of-hospital clinical examination or hospitalization). We developed two prognostic models using (1) elastic net regularization for Cox proportional hazard model (COX) and (2) a deep neural network survival method (PHNN), in which a neural network was used to represent a non-linear hazard function and explainability strategies are applied to estimate the influence of predictors on the risk function. Over a median follow-up of 65 months, 17.3% of the 10,614 patients developed HF. The PHNN model outperformed COX both in terms of discrimination (c-index 0.768 vs 0.734) and calibration (2-year integrated calibration index 0.008 vs 0.018). The AI approach led to the identification of 20 predictors of different domains (age, body mass index, echocardiographic and electrocardiographic features, laboratory measurements, comorbidities, therapies) whose relationship with the predicted risk correspond to known trends in the clinical practice. Our results suggest that prognostic models for HF in diabetic patients may improve using EHRs in combination with AI techniques for survival analysis, which provide high flexibility and better performance with respect to standard approaches.

Deep-learning model associating lateral cervical radiographic features with Cormack-Lehane grade 3 or 4 glottic view

Unanticipated difficult laryngoscopy is associated with serious airway-related complications. We aimed to develop and test a convolutional neural network-based deep-learning model that uses lateral cervical spine radiographs to predict Cormack-Lehane grade 3 or 4 direct laryngoscopy views of the glottis. We analysed the radiographs of 5939 thyroid surgery patients at our hospital, 253 (4%) of whom had grade 3 or 4 glottic views. We used 10 randomly sampled datasets to train a model. We compared the new model with six similar models (VGG, ResNet, Xception, ResNext, DenseNet and SENet). The Brier score (95%CI) of the new model, 0.023 (0.021-0.025), was lower ('better') than the other models: VGG, 0.034 (0.034-0.035); ResNet, 0.033 (0.033-0.035); Xception, 0.032 (0.031-0.033); ResNext, 0.033 (0.032-0.033); DenseNet, 0.030 (0.029-0.032); SENet, 0.031 (0.029-0.032), all p < 0.001. We calculated mean (95%CI) of the new model for: R² , 0.428 (0.388-0.468); mean squared error, 0.023 (0.021-0.025); mean absolute error, 0.048 (0.046-0.049); balanced accuracy, 0.713 (0.684-0.742); and area under the receiver operating characteristic curve, 0.965 (0.962-0.969). Radiographic features around the hyoid bone, pharynx and cervical spine were associated with grade 3 and 4 glottic views.

Bias and Class Imbalance in Oncologic Data-Towards Inclusive and Transferrable AI in Large Scale Oncology Data Sets

Recent technological developments have led to an increase in the size and types of data in the medical field derived from multiple platforms such as proteomic, genomic, imaging, and clinical data. Many machine learning models have been developed to support precision/personalized medicine initiatives such as computer-aided detection, diagnosis, prognosis, and treatment planning by using large-scale medical data. Bias and class imbalance represent two of the most pressing challenges for machine learning-based problems, particularly in medical (e.g., oncologic) data sets, due to the limitations in patient numbers, cost, privacy, and security of data sharing, and the complexity of generated data. Depending on the data set and the research question, the methods applied to address class imbalance problems can provide more effective, successful, and meaningful results. This review discusses the essential strategies for addressing and mitigating the class imbalance problems for different medical data types in the oncologic domain.

Performance Analysis of State-of-the-Art CNN Architectures for LUNA16

The convolutional neural network (CNN) has become a powerful tool in machine learning (ML) that is used to solve complex problems such as image recognition, natural language processing, and video analysis. Notably, the idea of exploring convolutional neural network architecture has gained substantial attention as well as popularity. This study focuses on the intrinsic various CNN architectures: LeNet, AlexNet, VGG16, ResNet-50, and Inception-V1, which have been scrutinized and compared with each other for the detection of lung cancer using publicly available LUNA16 datasets. Furthermore, multiple performance optimizers: root mean square propagation (RMSProp), adaptive moment estimation (Adam), and stochastic gradient descent (SGD), were applied for this comparative study. The performances of the three CNN architectures were measured for accuracy, specificity, sensitivity, positive predictive value, false omission rate, negative predictive value, and F1 score. The experimental results showed that the CNN AlexNet architecture with the SGD optimizer achieved the highest validation accuracy for CT lung cancer with an accuracy of 97.42%, misclassification rate of 2.58%, 97.58% sensitivity, 97.25% specificity, 97.58% positive predictive value, 97.25% negative predictive value, false omission rate of 2.75%, and F1 score of 97.58%. AlexNet with the SGD optimizer was the best and outperformed compared to the other state-of-the-art CNN architectures.