Radiologists with and without deep learning-based computer-aided diagnosis: comparison of performance and interobserver agreement for characterizing and diagnosing pulmonary nodules/masses

External validation of the performance of commercially available deep-learning-based lung nodule detection on low-dose CT images for lung cancer screening in Japan

PURPOSE

Artificial intelligence (AI) algorithms for lung nodule detection have been developed to assist radiologists. However, external validation of its performance on low-dose CT (LDCT) images is insufficient. We examined the performance of the commercially available deep-learning-based lung nodule detection (DL-LND) using LDCT images at Japanese lung cancer screening (LCS).

MATERIALS AND METHODS

Included were 43 patients with suspected lung cancer on LDCT images and pathologically confirmed lung cancer. The reference standard for nodules whose diameter exceeded 4 mm was set by a radiologist who referred to the reports of two other radiologists reading the LDCT images. After we applied commercially available DL-LND to the LDCT images, the radiologist reviewed all nodules detected by DL-LND. When he failed to identify an existing nodule, it was also included in the reference standard. To validate the performance of DL-LND, the sensitivity for lung nodules and lung cancer, the positive-predictive value (PPV) for lung nodules, and the mean number of false-positive (FP) nodules per CT scan were recorded.

RESULTS

The radiologist detected 97 nodules including 43 lung cancers and missed 3 solid nodules detected by DL-LND. A total of 100 nodules was included in the reference standard. DL-LND detected 396 nodules including 40 lung cancers. The sensitivity for the 100 nodules was 96.0%; the PPV was 24.2% (96/396). The mean number of FP nodules per CT scan was 7.0; sensitivity for lung cancer was 93.0% (40/43). DL-LND missed three lung cancers; 2 of these were atypical pulmonary cysts.

CONCLUSION

We externally verified that the sensitivity for lung nodules and lung cancer by DL-LND was very high. However, its low PPV and the increased FP nodules remains a serious drawback of DL-LND.

Ultra-low-dose chest computed tomography with model-based iterative reconstruction in the analysis of solid pulmonary nodules: A prospective study

BACKGROUND

Incidental pulmonary nodules are an increasingly common finding on computed tomography (CT) scans of the thorax due to the exponential rise in CT examinations in everyday practice. The majority of incidental pulmonary nodules are benign and correctly identifying the small number of malignant nodules is challenging. Ultra-low-dose CT (ULDCT) has been shown to be effective in diagnosis of respiratory pathology in comparison with traditional standard dose techniques. Our hypothesis was that ULDCT chest combined with model-based iterative reconstruction (MBIR) is comparable to standard dose CT (SDCT) chest in the analysis of pulmonary nodules with significant reduction in radiation dose.

AIM

To prospectively compare ULDCT chest combined with MBIR with SDCT chest in the analysis of solid pulmonary nodules.

METHODS

A prospective cohort study was conducted on adult patients (n = 30) attending a respiratory medicine outpatient clinic in a tertiary referral university hospital for surveillance of previously detected indeterminate pulmonary nodules on SDCT chest. This study involved the acquisition of a reference SDCT chest followed immediately by an ULDCT chest. Nodule identification, nodule characterisation, nodule measurement, objective and subjective image quality and radiation dose were compared between ULDCT with MBIR and SDCT chest.

RESULTS

One hundred solid nodules were detected on ULDCT chest and 98 on SDCT chest. There was no significant difference in the characteristics of correctly identified nodules when comparing SDCT chest to ULDCT chest protocols. Signal-to-noise ratio was significantly increased in the ULDCT chest in all areas except in the paraspinal muscle at the maximum cardiac diameter level (P < 0.001). The mean subjective image quality score for overall diagnostic acceptability was 8.9/10. The mean dose length product, computed tomography volume dose index and effective dose for the ULDCT chest protocol were 5.592 mGy.cm, 0.16 mGy and 0.08 mSv respectively. These were significantly less than the SDCT chest protocol (P < 0.001) and represent a radiation dose reduction of 97.6%.

CONCLUSION

ULDCT chest combined with MBIR is non-inferior to SDCT chest in the analysis of previously identified solid pulmonary nodules and facilitates a large reduction in radiation dose.

Automatic Segmentation and Radiomics for Identification and Activity Assessment of CTE Lesions in Crohn's Disease

BACKGROUND

The purpose of this article is to develop a deep learning automatic segmentation model for the segmentation of Crohn's disease (CD) lesions in computed tomography enterography (CTE) images. Additionally, the radiomics features extracted from the segmented CD lesions will be analyzed and multiple machine learning classifiers will be built to distinguish CD activity.

METHODS

This was a retrospective study with 2 sets of CTE image data. Segmentation datasets were used to establish nnU-Net neural network's automatic segmentation model. The classification dataset was processed using the automatic segmentation model to obtain segmentation results and extract radiomics features. The most optimal features were then selected to build 5 machine learning classifiers to distinguish CD activity. The performance of the automatic segmentation model was evaluated using the Dice similarity coefficient, while the performance of the machine learning classifier was evaluated using the area under the curve, sensitivity, specificity, and accuracy.

RESULTS

The segmentation dataset had 84 CTE examinations of CD patients (mean age 31 ± 13 years, 60 males), and the classification dataset had 193 (mean age 31 ± 12 years, 136 males). The deep learning segmentation model achieved a Dice similarity coefficient of 0.824 on the testing set. The logistic regression model showed the best performance among the 5 classifiers in the testing set, with an area under the curve, sensitivity, specificity, and accuracy of 0.862, 0.697, 0.840, and 0.759, respectively.

CONCLUSION

The automated segmentation model accurately segments CD lesions, and machine learning classifier distinguishes CD activity well. This method can assist radiologists in promptly and precisely evaluating CD activity.

A Self-supervised Learning-Based Fine-Grained Classification Model for Distinguishing Malignant From Benign Subcentimeter Solid Pulmonary Nodules

RATIONALE AND OBJECTIVES

Diagnosing subcentimeter solid pulmonary nodules (SSPNs) remains challenging in clinical practice. Deep learning may perform better than conventional methods in differentiating benign and malignant pulmonary nodules. This study aimed to develop and validate a model for differentiating malignant and benign SSPNs using CT images.

MATERIALS AND METHODS

This retrospective study included consecutive patients with SSPNs detected between January 2015 and October 2021 as an internal dataset. Malignancy was confirmed pathologically; benignity was confirmed pathologically or via follow-up evaluations. The SSPNs were segmented manually. A self-supervision pre-training-based fine-grained network was developed for predicting SSPN malignancy. The pre-trained model was established using data from the National Lung Screening Trial, Lung Nodule Analysis 2016, and a database of 5478 pulmonary nodules from the previous study, with subsequent fine-tuning using the internal dataset. The model's efficacy was investigated using an external cohort from another center, and its accuracy, sensitivity, specificity, and area under the receiver operating characteristic curve (AUC) were determined.

RESULTS

Overall, 1276 patients (mean age, 56 ± 10 years; 497 males) with 1389 SSPNs (mean diameter, 7.5 ± 2.0 mm; 625 benign) were enrolled. The internal dataset was specifically enriched for malignancy. The model's performance in the internal testing set (316 SSPNs) was: AUC, 0.964 (95% confidence interval (95%CI): 0.942-0.986); accuracy, 0.934; sensitivity, 0.965; and specificity, 0.908. The model's performance in the external test set (202 SSPNs) was: AUC, 0.945 (95% CI: 0.910-0.979); accuracy, 0.911; sensitivity, 0.977; and specificity, 0.860.

CONCLUSION

This deep learning model was robust and exhibited good performance in predicting the malignancy of SSPNs, which could help optimize patient management.

Prediction of benign and malignant ground glass pulmonary nodules based on multi-feature fusion of attention mechanism

Objectives

The purpose of this study was to develop and validate a new feature fusion algorithm to improve the classification performance of benign and malignant ground-glass nodules (GGNs) based on deep learning.

Methods

We retrospectively collected 385 cases of GGNs confirmed by surgical pathology from three hospitals. We utilized 239 GGNs from Hospital 1 as the training and internal validation set, and 115 and 31 GGNs from Hospital 2 and Hospital 3, respectively, as external test sets 1 and 2. Among these GGNs, 172 were benign and 203 were malignant. First, we evaluated clinical and morphological features of GGNs at baseline chest CT and simultaneously extracted whole-lung radiomics features. Then, deep convolutional neural networks (CNNs) and backpropagation neural networks (BPNNs) were applied to extract deep features from whole-lung CT images, clinical, morphological features, and whole-lung radiomics features separately. Finally, we integrated these four types of deep features using an attention mechanism. Multiple metrics were employed to evaluate the predictive performance of the model.

Results

The deep learning model integrating clinical, morphological, radiomics and whole lung CT image features with attention mechanism (CMRI-AM) achieved the best performance, with area under the curve (AUC) values of 0.941 (95% CI: 0.898-0.972), 0.861 (95% CI: 0.823-0.882), and 0.906 (95% CI: 0.878-0.932) on the internal validation set, external test set 1, and external test set 2, respectively. The AUC differences between the CMRI-AM model and other feature combination models were statistically significant in all three groups (all p<0.05).

Conclusion

Our experimental results demonstrated that (1) applying attention mechanism to fuse whole-lung CT images, radiomics features, clinical, and morphological features is feasible, (2) clinical, morphological, and radiomics features provide supplementary information for the classification of benign and malignant GGNs based on CT images, and (3) utilizing baseline whole-lung CT features to predict the benign and malignant of GGNs is an effective method. Therefore, optimizing the fusion of baseline whole-lung CT features can effectively improve the classification performance of GGNs.

Impact of artificial intelligence assistance on pulmonary nodule detection and localization in chest CT: a comparative study among radiologists of varying experience levels

The study aimed to evaluate the impact of AI assistance on pulmonary nodule detection rates among radiology residents and senior radiologists, along with assessing the effectiveness of two different commercialy available AI software systems in improving detection rates and LungRADS classification in chest CT. The study cohort included 198 participants with 221 pulmonary nodules. Residents' mean detection rate increased significantly from 64 to 77% with AI assist, while seniors' detection rate remained largely unchanged (85% vs. 86%). Residents showed significant improvement in segmental nodule localization with AI assistance, seniors did not. Software 2 slightly outperformed software 1 in increasing detection rates (67-77% vs. 80-86%), but neither significantly affected LungRADS classification. The study suggests that clinical experience mitigates the need for additional AI software, with the combination of CAD with residents being the most beneficial approach. Both software systems performed similarly, with software 2 showing a slightly higher but non-significant increase in detection rates.

Absolute ground truth-based validation of computer-aided nodule detection and volumetry in low-dose CT imaging

PURPOSE

To validate the performance of computer-aided detection (CAD) and volumetry software using an anthropomorphic phantom with a ground truth (GT) set of 3D-printed nodules.

METHODS

The Kyoto Kaguku Lungman phantom, containing 3D-printed solid nodules including six diameters (4 to 9 mm) and three morphologies (smooth, lobulated, spiculated), was scanned at varying CTDIvol levels (6.04, 1.54 and 0.20 mGy). Combinations of reconstruction algorithms (iterative and deep learning image reconstruction) and kernels (soft and hard) were applied. Detection, volumetry and density results recorded by a commercially available AI-based algorithm (AVIEW LCS + ) were compared to the absolute GT, which was determined through µCT scanning at 50 µm resolution. The associations between image acquisition parameters or nodule characteristics and accuracy of nodule detection and characterization were analyzed with chi square tests and multiple linear regression.

RESULTS

High levels of detection sensitivity and precision (minimal 83 % and 91 % respectively) were observed across all acquisitions. Neither reconstruction algorithm nor radiation dose showed significant associations with detection. Nodule diameter however showed a highly significant association with detection (p < 0.0001). Volumetric measurements for nodules > 6 mm were accurate within 10 % absolute range from volumeGT, regardless of dose and reconstruction. Nodule diameter and morphology are major determinants of volumetric accuracy (p < 0.001). Density assignment was not significantly influenced by any parameters.

CONCLUSIONS

Our study confirms the software's accurate performance in nodule volumetry, detection and density characterization with robustness for variations in CT imaging protocols. This study suggests the incorporation of similar phantom setups in quality assurance of CAD tools.

Diagnostic performance of deep learning models versus radiologists in COVID-19 pneumonia: A systematic review and meta-analysis

PURPOSE

Although several studies have compared the performance of deep learning (DL) models and radiologists for the diagnosis of COVID-19 pneumonia on CT of the chest, these results have not been collectively evaluated. We performed a meta-analysis of original articles comparing the performance of DL models versus radiologists in detecting COVID-19 pneumonia.

METHODS

A systematic search was conducted on the three main medical literature databases, Scopus, Web of Science, and PubMed, for articles published as of February 1st, 2023. We included original scientific articles that compared DL models trained to detect COVID-19 pneumonia on CT to radiologists. Meta-analysis was performed to determine DL versus radiologist performance in terms of model sensitivity and specificity, taking into account inter and intra-study heterogeneity.

RESULTS

Twenty-two articles met the inclusion criteria. Based on the meta-analytic calculations, DL models had significantly higher pooled sensitivity (0.933 vs. 0.829, p < 0.001) compared to radiologists with similar pooled specificity (0.905 vs. 0.897, p = 0.746). In the differentiation of COVID-19 versus community-acquired pneumonia, the DL models had significantly higher sensitivity compared to radiologists (0.915 vs. 0.836, p = 0.001).

CONCLUSIONS

DL models have high performance for screening of COVID-19 pneumonia on chest CT, offering the possibility of these models for augmenting radiologists in clinical practice.

Deep-learning for automated detection of MSU deposits on DECT: evaluating impact on efficiency and reader confidence

Introduction

Dual-energy CT (DECT) is a non-invasive way to determine the presence of monosodium urate (MSU) crystals in the workup of gout. Color-coding distinguishes MSU from calcium following material decomposition and post-processing. Manually identifying these foci (most commonly labeled green) is tedious, and an automated detection system could streamline the process. This study aims to evaluate the impact of a deep-learning (DL) algorithm developed for detecting green pixelations on DECT on reader time, accuracy, and confidence.

Methods

We collected a sample of positive and negative DECTs, reviewed twice-once with and once without the DL tool-with a 2-week washout period. An attending musculoskeletal radiologist and a fellow separately reviewed the cases, simulating clinical workflow. Metrics such as time taken, confidence in diagnosis, and the tool's helpfulness were recorded and statistically analyzed.

Results

We included thirty DECTs from different patients. The DL tool significantly reduced the reading time for the trainee radiologist (p = 0.02), but not for the attending radiologist (p = 0.15). Diagnostic confidence remained unchanged for both (p = 0.45). However, the DL model identified tiny MSU deposits that led to a change in diagnosis in two cases for the in-training radiologist and one case for the attending radiologist. In 3/3 of these cases, the diagnosis was correct when using DL.

Conclusions

The implementation of the developed DL model slightly reduced reading time for our less experienced reader and led to improved diagnostic accuracy. There was no statistically significant difference in diagnostic confidence when studies were interpreted without and with the DL model.

New trend in artificial intelligence-based assistive technology for thoracic imaging

Although there is no solid agreement for artificial intelligence (AI), it refers to a computer system with intelligence similar to that of humans. Deep learning appeared in 2006, and more than 10 years have passed since the third AI boom was triggered by improvements in computing power, algorithm development, and the use of big data. In recent years, the application and development of AI technology in the medical field have intensified internationally. There is no doubt that AI will be used in clinical practice to assist in diagnostic imaging in the future. In qualitative diagnosis, it is desirable to develop an explainable AI that at least represents the basis of the diagnostic process. However, it must be kept in mind that AI is a physician-assistant system, and the final decision should be made by the physician while understanding the limitations of AI. The aim of this article is to review the application of AI technology in diagnostic imaging from PubMed database while particularly focusing on diagnostic imaging in thorax such as lesion detection and qualitative diagnosis in order to help radiologists and clinicians to become more familiar with AI in thorax.

Development of a deep-learning model for classification of LI-RADS major features by using subtraction images of MRI: a preliminary study

PURPOSE

Liver Imaging Reporting and Data System (LI-RADS) is limited by interreader variability. Thus, our study aimed to develop a deep-learning model for classifying LI-RADS major features using subtraction images using magnetic resonance imaging (MRI).

METHODS

This single-center retrospective study included 222 consecutive patients who underwent resection for hepatocellular carcinoma (HCC) between January, 2015 and December, 2017. Subtraction arterial, portal venous, and transitional phase images of preoperative gadoxetic acid-enhanced MRI were used to train and test the deep-learning models. Initially, a three-dimensional (3D) nnU-Net-based deep-learning model was developed for HCC segmentation. Subsequently, a 3D U-Net-based deep-learning model was developed to assess three LI-RADS major features (nonrim arterial phase hyperenhancement [APHE], nonperipheral washout, and enhancing capsule [EC]), utilizing the results determined by board-certified radiologists as reference standards. The HCC segmentation performance was assessed using the Dice similarity coefficient (DSC), sensitivity, and precision. The sensitivity, specificity, and accuracy of the deep-learning model for classifying LI-RADS major features were calculated.

RESULTS

The average DSC, sensitivity, and precision of our model for HCC segmentation were 0.884, 0.891, and 0.887, respectively, across all the phases. Our model demonstrated a sensitivity, specificity, and accuracy of 96.6% (28/29), 66.7% (4/6), and 91.4% (32/35), respectively, for nonrim APHE; 95.0% (19/20), 50.0% (4/8), and 82.1% (23/28), respectively, for nonperipheral washout; and 86.7% (26/30), 54.2% (13/24), and 72.2% (39/54) for EC, respectively.

CONCLUSION

We developed an end-to-end deep-learning model that classifies the LI-RADS major features using subtraction MRI images. Our model exhibited satisfactory performance in classifying LI-RADS major features.

Systematic review of the performance evaluation of clinicians with or without the aid of machine learning clinical decision support system

Background

For the adoption of machine learning clinical decision support systems (ML-CDSS) it is critical to understand the performance aid of the ML-CDSS. However, it is not trivial, how the performance aid should be evaluated. To design reliable performance evaluation study, both the knowledge from the practical framework of experimental study design and the understanding of domain specific design factors are required.

Objective

The aim of this review study was to form a practical framework and identify key design factors for experimental design in evaluating the performance of clinicians with or without the aid of ML-CDSS.

Methods

The study was based on published ML-CDSS performance evaluation studies. We systematically searched articles published between January 2016 and December 2022. From the articles we collected a set of design factors. Only the articles comparing the performance of clinicians with or without the aid of ML-CDSS using experimental study methods were considered.

Results

The identified key design factors for the practical framework of ML-CDSS experimental study design were performance measures, user interface, ground truth data and the selection of samples and participants. In addition, we identified the importance of randomization, crossover design and training and practice rounds. Previous studies had shortcomings in the rationale and documentation of choices regarding the number of participants and the duration of the experiment.

Conclusion

The design factors of ML-CDSS experimental study are interdependent and all factors must be considered in individual choices.

Supplementary Information

The online version contains supplementary material available at 10.1007/s12553-023-00763-1.

Prediction of histologic types in solid lung lesions using preoperative contrast-enhanced CT

OBJECTIVES

This study aimed to develop and validate a predicting model for the histologic classification of solid lung lesions based on preoperative contrast-enhanced CT.

METHODS

A primary dataset of 1012 patients from Tianjin Medical University Cancer Institute and Hospital (TMUCIH) was randomly divided into a development cohort (708) and an internal validation cohort (304). Patients from the Second Hospital of Shanxi Medical University (SHSMU) were set as an external validation cohort (212). Two clinical factors (age, gender) and twenty-one characteristics on contrast-enhanced CT were used to construct a multinomial multivariable logistic regression model for the classification of seven common histologic types of solid lung lesions. The area under the receiver operating characteristic curve was used to assess the diagnostic performance of the model in the development and validation cohorts, separately.

RESULTS

Multivariable analysis showed that two clinical factors and twenty-one characteristics on contrast-enhanced CT were predictive in lung lesion histologic classification. The mean AUC of the proposed model for histologic classification was 0.95, 0.94, and 0.92 in the development, internal validation, and external validation cohort, respectively. When determining the malignancy of lung lesions based on histologic types, the mean AUC of the model was 0.88, 0.86, and 0.90 in three cohorts.

CONCLUSIONS

We demonstrated that by utilizing both clinical and CT characteristics on contrast-enhanced CT images, the proposed model could not only effectively stratify histologic types of solid lung lesions, but also enabled accurate assessment of lung lesion malignancy. Such a model has the potential to avoid unnecessary surgery for patients and to guide clinical decision-making for preoperative treatment.

KEY POINTS

• Clinical and CT characteristics on contrast-enhanced CT could be used to differentiate histologic types of solid lung lesions. • Predicting models using preoperative contrast-enhanced CT could accurately assessment of tumor malignancy based on predicted histologic types.