Putting artificial intelligence (AI) on the spot: machine learning evaluation of pulmonary nodules

Independent evaluation of the accuracy of 5 artificial intelligence software for detecting lung nodules on chest X-rays

Background

The integration of artificial intelligence (AI) into medicine is growing, with some experts predicting its standalone use soon. However, skepticism remains due to limited positive outcomes from independent validations. This research evaluates AI software's effectiveness in analyzing chest X-rays (CXR) to identify lung nodules, a possible lung cancer indicator.

Methods

This retrospective study analyzed 7,670,212 record pairs from radiological exams conducted between 2020 and 2022 during the Moscow Computer Vision Experiment, focusing on CXR and computed tomography (CT) scans. All images were acquired during clinical routine. The final dataset comprised 100 CXR images (50 with lung nodules, 50 without), selected consecutively and based on inclusion and exclusion criteria, to evaluate the performance of all five AI-based solutions, participating in the Moscow Computer Vision Experiment and analyzing CXR. The evaluation was performed in 3 stages. In the first stage, the probability of a nodule in the lung obtained from AI services was compared with the Ground Truth (1-there is a nodule, 0-there is no nodule). In the second stage, 3 radiologists evaluated the segmentation of nodules performed by the AI services (1-nodule correctly segmented, 0-nodule incorrectly segmented or not segmented at all). In the third stage, the same radiologists additionally evaluated the classification of the nodules (1-nodule correctly segmented and classified, 0-all other cases). The results obtained in stages 2 and 3 were compared with Ground Truth, which was common to all three stages. For each stage, diagnostic accuracy metrics were calculated for each AI service.

Results

Three software solutions (Celsus, Lunit INSIGHT CXR, and qXR) demonstrated diagnostic metrics that matched or surpassed the vendor specifications, and achieved the highest area under the receiver operating characteristic curve (AUC) of 0.956 [95% confidence interval (CI): 0.918 to 0.994]. However, when evaluated by three radiologists for accurate nodule segmentation and classification, all solutions performed below the vendor-declared metrics, with the highest AUC reaching 0.812 (95% CI: 0.744 to 0.879). Meanwhile, all AI services demonstrated 100% specificity at stages 2 and 3 of the study.

Conclusions

To ensure the reliability and applicability of AI-based software, it is crucial to validate performance metrics using high-quality datasets and engage radiologists in the evaluation process. Developers are recommended to improve the accuracy of the underlying models before allowing the standalone use of the software for lung nodule detection. The dataset created during the study may be accessed at https://mosmed.ai/datasets/mosmeddatargogksnalichiemiotsutstviemlegochnihuzlovtipvii/.

Imaging of Solid Pulmonary Nodules

Early detection with accurate classification of solid pulmonary nodules is critical in reducing lung cancer morbidity and mortality. Computed tomography (CT) remains the most widely used imaging examination for pulmonary nodule evaluation; however, other imaging modalities, such as PET/CT and MRI, are increasingly used for nodule characterization. Current advances in solid nodule imaging are largely due to developments in machine learning, including automated nodule segmentation and computer-aided detection. This review explores current multi-modality solid pulmonary nodule detection and characterization with discussion of radiomics and risk prediction models.

Revolutionizing Pulmonary Diagnostics: A Narrative Review of Artificial Intelligence Applications in Lung Imaging

Artificial intelligence (AI) has emerged as a transformative force in healthcare, particularly in pulmonary diagnostics. This comprehensive review explores the impact of AI on revolutionizing lung imaging, focusing on its applications in detecting abnormalities, diagnosing pulmonary conditions, and predicting disease prognosis. We provide an overview of traditional pulmonary diagnostic methods and highlight the importance of accurate and efficient lung imaging for early intervention and improved patient outcomes. Through the lens of AI, we examine machine learning algorithms, deep learning techniques, and natural language processing for analyzing radiology reports. Case studies and examples showcase the successful implementation of AI in pulmonary diagnostics, alongside challenges faced and lessons learned. Finally, we discuss future directions, including integrating AI into clinical workflows, ethical considerations, and the need for further research and collaboration in this rapidly evolving field. This review underscores the transformative potential of AI in enhancing the accuracy, efficiency, and accessibility of pulmonary healthcare.

Research on bronze wine vessel classification using improved SSA-CBAM-GNNs

This article proposes an advanced classification algorithm for bronze drinking utensils, taking into account the complexity of their cultural characteristics and the challenges of dynasty classification. The SSA-CBAM-GNNs algorithm integrates the Sparrow Search Algorithm (SSA), Spatial and Spectral Attention (CBAM) modules, and Graph Neural Networks (GNNs). The CBAM module is essential for optimizing feature extraction weights in graph neural networks, while SSA enhances the weighted network and expedites the convergence process. Experimental results, validated through various performance evaluation indicators, illustrate the outstanding performance of the improved SSA-CBAM-GNNs algorithm in accurately identifying and classifying cultural features of bronze drinking utensils. Comparative experiments confirm the algorithm's superiority over other methods. Overall, this study proposes a highly efficient identification and classification algorithm, and its effectiveness and excellence in extracting and identifying cultural features of bronze drinking utensils are experimentally demonstrated.

Clinical outcomes and actual consequence of lung nodules incidentally detected on chest radiographs by artificial intelligence

This study evaluated how often clinically significant lung nodules were detected unexpectedly on chest radiographs (CXR) by artificial intelligence (AI)-based detection software, and whether co-existing findings can aid in differential diagnosis of lung nodules. Patients (> 18 years old) with AI-detected lung nodules at their first visit from March 2021 to February 2022, except for those in the pulmonology or thoracic surgery departments, were retrospectively included. Three radiologists categorized nodules into malignancy, active inflammation, post-inflammatory sequelae, or "other" groups. Characteristics of the nodule and abnormality scores of co-existing lung lesions were compared. Approximately 1% of patients (152/14,563) had unexpected lung nodules. Among 73 patients with follow-up exams, 69.9% had true positive nodules. Increased abnormality scores for nodules were significantly associated with malignancy (odds ratio [OR] 1.076, P = 0.001). Increased abnormality scores for consolidation (OR 1.033, P = 0.040) and pleural effusion (OR 1.025, P = 0.041) were significantly correlated with active inflammation-type nodules. Abnormality scores for fibrosis (OR 1.036, P = 0.013) and nodules (OR 0.940, P = 0.001) were significantly associated with post-inflammatory sequelae categorization. AI-based lesion-detection software of CXRs in daily practice can help identify clinically significant incidental lung nodules, and referring accompanying lung lesions may help classify the nodule.

Pulmonary nodules segmentation based on domain adaptation

With the development of deep learning, the methods based on transfer learning have promoted the progress of medical image segmentation. However, the domain shift and complex background information of medical images limit the further improvement of the segmentation accuracy. Domain adaptation can compensate for the sample shortage by learning important information from a similar source dataset. Therefore, a segmentation method based on adversarial domain adaptation with background mask (ADAB) is proposed in this paper. Firstly, two ADAB networks are built for the source and target data segmentation, respectively. Next, to extract the foreground features that are the input of the discriminators, the background masks are generated according to the region growth algorithm. Then, to update the parameters in the target network without being affected by the conflict between the distinguishing differences of the discriminator and the domain shift reduction of the adversarial domain adaptation, a gradient reversal layer propagation is embedded in the ADAB model for the target data. Finally, an enhanced boundaries loss is deduced to make the target network sensitive to the edge of the area to be segmented. The performance of the proposed method is evaluated in the segmentation of pulmonary nodules in computed tomography images. Experimental results show that the proposed approach has a potential prospect in medical image processing.

The current status and future of FDA-approved artificial intelligence tools in chest radiology in the United States

Artificial intelligence (AI) is becoming more widespread within radiology. Capabilities that AI algorithms currently provide include detection, segmentation, classification, and quantification of pathological findings. Artificial intelligence software have created challenges for the traditional United States Food and Drug Administration (FDA) approval process for medical devices given their abilities to evolve over time with incremental data input. Currently, there are 190 FDA-approved radiology AI-based software devices, 42 of which pertain specifically to thoracic radiology. The majority of these algorithms are approved for the detection and/or analysis of pulmonary nodules, for monitoring placement of endotracheal tubes and indwelling catheters, for detection of emergent findings, and for assessment of pulmonary parenchyma; however, as technology evolves, there are many other potential applications that can be explored. For example, evaluation of non-idiopathic pulmonary fibrosis interstitial lung diseases, synthesis of imaging, clinical and/or laboratory data to yield comprehensive diagnoses, and survival or prognosis prediction of certain pathologies. With increasing physician and developer engagement, transparency and frequent communication between developers and regulatory agencies, such as the FDA, AI medical devices will be able to provide a critical supplement to patient management and ultimately enhance physicians' ability to improve patient care.

Research Progress of Respiratory Disease and Idiopathic Pulmonary Fibrosis Based on Artificial Intelligence

Machine Learning (ML) is an algorithm based on big data, which learns patterns from the previously observed data through classifying, predicting, and optimizing to accomplish specific tasks. In recent years, there has been rapid development in the field of ML in medicine, including lung imaging analysis, intensive medical monitoring, mechanical ventilation, and there is need for intubation etiology prediction evaluation, pulmonary function evaluation and prediction, obstructive sleep apnea, such as biological information monitoring and so on. ML can have good performance and is a great potential tool, especially in the imaging diagnosis of interstitial lung disease. Idiopathic pulmonary fibrosis (IPF) is a major problem in the treatment of respiratory diseases, due to the abnormal proliferation of fibroblasts, leading to lung tissue destruction. The diagnosis mainly depends on the early detection of imaging and early treatment, which can effectively prolong the life of patients. If the computer can be used to assist the examination results related to the effects of fibrosis, a timely diagnosis of such diseases will be of great value to both doctors and patients. We also previously proposed a machine learning algorithm model that can play a good clinical guiding role in early imaging prediction of idiopathic pulmonary fibrosis. At present, AI and machine learning have great potential and ability to transform many aspects of respiratory medicine and are the focus and hotspot of research. AI needs to become an invisible, seamless, and impartial auxiliary tool to help patients and doctors make better decisions in an efficient, effective, and acceptable way. The purpose of this paper is to review the current application of machine learning in various aspects of respiratory diseases, with the hope to provide some help and guidance for clinicians when applying algorithm models.

The Development of an Intelligent Agent to Detect and Non-Invasively Characterize Lung Lesions on CT Scans: Ready for the "Real World"?

(1) Background: Once lung lesions are identified on CT scans, they must be characterized by assessing the risk of malignancy. Despite the promising performance of computer-aided systems, some limitations related to the study design and technical issues undermine these tools' efficiency; an "intelligent agent" to detect and non-invasively characterize lung lesions on CT scans is proposed. (2) Methods: Two main modules tackled the detection of lung nodules on CT scans and the diagnosis of each nodule into benign and malignant categories. Computer-aided detection (CADe) and computer aided-diagnosis (CADx) modules relied on deep learning techniques such as Retina U-Net and the convolutional neural network; (3) Results: Tests were conducted on one publicly available dataset and two local datasets featuring CT scans acquired with different devices to reveal deep learning performances in "real-world" clinical scenarios. The CADe module reached an accuracy rate of 78%, while the CADx's accuracy, specificity, and sensitivity stand at 80%, 73%, and 85.7%, respectively; (4) Conclusions: Two different deep learning techniques have been adapted for CADe and CADx purposes in both publicly available and private CT scan datasets. Experiments have shown adequate performance in both detection and diagnosis tasks. Nevertheless, some drawbacks still characterize the supervised learning paradigm employed in networks such as CNN and Retina U-Net in real-world clinical scenarios, with CT scans from different devices with different sensors' fingerprints and spatial resolution. Continuous reassessment of CADe and CADx's performance is needed during their implementation in clinical practice.

Enhancing physicians' radiology diagnostics of COVID-19's effects on lung health by leveraging artificial intelligence

Introduction: This study aimed to develop an individualized artificial intelligence model to help radiologists assess the severity of COVID-19's effects on patients' lung health. Methods: Data was collected from medical records of 1103 patients diagnosed with COVID-19 using RT- qPCR between March and June 2020, in Hospital Madrid-Group (HM-Group, Spain). By using Convolutional Neural Networks, we determine the effects of COVID-19 in terms of lung area, opacities, and pulmonary air density. We then combine these variables with age and sex in a regression model to assess the severity of these conditions with respect to fatality risk (death or ICU). Results: Our model can predict high effect with an AUC of 0.736. Finally, we compare the performance of the model with respect to six physicians' diagnosis, and test for improvements on physicians' performance when using the prediction algorithm. Discussion: We find that the algorithm outperforms physicians (39.5% less error), and thus, physicians can significantly benefit from the information provided by the algorithm by reducing error by almost 30%.

Beyond high hopes: A scoping review of the 2019-2021 scientific discourse on machine learning in medical imaging

Machine learning has become a key driver of the digital health revolution. That comes with a fair share of high hopes and hype. We conducted a scoping review on machine learning in medical imaging, providing a comprehensive outlook of the field's potential, limitations, and future directions. Most reported strengths and promises included: improved (a) analytic power, (b) efficiency (c) decision making, and (d) equity. Most reported challenges included: (a) structural barriers and imaging heterogeneity, (b) scarcity of well-annotated, representative and interconnected imaging datasets (c) validity and performance limitations, including bias and equity issues, and (d) the still missing clinical integration. The boundaries between strengths and challenges, with cross-cutting ethical and regulatory implications, remain blurred. The literature emphasizes explainability and trustworthiness, with a largely missing discussion about the specific technical and regulatory challenges surrounding these concepts. Future trends are expected to shift towards multi-source models, combining imaging with an array of other data, in a more open access, and explainable manner.

Incidentally found resectable lung cancer with the usage of artificial intelligence on chest radiographs

PURPOSE

Detection of early lung cancer using chest radiograph remains challenging. We aimed to highlight the benefit of using artificial intelligence (AI) in chest radiograph with regard to its role in the unexpected detection of resectable early lung cancer.

MATERIALS AND METHODS

Patients with pathologically proven resectable lung cancer from March 2020 to February 2022 were retrospectively analyzed. Among them, we included patients with incidentally detected resectable lung cancer. Because commercially available AI-based lesion detection software was integrated for all chest radiographs in our hospital, we reviewed the clinical process of detecting lung cancer using AI in chest radiographs.

RESULTS

Among the 75 patients with pathologically proven resectable lung cancer, 13 (17.3%) had incidentally discovered lung cancer with a median size of 2.6 cm. Eight patients underwent chest radiograph for the evaluation of extrapulmonary diseases, while five underwent radiograph in preparation of an operation or procedure concerning other body parts. All lesions were detected as nodules by the AI-based software, and the median abnormality score for the nodules was 78%. Eight patients (61.5%) consulted a pulmonologist promptly on the same day when the chest radiograph was taken and before they received the radiologist's official report. Total and invasive sizes of the part-solid nodules were 2.3-3.3 cm and 0.75-2.2 cm, respectively.

CONCLUSION

This study demonstrates actual cases of unexpectedly detected resectable early lung cancer using AI-based lesion detection software. Our results suggest that AI is beneficial for incidental detection of early lung cancer in chest radiographs.

Artificial intelligence: A critical review of applications for lung nodule and lung cancer

Artificial intelligence (AI) is a broad concept that usually refers to computer programs that can learn from data and perform certain specific tasks. In the recent years, the growth of deep learning, a successful technique for computer vision tasks that does not require explicit programming, coupled with the availability of large imaging databases fostered the development of multiple applications in the medical imaging field, especially for lung nodules and lung cancer, mostly through convolutional neural networks (CNN). Some of the first applications of AI is this field were dedicated to automated detection of lung nodules on X-ray and computed tomography (CT) examinations, with performances now reaching or exceeding those of radiologists. For lung nodule segmentation, CNN-based algorithms applied to CT images show excellent spatial overlap index with manual segmentation, even for irregular and ground glass nodules. A third application of AI is the classification of lung nodules between malignant and benign, which could limit the number of follow-up CT examinations for less suspicious lesions. Several algorithms have demonstrated excellent capabilities for the prediction of the malignancy risk when a nodule is discovered. These different applications of AI for lung nodules are particularly appealing in the context of lung cancer screening. In the field of lung cancer, AI tools applied to lung imaging have been investigated for distinct aims. First, they could play a role for the non-invasive characterization of tumors, especially for histological subtype and somatic mutation predictions, with a potential therapeutic impact. Additionally, they could help predict the patient prognosis, in combination to clinical data. Despite these encouraging perspectives, clinical implementation of AI tools is only beginning because of the lack of generalizability of published studies, of an inner obscure working and because of limited data about the impact of such tools on the radiologists' decision and on the patient outcome. Radiologists must be active participants in the process of evaluating AI tools, as such tools could support their daily work and offer them more time for high added value tasks.

Pulmonary Nodule Clinical Trial Data Collection and Intelligent Differential Diagnosis for Medical Internet of Things

In this paper, the medical Internet of things (IoT) is used to pool data from clinical trials of pulmonary nodules, and on this basis, intelligent differential diagnosis techniques are investigated. A filtered orthogonal frequency division multiplexing model based on polarisation coding is proposed, where the input data are fed to a modulator after polarisation cascade coding, and the system performance is analysed under a medical Internet of things modulated additive Gaussian white noise channel. The above polarisation-coded filtered orthogonal frequency division multiplexing system components are applied to electroencephalogram (EEG) signal transmission, to which a threshold compression module and a vector reconstruction module are added to address the system power burden associated with the acquisition and transmission of large amounts of real-time EEG data in the medical IoT. In the threshold compression module, the inherent characteristics of EEG signals are analysed, and the generated EEG data are decomposed into multiple symbolic streams and compressed by applying different thresholds to improve the compression ratio while ensuring the quality of service of the application. A deep neural network-based approach is proposed for the detection and diagnosis of lung nodules. Automatic identification and measurement of simulated lung nodules and the corresponding volumes of nodules in images under different conditions are applied. The sensitivity of each AIADS in identifying lung nodules under different convolution kernel conditions, false positives (FP), false negatives (FN), relative volume errors (RVE), the miss detection rate (MDR) for different types of lung nodules, and the performance of each system in predicting the four types of nodules are calculated. In this paper, an interpretable multibranch feature convolutional neural network model is proposed for the diagnosis of benign and malignant lung nodules. It is demonstrated that the proposed model not only yields interpretable lung nodule classification results but also achieves better lung nodule classification performance with an accuracy rate of 97.8%.

Diagnostic validation of a deep learning nodule detection algorithm in low-dose chest CT: determination of optimized dose thresholds in a virtual screening scenario

OBJECTIVES

This study was conducted to evaluate the effect of dose reduction on the performance of a deep learning (DL)-based computer-aided diagnosis (CAD) system regarding pulmonary nodule detection in a virtual screening scenario.

METHODS

Sixty-eight anthropomorphic chest phantoms were equipped with 329 nodules (150 ground glass, 179 solid) with four sizes (5 mm, 8 mm, 10 mm, 12 mm) and scanned with nine tube voltage/current combinations. The examinations were analyzed by a commercially available DL-based CAD system. The results were compared by a comparison of proportions. Logistic regression was performed to evaluate the impact of tube voltage, tube current, nodule size, nodule density, and nodule location.

RESULTS

The combination with the lowest effective dose (E) and unimpaired detection rate was 80 kV/50 mAs (sensitivity: 97.9%, mean false-positive rate (FPR): 1.9, mean CTDIvol: 1.2 ± 0.4 mGy, mean E: 0.66 mSv). Logistic regression revealed that tube voltage and current had the greatest impact on the detection rate, while nodule size and density had no significant influence.

CONCLUSIONS

The optimal tube voltage/current combination proposed in this study (80 kV/50 mAs) is comparable to the proposed combinations in similar studies, which mostly dealt with conventional CAD software. Modification of tube voltage and tube current has a significant impact on the performance of DL-based CAD software in pulmonary nodule detection regardless of their size and composition.

KEY POINTS

• Modification of tube voltage and tube current has a significant impact on the performance of deep learning-based CAD software. • Nodule size and composition have no significant impact on the software's performance. • The optimal tube voltage/current combination for the examined software is 80 kV/50 mAs.

How AI Can Help in the Diagnostic Dilemma of Pulmonary Nodules

Simple Summary

Pulmonary nodules are considered a sign of bronchogenic carcinoma, detecting them early will reduce their progression and can save lives. Lung cancer is the second most common type of cancer in both men and women. This manuscript discusses the current applications of artificial intelligence (AI) in lung segmentation as well as pulmonary nodule segmentation and classification using computed tomography (CT) scans, published in the last two decades, in addition to the limitations and future prospects in the field of AI.

Abstract

Pulmonary nodules are the precursors of bronchogenic carcinoma, its early detection facilitates early treatment which save a lot of lives. Unfortunately, pulmonary nodule detection and classification are liable to subjective variations with high rate of missing small cancerous lesions which opens the way for implementation of artificial intelligence (AI) and computer aided diagnosis (CAD) systems. The field of deep learning and neural networks is expanding every day with new models designed to overcome diagnostic problems and provide more applicable and simply used models. We aim in this review to briefly discuss the current applications of AI in lung segmentation, pulmonary nodule detection and classification.

Performance of an AI based CAD system in solid lung nodule detection on chest phantom radiographs compared to radiology residents and fellow radiologists

Background

Despite the decreasing relevance of chest radiography in lung cancer screening, chest radiography is still frequently applied to assess for lung nodules. The aim of the current study was to determine the accuracy of a commercial AI based CAD system for the detection of artificial lung nodules on chest radiograph phantoms and compare the performance to radiologists in training.

Methods

Sixty-one anthropomorphic lung phantoms were equipped with 140 randomly deployed artificial lung nodules (5, 8, 10, 12 mm). A random generator chose nodule size and distribution before a two-plane chest X-ray (CXR) of each phantom was performed. Seven blinded radiologists in training (2 fellows, 5 residents) with 2 to 5 years of experience in chest imaging read the CXRs on a PACS-workstation independently. Results of the software were recorded separately. McNemar test was used to compare each radiologist’s results to the AI-computer-aided-diagnostic (CAD) software in a per-nodule and a per-phantom approach and Fleiss-Kappa was applied for inter-rater and intra-observer agreements.

Results

Five out of seven readers showed a significantly higher accuracy than the AI algorithm. The pooled accuracies of the radiologists in a nodule-based and a phantom-based approach were 0.59 and 0.82 respectively, whereas the AI-CAD showed accuracies of 0.47 and 0.67, respectively. Radiologists’ average sensitivity for 10 and 12 mm nodules was 0.80 and dropped to 0.66 for 8 mm (P=0.04) and 0.14 for 5 mm nodules (P<0.001). The radiologists and the algorithm both demonstrated a significant higher sensitivity for peripheral compared to central nodules (0.66 vs. 0.48; P=0.004 and 0.64 vs. 0.094; P=0.025, respectively). Inter-rater agreements were moderate among the radiologists and between radiologists and AI-CAD software (K’=0.58±0.13 and 0.51±0.1). Intra-observer agreement was calculated for two readers and was almost perfect for the phantom-based (K’=0.85±0.05; K’=0.80±0.02); and substantial to almost perfect for the nodule-based approach (K’=0.83±0.02; K’=0.78±0.02).

Conclusions

The AI based CAD system as a primary reader acts inferior to radiologists regarding lung nodule detection in chest phantoms. Chest radiography has reasonable accuracy in lung nodule detection if read by a radiologist alone and may be further optimized by an AI based CAD system as a second reader.