Validation of a deep learning computer aided system for CT based lung nodule detection, classification, and growth rate estimation in a routine clinical population

Performance of AI for preoperative CT assessment of lung metastases: Retrospective analysis of 167 patients

OBJECTIVES

To evaluate the performance of artificial intelligence (AI) in the preoperative detection of lung metastases on CT.

MATERIALS AND METHODS

Patients who underwent lung metastasectomy in our institution between 2016 and 2020 were enrolled, their preoperative CT reports having been performed before an AI solution (Veye Lung Nodules, version 3.9.2, Aidence) became available as a second reader in our department. All CT scans were retrospectively processed by AI. The sensitivities of unassisted radiologists (original CT radiology reports), AI reports alone and both combined were compared. Ground truth was established by a consensus reading of two radiologists, who analyzed whether the nodules mentioned in the pathology report were retrospectively visible on CT. Multivariate analysis was performed to identify nodule characteristics associated with detectability.

RESULTS

A total of 167 patients (men: 62.9 %; median age, 59 years [47-68]) with 475 resected nodules were included. AI detected an average of 4 nodules (0-17) per CT, of which 97 % were true nodules. The combination of radiologist plus AI (92.4 %) had significantly higher sensitivity than unassisted radiologists (80.4 %) (p < 0.001). In 27/57 (47.4 %) patients who had multiple preoperative CT scans, AI detected lung nodules earlier than the radiologist. Vascular contact was associated with non-detection by radiologists (OR:0.32[0.19, 0.54], p < 0.001), whilst the presence of cavitation (OR:0.26[0.13, 0.54], p < 0.001) or pleural contact (OR:0.10[0.04, 0.22], p < 0.001) was associated with non-detection by AI.

CONCLUSION

AI significantly increases the sensitivity of preoperative detection of lung metastases and enables earlier detection, with a significant potential benefit for patient management.

Validation of a commercially available CAD-system for lung nodule detection and characterization using CT-scans

OBJECTIVES

This study aims to externally validate a commercially available Computer-Aided Detection (CAD)-system for the automatic detection and characterization of solid, part-solid, and ground-glass lung nodules (LN) on CT scans.

METHODS

This retrospective study encompasses 263 chest CT scans performed between January 2020 and December 2021 at a Dutch university hospital. All scans were read by a radiologist (R1) and compared with the initial radiology report. Conflicting scans were assessed by an adjudicating radiologist (R2). All scans were also processed by CAD. The standalone performance of CAD in terms of sensitivity and false-positive (FP)-rate for detection was calculated together with the sensitivity for characterization, including texture, calcification, speculation, and location. The R1's detection sensitivity was also assessed.

RESULTS

A total of 183 true nodules were identified in 121 nodule-containing scans (142 non-nodule-containing scans), of which R1 identified 165/183 (90.2%). CAD detected 149 nodules, of which 12 were not identified by R1, achieving a sensitivity of 149/183 (81.4%) with an FP-rate of 49/121 (0.405). CAD's detection sensitivity for solid, part-solid, and ground-glass LNs was 82/94 (87.2%), 42/47 (89.4%), and 25/42 (59.5%), respectively. The classification accuracy for solid, part-solid, and ground-glass LNs was 81/82 (98.8%), 16/42 (38.1%), and 18/25 (72.0%), respectively. Additionally, CAD demonstrated overall classification accuracies of 137/149 (91.9%), 123/149 (82.6%), and 141/149 (94.6%) for calcification, spiculation, and location, respectively.

CONCLUSIONS

Although the overall detection rate of this system slightly lags behind that of a radiologist, CAD is capable of detecting different LNs and thereby has the potential to enhance a reader's detection rate. While promising characterization performances are obtained, the tool's performance in terms of texture classification remains a subject of concern.

CLINICAL RELEVANCE STATEMENT

Numerous lung nodule computer-aided detection-systems are commercially available, with some of them solely being externally validated based on their detection performance on solid nodules. We encourage researchers to assess performances by incorporating all relevant characteristics, including part-solid and ground-glass nodules.

KEY POINTS

Few computer-aided detection (CAD) systems are externally validated for automatic detection and characterization of lung nodules. A detection sensitivity of 81.4% and an overall texture classification sensitivity of 77.2% were measured utilizing CAD. CAD has the potential to increase single reader detection rate, however, improvement in texture classification is required.

Effect of emphysema on AI software and human reader performance in lung nodule detection from low-dose chest CT

BACKGROUND

Emphysema influences the appearance of lung tissue in computed tomography (CT). We evaluated whether this affects lung nodule detection by artificial intelligence (AI) and human readers (HR).

METHODS

Individuals were selected from the "Lifelines" cohort who had undergone low-dose chest CT. Nodules in individuals without emphysema were matched to similar-sized nodules in individuals with at least moderate emphysema. AI results for nodular findings of 30-100 mm3 and 101-300 mm3 were compared to those of HR; two expert radiologists blindly reviewed discrepancies. Sensitivity and false positives (FPs)/scan were compared for emphysema and non-emphysema groups.

RESULTS

Thirty-nine participants with and 82 without emphysema were included (n = 121, aged 61 ± 8 years (mean ± standard deviation), 58/121 males (47.9%)). AI and HR detected 196 and 206 nodular findings, respectively, yielding 109 concordant nodules and 184 discrepancies, including 118 true nodules. For AI, sensitivity was 0.68 (95% confidence interval 0.57-0.77) in emphysema versus 0.71 (0.62-0.78) in non-emphysema, with FPs/scan 0.51 and 0.22, respectively (p = 0.028). For HR, sensitivity was 0.76 (0.65-0.84) and 0.80 (0.72-0.86), with FPs/scan of 0.15 and 0.27 (p = 0.230). Overall sensitivity was slightly higher for HR than for AI, but this difference disappeared after the exclusion of benign lymph nodes. FPs/scan were higher for AI in emphysema than in non-emphysema (p = 0.028), while FPs/scan for HR were higher than AI for 30-100 mm3 nodules in non-emphysema (p = 0.009).

CONCLUSIONS

AI resulted in more FPs/scan in emphysema compared to non-emphysema, a difference not observed for HR.

RELEVANCE STATEMENT

In the creation of a benchmark dataset to validate AI software for lung nodule detection, the inclusion of emphysema cases is important due to the additional number of FPs.

KEY POINTS

• The sensitivity of nodule detection by AI was similar in emphysema and non-emphysema. • AI had more FPs/scan in emphysema compared to non-emphysema. • Sensitivity and FPs/scan by the human reader were comparable for emphysema and non-emphysema. • Emphysema and non-emphysema representation in benchmark dataset is important for validating AI.

HUNCHEST-II contributes to a shift to earlier-stage lung cancer detection: final results of a nationwide screening program

OBJECTIVES

The introduction of low-dose CT (LDCT) altered the landscape of lung cancer (LC) screening and contributed to the reduction of mortality rates worldwide. Here we report the final results of HUNCHEST-II, the largest population-based LDCT screening program in Hungary, including the screening and diagnostic outcomes, and the characteristics of the LC cases.

METHODS

A total of 4215 high-risk individuals aged between 50 and 75 years with a smoking history of at least 25 pack-years were assigned to undergo LDCT screening. Screening outcomes were determined based on the volume, growth, and volume doubling time of pulmonary nodules or masses. The clinical stage distribution of screen-detected cancers was compared with two independent practice-based databases consisting of unscreened LC patients.

RESULTS

The percentage of negative and indeterminate tests at baseline were 74.2% and 21.7%, respectively, whereas the prevalence of positive LDCT results was 4.1%. Overall, 76 LC patients were diagnosed throughout the screening rounds (1.8% of total participants), out of which 62 (1.5%) patients were already identified in the first screening round. The overall positive predictive value of a positive test was 58%. Most screen-detected malignancies were stage I LCs (60.7%), and only 16.4% of all cases could be classified as stage IV disease. The percentage of early-stage malignancies was significantly higher among HUNCHEST-II screen-detected individuals than among the LC patients in the National Koranyi Institute of Pulmonology's archive or the Hungarian Cancer Registry (p < 0.001).

CONCLUSIONS

HUNCHEST-II demonstrates that LDCT screening for LC facilitates early diagnosis, thus arguing in favor of introducing systematic LC screening in Hungary.

CLINICAL RELEVANCE STATEMENT

HUNCHEST-II is the so-far largest population-based low-dose CT screening program in Hungary. A positive test's overall positive predictive value was 58%, and most screen-detected malignancies were early-stage lesions. These results pave the way for expansive systematic screening in the region.

KEY POINTS

• Conducted in 18 medical facilities, HUNCHEST-II is the so far largest population-based low-dose CT screening program in Hungary. • The vast majority of screen-detected malignancies were early-stage lung cancers, and the overall positive predictive value of a positive test was 58%. • HUNCHEST-II facilitates early diagnosis, thus arguing in favor of introducing systematic lung cancer screening in Hungary.

Absolute ground truth-based validation of computer-aided nodule detection and volumetry in low-dose CT imaging

PURPOSE

To validate the performance of computer-aided detection (CAD) and volumetry software using an anthropomorphic phantom with a ground truth (GT) set of 3D-printed nodules.

METHODS

The Kyoto Kaguku Lungman phantom, containing 3D-printed solid nodules including six diameters (4 to 9 mm) and three morphologies (smooth, lobulated, spiculated), was scanned at varying CTDIvol levels (6.04, 1.54 and 0.20 mGy). Combinations of reconstruction algorithms (iterative and deep learning image reconstruction) and kernels (soft and hard) were applied. Detection, volumetry and density results recorded by a commercially available AI-based algorithm (AVIEW LCS + ) were compared to the absolute GT, which was determined through µCT scanning at 50 µm resolution. The associations between image acquisition parameters or nodule characteristics and accuracy of nodule detection and characterization were analyzed with chi square tests and multiple linear regression.

RESULTS

High levels of detection sensitivity and precision (minimal 83 % and 91 % respectively) were observed across all acquisitions. Neither reconstruction algorithm nor radiation dose showed significant associations with detection. Nodule diameter however showed a highly significant association with detection (p < 0.0001). Volumetric measurements for nodules > 6 mm were accurate within 10 % absolute range from volumeGT, regardless of dose and reconstruction. Nodule diameter and morphology are major determinants of volumetric accuracy (p < 0.001). Density assignment was not significantly influenced by any parameters.

CONCLUSIONS

Our study confirms the software's accurate performance in nodule volumetry, detection and density characterization with robustness for variations in CT imaging protocols. This study suggests the incorporation of similar phantom setups in quality assurance of CAD tools.

Early experiences of integrating an artificial intelligence-based diagnostic decision support system into radiology settings: a qualitative study

OBJECTIVES

Artificial intelligence (AI)-based clinical decision support systems to aid diagnosis are increasingly being developed and implemented but with limited understanding of how such systems integrate with existing clinical work and organizational practices. We explored the early experiences of stakeholders using an AI-based imaging software tool Veye Lung Nodules (VLN) aiding the detection, classification, and measurement of pulmonary nodules in computed tomography scans of the chest.

MATERIALS AND METHODS

We performed semistructured interviews and observations across early adopter deployment sites with clinicians, strategic decision-makers, suppliers, patients with long-term chest conditions, and academics with expertise in the use of diagnostic AI in radiology settings. We coded the data using the Technology, People, Organizations, and Macroenvironmental factors framework.

RESULTS

We conducted 39 interviews. Clinicians reported VLN to be easy to use with little disruption to the workflow. There were differences in patterns of use between experts and novice users with experts critically evaluating system recommendations and actively compensating for system limitations to achieve more reliable performance. Patients also viewed the tool positively. There were contextual variations in tool performance and use between different hospital sites and different use cases. Implementation challenges included integration with existing information systems, data protection, and perceived issues surrounding wider and sustained adoption, including procurement costs.

DISCUSSION

Tool performance was variable, affected by integration into workflows and divisions of labor and knowledge, as well as technical configuration and infrastructure.

CONCLUSION

The socio-organizational factors affecting performance of diagnostic AI are under-researched and require attention and further research.

Deep learning for the detection of benign and malignant pulmonary nodules in non-screening chest CT scans

BACKGROUND

Outside a screening program, early-stage lung cancer is generally diagnosed after the detection of incidental nodules in clinically ordered chest CT scans. Despite the advances in artificial intelligence (AI) systems for lung cancer detection, clinical validation of these systems is lacking in a non-screening setting.

METHOD

We developed a deep learning-based AI system and assessed its performance for the detection of actionable benign nodules (requiring follow-up), small lung cancers, and pulmonary metastases in CT scans acquired in two Dutch hospitals (internal and external validation). A panel of five thoracic radiologists labeled all nodules, and two additional radiologists verified the nodule malignancy status and searched for any missed cancers using data from the national Netherlands Cancer Registry. The detection performance was evaluated by measuring the sensitivity at predefined false positive rates on a free receiver operating characteristic curve and was compared with the panel of radiologists.

RESULTS

On the external test set (100 scans from 100 patients), the sensitivity of the AI system for detecting benign nodules, primary lung cancers, and metastases is respectively 94.3% (82/87, 95% CI: 88.1-98.8%), 96.9% (31/32, 95% CI: 91.7-100%), and 92.0% (104/113, 95% CI: 88.5-95.5%) at a clinically acceptable operating point of 1 false positive per scan (FP/s). These sensitivities are comparable to or higher than the radiologists, albeit with a slightly higher FP/s (average difference of 0.6).

CONCLUSIONS

The AI system reliably detects benign and malignant pulmonary nodules in clinically indicated CT scans and can potentially assist radiologists in this setting.

Development and evaluation of an integrated model based on a deep segmentation network and demography-added radiomics algorithm for segmentation and diagnosis of early lung adenocarcinoma

Non-invasive early detection and differentiation grading of lung adenocarcinoma using computed tomography (CT) images are clinically important for both clinicians and patients, including determining the extent of lung resection. However, these are difficult to accomplish using preoperative images, with CT-based diagnoses often being different from postoperative pathologic diagnoses. In this study, we proposed an integrated detection and classification algorithm (IDCal) for diagnosing ground-glass opacity nodules (GGN) using CT images and other patient informatics, and compared its performance with that of other diagnostic modalities. All labeling was confirmed by a thoracic surgeon by referring to the patient's CT image and biopsy report. The detection phase was implemented via a modified FC-DenseNet to contour the lesions as elaborately as possible and secure the reliability of the classification phase for subsequent applications. Then, by integrating radiomics features and other patients' general information, the lesions were dichotomously reclassified into "non-invasive" (atypical adenomatous hyperplasia, adenocarcinoma in situ, and minimally invasive adenocarcinoma) and "invasive" (invasive adenocarcinoma). Data from 168 GGN cases were used to develop the IDCal, which was then validated in 31 independent CT scans. IDCal showed a high accuracy of GGN detection (sensitivity, 0.970; false discovery rate, 0.697) and classification (accuracy, 0.97; f1-score, 0.98; ROAUC, 0.96). In conclusion, the proposed IDCal detects and classifies GGN with excellent performance. Thus, it can be suggested that our multimodal prediction model has high potential as an auxiliary diagnostic tool of GGN to help clinicians.

An Interpretable Three-Dimensional Artificial Intelligence Model for Computer-Aided Diagnosis of Lung Nodules in Computed Tomography Images

Lung cancer is typically classified into small-cell carcinoma and non-small-cell carcinoma. Non-small-cell carcinoma accounts for approximately 85% of all lung cancers. Low-dose chest computed tomography (CT) can quickly and non-invasively diagnose lung cancer. In the era of deep learning, an artificial intelligence (AI) computer-aided diagnosis system can be developed for the automatic recognition of CT images of patients, creating a new form of intelligent medical service. For many years, lung cancer has been the leading cause of cancer-related deaths in Taiwan, with smoking and air pollution increasing the likelihood of developing the disease. The incidence of lung adenocarcinoma in never-smoking women has also increased significantly in recent years, resulting in an important public health problem. Early detection of lung cancer and prompt treatment can help reduce the mortality rate of patients with lung cancer. In this study, an improved 3D interpretable hierarchical semantic convolutional neural network named HSNet was developed and validated for the automatic diagnosis of lung cancer based on a collection of lung nodule images. The interpretable AI model proposed in this study, with different training strategies and adjustment of model parameters, such as cyclic learning rate and random weight averaging, demonstrated better diagnostic performance than the previous literature, with results of a four-fold cross-validation procedure showing calcification: 0.9873 ± 0.006, margin: 0.9207 ± 0.009, subtlety: 0.9026 ± 0.014, texture: 0.9685 ± 0.006, sphericity: 0.8652 ± 0.021, and malignancy: 0.9685 ± 0.006.

The Effects of Artificial Intelligence Assistance on the Radiologists' Assessment of Lung Nodules on CT Scans: A Systematic Review

To reduce the number of missed or misdiagnosed lung nodules on CT scans by radiologists, many Artificial Intelligence (AI) algorithms have been developed. Some algorithms are currently being implemented in clinical practice, but the question is whether radiologists and patients really benefit from the use of these novel tools. This study aimed to review how AI assistance for lung nodule assessment on CT scans affects the performances of radiologists. We searched for studies that evaluated radiologists' performances in the detection or malignancy prediction of lung nodules with and without AI assistance. Concerning detection, radiologists achieved with AI assistance a higher sensitivity and AUC, while the specificity was slightly lower. Concerning malignancy prediction, radiologists achieved with AI assistance generally a higher sensitivity, specificity and AUC. The radiologists' workflows of using the AI assistance were often only described in limited detail in the papers. As recent studies showed improved performances of radiologists with AI assistance, AI assistance for lung nodule assessment holds great promise. To achieve added value of AI tools for lung nodule assessment in clinical practice, more research is required on the clinical validation of AI tools, impact on follow-up recommendations and ways of using AI tools.

Deep learning-based growth prediction for sub-solid pulmonary nodules on CT images

Background

Estimating the growth of pulmonary sub-solid nodules (SSNs) is crucial to the successful management of them during follow-up periods. The purpose of this study is to (1) investigate the measurement sensitivity of diameter, volume, and mass of SSNs for identifying growth and (2) seek to establish a deep learning-based model to predict the growth of SSNs.

Methods

A total of 2,523 patients underwent at least 2-year examination records retrospectively collected with sub-solid nodules. A total of 2,358 patients with 3,120 SSNs from the NLST dataset were randomly divided into training and validation sets. Patients from the Yibicom Health Management Center and Guangdong Provincial People’s Hospital were collected as an external test set (165 patients with 213 SSN). Trained models based on LUNA16 and Lndb19 datasets were employed to automatically obtain the diameter, volume, and mass of SSNs. Then, the increase rate in measurements between cancer and non-cancer groups was studied to evaluate the most appropriate way to identify growth-associated lung cancer. Further, according to the selected measurement, all SSNs were classified into two groups: growth and non-growth. Based on the data, the deep learning-based model (SiamModel) and radiomics model were developed and verified.

Results

The double time of diameter, volume, and mass were 711 vs. 963 days (P = 0.20), 552 vs. 621 days (P = 0.04) and 488 vs. 623 days (P< 0.001) in the cancer and non-cancer groups, respectively. Our proposed SiamModel performed better than the radiomics model in both the NLST validation set and external test set, with an AUC of 0.858 (95% CI 0.786–0.921) and 0.760 (95% CI 0.646–0.857) in the validation set and 0.862 (95% CI 0.789–0.927) and 0.681 (95% CI 0.506–0.841) in the external test set, respectively. Furthermore, our SiamModel could use the data from first-time CT to predict the growth of SSNs, with an AUC of 0.855 (95% CI 0.793–0.908) in the NLST validation set and 0.821 (95% CI 0.725–0.904) in the external test set.

Conclusion

Mass increase rate can reflect more sensitively the growth of SSNs associated with lung cancer than diameter and volume increase rates. A deep learning-based model has a great potential to predict the growth of SSNs.