Detection Sensitivity of a Commercial Lung Nodule CAD System in a Series of Pathologically Proven Lung Cancers

"Will I change nodule management recommendations if I change my CAD system?"-impact of volumetric deviation between different CAD systems on lesion management

OBJECTIVES

To evaluate and compare the measurement accuracy of two different computer-aided diagnosis (CAD) systems regarding artificial pulmonary nodules and assess the clinical impact of volumetric inaccuracies in a phantom study.

METHODS

In this phantom study, 59 different phantom arrangements with 326 artificial nodules (178 solid, 148 ground-glass) were scanned at 80 kV, 100 kV, and 120 kV. Four different nodule diameters were used: 5 mm, 8 mm, 10 mm, and 12 mm. Scans were analyzed by a deep-learning (DL)-based CAD and a standard CAD system. Relative volumetric errors (RVE) of each system vs. ground truth and the relative volume difference (RVD) DL-based vs. standard CAD were calculated. The Bland-Altman method was used to define the limits of agreement (LOA). The hypothetical impact on LungRADS classification was assessed for both systems.

RESULTS

There was no difference between the three voltage groups regarding nodule volumetry. Regarding the solid nodules, the RVE of the 5-mm-, 8-mm-, 10-mm-, and 12-mm-size groups for the DL CAD/standard CAD were 12.2/2.8%, 1.3/ - 2.8%, - 3.6/1.5%, and - 12.2/ - 0.3%, respectively. The corresponding values for the ground-glass nodules (GGN) were 25.6%/81.0%, 9.0%/28.0%, 7.6/20.6%, and 6.8/21.2%. The mean RVD for solid nodules/GGN was 1.3/ - 15.2%. Regarding the LungRADS classification, 88.5% and 79.8% of all solid nodules were correctly assigned by the DL CAD and the standard CAD, respectively. 14.9% of the nodules were assigned differently between the systems.

CONCLUSIONS

Patient management may be affected by the volumetric inaccuracy of the CAD systems and hence demands supervision and/or manual correction by a radiologist.

KEY POINTS

• The DL-based CAD system was more accurate in the volumetry of GGN and less accurate regarding solid nodules than the standard CAD system. • Nodule size and attenuation have an effect on the measurement accuracy of both systems; tube voltage has no effect on measurement accuracy. • Measurement inaccuracies of CAD systems can have an impact on patient management, which demands supervision by radiologists.

A practical and adaptive approach to lung cancer screening: a review of international evidence and position on CT lung cancer screening in the Singaporean population by the College of Radiologists Singapore

Lung cancer is the leading cause of cancer-related death around the world, being the top cause of cancer-related deaths among men and the second most common cause of cancer-related deaths among women in Singapore. Currently, no screening programme for lung cancer exists in Singapore. Since there is mounting evidence indicating a different epidemiology of lung cancer in Asian countries, including Singapore, compared to the rest of the world, a unique and adaptive approach must be taken for a screening programme to be successful at reducing mortality while maintaining cost-effectiveness and a favourable risk-benefit ratio. This review article promotes the use of low-dose computed tomography of the chest and explores the radiological challenges and future directions.

Screening baseline characteristics of early lung cancer on low-dose computed tomography with computer-aided detection in a Chinese population

OBJECTIVES

This study investigated appropriate baseline characteristics for screening a Chinese population at high risk of early lung cancer, assisted by low-dose computed tomography (LDCT) with computer-aided detection (CAD). Included is a discussion of the viability of using LDCT in the screening guideline and optimizing the guideline.

METHODS

In 2014, 1016 individuals from Sichuan Province were enrolled who satisfied the criteria for high risk according to the 2013 National Comprehensive Cancer Network (NCCN) Guidelines for Non-Small Cell Lung Cancer. From 2014 to 2018, each subject was followed using LDCT with CAD, and pathologically confirmed lung cancers and baseline nodule characteristics (size and density) were recorded. Positive risk was considered a non-calcified solid or part-solid nodule on LDCT with diameter ≥5 mm and ground-glass nodule ≥8 mm, as newly recommended by the China National Lung Cancer Screening Guideline.

RESULTS

From 2014-2018, 13 cases of lung cancer were detected; 5 of these were early stage (38.5%). According to the NCCN criteria, 54 women were included and one of these (1.8%) developed lung cancer. The prevalence of lung cancer was 0.7% at baseline. For the entire population (excluding subjects with a tumor mass at baseline, n = 4), the rate of positivity was 20.4% at baseline; applying the Chinese criteria, the false positive rate was 19.5% (197/1012).

CONCLUSIONS

Further studies are warranted to establish appropriate eligible criteria and management strategies for Chinese populations.

Comparison of computer-aided detection (CADe) capability for pulmonary nodules among standard-, reduced- and ultra-low-dose CTs with and without hybrid type iterative reconstruction technique

PURPOSE

To directly compare the effect of a reconstruction algorithm on nodule detection capability of the computer-aided detection (CADe) system using standard-dose, reduced-dose and ultra-low dose chest CTs with and without adaptive iterative dose reduction 3D (AIDR 3D).

MATERIALS AND METHODS

Our institutional review board approved this study, and written informed consent was obtained from each patient. Standard-, reduced- and ultra-low-dose chest CTs (250 mA, 50 mA and 10 mA) were used to examine 40 patients, 21 males (mean age ± standard deviation: 63.1 ± 11.0 years) and 19 females (mean age, 65.1 ± 12.7 years), and reconstructed as 1 mm-thick sections. Detection of nodule equal to more than 4 mm in dimeter was automatically performed by our proprietary CADe software. The utility of iterative reconstruction method for improving nodule detection capability, sensitivity and false positive rate (/case) of the CADe system using all protocols were compared by means of McNemar's test or signed rank test.

RESULTS

Sensitivity (SE: 0.43) and false-positive rate (FPR: 7.88) of ultra-low-dose CT without AIDR 3D was significantly inferior to those of standard-dose CTs (with AIDR 3D: SE, 0.78, p < .0001, FPR, 3.05, p < .0001; and without AIDR 3D: SE, 0.80, p < .0001, FPR: 2.63, p < .0001), reduced-dose CTs (with AIDR 3D: SE, 0.81, p < .0001, FPR, 3.05, p < .0001; and without AIDR 3D: SE, 0.62, p < .0001, FPR: 2.95, p < .0001) and ultra-low-dose CT with AIDR 3D (SE, 0.79, p < .0001, FPR, 4.88, p = .0001).

CONCLUSION

The AIDR 3D has a significant positive effect on nodule detection capability of the CADe system even when radiation dose is reduced.

Maximum-Intensity-Projection and Computer-Aided-Detection Algorithms as Stand-Alone Reader Devices in Lung Cancer Screening Using Different Dose Levels and Reconstruction Kernels

OBJECTIVE

The objective of our study was to evaluate lung nodule detection rates on standard and microdose chest CT with two different computer-aided detection systems (SyngoCT-CAD, VA 20, Siemens Healthcare [CAD1]; Lung CAD, IntelliSpace Portal DX Server, Philips Healthcare [CAD2]) as well as maximum-intensity-projection (MIP) images. We also assessed the impact of different reconstruction kernels.

MATERIALS AND METHODS

Standard and microdose CT using three reconstruction kernels (i30, i50, i70) was performed with an anthropomorphic chest phantom. We placed 133 ground-glass and 133 solid nodules (diameters of 5 mm, 8 mm, 10 mm, and 12 mm) in 55 phantoms. Four blinded readers evaluated the MIP images; one recorded the results of CAD1 and CAD2. Sensitivities for CAD and MIP nodule detection on standard dose and microdose CT were calculated for each reconstruction kernel.

RESULTS

Dose for microdose CT was significantly less than that for standard-dose CT (0.1323 mSv vs 1.65 mSv; p < 0.0001). CAD1 delivered superior results compared with CAD2 for standard-dose and microdose CT (p < 0.0001). At microdose level, the best stand-alone sensitivity (97.6%) was comparable with CAD1 sensitivity (96.0%; p = 0.36; both with i30 reconstruction kernel). Pooled sensitivities for all nodules, doses, and reconstruction kernels on CAD1 ranged from 88.9% to 97.3% versus 49.6% to 73.9% for CAD2. The best sensitivity was achieved with standard-dose CT, i50 kernel, and CAD1 (97.3%) versus 96% with microdose CT, i30 or i50 kernel, and CAD1. MIP images and CAD1 had similar performance at both dose levels (p = 0.1313 and p = 0.48).

CONCLUSION

Submillisievert CT is feasible for detecting solid and ground-glass nodules that require soft-tissue kernels for MIP and CAD systems to achieve acceptable sensitivities. MIP reconstructions remain a valuable adjunct to the interpretation of chest CT for increasing sensitivity and have the advantage of significantly lower false-positive rates.

Low-Dose CT Screening for Lung Cancer: Computer-aided Detection of Missed Lung Cancers

Purpose To update information regarding the usefulness of computer-aided detection (CAD) systems with a focus on the most critical category, that of missed cancers at earlier imaging, for cancers that manifest as a solid nodule. Materials and Methods By using a HIPAA-compliant institutional review board-approved protocol where informed consent was obtained, 50 lung cancers that manifested as a solid nodule on computed tomographic (CT) scans in annual rounds of screening (time 1) were retrospectively identified that could, in retrospect, be identified on the previous CT scans (time 0). Four CAD systems were compared, which were referred to as CAD 1, CAD 2, CAD 3, and CAD 4. The total number of accepted CAD-system-detected nodules at time 0 was determined by consensus of two radiologists and the number of CAD-system-detected nodules that were rejected by the radiologists was also documented. Results At time 0 when all the cancers had been missed, CAD system detection rates for the cancers were 56%, 70%, 68%, and 60% (κ = 0.45) for CAD systems 1, 2, 3, and 4, respectively. At time 1, the rates were 74%, 82%, 82%, and 78% (κ = 0.32), respectively. The average diameter of the 50 cancers at time 0 and time 1 was 4.8 mm and 11.4 mm, respectively. The number of CAD-system-detected nodules that were rejected per CT scan for CAD systems 1-4 at time 0 was 7.4, 1.7, 0.6, and 4.5 respectively. Conclusion CAD systems detected up to 70% of lung cancers that were not detected by the radiologist but failed to detect about 20% of the lung cancers when they were identified by the radiologist, which suggests that CAD may be useful in the role of second reader. (©) RSNA, 2016.

Large scale validation of the M5L lung CAD on heterogeneous CT datasets

PURPOSE

M5L, a fully automated computer-aided detection (CAD) system for the detection and segmentation of lung nodules in thoracic computed tomography (CT), is presented and validated on several image datasets.

METHODS

M5L is the combination of two independent subsystems, based on the Channeler Ant Model as a segmentation tool [lung channeler ant model (lungCAM)] and on the voxel-based neural approach. The lungCAM was upgraded with a scan equalization module and a new procedure to recover the nodules connected to other lung structures; its classification module, which makes use of a feed-forward neural network, is based of a small number of features (13), so as to minimize the risk of lacking generalization, which could be possible given the large difference between the size of the training and testing datasets, which contain 94 and 1019 CTs, respectively. The lungCAM (standalone) and M5L (combined) performance was extensively tested on 1043 CT scans from three independent datasets, including a detailed analysis of the full Lung Image Database Consortium/Image Database Resource Initiative database, which is not yet found in literature.

RESULTS

The lungCAM and M5L performance is consistent across the databases, with a sensitivity of about 70% and 80%, respectively, at eight false positive findings per scan, despite the variable annotation criteria and acquisition and reconstruction conditions. A reduced sensitivity is found for subtle nodules and ground glass opacities (GGO) structures. A comparison with other CAD systems is also presented.

CONCLUSIONS

The M5L performance on a large and heterogeneous dataset is stable and satisfactory, although the development of a dedicated module for GGOs detection could further improve it, as well as an iterative optimization of the training procedure. The main aim of the present study was accomplished: M5L results do not deteriorate when increasing the dataset size, making it a candidate for supporting radiologists on large scale screenings and clinical programs.

The 10 Pillars of Lung Cancer Screening: Rationale and Logistics of a Lung Cancer Screening Program

On the basis of the National Lung Screening Trial data released in 2011, the U.S. Preventive Services Task Force made lung cancer screening (LCS) with low-dose computed tomography (CT) a public health recommendation in 2013. The Centers for Medicare and Medicaid Services (CMS) currently reimburse LCS for asymptomatic individuals aged 55-77 years who have a tobacco smoking history of at least 30 pack-years and who are either currently smoking or had quit less than 15 years earlier. Commercial insurers reimburse the cost of LCS for individuals aged 55-80 years with the same smoking history. Effective care for the millions of Americans who qualify for LCS requires an organized step-wise approach. The 10-pillar model reflects the elements required to support a successful LCS program: eligibility, education, examination ordering, image acquisition, image review, communication, referral network, quality improvement, reimbursement, and research frontiers. Examination ordering can be coupled with decision support to ensure that only eligible individuals undergo LCS. Communication of results revolves around the Lung Imaging Reporting and Data System (Lung-RADS) from the American College of Radiology. Lung-RADS is a structured decision-oriented reporting system designed to minimize the rate of false-positive screening examination results. With nodule size and morphology as discriminators, Lung-RADS links nodule management pathways to the variety of nodules present on LCS CT studies. Tracking of patient outcomes is facilitated by a CMS-approved national registry maintained by the American College of Radiology. Online supplemental material is available for this article.

Lung cancer screening: state of the art

Results from the National Lung Screening Trial have confirmed that lung cancer mortality is reduced using low-dose CT screening. Opening a lung cancer screening program requires a multidisciplinary approach. While the fundamental aspects of a screening program are similar, such as scheduling, performing, and managing follow-up, there are aspects of a lung cancer screening program that are unique. This article will discuss factors important in establishing a state of the art lung cancer screening program.

Missed lung cancer on chest radiography and computed tomography

Missed lung cancer raises an important medicolegal issue and contributes to one of the most common causes for malpractice actions against radiologists. Lung cancer may be missed on either chest radiography or computed tomography. Although most malpractice cases involve lesions overlooked on the former, a small and increasing portion of cases are related to chest computed tomography scan. Factors contributing to overlooked lung cancer can be attributed to observer performance, lesion characteristics, and technical considerations.

Comparing and combining algorithms for computer-aided detection of pulmonary nodules in computed tomography scans: The ANODE09 study

Numerous publications and commercial systems are available that deal with automatic detection of pulmonary nodules in thoracic computed tomography scans, but a comparative study where many systems are applied to the same data set has not yet been performed. This paper introduces ANODE09 ( http://anode09.isi.uu.nl), a database of 55 scans from a lung cancer screening program and a web-based framework for objective evaluation of nodule detection algorithms. Any team can upload results to facilitate benchmarking. The performance of six algorithms for which results are available are compared; five from academic groups and one commercially available system. A method to combine the output of multiple systems is proposed. Results show a substantial performance difference between algorithms, and demonstrate that combining the output of algorithms leads to marked performance improvements.

Computer-aided detection (CAD) of lung nodules in CT scans: radiologist performance and reading time with incremental CAD assistance

OBJECTIVE

The diagnostic performance of radiologists using incremental CAD assistance for lung nodule detection on CT and their temporal variation in performance during CAD evaluation was assessed.

METHODS

CAD was applied to 20 chest multidetector-row computed tomography (MDCT) scans containing 190 non-calcified > or =3-mm nodules. After free search, three radiologists independently evaluated a maximum of up to 50 CAD detections/patient. Multiple free-response ROC curves were generated for free search and successive CAD evaluation, by incrementally adding CAD detections one at a time to the radiologists' performance.

RESULTS

The sensitivity for free search was 53% (range, 44%-59%) at 1.15 false positives (FP)/patient and increased with CAD to 69% (range, 59-82%) at 1.45 FP/patient. CAD evaluation initially resulted in a sharp rise in sensitivity of 14% with a minimal increase in FP over a time period of 100 s, followed by flattening of the sensitivity increase to only 2%. This transition resulted from a greater prevalence of true positive (TP) versus FP detections at early CAD evaluation and not by a temporal change in readers' performance. The time spent for TP (9.5 s +/- 4.5 s) and false negative (FN) (8.4 s +/- 6.7 s) detections was similar; FP decisions took two- to three-times longer (14.4 s +/- 8.7 s) than true negative (TN) decisions (4.7 s +/- 1.3 s).

CONCLUSIONS

When CAD output is ordered by CAD score, an initial period of rapid performance improvement slows significantly over time because of non-uniformity in the distribution of TP CAD output and not to a changing reader performance over time.

Commercially available computer-aided detection system for pulmonary nodules on thin-section images using 64 detectors-row CT: preliminary study of 48 cases

RATIONALE AND OBJECTIVES

Most studies of computer-aided detection (CAD) for pulmonary nodules have focused on solid nodule detection. The aim of this study was to evaluate the performance of a commercially available CAD system in the detection of pulmonary nodules with or without ground-glass opacity (GGO) using 64-detector-row computed tomography compared to visual interpretation.

MATERIALS AND METHODS

Computed tomographic examinations were performed on 48 patients with existing or suspicious pulmonary nodules on chest radiography. Three radiologists independently reported the location and pattern (GGO, solid, or part solid) of each nodule candidate on computed tomographic scans, assigned each a confidence score, and then analyzed all scans using the CAD system. A reference standard was established by a consensus panel of different radiologists, who found 229 noncalcified nodules with diameters > or = 4 mm. True-positive and false-positive results and confidence levels were used to generate jackknife alternative free-response receiver-operating characteristic plots.

RESULTS

The sensitivity of GGO for 3 radiologists (60%-80%) was significantly higher than that for the CAD system (21%) (McNemar's test, P < .0001). For overall and solid nodules, the figure-of-merit values without and with the CAD system were significantly different (P = .005-.04) on jackknife alternative free-response receiver-operating characteristic analysis. For GGO and part-solid nodules, the figure-of-merit values with the CAD system were greater than those without the CAD system, indicating no significant differences.

CONCLUSION

Radiologists are significantly superior to this CAD system in the detection of GGO, but the CAD system can still play a complementary role in detecting nodules with or without GGO.