Comparison of sensitivity of lung nodule detection between radiologists and technologists on low-dose CT lung cancer screening images

Validation of a deep learning computer aided system for CT based lung nodule detection, classification, and growth rate estimation in a routine clinical population

OBJECTIVE

In this study, we evaluated a commercially available computer assisted diagnosis system (CAD). The deep learning algorithm of the CAD was trained with a lung cancer screening cohort and developed for detection, classification, quantification, and growth of actionable pulmonary nodules on chest CT scans. Here, we evaluated the CAD in a retrospective cohort of a routine clinical population.

MATERIALS AND METHODS

In total, a number of 337 scans of 314 different subjects with reported nodules of 3-30 mm in size were included into the evaluation. Two independent thoracic radiologists alternately reviewed scans with or without CAD assistance to detect, classify, segment, and register pulmonary nodules. A third, more experienced, radiologist served as an adjudicator. In addition, the cohort was analyzed by the CAD alone. The study cohort was divided into five different groups: 1) 178 CT studies without reported pulmonary nodules, 2) 95 studies with 1-10 pulmonary nodules, 23 studies from the same patients with 3) baseline and 4) follow-up studies, and 5) 18 CT studies with subsolid nodules. A reference standard for nodules was based on majority consensus with the third thoracic radiologist as required. Sensitivity, false positive (FP) rate and Dice inter-reader coefficient were calculated.

RESULTS

After analysis of 470 pulmonary nodules, the sensitivity readings for radiologists without CAD and radiologist with CAD, were 71.9% (95% CI: 66.0%, 77.0%) and 80.3% (95% CI: 75.2%, 85.0%) (p < 0.01), with average FP rate of 0.11 and 0.16 per CT scan, respectively. Accuracy and kappa of CAD for classifying solid vs sub-solid nodules was 94.2% and 0.77, respectively. Average inter-reader Dice coefficient for nodule segmentation was 0.83 (95% CI: 0.39, 0.96) and 0.86 (95% CI: 0.51, 0.95) for CAD versus readers. Mean growth percentage discrepancy of readers and CAD alone was 1.30 (95% CI: 1.02, 2.21) and 1.35 (95% CI: 1.01, 4.99), respectively.

CONCLUSION

The applied CAD significantly increased radiologist's detection of actionable nodules yet also minimally increasing the false positive rate. The CAD can automatically classify and quantify nodules and calculate nodule growth rate in a cohort of a routine clinical population. Results suggest this Deep Learning software has the potential to assist chest radiologists in the tasks of pulmonary nodule detection and management within their routine clinical practice.

A Highly Reliable Convolutional Neural Network Based Soft Tissue Sarcoma Metastasis Detection from Chest X-ray Images: A Retrospective Cohort Study

Simple Summary

Soft tissue sarcomas are relatively rare malignant diseases. Part of the diagnosis and follow-up includes medical imaging of the thorax for detection of lung metastases. A Python script was created and trained using a set of lung X-rays and concordant CT scans from a high-volume German-speaking sarcoma center. It is capable of detecting malignant metastasis in the lung with a precision of 71.2%, specificity of 90.5%, sensitivity of 94% and accuracy of 91.2%. Furthermore, the program was able to detect even small nodules with a size <1 cm in conventional X-rays of the thorax. This algorithm was implemented into our daily clinical practice alongside with the radiologists’ findings. With this tool we aim to improve the quality of our service and reduce the expenditure of time.

Abstract

Introduction: soft tissue sarcomas are a subset of malignant tumors that are relatively rare and make up 1% of all malignant tumors in adulthood. Due to the rarity of these tumors, there are significant differences in quality in the diagnosis and treatment of these tumors. One paramount aspect is the diagnosis of hematogenous metastases in the lungs. Guidelines recommend routine lung imaging by means of X-rays. With the ever advancing AI-based diagnostic support, there has so far been no implementation for sarcomas. The aim of the study was to utilize AI to obtain analyzes regarding metastasis on lung X-rays in the most possible sensitive and specific manner in sarcoma patients. Methods: a Python script was created and trained using a set of lung X-rays with sarcoma metastases from a high-volume German-speaking sarcoma center. 26 patients with lung metastasis were included. For all patients chest X-ray with corresponding lung CT scans, and histological biopsies were available. The number of trainable images were expanded to 600. In order to evaluate the biological sensitivity and specificity, the script was tested on lung X-rays with a lung CT as control. Results: in this study we present a new type of convolutional neural network-based system with a precision of 71.2%, specificity of 90.5%, sensitivity of 94%, recall of 94% and accuracy of 91.2%. A good detection of even small findings was determined. Discussion: the created script establishes the option to check lung X-rays for metastases at a safe level, especially given this rare tumor entity.

Use of a Dual Artificial Intelligence Platform to Detect Unreported Lung Nodules

OBJECTIVE

To investigate the performance of Dual-AI Deep Learning Platform in detecting unreported pulmonary nodules that are 6 mm or greater, comprising computer-vision (CV) algorithm to detect pulmonary nodules, with positive results filtered by natural language processing (NLP) analysis of the dictated report.

METHODS

Retrospective analysis of 5047 chest CT scans and corresponding reports. Cases which were both CV algorithm positive (nodule ≥ 6 mm) and NLP negative (nodule not reported), were outputted for review by 2 chest radiologists.

RESULTS

The CV algorithm detected nodules that are 6 mm or greater in 1830 (36.3%) of 5047 cases. Three hundred fifty-five (19.4%) were unreported by the radiologist, as per NLP algorithm. Expert review determined that 139 (39.2%) of 355 cases were true positives (2.8% of all cases). One hundred thirty (36.7%) of 355 cases were unnecessary alerts-vague language in the report confounded the NLP algorithm. Eighty-six (24.2%) of 355 cases were false positives.

CONCLUSIONS

Dual-AI platform detected actionable unreported nodules in 2.8% of chest CT scans, yet minimized intrusion to radiologist's workflow by avoiding alerts for most already-reported nodules.

Radiologist performance in the detection of lung cancer using CT

AIM

To measure the level of radiologists' performance in lung cancer detection, and to explore radiologists' performance in cancer specialised and non-specialised centres.

MATERIALS AND METHODS

Thirty radiologists read 60 chest computed tomography (CT) examinations. Thirty cases had surgically or biopsy-proven lung cancer and 30 were cancer-free cases. The cancer cases were validated by four expert radiologists who located the malignant lung nodules. Reader performance was evaluated by calculating sensitivity, location sensitivity, specificity, and area under the receiver operating characteristic (ROC) curve (AUC). In addition, sensitivity at fixed specificity (0.794) was computed from each reader's estimated ROC curve.

RESULTS

The radiologists had a mean sensitivity of 0.749, sensitivity at fixed specificity of 0.744, location sensitivity of 0.666, specificity of 0.81 and AUC of 0.846. Radiologists in the specialised and non-specialised cancer centres had the following (specialised, non-specialised) pairs of values: sensitivity=(0.80, 0.719); sensitivity for fixed 0.794 specificity=(0.752, 0.740); location sensitivity=(0.712, 0.637); specificity=(0.794, 0.82) and AUC=(0.846, 0.846).

CONCLUSION

The efficacy of radiologists was comparable to other studies. Furthermore, AUC outcomes were similar for specialised and non-specialised cancer centre radiologists, suggesting they have similar discriminatory ability and that the higher sensitivity and lower specificity for specialised-centre radiologists can be attributed to them being less conservative in interpreting case images.

Lung cancer screening: nodule identification and characterization

The accurate identification and characterization of small pulmonary nodules at low-dose CT is an essential requirement for the implementation of effective lung cancer screening. Individual reader detection performance is influenced by nodule characteristics and technical CT parameters but can be improved by training, the application of CT techniques, and by computer-aided techniques. However, the evaluation of nodule detection in lung cancer screening trials differs from the assessment of individual readers as it incorporates multiple readers, their inter-observer variability, reporting thresholds, and reflects the program accuracy in identifying lung cancer. Understanding detection and interpretation errors in screening trials aids in the implementation of lung cancer screening in clinical practice. Indeed, as CT screening moves to ever lower radiation doses, radiologists must be cognisant of new technical challenges in nodule assessment. Screen detected lung cancers demonstrate distinct morphological features from incidentally or symptomatically detected lung cancers. Hence characterization of screen detected nodules requires an awareness of emerging concepts in early lung cancer appearances and their impact on radiological assessment and malignancy prediction models. Ultimately many nodules remain indeterminate, but further imaging evaluation can be appropriate with judicious utilization of contrast enhanced CT or MRI techniques or functional evaluation by PET-CT.

The impact of trained radiographers as concurrent readers on performance and reading time of experienced radiologists in the UK Lung Cancer Screening (UKLS) trial

Objectives

To compare radiologists’ performance reading CTs independently with their performance using radiographers as concurrent readers in lung cancer screening.

Methods

369 consecutive baseline CTs performed for the UK Lung Cancer Screening (UKLS) trial were double-read by radiologists reading either independently or concurrently with a radiographer. In concurrent reading, the radiologist reviewed radiographer-identified nodules and then detected any additional nodules. Radiologists recorded their independent and concurrent reading times. For each radiologist, sensitivity, average false-positive detections (FPs) per case and mean reading times for each method were calculated.

Results

694 nodules in 246/369 (66.7%) studies comprised the reference standard. Radiologists’ mean sensitivity and average FPs per case both increased with concurrent reading compared to independent reading (90.8 ± 5.6% vs. 77.5 ± 11.2%, and 0.60 ± 0.53 vs. 0.33 ± 0.20, respectively; p < 0.05 for 3/4 and 2/4 radiologists, respectively). The mean reading times per case decreased from 9.1 ± 2.3 min with independent reading to 7.2 ± 1.0 min with concurrent reading, decreasing significantly for 3/4 radiologists (p < 0.05).

Conclusions

The majority of radiologists demonstrated improved sensitivity, a small increase in FP detections and a statistically significantly reduced reading time using radiographers as concurrent readers.

Key Points

• Radiographers as concurrent readers could improve radiologists’ sensitivity in lung nodule detection.

• An increase in false-positive detections with radiographer-assisted concurrent reading occurred.

• The false-positive detection rate was still lower than reported for computer-aided detection.

• Concurrent reading with radiographers was also faster than single reading.

• The time saved per case using concurrently reading radiographers was relatively modest.

Electronic supplementary material

The online version of this article (doi:10.1007/s00330-017-4903-z) contains supplementary material, which is available to authorized users.

Comparing the performance of trained radiographers against experienced radiologists in the UK lung cancer screening (UKLS) trial

OBJECTIVE

To compare the performance of radiographers against that of radiologists for CT lung nodule detection in the UK Lung Cancer Screening (UKLS) pilot trial.

METHODS

Four radiographers, trained in CT nodule detection, and three radiologists were prospectively evaluated. 290 CTs performed for the UKLS were independently read by 2 radiologists and 2 radiographers. The reference standard comprised all radiologist-identified positive nodules after arbitration of discrepancies. For each radiographer and radiologist, relative sensitivity and average false positives (FPs) per case were compared for all cases read, as well as for subsets of cases read by each radiographer-radiologist combination (10 combinations).

RESULTS

599 nodules in 209/290 (72.1%) CT studies comprised the reference standard. The relative mean (±standard deviation) sensitivity of the four radiographers was 71.6 ± 8.5% compared with 83.3 ± 8.1% for the three radiologists. Radiographers were less sensitive and detected more FPs per case than radiologists in 7/10 and 8/10 radiographer-radiologist combinations, respectively (ranges of difference 11.2-33.8% and 0.4-2.6; p < 0.05). In 3/10 and 2/10 combinations, there was no difference in sensitivity and FPs per case between radiographers and radiologists. For nodules ≥100 mm(3) in volume or ≥5 mm in maximum diameter, radiographers were relatively less sensitive than radiologists in only 5/10 radiographer-radiologist combinations (range of difference 16.1-30.6%; p < 0.05) and not significantly different in the remaining 5/10 combinations.

CONCLUSION

Although overall radiographer performance was lower than that of experienced radiologists in this study, some radiographer performances were comparable with that of radiologists.

ADVANCES IN KNOWLEDGE

Overall, radiographers were less sensitive than radiologists reading the same CTs and also displayed higher average FP detections per case when compared with a reference standard derived from radiologist readings. However, some radiographers compared favourably with radiologists, especially when considering larger potentially clinically relevant nodules. Thus, while probably not sensitive enough to function as first readers, radiographers may still be able to fulfil the role of an assistant reader-that is, as a first or concurrent reader, who presents detected nodules for verification to a reading radiologist.

Inter- and intrascanner variability of pulmonary nodule volumetry on low-dose 64-row CT: an anthropomorphic phantom study

OBJECTIVE

To assess inter- and intrascanner variability in volumetry of solid pulmonary nodules in an anthropomorphic thoracic phantom using low-dose CT.

METHODS

Five spherical solid artificial nodules [diameters 3, 5, 8, 10 and 12 mm; CT density +100 Hounsfield units (HU)] were randomly placed inside an anthropomorphic thoracic phantom in different combinations. The phantom was examined on two 64-row multidetector CT (64-MDCT) systems (CT-A and CT-B) from different vendors with a low-dose protocol. Each CT examination was performed three times. The CT examinations were evaluated twice by independent blinded observers. Nodule volume was semi-automatically measured by dedicated software. Interscanner variability was evaluated by Bland-Altman analysis and expressed as 95% confidence interval (CI) of relative differences. Intrascanner variability was expressed as 95% CI of relative variation from the mean.

RESULTS

No significant difference in CT-derived volume was found between CT-A and CT-B, except for the 3-mm nodules (p<0.05). The 95% CI of interscanner variability was within ±41.6%, ±18.2% and ±4.9% for 3, 5 and ≥8 mm nodules, respectively. The 95% CI of intrascanner variability was within ±28.6%, ±13.4% and ±2.6% for 3, 5 and ≥8 mm nodules, respectively.

CONCLUSION

Different 64-MDCT scanners in low-dose settings yield good agreement in volumetry of artificial pulmonary nodules between 5 mm and 12 mm in diameter. Inter- and intrascanner variability decreases at a larger nodule size to a maximum of 4.9% for ≥8 mm nodules.

ADVANCES IN KNOWLEDGE

The commonly accepted cut-off of 25% to determine nodule growth has the potential to be reduced for ≥8 mm nodules. This offers the possibility of reducing the interval for repeated CT scans in lung cancer screenings.