Influence of computer-aided detection false-positives on reader performance and diagnostic confidence for CT colonography

Automated detection of pulmonary embolism from CT-angiograms using deep learning

Background

The aim of this study was to develop and evaluate a deep neural network model in the automated detection of pulmonary embolism (PE) from computed tomography pulmonary angiograms (CTPAs) using only weakly labelled training data.

Methods

We developed a deep neural network model consisting of two parts: a convolutional neural network architecture called InceptionResNet V2 and a long-short term memory network to process whole CTPA stacks as sequences of slices. Two versions of the model were created using either chest X-rays (Model A) or natural images (Model B) as pre-training data. We retrospectively collected 600 CTPAs to use in training and validation and 200 CTPAs to use in testing. CTPAs were annotated only with binary labels on both stack- and slice-based levels. Performance of the models was evaluated with ROC and precision–recall curves, specificity, sensitivity, accuracy, as well as positive and negative predictive values.

Results

Both models performed well on both stack- and slice-based levels. On the stack-based level, Model A reached specificity and sensitivity of 93.5% and 86.6%, respectively, outperforming Model B slightly (specificity 90.7% and sensitivity 83.5%). However, the difference between their ROC AUC scores was not statistically significant (0.94 vs 0.91, p = 0.07).

Conclusions

We show that a deep learning model trained with a relatively small, weakly annotated dataset can achieve excellent performance results in detecting PE from CTPAs.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12880-022-00763-z.

Artificial Intelligence Compared to Radiologists for the Initial Diagnosis of Prostate Cancer on Magnetic Resonance Imaging: A Systematic Review and Recommendations for Future Studies

Computer-aided diagnosis (CAD) of prostate cancer on multiparametric magnetic resonance imaging (mpMRI), using artificial intelligence (AI), may reduce missed cancers and unnecessary biopsies, increase inter-observer agreement between radiologists, and alleviate pressures caused by rising case incidence and a shortage of specialist radiologists to read prostate mpMRI. However, well-designed evaluation studies are required to prove efficacy above current clinical practice. A systematic search of the MEDLINE, EMBASE, and arXiv electronic databases was conducted for studies that compared CAD for prostate cancer detection or classification on MRI against radiologist interpretation and a histopathological reference standard, in treatment-naïve men with a clinical suspicion of prostate cancer. Twenty-seven studies were included in the final analysis. Due to substantial heterogeneities in the included studies, a narrative synthesis is presented. Several studies reported superior diagnostic accuracy for CAD over radiologist interpretation on small, internal patient datasets, though this was not observed in the few studies that performed evaluation using external patient data. Our review found insufficient evidence to suggest the clinical deployment of artificial intelligence algorithms at present. Further work is needed to develop and enforce methodological standards, promote access to large diverse datasets, and conduct prospective evaluations before clinical adoption can be considered.

Interobserver Variation of Colonic Polyp Measurement at Computed Tomography Colonography

BACKGROUND

The concept of "advanced polyps" is well accepted and is defined as polyps ≥10 mm and/or those having a villous component and/or demonstrating areas of dysplasia. Of these parameters, computed tomography colonography (CTC) can only document size. The accepted management of CTC-detected "advanced polyps" is to recommend excision if feasible, whereas the management of "intermediate" (6-9 mm) polyps is more controversial, and interval surveillance may be acceptable. Therefore, distinction between 6-9 mm and ≥10 mm is important.

METHODS

Datasets containing 26 polyps originally reported as between 8-12 mm in diameter were reviewed independently by 4 CTC-accredited radiologists. Observers tabulated the largest measurement for each polyp on axial, coronal, sagittal, and endoluminal views at lung-window settings. These measurements were also compared to those determined by the computer-aided detection (CAD) software.

RESULTS

The interobserver reliability intra-class correlation coefficient (ICC) for sagittal projection was 0.80 ("excellent" category of Hosmer and Lemeshow [2004]), 0.71 for axial ("acceptable"), 0.69 for coronal, and 0.41 for endoluminal ("unacceptable"). The largest of sagittal/axial/coronal measurement gave the best reliability with the smallest variance (ICC = 0.80; 95% CI 0.67-0.89). For 8 of 26 polyps, at least one radiologist's measurement placed the polyp in a different category compared to a colleague. For the majority of the polyps, the CAD significantly overestimated the readings compared to the largest of the manual measurements with an average difference of 1.6 mm (P < .0001 for sagittal/axial/coronal). This resulted in 33% of polyps falling into a different category-10% were lower and 23% were higher (P < .034).

CONCLUSION

It is apparent that around the cutoff point of 10 mm between "advanced" and "intermediate" polyps, interobserver performance is variable.

Diagnostic Performance and Patient Acceptance of Reduced-Laxative CT Colonography for the Detection of Polypoid and Non-Polypoid Neoplasms: A Multicenter Prospective Trial

Purpose To evaluate the diagnostic accuracy and patient acceptance of reduced-laxative computed tomographic (CT) colonography without computer-aided detection (CAD) for the detection of colorectal polypoid and non-polypoid neoplasms in a population with a positive recent fecal immunochemical test (FIT). Materials and Methods Institutional review board approval and written informed consent were obtained. This multicenter prospective trial enrolled patients who had positive FIT results. Reduced-laxative CT colonography and colonoscopy were performed on the same day. Patients received 380 mL polyethylene glycol solution, 20 mL iodinated oral contrast agent, and two doses of 20 mg mosapride the day before CT colonography. The main outcome measures were the accuracy of CT colonography for the detection of neoplasms 6 mm or larger in per-patient and per-lesion analyses and a survey of patient perceptions regarding the preparation and examination. The Clopper-Pearson method was used for assessing the 95% confidence intervals of per-patient and per-lesion accuracy. Survey scores were analyzed by using the Wilcoxon and χ² tests. Results Three hundred four patients underwent both CT colonography and colonoscopy. Per-patient sensitivity, specificity, positive predictive value, and negative predictive value of CT colonography for detecting neoplasms 10 mm or larger were 0.91 (40 of 44), 0.99 (255 of 258), 0.93 (40 of 43), and 0.98 (255 of 259), respectively; these values for neoplasms 6 mm or larger were 0.90 (71 of 79), 0.93 (207 of 223), 0.82 (71 of 87), and 0.96 (207 of 215), respectively. Per-lesion sensitivities for detection of polypoid and non-polypoid neoplasms 10 mm or larger were 0.95 (40 of 42) and 0.67 (six of nine), respectively; those for neoplasms 6 mm or larger were 0.90 (104 of 115) and 0.38 (eight of 21), respectively (P < .05 for both). Patient acceptance of preparation and examination with CT colonography was significantly higher than that with colonoscopy, and 62% (176 of 282) of patients would choose CT colonography as the first examination if they have a positive FIT result in the future. Conclusion Reduced-laxative CT colonography without CAD is accurate in the detection of polypoid neoplasms 6 mm or larger but is less accurate in the detection of non-polypoid neoplasms. Reduced-laxative CT colonography has high patient acceptance and is an efficient triage examination for patients with a positive FIT. ^© RSNA, 2016 Online supplemental material is available for this article.

Evaluation of computer-aided detection and diagnosis systems

Computer-aided detection and diagnosis (CAD) systems are increasingly being used as an aid by clinicians for detection and interpretation of diseases. Computer-aided detection systems mark regions of an image that may reveal specific abnormalities and are used to alert clinicians to these regions during image interpretation. Computer-aided diagnosis systems provide an assessment of a disease using image-based information alone or in combination with other relevant diagnostic data and are used by clinicians as a decision support in developing their diagnoses. While CAD systems are commercially available, standardized approaches for evaluating and reporting their performance have not yet been fully formalized in the literature or in a standardization effort. This deficiency has led to difficulty in the comparison of CAD devices and in understanding how the reported performance might translate into clinical practice. To address these important issues, the American Association of Physicists in Medicine (AAPM) formed the Computer Aided Detection in Diagnostic Imaging Subcommittee (CADSC), in part, to develop recommendations on approaches for assessing CAD system performance. The purpose of this paper is to convey the opinions of the AAPM CADSC members and to stimulate the development of consensus approaches and "best practices" for evaluating CAD systems. Both the assessment of a standalone CAD system and the evaluation of the impact of CAD on end-users are discussed. It is hoped that awareness of these important evaluation elements and the CADSC recommendations will lead to further development of structured guidelines for CAD performance assessment. Proper assessment of CAD system performance is expected to increase the understanding of a CAD system's effectiveness and limitations, which is expected to stimulate further research and development efforts on CAD technologies, reduce problems due to improper use, and eventually improve the utility and efficacy of CAD in clinical practice.

CAD-associated reader error in CT colonography

RATIONALE AND OBJECTIVES

Computed tomographic colonographic interpretation with computer-aided detection (CAD) may be superior to unaided viewing, although polyp characteristics may influence accuracy. Reader error due to polyp characteristics was evaluated in a multiple-case, multiple-reader trial of computed tomographic colonography with CAD.

MATERIALS AND METHODS

Two experts retrospectively reviewed 52 positive cases (74 polyps) and categorized them as hard, moderate, or easy to detect. Each case was evaluated without and with CAD. Features that may influence a reader's ability to detect a polyp or to accept or reject a CAD mark were tabulated. The association between polyp characteristics and detection rates in the trial was assessed. The difference in detection rates (CAD vs unassisted) was calculated, and regression analysis was performed.

RESULTS

Of 64 polyps found by CAD, experts categorized 20 as hard, 28 as moderate, and 16 as easy to detect. Reader characterization errors predominated (47.3%) over other errors. Factors associated with lower detection rates included small size, flat morphology, and resemblance to a thickened fold. CAD was superior for polyps resembling lipomas compared to those that did not resemble lipomas (average increase in detection rate with CAD, 12.8% vs 5.5%; P < .05).

CONCLUSIONS

Polyp characteristic may impair computed tomographic colonographic interpretation augmented by CAD. Readers can avoid errors of measurement by evaluating diminutive polyp candidates with sample measurements. Caution should be taken when evaluating focally thick folds and when using visual impression to dismiss a polyp candidate as a lipoma when it is submerged in densely tagged fluid.

Sources of false positives in computer-assisted CT colonography

The application of computer-aided detection (CAD) is expected to improve reader sensitivity and to reduce inter-observer variance in computed tomographic (CT) colonography. However, current CAD systems display a large number of false-positive (FP) detections. The reviewing of a large number of FP CAD detections increases interpretation time, and it may also reduce the specificity and/or sensitivity of a computer-assisted reader. Therefore, it is important to be aware of the patterns and pitfalls of FP CAD detections. This pictorial essay reviews common sources of FP CAD detections that have been observed in the literature and in our experiments in computer-assisted CT colonography. Also the recommended computer-assisted reading technique is described.

Colorectal cancer screening in women: an underutilized lifesaver

OBJECTIVE

Colorectal cancer (CRC) represents the third most common cancer diagnosed and a major cause of cancer-related deaths in women. Despite strong evidence that early screening decreases colorectal cancer incidence and mortality rates, colorectal cancer screening rates in women still lag significantly behind screening rates for breast and cervical cancers. Additionally, women have been found to be less likely than men to undergo CRC screening. This is despite the fact that the overall lifetime risk for the development of colorectal carcinoma is similar in both sexes. Barriers to screening have been found to be different for women compared with men. Screening adherence in women also appears to be associated with various social and demographic factors.

CONCLUSION

CT colonography (CTC) is an accurate, minimally invasive, and well-tolerated examination that is newly endorsed by the American Cancer Society, U.S. Multisociety Task Force, and the American College of Radiology. Improved screening compliance may occur in women with further dissemination of CTC.

Effect of computer-aided detection for CT colonography in a multireader, multicase trial

PURPOSE

To assess the effect of using computer-aided detection (CAD) in second-read mode on readers' accuracy in interpreting computed tomographic (CT) colonographic images.

MATERIALS AND METHODS

The contributing institutions performed the examinations under approval of their local institutional review board, with waiver of informed consent, for this HIPAA-compliant study. A cohort of 100 colonoscopy-proved cases was used: In 52 patients with findings positive for polyps, 74 polyps of 6 mm or larger were observed in 65 colonic segments; in 48 patients with findings negative for polyps, no polyps were found. Nineteen blinded readers interpreted each case at two different times, with and without the assistance of a commercial CAD system. The effect of CAD was assessed in segment-level and patient-level receiver operating characteristic (ROC) curve analyses.

RESULTS

Thirteen (68%) of 19 readers demonstrated higher accuracy with CAD, as measured with the segment-level area under the ROC curve (AUC). The readers' average segment-level AUC with CAD (0.758) was significantly greater (P = .015) than the average AUC in the unassisted read (0.737). Readers' per-segment, per-patient, and per-polyp sensitivity for all polyps of 6 mm or larger was higher (P < .011, .007, .005, respectively) for readings with CAD compared with unassisted readings (0.517 versus 0.465, 0.521 versus 0.466, and 0.477 versus 0.422, respectively). Sensitivity for patients with at least one large polyp of 10 mm or larger was also higher (P < .047) with CAD than without (0.777 versus 0.743). Average reader sensitivity also improved with CAD by more than 0.08 for small adenomas. Use of CAD reduced specificity of readers by 0.025 (P = .05).

CONCLUSION

Use of CAD resulted in a significant improvement in overall reader performance. CAD improves reader sensitivity when measured per segment, per patient, and per polyp for small polyps and adenomas and also reduces specificity by a small amount.

Electronic cleansing for CT colonography: does it help CAD software performance in a high-risk population for colorectal cancer?

OBJECTIVE

To compare the performance of computer-aided detection (CAD) for CT colonography (CTC) with and without electronic cleansing (EC) in a high-risk population tagged with a faecal tagging (FT) protocol.

METHODS

Thirty-two patients underwent CTC followed by same-day colonoscopy. All patients underwent bowel preparation and FT with barium and gastrografin. Each CTC dataset was processed with colon CAD with and without EC. Per-polyp sensitivity was calculated. The average number of false-positive (FP) results and their causes were also analysed and compared.

RESULTS

Eighty-six polyps were detected in 29 patients. Per-polyp sensitivities of CAD with EC (93.8% and 100%) were higher than those without EC (84.4% and 87.5%) for polyps >or=6 mm and >or=10 mm, respectively. However, the differences were not significant. The average number (6.3) of FPs of CAD with EC was significantly larger than that (3.1) without EC. The distribution of FPs in both CAD settings was also significantly different. The most common cause of FPs was the ileocaecal valve in both datasets. However, untagged faeces was a significantly less common cause of FPs with EC, EC-related artefacts being more common.

CONCLUSION

Electronic cleansing has the potential to improve per-polyp sensitivity of CTC CAD, although the significantly larger number of FPs with EC remains to be improved.

ACR Colon Cancer Committee white paper: status of CT colonography 2009

PURPOSE

To review the current status and rationale of the updated ACR practice guidelines for CT colonography (CTC).

METHODS

Clinical validation trials in both the United States and Europe are reviewed. Key technical aspects of the CTC examination are emphasized, including low-dose protocols, proper insufflation, and bowel preparation. Important issues of implementation are discussed, including training and certification, definition of the target lesion, reporting of colonic and extracolonic findings, quality metrics, reimbursement, and cost-effectiveness.

RESULTS

Successful validation trials in screening cohorts both in the United States with ACRIN and in Germany demonstrated sensitivity > or = 90% for patients with polyps >10 mm. Proper technique is critical, including low-dose techniques in screening cohorts, with an upper limit of the CT dose index by volume of 12.5 mGy per examination. Training new readers includes the requirement of interactive workstation training with 2-D and 3-D image display techniques. The target lesion is defined as a polyp > or = 6 mm, consistent with the American Cancer Society joint guidelines. Five quality metrics have been defined for CTC, with pilot data entered. Although the CMS national noncoverage decision in May 2009 was a disappointment, multiple third-party payers are reimbursing for screening CTC. Cost-effective modeling has shown CTC to be a dominant strategy, including in a Medicare cohort.

CONCLUSION

Supported by third-party payer reimbursement for screening, CTC will continue to further transition into community practice and can provide an important adjunctive examination for colorectal screening.

Polyp characteristics correctly annotated by computer-aided detection software but ignored by reporting radiologists during CT colonography

PURPOSE

To retrospectively describe the characteristics of polyps incorrectly dismissed by radiologists despite appropriate computer-aided detection (CAD) prompting during computed tomographic (CT) colonography.

MATERIALS AND METHODS

Ethics committee approval and patient informed consent were obtained from institutions that provided the data sets used in this HIPAA-compliant study. A total of 111 polyps that had a diameter of at least 6 mm and were detected with CAD were collated from three previous studies in which researchers investigated radiologist performance with and without CAD (total, 25 readers). Two new observers graded each polyp with predefined criteria, including polyp size, morphology, and location; data set quality; ease of visualization; tagging use and polyp coating; colonic curvature; CAD mark obscuration; and number of false-positive findings. The 86 polyps that were missed before CAD (those that were unreported by one or more original readers) were divided into those that remained unreported after CAD (no CAD gain, n = 36) and those that were reported correctly by at least one additional reader (CAD gain, n = 50). Logistic-regression analysis and the Fisher exact and Mann-Whitney tests were used to compare the results of both groups with each other and with a control group of 25 polyps, all of which were detected by readers without CAD.

RESULTS

Before CAD, polyps 10 mm in diameter or larger, those that were rated easy to visualize, and those that were uncoated by tagged fluid were less likely to be missed (72%, 76%, and 80% of control polyps vs 43%, 43%, and 59% of missed polyps, respectively; P < .001, P < .01, and P < .03, respectively). After CAD, the odds of CAD gain decreased with increasing polyp size (odds ratio, 0.92; 95% confidence interval: 0.85, 1.00; P = .04) and irregular morphology (odds ratio, 0.28; 95% confidence interval: 0.08, 0.92; P = .04).

CONCLUSION

Larger irregular polyps are a common source of incorrect radiologist dismissal, despite correct CAD prompting.