Virtual dissection CT colonography: evaluation of learning curves and reading times with and without computer-aided detection

The effect of computer-aided detection markers on visual search and reader performance during concurrent reading of CT colonography

OBJECTIVE

We aimed to identify the effect of computer-aided detection (CAD) on visual search and performance in CT Colonography (CTC) of inexperienced and experienced readers.

METHODS

Fifteen endoluminal CTC examinations were recorded, each with one polyp, and two videos were generated, one with and one without a CAD mark. Forty-two readers (17 experienced, 25 inexperienced) interpreted the videos during infrared visual search recording. CAD markers and polyps were treated as regions of interest in data processing. This multi-reader, multi-case study was analysed using multilevel modelling.

RESULTS

CAD drew readers' attention to polyps faster, accelerating identification times: median 'time to first pursuit' was 0.48 s (IQR 0.27 to 0.87 s) with CAD, versus 0.58 s (IQR 0.35 to 1.06 s) without. For inexperienced readers, CAD also held visual attention for longer. All visual search metrics used to assess visual gaze behaviour demonstrated statistically significant differences when "with" and "without" CAD were compared. A significant increase in the number of correct polyp identifications across all readers was seen with CAD (74 % without CAD, 87 % with CAD; p < 0.001).

CONCLUSIONS

CAD significantly alters visual search and polyp identification in readers viewing three-dimensional endoluminal CTC. For polyp and CAD marker pursuit times, CAD generally exerted a larger effect on inexperienced readers.

KEY POINTS

• Visual gaze is attracted by computer-assisted detection (CAD) marks on polyps • Inexperienced readers' gaze is affected more by CAD than experienced readers. • CAD marks could mean that the unannotated endoluminal surface is relatively neglected. • Correct polyp identification is increased significantly by CAD.

Digital tomosynthesis for evaluating metastatic lung nodules: nodule visibility, learning curves, and reading times

OBJECTIVE

To evaluate nodule visibility, learning curves, and reading times for digital tomosynthesis (DT).

MATERIALS AND METHODS

We included 80 patients who underwent computed tomography (CT) and DT before pulmonary metastasectomy. One experienced chest radiologist annotated all visible nodules on thin-section CT scans using computer-aided detection software. Two radiologists used CT as the reference standard and retrospectively graded the visibility of nodules on DT. Nodule detection performance was evaluated in four sessions of 20 cases each by six readers. After each session, readers were unblinded to the DT images by revealing the true-positive markings and were instructed to self-analyze their own misreads. Receiver-operating-characteristic curves were determined.

RESULTS

Among 414 nodules on CT, 53.3% (221/414) were visible on DT. The main reason for not seeing a nodule on DT was small size (93.3%, ≤ 5 mm). DT revealed a substantial number of malignant nodules (84.1%, 143/170). The proportion of malignant nodules among visible nodules on DT was significantly higher (64.7%, 143/221) than that on CT (41.1%, 170/414) (p < 0.001). Area under the curve (AUC) values at the initial session were > 0.8, and the average detection rate for malignant nodules was 85% (210/246). The inter-session analysis of the AUC showed no significant differences among the readers, and the detection rate for malignant nodules did not differ across sessions. A slight improvement in reading times was observed.

CONCLUSION

Most malignant nodules > 5 mm were visible on DT. As nodule detection performance was high from the initial session, DT may be readily applicable for radiology residents and board-certified radiologists.

Test-positive rate at CT colonography is increased by rectal bleeding and/or unexplained weight loss, unlike other common gastrointestinal symptoms

PURPOSE

We evaluated the rate of significant colonic and extra-colonic abnormalities at computed tomography colonography (CTC), according to symptoms and age.

MATERIALS AND METHODS

We retrospectively evaluated 7361 consecutive average-risk subjects (3073 males, average age: 60.3 ± 13.9; range 18-96 years) for colorectal cancer (CRC) who underwent CTC. They were divided into three groups according to clinical symptoms: 1343 asymptomatic individuals (group A), 899 patients with at least one "alarm" symptom for CRC, including rectal bleeding and unexplained weight loss (group C), and 5119 subjects with other gastrointestinal symptoms (group B). Diagnostic and test-positive rates of CTC were established using optical colonoscopy (OC) and/or surgery as reference standard. In addition, clinically significant extra-colonic findings were noted.

RESULTS

903 out of 7361 (12%, 95% confidence interval (CI) 0.11-0.13) subjects had at least one clinically significant colonic finding at CTC. CTC true positive fraction and false positive fraction were respectively 637/642 (99.2%, 95%CI 0.98-0.99) and 55/692 (7.95%, 95%CI 0.05-0.09). The pooled test-positive rate in group C (138/689, 20.0%, 95%CI 0.17-0.23) was significantly higher than in both groups A (79/1343, 5.9%, 95%CI 0.04-0.07) and B (420/5329, 7.5%, 95%CI 0.07-0.08) (p < 0.001). Aging and male gender were associated to a higher test positive rate. The rate of clinically significant extra-colonic findings was significantly higher in group C (44/689, 6.4%, 95%CI 0.04-0.08) versus groups A (26/1343, 1.9%, 95%CI 0.01-0.02) and B (64/5329, 1.2%, 95%CI 0.01-0.02) (p < 0.001).

CONCLUSION

Both test-positive and significant extra-colonic finding rates at CTC are significantly increased in the presence of "alarm" gastrointestinal symptoms especially in older patients.

Virtual colonoscopy: Utility, impact and overview

Computed tomography (CT) colonoscopy is a well-established technique for evaluation of colorectal cancer. Significant advances have been made in the technique of CT colonoscopy since its inception. Excellent results can be achieved in detecting both colorectal cancer and significant sized polyps as long as a meticulous technique is adopted while performing CT colonoscopy. Furthermore, it is important to realize that there is a learning curve involved in interpreting these studies and adequate experience is essential to achieve high sensitivity and specificity with this technique. Indications, contraindications, technique and interpretation, including potential pitfalls in CT colonoscopy imaging, are reviewed in this article. Recent advances and the current role of CT colonoscopy in colorectal cancer screening are also discussed.

Distributed human intelligence for colonic polyp classification in computer-aided detection for CT colonography

PURPOSE

To assess the diagnostic performance of distributed human intelligence for the classification of polyp candidates identified with computer-aided detection (CAD) for computed tomographic (CT) colonography.

MATERIALS AND METHODS

This study was approved by the institutional Office of Human Subjects Research. The requirement for informed consent was waived for this HIPAA-compliant study. CT images from 24 patients, each with at least one polyp of 6 mm or larger, were analyzed by using CAD software to identify 268 polyp candidates. Twenty knowledge workers (KWs) from a crowdsourcing platform labeled each polyp candidate as a true or false polyp. Two trials involving 228 KWs were conducted to assess reproducibility. Performance was assessed by comparing the area under the receiver operating characteristic curve (AUC) of KWs with the AUC of CAD for polyp classification.

RESULTS

The detection-level AUC for KWs was 0.845 ± 0.045 (standard error) in trial 1 and 0.855 ± 0.044 in trial 2. These were not significantly different from the AUC for CAD, which was 0.859 ± 0.043. When polyp candidates were stratified by difficulty, KWs performed better than CAD on easy detections; AUCs were 0.951 ± 0.032 in trial 1, 0.966 ± 0.027 in trial 2, and 0.877 ± 0.048 for CAD (P = .039 for trial 2). KWs who participated in both trials showed a significant improvement in performance going from trial 1 to trial 2; AUCs were 0.759 ± 0.052 in trial 1 and 0.839 ± 0.046 in trial 2 (P = .041).

CONCLUSION

The performance of distributed human intelligence is not significantly different from that of CAD for colonic polyp classification.

Evaluation of a standardized CT colonography training program for novice readers

PURPOSE

To determine how many computed tomographic (CT) colonography training studies have to be evaluated by novice readers to obtain an adequate level of competence in polyp detection.

MATERIALS AND METHODS

The study was approved by the Institutional Review Board. Informed consent was obtained from all participants. Six physicians (one radiologist, three radiology residents, two researchers) and three technicians completed a CT colonography training program. Two hundred CT colonographic examinations with colonoscopic verification were selected from a research database, with 100 CT colonographic examinations with at least one polyp 6 mm or larger. After a lecture session and short individual hands-on training, CT colonography training was done individually with immediate feedback of colonoscopy outcome. Per-polyp sensitivity was calculated for four sets of 50 CT colonographic examinations for lesions 6 mm or larger. By using logistic regression analyses, the number of CT colonographic examinations to reach 90% sensitivity for lesions 6 mm or larger was estimated. Reading times were registered.

RESULTS

The average per-polyp sensitivity for lesions 6 mm or larger was 76% (207 of 270) in the first set of 50 CT colonographic examinations, 77% (262 of 342) in the second (P = .96 vs first set), 80% (310 of 387) in the third (P = .67 vs first set), and 91% (261 of 288) in the fourth (P = .018). The estimated number of CT colonographic examinations for a sufficient sensitivity was 164. Six of nine readers reached this level of competence within 175 CT colonographic examinations. Reading times decreased significantly from the first to the second set of 50 CT colonographic examinations for six readers.

CONCLUSION

Novice CT colonography readers obtained sensitivity equal to that of experienced readers after practicing on average 164 CT colonographic studies.

Effect of computer-aided detection for CT colonography in a multireader, multicase trial

PURPOSE

To assess the effect of using computer-aided detection (CAD) in second-read mode on readers' accuracy in interpreting computed tomographic (CT) colonographic images.

MATERIALS AND METHODS

The contributing institutions performed the examinations under approval of their local institutional review board, with waiver of informed consent, for this HIPAA-compliant study. A cohort of 100 colonoscopy-proved cases was used: In 52 patients with findings positive for polyps, 74 polyps of 6 mm or larger were observed in 65 colonic segments; in 48 patients with findings negative for polyps, no polyps were found. Nineteen blinded readers interpreted each case at two different times, with and without the assistance of a commercial CAD system. The effect of CAD was assessed in segment-level and patient-level receiver operating characteristic (ROC) curve analyses.

RESULTS

Thirteen (68%) of 19 readers demonstrated higher accuracy with CAD, as measured with the segment-level area under the ROC curve (AUC). The readers' average segment-level AUC with CAD (0.758) was significantly greater (P = .015) than the average AUC in the unassisted read (0.737). Readers' per-segment, per-patient, and per-polyp sensitivity for all polyps of 6 mm or larger was higher (P < .011, .007, .005, respectively) for readings with CAD compared with unassisted readings (0.517 versus 0.465, 0.521 versus 0.466, and 0.477 versus 0.422, respectively). Sensitivity for patients with at least one large polyp of 10 mm or larger was also higher (P < .047) with CAD than without (0.777 versus 0.743). Average reader sensitivity also improved with CAD by more than 0.08 for small adenomas. Use of CAD reduced specificity of readers by 0.025 (P = .05).

CONCLUSION

Use of CAD resulted in a significant improvement in overall reader performance. CAD improves reader sensitivity when measured per segment, per patient, and per polyp for small polyps and adenomas and also reduces specificity by a small amount.

Improving the accuracy of CTC interpretation: computer-aided detection

Computer-aided polyp detection aims to improve the accuracy of the colonography interpretation. The computer searches the colonic wall to look for polyplike protrusions and presents a list of suspicious areas to a physician for further analysis. Computer-aided polyp detection has developed rapidly in the past decade in the laboratory setting and has sensitivities comparable with those of experts. Computer-aided polyp detection tends to help inexperienced readers more than experienced ones and may also lead to small reductions in specificity. In its currently proposed use as an adjunct to standard image interpretation, computer-aided polyp detection serves as a spellchecker rather than an efficiency enhancer.

ACR Colon Cancer Committee white paper: status of CT colonography 2009

PURPOSE

To review the current status and rationale of the updated ACR practice guidelines for CT colonography (CTC).

METHODS

Clinical validation trials in both the United States and Europe are reviewed. Key technical aspects of the CTC examination are emphasized, including low-dose protocols, proper insufflation, and bowel preparation. Important issues of implementation are discussed, including training and certification, definition of the target lesion, reporting of colonic and extracolonic findings, quality metrics, reimbursement, and cost-effectiveness.

RESULTS

Successful validation trials in screening cohorts both in the United States with ACRIN and in Germany demonstrated sensitivity > or = 90% for patients with polyps >10 mm. Proper technique is critical, including low-dose techniques in screening cohorts, with an upper limit of the CT dose index by volume of 12.5 mGy per examination. Training new readers includes the requirement of interactive workstation training with 2-D and 3-D image display techniques. The target lesion is defined as a polyp > or = 6 mm, consistent with the American Cancer Society joint guidelines. Five quality metrics have been defined for CTC, with pilot data entered. Although the CMS national noncoverage decision in May 2009 was a disappointment, multiple third-party payers are reimbursing for screening CTC. Cost-effective modeling has shown CTC to be a dominant strategy, including in a Medicare cohort.

CONCLUSION

Supported by third-party payer reimbursement for screening, CTC will continue to further transition into community practice and can provide an important adjunctive examination for colorectal screening.

Computer-aided detection (CAD) as a second reader using perspective filet view at CT colonography: effect on performance of inexperienced readers

AIM

To evaluate whether computer-aided detection (CAD) as a second reader using perspective filet view [three-dimensional (3D) filet] improves the performance of inexperienced readers at computed tomography colonography (CTC) compared with unassisted 3D filet and unassisted two-dimensional (2D) CTC.

MATERIAL AND METHODS

Fifty symptomatic patients underwent CTC and same-day colonoscopy with segmental unblinding. Two inexperienced readers read the CTC studies on 3D filet and 2D several weeks apart. Four months later, readers re-read the cases only evaluating CAD marks using 3D filet. Suspicious CAD marks not previously described on 3D filet were recorded. Jackknife free-response receiver operating characteristic (JAFROC-1) analysis was used to compare the observers' performances in detecting lesions with 3D filet, 2D and 3D filet with CAD.

RESULTS

One hundred and three lesions > or =3mm were detected at colonoscopy with segmental unblinding. CAD alone had a sensitivity of 73% (75/103) at a mean false-positive rate per patient of 12.8 in supine and 11.4 in prone. For inexperienced readers sensitivities with 3D filet with CAD were 58% (60/103) and 48% (50/103) with an improvement of 14-16 percentage points (p<0.05) compared with 2D and of 10-11 percentage points (p<0.05) compared with 3D filet. For inexperienced readers, the false-positive rate was 25-41% and 71-200% higher with 3D filet with CAD compared with 3D filet and 2D, respectively. JAFROC-1 analysis showed no significant differences in per-lesion overall performance among reading modes (p=0.8).

CONCLUSION

CAD applied as a second reader using 3D filet increased both sensitivity and the number of false positives by inexperienced readers compared with 3D filet and 2D, thus not improving overall performance, i.e., the ability to distinguish between lesions and non-lesions.