Polyp characteristics correctly annotated by computer-aided detection software but ignored by reporting radiologists during CT colonography

Improving the radiologist-CAD interaction: designing for appropriate trust

Computer-aided diagnosis (CAD) has great potential to improve radiologists' diagnostic performance. However, the reported performance of the radiologist-CAD team is lower than what might be expected based on the performance of the radiologist and the CAD system in isolation. This indicates that the interaction between radiologists and the CAD system is not optimal. An important factor in the interaction between humans and automated aids (such as CAD) is trust. Suboptimal performance of the human-automation team is often caused by an inappropriate level of trust in the automation. In this review, we examine the role of trust in the radiologist-CAD interaction and suggest ways to improve the output of the CAD system so that it allows radiologists to calibrate their trust in the CAD system more effectively. Observer studies of the CAD systems show that radiologists often have an inappropriate level of trust in the CAD system. They sometimes under-trust CAD, thereby reducing its potential benefits, and sometimes over-trust it, leading to diagnostic errors they would not have made without CAD. Based on the literature on trust in human-automation interaction and the results of CAD observer studies, we have identified four ways to improve the output of CAD so that it allows radiologists to form a more appropriate level of trust in CAD. Designing CAD systems for appropriate trust is important and can improve the performance of the radiologist-CAD team. Future CAD research and development should acknowledge the importance of the radiologist-CAD interaction, and specifically the role of trust therein, in order to create the perfect artificial partner for the radiologist. This review focuses on the role of trust in the radiologist-CAD interaction. The aim of the review is to encourage CAD developers to design for appropriate trust and thereby improve the performance of the radiologist-CAD team.

CT colonography: external clinical validation of an algorithm for computer-assisted prone and supine registration

PURPOSE

To perform external validation of a computer-assisted registration algorithm for prone and supine computed tomographic (CT) colonography and to compare the results with those of an existing centerline method.

MATERIALS AND METHODS

All contributing centers had institutional review board approval; participants provided informed consent. A validation sample of CT colonographic examinations of 51 patients with 68 polyps (6-55 mm) was selected from a publicly available, HIPAA compliant, anonymized archive. No patients were excluded because of poor preparation or inadequate distension. Corresponding prone and supine polyp coordinates were recorded, and endoluminal surfaces were registered automatically by using a computer algorithm. Two observers independently scored three-dimensional endoluminal polyp registration success. Results were compared with those obtained by using the normalized distance along the colonic centerline (NDACC) method. Pairwise Wilcoxon signed rank tests were used to compare gross registration error and McNemar tests were used to compare polyp conspicuity.

RESULTS

Registration was possible in all 51 patients, and 136 paired polyp coordinates were generated (68 polyps) to test the algorithm. Overall mean three-dimensional polyp registration error (mean ± standard deviation, 19.9 mm ± 20.4) was significantly less than that for the NDACC method (mean, 27.4 mm ± 15.1; P = .001). Accuracy was unaffected by colonic segment (P = .76) or luminal collapse (P = .066). During endoluminal review by two observers (272 matching tasks, 68 polyps, prone to supine and supine to prone coordinates), 223 (82%) polyp matches were visible (120° field of view) compared with just 129 (47%) when the NDACC method was used (P < .001). By using multiplanar visualization, 48 (70%) polyps were visible after scrolling ± 15 mm in any multiplanar axis compared with 16 (24%) for NDACC (P < .001).

CONCLUSION

Computer-assisted registration is more accurate than the NDACC method for mapping the endoluminal surface and matching the location of polyps in corresponding prone and supine CT colonographic acquisitions.

CT colonography: preliminary assessment of a double-read paradigm that uses computer-aided detection as the first reader

PURPOSE

To compare diagnostic performance and time efficiency of double-reading first-reader computer-aided detection (CAD) (DR FR CAD) followed by radiologist interpretation with that of an unassisted read using segmentally unblinded colonoscopy as reference standard.

MATERIALS AND METHODS

The local ethical committee approved this study. Written consent to use examinations was obtained from patients. Three experienced radiologists searched for polyps 6 mm or larger in 155 computed tomographic (CT) colonographic studies (57 containing 10 masses and 79 polyps ≥ 6 mm). Reading was randomized to either unassisted read or DR FR CAD. Data sets were reread 6 weeks later by using the opposite paradigm. DR FR CAD consists of evaluation of CAD prompts, followed by fast two-dimensional review for mass detection. CAD sensitivity was calculated. Readers' diagnoses and reviewing times with and without CAD were compared by using McNemar and Student t tests, respectively. Association between missed polyps and lesion characteristics was explored with multiple regression analysis.

RESULTS

With mean rate of 19 (standard deviation, 14; median, 15; range, 4-127) false-positive results per patient, CAD sensitivity was 90% for lesions 6 mm or larger. Readers' sensitivity and specificity for lesions 6 mm or larger were 74% (95% confidence interval [CI]: 65%, 84%) and 93% (95% CI: 89%, 97%), respectively, for the unassisted read and 77% (95% CI: 67%, 85%) and 90% (95% CI: 85%, 95%), respectively, for DR FR CAD (P = .343 and P = .189, respectively). Overall unassisted and DR FR CAD reviewing times were similar (243 vs 239 seconds; P = .623); DR FR CAD was faster when the number of CAD marks per patient was 20 or fewer (187 vs 220 seconds, P <01). Odds ratio of missing a polyp with CAD decreased as polyp size increased (0.6) and for polyps visible on both prone and supine scans (0.12); it increased for flat lesions (9.1).

CONCLUSION

DR FR CAD paradigm had similar performance compared with unassisted interpretation but better time efficiency when 20 or fewer CAD prompts per patient were generated.

Computer-Aided Detection of Colorectal Lesions with Super-Resolution CT Colonography: Pilot Evaluation

Reliable computer-aided detection (CADe) of small polyps and flat lesions is limited by the relatively low image resolution of computed tomographic colonography (CTC). We developed a sinogram-based super-resolution (SR) method to enhance the images of lesion candidates detected by CADe. First, CADe is used to detect lesion candidates at high sensitivity from conventional CTC images. Next, the signal patterns of the lesion candidates are enhanced in sinogram domain by use of non-uniform compressive sampling and iterative reconstruction to produce SR images of the lesion candidates. For pilot evaluation, an anthropomorphic phantom including simulated lesions was filled partially with fecal tagging and scanned by use of a CT scanner. A fully automated CADe scheme was used to detect lesion candidates in the images reconstructed at conventional 0.61-mm and at 0.10-mm SR image resolution. The proof-of-concept results indicate that the SR method has potential to reduce the number of FP CADe detections below that obtainable with the conventional CTC imaging technology.

Strategies for improved interpretation of computer-aided detections for CT colonography utilizing distributed human intelligence

Computer-aided detection (CAD) systems have been shown to improve the diagnostic performance of CT colonography (CTC) in the detection of premalignant colorectal polyps. Despite the improvement, the overall system is not optimal. CAD annotations on true lesions are incorrectly dismissed, and false positives are misinterpreted as true polyps. Here, we conduct an observer performance study utilizing distributed human intelligence in the form of anonymous knowledge workers (KWs) to investigate human performance in classifying polyp candidates under different presentation strategies. We evaluated 600 polyp candidates from 50 patients, each case having at least one polyp ≥6 mm, from a large database of CTC studies. Each polyp candidate was labeled independently as a true or false polyp by 20 KWs and an expert radiologist. We asked each labeler to determine whether the candidate was a true polyp after looking at a single 3D-rendered image of the candidate and after watching a video fly-around of the candidate. We found that distributed human intelligence improved significantly when presented with the additional information in the video fly-around. We noted that performance degraded with increasing interpretation time and increasing difficulty, but distributed human intelligence performed better than our CAD classifier for "easy" and "moderate" polyp candidates. Further, we observed numerous parallels between the expert radiologist and the KWs. Both showed similar improvement in classification moving from single-image to video interpretation. Additionally, difficulty estimates obtained from the KWs using an expectation maximization algorithm correlated well with the difficulty rating assigned by the expert radiologist. Our results suggest that distributed human intelligence is a powerful tool that will aid in the development of CAD for CTC.

CAD-associated reader error in CT colonography

RATIONALE AND OBJECTIVES

Computed tomographic colonographic interpretation with computer-aided detection (CAD) may be superior to unaided viewing, although polyp characteristics may influence accuracy. Reader error due to polyp characteristics was evaluated in a multiple-case, multiple-reader trial of computed tomographic colonography with CAD.

MATERIALS AND METHODS

Two experts retrospectively reviewed 52 positive cases (74 polyps) and categorized them as hard, moderate, or easy to detect. Each case was evaluated without and with CAD. Features that may influence a reader's ability to detect a polyp or to accept or reject a CAD mark were tabulated. The association between polyp characteristics and detection rates in the trial was assessed. The difference in detection rates (CAD vs unassisted) was calculated, and regression analysis was performed.

RESULTS

Of 64 polyps found by CAD, experts categorized 20 as hard, 28 as moderate, and 16 as easy to detect. Reader characterization errors predominated (47.3%) over other errors. Factors associated with lower detection rates included small size, flat morphology, and resemblance to a thickened fold. CAD was superior for polyps resembling lipomas compared to those that did not resemble lipomas (average increase in detection rate with CAD, 12.8% vs 5.5%; P < .05).

CONCLUSIONS

Polyp characteristic may impair computed tomographic colonographic interpretation augmented by CAD. Readers can avoid errors of measurement by evaluating diminutive polyp candidates with sample measurements. Caution should be taken when evaluating focally thick folds and when using visual impression to dismiss a polyp candidate as a lipoma when it is submerged in densely tagged fluid.

Predicting readers' diagnostic accuracy with a new CAD algorithm

RATIONALE AND OBJECTIVES

Before computer-aided detection (CAD) algorithms can be used in clinical practice, they must be shown to improve readers' diagnostic accuracy over their unaided performance. This is usually accomplished through a large multireader, multicase (MRMC) clinical trial. It is burdensome, however, for an MRMC study to be performed with each new release of a CAD algorithm. The aim of this report is to present an approach for building models to predict readers' accuracy with a new CAD algorithm.

MATERIALS AND METHODS

A modeling approach for predicting readers' results with a new CAD algorithm is described. Multiple-variable logistic regression was used to build models for readers' sensitivity and false-positive rate, given the results of an MRMC study with an older CAD algorithm and the stand-alone performance results of a new CAD algorithm. Data from a large lung MRMC CAD trial are used to illustrate the modeling approach and test the ability of the models to predict readers' accuracy with the new CAD algorithm.

RESULTS

The model overestimated the readers' actual sensitivity with the new CAD algorithm, but this did not reach statistical significance (0.621 vs 0.603, P = .147). The observed and predicted false-positive rates also did not differ significantly (0.275 vs 0.285, P = .250).

CONCLUSIONS

Using one clinical study as a test case, it is shown that the modeling approach is feasible. More testing of the approach is needed to determine if and under what circumstances it can be used as an alternative to a full-scale MRMC study. Meanwhile, the approach can be used to determine if a new CAD algorithm is likely to improve readers' accuracy before embarking on a full-scale MRMC study.

Evidence review and status update on computed tomography colonography

Computed tomographic (CT) colonography is being implemented increasingly in the USA and Europe, and in many centers it has become the radiological technique of choice for imaging the whole colorectum. Although high diagnostic accuracy has been demonstrated in both screening and symptomatic populations, controversy persists regarding implementation, who should interpret the examination, and its cost effectiveness, particularly in the context of primary colorectal cancer screening. Published research in recent years has demonstrated efficacy in a wide range of patient groups, striking technical improvements, and high levels of patient acceptability. New developments continue in the fields of computer aided detection, digital cleansing, and integration into positron emission tomography. The purpose of this review is to bring the reader up-to-date with the latest developments in CT colonography, in particular, those of the last year.

Sources of false positives in computer-assisted CT colonography

The application of computer-aided detection (CAD) is expected to improve reader sensitivity and to reduce inter-observer variance in computed tomographic (CT) colonography. However, current CAD systems display a large number of false-positive (FP) detections. The reviewing of a large number of FP CAD detections increases interpretation time, and it may also reduce the specificity and/or sensitivity of a computer-assisted reader. Therefore, it is important to be aware of the patterns and pitfalls of FP CAD detections. This pictorial essay reviews common sources of FP CAD detections that have been observed in the literature and in our experiments in computer-assisted CT colonography. Also the recommended computer-assisted reading technique is described.

Current trends in staging rectal cancer

Management of rectal cancer has evolved over the years. In this condition preoperative investigations assist in deciding the optimal treatment. The relation of the tumor edge to the circumferential margin (CRM) is an important factor in deciding the need for neoadjuvant treatment and determines the prognosis. Those with threatened or involved margins are offered long course chemoradiation to enable R₀ surgical resection. Endoanal ultrasound (EUS) is useful for tumor (T) staging; hence EUS is a useful imaging modality for early rectal cancer. Magnetic resonance imaging (MRI) is useful for assessing the mesorectum and the mesorectal fascia which has useful prognostic significance and for early identification of local recurrence. Computerized tomography (CT) of the chest, abdomen and pelvis is used to rule out distant metastasis. Identification of the malignant nodes using EUS, CT and MRI is based on the size, morphology and internal characteristics but has drawbacks. Most of the common imaging techniques are suboptimal for imaging following chemoradiation as they struggle to differentiate fibrotic changes and tumor. In this situation, EUS and MRI may provide complementary information to decide further treatment. Functional imaging using positron emission tomography (PET) is useful, particularly PET/CT fusion scans to identify areas of the functionally hot spots. In the current state, imaging has enabled the multidisciplinary team of surgeons, oncologists, radiologists and pathologists to decide on the patient centered management of rectal cancer. In future, functional imaging may play an active role in identifying patients with lymph node metastasis and those with residual and recurrent disease following neoadjuvant chemoradiotherapy.