Assessing the impact of CAD on the sensitivity and specificity of film readers

Added value of double reading in diagnostic radiology,a systematic review

Objectives

Double reading in diagnostic radiology can find discrepancies in the original report, but a systematic program of double reading is resource consuming. There are conflicting opinions on the value of double reading. The purpose of the current study was to perform a systematic review on the value of double reading.

Methods

A systematic review was performed to find studies calculating the rate of misses and overcalls with the aim of establishing the added value of double reading by human observers.

Results

The literature search resulted in 1610 hits. After abstract and full-text reading, 46 articles were selected for analysis. The rate of discrepancy varied from 0.4 to 22% depending on study setting. Double reading by a sub-specialist, in general, led to high rates of changed reports.

Conclusions

The systematic review found rather low discrepancy rates. The benefit of double reading must be balanced by the considerable number of working hours a systematic double-reading scheme requires. A more profitable scheme might be to use systematic double reading for selected, high-risk examination types. A second conclusion is that there seems to be a value of sub-specialisation for increased report quality. A consequent implementation of this would have far-reaching organisational effects.

Key Points

• In double reading, two or more radiologists read the same images.

• A systematic literature review was performed.

• The discrepancy rates varied from 0.4 to 22% in various studies.

• Double reading by sub-specialists found high discrepancy rates.

Electronic supplementary material

The online version of this article (10.1007/s13244-018-0599-0) contains supplementary material, which is available to authorised users.

Improving the radiologist-CAD interaction: designing for appropriate trust

Computer-aided diagnosis (CAD) has great potential to improve radiologists' diagnostic performance. However, the reported performance of the radiologist-CAD team is lower than what might be expected based on the performance of the radiologist and the CAD system in isolation. This indicates that the interaction between radiologists and the CAD system is not optimal. An important factor in the interaction between humans and automated aids (such as CAD) is trust. Suboptimal performance of the human-automation team is often caused by an inappropriate level of trust in the automation. In this review, we examine the role of trust in the radiologist-CAD interaction and suggest ways to improve the output of the CAD system so that it allows radiologists to calibrate their trust in the CAD system more effectively. Observer studies of the CAD systems show that radiologists often have an inappropriate level of trust in the CAD system. They sometimes under-trust CAD, thereby reducing its potential benefits, and sometimes over-trust it, leading to diagnostic errors they would not have made without CAD. Based on the literature on trust in human-automation interaction and the results of CAD observer studies, we have identified four ways to improve the output of CAD so that it allows radiologists to form a more appropriate level of trust in CAD. Designing CAD systems for appropriate trust is important and can improve the performance of the radiologist-CAD team. Future CAD research and development should acknowledge the importance of the radiologist-CAD interaction, and specifically the role of trust therein, in order to create the perfect artificial partner for the radiologist. This review focuses on the role of trust in the radiologist-CAD interaction. The aim of the review is to encourage CAD developers to design for appropriate trust and thereby improve the performance of the radiologist-CAD team.

Role of computer-aided detection in very small screening detected invasive breast cancers

This study aims to assess computer-aided detection (CAD) performance with full-field digital mammography (FFDM) in very small (equal to or less than 1 cm) invasive breast cancers. Sixty-eight invasive breast cancers less than or equal to 1 cm were retrospectively studied. All cases were detected with FFDM in women aged 49-69 years from our breast cancer screening program. Radiological characteristics of lesions following BI-RADS descriptors were recorded and compared with CAD sensitivity. Age, size, BI-RADS classification, breast density type, histological type of the neoplasm, and role of the CAD were also assessed. Per-study specificity and mass false-positive rate were determined by using 100 normal consecutive studies. Thirty-seven (54.4 %) masses, 17 (25 %) calcifications, 6 (8.8 %) masses with calcifications, 7 (10.3 %) architectural distortions, and 1 asymmetry (1.5 %) were found. CAD showed an overall sensitivity of 86.7 % (masses, 86.5 %; calcifications, 100 %; masses with calcifications, 100 %; and architectural distortion, 57.14 %), CAD failed to detect 9 out of 68 cases: 5 of 37 masses, 3 of 7 architectural distortions, and 1 of 1 asymmetry. Fifteen out of 37 masses were hyperdense, and all of them were detected by CAD. No association was seen among mass morphology or margins and detectability. Per-study specificity and CAD false-positive rate was 26 % and 1.76 false marks per study. In conclusion, CAD shows a high sensitivity and a low specificity. Lesion size, histology, and breast density do not influence sensitivity. Mammographic features, mass density, and thickness of the spicules in architectural distortions do influence.

Automatic detection of anomalies in screening mammograms

Background

Diagnostic performance in breast screening programs may be influenced by the prior probability of disease. Since breast cancer incidence is roughly half a percent in the general population there is a large probability that the screening exam will be normal. That factor may contribute to false negatives. Screening programs typically exhibit about 83% sensitivity and 91% specificity. This investigation was undertaken to determine if a system could be developed to pre-sort screening-images into normal and suspicious bins based on their likelihood to contain disease. Wavelets were investigated as a method to parse the image data, potentially removing confounding information. The development of a classification system based on features extracted from wavelet transformed mammograms is reported.

Methods

In the multi-step procedure images were processed using 2D discrete wavelet transforms to create a set of maps at different size scales. Next, statistical features were computed from each map, and a subset of these features was the input for a concerted-effort set of naïve Bayesian classifiers. The classifier network was constructed to calculate the probability that the parent mammography image contained an abnormality. The abnormalities were not identified, nor were they regionalized.

The algorithm was tested on two publicly available databases: the Digital Database for Screening Mammography (DDSM) and the Mammographic Images Analysis Society’s database (MIAS). These databases contain radiologist-verified images and feature common abnormalities including: spiculations, masses, geometric deformations and fibroid tissues.

Results

The classifier-network designs tested achieved sensitivities and specificities sufficient to be potentially useful in a clinical setting. This first series of tests identified networks with 100% sensitivity and up to 79% specificity for abnormalities. This performance significantly exceeds the mean sensitivity reported in literature for the unaided human expert.

Conclusions

Classifiers based on wavelet-derived features proved to be highly sensitive to a range of pathologies, as a result Type II errors were nearly eliminated. Pre-sorting the images changed the prior probability in the sorted database from 37% to 74%.

How to discriminate between computer-aided and computer-hindered decisions: a case study in mammography

BACKGROUND

Computer aids can affect decisions in complex ways, potentially even making them worse; common assessment methods may miss these effects. We developed a method for estimating the quality of decisions, as well as how computer aids affect it, and applied it to computer-aided detection (CAD) of cancer, reanalyzing data from a published study where 50 professionals ("readers") interpreted 180 mammograms, both with and without computer support.

METHOD

We used stepwise regression to estimate how CAD affected the probability of a reader making a correct screening decision on a patient with cancer (sensitivity), thereby taking into account the effects of the difficulty of the cancer (proportion of readers who missed it) and the reader's discriminating ability (Youden's determinant). Using regression estimates, we obtained thresholds for classifying a posteriori the cases (by difficulty) and the readers (by discriminating ability).

RESULTS

Use of CAD was associated with a 0.016 increase in sensitivity (95% confidence interval [CI], 0.003-0.028) for the 44 least discriminating radiologists for 45 relatively easy, mostly CAD-detected cancers. However, for the 6 most discriminating radiologists, with CAD, sensitivity decreased by 0.145 (95% CI, 0.034-0.257) for the 15 relatively difficult cancers.

CONCLUSIONS

Our exploratory analysis method reveals unexpected effects. It indicates that, despite the original study detecting no significant average effect, CAD helped the less discriminating readers but hindered the more discriminating readers. Such differential effects, although subtle, may be clinically significant and important for improving both computer algorithms and protocols for their use. They should be assessed when evaluating CAD and similar warning systems.

Mammographic features of breast cancers at single reading with computer-aided detection and at double reading in a large multicenter prospective trial of computer-aided detection: CADET II

PURPOSE

To evaluate the mammographic features of breast cancer that favor lesion detection with single reading and computer-aided detection (CAD) or with double reading.

MATERIALS AND METHODS

The Computer Aided Detection Evaluation Trial II study was approved by the ethics committee, and all participants provided written informed consent. A total of 31,057 women were recruited from three screening centers between September 2006 and August 2007. They were randomly allocated to the double reading group, the single reading with CAD group, or the double reading and single reading with CAD group at a ratio of 1:1:28, respectively. In this study, cancers in the women whose mammograms were read with both single reading with CAD and double reading were retrospectively reviewed. The original mammograms were obtained for each case and reviewed by two of three experienced breast radiologists in consensus. The method of detection was noted. The size and predominant mammographic feature of the cancer were recorded, as was the breast density. CAD marking data were reviewed to determine if the cancer had been correctly marked.

RESULTS

A total of 227 cancers were detected in 28,204 women. A total of 170 cases were recalled with both reading regimens. Lesion types were masses (66%), microcalcifications (25%), parenchymal deformities (6%), and asymmetric densities (3%). The ability of the reading regimens to correctly prompt the reader to recall cases varied significantly by lesion type (P < .001). More parenchymal deformities were recalled with double reading, whereas more asymmetric densities were recalled with single reading with CAD. There was no difference in the ability of either reading regimen to prompt the reader to correctly recall masses or microcalcifications. CAD correctly prompted 100% of microcalcifications, 87% of mass lesions, 80% of asymmetric densities, and 50% of parenchymal deformities. CAD correctly marked 93% of spiculated masses compared with 80% of ill-defined masses (P = .054). There was a significant trend for cancers detected with double reading to occur only in women with a denser mammographic background pattern (P = .02). Size had no effect on lesion detection.

CONCLUSION

Readers using either single reading with CAD or double reading need to be aware of the strengths and weaknesses of reading regimens to avoid missing the more challenging cancer cases.

Using computer-aided detection in mammography as a decision support

Objective:

To evaluate an interactive computer-aided detection (CAD) system for reading mammograms to improve decision making.

Methods:

A dedicated mammographic workstation has been developed in which readers can probe image locations for the presence of CAD information. If present, CAD findings are displayed with the computed malignancy rating. A reader study was conducted in which four screening radiologists and five non-radiologists participated to study the effect of this system on detection performance. The participants read 120 cases of which 40 cases had a malignant mass that was missed at the original screening. The readers read each mammogram both with and without CAD in separate sessions. Each reader reported localized findings and assigned a malignancy score per finding. Mean sensitivity was computed in an interval of false-positive fractions less than 10%.

Results:

Mean sensitivity was 25.1% in the sessions without CAD and 34.8% in the CAD-assisted sessions. The increase in detection performance was significant (p = 0.012). Average reading time was 84.7 ± 61.5 s/case in the unaided sessions and was not significantly higher when interactive CAD was used (85.9 ± 57.8 s/case).

Conclusion:

Interactive use of CAD in mammography may be more effective than traditional CAD for improving mass detection without affecting reading time.

Variable size computer-aided detection prompts and mammography film reader decisions

Introduction

The purpose of the present study was to investigate the effect of computer-aided detection (CAD) prompts on reader behaviour in a large sample of breast screening mammograms by analysing the relationship of the presence and size of prompts to the recall decision.

Methods

Local research ethics committee approval was obtained; informed consent was not required. Mammograms were obtained from women attending routine mammography at two breast screening centres in 1996. Films, previously double read, were re-read by a different reader using CAD. The study material included 315 cancer cases comprising all screen-detected cancer cases, all subsequent interval cancers and 861 normal cases randomly selected from 10,267 cases. Ground truth data were used to assess the efficacy of CAD prompting. Associations between prompt attributes and tumour features or reader recall decisions were assessed by chi-squared tests.

Results

There was a highly significant relationship between prompting and a decision to recall for cancer cases and for a random sample of normal cases (P < 0.001). Sixty-four per cent of all cases contained at least one CAD prompt. In cancer cases, larger prompts were more likely to be recalled (P = 0.02) for masses but there was no such association for calcifications (P = 0.9). In a random sample of 861 normal cases, larger prompts were more likely to be recalled (P = 0.02) for both mass and calcification prompts. Significant associations were observed with prompting and breast density (p = 0.009) for cancer cases but not for normal cases (P = 0.05).

Conclusions

For both normal cases and cancer cases, prompted mammograms were more likely to be recalled and the prompt size was also associated with a recall decision.

Evaluation of computer-aided detection systems in the detection of small invasive breast carcinoma

PURPOSE

To retrospectively compare two CAD systems for detecting invasive breast cancers manifesting as noncalcified masses smaller than 16 mm.

MATERIALS AND METHODS

Waiver of informed consent was granted by the Institutional Review Board that approved this HIPAA-compliant study. Mammograms obtained from two institutions providing consecutive invasive carcinomas manifesting as noncalcified masses smaller than 16 mm were evaluated by using two commercially available CAD systems (R2 ImageChecker M1000, version 5.0A and iCAD Second Look, version 6.0 mid operating point). To provide statistical power to test for a possible 10% difference in the sensitivity performance between the systems, 192 consecutive mammographic studies (182 unifocal, six multifocal, and four bilateral cancers) were collected. Masses were characterized using the Breast Imaging Reporting and Data System (BI-RADS). Per study specificity and mass false marker rate were determined by using 51 normal four-view studies, while scoring only the mass false-positive marks for noncalcified masses. Associations between mass characteristics and supplying institution were compared by using chi2 tests. A P value of .05 was considered to indicate a significant difference.

RESULTS

The respective per study sensitivity, per image (ie, per view) sensitivity, per study specificity, and mass false-positive marker rates were 81.8%, 64.7%, 39.2%, and 1.08 for the R2 ImageChecker M1000 system, and 60.9%, 42.6%, 31.4%, and 1.41 for the iCAD Second Look system. The overall per study and per image sensitivities were significantly better for R2 than for iCAD (McNemar test, all P<.001), with a nonsignificant higher per study specificity and lower mass false marker rate on normal studies. CAD results demonstrated at least a 20% variation between BI-RADS categories 4a and 5 for per study and per image sensitivity.

CONCLUSION

A statistically significant difference was observed in per study and per image sensitivity in our mammography data set with small (<16 mm), noncalcified invasive breast malignancies between two CAD systems. Differences in per study specificity and mass false marker rate were noted but were not statistically significant.

Does the accuracy of single reading with CAD (computer-aided detection) compare with that of double reading?: A review of the literature

AIM

To examine current evidence to determine whether the accuracy of single reading with computed-aided detection (CAD) compares with that of double reading.

METHODS

We performed a literature review to identify studies where both protocols had been investigated and compared. We identified eight studies that compared single reading with CAD against double reading, of which six reported on comparisons of both sensitivity and specificity.

RESULTS

Of the six studies identified, three showed no differences in either sensitivity or specificity. One showed single reading with CAD had a higher sensitivity at the same specificity, another that single reading with CAD had a higher specificity at the same sensitivity. However, one study, in a real-life setting, showed that single reading with CAD had a higher sensitivity but a lower specificity.

CONCLUSION

As the majority of the studies were not in a real-life setting, used test sets, lacked sufficient training in the use of CAD and simulated double reading (using a protocol of recall if one suggests), current evidence is therefore limited as to the accuracy, in terms of sensitivity and specificity, of single reading with CAD in comparison with the most common practice in the UK of double reading using a protocol of consensus or arbitration.

Single reading with computer-aided detection and double reading of screening mammograms in the United Kingdom National Breast Screening Program

PURPOSE

To retrospectively determine if the use of a computer-aided detection (CAD) system can improve the performance of single reading of screening mammograms to match that of double reading in the United Kingdom.

MATERIALS AND METHODS

Local research ethics committee approval was obtained; informed consent was not required. This study included a sample of 10 267 mammograms obtained in women aged 50 years or older who underwent routine screening at one of two breast screening centers in 1996. Mammograms that were double read in 1996 were randomly allocated to be re-read by eight different radiologists using CAD. The cancer detection and recall rates from double reading and single reading with CAD were compared. Statistical significance and confidence intervals were calculated with the McNemar test to account for the matched nature of the data.

RESULTS

Single reading with CAD led to a cancer detection rate that was significantly (P = .02) higher than that achieved with double reading: 6.5% more cancers were detected by means of single reading with CAD than by means of double reading. However, the recall rate was higher for single reading with CAD than for double reading (8.6% vs 6.5%, respectively; P < .001). This was equivalent to relative increases of 15% and 32% in the cancer detection and recall rates, respectively.

CONCLUSION

Single reading with CAD leads to an improved cancer detection rate and an increased recall rate.

Computer-aided detection in the United Kingdom National Breast Screening Programme: prospective study

PURPOSE

To evaluate prospectively the recall and cancer detection rates with and without computer-aided detection (CAD) in the United Kingdom National Health Service Breast Screening Programme.

MATERIALS AND METHODS

The study had appropriate ethics committee approval. Informed consent was not required; however, patients were informed that their mammograms might be used in research efforts, and all patients agreed to participate. Mammograms obtained in 6111 women (mean age, 58.4 years) undergoing routine screening every 3 years were analyzed with a CAD system. Mammograms were independently double read. Twelve readers participated. Readers recorded an initial evaluation, viewed the CAD prompts, and recorded a final evaluation. Recall to assessment was decided after arbitration. Sensitivities were calculated for single reading, single reading with CAD, and double reading, as a proportion of the total number of cancers detected by using double reading with CAD.

RESULTS

A total of 62 cancers were detected in 61 women. CAD prompted 51 (84%) of 61 radiographically detected cancers. Of 12 cancers missed on single reading, nine were correctly prompted; however, seven prompts were overruled by the reader. Sensitivity of single reading was 90.2% (95% confidence interval [CI]: 83.0%, 95.0%), single reading with CAD was 91.5% (95% CI: 85.0%, 96.0%), and double reading without CAD was 98.4% (95% CI: 91.0%, 100%). Cancer detection rate was 1%. Recall to assessment rate was 6.1%, with an increase of 5.8% because of CAD. Average time required, per reader, to read a case was 25 seconds without CAD and 45 seconds with CAD.

CONCLUSION

CAD increases sensitivity of single reading by 1.3%, whereas double reading increases sensitivity by 8.2%.

Evaluation of computer-aided detection (CAD) devices

We present a review of three major UK studies of computer-aided detection (CAD) for mammography. A short account of the motivation, methods and results is given for each of the three. A number of conclusions are drawn, particularly about the merits and difficulties of research in the field. The first two studies measured the impact of CAD on the sensitivity and specificity of film readers interpreting cases with known outcomes displayed on rollers with an artificially high frequency of cancers. In the first study 50 film readers each read 180 cases, including 60 cancers (40 screen-detected and 20 interval). In the second study 35 film readers viewed 120 cases including 44 cancers, of which 40 were selected to be difficult cases that CAD prompted correctly. The third study was carried out prospectively. 6111 films were independently double read by film readers who recorded a judgement before and after viewing CAD prompts. In addition to this, intraobserver measure of the impact of CAD, we compared the cancer detection rate in these cases with that in 1339 cases read over the same period without the benefit of CAD. None of the three studies showed a statistically significant effect attributable to CAD. There is evidence that a high proportion of missed cancers are prompted and that "emphasised" prompts, which have a greater positive predictive value, have a stronger impact on decision-making that other prompts.