Radiologists' perceptions of computer aided detection versus double reading for mammography interpretation

Service and clinical impacts of reader bias in breast cancer screening: a retrospective study

OBJECTIVES

To determine factors influencing reader agreement in breast screening and investigate the relationship between agreement level and patient outcomes.

METHODS

Reader pair agreement for 83 265 sets of mammograms from the Scottish Breast Screening service (2015-2020) was evaluated using Cohen's kappa statistic. Each mammography examination was read by two readers, per routine screening practice, with the second initially blinded but able to choose to view the first reader's opinion. If the two readers disagreed, a third reader arbitrated. Variation in reader agreement was examined by: whether the reader acted as the first or second reader, reader experience, and recall, cancer detection and arbitration recall rate.

RESULTS

Readers' opinions varied by whether they acted as the first or second reader. Furthermore, reader 2 was more likely to agree with reader 1 if reader 1 was more experienced than they were, and less likely to agree if they themselves were more experienced than reader 1 (P < .001). Agreement was not significantly associated with cancer detection rate, overall recall rate or arbitration recall rates (P > .05). Lower agreement between readers led to a higher arbiter workload (P < .001).

CONCLUSIONS

In mammography screening, the second reader's opinion is influenced by the first reader's opinion, with the degree of influence dependent on the readers' relative experience levels.

ADVANCES IN KNOWLEDGE

While less-experienced readers relied on their more experienced reading partner, no adverse impact on service outcomes was observed. Allowing access to the first reader's opinion may benefit newly qualified readers, but reduces independent evaluation, which may lower cancer detection rates.

Emerging uses of artificial intelligence in breast and axillary ultrasound

Breast ultrasound is a valuable adjunctive tool to mammography in detecting breast cancer, especially in women with dense breasts. Ultrasound also plays an important role in staging breast cancer by assessing axillary lymph nodes. However, its utility is limited by operator dependence, high recall rate, low positive predictive value and low specificity. These limitations present an opportunity for artificial intelligence (AI) to improve diagnostic performance and pioneer novel uses of ultrasound. Research in developing AI for radiology has flourished over the past few years. A subset of AI, deep learning, uses interconnected computational nodes to form a neural network, which extracts complex visual features from image data to train itself into a predictive model. This review summarizes several key studies evaluating AI programs' performance in predicting breast cancer and demonstrates that AI can assist radiologists and address limitations of ultrasound by acting as a decision support tool. This review also touches on how AI programs allow for novel predictive uses of ultrasound, particularly predicting molecular subtypes of breast cancer and response to neoadjuvant chemotherapy, which have the potential to change how breast cancer is managed by providing non-invasive prognostic and treatment data from ultrasound images. Lastly, this review explores how AI programs demonstrate improved diagnostic accuracy in predicting axillary lymph node metastasis. The limitations and future challenges in developing and implementing AI for breast and axillary ultrasound will also be discussed.

Diagnostic Errors in Cerebrovascular Pathology: Retrospective Analysis of a Neuroradiology Database at a Large Tertiary Academic Medical Center

BACKGROUND AND PURPOSE

Diagnostic errors affect 2%-8% of neuroradiology studies, resulting in significant potential morbidity and mortality. This retrospective analysis of a large database at a single tertiary academic institution focuses on diagnostic misses in cerebrovascular pathology and suggests error-reduction strategies.

MATERIALS AND METHODS

CT and MR imaging reports from a consecutive database spanning 2015-2020 were searched for errors of attending physicians in cerebrovascular pathology. Data were collected on missed findings, study types, and interpretation settings. Errors were categorized as ischemic, arterial, venous, hemorrhagic, and "other."

RESULTS

A total of 245,762 CT and MR imaging neuroradiology examinations were interpreted during the study period. Vascular diagnostic errors were present in 165 reports, with a mean of 49.6 (SD, 23.3) studies on the shifts when an error was made, compared with 34.9 (SD, 19.2) on shifts without detected errors (P < .0001). Seventy percent of examinations occurred in the hospital setting; 93.3% of errors were perceptual; 6.7% were interpretive; and 93.9% (n = 155) were clinically significant (RADPEER 2B or 3B). The distribution of errors was arterial and ischemic each with 33.3%, hemorrhagic with 21.8%, and venous with 7.5%. Most errors involved brain MR imaging (30.3%) followed by head CTA (27.9%) and noncontrast head CT (26.1%). The most common misses were acute/subacute infarcts (25.1%), followed by aneurysms (13.7%) and subdural hematomas (9.7%).

CONCLUSIONS

Most cerebrovascular diagnostic errors were perceptual and clinically significant, occurred in the emergency/inpatient setting, and were associated with higher-volume shifts. Diagnostic errors could be minimized by adjusting search patterns to ensure vigilance on the sites of the frequently missed pathologies.

Radiology for Thoracic Conditions in Low- and Middle-Income Countries

With a disproportionately high burden of global morbidity and mortality caused by chronic respiratory diseases (CRDs) in low and middle-income countries (LMICs), access to radiological services is of critical importance for screening, diagnosis, and treatment guidance.

Mandating Limits on Workload, Duty, and Speed in Radiology

Research has not yet quantified the effects of workload or duty hours on the accuracy of radiologists. With the exception of a brief reduction in imaging studies during the 2020 peak of the COVID-19 pandemic, the workload of radiologists in the United States has seen relentless growth in recent years. One concern is that this increased demand could lead to reduced accuracy. Behavioral studies in species ranging from insects to humans have shown that decision speed is inversely correlated to decision accuracy. A potential solution is to institute workload and duty limits to optimize radiologist performance and patient safety. The concern, however, is that any prescribed mandated limits would be arbitrary and thus no more advantageous than allowing radiologists to self-regulate. Specific studies have been proposed to determine whether limits reduce error, and if so, to provide a principled basis for such limits. This could determine the precise susceptibility of individual radiologists to medical error as a function of speed during image viewing, the maximum number of studies that could be read during a work shift, and the appropriate shift duration as a function of time of day. Before principled recommendations for restrictions are made, however, it is important to understand how radiologists function both optimally and at the margins of adequate performance. This study examines the relationship between interpretation speed and error rates in radiology, the potential influence of artificial intelligence on reading speed and error rates, and the possible outcomes of imposed limits on both caseload and duty hours. This review concludes that the scientific evidence needed to make meaningful rules is lacking and notes that regulating workloads without scientific principles can be more harmful than not regulating at all.

Improving radiologist's ability in identifying particular abnormal lesions on mammograms through training test set with immediate feedback

It has been shown that there are differences in diagnostic accuracy of cancer detection on mammograms, from below 50% in developing countries to over 80% in developed world. One previous study reported that radiologists from a population in Asia displayed a low mammographic cancer detection of 48% compared with over 80% in developed countries, and more importantly, that most lesions missed by these radiologists were spiculated masses or stellate lesions. The aim of this study was to explore the performance of radiologists after undertaking a training test set which had been designed to improve the capability in detecting a specific type of cancers on mammograms. Twenty-five radiologists read two sets of 60 mammograms in a standardized mammogram reading room. The first test set focused on stellate or spiculated masses. When radiologists completed the first set, the system displayed immediate feedback to the readers comparing their performances in each case with the truth of cancer cases and cancer types so that the readers could identify individual-based errors. Later radiologists were asked to read the second set of mammograms which contained different types of cancers including stellate/spiculated masses, asymmetric density, calcification, discrete mass and architectural distortion. Case sensitivity, lesion sensitivity, specificity, receiver operating characteristics (ROC) and Jackknife alternative free-response receiver operating characteristics (JAFROC) were calculated for each participant and their diagnostic accuracy was compared between two sessions. Results showed significant improvement among radiologists in case sensitivity (+ 11.4%; P < 0.05), lesion sensitivity (+ 18.7%; P < 0.01) and JAFROC (+ 11%; P < 0.01) in the second set compared with the first set. The increase in diagnostic accuracy was also recorded in the detection of stellate/spiculated mass (+ 20.6%; P < 0.05). This indicated that the performance of radiologists in detecting malignant lesions on mammograms can be improved if an appropriate training intervention is applied after the readers' weakness and strength are identified.

Missed Breast Cancer: Effects of Subconscious Bias and Lesion Characteristics

Medical errors are a substantial cause of morbidity and mortality and the third leading cause of death in the United States. Errors resulting in missed breast cancer are the most common reason for medical malpractice lawsuits against all physicians. Missed breast cancers are breast malignancies that are detectable at retrospective review of a previously obtained mammogram that was prospectively reported as showing negative, benign, or probably benign findings. Investigators in prior studies have found that up to 35% of both interval cancers and screen-detected cancers could be classified as missed. As such, in conjunction with having awareness of the most common misleading appearances of breast cancer, it is important to understand the cognitive processes and unconscious biases that can impact image interpretation, thereby helping to decrease the number of missed breast cancers. The various cognitive processes that lead to unconscious bias in breast imaging, such as satisfaction of search, inattention blindness, hindsight, anchoring, premature closing, and satisfaction of reporting, are outlined in this pictorial review of missed breast cancers. In addition, strategies for reducing the rates of these missed cancers are highlighted. The most commonly missed and misinterpreted lesions, including stable lesions, benign-appearing masses, one-view findings, developing asymmetries, subtle calcifications, and architectural distortion, also are reviewed. This information will help illustrate why and how breast cancers are missed and aid in the development of appropriate minimization strategies in breast imaging. ^©RSNA, 2020.

Computer-Aided Detection for Breast Cancer Screening in Clinical Settings: Scoping Review

Background

With the growth of machine learning applications, the practice of medicine is evolving. Computer-aided detection (CAD) is a software technology that has become widespread in radiology practices, particularly in breast cancer screening for improving detection rates at earlier stages. Many studies have investigated the diagnostic accuracy of CAD, but its implementation in clinical settings has been largely overlooked.

Objective

The aim of this scoping review was to summarize recent literature on the adoption and implementation of CAD during breast cancer screening by radiologists and to describe barriers and facilitators for CAD use.

Methods

The MEDLINE database was searched for English, peer-reviewed articles that described CAD implementation, including barriers or facilitators, in breast cancer screening and were published between January 2010 and March 2018. Articles describing the diagnostic accuracy of CAD for breast cancer detection were excluded. The search returned 526 citations, which were reviewed in duplicate through abstract and full-text screening. Reference lists and cited references in the included studies were reviewed.

Results

A total of nine articles met the inclusion criteria. The included articles showed that there is a tradeoff between the facilitators and barriers for CAD use. Facilitators for CAD use were improved breast cancer detection rates, increased profitability of breast imaging, and time saved by replacing double reading. Identified barriers were less favorable perceptions of CAD compared to double reading by radiologists, an increase in recall rates of patients for further testing, increased costs, and unclear effect on patient outcomes.

Conclusions

There is a gap in the literature between CAD’s well-established diagnostic accuracy and its implementation and use by radiologists. Generally, the perceptions of radiologists have not been considered and details of implementation approaches for adoption of CAD have not been reported. The cost-effectiveness of CAD has not been well established for breast cancer screening in various populations. Further research is needed on how to best facilitate CAD in radiology practices in order to optimize patient outcomes, and the views of radiologists need to be better considered when advancing CAD use.

256 Shades of gray: uncertainty and diagnostic error in radiology

Radiologists practice in an environment of extraordinarily high uncertainty, which results partly from the high variability of the physical and technical aspects of imaging, partly from the inherent limitations in the diagnostic power of the various imaging modalities, and partly from the complex visual-perceptual and cognitive processes involved in image interpretation. This paper reviews the high level of uncertainty inherent to the process of radiological imaging and image interpretation vis-à-vis the issue of radiological interpretive error, in order to highlight the considerable degree of overlap that exists between these. The scope of radiological error, its many potential causes and various error-reduction strategies in radiology are also reviewed.

Utilization of Computer-Aided Detection for Digital Screening Mammography in the United States, 2008 to 2016

PURPOSE

Computer-aided detection (CAD) for screening mammography is a software technology designed to improve radiologists' reading performance. Since 2007, multiple Breast Cancer Surveillance Consortium research papers have shown that CAD decreases performance by increasing recalls and decreasing the detection of invasive cancer while increasing the detection of ductal carcinoma in situ. The aim of this study was to test the hypothesis that CAD use by digital mammography facilities would decrease over time.

METHODS

In August 2007, August 2011, and March 2016, the FDA database of certified mammography facilities was accessed, and a random sample of 400 of approximately 8,500 total facilities was generated. In 2008 and 2011, a telephone survey was conducted of the facilities regarding digital mammography and CAD use. In 2016, facility websites were reviewed before calling the facilities. Bonferroni-corrected P values were used to assess statistical differences in the proportion of CAD at digital facilities for the three surveys.

RESULTS

The mean proportion of digital facilities using CAD was 91.4%, including 91.4% (128 of 140) in 2008, 90.2% (238 of 264) in 2011, and 92.3% (358 of 388) in 2016. The difference for 2008 versus 2011 was 1.3% (95% confidence interval [CI], -0.5% to 7.7%), for 2011 versus 2016 was -2.1% (95% CI, -6.9% to 2.7%), and for 2008 versus 2016 was -0.8% (95% CI, -6.7% to 5.0%).

CONCLUSIONS

In three national surveys, it was found that CAD use at US digital screening mammography facilities was stable from 2008 to 2016. This persistent utilization is relevant to the debate on the value of targeting ductal carcinoma in situ in screening.

Breast cancer screening in England and the United States: a comparison of provision and utilisation

OBJECTIVES

Comparing breast cancer screening across countries within the context of some of the benefits and harms offers the opportunity to improve effectiveness through mutual learning.

METHODS

This paper describes the provision of breast cancer screening in England and the United States. The various recommendations for accessing breast cancer screening in the two countries are set out and the organisation of services including quality assurance, incentives and performance mechanisms considered.

RESULTS

In the United States, younger women are routinely screened; they are less likely to benefit and more likely to be harmed. The utilisation of breast cancer screening amongst eligible women is broadly comparable in the two countries. However, there are differences in technical performance; the reasons for these including radiological reading procedures and cultural factors are explored.

CONCLUSIONS

Despite a well-functioning screening programme, breast cancer mortality and survival in England are poor relative to other countries. Emphasis for American improvement should be on reducing false-positive recall rates, while the English NHS could supplement existing efforts to understand and improve comparatively poor survival and mortality.

Computer-aided detection of cancer in automated 3-D breast ultrasound

Automated 3-D breast ultrasound (ABUS) has gained a lot of interest and may become widely used in screening of dense breasts, where sensitivity of mammography is poor. However, reading ABUS images is time consuming, and subtle abnormalities may be missed. Therefore, we are developing a computer aided detection (CAD) system to help reduce reading time and prevent errors. In the multi-stage system we propose, segmentations of the breast, the nipple and the chestwall are performed, providing landmarks for the detection algorithm. Subsequently, voxel features characterizing coronal spiculation patterns, blobness, contrast, and depth are extracted. Using an ensemble of neural-network classifiers, a likelihood map indicating potential abnormality is computed. Local maxima in the likelihood map are determined and form a set of candidates in each image. These candidates are further processed in a second detection stage, which includes region segmentation, feature extraction and a final classification. On region level, classification experiments were performed using different classifiers including an ensemble of neural networks, a support vector machine, a k-nearest neighbors, a linear discriminant, and a gentle boost classifier. Performance was determined using a dataset of 238 patients with 348 images (views), including 169 malignant and 154 benign lesions. Using free response receiver operating characteristic (FROC) analysis, the system obtains a view-based sensitivity of 64% at 1 false positives per image using an ensemble of neural-network classifiers.

Development and evaluation of a software tool for the generation of virtual liver lesions in multidetector-row CT datasets

RATIONALE AND OBJECTIVES

Development and evaluation of a software tool for the insertion of simulated hypodense liver lesions in multidetector-row computed tomography (CT) datasets.

MATERIALS AND METHODS

Forty software-generated hypodense liver lesions were inserted at random locations in 20 CT datasets by using the "alpha blending" technique and compared with 40 real metastatic lesions. The location, diameter (5-20 mm) and density of the simulated lesions were individually adjusted to closely resemble real lesions in each patient. Three blinded readers evaluated all 80 lesions twice in a 2-week interval using a five-point Likert confidence scale under standardized conditions. Nonparametric tests were used to statistically evaluate possible differences in scoring between real and simulated lesions. The correctness of the observer rating for real and simulated lesions was compared to chance distribution using the chi-squared statistics. The inter- and intraobserver variability was determined using Kendall's coefficient of concordance.

RESULTS

The observer study did not reveal significant differences between the scoring for real versus simulated lesions for any of the readers (P > .05). The distribution of correct and false scoring of the lesions was not significantly different from chance distribution (P > .05). Inter- and intraobserver agreement was poor (Kendall W coefficient = 0.12/0.13).

CONCLUSION

The proposed algorithm is suitable for creating realistic virtual liver lesions in CT datasets.