Arbitration of discrepant BI-RADS 0 recalls by a third reader at screening mammography lowers recall rate but not the cancer detection rate and sensitivity at blinded and non-blinded double reading

Service and clinical impacts of reader bias in breast cancer screening: a retrospective study

OBJECTIVES

To determine factors influencing reader agreement in breast screening and investigate the relationship between agreement level and patient outcomes.

METHODS

Reader pair agreement for 83 265 sets of mammograms from the Scottish Breast Screening service (2015-2020) was evaluated using Cohen's kappa statistic. Each mammography examination was read by two readers, per routine screening practice, with the second initially blinded but able to choose to view the first reader's opinion. If the two readers disagreed, a third reader arbitrated. Variation in reader agreement was examined by: whether the reader acted as the first or second reader, reader experience, and recall, cancer detection and arbitration recall rate.

RESULTS

Readers' opinions varied by whether they acted as the first or second reader. Furthermore, reader 2 was more likely to agree with reader 1 if reader 1 was more experienced than they were, and less likely to agree if they themselves were more experienced than reader 1 (P < .001). Agreement was not significantly associated with cancer detection rate, overall recall rate or arbitration recall rates (P > .05). Lower agreement between readers led to a higher arbiter workload (P < .001).

CONCLUSIONS

In mammography screening, the second reader's opinion is influenced by the first reader's opinion, with the degree of influence dependent on the readers' relative experience levels.

ADVANCES IN KNOWLEDGE

While less-experienced readers relied on their more experienced reading partner, no adverse impact on service outcomes was observed. Allowing access to the first reader's opinion may benefit newly qualified readers, but reduces independent evaluation, which may lower cancer detection rates.

The added value of an artificial intelligence system in assisting radiologists on indeterminate BI-RADS 0 mammograms

OBJECTIVES

To investigate the value of an artificial intelligence (AI) system in assisting radiologists to improve the assessment accuracy of BI-RADS 0 cases in mammograms.

METHODS

We included 34,654 consecutive digital mammography studies, collected between January 2011 and January 2019, among which, 1088 cases from 1010 unique patients with initial BI-RADS 0 assessment who were recalled during 2 years of follow-up were used in this study. Two mid-level radiologists retrospectively re-assessed these BI-RADS 0 cases with the assistance of an AI system developed by us previously. In addition, four entry-level radiologists were split into two groups to cross-read 80 cases with and without the AI. Diagnostic performance was evaluated using the follow-up diagnosis or biopsy results as the reference standard.

RESULTS

Of the 1088 cases, 626 were actually normal (BI-RADS 1 and no recall required). Assisted by the AI system, 351 (56%) and 362 (58%) normal cases were correctly identified by the two mid-level radiologists hence can be avoided for unnecessary follow-ups. However, they would have missed 12 (10 invasive cancers and 2 ductal carcinoma in situ cancers) and 6 (invasive cancers) malignant lesions respectively as a result. These missed lesions were not highly malignant tumors. The inter-rater reliability of entry-level radiologists increased from 0.20 to 0.30 (p < 0.005) by introducing the AI.

CONCLUSION

The AI system can effectively assist mid-level radiologists in reducing unnecessary follow-ups of mammographically indeterminate breast lesions and reducing the benign biopsy rate without missing highly malignant tumors.

KEY POINTS

• The artificial intelligence system could assist mid-level radiologists in effectively reducing unnecessary BI-RADS 0 mammogram recalls and the benign biopsy rate without missing highly malignant tumors. • The artificial intelligence system was capable of detecting low suspicion lesions from heterogeneously and extremely dense breasts that radiologists tended to miss. • The use of an artificial intelligence system may improve the inter-rater reliability and sensitivity, and reduce the reading time of entry-level radiologists in assessing potential lesions in BI-RADS 0 mammograms.

Incorporating Peer Learning Into Your Breast Imaging Practice

Traditional score-based peer review has come under scrutiny in recent years, as studies have demonstrated it to be generally ineffective at improving quality. Many practices and programs are transitioning to a peer learning model to replace or supplement traditional peer review. Peer learning differs from traditional score-based peer review in that the emphasis is on sharing learning opportunities and creating an environment that fosters discussion of errors in a nonpunitive forum with the goal of improved patient care. Creating a just culture is central to fostering successful peer learning. In a just culture, mistakes can be discussed without shame or fear of retribution and the focus is on systems improvement rather than individual blame. Peer learning, as it pertains to breast imaging, can occur in many forms and venues. Examples of the various formats in which peer learning can occur include through individual colleague interaction, as well as divisional, multidisciplinary, department-wide, and virtual conferences, and with the assistance of artificial intelligence. Incorporating peer learning into the practice of breast imaging aims to reduce delayed diagnoses of breast cancer and optimize patient care.

Optimising breast cancer screening reading: blinding the second reader to the first reader's decisions

Objectives

In breast cancer screening, two readers separately examine each woman’s mammograms for signs of cancer. We examined whether preventing the two readers from seeing each other’s decisions (blinding) affects behaviour and outcomes.

Methods

This cohort study used data from the CO-OPS breast-screening trial (1,119,191 women from 43 screening centres in England) where all discrepant readings were arbitrated. Multilevel models were fitted using Markov chain Monte Carlo to measure whether reader 2 conformed to the decisions of reader 1 when they were not blinded, and the effect of blinding on overall rates of recall for further tests and cancer detection. Differences in positive predictive value (PPV) were assessed using Pearson’s chi-squared test.

Results

When reader 1 recalls, the probability of reader 2 also recalling was higher when not blinded than when blinded, suggesting readers may be influenced by the other’s decision. Overall, women were less likely to be recalled when reader 2 was blinded (OR 0.923; 95% credible interval 0.864, 0.986), with no clear pattern in cancer detection rate (OR 1.029; 95% credible interval 0.970, 1.089; Bayesian p value 0.832). PPV was 22.1% for blinded versus 20.6% for not blinded (p < 0.001).

Conclusions

Our results suggest that when not blinded, reader 2 is influenced by reader 1’s decisions to recall (alliterative bias) which would result in bypassing arbitration and negate some of the benefits of double-reading. We found a relationship between blinding the second reader and slightly higher PPV of breast cancer screening, although this analysis may be confounded by other centre characteristics.

Key Points

• In Europe, it is recommended that breast screening mammograms are analysed by two readers but there is little evidence on the effect of ‘blinding’ the readers so they cannot see each other’s decisions.

• We found evidence that when the second reader is not blinded, they are more likely to agree with a recall decision from the first reader and less likely to make an independent judgement (alliterative error). This may reduce overall accuracy through bypassing arbitration.

• This observational study suggests an association between blinding the second reader and higher positive predictive value of screening, but this may be confounded by centre characteristics.

Supplementary Information

The online version contains supplementary material available at 10.1007/s00330-021-07965-z.

A Brief Review on Breast Carcinoma and Deliberation on Current Non Invasive Imaging Techniques for Detection

BACKGROUND

Breast carcinoma is a life threatening disease that accounts for 25.1% of all carcinoma among women worldwide. Early detection of the disease enhances the chance for survival.

DISCUSSION

This paper presents comprehensive report on breast carcinoma disease and its modalities available for detection and diagnosis, as it delves into the screening and detection modalities with special focus placed on the non-invasive techniques and its recent advancement work done, as well as a proposal on a novel method for the application of early breast carcinoma detection.

CONCLUSION

This paper aims to serve as a foundation guidance for the reader to attain bird's eye understanding on breast carcinoma disease and its current non-invasive modalities.

Consensus Review of Discordant Imaging Findings after the Introduction of Digital Screening Mammography: Irish National Breast Cancer Screening Program Experience

Background When there are discordant results between individual readers interpreting screening mammograms, consensus by independent readers may reduce unnecessary recalls for further work-up. Few studies have looked at consensus outcomes following the introduction of full-field digital mammography (FFDM). Purpose To determine outcomes of women discussed at consensus meetings during a 5-year period after introduction of FFDM, including recall rates, cancer detection, and interval cancers. Materials and Methods In this retrospective study from January 2010 to December 2014, the authors reviewed all screening mammograms from a single unit of a biennial Irish national breast screening program after the introduction of FFDM. Screening mammograms were double reported. Abnormalities detected at discordant screening mammography readings were discussed at biweekly consensus meetings. Outcomes of consensus meetings were reviewed in terms of referral for assessment, biopsy rates, cancer detection, and outcomes from later rounds of screening. Statistical analysis was performed by using a χ² test to compare recall rate and cancer detection rates between FFDM and screen-film mammography based on a previously published study from the authors' institution. Results A total of 2565 women (age range, 50-64 years) with discordant mammographic findings were discussed at consensus meetings. Of these 2565 women, 1037 (40%) were referred for further assessment; 108 cancers were detected in these women. Of the 1285 women who returned to biennial screening, malignancy was detected at the site of original concern in 12 women at a further round of screening. Three true interval cancers were identified. Sensitivity (88.5% [108 of 122]; 95% confidence interval [CI]: 81.5%, 93.6%) and negative predictive value (99.1% [1528 of 1542]; 95% CI: 98.5%, 99.4%) of consensus review remained stable after the introduction of FFDM. Specificity of consensus review increased from 57.6% (729 of 1264; 95% CI: 54.9%, 60.4%) to 62.2% (1528 of 2457; 95% CI: 60.2%, 64.1%) (P = .008). Conclusion Consensus review of discordant mammographic screening-detected abnormalities remains a valuable tool after introduction of full-field digital mammography as it reduces recall for assessment and demonstrates persistently high sensitivity and negative predictive values. © RSNA, 2020 See also the editorial by Hofvind and Lee in this issue.

Impact of the second reader on screening outcome at blinded double reading of digital screening mammograms

Background

To determine the impact of the second reader on screening outcome at blinded double reading of digital screening mammograms.

Methods

We included a consecutive series of 99,013 digital screening mammograms, obtained between July 2013 and January 2015 and double read in a blinded fashion. During 2-year follow-up, we collected radiology, surgery and pathology reports of recalled women.

Results

Single reading resulted in 2928 recalls and 616 screen-detected cancers (SDCs). The second reader recalled another 612 women, resulting in 82 additional SDCs. Addition of the second reader increased the recall rate (3.0% to 3.6%, p < 0.001), cancer detection rate (6.2–7.0 per 1000 screens, p < 0.001) and false positive recall rate (24.4–28.7 per 1000 screens, p < 0.001). Positive predictive value of recall (21.0% vs. 19.7%, p = 0.20) and of biopsy (52.1% vs. 50.9%, p = 0.56) were comparable for single reading and blinded double reading. Tumour characteristics were comparable for cancers detected by the first reader and cancers additionally detected by the second reader, except of a more favourable tumour grade in the latter group.

Conclusions

At blinded double reading, the second reader significantly increases the cancer detection rate, at the expense of an increased recall rate and false positive recall rate.

Characteristics of screen-detected cancers following concordant or discordant recalls at blinded double reading in biennial digital screening mammography

OBJECTIVES

To analyse which mammographic and tumour characteristics led to concordant versus discordant recalls at blinded double reading to further optimise our breast cancer screening programme.

METHODS

We included a consecutive series of 99,013 screening mammograms obtained between July 2013 and January 2015. All mammograms were double read in a blinded fashion. Discordant readings were routinely recalled without consensus or arbitration. During the 2-year follow-up, relevant data of the recalled women were collected. We compared mammographic characteristics, screening outcome and tumour characteristics between concordant and discordant recalls.

RESULTS

There were 2,543 concordant recalls (71.4%) and 997 discordant recalls (28.0%). The positive predictive value of a concordant recall was significantly higher (23.5% vs. 10.0%, p < 0.001). The proportion of BI-RADS 0 was significantly higher in the discordant recall group (75.7% vs. 56.3%, p < 0.001). Discordant recalls were more often an asymmetry or architectural distortion (21.8% vs. 13.2% and 9.3% vs. 6.5%, respectively, p < 0.001). There were no differences in the distribution of DCIS and invasive cancers and tumour characteristics were comparable for the two groups, except for a more favourable tumour grade in the discordant recall group (54.7% vs. 39.9% grade I tumours, p = 0.022).

CONCLUSIONS

Screen-detected cancers detected by a discordant reading show a more favourable tumour grade than cancers diagnosed after a concordant recall. The higher proportion of asymmetries and architectural distortions in this group provide a possible target for improving screening programmes by additional training of screening radiologists and the implementation of digital breast tomosynthesis.

KEY POINTS

• With blinded double reading of screening mammograms, screen-detected cancers detected by a discordant reading show a more favourable tumour grade than cancers diagnosed after a concordant recall. • The proportions of asymmetries and architectural distortions are higher in case of a discordant reading. • Possible improvement strategies could target additional training of screening radiologists and the implementation of digital breast tomosynthesis in breast cancer screening programmes.

Strategies for Decreasing Screening Mammography Recall Rates While Maintaining Performance Metrics

RATIONALE AND OBJECTIVE

This study aimed to determine the impact of interventions designed to reduce screening mammography recall rates on screening performance metrics.

MATERIALS AND METHODS

We assessed baseline performance for full-field digital mammography (FFDM) and digital breast tomosynthesis mammography (DBT) for a 3-year period before intervention. The first intervention sought to increase awareness of recalls from screening mammography. Breast imagers discussed their perceptions regarding screening recalls and were required to review their own recalled cases, including outcomes of diagnostic evaluation and biopsy. The second intervention implemented consensus double reading of all recalls, requiring two radiologists to agree if recall was necessary. Recall rates, cancer detection rates, and positive predictive value 1 (PPV1) were compared before and after each intervention.

RESULTS

The baseline recall rate, cancer detection rate, and PPV1 were 11.1%, 3.8/1000, and 3.4%, respectively, for FFDM, and 7.6%, 4.8/1000, and 6.0%, respectively, for DBT. Recall rates decreased significantly to 9.2% for FFDM and to 6.6% for DBT after the first intervention promoting awareness, as well as to 9.9% for FFDM after the second intervention implementing group consensus. PPV1 increased significantly to 5.7% for FFDM and to 9.0% for DBT after the second intervention. Cancer detection rate did not significantly change with the implementation of these interventions. An average of 2.3 minutes was spent consulting for each recall.

CONCLUSION

Reduction in recall rates is desirable, provided performance metrics remain favorable. Our interventions improved performance and could be implemented in other breast imaging settings.

Is the false-positive rate in mammography in North America too high?

The practice of investigating pathological abnormalities in the breasts of females who are asymptomatic is primarily employed using X-ray mammography. The importance of breast screening is reflected in the mortality-based benefits observed among females who are found to possess invasive breast carcinoma prior to the manifestation of clinical symptoms. It is estimated that population-based screening constitutes a 17% reduction in the breast cancer mortality rate among females affected by invasive breast carcinoma. In spite of the significant utility that screening confers in those affected by invasive cancer, limitations associated with screening manifest as potential harms affecting individuals who are free of invasive disease. Disease-free and benign tumour-bearing individuals who are subjected to diagnostic work-up following a screening examination constitute a population of cases referred to as false positives (FPs). This article discusses factors contributing to the FP rate in mammography and extends the discussion to an assessment of the consequences associated with FP reporting. We conclude that the mammography FP rate in North America is in excess based upon the observation of overtreatment of in situ lesions and the disproportionate distribution of detriment and benefit among the population of individuals recalled for diagnostic work-up subsequent to screening. To address the excessive incidence of FPs in mammography, we investigate solutions that may be employed to remediate the current status of the FP rate. Subsequently, it can be suggested that improvements in the breast-screening protocol, medical litigation risk, image interpretation software and the implementation of image acquisition modalities that overcome superimposition effects are promising solutions.