Correlation Between Screening Mammography Interpretive Performance on a Test Set and Performance in Clinical Practice

Fatigue and vigilance in medical experts detecting breast cancer

An abundance of laboratory-based experiments has described a vigilance decrement of reducing accuracy to detect targets with time on task, but there are few real-world studies, none of which have previously controlled the environment to control for bias. We describe accuracy in clinical practice for 360 experts who examined >1 million women's mammograms for signs of cancer, whilst controlling for potential biases. The vigilance decrement pattern was not observed. Instead, test accuracy improved over time, through a reduction in false alarms and an increase in speed, with no significant change in sensitivity. The multiple-decision model explains why experts miss targets in low prevalence settings through a change in decision threshold and search quit threshold and propose it should be adapted to explain these observed patterns of accuracy with time on task. What is typically thought of as standard and robust research findings in controlled laboratory settings may not directly apply to real-world environments and instead large, controlled studies in relevant environments are needed.

Clinical performance progress of BREAST participants: the impact of test-set participation

AIM

To investigate if positive changes in the clinical performance of radiologists are associated with reading mammographic test sets.

MATERIALS AND METHODS

This study investigated the clinical audit history for a cohort of 39 participants in the BreastScreen Reader Assessment Strategy who have read for BreastScreen New South Wales in the period between 2010 and 2018, inclusively. Based on the year in which each radiologist completed his or her first test set, data of multiple clinical audit metrics from two calendar years before test-set reading were compared against similar data from three different periods after test-set completion. The same process was repeated after dividing radiologists into two subgroups based on their median screen-reading volume (3,688), to test if experience is a determinant of post-test set performance.

RESULTS

On average, radiologists showed significant improvements (p<0.05) in the recall rate for subsequent screening rounds, in positive predictive value 1 (PPV1), and in specificity. When dividing radiologists based on their average annual reading volume, radiologists with higher reading numbers demonstrated similar significant improvements in the recall rate and in PPV1. In addition, they showed significant improvements in the detection rates of invasive breast cancer and ductal carcinoma in situ (DCIS). In contrast, the radiologists with lower reading volume indicated significant changes in the recall rate and in PPV1, both accruing in one of the three compared periods.

CONCLUSION

Mammographic test-set participants improve over time in identifying normal breast screens and detecting breast cancer in association with reading higher volumes of breast screening cases.

Mammography self-evaluation online test for screening readers: an Italian Society of Medical Radiology (SIRM) initiative

OBJECTIVES

To report and analyse the characteristics and performance of the first cohort of Italian radiologists completing the national mammography self-evaluation online test established by the Italian Society of Medical Radiology (SIRM).

METHODS

A specifically-built dataset of 132 mammograms (24 with screen-detected cancers and 108 negative cases) was preliminarily tested on 48 radiologists to define pass thresholds (62% sensitivity and 86% specificity) and subsequently made available online to SIRM members during a 13-month timeframe between 2018 and 2019. Associations between participants' characteristics, pass rates, and diagnostic accuracy were then investigated with descriptive statistics and univariate and multivariable regression analyses.

RESULTS

A total of 342 radiologists completed the test, 151/342 (44.2%) with success. All individual variables, except gender, showed a significant correlation with pass rates and diagnostic sensitivity, confirmed by univariate logistic regression, while only involvement in organised screening programs and number of mammograms read per year showed a positive association with specificity at univariate logistic regression. In the multivariable regression analysis, fewer variables remained significant: > 3000 mammograms read per year for success rate; female gender, public practice setting, and higher experience self-judgement for sensitivity; no variables were significantly associated with specificity.

CONCLUSIONS

This national self-evaluation test effectively differentiated multiple aspects of mammographic reading experience, but specific breast imaging experience was shown not to strictly guarantee good diagnostic accuracy. Due to its easy use and the validity of obtained results, this test could be extended to all Italian breast radiologists, regardless of their experience, also as a Breast Unit accreditation criterion.

KEY POINTS

• This self-evaluation test was found to be able to differentiate various degrees of mammographic interpretation experience. • Breast cancer screening readers should undergo a self-assessment test, since experience parameters alone do not guarantee diagnostic ability.

The relationship between missed breast cancers on mammography in a test-set based assessment scheme and real-life performance in a National Breast Screening Programme

PURPOSE

This retrospective study determined whether a test-set based assessment scheme (PERFORMS) used in a national breast screening programme could be used to predict real-life performance by investigating if the number of cancers missed by mammography readers in real-life related to the number of cancers missed in the PERFORMS test-set and whether real-life reading volumes affected performance.

METHOD

Data was obtained from consenting readers in the screening programme in England (NHSBSP) where double reading is standard. The rate of cancers missed by individual first readers but correctly identified by second readers was compared with the number of cancers missed in the PERFORMS test-set over a 3-year period. NHSBSP readers are required to interpret at least 1500 cases per year as a first reader, so results were compared between readers who exceeded this target and those that did not. Parametric and non-parametric correlations were calculated.

RESULTS

Amongst the 536 readers, there was a highly significant positive correlation between the real-life and PERFORMS test-set missed cancer metrics (Pearson Correlation = 0.228, n = 536, p < .0001, Spearman's rho = 0.265, n = 536, p < .0001). There was no significant difference in rates of missed cancers between the 452 readers who exceeded the 1500 first read per year target and those who did not (t(94.2) = -1.87, p = .0643, r = 0.19).

CONCLUSIONS

The use of a test-set based assessment scheme accurately reflects real-life mammography reading performance, indicating that it can be a useful tool in identifying poor reader performance.

Artificial intelligence in breast cancer screening: primary care provider preferences

BACKGROUND

Artificial intelligence (AI) is increasingly being proposed for use in medicine, including breast cancer screening (BCS). Little is known, however, about referring primary care providers' (PCPs') preferences for this technology.

METHODS

We identified the most important attributes of AI BCS for ordering PCPs using qualitative interviews: sensitivity, specificity, radiologist involvement, understandability of AI decision-making, supporting evidence, and diversity of training data. We invited US-based PCPs to participate in an internet-based experiment designed to force participants to trade off among the attributes of hypothetical AI BCS products. Responses were analyzed with random parameters logit and latent class models to assess how different attributes affect the choice to recommend AI-enhanced screening.

RESULTS

Ninety-one PCPs participated. Sensitivity was most important, and most PCPs viewed radiologist participation in mammography interpretation as important. Other important attributes were specificity, understandability of AI decision-making, and diversity of data. We identified 3 classes of respondents: "Sensitivity First" (41%) found sensitivity to be more than twice as important as other attributes; "Against AI Autonomy" (24%) wanted radiologists to confirm every image; "Uncertain Trade-Offs" (35%) viewed most attributes as having similar importance. A majority (76%) accepted the use of AI in a "triage" role that would allow it to filter out likely negatives without radiologist confirmation.

CONCLUSIONS AND RELEVANCE

Sensitivity was the most important attribute overall, but other key attributes should be addressed to produce clinically acceptable products. We also found that most PCPs accept the use of AI to make determinations about likely negative mammograms without radiologist confirmation.

The roles of clinical audit and test sets in promoting the quality of breast screening: a scoping review

Breast screening programmes enhance the probability of early breast cancer detection in many countries worldwide; however, the success of these efforts is highly dependent on the ability of breast screen readers to detect abnormalities in the screened population, which has low prevalence. Therefore, this task can be challenging. Clinical audit is a key quality assurance measure that aims to keep the screen reading performance within acceptable standards. Auditing, nonetheless, is a lengthy process, and its accuracy is dependent on available clinical data, which often can be limited. Mammographic standardised test sets are a different screen reading evaluation approach that provides participants with instant feedback based on a simulated environment. Although a test set provides unique evaluative qualities, its ability to represent clinical performance is debated. This article describes the distinctive roles of clinical audit and test sets in measuring and improving the quality of breast screening and highlights the relationship between test sets and clinical performance.

Fatigue in radiology: a fertile area for future research

Fatigue in radiologists may be responsible for a large number of medical errors. This review describes the latest research on fatigue in radiology. This includes measurement methods, and recent evidence on how fatigue affects accuracy in laboratory test conditions and in clinical practice. The extensive opportunities for future research in the area are explored, including testing interventions to reduce fatigue-related error, and further understanding of which fatigue measures correlate with errors. Finally we explore the possibility of answering these questions using large population-based observational studies and pragmatic integrated randomised controlled trials.