The roles of clinical audit and test sets in promoting the quality of breast screening: a scoping review

Radiomic analysis of cohort-specific diagnostic errors in reading dense mammograms using artificial intelligence

OBJECTIVES

This study aims to investigate radiologists' interpretation errors when reading dense screening mammograms using a radiomics-based artificial intelligence approach.

METHODS

Thirty-six radiologists from China and Australia read 60 dense mammograms. For each cohort, we identified normal areas that looked suspicious of cancer and the malignant areas containing cancers. Then radiomic features were extracted from these identified areas and random forest models were trained to recognize the areas that were most frequently linked to diagnostic errors within each cohort. The performance of the model and discriminatory power of significant radiomic features were assessed.

RESULTS

We found that in the Chinese cohort, the AUC values for predicting false positives were 0.864 (CC) and 0.829 (MLO), while in the Australian cohort, they were 0.652 (CC) and 0.747 (MLO). For false negatives, the AUC values in the Chinese cohort were 0.677 (CC) and 0.673 (MLO), and in the Australian cohort, they were 0.600 (CC) and 0.505 (MLO). In both cohorts, regions with higher Gabor and maximum response filter outputs were more prone to false positives, while areas with significant intensity changes and coarse textures were more likely to yield false negatives.

CONCLUSIONS

This cohort-based pipeline proves effective in identifying common errors for specific reader cohorts based on image-derived radiomic features.

ADVANCES IN KNOWLEDGE

This study demonstrates that radiomics-based AI can effectively identify and predict radiologists' interpretation errors in dense mammograms, with distinct radiomic features linked to false positives and false negatives in Chinese and Australian cohorts.

An initial exploration of factors that may impact radiographer performance in reporting mammograms

OBJECTIVES

In the United Kingdom, radiographers with a qualification in image interpretation have interpreted mammograms since 1995. These radiographers work under the title of radiography advanced practitioners (RAP) or Consultant Radiographer. This study extends upon what has been very recently published by exploring further clinical, non-clinical and experiential factors that may impact the reporting performance of RAPs.

METHODS

Fifteen RAPs interpreted an image test set of 60 2D mammograms of known truth using the Detected-X software platform. Unknown to the reader, twenty cases contained a malignancy. Sensitivity, specificity, lesion sensitivity, receiver operating characteristic (ROC) and jack-knife free response operating characteristic (AFROC) values were established for each RAP. Specific features that had significant impact on accuracy were identified using Student's-T and Mann Whitney tests.

RESULTS

RAPs with more than 10 years' experience in image interpretation, compared to those with less than 10 years' experience, demonstrated lower specificity (51.3% vs 84.8%, p = 0.0264), ROC (0.83 vs 0.91, p = 0.0264) and AFROC (0.75 vs 0.87, p = 0.0037) values. Further, higher sensitivity values of 90.7% were seen in those RAPs who had an eye test in the last year compared to those who had not, 82% (p = 0.021). Other changes are presented in the paper.

CONCLUSION

These data reveal previously unidentified factors that impact the diagnostic efficacy of RAPs when interpreting mammographic images. Highlighting such findings will empower screening authorities to better examine ways of standardising performance and offer a baseline for performance benchmarks.

IMPLICATIONS FOR PRACTICE

This study for the first time performs an initial exploration of the factors that may be associated with RAP performance when interpreting screening mammograms.

Test-set results can predict participants' development in breast-screen cancer detection: An observational cohort study

Background and Aim

Test-sets are standardized assessments used to evaluate reader performance in breast screening. Understanding how test-set results affect real-world performance can help refine their use as a quality improvement tool. The aim of this study is to explore if mammographic test-set results could identify breast-screening readers who improved their cancer detection in association with test-set training.

Methods

Test-set results of 41 participants were linked to their annual cancer detection rate change in two periods oriented around their first test-set participation year. Correlation tests and a multiple linear regression model investigated the relationship between each metric in the test-set results and the change in detection rates. Additionally, participants were divided based on their improvement status between the two periods, and Mann-Whitney U test was used to determine if the subgroups differed in their test-set metrics.

Results

Test-set records indicated multiple significant correlations with the change in breast cancer detection rate: a moderate positive correlation with sensitivity (0.688, p < 0.001), a moderate negative correlation with specificity (-0.528, p < 0.001), and a low to moderate positive correlation with lesion sensitivity (0.469, p = 0.002), and the number of years screen-reading mammograms (0.365, p = 0.02). In addition, the overall regression was statistically significant (F (2,38) = 18.456 p < 0.001), with an R² of 0.493 (adjusted R² = 0.466) based on sensitivity (F = 27.132, p < 0.001) and specificity (F = 9.78, p = 0.003). Subgrouping the cohort based on the change in cancer detection indicated that the improved group is significantly higher in sensitivity (p < 0.001) and lesion sensitivity (p = 0.02) but lower in specificity (p = 0.003).

Conclusion

Sensitivity and specificity are the strongest test-set performance measures to predict the change in breast cancer detection in real-world breast screening settings following test-set participation.

Using Radiomics-Based Machine Learning to Create Targeted Test Sets to Improve Specific Mammography Reader Cohort Performance: A Feasibility Study

Mammography interpretation is challenging with high error rates. This study aims to reduce the errors in mammography reading by mapping diagnostic errors against global mammographic characteristics using a radiomics-based machine learning approach. A total of 36 radiologists from cohort A (n = 20) and cohort B (n = 16) read 60 high-density mammographic cases. Radiomic features were extracted from three regions of interest (ROIs), and random forest models were trained to predict diagnostic errors for each cohort. Performance was evaluated using sensitivity, specificity, accuracy, and AUC. The impact of ROI placement and normalization on prediction was investigated. Our approach successfully predicted both the false positive and false negative errors of both cohorts but did not consistently predict location errors. The errors produced by radiologists from cohort B were less predictable compared to those in cohort A. The performance of the models did not show significant improvement after feature normalization, despite the mammograms being produced by different vendors. Our novel radiomics-based machine learning pipeline focusing on global radiomic features could predict false positive and false negative errors. The proposed method can be used to develop group-tailored mammographic educational strategies to help improve future mammography reader performance.

Test-set training improves the detection rates of invasive cancer in screening mammography

AIM

To investigate if mammographic test-set participation affects routine breast cancer screening performance.

MATERIALS AND METHODS

Clinical audit data between 2008 and 2018 were collected for 35 breast screen readers who participated in the BreastScreen Reader Assessment Strategy (BREAST) and 22 readers with no history of test-set participation. For BREAST readers, the annual audit data were divided according to the year they completed their first test set, and the same years were used randomly to align and divide the data of non-BREAST readers into pre- and post-test set periods. Multiple audit parameters were inspected retrospectively for the two cohorts to identify how their reading performance has evolved in screening mammography.

RESULTS

Investigating 2 calendar years before and after test-set participation, BREAST and non-BREAST readers recalled lower rates of women in the latter period (p=0.03 and p=0.02, respectively). They also improved their positive predictive value (PPV; p=0.01 and p=0.02, respectively). BREAST readers additionally improved their detection rates of invasive cancer (p=0.02) and all cancers (p=0.01). In an extended 3-year comparison, similar improvements occurred in the recall rate for BREAST (p=0.02) and non-BREAST readers (p=0.02) and in PPV (p=0.001, 0.01, respectively); however, improvements in detection rates also occurred exclusively in BREAST readers' performance for invasive cancer (p=0.04), DCIS (p=0.05), and all cancers (p=0.02); however, significant improvements in detection did not involve <15 mm invasive cancers in both periods. Meanwhile, non-BREAST readers demonstrated a decrease in sensitivity (p=0.02).

CONCLUSION

Participation in test sets is linked to over-time improvements in most audit-measured cancer detection rates.

Differences in lesion interpretation between radiologists in two countries: Lessons from a digital breast tomosynthesis training test set

INTRODUCTION

In many western countries, there is good evidence documenting the performance of radiologists reading digital breast tomosynthesis (DBT) images. However, the diagnostic efficiency of Chinese radiologists using DBT, particularly type of errors being made and type of cancers being missed, is understudied. This study aims to investigate the pattern of diagnostic errors across different lesion types produced by Chinese radiologists diagnosing from DBT images. Australian radiologists will be used as a benchmark.

METHODS

Twelve Chinese radiologists read a DBT test set and located each perceived cancer lesion. True positives, false positives (FP), true negatives and false negatives (FN) were generated. The same test set was also read by 14 Australian radiologists. Z-scores and Pearson correlations were used to compare interpretation of lesions and identification of normal appearances between two groups of radiologists.

RESULTS

Architectural distortions (p < .001) and stellate masses (p = .02) were more difficult for Chinese radiologists to correctly diagnose compared to their Australian counterparts. Chinese readers categorised more FPs as discrete masses (p < .001) and fewer FPs as architectural distortions (p < .001) comparing with Australian radiologists. The percentages of FN for each cancer case were not correlated (r = 0.37, p = .18) but the percentages of FP for each normal case were moderately correlated (r = 0.52, p = .02) between two groups of readers.

CONCLUSIONS

Architectural distortions and stellate masses were challenging to Chinese radiologists when reading DBT. Our findings proposed the need of development of training and education programs focussing on imaging cases tailored for specific groups of readers with certain interpretation patterns.

Clinical performance progress of BREAST participants: the impact of test-set participation

AIM

To investigate if positive changes in the clinical performance of radiologists are associated with reading mammographic test sets.

MATERIALS AND METHODS

This study investigated the clinical audit history for a cohort of 39 participants in the BreastScreen Reader Assessment Strategy who have read for BreastScreen New South Wales in the period between 2010 and 2018, inclusively. Based on the year in which each radiologist completed his or her first test set, data of multiple clinical audit metrics from two calendar years before test-set reading were compared against similar data from three different periods after test-set completion. The same process was repeated after dividing radiologists into two subgroups based on their median screen-reading volume (3,688), to test if experience is a determinant of post-test set performance.

RESULTS

On average, radiologists showed significant improvements (p<0.05) in the recall rate for subsequent screening rounds, in positive predictive value 1 (PPV1), and in specificity. When dividing radiologists based on their average annual reading volume, radiologists with higher reading numbers demonstrated similar significant improvements in the recall rate and in PPV1. In addition, they showed significant improvements in the detection rates of invasive breast cancer and ductal carcinoma in situ (DCIS). In contrast, the radiologists with lower reading volume indicated significant changes in the recall rate and in PPV1, both accruing in one of the three compared periods.

CONCLUSION

Mammographic test-set participants improve over time in identifying normal breast screens and detecting breast cancer in association with reading higher volumes of breast screening cases.

Mammography self-evaluation online test for screening readers: an Italian Society of Medical Radiology (SIRM) initiative

OBJECTIVES

To report and analyse the characteristics and performance of the first cohort of Italian radiologists completing the national mammography self-evaluation online test established by the Italian Society of Medical Radiology (SIRM).

METHODS

A specifically-built dataset of 132 mammograms (24 with screen-detected cancers and 108 negative cases) was preliminarily tested on 48 radiologists to define pass thresholds (62% sensitivity and 86% specificity) and subsequently made available online to SIRM members during a 13-month timeframe between 2018 and 2019. Associations between participants' characteristics, pass rates, and diagnostic accuracy were then investigated with descriptive statistics and univariate and multivariable regression analyses.

RESULTS

A total of 342 radiologists completed the test, 151/342 (44.2%) with success. All individual variables, except gender, showed a significant correlation with pass rates and diagnostic sensitivity, confirmed by univariate logistic regression, while only involvement in organised screening programs and number of mammograms read per year showed a positive association with specificity at univariate logistic regression. In the multivariable regression analysis, fewer variables remained significant: > 3000 mammograms read per year for success rate; female gender, public practice setting, and higher experience self-judgement for sensitivity; no variables were significantly associated with specificity.

CONCLUSIONS

This national self-evaluation test effectively differentiated multiple aspects of mammographic reading experience, but specific breast imaging experience was shown not to strictly guarantee good diagnostic accuracy. Due to its easy use and the validity of obtained results, this test could be extended to all Italian breast radiologists, regardless of their experience, also as a Breast Unit accreditation criterion.

KEY POINTS

• This self-evaluation test was found to be able to differentiate various degrees of mammographic interpretation experience. • Breast cancer screening readers should undergo a self-assessment test, since experience parameters alone do not guarantee diagnostic ability.