Number of mammography cases read per year is a strong predictor of sensitivity

Test-set results can predict participants' development in breast-screen cancer detection: An observational cohort study

Background and Aim

Test-sets are standardized assessments used to evaluate reader performance in breast screening. Understanding how test-set results affect real-world performance can help refine their use as a quality improvement tool. The aim of this study is to explore if mammographic test-set results could identify breast-screening readers who improved their cancer detection in association with test-set training.

Methods

Test-set results of 41 participants were linked to their annual cancer detection rate change in two periods oriented around their first test-set participation year. Correlation tests and a multiple linear regression model investigated the relationship between each metric in the test-set results and the change in detection rates. Additionally, participants were divided based on their improvement status between the two periods, and Mann-Whitney U test was used to determine if the subgroups differed in their test-set metrics.

Results

Test-set records indicated multiple significant correlations with the change in breast cancer detection rate: a moderate positive correlation with sensitivity (0.688, p < 0.001), a moderate negative correlation with specificity (-0.528, p < 0.001), and a low to moderate positive correlation with lesion sensitivity (0.469, p = 0.002), and the number of years screen-reading mammograms (0.365, p = 0.02). In addition, the overall regression was statistically significant (F (2,38) = 18.456 p < 0.001), with an R² of 0.493 (adjusted R² = 0.466) based on sensitivity (F = 27.132, p < 0.001) and specificity (F = 9.78, p = 0.003). Subgrouping the cohort based on the change in cancer detection indicated that the improved group is significantly higher in sensitivity (p < 0.001) and lesion sensitivity (p = 0.02) but lower in specificity (p = 0.003).

Conclusion

Sensitivity and specificity are the strongest test-set performance measures to predict the change in breast cancer detection in real-world breast screening settings following test-set participation.

Do Reader Characteristics Affect Diagnostic Efficacy in Screening Mammography? A Systematic Review

To examine reader characteristics associated with diagnostic efficacy in the interpretation of screening mammograms. A systematic search of the literature was conducted using databases such as Cochrane, Scopus, Medline, Embase, Web of Science, and PubMed. Search terms were combined with "AND" or "OR" and included: "Radiologist's characteristics AND performance"; "radiologist experience AND screening mammography"; "annual volume read AND diagnostic efficacy"; "screening mammography performance OR diagnostic efficacy". Studies were included if they assessed reader performance in screening mammography interpretation, breast readers, used a reference standard to assess the performance, and were published in the English language. Twenty-eight studies were reviewed. Increasing reader's age was associated with lower false positive rates. No association was found between gender and performance. Half of the studies showed no association between years of reading mammograms and performance. Most studies showed that high reading volume was more likely to be associated with increased sensitivity, cancer detection rates (CDR), lower recall rate, and lower false positive rates. Inconsistent associations were found between fellowship training in breast imaging and reader performance. Specialization in breast imaging was associated with better CDR, sensitivity, and specificity. Limited studies were available to establish the association between performance and factors such as time spent in breast imaging (n = 2), screening focus (n = 1), formal rotation in mammography (n = 1), owner of practice (n = 1), and practice type (n = 1). No individual characteristics is associated with versatility in diagnostic efficacy, albeit reading volume and specialization in breast imaging appear to be associated with with increased sensitivity and CDR without significantly affecting other performance metrics.

A machine learning model based on readers' characteristics to predict their performances in reading screening mammograms

OBJECTIVES

Proposing a machine learning model to predict readers' performances, as measured by the area under the receiver operating characteristics curve (AUC) and lesion sensitivity, using the readers' characteristics.

METHODS

Data were collected from 905 radiologists and breast physicians who completed at least one case-set of 60 mammographic images containing 40 normal and 20 biopsy-proven cancer cases. Nine different case-sets were available. Using a questionnaire, we collected radiologists' demographic details, such as reading volume and years of experience. These characteristics along with a case set difficulty measure were fed into two ensemble of regression trees to predict the readers' AUCs and lesion sensitivities. We calculated the Pearson correlation coefficient between the predicted values by the model and the actual AUC and lesion sensitivity. The usefulness of the model to categorize readers as low and high performers based on different criteria was also evaluated. The performances of the models were evaluated using leave-one-out cross-validation.

RESULTS

The Pearson correlation coefficient between the predicted AUC and actual one was 0.60 (p < 0.001). The model's performance for differentiating the reader in the first and fourth quartile based on the AUC values was 0.86 (95% CI 0.83-0.89). The model reached an AUC of 0.91 (95% CI 0.88-0.93) for distinguishing the readers in the first quartile from the fourth one based on the lesion sensitivity.

CONCLUSION

A machine learning model can be used to categorize readers as high- or low-performing. Such model could be useful for screening programs for designing a targeted quality assurance and optimizing the double reading practice.

Breast cancer detection across dense and non-dense breasts: Markers of diagnostic confidence and efficacy

Background

The impact of radiologists’ characteristics has become a major focus of recent research. However, the markers of diagnostic efficacy and confidence in dense and non-dense breasts are poorly understood.

Purpose

This study aims to assess the relationship between radiologists’ characteristics and diagnostic performance across dense and non-dense breasts.

Materials and methods

Radiologists specialising in breast imaging (n = 128) who had 0.5–40 (13±10.6) years of experience reading mammograms were recruited. Participants independently interpreted a test set containing 60 digital mammograms (40 normal and 20 abnormal) with similarly distributed breast densities. Diagnostic performance measures were analysed via Jamovi software (version 1.6.22).

Results

In dense breasts, breast-imaging fellowship completion significantly improved specificity (p = 0.004), location sensitivity (p = 0.01) and the area under the curve (AUC) of the receiver operating characteristic (p = 0.03). Only participation in BreastScreen reading significantly improved all performance metrics: specificity (p = 0.04), sensitivity (p = 0.005), location sensitivity (p < 0.001) and AUC (p < 0.001). Reading > 100 mammograms weekly significantly improved sensitivity (p = 0.03), location sensitivity (p = 0.001), and AUC (p = 0.03).In non-dense breasts, breast fellowship completion significantly improved sensitivity (p = 0.02), location sensitivity (p = 0.04) and AUC (p = 0.002). Participation in BreastScreen reading and reading > 100 mammograms weekly significantly improved only sensitivity (p = 0.002 and p = 0.003, respectively) and location sensitivity (p < 0.001 and p < 0.001, respectively).

Conclusion

Participating in screening programs, breast fellowships and reading > 100 mammograms weekly are important indicators of the diagnostic performance of radiologists across dense and non-dense breasts. In dense breasts, optimal performance resulted from participation in a breast screening program.

SCREENING MAMMOGRAPHY: DIAGNOSTIC EFFICACY-ISSUES AND CONSIDERATIONS FOR THE 2020S

Diagnostic efficacy in medical imaging is ultimately a reflection of radiologist performance. This can be influenced by numerous factors, some of which are patient related, such as the physical size and density of the breast, and machine related, where some lesions are difficult to visualise on traditional imaging techniques. Other factors are human reader errors that occur during the diagnostic process, which relate to reader experience and their perceptual and cognitive oversights. Given the large-scale nature of breast cancer screening, even small increases in diagnostic performance equate to large numbers of women saved. It is important to identify the causes of diagnostic errors and how detection efficacy can be improved. This narrative review will therefore explore the various factors that influence mammographic performance and the potential solutions used in an attempt to ameliorate the errors made.

Differences in lesion interpretation between radiologists in two countries: Lessons from a digital breast tomosynthesis training test set

INTRODUCTION

In many western countries, there is good evidence documenting the performance of radiologists reading digital breast tomosynthesis (DBT) images. However, the diagnostic efficiency of Chinese radiologists using DBT, particularly type of errors being made and type of cancers being missed, is understudied. This study aims to investigate the pattern of diagnostic errors across different lesion types produced by Chinese radiologists diagnosing from DBT images. Australian radiologists will be used as a benchmark.

METHODS

Twelve Chinese radiologists read a DBT test set and located each perceived cancer lesion. True positives, false positives (FP), true negatives and false negatives (FN) were generated. The same test set was also read by 14 Australian radiologists. Z-scores and Pearson correlations were used to compare interpretation of lesions and identification of normal appearances between two groups of radiologists.

RESULTS

Architectural distortions (p < .001) and stellate masses (p = .02) were more difficult for Chinese radiologists to correctly diagnose compared to their Australian counterparts. Chinese readers categorised more FPs as discrete masses (p < .001) and fewer FPs as architectural distortions (p < .001) comparing with Australian radiologists. The percentages of FN for each cancer case were not correlated (r = 0.37, p = .18) but the percentages of FP for each normal case were moderately correlated (r = 0.52, p = .02) between two groups of readers.

CONCLUSIONS

Architectural distortions and stellate masses were challenging to Chinese radiologists when reading DBT. Our findings proposed the need of development of training and education programs focussing on imaging cases tailored for specific groups of readers with certain interpretation patterns.

Mammography self-evaluation online test for screening readers: an Italian Society of Medical Radiology (SIRM) initiative

OBJECTIVES

To report and analyse the characteristics and performance of the first cohort of Italian radiologists completing the national mammography self-evaluation online test established by the Italian Society of Medical Radiology (SIRM).

METHODS

A specifically-built dataset of 132 mammograms (24 with screen-detected cancers and 108 negative cases) was preliminarily tested on 48 radiologists to define pass thresholds (62% sensitivity and 86% specificity) and subsequently made available online to SIRM members during a 13-month timeframe between 2018 and 2019. Associations between participants' characteristics, pass rates, and diagnostic accuracy were then investigated with descriptive statistics and univariate and multivariable regression analyses.

RESULTS

A total of 342 radiologists completed the test, 151/342 (44.2%) with success. All individual variables, except gender, showed a significant correlation with pass rates and diagnostic sensitivity, confirmed by univariate logistic regression, while only involvement in organised screening programs and number of mammograms read per year showed a positive association with specificity at univariate logistic regression. In the multivariable regression analysis, fewer variables remained significant: > 3000 mammograms read per year for success rate; female gender, public practice setting, and higher experience self-judgement for sensitivity; no variables were significantly associated with specificity.

CONCLUSIONS

This national self-evaluation test effectively differentiated multiple aspects of mammographic reading experience, but specific breast imaging experience was shown not to strictly guarantee good diagnostic accuracy. Due to its easy use and the validity of obtained results, this test could be extended to all Italian breast radiologists, regardless of their experience, also as a Breast Unit accreditation criterion.

KEY POINTS

• This self-evaluation test was found to be able to differentiate various degrees of mammographic interpretation experience. • Breast cancer screening readers should undergo a self-assessment test, since experience parameters alone do not guarantee diagnostic ability.

Reader characteristics and mammogram features associated with breast imaging reporting scores

Objectives:

This study aims to explore the reading performances of radiologists in detecting cancers on mammograms using Tabar Breast Imaging Reporting and Data System (BIRADS) classification and identify factors related to breast imaging reporting scores.

Methods:

117 readings of five different mammogram test sets with each set containing 20 cancer and 40 normal cases were performed by Australian radiologists. Each radiologist evaluated the mammograms using the BIRADS lexicon with category 1 - negative, category 2 - benign findings, category 3 - equivocal findings (Recall), category 4 - suspicious findings (Recall), and category 5 - highly suggestive of malignant findings (Recall). Performance metrics (true positive, false positive, true negative, and false negative) were calculated for each radiologist and the distribution of reporting categories was analyzed in reader-based and case-based groups. The association of reader characteristics and case features among categories was examined using Mann-Whitney U and Kruskal-Wallis tests.

Results:

38% of cancer-containing mammograms were reported with category 3 which decreased to 32.3% with category 4 and 16.2% with category 5 while 16.6 and 10.3% of cancer cases were marked with categories 1 and 2. Female readers had less false-negative rates when using categories 1 and 2 for cancer cases than male readers (p < 0.01). A similar pattern as gender category was also found in Breast Screen readers and readers completed breast reading fellowships compared with non-Breast Screen and non-fellowship readers (p < 0.05). Radiologists with low number of cases read per week were more likely to record the cancer cases with category 4 while the ones with high number of cases were with category 3 (p < 0.01). Discrete mass and asymmetric density were the two types of abnormalities reported mostly as equivocal findings with category 3 (47–50%; p = 0.005) while spiculated mass or stellate lesions were mostly selected as highly suggestive of malignancy with category 5 (26%, p = 0.001).

Conclusions:

Most radiologists used category 3 when reporting cancer mammograms. Gender, working for BreastScreen, fellowship completion, and number of cases read per week were factors associated with scoring selection. Radiologists reported higher Tabar BIRADS category for specific types of abnormalities on mammograms than others.

Advances in knowledge:

The study identified factors associated with the decision of radiologists in assigning a BIRADS Tabar score for mammograms with abnormality. These findings will be useful for individual training programs to improve the confidence of radiologists in recognizing abnormal lesions on screening mammograms.

Interpretative characteristics and case features associated with the performances of radiologists in reading mammograms: A study from a non-screening population in Asia

AIMS

To explore radiologist characteristics and case features associated with diagnostic performances in cancer detection on mammograms in a South East Asian population.

METHODS

Fifty-three radiologists reported 60 mammographic examinations which consisted of 40 normal and 20 cancer-containing cases at the BREAST workshops. Radiologists were asked to examine each mammogram using the BIRADS on diagnostic monitors. Differences in reader characteristics and case features between correct and incorrect decisions were assessed separately for cancer and normal cases. Univariate and multivariate logistic regressions were applied to generate odds ratios (OR) for significant factors related to correct decisions.

RESULTS

Radiologists who spent ≥10 hours/week reporting mammograms had a higher possibility of detecting cancer lesions (OR = 1.6; P = 0.01). A higher rate of accuracy in reporting negative cases was associated with female radiologists (OR = 1.4; P = 0.002), radiologists who read ≤20 mammograms per week (OR = 1.5; P < 0.0001), had completed training course (OR = 1.7; P < 0.0001) or wore eyeglasses (OR = 1.4; P = 0.01). Cancer cases with breast density >50% (OR = 2.1; P < 0.0001), having abnormal lesions ≥9 mm (OR = 1.8; P < 0.0001), or displaying calcifications, a discrete mass or nonspecific density (OR = 1.6; P < 0.0001) were recorded with a higher detection rate by radiologists than other cases. Lesions located on the right breasts (OR = 1.8; P < 0.0001) or found in the lower inner, upper outer or mixed locations (OR = 2.7; P < 0.0001) were also recorded with a better diagnostic possibility compared with other lesions.

CONCLUSION

This work identified key features related to diagnostic accuracy of breast cancer on mammograms in a nonscreening population, which is helpful for developing appropriate strategies to improve breast cancer detectability of radiologists.

The roles of clinical audit and test sets in promoting the quality of breast screening: a scoping review

Breast screening programmes enhance the probability of early breast cancer detection in many countries worldwide; however, the success of these efforts is highly dependent on the ability of breast screen readers to detect abnormalities in the screened population, which has low prevalence. Therefore, this task can be challenging. Clinical audit is a key quality assurance measure that aims to keep the screen reading performance within acceptable standards. Auditing, nonetheless, is a lengthy process, and its accuracy is dependent on available clinical data, which often can be limited. Mammographic standardised test sets are a different screen reading evaluation approach that provides participants with instant feedback based on a simulated environment. Although a test set provides unique evaluative qualities, its ability to represent clinical performance is debated. This article describes the distinctive roles of clinical audit and test sets in measuring and improving the quality of breast screening and highlights the relationship between test sets and clinical performance.

Breast Cancer Diagnostic Efficacy in a Developing South-East Asian Country

Background: Breast cancer, is increasing in prevalence amongst South East (SE) Asian women, highlighting the need for high quality, early diagnoses. This study investigated radiologists’ detection efficacy in a developing (DC) and developed (DDC) SE Asian country, as compared to Australian radiologists. Methods: Using a test-set of 60 mammographic cases, 20 containing cancer, JAFROC figures of merit (FOM) and ROC area under the curves (AUC) were calculated as well as location sensitivity, sensitivity and specificity. The test set was examined by 35, 15, and 53 radiologists from DC, a DDC and Australia, respectively. Results: DC radiologists, compared to both groups of counterparts, demonstrated significantly lower JAFROC FOM, ROC AUC and specificity scores. DC radiologists had a significantly lower location sensitivity than Australian radiologists. DC radiologists also demonstrated significantly lower values for age, hours of reading per week, and years of mammography experience when compared with other radiologists. Conclusion: Significant differences in breast cancer detection parameters can be attributed to the experience of DC radiologists. The development of inexpensive, innovative, interactive training programs are discussed. This nonuniform level of breast cancer detection between countries must be addressed to achieve the World Health Organisation goal of health equity.

Observer Variability in Breast Cancer Diagnosis between Countries with and without Breast Screening

RATIONAL AND OBJECTIVES

Image reporting is a vital component of patient management depending on individual radiologists' performance. Our objective was to explore mammographic diagnostic efficacy in a country where breast cancer screening does not exist.

MATERIALS AND METHODS

Two mammographic test sets were used: a typical screening (TS) and high-difficulty (HD) test set. Nonscreening (NS) radiologists (n = 11) read both test sets, while 52 and 49 screening radiologists read the TS and HD test sets, respectively. The screening radiologists were classified into two groups: a less experienced (LE) group with ≤5 years' experience and a more experienced (ME) group with ≥5 years' experience. A Kruskal-Wallis and Tukey-Kramer post hoc test were used to compare reading performance among reader groups, and the Wilcoxon matched pairs tests was used to compare TS and ND test sets for the NS radiologists.

RESULTS

Across the three reader groups, there were significant differences in case sensitivity (χ² [2] = 9.4, P = .008), specificity (χ² [2] = 10.3, P = .006), location sensitivity (χ² [2] = 19.8, P < .001), receiver operating characteristics, area under the curve (χ² [2] = 19.7, P < .001) and jack-knife free-response receiver operating characteristics (JAFROCs) (χ² [2] = 18.1, P < .001). NS performance for all measured scores was significantly lower than those for the ME readers (P < .006), while only location sensitivity was lower (χ² [2] = 17.5, P = .026) for the NS compared to the LE group. No other significant differences were observed.

CONCLUSION

Large variations in mammographic performance exist between radiologists from screening and nonscreening countries.

Mammographic detection of breast cancer in a non-screening country

OBJECTIVE:

To compare the diagnostic accuracy between radiologists' from a country with and without breast cancer screening.

METHODS:

All participating radiologists gave informed consent. A test-set involving 60 mammographic cases (20 cancer and 40 non-cancer) were read by 11 radiologists from a non-screening (NS) country during a workshop in July 2016. 52 radiologists from a screening country read the same test-set at the Royal Australian and New Zealand College of Radiologists' meetings in July 2015. The screening radiologists were classified into two groups: those with less than or equal to 5 years of experience; those with more than 5 years of experience, and each group was compared to the group of NS radiologists. A Kruskal-Wallis test followed by post-hoc multiple comparisons test were used to compare measures of diagnostic accuracy among the reader groups.

RESULTS:

The diagnostic accuracy of the NS radiologists was significantly lower in terms of sensitivity [mean = 54.0; 95% confidence interval (CI) (40.0-67.0)], location sensitivity [mean = 26.0; 95% CI (16.0-37.0)], receive roperating characteristic area under curve [mean = 73.0; 95% CI (66.5-81.0)] and Jackknifefree-response receiver operating characteristics figure-of-merit [mean = 45.0; 95% CI (40.0-50.0)] when compared with the less and more experienced screening radiologists, whilst no difference in specificity [mean = 75.0; 95% CI (70.0- 81.0)] was found. No significant differences in all measured diagnostic accuracy were found between the two groups of screening radiologists.

CONCLUSION:

The mammographic performance of a group of radiologists from a country without screening program was suboptimal compared with radiologists from Australia.

ADVANCES IN KNOWLEDGE:

Identifying mammographic performance in developing countries is required to optimize breast cancer diagnosis.

Errors in Mammography Cannot be Solved Through Technology Alone

Mammography has been the frontline screening tool for breast cancer for decades. However, high error rates in the form of false negatives (FNs) and false positives (FPs) have persisted despite technological improvements. Radiologists still miss between 10% and 30% of cancers while 80% of woman recalled for additional views have normal outcomes, with 40% of biopsied lesions being benign. Research show that the majority of cancers missed is actually visible and looked at, but either go unnoticed or are deemed to be benign. Causal agents for these errors include human related characteristics resulting in contributory search, perception and decision-making behaviours. Technical, patient and lesion factors are also important relating to positioning, compression, patient size, breast density and presence of breast implants as well as the nature and subtype of the cancer itself, where features such as architectural distortion and triple-negative cancers remain challenging to detect on screening. A better understanding of these causal agents as well as the adoption of technological and educational interventions, which audits reader performance and provide immediate perceptual feedback, should help. This paper reviews the current status of our knowledge around error rates in mammography and explores the factors impacting it. It also presents potential solutions for maximizing diagnostic efficacy thus benefiting the millions of women who undergo this procedure each year.

Personal and Network Dynamics in Performance of Knowledge Workers: A Study of Australian Breast Radiologists

Materials and Methods

In this paper, we propose a theoretical model based upon previous studies about personal and social network dynamics of job performance. We provide empirical support for this model using real-world data within the context of the Australian radiology profession. An examination of radiologists’ professional network topology through structural-positional and relational dimensions and radiologists’ personal characteristics in terms of knowledge, experience and self-esteem is provided. Thirty one breast imaging radiologists completed a purpose designed questionnaire regarding their network characteristics and personal attributes. These radiologists also independently read a test set of 60 mammographic cases: 20 cases with cancer and 40 normal cases. A Jackknife free response operating characteristic (JAFROC) method was used to measure the performance of the radiologists’ in detecting breast cancers.

Results

Correlational analyses showed that reader performance was positively correlated with the social network variables of degree centrality and effective size, but negatively correlated with constraint and hierarchy. For personal characteristics, the number of mammograms read per year and self-esteem (self-evaluation) positively correlated with reader performance. Hierarchical multiple regression analysis indicated that the combination of number of mammograms read per year and network’s effective size, hierarchy and tie strength was the best fitting model, explaining 63.4% of the variance in reader performance. The results from this study indicate the positive relationship between reading high volumes of cases by radiologists and expertise development, but also strongly emphasise the association between effective social/professional interactions and informal knowledge sharing with high performance.