A review of the PERFORMS scheme in breast screening

Access to prior screening mammograms affects the specificity but not sensitivity of radiologists' performance

AIMS

To establish the impact that access to prior mammograms has on radiologists' performance and the influence of radiologists' characteristics and breast density on their subsequent performance.

METHODS

Eight participants independently interpreted 72 digital screening mammograms in two reading sessions using the Royal Australian and New Zealand College of Radiologist's classification. In the first reading session, participants were given access to current and prior mammograms. In the second reading session six months later, participants only had access to the current mammograms. Radiologists' specificity, sensitivity, lesion sensitivity, Receiver Operating Characteristic (ROC) curve, and Jacknife Alternative Free-response ROC (JAFROC) were calculated. A Paired T-test was used to compare readings with and without prior mammograms, and to assess if breast density influenced participants performance. Independent Sample T-test was used to compare performance across radiologists' characteristics. A relative risk analysis was conducted to assess the probability of false positives and false negatives when prior mammograms were available.

RESULTS

Access to prior mammograms improved specificity in dense and non-dense breasts (p≤0.01) and reduced false positives (p = 0.01) but had no effect on sensitivity (p = 0.37), lesion sensitivity (p = 0.67), ROC (p = 0.16), and JAFROC (p = 0.24). Prior mammogram also reduced the probability of false positives (RR = 0.38; 95%CI:0.26-0.57, p<0.0001) without affecting the false negative rate (RR = 1.14; 95%CI:0.88-1.49, p = 0.30). The impact of prior mammograms on performance was not influenced by breast density or radiologists' characteristics.

CONCLUSIONS

Access to prior mammograms improves radiologists' specificity and reduces false positives without affecting sensitivity and the false negative rate regardless of radiologists' characteristics and breast density.

Optimizing mammography interpretation education: leveraging deep learning for cohort-specific error detection to enhance radiologist training

Purpose

Accurate interpretation of mammograms presents challenges. Tailoring mammography training to reader profiles holds the promise of an effective strategy to reduce these errors. This proof-of-concept study investigated the feasibility of employing convolutional neural networks (CNNs) with transfer learning to categorize regions associated with false-positive (FP) errors within screening mammograms into categories of "low" or "high" likelihood of being a false-positive detection for radiologists sharing similar geographic characteristics.

Approach

Mammography test sets assessed by two geographically distant cohorts of radiologists (cohorts A and B) were collected. FP patches within these mammograms were segmented and categorized as "difficult" or "easy" based on the number of readers committing FP errors. Patches outside 1.5 times the interquartile range above the upper quartile were labeled as difficult, whereas the remaining patches were labeled as easy. Using transfer learning, a patch-wise CNN model for binary patch classification was developed utilizing ResNet as the feature extractor, with modified fully connected layers for the target task. Model performance was assessed using 10-fold cross-validation.

Results

Compared with other architectures, the transferred ResNet-50 achieved the highest performance, obtaining receiver operating characteristics area under the curve values of 0.933 ( ± 0.012 ) and 0.975 ( ± 0.011 ) on the validation sets for cohorts A and B, respectively.

Conclusions

The findings highlight the feasibility of employing CNN-based transfer learning to predict the difficulty levels of local FP patches in screening mammograms for specific radiologist cohort with similar geographic characteristics.

A comparison of breast screening performance based on a standardised test, during and outside of the COVID-19 lockdown period

PURPOSE

In the UK, the COVID-19 pandemicand the resulting lockdown significantly impacted routine breast screening and led to reduced case volumes for breast screening readers. The aim of the study was to evaluate whether breast screening performance on a test-set based assessment scheme (PERFORMS) varied over time, in association with the lockdown period.

METHOD

In this retrospective study, performance data were obtained for all breast cancer screening readers in England who completed the PERFORMS schemes based on digital mammography (from 2015 to 2022). Cancer detection (sensitivity), correct return to screen (specificity) and the difference between the sensitivity and specificity were calculated for each breast screener in each PERFORMS scheme.

RESULTS

During the study period, 4906 readers participated in eight PERFORMS schemes. Both the cancer detection (H(7) = 775.56, p <.00001) and correct return to screen rate (H(7) = 401.13, p <.00001) varied significantly by scheme. The difference between cancer detection and correct return to screen rate in the scheme administered during the lockdown period was significantly higher than in all other schemes (p <.05, Bonferroni correction applied).

CONCLUSIONS

The performance of breast screeners on the PERFORMS test-set based assessment scheme which was administered during the lockdown period, was characterised by markedly high sensitivity and low specificity when compared to their performance on other tests taken pre-COVID and following the lockdown.

Mammography Breast Cancer Screening Triage Using Deep Learning: A UK Retrospective Study

Background Breast screening enables early detection of cancers; however, most women have normal mammograms, resulting in repetitive and resource-intensive reading tasks. Purpose To investigate if deep learning (DL) algorithms can be used to triage mammograms by identifying normal results to reduce workload or flag cancers that may be overlooked. Materials and Methods In this retrospective study, three commercial DL algorithms were investigated using consecutive mammograms from two UK Breast Screening Program sites from January 2015 to December 2017 and January 2017 to December 2018 on devices from two mammography vendors. Normal mammograms with a 3-year follow-up and histopathologically proven cancer detected at screening, the subsequent round, or in the 3-year interval were included. Two algorithm thresholds were set: in scenario A, 99.0% sensitivity for rule-out triage to a lone reader, and in scenario B, approximately 1.0% additional recall providing a rule-in triage for further assessment. Both thresholds were then applied to the screening workflow in scenario C. The sensitivity and specificity were used to assess the overall predictive performance of each DL algorithm. Results The data set comprised 78 849 patients (median age, 59 years [IQR, 53-63 years]) and 887 screening-detected, 439 interval, and 688 subsequent screening round-detected cancers. In scenario A (rule-out triage), models DL-1, DL-2, and DL-3 triaged 35.0% (27 565 of 78 849), 53.2% (41 937 of 78 849), and 55.6% (43 869 of 78 849) of mammograms, respectively, with 0.0% (0 of 887) to 0.1% (one of 887) of screening-detected cancers undetected. In scenario B, DL algorithms triaged in 4.6% (20 of 439) to 8.2% (36 of 439) of interval and 5.2% (36 of 688) to 6.1% (42 of 688) of subsequent-round cancers when applied after the routine double-reading workflow. Combining both approaches in scenario C resulted in an overall noninferior specificity (difference, -0.9%; P < .001) and superior sensitivity (difference, 2.7%; P < .001) for the adaptive workflow compared with routine double reading for all three algorithms. Conclusion Rule-out and rule-in DL-adapted triage workflows can improve the efficiency and efficacy of mammography breast cancer screening. © RSNA, 2023 Supplemental material is available for this article. See also the editorial by Nishikawa and Lu in this issue.

Global Radiomic Features from Mammography for Predicting Difficult-To-Interpret Normal Cases

This work aimed to investigate whether global radiomic features (GRFs) from mammograms can predict difficult-to-interpret normal cases (NCs). Assessments from 537 readers interpreting 239 normal mammograms were used to categorise cases as 120 difficult-to-interpret and 119 easy-to-interpret based on cases having the highest and lowest difficulty scores, respectively. Using lattice- and squared-based approaches, 34 handcrafted GRFs per image were extracted and normalised. Three classifiers were constructed: (i) CC and (ii) MLO using the GRFs from corresponding craniocaudal and mediolateral oblique images only, based on the random forest technique for distinguishing difficult- from easy-to-interpret NCs, and (iii) CC + MLO using the median predictive scores from both CC and MLO models. Useful GRFs for the CC and MLO models were recognised using a scree test. The CC and MLO models were trained and validated using the leave-one-out-cross-validation. The models' performances were assessed by the AUC and compared using the DeLong test. A Kruskal-Wallis test was used to examine if the 34 GRFs differed between difficult- and easy-to-interpret NCs and if difficulty level based on the traditional breast density (BD) categories differed among 115 low-BD and 124 high-BD NCs. The CC + MLO model achieved higher performance (0.71 AUC) than the individual CC and MLO model alone (0.66 each), but statistically non-significant difference was found (all p > 0.05). Six GRFs were identified to be valuable in describing difficult-to-interpret NCs. Twenty features, when compared between difficult- and easy-to-interpret NCs, differed significantly (p < 0.05). No statistically significant difference was observed in difficulty between low- and high-BD NCs (p = 0.709). GRF mammographic analysis can predict difficult-to-interpret NCs.

Screening outcome for interpretation by the first and second reader in a population-based mammographic screening program with independent double reading

BACKGROUND

Double reading of screening mammograms is associated with a higher rate of screen-detected cancer than single reading, but different strategies exist regarding reader pairing and blinding. Knowledge about these aspects is important when considering strategies for future use of artificial intelligence in mammographic screening.

PURPOSE

To investigate screening outcome, histopathological tumor characteristics, and mammographic features stratified by the first and the second reader in a population based screening program for breast cancer.

MATERIAL AND METHODS

The study sample consisted of data from 3,499,048 screening examinations from 834,691 women performed during 1996-2018 in BreastScreen Norway. All examinations were interpreted independently by two radiologists, 272 in total. We analyzed interpretation score, recall, and cancer detection, as well as histopathological tumor characteristics and mammographic features of the cancers, stratified by the first and second readers.

RESULTS

For Reader 1, the rate of positive interpretations was 4.8%, recall 2.3%, and cancer detection 0.5%. The corresponding percentages for Reader 2 were 4.9%, 2.5%, and 0.5% (P < 0.05 compared with Reader 1). No statistical difference was observed for histopathological tumor characteristics or mammographic features when stratified by Readers 1 and 2. Recall and cancer detection were statistically higher and histopathological tumor characteristics less favorable for cases detected after concordant positive compared with discordant interpretations.

CONCLUSION

Despite reaching statistical significance, mainly due to the large study sample, we consider the differences in interpretation scores, recall, and cancer detection between the first and second readers to be clinically negligible. For practical and clinical purposes, double reading in BreastScreen Norway is independent.

Test-set training improves the detection rates of invasive cancer in screening mammography

AIM

To investigate if mammographic test-set participation affects routine breast cancer screening performance.

MATERIALS AND METHODS

Clinical audit data between 2008 and 2018 were collected for 35 breast screen readers who participated in the BreastScreen Reader Assessment Strategy (BREAST) and 22 readers with no history of test-set participation. For BREAST readers, the annual audit data were divided according to the year they completed their first test set, and the same years were used randomly to align and divide the data of non-BREAST readers into pre- and post-test set periods. Multiple audit parameters were inspected retrospectively for the two cohorts to identify how their reading performance has evolved in screening mammography.

RESULTS

Investigating 2 calendar years before and after test-set participation, BREAST and non-BREAST readers recalled lower rates of women in the latter period (p=0.03 and p=0.02, respectively). They also improved their positive predictive value (PPV; p=0.01 and p=0.02, respectively). BREAST readers additionally improved their detection rates of invasive cancer (p=0.02) and all cancers (p=0.01). In an extended 3-year comparison, similar improvements occurred in the recall rate for BREAST (p=0.02) and non-BREAST readers (p=0.02) and in PPV (p=0.001, 0.01, respectively); however, improvements in detection rates also occurred exclusively in BREAST readers' performance for invasive cancer (p=0.04), DCIS (p=0.05), and all cancers (p=0.02); however, significant improvements in detection did not involve <15 mm invasive cancers in both periods. Meanwhile, non-BREAST readers demonstrated a decrease in sensitivity (p=0.02).

CONCLUSION

Participation in test sets is linked to over-time improvements in most audit-measured cancer detection rates.

Colorectal Cancer: Performance and Evaluation for CT Colonography Screening- A Multicenter Cluster-randomized Controlled Trial

Background Most radiologists reporting CT colonography (CTC) do not undergo compulsory performance accreditation, potentially lowering diagnostic sensitivity. Purpose To determine whether 1-day individualized training in CTC reporting improves diagnostic sensitivity of experienced radiologists for 6-mm or larger lesions, the durability of any improvement, and any associated factors. Materials and Methods This prospective, multicenter cluster-randomized controlled trial was performed in National Health Service hospitals in England and Wales between April 2017 and January 2020. CTC services were cluster randomized into intervention (1-day training plus feedback) or control (no training or feedback) arms. Radiologists in the intervention arm attended a 1-day workshop focusing on CTC reporting pitfalls with individualized feedback. Radiologists in the control group received no training. Sensitivity for 6-mm or larger lesions was tested at baseline and 1, 6, and 12 months thereafter via interpretation of 10 CTC scans at each time point. The primary outcome was the mean difference in per-lesion sensitivity between arms at 1 month, analyzed using multilevel regression after adjustment for baseline sensitivity. Secondary outcomes included per-lesion sensitivity at 6- and 12-month follow-up, sensitivity for flat neoplasia, and effect of prior CTC experience. Results A total of 69 hospitals were randomly assigned to the intervention (31 clusters, 80 radiologists) or control (38 clusters, 59 radiologists) arm. Radiologists were experienced (median, 500-999 CTC scans interpreted) and reported CTC scans routinely (median, 151-200 scans per year). One-month sensitivity improved after intervention (66.4% [659 of 992]) compared with sensitivity in the control group (42.4% [278 of 655]; difference = 20.8%; 95% CI: 14.6, 27.0; P < .001). Improvements were maintained at 6 (66.4% [572 of 861] vs 50.5% [283 of 560]; difference = 13.0%; 95% CI: 7.4, 18.5; P < .001) and 12 (63.7% [310 of 487] vs 44.4% [187 of 421]; difference = 16.7%; 95% CI: 10.3, 23.1; P < .001) months. This beneficial effect applied to flat lesions (difference = 22.7%; 95% CI: 15.5, 29.9; P < .001) and was independent of career experience (≥1500 CTC scans: odds ratio = 1.09; 95% CI: 0.88, 1.36; P = .22). Conclusion For radiologists evaluating CT colonography studies, a 1-day training intervention yielded sustained improvement in detection of clinically relevant colorectal neoplasia, independent of previous career experience. Clinical trial registration no. NCT02892721 © RSNA, 2022 Online supplemental material is available for this article. See also the editorial by Pickhardt in this issue.