When radiologists perform best: the learning curve in screening mammogram interpretation

Changes in prostate cancer detection rate of MRI-TRUS fusion vs systematic biopsy over time: evidence of a learning curve

BACKGROUND

To determine the effect of urologist and radiologist learning curves and changes in MRI-TRUS fusion platform during 9 years of NCI's experience with multiparametric magnetic resonance imaging (mpMRI)/TRUS fusion biopsy.

METHODS

A prospectively maintained database of patients undergoing mpMRI followed by fusion biopsy (Fbx) and systematic biopsy (Sbx) from 2007 to 2016 was reviewed. The patients were stratified based on the timing of first biopsy. Cohort 1 (7/2007-12/2010) accounted for learning curve. Cohort 2 (1/2011-5/2013) and cohort 3 (5/2013-4/2016) included patients biopsied prior to and after debut of a new software platform, respectively. Clinically significant (CS) disease was defined as Gleason 7 (3+4) or higher. McNemar's test compared cancer detection rates (CDRs) of Sbx and Fbx between time periods.

RESULTS

1528 patients were included in the study with 230, 537 and 761 patients included in three respective cohorts. Median age (interquartile range) was 61.0 (±9.0), 62.0 (±7.3), and 64.0 (±11.0) years in three cohorts, respectively (P<0.001). Fbx and Sbx had comparable CS CDR in cohort 1 (24.8 vs 22.2%, P=0.377). Fbx detected significantly more CS disease compared to Sbx in the following two periods (cohort 2: 31.5 vs 25.0%, P=0.001; cohort 3: 36.4 vs 30.3%, P<0.001) and detected significantly less low risk disease in the same period (cohort 2: 14.5 vs 19.6%, P<0.001; cohort 3: 12.6 vs 16.7%, P<0.001). Even after multivariate adjustment with age, PSA, race, clinical stage and MRI suspicion score, Fbx CS cancer detection increased in successive cohorts (cohort 2: OR 2.23, P=0.043; cohort 3: OR 2.92, P=0.007).

CONCLUSIONS

In the past 9 years, there has been significant improvement in the accuracy of Fbx. Our results show that after an early learning period, Fbx detected higher rates of CS cancer and lower rates of clinically insignificant cancer than Sbx. Software advances allowed for even greater detection of CS disease.

Lack of agreement between radiologists: implications for image-based model observers

We tested the agreement of radiologists' rankings of different reconstructions of breast computed tomography images based on their diagnostic (classification) performance and on their subjective image quality assessments. We used 102 pathology proven cases (62 malignant, 40 benign), and an iterative image reconstruction (IIR) algorithm to obtain 24 reconstructions per case with different image appearances. Using image feature analysis, we selected 3 IIRs and 1 clinical reconstruction and 50 lesions. The reconstructions produced a range of image quality from smooth/low-noise to sharp/high-noise, which had a range in classifier performance corresponding to AUCs of 0.62 to 0.96. Six experienced Mammography Quality Standards Act (MQSA) radiologists rated the likelihood of malignancy for each lesion. We conducted an additional reader study with the same radiologists and a subset of 30 lesions. Radiologists ranked each reconstruction according to their preference. There was disagreement among the six radiologists on which reconstruction produced images with the highest diagnostic content, but they preferred the midsharp/noise image appearance over the others. However, the reconstruction they preferred most did not match with their performance. Due to these disagreements, it may be difficult to develop a single image-based model observer that is representative of a population of radiologists for this particular imaging task.

Learning curve estimation in medical devices and procedures: hierarchical modeling

In the use of medical device procedures, learning effects have been shown to be a critical component of medical device safety surveillance. To support their estimation of these effects, we evaluated multiple methods for modeling these rates within a complex simulated dataset representing patients treated by physicians clustered within institutions. We employed unique modeling for the learning curves to incorporate the learning hierarchy between institution and physicians and then modeled them within established methods that work with hierarchical data such as generalized estimating equations (GEE) and generalized linear mixed effect models. We found that both methods performed well, but that the GEE may have some advantages over the generalized linear mixed effect models for ease of modeling and a substantially lower rate of model convergence failures. We then focused more on using GEE and performed a separate simulation to vary the shape of the learning curve as well as employed various smoothing methods to the plots. We concluded that while both hierarchical methods can be used with our mathematical modeling of the learning curve, the GEE tended to perform better across multiple simulated scenarios in order to accurately model the learning effect as a function of physician and hospital hierarchical data in the use of a novel medical device. We found that the choice of shape used to produce the 'learning-free' dataset would be dataset specific, while the choice of smoothing method was negligibly different from one another. This was an important application to understand how best to fit this unique learning curve function for hierarchical physician and hospital data. Copyright © 2017 John Wiley & Sons, Ltd.

Reducing Costs of Breast Examination: Ultrasound Performance and Inter-Observer Variability of Expert Radiologists Versus Residents

AIM

To compare efficiency levels between radiologist and radiology resident and any significant or clinically relevant differences in breast ultrasound diagnosis, thus reducing extra costs.

MATERIAL AND METHODS

100 patients attending for breast ultrasound were included. Each patient was examined by a radiologist, and subsequently by a resident of the radiology department. Both operators noted their findings and wrote a concluding report. Reports were compared for histological and biological analysis.

RESULTS

100 female patients with a mean age about 49 years were examined. The proportions of correct diagnoses of lesions individuated by radiologist and resident were 26.90 > 13.71% (p-value = 10.7), i.e. the radiologist was more accurate in comparison to resident in the individuation of breast lesions.

CONCLUSIONS

The radiologist was more accurate in comparison to radiology resident in the evaluation of breast pathology in ultrasonography diagnoses, and this could reduce cost and/or in-depth analysis.

Analysis of mammographic diagnostic errors in breast clinic

Mammography is the gold standard for detection of early breast cancer and it is still the only diagnostic tool which shows reduction of the mortality from that. Despite that, there is a high chance of false negatives that can lead to diagnostic errors resulting in delays of treatment and worsening of prognosis. The aim of this study is to analyze the rate of false negative in mammography and assess the source of diagnostic errors. Two radiologists have retrospectively evaluated 500 mammograms performed between January 2008 and December 2011 in Breast Imaging Clinic. 250 patients (Group A) had been operated for breast cancer and 250 patients (Group B) were healthy woman submitted to mammography according to the guideline for early detection of breast cancer. In Group A, 138 patients (55.2 %) were true missed cancer, 61 had minimal sign (24.4 %) and 53 were false negative (FN) (20.4 %). The source of errors amongst the FN were in 42 % of cases due to perception, in 15 % to interpretation, in 10 % to subtle/unusual lesion characteristics, in 9 % error for satisfaction of search, in 7 % to inherent limitations of mammography, in 4 % to poor technique and 13 % for inadequate clinical management. The diagnostic errors in breast clinic services are not negligible. The largest number of FN results from perception errors, misinterpretation and inadequate clinical management. These can be related to factors such as inattention, fatigue or lack of experience. To reduce it, it is necessary to have a dedicated multidisciplinary staff and adequate equipment and workloads.

Identifying and processing the gap between perceived and actual agreement in breast pathology interpretation

We examined how pathologists' process their perceptions of how their interpretations on diagnoses for breast pathology cases agree with a reference standard. To accomplish this, we created an individualized self-directed continuing medical education program that showed pathologists interpreting breast specimens how their interpretations on a test set compared with a reference diagnosis developed by a consensus panel of experienced breast pathologists. After interpreting a test set of 60 cases, 92 participating pathologists were asked to estimate how their interpretations compared with the standard for benign without atypia, atypia, ductal carcinoma in situ and invasive cancer. We then asked pathologists their thoughts about learning about differences in their perceptions compared with actual agreement. Overall, participants tended to overestimate their agreement with the reference standard, with a mean difference of 5.5% (75.9% actual agreement; 81.4% estimated agreement), especially for atypia and were least likely to overestimate it for invasive breast cancer. Non-academic affiliated pathologists were more likely to more closely estimate their performance relative to academic affiliated pathologists (77.6 vs 48%; P=0.001), whereas participants affiliated with an academic medical center were more likely to underestimate agreement with their diagnoses compared with non-academic affiliated pathologists (40 vs 6%). Before the continuing medical education program, nearly 55% (54.9%) of participants could not estimate whether they would overinterpret the cases or underinterpret them relative to the reference diagnosis. Nearly 80% (79.8%) reported learning new information from this individualized web-based continuing medical education program, and 23.9% of pathologists identified strategies they would change their practice to improve. In conclusion, when evaluating breast pathology specimens, pathologists do a good job of estimating their diagnostic agreement with a reference standard, but for atypia cases, pathologists tend to overestimate diagnostic agreement. Many participants were able to identify ways to improve.

Imaging-based screening: maximizing benefits and minimizing harms

Advanced imaging technologies play a central role in screening asymptomatic patients. However, the balance between imaging-based screening's potential benefits versus risks is sometimes unclear. Radiologists will have to address ongoing concerns, including high false-positive rates, incidental findings outside the organ of interest, overdiagnosis, and potential risks from radiation exposure. In this article, we provide a brief overview of these recurring controversies and suggest the following as areas that radiologists should focus on in order to tip the balance toward more benefits and less harms for patients undergoing imaging-based screening: interpretive variability, abnormal finding thresholds, and personalized, risk-based screening.

The National Mammography Database: Preliminary Data

OBJECTIVE

The purposes of our study were to analyze screening mammography data submitted to the National Mammography Database (NMD) since its inception to confirm data collection feasibility, to draw parallels to data from the Breast Cancer Surveillance Consortium (BCSC), and to examine trends over time. We also retrospectively evaluated practice-level variation in terms of practice type, practice setting, census region, and annual volume.

MATERIALS AND METHODS

Data from 90 mammography facilities in the NMD registry were analyzed. The registry receives mammography data collected as part of standard clinical practice, including self-reported demographic information, clinical findings, screening mammography interpretation, and biopsy results. Outcome metrics calculated were cancer detection rate, recall rate, and positive predictive values for biopsy recommended (PPV2) and biopsy performed (PPV3).

RESULTS

The NMD successfully collected and analyzed data for 3,181,437 screening mammograms performed between January 2008 and December 2012. Mean values for outcomes were cancer detection rate of 3.43 per 1000 (95% CI, 3.2-3.7), recall rate of 10% (95% CI, 9.3-10.7%), PPV2 of 18.5% (95% CI, 16.7-20.2%), and PPV3 of 29.2% (95% CI, 26.2-32.3%). No statistically significant difference was seen in performance measurements on the basis of practice type, practice setting, census region, or annual volume. NMD performance measurements parallel those reported by the BCSC.

CONCLUSION

The NMD has become the fastest growing mammography registry in the United States, providing nationwide performance metrics and permitting comparison with published benchmarks. Our study shows the feasibility of using the NMD to audit mammography facilities and to provide current, ongoing benchmark data.

Initial Performance of Radiologists and Radiology Residents in Interpreting Low-Dose (2-mSv) Appendiceal CT

OBJECTIVE

The objective of our study was to prospectively evaluate the initial diagnostic performance and learning curve of a community of radiologists and residents in interpreting 2-mSv appendiceal CT.

SUBJECTS AND METHODS

We included 46 attending radiologists and 153 radiology residents from 22 hospitals who completed an online training course of 30 2-mSv CT cases. Appendicitis was confirmed in 14 cases. Most of the readers had limited (≤ 10 cases, n = 32) or no (n = 118) prior experience with low-dose appendiceal CT. The order of cases was randomized for each reader. A multireader multicase ROC analysis was performed. Generalized estimating equations were used to model the learning curves in diagnostic performance.

RESULTS

Diagnostic performance gradually improved with years of training. The average AUC was 0.94 (95% CI, 0.90-0.98), 0.92 (0.88-0.96), 0.90 (0.85-0.96), and 0.86 (0.80-0.92) for the attending radiologists, senior residents, 2nd-year residents, and 1st-year residents, respectively. We did not observe any notable intrareader learning curves over the training course of the 30 cases except a decrease in reading time. Diagnostic accuracy and sensitivity were significantly affected by the reader training level and prior overall experience with appendiceal CT but not by the prior specific experience with low-dose appendiceal CT.

CONCLUSION

The learning curve is likely prolonged and forms gradually over years by overall radiology training and clinical experience in general rather than by experience with low-dose appendiceal CT specifically.

Acquisition and maintenance of medical expertise: a perspective from the expert-performance approach with deliberate practice

As a part of a special collection in this issue of Academic Medicine, which is focused on mastery learning in medical education, this Perspective describes how the expert-performance approach with deliberate practice is consistent with many characteristics of mastery learning. Importantly, this Perspective also explains how the expert-performance approach provides a very different perspective on the acquisition of skill. Whereas traditional education with mastery learning focuses on having students attain an adequate level of performance that is based on goals set by the existing curricula, the expert-performance approach takes an empirical approach and first identifies the final goal of training-namely, reproducibly superior objective performance (superior patient outcomes) for individuals in particular medical specialties. Analyzing this superior complex performance reveals three types of mental representations that permit expert performers to plan, execute, and monitor their own performance. By reviewing research on medical performance and education, the author describes evidence for these representations and their development within the expert-performance framework. He uses the research to generate suggestions for improved training of medical students and professionals. Two strategies-designing learning environments with libraries of cases and creating opportunities for individualized teacher-guided training-should enable motivated individuals to acquire a full set of refined mental representations. Providing the right resources to support the expert-performance approach will allow such individuals to become self-regulated learners-that is, members of the medical community who have the tools to improve their own and their team members' performances throughout their entire professional careers.

Racial and Ethnic Disparity in Symptomatic Breast Cancer Awareness despite a Recent Screen: The Role of Tumor Biology and Mammography Facility Characteristics

BACKGROUND

In a racially and ethnically diverse sample of recently diagnosed urban patients with breast cancer, we examined associations of patient, tumor biology, and mammography facility characteristics on the probability of symptomatic discovery of their breast cancer despite a recent prior screening mammogram.

METHODS

In the Breast Cancer Care in Chicago study, self-reports at interview were used to define patients as having a screen-detected breast cancer or having symptomatic awareness despite a recent screening mammogram (SADRS), in the past 1 or 2 years. Patients with symptomatic breast cancer who did not report a recent prior screen were excluded from these analyses. Characteristics associated with more aggressive disease [estrogen receptor (ER)- and progesterone receptor (PR)-negative status and higher tumor grade] were abstracted from medical records. Mammogram facility characteristics that might indicate aspects of screening quality were defined and controlled for in some analyses.

RESULTS

SADRS was more common among non-Hispanic black and Hispanic than among non-Hispanic white patients (36% and 42% vs. 25%, respectively, P = 0.0004). SADRS was associated with ER/PR-negative and higher-grade disease. Patients screened at sites that relied on dedicated radiologists and sites that were breast imaging centers of excellence were less likely to report SADRS. Tumor and facility factors together accounted for two thirds of the disparity in SADRS (proportion mediated = 70%, P = 0.02).

CONCLUSION

Facility resources and tumor aggressiveness explain much of the racial/ethnic disparity in symptomatic breast cancer among recently screened patients.

IMPACT

A more equitable distribution of high-quality screening would ameliorate but not eliminate this disparity.

Criteria for identifying radiologists with acceptable screening mammography interpretive performance on basis of multiple performance measures

OBJECTIVE

Using a combination of performance measures, we updated previously proposed criteria for identifying physicians whose performance interpreting screening mammography may indicate suboptimal interpretation skills.

MATERIALS AND METHODS

In this study, six expert breast imagers used a method based on the Angoff approach to update criteria for acceptable mammography performance on the basis of two sets of combined performance measures: set 1, sensitivity and specificity for facilities with complete capture of false-negative cancers; and set 2, cancer detection rate (CDR), recall rate, and positive predictive value of a recall (PPV1) for facilities that cannot capture false-negative cancers but have reliable cancer follow-up information for positive mammography results. Decisions were informed by normative data from the Breast Cancer Surveillance Consortium (BCSC).

RESULTS

Updated combined ranges for acceptable sensitivity and specificity of screening mammography are sensitivity≥80% and specificity≥85% or sensitivity 75-79% and specificity 88-97%. Updated ranges for CDR, recall rate, and PPV1 are: CDR≥6 per 1000, recall rate 3-20%, and any PPV1; CDR 4-6 per 1000, recall rate 3-15%, and PPV1≥3%; or CDR 2.5-4.0 per 1000, recall rate 5-12%, and PPV1 3-8%. Using the original criteria, 51% of BCSC radiologists had acceptable sensitivity and specificity; 40% had acceptable CDR, recall rate, and PPV1. Using the combined criteria, 69% had acceptable sensitivity and specificity and 62% had acceptable CDR, recall rate, and PPV1.

CONCLUSION

The combined criteria improve previous criteria by considering the interrelationships of multiple performance measures and broaden the acceptable performance ranges compared with previous criteria based on individual measures.

Radiographers supporting radiologists in the interpretation of screening mammography: a viable strategy to meet the shortage in the number of radiologists

BACKGROUND

An alternative approach to the traditional model of radiologists interpreting screening mammography is necessary due to the shortage of radiologists to interpret screening mammograms in many countries.

METHODS

We evaluated the performance of 15 Mexican radiographers, also known as radiologic technologists, in the interpretation of screening mammography after a 6 months training period in a screening setting. Fifteen radiographers received 6 months standardized training with radiologists in the interpretation of screening mammography using the Breast Imaging Reporting and Data System (BI-RADS) system. A challenging test set of 110 cases developed by the Breast Cancer Surveillance Consortium was used to evaluate their performance. We estimated sensitivity, specificity, false positive rates, likelihood ratio of a positive test (LR+) and the area under the subject-specific Receiver Operating Characteristic (ROC) curve (AUC) for diagnostic accuracy. A mathematical model simulating the consequences in costs and performance of two hypothetical scenarios compared to the status quo in which a radiologist reads all screening mammograms was also performed.

RESULTS

Radiographer's sensitivity was comparable to the sensitivity scores achieved by U.S. radiologists who took the test but their false-positive rate was higher. Median sensitivity was 73.3 % (Interquartile range, IQR: 46.7-86.7 %) and the median false positive rate was 49.5 % (IQR: 34.7-57.9 %). The median LR+ was 1.4 (IQR: 1.3-1.7 %) and the median AUC was 0.6 (IQR: 0.6-0.7). A scenario in which a radiographer reads all mammograms first, and a radiologist reads only those that were difficult for the radiographer, was more cost-effective than a scenario in which either the radiographer or radiologist reads all mammograms.

CONCLUSIONS

Given the comparable sensitivity achieved by Mexican radiographers and U.S. radiologists on a test set, screening mammography interpretation by radiographers appears to be a possible adjunct to radiologists in countries with shortages of radiologists. Further studies are required to assess the effectiveness of different training programs in order to obtain acceptable screening accuracy, as well as the best approaches for the use of non-physician readers to interpret screening mammography.

The influence of mammographic technologists on radiologists' ability to interpret screening mammograms in community practice

RATIONALE AND OBJECTIVES

To determine whether the mammographic technologist has an effect on the radiologists' interpretative performance of screening mammography in community practice.

MATERIALS AND METHODS

In this institutional review board-approved retrospective cohort study, we included Carolina Mammography Registry data from 372 radiologists and 356 mammographic technologists from 1994 to 2009 who performed 1,003,276 screening mammograms. Measures of interpretative performance (recall rate, sensitivity, specificity, positive predictive value [PPV1], and cancer detection rate [CDR]) were ascertained prospectively with cancer outcomes collected from the state cancer registry and pathology reports. To determine if the mammographic technologist influenced the radiologists' performance, we used mixed effects logistic regression models, including a radiologist-specific random effect and taking into account the clustering of examinations across women, separately for screen-film mammography (SFM) and full-field digital mammography (FFDM).

RESULTS

Of the 356 mammographic technologists included, 343 performed 889,347 SFM examinations, 51 performed 113,929 FFDM examinations, and 38 performed both SFM and FFDM examinations. A total of 4328 cancers were reported for SFM and 564 cancers for FFDM. The technologists had a statistically significant effect on the radiologists' recall rate, sensitivity, specificity, and CDR for both SFM and FFDM (P values <.01). For PPV1, variability by technologist was observed for SFM (P value <.0001) but not for FFDM (P value = .088).

CONCLUSIONS

The interpretative performance of radiologists in screening mammography varies substantially by the technologist performing the examination. Additional studies should aim to identify technologist characteristics that may explain this variation.

Educational interventions to improve screening mammography interpretation: a randomized controlled trial

OBJECTIVE

The objective of our study was to conduct a randomized controlled trial of educational interventions that were created to improve performance of screening mammography interpretation.

MATERIALS AND METHODS

We randomly assigned physicians who interpret mammography to one of three groups: self-paced DVD, live expert-led educational seminar, or control. The DVD and seminar interventions used mammography cases of varying difficulty and provided associated teaching points. Interpretive performance was compared using a pretest-posttest design. Sensitivity, specificity, and positive predictive value (PPV) were calculated relative to two outcomes: cancer status and consensus of three experts about recall. The performance measures for each group were compared using logistic regression adjusting for pretest performance.

RESULTS

One hundred two radiologists completed all aspects of the trial. After adjustment for preintervention performance, the odds of improved sensitivity for correctly identifying a lesion relative to expert recall were 1.34 times higher for DVD participants than for control subjects (95% CI, 1.00-1.81; p = 0.050). The odds of an improved PPV for correctly identifying a lesion relative to both expert recall (odds ratio [OR] = 1.94; 95% CI, 1.24-3.05; p = 0.004) and cancer status (OR = 1.81; 95% CI, 1.01-3.23; p = 0.045) were significantly improved for DVD participants compared with control subjects, with no significant change in specificity. For the seminar group, specificity was significantly lower than the control group (OR relative to expert recall = 0.80; 95% CI, 0.64-1.00; p = 0.048; OR relative to cancer status = 0.79; 95% CI, 0.65-0.95; p = 0.015).

CONCLUSION

In this randomized controlled trial, the DVD educational intervention resulted in a significant improvement in screening mammography interpretive performance on a test set, which could translate into improved interpretative performance in clinical practice.

Radiology resident mammography training: interpretation difficulty and error-making patterns

RATIONALE AND OBJECTIVES

The purpose of this study was to better understand the concept of mammography difficulty and how it affects radiology resident performance.

MATERIALS AND METHODS

Seven radiology residents and three expert breast imagers reviewed 100 mammograms, consisting of bilateral medial lateral oblique and craniocaudal views, using a research workstation. The cases consisted of normal, benign, and malignant findings. Participants identified abnormalities and scored the difficulty and malignant potential for each case. Resident performance (sensitivity, specificity, and area under the receiver operating characteristic curve [AUC]) was calculated for self- and expert-assessed high and low difficulties.

RESULTS

For cases classified by self-assessed difficulty, the resident AUCs were 0.667 for high difficulty and 0.771 for low difficulty cases (P = .010). Resident sensitivities were 0.707 for high and 0.614 for low difficulty cases (P = .113). Resident specificities were 0.583 for high and 0.905 for low difficulty cases (P < .001). For cases classified by expert-assessed difficulty, the resident AUCs were 0.583 for high and 0.783 for low difficulty cases (P = .001). Resident sensitivities were 0.558 for high and 0.796 for low difficulty cases (P < .001). Resident specificities were 0.714 for high and 0.740 for low difficulty cases (P = .807).

CONCLUSIONS

Increased self- and expert-assessed difficulty is associated with a decrease in resident performance in mammography. However, while this lower performance is due to a decrease in specificity for self-assessed difficulty, it is due to a decrease in sensitivity for expert-assessed difficulty. These trends suggest that educators should provide a mix of self- and expert-assessed difficult cases in educational materials to maximize the effect of training on resident performance and confidence.

Belief elicitation to populate health economic models of medical diagnostic devices in development

BACKGROUND AND OBJECTIVE

Bayesian methods can be used to elicit experts' beliefs about the clinical value of healthcare technologies. This study investigates a belief-elicitation method for estimating diagnostic performance in an early stage of development of photoacoustic mammography (PAM) imaging versus magnetic resonance imaging (MRI) for detecting breast cancer.

RESEARCH DESIGN

Eighteen experienced radiologists ranked tumor characteristics regarding their importance to detect malignancies. With reference to MRI, radiologists estimated the true positives and negatives of PAM using the variable interval method. An overall probability density function was determined using linear opinion pooling, weighted for individual experts' experience.

RESULT

The most important tumor characteristics are mass margins and mass shape. Respondents considered MRI the better technology to visualize these characteristics. Belief elicitation confirmed this by providing an overall sensitivity of PAM ranging from 58.9 to 85.1% (mode 75.6%) and specificity ranging from 52.2 to 77.6% (mode 66.5%).

CONCLUSION

Belief elicitation allowed estimates to be obtained for the expected diagnostic performance of PAM, although radiologists expressed difficulties in doing so. Heterogeneity within and between experts reflects this uncertainty and the infancy of PAM. Further clinical trials are required to validate the extent to which this belief-elicitation method is predictive for observed test performance.

Trends in surgery for screen-detected and interval breast cancers in a national screening programme

Background

This population-based study aimed to evaluate trends in surgical approach for screen-detected cancer versus interval breast cancer, and to determine the factors associated with positive resection margins.

Methods

Screening mammograms of women aged 50–75 years, who underwent biennial screening in a Dutch breast-screening region between 1997 and 2011, were included. Patient and tumour characteristics were compared between women who underwent mastectomy or breast-conserving surgery (BCS) for screen-detected or interval cancer, and women with a negative or positive resection margin after BCS.

Results

Some 417 013 consecutive screening mammograms were included. A total of 2224 screen-detected and 825 interval cancers were diagnosed. The BCS rate remained stable (mean 6·1 per 1000 screened women; P = 0·099), whereas mastectomy rates increased significantly during the study from 0·9 (1997–1998) to 1·9 (2009–2010) per 1000 screened women (P &lt; 0·001). The proportion of positive resection margins for invasive cancer was 19·6 and 7·6 per cent in 1997–1998 and 2009–2010 respectively (P &lt; 0·001), with significant variation between hospitals. Dense breasts, preoperative magnetic resonance imaging, microcalcifications, architectural distortion, tumour size over 20 mm, axillary lymph node metastasis and treating hospital were independent risk factors for mastectomy. Interval cancer, image-guided tumour localization, microcalcifications, breast parenchyma asymmetry, tumour size greater than 20 mm, lobular tumour histology, low tumour grade, extensive invasive component and treating hospital were independent risk factors for positive resection margins.

Conclusion

Mastectomy rates doubled during a 14-year period of screening mammography and the proportion of positive resection margins decreased, with variation among hospitals. The latter observation stresses the importance of quality control programmes for hospitals treating women with breast cancer.

Variations in screening outcome among pairs of screening radiologists at non-blinded double reading of screening mammograms: a population-based study

OBJECTIVES

Substantial inter-observer variability in screening mammography interpretation has been reported at single reading. However, screening results of pairs of screening radiologists have not yet been published. We determined variations in screening performances among pairs of screening radiologists at non-blinded double reading.

METHODS

We included pairs of screening radiologists with at least 7,500 screening examinations per pair, obtained between 1997 and 2011. During 2-year follow-up, breast imaging reports, surgical reports and pathology results were collected of all referred women and interval cancers. Referral rate, cancer detection rate, positive predictive value and sensitivity were calculated for each pair.

RESULTS

A total of 310,906 screening mammograms, read by 26 pairs of screening radiologists, were included for analysis. The referral rate ranged from 1.0 % (95 % CI 0.8 %-1.2 %) to 1.5 % (95 % CI 1.3 %-1.8 %), the cancer detection rate from 4.0 (95 % CI 2.8-5.2) to 6.3 (95 % CI 4.5-8.0) per 1,000 screens. The programme sensitivity and positive predictive value of referral ranged from 55.1 % (95 % CI 45.1 %-65.1 %) to 81.5 % (95 % CI 73.4 %-89.6 %) and from 28.7 % (95 % CI 20.8 %-36.6 %) to 49.5 % (95 % CI 39.7 %-59.3 %), respectively.

CONCLUSION

We found significant variations in screening outcomes among pairs of screening radiologists at non-blinded double reading. This stresses the importance of monitoring screening results on a local scale.

KEY POINTS

• Substantial inter-observer variability in screening mammography interpretation is known at single reading • Population-based study showed significant variations in outcomes among pairs of screening radiologists • Local monitoring and regular feedback are important to optimise screening outcome.

Eyes-on training and radiological expertise: an examination of expertise development and its effects on visual working memory

OBJECTIVE

Our aim was to examine the specificity of the effects of acquiring expertise on visual working memory (VWM) and the degree to which higher levels of experience within the domain of expertise are associated with more efficient use of VWM.

BACKGROUND

Previous research is inconsistent on whether expertise effects are specific to the area of expertise or generalize to other tasks that also involve the same cognitive processes. It is also unclear whether more training and/or experience will lead to continued improvement on domain-relevant tasks or whether a plateau could be reached.

METHOD

In Experiment I, veterinary medicine students completed a one-shot visual change detection task. In Experiment 2, veterinarians completed a flicker change detection task. Both experiments involved stimuli specific to the domain of radiology and general stimuli.

RESULTS

In Experiment I, veterinary medicine students who had completed an "eyes-on" radiological training demonstrated a domain-specific effect in which performance was better on the domain-specific stimuli than on the domain-general stimuli. In Experiment 2, veterinarians again showed a domain-specific effect, but performance was unrelated to the amount of experience veterinarians had accumulated.

CONCLUSION

The effect of experience is domain specific and occurs during the first few years of training, after which a plateau is reached.

APPLICATION

VWM training in one domain may not lead to improved performance on other VWM tasks. In acquiring expertise, eyes-on training is important initially, but continued experience may not be associated with further improvements in the efficiency of VWM.

Short-term outcomes of screening mammography using computer-aided detection: a population-based study of medicare enrollees

BACKGROUND

Computer-aided detection (CAD) has rapidly diffused into screening mammography practice despite limited and conflicting data on its clinical effect.

OBJECTIVE

To determine associations between CAD use during screening mammography and the incidence of ductal carcinoma in situ (DCIS) and invasive breast cancer, invasive cancer stage, and diagnostic testing.

DESIGN

Retrospective cohort study.

SETTING

Medicare program.

PARTICIPANTS

Women aged 67 to 89 years having screening mammography between 2001 and 2006 in U.S. SEER (Surveillance, Epidemiology and End Results) regions (409 459 mammograms from 163 099 women).

MEASUREMENTS

Incident DCIS and invasive breast cancer within 1 year after mammography, invasive cancer stage, and diagnostic testing within 90 days after screening among women without breast cancer.

RESULTS

From 2001 to 2006, CAD prevalence increased from 3.6% to 60.5%. Use of CAD was associated with greater DCIS incidence (adjusted odds ratio [OR], 1.17 [95% CI, 1.11 to 1.23]) but no difference in invasive breast cancer incidence (adjusted OR, 1.00 [CI, 0.97 to 1.03]). Among women with invasive cancer, CAD was associated with greater likelihood of stage I to II versus III to IV cancer (adjusted OR, 1.27 [CI, 1.14 to 1.41]). In women without breast cancer, CAD was associated with increased odds of diagnostic mammography (adjusted OR, 1.28 [CI, 1.27 to 1.29]), breast ultrasonography (adjusted OR, 1.07 [CI, 1.06 to 1.09]), and breast biopsy (adjusted OR, 1.10 [CI, 1.08 to 1.12]).

LIMITATION

Short follow-up for cancer stage, potential unmeasured confounding, and uncertain generalizability to younger women.

CONCLUSION

Use of CAD during screening mammography among Medicare enrollees is associated with increased DCIS incidence, the diagnosis of invasive breast cancer at earlier stages, and increased diagnostic testing among women without breast cancer.

PRIMARY FUNDING SOURCE

Center for Healthcare Policy and Research, University of California, Davis.

Potentially missed detection with screening mammography: does the quality of radiologist's interpretation vary by patient socioeconomic advantage/disadvantage?

PURPOSE

We examined whether quality of mammography interpretation as performed by the original reading radiologist varied by patient sociodemographic characteristics.

METHODS

For 149 patients residing in Chicago and diagnosed in 2005-2008, we obtained the original index mammogram that detected the breast cancer and at least one prior mammogram that did not detect the cancer performed within 2 years of the index mammogram. A single breast imaging specialist performed a blinded review of the prior mammogram. Potentially missed detection (PMD) was defined as an actionable lesion seen during a blinded review of the prior mammogram that was in the same quadrant as the cancer on the index mammogram.

RESULTS

Of 149 prior mammograms originally read as nonmalignant, 46% (N = 68) had a potentially detectable lesion. In unadjusted analyses, PMD was greater among minority patients (54% vs. 39%, P = .07) and for patients with incomes below $30,000 (65% vs. 36%, P < .01), less education (58% vs. 39%, P = .02), and lacking private health insurance (63% vs. 40%, P = .02). Likelihood ratio tests for the inclusion of socioeconomic variables in multivariable logistic regression models were highly significant (P ≤ .02).

CONCLUSIONS

Disadvantaged socioeconomic status appears to be associated with PMD of breast cancer at mammography screening.

Effects of digital mammography uptake on downstream breast-related care among older women

BACKGROUND

Digital mammography is the dominant modality for breast cancer screening in the United States. No previous studies have investigated as to how introducing digital mammography affects downstream breast-related care.

OBJECTIVE

Compare breast-related health care use after a screening mammogram before and after introduction of digital mammography.

RESEARCH DESIGN AND SUBJECTS

Longitudinal study of screening mammograms from 14 radiology facilities contributing data to the Breast Cancer Surveillance Consortium performed 1 year before and 4 years after each facility introduced digital mammography, along with linked Medicare claims. We included 30,211 mammograms for women aged 66 years and older without breast cancer.

MEASURES

Rates of false-positive recall and short-interval follow-up were based on radiologists' assessments and recommendations; rates of follow-up mammography, ultrasound, and breast biopsy use were based on Medicare claims.

RESULTS

False-positive recall rates increased after the introduction of digital mammography. Follow-up mammography use was significantly higher across all 4 years after a facility began using digital mammography compared with the year before [year 1 odds ratio (OR) = 1.7, 95% confidence interval (CI), 1.4-2.1]. Among women with false-positive mammography results, use of ultrasound decreased significantly in the second through fourth years after digital mammography began (year 2 OR = 0.4, 95% CI, 0.3-0.6).

CONCLUSIONS

Introduction of a new technology led to changes in health care use that persisted for at least 4 years. Comparative effectiveness research on new technologies should consider not only diagnostic performance but also downstream utilization attributable to this apparent learning curve.

Diagnostic mammography: identifying minimally acceptable interpretive performance criteria

PURPOSE

To develop criteria to identify thresholds for the minimally acceptable performance of physicians interpreting diagnostic mammography studies.

MATERIALS AND METHODS

In an institutional review board-approved HIPAA-compliant study, an Angoff approach was used to set criteria for identifying minimally acceptable interpretive performance for both workup after abnormal screening examinations and workup of a breast lump. Normative data from the Breast Cancer Surveillance Consortium (BCSC) was used to help the expert radiologist identify the impact of cut points. Simulations, also using data from the BCSC, were used to estimate the expected clinical impact from the recommended performance thresholds.

RESULTS

Final cut points for workup of abnormal screening examinations were as follows: sensitivity, less than 80%; specificity, less than 80% or greater than 95%; abnormal interpretation rate, less than 8% or greater than 25%; positive predictive value (PPV) of biopsy recommendation (PPV2), less than 15% or greater than 40%; PPV of biopsy performed (PPV3), less than 20% or greater than 45%; and cancer diagnosis rate, less than 20 per 1000 interpretations. Final cut points for workup of a breast lump were as follows: sensitivity, less than 85%; specificity, less than 83% or greater than 95%; abnormal interpretation rate, less than 10% or greater than 25%; PPV2, less than 25% or greater than 50%; PPV3, less than 30% or greater than 55%; and cancer diagnosis rate, less than 40 per 1000 interpretations. If underperforming physicians moved into the acceptable range after remedial training, the expected result would be (a) diagnosis of an additional 86 cancers per 100,000 women undergoing workup after screening examinations, with a reduction in the number of false-positive examinations by 1067 per 100,000 women undergoing this workup, and (b) diagnosis of an additional 335 cancers per 100,000 women undergoing workup of a breast lump, with a reduction in the number of false-positive examinations by 634 per 100,000 women undergoing this workup.

CONCLUSION

Interpreting physicians who fall outside one or more of the identified cut points should be reviewed in the context of an overall assessment of all their performance measures and their specific practice setting to determine if remedial training is indicated.

Assessing health care use and cost consequences of a new screening modality: the case of digital mammography

BACKGROUND

Full-field digital mammography (FFDM) has largely replaced screen-film mammography (SFM) for breast cancer screening, but how this affects downstream breast-related use and costs is unknown.

OBJECTIVES

To compare breast-related health care use and costs among Medicare beneficiaries undergoing SFM versus FFDM from 1999 to 2005.

DESIGN

Retrospective cohort study.

SUBJECTS

Medicare-enrolled women aged 66 and older with mammograms in Breast Cancer Surveillance Consortium registries.

MEASURES

Subsequent follow-up with additional imaging or breast biopsy within 12 months was ascertained through Medicare claims. Associated mean costs were estimated by screening modality and year, adjusting for confounding factors, and clustering within mammography facilities using Generalized Estimating Equations.

RESULTS

Among 138,199 women, 332,324 SFM and 22,407 FFDM mammograms were analyzed. Approximately 6.5% of SFM and 9.0% of FFDM had positive findings. In 2001, subsequent imaging was higher among FFDM versus SFM (127.5 vs. 97.4 follow-up mammography claims per 1000 index mammograms), whereas subsequent biopsy was lower among FFDM versus SFM (19.2 vs. 24.9 per 1000 index mammograms) with differences decreasing over time. From 2001 to 2004, mammography subsequent to FFDM had higher mean costs than SFM ($82.60 vs. $64.31 in 2001). The only cost differences between SFM and FFDM for ultrasound or biopsy were in 2001.

CONCLUSIONS

Subsequent breast-related health care use differed early in FFDM introduction, but diminished over time with differences attributable to higher recall rates for additional imaging and lower rates of biopsy in those undergoing FFDM versus SFM. Remaining cost differences are because of higher reimbursement rates for FFDM versus SFM.

Are radiologists' goals for mammography accuracy consistent with published recommendations?

RATIONALE AND OBJECTIVES

Mammography quality assurance programs have been in place for more than a decade. We studied radiologists' self-reported performance goals for accuracy in screening mammography and compared them to published recommendations.

MATERIALS AND METHODS

A mailed survey of radiologists at mammography registries in seven states within the Breast Cancer Surveillance Consortium (BCSC) assessed radiologists' performance goals for interpreting screening mammograms. Self-reported goals were compared to published American College of Radiology (ACR) recommended desirable ranges for recall rate, false-positive rate, positive predictive value of biopsy recommendation (PPV2), and cancer detection rate. Radiologists' goals for interpretive accuracy within desirable range were evaluated for associations with their demographic characteristics, clinical experience, and receipt of audit reports.

RESULTS

The survey response rate was 71% (257 of 364 radiologists). The percentage of radiologists reporting goals within desirable ranges was 79% for recall rate, 22% for false-positive rate, 39% for PPV2, and 61% for cancer detection rate. The range of reported goals was 0%-100% for false-positive rate and PPV2. Primary academic affiliation, receiving more hours of breast imaging continuing medical education, and receiving audit reports at least annually were associated with desirable PPV2 goals. Radiologists reporting desirable cancer detection rate goals were more likely to have interpreted mammograms for 10 or more years, and >1000 mammograms per year.

CONCLUSION

Many radiologists report goals for their accuracy when interpreting screening mammograms that fall outside of published desirable benchmarks, particularly for false-positive rate and PPV2, indicating an opportunity for education.

Use of clinical history affects accuracy of interpretive performance of screening mammography

OBJECTIVE

To examine how use of clinical history affects radiologist's interpretation of screening mammography.

STUDY DESIGN AND SETTING

Using a self-administered survey and actual interpretive performance, we examined associations between use of clinical history and sensitivity, false-positive rate, recall rate, and positive predictive value, after adjusting for relevant covariates using conditional logistic regression.

RESULTS

Of the 216 radiologists surveyed (63.4%), most radiologists reported usually or always using clinical history when interpreting screening mammography. Compared with radiologists who rarely use clinical history, radiologists who usually or always use it had a higher false-positive rate with younger women (10.7 vs. 9.7), denser breast tissue (10.1 for heterogeneously dense to 10.9 for extremely dense vs. 8.9 for fatty tissue), or longer screening intervals (> prior 5 years) (12.5 vs. 10.5). Effect of current hormone therapy (HT) use on false-positive rate was weaker among radiologists who use clinical history compared with those who did not (P=0.01), resulting in fewer false-positive examinations and a nonsignificant lower sensitivity (79.2 vs. 85.2) among HT users.

CONCLUSION

Interpretive performance appears to be influenced by patient age, breast density, screening interval, and HT use. This influence does not always result in improved interpretive performance.

Postoperative radiographs after maxillofacial trauma: Sense or nonsense?

The purpose of the present study was to investigate the necessity of routine postoperative radiographic analysis in patients with maxillofacial trauma. Between January 2000 and January 2010, 579 patients were treated surgically for 646 maxillofacial fractures including complex maxillofacial trauma. The incidence of surgical retreatments based on postoperative radiographs after maxillofacial trauma were investigated. 16 patients needed surgical retreatment. The decision to revise was based on postoperative imaging alone in one patient (0.2%). The available data in the literature concerning postoperative radiography in maxillofacial trauma was reviewed. Six useful studies concerning postoperative radiography in maxillofacial trauma were available for review. When combining these studies a total of 1377 patients underwent surgery for correction of a maxillofacial fracture. Nine patients returned to the operating theatre for correction of the initial procedure after trauma (0.7%). The present results are in line with the available literature. Routine postoperative radiography is not necessary after surgical treatment of maxillofacial trauma. Avoiding routine postoperative radiography will lead to a reduction in exposure of patients to ionizing radiation, a reduction of costs and probably a more efficient discharge.

Using a tailored web-based intervention to set goals to reduce unnecessary recall

RATIONALE AND OBJECTIVES

To examine whether an intervention strategy consisting of a tailored web-based intervention, which provides individualized audit data with peer comparisons and other data that can affect recall, can assist radiologists in setting goals for reducing unnecessary recall.

MATERIALS AND METHODS

In a multisite randomized controlled study, we used a tailored web-based intervention to assess radiologists' ability to set goals to improve interpretive performance. The intervention provided peer comparison audit data, profiled breast cancer risk in each radiologist's respective patient populations, and evaluated the possible impact of medical malpractice concerns. We calculated the percentage of radiologists who would consider changing their recall rates, and examined the specific goals they set to reduce recall rates. We describe characteristics of radiologists who developed realistic goals to reduce their recall rates, and their reactions to the importance of patient risk factors and medical malpractice concerns.

RESULTS

Forty-one of 46 radiologists (89.1%) who started the intervention completed it. Thirty-one (72.1%) indicated they would like to change their recall rates and 30 (69.8%) entered a text response about changing their rates. Sixteen of the 30 (53.3%) radiologists who included a text response set realistic goals that would likely result in reducing unnecessary recall. The actual recall rates of those who set realistic goals were not statistically different from those who did not (13.8% vs. 15.1%, respectively). The majority of selected goals involved re-reviewing cases initially interpreted as Breast Imaging Reporting and Data System category 0. More than half of radiologists who commented on the influence of patient risk (56.3%) indicated that radiologists planned to pay more attention to risk factors, and 100% of participants commented on concerns radiologists have about malpractice with the primary concern (37.5%) being fear of lawsuits.

CONCLUSIONS

Interventions designed to reduce unnecessary recall can succeed in assisting radiologists to develop goals that may ultimately reduce unnecessary recall.

Effect of previous benign breast biopsy on the interpretive performance of subsequent screening mammography

BACKGROUND

Most breast biopsies will be negative for cancer. Benign breast biopsy can cause changes in the breast tissue, but whether such changes affect the interpretive performance of future screening mammography is not known.

METHODS

We prospectively evaluated whether self-reported benign breast biopsy was associated with reduced subsequent screening mammography performance using examination data from the mammography registries of the Breast Cancer Surveillance Consortium from January 2, 1996, through December 31, 2005. A positive interpretation was defined as a recommendation for any additional evaluation. Cancer was defined as any invasive breast cancer or ductal carcinoma in situ diagnosed within 1 year of mammography screening. Measures of mammography performance (sensitivity, specificity, and positive predictive value 1 [PPV1]) were compared both at woman level and breast level in the presence and absence of self-reported benign biopsy history. Referral to biopsy was considered a positive interpretation to calculate positive predictive value 2 (PPV2). Multivariable analysis of a correct interpretation on each performance measure was conducted after adjusting for registry, year of examination, patient characteristics, months since last mammogram, and availability of comparison film. Accuracy of the mammogram interpretation was measured using area under the receiver operating characteristic curve (AUC). All statistical tests were two-sided.

RESULTS

A total of 2,007,381 screening mammograms were identified among 799,613 women, of which 14.6% mammograms were associated with self-reported previous breast biopsy. Multivariable adjusted models for mammography performance showed reduced specificity (odds ratio [OR] = 0.74, 95% confidence interval [CI] = 0.73 to 0.75, P < .001), PPV2 (OR = 0.85, 95% CI = 0.79 to 0.92, P < .001), and AUC (AUC 0.892 vs 0.925, P < .001) among women with self-reported benign biopsy. There was no difference in sensitivity or PPV1 in the same adjusted models, although unadjusted differences in both were found. Specificity was lowest among women with documented fine needle aspiration-the least invasive biopsy technique (OR = 0.58, 95% CI = 0.55 to 0.61, P < .001). Repeating the analysis among women with documented biopsy history, unilateral biopsy history, or restricted to invasive cancers did not change the results.

CONCLUSIONS

Self-reported benign breast biopsy history was associated with statistically significantly reduced mammography performance. The difference in performance was likely because of tissue characteristics rather than the biopsy itself.

Radiologists' attitudes and use of mammography audit reports

RATIONALE AND OBJECTIVES

The US Mammography Quality Standards Act mandates medical audits to track breast cancer outcomes data associated with interpretive performance. The objectives of our study were to assess the content and style of audits and examine use of, attitudes toward, and perceptions of the value that radiologists' have regarding mandated medical audits.

MATERIALS AND METHODS

Radiologists (n = 364) at mammography registries in seven US states contributing data to the Breast Cancer Surveillance Consortium (BCSC) were invited to participate. We examined radiologists' demographic characteristics, clinical experience, use, attitudes, and perceived value of audit reports from results of a self-administered survey. Information on the content and style of BCSC audits provided to radiologists and facilities was obtained from site investigators. Radiologists' characteristics were analyzed according to whether or not they self-reported receiving regular mammography audit reports. Latent class analysis was used to classify radiologists' individual perceptions of audit reports into overall probabilities of having "favorable," "less favorable," "neutral," or "unfavorable" attitudes toward audit reports.

RESULTS

Seventy-one percent (257 of 364) of radiologists completed the survey; two radiologists did not complete the audit survey question, leaving 255 for the final study cohort. Most survey respondents received regular audits (91%), paid close attention to their audit numbers (83%), found the reports valuable (87%), and felt that audit reports prompted them to improve interpretative performance (75%). Variability was noted in the style, target audience, and frequency of reports provided by the BCSC registries. One in four radiologists reported that if Congress mandates more intensive auditing requirements, but does not provide funding to support this regulation they may stop interpreting mammograms.

CONCLUSION

Radiologists working in breast imaging generally had favorable opinions of audit reports, which were mandated by Congress; however, almost 1 in 10 radiologists reported that they did not receive audits.