Variability in interpretive performance at screening mammography and radiologists' characteristics associated with accuracy

Range of Radiologist Performance in a Population-based Screening Cohort of 1 Million Digital Mammography Examinations

Background There is great interest in developing artificial intelligence (AI)-based computer-aided detection (CAD) systems for use in screening mammography. Comparative performance benchmarks from true screening cohorts are needed. Purpose To determine the range of human first-reader performance measures within a population-based screening cohort of 1 million screening mammograms to gauge the performance of emerging AI CAD systems. Materials and Methods This retrospective study consisted of all screening mammograms in women aged 40-74 years in Stockholm County, Sweden, who underwent screening with full-field digital mammography between 2008 and 2015. There were 110 interpreting radiologists, of whom 24 were defined as high-volume readers (ie, those who interpreted more than 5000 annual screening mammograms). A true-positive finding was defined as the presence of a pathology-confirmed cancer within 12 months. Performance benchmarks included sensitivity and specificity, examined per quartile of radiologists' performance. First-reader sensitivity was determined for each tumor subgroup, overall and by quartile of high-volume reader sensitivity. Screening outcomes were examined based on the first reader's sensitivity quartile with 10 000 screening mammograms per quartile. Linear regression models were fitted to test for a linear trend across quartiles of performance. Results A total of 418 041 women (mean age, 54 years ± 10 [standard deviation]) were included, and 1 186 045 digital mammograms were evaluated, with 972 899 assessed by high-volume readers. Overall sensitivity was 73% (95% confidence interval [CI]: 69%, 77%), and overall specificity was 96% (95% CI: 95%, 97%). The mean values per quartile of high-volume reader performance ranged from 63% to 84% for sensitivity and from 95% to 98% for specificity. The sensitivity difference was very large for basal cancers, with the least sensitive and most sensitive high-volume readers detecting 53% and 89% of cancers, respectively (P < .001). Conclusion Benchmarks showed a wide range of performance differences between high-volume readers. Sensitivity varied by tumor characteristics. © RSNA, 2020 Online supplemental material is available for this article.

A deep learning-based automated diagnostic system for classifying mammographic lesions

BACKGROUND

Screening mammography has led to reduced breast cancer-specific mortality and is recommended worldwide. However, the resultant doctors' workload of reading mammographic scans needs to be addressed. Although computer-aided detection (CAD) systems have been developed to support readers, the findings are conflicting regarding whether traditional CAD systems improve reading performance. Rapid progress in the artificial intelligence (AI) field has led to the advent of newer CAD systems using deep learning-based algorithms which have the potential to reach human performance levels. Those systems, however, have been developed using mammography images mainly from women in western countries. Because Asian women characteristically have higher-density breasts, it is uncertain whether those AI systems can apply to Japanese women. In this study, we will construct a deep learning-based CAD system trained using mammography images from a large number of Japanese women with high quality reading.

METHODS

We will collect digital mammography images taken for screening or diagnostic purposes at multiple institutions in Japan. A total of 15,000 images, consisting of 5000 images with breast cancer and 10,000 images with benign lesions, will be collected. At least 1000 images of normal breasts will also be collected for use as reference data. With these data, we will construct a deep learning-based AI system to detect breast cancer on mammograms. The primary endpoint will be the sensitivity and specificity of the AI system with the test image set.

DISCUSSION

When the ability of AI reading is shown to be on a par with that of human reading, images of normal breasts or benign lesions that do not have to be read by a human can be selected by AI beforehand. Our AI might work well in Asian women who have similar breast density, size, and shape to those of Japanese women.

TRIAL REGISTRATION

UMIN, trial number UMIN000039009. Registered 26 December 2019, https://www.umin.ac.jp/ctr/.

BI-RADS 3 (short-interval follow-up) assessment rate at diagnostic mammography: Correlation with recall rates and utilization as a performance benchmark

The purpose of this study was to identify a correlation between the screening BI-RADS 0 (recall) rates and diagnostic BI-RADS 3 (short-interval follow-up) rates of individual interpreting radiologists, with the goal of utilizing the BI-RADS 3 rate as an acceptable performance metric in the diagnostic population. A multicenter retrospective analysis of medical audit statistics was conducted on annual radiologist performance data collected over a 14-year period in a community hospital-based practice. Mixed regression models were used to estimate the association between screening BI-RADS 0 and diagnostic BI-RADS 3 examinations while adjusting for calendar year, annual radiologist screening volume, annual radiologist diagnostic volume, and diagnostic examination indication. A moderate statistically significant positive correlation was established between the screening BI-RADS 0 rates and Diagnostic BI-RADS 3 rates (Pearson correlation coefficient + 0.349, P ≤ .001). Furthermore, when utilizing a national benchmark range of 8%-12% as an acceptable BI-RADS 0 rate within a screening population, the correlative BI-RADS 3 assessment rate was demonstrated to be approximately 16%. We propose that this BI-RADS category 3 rate may represent an additional acceptable performance metric in the diagnostic population. Routine inclusion of an interpreting mammographer's diagnostic BI-RADS 3 rate in the annual medical audit may help reduce inappropriate and/or excess use of the BI-RADS 3 category, which may lead to significant potential reductions in follow-up examinations with their associated healthcare-related costs, resource expenditure, and induced patient anxiety.

The roles of clinical audit and test sets in promoting the quality of breast screening: a scoping review

Breast screening programmes enhance the probability of early breast cancer detection in many countries worldwide; however, the success of these efforts is highly dependent on the ability of breast screen readers to detect abnormalities in the screened population, which has low prevalence. Therefore, this task can be challenging. Clinical audit is a key quality assurance measure that aims to keep the screen reading performance within acceptable standards. Auditing, nonetheless, is a lengthy process, and its accuracy is dependent on available clinical data, which often can be limited. Mammographic standardised test sets are a different screen reading evaluation approach that provides participants with instant feedback based on a simulated environment. Although a test set provides unique evaluative qualities, its ability to represent clinical performance is debated. This article describes the distinctive roles of clinical audit and test sets in measuring and improving the quality of breast screening and highlights the relationship between test sets and clinical performance.

Is there a safety-net effect with computer-aided detection?

Computer-aided detection (CAD) systems are used to aid readers interpreting screening mammograms. An expert reader searches the image initially unaided and then once again with the aid of CAD, which prompts automatically detected suspicious regions. This could lead to a "safety-net" effect, where the initial unaided search of the image is adversely affected by the fact that it is preliminary to an additional search with CAD and may, therefore, be less thorough. To investigate the existence of such an effect, we created a visual search experiment for nonexpert observers mirroring breast screening with CAD. Each observer searched 100 images for microcalcification clusters within synthetic images in both prompted (CAD) and unprompted (no-CAD) conditions. Fifty-two participants were recruited for the study, 48 of whom had their eye movements tracked in real-time; the other 4 participants could not be accurately calibrated, so only behavioral data were collected. In the CAD condition, before prompts were displayed, image coverage was significantly lower than coverage in the no-CAD condition (t 47 = 5.29 , p < 0.0001 ). Observer sensitivity was significantly greater for targets marked by CAD than the same targets in the no-CAD condition (t 51 = 6.56 , p < 0.001 ). For targets not marked by CAD, there was no significant difference in observer sensitivity in the CAD condition compared with the same targets in the no-CAD condition (t 51 = 0.54 , p = 0.59 ). These results suggest that the initial search may be influenced by the subsequent availability of CAD; if so, cross-sectional CAD efficacy studies should account for the effect when estimating benefit.

Changes in cancer detection and false-positive recall in mammography using artificial intelligence: a retrospective, multireader study

BACKGROUND

Mammography is the current standard for breast cancer screening. This study aimed to develop an artificial intelligence (AI) algorithm for diagnosis of breast cancer in mammography, and explore whether it could benefit radiologists by improving accuracy of diagnosis.

METHODS

In this retrospective study, an AI algorithm was developed and validated with 170 230 mammography examinations collected from five institutions in South Korea, the USA, and the UK, including 36 468 cancer positive confirmed by biopsy, 59 544 benign confirmed by biopsy (8827 mammograms) or follow-up imaging (50 717 mammograms), and 74 218 normal. For the multicentre, observer-blinded, reader study, 320 mammograms (160 cancer positive, 64 benign, 96 normal) were independently obtained from two institutions. 14 radiologists participated as readers and assessed each mammogram in terms of likelihood of malignancy (LOM), location of malignancy, and necessity to recall the patient, first without and then with assistance of the AI algorithm. The performance of AI and radiologists was evaluated in terms of LOM-based area under the receiver operating characteristic curve (AUROC) and recall-based sensitivity and specificity.

FINDINGS

The AI standalone performance was AUROC 0·959 (95% CI 0·952-0·966) overall, and 0·970 (0·963-0·978) in the South Korea dataset, 0·953 (0·938-0·968) in the USA dataset, and 0·938 (0·918-0·958) in the UK dataset. In the reader study, the performance level of AI was 0·940 (0·915-0·965), significantly higher than that of the radiologists without AI assistance (0·810, 95% CI 0·770-0·850; p<0·0001). With the assistance of AI, radiologists' performance was improved to 0·881 (0·850-0·911; p<0·0001). AI was more sensitive to detect cancers with mass (53 [90%] vs 46 [78%] of 59 cancers detected; p=0·044) or distortion or asymmetry (18 [90%] vs ten [50%] of 20 cancers detected; p=0·023) than radiologists. AI was better in detection of T1 cancers (73 [91%] vs 59 [74%] of 80; p=0·0039) or node-negative cancers (104 [87%] vs 88 [74%] of 119; p=0·0025) than radiologists.

INTERPRETATION

The AI algorithm developed with large-scale mammography data showed better diagnostic performance in breast cancer detection compared with radiologists. The significant improvement in radiologists' performance when aided by AI supports application of AI to mammograms as a diagnostic support tool.

FUNDING

Lunit.

International evaluation of an AI system for breast cancer screening

Screening mammography aims to identify breast cancer at earlier stages of the disease, when treatment can be more successful1. Despite the existence of screening programmes worldwide, the interpretation of mammograms is affected by high rates of false positives and false negatives2. Here we present an artificial intelligence (AI) system that is capable of surpassing human experts in breast cancer prediction. To assess its performance in the clinical setting, we curated a large representative dataset from the UK and a large enriched dataset from the USA. We show an absolute reduction of 5.7% and 1.2% (USA and UK) in false positives and 9.4% and 2.7% in false negatives. We provide evidence of the ability of the system to generalize from the UK to the USA. In an independent study of six radiologists, the AI system outperformed all of the human readers: the area under the receiver operating characteristic curve (AUC-ROC) for the AI system was greater than the AUC-ROC for the average radiologist by an absolute margin of 11.5%. We ran a simulation in which the AI system participated in the double-reading process that is used in the UK, and found that the AI system maintained non-inferior performance and reduced the workload of the second reader by 88%. This robust assessment of the AI system paves the way for clinical trials to improve the accuracy and efficiency of breast cancer screening.

Improvement of Cancer Detection on Mammograms via BREAST Test Sets

BACKGROUND

Breast Screen Reader Assessment Strategy (BREAST) is an innovative training and research program for radiologists in Australia and New Zealand. The aim of this study is to evaluate the efficacy of BREAST test sets in improving readers' performance in detecting cancers on mammograms.

MATERIALS AND METHODS

Between 2011 and 2018, 50 radiologists (40 fellows, 10 registrars) completed three BREAST test sets and 17 radiologists completed four test sets. Each test set contained 20 biopsy-proven cancer and 40 normal cases. Immediate image-based feedback was available to readers after they completed each test set which allowed the comparison of their selections with the truth. Case specificity, case sensitivity, lesion sensitivity, the Receiver Operating Characteristic (ROC) Area Under the Curve (AUC) and Jackknife Free-Response Receiver Operating Characteristic (JAFROC) Figure of Merit (FOM) were calculated for each reader. Kruskal-Wallis test was utilized to compare scores of the radiologist and registrars across all test-sets whilst Wilcoxon signed rank test was to compare the scores between pairs of test sets.

RESULTS

Significant improvements in lesion sensitivity ranging from 21% to 31% were found in radiologists completing later test sets compared to first test set (p ≤ 0.01). Eighty three percent of radiologists achieved higher performance in lesion sensitivity after they completed the first read. Registrars had significantly better scores in the third test set compared to the first set with mean increases of 79% in lesion sensitivity (p = 0.005) and 37% in JAFROC (p = 0.02). Sixty percent and 100% of registrars increased their scores in lesion sensitivity in the second and third reads compared to the first read while the percentage of registrars with higher scores in JAFROC was 80%.

CONCLUSION

Introduction of BREAST into national training programs appears to have an important impact in promoting diagnostic efficacy amongst radiologists and radiology registrars undergoing mammographic readings.

Effects of time of day on radiological interpretation

Accurate interpretation of radiological images involves a complex visual search that relies on several cognitive processes, including selective attention, working memory, and decision-making. Patient outcomes often depend on the accuracy of image interpretations, and yet research has revealed that conclusions vary significantly from one radiologist to another. A myriad of factors has been shown to contribute to the likelihood of interpretative errors and discrepancies, including the radiologist's level of experience and fatigue, and these factors are well reported elsewhere; however, a potentially important factor that has been given little previous consideration is how radiologists' interpretations might be impacted by the time of day at which the reading takes place, a factor that other disciplines have shown to be a determinant of competency. The available literature shows that while the time of day is known to significantly impact some cognitive functions that likely relate to reading competence, including selective visual attention and visual working memory, little is known about the impact of the time of day on radiology interpretation performance. This review explores the evidence regarding the relationship between time of day and performance, with a particular emphasis on radiological activities.

Interpretive Performance and Inter-Observer Agreement on Digital Mammography Test Sets

Objective

To evaluate the interpretive performance and inter-observer agreement on digital mammographs among radiologists and to investigate whether radiologist characteristics affect performance and agreement.

Materials and Methods

The test sets consisted of full-field digital mammograms and contained 12 cancer cases among 1000 total cases. Twelve radiologists independently interpreted all mammograms. Performance indicators included the recall rate, cancer detection rate (CDR), positive predictive value (PPV), sensitivity, specificity, false positive rate (FPR), and area under the receiver operating characteristic curve (AUC). Inter-radiologist agreement was measured. The reporting radiologist characteristics included number of years of experience interpreting mammography, fellowship training in breast imaging, and annual volume of mammography interpretation.

Results

The mean and range of interpretive performance were as follows: recall rate, 7.5% (3.3–10.2%); CDR, 10.6 (8.0–12.0 per 1000 examinations); PPV, 15.9% (8.8–33.3%); sensitivity, 88.2% (66.7–100%); specificity, 93.5% (90.6–97.8%); FPR, 6.5% (2.2–9.4%); and AUC, 0.93 (0.82–0.99). Radiologists who annually interpreted more than 3000 screening mammograms tended to exhibit higher CDRs and sensitivities than those who interpreted fewer than 3000 mammograms (p = 0.064). The inter-radiologist agreement showed a percent agreement of 77.2–88.8% and a kappa value of 0.27–0.34. Radiologist characteristics did not affect agreement.

Conclusion

The interpretative performance of the radiologists fulfilled the mammography screening goal of the American College of Radiology, although there was inter-observer variability. Radiologists who interpreted more than 3000 screening mammograms annually tended to perform better than radiologists who did not.

Surveillance Breast MRI and Mammography: Comparison in Women with a Personal History of Breast Cancer

Background There is lack of consensus regarding the use of breast MRI for routine surveillance for second breast cancer events in women with a personal history of breast cancer. Purpose To compare performance of surveillance mammography with breast MRI. Materials and Methods This observational cohort study used prospectively collected data and included 13 266 women age 18 years and older (mean age, 60 years ± 13) with stage 0-III breast cancer who underwent 33 938 mammographic examinations and 2506 breast MRI examinations from 2005 to 2012 in the Breast Cancer Surveillance Consortium. Women were categorized into two groups: mammography alone (n = 11 745) or breast MRI (n = 1521). Performance measures were calculated by using end-of-day assessment and occurrence of second breast cancer events within 1 year of imaging. Logistic regression was used to compare performance for breast MRI versus mammography alone, adjusting for women, examination, and primary breast cancer characteristics. Analysis was conducted on a per-examination basis. Results Breast MRI was associated with younger age at diagnosis, chemotherapy, and higher education and income. Raw performance measures for breast MRI versus mammography were as follows, respectively: cancer detection rates, 10.8 (95% confidence interval [CI]: 6.7, 14.8) versus 8.2 (95% CI: 7.3, 9.2) per 1000 examinations; sensitivity, 61.4% (27 of 44; 95% CI: 46.5%, 76.2%) versus 70.3% (279 of 397; 95% CI: 65.8%, 74.8%); and biopsy rate, 10.1% (253 of 2506; 95% CI: 8.9%, 11.3%) versus 4.0% (1343 of 33 938; 95% CI: 3.7%, 4.2%). In multivariable models, breast MRI was associated with higher biopsy rate (odds ratio [OR], 2.2; 95% CI: 1.9, 2.7; P < .001) and cancer detection rate (OR, 1.7; 95% CI: 1.1, 2.7; P = .03) than mammography alone. However, there were no differences in sensitivity (OR, 1.1; 95% CI: 0.4, 2.9; P = .84) or interval cancer rate (OR, 1.1; 95% CI: 0.6, 2.2; P = .70). Conclusion Comparison of the performance of surveillance breast MRI with mammography must account for patient characteristics. Whereas breast MRI leads to higher biopsy and cancer detection rates, there were no significant differences in sensitivity or interval cancers compared with mammography. © RSNA, 2019 Online supplemental material is available for this article. See also the editorial by Newell in this issue.

Artificial intelligence in breast imaging

This article reviews current limitations and future opportunities for the application of computer-aided detection (CAD) systems and artificial intelligence in breast imaging. Traditional CAD systems in mammography screening have followed a rules-based approach, incorporating domain knowledge into hand-crafted features before using classical machine learning techniques as a classifier. The first commercial CAD system, ImageChecker M1000, relies on computer vision techniques for pattern recognition. Unfortunately, CAD systems have been shown to adversely affect some radiologists' performance and increase recall rates. The Digital Mammography DREAM Challenge was a multidisciplinary collaboration that provided 640,000 mammography images for teams to help decrease false-positive rates in breast cancer screening. Winning solutions leveraged deep learning's (DL) automatic hierarchical feature learning capabilities and used convolutional neural networks. Start-ups Therapixel and Kheiron Medical Technologies are using DL for breast cancer screening. With increasing use of digital breast tomosynthesis, specific artificial intelligence (AI)-CAD systems are emerging to include iCAD's PowerLook Tomo Detection and ScreenPoint Medical's Transpara. Other AI-CAD systems are focusing on breast diagnostic techniques such as ultrasound and magnetic resonance imaging (MRI). There is a gap in the market for contrast-enhanced spectral mammography AI-CAD tools. Clinical implementation of AI-CAD tools requires testing in scenarios mimicking real life to prove its usefulness in the clinical environment. This requires a large and representative dataset for testing and assessment of the reader's interaction with the tools. A cost-effectiveness assessment should be undertaken, with a large feasibility study carried out to ensure there are no unintended consequences. AI-CAD systems should incorporate explainable AI in accordance with the European Union General Data Protection Regulation (GDPR).

A Probabilistic Model to Support Radiologists' Classification Decisions in Mammography Practice

We developed a probabilistic model to support the classification decisions made by radiologists in mammography practice. Using the feature observations and Breast Imaging Reporting and Data System (BI-RADS) classifications from radiologists examining diagnostic and screening mammograms, we modeled their decisions to understand their judgments. Our model could help improve the decisions made by radiologists using their own feature observations and classifications while maintaining their observed sensitivities. Based on 112,433 mammographic cases from 36,111 patients and 13 radiologists at 2 separate institutions with a 1.1% prevalence of malignancy, we trained a probabilistic Bayesian network (BN) to estimate the malignancy probabilities of lesions. For each radiologist, we learned an observed probabilistic threshold within the model. We compared the sensitivity and specificity of each radiologist against the BN model using either their observed threshold or the standard 2% threshold recommended by BI-RADS. We found significant variability among the radiologists' observed thresholds. By applying the observed thresholds, the BN model showed a 0.01% (1 case) increase in false negatives and a 28.9% (3612 cases) reduction in false positives. When using the standard 2% BI-RADS-recommended threshold, there was a 26.7% (47 cases) increase in false negatives and a 47.3% (5911 cases) reduction in false positives. Our results show that we can significantly reduce screening mammography false positives with a minimal increase in false negatives. We find that learning radiologists' observed thresholds provides valuable information regarding the conservativeness of clinical practice and allows us to quantify the variability in sensitivity across and within institutions. Our model could provide support to radiologists to improve their performance and consistency within mammography practice.

Current Clinical Practice Patterns of Self-Identified Nuclear Medicine Specialists

OBJECTIVE

The objective of our study was to study patterns of services rendered by U.S. physicians who self-identify as nuclear medicine (NM) specialists.

MATERIALS AND METHODS

Recent Medicare physician claims and demographic files were obtained and linked. NM specialists were defined as physicians self-identifying NM as their primary specialty on claims or as any of their specialties during enrollment. Using other self-identified specialties, we classified physicians as nuclear radiologists, nuclear cardiologists, exclusively NM physicians, or Others. Our primary outcome measure was the percentage of NM effort (in work relative value units [WRVUs]) per physician per specialty group. Secondary outcome measures included physician sociodemographic parameters and most common uniquely rendered services.

RESULTS

Nationally, 1583 physicians self-identified as NM specialists during the calendar years 2012 through 2015. The distribution of WRVUs attributed to NM varied widely by specialty group; most nuclear radiologists and nuclear cardiologists devoted 10% or less of their effort to NM services whereas most NM physicians devoted 90% or more of their effort to NM services. NM specialists were most commonly nuclear radiologists (52.2%) and men (80.3%) and practiced in urban (98.4%) and nonacademic settings (62.9%). NM physicians interpreted more general NM studies, nuclear radiologists interpreted more cross-sectional imaging studies, and nuclear cardiologists interpreted mostly nuclear cardiology studies, with a majority of their overall work attributed to clinical evaluation and management (E/M). E/M services accounted for less than 2% of WRVUs for both nuclear radiologists and NM physicians.

CONCLUSION

The work patterns of U.S. NM specialists is highly variable. Most NM physicians practice 90% or more NM, whereas most nuclear radiologists and nuclear cardiologists practice 10% or less NM. Commonly performed services vary considerably by specialty group.

Mammographic detection of breast cancer in a non-screening country

OBJECTIVE:

To compare the diagnostic accuracy between radiologists' from a country with and without breast cancer screening.

METHODS:

All participating radiologists gave informed consent. A test-set involving 60 mammographic cases (20 cancer and 40 non-cancer) were read by 11 radiologists from a non-screening (NS) country during a workshop in July 2016. 52 radiologists from a screening country read the same test-set at the Royal Australian and New Zealand College of Radiologists' meetings in July 2015. The screening radiologists were classified into two groups: those with less than or equal to 5 years of experience; those with more than 5 years of experience, and each group was compared to the group of NS radiologists. A Kruskal-Wallis test followed by post-hoc multiple comparisons test were used to compare measures of diagnostic accuracy among the reader groups.

RESULTS:

The diagnostic accuracy of the NS radiologists was significantly lower in terms of sensitivity [mean = 54.0; 95% confidence interval (CI) (40.0-67.0)], location sensitivity [mean = 26.0; 95% CI (16.0-37.0)], receive roperating characteristic area under curve [mean = 73.0; 95% CI (66.5-81.0)] and Jackknifefree-response receiver operating characteristics figure-of-merit [mean = 45.0; 95% CI (40.0-50.0)] when compared with the less and more experienced screening radiologists, whilst no difference in specificity [mean = 75.0; 95% CI (70.0- 81.0)] was found. No significant differences in all measured diagnostic accuracy were found between the two groups of screening radiologists.

CONCLUSION:

The mammographic performance of a group of radiologists from a country without screening program was suboptimal compared with radiologists from Australia.

ADVANCES IN KNOWLEDGE:

Identifying mammographic performance in developing countries is required to optimize breast cancer diagnosis.

Second opinion interpretation of breast ultrasound images-Is it worth another look?

PURPOSE

To determine if real-time breast ultrasound (RTUR) after second opinion reinterpretation of submitted static ultrasound images at a comprehensive cancer center impacts clinical management, specifically by detecting additional cancer and preventing unnecessary biopsy.

MATERIALS AND METHODS

In this IRB-approved and HIPAA-compliant retrospective study, 209 patients were included who had breast ultrasound studies from outside facilities submitted for second opinion review between January 2013 and May 2014, and who subsequently underwent RTUR at our institution within three months of the outside study. Findings on submitted exams were compared with those on RTUR and disagreements between them were annotated to indicate the presence or absence of suspicious lesions and recommendation for biopsy. Changes in management were defined as any additional biopsies performed or biopsies averted after RTUR and reported as frequencies and percentages using 95% confidence intervals.

RESULTS

Following RTUR, 49 additional biopsies were performed in 43/209 patients (20.6%; 95% CI 15.1-26.1%). Additional cancer was found in 12/49 (24.5%) biopsies in 11/209 patients (5.3%; 95% CI 2.2-8.2%). Forty biopsies in 31/209 (14.8%; CI 10.0-19.7%) patients originally recommended were canceled after RTUR. Overall, a change in management after RTUR was observed in 68/209 patients (32.5%; 95% CI 26.1-38.9%), including patients with either additional biopsies performed or biopsies averted.

CONCLUSION

RTUR was found to be an important tool in the management of patients at our comprehensive cancer center. Although additional false-positive lesions may be detected on RTUR, a great number of patients will benefit from RTUR in finding additional cancers or avoiding unnecessary biopsies.

Ensemble of convolutional neural networks for classification of breast microcalcification from mammograms

Human level recall performance in detecting breast cancer considering microcalcifications from mammograms has a recall value between 74.5% and 92.3%. In this research, we approach to breast microcalcification classification problem using convolutional neural networks along with various preprocessing methods such as contrast scaling, dilation, cropping etc. and decision fusion using ensemble of networks. Various experiments on Digital Database for Screening Mammography dataset showed that preprocessing poses great importance on the classification performance. The stand-alone models using the dilation and cropping preprocessing techniques achieved the highest recall value of 91.3%. The ensembles of the stand-alone models surpass this recall value and a 97.3% value of recall is achieved. The ensemble having the highest F1 Score (harmonic mean of precision and recall), which is 94.5%, has a recall value of 94.0% and a precision value of 95.0%. This recall is still above human level performance and the models achieve competitive results in terms of accuracy, precision, recall and F1 score measures.

Deep Convolutional Neural Networks for breast cancer screening

BACKGROUND AND OBJECTIVE

Radiologists often have a hard time classifying mammography mass lesions which leads to unnecessary breast biopsies to remove suspicions and this ends up adding exorbitant expenses to an already burdened patient and health care system.

METHODS

In this paper we developed a Computer-aided Diagnosis (CAD) system based on deep Convolutional Neural Networks (CNN) that aims to help the radiologist classify mammography mass lesions. Deep learning usually requires large datasets to train networks of a certain depth from scratch. Transfer learning is an effective method to deal with relatively small datasets as in the case of medical images, although it can be tricky as we can easily start overfitting.

RESULTS

In this work, we explore the importance of transfer learning and we experimentally determine the best fine-tuning strategy to adopt when training a CNN model. We were able to successfully fine-tune some of the recent, most powerful CNNs and achieved better results compared to other state-of-the-art methods which classified the same public datasets. For instance we achieved 97.35% accuracy and 0.98 AUC on the DDSM database, 95.50% accuracy and 0.97 AUC on the INbreast database and 96.67% accuracy and 0.96 AUC on the BCDR database. Furthermore, after pre-processing and normalizing all the extracted Regions of Interest (ROIs) from the full mammograms, we merged all the datasets to build one large set of images and used it to fine-tune our CNNs. The CNN model which achieved the best results, a 98.94% accuracy, was used as a baseline to build the Breast Cancer Screening Framework. To evaluate the proposed CAD system and its efficiency to classify new images, we tested it on an independent database (MIAS) and got 98.23% accuracy and 0.99 AUC.

CONCLUSION

The results obtained demonstrate that the proposed framework is performant and can indeed be used to predict if the mass lesions are benign or malignant.

Diagnostic Accuracy of Digital Breast Tomosynthesis in the Evaluation of Palpable Breast Abnormalities

RATIONALE AND OBJECTIVES

The role of digital breast tomosynthesis (DBT) in evaluating palpable abnormalities has not been evaluated and its accuracy compared to 2D mammography is unknown. The purpose of this study was to evaluate combined 2D mammography, DBT, and ultrasound (US) at palpable sites.

MATERIALS AND METHODS

Two breast imagers reviewed blinded consecutive cases with combined 2D mammograms and DBT examinations performed for palpable complaints. By consensus, 2D and DBT findings were recorded and compared to US. Patient characteristics, demographics, subsequent workup, and outcome were recorded.

RESULTS

A total of 229 sites in 188 patients were included, with 50 biopsies performed identifying 18 cancers. All 18 cancers were identified on 2D and US, whereas 17 cancers were identified on DBT. Cancer detection sensitivities for 2D, DBT, and US were 100.0%, 94.4%, and 100.0%. The negative predictive value, when combined with US, was 100% for both. The sensitivity and the specificity for both benign and malignant findings with 2D and DBT were 70.5% versus 75.4% (P = 0.07) and 95.3% versus 99.1% (P = 0.125). Palpable findings not identified by 2D and DBT were smaller than those identified (11.5 ± 8.3 mm vs 23.9 ± 12.8 mm, P < 0.001). Patients with dense breasts were more likely to have mammographically occult findings than patients with nondense breasts (27.4% vs 8.3%).

CONCLUSIONS

DBT did not improve cancer detection over 2D or US. Both mammographic modalities failed to identify sonographically confirmed findings primarily in dense breasts. The diagnostic use of DBT at palpable sites provided limited benefit over combined 2D and US. When utilizing DBT, US should be performed to adequately characterize palpable sites.

Increasing the Efficiency on Producing Radiology Reports for Breast Cancer Diagnosis by Means of Structured Reports

Background: Radiology reports are commonly written on free-text using voice recognition devices. Structured reports (SR) have a high potential but they are usually considered more difficult to fill-in so their adoption in clinical practice leads to a lower efficiency. However, some studies have demonstrated that in some cases, producing SRs may require shorter time than plain-text ones. This work focuses on the definition and demonstration of a methodology to evaluate the productivity of software tools for producing radiology reports. A set of SRs for breast cancer diagnosis based on BI-RADS have been developed using this method. An analysis of their efficiency with respect to free-text reports has been performed.

Material and Methods: The methodology proposed compares the Elapsed Time (ET) on a set of radiological reports. Free-text reports are produced with the speech recognition devices used in the clinical practice. Structured reports are generated using a web application generated with TRENCADIS framework. A team of six radiologists with three different levels of experience in the breast cancer diagnosis was recruited. These radiologists performed the evaluation, each one introducing 50 reports for mammography, 50 for ultrasound scan and 50 for MRI using both approaches. Also, the Relative Efficiency (REF) was computed for each report, dividing the ET of both methods. We applied the T-Student (T-S) test to compare the ETs and the ANOVA test to compare the REFs. Both tests were computed using the SPSS software.

Results: The study produced three DICOM- SR templates for Breast Cancer Diagnosis on mammography, ultrasound and MRI, using RADLEX terms based on BIRADs 5th edition. The T-S test on radiologists with high or intermediate profile, showed that the difference between the ET was only statistically significant for mammography and ultrasound. The ANOVA test performed grouping the REF by modalities, indicated that there were no significant differences between mammograms and ultrasound scans, but both have significant statistical differences with MRI. The ANOVA test of the REF for each modality, indicated that there were only significant differences in Mammography (ANOVA p = 0.024) and Ultrasound (ANOVA p = 0.008). The ANOVA test for each radiologist profile, indicated that there were significant differences on the high profile (ANOVA p = 0.028) and medium (ANOVA p=0.045).

Conclusions: In this work, we have defined and demonstrated a methodology to evaluate the productivity of software tools for producing radiology reports in Breast Cancer. We have evaluated that adopting Structured Reporting in mammography and ultrasound studies in breast cancer diagnosis improves the performance in producing reports.

ACR BI-RADS Assessment Category 4 Subdivisions in Diagnostic Mammography: Utilization and Outcomes in the National Mammography Database

Purpose To determine the utilization and positive predictive value (PPV) of the American College of Radiology (ACR) Breast Imaging Data and Reporting System (BI-RADS) category 4 subdivisions in diagnostic mammography in the National Mammography Database (NMD). Materials and Methods This study involved retrospective review of diagnostic mammography data submitted to the NMD from January 1, 2008 to December 30, 2014. Utilization rates of BI-RADS category 4 subdivisions were compared by year, facility (type, location, census region), and examination (indication, finding type) characteristics. PPV3 (positive predictive value for biopsies performed) was calculated overall and according to category 4 subdivision. The χ² test was used to test for significant associations. Results Of 1 309 950 diagnostic mammograms, 125 447 (9.6%) were category 4, of which 33.3% (41 841 of 125 447) were subdivided. Subdivision utilization rates were higher (P < .001) in practices that were community, suburban, or in the West; for examination indication of prior history of breast cancer; and for the imaging finding of architectural distortion. Of 41 841 category 4 subdivided examinations, 4A constituted 55.6% (23 258 of 41 841) of the examinations; 4B, 31.8% (13 302 of 41 841) of the examinations; and 4C, 12.6% (5281 of 41 841) of the examinations. Pathologic outcomes were available in 91 563 examinations, and overall category 4 PPV3 was 21.1% (19 285 of 91 563). There was a statistically significant difference in PPV3 according to category 4 subdivision (P < .001): The PPV of 4A was 7.6% (1274 of 16 784), that of 4B was 22% (2317 of 10 408), and that of 4C was 69.3% (2839 of 4099). Conclusion Although BI-RADS suggests their use, subdivisions were utilized in the minority (33.3% [41 841 of 125 447]) of category 4 diagnostic mammograms, with variability based on facility and examination characteristics. When subdivisions were used, PPV3s were in BI-RADS-specified malignancy ranges. This analysis supports the use of subdivisions in broad practice and, given benefits for patient care, should motivate increased utilization. ^© RSNA, 2018 Online supplemental material is available for this article.

Correlation Between Screening Mammography Interpretive Performance on a Test Set and Performance in Clinical Practice

RATIONALE AND OBJECTIVES

Evidence is inconsistent about whether radiologists' interpretive performance on a screening mammography test set reflects their performance in clinical practice. This study aimed to estimate the correlation between test set and clinical performance and determine if the correlation is influenced by cancer prevalence or lesion difficulty in the test set.

MATERIALS AND METHODS

This institutional review board-approved study randomized 83 radiologists from six Breast Cancer Surveillance Consortium registries to assess one of four test sets of 109 screening mammograms each; 48 radiologists completed a fifth test set of 110 mammograms 2 years later. Test sets differed in number of cancer cases and difficulty of lesion detection. Test set sensitivity and specificity were estimated using woman-level and breast-level recall with cancer status and expert opinion as gold standards. Clinical performance was estimated using women-level recall with cancer status as the gold standard. Spearman rank correlations between test set and clinical performance with 95% confidence intervals (CI) were estimated.

RESULTS

For test sets with fewer cancers (N = 15) that were more difficult to detect, correlations were weak to moderate for sensitivity (woman level = 0.46, 95% CI = 0.16, 0.69; breast level = 0.35, 95% CI = 0.03, 0.61) and weak for specificity (0.24, 95% CI = 0.01, 0.45) relative to expert recall. Correlations for test sets with more cancers (N = 30) were close to 0 and not statistically significant.

CONCLUSIONS

Correlations between screening performance on a test set and performance in clinical practice are not strong. Test set performance more accurately reflects performance in clinical practice if cancer prevalence is low and lesions are challenging to detect.

Breast Density Legislation in New England: A Survey Study of Practicing Radiologists

RATIONALE AND OBJECTIVES

This study aimed to assess radiologists' knowledge about breast density legislation as well as perceived practice changes resulting from the enactment of breast density legislation.

MATERIALS AND METHODS

This is an institutional review board-exempt anonymous email survey of 523 members of the New England Roentgen Ray Society. In addition to radiologist demographics, survey questions addressed radiologist knowledge of breast density legislation, knowledge of breast density as a risk factor for breast cancer, recommendations for supplemental screening, and perceived practice changes resulting from density notification legislation.

RESULTS

Of the 523 members, 96 responded, yielding an 18% response rate. Seventy-three percent of respondents practiced in a state with breast density legislation. Sixty-nine percent felt that breast density notification increased patient anxiety about breast cancer, but also increased patient (74%) and provider (66%) understanding of the effect of breast density on mammographic sensitivity. Radiologist knowledge of the relative risk of breast cancer when comparing breasts of different density was variable.

CONCLUSIONS

Considerable confusion and controversy regarding breast density persists, even among practicing radiologists.

An investigation into the mammographic appearances of missed breast cancers when recall rates are reduced

OBJECTIVE

This study investigated whether certain mammographic appearances of breast cancer are missed when radiologists read at lower recall rates.

METHODS

5 radiologists read 1 identical test set of 200 mammographic (180 normal cases and 20 abnormal cases) 3 times and were requested to adhere to 3 different recall rate conditions: free recall, 15% and 10%. The radiologists were asked to mark the locations of suspicious lesions and provide a confidence rating for each decision. An independent expert radiologist identified the various types of cancers in the test set, including the presence of calcifications and the lesion location, including specific mammographic density.

RESULTS

Radiologists demonstrated lower sensitivity and receiver operating characteristic area under the curve for non-specific density/asymmetric density (H = 6.27, p = 0.04 and H = 7.35, p = 0.03, respectively) and mixed features (H = 9.97, p = 0.01 and H = 6.50, p = 0.04, respectively) when reading at 15% and 10% recall rates. No significant change was observed on cancer characterized with stellate masses (H = 3.43, p = 0.18 and H = 1.23, p = 0.54, respectively) and architectural distortion (H = 0.00, p = 1.00 and H = 2.00, p = 0.37, respectively). Across all recall conditions, stellate masses were likely to be recalled (90.0%), whereas non-specific densities were likely to be missed (45.6%).

CONCLUSION

Cancers with a stellate mass were more easily detected and were more likely to continue to be recalled, even at lower recall rates. Cancers with non-specific density and mixed features were most likely to be missed at reduced recall rates. Advances in knowledge: Internationally, recall rates vary within screening mammography programs considerably, with a range between 1% and 15%, and very little is known about the type of breast cancer appearances found when radiologists interpret screening mammograms at these various recall rates. Therefore, understanding the lesion types and the mammographic appearances of breast cancers that are affected by readers' recall decisions should be investigated.

Analysis of Participant Factors That Affect the Diagnostic Performance of Screening Mammography: A Report of the Alliance for Breast Cancer Screening in Korea

OBJECTIVE

To analyze participant factors that affect the diagnostic performance of screening mammography.

MATERIALS AND METHODS

We enrolled 128756 cases from 10 hospitals between 2005 and 2010. We analyzed recall rate, cancer detection rate (CDR) per 1000 examinations, positive predictive value (PPV), sensitivity, specificity, false positive rate (FPR), and interval cancer rate (ICR) per 1000 negative examinations according to participant factors including age, breast density, and number of visit to the same institution, and adjusted for confounding variables.

RESULTS

Increasing age improved recall rates (27.4% in 40's, 17.5% in 50's, 11.1% in 60's, and 8.6% in 70's), CDR (2.7, 3.2, 2.0, and 2.4), PPV (1.0, 1.8, 1.8, and 2.8%), sensitivity (81.3, 88.8, 90.3, and 94.7%), specificity (72.7, 82.7, 89.0, and 91.7%), and FPR (27.3, 17.3, 11.0, and 8.4%) (p < 0.05). Higher breast density impaired recall rates (4.0% in P1, 9.0% in P2, 28.9% in P3, and 27.8% in P4), PPV (3.3, 2.3, 1.2, and 1.3%), specificity (96.1, 91.2, 71.4, and 72.5%), and FPR (3.9, 8.9, 28.6, and 27.6%) (p < 0.001). It also increased CDR (1.3, 2.1, 3.3, and 3.6) and ICR (0.2, 0.3, 0.6, and 1.6) (p < 0.05). Successive visits to the same institution improved recall rates (20.9% for one visit, 10.7% for two visits, 7.7% for more than three visits), PPV (1.6, 2.8, and 2.7%), specificity (79.4, 89.6, and 92.5%), and FPR (20.6, 10.4, and 7.5%) (p < 0.001).

CONCLUSION

Young age and dense breasts negatively affected diagnostic performance in mammography screening, whereas successive visits to the same institution had a positive effect. Examinee education for successive visits to the same institution would improve the diagnostic performance.

Lack of agreement between radiologists: implications for image-based model observers

We tested the agreement of radiologists' rankings of different reconstructions of breast computed tomography images based on their diagnostic (classification) performance and on their subjective image quality assessments. We used 102 pathology proven cases (62 malignant, 40 benign), and an iterative image reconstruction (IIR) algorithm to obtain 24 reconstructions per case with different image appearances. Using image feature analysis, we selected 3 IIRs and 1 clinical reconstruction and 50 lesions. The reconstructions produced a range of image quality from smooth/low-noise to sharp/high-noise, which had a range in classifier performance corresponding to AUCs of 0.62 to 0.96. Six experienced Mammography Quality Standards Act (MQSA) radiologists rated the likelihood of malignancy for each lesion. We conducted an additional reader study with the same radiologists and a subset of 30 lesions. Radiologists ranked each reconstruction according to their preference. There was disagreement among the six radiologists on which reconstruction produced images with the highest diagnostic content, but they preferred the midsharp/noise image appearance over the others. However, the reconstruction they preferred most did not match with their performance. Due to these disagreements, it may be difficult to develop a single image-based model observer that is representative of a population of radiologists for this particular imaging task.

Performance of a subsidised mammographic screening programme in Malaysia, a middle-income Asian country

BACKGROUND

The incidence of breast cancer in Asia is increasing because of urbanization and lifestyle changes. In the developing countries in Asia, women present at late stages, and mortality is high. Mammographic screening is the only evidence-based screening modality that reduces breast cancer mortality. To date, only opportunistic screening is offered in the majority of Asian countries because of the lack of justification and funding. Nevertheless, there have been few reports on the effectiveness of such programmes. In this study, we describe the cancer detection rate and challenges experienced in an opportunistic mammographic screening programme in Malaysia.

METHODS

From October 2011 to June 2015, 1,778 asymptomatic women, aged 40-74 years, underwent subsidised mammographic screening. All patients had a clinical breast examination before mammographic screening, and women with mammographic abnormalities were referred to a surgeon. The cancer detection rate and variables associated with a recommendation for adjunct ultrasonography were determined.

RESULTS

The mean age for screening was 50.8 years and seven cancers (0.39%) were detected. The detection rate was 0.64% in women aged 50 years and above, and 0.12% in women below 50 years old. Adjunct ultrasonography was recommended in 30.7% of women, and was significantly associated with age, menopausal status, mammographic density and radiologist's experience. The main reasons cited for recommendation of an adjunct ultrasound was dense breasts and mammographic abnormalities.

DISCUSSION

The cancer detection rate is similar to population-based screening mammography programmes in high-income Asian countries. Unlike population-based screening programmes in Caucasian populations where the adjunct ultrasonography rate is 2-4%, we report that 3 out of 10 women attending screening mammography were recommended for adjunct ultrasonography. This could be because Asian women attending screening are likely premenopausal and hence have denser breasts. Radiologists who reported more than 360 mammograms were more confident in reporting a mammogram as normal without adjunct ultrasonography compared to those who reported less than 180 mammograms.

CONCLUSION

Our subsidised opportunistic mammographic screening programme is able to provide equivalent cancer detection rates but the high recall for adjunct ultrasonography would make screening less cost-effective.

A deep learning approach for the analysis of masses in mammograms with minimal user intervention

We present an integrated methodology for detecting, segmenting and classifying breast masses from mammograms with minimal user intervention. This is a long standing problem due to low signal-to-noise ratio in the visualisation of breast masses, combined with their large variability in terms of shape, size, appearance and location. We break the problem down into three stages: mass detection, mass segmentation, and mass classification. For the detection, we propose a cascade of deep learning methods to select hypotheses that are refined based on Bayesian optimisation. For the segmentation, we propose the use of deep structured output learning that is subsequently refined by a level set method. Finally, for the classification, we propose the use of a deep learning classifier, which is pre-trained with a regression to hand-crafted feature values and fine-tuned based on the annotations of the breast mass classification dataset. We test our proposed system on the publicly available INbreast dataset and compare the results with the current state-of-the-art methodologies. This evaluation shows that our system detects 90% of masses at 1 false positive per image, has a segmentation accuracy of around 0.85 (Dice index) on the correctly detected masses, and overall classifies masses as malignant or benign with sensitivity (Se) of 0.98 and specificity (Sp) of 0.7.

Inter-reader Variability in the Use of BI-RADS Descriptors for Suspicious Findings on Diagnostic Mammography: A Multi-institution Study of 10 Academic Radiologists

RATIONALE AND OBJECTIVES

The study aimed to determine the inter-observer agreement among academic breast radiologists when using the Breast Imaging Reporting and Data System (BI-RADS) lesion descriptors for suspicious findings on diagnostic mammography.

MATERIALS AND METHODS

Ten experienced academic breast radiologists across five medical centers independently reviewed 250 de-identified diagnostic mammographic cases that were previously assessed as BI-RADS 4 or 5 with subsequent pathologic diagnosis by percutaneous or surgical biopsy. Each radiologist assessed the presence of the following suspicious mammographic findings: mass, asymmetry (one view), focal asymmetry (two views), architectural distortion, and calcifications. For any identified calcifications, the radiologist also described the morphology and distribution. Inter-observer agreement was determined with Fleiss kappa statistic. Agreement was also calculated by years of experience.

RESULTS

Of the 250 lesions, 156 (62%) were benign and 94 (38%) were malignant. Agreement among the 10 readers was strongest for recognizing the presence of calcifications (k = 0.82). There was substantial agreement among the readers for the identification of a mass (k = 0.67), whereas agreement was fair for the presence of a focal asymmetry (k = 0.21) or architectural distortion (k = 0.28). Agreement for asymmetries (one view) was slight (k = 0.09). Among the categories of calcification morphology and distribution, reader agreement was moderate (k = 0.51 and k = 0.60, respectively). Readers with more experience (10 or more years in clinical practice) did not demonstrate higher levels of agreement compared to those with less experience.

CONCLUSIONS

Strength of agreement varies widely for different types of mammographic findings, even among dedicated academic breast radiologists. More subtle findings such as asymmetries and architectural distortion demonstrated the weakest agreement. Studies that seek to evaluate the predictive value of certain mammographic features for malignancy should take into consideration the inherent interpretive variability for these findings.

Reducing Costs of Breast Examination: Ultrasound Performance and Inter-Observer Variability of Expert Radiologists Versus Residents

AIM

To compare efficiency levels between radiologist and radiology resident and any significant or clinically relevant differences in breast ultrasound diagnosis, thus reducing extra costs.

MATERIAL AND METHODS

100 patients attending for breast ultrasound were included. Each patient was examined by a radiologist, and subsequently by a resident of the radiology department. Both operators noted their findings and wrote a concluding report. Reports were compared for histological and biological analysis.

RESULTS

100 female patients with a mean age about 49 years were examined. The proportions of correct diagnoses of lesions individuated by radiologist and resident were 26.90 > 13.71% (p-value = 10.7), i.e. the radiologist was more accurate in comparison to resident in the individuation of breast lesions.

CONCLUSIONS

The radiologist was more accurate in comparison to radiology resident in the evaluation of breast pathology in ultrasonography diagnoses, and this could reduce cost and/or in-depth analysis.

Analysis of mammographic diagnostic errors in breast clinic

Mammography is the gold standard for detection of early breast cancer and it is still the only diagnostic tool which shows reduction of the mortality from that. Despite that, there is a high chance of false negatives that can lead to diagnostic errors resulting in delays of treatment and worsening of prognosis. The aim of this study is to analyze the rate of false negative in mammography and assess the source of diagnostic errors. Two radiologists have retrospectively evaluated 500 mammograms performed between January 2008 and December 2011 in Breast Imaging Clinic. 250 patients (Group A) had been operated for breast cancer and 250 patients (Group B) were healthy woman submitted to mammography according to the guideline for early detection of breast cancer. In Group A, 138 patients (55.2 %) were true missed cancer, 61 had minimal sign (24.4 %) and 53 were false negative (FN) (20.4 %). The source of errors amongst the FN were in 42 % of cases due to perception, in 15 % to interpretation, in 10 % to subtle/unusual lesion characteristics, in 9 % error for satisfaction of search, in 7 % to inherent limitations of mammography, in 4 % to poor technique and 13 % for inadequate clinical management. The diagnostic errors in breast clinic services are not negligible. The largest number of FN results from perception errors, misinterpretation and inadequate clinical management. These can be related to factors such as inattention, fatigue or lack of experience. To reduce it, it is necessary to have a dedicated multidisciplinary staff and adequate equipment and workloads.

Using automatically extracted information from mammography reports for decision-support

OBJECTIVE

To evaluate a system we developed that connects natural language processing (NLP) for information extraction from narrative text mammography reports with a Bayesian network for decision-support about breast cancer diagnosis. The ultimate goal of this system is to provide decision support as part of the workflow of producing the radiology report.

MATERIALS AND METHODS

We built a system that uses an NLP information extraction system (which extract BI-RADS descriptors and clinical information from mammography reports) to provide the necessary inputs to a Bayesian network (BN) decision support system (DSS) that estimates lesion malignancy from BI-RADS descriptors. We used this integrated system to predict diagnosis of breast cancer from radiology text reports and evaluated it with a reference standard of 300 mammography reports. We collected two different outputs from the DSS: (1) the probability of malignancy and (2) the BI-RADS final assessment category. Since NLP may produce imperfect inputs to the DSS, we compared the difference between using perfect ("reference standard") structured inputs to the DSS ("RS-DSS") vs NLP-derived inputs ("NLP-DSS") on the output of the DSS using the concordance correlation coefficient. We measured the classification accuracy of the BI-RADS final assessment category when using NLP-DSS, compared with the ground truth category established by the radiologist.

RESULTS

The NLP-DSS and RS-DSS had closely matched probabilities, with a mean paired difference of 0.004±0.025. The concordance correlation of these paired measures was 0.95. The accuracy of the NLP-DSS to predict the correct BI-RADS final assessment category was 97.58%.

CONCLUSION

The accuracy of the information extracted from mammography reports using the NLP system was sufficient to provide accurate DSS results. We believe our system could ultimately reduce the variation in practice in mammography related to assessment of malignant lesions and improve management decisions.

Radiologists' interpretive skills in screening vs. diagnostic mammography: are they related?

PURPOSE

This study aims to determine whether radiologists who perform well in screening also perform well in interpreting diagnostic mammography.

MATERIALS AND METHODS

We evaluated the accuracy of 468 radiologists interpreting 2,234,947 screening and 196,164 diagnostic mammograms. Adjusting for site, radiologist, and patient characteristics, we identified radiologists with performance in the highest tertile and compared to those with lower performance.

RESULTS

A moderate correlation was noted for radiologists' accuracy when interpreting screening versus their accuracy on diagnostic examinations: sensitivity (r_spearman=0.51, 95% CI: 0.22, 0.80; P=.0006) and specificity (r_spearman=0.40, 95% CI: 0.30, 0.49; P<.0001).

CONCLUSION

Different educational approaches to screening and diagnostic imaging should be considered.

Validation of a Medicare Claims-based Algorithm for Identifying Breast Cancers Detected at Screening Mammography

BACKGROUND

The breast cancer detection rate is a benchmark measure of screening mammography quality, but its computation requires linkage of mammography interpretive performance information with cancer incidence data. A Medicare claims-based measure of detected breast cancers could simplify measurement of this benchmark and facilitate mammography quality assessment and research.

OBJECTIVES

To validate a claims-based algorithm that can identify with high positive predictive value (PPV) incident breast cancers that were detected at screening mammography.

RESEARCH DESIGN

Development of a claims-derived algorithm using classification and regression tree analyses within a random half-sample of Medicare screening mammography claims followed by validation of the algorithm in the remaining half-sample using clinical data on mammography results and cancer incidence from the Breast Cancer Surveillance Consortium (BCSC).

SUBJECTS

Female fee-for-service Medicare enrollees aged 68 years and older who underwent screening mammography from 2001 to 2005 within BCSC registries in 4 states (CA, NC, NH, and VT), enabling linkage of claims and BCSC mammography data (N=233,044 mammograms obtained by 104,997 women).

MEASURES

Sensitivity, specificity, and PPV of algorithmic identification of incident breast cancers that were detected by radiologists relative to a reference standard based on BCSC mammography and cancer incidence data.

RESULTS

An algorithm based on subsequent codes for breast cancer diagnoses and treatments and follow-up mammography identified incident screen-detected breast cancers with 92.9% sensitivity [95% confidence interval (CI), 91.0%-94.8%], 99.9% specificity (95% CI, 99.9%-99.9%), and a PPV of 88.0% (95% CI, 85.7%-90.4%).

CONCLUSIONS

A simple claims-based algorithm can accurately identify incident breast cancers detected at screening mammography among Medicare enrollees. The algorithm may enable mammography quality assessment using Medicare claims alone.

Risk of breast cancer after false-positive results in mammographic screening

Women with false-positive results are commonly referred back to routine screening. Questions remain regarding their long-term outcome of breast cancer. We assessed the risk of screen-detected breast cancer in women with false-positive results. We conducted a joint analysis using individual level data from the population-based screening programs in Copenhagen and Funen in Denmark, Norway, and Spain. Overall, 150,383 screened women from Denmark (1991-2008), 612,138 from Norway (1996-2010), and 1,172,572 from Spain (1990-2006) were included. Poisson regression was used to estimate the relative risk (RR) of screen-detected cancer for women with false-positive versus negative results. We analyzed information from 1,935,093 women 50-69 years who underwent 6,094,515 screening exams. During an average 5.8 years of follow-up, 230,609 (11.9%) women received a false-positive result and 27,849 (1.4%) were diagnosed with screen-detected cancer. The adjusted RR of screen-detected cancer after a false-positive result was 2.01 (95% CI: 1.93-2.09). Women who tested false-positive at first screen had a RR of 1.86 (95% CI: 1.77-1.96), whereas those who tested false-positive at third screening had a RR of 2.42 (95% CI: 2.21-2.64). The RR of breast cancer at the screening test after the false-positive result was 3.95 (95% CI: 3.71-4.21), whereas it decreased to 1.25 (95% CI: 1.17-1.34) three or more screens after the false-positive result. Women with false-positive results had a twofold risk of screen-detected breast cancer compared to women with negative tests. The risk remained significantly higher three or more screens after the false-positive result. The increased risk should be considered when discussing stratified screening strategies.

A measure of association for ordered categorical data in population-based studies

Ordinal classification scales are commonly used to define a patient's disease status in screening and diagnostic tests such as mammography. Challenges arise in agreement studies when evaluating the association between many raters' classifications of patients' disease or health status when an ordered categorical scale is used. In this paper, we describe a population-based approach and chance-corrected measure of association to evaluate the strength of relationship between multiple raters' ordinal classifications where any number of raters can be accommodated. In contrast to Shrout and Fleiss' intraclass correlation coefficient, the proposed measure of association is invariant with respect to changes in disease prevalence. We demonstrate how unique characteristics of individual raters can be explored using random effects. Simulation studies are conducted to demonstrate the properties of the proposed method under varying assumptions. The methods are applied to two large-scale agreement studies of breast cancer screening and prostate cancer severity.

Digital Breast Tomosynthesis: State of the Art

This topical review on digital breast tomosynthesis (DBT) is provided with the intent of describing the state of the art in terms of technology, results from recent clinical studies, advanced applications, and ongoing efforts to develop multimodality imaging systems that include DBT. Particular emphasis is placed on clinical studies. The observations of increase in cancer detection rates, particularly for invasive cancers, and the reduction in false-positive rates with DBT in prospective trials indicate its benefit for breast cancer screening. Retrospective multireader multicase studies show either noninferiority or superiority of DBT compared with mammography. Methods to curtail radiation dose are of importance. (©) RSNA, 2015.

Imaging-based screening: maximizing benefits and minimizing harms

Advanced imaging technologies play a central role in screening asymptomatic patients. However, the balance between imaging-based screening's potential benefits versus risks is sometimes unclear. Radiologists will have to address ongoing concerns, including high false-positive rates, incidental findings outside the organ of interest, overdiagnosis, and potential risks from radiation exposure. In this article, we provide a brief overview of these recurring controversies and suggest the following as areas that radiologists should focus on in order to tip the balance toward more benefits and less harms for patients undergoing imaging-based screening: interpretive variability, abnormal finding thresholds, and personalized, risk-based screening.