Understanding diagnostic variability in breast pathology: lessons learned from an expert consensus review panel

Region of interest identification and diagnostic agreement in breast pathology

A pathologist's accurate interpretation relies on identifying relevant histopathological features. Little is known about the precise relationship between feature identification and diagnostic decision making. We hypothesized that greater overlap between a pathologist's selected diagnostic region of interest (ROI) and a consensus derived ROI is associated with higher diagnostic accuracy. We developed breast biopsy test cases that included atypical ductal hyperplasia (n=80); ductal carcinoma in situ (n=78); and invasive breast cancer (n=22). Benign cases were excluded due to the absence of specific abnormalities. Three experienced breast pathologists conducted an independent review of the 180 digital whole slide images, established a reference consensus diagnosis and marked one or more diagnostic ROIs for each case. Forty-four participating pathologists independently diagnosed and marked ROIs on the images. Participant diagnoses and ROI were compared with consensus reference diagnoses and ROI. Regression models tested whether percent overlap between participant ROI and consensus reference ROI predicted diagnostic accuracy. Each of the 44 participants interpreted 39-50 cases for a total of 1972 individual diagnoses. Percent ROI overlap with the expert reference ROI was higher in pathologists who self-reported academic affiliation (69 vs 65%, P=0.002). Percent overlap between participants' ROI and consensus reference ROI was then classified into ordinal categories: 0, 1-33, 34-65, 66-99 and 100% overlap. For each incremental change in the ordinal percent ROI overlap, diagnostic agreement increased by 60% (OR 1.6, 95% CI (1.5-1.7), P<0.001) and the association remained significant even after adjustment for other covariates. The magnitude of the association between ROI overlap and diagnostic agreement increased with increasing diagnostic severity. The findings indicate that pathologists are more likely to converge with an expert reference diagnosis when they identify an overlapping diagnostic image region, suggesting that future computer-aided detection systems that highlight potential diagnostic regions could be a helpful tool to improve accuracy and education.

Pupil diameter changes reflect difficulty and diagnostic accuracy during medical image interpretation

Background

No automated methods exist to objectively monitor and evaluate the diagnostic process while physicians review computerized medical images. The present study tested whether using eye tracking to monitor tonic and phasic pupil dynamics may prove valuable in tracking interpretive difficulty and predicting diagnostic accuracy.

Methods

Pathologists interpreted digitized breast biopsies varying in diagnosis and rated difficulty, while pupil diameter was monitored. Tonic diameter was recorded during the entire duration of interpretation, and phasic diameter was examined when the eyes fixated on a pre-determined diagnostic region during inspection.

Results

Tonic pupil diameter was higher with increasing rated difficulty levels of cases. Phasic diameter was interactively influenced by case difficulty and the eventual agreement with consensus diagnosis. More difficult cases produced increases in pupil diameter, but only when the pathologists’ diagnoses were ultimately correct. All results were robust after adjusting for the potential impact of screen brightness on pupil diameter.

Conclusions

Results contribute new understandings of the diagnostic process, theoretical positions regarding locus coeruleus-norepinephrine system function, and suggest novel approaches to monitoring, evaluating, and guiding medical image interpretation.

Evaluation of 12 strategies for obtaining second opinions to improve interpretation of breast histopathology: simulation study

OBJECTIVE

 To evaluate the potential effect of second opinions on improving the accuracy of diagnostic interpretation of breast histopathology.

DESIGN

 Simulation study.

SETTING

 12 different strategies for acquiring independent second opinions.

PARTICIPANTS

 Interpretations of 240 breast biopsy specimens by 115 pathologists, one slide for each case, compared with reference diagnoses derived by expert consensus.

MAIN OUTCOME MEASURES

 Misclassification rates for individual pathologists and for 12 simulated strategies for second opinions. Simulations compared accuracy of diagnoses from single pathologists with that of diagnoses based on pairing interpretations from first and second independent pathologists, where resolution of disagreements was by an independent third pathologist. 12 strategies were evaluated in which acquisition of second opinions depended on initial diagnoses, assessment of case difficulty or borderline characteristics, pathologists' clinical volumes, or whether a second opinion was required by policy or desired by the pathologists. The 240 cases included benign without atypia (10% non-proliferative, 20% proliferative without atypia), atypia (30%), ductal carcinoma in situ (DCIS, 30%), and invasive cancer (10%). Overall misclassification rates and agreement statistics depended on the composition of the test set, which included a higher prevalence of difficult cases than in typical practice.

RESULTS

 Misclassification rates significantly decreased (P<0.001) with all second opinion strategies except for the strategy limiting second opinions only to cases of invasive cancer. The overall misclassification rate decreased from 24.7% to 18.1% when all cases received second opinions (P<0.001). Obtaining both first and second opinions from pathologists with a high volume (≥10 breast biopsy specimens weekly) resulted in the lowest misclassification rate in this test set (14.3%, 95% confidence interval 10.9% to 18.0%). Obtaining second opinions only for cases with initial interpretations of atypia, DCIS, or invasive cancer decreased the over-interpretation of benign cases without atypia from 12.9% to 6.0%. Atypia cases had the highest misclassification rate after single interpretation (52.2%), remaining at more than 34% in all second opinion scenarios.

CONCLUSION

 Second opinions can statistically significantly improve diagnostic agreement for pathologists' interpretations of breast biopsy specimens; however, variability in diagnosis will not be completely eliminated, especially for breast specimens with atypia.

Variability in Pathologists' Interpretations of Individual Breast Biopsy Slides: A Population Perspective

BACKGROUND

The effect of physician diagnostic variability on accuracy at a population level depends on the prevalence of diagnoses.

OBJECTIVE

To estimate how diagnostic variability affects accuracy from the perspective of a U.S. woman aged 50 to 59 years having a breast biopsy.

DESIGN

Applied probability using Bayes' theorem.

SETTING

B-Path (Breast Pathology) Study comparing pathologists' interpretations of a single biopsy slide versus a reference consensus interpretation from 3 experts.

PARTICIPANTS

115 practicing pathologists (6900 total interpretations from 240 distinct cases).

MEASUREMENTS

A single representative slide from each of the 240 cases was used to estimate the proportion of biopsies with a diagnosis that would be verified if the same slide were interpreted by a reference group of 3 expert pathologists. Probabilities of confirmation (predictive values) were estimated using B-Path Study results and prevalence of biopsy diagnoses for women aged 50 to 59 years in the Breast Cancer Surveillance Consortium.

RESULTS

Overall, if 1 representative slide were used per case, 92.3% (95% CI, 91.4% to 93.1%) of breast biopsy diagnoses would be verified by reference consensus diagnoses, with 4.6% (CI, 3.9% to 5.3%) overinterpreted and 3.2% (CI, 2.7% to 3.6%) underinterpreted. Verification of invasive breast cancer and benign without atypia diagnoses is highly probable; estimated predictive values were 97.7% (CI, 96.5% to 98.7%) and 97.1% (CI, 96.7% to 97.4%), respectively. Verification is less probable for atypia (53.6% overinterpreted and 8.6% underinterpreted) and ductal carcinoma in situ (DCIS) (18.5% overinterpreted and 11.8% underinterpreted).

LIMITATIONS

Estimates are based on a testing situation with 1 slide used per case and without access to second opinions. Population-adjusted estimates may differ for women from other age groups, unscreened women, or women in different practice settings.

CONCLUSION

This analysis, based on interpretation of a single breast biopsy slide per case, predicts a low likelihood that a diagnosis of atypia or DCIS would be verified by a reference consensus diagnosis. This diagnostic grey zone should be considered in clinical management decisions in patients with these diagnoses.

PRIMARY FUNDING SOURCE

National Cancer Institute.

Reliability of diagnosis from Mohs slides: interpersonal and intrapersonal agreement on basal cell carcinoma presence and histological subtype

BACKGROUND

The success of Mohs micrographic surgery (MMS) depends partly on the correct diagnosis of slides.

OBJECTIVES

To determine reliability of diagnosis from Mohs slides.

METHODS

This was a prospective study evaluating the reliability of diagnosis from Mohs slides of basal cell carcinoma (BCC) presence, BCC location on the slide and BCC subtype among six raters who independently assessed 50 Mohs slides twice with a 2-month interval. Slides were randomly selected whereby difficult-to-diagnose slides were oversampled. For each slide, a reference diagnosis was established by an expert panel. Cohen's kappa (κ) was calculated to determine levels of agreement interpersonally (rater vs. reference diagnosis) and intrapersonally (rater at T1 vs. T2). Multivariable logistic regression was used to determine independent risk factors for slides with interpersonal discordant diagnosis. The variables studied were BCC presence, whether a slide was scored as easy or difficult to diagnose, review duration of the 50 slides, profession and years of experience in diagnosis from Mohs slides.

RESULTS

Interpersonal and intrapersonal agreement were substantial on BCC presence (κ = 0·66 and 0·68) and moderate on BCC subtype (κ = 0·45 and 0·55). Slides that were scored as difficult to diagnose were an independent risk factor for interpersonal discordant diagnosis on BCC presence (odds ratio 3·54, 95% confidence interval 1·81-6·84).

CONCLUSIONS

Reliability of diagnosis from Mohs slides was substantial on BCC presence and moderate on BCC subtype. For slides that are scored difficult to diagnose, a second opinion is recommended to prevent misinterpretation and thereby recurrence of skin cancer.

The Influence of Disease Severity of Preceding Clinical Cases on Pathologists' Medical Decision Making

BACKGROUND

Medical decision making may be influenced by contextual factors. We evaluated whether pathologists are influenced by disease severity of recently observed cases.

METHODS

Pathologists independently interpreted 60 breast biopsy specimens (one slide per case; 240 total cases in the study) in a prospective randomized observational study. Pathologists interpreted the same cases in 2 phases, separated by a washout period of >6 months. Participants were not informed that the cases were identical in each phase, and the sequence was reordered randomly for each pathologist and between phases. A consensus reference diagnosis was established for each case by 3 experienced breast pathologists. Ordered logit models examined the effect the pathologists' diagnoses on the preceding case or the 5 preceding cases had on their diagnosis for the subsequent index case.

RESULTS

Among 152 pathologists, 49 provided interpretive data in both phases I and II, 66 from only phase I, and 37 from phase II only. In phase I, pathologists were more likely to indicate a more severe diagnosis than the reference diagnosis when the preceding case was diagnosed as ductal carcinoma in situ (DCIS) or invasive cancer (proportional odds ratio [POR], 1.28; 95% confidence interval [CI], 1.15-1.42). Results were similar when considering the preceding 5 cases and for the pathologists in phase II who interpreted the same cases in a different order compared with phase I (POR, 1.17; 95% CI, 1.05-1.31).

CONCLUSION

Physicians appear to be influenced by the severity of previously interpreted test cases. Understanding types and sources of diagnostic bias may lead to improved assessment of accuracy and better patient care.

Imaging-based screening: maximizing benefits and minimizing harms

Advanced imaging technologies play a central role in screening asymptomatic patients. However, the balance between imaging-based screening's potential benefits versus risks is sometimes unclear. Radiologists will have to address ongoing concerns, including high false-positive rates, incidental findings outside the organ of interest, overdiagnosis, and potential risks from radiation exposure. In this article, we provide a brief overview of these recurring controversies and suggest the following as areas that radiologists should focus on in order to tip the balance toward more benefits and less harms for patients undergoing imaging-based screening: interpretive variability, abnormal finding thresholds, and personalized, risk-based screening.

Medical malpractice concerns and defensive medicine: a nationwide survey of breast pathologists

OBJECTIVES

"Assurance behaviors" in medical practice involve providing additional services of marginal or no medical value to avoid adverse outcomes, deter patients from filing malpractice claims, or ensure that legal standards of care were met. The extent to which concerns about medical malpractice influence assurance behaviors of pathologists interpreting breast specimens is unknown.

METHODS

Breast pathologists (n = 252) enrolled in a nationwide study completed an online survey of attitudes regarding malpractice and perceived alterations in interpretive behavior due to concerns of malpractice. Associations between pathologist characteristics and the impact of malpractice concerns on personal and colleagues' assurance behaviors were determined by χ(2) and logistic regression analysis.

RESULTS

Most participants reported using one or more assurance behaviors due to concerns about medical malpractice for both their personal (88%) and colleagues' (88%) practices, including ordering additional stains, recommending additional surgical sampling, obtaining second reviews, or choosing the more severe diagnosis for borderline cases. Nervousness over breast pathology was positively associated with assurance behavior and remained statistically significant in a multivariable logistic regression model (odds ratio, 2.5; 95% confidence interval, 1.0-6.1; P = .043).

CONCLUSIONS

Practicing US breast pathologists report exercising defensive medicine by using assurance behaviors due to malpractice concerns.

Validation of natural language processing to extract breast cancer pathology procedures and results

Background:

Pathology reports typically require manual review to abstract research data. We developed a natural language processing (NLP) system to automatically interpret free-text breast pathology reports with limited assistance from manual abstraction.

Methods:

We used an iterative approach of machine learning algorithms and constructed groups of related findings to identify breast-related procedures and results from free-text pathology reports. We evaluated the NLP system using an all-or-nothing approach to determine which reports could be processed entirely using NLP and which reports needed manual review beyond NLP. We divided 3234 reports for development (2910, 90%), and evaluation (324, 10%) purposes using manually reviewed pathology data as our gold standard.

Results:

NLP correctly coded 12.7% of the evaluation set, flagged 49.1% of reports for manual review, incorrectly coded 30.8%, and correctly omitted 7.4% from the evaluation set due to irrelevancy (i.e. not breast-related). Common procedures and results were identified correctly (e.g. invasive ductal with 95.5% precision and 94.0% sensitivity), but entire reports were flagged for manual review because of rare findings and substantial variation in pathology report text.

Conclusions:

The NLP system we developed did not perform sufficiently for abstracting entire breast pathology reports. The all-or-nothing approach resulted in too broad of a scope of work and limited our flexibility to identify breast pathology procedures and results. Our NLP system was also limited by the lack of the gold standard data on rare findings and wide variation in pathology text. Focusing on individual, common elements and improving pathology text report standardization may improve performance.

Extracting knowledge from chemical imaging data using computational algorithms for digital cancer diagnosis

Fourier transform infrared (FTIR) spectroscopic imaging is an emerging microscopy modality for clinical histopathologic diagnoses as well as for biomedical research. Spectral data recorded in this modality are indicative of the underlying, spatially resolved biochemical composition but need computerized algorithms to digitally recognize and transform this information to a diagnostic tool to identify cancer or other physiologic conditions. Statistical pattern recognition forms the backbone of these recognition protocols and can be used for highly accurate results. Aided by biochemical correlations with normal and diseased states and the power of modern computer-aided pattern recognition, this approach is capable of combating many standing questions of traditional histology-based diagnosis models. For example, a simple diagnostic test can be developed to determine cell types in tissue. As a more advanced application, IR spectral data can be integrated with patient information to predict risk of cancer, providing a potential road to precision medicine and personalized care in cancer treatment. The IR imaging approach can be implemented to complement conventional diagnoses, as the samples remain unperturbed and are not destroyed. Despite high potential and utility of this approach, clinical implementation has not yet been achieved due to practical hurdles like speed of data acquisition and lack of optimized computational procedures for extracting clinically actionable information rapidly. The latter problem has been addressed by developing highly efficient ways to process IR imaging data but remains one that has considerable scope for progress. Here, we summarize the major issues and provide practical considerations in implementing a modified Bayesian classification protocol for digital molecular pathology. We hope to familiarize readers with analysis methods in IR imaging data and enable researchers to develop methods that can lead to the use of this promising technique for digital diagnosis of cancer.

Diagnostic concordance among pathologists interpreting breast biopsy specimens

IMPORTANCE

A breast pathology diagnosis provides the basis for clinical treatment and management decisions; however, its accuracy is inadequately understood.

OBJECTIVES

To quantify the magnitude of diagnostic disagreement among pathologists compared with a consensus panel reference diagnosis and to evaluate associated patient and pathologist characteristics.

DESIGN, SETTING, AND PARTICIPANTS

Study of pathologists who interpret breast biopsies in clinical practices in 8 US states.

EXPOSURES

Participants independently interpreted slides between November 2011 and May 2014 from test sets of 60 breast biopsies (240 total cases, 1 slide per case), including 23 cases of invasive breast cancer, 73 ductal carcinoma in situ (DCIS), 72 with atypical hyperplasia (atypia), and 72 benign cases without atypia. Participants were blinded to the interpretations of other study pathologists and consensus panel members. Among the 3 consensus panel members, unanimous agreement of their independent diagnoses was 75%, and concordance with the consensus-derived reference diagnoses was 90.3%.

MAIN OUTCOMES AND MEASURES

The proportions of diagnoses overinterpreted and underinterpreted relative to the consensus-derived reference diagnoses were assessed.

RESULTS

Sixty-five percent of invited, responding pathologists were eligible and consented to participate. Of these, 91% (N = 115) completed the study, providing 6900 individual case diagnoses. Compared with the consensus-derived reference diagnosis, the overall concordance rate of diagnostic interpretations of participating pathologists was 75.3% (95% CI, 73.4%-77.0%; 5194 of 6900 interpretations). Among invasive carcinoma cases (663 interpretations), 96% (95% CI, 94%-97%) were concordant, and 4% (95% CI, 3%-6%) were underinterpreted; among DCIS cases (2097 interpretations), 84% (95% CI, 82%-86%) were concordant, 3% (95% CI, 2%-4%) were overinterpreted, and 13% (95% CI, 12%-15%) were underinterpreted; among atypia cases (2070 interpretations), 48% (95% CI, 44%-52%) were concordant, 17% (95% CI, 15%-21%) were overinterpreted, and 35% (95% CI, 31%-39%) were underinterpreted; and among benign cases without atypia (2070 interpretations), 87% (95% CI, 85%-89%) were concordant and 13% (95% CI, 11%-15%) were overinterpreted. Disagreement with the reference diagnosis was statistically significantly higher among biopsies from women with higher (n = 122) vs lower (n = 118) breast density on prior mammograms (overall concordance rate, 73% [95% CI, 71%-75%] for higher vs 77% [95% CI, 75%-80%] for lower, P < .001), and among pathologists who interpreted lower weekly case volumes (P < .001) or worked in smaller practices (P = .034) or nonacademic settings (P = .007).

CONCLUSIONS AND RELEVANCE

In this study of pathologists, in which diagnostic interpretation was based on a single breast biopsy slide, overall agreement between the individual pathologists' interpretations and the expert consensus-derived reference diagnoses was 75.3%, with the highest level of concordance for invasive carcinoma and lower levels of concordance for DCIS and atypia. Further research is needed to understand the relationship of these findings with patient management.

What is atypia? Use, misuse and overuse of the term atypia in diagnostic cytopathology

The term "atypical" was introduced by the founder of modern cytodiagnosis, Dr. George N. Papanicolaou, to convey a very low suspicion of (pre)malignancy. Despite controversies concerning its ambiguous and imprecise definition and its uncertain optimal use, the term "atypia" has continued to be used in cytopathology, and has recently been increasingly used in standardized nongynecologic cytopathology diagnostic reporting terminologies. Its increasing use suggests that "atypia" continues to be a useful category to fill the gap between what we can recognize as entirely normal (including reactive changes) and what we can recognize as clearly abnormal (premalignant or malignant). However, this diagnosis should be used parsimoniously, since the potential overuse of "atypia" diagnoses can lead to the erosion of clinicians' confidence in cytopathology, their misunderstanding of the cytopathology report, and to an increase the clinicians' diagnostic uncertainty, with negative consequences on patients' satisfaction and wellbeing, and on health care costs. A clinically meaningful, standardized cytodiagnostic category of "atypia" requires a narrow definition, quantitative criteria, agreed-upon reference images, a clear clinical meaning (likelihood of underlying malignancy or premalignancy) and, ideally, well-defined management options. The successful implementation of such a standardized "atypia" diagnostic category requires continuous education of cytology professionals and quality assurance efforts to monitor its use. The interobserver variability and potential excessive use of the diagnosis of "atypia" may be reduced by considering and addressing the major factors involved in its variable use, namely the quality of the sample, the definition of "atypia", the education/training of the cytologist/pathologist, and cytologist/pathologist-related "supracytologic" factors.

Evaluation environment for digital and analog pathology: a platform for validation studies

We present a platform for designing and executing studies that compare pathologists interpreting histopathology of whole slide images (WSIs) on a computer display to pathologists interpreting glass slides on an optical microscope. eeDAP is an evaluation environment for digital and analog pathology. The key element in eeDAP is the registration of the WSI to the glass slide. Registration is accomplished through computer control of the microscope stage and a camera mounted on the microscope that acquires real-time images of the microscope field of view (FOV). Registration allows for the evaluation of the same regions of interest (ROIs) in both domains. This can reduce or eliminate disagreements that arise from pathologists interpreting different areas and focuses on the comparison of image quality. We reduced the pathologist interpretation area from an entire glass slide (10 to [Formula: see text]) to small ROIs ([Formula: see text]). We also made possible the evaluation of individual cells. We summarize eeDAP's software and hardware and provide calculations and corresponding images of the microscope FOV and the ROIs extracted from the WSIs. The eeDAP software can be downloaded from the Google code website (project: eeDAP) as a MATLAB source or as a precompiled stand-alone license-free application.

Development of an electronic breast pathology database in a community health system

Background:

Health care systems rely on electronic patient data, yet access to breast tissue pathology results continues to depend on interpreting dictated free-text reports.

Objective:

The objective was to develop a method to electronically search and categorize pathologic diagnoses of patients’ breast tissue specimens from dictated free-text pathology reports in a large health system for multiple users including clinicians.

Design:

A database integrating existing patient-level administrative and clinical information for breast cancer screening and diagnostic services and a web-based application for comprehensive searching of pathology reports were developed by a health system team led by pathologists. The Breast Pathology Assessment Tool and Hierarchy for Diagnosis (BPATH-Dx) provided search terms and guided electronic transcription of diagnoses from text fields on breast pathology clinical reports to standardized categories.

Approach:

Breast pathology encounters in the pathology database were matched with administrative data for 7332 women with breast tissue specimens obtained from an initial procedure in the health system from January 1, 2008 to December 31, 2011. Sequential queries of the pathology text based on BPATH-Dx categorized biopsies according to their worst pathological diagnosis, as is standard practice. Diagnoses ranged from invasive breast cancer (23.3%), carcinoma in situ (7.8%), atypical lesions (6.39%), proliferative lesions without atypia (27.9%), and nonproliferative lesions (34.7%), and were further classified into subcategories. A random sample of 5% of reports that were manually reviewed indicated 97.5% agreement.

Conclusions:

Sequential queries of free-text pathology reports guided by a standardized assessment tool in conjunction with a web-based search application provide an efficient and reproducible approach to accessing nonmalignant breast pathology diagnoses. This method advances the use of pathology data and electronic health records to improve health care quality, patient care, outcomes, and research.