Pathologists should probably forget about kappa. Percent agreement, diagnostic specificity and related metrics provide more clinically applicable measures of interobserver variability

Seeing the random forest through the decision trees. Supporting learning health systems from histopathology with machine learning models: Challenges and opportunities

This paper discusses some overlooked challenges faced when working with machine learning models for histopathology and presents a novel opportunity to support "Learning Health Systems" with them. Initially, the authors elaborate on these challenges after separating them according to their mitigation strategies: those that need innovative approaches, time, or future technological capabilities and those that require a conceptual reappraisal from a critical perspective. Then, a novel opportunity to support "Learning Health Systems" by integrating hidden information extracted by ML models from digitalized histopathology slides with other healthcare big data is presented.

Inter-Rater and Intra-Rater Agreement in Scoring Severity of Rodent Cardiomyopathy and Relation to Artificial Intelligence-Based Scoring

We previously developed a computer-assisted image analysis algorithm to detect and quantify the microscopic features of rodent progressive cardiomyopathy (PCM) in rat heart histologic sections and validated the results with a panel of five veterinary toxicologic pathologists using a multinomial logistic model. In this study, we assessed both the inter-rater and intra-rater agreement of the pathologists and compared pathologists' ratings to the artificial intelligence (AI)-predicted scores. Pathologists and the AI algorithm were presented with 500 slides of rodent heart. They quantified the amount of cardiomyopathy in each slide. A total of 200 of these slides were novel to this study, whereas 100 slides were intentionally selected for repetition from the previous study. After a washout period of more than six months, the repeated slides were examined to assess intra-rater agreement among pathologists. We found the intra-rater agreement to be substantial, with weighted Cohen's kappa values ranging from k = 0.64 to 0.80. Intra-rater variability is not a concern for the deterministic AI. The inter-rater agreement across pathologists was moderate (Cohen's kappa k = 0.56). These results demonstrate the utility of AI algorithms as a tool for pathologists to increase sensitivity and specificity for the histopathologic assessment of the heart in toxicology studies.

A machine learning approach for quantifying age-related histological changes in the mouse kidney

The ability to quantify aging-related changes in histological samples is important, as it allows for evaluation of interventions intended to effect health span. We used a machine learning architecture that can be trained to detect and quantify these changes in the mouse kidney. Using additional held out data, we show validation of our model, correlation with scores given by pathologists using the Geropathology Research Network aging grading scheme, and its application in providing reproducible and quantifiable age scores for histological samples. Aging quantification also provides the insights into possible changes in image appearance that are independent of specific geropathology-specified lesions. Furthermore, we provide trained classifiers for H&E-stained slides, as well as tutorials on how to use these and how to create additional classifiers for other histological stains and tissues using our architecture. This architecture and combined resources allow for the high throughput quantification of mouse aging studies in general and specifically applicable to kidney tissues.

Update on the reliability of gait analysis interpretation in cerebral palsy: Inter-institution agreement

BACKGROUND

Studies have shown good reliability for gait analysis interpretation among surgeons from the same institution. However, reliability among surgeons from different institutions remains to be determined.

RESEARCH QUESTION

Is gait analysis interpretation by surgeons from different institutions as reliable as it is for surgeons from the same institution?

METHODS

Gait analysis data for 67 patients with cerebral palsy (CP) were reviewed prospectively by two orthopedic surgeons from different institutions in the same state, each with > 10 years' experience interpreting gait analysis data. The surgeons identified gait problems and made treatment recommendations for each patient using a rating form. Percent agreement between raters was calculated for each problem and treatment, and compared to expected agreement based on chance using Cohen's kappa.

RESULTS

For problem identification, the greatest agreement was seen for equinus (85% agreement), calcaneus (88%), in-toeing (89%), and out-toeing (90%). Agreement for the remaining problems ranged between 66-78%. Percent agreement was significantly higher than expected due to chance for all issues (p ≤ 0.01) with modest kappa values ranging from 0.12 to 0.51. Agreement between surgeons for treatment recommendations was highest for triceps surae lengthening (89% agreement), tibial derotation osteotomy (90%), and foot osteotomy (87%). Agreement for the remaining treatments ranged between 72-78%. Percent agreement for all treatments was significantly higher than the expected values (p ≤ 0.002) with modest kappa values ranging from 0.22 to 0.52.

SIGNIFICANCE

Previous research established that computerized gait analysis data interpretation is reliable for surgeons within a single institution. The current study demonstrates that gait analysis interpretation can also be reliable among surgeons from different institutions. Future research should examine reliability among physicians from more institutions to confirm these results.

Tissue Contamination Challenges the Credibility of Machine Learning Models in Real World Digital Pathology

Machine learning (ML) models are poised to transform surgical pathology practice. The most successful use attention mechanisms to examine whole slides, identify which areas of tissue are diagnostic, and use them to guide diagnosis. Tissue contaminants, such as floaters, represent unexpected tissue. Although human pathologists are extensively trained to consider and detect tissue contaminants, we examined their impact on ML models. We trained 4 whole-slide models. Three operate in placenta for the following functions: (1) detection of decidual arteriopathy, (2) estimation of gestational age, and (3) classification of macroscopic placental lesions. We also developed a model to detect prostate cancer in needle biopsies. We designed experiments wherein patches of contaminant tissue are randomly sampled from known slides and digitally added to patient slides and measured model performance. We measured the proportion of attention given to contaminants and examined the impact of contaminants in the t-distributed stochastic neighbor embedding feature space. Every model showed performance degradation in response to one or more tissue contaminants. Decidual arteriopathy detection--balanced accuracy decreased from 0.74 to 0.69 ± 0.01 with addition of 1 patch of prostate tissue for every 100 patches of placenta (1% contaminant). Bladder, added at 10% contaminant, raised the mean absolute error in estimating gestational age from 1.626 weeks to 2.371 ± 0.003 weeks. Blood, incorporated into placental sections, induced false-negative diagnoses of intervillous thrombi. Addition of bladder to prostate cancer needle biopsies induced false positives, a selection of high-attention patches, representing 0.033 mm2, and resulted in a 97% false-positive rate when added to needle biopsies. Contaminant patches received attention at or above the rate of the average patch of patient tissue. Tissue contaminants induce errors in modern ML models. The high level of attention given to contaminants indicates a failure to encode biological phenomena. Practitioners should move to quantify and ameliorate this problem.

A machine learning approach for quantifying age-related histological changes in the mouse kidney

The ability to quantify aging-related changes in histological samples is important, as it allows for evaluation of interventions intended to effect health span. We used a machine learning architecture that can be trained to detect and quantify these changes in the mouse kidney. Using additional held out data, we show validation of our model, correlation with scores given by pathologists using the Geropathology Research Network aging grading scheme, and its application in providing reproducible and quantifiable age scores for histological samples. Aging quantification also provides the insights into possible changes in image appearance that are independent of specific geropathology-specified lesions. Furthermore, we provide trained classifiers for H&E-stained slides, as well as tutorials on how to use these and how to create additional classifiers for other histological stains and tissues using our architecture.This architecture and combined resources allow for the high throughput quantification of mouse aging studies in general and specifically applicable to kidney tissues.

Tissue contamination challenges the credibility of machine learning models in real world digital pathology

Machine learning (ML) models are poised to transform surgical pathology practice. The most successful use attention mechanisms to examine whole slides, identify which areas of tissue are diagnostic, and use them to guide diagnosis. Tissue contaminants, such as floaters, represent unexpected tissue. While human pathologists are extensively trained to consider and detect tissue contaminants, we examined their impact on ML models. We trained 4 whole slide models. Three operate in placenta for 1) detection of decidual arteriopathy (DA), 2) estimation of gestational age (GA), and 3) classification of macroscopic placental lesions. We also developed a model to detect prostate cancer in needle biopsies. We designed experiments wherein patches of contaminant tissue are randomly sampled from known slides and digitally added to patient slides and measured model performance. We measured the proportion of attention given to contaminants and examined the impact of contaminants in T-distributed Stochastic Neighbor Embedding (tSNE) feature space. Every model showed performance degradation in response to one or more tissue contaminants. DA detection balanced accuracy decreased from 0.74 to 0.69 +/- 0.01 with addition of 1 patch of prostate tissue for every 100 patches of placenta (1% contaminant). Bladder, added at 10% contaminant raised the mean absolute error in estimating gestation age from 1.626 weeks to 2.371 +/ 0.003 weeks. Blood, incorporated into placental sections, induced false negative diagnoses of intervillous thrombi. Addition of bladder to prostate cancer needle biopsies induced false positives, a selection of high-attention patches, representing 0.033mm2, resulted in a 97% false positive rate when added to needle biopsies. Contaminant patches received attention at or above the rate of the average patch of patient tissue. Tissue contaminants induce errors in modern ML models. The high level of attention given to contaminants indicates a failure to encode biological phenomena. Practitioners should move to quantify and ameliorate this problem.

Companion diagnostic requirements for spatial biology using multiplex immunofluorescence and multispectral imaging

Immunohistochemistry has long been held as the gold standard for understanding the expression patterns of therapeutically relevant proteins to identify prognostic and predictive biomarkers. Patient selection for targeted therapy in oncology has successfully relied upon standard microscopy-based methodologies, such as single-marker brightfield chromogenic immunohistochemistry. As promising as these results are, the analysis of one protein, with few exceptions, no longer provides enough information to draw effective conclusions about the probability of treatment response. More multifaceted scientific queries have driven the development of high-throughput and high-order technologies to interrogate biomarker expression patterns and spatial interactions between cell phenotypes in the tumor microenvironment. Such multi-parameter data analysis has been historically reserved for technologies that lack the spatial context that is provided by immunohistochemistry. Over the past decade, technical developments in multiplex fluorescence immunohistochemistry and discoveries made with improving image data analysis platforms have highlighted the importance of spatial relationships between certain biomarkers in understanding a patient's likelihood to respond to, typically, immune checkpoint inhibitors. At the same time, personalized medicine has instigated changes in both clinical trial design and its conduct in a push to make drug development and cancer treatment more efficient, precise, and economical. Precision medicine in immuno-oncology is being steered by data-driven approaches to gain insight into the tumor and its dynamic interaction with the immune system. This is particularly necessary given the rapid growth in the number of trials involving more than one immune checkpoint drug, and/or using those in combination with conventional cancer treatments. As multiplex methods, like immunofluorescence, push the boundaries of immunohistochemistry, it becomes critical to understand the foundation of this technology and how it can be deployed for use as a regulated test to identify the prospect of response from mono- and combination therapies. To that end, this work will focus on: 1) the scientific, clinical, and economic requirements for developing clinical multiplex immunofluorescence assays; 2) the attributes of the Akoya Phenoptics workflow to support predictive tests, including design principles, verification, and validation needs; 3) regulatory, safety and quality considerations; 4) application of multiplex immunohistochemistry through lab-developed-tests and regulated in vitro diagnostic devices.

Defining Morphologic Features of Invasion in Pulmonary Nonmucinous Adenocarcinoma With Lepidic Growth: A Proposal by the International Association for the Study of Lung Cancer Pathology Committee

INTRODUCTION

Since the eight edition of the Union for International Cancer Control and American Joint Committee on Cancer TNM classification system, the primary tumor pT stage is determined on the basis of presence and size of the invasive components. The aim of this study was to identify histologic features in tumors with lepidic growth pattern which may be used to establish criteria for distinguishing invasive from noninvasive areas.

METHODS

A Delphi approach was used with two rounds of blinded anonymized analysis of resected nonmucinous lung adenocarcinoma cases with presumed invasive and noninvasive components, followed by one round of reviewer de-anonymized and unblinded review of cases with known outcomes. A digital pathology platform was used for measuring total tumor size and invasive tumor size.

RESULTS

The mean coefficient of variation for measuring total tumor size and tumor invasive size was 6.9% (range: 1.7%-22.3%) and 54% (range: 14.7%-155%), respectively, with substantial variations in interpretation of the size and location of invasion among pathologists. Following the presentation of the results and further discussion among members at large of the International Association for the Study of Lung Cancer Pathology Committee, extensive epithelial proliferation (EEP) in areas of collapsed lepidic growth pattern is recognized as a feature likely to be associated with invasive growth. The EEP is characterized by multilayered luminal epithelial cell growth, usually with high-grade cytologic features in several alveolar spaces.

CONCLUSIONS

Collapsed alveoli and transition zones with EEP were identified by the Delphi process as morphologic features that were a source of interobserver variability. Definition criteria for collapse and EEP are proposed to improve reproducibility of invasion measurement.

Histological interpretation of differentiated vulvar intraepithelial neoplasia (dVIN) remains challenging-observations from a bi-national ring-study

Differentiated vulvar intraepithelial neoplasia (dVIN) is a premalignant lesion that is known to progress rapidly to invasive carcinoma. Accurate histological diagnosis is therefore crucial to allow appropriate treatment. To identify reliable diagnostic features, we evaluated the inter-observer agreement in the histological assessment of dVIN, among a bi-national, multi-institutional group of pathologists. Two investigators from Erasmus MC selected 36 hematoxylin-eosin-stained glass slides of dVIN and no-dysplasia, and prepared a list of 15 histological features of dVIN. Nine participating pathologists (i) diagnosed each slide as dVIN or no-dysplasia, (ii) indicated which features they used for the diagnosis, and (iii) rated these features in terms of their diagnostic usefulness. Diagnoses rendered by > 50% participants were taken as the consensus (gold standard). p53-immunohistochemistry (IHC) was performed for all cases, and the expression patterns were correlated with the consensus diagnoses. Kappa (ĸ)-statistics were computed to measure inter-observer agreements, and concordance of the p53-IHC patterns with the consensus diagnoses. For the diagnosis of dVIN, overall agreement was moderate (ĸ = 0.42), and pair-wise agreements ranged from slight (ĸ = 0.10) to substantial (ĸ = 0.73). Based on the levels of agreement and ratings of usefulness, the most helpful diagnostic features were parakeratosis, cobblestone appearance, chromatin abnormality, angulated nuclei, atypia discernable under × 100, and altered cellular alignment. p53-IHC patterns showed substantial concordance (ĸ = 0.67) with the consensus diagnoses. Histological interpretation of dVIN remains challenging with suboptimal inter-observer agreement. We identified the histological features that may facilitate the diagnosis of dVIN. For cases with a histological suspicion of dVIN, consensus-based pathological evaluation may improve the reliability of the diagnosis.

Reliability of histopathologic diagnosis of fibrotic interstitial lung disease: an international collaborative standardization project

Background

Current interstitial lung disease (ILD) diagnostic guidelines assess criteria across clinical, radiologic and pathologic domains. Significant interobserver variation in histopathologic evaluation has previously been shown but the specific source of these discrepancies is poorly documented. We sought to document specific areas of difficulty and develop improved criteria that would reduce overall interobserver variation.

Methods

Using an internet-based approach, we reviewed selected images of specific diagnostic features of ILD histopathology and whole slide images of fibrotic ILD. After an initial round of review, we confirmed the presence of interobserver variation among our group. We then developed refined criteria and reviewed a second set of cases.

Results

The initial round reproduced the existing literature on interobserver variation in diagnosis of ILD. Cases which were pre-selected as inconsistent with usual interstitial pneumonia/idiopathic pulmonary fibrosis (UIP/IPF) were confirmed as such by multi-observer review. Cases which were thought to be in the spectrum of chronic fibrotic ILD for which UIP/IPF were in the differential showed marked variation in nearly all aspects of ILD evaluation including extent of inflammation and extent and pattern of fibrosis. A proposed set of more explicit criteria had only modest effects on this outcome. While we were only modestly successful in reducing interobserver variation, we did identify specific reasons that current histopathologic criteria of fibrotic ILD are not well defined in practice.

Conclusions

Any additional classification scheme must address interobserver variation in histopathologic diagnosis of fibrotic ILD order to remain clinically relevant. Improvements to tissue-based diagnostics may require substantial resources such as larger datasets or novel technologies to improve reproducibility. Benchmarks should be established for expected outcomes among clinically defined subgroups as a quality metric.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12890-021-01522-6.