Pilot study to evaluate tools to collect pathologist annotations for validating machine learning algorithms

Purpose: Validation of artificial intelligence (AI) algorithms in digital pathology with a reference standard is necessary before widespread clinical use, but few examples focus on creating a reference standard based on pathologist annotations. This work assesses the results of a pilot study that collects density estimates of stromal tumor-infiltrating lymphocytes (sTILs) in breast cancer biopsy specimens. This work will inform the creation of a validation dataset for the evaluation of AI algorithms fit for a regulatory purpose. Approach: Collaborators and crowdsourced pathologists contributed glass slides, digital images, and annotations. Here, "annotations" refer to any marks, segmentations, measurements, or labels a pathologist adds to a report, image, region of interest (ROI), or biological feature. Pathologists estimated sTILs density in 640 ROIs from hematoxylin and eosin stained slides of 64 patients via two modalities: an optical light microscope and two digital image viewing platforms. Results: The pilot study generated 7373 sTILs density estimates from 29 pathologists. Analysis of annotations found the variability of density estimates per ROI increases with the mean; the root mean square differences were 4.46, 14.25, and 26.25 as the mean density ranged from 0% to 10%, 11% to 40%, and 41% to 100%, respectively. The pilot study informs three areas of improvement for future work: technical workflows, annotation platforms, and agreement analysis methods. Upgrades to the workflows and platforms will improve operability and increase annotation speed and consistency. Conclusions: Exploratory data analysis demonstrates the need to develop new statistical approaches for agreement. The pilot study dataset and analysis methods are publicly available to allow community feedback. The development and results of the validation dataset will be publicly available to serve as an instructive tool that can be replicated by developers and researchers.

Abstract 460: Tools for collecting pathologist annotations and understanding interobserver variability

Background: The High Throughput Truthing (HTT) project is assessing pathologist agreement estimates of stromal tumor-infiltrating lymphocytes (sTILs) density in hematoxylin and eosin (H&E) stained breast cancer biopsy slides. The HTT project will create a validation dataset for artificial intelligence and machine learning (AI/ML) algorithms in digital pathology fit for a training, proficiency testing, and regulatory purpose.

Methods: The pilot study crowdsourced pathologists to estimate sTIL density in 640 regions of interest (ROIs) across 64 slides via two modalities: an optical microscope (eeDAP) and two digital platforms (caMicroscope and PathPresenter). eeDAP is a hardware-software interface that presents the observer with pre-defined fields of view on H&E slides that corresponds to the ROI on a whole slide image. The PathPresenter and caMicroscope web-applications replicate the eeDAP workflow on the whole slide image without microscope hardware. In the workflow, pathologists evaluated the eligibility of an ROI for sTILs content then estimated the densities of tumor-associated stroma and sTILs in the ROI. Inter-pathologist agreement within ROIs was characterized with the root mean-squared difference. Using 72 of the highest variability ROIs selected from the pilot study, seven practicing pathologists participated in a subsequent focus group to improve the clinical training and data-collection workflows.

Results: The pilot study collected 7,373 sTIL density estimates from 35 pathologists between February 2020 and May 2021. The focus group provided an additional 411 evaluations on 72 ROIs and in-depth discussions to identify pitfalls, gaps in training, and workflow improvements. Installation of eeDAP for physical data collection guided improvements in documentation and operation capabilities. Updated training materials refine the definition of tumor-associated stroma, provide reference images to differentiate sTILs from other cell types, and provide feedback during training. Digital and microscope platforms benefitted from enforcing registration and training, standardizing workflows, and accelerating eeDAP slide-image registration.

Conclusions: The slides, images, and annotations provided by volunteer collaborators and participants for our pilot study led to improvements in data collection tools and crowdsourcing workflows to ensure consistency and minimize annotation variability. Our pilot dataset and analysis methods are available on a public HTT Github repository to allow open access to our methodology and feedback from the digital pathology and statistics communities. These data-collection and analysis methods are applicable to other quantitative biomarkers for validation of AI/ML algorithms. The lessons learned from this work will be applied to the HTT pivotal study and inform future quality data-collection methods of pathologist annotations.

Citation Format: Katherine N. Elfer, Kim Blenman, Sarah N. Dudgeon, Victor Garcia, Anna Ehinger, Xiaoxian Li, Amy Ly, Dieter Peeters, Bruce Werness, Matthew Hanna, Roberto Salgado. Tools for collecting pathologist annotations and understanding interobserver variability [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2022; 2022 Apr 8-13. Philadelphia (PA): AACR; Cancer Res 2022;82(12_Suppl):Abstract nr 460.

A Pathologist-Annotated Dataset for Validating Artificial Intelligence: A Project Description and Pilot Study

Purpose:

Validating artificial intelligence algorithms for clinical use in medical images is a challenging endeavor due to a lack of standard reference data (ground truth). This topic typically occupies a small portion of the discussion in research papers since most of the efforts are focused on developing novel algorithms. In this work, we present a collaboration to create a validation dataset of pathologist annotations for algorithms that process whole slide images. We focus on data collection and evaluation of algorithm performance in the context of estimating the density of stromal tumor-infiltrating lymphocytes (sTILs) in breast cancer.

Methods:

We digitized 64 glass slides of hematoxylin- and eosin-stained invasive ductal carcinoma core biopsies prepared at a single clinical site. A collaborating pathologist selected 10 regions of interest (ROIs) per slide for evaluation. We created training materials and workflows to crowdsource pathologist image annotations on two modes: an optical microscope and two digital platforms. The microscope platform allows the same ROIs to be evaluated in both modes. The workflows collect the ROI type, a decision on whether the ROI is appropriate for estimating the density of sTILs, and if appropriate, the sTIL density value for that ROI.

Results:

In total, 19 pathologists made 1645 ROI evaluations during a data collection event and the following 2 weeks. The pilot study yielded an abundant number of cases with nominal sTIL infiltration. Furthermore, we found that the sTIL densities are correlated within a case, and there is notable pathologist variability. Consequently, we outline plans to improve our ROI and case sampling methods. We also outline statistical methods to account for ROI correlations within a case and pathologist variability when validating an algorithm.

Conclusion:

We have built workflows for efficient data collection and tested them in a pilot study. As we prepare for pivotal studies, we will investigate methods to use the dataset as an external validation tool for algorithms. We will also consider what it will take for the dataset to be fit for a regulatory purpose: study size, patient population, and pathologist training and qualifications. To this end, we will elicit feedback from the Food and Drug Administration via the Medical Device Development Tool program and from the broader digital pathology and AI community. Ultimately, we intend to share the dataset, statistical methods, and lessons learned.

BRCA1/2 mutation status influences somatic genetic progression in inherited and sporadic epithelial ovarian cancer cases

Metaphase comparative genomic hybridization was used to analyze the spectrum of genetic alterations in 141 epithelial ovarian cancers from BRCA1 and BRCA2 mutation carriers, individuals with familial non-BRCA1/2 epithelial ovarian cancer, and women with nonfamilial epithelial ovarian cancer. Multiple genetic alterations were identified in almost all tumors. The high frequency with which some alterations were identified suggests the location of genes that are commonly altered during ovarian tumor development. In multiple chromosome regions, there were significant differences in alteration frequency between the four tumor types suggesting that BRCA1/2 mutation status and a family history of ovarian cancer influences the somatic genetic pathway of ovarian cancer progression. These findings were supported by hierarchical cluster analysis, which identified genetic events that tend to occur together during tumorigenesis and several alterations that were specific to tumors of a particular type. In addition, some genetic alterations were strongly associated with differences in tumor differentiation and disease stage. Taken together, these data provide molecular genetic evidence to support previous findings from histopathological studies, which suggest that clinical features of ovarian and breast tumors differ with respect to BRCA1/2 mutation status and/or cancer family history.

Telomerase activity in benign and malignant thyroid tumors

Thyroid nodules are found in 5% to 10% of the population. While these nodules carry only a 5% to 10% risk of malignancy, tests that complement fine-needle aspiration (FNA) cytology in preoperative diagnosis and risk stratification are lacking. Telomerase is a ribonucleoprotein polymerase with activity found in many malignant tissues, but absent from most normal adult tissue. In this study, we have investigated telomerase activity in 24 thyroid tumors, 14 matched adjacent thyroid tissues, and 3 chronic thyroiditis tissue samples. Using a telomeric repeat amplification protocol (TRAP) assay on frozen tissue, telomerase activity was detected in 11 of 20 thyroid carcinomas, including 10 of 14 papillary carcinomas and a Hurthle cell carcinoma. Telomerase activity was not detected in 4 benign adenomas, 3 follicular carcinomas, or a single case each of medullary and anaplastic thyroid carcinoma. Telomerase activity was detected in 3 of 14 samples of adjacent thyroid tissue from patients with thyroid tumors. Interestingly, all 3 cases of adjacent thyroid tissue that tested positive had a moderate to marked degree of chronic inflammation. In addition, 3 of 3 samples from chronic thyroiditis specimens tested positive for telomerase activity. When tumor invasiveness (vascular and/or capsular) was compared with telomerase activity in papillary carcinomas, only 1 of 4 telomerase-negative tumors was invasive, while 6 of 10 of telomerase-positive tumors were invasive. Moreover, 6 of 7 invasive papillary carcinomas had telomerase activity. In summary, this is the first report of telomerase activity in thyroid tissue and nodules. This activity was detected in a large percentage of papillary thyroid carcinomas, but not benign adenomas, follicular carcinomas, or most normal thyroid tissue. Telomerase activity may also correlate with tumor invasiveness. Further studies will focus on larger numbers of tumors, metastatic tissue, and undifferentiated carcinomas, as well as application of this assay to products from fine-needle aspirates as a potential diagnostic and prognostic marker in thyroid neoplasms.