A Containerized Software System for Generation, Management, and Exploration of Features from Whole Slide Tissue Images

Keratin 17 modulates the immune topography of pancreatic cancer

BACKGROUND

The immune microenvironment impacts tumor growth, invasion, metastasis, and patient survival and may provide opportunities for therapeutic intervention in pancreatic ductal adenocarcinoma (PDAC). Although never studied as a potential modulator of the immune response in most cancers, Keratin 17 (K17), a biomarker of the most aggressive (basal) molecular subtype of PDAC, is intimately involved in the histogenesis of the immune response in psoriasis, basal cell carcinoma, and cervical squamous cell carcinoma. Thus, we hypothesized that K17 expression could also impact the immune cell response in PDAC, and that uncovering this relationship could provide insight to guide the development of immunotherapeutic opportunities to extend patient survival.

METHODS

Multiplex immunohistochemistry (mIHC) and automated image analysis based on novel computational imaging technology were used to decipher the abundance and spatial distribution of T cells, macrophages, and tumor cells, relative to K17 expression in 235 PDACs.

RESULTS

K17 expression had profound effects on the exclusion of intratumoral CD8+ T cells and was also associated with decreased numbers of peritumoral CD8+ T cells, CD16+ macrophages, and CD163+ macrophages (p < 0.0001). The differences in the intratumor and peritumoral CD8+ T cell abundance were not impacted by neoadjuvant therapy, tumor stage, grade, lymph node status, histologic subtype, nor KRAS, p53, SMAD4, or CDKN2A mutations.

CONCLUSIONS

Thus, K17 expression correlates with major differences in the immune microenvironment that are independent of any tested clinicopathologic or tumor intrinsic variables, suggesting that targeting K17-mediated immune effects on the immune system could restore the innate immunologic response to PDAC and might provide novel opportunities to restore immunotherapeutic approaches for this most deadly form of cancer.

Training pathologists to assess stromal tumour-infiltrating lymphocytes in breast cancer synergises efforts in clinical care and scientific research

A growing body of research supports stromal tumour-infiltrating lymphocyte (TIL) density in breast cancer to be a robust prognostic and predicive biomarker. The gold standard for stromal TIL density quantitation in breast cancer is pathologist visual assessment using haematoxylin and eosin-stained slides. Artificial intelligence/machine-learning algorithms are in development to automate the stromal TIL scoring process, and must be validated against a reference standard such as pathologist visual assessment. Visual TIL assessment may suffer from significant interobserver variability. To improve interobserver agreement, regulatory science experts at the US Food and Drug Administration partnered with academic pathologists internationally to create a freely available online continuing medical education (CME) course to train pathologists in assessing breast cancer stromal TILs using an interactive format with expert commentary. Here we describe and provide a user guide to this CME course, whose content was designed to improve pathologist accuracy in scoring breast cancer TILs. We also suggest subsequent steps to translate knowledge into clinical practice with proficiency testing.

Reproducible Reporting of the Collection and Evaluation of Annotations for Artificial Intelligence Models

This work puts forth and demonstrates the utility of a reporting framework for collecting and evaluating annotations of medical images used for training and testing artificial intelligence (AI) models in assisting detection and diagnosis. AI has unique reporting requirements, as shown by the AI extensions to the Consolidated Standards of Reporting Trials (CONSORT) and Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) checklists and the proposed AI extensions to the Standards for Reporting Diagnostic Accuracy (STARD) and Transparent Reporting of a Multivariable Prediction model for Individual Prognosis or Diagnosis (TRIPOD) checklists. AI for detection and/or diagnostic image analysis requires complete, reproducible, and transparent reporting of the annotations and metadata used in training and testing data sets. In an earlier work by other researchers, an annotation workflow and quality checklist for computational pathology annotations were proposed. In this manuscript, we operationalize this workflow into an evaluable quality checklist that applies to any reader-interpreted medical images, and we demonstrate its use for an annotation effort in digital pathology. We refer to this quality framework as the Collection and Evaluation of Annotations for Reproducible Reporting of Artificial Intelligence (CLEARR-AI).

Initial interactions with the FDA on developing a validation dataset as a medical device development tool

Quantifying tumor-infiltrating lymphocytes (TILs) in breast cancer tumors is a challenging task for pathologists. With the advent of whole slide imaging that digitizes glass slides, it is possible to apply computational models to quantify TILs for pathologists. Development of computational models requires significant time, expertise, consensus, and investment. To reduce this burden, we are preparing a dataset for developers to validate their models and a proposal to the Medical Device Development Tool (MDDT) program in the Center for Devices and Radiological Health of the U.S. Food and Drug Administration (FDA). If the FDA qualifies the dataset for its submitted context of use, model developers can use it in a regulatory submission within the qualified context of use without additional documentation. Our dataset aims at reducing the regulatory burden placed on developers of models that estimate the density of TILs and will allow head-to-head comparison of multiple computational models on the same data. In this paper, we discuss the MDDT preparation and submission process, including the feedback we received from our initial interactions with the FDA and propose how a qualified MDDT validation dataset could be a mechanism for open, fair, and consistent measures of computational model performance. Our experiences will help the community understand what the FDA considers relevant and appropriate (from the perspective of the submitter), at the early stages of the MDDT submission process, for validating stromal TIL density estimation models and other potential computational models. © 2023 The Authors. The Journal of Pathology published by John Wiley & Sons Ltd on behalf of The Pathological Society of Great Britain and Ireland. This article has been contributed to by U.S. Government employees and their work is in the public domain in the USA.

Personalizable AI platform for universal access to research and diagnosis in digital pathology

BACKGROUND AND MOTIVATION

Digital pathology has been evolving over the last years, proposing significant workflow advantages that have fostered its adoption in professional environments. Patient clinical and image data are readily available in remote data banks that can be consumed efficiently over standard communication technologies. The appearance of new imaging techniques and advanced artificial intelligence algorithms has significantly reduced the burden on medical professionals by speeding up the screening process. Despite these advancements, the usage of digital pathology in professional environments has been slowed down by poor interoperability between services resulting from a lack of standard interfaces and integrative solutions. This work addresses this issue by proposing a cloud-based digital pathology platform built on standard and open interfaces.

METHODS

The work proposes and describes a vendor-neutral platform that provides interfaces for managing digital slides, and medical reports, and integrating digital image analysis services compatible with existing standards. The solution integrates the open-source plugin-based Dicoogle PACS for interoperability and extensibility, which grants the proposed solution great feature customization.

RESULTS

The solution was developed in collaboration with iPATH research project partners, including the validation by medical pathologists. The result is a pure Web collaborative framework that supports both research and production environments. A total of 566 digital slides from different pathologies were successfully uploaded to the platform. Using the integration interfaces, a mitosis detection algorithm was successfully installed into the platform, and it was trained with 2400 annotations collected from breast carcinoma images.

CONCLUSION

Interoperability is a key factor when discussing digital pathology solutions, as it facilitates their integration into existing institutions' information systems. Moreover, it improves data sharing and integration of third-party services such as image analysis services, which have become relevant in today's digital pathology workflow. The proposed solution fully embraces the DICOM standard for digital pathology, presenting an interoperable cloud-based solution that provides great feature customization thanks to its extensible architecture.

The Importance of Quality Assurance in Radiation Oncology Clinical Trials

Clinical trials have been the center of progress in modern medicine. In oncology, we are fortunate to have a structure in place through the National Clinical Trials Network (NCTN). The NCTN provides the infrastructure and a forum for scientific discussion to develop clinical concepts for trial design. The NCTN also provides a network group structure to administer trials for successful trial management and outcome analyses. There are many important aspects to trial design and conduct. Modern trials need to ensure appropriate trial conduct and secure data management processes. Of equal importance is the quality assurance of a clinical trial. If progress is to be made in oncology clinical medicine, investigators and patient care providers of service need to feel secure that trial data is complete, accurate, and well-controlled in order to be confident in trial analysis and move trial outcome results into daily practice. As our technology has matured, so has our need to apply technology in a uniform manner for appropriate interpretation of trial outcomes. In this article, we review the importance of quality assurance in clinical trials involving radiation therapy. We will include important aspects of institution and investigator credentialing for participation as well as ongoing processes to ensure that each trial is being managed in a compliant manner. We will provide examples of the importance of complete datasets to ensure study interpretation. We will describe how successful strategies for quality assurance in the past will support new initiatives moving forward.

Computational immunohistochemical mapping adds immune context to histological phenotypes in mouse models of colitis

Inflammatory bowel disease (IBD) is characterized by chronic, dysregulated inflammation in the gastrointestinal tract. The heterogeneity of IBD is reflected through two major subtypes, Crohn's Disease (CD) and Ulcerative Colitis (UC). CD and UC differ across symptomatic presentation, histology, immune responses, and treatment. While colitis mouse models have been influential in deciphering IBD pathogenesis, no single model captures the full heterogeneity of clinical disease. The translational capacity of mouse models may be augmented by shifting to multi-mouse model studies that aggregate analysis across various well-controlled phenotypes. Here, we evaluate the value of histology in multi-mouse model characterizations by building upon a previous pipeline that detects histological disease classes in hematoxylin and eosin (H&E)-stained murine colons. Specifically, we map immune marker positivity across serially-sectioned slides to H&E histological classes across the dextran sodium sulfate (DSS) chemical induction model and the intestinal epithelium-specific, inducible Villin-CreERT2;Klf5fl/fl (Klf5ΔIND) genetic model. In this study, we construct the beginning frameworks to define H&E-patch-based immunophenotypes based on IHC-H&E mappings.

Deep learning in computational dermatopathology of melanoma: A technical systematic literature review

Deep learning (DL) has become one of the major approaches in computational dermatopathology, evidenced by a significant increase in this topic in the current literature. We aim to provide a structured and comprehensive overview of peer-reviewed publications on DL applied to dermatopathology focused on melanoma. In comparison to well-published DL methods on non-medical images (e.g., classification on ImageNet), this field of application comprises a specific set of challenges, such as staining artifacts, large gigapixel images, and various magnification levels. Thus, we are particularly interested in the pathology-specific technical state-of-the-art. We also aim to summarize the best performances achieved thus far with respect to accuracy, along with an overview of self-reported limitations. Accordingly, we conducted a systematic literature review of peer-reviewed journal and conference articles published between 2012 and 2022 in the databases ACM Digital Library, Embase, IEEE Xplore, PubMed, and Scopus, expanded by forward and backward searches to identify 495 potentially eligible studies. After screening for relevance and quality, a total of 54 studies were included. We qualitatively summarized and analyzed these studies from technical, problem-oriented, and task-oriented perspectives. Our findings suggest that the technical aspects of DL for histopathology in melanoma can be further improved. The DL methodology was adopted later in this field, and still lacks the wider adoption of DL methods already shown to be effective for other applications. We also discuss upcoming trends toward ImageNet-based feature extraction and larger models. While DL has achieved human-competitive accuracy in routine pathological tasks, its performance on advanced tasks is still inferior to wet-lab testing (for example). Finally, we discuss the challenges impeding the translation of DL methods to clinical practice and provide insight into future research directions.

Interoperable slide microscopy viewer and annotation tool for imaging data science and computational pathology

The exchange of large and complex slide microscopy imaging data in biomedical research and pathology practice is impeded by a lack of data standardization and interoperability, which is detrimental to the reproducibility of scientific findings and clinical integration of technological innovations. We introduce Slim, an open-source, web-based slide microscopy viewer that implements the internationally accepted Digital Imaging and Communications in Medicine (DICOM) standard to achieve interoperability with a multitude of existing medical imaging systems. We showcase the capabilities of Slim as the slide microscopy viewer of the NCI Imaging Data Commons and demonstrate how the viewer enables interactive visualization of traditional brightfield microscopy and highly-multiplexed immunofluorescence microscopy images from The Cancer Genome Atlas and Human Tissue Atlas Network, respectively, using standard DICOMweb services. We further show how Slim enables the collection of standardized image annotations for the development or validation of machine learning models and the visual interpretation of model inference results in the form of segmentation masks, spatial heat maps, or image-derived measurements.

Histology Image Viewer and Converter (HIVC): A High-Speed Freeware Software to View and Convert Whole Slide Histology Images

Histology images are widely used to assess the microstructure of biological tissues, but scanners often save images in bulky SVS and multi-layered TIFF formats. These formats were designed to archive image blocks and high-resolution textual information and are not compatible with conventional image analysis software. Our goal was to create a freeware Histology Image Viewer and Converter (HIVC) with a graphical user interface that allows viewing and converting whole-slide images in batch. HIVC was developed using C# Language for Windows x64 operating system. HIVC's performance was assessed by converting 20 whole-slide images to a JPG format at 20x and 40x resolution and comparing the results to ImageJ, Cell Profiler, QuPath, Nanoborb, and Aperio ImageScope. HIVC was more than 8-times faster in converting images than other software packages. This software allows high-speed batch conversion of histology images to traditional formats, permitting platform-independent secondary analyses.

Radiation Oncology: Future Vision for Quality Assurance and Data Management in Clinical Trials and Translational Science

The future of radiation oncology is exceptionally strong as we are increasingly involved in nearly all oncology disease sites due to extraordinary advances in radiation oncology treatment management platforms and improvements in treatment execution. Due to our technology and consistent accuracy, compressed radiation oncology treatment strategies are becoming more commonplace secondary to our ability to successfully treat tumor targets with increased normal tissue avoidance. In many disease sites including the central nervous system, pulmonary parenchyma, liver, and other areas, our service is redefining the standards of care. Targeting of disease has improved due to advances in tumor imaging and application of integrated imaging datasets into sophisticated planning systems which can optimize volume driven plans created by talented personnel. Treatment times have significantly decreased due to volume driven arc therapy and positioning is secured by real time imaging and optical tracking. Normal tissue exclusion has permitted compressed treatment schedules making treatment more convenient for the patient. These changes require additional study to further optimize care. Because data exchange worldwide have evolved through digital platforms and prisms, images and radiation datasets worldwide can be shared/reviewed on a same day basis using established de-identification and anonymization methods. Data storage post-trial completion can co-exist with digital pathomic and radiomic information in a single database coupled with patient specific outcome information and serve to move our translational science forward with nimble query elements and artificial intelligence to ask better questions of the data we collect and collate. This will be important moving forward to validate our process improvements at an enterprise level and support our science. We have to be thorough and complete in our data acquisition processes, however if we remain disciplined in our data management plan, our field can grow further and become more successful generating new standards of care from validated datasets.

Pilot study to evaluate tools to collect pathologist annotations for validating machine learning algorithms

Purpose: Validation of artificial intelligence (AI) algorithms in digital pathology with a reference standard is necessary before widespread clinical use, but few examples focus on creating a reference standard based on pathologist annotations. This work assesses the results of a pilot study that collects density estimates of stromal tumor-infiltrating lymphocytes (sTILs) in breast cancer biopsy specimens. This work will inform the creation of a validation dataset for the evaluation of AI algorithms fit for a regulatory purpose. Approach: Collaborators and crowdsourced pathologists contributed glass slides, digital images, and annotations. Here, "annotations" refer to any marks, segmentations, measurements, or labels a pathologist adds to a report, image, region of interest (ROI), or biological feature. Pathologists estimated sTILs density in 640 ROIs from hematoxylin and eosin stained slides of 64 patients via two modalities: an optical light microscope and two digital image viewing platforms. Results: The pilot study generated 7373 sTILs density estimates from 29 pathologists. Analysis of annotations found the variability of density estimates per ROI increases with the mean; the root mean square differences were 4.46, 14.25, and 26.25 as the mean density ranged from 0% to 10%, 11% to 40%, and 41% to 100%, respectively. The pilot study informs three areas of improvement for future work: technical workflows, annotation platforms, and agreement analysis methods. Upgrades to the workflows and platforms will improve operability and increase annotation speed and consistency. Conclusions: Exploratory data analysis demonstrates the need to develop new statistical approaches for agreement. The pilot study dataset and analysis methods are publicly available to allow community feedback. The development and results of the validation dataset will be publicly available to serve as an instructive tool that can be replicated by developers and researchers.

General Roadmap and Core Steps for the Development of AI Tools in Digital Pathology

Integrating Artificial Intelligence (AI) tools in the tissue diagnostic workflow will benefit the pathologist and, ultimately, the patient. The generation of such AI tools has two parallel yet interconnected processes, namely the definition of the pathologist’s task to be delivered in silico, and the software development requirements. In this review paper, we demystify this process, from a viewpoint that joins experienced pathologists and data scientists, by proposing a general pathway and describing the core steps to build an AI digital pathology tool. In doing so, we highlight the importance of the collaboration between AI scientists and pathologists, from the initial formulation of the hypothesis to the final, ready-to-use product.

Development of Training Materials for Pathologists to Provide Machine Learning Validation Data of Tumor-Infiltrating Lymphocytes in Breast Cancer

Simple Summary

The High Throughput Truthing project aims to develop a dataset of stromal tumor-infiltrating lymphocytes (sTILs) density evaluations in hematoxylin and eosin-stained invasive breast cancer specimens fit for a regulatory purpose. After completion of the pilot study, the analysis demonstrated inconsistencies and gaps in the provided training to pathologists. Select regions of interest (ROIs) were reviewed by an expert panel, who provided annotations and commentary on the challenges of the sTILs assessment. We used these annotations to develop a training document and reference standard for new training materials. These materials will train crowd-sourced pathologists to help create an algorithm validation dataset and contribute to sTILs evaluations in clinical practice.

Abstract

The High Throughput Truthing project aims to develop a dataset for validating artificial intelligence and machine learning models (AI/ML) fit for regulatory purposes. The context of this AI/ML validation dataset is the reporting of stromal tumor-infiltrating lymphocytes (sTILs) density evaluations in hematoxylin and eosin-stained invasive breast cancer biopsy specimens. After completing the pilot study, we found notable variability in the sTILs estimates as well as inconsistencies and gaps in the provided training to pathologists. Using the pilot study data and an expert panel, we created custom training materials to improve pathologist annotation quality for the pivotal study. We categorized regions of interest (ROIs) based on their mean sTILs density and selected ROIs with the highest and lowest sTILs variability. In a series of eight one-hour sessions, the expert panel reviewed each ROI and provided verbal density estimates and comments on features that confounded the sTILs evaluation. We aggregated and shaped the comments to identify pitfalls and instructions to improve our training materials. From these selected ROIs, we created a training set and proficiency test set to improve pathologist training with the goal to improve data collection for the pivotal study. We are not exploring AI/ML performance in this paper. Instead, we are creating materials that will train crowd-sourced pathologists to be the reference standard in a pivotal study to create an AI/ML model validation dataset. The issues discussed here are also important for clinicians to understand about the evaluation of sTILs in clinical practice and can provide insight to developers of AI/ML models.

Deep Learning-Based Mapping of Tumor Infiltrating Lymphocytes in Whole Slide Images of 23 Types of Cancer

The role of tumor infiltrating lymphocytes (TILs) as a biomarker to predict disease progression and clinical outcomes has generated tremendous interest in translational cancer research. We present an updated and enhanced deep learning workflow to classify 50x50 um tiled image patches (100x100 pixels at 20x magnification) as TIL positive or negative based on the presence of 2 or more TILs in gigapixel whole slide images (WSIs) from the Cancer Genome Atlas (TCGA). This workflow generates TIL maps to study the abundance and spatial distribution of TILs in 23 different types of cancer. We trained three state-of-the-art, popular convolutional neural network (CNN) architectures (namely VGG16, Inception-V4, and ResNet-34) with a large volume of training data, which combined manual annotations from pathologists (strong annotations) and computer-generated labels from our previously reported first-generation TIL model for 13 cancer types (model-generated annotations). Specifically, this training dataset contains TIL positive and negative patches from cancers in additional organ sites and curated data to help improve algorithmic performance by decreasing known false positives and false negatives. Our new TIL workflow also incorporates automated thresholding to convert model predictions into binary classifications to generate TIL maps. The new TIL models all achieve better performance with improvements of up to 13% in accuracy and 15% in F-score. We report these new TIL models and a curated dataset of TIL maps, referred to as TIL-Maps-23, for 7983 WSIs spanning 23 types of cancer with complex and diverse visual appearances, which will be publicly available along with the code to evaluate performance. Code Available at: https://github.com/ShahiraAbousamra/til_classification.

Multimodal analysis suggests differential immuno-metabolic crosstalk in lung squamous cell carcinoma and adenocarcinoma

Immunometabolism within the tumor microenvironment is an appealing target for precision therapy approaches in lung cancer. Interestingly, obesity confers an improved response to immune checkpoint inhibition in non-small cell lung cancer (NSCLC), suggesting intriguing relationships between systemic metabolism and the immunometabolic environment in lung tumors. We hypothesized that visceral fat and ¹⁸F-Fluorodeoxyglucose uptake influenced the tumor immunometabolic environment and that these bidirectional relationships differ in NSCLC subtypes, lung adenocarcinoma (LUAD) and lung squamous cell carcinoma (LUSC). By integrating ¹⁸F-FDG PET/CT imaging, bulk and single-cell RNA-sequencing, and histology, we observed that LUSC had a greater dependence on glucose than LUAD. In LUAD tumors with high glucose uptake, glutaminase was downregulated, suggesting a tradeoff between glucose and glutamine metabolism, while in LUSC tumors with high glucose uptake, genes related to fatty acid and amino acid metabolism were also increased. We found that tumor-infiltrating T cells had the highest expression of glutaminase, ribosomal protein 37, and cystathionine gamma-lyase in NSCLC, highlighting the metabolic flexibility of this cell type. Further, we demonstrate that visceral adiposity, but not body mass index (BMI), was positively associated with tumor glucose uptake in LUAD and that patients with high BMI had favorable prognostic transcriptional profiles, while tumors of patients with high visceral fat had poor prognostic gene expression. We posit that metabolic adjunct therapy may be more successful in LUSC rather than LUAD due to LUAD's metabolic flexibility and that visceral adiposity, not BMI alone, should be considered when developing precision medicine approaches for the treatment of NSCLC.

An Expandable Informatics Framework for Enhancing Central Cancer Registries with Digital Pathology Specimens, Computational Imaging Tools, and Advanced Mining Capabilities

BACKGROUND

Population-based state cancer registries are an authoritative source for cancer statistics in the United States. They routinely collect a variety of data, including patient demographics, primary tumor site, stage at diagnosis, first course of treatment, and survival, on every cancer case that is reported across all U.S. states and territories. The goal of our project is to enrich NCI's Surveillance, Epidemiology, and End Results (SEER) registry data with high-quality population-based biospecimen data in the form of digital pathology, machine-learning-based classifications, and quantitative histopathology imaging feature sets (referred to here as Pathomics features).

MATERIALS AND METHODS

As part of the project, the underlying informatics infrastructure was designed, tested, and implemented through close collaboration with several participating SEER registries to ensure consistency with registry processes, computational scalability, and ability to support creation of population cohorts that span multiple sites. Utilizing computational imaging algorithms and methods to both generate indices and search for matches makes it possible to reduce inter- and intra-observer inconsistencies and to improve the objectivity with which large image repositories are interrogated.

RESULTS

Our team has created and continues to expand a well-curated repository of high-quality digitized pathology images corresponding to subjects whose data are routinely collected by the collaborating registries. Our team has systematically deployed and tested key, visual analytic methods to facilitate automated creation of population cohorts for epidemiological studies and tools to support visualization of feature clusters and evaluation of whole-slide images. As part of these efforts, we are developing and optimizing advanced search and matching algorithms to facilitate automated, content-based retrieval of digitized specimens based on their underlying image features and staining characteristics.

CONCLUSION

To meet the challenges of this project, we established the analytic pipelines, methods, and workflows to support the expansion and management of a growing repository of high-quality digitized pathology and information-rich, population cohorts containing objective imaging and clinical attributes to facilitate studies that seek to discriminate among different subtypes of disease, stratify patient populations, and perform comparisons of tumor characteristics within and across patient cohorts. We have also successfully developed a suite of tools based on a deep-learning method to perform quantitative characterizations of tumor regions, assess infiltrating lymphocyte distributions, and generate objective nuclear feature measurements. As part of these efforts, our team has implemented reliable methods that enable investigators to systematically search through large repositories to automatically retrieve digitized pathology specimens and correlated clinical data based on their computational signatures.

A Pathologist-Annotated Dataset for Validating Artificial Intelligence: A Project Description and Pilot Study

Purpose:

Validating artificial intelligence algorithms for clinical use in medical images is a challenging endeavor due to a lack of standard reference data (ground truth). This topic typically occupies a small portion of the discussion in research papers since most of the efforts are focused on developing novel algorithms. In this work, we present a collaboration to create a validation dataset of pathologist annotations for algorithms that process whole slide images. We focus on data collection and evaluation of algorithm performance in the context of estimating the density of stromal tumor-infiltrating lymphocytes (sTILs) in breast cancer.

Methods:

We digitized 64 glass slides of hematoxylin- and eosin-stained invasive ductal carcinoma core biopsies prepared at a single clinical site. A collaborating pathologist selected 10 regions of interest (ROIs) per slide for evaluation. We created training materials and workflows to crowdsource pathologist image annotations on two modes: an optical microscope and two digital platforms. The microscope platform allows the same ROIs to be evaluated in both modes. The workflows collect the ROI type, a decision on whether the ROI is appropriate for estimating the density of sTILs, and if appropriate, the sTIL density value for that ROI.

Results:

In total, 19 pathologists made 1645 ROI evaluations during a data collection event and the following 2 weeks. The pilot study yielded an abundant number of cases with nominal sTIL infiltration. Furthermore, we found that the sTIL densities are correlated within a case, and there is notable pathologist variability. Consequently, we outline plans to improve our ROI and case sampling methods. We also outline statistical methods to account for ROI correlations within a case and pathologist variability when validating an algorithm.

Conclusion:

We have built workflows for efficient data collection and tested them in a pilot study. As we prepare for pivotal studies, we will investigate methods to use the dataset as an external validation tool for algorithms. We will also consider what it will take for the dataset to be fit for a regulatory purpose: study size, patient population, and pathologist training and qualifications. To this end, we will elicit feedback from the Food and Drug Administration via the Medical Device Development Tool program and from the broader digital pathology and AI community. Ultimately, we intend to share the dataset, statistical methods, and lessons learned.

PRISM: A Platform for Imaging in Precision Medicine

PURPOSE

Precision medicine requires an understanding of individual variability, which can only be acquired from large data collections such as those supported by the Cancer Imaging Archive (TCIA). We have undertaken a program to extend the types of data TCIA can support. This, in turn, will enable TCIA to play a key role in precision medicine research by collecting and disseminating high-quality, state-of-the-art, quantitative imaging data that meet the evolving needs of the cancer research community.

METHODS

A modular technology platform is presented that would allow existing data resources, such as TCIA, to evolve into a comprehensive data resource that meets the needs of users engaged in translational research for imaging-based precision medicine. This Platform for Imaging in Precision Medicine (PRISM) helps streamline the deployment and improve TCIA's efficiency and sustainability. More importantly, its inherent modular architecture facilitates a piecemeal adoption by other data repositories.

RESULTS

PRISM includes services for managing radiology and pathology images and features and associated clinical data. A semantic layer is being built to help users explore diverse collections and pool data sets to create specialized cohorts. PRISM includes tools for image curation and de-identification. It includes image visualization and feature exploration tools. The entire platform is distributed as a series of containerized microservices with representational state transfer interfaces.

CONCLUSION

PRISM is helping modernize, scale, and sustain the technology stack that powers TCIA. Repositories can take advantage of individual PRISM services such as de-identification and quality control. PRISM is helping scale image informatics for cancer research at a time when the size, complexity, and demands to integrate image data with other precision medicine data-intensive commons are mounting.

CytoBrowser: a browser-based collaborative annotation platform for whole slide images

We present CytoBrowser, an open-source (GPLv3) JavaScript and Node.js driven environment for fast and accessible collaborative online visualization, assessment, and annotation of very large microscopy images, including, but not limited to, z-stacks (focus stacks) of cytology or histology whole slide images. CytoBrowser provides a web-based viewer for high-resolution zoomable images and facilitates easy remote collaboration, with options for joint-view visualization and simultaneous collaborative annotation of very large datasets. It delivers a unique combination of functionalities not found in other software solutions, making it a preferred tool for large scale annotation of whole slide image data. The web browser interface is directly accessible on any modern computer or even on a mobile phone, without need for additional software. By sharing a "session", several remote users can interactively explore and jointly annotate whole slide image data, thereby reaching improved data understanding and annotation quality, effortless project scaling and distribution of resources to/from remote locations, efficient creation of "ground truth" annotations for methods' evaluation and training of machine learning-based approaches, a user-friendly learning environment for medical students, to just name a few. Rectangle and polygon region annotations complement point-based annotations, each with a selectable annotation-class as well as free-form text fields. The default setting of CytoBrowser presents an interface for the Bethesda cancer grading system, while other annotation schemes can easily be incorporated. Automatic server side storage of annotations is complemented by JSON-based import/export options facilitating easy interoperability with other tools. CytoBrowser is available here: https://mida-group.github.io/CytoBrowser/.

Cancer Informatics for Cancer Centers (CI4CC): Building a Community Focused on Sharing Ideas and Best Practices to Improve Cancer Care and Patient Outcomes

Cancer Informatics for Cancer Centers (CI4CC) is a grassroots, nonprofit 501c3 organization intended to provide a focused national forum for engagement of senior cancer informatics leaders, primarily aimed at academic cancer centers anywhere in the world but with a special emphasis on the 70 National Cancer Institute-funded cancer centers. Although each of the participating cancer centers is structured differently, and leaders' titles vary, we know firsthand there are similarities in both the issues we face and the solutions we achieve. As a consortium, we have initiated a dedicated listserv, an open-initiatives program, and targeted biannual face-to-face meetings. These meetings are a place to review our priorities and initiatives, providing a forum for discussion of the strategic and pragmatic issues we, as informatics leaders, individually face at our respective institutions and cancer centers. Here we provide a brief history of the CI4CC organization and meeting highlights from the latest CI4CC meeting that took place in Napa, California from October 14-16, 2019. The focus of this meeting was "intersections between informatics, data science, and population science." We conclude with a discussion on "hot topics" on the horizon for cancer informatics.

AI in Medical Imaging Informatics: Current Challenges and Future Directions

This paper reviews state-of-the-art research solutions across the spectrum of medical imaging informatics, discusses clinical translation, and provides future directions for advancing clinical practice. More specifically, it summarizes advances in medical imaging acquisition technologies for different modalities, highlighting the necessity for efficient medical data management strategies in the context of AI in big healthcare data analytics. It then provides a synopsis of contemporary and emerging algorithmic methods for disease classification and organ/ tissue segmentation, focusing on AI and deep learning architectures that have already become the de facto approach. The clinical benefits of in-silico modelling advances linked with evolving 3D reconstruction and visualization applications are further documented. Concluding, integrative analytics approaches driven by associate research branches highlighted in this study promise to revolutionize imaging informatics as known today across the healthcare continuum for both radiology and digital pathology applications. The latter, is projected to enable informed, more accurate diagnosis, timely prognosis, and effective treatment planning, underpinning precision medicine.

Utilizing Automated Breast Cancer Detection to Identify Spatial Distributions of Tumor-Infiltrating Lymphocytes in Invasive Breast Cancer

Quantitative assessment of spatial relations between tumor and tumor-infiltrating lymphocytes (TIL) is increasingly important in both basic science and clinical aspects of breast cancer research. We have developed and evaluated convolutional neural network analysis pipelines to generate combined maps of cancer regions and TILs in routine diagnostic breast cancer whole slide tissue images. The combined maps provide insight about the structural patterns and spatial distribution of lymphocytic infiltrates and facilitate improved quantification of TILs. Both tumor and TIL analyses were evaluated by using three convolutional neural network networks (34-layer ResNet, 16-layer VGG, and Inception v4); the results compared favorably with those obtained by using the best published methods. We have produced open-source tools and a public data set consisting of tumor/TIL maps for 1090 invasive breast cancer images from The Cancer Genome Atlas. The maps can be downloaded for further downstream analyses.

AIDEx - An Open-source Platform for Real-Time Forecasting Sepsis and A Case Study on Taking ML Algorithms to Production

Sepsis, a dysregulated immune response to infection, has been the leading cause of morbidity and mortality in critically ill patients. Multiple studies have demonstrated improved survival outcomes when early treatment is initiated for septic patients. In our previous work, we developed a real-time machine learning algorithm capable of predicting onset of sepsis four to six hours prior to clinical recognition. In this work, we develop AIDEx, an open-source platform that consumes data as FHIR resources. It is capable of consuming live patient data, securely transporting it into a cloud environment, and monitoring patients in real-time. We build AIDEx as an EHR vendor-agnostic open-source platform that can be easily deployed in clinical environments. Finally, the computation of the sepsis risk scores uses a common design pattern that is seen in streaming clinical informatics and predictive analytics applications. AIDEx provides a comprehensive case study in the design and development of a production-ready ML platform that integrates with Healthcare IT systems.

Dataset of segmented nuclei in hematoxylin and eosin stained histopathology images of ten cancer types

The distribution and appearance of nuclei are essential markers for the diagnosis and study of cancer. Despite the importance of nuclear morphology, there is a lack of large scale, accurate, publicly accessible nucleus segmentation data. To address this, we developed an analysis pipeline that segments nuclei in whole slide tissue images from multiple cancer types with a quality control process. We have generated nucleus segmentation results in 5,060 Whole Slide Tissue images from 10 cancer types in The Cancer Genome Atlas. One key component of our work is that we carried out a multi-level quality control process (WSI-level and image patch-level), to evaluate the quality of our segmentation results. The image patch-level quality control used manual segmentation ground truth data from 1,356 sampled image patches. The datasets we publish in this work consist of roughly 5 billion quality controlled nuclei from more than 5,060 TCGA WSIs from 10 different TCGA cancer types and 1,356 manually segmented TCGA image patches from the same 10 cancer types plus additional 4 cancer types.

MLCD: A Unified Software Package for Cancer Diagnosis

PURPOSE

Machine Learning Package for Cancer Diagnosis (MLCD) is the result of a National Institutes of Health/National Cancer Institute (NIH/NCI)-sponsored project for developing a unified software package from state-of-the-art breast cancer biopsy diagnosis and machine learning algorithms that can improve the quality of both clinical practice and ongoing research.

METHODS

Whole-slide images of 240 well-characterized breast biopsy cases, initially assembled under R01 CA140560, were used for developing the algorithms and training the machine learning models. This software package is based on the methodology developed and published under our recent NIH/NCI-sponsored research grant (R01 CA172343) for finding regions of interest (ROIs) in whole-slide breast biopsy images, for segmenting ROIs into histopathologic tissue types and for using this segmentation in classifiers that can suggest final diagnoses.

RESULT

The package provides an ROI detector for whole-slide images and modules for semantic segmentation into tissue classes and diagnostic classification into 4 classes (benign, atypia, ductal carcinoma in situ, invasive cancer) of the ROIs. It is available through the GitHub repository under the Massachusetts Institute of Technology license and will later be distributed with the Pathology Image Informatics Platform system. A Web page provides instructions for use.

CONCLUSION

Our tools have the potential to provide help to other cancer researchers and, ultimately, to practicing physicians and will motivate future research in this field. This article describes the methodology behind the software development and gives sample outputs to guide those interested in using this package.

Open access image repositories: high-quality data to enable machine learning research

Originally motivated by the need for research reproducibility and data reuse, large-scale, open access information repositories have become key resources for training and testing of advanced machine learning applications in biomedical and clinical research. To be of value, such repositories must provide large, high-quality data sets, where quality is defined as minimising variance due to data collection protocols and data misrepresentations. Curation is the key to quality. We have constructed a large public access image repository, The Cancer Imaging Archive, dedicated to the promotion of open science to advance the global effort to diagnose and treat cancer. Drawing on this experience and our experience in applying machine learning techniques to the analysis of radiology and pathology image data, we will review the requirements placed on such information repositories by state-of-the-art machine learning applications and how these requirements can be met.

Dual-Personality DICOM-TIFF for Whole Slide Images: A Migration Technique for Legacy Software

Despite recently organized Digital Imaging and Communications in Medicine (DICOM) testing and demonstration events involving numerous participating vendors, it is still the case that scanner manufacturers, software developers, and users continue to depend on proprietary file formats rather than adopting the standard DICOM whole slide microscopic image object. Many proprietary formats are Tagged Image File Format (TIFF) based, and existing applications and libraries can read tiled TIFF files. The sluggish adoption of DICOM for whole slide image encoding can be temporarily mitigated by the use of dual-personality DICOM-TIFF files. These are compatible with the installed base of TIFF-based software, as well as newer DICOM-based software. The DICOM file format was deliberately designed to support this dual-personality capability for such transitional situations, although it is rarely used. Furthermore, existing TIFF files can be converted into dual-personality DICOM-TIFF without changing the pixel data. This paper demonstrates the feasibility of extending the dual-personality concept to multiframe-tiled pyramidal whole slide images and explores the issues encountered. Open source code and sample converted images are provided for testing.

Twenty Years of Digital Pathology: An Overview of the Road Travelled, What is on the Horizon, and the Emergence of Vendor-Neutral Archives

Almost 20 years have passed since the commercial introduction of whole-slide imaging (WSI) scanners. During this time, the creation of various WSI devices with the ability to digitize an entire glass slide has transformed the field of pathology. Parallel advances in computational technology and storage have permitted rapid processing of large-scale WSI datasets. This article provides an overview of important past and present efforts related to WSI. An account of how the virtual microscope evolved from the need to visualize and manage satellite data for earth science applications is provided. The article also discusses important milestones beginning from the first WSI scanner designed by Bacus to the Food and Drug Administration approval of the first digital pathology system for primary diagnosis in surgical pathology. As pathology laboratories commit to going fully digitalize, the need has emerged to include WSIs into an enterprise-level vendor-neutral archive (VNA). The different types of VNAs available are reviewed as well as how best to implement them and how pathology can benefit from participating in this effort. Differences between traditional image algorithms that extract pixel-, object-, and semantic-level features versus deep learning methods are highlighted. The need for large-scale data management, analysis, and visualization in computational pathology is also addressed.

Spatial Organization and Molecular Correlation of Tumor-Infiltrating Lymphocytes Using Deep Learning on Pathology Images

Beyond sample curation and basic pathologic characterization, the digitized H&E-stained images of TCGA samples remain underutilized. To highlight this resource, we present mappings of tumor-infiltrating lymphocytes (TILs) based on H&E images from 13 TCGA tumor types. These TIL maps are derived through computational staining using a convolutional neural network trained to classify patches of images. Affinity propagation revealed local spatial structure in TIL patterns and correlation with overall survival. TIL map structural patterns were grouped using standard histopathological parameters. These patterns are enriched in particular T cell subpopulations derived from molecular measures. TIL densities and spatial structure were differentially enriched among tumor types, immune subtypes, and tumor molecular subtypes, implying that spatial infiltrate state could reflect particular tumor cell aberration states. Obtaining spatial lymphocytic patterns linked to the rich genomic characterization of TCGA samples demonstrates one use for the TCGA image archives with insights into the tumor-immune microenvironment.

PanCancer insights from The Cancer Genome Atlas: the pathologist's perspective

The Cancer Genome Atlas (TCGA) represents one of several international consortia dedicated to performing comprehensive genomic and epigenomic analyses of selected tumour types to advance our understanding of disease and provide an open-access resource for worldwide cancer research. Thirty-three tumour types (selected by histology or tissue of origin, to include both common and rare diseases), comprising >11 000 specimens, were subjected to DNA sequencing, copy number and methylation analysis, and transcriptomic, proteomic and histological evaluation. Each cancer type was analysed individually to identify tissue-specific alterations, and make correlations across different molecular platforms. The final dataset was then normalized and combined for the PanCancer Initiative, which seeks to identify commonalities across different cancer types or cells of origin/lineage, or within anatomically or morphologically related groups. An important resource generated along with the rich molecular studies is an extensive digital pathology slide archive, composed of frozen section tissue directly related to the tissues analysed as part of TCGA, and representative formalin-fixed paraffin-embedded, haematoxylin and eosin (H&E)-stained diagnostic slides. These H&E image resources have primarily been used to verify diagnoses and histological subtypes with some limited extraction of standard pathological variables such as mitotic activity, grade, and lymphocytic infiltrates. Largely overlooked is the richness of these scanned images for more sophisticated feature extraction approaches coupled with machine learning, and ultimately correlation with molecular features and clinical endpoints. Here, we document initial attempts to exploit TCGA imaging archives, and describe some of the tools, and the rapidly evolving image analysis/feature extraction landscape. Our hope is to inform, and ultimately inspire and challenge, the pathology and cancer research communities to exploit these imaging resources so that the full potential of this integral platform of TCGA can be used to complement and enhance the insightful integrated analyses from the genomic and epigenomic platforms. Copyright © 2017 Pathological Society of Great Britain and Ireland. Published by John Wiley & Sons, Ltd.