Recommendations for Performance Evaluation of Machine Learning in Pathology: A Concept Paper From the College of American Pathologists

CONTEXT.—

Machine learning applications in the pathology clinical domain are emerging rapidly. As decision support systems continue to mature, laboratories will increasingly need guidance to evaluate their performance in clinical practice. Currently there are no formal guidelines to assist pathology laboratories in verification and/or validation of such systems. These recommendations are being proposed for the evaluation of machine learning systems in the clinical practice of pathology.

OBJECTIVE.—

To propose recommendations for performance evaluation of in vitro diagnostic tests on patient samples that incorporate machine learning as part of the preanalytical, analytical, or postanalytical phases of the laboratory workflow. Topics described include considerations for machine learning model evaluation including risk assessment, predeployment requirements, data sourcing and curation, verification and validation, change control management, human-computer interaction, practitioner training, and competency evaluation.

DATA SOURCES.—

An expert panel performed a review of the literature, Clinical and Laboratory Standards Institute guidance, and laboratory and government regulatory frameworks.

CONCLUSIONS.—

Review of the literature and existing documents enabled the development of proposed recommendations. This white paper pertains to performance evaluation of machine learning systems intended to be implemented for clinical patient testing. Further studies with real-world clinical data are encouraged to support these proposed recommendations. Performance evaluation of machine learning models is critical to verification and/or validation of in vitro diagnostic tests using machine learning intended for clinical practice.

The NCI Imaging Data Commons as a platform for reproducible research in computational pathology

BACKGROUND AND OBJECTIVES

Reproducibility is a major challenge in developing machine learning (ML)-based solutions in computational pathology (CompPath). The NCI Imaging Data Commons (IDC) provides >120 cancer image collections according to the FAIR principles and is designed to be used with cloud ML services. Here, we explore its potential to facilitate reproducibility in CompPath research.

METHODS

Using the IDC, we implemented two experiments in which a representative ML-based method for classifying lung tumor tissue was trained and/or evaluated on different datasets. To assess reproducibility, the experiments were run multiple times with separate but identically configured instances of common ML services.

RESULTS

The results of different runs of the same experiment were reproducible to a large extent. However, we observed occasional, small variations in AUC values, indicating a practical limit to reproducibility.

CONCLUSIONS

We conclude that the IDC facilitates approaching the reproducibility limit of CompPath research (i) by enabling researchers to reuse exactly the same datasets and (ii) by integrating with cloud ML services so that experiments can be run in identically configured computing environments.

National Cancer Institute Imaging Data Commons: Toward Transparency, Reproducibility, and Scalability in Imaging Artificial Intelligence

The remarkable advances of artificial intelligence (AI) technology are revolutionizing established approaches to the acquisition, interpretation, and analysis of biomedical imaging data. Development, validation, and continuous refinement of AI tools requires easy access to large high-quality annotated datasets, which are both representative and diverse. The National Cancer Institute (NCI) Imaging Data Commons (IDC) hosts large and diverse publicly available cancer image data collections. By harmonizing all data based on industry standards and colocalizing it with analysis and exploration resources, the IDC aims to facilitate the development, validation, and clinical translation of AI tools and address the well-documented challenges of establishing reproducible and transparent AI processing pipelines. Balanced use of established commercial products with open-source solutions, interconnected by standard interfaces, provides value and performance, while preserving sufficient agility to address the evolving needs of the research community. Emphasis on the development of tools, use cases to demonstrate the utility of uniform data representation, and cloud-based analysis aim to ease adoption and help define best practices. Integration with other data in the broader NCI Cancer Research Data Commons infrastructure opens opportunities for multiomics studies incorporating imaging data to further empower the research community to accelerate breakthroughs in cancer detection, diagnosis, and treatment. Published under a CC BY 4.0 license.

Interoperable slide microscopy viewer and annotation tool for imaging data science and computational pathology

The exchange of large and complex slide microscopy imaging data in biomedical research and pathology practice is impeded by a lack of data standardization and interoperability, which is detrimental to the reproducibility of scientific findings and clinical integration of technological innovations. We introduce Slim, an open-source, web-based slide microscopy viewer that implements the internationally accepted Digital Imaging and Communications in Medicine (DICOM) standard to achieve interoperability with a multitude of existing medical imaging systems. We showcase the capabilities of Slim as the slide microscopy viewer of the NCI Imaging Data Commons and demonstrate how the viewer enables interactive visualization of traditional brightfield microscopy and highly-multiplexed immunofluorescence microscopy images from The Cancer Genome Atlas and Human Tissue Atlas Network, respectively, using standard DICOMweb services. We further show how Slim enables the collection of standardized image annotations for the development or validation of machine learning models and the visual interpretation of model inference results in the form of segmentation masks, spatial heat maps, or image-derived measurements.

Volumetric Tissue Imaging of Surgical Tissue Specimens Using Micro-Computed Tomography: An Emerging Digital Pathology Modality for Nondestructive, Slide-Free Microscopy-Clinical Applications of Digital Pathology in 3 Dimensions

OBJECTIVES

Micro-computed tomography (micro-CT) is a novel, nondestructive, slide-free digital imaging modality that enables the acquisition of high-resolution, volumetric images of intact surgical tissue specimens. The aim of this systematic mapping review is to provide a comprehensive overview of the available literature on clinical applications of micro-CT tissue imaging and to assess its relevance and readiness for pathology practice.

METHODS

A computerized literature search was performed in the PubMed, Scopus, Web of Science, and CENTRAL databases. To gain insight into regulatory and financial considerations for performing and examining micro-CT imaging procedures in a clinical setting, additional searches were performed in medical device databases.

RESULTS

Our search identified 141 scientific articles published between 2000 and 2021 that described clinical applications of micro-CT tissue imaging. The number of relevant publications is progressively increasing, with the specialties of pulmonology, cardiology, otolaryngology, and oncology being most commonly concerned. The included studies were mostly performed in pathology departments. Current micro-CT devices have already been cleared for clinical use, and a Current Procedural Terminology (CPT) code exists for reimbursement of micro-CT imaging procedures.

CONCLUSIONS

Micro-CT tissue imaging enables accurate volumetric measurements and evaluations of entire surgical specimens at microscopic resolution across a wide range of clinical applications.

Computational pathology in 2030: a Delphi study forecasting the role of AI in pathology within the next decade

BACKGROUND

Artificial intelligence (AI) is rapidly fuelling a fundamental transformation in the practice of pathology. However, clinical integration remains challenging, with no AI algorithms to date in routine adoption within typical anatomic pathology (AP) laboratories. This survey gathered current expert perspectives and expectations regarding the role of AI in AP from those with first-hand computational pathology and AI experience.

METHODS

Perspectives were solicited using the Delphi method from 24 subject matter experts between December 2020 and February 2021 regarding the anticipated role of AI in pathology by the year 2030. The study consisted of three consecutive rounds: 1) an open-ended, free response questionnaire generating a list of survey items; 2) a Likert-scale survey scored by experts and analysed for consensus; and 3) a repeat survey of items not reaching consensus to obtain further expert consensus.

FINDINGS

Consensus opinions were reached on 141 of 180 survey items (78.3%). Experts agreed that AI would be routinely and impactfully used within AP laboratory and pathologist clinical workflows by 2030. High consensus was reached on 100 items across nine categories encompassing the impact of AI on (1) pathology key performance indicators (KPIs) and (2) the pathology workforce and specific tasks performed by (3) pathologists and (4) AP lab technicians, as well as (5) specific AI applications and their likelihood of routine use by 2030, (6) AI's role in integrated diagnostics, (7) pathology tasks likely to be fully automated using AI, and (8) regulatory/legal and (9) ethical aspects of AI integration in pathology.

INTERPRETATION

This systematic consensus study details the expected short-to-mid-term impact of AI on pathology practice. These findings provide timely and relevant information regarding future care delivery in pathology and raise key practical, ethical, and legal challenges that must be addressed prior to AI's successful clinical implementation.

FUNDING

No specific funding was provided for this study.

MITI minimum information guidelines for highly multiplexed tissue images

The imminent release of tissue atlases combining multi-channel microscopy with single cell sequencing and other omics data from normal and diseased specimens creates an urgent need for data and metadata standards that guide data deposition, curation and release. We describe a Minimum Information about highly multiplexed Tissue Imaging (MITI) standard that applies best practices developed for genomics and other microscopy data to highly multiplexed tissue images and traditional histology.

A Pathologist-Annotated Dataset for Validating Artificial Intelligence: A Project Description and Pilot Study

Purpose:

Validating artificial intelligence algorithms for clinical use in medical images is a challenging endeavor due to a lack of standard reference data (ground truth). This topic typically occupies a small portion of the discussion in research papers since most of the efforts are focused on developing novel algorithms. In this work, we present a collaboration to create a validation dataset of pathologist annotations for algorithms that process whole slide images. We focus on data collection and evaluation of algorithm performance in the context of estimating the density of stromal tumor-infiltrating lymphocytes (sTILs) in breast cancer.

Methods:

We digitized 64 glass slides of hematoxylin- and eosin-stained invasive ductal carcinoma core biopsies prepared at a single clinical site. A collaborating pathologist selected 10 regions of interest (ROIs) per slide for evaluation. We created training materials and workflows to crowdsource pathologist image annotations on two modes: an optical microscope and two digital platforms. The microscope platform allows the same ROIs to be evaluated in both modes. The workflows collect the ROI type, a decision on whether the ROI is appropriate for estimating the density of sTILs, and if appropriate, the sTIL density value for that ROI.

Results:

In total, 19 pathologists made 1645 ROI evaluations during a data collection event and the following 2 weeks. The pilot study yielded an abundant number of cases with nominal sTIL infiltration. Furthermore, we found that the sTIL densities are correlated within a case, and there is notable pathologist variability. Consequently, we outline plans to improve our ROI and case sampling methods. We also outline statistical methods to account for ROI correlations within a case and pathologist variability when validating an algorithm.

Conclusion:

We have built workflows for efficient data collection and tested them in a pilot study. As we prepare for pivotal studies, we will investigate methods to use the dataset as an external validation tool for algorithms. We will also consider what it will take for the dataset to be fit for a regulatory purpose: study size, patient population, and pathologist training and qualifications. To this end, we will elicit feedback from the Food and Drug Administration via the Medical Device Development Tool program and from the broader digital pathology and AI community. Ultimately, we intend to share the dataset, statistical methods, and lessons learned.

Volumetric Imaging of Lung Tissue at Micrometer Resolution: Clinical Applications of Micro-CT for the Diagnosis of Pulmonary Diseases

Micro-computed tomography (micro-CT) is a promising novel medical imaging modality that allows for non-destructive volumetric imaging of surgical tissue specimens at high spatial resolution. The aim of this study is to provide a comprehensive assessment of the clinical applications of micro-CT for the tissue-based diagnosis of lung diseases. This scoping review was conducted in accordance with the PRISMA Extension for Scoping Reviews, aiming to include every clinical study reporting on micro-CT imaging of human lung tissues. A literature search yielded 570 candidate articles, out of which 37 were finally included in the review. Of the selected studies, 9 studies explored via micro-CT imaging the morphology and anatomy of normal human lung tissue; 21 studies investigated microanatomic pulmonary alterations due to obstructive or restrictive lung diseases, such as chronic obstructive pulmonary disease, idiopathic pulmonary fibrosis, and cystic fibrosis; and 7 studies examined the utility of micro-CT imaging in assessing lung cancer lesions (n = 4) or in transplantation-related pulmonary alterations (n = 3). The selected studies reported that micro-CT could successfully detect several lung diseases providing three-dimensional images of greater detail and resolution than routine optical slide microscopy, and could additionally provide valuable volumetric insight in both restrictive and obstructive lung diseases. In conclusion, micro-CT-based volumetric measurements and qualitative evaluations of pulmonary tissue structures can be utilized for the clinical management of a variety of lung diseases. With micro-CT devices becoming more accessible, the technology has the potential to establish itself as a core diagnostic imaging modality in pathology and to enable integrated histopathologic and radiologic assessment of lung cancer and other lung diseases.

Quantitative Imaging Informatics for Cancer Research

PURPOSE

We summarize Quantitative Imaging Informatics for Cancer Research (QIICR; U24 CA180918), one of the first projects funded by the National Cancer Institute (NCI) Informatics Technology for Cancer Research program.

METHODS

QIICR was motivated by the 3 use cases from the NCI Quantitative Imaging Network. 3D Slicer was selected as the platform for implementation of open-source quantitative imaging (QI) tools. Digital Imaging and Communications in Medicine (DICOM) was chosen for standardization of QI analysis outputs. Support of improved integration with community repositories focused on The Cancer Imaging Archive (TCIA). Priorities included improved capabilities of the standard, toolkits and tools, reference datasets, collaborations, and training and outreach.

RESULTS

Fourteen new tools to support head and neck cancer, glioblastoma, and prostate cancer QI research were introduced and downloaded over 100,000 times. DICOM was amended, with over 40 correction proposals addressing QI needs. Reference implementations of the standard in a popular toolkit and standalone tools were introduced. Eight datasets exemplifying the application of the standard and tools were contributed. An open demonstration/connectathon was organized, attracting the participation of academic groups and commercial vendors. Integration of tools with TCIA was improved by implementing programmatic communication interface and by refining best practices for QI analysis results curation.

CONCLUSION

Tools, capabilities of the DICOM standard, and datasets we introduced found adoption and utility within the cancer imaging community. A collaborative approach is critical to addressing challenges in imaging informatics at the national and international levels. Numerous challenges remain in establishing and maintaining the infrastructure of analysis tools and standardized datasets for the imaging community. Ideas and technology developed by the QIICR project are contributing to the NCI Imaging Data Commons currently being developed.

NCI Imaging Data Commons

The National Cancer Institute (NCI) Cancer Research Data Commons (CRDC) aims to establish a national cloud-based data science infrastructure. Imaging Data Commons (IDC) is a new component of CRDC supported by the Cancer Moonshot. The goal of IDC is to enable a broad spectrum of cancer researchers, with and without imaging expertise, to easily access and explore the value of deidentified imaging data and to support integrated analyses with nonimaging data. We achieve this goal by colocating versatile imaging collections with cloud-based computing resources and data exploration, visualization, and analysis tools. The IDC pilot was released in October 2020 and is being continuously populated with radiology and histopathology collections. IDC provides access to curated imaging collections, accompanied by documentation, a user forum, and a growing number of analysis use cases that aim to demonstrate the value of a data commons framework applied to cancer imaging research. SIGNIFICANCE: This study introduces NCI Imaging Data Commons, a new repository of the NCI Cancer Research Data Commons, which will support cancer imaging research on the cloud.

Current clinical applications and potential perspective of micro-computed tomography in cardiovascular imaging: A systematic scoping review

Micro-computed tomography (micro-CT) constitutes an emerging imaging technique, which can be utilized in cardiovascular medicine to study in-detail the microstructure of heart and vessels. This paper aims to systematically review the clinical utility of micro-CT in cardiovascular imaging and propose future applications of micro-CT imaging in cardiovascular research. A systematic scoping review was conducted by searching for original studies written in English according to Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) extension for scoping reviews. Medline, Scopus, ClinicalTrials.gov, and the Cochrane library were systematically searched through December 11, 2020 to identify publications concerning micro-CT applications in cardiovascular imaging. Preclinical-animal studies and case reports were excluded. The Newcastle-Ottawa assessment scale for observational studies was used to evaluate study quality. In total, the search strategy identified 30 studies that report on micro-CT-based cardiovascular imaging and satisfy our eligibility criteria. Across all included studies, the total number of micro-CT scanned specimens was 1,227. Six studies involved postmortem 3D-reconstruction of congenital heart defects, while eleven studies described atherosclerotic vessel (coronary or carotid) characteristics. Thirteen other studies employed micro-CT for the assessment of medical devices (mainly stents or prosthetic valves). In conclusion, micro-CT is a novel imaging modality, effectively adapted for the 3D visualization and analysis of cardiac soft tissues and devices at high spatial resolution. Its increasing use could make significant contributions to our improved understanding of the histopathophysiology of cardiovascular diseases, and, thus, has the potential to optimize interventional procedures and technologies, and ultimately improve patient outcomes.

Integrating the Health-care Enterprise Pathology and Laboratory Medicine Guideline for Digital Pathology Interoperability

Integrating the health-care enterprise (IHE) is an international initiative to promote the use of standards to achieve interoperability among health information technology systems. The Pathology and Laboratory Medicine domain within IHE has brought together subject matter experts, electronic health record vendors, and digital imaging vendors, to initiate development of a series of digital pathology interoperability guidelines, called “integration profiles” within IHE. This effort begins with documentation of common use cases, followed by identification of available data and technology standards best utilized to achieve those use cases. An integration profile that describes the information flow and technology interactions is then published for trial use. Real world testing occurs in “connectathon” events, in which multiple vendors attempt to connect their products following the interoperability guidance parameters set forth in the profile. This paper describes the overarching set of integration profiles, one of which has been published, to support key digital pathology use cases.

[Technical, operational, and regulatory considerations for the adoption of digital and computational pathology]

BACKGROUND

Innovative information technologies open new possibilities for diagnostics and promise to improve patient care. However, the integration of data- and computing-intensive procedures into diagnostic workflows also poses risks and considerable challenges for pathologists.

OBJECTIVES

Considering technical, operational, and regulatory aspects, we present a holistic and systematic approach for the adoption of digital and computational pathology.

MATERIAL AND METHODS

We discuss challenges for the implementation of computational diagnostic procedures and analyze regulatory frameworks for risk-based assessment and monitoring of software as an in vitro diagnostic device. Applying regulatory science, we develop an approach to streamline adoption of digital workflows in pathology.

RESULTS

Data- and computing-intensive workflows in digital pathology are complex and underscore the need for computational and regulatory science as a central part of pathological diagnostics. To promote the adoption of computational diagnostics, we have founded an interdisciplinary initiative (the Alliance) that focuses on regulatory research in the field of digital pathology and works closely with a number of expert and interest groups on the precompetitive development of standards for computational workflows.

DISCUSSION

The inclusion of different stakeholder groups and the coordination of technical, operational, and regulatory aspects is necessary to maintain the balance between progress and safety in diagnostics and to make innovations quickly and safely available for patient care.

A Regulatory Science Initiative to Harmonize and Standardize Digital Pathology and Machine Learning Processes to Speed up Clinical Innovation to Patients

Unlocking the full potential of pathology data by gaining computational access to histological pixel data and metadata (digital pathology) is one of the key promises of computational pathology. Despite scientific progress and several regulatory approvals for primary diagnosis using whole-slide imaging, true clinical adoption at scale is slower than anticipated. In the U.S., advances in digital pathology are often siloed pursuits by individual stakeholders, and to our knowledge, there has not been a systematic approach to advance the field through a regulatory science initiative. The Alliance for Digital Pathology (the Alliance) is a recently established, volunteer, collaborative, regulatory science initiative to standardize digital pathology processes to speed up innovation to patients. The purpose is: (1) to account for the patient perspective by including patient advocacy; (2) to investigate and develop methods and tools for the evaluation of effectiveness, safety, and quality to specify risks and benefits in the precompetitive phase; (3) to help strategize the sequence of clinically meaningful deliverables; (4) to encourage and streamline the development of ground-truth data sets for machine learning model development and validation; and (5) to clarify regulatory pathways by investigating relevant regulatory science questions. The Alliance accepts participation from all stakeholders, and we solicit clinically relevant proposals that will benefit the field at large. The initiative will dissolve once a clinical, interoperable, modularized, integrated solution (from tissue acquisition to diagnostic algorithm) has been implemented. In times of rapidly evolving discoveries, scientific input from subject-matter experts is one essential element to inform regulatory guidance and decision-making. The Alliance aims to establish and promote synergistic regulatory science efforts that will leverage diverse inputs to move digital pathology forward and ultimately improve patient care.

Implementing the DICOM Standard for Digital Pathology

Background:

Digital Imaging and Communications in Medicine (DICOM^®) is the standard for the representation, storage, and communication of medical images and related information. A DICOM file format and communication protocol for pathology have been defined; however, adoption by vendors and in the field is pending. Here, we implemented the essential aspects of the standard and assessed its capabilities and limitations in a multisite, multivendor healthcare network.

Methods:

We selected relevant DICOM attributes, developed a program that extracts pixel data and pixel-related metadata, integrated patient and specimen-related metadata, populated and encoded DICOM attributes, and stored DICOM files. We generated the files using image data from four vendor-specific image file formats and clinical metadata from two departments with different laboratory information systems. We validated the generated DICOM files using recognized DICOM validation tools and measured encoding, storage, and access efficiency for three image compression methods. Finally, we evaluated storing, querying, and retrieving data over the web using existing DICOM archive software.

Results:

Whole slide image data can be encoded together with relevant patient and specimen-related metadata as DICOM objects. These objects can be accessed efficiently from files or through RESTful web services using existing software implementations. Performance measurements show that the choice of image compression method has a major impact on data access efficiency. For lossy compression, JPEG achieves the fastest compression/decompression rates. For lossless compression, JPEG-LS significantly outperforms JPEG 2000 with respect to data encoding and decoding speed.

Conclusion:

Implementation of DICOM allows efficient access to image data as well as associated metadata. By leveraging a wealth of existing infrastructure solutions, the use of DICOM facilitates enterprise integration and data exchange for digital pathology.

Multiplexed protein maps link subcellular organization to cellular states

Obtaining highly multiplexed protein measurements across multiple length scales has enormous potential for biomedicine. Here, we measured, by iterative indirect immunofluorescence imaging (4i), 40-plex protein readouts from biological samples at high-throughput from the millimeter to the nanometer scale. This approach simultaneously captures properties apparent at the population, cellular, and subcellular levels, including microenvironment, cell shape, and cell cycle state. It also captures the detailed morphology of organelles, cytoskeletal structures, nuclear subcompartments, and the fate of signaling receptors in thousands of single cells in situ. We used computer vision and systems biology approaches to achieve unsupervised comprehensive quantification of protein subcompartmentalization within various multicellular, cellular, and pharmacological contexts. Thus, highly multiplexed subcellular protein maps can be used to identify functionally relevant single-cell states.