Developing the Quantitative Histopathology Image Ontology (QHIO): A case study using the hot spot detection problem

Modelling digital health data: The ExaMode ontology for computational pathology

Computational pathology can significantly benefit from ontologies to standardize the employed nomenclature and help with knowledge extraction processes for high-quality annotated image datasets. The end goal is to reach a shared model for digital pathology to overcome data variability and integration problems. Indeed, data annotation in such a specific domain is still an unsolved challenge and datasets cannot be steadily reused in diverse contexts due to heterogeneity issues of the adopted labels, multilingualism, and different clinical practices.

Material and methods

This paper presents the ExaMode ontology, modeling the histopathology process by considering 3 key cancer diseases (colon, cervical, and lung tumors) and celiac disease. The ExaMode ontology has been designed bottom-up in an iterative fashion with continuous feedback and validation from pathologists and clinicians. The ontology is organized into 5 semantic areas that defines an ontological template to model any disease of interest in histopathology.

Results

The ExaMode ontology is currently being used as a common semantic layer in: (i) an entity linking tool for the automatic annotation of medical records; (ii) a web-based collaborative annotation tool for histopathology text reports; and (iii) a software platform for building holistic solutions integrating multimodal histopathology data.

Discussion

The ontology ExaMode is a key means to store data in a graph database according to the RDF data model. The creation of an RDF dataset can help develop more accurate algorithms for image analysis, especially in the field of digital pathology. This approach allows for seamless data integration and a unified query access point, from which we can extract relevant clinical insights about the considered diseases using SPARQL queries.

An autofocus algorithm considering wavelength changes for large scale microscopic hyperspectral pathological imaging system

Microscopic hyperspectral imaging technology has been widely used to acquire pathological information of tissue sections. Autofocus is one of the most important steps in microscopic hyperspectral imaging systems to capture large scale or even whole slide images of pathological slides with high quality and high speed. However, there are quite few autofocus algorithm put forward for the microscopic hyperspectral imaging system. Therefore, this article proposes a Laplace operator based autofocus algorithm for microscopic hyperspectral imaging system which takes the influence of wavelength changes into consideration. Through the proposed algorithm, the focal length for each wavelength can be adjusted automatically to ensure that each single band image can be autofocused precisely with adaptive image sharpness evaluation method. In addition, to increase the capture speed, the relationship of wavelength and focal length is derived and the focal offsets among different single band images are calculated for pre-focusing. We have employed the proposed method on our own datasets and the experimental results show that it can capture large-scale microscopic hyperspectral pathology images with high precise.

Ontologies and Knowledge Graphs in Oncology Research

The complexity of cancer research stems from leaning on several biomedical disciplines for relevant sources of data, many of which are complex in their own right. A holistic view of cancer—which is critical for precision medicine approaches—hinges on integrating a variety of heterogeneous data sources under a cohesive knowledge model, a role which biomedical ontologies can fill. This study reviews the application of ontologies and knowledge graphs in cancer research. In total, our review encompasses 141 published works, which we categorized under 14 hierarchical categories according to their usage of ontologies and knowledge graphs. We also review the most commonly used ontologies and newly developed ones. Our review highlights the growing traction of ontologies in biomedical research in general, and cancer research in particular. Ontologies enable data accessibility, interoperability and integration, support data analysis, facilitate data interpretation and data mining, and more recently, with the emergence of the knowledge graph paradigm, support the application of Artificial Intelligence methods to unlock new knowledge from a holistic view of the available large volumes of heterogeneous data.

Integrating Image Analysis and Dental Radiography for Periodontal and Peri-Implant Diagnosis

The recent change in classification of periodontal and peri-implant diseases includes objective evaluation of intra-oral radiographs and quantification of bone loss for disease staging and grading. Assessment of the progression of periodontal disease requires deduction of bone loss longitudinally, and its interpretation as (1) a percentage in relation to tooth root and (2) as a function of the patient's age. Similarly, bone loss around dental implants, after accounting for initial remodeling, is central for determining diagnosis, severity, and progression of peri-implantitis. Bone gain secondary to periodontal regeneration can be measured using standardized dental radiographs and compared to baseline morphology to determine treatment success. Computational image analysis, including machine learning (ML), has the potential to develop and automate quantitative measures of tooth, implant, bone volumes, and predict disease progression. The developed algorithms need to be standardized while considering pre-analytic, analytic, and post-analytic factors for successful translation to clinic. This review will introduce image analysis and machine learning in the context of dental radiography, and expand on the potential for integration of image analysis for assisted diagnosis of periodontitis and peri-implantitis.

Expression profiles of GPR21, GPR39, GPR135, and GPR153 orphan receptors in different cancers

Orphan receptors have unknown endogenous ligands, are expressed in different tissues, and participate in various diseases such as diabetes, hypertension and cancer. We studied the expression profiles of GPR21, GPR39, GPR135 and GPR153 orphan receptors in several tumour tissues. Cervical, breast, skin, prostate, and astrocytoma tissues were analysed for orphan receptor gene expression using Real time PCR analysis. GPR39 is over-expressed in cervical and prostate cancer tissues, and GPR21 and GPR135 receptors are significantly decreased in cervical, breast, skin, prostate, and astrocytoma tissues, when compared with healthy human fibroblasts. In conclusion, GPR21 and GPR135 receptor gene expression is reduced in cancerous tissues. GPR39 may have a role in the development and evolution of cervical and prostate cancer. These data suggest these receptors may be alternative molecules for new diagnostic approaches, and the design of novel therapeutics against oncological pathologies.

Characterizing Immune Responses in Whole Slide Images of Cancer With Digital Pathology and Pathomics

Purpose of Review

Our goal is to show how readily available Pathomics tissue analytics can be used to study tumor immune interactions in cancer. We provide a brief overview of how Pathomics complements traditional histopathologic examination of cancer tissue samples. We highlight a novel Pathomics application, Tumor-TILs, that quantitatively measures and generates maps of tumor infiltrating lymphocytes in breast, pancreatic, and lung cancer by leveraging deep learning computer vision applications to perform automated analyses of whole slide images.

Recent Findings

Tumor-TIL maps have been generated to analyze WSIs from thousands of cases of breast, pancreatic, and lung cancer. We report the availability of these tools in an effort to promote collaborative research and motivate future development of ensemble Pathomics applications to discover novel biomarkers and perform a wide range of correlative clinicopathologic research in cancer immunopathology and beyond.

Summary

Tumor immune interactions in cancer are a fascinating aspect of cancer pathobiology with particular significance due to the emergence of immunotherapy. We present simple yet powerful specialized Pathomics methods that serve as powerful clinical research tools and potential standalone clinical screening tests to predict clinical outcomes and treatment responses for precision medicine applications in immunotherapy.

Digital pathology and computational image analysis in nephropathology

The emergence of digital pathology - an image-based environment for the acquisition, management and interpretation of pathology information supported by computational techniques for data extraction and analysis - is changing the pathology ecosystem. In particular, by virtue of our new-found ability to generate and curate digital libraries, the field of machine vision can now be effectively applied to histopathological subject matter by individuals who do not have deep expertise in machine vision techniques. Although these novel approaches have already advanced the detection, classification, and prognostication of diseases in the fields of radiology and oncology, renal pathology is just entering the digital era, with the establishment of consortia and digital pathology repositories for the collection, analysis and integration of pathology data with other domains. The development of machine-learning approaches for the extraction of information from image data, allows for tissue interrogation in a way that was not previously possible. The application of these novel tools are placing pathology centre stage in the process of defining new, integrated, biologically and clinically homogeneous disease categories, to identify patients at risk of progression, and shifting current paradigms for the treatment and prevention of kidney diseases.

COVID-19 Biomarkers in research: Extension of the OncoMX cancer biomarker data model to capture biomarker data from other diseases

Scientists, medical researchers, and health care workers have mobilized worldwide in response to the outbreak of COVID-19, caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2; SCoV2). Preliminary data have captured a wide range of host responses, symptoms, and lingering problems post-recovery within the human population. These variable clinical manifestations suggest differences in influential factors, such as innate and adaptive host immunity, existing or underlying health conditions, co-morbidities, genetics, and other factors. As COVID-19-related data continue to accumulate from disparate groups, the heterogeneous nature of these datasets poses challenges for efficient extrapolation of meaningful observations, hindering translation of information into clinical applications. Attempts to utilize, analyze, or combine biomarker datasets from multiple sources have shown to be inefficient and complicated, without a unifying resource. As such, there is an urgent need within the research community for the rapid development of an integrated and harmonized COVID-19 Biomarker Knowledgebase. By leveraging data collection and integration methods, backed by a robust data model developed to capture cancer biomarker data we have rapidly crowdsourced the collection and harmonization of COVID-19 biomarkers. Our resource currently has 138 unique biomarkers. We found multiple instances of the same biomarker substance being suggested as multiple biomarker types during our extensive cross-validation and manual curation. As a result, our Knowledgebase currently has 265 biomarker type combinations. Every biomarker entry is made comprehensive by bringing in together ancillary data from multiple sources such as biomarker accessions (canonical UniProtKB accession, PubChem Compound ID, Cell Ontology ID, Protein Ontology ID, NCI Thesaurus Code, and Disease Ontology ID), BEST biomarker category, and specimen type (Uberon Anatomy Ontology) unified with ontology standards. Our preliminary observations show distinct trends in the collated biomarkers. Most biomarkers are related to the immune system (SAA,TNF-∝, and IP-10) or coagulopathies (D-dimer, antithrombin, and VWF) and a few have already been established as cancer biomarkers (ACE2, IL-6, IL-4 and IL-2). These trends align with proposed hypotheses of clinical manifestations compounding the complexity of COVID-19 pathobiology. We explore these trends as we put forth a COVID-19 biomarker resource that will help researchers and diagnosticians alike. All biomarker data are freely available from https://data.oncomx.org/covid19 .

Deep learning-based image analysis methods for brightfield-acquired multiplex immunohistochemistry images

BACKGROUND

Multiplex immunohistochemistry (mIHC) permits the labeling of six or more distinct cell types within a single histologic tissue section. The classification of each cell type requires detection of the unique colored chromogens localized to cells expressing biomarkers of interest. The most comprehensive and reproducible method to evaluate such slides is to employ digital pathology and image analysis pipelines to whole-slide images (WSIs). Our suite of deep learning tools quantitatively evaluates the expression of six biomarkers in mIHC WSIs. These methods address the current lack of readily available methods to evaluate more than four biomarkers and circumvent the need for specialized instrumentation to spectrally separate different colors. The use case application for our methods is a study that investigates tumor immune interactions in pancreatic ductal adenocarcinoma (PDAC) with a customized mIHC panel.

METHODS

Six different colored chromogens were utilized to label T-cells (CD3, CD4, CD8), B-cells (CD20), macrophages (CD16), and tumor cells (K17) in formalin-fixed paraffin-embedded (FFPE) PDAC tissue sections. We leveraged pathologist annotations to develop complementary deep learning-based methods: (1) ColorAE is a deep autoencoder which segments stained objects based on color; (2) U-Net is a convolutional neural network (CNN) trained to segment cells based on color, texture and shape; and ensemble methods that employ both ColorAE and U-Net, collectively referred to as (3) ColorAE:U-Net. We assessed the performance of our methods using: structural similarity and DICE score to evaluate segmentation results of ColorAE against traditional color deconvolution; F1 score, sensitivity, positive predictive value, and DICE score to evaluate the predictions from ColorAE, U-Net, and ColorAE:U-Net ensemble methods against pathologist-generated ground truth. We then used prediction results for spatial analysis (nearest neighbor).

RESULTS

We observed that (1) the performance of ColorAE is comparable to traditional color deconvolution for single-stain IHC images (note: traditional color deconvolution cannot be used for mIHC); (2) ColorAE and U-Net are complementary methods that detect 6 different classes of cells with comparable performance; (3) combinations of ColorAE and U-Net into ensemble methods outperform using either ColorAE and U-Net alone; and (4) ColorAE:U-Net ensemble methods can be employed for detailed analysis of the tumor microenvironment (TME). We developed a suite of scalable deep learning methods to analyze 6 distinctly labeled cell populations in mIHC WSIs. We evaluated our methods and found that they reliably detected and classified cells in the PDAC tumor microenvironment. We also present a use case, wherein we apply the ColorAE:U-Net ensemble method across 3 mIHC WSIs and use the predictions to quantify all stained cell populations and perform nearest neighbor spatial analysis. Thus, we provide proof of concept that these methods can be employed to quantitatively describe the spatial distribution immune cells within the tumor microenvironment. These complementary deep learning methods are readily deployable for use in clinical research studies.

Ontologies for Liver Diseases Representation: A Systematic Literature Review

Ontology, as a useful knowledge engineering technique, has been widely used for reducing ambiguity and helping with information sharing. It is considered originally to be clear, comprehensive, and with well-defined format. It characterizes several domains purposes description through structured and formalized languages. In various areas of research, it has become a significant way to realize successful and powerful accomplishments. Actually, medical ontologies were turned into an efficient application in medical domains. They also become a relevant approach to process large medical data volumes. Consequently, they are behaving as a support decision system in some cases. Also, they ensure diagnosis process acceleration and assistance. Additionally, they have been integrated especially to represent human healthcare concepts. For that reason, plenty of research works applied ontologies to design and treat liver diseases. In this article, we present a general overview of medical ontologies to stand for this type of disease. We expose and discuss these works in details by a complete comparison. Also, we show their performance to arrange clinical data and extract results.

Annotations, Ontologies, and Whole Slide Images - Development of an Annotated Ontology-Driven Whole Slide Image Library of Normal and Abnormal Human Tissue

Objective:

Digital pathology is today a widely used technology, and the digitalization of microscopic slides into whole slide images (WSIs) allows the use of machine learning algorithms as a tool in the diagnostic process. In recent years, “deep learning” algorithms for image analysis have been applied to digital pathology with great success. The training of these algorithms requires a large volume of high-quality images and image annotations. These large image collections are a potent source of information, and to use and share the information, standardization of the content through a consistent terminology is essential. The aim of this project was to develop a pilot dataset of exhaustive annotated WSI of normal and abnormal human tissue and link the annotations to appropriate ontological information.

Materials and Methods:

Several biomedical ontologies and controlled vocabularies were investigated with the aim of selecting the most suitable ontology for this project. The selection criteria required an ontology that covered anatomical locations, histological subcompartments, histopathologic diagnoses, histopathologic terms, and generic terms such as normal, abnormal, and artifact. WSIs of normal and abnormal tissue from 50 colon resections and 69 skin excisions, diagnosed 2015-2016 at the Department of Clinical Pathology in Linköping, were randomly collected. These images were manually and exhaustively annotated at the level of major subcompartments, including normal or abnormal findings and artifacts.

Results:

Systemized nomenclature of medicine clinical terms (SNOMED CT) was chosen, and the annotations were linked to its codes and terms. Two hundred WSI were collected and annotated, resulting in 17,497 annotations, covering a total area of 302.19 cm², equivalent to 107,7 gigapixels. Ninety-five unique SNOMED CT codes were used. The time taken to annotate a WSI varied from 45 s to over 360 min, a total time of approximately 360 h.

Conclusion:

This work resulted in a dataset of 200 exhaustive annotated WSIs of normal and abnormal tissue from the colon and skin, and it has informed plans to build a comprehensive library of annotated WSIs. SNOMED CT was found to be the best ontology for annotation labeling. This project also demonstrates the need for future development of annotation tools in order to make the annotation process more efficient.

An Image Analysis Resource for Cancer Research: PIIP-Pathology Image Informatics Platform for Visualization, Analysis, and Management

Pathology Image Informatics Platform (PIIP) is an NCI/NIH sponsored project intended for managing, annotating, sharing, and quantitatively analyzing digital pathology imaging data. It expands on an existing, freely available pathology image viewer, Sedeen. The goal of this project is to develop and embed some commonly used image analysis applications into the Sedeen viewer to create a freely available resource for the digital pathology and cancer research communities. Thus far, new plugins have been developed and incorporated into the platform for out of focus detection, region of interest transformation, and IHC slide analysis. Our biomarker quantification and nuclear segmentation algorithms, written in MATLAB, have also been integrated into the viewer. This article describes the viewing software and the mechanism to extend functionality by plugins, brief descriptions of which are provided as examples, to guide users who want to use this platform. PIIP project materials, including a video describing its usage and applications, and links for the Sedeen Viewer, plug-ins, and user manuals are freely available through the project web page: http://pathiip.org Cancer Res; 77(21); e83-86. ©2017 AACR.