How to measure image quality in tissue-based diagnosis (diagnostic surgical pathology)

Pixelwise H-score: A novel digital image analysis-based metric to quantify membrane biomarker expression from immunohistochemistry images

Immunohistochemistry (IHC) assays play a central role in evaluating biomarker expression in tissue sections for diagnostic and research applications. Manual scoring of IHC images, which is the current standard of practice, is known to have several shortcomings in terms of reproducibility and scalability to large scale studies. Here, by using a digital image analysis-based approach, we introduce a new metric called the pixelwise H-score (pix H-score) that quantifies biomarker expression from whole-slide scanned IHC images. The pix H-score is an unsupervised algorithm that only requires the specification of intensity thresholds for the biomarker and the nuclear-counterstain channels. We present the detailed implementation of the pix H-score in two different whole-slide image analysis software packages Visiopharm and HALO. We consider three biomarkers P-cadherin, PD-L1, and 5T4, and show how the pix H-score exhibits tight concordance to multiple orthogonal measurements of biomarker abundance such as the biomarker mRNA transcript and the pathologist H-score. We also compare the pix H-score to existing automated image analysis algorithms and demonstrate that the pix H-score provides either comparable or significantly better performance over these methodologies. We also present results of an empirical resampling approach to assess the performance of the pix H-score in estimating biomarker abundance from select regions within the tumor tissue relative to the whole tumor resection. We anticipate that the new metric will be broadly applicable to quantify biomarker expression from a wide variety of IHC images. Moreover, these results underscore the benefit of digital image analysis-based approaches which offer an objective, reproducible, and highly scalable strategy to quantitatively analyze IHC images.

The Effect of Quality Control on Accuracy of Digital Pathology Image Analysis

Digital slide images produced from routine diagnostic histopathological preparations suffer from variation arising at every step of the processing pipeline. Typically, pathologists compensate for such variation using expert knowledge and experience, which is difficult to replicate in automated solutions. The extent to which inconsistencies affect image analysis is explored in this work, examining in detail, the results from a previously published algorithm automating the generation of tumor:stroma ratio (TSR) in colorectal clinical trial datasets. One dataset consisting of 2,211 cases and 106,268 expert-labelled images is used to identify quality issues, by visually inspecting cases where algorithm-pathologist agreement is lowest. Twelve categories are identified and used to analyze pathologist-algorithm agreement in relation to these categories. Of the 2,211 cases, 701 were found to be free from any image quality issues. Algorithm performance was then assessed, comparing pathologist agreement with image quality classification. It was found that agreement was lowest on poorly differentiated tissue, with a mean TSR difference of 0.25 (sd = 0.24). Removing images that contained quality issues increased accuracy from 80% to 83%, at the expense of reducing the dataset to 33,736 images (32%). Training the algorithm on the optimized dataset, prior to testing on all images saw a decrease in accuracy of 4%, indicating that the optimized dataset did not contain enough variation to generate a fully representative model. The results provide an in-depth perspective on image quality, highlighting the importance of the effects on downstream image analysis.

An automatic nuclei segmentation method based on deep convolutional neural networks for histopathology images

BACKGROUND

Since nuclei segmentation in histopathology images can provide key information for identifying the presence or stage of a disease, the images need to be assessed carefully. However, color variation in histopathology images, and various structures of nuclei are two major obstacles in accurately segmenting and analyzing histopathology images. Several machine learning methods heavily rely on hand-crafted features which have limitations due to manual thresholding.

RESULTS

To obtain robust results, deep learning based methods have been proposed. Deep convolutional neural networks (DCNN) used for automatically extracting features from raw image data have been proven to achieve great performance. Inspired by such achievements, we propose a nuclei segmentation method based on DCNNs. To normalize the color of histopathology images, we use a deep convolutional Gaussian mixture color normalization model which is able to cluster pixels while considering the structures of nuclei. To segment nuclei, we use Mask R-CNN which achieves state-of-the-art object segmentation performance in the field of computer vision. In addition, we perform multiple inference as a post-processing step to boost segmentation performance. We evaluate our segmentation method on two different datasets. The first dataset consists of histopathology images of various organ while the other consists histopathology images of the same organ. Performance of our segmentation method is measured in various experimental setups at the object-level and the pixel-level. In addition, we compare the performance of our method with that of existing state-of-the-art methods. The experimental results show that our nuclei segmentation method outperforms the existing methods.

CONCLUSIONS

We propose a nuclei segmentation method based on DCNNs for histopathology images. The proposed method which uses Mask R-CNN with color normalization and multiple inference post-processing provides robust nuclei segmentation results. Our method also can facilitate downstream nuclei morphological analyses as it provides high-quality features extracted from histopathology images.

Introduction to Digital Image Analysis in Whole-slide Imaging: A White Paper from the Digital Pathology Association

The advent of whole-slide imaging in digital pathology has brought about the advancement of computer-aided examination of tissue via digital image analysis. Digitized slides can now be easily annotated and analyzed via a variety of algorithms. This study reviews the fundamentals of tissue image analysis and aims to provide pathologists with basic information regarding the features, applications, and general workflow of these new tools. The review gives an overview of the basic categories of software solutions available, potential analysis strategies, technical considerations, and general algorithm readouts. Advantages and limitations of tissue image analysis are discussed, and emerging concepts, such as artificial intelligence and machine learning, are introduced. Finally, examples of how digital image analysis tools are currently being used in diagnostic laboratories, translational research, and drug development are discussed.

An alternative reference space for H&E color normalization

Digital imaging of H&E stained slides has enabled the application of image processing to support pathology workflows. Potential applications include computer-aided diagnostics, advanced quantification tools, and innovative visualization platforms. However, the intrinsic variability of biological tissue and the vast differences in tissue preparation protocols often lead to significant image variability that can hamper the effectiveness of these computational tools. We developed an alternative representation for H&E images that operates within a space that is more amenable to many of these image processing tools. The algorithm to derive this representation operates by exploiting the correlation between color and the spatial properties of the biological structures present in most H&E images. In this way, images are transformed into a structure-centric space in which images are segregated into tissue structure channels. We demonstrate that this framework can be extended to achieve color normalization, effectively reducing inter-slide variability.

An optimized color transformation for the analysis of digital images of hematoxylin & eosin stained slides

Hematoxylin and eosin (H&E) staining is ubiquitous in pathology practice and research. As digital pathology has evolved, the reliance of quantitative methods that make use of H&E images has similarly expanded. For example, cell counting and nuclear morphometry rely on the accurate demarcation of nuclei from other structures and each other. One of the major obstacles to quantitative analysis of H&E images is the high degree of variability observed between different samples and different laboratories. In an effort to characterize this variability, as well as to provide a substrate that can potentially mitigate this factor in quantitative image analysis, we developed a technique to project H&E images into an optimized space more appropriate for many image analysis procedures. We used a decision tree-based support vector machine learning algorithm to classify 44 H&E stained whole slide images of resected breast tumors according to the histological structures that are present. This procedure takes an H&E image as an input and produces a classification map of the image that predicts the likelihood of a pixel belonging to any one of a set of user-defined structures (e.g., cytoplasm, stroma). By reducing these maps into their constituent pixels in color space, an optimal reference vector is obtained for each structure, which identifies the color attributes that maximally distinguish one structure from other elements in the image. We show that tissue structures can be identified using this semi-automated technique. By comparing structure centroids across different images, we obtained a quantitative depiction of H&E variability for each structure. This measurement can potentially be utilized in the laboratory to help calibrate daily staining or identify troublesome slides. Moreover, by aligning reference vectors derived from this technique, images can be transformed in a way that standardizes their color properties and makes them more amenable to image processing.

An automated system for whole microscopic image acquisition and analysis

The field of anatomic pathology has experienced major changes over the last decade. Virtual microscopy (VM) systems have allowed experts in pathology and other biomedical areas to work in a safer and more collaborative way. VMs are automated systems capable of digitizing microscopic samples that were traditionally examined one by one. The possibility of having digital copies reduces the risk of damaging original samples, and also makes it easier to distribute copies among other pathologists. This article describes the development of an automated high-resolution whole slide imaging (WSI) system tailored to the needs and problems encountered in digital imaging for pathology, from hardware control to the full digitization of samples. The system has been built with an additional digital monochromatic camera together with the color camera by default and LED transmitted illumination (RGB). Monochrome cameras are the preferred method of acquisition for fluorescence microscopy. The system is able to digitize correctly and form large high resolution microscope images for both brightfield and fluorescence. The quality of the digital images has been quantified using three metrics based on sharpness, contrast and focus. It has been proved on 150 tissue samples of brain autopsies, prostate biopsies and lung cytologies, at five magnifications: 2.5×, 10×, 20×, 40×, and 63×. The article is focused on the hardware set-up and the acquisition software, although results of the implemented image processing techniques included in the software and applied to the different tissue samples are also presented.

Automatic nuclei segmentation in H&E stained breast cancer histopathology images

The introduction of fast digital slide scanners that provide whole slide images has led to a revival of interest in image analysis applications in pathology. Segmentation of cells and nuclei is an important first step towards automatic analysis of digitized microscopy images. We therefore developed an automated nuclei segmentation method that works with hematoxylin and eosin (H&E) stained breast cancer histopathology images, which represent regions of whole digital slides. The procedure can be divided into four main steps: 1) pre-processing with color unmixing and morphological operators, 2) marker-controlled watershed segmentation at multiple scales and with different markers, 3) post-processing for rejection of false regions and 4) merging of the results from multiple scales. The procedure was developed on a set of 21 breast cancer cases (subset A) and tested on a separate validation set of 18 cases (subset B). The evaluation was done in terms of both detection accuracy (sensitivity and positive predictive value) and segmentation accuracy (Dice coefficient). The mean estimated sensitivity for subset A was 0.875 (±0.092) and for subset B 0.853 (±0.077). The mean estimated positive predictive value was 0.904 (±0.075) and 0.886 (±0.069) for subsets A and B, respectively. For both subsets, the distribution of the Dice coefficients had a high peak around 0.9, with the vast majority of segmentations having values larger than 0.8.

From Biomedical Image Analysis to Biomedical Image Understanding Using Machine Learning

This chapter introduces the reader into the main topics covered by the book: biomedical images, biomedical image analysis and machine learning. The general concepts of each topic are presented and the most representative techniques are briefly discussed. Nevertheless, the chapter focuses on the problem of image understanding (i.e., the problem of mapping the low-level image visual content to its high-level semantic meaning). The chapter discusses different important biomedical problems, such as computer assisted diagnosis, biomedical image retrieval, image-user interaction and medical image navigation, which require solutions involving image understanding. Image understanding, thought of as the strategy to associate semantic meaning to the image visual contents, is a difficult problem that opens up many research challenges. In the context of actual biomedical problems, this is probably an invaluable tool for improving the amount of knowledge that medical doctors are currently extracting from their day-to-day work. Finally, the chapter explores some general ideas that may guide the future research in the field.

History and structures of telecommunication in pathology, focusing on open access platforms

Background

Telecommunication has matured to a broadly applied tool in diagnostic pathology.

Technology and Systems

Contemporary with the development of fast electronic communication lines (Integrated digital network services (ISDN), broad band connections, and fibre optics, as well as the digital imaging technology (digital camera), telecommunication in tissue - based diagnosis (telepathology) has matured. Open access (internet) and server - based communication have induced the development of specific medical information platforms, such as iPATH, UICC-TPCC (telepathology consultation centre of the Union International against Cancer), or the Armed Forces Institute of Pathology (AFIP) teleconsultation system. They have been closed, and are subject to be replaced by specific open access forums (Medical Electronic Expert Communication System (MECES) with embedded virtual slide (VS) technology). MECES uses php language, data base driven mySqL architecture, X/L-AMPP infrastructure, and browser friendly W3C conform standards.

Experiences

The server - based medical communication systems (AFIP, iPATH, UICC-TPCC) have been reported to be a useful and easy to handle tool for expert consultation. Correct sampling and evaluation of transmitted still images by experts reported revealed no or only minor differences to the original images and good practice of the involved experts. β tests with the new generation medical expert consultation systems (MECES) revealed superior results in terms of performance, still image viewing, and system handling, especially as this is closely related to the use of so - called social forums (facebook, youtube, etc.).

Benefits and Expectations

In addition to the acknowledged advantages of the former established systems (assistance of pathologists working in developing countries, diagnosis confirmation, international information exchange, etc.), the new generation offers additional benefits such as acoustic information transfer, assistance in image screening, VS technology, and teaching in diagnostic sampling, judgement, and verification.

Distributed computing in image analysis using open source frameworks and application to image sharpness assessment of histological whole slide images

Background

Automated image analysis on virtual slides is evolving rapidly and will play an important role in the future of digital pathology. Due to the image size, the computational cost of processing whole slide images (WSIs) in full resolution is immense. Moreover, image analysis requires well focused images in high magnification.

Methods

We present a system that merges virtual microscopy techniques, open source image analysis software, and distributed parallel processing. We have integrated the parallel processing framework JPPF, so batch processing can be performed distributed and in parallel. All resulting meta data and image data are collected and merged. As an example the system is applied to the specific task of image sharpness assessment. ImageJ is an open source image editing and processing framework developed at the NIH having a large user community that contributes image processing algorithms wrapped as plug-ins in a wide field of life science applications. We developed an ImageJ plug-in that supports both basic interactive virtual microscope and batch processing functionality. For the application of sharpness inspection we employ an approach with non-overlapping tiles. Compute nodes retrieve image tiles of moderate size from the streaming server and compute the focus measure. Each tile is divided into small sub images to calculate an edge based sharpness criterion which is used for classification. The results are aggregated in a sharpness map.

Results

Based on the system we calculate a sharpness measure and classify virtual slides into one of the following categories - excellent, okay, review and defective. Generating a scaled sharpness map enables the user to evaluate sharpness of WSIs and shows overall quality at a glance thus reducing tedious assessment work.

Conclusions

Using sharpness assessment as an example, the introduced system can be used to process, analyze and parallelize analysis of whole slide images based on open source software.

Using a visual discrimination model for the detection of compression artifacts in virtual pathology images

A major issue in telepathology is the extremely large and growing size of digitized "virtual" slides, which can require several gigabytes of storage and cause significant delays in data transmission for remote image interpretation and interactive visualization by pathologists. Compression can reduce this massive amount of virtual slide data, but reversible (lossless) methods limit data reduction to less than 50%, while lossy compression can degrade image quality and diagnostic accuracy. "Visually lossless" compression offers the potential for using higher compression levels without noticeable artifacts, but requires a rate-control strategy that adapts to image content and loss visibility. We investigated the utility of a visual discrimination model (VDM) and other distortion metrics for predicting JPEG 2000 bit rates corresponding to visually lossless compression of virtual slides for breast biopsy specimens. Threshold bit rates were determined experimentally with human observers for a variety of tissue regions cropped from virtual slides. For test images compressed to their visually lossless thresholds, just-noticeable difference (JND) metrics computed by the VDM were nearly constant at the 95th percentile level or higher, and were significantly less variable than peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) metrics. Our results suggest that VDM metrics could be used to guide the compression of virtual slides to achieve visually lossless compression while providing 5-12 times the data reduction of reversible methods.

Quantification of virtual slides: Approaches to analysis of content-based image information

Virtual microscopy, which is the diagnostic work on completely digitized histological and cytological slides as well as blood smears, is at the stage to be implemented in routine diagnostic surgical pathology (tissue-based diagnosis) in the near future, once it has been accepted by the US Food and Drug Administration. The principle of content-based image information, its mandatory prerequisites to obtain reproducible and stable image information as well as the different compartments that contribute to image information are described in detail. Automated extraction of content-based image information requires shading correction, constant maximum of grey values, and standardized grey value histograms. The different compartments to evaluate image information include objects, structure, and texture. Identification of objects and derived structure depend on segmentation accuracy and applied procedures; textures contain pixel-based image information only. All together, these image compartments posses the discrimination power to distinguish between object space and background, and, in addition, to reproducibly define regions of interest (ROIs). ROIs are image areas which display the information that is of preferable interest to the viewing pathologist. They contribute to the derived diagnosis to a higher level when compared with other image areas. The implementation of content-based image information algorithms to be applied for predictive tissue-based diagnoses is described in detail.

Toluidine blue color perception in identification of oral mucosal lesions

The objective of this study is to examine observer agreement on the rank of the color tones after toluidine blue staining of a mucosal lesion. Cohort study with repeated measures is the design of the study. Twenty observers ranked and scored 8 specified areas on the color images of a lesion before and after toluidine blue application in two sessions. Inter and intra-observer variations were analyzed with Cohen's kappa. The L* (the black-white axis), a* (red-green axis), and b* (yellow-blue axis) values were measured and set as the gold standards. Intra and inter-observer agreements were к = 0.86 and к = 0.854. All color parameters were effective on the color ranking order (pL* = 0.00, pa* = 0.007, pb* = 0.00), although L* and b* were more effective on the ranking of the samples than a*. Areas that appeared pale blue visually had a significant blue component, but the observers were confused with the effect of whiteness of the area in clinical decision making.

Automatic measurement of epithelium differentiation and classification of cervical intraneoplasia by computerized image analysis

BACKGROUND

The feasibility of evaluating an objective grading of cervical intraneoplasia lesions (CIN) is attempted using an automatic computerized system able to measure several valuable parameters with special reference to epithelium differentiation.

METHODS

4 groups of 10 images each were selected at random from 68 consensus images coming from 80 archival cervical biopsies, normal (n = 10), CIN 1 (n = 10), CIN 2 (n = 10), CIN 3 (n = 10). Representative images of lesions were captured from the microscopic slides and were analyzed using mathematical morphology, with special reference toVoronoï tessellation and Delaunay triangulation. Epithelium surface, nuclear and cytoplasm area, triangle edge and area, total and upper nuclear index were precisely measured in each lesion, and discriminant coefficients were calculated therewith. A dilation/erosion coefficient was automatically defined using triangle edge length and nuclear radius in order to measure the epithelium ratio of differentiation. A histogram ratio was also automatically established between total nuclei and upper nuclei on top of differentiated epithelium. With the latter two ratios added to the nucleo-cytoplasmic ratio, a cervical score able to classify CIN is proposed.

RESULTS

There is a quasi-linear increase of mean cervical score values between normal epithelium and CIN 3: (27) for normal epithelium, (51) for CIN 1, (78) for CIN 2 and (100) for CIN 3, with significant differences (P < 0.05).

CONCLUSION

Our results highlight the possibility of applying a cervical score for the automatic grading of CIN lesions and thereby assisting the pathologist for improvement of grading. The automatic measure of epithelium differentiation ratio appears to be a new interesting parameter in computerized image analysis of cervical lesions.

Requirements for the valid quantification of immunostains on tissue microarray materials using image analysis

Antibody-based proteomics applied to tissue microarray (TMA) technology provides a very efficient means of visualizing and locating antigen expression in large collections of normal and pathological tissue samples. To characterize antigen expression on TMAs, the use of image analysis methods avoids the effects of human subjectivity evidenced in manual microscopical analysis. Thus, these methods have the potential to significantly enhance both precision and reproducibility. Although some commercial systems include tools for the quantitative evaluation of immunohistochemistry-stained images, there exists no clear agreement on best practices to allow for correct and reproducible quantification results. Our study focuses on practical aspects regarding (i) image acquisition (ii) segmentation of staining and counterstaining areas and (iii) extraction of quantitative features. We illustrate our findings using a commercial system to quantify different immunohistochemistry markers targeting proteins with different expression patterns (cytoplasmic, nuclear or membranous) in colon cancer or brain tumor TMAs. Our investigations led us to identify several steps that we consider essential for standardizing computer-assisted immunostaining quantification experiments. In addition, we propose a data normalization process based on reference materials to be able to compare measurements between studies involving different TMAs. In conclusion, we recommend certain critical prerequisites that commercial or in-house systems should satisfy in order to permit valid immunostaining quantification.

Immunohistochemistry in diagnostic surgical pathology: contributions of protein life-cycle, use of evidence-based methods and data normalization on interpretation of immunohistochemical stains

Immunohistochemical (IHC) staining of formalin-fixed and paraffin-embedded tissues (FFPE) is widely used in diagnostic surgical pathology. All anatomical and surgical pathologists use IHC to confirm cancer cell type and possible origin of metastatic cancer of unknown primary site. What kinds of improvements in IHC are needed to boost and strengthen the use of IHC in future diagnostic pathology practice? The aim of this perspective is to suggest that continuing reliance on immunohistochemistry in cancer diagnosis, search and validation of biomarkers for predictive and prognostic studies and utility in cancer treatment selection means that minimum IHC data sets including "normalization methods" for IHC scoring, use of relative protein expression levels, use of protein functional pathways and modifications and protein cell type specificity may be needed when markers are proposed for use in diagnostic pathology. Furthermore evidence based methods (EBM), minimum criteria for diagnostic accuracy (STARD), will help in selecting antibodies for use in diagnostic pathology. In the near future, quantitative methods of proteomics, quantitative real-time polymerase chain reaction (qRT-PCR) and the use of high-throughput genomics for diagnosis and predictive decisions may become preferred tools in medicine.

Theory of sampling and its application in tissue based diagnosis

BACKGROUND

A general theory of sampling and its application in tissue based diagnosis is presented. Sampling is defined as extraction of information from certain limited spaces and its transformation into a statement or measure that is valid for the entire (reference) space. The procedure should be reproducible in time and space, i.e. give the same results when applied under similar circumstances. Sampling includes two different aspects, the procedure of sample selection and the efficiency of its performance. The practical performance of sample selection focuses on search for localization of specific compartments within the basic space, and search for presence of specific compartments.

METHODS

When a sampling procedure is applied in diagnostic processes two different procedures can be distinguished: I) the evaluation of a diagnostic significance of a certain object, which is the probability that the object can be grouped into a certain diagnosis, and II) the probability to detect these basic units. Sampling can be performed without or with external knowledge, such as size of searched objects, neighbourhood conditions, spatial distribution of objects, etc. If the sample size is much larger than the object size, the application of a translation invariant transformation results in Kriege's formula, which is widely used in search for ores. Usually, sampling is performed in a series of area (space) selections of identical size. The size can be defined in relation to the reference space or according to interspatial relationship. The first method is called random sampling, the second stratified sampling.

RESULTS

Random sampling does not require knowledge about the reference space, and is used to estimate the number and size of objects. Estimated features include area (volume) fraction, numerical, boundary and surface densities. Stratified sampling requires the knowledge of objects (and their features) and evaluates spatial features in relation to the detected objects (for example grey value distribution around an object). It serves also for the definition of parameters of the probability function in so-called active segmentation.

CONCLUSION

The method is useful in standardization of images derived from immunohistochemically stained slides, and implemented in the EAMUS system http://www.diagnomX.de. It can also be applied for the search of "objects possessing an amplification function", i.e. a rare event with "steering function". A formula to calculate the efficiency and potential error rate of the described sampling procedures is given.

Integration and acceleration of virtual microscopy as the key to successful implementation into the routine diagnostic process

Background

The virtual microscopy is widely accepted in Pathology for educational purposes and teleconsultation but is far from the routine use in surgical pathology due to the technical requirements and some limitations. A technical problem is the limited bandwidth of a usual network and the delayed transmission rate and presentation time on the screen.

Methods

In this study the process of secondary diagnostic was evaluated using the "T.Konsult Pathologie" service of the Professional Association of German Pathologists within the German breast cancer screening program. The characteristics of the access to the WSI (Whole Slide Images) have been analyzed to explore the possibilities of prefetching and caching to reduce the presentation and transfer time with the goal to increase user acceptance. The log files of the web server were analyzed to reconstruct the movements of the pathologist on the WSI and to create the observation path. Using a specialized tool the observation paths were extracted automatically from the log files. The attributes linearity, 3-point-linearity, changes per request, and number of consecutive requests were calculated to design, develop and evaluate different caching and prefetching strategies.

Results

The analysis of the observation paths showed that a complete accordance of two image requests is a very rare event. But more frequently a partial covering of two requested image areas can be found. In total 257 diagnostic paths from 131 WSI have been extracted and analysed. On average a diagnostic path consists of 16 image requests and takes 189 seconds between first and last image request. The mean linearity was 0,41 and the mean 3-point-linearity 0,85. Three different caching algorithms have been compared with respect to hit rate and additional image requests on the WSI server. Tests demonstrated that 95% of the diagnostic paths could be loaded without any deletion of entries in the cache (cache size 12,2 Megapixel). If the image parts are stored after JPEG compression this complies with less than 2 MB.

Discussion

WSI telepathology is a technology which offers the possibility to break the limitations of conventional static telepathology. The complete histological slide may be investigated instead of sets of images of lesions sampled by the presenting pathologist. The benefit is demonstrated by the high diagnostic security of 95% accordance between first and second diagnosis.