1
|
Al-Thelaya K, Gilal NU, Alzubaidi M, Majeed F, Agus M, Schneider J, Househ M. Applications of discriminative and deep learning feature extraction methods for whole slide image analysis: A survey. J Pathol Inform 2023; 14:100335. [PMID: 37928897 PMCID: PMC10622844 DOI: 10.1016/j.jpi.2023.100335] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/29/2023] [Revised: 07/17/2023] [Accepted: 07/19/2023] [Indexed: 11/07/2023] Open
Abstract
Digital pathology technologies, including whole slide imaging (WSI), have significantly improved modern clinical practices by facilitating storing, viewing, processing, and sharing digital scans of tissue glass slides. Researchers have proposed various artificial intelligence (AI) solutions for digital pathology applications, such as automated image analysis, to extract diagnostic information from WSI for improving pathology productivity, accuracy, and reproducibility. Feature extraction methods play a crucial role in transforming raw image data into meaningful representations for analysis, facilitating the characterization of tissue structures, cellular properties, and pathological patterns. These features have diverse applications in several digital pathology applications, such as cancer prognosis and diagnosis. Deep learning-based feature extraction methods have emerged as a promising approach to accurately represent WSI contents and have demonstrated superior performance in histology-related tasks. In this survey, we provide a comprehensive overview of feature extraction methods, including both manual and deep learning-based techniques, for the analysis of WSIs. We review relevant literature, analyze the discriminative and geometric features of WSIs (i.e., features suited to support the diagnostic process and extracted by "engineered" methods as opposed to AI), and explore predictive modeling techniques using AI and deep learning. This survey examines the advances, challenges, and opportunities in this rapidly evolving field, emphasizing the potential for accurate diagnosis, prognosis, and decision-making in digital pathology.
Collapse
Affiliation(s)
- Khaled Al-Thelaya
- Department of Information and Computing Technology, College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar
| | - Nauman Ullah Gilal
- Department of Information and Computing Technology, College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar
| | - Mahmood Alzubaidi
- Department of Information and Computing Technology, College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar
| | - Fahad Majeed
- Department of Information and Computing Technology, College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar
| | - Marco Agus
- Department of Information and Computing Technology, College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar
| | - Jens Schneider
- Department of Information and Computing Technology, College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar
| | - Mowafa Househ
- Department of Information and Computing Technology, College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar
| |
Collapse
|
2
|
Han W, Johnson C, Warner A, Gaed M, Gomez JA, Moussa M, Chin J, Pautler S, Bauman G, Ward AD. Automatic cancer detection on digital histopathology images of mid-gland radical prostatectomy specimens. J Med Imaging (Bellingham) 2020; 7:047501. [PMID: 32715024 DOI: 10.1117/1.jmi.7.4.047501] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/08/2019] [Accepted: 07/06/2020] [Indexed: 11/14/2022] Open
Abstract
Purpose: Automatic cancer detection on radical prostatectomy (RP) sections facilitates graphical and quantitative surgical pathology reporting, which can potentially benefit postsurgery follow-up care and treatment planning. It can also support imaging validation studies using a histologic reference standard and pathology research studies. This problem is challenging due to the large sizes of digital histopathology whole-mount whole-slide images (WSIs) of RP sections and staining variability across different WSIs. Approach: We proposed a calibration-free adaptive thresholding algorithm, which compensates for staining variability and yields consistent tissue component maps (TCMs) of the nuclei, lumina, and other tissues. We used and compared three machine learning methods for classifying each cancer versus noncancer region of interest (ROI) throughout each WSI: (1) conventional machine learning methods and 14 texture features extracted from TCMs, (2) transfer learning with pretrained AlexNet fine-tuned by TCM ROIs, and (3) transfer learning with pretrained AlexNet fine-tuned with raw image ROIs. Results: The three methods yielded areas under the receiver operating characteristic curve of 0.96, 0.98, and 0.98, respectively, in leave-one-patient-out cross validation using 1.3 million ROIs from 286 mid-gland whole-mount WSIs from 68 patients. Conclusion: Transfer learning with the use of TCMs demonstrated state-of-the-art overall performance and is more stable with respect to sample size across different tissue types. For the tissue types involving Gleason 5 (most aggressive) cancer, it achieved the best performance compared to the other tested methods. This tool can be translated to clinical workflow to assist graphical and quantitative pathology reporting for surgical specimens upon further multicenter validation.
Collapse
Affiliation(s)
- Wenchao Han
- Baines Imaging Research Laboratory, London Regional Cancer Program, London, Canada.,Lawson Health Research Institute, London, Ontario, Canada.,Western University, Department of Medical Biophysics, London, Ontario, Canada
| | - Carol Johnson
- Baines Imaging Research Laboratory, London Regional Cancer Program, London, Canada
| | - Andrew Warner
- Baines Imaging Research Laboratory, London Regional Cancer Program, London, Canada
| | - Mena Gaed
- Western University, Department of Pathology and Laboratory Medicine, London, Ontario, Canada
| | - Jose A Gomez
- Western University, Department of Pathology and Laboratory Medicine, London, Ontario, Canada
| | - Madeleine Moussa
- Western University, Department of Pathology and Laboratory Medicine, London, Ontario, Canada
| | - Joseph Chin
- Western University, Department of Oncology, London, Ontario, Canada.,Western University, Department of Surgery, London, Ontario, Canada
| | - Stephen Pautler
- Western University, Department of Oncology, London, Ontario, Canada.,Western University, Department of Surgery, London, Ontario, Canada
| | - Glenn Bauman
- Lawson Health Research Institute, London, Ontario, Canada.,Western University, Department of Medical Biophysics, London, Ontario, Canada.,Western University, Department of Oncology, London, Ontario, Canada
| | - Aaron D Ward
- Baines Imaging Research Laboratory, London Regional Cancer Program, London, Canada.,Lawson Health Research Institute, London, Ontario, Canada.,Western University, Department of Medical Biophysics, London, Ontario, Canada.,Western University, Department of Oncology, London, Ontario, Canada
| |
Collapse
|
3
|
Bhargava R, Madabhushi A. Emerging Themes in Image Informatics and Molecular Analysis for Digital Pathology. Annu Rev Biomed Eng 2017; 18:387-412. [PMID: 27420575 DOI: 10.1146/annurev-bioeng-112415-114722] [Citation(s) in RCA: 86] [Impact Index Per Article: 12.3] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/17/2022]
Abstract
Pathology is essential for research in disease and development, as well as for clinical decision making. For more than 100 years, pathology practice has involved analyzing images of stained, thin tissue sections by a trained human using an optical microscope. Technological advances are now driving major changes in this paradigm toward digital pathology (DP). The digital transformation of pathology goes beyond recording, archiving, and retrieving images, providing new computational tools to inform better decision making for precision medicine. First, we discuss some emerging innovations in both computational image analytics and imaging instrumentation in DP. Second, we discuss molecular contrast in pathology. Molecular DP has traditionally been an extension of pathology with molecularly specific dyes. Label-free, spectroscopic images are rapidly emerging as another important information source, and we describe the benefits and potential of this evolution. Third, we describe multimodal DP, which is enabled by computational algorithms and combines the best characteristics of structural and molecular pathology. Finally, we provide examples of application areas in telepathology, education, and precision medicine. We conclude by discussing challenges and emerging opportunities in this area.
Collapse
Affiliation(s)
- Rohit Bhargava
- Departments of Bioengineering, Chemical and Biomolecular Engineering, Electrical and Computer Engineering, Mechanical Science and Engineering, and Chemistry, and Beckman Institute for Advanced Science and Technology, University of Illinois at Urbana-Champaign, Urbana, Illinois 61801;
| | - Anant Madabhushi
- Center for Computational Imaging and Personalized Diagnostics; Departments of Biomedical Engineering, Urology, Pathology, Radiology, Radiation Oncology, General Medical Sciences, Electrical Engineering, and Computer Science; and Case Comprehensive Cancer Center, Case Western Reserve University, Cleveland, Ohio 44106;
| |
Collapse
|
4
|
Leo P, Lee G, Shih NNC, Elliott R, Feldman MD, Madabhushi A. Evaluating stability of histomorphometric features across scanner and staining variations: prostate cancer diagnosis from whole slide images. J Med Imaging (Bellingham) 2016; 3:047502. [PMID: 27803941 DOI: 10.1117/1.jmi.3.4.047502] [Citation(s) in RCA: 42] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/03/2016] [Accepted: 09/16/2016] [Indexed: 01/04/2023] Open
Abstract
Quantitative histomorphometry (QH) is the process of computerized feature extraction from digitized tissue slide images to predict disease presence, behavior, and outcome. Feature stability between sites may be compromised by laboratory-specific variables including dye batch, slice thickness, and the whole slide scanner used. We present two new measures, preparation-induced instability score and latent instability score, to quantify feature instability across and within datasets. In a use case involving prostate cancer, we examined QH features which may detect cancer on whole slide images. Using our method, we found that five feature families (graph, shape, co-occurring gland tensor, sub-graph, and texture) were different between datasets in 19.7% to 48.6% of comparisons while the values expected without site variation were 4.2% to 4.6%. Color normalizing all images to a template did not reduce instability. Scanning the same 34 slides on three scanners demonstrated that Haralick features were most substantively affected by scanner variation, being unstable in 62% of comparisons. We found that unstable feature families performed significantly worse in inter- than intrasite classification. Our results appear to suggest QH features should be evaluated across sites to assess robustness, and class discriminability alone should not represent the benchmark for digital pathology feature selection.
Collapse
Affiliation(s)
- Patrick Leo
- Case Western Reserve University , Department of Biomedical Engineering, 2071 Martin Luther King Jr. Drive, Cleveland, Ohio 44106, United States
| | - George Lee
- Case Western Reserve University , Department of Biomedical Engineering, 2071 Martin Luther King Jr. Drive, Cleveland, Ohio 44106, United States
| | - Natalie N C Shih
- University of Pennsylvania , Department of Pathology, 3400 Spruce Street, Philadelphia, Pennsylvania 19104, United States
| | - Robin Elliott
- Case Western Reserve University , Department of Pathology, 11100 Euclid Avenue, Cleveland, Ohio 44106, United States
| | - Michael D Feldman
- University of Pennsylvania , Department of Pathology, 3400 Spruce Street, Philadelphia, Pennsylvania 19104, United States
| | - Anant Madabhushi
- Case Western Reserve University , Department of Biomedical Engineering, 2071 Martin Luther King Jr. Drive, Cleveland, Ohio 44106, United States
| |
Collapse
|
5
|
Petrolis R, Ramonaitė R, Jančiauskas D, Kupčinskas J, Pečiulis R, Kupčinskas L, Kriščiukaitis A. Digital imaging of colon tissue: method for evaluation of inflammation severity by spatial frequency features of the histological images. Diagn Pathol 2015; 10:159. [PMID: 26370784 PMCID: PMC4570696 DOI: 10.1186/s13000-015-0389-7] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/27/2015] [Accepted: 08/28/2015] [Indexed: 12/22/2022] Open
Abstract
BACKGROUND The efficacy of histological analysis of colon sections used for evaluation of inflammation severity can be improved by means of digital imaging giving quantitative estimates of main diagnostic features. The aim of this study was to reveal most valuable diagnostic features reflecting inflammation severity in colon and elaborate the evaluation method for computer-aided diagnostics. METHODS Tissue specimens from 24 BALB/c mice and 15 patients were included in the study. Chronic and acute colon inflammation in mice was induced by oral administration of dextran sulphate sodium (DSS) solution, while mice in the control group did not get DSS. Human samples of inflamed colon tissue were obtained from patients with ulcerative colitis (n = 6). Non-inflamed colon tissue of control subjects (n = 9) was obtained from patients with irritable bowel syndrome or functional obstipation. Analysis of morphological changes in mice and human colon mucosa was performed using 4-μm haematoxylin-eosin (HE) sections. The features reflecting morphological changes in the images of colon mucosa were calculated by convolution of Gabor filter bank and array of pixel values. All features were generalized by calculating mean, histogram skewness and entropy of every image response. Principal component analysis was used to construct optimal representation of morphological changes. RESULTS First principal component (PC1) was representing the major part of features variation (97 % in mice and 71 % in human specimens) and was selected as a measure of inflammation severity. Validation of new measure was performed by means of custom-made software realizing double blind comparison of differences in PC1 with expert's opinion about inflammation severity presented in two compared pictures. Overall accuracy of 80 % for mice and 67 % for human was reached. CONCLUSION Principal component analysis of spatial frequency features of histological images may provide continuous scale estimation of inflammation severity of colon tissue.
Collapse
Affiliation(s)
- Robertas Petrolis
- Neuroscience Institute, Lithuanian University of Health Sciences, Eiveniu str. 2, LT 50009, Kaunas, Lithuania.
| | - Rima Ramonaitė
- Institute for Digestive Research, Lithuanian University of Health Sciences, Kaunas, LT 44307, Lithuania
| | - Dainius Jančiauskas
- Clinic of Pathology, Lithuanian University of Health Sciences, Kaunas, LT 50009, Lithuania
| | - Juozas Kupčinskas
- Institute for Digestive Research, Lithuanian University of Health Sciences, Kaunas, LT 44307, Lithuania
- Department of Gastroenterology, Lithuanian University of Health Sciences, Kaunas, LT 50161, Lithuania
| | - Rokas Pečiulis
- Lithuanian University of Health Sciences, Kaunas, LT 50009, Lithuania
| | - Limas Kupčinskas
- Institute for Digestive Research, Lithuanian University of Health Sciences, Kaunas, LT 44307, Lithuania
- Department of Gastroenterology, Lithuanian University of Health Sciences, Kaunas, LT 50161, Lithuania
| | - Algimantas Kriščiukaitis
- Neuroscience Institute, Lithuanian University of Health Sciences, Eiveniu str. 2, LT 50009, Kaunas, Lithuania
- Department of Physics, Mathematics and Biophysics, Lithuanian University of Health Sciences, Kaunas, LT 50009, Lithuania
| |
Collapse
|
6
|
Gertych A, Ing N, Ma Z, Fuchs TJ, Salman S, Mohanty S, Bhele S, Velásquez-Vacca A, Amin MB, Knudsen BS. Machine learning approaches to analyze histological images of tissues from radical prostatectomies. Comput Med Imaging Graph 2015; 46 Pt 2:197-208. [PMID: 26362074 DOI: 10.1016/j.compmedimag.2015.08.002] [Citation(s) in RCA: 53] [Impact Index Per Article: 5.9] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/13/2015] [Revised: 06/30/2015] [Accepted: 08/07/2015] [Indexed: 11/18/2022]
Abstract
Computerized evaluation of histological preparations of prostate tissues involves identification of tissue components such as stroma (ST), benign/normal epithelium (BN) and prostate cancer (PCa). Image classification approaches have been developed to identify and classify glandular regions in digital images of prostate tissues; however their success has been limited by difficulties in cellular segmentation and tissue heterogeneity. We hypothesized that utilizing image pixels to generate intensity histograms of hematoxylin (H) and eosin (E) stains deconvoluted from H&E images numerically captures the architectural difference between glands and stroma. In addition, we postulated that joint histograms of local binary patterns and local variance (LBPxVAR) can be used as sensitive textural features to differentiate benign/normal tissue from cancer. Here we utilized a machine learning approach comprising of a support vector machine (SVM) followed by a random forest (RF) classifier to digitally stratify prostate tissue into ST, BN and PCa areas. Two pathologists manually annotated 210 images of low- and high-grade tumors from slides that were selected from 20 radical prostatectomies and digitized at high-resolution. The 210 images were split into the training (n=19) and test (n=191) sets. Local intensity histograms of H and E were used to train a SVM classifier to separate ST from epithelium (BN+PCa). The performance of SVM prediction was evaluated by measuring the accuracy of delineating epithelial areas. The Jaccard J=59.5 ± 14.6 and Rand Ri=62.0 ± 7.5 indices reported a significantly better prediction when compared to a reference method (Chen et al., Clinical Proteomics 2013, 10:18) based on the averaged values from the test set. To distinguish BN from PCa we trained a RF classifier with LBPxVAR and local intensity histograms and obtained separate performance values for BN and PCa: JBN=35.2 ± 24.9, OBN=49.6 ± 32, JPCa=49.5 ± 18.5, OPCa=72.7 ± 14.8 and Ri=60.6 ± 7.6 in the test set. Our pixel-based classification does not rely on the detection of lumens, which is prone to errors and has limitations in high-grade cancers and has the potential to aid in clinical studies in which the quantification of tumor content is necessary to prognosticate the course of the disease. The image data set with ground truth annotation is available for public use to stimulate further research in this area.
Collapse
Affiliation(s)
- Arkadiusz Gertych
- Department of Surgery, Cedars-Sinai Medical Center, 8700 Beverly Blvd, Los Angeles, CA 90048, USA; Department of Pathology and Laboratory Medicine, Cedars-Sinai Medical Center, 8700 Beverly Blvd, Los Angeles, CA 90048, USA.
| | - Nathan Ing
- Department of Surgery, Cedars-Sinai Medical Center, 8700 Beverly Blvd, Los Angeles, CA 90048, USA
| | - Zhaoxuan Ma
- Department of Biomedical Sciences Cedars-Sinai Medical Center, 8700 Beverly Blvd, Los Angeles, CA 90048, USA
| | - Thomas J Fuchs
- Jet Propulsion Laboratory, California Institute of Technology, 4800 Oak Grove Drive, Pasadena, CA 91109, USA; Department of Medical Physics, Memorial Sloan Kettering Cancer Center, 1275 York Avenue, New York, NY 10065, USA; Department of Pathology, Memorial Sloan Kettering Cancer Center, 1275 York Avenue, New York, NY 10065, USA
| | - Sadri Salman
- Department of Surgery, Cedars-Sinai Medical Center, 8700 Beverly Blvd, Los Angeles, CA 90048, USA
| | - Sambit Mohanty
- Department of Pathology and Laboratory Medicine, Cedars-Sinai Medical Center, 8700 Beverly Blvd, Los Angeles, CA 90048, USA
| | - Sanica Bhele
- Department of Pathology and Laboratory Medicine, Cedars-Sinai Medical Center, 8700 Beverly Blvd, Los Angeles, CA 90048, USA
| | - Adriana Velásquez-Vacca
- Department of Surgery, Cedars-Sinai Medical Center, 8700 Beverly Blvd, Los Angeles, CA 90048, USA
| | - Mahul B Amin
- Department of Pathology and Laboratory Medicine, Cedars-Sinai Medical Center, 8700 Beverly Blvd, Los Angeles, CA 90048, USA
| | - Beatrice S Knudsen
- Department of Pathology and Laboratory Medicine, Cedars-Sinai Medical Center, 8700 Beverly Blvd, Los Angeles, CA 90048, USA; Department of Biomedical Sciences Cedars-Sinai Medical Center, 8700 Beverly Blvd, Los Angeles, CA 90048, USA
| |
Collapse
|
7
|
Mosquera-Lopez C, Agaian S, Velez-Hoyos A, Thompson I. Computer-Aided Prostate Cancer Diagnosis From Digitized Histopathology: A Review on Texture-Based Systems. IEEE Rev Biomed Eng 2014; 8:98-113. [PMID: 25055385 DOI: 10.1109/rbme.2014.2340401] [Citation(s) in RCA: 51] [Impact Index Per Article: 5.1] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Abstract
Prostate cancer (PCa) is currently diagnosed by microscopic evaluation of biopsy samples. Since tissue assessment heavily relies on the pathologists level of expertise and interpretation criteria, it is still a subjective process with high intra- and interobserver variabilities. Computer-aided diagnosis (CAD) may have a major impact on detection and grading of PCa by reducing the pathologists reading time, and increasing the accuracy and reproducibility of diagnosis outcomes. However, the complexity of the prostatic tissue and the large volumes of data generated by biopsy procedures make the development of CAD systems for PCa a challenging task. The problem of automated diagnosis of prostatic carcinoma from histopathology has received a lot of attention. As a result, a number of CAD systems, have been proposed for quantitative image analysis and classification. This review aims at providing a detailed description of selected literature in the field of CAD of PCa, emphasizing the role of texture analysis methods in tissue description. It includes a review of image analysis tools for image preprocessing, feature extraction, classification, and validation techniques used in PCa detection and grading, as well as future directions in pursuit of better texture-based CAD systems.
Collapse
|
8
|
Sadimin ET, Foran DJ. Pathology Imaging Informatics for Clinical Practice and Investigative and Translational Research. ACTA ACUST UNITED AC 2012; 5:103-109. [PMID: 22855694 DOI: 10.7156/v5i2p103] [Citation(s) in RCA: 11] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/20/2022]
Abstract
Pathologists routinely interpret gross and microscopic specimens to render diagnoses and to engage in a broad spectrum of investigative research. Multiple studies have demonstrated that imaging technologies have progressed to a level at which properly digitized specimens provide sufficient quality comparable to the traditional glass slides examinations. Continued advancements in this area will have a profound impact on the manner in which pathology is conducted from this point on. Several leading institutions have already undertaken ambitious projects directed toward digitally imaging, archiving, and sharing pathology specimens. As a result of these advances, the use of informatics in diagnostic and investigative pathology applications is expanding rapidly. In addition, the advent of novel technologies such as multispectral imaging makes it possible to visualize and analyze imaged specimens using multiple wavelengths simultaneously. As these powerful technologies become increasingly accepted and adopted, the opportunities for gaining new insight into the underlying mechanisms of diseases as well as the potential for discriminating among subtypes of pathologies are growing accordingly.
Collapse
Affiliation(s)
- Evita T Sadimin
- Department of Pathology, Robert Wood Johnson Medical School, New Brunswick, NJ
| | | |
Collapse
|