1
|
Gouzou D, Taimori A, Haloubi T, Finlayson N, Wang Q, Hopgood JR, Vallejo M. Applications of machine learning in time-domain fluorescence lifetime imaging: a review. Methods Appl Fluoresc 2024; 12:022001. [PMID: 38055998 PMCID: PMC10851337 DOI: 10.1088/2050-6120/ad12f7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/30/2023] [Revised: 09/25/2023] [Accepted: 12/06/2023] [Indexed: 12/08/2023]
Abstract
Many medical imaging modalities have benefited from recent advances in Machine Learning (ML), specifically in deep learning, such as neural networks. Computers can be trained to investigate and enhance medical imaging methods without using valuable human resources. In recent years, Fluorescence Lifetime Imaging (FLIm) has received increasing attention from the ML community. FLIm goes beyond conventional spectral imaging, providing additional lifetime information, and could lead to optical histopathology supporting real-time diagnostics. However, most current studies do not use the full potential of machine/deep learning models. As a developing image modality, FLIm data are not easily obtainable, which, coupled with an absence of standardisation, is pushing back the research to develop models which could advance automated diagnosis and help promote FLIm. In this paper, we describe recent developments that improve FLIm image quality, specifically time-domain systems, and we summarise sensing, signal-to-noise analysis and the advances in registration and low-level tracking. We review the two main applications of ML for FLIm: lifetime estimation and image analysis through classification and segmentation. We suggest a course of action to improve the quality of ML studies applied to FLIm. Our final goal is to promote FLIm and attract more ML practitioners to explore the potential of lifetime imaging.
Collapse
Affiliation(s)
- Dorian Gouzou
- Dorian Gouzou and Marta Vallejo are with Institute of Signals, Sensors and Systems, School of Engineering and Physical Sciences, Heriot Watt University, Edinburgh, EH14 4AS, United Kingdom
| | - Ali Taimori
- Tarek Haloubi, Ali Taimori, and James R. Hopgood are with Institute for Imaging, Data and Communication, School of Engineering, University of Edinburgh, Edinburgh, EH9 3FG, United Kingdom
| | - Tarek Haloubi
- Tarek Haloubi, Ali Taimori, and James R. Hopgood are with Institute for Imaging, Data and Communication, School of Engineering, University of Edinburgh, Edinburgh, EH9 3FG, United Kingdom
| | - Neil Finlayson
- Neil Finlayson is with Institute for Integrated Micro and Nano Systems, School of Engineering, University ofEdinburgh, Edinburgh EH9 3FF, United Kingdom
| | - Qiang Wang
- Qiang Wang is with Centre for Inflammation Research, University of Edinburgh, Edinburgh, EH16 4TJ, United Kingdom
| | - James R Hopgood
- Tarek Haloubi, Ali Taimori, and James R. Hopgood are with Institute for Imaging, Data and Communication, School of Engineering, University of Edinburgh, Edinburgh, EH9 3FG, United Kingdom
| | - Marta Vallejo
- Dorian Gouzou and Marta Vallejo are with Institute of Signals, Sensors and Systems, School of Engineering and Physical Sciences, Heriot Watt University, Edinburgh, EH14 4AS, United Kingdom
| |
Collapse
|
2
|
Doyle J, Green BF, Eminizer M, Jimenez-Sanchez D, Lu S, Engle EL, Xu H, Ogurtsova A, Lai J, Soto-Diaz S, Roskes JS, Deutsch JS, Taube JM, Sunshine JC, Szalay AS. Whole-Slide Imaging, Mutual Information Registration for Multiplex Immunohistochemistry and Immunofluorescence. J Transl Med 2023; 103:100175. [PMID: 37196983 PMCID: PMC10527458 DOI: 10.1016/j.labinv.2023.100175] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/06/2022] [Revised: 03/24/2023] [Accepted: 05/08/2023] [Indexed: 05/19/2023] Open
Abstract
Multiplex immunohistochemistry/immunofluorescence (mIHC/mIF) is a developing technology that facilitates the evaluation of multiple, simultaneous protein expressions at single-cell resolution while preserving tissue architecture. These approaches have shown great potential for biomarker discovery, yet many challenges remain. Importantly, streamlined cross-registration of multiplex immunofluorescence images with additional imaging modalities and immunohistochemistry (IHC) can help increase the plex and/or improve the quality of the data generated by potentiating downstream processes such as cell segmentation. To address this problem, a fully automated process was designed to perform a hierarchical, parallelizable, and deformable registration of multiplexed digital whole-slide images (WSIs). We generalized the calculation of mutual information as a registration criterion to an arbitrary number of dimensions, making it well suited for multiplexed imaging. We also used the self-information of a given IF channel as a criterion to select the optimal channels to use for registration. Additionally, as precise labeling of cellular membranes in situ is essential for robust cell segmentation, a pan-membrane immunohistochemical staining method was developed for incorporation into mIF panels or for use as an IHC followed by cross-registration. In this study, we demonstrate this process by registering whole-slide 6-plex/7-color mIF images with whole-slide brightfield mIHC images, including a CD3 and a pan-membrane stain. Our algorithm, WSI, mutual information registration (WSIMIR), performed highly accurate registration allowing the retrospective generation of an 8-plex/9-color, WSI, and outperformed 2 alternative automated methods for cross-registration by Jaccard index and Dice similarity coefficient (WSIMIR vs automated WARPY, P < .01 and P < .01, respectively, vs HALO + transformix, P = .083 and P = .049, respectively). Furthermore, the addition of a pan-membrane IHC stain cross-registered to an mIF panel facilitated improved automated cell segmentation across mIF WSIs, as measured by significantly increased correct detections, Jaccard index (0.78 vs 0.65), and Dice similarity coefficient (0.88 vs 0.79).
Collapse
Affiliation(s)
- Joshua Doyle
- Department of Astronomy and Physics, Johns Hopkins University, Baltimore, Maryland
| | - Benjamin F Green
- Department of Dermatology, Johns Hopkins University School of Medicine, Baltimore, Maryland; The Mark Foundation Center for Advanced Genomics and Imaging, Johns Hopkins University, Baltimore, Maryland; Bloomberg∼Kimmel Institute for Cancer Immunotherapy and Sidney Kimmel Comprehensive Cancer Center, Johns Hopkins University, Baltimore, Maryland
| | - Margaret Eminizer
- Department of Astronomy and Physics, Johns Hopkins University, Baltimore, Maryland; Institute for Data Intensive Engineering and Science, Johns Hopkins University, Baltimore, Maryland
| | - Daniel Jimenez-Sanchez
- Department of Dermatology, Johns Hopkins University School of Medicine, Baltimore, Maryland
| | - Steve Lu
- Department of Dermatology, Johns Hopkins University School of Medicine, Baltimore, Maryland
| | - Elizabeth L Engle
- Department of Dermatology, Johns Hopkins University School of Medicine, Baltimore, Maryland; The Mark Foundation Center for Advanced Genomics and Imaging, Johns Hopkins University, Baltimore, Maryland; Bloomberg∼Kimmel Institute for Cancer Immunotherapy and Sidney Kimmel Comprehensive Cancer Center, Johns Hopkins University, Baltimore, Maryland
| | - Haiying Xu
- Department of Dermatology, Johns Hopkins University School of Medicine, Baltimore, Maryland; The Mark Foundation Center for Advanced Genomics and Imaging, Johns Hopkins University, Baltimore, Maryland; Bloomberg∼Kimmel Institute for Cancer Immunotherapy and Sidney Kimmel Comprehensive Cancer Center, Johns Hopkins University, Baltimore, Maryland
| | - Aleksandra Ogurtsova
- Department of Dermatology, Johns Hopkins University School of Medicine, Baltimore, Maryland; The Mark Foundation Center for Advanced Genomics and Imaging, Johns Hopkins University, Baltimore, Maryland; Bloomberg∼Kimmel Institute for Cancer Immunotherapy and Sidney Kimmel Comprehensive Cancer Center, Johns Hopkins University, Baltimore, Maryland
| | - Jonathan Lai
- Department of Dermatology, Johns Hopkins University School of Medicine, Baltimore, Maryland
| | - Sigfredo Soto-Diaz
- Department of Dermatology, Johns Hopkins University School of Medicine, Baltimore, Maryland
| | - Jeffrey S Roskes
- Department of Astronomy and Physics, Johns Hopkins University, Baltimore, Maryland; Institute for Data Intensive Engineering and Science, Johns Hopkins University, Baltimore, Maryland
| | - Julie S Deutsch
- Department of Dermatology, Johns Hopkins University School of Medicine, Baltimore, Maryland
| | - Janis M Taube
- Department of Dermatology, Johns Hopkins University School of Medicine, Baltimore, Maryland; The Mark Foundation Center for Advanced Genomics and Imaging, Johns Hopkins University, Baltimore, Maryland; Bloomberg∼Kimmel Institute for Cancer Immunotherapy and Sidney Kimmel Comprehensive Cancer Center, Johns Hopkins University, Baltimore, Maryland; Department of Pathology, Johns Hopkins University School of Medicine, Baltimore, Maryland
| | - Joel C Sunshine
- Department of Dermatology, Johns Hopkins University School of Medicine, Baltimore, Maryland; Bloomberg∼Kimmel Institute for Cancer Immunotherapy and Sidney Kimmel Comprehensive Cancer Center, Johns Hopkins University, Baltimore, Maryland; Department of Pathology, Johns Hopkins University School of Medicine, Baltimore, Maryland; Department of Biomedical Engineering, Johns Hopkins University School of Medicine, Baltimore, Maryland; Johns Hopkins Center for Translational Immunoengineering, Johns Hopkins University School of Medicine, Baltimore, Maryland.
| | - Alexander S Szalay
- Department of Astronomy and Physics, Johns Hopkins University, Baltimore, Maryland; The Mark Foundation Center for Advanced Genomics and Imaging, Johns Hopkins University, Baltimore, Maryland; Institute for Data Intensive Engineering and Science, Johns Hopkins University, Baltimore, Maryland
| |
Collapse
|
3
|
Gatenbee CD, Baker AM, Prabhakaran S, Swinyard O, Slebos RJC, Mandal G, Mulholland E, Andor N, Marusyk A, Leedham S, Conejo-Garcia JR, Chung CH, Robertson-Tessi M, Graham TA, Anderson ARA. Virtual alignment of pathology image series for multi-gigapixel whole slide images. Nat Commun 2023; 14:4502. [PMID: 37495577 PMCID: PMC10372014 DOI: 10.1038/s41467-023-40218-9] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/16/2022] [Accepted: 07/13/2023] [Indexed: 07/28/2023] Open
Abstract
Interest in spatial omics is on the rise, but generation of highly multiplexed images remains challenging, due to cost, expertise, methodical constraints, and access to technology. An alternative approach is to register collections of whole slide images (WSI), generating spatially aligned datasets. WSI registration is a two-part problem, the first being the alignment itself and the second the application of transformations to huge multi-gigapixel images. To address both challenges, we developed Virtual Alignment of pathoLogy Image Series (VALIS), software which enables generation of highly multiplexed images by aligning any number of brightfield and/or immunofluorescent WSI, the results of which can be saved in the ome.tiff format. Benchmarking using publicly available datasets indicates VALIS provides state-of-the-art accuracy in WSI registration and 3D reconstruction. Leveraging existing open-source software tools, VALIS is written in Python, providing a free, fast, scalable, robust, and easy-to-use pipeline for registering multi-gigapixel WSI, facilitating downstream spatial analyses.
Collapse
Affiliation(s)
- Chandler D Gatenbee
- Department of Integrated Mathematical Oncology, H. Lee Moffitt Cancer Center & Research Institute, 12902 Magnolia Drive, SRB 4, Tampa, FL, 336122, USA.
| | - Ann-Marie Baker
- Evolution and Cancer Laboratory, Centre for Genomics and Computational Biology, Barts Cancer Institute, Queen Mary University of London, London, EC1M 6BQ, UK
| | - Sandhya Prabhakaran
- Department of Integrated Mathematical Oncology, H. Lee Moffitt Cancer Center & Research Institute, 12902 Magnolia Drive, SRB 4, Tampa, FL, 336122, USA
| | - Ottilie Swinyard
- Evolution and Cancer Laboratory, Centre for Genomics and Computational Biology, Barts Cancer Institute, Queen Mary University of London, London, EC1M 6BQ, UK
| | - Robbert J C Slebos
- Department of Head and Neck-Endocrine Oncology, H. Lee Moffitt Cancer Center & Research Institute, 12902 Magnolia Drive, CSB 6, Tampa, FL, USA
| | - Gunjan Mandal
- Department of Immunology, H. Lee Moffitt Cancer Center & Research Institute, 12902 Magnolia Drive, MRC, Tampa, FL, 336122, USA
| | - Eoghan Mulholland
- Wellcome Centre for Human Genetics, University of Oxford, Oxford, OX37BN, UK
| | - Noemi Andor
- Department of Integrated Mathematical Oncology, H. Lee Moffitt Cancer Center & Research Institute, 12902 Magnolia Drive, SRB 4, Tampa, FL, 336122, USA
| | - Andriy Marusyk
- Department of Cancer Physiology, H. Lee Moffitt Cancer Center & Research Institute, 12902 Magnolia Drive, SRB 4, Tampa, FL, USA
| | - Simon Leedham
- Wellcome Centre for Human Genetics, University of Oxford, Oxford, OX37BN, UK
| | - Jose R Conejo-Garcia
- Department of Immunology, H. Lee Moffitt Cancer Center & Research Institute, 12902 Magnolia Drive, MRC, Tampa, FL, 336122, USA
| | - Christine H Chung
- Department of Head and Neck-Endocrine Oncology, H. Lee Moffitt Cancer Center & Research Institute, 12902 Magnolia Drive, CSB 6, Tampa, FL, USA
| | - Mark Robertson-Tessi
- Department of Integrated Mathematical Oncology, H. Lee Moffitt Cancer Center & Research Institute, 12902 Magnolia Drive, SRB 4, Tampa, FL, 336122, USA
| | - Trevor A Graham
- Evolution and Cancer Laboratory, Centre for Genomics and Computational Biology, Barts Cancer Institute, Queen Mary University of London, London, EC1M 6BQ, UK
| | - Alexander R A Anderson
- Department of Integrated Mathematical Oncology, H. Lee Moffitt Cancer Center & Research Institute, 12902 Magnolia Drive, SRB 4, Tampa, FL, 336122, USA.
| |
Collapse
|
4
|
Jiang J, Moore R, Jordan CE, Guo R, Maus RL, Liu H, Goode E, Markovic SN, Wang C. Multiplex Immunofluorescence Image Quality Checking Using DAPI Channel-referenced Evaluation. J Histochem Cytochem 2023; 71:121-130. [PMID: 36960831 PMCID: PMC10084566 DOI: 10.1369/00221554231161693] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/02/2022] [Accepted: 02/14/2023] [Indexed: 03/25/2023] Open
Abstract
Multiplex immunofluorescence (MxIF) images provide detailed information of cell composition and spatial context for biomedical research. However, compromised data quality could lead to research biases. Comprehensive image quality checking (QC) is essential for reliable downstream analysis. As a reliable and specific staining of cell nuclei, 4',6-diamidino-2-phenylindole (DAPI) signals were used as references for tissue localization and auto-focusing across MxIF staining-scanning-bleaching iterations and could potentially be reused for QC. To confirm the feasibility of using DAPI as QC reference, pixel-level DAPI values were extracted to calculate signal fluctuations and tissue content similarities in staining-scanning-bleaching iterations for identifying quality issues. Concordance between automatic quantification and human experts' annotations were evaluated on a data set consisting of 348 fields of view (FOVs) with 45 immune and tumor cell markers. Cell distribution differences between subsets of QC-pass vs QC-failed FOVs were compared to investigate the downstream effects. Results showed that 87.3% FOVs with tissue damage and 73.4% of artifacts were identified. QC-failed FOVs showed elevated regional gathering in cellular feature space compared with the QC-pass FOVs. Our results supported that DAPI signals could be used as references for MxIF image QC, and low-quality FOVs identified by our method must be cautiously considered for downstream analyses.
Collapse
Affiliation(s)
- Jun Jiang
- Department of Quantitative Health Sciences, Mayo Clinic, Rochester, Minnesota
| | - Raymond Moore
- Department of Quantitative Health Sciences, Mayo Clinic, Rochester, Minnesota
| | - Clarissa E. Jordan
- Department of Laboratory Medicine and Pathology, Mayo Clinic, Rochester, Minnesota
| | - Ruifeng Guo
- Department of Laboratory Medicine and Pathology, Mayo Clinic, Rochester, Minnesota
| | - Rachel L. Maus
- Department of Oncology, Mayo Clinic, Rochester, Minnesota
| | - Hongfang Liu
- Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, Minnesota
| | - Ellen Goode
- Department of Quantitative Health Sciences, Mayo Clinic, Rochester, Minnesota
| | | | - Chen Wang
- Department of Quantitative Health Sciences, Mayo Clinic, Rochester, Minnesota
| |
Collapse
|
5
|
Oliveira LC, Lai Z, Harvey D, Nzenkue K, Jin LW, Decarli C, Chuah CN, Dugger BN. Preanalytic variable effects on segmentation and quantification machine learning algorithms for amyloid-β analyses on digitized human brain slides. J Neuropathol Exp Neurol 2023; 82:212-220. [PMID: 36692190 PMCID: PMC9941825 DOI: 10.1093/jnen/nlac132] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/25/2023] Open
Abstract
Computational machine learning (ML)-based frameworks could be advantageous for scalable analyses in neuropathology. A recent deep learning (DL) framework has shown promise in automating the processes of visualizing and quantifying different types of amyloid-β deposits as well as segmenting white matter (WM) from gray matter (GM) on digitized immunohistochemically stained slides. However, this framework has only been trained and evaluated on amyloid-β-stained slides with minimal changes in preanalytic variables. In this study, we evaluated select preanalytical variables including magnification, compression rate, and storage format using three digital slides scanners (Zeiss Axioscan Z1, Leica Aperio AT2, and Leica Aperio GT 450), on over 60 whole slide images, in a cohort of 14 cases having a spectrum of amyloid-β deposits. We conducted statistical comparisons of preanalytic variables with repeated measures analysis of variance evaluating the outputs of two DL frameworks for segmentation and object classification tasks. For both WM/GM segmentation and amyloid-β plaque classification tasks, there were statistical differences with respect to scanner types (p < 0.05) and magnifications (p < 0.05). Although small numbers of cases were analyzed, this pilot study highlights the significance of preanalytic variables that may alter the performance of ML algorithms.
Collapse
Affiliation(s)
- Luca Cerny Oliveira
- Department of Electrical and Computer Engineering, University of California Davis, Davis, California, USA
| | - Zhengfeng Lai
- Department of Electrical and Computer Engineering, University of California Davis, Davis, California, USA
| | - Danielle Harvey
- Department of Public Health Sciences, University of California, Davis, Davis, California, USA
| | - Kevin Nzenkue
- Department of Pathology and Laboratory Medicine, University of California Davis, Sacramento, California, USA
| | - Lee-Way Jin
- Department of Pathology and Laboratory Medicine, University of California Davis, Sacramento, California, USA
| | - Charles Decarli
- Department of Neurology, University of California, Davis, Sacramento, California, USA
| | - Chen-Nee Chuah
- Department of Electrical and Computer Engineering, University of California Davis, Davis, California, USA
| | - Brittany N Dugger
- Department of Pathology and Laboratory Medicine, University of California Davis, Sacramento, California, USA
| |
Collapse
|
6
|
Wang CW, Lee YC, Khalil MA, Lin KY, Yu CP, Lien HC. Fast cross-staining alignment of gigapixel whole slide images with application to prostate cancer and breast cancer analysis. Sci Rep 2022; 12:11623. [PMID: 35803996 PMCID: PMC9270377 DOI: 10.1038/s41598-022-15962-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/30/2021] [Accepted: 07/01/2022] [Indexed: 12/24/2022] Open
Abstract
Joint analysis of multiple protein expressions and tissue morphology patterns is important for disease diagnosis, treatment planning, and drug development, requiring cross-staining alignment of multiple immunohistochemical and histopathological slides. However, cross-staining alignment of enormous gigapixel whole slide images (WSIs) at single cell precision is difficult. Apart from gigantic data dimensions of WSIs, there are large variations on the cell appearance and tissue morphology across different staining together with morphological deformations caused by slide preparation. The goal of this study is to build an image registration framework for cross-staining alignment of gigapixel WSIs of histopathological and immunohistochemical microscopic slides and assess its clinical applicability. To the authors' best knowledge, this is the first study to perform real time fully automatic cross staining alignment of WSIs with 40× and 20× objective magnification. The proposed WSI registration framework consists of a rapid global image registration module, a real time interactive field of view (FOV) localization model and a real time propagated multi-level image registration module. In this study, the proposed method is evaluated on two kinds of cancer datasets from two hospitals using different digital scanners, including a dual staining breast cancer data set with 43 hematoxylin and eosin (H&E) WSIs and 43 immunohistochemical (IHC) CK(AE1/AE3) WSIs, and a triple staining prostate cancer data set containing 30 H&E WSIs, 30 IHC CK18 WSIs, and 30 IHC HMCK WSIs. In evaluation, the registration performance is measured by not only registration accuracy but also computational time. The results show that the proposed method achieves high accuracy of 0.833 ± 0.0674 for the triple-staining prostate cancer data set and 0.931 ± 0.0455 for the dual-staining breast cancer data set, respectively, and takes only 4.34 s per WSI registration on average. In addition, for 30.23% data, the proposed method takes less than 1 s for WSI registration. In comparison with the benchmark methods, the proposed method demonstrates superior performance in registration accuracy and computational time, which has great potentials for assisting medical doctors to identify cancerous tissues and determine the cancer stage in clinical practice.
Collapse
Affiliation(s)
- Ching-Wei Wang
- Graduate Institute of Applied Science and Technology, National Taiwan University of Science and Technology, Taipei, Taiwan. .,Graduate Institute of Biomedical Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan.
| | - Yu-Ching Lee
- Graduate Institute of Applied Science and Technology, National Taiwan University of Science and Technology, Taipei, Taiwan
| | - Muhammad-Adil Khalil
- Graduate Institute of Applied Science and Technology, National Taiwan University of Science and Technology, Taipei, Taiwan
| | - Kuan-Yu Lin
- Graduate Institute of Biomedical Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan
| | - Cheng-Ping Yu
- Department of Pathology, Tri-Service General Hospital, Taipei, Taiwan.,Institute of Pathology and Parasitology, National Defense Medical Center, Taipei, Taiwan
| | - Huang-Chun Lien
- Department of Pathology, National Taiwan University Hospital, Taipei, Taiwan.,Graduate Institute of Pathology, National Taiwan University, Taipei, Taiwan
| |
Collapse
|
7
|
Bertram CA, Aubreville M, Donovan TA, Bartel A, Wilm F, Marzahl C, Assenmacher CA, Becker K, Bennett M, Corner S, Cossic B, Denk D, Dettwiler M, Gonzalez BG, Gurtner C, Haverkamp AK, Heier A, Lehmbecker A, Merz S, Noland EL, Plog S, Schmidt A, Sebastian F, Sledge DG, Smedley RC, Tecilla M, Thaiwong T, Fuchs-Baumgartinger A, Meuten DJ, Breininger K, Kiupel M, Maier A, Klopfleisch R. Computer-assisted mitotic count using a deep learning–based algorithm improves interobserver reproducibility and accuracy. Vet Pathol 2021; 59:211-226. [PMID: 34965805 PMCID: PMC8928234 DOI: 10.1177/03009858211067478] [Citation(s) in RCA: 16] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/15/2022]
Abstract
The mitotic count (MC) is an important histological parameter for prognostication of malignant neoplasms. However, it has inter- and intraobserver discrepancies due to difficulties in selecting the region of interest (MC-ROI) and in identifying or classifying mitotic figures (MFs). Recent progress in the field of artificial intelligence has allowed the development of high-performance algorithms that may improve standardization of the MC. As algorithmic predictions are not flawless, computer-assisted review by pathologists may ensure reliability. In the present study, we compared partial (MC-ROI preselection) and full (additional visualization of MF candidates and display of algorithmic confidence values) computer-assisted MC analysis to the routine (unaided) MC analysis by 23 pathologists for whole-slide images of 50 canine cutaneous mast cell tumors (ccMCTs). Algorithmic predictions aimed to assist pathologists in detecting mitotic hotspot locations, reducing omission of MFs, and improving classification against imposters. The interobserver consistency for the MC significantly increased with computer assistance (interobserver correlation coefficient, ICC = 0.92) compared to the unaided approach (ICC = 0.70). Classification into prognostic stratifications had a higher accuracy with computer assistance. The algorithmically preselected hotspot MC-ROIs had a consistently higher MCs than the manually selected MC-ROIs. Compared to a ground truth (developed with immunohistochemistry for phosphohistone H3), pathologist performance in detecting individual MF was augmented when using computer assistance (F1-score of 0.68 increased to 0.79) with a reduction in false negatives by 38%. The results of this study demonstrate that computer assistance may lead to more reproducible and accurate MCs in ccMCTs.
Collapse
Affiliation(s)
- Christof A. Bertram
- University of Veterinary Medicine, Vienna, Austria
- Freie Universität Berlin, Berlin, Germany
| | | | | | | | - Frauke Wilm
- Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany
| | - Christian Marzahl
- Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany
| | | | | | | | | | | | | | | | | | | | | | | | | | - Sophie Merz
- IDEXX Vet Med Labor GmbH, Kornwestheim, Germany
| | | | | | | | | | | | | | - Marco Tecilla
- Roche Pharmaceutical Research and Early Development (pRED), Basel, Switzerland
| | | | | | | | | | | | - Andreas Maier
- Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany
| | | |
Collapse
|
8
|
Paknezhad M, Loh SYM, Choudhury Y, Koh VKC, Yong TTK, Tan HS, Kanesvaran R, Tan PH, Peng JYS, Yu W, Tan YB, Loy YZ, Tan MH, Lee HK. Regional registration of whole slide image stacks containing major histological artifacts. BMC Bioinformatics 2020; 21:558. [PMID: 33276732 PMCID: PMC7718714 DOI: 10.1186/s12859-020-03907-6] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/10/2020] [Accepted: 11/25/2020] [Indexed: 11/29/2022] Open
Abstract
Background High resolution 2D whole slide imaging provides rich information about the tissue structure. This information can be a lot richer if these 2D images can be stacked into a 3D tissue volume. A 3D analysis, however, requires accurate reconstruction of the tissue volume from the 2D image stack. This task is not trivial due to the distortions such as tissue tearing, folding and missing at each slide. Performing registration for the whole tissue slices may be adversely affected by distorted tissue regions. Consequently, regional registration is found to be more effective. In this paper, we propose a new approach to an accurate and robust registration of regions of interest for whole slide images. We introduce the idea of multi-scale attention for registration. Results Using mean similarity index as the metric, the proposed algorithm (mean ± SD \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$0.84 \pm 0.11$$\end{document}0.84±0.11) followed by a fine registration algorithm (\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$0.86 \pm 0.08$$\end{document}0.86±0.08) outperformed the state-of-the-art linear whole tissue registration algorithm (\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$0.74 \pm 0.19$$\end{document}0.74±0.19) and the regional version of this algorithm (\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$0.81 \pm 0.15$$\end{document}0.81±0.15). The proposed algorithm also outperforms the state-of-the-art nonlinear registration algorithm (original: \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$0.82 \pm 0.12$$\end{document}0.82±0.12, regional: \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$0.77 \pm 0.22$$\end{document}0.77±0.22) for whole slide images and a recently proposed patch-based registration algorithm (patch size 256: \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$0.79 \pm 0.16$$\end{document}0.79±0.16 , patch size 512: \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$0.77 \pm 0.16$$\end{document}0.77±0.16) for medical images. Conclusion Using multi-scale attention mechanism leads to a more robust and accurate solution to the problem of regional registration of whole slide images corrupted in some parts by major histological artifacts in the imaged tissue.
Collapse
Affiliation(s)
- Mahsa Paknezhad
- Imaging Informatics Division, Bioinformatics Institute (BII), Agency for Science, Technology and Research (A*STAR), 30 Biopolis Street, 07-01, Matrix, 138671, Singapore, Singapore.
| | - Sheng Yang Michael Loh
- Imaging Informatics Division, Bioinformatics Institute (BII), Agency for Science, Technology and Research (A*STAR), 30 Biopolis Street, 07-01, Matrix, 138671, Singapore, Singapore
| | - Yukti Choudhury
- Lucence Diagnostics, 217 Henderson Road, 03-08, Henderson Industrial Park, 159555, Singapore, Singapore.,Institute of Bioengineering and Nanotechnology, 31 Biopolis Way, The Nanos 09-01, 138669, Singapore, Singapore
| | | | - Timothy Tay Kwang Yong
- Institute of Molecular and Cell Biology, 61 Biopolis Drive, 138673, Singapore, Singapore
| | - Hui Shan Tan
- Image and Pervasive Access Lab (IPAL), CNRS UMI 2955, 1 Fusionopolis Way, 138632, Singapore, Singapore
| | - Ravindran Kanesvaran
- Institute of Molecular and Cell Biology, 61 Biopolis Drive, 138673, Singapore, Singapore
| | - Puay Hoon Tan
- Singapore General Hospital, Outram Road, 169608, Singapore, Singapore
| | | | - Weimiao Yu
- National Cancer Centre Singapore, 11 Hospital Drive, 169610, Singapore, Singapore
| | - Yongcheng Benjamin Tan
- Institute of Molecular and Cell Biology, 61 Biopolis Drive, 138673, Singapore, Singapore
| | - Yong Zhen Loy
- Singapore General Hospital, Outram Road, 169608, Singapore, Singapore
| | - Min-Han Tan
- Lucence Diagnostics, 217 Henderson Road, 03-08, Henderson Industrial Park, 159555, Singapore, Singapore.,Institute of Bioengineering and Nanotechnology, 31 Biopolis Way, The Nanos 09-01, 138669, Singapore, Singapore.,Institute of Molecular and Cell Biology, 61 Biopolis Drive, 138673, Singapore, Singapore
| | - Hwee Kuan Lee
- Imaging Informatics Division, Bioinformatics Institute (BII), Agency for Science, Technology and Research (A*STAR), 30 Biopolis Street, 07-01, Matrix, 138671, Singapore, Singapore.,National University of Singapore, 21 Lower Kent Ridge Rd, 119077, Singapore, Singapore.,Image and Pervasive Access Lab (IPAL), CNRS UMI 2955, 1 Fusionopolis Way, 138632, Singapore, Singapore.,Singapore Eye Research Institute, 20 College Road, 169856, Singapore, Singapore
| |
Collapse
|
9
|
Jiang J, Prodduturi N, Chen D, Gu Q, Flotte T, Feng Q, Hart S. Image-to-image translation for automatic ink removal in whole slide images. J Med Imaging (Bellingham) 2020; 7:057502. [PMID: 33102624 DOI: 10.1117/1.jmi.7.5.057502] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/13/2020] [Accepted: 09/21/2020] [Indexed: 11/14/2022] Open
Abstract
Purpose: Deep learning models are showing promise in digital pathology to aid diagnoses. Training complex models requires a significant amount and diversity of well-annotated data, typically housed in institutional archives. These slides often contain clinically meaningful markings to indicate regions of interest. If slides are scanned with the ink present, then the downstream model may end up looking for regions with ink before making a classification. If scanned without the markings, the information regarding where the relevant regions are located is lost. A compromise solution is to scan the slide with the annotations present but digitally remove them. Approach: We proposed a straightforward framework to digitally remove ink markings from whole slide images using a conditional generative adversarial network based on Pix2Pix. Results: The peak signal-to-noise ratio increased 30%, structural similarity index increased 20%, and visual information fidelity increased 200% relative to previous methods. Conclusions: When comparing our digital removal of marked images with rescans of clean slides, our method qualitatively and quantitatively exceeds current benchmarks, opening the possibility of using archived clinical samples as resources to fuel the next generation of deep learning models for digital pathology.
Collapse
Affiliation(s)
- Jun Jiang
- Mayo Clinic, Health Science Research Department, Rochester, United States
| | - Naresh Prodduturi
- Mayo Clinic, Health Science Research Department, Rochester, United States
| | - David Chen
- Mayo Clinic, Health Science Research Department, Rochester, United States
| | - Qiangqiang Gu
- Mayo Clinic, Health Science Research Department, Rochester, United States
| | - Thomas Flotte
- Mayo Clinic, Health Science Research Department, Rochester, United States
| | - Qianjin Feng
- Southern Medical University, School of Biomedical Engineering, Guangzhou, China
| | - Steven Hart
- Mayo Clinic, Health Science Research Department, Rochester, United States
| |
Collapse
|