1
|
Zhang P, Gao C, Huang Y, Chen X, Pan Z, Wang L, Dong D, Li S, Qi X. Artificial intelligence in liver imaging: methods and applications. Hepatol Int 2024; 18:422-434. [PMID: 38376649 DOI: 10.1007/s12072-023-10630-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/15/2023] [Accepted: 12/18/2023] [Indexed: 02/21/2024]
Abstract
Liver disease is regarded as one of the major health threats to humans. Radiographic assessments hold promise in terms of addressing the current demands for precisely diagnosing and treating liver diseases, and artificial intelligence (AI), which excels at automatically making quantitative assessments of complex medical image characteristics, has made great strides regarding the qualitative interpretation of medical imaging by clinicians. Here, we review the current state of medical-imaging-based AI methodologies and their applications concerning the management of liver diseases. We summarize the representative AI methodologies in liver imaging with focusing on deep learning, and illustrate their promising clinical applications across the spectrum of precise liver disease detection, diagnosis and treatment. We also address the current challenges and future perspectives of AI in liver imaging, with an emphasis on feature interpretability, multimodal data integration and multicenter study. Taken together, it is revealed that AI methodologies, together with the large volume of available medical image data, might impact the future of liver disease care.
Collapse
Affiliation(s)
- Peng Zhang
- Institute for TCM-X, MOE Key Laboratory of Bioinformatics, Bioinformatics Division, BNRIST, Department of Automation, Tsinghua University, Beijing, China
| | - Chaofei Gao
- Institute for TCM-X, MOE Key Laboratory of Bioinformatics, Bioinformatics Division, BNRIST, Department of Automation, Tsinghua University, Beijing, China
| | - Yifei Huang
- Department of Gastroenterology, The Third Affiliated Hospital of Sun Yat-sen University, Guangzhou, China
| | - Xiangyi Chen
- Institute for TCM-X, MOE Key Laboratory of Bioinformatics, Bioinformatics Division, BNRIST, Department of Automation, Tsinghua University, Beijing, China
| | - Zhuoshi Pan
- Institute for TCM-X, MOE Key Laboratory of Bioinformatics, Bioinformatics Division, BNRIST, Department of Automation, Tsinghua University, Beijing, China
| | - Lan Wang
- Institute for TCM-X, MOE Key Laboratory of Bioinformatics, Bioinformatics Division, BNRIST, Department of Automation, Tsinghua University, Beijing, China
| | - Di Dong
- CAS Key Laboratory of Molecular Imaging, Beijing Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China
| | - Shao Li
- Institute for TCM-X, MOE Key Laboratory of Bioinformatics, Bioinformatics Division, BNRIST, Department of Automation, Tsinghua University, Beijing, China.
| | - Xiaolong Qi
- Center of Portal Hypertension, Department of Radiology, Zhongda Hospital, Medical School, Nurturing Center of Jiangsu Province for State Laboratory of AI Imaging & Interventional Radiology, Southeast University, Nanjing, China.
| |
Collapse
|
2
|
Bashir RMS, Qaiser T, Raza SEA, Rajpoot NM. Consistency regularisation in varying contexts and feature perturbations for semi-supervised semantic segmentation of histology images. Med Image Anal 2024; 91:102997. [PMID: 37866169 DOI: 10.1016/j.media.2023.102997] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/12/2023] [Revised: 10/05/2023] [Accepted: 10/06/2023] [Indexed: 10/24/2023]
Abstract
Semantic segmentation of various tissue and nuclei types in histology images is fundamental to many downstream tasks in the area of computational pathology (CPath). In recent years, Deep Learning (DL) methods have been shown to perform well on segmentation tasks but DL methods generally require a large amount of pixel-wise annotated data. Pixel-wise annotation sometimes requires expert's knowledge and time which is laborious and costly to obtain. In this paper, we present a consistency based semi-supervised learning (SSL) approach that can help mitigate this challenge by exploiting a large amount of unlabelled data for model training thus alleviating the need for a large annotated dataset. However, SSL models might also be susceptible to changing context and features perturbations exhibiting poor generalisation due to the limited training data. We propose an SSL method that learns robust features from both labelled and unlabelled images by enforcing consistency against varying contexts and feature perturbations. The proposed method incorporates context-aware consistency by contrasting pairs of overlapping images in a pixel-wise manner from changing contexts resulting in robust and context invariant features. We show that cross-consistency training makes the encoder features invariant to different perturbations and improves the prediction confidence. Finally, entropy minimisation is employed to further boost the confidence of the final prediction maps from unlabelled data. We conduct an extensive set of experiments on two publicly available large datasets (BCSS and MoNuSeg) and show superior performance compared to the state-of-the-art methods.
Collapse
Affiliation(s)
| | - Talha Qaiser
- Tissue Image Analytics Centre, University of Warwick, Coventry, United Kingdom.
| | - Shan E Ahmed Raza
- Tissue Image Analytics Centre, University of Warwick, Coventry, United Kingdom.
| | - Nasir M Rajpoot
- Tissue Image Analytics Centre, University of Warwick, Coventry, United Kingdom; The Alan Turing Institute, London, United Kingdom; Histofy Ltd, United Kingdom; Department of Pathology, University Hospitals Coventry & Warwickshire, United Kingdom.
| |
Collapse
|
3
|
Evans H, Hero E, Minhas F, Wahab N, Dodd K, Sahota H, Ganguly R, Robinson A, Neerudu M, Blessing E, Borkar P, Snead D. Standardized Clinical Annotation of Digital Histopathology Slides at the Point of Diagnosis. Mod Pathol 2023; 36:100297. [PMID: 37544362 DOI: 10.1016/j.modpat.2023.100297] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/21/2023] [Revised: 06/26/2023] [Accepted: 07/25/2023] [Indexed: 08/08/2023]
Abstract
As digital pathology replaces conventional glass slide microscopy as a means of reporting cellular pathology samples, the annotation of digital pathology whole slide images is rapidly becoming part of a pathologist's regular practice. Currently, there is no recognizable organization of these annotations, and as a result, pathologists adopt an arbitrary approach to defining regions of interest, leading to irregularity and inconsistency and limiting the downstream efficient use of this valuable effort. In this study, we propose a Standardized Annotation Reporting Style for digital whole slide images. We formed a list of 167 commonly annotated entities (under 12 specialty subcategories) based on review of Royal College of Pathologists and College of American Pathologists documents, feedback from reporting pathologists in our NHS department, and experience in developing annotation dictionaries for PathLAKE research projects. Each entity was assigned a suitable annotation shape, SNOMED CT (SNOMED International) code, and unique color. Additionally, as an example of how the approach could be expanded to specific tumor types, all lung tumors in the fifth World Health Organization of thoracic tumors 2021 were included. The proposed standardization of annotations increases their utility, making them identifiable at low power and searchable across and between cases. This would aid pathologists reporting and reviewing cases and enable annotations to be used for research. This structured approach could serve as the basis for an industry standard and be easily adopted to ensure maximum functionality and efficiency in the use of annotations made during routine clinical examination of digital slides.
Collapse
Affiliation(s)
- Harriet Evans
- Histopathology Department, University Hospitals Coventry and Warwickshire NHS Trust, Coventry, United Kingdom; Warwick Medical School, University of Warwick, Coventry, United Kingdom.
| | - Emily Hero
- Histopathology Department, University Hospitals Coventry and Warwickshire NHS Trust, Coventry, United Kingdom; Histopathology Department, University Hospitals of Leicester NHS Trust, Leicester, United Kingdom
| | - Fayyaz Minhas
- Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, Coventry, United Kingdom
| | - Noorul Wahab
- Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, Coventry, United Kingdom
| | - Katherine Dodd
- Histopathology Department, University Hospitals Coventry and Warwickshire NHS Trust, Coventry, United Kingdom
| | - Harvir Sahota
- Histopathology Department, University Hospitals Coventry and Warwickshire NHS Trust, Coventry, United Kingdom; Department of Psychiatry, Coventry and Warwickshire Partnership Trust, Coventry, United Kingdom
| | - Ratnadeep Ganguly
- Histopathology Department, University Hospitals Coventry and Warwickshire NHS Trust, Coventry, United Kingdom
| | - Andrew Robinson
- Histopathology Department, University Hospitals Coventry and Warwickshire NHS Trust, Coventry, United Kingdom
| | - Manjuvani Neerudu
- Histopathology Department, University Hospitals Coventry and Warwickshire NHS Trust, Coventry, United Kingdom
| | - Elaine Blessing
- Histopathology Department, University Hospitals Coventry and Warwickshire NHS Trust, Coventry, United Kingdom
| | - Pallavi Borkar
- Histopathology Department, University Hospitals Coventry and Warwickshire NHS Trust, Coventry, United Kingdom
| | - David Snead
- Histopathology Department, University Hospitals Coventry and Warwickshire NHS Trust, Coventry, United Kingdom; Warwick Medical School, University of Warwick, Coventry, United Kingdom
| |
Collapse
|
4
|
Menotti L, Silvello G, Atzori M, Boytcheva S, Ciompi F, Di Nunzio GM, Fraggetta F, Giachelle F, Irrera O, Marchesin S, Marini N, Müller H, Primov T. Modelling digital health data: The ExaMode ontology for computational pathology. J Pathol Inform 2023; 14:100332. [PMID: 37705689 PMCID: PMC10495665 DOI: 10.1016/j.jpi.2023.100332] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/09/2023] [Revised: 07/14/2023] [Accepted: 08/16/2023] [Indexed: 09/15/2023] Open
Abstract
Computational pathology can significantly benefit from ontologies to standardize the employed nomenclature and help with knowledge extraction processes for high-quality annotated image datasets. The end goal is to reach a shared model for digital pathology to overcome data variability and integration problems. Indeed, data annotation in such a specific domain is still an unsolved challenge and datasets cannot be steadily reused in diverse contexts due to heterogeneity issues of the adopted labels, multilingualism, and different clinical practices. Material and methods This paper presents the ExaMode ontology, modeling the histopathology process by considering 3 key cancer diseases (colon, cervical, and lung tumors) and celiac disease. The ExaMode ontology has been designed bottom-up in an iterative fashion with continuous feedback and validation from pathologists and clinicians. The ontology is organized into 5 semantic areas that defines an ontological template to model any disease of interest in histopathology. Results The ExaMode ontology is currently being used as a common semantic layer in: (i) an entity linking tool for the automatic annotation of medical records; (ii) a web-based collaborative annotation tool for histopathology text reports; and (iii) a software platform for building holistic solutions integrating multimodal histopathology data. Discussion The ontology ExaMode is a key means to store data in a graph database according to the RDF data model. The creation of an RDF dataset can help develop more accurate algorithms for image analysis, especially in the field of digital pathology. This approach allows for seamless data integration and a unified query access point, from which we can extract relevant clinical insights about the considered diseases using SPARQL queries.
Collapse
Affiliation(s)
- Laura Menotti
- Department of Information Engineering, University of Padua, Padova, Italy
| | - Gianmaria Silvello
- Department of Information Engineering, University of Padua, Padova, Italy
| | - Manfredo Atzori
- Information Systems Institute, University of Applied Sciences Western Switzerland, Delémont, Switzerland
- Department of Neuroscience, University of Padua, Padova, Italy
| | | | - Francesco Ciompi
- Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands
| | | | | | - Fabio Giachelle
- Department of Information Engineering, University of Padua, Padova, Italy
| | - Ornella Irrera
- Department of Information Engineering, University of Padua, Padova, Italy
| | - Stefano Marchesin
- Department of Information Engineering, University of Padua, Padova, Italy
| | - Niccolò Marini
- Information Systems Institute, University of Applied Sciences Western Switzerland, Delémont, Switzerland
| | - Henning Müller
- Information Systems Institute, University of Applied Sciences Western Switzerland, Delémont, Switzerland
| | | |
Collapse
|
5
|
Ding K, Zhou M, Wang H, Gevaert O, Metaxas D, Zhang S. A Large-scale Synthetic Pathological Dataset for Deep Learning-enabled Segmentation of Breast Cancer. Sci Data 2023; 10:231. [PMID: 37085533 PMCID: PMC10121551 DOI: 10.1038/s41597-023-02125-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/12/2022] [Accepted: 03/31/2023] [Indexed: 04/23/2023] Open
Abstract
The success of training computer-vision models heavily relies on the support of large-scale, real-world images with annotations. Yet such an annotation-ready dataset is difficult to curate in pathology due to the privacy protection and excessive annotation burden. To aid in computational pathology, synthetic data generation, curation, and annotation present a cost-effective means to quickly enable data diversity that is required to boost model performance at different stages. In this study, we introduce a large-scale synthetic pathological image dataset paired with the annotation for nuclei semantic segmentation, termed as Synthetic Nuclei and annOtation Wizard (SNOW). The proposed SNOW is developed via a standardized workflow by applying the off-the-shelf image generator and nuclei annotator. The dataset contains overall 20k image tiles and 1,448,522 annotated nuclei with the CC-BY license. We show that SNOW can be used in both supervised and semi-supervised training scenarios. Extensive results suggest that synthetic-data-trained models are competitive under a variety of model training settings, expanding the scope of better using synthetic images for enhancing downstream data-driven clinical tasks.
Collapse
Affiliation(s)
- Kexin Ding
- Department of Computer Science, University of North Carolina at Charlotte, Charlotte, NC, 28262, USA
| | - Mu Zhou
- Sensebrain Research, San Jose, CA, 95131, USA
| | - He Wang
- Department of Pathology, Yale University, New Haven, CT, 06520, USA
| | - Olivier Gevaert
- Stanford Center for Biomedical Informatics Research, Department of Medicine and Biomedical Data Science, Stanford University, Stanford, CA, 94305, USA
| | - Dimitris Metaxas
- Department of Computer Science, Rutgers University, New Brunswick, NJ, 08901, USA
| | - Shaoting Zhang
- Shanghai Artificial Intelligence Laboratory, Shanghai, 200232, China.
| |
Collapse
|
6
|
Annotating for Artificial Intelligence Applications in Digital Pathology: A Practical Guide for Pathologists and Researchers. Mod Pathol 2023; 36:100086. [PMID: 36788085 DOI: 10.1016/j.modpat.2022.100086] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/26/2022] [Revised: 11/24/2022] [Accepted: 12/14/2022] [Indexed: 01/13/2023]
Abstract
Training machine learning models for artificial intelligence (AI) applications in pathology often requires extensive annotation by human experts, but there is little guidance on the subject. In this work, we aimed to describe our experience and provide a simple, useful, and practical guide addressing annotation strategies for AI development in computational pathology. Annotation methodology will vary significantly depending on the specific study's objectives, but common difficulties will be present across different settings. We summarize key aspects and issue guiding principles regarding team interaction, ground-truth quality assessment, different annotation types, and available software and hardware options and address common difficulties while annotating. This guide was specifically designed for pathology annotation, intending to help pathologists, other researchers, and AI developers with this process.
Collapse
|
7
|
Li D, Pehrson LM, Tøttrup L, Fraccaro M, Bonnevie R, Thrane J, Sørensen PJ, Rykkje A, Andersen TT, Steglich-Arnholm H, Stærk DMR, Borgwardt L, Hansen KL, Darkner S, Carlsen JF, Nielsen MB. Inter- and Intra-Observer Agreement When Using a Diagnostic Labeling Scheme for Annotating Findings on Chest X-rays-An Early Step in the Development of a Deep Learning-Based Decision Support System. Diagnostics (Basel) 2022; 12:diagnostics12123112. [PMID: 36553118 PMCID: PMC9776917 DOI: 10.3390/diagnostics12123112] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/19/2022] [Revised: 11/21/2022] [Accepted: 11/26/2022] [Indexed: 12/14/2022] Open
Abstract
Consistent annotation of data is a prerequisite for the successful training and testing of artificial intelligence-based decision support systems in radiology. This can be obtained by standardizing terminology when annotating diagnostic images. The purpose of this study was to evaluate the annotation consistency among radiologists when using a novel diagnostic labeling scheme for chest X-rays. Six radiologists with experience ranging from one to sixteen years, annotated a set of 100 fully anonymized chest X-rays. The blinded radiologists annotated on two separate occasions. Statistical analyses were done using Randolph's kappa and PABAK, and the proportions of specific agreements were calculated. Fair-to-excellent agreement was found for all labels among the annotators (Randolph's Kappa, 0.40-0.99). The PABAK ranged from 0.12 to 1 for the two-reader inter-rater agreement and 0.26 to 1 for the intra-rater agreement. Descriptive and broad labels achieved the highest proportion of positive agreement in both the inter- and intra-reader analyses. Annotating findings with specific, interpretive labels were found to be difficult for less experienced radiologists. Annotating images with descriptive labels may increase agreement between radiologists with different experience levels compared to annotation with interpretive labels.
Collapse
Affiliation(s)
- Dana Li
- Department of Diagnostic Radiology, Copenhagen University Hospital, Rigshospitalet, 2100 Copenhagen, Denmark
- Department of Clinical Medicine, University of Copenhagen, 2100 Copenhagen, Denmark
- Correspondence:
| | - Lea Marie Pehrson
- Department of Diagnostic Radiology, Copenhagen University Hospital, Rigshospitalet, 2100 Copenhagen, Denmark
- Department of Computer Science, University of Copenhagen, 2100 Copenhagen, Denmark
| | | | | | | | | | - Peter Jagd Sørensen
- Department of Diagnostic Radiology, Copenhagen University Hospital, Rigshospitalet, 2100 Copenhagen, Denmark
- Department of Clinical Medicine, University of Copenhagen, 2100 Copenhagen, Denmark
| | - Alexander Rykkje
- Department of Diagnostic Radiology, Copenhagen University Hospital, Rigshospitalet, 2100 Copenhagen, Denmark
- Department of Clinical Medicine, University of Copenhagen, 2100 Copenhagen, Denmark
| | - Tobias Thostrup Andersen
- Department of Diagnostic Radiology, Copenhagen University Hospital, Rigshospitalet, 2100 Copenhagen, Denmark
| | - Henrik Steglich-Arnholm
- Department of Diagnostic Radiology, Copenhagen University Hospital, Rigshospitalet, 2100 Copenhagen, Denmark
| | - Dorte Marianne Rohde Stærk
- Department of Diagnostic Radiology, Copenhagen University Hospital, Rigshospitalet, 2100 Copenhagen, Denmark
| | - Lotte Borgwardt
- Department of Diagnostic Radiology, Copenhagen University Hospital, Rigshospitalet, 2100 Copenhagen, Denmark
| | - Kristoffer Lindskov Hansen
- Department of Diagnostic Radiology, Copenhagen University Hospital, Rigshospitalet, 2100 Copenhagen, Denmark
- Department of Clinical Medicine, University of Copenhagen, 2100 Copenhagen, Denmark
| | - Sune Darkner
- Department of Computer Science, University of Copenhagen, 2100 Copenhagen, Denmark
| | - Jonathan Frederik Carlsen
- Department of Diagnostic Radiology, Copenhagen University Hospital, Rigshospitalet, 2100 Copenhagen, Denmark
- Department of Clinical Medicine, University of Copenhagen, 2100 Copenhagen, Denmark
| | - Michael Bachmann Nielsen
- Department of Diagnostic Radiology, Copenhagen University Hospital, Rigshospitalet, 2100 Copenhagen, Denmark
- Department of Clinical Medicine, University of Copenhagen, 2100 Copenhagen, Denmark
| |
Collapse
|
8
|
Bassani S, Santonicco N, Eccher A, Scarpa A, Vianini M, Brunelli M, Bisi N, Nocini R, Sacchetto L, Munari E, Pantanowitz L, Girolami I, Molteni G. Artificial intelligence in head and neck cancer diagnosis. J Pathol Inform 2022; 13:100153. [PMID: 36605112 PMCID: PMC9808017 DOI: 10.1016/j.jpi.2022.100153] [Citation(s) in RCA: 8] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/02/2022] [Revised: 10/17/2022] [Accepted: 11/02/2022] [Indexed: 11/09/2022] Open
Abstract
Introduction Artificial intelligence (AI) is currently being used to augment histopathological diagnostics in pathology. This systematic review aims to evaluate the evolution of these AI-based diagnostic techniques for diagnosing head and neck neoplasms. Materials and methods Articles regarding the use of AI for head and neck pathology published from 1982 until March 2022 were evaluated based on a search strategy determined by a multidisciplinary team of pathologists and otolaryngologists. Data from eligible articles were summarized according to author, year of publication, country, study population, tumor details, study results, and limitations. Results Thirteen articles were included according to inclusion criteria. The selected studies were published between 2012 and March 1, 2022. Most of these studies concern the diagnosis of oral cancer; in particular, 6 are related to the oral cavity, 2 to the larynx, 1 to the salivary glands, and 4 to head and neck squamous cell carcinoma not otherwise specified (NOS). As for the type of diagnostics considered, 12 concerned histopathology and 1 cytology. Discussion Starting from the pathological examination, artificial intelligence tools are an excellent solution for implementing diagnosis capability. Nevertheless, today the unavailability of large training datasets is a main issue that needs to be overcome to realize the true potential.
Collapse
Affiliation(s)
- Sara Bassani
- Otolaryngology-Head and Neck Surgery Department, University of Verona, Verona, Italy
| | - Nicola Santonicco
- Otolaryngology-Head and Neck Surgery Department, University of Verona, Verona, Italy,Department of Diagnostics and Public Health, Section of Pathology, University and Hospital Trust of Verona, Verona, Italy
| | - Albino Eccher
- Department of Pathology and Diagnostics, University and Hospital Trust of Verona, Verona, Italy,Corresponding author at: Department of Pathology and Diagnostics, University and Hospital Trust of Verona, Verona, Italy.
| | - Aldo Scarpa
- Department of Diagnostics and Public Health, Section of Pathology, University and Hospital Trust of Verona, Verona, Italy
| | - Matteo Vianini
- Department of Otolaryngology, Villafranca Hospital, Verona, Italy
| | - Matteo Brunelli
- Department of Diagnostics and Public Health, Section of Pathology, University and Hospital Trust of Verona, Verona, Italy
| | - Nicola Bisi
- Otolaryngology-Head and Neck Surgery Department, University of Verona, Verona, Italy
| | - Riccardo Nocini
- Otolaryngology-Head and Neck Surgery Department, University of Verona, Verona, Italy
| | - Luca Sacchetto
- Otolaryngology-Head and Neck Surgery Department, University of Verona, Verona, Italy
| | - Enrico Munari
- Department of Molecular and Translational Medicine, University of Brescia, 25121 Brescia, Italy
| | | | - Ilaria Girolami
- Department of Pathology, Provincial Hospital of Bolzano (SABES-ASDAA), Bolzano-Bozen, Italy; Lehrkrankenhaus der Paracelsus Medizinischen Privatuniversität
| | - Gabriele Molteni
- Otolaryngology-Head and Neck Surgery Department, University of Verona, Verona, Italy
| |
Collapse
|
9
|
Viswanathan VS, Toro P, Corredor G, Mukhopadhyay S, Madabhushi A. The state of the art for artificial intelligence in lung digital pathology. J Pathol 2022; 257:413-429. [PMID: 35579955 PMCID: PMC9254900 DOI: 10.1002/path.5966] [Citation(s) in RCA: 27] [Impact Index Per Article: 13.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/22/2022] [Revised: 04/26/2022] [Accepted: 05/15/2022] [Indexed: 12/03/2022]
Abstract
Lung diseases carry a significant burden of morbidity and mortality worldwide. The advent of digital pathology (DP) and an increase in computational power have led to the development of artificial intelligence (AI)-based tools that can assist pathologists and pulmonologists in improving clinical workflow and patient management. While previous works have explored the advances in computational approaches for breast, prostate, and head and neck cancers, there has been a growing interest in applying these technologies to lung diseases as well. The application of AI tools on radiology images for better characterization of indeterminate lung nodules, fibrotic lung disease, and lung cancer risk stratification has been well documented. In this article, we discuss methodologies used to build AI tools in lung DP, describing the various hand-crafted and deep learning-based unsupervised feature approaches. Next, we review AI tools across a wide spectrum of lung diseases including cancer, tuberculosis, idiopathic pulmonary fibrosis, and COVID-19. We discuss the utility of novel imaging biomarkers for different types of clinical problems including quantification of biomarkers like PD-L1, lung disease diagnosis, risk stratification, and prediction of response to treatments such as immune checkpoint inhibitors. We also look briefly at some emerging applications of AI tools in lung DP such as multimodal data analysis, 3D pathology, and transplant rejection. Lastly, we discuss the future of DP-based AI tools, describing the challenges with regulatory approval, developing reimbursement models, planning clinical deployment, and addressing AI biases. © 2022 The Authors. The Journal of Pathology published by John Wiley & Sons Ltd on behalf of The Pathological Society of Great Britain and Ireland.
Collapse
Affiliation(s)
| | - Paula Toro
- Department of PathologyCleveland ClinicClevelandOHUSA
| | - Germán Corredor
- Department of Biomedical EngineeringCase Western Reserve UniversityClevelandOHUSA
- Louis Stokes Cleveland VA Medical CenterClevelandOHUSA
| | | | - Anant Madabhushi
- Department of Biomedical EngineeringCase Western Reserve UniversityClevelandOHUSA
- Louis Stokes Cleveland VA Medical CenterClevelandOHUSA
| |
Collapse
|
10
|
Mariam K, Afzal OM, Hussain W, Javed MU, Kiyani A, Rajpoot N, Khurram SA, Khan HA. On Smart Gaze based Annotation of Histopathology Images for Training of Deep Convolutional Neural Networks. IEEE J Biomed Health Inform 2022; 26:3025-3036. [PMID: 35130177 DOI: 10.1109/jbhi.2022.3148944] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Abstract
Unavailability of large training datasets is a bottleneck that needs to be overcome to realize the true potential of deep learning in histopathology applications. Although slide digitization via whole slide imaging scanners has increased the speed of data acquisition, labeling of virtual slides requires a substantial time investment from pathologists. Eye gaze annotations have the potential to speed up the slide labeling process. This work explores the viability and timing comparisons of eye gaze labeling compared to conventional manual labeling for training object detectors. Challenges associated with gaze based labeling and methods to refine the coarse data annotations for subsequent object detection are also discussed. Results demonstrate that gaze tracking based labeling can save valuable pathologist time and delivers good performance when employed for training a deep object detector. Using the task of localization of Keratin Pearls in cases of oral squamous cell carcinoma as a test case, we compare the performance gap between deep object detectors trained using hand-labelled and gaze-labelled data. On average, compared to 'Bounding-box' based hand-labeling, gaze-labeling required 57.6% less time per label and compared to 'Freehand' labeling, gaze-labeling required on average 85% less time per label.
Collapse
|
11
|
Stadler CB, Lindvall M, Lundström C, Bodén A, Lindman K, Rose J, Treanor D, Blomma J, Stacke K, Pinchaud N, Hedlund M, Landgren F, Woisetschläger M, Forsberg D. Proactive Construction of an Annotated Imaging Database for Artificial Intelligence Training. J Digit Imaging 2021; 34:105-115. [PMID: 33169211 PMCID: PMC7887127 DOI: 10.1007/s10278-020-00384-4] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/10/2023] Open
Abstract
Artificial intelligence (AI) holds much promise for enabling highly desired imaging diagnostics improvements. One of the most limiting bottlenecks for the development of useful clinical-grade AI models is the lack of training data. One aspect is the large amount of cases needed and another is the necessity of high-quality ground truth annotation. The aim of the project was to establish and describe the construction of a database with substantial amounts of detail-annotated oncology imaging data from pathology and radiology. A specific objective was to be proactive, that is, to support undefined subsequent AI training across a wide range of tasks, such as detection, quantification, segmentation, and classification, which puts particular focus on the quality and generality of the annotations. The main outcome of this project was the database as such, with a collection of labeled image data from breast, ovary, skin, colon, skeleton, and liver. In addition, this effort also served as an exploration of best practices for further scalability of high-quality image collections, and a main contribution of the study was generic lessons learned regarding how to successfully organize efforts to construct medical imaging databases for AI training, summarized as eight guiding principles covering team, process, and execution aspects.
Collapse
Affiliation(s)
- Caroline Bivik Stadler
- Center for Medical Image Science and Visualization (CMIV), Linköping University Hospital, Linköping University, SE-581 85, Linköping, Sweden. .,Department of Health, Medicine and Caring Sciences (HMV), Linköping University, SE-581 85, Linköping, Sweden.
| | - Martin Lindvall
- Sectra AB, Teknikringen 20, SE-583 30, Linköping, Sweden.,Department of Science and Technology (ITN), Linköping University, Campus Norrköping, SE-601 74, Norrköping, Sweden
| | - Claes Lundström
- Center for Medical Image Science and Visualization (CMIV), Linköping University Hospital, Linköping University, SE-581 85, Linköping, Sweden.,Sectra AB, Teknikringen 20, SE-583 30, Linköping, Sweden.,Department of Science and Technology (ITN), Linköping University, Campus Norrköping, SE-601 74, Norrköping, Sweden
| | - Anna Bodén
- Center for Medical Image Science and Visualization (CMIV), Linköping University Hospital, Linköping University, SE-581 85, Linköping, Sweden.,Department of Clinical Pathology, Region Östergötland, Linköping University Hospital, SE-581 85, Linköping, Sweden.,Department of Biomedical and Clinical Sciences (BKV), Linköping University, SE-581 85, Linköping, Sweden
| | - Karin Lindman
- Center for Medical Image Science and Visualization (CMIV), Linköping University Hospital, Linköping University, SE-581 85, Linköping, Sweden.,Department of Clinical Pathology, Region Östergötland, Linköping University Hospital, SE-581 85, Linköping, Sweden.,Department of Biomedical and Clinical Sciences (BKV), Linköping University, SE-581 85, Linköping, Sweden
| | - Jeronimo Rose
- Center for Medical Image Science and Visualization (CMIV), Linköping University Hospital, Linköping University, SE-581 85, Linköping, Sweden
| | - Darren Treanor
- Center for Medical Image Science and Visualization (CMIV), Linköping University Hospital, Linköping University, SE-581 85, Linköping, Sweden.,Department of Clinical Pathology, Region Östergötland, Linköping University Hospital, SE-581 85, Linköping, Sweden.,Department of Biomedical and Clinical Sciences (BKV), Linköping University, SE-581 85, Linköping, Sweden.,Department of Cellular Pathology, Leeds Teaching Hospital NHS Trust, Beckett St, Leeds, LS9 7TF, UK.,University of Leeds, Leeds, LS2 9JT, UK
| | - Johan Blomma
- Department of Radiology, Region Östergötland, Linköping University Hospital, SE-581 85, Linköping, Sweden
| | - Karin Stacke
- Sectra AB, Teknikringen 20, SE-583 30, Linköping, Sweden.,Department of Science and Technology (ITN), Linköping University, Campus Norrköping, SE-601 74, Norrköping, Sweden
| | - Nicolas Pinchaud
- ContextVision AB, Klara Norra Kyrkogata 31, SE-111 22, Stockholm, Sweden
| | - Martin Hedlund
- ContextVision AB, Klara Norra Kyrkogata 31, SE-111 22, Stockholm, Sweden
| | - Filip Landgren
- Department of Radiology, Region Östergötland, Linköping University Hospital, SE-581 85, Linköping, Sweden
| | - Mischa Woisetschläger
- Center for Medical Image Science and Visualization (CMIV), Linköping University Hospital, Linköping University, SE-581 85, Linköping, Sweden.,Department of Radiology, Region Östergötland, Linköping University Hospital, SE-581 85, Linköping, Sweden
| | | |
Collapse
|
12
|
Abstract
PURPOSE OF REVIEW Successful integration of artificial intelligence into extant clinical workflows is contingent upon a number of factors including clinician comprehension and interpretation of computer vision. This article discusses how image analysis and machine learning have enabled comprehensive characterization of kidney morphology for development of automated diagnostic and prognostic renal pathology applications. RECENT FINDINGS The primordial digital pathology informatics work employed classical image analysis and machine learning to prognosticate renal disease. Although this classical approach demonstrated tremendous potential, subsequent advancements in hardware technology rendered artificial neural networks '(ANNs) the method of choice for machine vision in computational pathology'. Offering rapid and reproducible detection, characterization and classification of kidney morphology, ANNs have facilitated the development of diagnostic and prognostic applications. In addition, modern machine learning with ANNs has revealed novel biomarkers in kidney disease, demonstrating the potential for machine vision to elucidate novel pathologic mechanisms beyond extant clinical knowledge. SUMMARY Despite the revolutionary developments potentiated by modern machine learning, several challenges remain, including data quality control and curation, image annotation and ontology, integration of multimodal data and interpretation of machine vision or 'opening the black box'. Resolution of these challenges will not only revolutionize diagnostic pathology but also pave the way for precision medicine and integration of artificial intelligence in the process of care.
Collapse
|
13
|
Harrison JH, Gilbertson JR, Hanna MG, Olson NH, Seheult JN, Sorace JM, Stram MN. Introduction to Artificial Intelligence and Machine Learning for Pathology. Arch Pathol Lab Med 2021; 145:1228-1254. [PMID: 33493264 DOI: 10.5858/arpa.2020-0541-cp] [Citation(s) in RCA: 32] [Impact Index Per Article: 10.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 11/10/2020] [Indexed: 11/06/2022]
Abstract
CONTEXT.— Recent developments in machine learning have stimulated intense interest in software that may augment or replace human experts. Machine learning may impact pathology practice by offering new capabilities in analysis, interpretation, and outcomes prediction using images and other data. The principles of operation and management of machine learning systems are unfamiliar to pathologists, who anticipate a need for additional education to be effective as expert users and managers of the new tools. OBJECTIVE.— To provide a background on machine learning for practicing pathologists, including an overview of algorithms, model development, and performance evaluation; to examine the current status of machine learning in pathology and consider possible roles and requirements for pathologists in local deployment and management of machine learning systems; and to highlight existing challenges and gaps in deployment methodology and regulation. DATA SOURCES.— Sources include the biomedical and engineering literature, white papers from professional organizations, government reports, electronic resources, and authors' experience in machine learning. References were chosen when possible for accessibility to practicing pathologists without specialized training in mathematics, statistics, or software development. CONCLUSIONS.— Machine learning offers an array of techniques that in recent published results show substantial promise. Data suggest that human experts working with machine learning tools outperform humans or machines separately, but the optimal form for this combination in pathology has not been established. Significant questions related to the generalizability of machine learning systems, local site verification, and performance monitoring remain to be resolved before a consensus on best practices and a regulatory environment can be established.
Collapse
Affiliation(s)
- James H Harrison
- From the Department of Pathology, University of Virginia School of Medicine, Charlottesville (Harrison)
| | - John R Gilbertson
- the Departments of Biomedical Informatics and Pathology, University of Pittsburgh, Pittsburgh, Pennsylvania (Gilbertson)
| | - Matthew G Hanna
- the Department of Pathology, Memorial Sloan Kettering Cancer Center, New York, New York (Hanna)
| | - Niels H Olson
- the Defense Innovation Unit, Mountain View, California (Olson).,the Department of Pathology, Uniformed Services University, Bethesda, Maryland (Olson)
| | - Jansen N Seheult
- the Department of Pathology, University of Pittsburgh, and Vitalant Specialty Labs, Pittsburgh, Pennsylvania (Seheult)
| | - James M Sorace
- the US Department of Health and Human Services, retired, Lutherville, Maryland (Sorace)
| | - Michelle N Stram
- the Department of Forensic Medicine, New York University, and Office of Chief Medical Examiner, New York, New York (Stram)
| |
Collapse
|
14
|
Lindvall M, Sanner A, Petré F, Lindman K, Treanor D, Lundström C, Löwgren J. TissueWand, a Rapid Histopathology Annotation Tool. J Pathol Inform 2020; 11:27. [PMID: 33042606 PMCID: PMC7518350 DOI: 10.4103/jpi.jpi_5_20] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/10/2020] [Revised: 03/23/2020] [Accepted: 05/19/2020] [Indexed: 12/28/2022] Open
Abstract
Background: Recent advancements in machine learning (ML) bring great possibilities for the development of tools to assist with diagnostic tasks within histopathology. However, these approaches typically require a large amount of ground truth training data in the form of image annotations made by human experts. As such annotation work is a very time-consuming task, there is a great need for tools that can assist in this process, saving time while not sacrificing annotation quality. Methods: In an iterative design process, we developed TissueWand – an interactive tool designed for efficient annotation of gigapixel-sized histopathological images, not being constrained to a predefined annotation task. Results: Several findings regarding appropriate interaction concepts were made, where a key design component was semi-automation based on rapid interaction feedback in a local region. In a user study, the resulting tool was shown to cause substantial speed-up compared to manual work while maintaining quality. Conclusions: The TissueWand tool shows promise to replace manual methods for early stages of dataset curation where no task-specific ML model yet exists to aid the effort.
Collapse
Affiliation(s)
- Martin Lindvall
- Sectra AB, Research Department, Linköping, Sweden.,Center for Medical Image Science and Visualization, Linköping University, Linköping, Sweden.,Department of Science and Technology (ITN), Linköping University, Norrköping, Sweden
| | | | | | - Karin Lindman
- Department of Clinical Pathology, Region Östergötland, Linköping, Sweden.,Department of Biomedical and Clinical Sciences, Linköping University, Linköping, Sweden
| | - Darren Treanor
- Center for Medical Image Science and Visualization, Linköping University, Linköping, Sweden.,Department of Cellular Pathology, Leeds Teaching Hospitals NHS Trust, Leeds, UK.,University of Leeds, Leeds, UK
| | - Claes Lundström
- Sectra AB, Research Department, Linköping, Sweden.,Center for Medical Image Science and Visualization, Linköping University, Linköping, Sweden.,Department of Science and Technology (ITN), Linköping University, Norrköping, Sweden
| | - Jonas Löwgren
- Department of Science and Technology (ITN), Linköping University, Norrköping, Sweden
| |
Collapse
|