301
|
Howard FM, Dolezal J, Kochanny S, Schulte J, Chen H, Heij L, Huo D, Nanda R, Olopade OI, Kather JN, Cipriani N, Grossman RL, Pearson AT. The impact of site-specific digital histology signatures on deep learning model accuracy and bias. Nat Commun 2021; 12:4423. [PMID: 34285218 PMCID: PMC8292530 DOI: 10.1038/s41467-021-24698-1] [Citation(s) in RCA: 99] [Impact Index Per Article: 33.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/09/2020] [Accepted: 07/01/2021] [Indexed: 12/20/2022] Open
Abstract
The Cancer Genome Atlas (TCGA) is one of the largest biorepositories of digital histology. Deep learning (DL) models have been trained on TCGA to predict numerous features directly from histology, including survival, gene expression patterns, and driver mutations. However, we demonstrate that these features vary substantially across tissue submitting sites in TCGA for over 3,000 patients with six cancer subtypes. Additionally, we show that histologic image differences between submitting sites can easily be identified with DL. Site detection remains possible despite commonly used color normalization and augmentation methods, and we quantify the image characteristics constituting this site-specific digital histology signature. We demonstrate that these site-specific signatures lead to biased accuracy for prediction of features including survival, genomic mutations, and tumor stage. Furthermore, ethnicity can also be inferred from site-specific signatures, which must be accounted for to ensure equitable application of DL. These site-specific signatures can lead to overoptimistic estimates of model performance, and we propose a quadratic programming method that abrogates this bias by ensuring models are not trained and validated on samples from the same site.
Collapse
Affiliation(s)
- Frederick M Howard
- Section of Hematology/Oncology, Department of Medicine, University of Chicago, Chicago, IL, USA
| | - James Dolezal
- Section of Hematology/Oncology, Department of Medicine, University of Chicago, Chicago, IL, USA
| | - Sara Kochanny
- Section of Hematology/Oncology, Department of Medicine, University of Chicago, Chicago, IL, USA
| | - Jefree Schulte
- Department of Pathology, University of Chicago, Chicago, IL, USA
| | - Heather Chen
- Department of Pathology, University of Chicago, Chicago, IL, USA
| | - Lara Heij
- Department of Surgery and Transplantation, University Hospital RWTH Aachen, Aachen, Germany
- Institute of Pathology, University Hospital RWTH Aachen, Aachen, Germany
| | - Dezheng Huo
- Department of Public Health Sciences, University of Chicago, Chicago, IL, USA
- University of Chicago Comprehensive Cancer Center, Chicago, IL, USA
| | - Rita Nanda
- Section of Hematology/Oncology, Department of Medicine, University of Chicago, Chicago, IL, USA
- University of Chicago Comprehensive Cancer Center, Chicago, IL, USA
| | - Olufunmilayo I Olopade
- Section of Hematology/Oncology, Department of Medicine, University of Chicago, Chicago, IL, USA
- University of Chicago Comprehensive Cancer Center, Chicago, IL, USA
| | - Jakob N Kather
- Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
- Pathology and Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, UK
- Medical Oncology, National Center for Tumor Diseases, University Hospital Heidelberg, Heidelberg, Germany
| | - Nicole Cipriani
- Department of Pathology, University of Chicago, Chicago, IL, USA
- University of Chicago Comprehensive Cancer Center, Chicago, IL, USA
| | - Robert L Grossman
- Section of Hematology/Oncology, Department of Medicine, University of Chicago, Chicago, IL, USA.
- University of Chicago Comprehensive Cancer Center, Chicago, IL, USA.
| | - Alexander T Pearson
- Section of Hematology/Oncology, Department of Medicine, University of Chicago, Chicago, IL, USA.
- University of Chicago Comprehensive Cancer Center, Chicago, IL, USA.
| |
Collapse
|
302
|
Brinker TJ, Kiehl L, Schmitt M, Jutzi TB, Krieghoff-Henning EI, Krahl D, Kutzner H, Gholam P, Haferkamp S, Klode J, Schadendorf D, Hekler A, Fröhling S, Kather JN, Haggenmüller S, von Kalle C, Heppt M, Hilke F, Ghoreschi K, Tiemann M, Wehkamp U, Hauschild A, Weichenthal M, Utikal JS. Deep learning approach to predict sentinel lymph node status directly from routine histology of primary melanoma tumours. Eur J Cancer 2021; 154:227-234. [PMID: 34298373 DOI: 10.1016/j.ejca.2021.05.026] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/26/2021] [Revised: 05/18/2021] [Accepted: 05/20/2021] [Indexed: 12/28/2022]
Abstract
AIM Sentinel lymph node status is a central prognostic factor for melanomas. However, the surgical excision involves some risks for affected patients. In this study, we therefore aimed to develop a digital biomarker that can predict lymph node metastasis non-invasively from digitised H&E slides of primary melanoma tumours. METHODS A total of 415 H&E slides from primary melanoma tumours with known sentinel node (SN) status from three German university hospitals and one private pathological practice were digitised (150 SN positive/265 SN negative). Two hundred ninety-one slides were used to train artificial neural networks (ANNs). The remaining 124 slides were used to test the ability of the ANNs to predict sentinel status. ANNs were trained and/or tested on data sets that were matched or not matched between SN-positive and SN-negative cases for patient age, ulceration, and tumour thickness, factors that are known to correlate with lymph node status. RESULTS The best accuracy was achieved by an ANN that was trained and tested on unmatched cases (61.8% ± 0.2%) area under the receiver operating characteristic (AUROC). In contrast, ANNs that were trained and/or tested on matched cases achieved (55.0% ± 3.5%) AUROC or less. CONCLUSION Our results indicate that the image classifier can predict lymph node status to some, albeit so far not clinically relevant, extent. It may do so by mostly detecting equivalents of factors on histological slides that are already known to correlate with lymph node status. Our results provide a basis for future research with larger data cohorts.
Collapse
Affiliation(s)
- Titus J Brinker
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany.
| | - Lennard Kiehl
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany
| | - Max Schmitt
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany
| | - Tanja B Jutzi
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany
| | - Eva I Krieghoff-Henning
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany
| | - Dieter Krahl
- Private Laboratory of Dermatohistopathology, Mönchhofstraße 52, 69120, Heidelberg, Germany
| | - Heinz Kutzner
- Dermatopathology Laboratory, Friedrichshafen, Germany
| | - Patrick Gholam
- Department of Dermatology, University Hospital Heidelberg, Heidelberg. Germany
| | - Sebastian Haferkamp
- Department of Dermatology, University Hospital Regensburg, Regensburg, Germany
| | - Joachim Klode
- Department of Dermatology, University Hospital Essen, Essen, Germany
| | - Dirk Schadendorf
- Department of Dermatology, University Hospital Essen, Essen, Germany
| | - Achim Hekler
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany
| | - Stefan Fröhling
- Translational Medical Oncology, German Cancer Research Center (DKFZ), National Center for Tumor Diseases (NCT), 69120, Heidelberg, Germany
| | - Jakob N Kather
- Translational Medical Oncology, German Cancer Research Center (DKFZ), National Center for Tumor Diseases (NCT), 69120, Heidelberg, Germany; Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
| | - Sarah Haggenmüller
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany
| | - Christof von Kalle
- Department of Clinical-Translational Sciences, Charité University Medicine and Berlin Institute of Health (BIH), Berlin, Germany
| | - Markus Heppt
- Department of Dermatology, University Hospital Erlangen, Erlangen, Germany
| | - Franz Hilke
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin, Berlin, Germany
| | - Kamran Ghoreschi
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin, Berlin, Germany
| | | | - Ulrike Wehkamp
- Skin Cancer Unit, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Axel Hauschild
- Skin Cancer Unit, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | | | - Jochen S Utikal
- Department of Dermatology, Venereology and Allergology, University Medical Center Mannheim, Ruprecht-Karl University of Heidelberg, Mannheim, Germany; Department of Dermatology, University Hospital (UKSH), Kiel, Germany
| |
Collapse
|
303
|
Abstract
It is very important to make an objective evaluation of colorectal cancer histological images. Current approaches are generally based on the use of different combinations of textual features and classifiers to assess the classification performance, or transfer learning to classify different organizational types. However, since histological images contain multiple tissue types and characteristics, classification is still challenging. In this study, we proposed the best classification methodology based on the selected optimizer and modified the parameters of CNN methods. Then, we used deep learning technology to distinguish between healthy and diseased large intestine tissues. Firstly, we trained a neural network and compared the network architecture optimizers. Secondly, we modified the parameters of the network layer to optimize the superior architecture. Finally, we compared our well-trained deep learning methods on two different histological image open datasets, which comprised 5000 H&E images of colorectal cancer. The other dataset was composed of nine organizational categories of 100,000 images with an external validation of 7180 images. The results showed that the accuracy of the recognition of histopathological images was significantly better than that of existing methods. Therefore, this method is expected to have great potential to assist physicians to make clinical diagnoses and reduce the number of disparate assessments based on the use of artificial intelligence to classify colorectal cancer tissue.
Collapse
|
304
|
Bae H, Paludan M, Knoblauch J, Jensen KH. Neural networks and robotic microneedles enable autonomous extraction of plant metabolites. PLANT PHYSIOLOGY 2021; 186:1435-1441. [PMID: 34014283 PMCID: PMC8260139 DOI: 10.1093/plphys/kiab178] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/08/2020] [Accepted: 03/28/2021] [Indexed: 06/12/2023]
Abstract
Plant metabolites comprise a wide range of extremely important chemicals. In many cases, like savory spices, they combine distinctive functional properties-deterrence against herbivory-with an unmistakable flavor. Others have remarkable therapeutic qualities, for instance, the malaria drug artemisinin, or mechanical properties, for example natural rubber. We present a breakthrough in plant metabolite extraction technology. Using a neural network, we teach a computer how to recognize metabolite-rich cells of the herbal plant rosemary (Rosmarinus officinalis) and automatically extract the chemicals using a microrobot while leaving the rest of the plant undisturbed. Our approach obviates the need for chemical and mechanical separation and enables the extraction of plant metabolites that currently lack proper methods for efficient biomass use. Computer code required to train the neural network, identify regions of interest, and control the micromanipulator is available as part of the Supplementary Material.
Collapse
Affiliation(s)
- Hansol Bae
- Department of Physics, Technical University of Denmark, DK-2800 Kongens Lyngby, Denmark
| | - Magnus Paludan
- Department of Physics, Technical University of Denmark, DK-2800 Kongens Lyngby, Denmark
| | - Jan Knoblauch
- Department of Physics, Technical University of Denmark, DK-2800 Kongens Lyngby, Denmark
| | - Kaare H. Jensen
- Department of Physics, Technical University of Denmark, DK-2800 Kongens Lyngby, Denmark
| |
Collapse
|
305
|
Liu X, Zhang D, Liu Z, Li Z, Xie P, Sun K, Wei W, Dai W, Tang Z, Ding Y, Cai G, Tong T, Meng X, Tian J. Deep learning radiomics-based prediction of distant metastasis in patients with locally advanced rectal cancer after neoadjuvant chemoradiotherapy: A multicentre study. EBioMedicine 2021; 69:103442. [PMID: 34157487 PMCID: PMC8237293 DOI: 10.1016/j.ebiom.2021.103442] [Citation(s) in RCA: 55] [Impact Index Per Article: 18.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/03/2021] [Revised: 04/17/2021] [Accepted: 06/01/2021] [Indexed: 12/22/2022] Open
Abstract
BACKGROUND Accurate predictions of distant metastasis (DM) in locally advanced rectal cancer (LARC) patients receiving neoadjuvant chemoradiotherapy (nCRT) are helpful in developing appropriate treatment plans. This study aimed to perform DM prediction through deep learning radiomics. METHODS We retrospectively sampled 235 patients receiving nCRT with the minimum 36 months' postoperative follow-up from three hospitals. Through transfer learning, a deep learning radiomic signature (DLRS) based on multiparametric magnetic resonance imaging (MRI) was constructed. A nomogram was established integrating deep MRI information and clinicopathologic factors for better prediction. Harrell's concordance index (C-index) and time-dependent receiver operating characteristic (ROC) were used as performance metrics. Furthermore, the risk of DM in patients with different response to nCRT was evaluated with the nomogram. FINDINGS DLRS performed well in DM prediction, with a C-index of 0·747 and an area under curve (AUC) at three years of 0·894 in the validation cohort. The performance of nomogram was better, with a C-index of 0·775. In addition, the nomogram could stratify patients with different responses to nCRT into high- and low-risk groups of DM (P < 0·05). INTERPRETATION MRI-based deep learning radiomics had potential in predicting the DM of LARC patients receiving nCRT and could help evaluate the risk of DM in patients who have different responses to nCRT. FUNDING The funding bodies that contributed to this study are listed in the Acknowledgements section.
Collapse
Affiliation(s)
- Xiangyu Liu
- Engineering Research Center of Molecular and Neuro Imaging of Ministry of Education, School of Life Science and Technology, Xidian University, Xi'an, Shaanxi 710126, China; CAS Key Laboratory of Molecular Imaging, Beijing Key Laboratory of Molecular Imaging, the State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
| | - Dafu Zhang
- Department of Radiology, the Third Affiliated Hospital of Kunming Medical University, Yunnan Cancer Hospital, Yunnan Cancer Center, Kunming 650118, China
| | - Zhenyu Liu
- CAS Key Laboratory of Molecular Imaging, Beijing Key Laboratory of Molecular Imaging, the State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China; CAS Center for Excellence in Brain Science and Intelligence Technology, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China; School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100080, China
| | - Zhenhui Li
- Department of Radiology, the Third Affiliated Hospital of Kunming Medical University, Yunnan Cancer Hospital, Yunnan Cancer Center, Kunming 650118, China
| | - Peiyi Xie
- Department of Radiology, The Sixth Affiliated Hospital of Sun Yat-sen University, Guangzhou 510655, China
| | - Kai Sun
- Engineering Research Center of Molecular and Neuro Imaging of Ministry of Education, School of Life Science and Technology, Xidian University, Xi'an, Shaanxi 710126, China; CAS Key Laboratory of Molecular Imaging, Beijing Key Laboratory of Molecular Imaging, the State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
| | - Wei Wei
- CAS Key Laboratory of Molecular Imaging, Beijing Key Laboratory of Molecular Imaging, the State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
| | - Weixing Dai
- Department of Colorectal Surgery, Fudan University Shanghai Cancer Center, Shanghai 200032, China
| | - Zhenchao Tang
- Advanced Innovation Center for Big Data-Based Precision Medicine, School of Medicine and Engineering, Beihang University, Beijing 100191, China
| | - Yingying Ding
- Department of Radiology, the Third Affiliated Hospital of Kunming Medical University, Yunnan Cancer Hospital, Yunnan Cancer Center, Kunming 650118, China
| | - Guoxiang Cai
- Department of Colorectal Surgery, Fudan University Shanghai Cancer Center, Shanghai 200032, China
| | - Tong Tong
- Department of Radiology, Fudan University Shanghai Cancer Center, Department of Oncology, Shanghai Medical College, Fudan University, Shanghai 200032, China.
| | - Xiaochun Meng
- Department of Radiology, The Sixth Affiliated Hospital of Sun Yat-sen University, Guangzhou 510655, China.
| | - Jie Tian
- Engineering Research Center of Molecular and Neuro Imaging of Ministry of Education, School of Life Science and Technology, Xidian University, Xi'an, Shaanxi 710126, China; CAS Key Laboratory of Molecular Imaging, Beijing Key Laboratory of Molecular Imaging, the State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China; Advanced Innovation Center for Big Data-Based Precision Medicine, School of Medicine and Engineering, Beihang University, Beijing 100191, China; Key Laboratory of Big Data-Based Precision Medicine (Beihang University), Ministry of Industry and Information Technology, Beijing 100191, China.
| |
Collapse
|
306
|
Liang Y, Zhu Y, Lin H, Zhang S, Li S, Huang Y, Liu C, Qu J, Liang C, Zhao K, Li Z, Liu Z. The value of the tumour-stroma ratio for predicting neoadjuvant chemoradiotherapy response in locally advanced rectal cancer: a case control study. BMC Cancer 2021; 21:729. [PMID: 34172021 PMCID: PMC8235870 DOI: 10.1186/s12885-021-08516-x] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/19/2021] [Accepted: 06/16/2021] [Indexed: 01/05/2023] Open
Abstract
BACKGROUND The tumour-stroma ratio (TSR) is recognized as a practical prognostic factor in colorectal cancer. However, TSR assessment generally utilizes surgical specimens. This study aims to investigate whether the TSR evaluated from preoperative biopsy specimens by a semi-automatic quantification method can predict the response after neoadjuvant chemoradiotherapy (nCRT) of patients with locally advanced rectal cancer (LARC). METHODS A total of 248 consecutive patients diagnosed with LARC and treated with nCRT followed by resection were included. Haematoxylin and eosin (HE)-stained sections of biopsy specimens were collected, and the TSR was evaluated by a semi-automatic quantification method and was divided into three categories, using the cut-offs determined in the whole cohort to balance the proportion of patients in each category. The response to nCRT was evaluated on the primary tumour resection specimen by an expert pathologist using the four-tier tumour regression grade (TRG) system. RESULTS The TSR can discriminate patients that are major-responders (TRG 0-1) from patients that are non-responders (TRG 2-3). Patients were divided into stroma-low (33.5%), stroma-intermediate (33.9%), and stroma-high (32.7%) groups using 56.3 and 72.8% as the cutoffs. In the stroma-low group, 58 (69.9%) patients were major-responders, and only 39 (48.1%) patients were considered major-responders in the stroma-high group (P = 0.018). Multivariate analysis showed that the TSR was the only pre-treatment predictor of response to nCRT (adjusted odds ratio 0.40, 95% confidence interval 0.21-0.76, P = 0.002). CONCLUSION An elevated TSR in preoperative biopsy specimens is an independent predictor of nCRT response in LARC. This semi-automatic quantified TSR could be easily translated into routine pathologic assessment due to its reproducibility and reliability.
Collapse
Affiliation(s)
- Yanting Liang
- Guangdong Cardiovascular Institute, Guangdong Provincial People's Hospital, Guangdong Academy of Medical Sciences, Guangzhou, China
| | - Yaxi Zhu
- Department of Pathology, The Sixth Affiliated Hospital of Sun Yat-sen University, Guangzhou, China
| | - Huan Lin
- School of Medicine, South China University of Technology, Guangzhou, China
- Department of Radiology, Guangdong Provincial People's Hospital, Guangdong Academy of Medical Sciences, 106 Zhongshan Er Road, Guangzhou, 510080, China
| | - Shenyan Zhang
- Department of Pathology, The Sixth Affiliated Hospital of Sun Yat-sen University, Guangzhou, China
| | - Suyun Li
- School of Medicine, South China University of Technology, Guangzhou, China
- Department of Radiology, Guangdong Provincial People's Hospital, Guangdong Academy of Medical Sciences, 106 Zhongshan Er Road, Guangzhou, 510080, China
| | - Yanqi Huang
- Department of Radiology, Guangdong Provincial People's Hospital, Guangdong Academy of Medical Sciences, 106 Zhongshan Er Road, Guangzhou, 510080, China
- The Second School of Clinical Medicine, Southern Medical University, Guangzhou, China
| | - Chen Liu
- Department of Radiology, Guangdong Provincial People's Hospital, Guangdong Academy of Medical Sciences, 106 Zhongshan Er Road, Guangzhou, 510080, China
- The Second School of Clinical Medicine, Southern Medical University, Guangzhou, China
| | - Jinrong Qu
- Department of Radiology, The Affiliated Cancer Hospital of Zhengzhou University & Henan Cancer Hospital, Zhengzhou, China
| | - Changhong Liang
- Department of Radiology, Guangdong Provincial People's Hospital, Guangdong Academy of Medical Sciences, 106 Zhongshan Er Road, Guangzhou, 510080, China
| | - Ke Zhao
- School of Medicine, South China University of Technology, Guangzhou, China.
- Department of Radiology, Guangdong Provincial People's Hospital, Guangdong Academy of Medical Sciences, 106 Zhongshan Er Road, Guangzhou, 510080, China.
| | - Zhenhui Li
- Department of Radiology, Yunnan Cancer Center, The Third Affiliated Hospital of Kunming Medical University, Yunnan Cancer Hospital, Kunming, 650118, China.
| | - Zaiyi Liu
- Department of Radiology, Guangdong Provincial People's Hospital, Guangdong Academy of Medical Sciences, 106 Zhongshan Er Road, Guangzhou, 510080, China.
| |
Collapse
|
307
|
Yoshida H, Kiyuna T. Requirements for implementation of artificial intelligence in the practice of gastrointestinal pathology. World J Gastroenterol 2021; 27:2818-2833. [PMID: 34135556 PMCID: PMC8173389 DOI: 10.3748/wjg.v27.i21.2818] [Citation(s) in RCA: 15] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/04/2021] [Revised: 03/16/2021] [Accepted: 04/28/2021] [Indexed: 02/06/2023] Open
Abstract
Tremendous advances in artificial intelligence (AI) in medical image analysis have been achieved in recent years. The integration of AI is expected to cause a revolution in various areas of medicine, including gastrointestinal (GI) pathology. Currently, deep learning algorithms have shown promising benefits in areas of diagnostic histopathology, such as tumor identification, classification, prognosis prediction, and biomarker/genetic alteration prediction. While AI cannot substitute pathologists, carefully constructed AI applications may increase workforce productivity and diagnostic accuracy in pathology practice. Regardless of these promising advances, unlike the areas of radiology or cardiology imaging, no histopathology-based AI application has been approved by a regulatory authority or for public reimbursement. Thus, implying that there are still some obstacles to be overcome before AI applications can be safely and effectively implemented in real-life pathology practice. The challenges have been identified at different stages of the development process, such as needs identification, data curation, model development, validation, regulation, modification of daily workflow, and cost-effectiveness balance. The aim of this review is to present challenges in the process of AI development, validation, and regulation that should be overcome for its implementation in real-life GI pathology practice.
Collapse
Affiliation(s)
- Hiroshi Yoshida
- Department of Diagnostic Pathology, National Cancer Center Hospital, Tokyo 104-0045, Japan
| | - Tomoharu Kiyuna
- Digital Healthcare Business Development Office, NEC Corporation, Tokyo 108-8556, Japan
| |
Collapse
|
308
|
Khened M, Kori A, Rajkumar H, Krishnamurthi G, Srinivasan B. A generalized deep learning framework for whole-slide image segmentation and analysis. Sci Rep 2021; 11:11579. [PMID: 34078928 PMCID: PMC8172839 DOI: 10.1038/s41598-021-90444-8] [Citation(s) in RCA: 61] [Impact Index Per Article: 20.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/04/2021] [Accepted: 05/04/2021] [Indexed: 12/22/2022] Open
Abstract
Histopathology tissue analysis is considered the gold standard in cancer diagnosis and prognosis. Whole-slide imaging (WSI), i.e., the scanning and digitization of entire histology slides, are now being adopted across the world in pathology labs. Trained histopathologists can provide an accurate diagnosis of biopsy specimens based on WSI data. Given the dimensionality of WSIs and the increase in the number of potential cancer cases, analyzing these images is a time-consuming process. Automated segmentation of tumorous tissue helps in elevating the precision, speed, and reproducibility of research. In the recent past, deep learning-based techniques have provided state-of-the-art results in a wide variety of image analysis tasks, including the analysis of digitized slides. However, deep learning-based solutions pose many technical challenges, including the large size of WSI data, heterogeneity in images, and complexity of features. In this study, we propose a generalized deep learning-based framework for histopathology tissue analysis to address these challenges. Our framework is, in essence, a sequence of individual techniques in the preprocessing-training-inference pipeline which, in conjunction, improve the efficiency and the generalizability of the analysis. The combination of techniques we have introduced includes an ensemble segmentation model, division of the WSI into smaller overlapping patches while addressing class imbalances, efficient techniques for inference, and an efficient, patch-based uncertainty estimation framework. Our ensemble consists of DenseNet-121, Inception-ResNet-V2, and DeeplabV3Plus, where all the networks were trained end to end for every task. We demonstrate the efficacy and improved generalizability of our framework by evaluating it on a variety of histopathology tasks including breast cancer metastases (CAMELYON), colon cancer (DigestPath), and liver cancer (PAIP). Our proposed framework has state-of-the-art performance across all these tasks and is ranked within the top 5 currently for the challenges based on these datasets. The entire framework along with the trained models and the related documentation are made freely available at GitHub and PyPi. Our framework is expected to aid histopathologists in accurate and efficient initial diagnosis. Moreover, the estimated uncertainty maps will help clinicians to make informed decisions and further treatment planning or analysis.
Collapse
Affiliation(s)
- Mahendra Khened
- Department of Engineering Design, Indian Institute of Technology Madras, Chennai, 600036, India
| | - Avinash Kori
- Department of Engineering Design, Indian Institute of Technology Madras, Chennai, 600036, India
| | - Haran Rajkumar
- Department of Engineering Design, Indian Institute of Technology Madras, Chennai, 600036, India
| | - Ganapathy Krishnamurthi
- Department of Engineering Design, Indian Institute of Technology Madras, Chennai, 600036, India.
| | - Balaji Srinivasan
- Department of Mechanical Engineering, Indian Institute of Technology Madras, Chennai, 600036, India
| |
Collapse
|
309
|
Jiao Y, Li J, Qian C, Fei S. Deep learning-based tumor microenvironment analysis in colon adenocarcinoma histopathological whole-slide images. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2021; 204:106047. [PMID: 33789213 DOI: 10.1016/j.cmpb.2021.106047] [Citation(s) in RCA: 30] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/30/2020] [Accepted: 03/06/2021] [Indexed: 06/12/2023]
Abstract
BACKGROUND AND OBJECTIVE Colon cancer is a fatal disease, and a comprehensive understanding of the tumor microenvironment (TME) could lead to better risk stratification, prognosis prediction, and therapy management. In this paper, we focused on the automatic evaluation of TME in giga-pixel digital histopathology whole-slide images. METHODS A convolutional neural network is used to recognize nine different content presented in colon cancer whole-slide images. Several implementation details, including the foreground filtering and stain normalization are discussed. Based on the whole-slide segmentation, several TME descriptors are quantified and correlated with the clinical outcome by Kaplan-Meier analysis and Cox regression. Specifically, the stroma, tumor, necrosis, and lymphocyte components are discussed. RESULTS We validated the method on colon adenocarcinoma cases from The Cancer Genome Atlas project. The result shows that the stroma is an independent predictor of progression-free interval (PFI) after corrected by age and pathological stage, with a hazard ratio of 1.665 (95%CI: 1.110~2.495, p = 0.014). High-level necrosis component and lymphocytes component tend to be correlated with poor PFI, with a hazard ratio of 1.552 (95%CI: 0.943~2.554, p = 0.084) and 1.512 (95%CI: 0.979~2.336, p = 0.062), respectively. CONCLUSIONS The result reveals the complex role of the tumor microenvironment in colon adenocarcinoma, and the quantified descriptors are potential predictors of disease progression. The method could be considered for risk stratification and targeted therapy and extend to other types of cancer, leading to a better understanding of the tumor microenvironment.
Collapse
Affiliation(s)
- Yiping Jiao
- Shool of Automation, Southeast University, 2nd Sipailou Road, Nanjing, China.
| | - Junhong Li
- Luoyang Central Hospital affiliated to Zhengzhou University, Luoyang, China
| | - Chenqi Qian
- Jiangsu Chunyu Education Group CO., 88th Zhongshan North Road, Nanjing, China
| | - Shumin Fei
- Shool of Automation, Southeast University, 2nd Sipailou Road, Nanjing, China.
| |
Collapse
|
310
|
Calderaro J, Kather JN. Artificial intelligence-based pathology for gastrointestinal and hepatobiliary cancers. Gut 2021; 70:1183-1193. [PMID: 33214163 DOI: 10.1136/gutjnl-2020-322880] [Citation(s) in RCA: 51] [Impact Index Per Article: 17.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/24/2020] [Revised: 10/03/2020] [Accepted: 10/27/2020] [Indexed: 12/11/2022]
Abstract
Artificial intelligence (AI) can extract complex information from visual data. Histopathology images of gastrointestinal (GI) and liver cancer contain a very high amount of information which human observers can only partially make sense of. Complementing human observers, AI allows an in-depth analysis of digitised histological slides of GI and liver cancer and offers a wide range of clinically relevant applications. First, AI can automatically detect tumour tissue, easing the exponentially increasing workload on pathologists. In addition, and possibly exceeding pathologist's capacities, AI can capture prognostically relevant tissue features and thus predict clinical outcome across GI and liver cancer types. Finally, AI has demonstrated its capacity to infer molecular and genetic alterations of cancer tissues from histological digital slides. These are likely only the first of many AI applications that will have important clinical implications. Thus, pathologists and clinicians alike should be aware of the principles of AI-based pathology and its ability to solve clinically relevant problems, along with its limitations and biases.
Collapse
Affiliation(s)
- Julien Calderaro
- U955, INSERM, Créteil, France .,Pathology, Hopital Henri Mondor, Creteil, Île-de-France, France
| | - Jakob Nikolas Kather
- Applied Tumor Immunity, Deutsches Krebsforschungszentrum, Heidelberg, BW, Germany.,Department of Medicine III, University Hospital RWTH, Aachen, Germany
| |
Collapse
|
311
|
Kobayashi S, Saltz JH, Yang VW. State of machine and deep learning in histopathological applications in digestive diseases. World J Gastroenterol 2021; 27:2545-2575. [PMID: 34092975 PMCID: PMC8160628 DOI: 10.3748/wjg.v27.i20.2545] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/28/2021] [Revised: 03/27/2021] [Accepted: 04/29/2021] [Indexed: 02/06/2023] Open
Abstract
Machine learning (ML)- and deep learning (DL)-based imaging modalities have exhibited the capacity to handle extremely high dimensional data for a number of computer vision tasks. While these approaches have been applied to numerous data types, this capacity can be especially leveraged by application on histopathological images, which capture cellular and structural features with their high-resolution, microscopic perspectives. Already, these methodologies have demonstrated promising performance in a variety of applications like disease classification, cancer grading, structure and cellular localizations, and prognostic predictions. A wide range of pathologies requiring histopathological evaluation exist in gastroenterology and hepatology, indicating these as disciplines highly targetable for integration of these technologies. Gastroenterologists have also already been primed to consider the impact of these algorithms, as development of real-time endoscopic video analysis software has been an active and popular field of research. This heightened clinical awareness will likely be important for future integration of these methods and to drive interdisciplinary collaborations on emerging studies. To provide an overview on the application of these methodologies for gastrointestinal and hepatological histopathological slides, this review will discuss general ML and DL concepts, introduce recent and emerging literature using these methods, and cover challenges moving forward to further advance the field.
Collapse
Affiliation(s)
- Soma Kobayashi
- Department of Biomedical Informatics, Renaissance School of Medicine, Stony Brook University, Stony Brook, NY 11794, United States
| | - Joel H Saltz
- Department of Biomedical Informatics, Renaissance School of Medicine, Stony Brook University, Stony Brook, NY 11794, United States
| | - Vincent W Yang
- Department of Medicine, Renaissance School of Medicine, Stony Brook University, Stony Brook, NY 11794, United States
- Department of Physiology and Biophysics, Renaissance School of Medicine, Stony Brook University, Stony Brook , NY 11794, United States
| |
Collapse
|
312
|
Javed S, Mahmood A, Dias J, Werghi N, Rajpoot N. Spatially Constrained Context-Aware Hierarchical Deep Correlation Filters for Nucleus Detection in Histology Images. Med Image Anal 2021; 72:102104. [PMID: 34242872 DOI: 10.1016/j.media.2021.102104] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/04/2020] [Revised: 05/10/2021] [Accepted: 05/12/2021] [Indexed: 09/30/2022]
Abstract
Nucleus detection in histology images is a fundamental step for cellular-level analysis in computational pathology. In clinical practice, quantitative nuclear morphology can be used for diagnostic decision making, prognostic stratification, and treatment outcome prediction. Nucleus detection is a challenging task because of large variations in the shape of different types of nucleus such as nuclear clutter, heterogeneous chromatin distribution, and irregular and fuzzy boundaries. To address these challenges, we aim to accurately detect nuclei using spatially constrained context-aware correlation filters using hierarchical deep features extracted from multiple layers of a pre-trained network. During training, we extract contextual patches around each nucleus which are used as negative examples while the actual nucleus patch is used as a positive example. In order to spatially constrain the correlation filters, we propose to construct a spatial structural graph across different nucleus components encoding pairwise similarities. The correlation filters are constrained to act as eigenvectors of the Laplacian of the spatial graphs enforcing these to capture the nucleus structure. A novel objective function is proposed by embedding graph-based structural information as well as the contextual information within the discriminative correlation filter framework. The learned filters are constrained to be orthogonal to both the contextual patches and the spatial graph-Laplacian basis to improve the localization and discriminative performance. The proposed objective function trains a hierarchy of correlation filters on different deep feature layers to capture the heterogeneity in nuclear shape and texture. The proposed algorithm is evaluated on three publicly available datasets and compared with 15 current state-of-the-art methods demonstrating competitive performance in terms of accuracy, speed, and generalization.
Collapse
Affiliation(s)
- Sajid Javed
- Khalifa University Center for Autonomous Robotic Systems (KUCARS), Khalifa University, Abu Dhabi, UAE.; Department of Electrical Engineering and Computer Science, Khalifa University, Abu Dhabi, UAE
| | - Arif Mahmood
- Department of Computer Science, Information Technology University, Lahore, Pakistan
| | - Jorge Dias
- Khalifa University Center for Autonomous Robotic Systems (KUCARS), Khalifa University, Abu Dhabi, UAE.; Department of Electrical Engineering and Computer Science, Khalifa University, Abu Dhabi, UAE
| | - Naoufel Werghi
- Khalifa University Center for Autonomous Robotic Systems (KUCARS), Khalifa University, Abu Dhabi, UAE.; Department of Electrical Engineering and Computer Science, Khalifa University, Abu Dhabi, UAE..
| | - Nasir Rajpoot
- Department of Computer Science, University of Warwick, Coventry, CV4 7AL, U.K.; Department of Pathology, University Hospitals Coventry and Warwickshire, Walsgrave, Coventry, CV2 2DX, U.K.; The Alan Turing Institute, London, NW1 2DB, U.K
| |
Collapse
|
313
|
Shi JY, Wang X, Ding GY, Dong Z, Han J, Guan Z, Ma LJ, Zheng Y, Zhang L, Yu GZ, Wang XY, Ding ZB, Ke AW, Yang H, Wang L, Ai L, Cao Y, Zhou J, Fan J, Liu X, Gao Q. Exploring prognostic indicators in the pathological images of hepatocellular carcinoma based on deep learning. Gut 2021; 70:951-961. [PMID: 32998878 DOI: 10.1136/gutjnl-2020-320930] [Citation(s) in RCA: 84] [Impact Index Per Article: 28.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/19/2020] [Revised: 07/02/2020] [Accepted: 07/20/2020] [Indexed: 02/06/2023]
Abstract
OBJECTIVE Tumour pathology contains rich information, including tissue structure and cell morphology, that reflects disease progression and patient survival. However, phenotypic information is subtle and complex, making the discovery of prognostic indicators from pathological images challenging. DESIGN An interpretable, weakly supervised deep learning framework incorporating prior knowledge was proposed to analyse hepatocellular carcinoma (HCC) and explore new prognostic phenotypes on pathological whole-slide images (WSIs) from the Zhongshan cohort of 1125 HCC patients (2451 WSIs) and TCGA cohort of 320 HCC patients (320 WSIs). A 'tumour risk score (TRS)' was established to evaluate patient outcomes, and then risk activation mapping (RAM) was applied to visualise the pathological phenotypes of TRS. The multi-omics data of The Cancer Genome Atlas(TCGA) HCC were used to assess the potential pathogenesis underlying TRS. RESULTS Survival analysis revealed that TRS was an independent prognosticator in both the Zhongshan cohort (p<0.0001) and TCGA cohort (p=0.0003). The predictive ability of TRS was superior to and independent of clinical staging systems, and TRS could evenly stratify patients into up to five groups with significantly different prognoses. Notably, sinusoidal capillarisation, prominent nucleoli and karyotheca, the nucleus/cytoplasm ratio and infiltrating inflammatory cells were identified as the main underlying features of TRS. The multi-omics data of TCGA HCC hint at the relevance of TRS to tumour immune infiltration and genetic alterations such as the FAT3 and RYR2 mutations. CONCLUSION Our deep learning framework is an effective and labour-saving method for decoding pathological images, providing a valuable means for HCC risk stratification and precise patient treatment.
Collapse
Affiliation(s)
- Jie-Yi Shi
- Department of Liver Surgery and Transplantation, Liver Cancer Institute, Zhongshan Hospital, and Key Laboratory of Carcinogenesis and Cancer Invasion (Ministry of Education), Fudan University, Shanghai, P. R. China
| | - Xiaodong Wang
- School of Computer Science and Technology, Xidian University, Xi'an, P. R. China
| | - Guang-Yu Ding
- Department of Liver Surgery and Transplantation, Liver Cancer Institute, Zhongshan Hospital, and Key Laboratory of Carcinogenesis and Cancer Invasion (Ministry of Education), Fudan University, Shanghai, P. R. China
| | - Zhou Dong
- School of Computer Science, Northwestern Polytechnical University, Xi'an, P. R. China
| | - Jing Han
- Department of Pathology, Zhongshan Hospital Fudan University, Shanghai, P. R. China
| | - Zehui Guan
- School of Computer Science, Northwestern Polytechnical University, Xi'an, P. R. China
| | - Li-Jie Ma
- Department of General Surgery, Zhongshan Hospital (South), Public Health Clinical Centre, Fudan University, Shanghai, P. R. China
| | - Yuxuan Zheng
- School of Computer Science and Technology, Xidian University, Xi'an, P. R. China
| | - Lei Zhang
- School of Computer Science and Technology, Xidian University, Xi'an, P. R. China
| | - Guan-Zhen Yu
- Shanghai Key Laboratory of Multidimensional Information Processing, East China Normal University, Shanghai, P. R. China
| | - Xiao-Ying Wang
- Department of Liver Surgery and Transplantation, Liver Cancer Institute, Zhongshan Hospital, and Key Laboratory of Carcinogenesis and Cancer Invasion (Ministry of Education), Fudan University, Shanghai, P. R. China
| | - Zhen-Bin Ding
- Department of Liver Surgery and Transplantation, Liver Cancer Institute, Zhongshan Hospital, and Key Laboratory of Carcinogenesis and Cancer Invasion (Ministry of Education), Fudan University, Shanghai, P. R. China
| | - Ai-Wu Ke
- Department of Liver Surgery and Transplantation, Liver Cancer Institute, Zhongshan Hospital, and Key Laboratory of Carcinogenesis and Cancer Invasion (Ministry of Education), Fudan University, Shanghai, P. R. China
| | - Haoqing Yang
- School of Computer Science and Technology, Xidian University, Xi'an, P. R. China
| | - Liming Wang
- School of Computer Science and Technology, Xidian University, Xi'an, P. R. China
| | - Lirong Ai
- School of Computer Science, Northwestern Polytechnical University, Xi'an, P. R. China
| | - Ya Cao
- Cancer Research Institute, Xiangya School of Medicine, Central South University, Hunan, P. R. China
| | - Jian Zhou
- Department of Liver Surgery and Transplantation, Liver Cancer Institute, Zhongshan Hospital, and Key Laboratory of Carcinogenesis and Cancer Invasion (Ministry of Education), Fudan University, Shanghai, P. R. China
- Institute of Biomedical Sciences, Fudan University, Shanghai, P. R. China
| | - Jia Fan
- Department of Liver Surgery and Transplantation, Liver Cancer Institute, Zhongshan Hospital, and Key Laboratory of Carcinogenesis and Cancer Invasion (Ministry of Education), Fudan University, Shanghai, P. R. China
- Institute of Biomedical Sciences, Fudan University, Shanghai, P. R. China
| | - Xiyang Liu
- School of Computer Science and Technology, Xidian University, Xi'an, P. R. China
| | - Qiang Gao
- Department of Liver Surgery and Transplantation, Liver Cancer Institute, Zhongshan Hospital, and Key Laboratory of Carcinogenesis and Cancer Invasion (Ministry of Education), Fudan University, Shanghai, P. R. China
- Institute of Biomedical Sciences, Fudan University, Shanghai, P. R. China
- State Key Laboratory of Genetic Engineering at Fudan University, Shanghai, P. R. China
| |
Collapse
|
314
|
Gehrung M, Crispin-Ortuzar M, Berman AG, O'Donovan M, Fitzgerald RC, Markowetz F. Triage-driven diagnosis of Barrett's esophagus for early detection of esophageal adenocarcinoma using deep learning. Nat Med 2021; 27:833-841. [PMID: 33859411 DOI: 10.1038/s41591-021-01287-9] [Citation(s) in RCA: 51] [Impact Index Per Article: 17.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/24/2020] [Accepted: 02/17/2021] [Indexed: 12/18/2022]
Abstract
Deep learning methods have been shown to achieve excellent performance on diagnostic tasks, but how to optimally combine them with expert knowledge and existing clinical decision pathways is still an open challenge. This question is particularly important for the early detection of cancer, where high-volume workflows may benefit from (semi-)automated analysis. Here we present a deep learning framework to analyze samples of the Cytosponge-TFF3 test, a minimally invasive alternative to endoscopy, for detecting Barrett's esophagus, which is the main precursor of esophageal adenocarcinoma. We trained and independently validated the framework on data from two clinical trials, analyzing a combined total of 4,662 pathology slides from 2,331 patients. Our approach exploits decision patterns of gastrointestinal pathologists to define eight triage classes of varying priority for manual expert review. By substituting manual review with automated review in low-priority classes, we can reduce pathologist workload by 57% while matching the diagnostic performance of experienced pathologists.
Collapse
Affiliation(s)
- Marcel Gehrung
- Cancer Research UK Cambridge Institute, University of Cambridge, Cambridge, UK
- The Alan Turing Institute, London, UK
| | | | - Adam G Berman
- Cancer Research UK Cambridge Institute, University of Cambridge, Cambridge, UK
| | - Maria O'Donovan
- MRC Cancer Unit, University of Cambridge, Cambridge, UK
- Department of Pathology, Cambridge University Hospitals NHS Trust, Cambridge, UK
| | | | - Florian Markowetz
- Cancer Research UK Cambridge Institute, University of Cambridge, Cambridge, UK.
| |
Collapse
|
315
|
Abstract
PURPOSE OF REVIEW Pathology is the cornerstone of cancer care. Pathomics, which represents the use of artificial intelligence in digital pathology, is an emerging and promising field that will revolutionize medical and surgical pathology in the coming years. This review provides an overview of pathomics, its current and future applications and its most relevant applications in Head and Neck cancer care. RECENT FINDINGS The number of studies investigating the use of artificial intelligence in pathology is rapidly growing, especially as the utilization of deep learning has shown great potential with Whole Slide Images. Even though numerous steps still remain before its clinical use, Pathomics has been used for varied applications comprising of computer-assisted diagnosis, molecular anomalies prediction, tumor microenvironment and biomarker identification as well as prognosis evaluation. The majority of studies were performed on the most frequent cancers, notably breast, prostate, and lung. Interesting results were also found in Head and Neck cancers. SUMMARY Even if its use in Head and Neck cancer care is still low, Pathomics is a powerful tool to improve diagnosis, identify prognostic factors and new biomarkers. Important challenges lie ahead before its use in a clinical practice, notably the lack of information on how AI makes its decisions, the slow deployment of digital pathology, and the need for extensively validated data in order to obtain authorities approval. Regardless, pathomics will most likely improve pathology in general, including Head and Neck cancer care in the coming years.
Collapse
|
316
|
van der Laak J, Litjens G, Ciompi F. Deep learning in histopathology: the path to the clinic. Nat Med 2021; 27:775-784. [PMID: 33990804 DOI: 10.1038/s41591-021-01343-4] [Citation(s) in RCA: 298] [Impact Index Per Article: 99.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/18/2020] [Accepted: 03/31/2021] [Indexed: 02/08/2023]
Abstract
Machine learning techniques have great potential to improve medical diagnostics, offering ways to improve accuracy, reproducibility and speed, and to ease workloads for clinicians. In the field of histopathology, deep learning algorithms have been developed that perform similarly to trained pathologists for tasks such as tumor detection and grading. However, despite these promising results, very few algorithms have reached clinical implementation, challenging the balance between hope and hype for these new techniques. This Review provides an overview of the current state of the field, as well as describing the challenges that still need to be addressed before artificial intelligence in histopathology can achieve clinical value.
Collapse
Affiliation(s)
- Jeroen van der Laak
- Department of Pathology, Radboud University Medical Center, Nijmegen, the Netherlands. .,Center for Medical Image Science and Visualization, Linköping University, Linköping, Sweden.
| | - Geert Litjens
- Department of Pathology, Radboud University Medical Center, Nijmegen, the Netherlands
| | - Francesco Ciompi
- Department of Pathology, Radboud University Medical Center, Nijmegen, the Netherlands
| |
Collapse
|
317
|
Theodosi A, Ouzounis S, Kostopoulos S, Glotsos D, Kalatzis I, Asvestas P, Tzelepi V, Ravazoula P, Cavouras D, Sakellaropoulos G. Employing machine learning and microscopy images of AIB1-stained biopsy material to assess the 5-year survival of patients with colorectal cancer. Microsc Res Tech 2021; 84:2421-2433. [PMID: 33929071 DOI: 10.1002/jemt.23797] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/16/2020] [Revised: 02/04/2021] [Accepted: 04/10/2021] [Indexed: 01/07/2023]
Abstract
Our purpose was to employ microscopy images of amplified in breast cancer 1 (AIB1)-stained biopsy material of patients with colorectal cancer (CRC) to: (a) find statistically significant differences (SSDs) in the texture and color of the epithelial gland tissue, between 5-year survivors and non-survivors after the first diagnosis and (b) employ machine learning (ML) methods for predicting the CRC-patient 5-year survival. We collected biopsy material from 54 patients with diagnosed CRC from the archives of the University Hospital of Patras, Greece. Twenty-six of the patients had survived 5 years after the first diagnosis. We selected regions of interest containing the epithelial gland at different microscope lens magnifications. We computed 69 textural and color features. Furthermore, we identified features with SSDs between the two groups of patients and we designed a supervised ML system for predicting the CRC-patient 5-year survival. Additionally, we employed the VGG16 pretrained convolution neural network to extract deep learning (DL) features, the support vector machines classifier, and the bootstrap cross-validation method for boosting the accuracy of predicting 5-year survival. Fourteen features sustained SSDs between the two groups of patients. The supervised ML system achieved 87% accuracy in predicting 5-year survival. In comparison, the DL system, using images from all magnifications, gave 97% classification accuracy. Glandular texture in 5-year non-survivors appeared to be of lower contrast, coarseness, roughness, local pixel correlation, and lower AIB1 variation, all indicating loss of textural definition. The supervised ML system revealed useful information regarding features that discriminate between 5-year survivors and non-survivors while the DL system displayed superior accuracy by employing DL features.
Collapse
Affiliation(s)
- Angeliki Theodosi
- Department of Medical Physics, School of Health Sciences, Faculty of Medicine, University of Patras, Patras, Greece
| | - Sotiris Ouzounis
- Department of Biomedical Engineering, Medical Image and Signal Processing Laboratory, University of West Attica, Athens, Greece
| | - Spiros Kostopoulos
- Department of Biomedical Engineering, Medical Image and Signal Processing Laboratory, University of West Attica, Athens, Greece
| | - Dimitris Glotsos
- Department of Biomedical Engineering, Medical Image and Signal Processing Laboratory, University of West Attica, Athens, Greece
| | - Ioannis Kalatzis
- Department of Biomedical Engineering, Medical Image and Signal Processing Laboratory, University of West Attica, Athens, Greece
| | - Pantelis Asvestas
- Department of Biomedical Engineering, Medical Image and Signal Processing Laboratory, University of West Attica, Athens, Greece
| | - Vassiliki Tzelepi
- Department of Pathology, University Hospital of Patras, Patras, Greece
| | | | - Dionisis Cavouras
- Department of Biomedical Engineering, Medical Image and Signal Processing Laboratory, University of West Attica, Athens, Greece
| | - George Sakellaropoulos
- Department of Medical Physics, School of Health Sciences, Faculty of Medicine, University of Patras, Patras, Greece
| |
Collapse
|
318
|
Cao JS, Lu ZY, Chen MY, Zhang B, Juengpanich S, Hu JH, Li SJ, Topatana W, Zhou XY, Feng X, Shen JL, Liu Y, Cai XJ. Artificial intelligence in gastroenterology and hepatology: Status and challenges. World J Gastroenterol 2021; 27:1664-1690. [PMID: 33967550 PMCID: PMC8072192 DOI: 10.3748/wjg.v27.i16.1664] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/15/2021] [Revised: 02/11/2021] [Accepted: 03/17/2021] [Indexed: 02/06/2023] Open
Abstract
Originally proposed by John McCarthy in 1955, artificial intelligence (AI) has achieved a breakthrough and revolutionized the processing methods of clinical medicine with the increasing workloads of medical records and digital images. Doctors are paying attention to AI technologies for various diseases in the fields of gastroenterology and hepatology. This review will illustrate AI technology procedures for medical image analysis, including data processing, model establishment, and model validation. Furthermore, we will summarize AI applications in endoscopy, radiology, and pathology, such as detecting and evaluating lesions, facilitating treatment, and predicting treatment response and prognosis with excellent model performance. The current challenges for AI in clinical application include potential inherent bias in retrospective studies that requires larger samples for validation, ethics and legal concerns, and the incomprehensibility of the output results. Therefore, doctors and researchers should cooperate to address the current challenges and carry out further investigations to develop more accurate AI tools for improved clinical applications.
Collapse
Affiliation(s)
- Jia-Sheng Cao
- Department of General Surgery, Sir Run-Run Shaw Hospital, Zhejiang University, Hangzhou 310016, Zhejiang Province, China
| | - Zi-Yi Lu
- Zhejiang University School of Medicine, Zhejiang University, Hangzhou 310058, Zhejiang Province, China
| | - Ming-Yu Chen
- Department of General Surgery, Sir Run-Run Shaw Hospital, Zhejiang University, Hangzhou 310016, Zhejiang Province, China
| | - Bin Zhang
- Department of General Surgery, Sir Run-Run Shaw Hospital, Zhejiang University, Hangzhou 310016, Zhejiang Province, China
| | - Sarun Juengpanich
- Zhejiang University School of Medicine, Zhejiang University, Hangzhou 310058, Zhejiang Province, China
| | - Jia-Hao Hu
- Department of General Surgery, Sir Run-Run Shaw Hospital, Zhejiang University, Hangzhou 310016, Zhejiang Province, China
| | - Shi-Jie Li
- Department of General Surgery, Sir Run-Run Shaw Hospital, Zhejiang University, Hangzhou 310016, Zhejiang Province, China
| | - Win Topatana
- Zhejiang University School of Medicine, Zhejiang University, Hangzhou 310058, Zhejiang Province, China
| | - Xue-Yin Zhou
- School of Medicine, Wenzhou Medical University, Wenzhou 325035, Zhejiang Province, China
| | - Xu Feng
- Department of General Surgery, Sir Run-Run Shaw Hospital, Zhejiang University, Hangzhou 310016, Zhejiang Province, China
| | - Ji-Liang Shen
- Department of General Surgery, Sir Run-Run Shaw Hospital, Zhejiang University, Hangzhou 310016, Zhejiang Province, China
| | - Yu Liu
- College of Life Sciences, Zhejiang University, Hangzhou 310058, Zhejiang Province, China
| | - Xiu-Jun Cai
- Department of General Surgery, Sir Run-Run Shaw Hospital, Zhejiang University, Hangzhou 310016, Zhejiang Province, China
| |
Collapse
|
319
|
Lee SH, Song IH, Jang HJ. Feasibility of deep learning-based fully automated classification of microsatellite instability in tissue slides of colorectal cancer. Int J Cancer 2021; 149:728-740. [PMID: 33851412 DOI: 10.1002/ijc.33599] [Citation(s) in RCA: 33] [Impact Index Per Article: 11.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/29/2020] [Revised: 02/19/2021] [Accepted: 04/07/2021] [Indexed: 12/16/2022]
Abstract
High levels of microsatellite instability (MSI-H) occurs in about 15% of sporadic colorectal cancer (CRC) and is an important predictive marker for response to immune checkpoint inhibitors. To test the feasibility of a deep learning (DL)-based classifier as a screening tool for MSI status, we built a fully automated DL-based MSI classifier using pathology whole-slide images (WSIs) of CRCs. On small image patches of The Cancer Genome Atlas (TCGA) CRC WSI dataset, tissue/non-tissue, normal/tumor and MSS/MSI-H classifiers were applied sequentially for the fully automated prediction of the MSI status. The classifiers were also tested on an independent cohort. Furthermore, to test how the expansion of the training data affects the performance of the DL-based classifier, additional classifier trained on both TCGA and external datasets was tested. The areas under the receiver operating characteristic curves were 0.892 and 0.972 for the TCGA and external datasets, respectively, by a classifier trained on both datasets. The performance of the DL-based classifier was much better than that of previously reported histomorphology-based methods. We speculated that about 40% of CRC slides could be screened for MSI status without molecular testing by the DL-based classifier. These results demonstrated that the DL-based method has potential as a screening tool to discriminate molecular alteration in tissue slides.
Collapse
Affiliation(s)
- Sung Hak Lee
- Department of Hospital Pathology, Seoul St. Mary's Hospital, Seoul, South Korea
| | - In Hye Song
- Department of Hospital Pathology, Seoul St. Mary's Hospital, Seoul, South Korea
| | - Hyun-Jong Jang
- Catholic Big Data Integration Center, Department of Physiology, College of Medicine, The Catholic University of Korea, Seoul, South Korea
| |
Collapse
|
320
|
Pai RK, Hartman D, Schaeffer DF, Rosty C, Shivji S, Kirsch R, Pai RK. Development and initial validation of a deep learning algorithm to quantify histological features in colorectal carcinoma including tumour budding/poorly differentiated clusters. Histopathology 2021; 79:391-405. [PMID: 33590485 DOI: 10.1111/his.14353] [Citation(s) in RCA: 24] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/09/2020] [Revised: 02/03/2021] [Accepted: 02/14/2021] [Indexed: 12/14/2022]
Abstract
AIMS To develop and validate a deep learning algorithm to quantify a broad spectrum of histological features in colorectal carcinoma. METHODS AND RESULTS A deep learning algorithm was trained on haematoxylin and eosin-stained slides from tissue microarrays of colorectal carcinomas (N = 230) to segment colorectal carcinoma digitised images into 13 regions and one object. The segmentation algorithm demonstrated moderate to almost perfect agreement with interpretations by gastrointestinal pathologists, and was applied to an independent test cohort of digitised whole slides of colorectal carcinoma (N = 136). The algorithm correctly classified mucinous and high-grade tumours, and identified significant differences between mismatch repair-proficient and mismatch repair-deficient (MMRD) tumours with regard to mucin, inflammatory stroma, and tumour-infiltrating lymphocytes (TILs). A cutoff of >44.4 TILs per mm2 carcinoma gave a sensitivity of 88% and a specificity of 73% in classifying MMRD carcinomas. Algorithm measures of tumour budding (TB) and poorly differentiated clusters (PDCs) outperformed TB grade derived from routine sign-out, and compared favourably with manual counts of TB/PDCs with regard to lymphatic, venous and perineural invasion. Comparable associations were seen between algorithm measures of TB/PDCs and manual counts of TB/PDCs for lymph node metastasis (all P < 0.001); however, stronger correlations were seen between the proportion of positive lymph nodes and algorithm measures of TB/PDCs. Stronger associations were also seen between distant metastasis and algorithm measures of TB/PDCs (P = 0.004) than between distant metastasis and TB (P = 0.04) and TB/PDC counts (P = 0.06). CONCLUSIONS Our results highlight the potential of deep learning to identify and quantify a broad spectrum of histological features in colorectal carcinoma.
Collapse
Affiliation(s)
- Reetesh K Pai
- Department of Pathology, University of Pittsburgh Medical Center, Pittsburgh, PA, USA
| | - Douglas Hartman
- Department of Pathology, University of Pittsburgh Medical Center, Pittsburgh, PA, USA
| | - David F Schaeffer
- Department of Pathology, University of British Columbia, Vancouver, British Columbia, Canada
| | - Christophe Rosty
- Colorectal Oncogenomics Group, Department of Clinical Pathology, University of Melbourne, Parkville, Victoria, Australia.,Envoi Specialist Pathologists, University of Queensland, Brisbane, Queensland, Australia.,Faculty of Medicine, University of Queensland, Brisbane, Queensland, Australia
| | - Sameer Shivji
- Department of Pathology, Mount Sinai Hospital, Toronto, Ontario, Canada
| | - Richard Kirsch
- Department of Pathology, Mount Sinai Hospital, Toronto, Ontario, Canada
| | - Rish K Pai
- Department of Pathology and Laboratory Medicine, Mayo Clinic Arizona, Scottsdale, AZ, USA
| |
Collapse
|
321
|
Schiele S, Arndt TT, Martin B, Miller S, Bauer S, Banner BM, Brendel EM, Schenkirsch G, Anthuber M, Huss R, Märkl B, Müller G. Deep Learning Prediction of Metastasis in Locally Advanced Colon Cancer Using Binary Histologic Tumor Images. Cancers (Basel) 2021; 13:2074. [PMID: 33922988 PMCID: PMC8123276 DOI: 10.3390/cancers13092074] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/09/2021] [Revised: 04/15/2021] [Accepted: 04/21/2021] [Indexed: 12/12/2022] Open
Abstract
In this study, we developed the Binary ImaGe Colon Metastasis classifier (BIg-CoMet), a semi-guided approach for the stratification of colon cancer patients into two risk groups for the occurrence of distant metastasis, using an InceptionResNetV2-based deep learning model trained on binary images. We enrolled 291 colon cancer patients with pT3 and pT4 adenocarcinomas and converted one cytokeratin-stained representative tumor section per case into a binary image. Image augmentation and dropout layers were incorporated to avoid overfitting. In a validation collective (n = 128), BIg-CoMet was able to discriminate well between patients with and without metastasis (AUC: 0.842, 95% CI: 0.774-0.911). Further, the Kaplan-Meier curves of the metastasis-free survival showed a highly significant worse clinical course for the high-risk group (log-rank test: p < 0.001), and we demonstrated superiority over other established risk factors. A multivariable Cox regression analysis adjusted for confounders supported the use of risk groups as a prognostic factor for the occurrence of metastasis (hazard ratio (HR): 5.4, 95% CI: 2.5-11.7, p < 0.001). BIg-CoMet achieved good performance for both UICC subgroups, especially for UICC III (n = 53), with a positive predictive value of 80%. Our study demonstrates the ability to stratify colon cancer patients via a semi-guided process on images that primarily reflect tumor architecture.
Collapse
Affiliation(s)
- Stefan Schiele
- Institute of Mathematics, Augsburg University, 86159 Augsburg, Germany; (T.T.A.); (G.M.)
| | - Tim Tobias Arndt
- Institute of Mathematics, Augsburg University, 86159 Augsburg, Germany; (T.T.A.); (G.M.)
- General Pathology and Molecular Diagnostics, Medical Faculty, 86156 Augsburg, Germany; (B.M.); (S.M.); (S.B.); (B.M.B.); (E.-M.B.); (R.H.); (B.M.)
| | - Benedikt Martin
- General Pathology and Molecular Diagnostics, Medical Faculty, 86156 Augsburg, Germany; (B.M.); (S.M.); (S.B.); (B.M.B.); (E.-M.B.); (R.H.); (B.M.)
| | - Silvia Miller
- General Pathology and Molecular Diagnostics, Medical Faculty, 86156 Augsburg, Germany; (B.M.); (S.M.); (S.B.); (B.M.B.); (E.-M.B.); (R.H.); (B.M.)
| | - Svenja Bauer
- General Pathology and Molecular Diagnostics, Medical Faculty, 86156 Augsburg, Germany; (B.M.); (S.M.); (S.B.); (B.M.B.); (E.-M.B.); (R.H.); (B.M.)
| | - Bettina Monika Banner
- General Pathology and Molecular Diagnostics, Medical Faculty, 86156 Augsburg, Germany; (B.M.); (S.M.); (S.B.); (B.M.B.); (E.-M.B.); (R.H.); (B.M.)
| | - Eva-Maria Brendel
- General Pathology and Molecular Diagnostics, Medical Faculty, 86156 Augsburg, Germany; (B.M.); (S.M.); (S.B.); (B.M.B.); (E.-M.B.); (R.H.); (B.M.)
| | - Gerhard Schenkirsch
- Tumor Data Management, University Hospital of Augsburg, 86156 Augsburg, Germany;
| | - Matthias Anthuber
- General, Visceral, and Transplantation Surgery, University Hospital of Augsburg, 86156 Augsburg, Germany;
| | - Ralf Huss
- General Pathology and Molecular Diagnostics, Medical Faculty, 86156 Augsburg, Germany; (B.M.); (S.M.); (S.B.); (B.M.B.); (E.-M.B.); (R.H.); (B.M.)
| | - Bruno Märkl
- General Pathology and Molecular Diagnostics, Medical Faculty, 86156 Augsburg, Germany; (B.M.); (S.M.); (S.B.); (B.M.B.); (E.-M.B.); (R.H.); (B.M.)
| | - Gernot Müller
- Institute of Mathematics, Augsburg University, 86159 Augsburg, Germany; (T.T.A.); (G.M.)
| |
Collapse
|
322
|
Interpretable survival prediction for colorectal cancer using deep learning. NPJ Digit Med 2021; 4:71. [PMID: 33875798 PMCID: PMC8055695 DOI: 10.1038/s41746-021-00427-2] [Citation(s) in RCA: 79] [Impact Index Per Article: 26.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/31/2020] [Accepted: 02/26/2021] [Indexed: 02/07/2023] Open
Abstract
Deriving interpretable prognostic features from deep-learning-based prognostic histopathology models remains a challenge. In this study, we developed a deep learning system (DLS) for predicting disease-specific survival for stage II and III colorectal cancer using 3652 cases (27,300 slides). When evaluated on two validation datasets containing 1239 cases (9340 slides) and 738 cases (7140 slides), respectively, the DLS achieved a 5-year disease-specific survival AUC of 0.70 (95% CI: 0.66–0.73) and 0.69 (95% CI: 0.64–0.72), and added significant predictive value to a set of nine clinicopathologic features. To interpret the DLS, we explored the ability of different human-interpretable features to explain the variance in DLS scores. We observed that clinicopathologic features such as T-category, N-category, and grade explained a small fraction of the variance in DLS scores (R2 = 18% in both validation sets). Next, we generated human-interpretable histologic features by clustering embeddings from a deep-learning-based image-similarity model and showed that they explained the majority of the variance (R2 of 73–80%). Furthermore, the clustering-derived feature most strongly associated with high DLS scores was also highly prognostic in isolation. With a distinct visual appearance (poorly differentiated tumor cell clusters adjacent to adipose tissue), this feature was identified by annotators with 87.0–95.5% accuracy. Our approach can be used to explain predictions from a prognostic deep learning model and uncover potentially-novel prognostic features that can be reliably identified by people for future validation studies.
Collapse
|
323
|
Werner J, Kronberg RM, Stachura P, Ostermann PN, Müller L, Schaal H, Bhatia S, Kather JN, Borkhardt A, Pandyra AA, Lang KS, Lang PA. Deep Transfer Learning Approach for Automatic Recognition of Drug Toxicity and Inhibition of SARS-CoV-2. Viruses 2021; 13:v13040610. [PMID: 33918368 PMCID: PMC8066066 DOI: 10.3390/v13040610] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/27/2021] [Revised: 03/25/2021] [Accepted: 03/30/2021] [Indexed: 12/11/2022] Open
Abstract
Severe acute respiratory syndrome coronavirus-2 (SARS-CoV-2) causes COVID-19 and is responsible for the ongoing pandemic. Screening of potential antiviral drugs against SARS-CoV-2 depend on in vitro experiments, which are based on the quantification of the virus titer. Here, we used virus-induced cytopathic effects (CPE) in brightfield microscopy of SARS-CoV-2-infected monolayers to quantify the virus titer. Images were classified using deep transfer learning (DTL) that fine-tune the last layers of a pre-trained Resnet18 (ImageNet). To exclude toxic concentrations of potential drugs, the network was expanded to include a toxic score (TOX) that detected cell death (CPETOXnet). With this analytic tool, the inhibitory effects of chloroquine, hydroxychloroquine, remdesivir, and emetine were validated. Taken together we developed a simple method and provided open access implementation to quantify SARS-CoV-2 titers and drug toxicity in experimental settings, which may be adaptable to assays with other viruses. The quantification of virus titers from brightfield images could accelerate the experimental approach for antiviral testing.
Collapse
Affiliation(s)
- Julia Werner
- Department of Molecular Medicine II, Medical Faculty, Heinrich-Heine-University, 40225 Düsseldorf, Germany; (J.W.); (R.M.K.); (P.S.)
| | - Raphael M. Kronberg
- Department of Molecular Medicine II, Medical Faculty, Heinrich-Heine-University, 40225 Düsseldorf, Germany; (J.W.); (R.M.K.); (P.S.)
- Mathematical Modelling of Biological Systems, Heinrich-Heine-University, 40225 Düsseldorf, Germany
| | - Pawel Stachura
- Department of Molecular Medicine II, Medical Faculty, Heinrich-Heine-University, 40225 Düsseldorf, Germany; (J.W.); (R.M.K.); (P.S.)
| | - Philipp N. Ostermann
- Institute of Virology, Medical Faculty, Heinrich-Heine-University, 40225 Düsseldorf, Germany; (P.N.O.); (L.M.); (H.S.)
| | - Lisa Müller
- Institute of Virology, Medical Faculty, Heinrich-Heine-University, 40225 Düsseldorf, Germany; (P.N.O.); (L.M.); (H.S.)
| | - Heiner Schaal
- Institute of Virology, Medical Faculty, Heinrich-Heine-University, 40225 Düsseldorf, Germany; (P.N.O.); (L.M.); (H.S.)
| | - Sanil Bhatia
- Department of Pediatric Oncology, Hematology and Clinical Immunology, Medical Faculty, Center of Child and Adolescent Health, Heinrich-Heine-University, 40225 Düsseldorf, Germany; (S.B.); (A.B.); (A.A.P.)
| | - Jakob N. Kather
- Department of Medicine III, University Hospital RWTH Aachen, 52074 Aachen, Germany;
| | - Arndt Borkhardt
- Department of Pediatric Oncology, Hematology and Clinical Immunology, Medical Faculty, Center of Child and Adolescent Health, Heinrich-Heine-University, 40225 Düsseldorf, Germany; (S.B.); (A.B.); (A.A.P.)
| | - Aleksandra A. Pandyra
- Department of Pediatric Oncology, Hematology and Clinical Immunology, Medical Faculty, Center of Child and Adolescent Health, Heinrich-Heine-University, 40225 Düsseldorf, Germany; (S.B.); (A.B.); (A.A.P.)
| | - Karl S. Lang
- Institute of Immunology, Medical Faculty, University of Duisburg-Essen, 45147 Essen, Germany;
| | - Philipp A. Lang
- Department of Molecular Medicine II, Medical Faculty, Heinrich-Heine-University, 40225 Düsseldorf, Germany; (J.W.); (R.M.K.); (P.S.)
- Correspondence:
| |
Collapse
|
324
|
Lancellotti C, Cancian P, Savevski V, Kotha SRR, Fraggetta F, Graziano P, Di Tommaso L. Artificial Intelligence & Tissue Biomarkers: Advantages, Risks and Perspectives for Pathology. Cells 2021; 10:787. [PMID: 33918173 PMCID: PMC8066881 DOI: 10.3390/cells10040787] [Citation(s) in RCA: 23] [Impact Index Per Article: 7.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/26/2021] [Revised: 03/30/2021] [Accepted: 03/31/2021] [Indexed: 12/18/2022] Open
Abstract
Tissue Biomarkers are information written in the tissue and used in Pathology to recognize specific subsets of patients with diagnostic, prognostic or predictive purposes, thus representing the key elements of Personalized Medicine. The advent of Artificial Intelligence (AI) promises to further reinforce the role of Pathology in the scenario of Personalized Medicine: AI-based devices are expected to standardize the evaluation of tissue biomarkers and also to discover novel information, which would otherwise be ignored by human review, and use them to make specific predictions. In this review we will present how AI has been used to support Tissue Biomarkers evaluation in the specific field of Pathology, give an insight to the intriguing field of AI-based biomarkers and discuss possible advantages, risk and perspectives for Pathology.
Collapse
Affiliation(s)
- Cesare Lancellotti
- Department of Biomedical Sciences, Humanitas University, Via Rita Levi Montalcini 4, 20090 Pieve Emanuele, Milan, Italy; (C.L.); (S.R.R.K.)
- Pathology Unit, Humanitas Research Hospital, Via Manzoni 56, 20089 Rozzano, Milan, Italy
| | - Pierandrea Cancian
- Artificial Intelligence Center IRCCS Humanitas Research Hospital, Via Manzoni 56, 20089 Rozzano, Milan, Italy; (P.C.); (V.S.)
| | - Victor Savevski
- Artificial Intelligence Center IRCCS Humanitas Research Hospital, Via Manzoni 56, 20089 Rozzano, Milan, Italy; (P.C.); (V.S.)
| | - Soumya Rupa Reddy Kotha
- Department of Biomedical Sciences, Humanitas University, Via Rita Levi Montalcini 4, 20090 Pieve Emanuele, Milan, Italy; (C.L.); (S.R.R.K.)
- Pathology Unit, Humanitas Research Hospital, Via Manzoni 56, 20089 Rozzano, Milan, Italy
| | | | - Paolo Graziano
- Pathology Unit, Fondazione IRCCS Casa Sollievo della Sofferenza, 71013 San Giovanni Rotondo, FG, Italy;
| | - Luca Di Tommaso
- Department of Biomedical Sciences, Humanitas University, Via Rita Levi Montalcini 4, 20090 Pieve Emanuele, Milan, Italy; (C.L.); (S.R.R.K.)
- Pathology Unit, Humanitas Research Hospital, Via Manzoni 56, 20089 Rozzano, Milan, Italy
| |
Collapse
|
325
|
Manzari MT, Shamay Y, Kiguchi H, Rosen N, Scaltriti M, Heller DA. Targeted drug delivery strategies for precision medicines. NATURE REVIEWS. MATERIALS 2021; 6:351-370. [PMID: 34950512 PMCID: PMC8691416 DOI: 10.1038/s41578-020-00269-6] [Citation(s) in RCA: 356] [Impact Index Per Article: 118.7] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Accepted: 11/24/2020] [Indexed: 05/05/2023]
Abstract
Progress in the field of precision medicine has changed the landscape of cancer therapy. Precision medicine is propelled by technologies that enable molecular profiling, genomic analysis, and optimized drug design to tailor treatments for individual patients. Although precision medicines have resulted in some clinical successes, the use of many potential therapeutics has been hindered by pharmacological issues, including toxicities and drug resistance. Drug delivery materials and approaches have now advanced to a point where they can enable the modulation of a drug's pharmacological parameters without compromising the desired effect on molecular targets. Specifically, they can modulate a drug's pharmacokinetics, stability, absorption, and exposure to tumours and healthy tissues, and facilitate the administration of synergistic drug combinations. This Review highlights recent progress in precision therapeutics and drug delivery, and identifies opportunities for strategies to improve the therapeutic index of cancer drugs, and consequently, clinical outcomes.
Collapse
Affiliation(s)
- Mandana T. Manzari
- Molecular Pharmacology Program, Memorial Sloan Kettering Cancer Center, New York, NY, USA
- These authors have contributed equally to this work
| | - Yosi Shamay
- Faculty of Biomedical Engineering, Technion-Israel Institute of Technology, Haifa, Israel
- These authors have contributed equally to this work
| | - Hiroto Kiguchi
- Department of Pediatrics, Memorial Sloan Kettering Cancer Center, New York, NY, USA
- Division of Oncology, Children’s Hospital of Philadelphia, Philadelphia, PA, USA
- These authors have contributed equally to this work
| | - Neal Rosen
- Department of Medicine, Memorial Sloan Kettering Cancer Center, New York, NY, USA
- Weill Cornell Medical College, New York, NY, USA
- Human Oncology and Pathogenesis Program, Memorial Sloan Kettering Cancer, New York, NY, USA
| | - Maurizio Scaltriti
- Human Oncology and Pathogenesis Program, Memorial Sloan Kettering Cancer, New York, NY, USA
- Department of Pathology, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| | - Daniel A. Heller
- Molecular Pharmacology Program, Memorial Sloan Kettering Cancer Center, New York, NY, USA
- Weill Cornell Medical College, New York, NY, USA
| |
Collapse
|
326
|
Zeng T, Yu X, Chen Z. Applying artificial intelligence in the microbiome for gastrointestinal diseases: A review. J Gastroenterol Hepatol 2021; 36:832-840. [PMID: 33880762 DOI: 10.1111/jgh.15503] [Citation(s) in RCA: 12] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/09/2021] [Revised: 03/18/2021] [Accepted: 03/18/2021] [Indexed: 12/20/2022]
Abstract
For a long time, gut bacteria have been recognized for their important roles in the occurrence and progression of gastrointestinal diseases like colorectal cancer, and the ever-increasing amounts of microbiome data combined with other high-quality clinical and imaging datasets are leading the study of gastrointestinal diseases into an era of biomedical big data. The "omics" technologies used for microbiome analysis continuously evolve, and the machine learning or artificial intelligence technologies are key to extract the relevant information from microbiome data. This review intends to provide a focused summary of recent research and applications of microbiome big data and to discuss the use of artificial intelligence to combat gastrointestinal diseases.
Collapse
Affiliation(s)
- Tao Zeng
- CAS Key Laboratory of Computational Biology, Bio-Med Big Data Center, Shanghai Institute of Nutrition and Health, University of Chinese Academy of Sciences, Chinese Academy of Sciences, Shanghai, China
- Key Laboratory of Systems Biology, Shanghai Institute of Biochemistry and Cell Biology, Center for Excellence in Molecular Cell Science, Chinese Academy of Sciences, Shanghai, China
| | - Xiangtian Yu
- Clinical Reasearch Center, Shanghai Jiao Tong University Affiliated Sixth People's Hospital, Shanghai, China
| | - Zhangran Chen
- Institute for Microbial Ecology, School of Medicine, Xiamen University, Xiamen, China
| |
Collapse
|
327
|
Ke J, Shen Y, Lu Y, Deng J, Wright JD, Zhang Y, Huang Q, Wang D, Jing N, Liang X, Jiang F. Quantitative analysis of abnormalities in gynecologic cytopathology with deep learning. J Transl Med 2021; 101:513-524. [PMID: 33526806 DOI: 10.1038/s41374-021-00537-1] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/16/2020] [Revised: 12/21/2020] [Accepted: 01/04/2021] [Indexed: 12/19/2022] Open
Abstract
Cervical cancer is one of the most frequent cancers in women worldwide, yet the early detection and treatment of lesions via regular cervical screening have led to a drastic reduction in the mortality rate. However, the routine examination of screening as a regular health checkup of women is characterized as time-consuming and labor-intensive, while there is lack of characteristic phenotypic profile and quantitative analysis. In this research, over the analysis of a privately collected and manually annotated dataset of 130 cytological whole-slide images, the authors proposed a deep-learning diagnostic system to localize, grade, and quantify squamous cell abnormalities. The system can distinguish abnormalities at the morphology level, namely atypical squamous cells of undetermined significance, low-grade squamous intraepithelial lesion, high-grade squamous intraepithelial lesion, and squamous cell carcinoma, as well as differential phenotypes of normal cells. The case study covered 51 positive and 79 negative digital gynecologic cytology slides collected from 2016 to 2018. Our automatic diagnostic system demonstrated its sensitivity of 100% at slide-level abnormality prediction, with the confirmation with three pathologists who performed slide-level diagnosis and training sample annotations. In the cellular-level classification, we yielded an accuracy of 94.5% in the binary classification between normality and abnormality, and the AUC was above 85% for each subtype of epithelial abnormality. Although the final confirmation from pathologists is often a must, empirically, computer-aided methods are capable of the effective extraction, interpretation, and quantification of morphological features, while also making it more objective and reproducible.
Collapse
Affiliation(s)
- Jing Ke
- Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China.
- School of Computer Science and Engineering, University of New South Wales, Sydney, NSW, Australia.
| | - Yiqing Shen
- School of Mathematical Sciences, Shanghai Jiao Tong University, Shanghai, China
| | - Yizhou Lu
- School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, China
| | - Junwei Deng
- School of Information, University of Michigan, Ann Arbor, MI, USA
| | - Jason D Wright
- Department of Obstetrics and Gynecology, Columbia University, New York, NY, USA
| | - Yan Zhang
- Department of Pathology, Shanghai Tongshu Medical Laboratory Co.Ltd, Shanghai, China
| | - Qin Huang
- Department of Endocrinology and Metabolism, Shanghai Jiao Tong University Affiliated Sixth People's Hospital, Shanghai Clinical Center for Diabetes, Shanghai, China
| | - Dadong Wang
- Quantitative Imaging, Data61 CSIRO, Sydney, NSW, Australia
| | - Naifeng Jing
- Department of Micro-Nano Electronics, Shanghai Jiao Tong University, Shanghai, China
| | - Xiaoyao Liang
- Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China
- Biren Research, Shanghai, China, Shanghai, China
| | - Fusong Jiang
- Department of Endocrinology and Metabolism, Shanghai Jiao Tong University Affiliated Sixth People's Hospital, Shanghai Clinical Center for Diabetes, Shanghai, China
| |
Collapse
|
328
|
Cheng J, Liu Y, Huang W, Hong W, Wang L, Zhan X, Han Z, Ni D, Huang K, Zhang J. Computational Image Analysis Identifies Histopathological Image Features Associated With Somatic Mutations and Patient Survival in Gastric Adenocarcinoma. Front Oncol 2021; 11:623382. [PMID: 33869007 PMCID: PMC8045755 DOI: 10.3389/fonc.2021.623382] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/30/2020] [Accepted: 03/15/2021] [Indexed: 12/24/2022] Open
Abstract
Computational analysis of histopathological images can identify sub-visual objective image features that may not be visually distinguishable by human eyes, and hence provides better modeling of disease phenotypes. This study aims to investigate whether specific image features are associated with somatic mutations and patient survival in gastric adenocarcinoma (sample size = 310). An automated image analysis pipeline was developed to extract quantitative morphological features from H&E stained whole-slide images. We found that four frequently somatically mutated genes (TP53, ARID1A, OBSCN, and PIK3CA) were significantly associated with tumor morphological changes. A prognostic model built on the image features significantly stratified patients into low-risk and high-risk groups (log-rank test p-value = 2.6e-4). Multivariable Cox regression showed the model predicted risk index was an additional prognostic factor besides tumor grade and stage. Gene ontology enrichment analysis showed that the genes whose expressions mostly correlated with the contributing features in the prognostic model were enriched on biological processes such as cell cycle and muscle contraction. These results demonstrate that histopathological image features can reflect underlying somatic mutations and identify high-risk patients that may benefit from more precise treatment regimens. Both the image features and pipeline are highly interpretable to enable translational applications.
Collapse
Affiliation(s)
- Jun Cheng
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, Shenzhen University, Shenzhen, China.,Guangdong Key Laboratory for Biomedical Measurements and Ultrasound Imaging, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, China.,Marshall Laboratory of Biomedical Engineering Shenzhen University, Shenzhen, China
| | - Yuting Liu
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, Shenzhen University, Shenzhen, China.,Guangdong Key Laboratory for Biomedical Measurements and Ultrasound Imaging, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, China
| | - Wei Huang
- Department of Radiation Oncology, Guangdong Provincial People's Hospital, Guangdong Academy of Medical Sciences, School of Medicine, South China University of Technology, Guangzhou, China
| | - Wenhui Hong
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, Shenzhen University, Shenzhen, China.,Guangdong Key Laboratory for Biomedical Measurements and Ultrasound Imaging, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, China
| | - Lingling Wang
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, Shenzhen University, Shenzhen, China.,Guangdong Key Laboratory for Biomedical Measurements and Ultrasound Imaging, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, China
| | - Xiaohui Zhan
- School of Basic Medicine, Chongqing Medical University, Chongqin, China
| | - Zhi Han
- Department of Medicine, Indiana University, School of Medicine, Indianapolis, IN, United States
| | - Dong Ni
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, Shenzhen University, Shenzhen, China.,Guangdong Key Laboratory for Biomedical Measurements and Ultrasound Imaging, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, China.,Marshall Laboratory of Biomedical Engineering Shenzhen University, Shenzhen, China
| | - Kun Huang
- Department of Medicine, Indiana University, School of Medicine, Indianapolis, IN, United States
| | - Jie Zhang
- Department of Medical and Molecular Genetics, Indiana University School of Medicine, Indianapolis, IN, United States
| |
Collapse
|
329
|
Wang KS, Yu G, Xu C, Meng XH, Zhou J, Zheng C, Deng Z, Shang L, Liu R, Su S, Zhou X, Li Q, Li J, Wang J, Ma K, Qi J, Hu Z, Tang P, Deng J, Qiu X, Li BY, Shen WD, Quan RP, Yang JT, Huang LY, Xiao Y, Yang ZC, Li Z, Wang SC, Ren H, Liang C, Guo W, Li Y, Xiao H, Gu Y, Yun JP, Huang D, Song Z, Fan X, Chen L, Yan X, Li Z, Huang ZC, Huang J, Luttrell J, Zhang CY, Zhou W, Zhang K, Yi C, Wu C, Shen H, Wang YP, Xiao HM, Deng HW. Accurate diagnosis of colorectal cancer based on histopathology images using artificial intelligence. BMC Med 2021; 19:76. [PMID: 33752648 PMCID: PMC7986569 DOI: 10.1186/s12916-021-01942-5] [Citation(s) in RCA: 55] [Impact Index Per Article: 18.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/22/2020] [Accepted: 02/16/2021] [Indexed: 02/06/2023] Open
Abstract
BACKGROUND Accurate and robust pathological image analysis for colorectal cancer (CRC) diagnosis is time-consuming and knowledge-intensive, but is essential for CRC patients' treatment. The current heavy workload of pathologists in clinics/hospitals may easily lead to unconscious misdiagnosis of CRC based on daily image analyses. METHODS Based on a state-of-the-art transfer-learned deep convolutional neural network in artificial intelligence (AI), we proposed a novel patch aggregation strategy for clinic CRC diagnosis using weakly labeled pathological whole-slide image (WSI) patches. This approach was trained and validated using an unprecedented and enormously large number of 170,099 patches, > 14,680 WSIs, from > 9631 subjects that covered diverse and representative clinical cases from multi-independent-sources across China, the USA, and Germany. RESULTS Our innovative AI tool consistently and nearly perfectly agreed with (average Kappa statistic 0.896) and even often better than most of the experienced expert pathologists when tested in diagnosing CRC WSIs from multicenters. The average area under the receiver operating characteristics curve (AUC) of AI was greater than that of the pathologists (0.988 vs 0.970) and achieved the best performance among the application of other AI methods to CRC diagnosis. Our AI-generated heatmap highlights the image regions of cancer tissue/cells. CONCLUSIONS This first-ever generalizable AI system can handle large amounts of WSIs consistently and robustly without potential bias due to fatigue commonly experienced by clinical pathologists. It will drastically alleviate the heavy clinical burden of daily pathology diagnosis and improve the treatment for CRC patients. This tool is generalizable to other cancer diagnosis based on image recognition.
Collapse
Affiliation(s)
- K S Wang
- Department of Pathology, Xiangya Hospital, Central South University, Changsha, 410078, Hunan, China
- Department of Pathology, School of Basic Medical Science, Central South University, Changsha, 410013, Hunan, China
| | - G Yu
- Department of Biomedical Engineering, School of Basic Medical Science, Central South University, Changsha, 410013, Hunan, China
| | - C Xu
- Department of Biostatistics and Epidemiology, The University of Oklahoma Health Sciences Center, Oklahoma City, OK, 73104, USA
| | - X H Meng
- Laboratory of Molecular and Statistical Genetics, College of Life Sciences, Hunan Normal University, Changsha, 410081, Hunan, China
| | - J Zhou
- Department of Pathology, Xiangya Hospital, Central South University, Changsha, 410078, Hunan, China
- Department of Pathology, School of Basic Medical Science, Central South University, Changsha, 410013, Hunan, China
| | - C Zheng
- Department of Pathology, Xiangya Hospital, Central South University, Changsha, 410078, Hunan, China
- Department of Pathology, School of Basic Medical Science, Central South University, Changsha, 410013, Hunan, China
| | - Z Deng
- Department of Pathology, Xiangya Hospital, Central South University, Changsha, 410078, Hunan, China
- Department of Pathology, School of Basic Medical Science, Central South University, Changsha, 410013, Hunan, China
| | - L Shang
- Department of Pathology, Xiangya Hospital, Central South University, Changsha, 410078, Hunan, China
| | - R Liu
- Department of Pathology, Xiangya Hospital, Central South University, Changsha, 410078, Hunan, China
| | - S Su
- Department of Pathology, Xiangya Hospital, Central South University, Changsha, 410078, Hunan, China
| | - X Zhou
- Department of Pathology, Xiangya Hospital, Central South University, Changsha, 410078, Hunan, China
| | - Q Li
- Department of Pathology, Xiangya Hospital, Central South University, Changsha, 410078, Hunan, China
| | - J Li
- Department of Pathology, Xiangya Hospital, Central South University, Changsha, 410078, Hunan, China
| | - J Wang
- Department of Pathology, Xiangya Hospital, Central South University, Changsha, 410078, Hunan, China
| | - K Ma
- Department of Pathology, School of Basic Medical Science, Central South University, Changsha, 410013, Hunan, China
| | - J Qi
- Department of Pathology, School of Basic Medical Science, Central South University, Changsha, 410013, Hunan, China
| | - Z Hu
- Department of Pathology, School of Basic Medical Science, Central South University, Changsha, 410013, Hunan, China
| | - P Tang
- Department of Pathology, School of Basic Medical Science, Central South University, Changsha, 410013, Hunan, China
| | - J Deng
- Department of Deming Department of Medicine, Tulane Center of Biomedical Informatics and Genomics, Tulane University School of Medicine, 1440 Canal Street, Suite 1610, New Orleans, LA, 70112, USA
| | - X Qiu
- Centers of System Biology, Data Information and Reproductive Health, School of Basic Medical Science, School of Basic Medical Science, Central South University, Changsha, 410008, Hunan, China
| | - B Y Li
- Centers of System Biology, Data Information and Reproductive Health, School of Basic Medical Science, School of Basic Medical Science, Central South University, Changsha, 410008, Hunan, China
| | - W D Shen
- Centers of System Biology, Data Information and Reproductive Health, School of Basic Medical Science, School of Basic Medical Science, Central South University, Changsha, 410008, Hunan, China
| | - R P Quan
- Centers of System Biology, Data Information and Reproductive Health, School of Basic Medical Science, School of Basic Medical Science, Central South University, Changsha, 410008, Hunan, China
| | - J T Yang
- Centers of System Biology, Data Information and Reproductive Health, School of Basic Medical Science, School of Basic Medical Science, Central South University, Changsha, 410008, Hunan, China
| | - L Y Huang
- Centers of System Biology, Data Information and Reproductive Health, School of Basic Medical Science, School of Basic Medical Science, Central South University, Changsha, 410008, Hunan, China
| | - Y Xiao
- Centers of System Biology, Data Information and Reproductive Health, School of Basic Medical Science, School of Basic Medical Science, Central South University, Changsha, 410008, Hunan, China
| | - Z C Yang
- Department of Pharmacology, Xiangya School of Pharmaceutical Sciences, Central South University, Changsha, 410078, Hunan, China
| | - Z Li
- School of Life Sciences, Central South University, Changsha, 410013, Hunan, China
| | - S C Wang
- College of Information Science and Engineering, Hunan Normal University, Changsha, 410081, Hunan, China
| | - H Ren
- Department of Pathology, Gongli Hospital, Second Military Medical University, Shanghai, 200135, China
- Department of Pathology, the Peace Hospital Affiliated to Changzhi Medical College, Changzhi, 046000, China
| | - C Liang
- Pathological Laboratory of Adicon Medical Laboratory Co., Ltd, Hangzhou, 310023, Zhejiang, China
| | - W Guo
- Department of Pathology, First Affiliated Hospital of Hunan Normal University, The People's Hospital of Hunan Province, Changsha, 410005, Hunan, China
| | - Y Li
- Department of Pathology, First Affiliated Hospital of Hunan Normal University, The People's Hospital of Hunan Province, Changsha, 410005, Hunan, China
| | - H Xiao
- Department of Pathology, the Third Xiangya Hospital, Central South University, Changsha, 410013, Hunan, China
| | - Y Gu
- Department of Pathology, the Third Xiangya Hospital, Central South University, Changsha, 410013, Hunan, China
| | - J P Yun
- Department of Pathology, Sun Yat-Sen University Cancer Center, Guangzhou, 510060, China
| | - D Huang
- Department of Pathology, Fudan University Shanghai Cancer Center, Shanghai, 200032, China
| | - Z Song
- Department of Pathology, Chinese PLA General Hospital, Beijing, 100853, China
| | - X Fan
- Department of Pathology, Nanjing Drum Tower Hospital, the Affiliated Hospital of Nanjing University Medical School, Nanjing, 210008, China
| | - L Chen
- Department of Pathology, The first affiliated hospital, Air Force Medical University, Xi'an, 710032, China
| | - X Yan
- Institute of Pathology and southwest cancer center, Southwest Hospital, Third Military Medical University, Chongqing, 400038, China
| | - Z Li
- Department of Pathology, Guangdong Provincial People's Hospital, Guangdong Academy of Medical Sciences, Guangzhou, 510080, China
| | - Z C Huang
- Department of Biomedical Engineering, School of Basic Medical Science, Central South University, Changsha, 410013, Hunan, China
| | - J Huang
- Department of Anatomy and Neurobiology, School of Basic Medical Science, Central South University, Changsha, 410013, Hunan, China
| | - J Luttrell
- School of Computing Sciences and Computer Engineering, University of Southern Mississippi, Hattiesburg, MS, 39406, USA
| | - C Y Zhang
- School of Computing Sciences and Computer Engineering, University of Southern Mississippi, Hattiesburg, MS, 39406, USA
| | - W Zhou
- College of Computing, Michigan Technological University, Houghton, MI, 49931, USA
| | - K Zhang
- Department of Computer Science, Bioinformatics Facility of Xavier NIH RCMI Cancer Research Center, Xavier University of Louisiana, New Orleans, LA, 70125, USA
| | - C Yi
- Department of Pathology, Ochsner Medical Center, New Orleans, LA, 70121, USA
| | - C Wu
- Department of Statistics, Florida State University, Tallahassee, FL, 32306, USA
| | - H Shen
- Department of Deming Department of Medicine, Tulane Center of Biomedical Informatics and Genomics, Tulane University School of Medicine, 1440 Canal Street, Suite 1610, New Orleans, LA, 70112, USA
- Division of Biomedical Informatics and Genomics, Deming Department of Medicine, Tulane University School of Medicine, New Orleans, LA, 70112, USA
| | - Y P Wang
- Department of Deming Department of Medicine, Tulane Center of Biomedical Informatics and Genomics, Tulane University School of Medicine, 1440 Canal Street, Suite 1610, New Orleans, LA, 70112, USA
- Department of Biomedical Engineering, Tulane University, New Orleans, LA, 70118, USA
| | - H M Xiao
- Centers of System Biology, Data Information and Reproductive Health, School of Basic Medical Science, School of Basic Medical Science, Central South University, Changsha, 410008, Hunan, China.
| | - H W Deng
- Department of Deming Department of Medicine, Tulane Center of Biomedical Informatics and Genomics, Tulane University School of Medicine, 1440 Canal Street, Suite 1610, New Orleans, LA, 70112, USA.
- Centers of System Biology, Data Information and Reproductive Health, School of Basic Medical Science, School of Basic Medical Science, Central South University, Changsha, 410008, Hunan, China.
- Division of Biomedical Informatics and Genomics, Deming Department of Medicine, Tulane University School of Medicine, New Orleans, LA, 70112, USA.
| |
Collapse
|
330
|
Two Ensemble-CNN Approaches for Colorectal Cancer Tissue Type Classification. J Imaging 2021; 7:jimaging7030051. [PMID: 34460707 PMCID: PMC8321410 DOI: 10.3390/jimaging7030051] [Citation(s) in RCA: 13] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/23/2021] [Revised: 02/16/2021] [Accepted: 02/26/2021] [Indexed: 02/06/2023] Open
Abstract
In recent years, automatic tissue phenotyping has attracted increasing interest in the Digital Pathology (DP) field. For Colorectal Cancer (CRC), tissue phenotyping can diagnose the cancer and differentiate between different cancer grades. The development of Whole Slide Images (WSIs) has provided the required data for creating automatic tissue phenotyping systems. In this paper, we study different hand-crafted feature-based and deep learning methods using two popular multi-classes CRC-tissue-type databases: Kather-CRC-2016 and CRC-TP. For the hand-crafted features, we use two texture descriptors (LPQ and BSIF) and their combination. In addition, two classifiers are used (SVM and NN) to classify the texture features into distinct CRC tissue types. For the deep learning methods, we evaluate four Convolutional Neural Network (CNN) architectures (ResNet-101, ResNeXt-50, Inception-v3, and DenseNet-161). Moreover, we propose two Ensemble CNN approaches: Mean-Ensemble-CNN and NN-Ensemble-CNN. The experimental results show that the proposed approaches outperformed the hand-crafted feature-based methods, CNN architectures and the state-of-the-art methods in both databases.
Collapse
|
331
|
Maassen O, Fritsch S, Palm J, Deffge S, Kunze J, Marx G, Riedel M, Schuppert A, Bickenbach J. Future Medical Artificial Intelligence Application Requirements and Expectations of Physicians in German University Hospitals: Web-Based Survey. J Med Internet Res 2021; 23:e26646. [PMID: 33666563 PMCID: PMC7980122 DOI: 10.2196/26646] [Citation(s) in RCA: 27] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/19/2020] [Revised: 01/29/2021] [Accepted: 02/15/2021] [Indexed: 12/13/2022] Open
Abstract
BACKGROUND The increasing development of artificial intelligence (AI) systems in medicine driven by researchers and entrepreneurs goes along with enormous expectations for medical care advancement. AI might change the clinical practice of physicians from almost all medical disciplines and in most areas of health care. While expectations for AI in medicine are high, practical implementations of AI for clinical practice are still scarce in Germany. Moreover, physicians' requirements and expectations of AI in medicine and their opinion on the usage of anonymized patient data for clinical and biomedical research have not been investigated widely in German university hospitals. OBJECTIVE This study aimed to evaluate physicians' requirements and expectations of AI in medicine and their opinion on the secondary usage of patient data for (bio)medical research (eg, for the development of machine learning algorithms) in university hospitals in Germany. METHODS A web-based survey was conducted addressing physicians of all medical disciplines in 8 German university hospitals. Answers were given using Likert scales and general demographic responses. Physicians were asked to participate locally via email in the respective hospitals. RESULTS The online survey was completed by 303 physicians (female: 121/303, 39.9%; male: 173/303, 57.1%; no response: 9/303, 3.0%) from a wide range of medical disciplines and work experience levels. Most respondents either had a positive (130/303, 42.9%) or a very positive attitude (82/303, 27.1%) towards AI in medicine. There was a significant association between the personal rating of AI in medicine and the self-reported technical affinity level (H4=48.3, P<.001). A vast majority of physicians expected the future of medicine to be a mix of human and artificial intelligence (273/303, 90.1%) but also requested a scientific evaluation before the routine implementation of AI-based systems (276/303, 91.1%). Physicians were most optimistic that AI applications would identify drug interactions (280/303, 92.4%) to improve patient care substantially but were quite reserved regarding AI-supported diagnosis of psychiatric diseases (62/303, 20.5%). Of the respondents, 82.5% (250/303) agreed that there should be open access to anonymized patient databases for medical and biomedical research. CONCLUSIONS Physicians in stationary patient care in German university hospitals show a generally positive attitude towards using most AI applications in medicine. Along with this optimism comes several expectations and hopes that AI will assist physicians in clinical decision making. Especially in fields of medicine where huge amounts of data are processed (eg, imaging procedures in radiology and pathology) or data are collected continuously (eg, cardiology and intensive care medicine), physicians' expectations of AI to substantially improve future patient care are high. In the study, the greatest potential was seen in the application of AI for the identification of drug interactions, assumedly due to the rising complexity of drug administration to polymorbid, polypharmacy patients. However, for the practical usage of AI in health care, regulatory and organizational challenges still have to be mastered.
Collapse
Affiliation(s)
- Oliver Maassen
- Department of Intensive Care Medicine, University Hospital RWTH Aachen, Aachen, Germany
- SMITH Consortium of the German Medical Informatics Initiative, Leipzig, Germany
| | - Sebastian Fritsch
- Department of Intensive Care Medicine, University Hospital RWTH Aachen, Aachen, Germany
- SMITH Consortium of the German Medical Informatics Initiative, Leipzig, Germany
- Jülich Supercomputing Centre, Forschungszentrum Jülich, Jülich, Germany
| | - Julia Palm
- SMITH Consortium of the German Medical Informatics Initiative, Leipzig, Germany
- Institute of Medical Statistics, Computer and Data Sciences, Jena University Hospital, Jena, Germany
| | - Saskia Deffge
- Department of Intensive Care Medicine, University Hospital RWTH Aachen, Aachen, Germany
- SMITH Consortium of the German Medical Informatics Initiative, Leipzig, Germany
| | - Julian Kunze
- Department of Intensive Care Medicine, University Hospital RWTH Aachen, Aachen, Germany
- SMITH Consortium of the German Medical Informatics Initiative, Leipzig, Germany
| | - Gernot Marx
- Department of Intensive Care Medicine, University Hospital RWTH Aachen, Aachen, Germany
- SMITH Consortium of the German Medical Informatics Initiative, Leipzig, Germany
| | - Morris Riedel
- SMITH Consortium of the German Medical Informatics Initiative, Leipzig, Germany
- Jülich Supercomputing Centre, Forschungszentrum Jülich, Jülich, Germany
- School of Natural Sciences and Engineering, University of Iceland, Reykjavik, Iceland
| | - Andreas Schuppert
- SMITH Consortium of the German Medical Informatics Initiative, Leipzig, Germany
- Institute for Computational Biomedicine II, University Hospital RWTH Aachen, Aachen, Germany
| | - Johannes Bickenbach
- Department of Intensive Care Medicine, University Hospital RWTH Aachen, Aachen, Germany
- SMITH Consortium of the German Medical Informatics Initiative, Leipzig, Germany
| |
Collapse
|
332
|
Kleppe A, Skrede OJ, De Raedt S, Liestøl K, Kerr DJ, Danielsen HE. Designing deep learning studies in cancer diagnostics. Nat Rev Cancer 2021; 21:199-211. [PMID: 33514930 DOI: 10.1038/s41568-020-00327-9] [Citation(s) in RCA: 140] [Impact Index Per Article: 46.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 12/09/2020] [Indexed: 12/16/2022]
Abstract
The number of publications on deep learning for cancer diagnostics is rapidly increasing, and systems are frequently claimed to perform comparable with or better than clinicians. However, few systems have yet demonstrated real-world medical utility. In this Perspective, we discuss reasons for the moderate progress and describe remedies designed to facilitate transition to the clinic. Recent, presumably influential, deep learning studies in cancer diagnostics, of which the vast majority used images as input to the system, are evaluated to reveal the status of the field. By manipulating real data, we then exemplify that much and varied training data facilitate the generalizability of neural networks and thus the ability to use them clinically. To reduce the risk of biased performance estimation of deep learning systems, we advocate evaluation in external cohorts and strongly advise that the planned analyses, including a predefined primary analysis, are described in a protocol preferentially stored in an online repository. Recommended protocol items should be established for the field, and we present our suggestions.
Collapse
Affiliation(s)
- Andreas Kleppe
- Institute for Cancer Genetics and Informatics, Oslo University Hospital, Oslo, Norway
- Department of Informatics, University of Oslo, Oslo, Norway
| | - Ole-Johan Skrede
- Institute for Cancer Genetics and Informatics, Oslo University Hospital, Oslo, Norway
- Department of Informatics, University of Oslo, Oslo, Norway
| | - Sepp De Raedt
- Institute for Cancer Genetics and Informatics, Oslo University Hospital, Oslo, Norway
- Department of Informatics, University of Oslo, Oslo, Norway
| | - Knut Liestøl
- Institute for Cancer Genetics and Informatics, Oslo University Hospital, Oslo, Norway
- Department of Informatics, University of Oslo, Oslo, Norway
| | - David J Kerr
- Nuffield Division of Clinical Laboratory Sciences, University of Oxford, Oxford, UK
| | - Håvard E Danielsen
- Institute for Cancer Genetics and Informatics, Oslo University Hospital, Oslo, Norway.
- Department of Informatics, University of Oslo, Oslo, Norway.
- Nuffield Division of Clinical Laboratory Sciences, University of Oxford, Oxford, UK.
| |
Collapse
|
333
|
Zhao K, Wu L, Huang Y, Yao S, Xu Z, Lin H, Wang H, Liang Y, Xu Y, Chen X, Zhao M, Peng J, Huang Y, Liang C, Li Z, Li Y, Liu Z. Deep learning quantified mucus-tumor ratio predicting survival of patients with colorectal cancer using whole-slide images. PRECISION CLINICAL MEDICINE 2021; 4:17-24. [PMID: 35693123 PMCID: PMC8982603 DOI: 10.1093/pcmedi/pbab002] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/25/2020] [Revised: 01/14/2021] [Accepted: 01/24/2021] [Indexed: 02/05/2023] Open
Abstract
Background In colorectal cancer (CRC), mucinous adenocarcinoma differs from other adenocarcinomas in gene-phenotype, morphology, and prognosis. However, mucinous components are present in a large number of adenocarcinomas, and the prognostic value of mucus proportion has not been investigated. Artificial intelligence provides a way to quantify mucus proportion on whole-slide images (WSIs) accurately. We aimed to quantify mucus proportion by deep learning and further investigate its prognostic value in two CRC patient cohorts. Methods Deep learning was used to segment WSIs stained with hematoxylin and eosin. Mucus-tumor ratio (MTR) was defined as the proportion of mucinous component in the tumor area. A training cohort (N = 419) and a validation cohort (N = 315) were used to evaluate the prognostic value of MTR. Survival analysis was performed using the Cox proportional hazard model. Result Patients were stratified to mucus-low and mucus-high groups, with 24.1% as the threshold. In the training cohort, patients with mucus-high had unfavorable outcomes (hazard ratio for high vs. low 1.88, 95% confidence interval 1.18-2.99, P = 0.008), with 5-year overall survival rates of 54.8% and 73.7% in mucus-high and mucus-low groups, respectively. The results were confirmed in the validation cohort (2.09, 1.21-3.60, 0.008; 62.8% vs. 79.8%). The prognostic value of MTR was maintained in multivariate analysis for both cohorts. Conclusion The deep learning quantified MTR was an independent prognostic factor in CRC. With the advantages of advanced efficiency and high consistency, our method is suitable for clinical application and promotes precision medicine development.
Collapse
Affiliation(s)
- Ke Zhao
- Department of Radiology, Guangdong Provincial People's Hospital, Guangdong Academy of Medical Sciences, Guangzhou 510080, China
- School of Medicine, South China University of Technology, Guangzhou 510006, China
| | - Lin Wu
- Department of Pathology, the Third Affiliated Hospital of Kunming Medical University, Yunnan Cancer Hospital, Yunnan Cancer Center, Kunming 650118, China
| | - Yanqi Huang
- Department of Radiology, Guangdong Provincial People's Hospital, Guangdong Academy of Medical Sciences, Guangzhou 510080, China
- The Second School of Clinical Medicine, Southern Medical University, Guangzhou 510080, China
| | - Su Yao
- Department of Pathology, Guangdong Provincial People's Hospital, Guangdong Academy of Medical Sciences, Guangzhou 510080, China
| | - Zeyan Xu
- Department of Radiology, Guangdong Provincial People's Hospital, Guangdong Academy of Medical Sciences, Guangzhou 510080, China
- School of Medicine, South China University of Technology, Guangzhou 510006, China
| | - Huan Lin
- Department of Radiology, Guangdong Provincial People's Hospital, Guangdong Academy of Medical Sciences, Guangzhou 510080, China
- School of Medicine, South China University of Technology, Guangzhou 510006, China
| | - Huihui Wang
- Department of Radiology, Guangdong Provincial People's Hospital, Guangdong Academy of Medical Sciences, Guangzhou 510080, China
- Shantou University Medical College, Shantou 515041, China
| | - Yanting Liang
- Department of Radiology, Guangdong Provincial People's Hospital, Guangdong Academy of Medical Sciences, Guangzhou 510080, China
| | - Yao Xu
- School of Bioengineering, Chongqing University, Chongqing 400044, China
| | - Xin Chen
- Department of Radiology, Guangzhou First People's Hospital, School of Medicine, South China University of Technology, Guangzhou 510180, China
| | - Minning Zhao
- Department of Radiology, Guangdong Provincial People's Hospital, Guangdong Academy of Medical Sciences, Guangzhou 510080, China
- The Second School of Clinical Medicine, Southern Medical University, Guangzhou 510080, China
| | - Jiaming Peng
- School of Life Science and Technology, Xidian University, Xi'an 710071, China
| | - Yuli Huang
- School of Life Science and Technology, Xidian University, Xi'an 710071, China
| | - Changhong Liang
- Department of Radiology, Guangdong Provincial People's Hospital, Guangdong Academy of Medical Sciences, Guangzhou 510080, China
| | - Zhenhui Li
- Department of Radiology, the Third Affiliated Hospital of Kunming Medical University, Yunnan Cancer Hospital, Yunnan Cancer Center, Kunming 650118, China
| | - Yong Li
- Department of General Surgery, Guangdong Provincial People's Hospital, Guangdong Academy of Medical Sciences, Guangzhou 510080, China
| | - Zaiyi Liu
- Department of Radiology, Guangdong Provincial People's Hospital, Guangdong Academy of Medical Sciences, Guangzhou 510080, China
| |
Collapse
|
334
|
Klimov S, Xue Y, Gertych A, Graham RP, Jiang Y, Bhattarai S, Pandol SJ, Rakha EA, Reid MD, Aneja R. Predicting Metastasis Risk in Pancreatic Neuroendocrine Tumors Using Deep Learning Image Analysis. Front Oncol 2021; 10:593211. [PMID: 33718106 PMCID: PMC7946991 DOI: 10.3389/fonc.2020.593211] [Citation(s) in RCA: 11] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/10/2020] [Accepted: 12/18/2020] [Indexed: 12/20/2022] Open
Abstract
BACKGROUND The prognosis of patients with pancreatic neuroendocrine tumors (PanNET), the second most common type of pancreatic cancer, varies significantly, and up to 15% of patients develop metastasis. Although certain morphological characteristics of PanNETs have been associated with patient outcome, there are no available morphology-based prognostic markers. Given that current clinical histopathology markers are unable to identify high-risk PanNET patients, the development of accurate prognostic biomarkers is needed. Here, we describe a novel machine learning, multiclassification pipeline to predict the risk of metastasis using morphological information from whole tissue slides. METHODS Digital images from surgically resected tissues from 89 PanNET patients were used. Pathologist-annotated regions were extracted to train a convolutional neural network (CNN) to identify tiles consisting of PanNET, stroma, normal pancreas parenchyma, and fat. Computationally annotated cancer or stroma tiles and patient metastasis status were used to train CNN to calculate a region based metastatic risk score. Aggregation of the metastatic probability scores across the slide was performed to predict the risk of metastasis. RESULTS The ability of CNN to discriminate different tissues was high (per-tile accuracy >95%; whole slide cancer regions Jaccard index = 79%). Cancer and stromal tiles with high evaluated probability provided F1 scores of 0.82 and 0.69, respectively, when we compared tissues from patients who developed metastasis and those who did not. The final model identified low-risk (n = 76) and high-risk (n = 13) patients, as well as predicted metastasis-free survival (hazard ratio: 4.71) after adjusting for common clinicopathological variables, especially in grade I/II patients. CONCLUSION Using slides from surgically resected PanNETs, our novel, multiclassification, deep learning pipeline was able to predict the risk of metastasis in PanNET patients. Our results suggest the presence of prognostic morphological patterns in PanNET tissues, and that these patterns may help guide clinical decision making.
Collapse
Affiliation(s)
- Sergey Klimov
- Department of Biology, Georgia State University, Atlanta, GA, United States
- Department of Mathematics and Statistics, Georgia State University, Atlanta, GA, United States
| | - Yue Xue
- Department of Pathology, Northwestern University, Chicago, IL, United States
| | - Arkadiusz Gertych
- Department of Surgery, Cedars-Sinai Medical Center, Los Angeles, CA, United States
- Department of Pathology and Laboratory Medicine, Cedars-Sinai Medical Center, Los Angeles, CA, United States
- Faculty of Biomedical Engineering, Silesian University of Technology, Zabrze, Poland
| | - Rondell P Graham
- Department of Laboratory Medicine and Pathology, Mayo Clinic, Rochester, MN, United States
| | - Yi Jiang
- Department of Mathematics and Statistics, Georgia State University, Atlanta, GA, United States
| | - Shristi Bhattarai
- Department of Biology, Georgia State University, Atlanta, GA, United States
| | - Stephen J Pandol
- Department of Medicine, Cedars-Sinai Medical Center, Los Angeles, CA, United States
| | - Emad A Rakha
- Department of Cellular Pathology, University of Nottingham, Nottingham, United Kingdom
| | - Michelle D Reid
- Department of Pathology, Emory University, Atlanta, GA, United States
| | - Ritu Aneja
- Department of Biology, Georgia State University, Atlanta, GA, United States
| |
Collapse
|
335
|
Vu QD, Kim K, Kwak JT. Unsupervised Tumor Characterization via Conditional Generative Adversarial Networks. IEEE J Biomed Health Inform 2021; 25:348-357. [PMID: 32396112 DOI: 10.1109/jbhi.2020.2993560] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
Abstract
Grading for cancer, based upon the degree of cancer differentiation, plays a major role in describing the characteristics and behavior of the cancer and determining treatment plan for patients. The grade is determined by a subjective and qualitative assessment of tissues under microscope, which suffers from high inter- and intra-observer variability among pathologists. Digital pathology offers an alternative means to automate the procedure as well as to improve the accuracy and robustness of cancer grading. However, most of such methods tend to mimic or reproduce cancer grade determined by human experts. Herein, we propose an alternative, quantitative means of assessing and characterizing cancers in an unsupervised manner. The proposed method utilizes conditional generative adversarial networks to characterize tissues. The proposed method is evaluated using whole slide images (WSIs) and tissue microarrays (TMAs) of colorectal cancer specimens. The results suggest that the proposed method holds a potential for quantifying cancer characteristics and improving cancer pathology.
Collapse
|
336
|
Echle A, Rindtorff NT, Brinker TJ, Luedde T, Pearson AT, Kather JN. Deep learning in cancer pathology: a new generation of clinical biomarkers. Br J Cancer 2021; 124:686-696. [PMID: 33204028 PMCID: PMC7884739 DOI: 10.1038/s41416-020-01122-x] [Citation(s) in RCA: 243] [Impact Index Per Article: 81.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/11/2020] [Revised: 09/06/2020] [Accepted: 09/30/2020] [Indexed: 12/14/2022] Open
Abstract
Clinical workflows in oncology rely on predictive and prognostic molecular biomarkers. However, the growing number of these complex biomarkers tends to increase the cost and time for decision-making in routine daily oncology practice; furthermore, biomarkers often require tumour tissue on top of routine diagnostic material. Nevertheless, routinely available tumour tissue contains an abundance of clinically relevant information that is currently not fully exploited. Advances in deep learning (DL), an artificial intelligence (AI) technology, have enabled the extraction of previously hidden information directly from routine histology images of cancer, providing potentially clinically useful information. Here, we outline emerging concepts of how DL can extract biomarkers directly from histology images and summarise studies of basic and advanced image analysis for cancer histology. Basic image analysis tasks include detection, grading and subtyping of tumour tissue in histology images; they are aimed at automating pathology workflows and consequently do not immediately translate into clinical decisions. Exceeding such basic approaches, DL has also been used for advanced image analysis tasks, which have the potential of directly affecting clinical decision-making processes. These advanced approaches include inference of molecular features, prediction of survival and end-to-end prediction of therapy response. Predictions made by such DL systems could simplify and enrich clinical decision-making, but require rigorous external validation in clinical settings.
Collapse
Affiliation(s)
- Amelie Echle
- Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
| | | | - Titus Josef Brinker
- National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Tom Luedde
- Department of Gastroenterology, Hepatology and Infectious Diseases, University Hospital Duesseldorf, Düsseldorf, Germany
| | - Alexander Thomas Pearson
- Section of Hematology/Oncology, Department of Medicine, The University of Chicago, Chicago, IL, USA
| | - Jakob Nikolas Kather
- Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany.
- German Cancer Research Center (DKFZ), Heidelberg, Germany.
| |
Collapse
|
337
|
Nguyen HG, Blank A, Dawson HE, Lugli A, Zlobec I. Classification of colorectal tissue images from high throughput tissue microarrays by ensemble deep learning methods. Sci Rep 2021; 11:2371. [PMID: 33504830 PMCID: PMC7840737 DOI: 10.1038/s41598-021-81352-y] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/14/2020] [Accepted: 01/05/2021] [Indexed: 12/13/2022] Open
Abstract
Tissue microarray (TMA) core images are a treasure trove for artificial intelligence applications. However, a common problem of TMAs is multiple sectioning, which can change the content of the intended tissue core and requires re-labelling. Here, we investigate different ensemble methods for colorectal tissue classification using high-throughput TMAs. Hematoxylin and Eosin (H&E) core images of 0.6 mm or 1.0 mm diameter from three international cohorts were extracted from 54 digital slides (n = 15,150 cores). After TMA core extraction and color enhancement, five different flows of independent and ensemble deep learning were applied. Training and testing data with 2144 and 13,006 cores included three classes: tumor, normal or "other" tissue. Ground-truth data were collected from 30 ngTMA slides (n = 8689 cores). A test augmentation is applied to reduce the uncertain prediction. Predictive accuracy of the best method, namely Soft Voting Ensemble of one VGG and one CapsNet models was 0.982, 0.947 and 0.939 for normal, "other" and tumor, which outperformed to independent or ensemble learning with one base-estimator. Our high-accuracy algorithm for colorectal tissue classification in high-throughput TMAs is amenable to images from different institutions, core sizes and stain intensity. It helps to reduce error in TMA core evaluations with previously given labels.
Collapse
Affiliation(s)
- Huu-Giao Nguyen
- Institute of Pathology, University of Bern, Murtenstrasse 31, 3008, Bern, Switzerland
| | - Annika Blank
- Institute of Pathology, University of Bern, Murtenstrasse 31, 3008, Bern, Switzerland
- Institute of Pathology, Triemli City Hospital, Birmensdorferstrasse 497, 8063, Zurich, Switzerland
| | - Heather E Dawson
- Institute of Pathology, University of Bern, Murtenstrasse 31, 3008, Bern, Switzerland
| | - Alessandro Lugli
- Institute of Pathology, University of Bern, Murtenstrasse 31, 3008, Bern, Switzerland
| | - Inti Zlobec
- Institute of Pathology, University of Bern, Murtenstrasse 31, 3008, Bern, Switzerland.
| |
Collapse
|
338
|
Yamashita R, Long J, Longacre T, Peng L, Berry G, Martin B, Higgins J, Rubin DL, Shen J. Deep learning model for the prediction of microsatellite instability in colorectal cancer: a diagnostic study. Lancet Oncol 2021; 22:132-141. [PMID: 33387492 DOI: 10.1016/s1470-2045(20)30535-0] [Citation(s) in RCA: 191] [Impact Index Per Article: 63.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/31/2020] [Revised: 08/25/2020] [Accepted: 08/26/2020] [Indexed: 02/06/2023]
Abstract
BACKGROUND Detecting microsatellite instability (MSI) in colorectal cancer is crucial for clinical decision making, as it identifies patients with differential treatment response and prognosis. Universal MSI testing is recommended, but many patients remain untested. A critical need exists for broadly accessible, cost-efficient tools to aid patient selection for testing. Here, we investigate the potential of a deep learning-based system for automated MSI prediction directly from haematoxylin and eosin (H&E)-stained whole-slide images (WSIs). METHODS Our deep learning model (MSINet) was developed using 100 H&E-stained WSIs (50 with microsatellite stability [MSS] and 50 with MSI) scanned at 40× magnification, each from a patient randomly selected in a class-balanced manner from the pool of 343 patients who underwent primary colorectal cancer resection at Stanford University Medical Center (Stanford, CA, USA; internal dataset) between Jan 1, 2015, and Dec 31, 2017. We internally validated the model on a holdout test set (15 H&E-stained WSIs from 15 patients; seven cases with MSS and eight with MSI) and externally validated the model on 484 H&E-stained WSIs (402 cases with MSS and 77 with MSI; 479 patients) from The Cancer Genome Atlas, containing WSIs scanned at 40× and 20× magnification. Performance was primarily evaluated using the sensitivity, specificity, negative predictive value (NPV), and area under the receiver operating characteristic curve (AUROC). We compared the model's performance with that of five gastrointestinal pathologists on a class-balanced, randomly selected subset of 40× magnification WSIs from the external dataset (20 with MSS and 20 with MSI). FINDINGS The MSINet model achieved an AUROC of 0·931 (95% CI 0·771-1·000) on the holdout test set from the internal dataset and 0·779 (0·720-0·838) on the external dataset. On the external dataset, using a sensitivity-weighted operating point, the model achieved an NPV of 93·7% (95% CI 90·3-96·2), sensitivity of 76·0% (64·8-85·1), and specificity of 66·6% (61·8-71·2). On the reader experiment (40 cases), the model achieved an AUROC of 0·865 (95% CI 0·735-0·995). The mean AUROC performance of the five pathologists was 0·605 (95% CI 0·453-0·757). INTERPRETATION Our deep learning model exceeded the performance of experienced gastrointestinal pathologists at predicting MSI on H&E-stained WSIs. Within the current universal MSI testing paradigm, such a model might contribute value as an automated screening tool to triage patients for confirmatory testing, potentially reducing the number of tested patients, thereby resulting in substantial test-related labour and cost savings. FUNDING Stanford Cancer Institute and Stanford Departments of Pathology and Biomedical Data Science.
Collapse
Affiliation(s)
- Rikiya Yamashita
- Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, CA, USA; Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Stanford, CA, USA
| | - Jin Long
- Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Stanford, CA, USA
| | - Teri Longacre
- Department of Pathology, Stanford University School of Medicine, Stanford, CA, USA
| | - Lan Peng
- Department of Pathology, University of Texas, Southwestern Medical Center, Dallas, TX, USA
| | - Gerald Berry
- Department of Pathology, Stanford University School of Medicine, Stanford, CA, USA
| | - Brock Martin
- Department of Pathology, Stanford University School of Medicine, Stanford, CA, USA
| | - John Higgins
- Department of Pathology, Stanford University School of Medicine, Stanford, CA, USA
| | - Daniel L Rubin
- Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, CA, USA; Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Stanford, CA, USA
| | - Jeanne Shen
- Department of Pathology, Stanford University School of Medicine, Stanford, CA, USA; Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Stanford, CA, USA.
| |
Collapse
|
339
|
Hildebrand LA, Pierce CJ, Dennis M, Paracha M, Maoz A. Artificial Intelligence for Histology-Based Detection of Microsatellite Instability and Prediction of Response to Immunotherapy in Colorectal Cancer. Cancers (Basel) 2021; 13:391. [PMID: 33494280 PMCID: PMC7864494 DOI: 10.3390/cancers13030391] [Citation(s) in RCA: 44] [Impact Index Per Article: 14.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/25/2020] [Revised: 01/18/2021] [Accepted: 01/18/2021] [Indexed: 12/14/2022] Open
Abstract
Microsatellite instability (MSI) is a molecular marker of deficient DNA mismatch repair (dMMR) that is found in approximately 15% of colorectal cancer (CRC) patients. Testing all CRC patients for MSI/dMMR is recommended as screening for Lynch Syndrome and, more recently, to determine eligibility for immune checkpoint inhibitors in advanced disease. However, universal testing for MSI/dMMR has not been uniformly implemented because of cost and resource limitations. Artificial intelligence has been used to predict MSI/dMMR directly from hematoxylin and eosin (H&E) stained tissue slides. We review the emerging data regarding the utility of machine learning for MSI classification, focusing on CRC. We also provide the clinician with an introduction to image analysis with machine learning and convolutional neural networks. Machine learning can predict MSI/dMMR with high accuracy in high quality, curated datasets. Accuracy can be significantly decreased when applied to cohorts with different ethnic and/or clinical characteristics, or different tissue preparation protocols. Research is ongoing to determine the optimal machine learning methods for predicting MSI, which will need to be compared to current clinical practices, including next-generation sequencing. Predicting response to immunotherapy remains an unmet need.
Collapse
Affiliation(s)
- Lindsey A. Hildebrand
- Department of Medicine, Boston University School of Medicine and Boston Medical Center, Boston, MA 02118, USA; (L.A.H.); (C.J.P.); (M.D.); (M.P.)
| | - Colin J. Pierce
- Department of Medicine, Boston University School of Medicine and Boston Medical Center, Boston, MA 02118, USA; (L.A.H.); (C.J.P.); (M.D.); (M.P.)
| | - Michael Dennis
- Department of Medicine, Boston University School of Medicine and Boston Medical Center, Boston, MA 02118, USA; (L.A.H.); (C.J.P.); (M.D.); (M.P.)
- Division of Hematology Oncology, Department of Medicine, University of California San Diego, San Diego, CA 92093, USA
| | - Munizay Paracha
- Department of Medicine, Boston University School of Medicine and Boston Medical Center, Boston, MA 02118, USA; (L.A.H.); (C.J.P.); (M.D.); (M.P.)
| | - Asaf Maoz
- Department of Medicine, Boston University School of Medicine and Boston Medical Center, Boston, MA 02118, USA; (L.A.H.); (C.J.P.); (M.D.); (M.P.)
- Department of Medical Oncology, Dana-Farber Cancer Institute and Harvard Medical School, Boston, MA 02215, USA
| |
Collapse
|
340
|
Deep learning predicts postsurgical recurrence of hepatocellular carcinoma from digital histopathologic images. Sci Rep 2021; 11:2047. [PMID: 33479370 PMCID: PMC7820423 DOI: 10.1038/s41598-021-81506-y] [Citation(s) in RCA: 29] [Impact Index Per Article: 9.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/27/2020] [Accepted: 01/05/2021] [Indexed: 12/22/2022] Open
Abstract
Recurrence risk stratification of patients undergoing primary surgical resection for hepatocellular carcinoma (HCC) is an area of active investigation, and several staging systems have been proposed to optimize treatment strategies. However, as many as 70% of patients still experience tumor recurrence at 5 years post-surgery. We developed and validated a deep learning-based system (HCC-SurvNet) that provides risk scores for disease recurrence after primary resection, directly from hematoxylin and eosin-stained digital whole-slide images of formalin-fixed, paraffin embedded liver resections. Our model achieved concordance indices of 0.724 and 0.683 on the internal and external test cohorts, respectively, exceeding the performance of the standard Tumor-Node-Metastasis classification system. The model’s risk score stratified patients into low- and high-risk subgroups with statistically significant differences in their survival distributions, and was an independent risk factor for post-surgical recurrence in both test cohorts. Our results suggest that deep learning-based models can provide recurrence risk scores which may augment current patient stratification methods and help refine the clinical management of patients undergoing primary surgical resection for HCC.
Collapse
|
341
|
Kwak MS, Lee HH, Yang JM, Cha JM, Jeon JW, Yoon JY, Kim HI. Deep Convolutional Neural Network-Based Lymph Node Metastasis Prediction for Colon Cancer Using Histopathological Images. Front Oncol 2021; 10:619803. [PMID: 33520727 PMCID: PMC7838556 DOI: 10.3389/fonc.2020.619803] [Citation(s) in RCA: 27] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/21/2020] [Accepted: 11/30/2020] [Indexed: 12/24/2022] Open
Abstract
Background Human evaluation of pathological slides cannot accurately predict lymph node metastasis (LNM), although accurate prediction is essential to determine treatment and follow-up strategies for colon cancer. We aimed to develop accurate histopathological features for LNM in colon cancer. Methods We developed a deep convolutional neural network model to distinguish the cancer tissue component of colon cancer using data from the tissue bank of the National Center for Tumor Diseases and the pathology archive at the University Medical Center Mannheim, Germany. This model was applied to whole-slide pathological images of colon cancer patients from The Cancer Genome Atlas (TCGA). The predictive value of the peri-tumoral stroma (PTS) score for LNM was assessed. Results A total of 164 patients with stages I, II, and III colon cancer from TCGA were analyzed. The mean PTS score was 0.380 (± SD = 0.285), and significantly higher PTS scores were observed in patients in the LNM-positive group than those in the LNM-negative group (P < 0.001). In the univariate analyses, the PTS scores for the LNM-positive group were significantly higher than those for the LNM-negative group (P < 0.001). Further, the PTS scores in lymphatic invasion and any one of perineural, lymphatic, or venous invasion were significantly increased in the LNM-positive group (P < 0.001 and P < 0.001). Conclusion We established the PTS score, a simplified reproducible parameter, for predicting LNM in colon cancer using computer-based analysis that could be used to guide treatment decisions. These findings warrant further confirmation through large-scale prospective clinical trials.
Collapse
Affiliation(s)
- Min Seob Kwak
- Department of Internal Medicine, Kyung Hee University Hospital at Gangdong, Kyung Hee University College of Medicine, Seoul, South Korea
| | - Hun Hee Lee
- Department of Internal Medicine, Kyung Hee University Hospital at Gangdong, Kyung Hee University College of Medicine, Seoul, South Korea
| | - Jae Min Yang
- Department of Computer Science and Engineering, Konkuk University, Seoul, South Korea
| | - Jae Myung Cha
- Department of Internal Medicine, Kyung Hee University Hospital at Gangdong, Kyung Hee University College of Medicine, Seoul, South Korea
| | - Jung Won Jeon
- Department of Internal Medicine, Kyung Hee University Hospital at Gangdong, Kyung Hee University College of Medicine, Seoul, South Korea
| | - Jin Young Yoon
- Department of Internal Medicine, Kyung Hee University Hospital at Gangdong, Kyung Hee University College of Medicine, Seoul, South Korea
| | - Ha Il Kim
- Department of Internal Medicine, Kyung Hee University Hospital at Gangdong, Kyung Hee University College of Medicine, Seoul, South Korea
| |
Collapse
|
342
|
Liu X, Lu J, Zhang G, Han J, Zhou W, Chen H, Zhang H, Yang Z. A Machine Learning Approach Yields a Multiparameter Prognostic Marker in Liver Cancer. Cancer Immunol Res 2021; 9:337-347. [PMID: 33431375 DOI: 10.1158/2326-6066.cir-20-0616] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/23/2020] [Revised: 11/13/2020] [Accepted: 01/07/2021] [Indexed: 01/19/2023]
Abstract
A number of staging systems have been developed to predict clinical outcomes in hepatocellular carcinoma (HCC). However, no general consensus has been reached regarding the optimal model. New approaches such as machine learning (ML) strategies are powerful tools for incorporating risk factors from multiple platforms. We retrospectively reviewed the baseline information, including clinicopathologic characteristics, laboratory parameters, and peripheral immune features reflecting T-cell function, from three HCC cohorts. A gradient-boosting survival (GBS) classifier was trained with prognosis-related variables in the training dataset and validated in two independent cohorts. We constructed a 20-feature GBS model classifier incorporating one clinical feature, 14 laboratory parameters, and five T-cell function parameters obtained from peripheral blood mononuclear cells. The GBS model-derived risk scores demonstrated high concordance indexes (C-indexes): 0.844, 0.827, and 0.806 in the training set and validation sets 1 and 2, respectively. The GBS classifier could separate patients into high-, medium- and low-risk subgroups with respect to death in all datasets (P < 0.05 for all comparisons). A higher risk score was positively correlated with a higher clinical stage and the presence of portal vein tumor thrombus (PVTT). Subgroup analyses with respect to Child-Pugh class, Barcelona Clinic Liver Cancer stage, and PVTT status supported the prognostic relevance of the GBS-derived risk algorithm independent of the conventional tumor staging system. In summary, a multiparameter ML algorithm incorporating clinical characteristics, laboratory parameters, and peripheral immune signatures offers a different approach to identify patients with the greatest risk of HCC-related death.
Collapse
Affiliation(s)
- Xiaoli Liu
- Center for Integrative Medicine, Beijing Ditan Hospital, Capital Medical University, Beijing, P.R. China
| | - Jilin Lu
- Department of General Surgery, Huashan Hospital, Fudan University, Shanghai, P.R. China
| | - Guanxiong Zhang
- Genecast Precision Medicine Technology Institute, Beijing, P.R. China
| | - Junyan Han
- Institute of Infectious Diseases, Beijing Ditan Hospital, Capital Medical University, Beijing, P.R. China
| | - Wei Zhou
- Genecast Precision Medicine Technology Institute, Beijing, P.R. China
| | - Huan Chen
- Genecast Precision Medicine Technology Institute, Beijing, P.R. China.
| | - Henghui Zhang
- Institute of Infectious Diseases, Beijing Ditan Hospital, Capital Medical University, Beijing, P.R. China.
| | - Zhiyun Yang
- Center for Integrative Medicine, Beijing Ditan Hospital, Capital Medical University, Beijing, P.R. China.
| |
Collapse
|
343
|
Cai D, Duan X, Wang W, Huang ZP, Zhu Q, Zhong ME, Lv MY, Li CH, Kou WB, Wu XJ, Gao F. A Metabolism-Related Radiomics Signature for Predicting the Prognosis of Colorectal Cancer. Front Mol Biosci 2021; 7:613918. [PMID: 33490106 PMCID: PMC7817969 DOI: 10.3389/fmolb.2020.613918] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/04/2020] [Accepted: 12/07/2020] [Indexed: 01/05/2023] Open
Abstract
Background: Radiomics refers to the extraction of a large amount of image information from medical images, which can provide decision support for clinicians. In this study, we developed and validated a radiomics-based nomogram to predict the prognosis of colorectal cancer (CRC). Methods: A total of 381 patients with colorectal cancer (primary cohort: n = 242; validation cohort: n = 139) were enrolled and radiomic features were extracted from the vein phase of preoperative computed tomography (CT). The radiomics score was generated by using the least absolute shrinkage and selection operator algorithm (LASSO). A nomogram was constructed by combining the radiomics score with clinicopathological risk factors for predicting the prognosis of CRC patients. The performance of the nomogram was evaluated by the calibration curve, receiver operating characteristic (ROC) curve and C-index statistics. Functional analysis and correlation analysis were used to explore the underlying association between radiomic feature and the gene-expression patterns. Results: Five radiomic features were selected to calculate the radiomics score by using the LASSO regression model. The Kaplan-Meier analysis showed that radiomics score was significantly associated with disease-free survival (DFS) [primary cohort: hazard ratio (HR): 5.65, 95% CI: 2.26–14.13, P < 0.001; validation cohort: HR: 8.49, 95% CI: 2.05–35.17, P < 0.001]. Multivariable analysis confirmed the independent prognostic value of radiomics score (primary cohort: HR: 5.35, 95% CI: 2.14–13.39, P < 0.001; validation cohort: HR: 5.19, 95% CI: 1.22–22.00, P = 0.026). We incorporated radiomics signature with the TNM stage to build a nomogram, which performed better than TNM stage alone. The C-index of the nomogram achieved 0.74 (0.69–0.80) in the primary cohort and 0.82 (0.77–0.87) in the validation cohort. Functional analysis and correlation analysis found that the radiomic signatures were mainly associated with metabolism related pathways. Conclusions: The radiomics score derived from the preoperative CT image was an independent prognostic factor and could be a complement to the current staging strategies of colorectal cancer.
Collapse
Affiliation(s)
- Du Cai
- Department of Colorectal Surgery, The Sixth Affiliated Hospital of Sun Yat-sen University, Guangzhou, China.,Guangdong Provincial Key Laboratory of Colorectal and Pelvic Floor Diseases, Supported by National Key Clinical Discipline, Guangdong Institute of Gastroenterology, Guangzhou, China
| | - Xin Duan
- Department of Colorectal Surgery, The Sixth Affiliated Hospital of Sun Yat-sen University, Guangzhou, China.,Guangdong Provincial Key Laboratory of Colorectal and Pelvic Floor Diseases, Supported by National Key Clinical Discipline, Guangdong Institute of Gastroenterology, Guangzhou, China
| | - Wei Wang
- Department of Gynecology, Huzhou Maternity & Child Health Care Hospital, Huzhou, China
| | - Ze-Ping Huang
- Department of Colorectal Surgery, The Sixth Affiliated Hospital of Sun Yat-sen University, Guangzhou, China.,Guangdong Provincial Key Laboratory of Colorectal and Pelvic Floor Diseases, Supported by National Key Clinical Discipline, Guangdong Institute of Gastroenterology, Guangzhou, China
| | - Qiqi Zhu
- Department of Colorectal Surgery, The Sixth Affiliated Hospital of Sun Yat-sen University, Guangzhou, China.,Guangdong Provincial Key Laboratory of Colorectal and Pelvic Floor Diseases, Supported by National Key Clinical Discipline, Guangdong Institute of Gastroenterology, Guangzhou, China
| | - Min-Er Zhong
- Department of Colorectal Surgery, The Sixth Affiliated Hospital of Sun Yat-sen University, Guangzhou, China.,Guangdong Provincial Key Laboratory of Colorectal and Pelvic Floor Diseases, Supported by National Key Clinical Discipline, Guangdong Institute of Gastroenterology, Guangzhou, China
| | - Min-Yi Lv
- Department of Colorectal Surgery, The Sixth Affiliated Hospital of Sun Yat-sen University, Guangzhou, China.,Guangdong Provincial Key Laboratory of Colorectal and Pelvic Floor Diseases, Supported by National Key Clinical Discipline, Guangdong Institute of Gastroenterology, Guangzhou, China
| | - Cheng-Hang Li
- Department of Colorectal Surgery, The Sixth Affiliated Hospital of Sun Yat-sen University, Guangzhou, China.,Guangdong Provincial Key Laboratory of Colorectal and Pelvic Floor Diseases, Supported by National Key Clinical Discipline, Guangdong Institute of Gastroenterology, Guangzhou, China
| | - Wei-Bin Kou
- Department of Colorectal Surgery, The Sixth Affiliated Hospital of Sun Yat-sen University, Guangzhou, China.,Guangdong Provincial Key Laboratory of Colorectal and Pelvic Floor Diseases, Supported by National Key Clinical Discipline, Guangdong Institute of Gastroenterology, Guangzhou, China
| | - Xiao-Jian Wu
- Department of Colorectal Surgery, The Sixth Affiliated Hospital of Sun Yat-sen University, Guangzhou, China.,Guangdong Provincial Key Laboratory of Colorectal and Pelvic Floor Diseases, Supported by National Key Clinical Discipline, Guangdong Institute of Gastroenterology, Guangzhou, China
| | - Feng Gao
- Department of Colorectal Surgery, The Sixth Affiliated Hospital of Sun Yat-sen University, Guangzhou, China.,Guangdong Provincial Key Laboratory of Colorectal and Pelvic Floor Diseases, Supported by National Key Clinical Discipline, Guangdong Institute of Gastroenterology, Guangzhou, China
| |
Collapse
|
344
|
Robustness of convolutional neural networks in recognition of pigmented skin lesions. Eur J Cancer 2021; 145:81-91. [PMID: 33423009 DOI: 10.1016/j.ejca.2020.11.020] [Citation(s) in RCA: 25] [Impact Index Per Article: 8.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/05/2020] [Revised: 11/06/2020] [Accepted: 11/15/2020] [Indexed: 12/26/2022]
Abstract
BACKGROUND A basic requirement for artificial intelligence (AI)-based image analysis systems, which are to be integrated into clinical practice, is a high robustness. Minor changes in how those images are acquired, for example, during routine skin cancer screening, should not change the diagnosis of such assistance systems. OBJECTIVE To quantify to what extent minor image perturbations affect the convolutional neural network (CNN)-mediated skin lesion classification and to evaluate three possible solutions for this problem (additional data augmentation, test-time augmentation, anti-aliasing). METHODS We trained three commonly used CNN architectures to differentiate between dermoscopic melanoma and nevus images. Subsequently, their performance and susceptibility to minor changes ('brittleness') was tested on two distinct test sets with multiple images per lesion. For the first set, image changes, such as rotations or zooms, were generated artificially. The second set contained natural changes that stemmed from multiple photographs taken of the same lesions. RESULTS All architectures exhibited brittleness on the artificial and natural test set. The three reviewed methods were able to decrease brittleness to varying degrees while still maintaining performance. The observed improvement was greater for the artificial than for the natural test set, where enhancements were minor. CONCLUSIONS Minor image changes, relatively inconspicuous for humans, can have an effect on the robustness of CNNs differentiating skin lesions. By the methods tested here, this effect can be reduced, but not fully eliminated. Thus, further research to sustain the performance of AI classifiers is needed to facilitate the translation of such systems into the clinic.
Collapse
|
345
|
Srinidhi CL, Ciga O, Martel AL. Deep neural network models for computational histopathology: A survey. Med Image Anal 2021; 67:101813. [PMID: 33049577 PMCID: PMC7725956 DOI: 10.1016/j.media.2020.101813] [Citation(s) in RCA: 212] [Impact Index Per Article: 70.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/27/2019] [Revised: 05/12/2020] [Accepted: 08/09/2020] [Indexed: 12/14/2022]
Abstract
Histopathological images contain rich phenotypic information that can be used to monitor underlying mechanisms contributing to disease progression and patient survival outcomes. Recently, deep learning has become the mainstream methodological choice for analyzing and interpreting histology images. In this paper, we present a comprehensive review of state-of-the-art deep learning approaches that have been used in the context of histopathological image analysis. From the survey of over 130 papers, we review the field's progress based on the methodological aspect of different machine learning strategies such as supervised, weakly supervised, unsupervised, transfer learning and various other sub-variants of these methods. We also provide an overview of deep learning based survival models that are applicable for disease-specific prognosis tasks. Finally, we summarize several existing open datasets and highlight critical challenges and limitations with current deep learning approaches, along with possible avenues for future research.
Collapse
Affiliation(s)
- Chetan L Srinidhi
- Physical Sciences, Sunnybrook Research Institute, Toronto, Canada; Department of Medical Biophysics, University of Toronto, Canada.
| | - Ozan Ciga
- Department of Medical Biophysics, University of Toronto, Canada
| | - Anne L Martel
- Physical Sciences, Sunnybrook Research Institute, Toronto, Canada; Department of Medical Biophysics, University of Toronto, Canada
| |
Collapse
|
346
|
Warner E, Wang N, Lee J, Rao A. Meaningful incorporation of artificial intelligence for personalized patient management during cancer: Quantitative imaging, risk assessment, and therapeutic outcomes. Artif Intell Med 2021. [DOI: 10.1016/b978-0-12-821259-2.00017-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/28/2022]
|
347
|
Chen H, Sung JJY. Potentials of AI in medical image analysis in Gastroenterology and Hepatology. J Gastroenterol Hepatol 2021; 36:31-38. [PMID: 33140875 DOI: 10.1111/jgh.15327] [Citation(s) in RCA: 19] [Impact Index Per Article: 6.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/16/2020] [Revised: 10/30/2020] [Accepted: 10/30/2020] [Indexed: 12/15/2022]
Abstract
With the advancement of artificial intelligence (AI) technology, it comes in a big wave carrying possibly huge impact in the field of medicine. Gastroenterology and hepatology, being a specialty relying much on diagnostic imaging, endoscopy, and histopathology, AI technology has promised improving the quality and consistency of care to the patients. In this review, we will elucidate the development of machine learning methods, especially the visual representation mechanism in deep learning on recognition tasks. Various AI-image analysis applications in endoscopy, radiology, and pathology are covered in gastroenterology and hepatology and reveal the enormous potentials for AI in assisting diagnosis, prognosis, and treatment. We also discuss the promises as well as pitfalls for AI in medical image analysis and pointing out future research directions.
Collapse
Affiliation(s)
- Hao Chen
- Department of Computer Science and Engineering, The Chinese University of Hong Kong, Shatin, Hong Kong
| | - Joseph J Y Sung
- Department of Medicine and Therapeutics, The Chinese University of Hong Kong, Shatin, Hong Kong
| |
Collapse
|
348
|
Reducing annotation effort in digital pathology: A Co-Representation learning framework for classification tasks. Med Image Anal 2021; 67:101859. [DOI: 10.1016/j.media.2020.101859] [Citation(s) in RCA: 11] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/31/2019] [Revised: 09/14/2020] [Accepted: 09/25/2020] [Indexed: 01/07/2023]
|
349
|
Schömig-Markiefka B, Pryalukhin A, Hulla W, Bychkov A, Fukuoka J, Madabhushi A, Achter V, Nieroda L, Büttner R, Quaas A, Tolkach Y. Quality control stress test for deep learning-based diagnostic model in digital pathology. Mod Pathol 2021; 34:2098-2108. [PMID: 34168282 PMCID: PMC8592835 DOI: 10.1038/s41379-021-00859-x] [Citation(s) in RCA: 60] [Impact Index Per Article: 20.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/14/2021] [Revised: 06/06/2021] [Accepted: 06/06/2021] [Indexed: 11/23/2022]
Abstract
Digital pathology provides a possibility for computational analysis of histological slides and automatization of routine pathological tasks. Histological slides are very heterogeneous concerning staining, sections' thickness, and artifacts arising during tissue processing, cutting, staining, and digitization. In this study, we digitally reproduce major types of artifacts. Using six datasets from four different institutions digitized by different scanner systems, we systematically explore artifacts' influence on the accuracy of the pre-trained, validated, deep learning-based model for prostate cancer detection in histological slides. We provide evidence that any histological artifact dependent on severity can lead to a substantial loss in model performance. Strategies for the prevention of diagnostic model accuracy losses in the context of artifacts are warranted. Stress-testing of diagnostic models using synthetically generated artifacts might be an essential step during clinical validation of deep learning-based algorithms.
Collapse
Affiliation(s)
- Birgid Schömig-Markiefka
- grid.411097.a0000 0000 8852 305XInstitute of Pathology, University Hospital Cologne, Cologne, Germany
| | - Alexey Pryalukhin
- Institute of Pathology, Landesklinikum Wiener Neustadt, Wiener Neustadt, Austria
| | - Wolfgang Hulla
- Institute of Pathology, Landesklinikum Wiener Neustadt, Wiener Neustadt, Austria
| | - Andrey Bychkov
- grid.174567.60000 0000 8902 2273Department of Pathology, Nagasaki University Graduate School of Biomedical Sciences, Nagasaki, Japan ,grid.414927.d0000 0004 0378 2140Department of Pathology, Kameda Medical Center, Kamogawa, Japan
| | - Junya Fukuoka
- grid.174567.60000 0000 8902 2273Department of Pathology, Nagasaki University Graduate School of Biomedical Sciences, Nagasaki, Japan ,grid.414927.d0000 0004 0378 2140Department of Pathology, Kameda Medical Center, Kamogawa, Japan
| | - Anant Madabhushi
- grid.67105.350000 0001 2164 3847Department of Biomedical Engineering, Case Western Reserve University, Cleveland, OH USA ,grid.410349.b0000 0004 5912 6484Louis Stokes Cleveland Veterans Administration Medical Center, Cleveland, OH USA
| | - Viktor Achter
- grid.6190.e0000 0000 8580 3777Regional computing center (RRZK), University of Cologne, Cologne, Germany
| | - Lech Nieroda
- grid.6190.e0000 0000 8580 3777Regional computing center (RRZK), University of Cologne, Cologne, Germany
| | - Reinhard Büttner
- grid.411097.a0000 0000 8852 305XInstitute of Pathology, University Hospital Cologne, Cologne, Germany
| | - Alexander Quaas
- grid.411097.a0000 0000 8852 305XInstitute of Pathology, University Hospital Cologne, Cologne, Germany
| | - Yuri Tolkach
- Institute of Pathology, University Hospital Cologne, Cologne, Germany.
| |
Collapse
|
350
|
Singh R, Ahmed T, Kumar A, Singh AK, Pandey AK, Singh SK. Imbalanced Breast Cancer Classification Using Transfer Learning. IEEE/ACM TRANSACTIONS ON COMPUTATIONAL BIOLOGY AND BIOINFORMATICS 2021; 18:83-93. [PMID: 32175873 DOI: 10.1109/tcbb.2020.2980831] [Citation(s) in RCA: 16] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/10/2023]
Abstract
Accurate breast cancer detection using automated algorithms remains a problem within the literature. Although a plethora of work has tried to address this issue, an exact solution is yet to be found. This problem is further exacerbated by the fact that most of the existing datasets are imbalanced, i.e., the number of instances of a particular class far exceeds that of the others. In this paper, we propose a framework based on the notion of transfer learning to address this issue and focus our efforts on histopathological and imbalanced image classification. We use the popular VGG-19 as the base model and complement it with several state-of-the-art techniques to improve the overall performance of the system. With the ImageNet dataset taken as the source domain, we apply the learned knowledge in the target domain consisting of histopathological images. With experimentation performed on a large-scale dataset consisting of 277,524 images, we show that the framework proposed in this paper gives superior performance than those available in the existing literature. Through numerical simulations conducted on a supercomputer, we also present guidelines for work in transfer learning and imbalanced image classification.
Collapse
|