101
|
Fu X, Sahai E, Wilkins A. Application of digital pathology-based advanced analytics of tumour microenvironment organisation to predict prognosis and therapeutic response. J Pathol 2023; 260:578-591. [PMID: 37551703 PMCID: PMC10952145 DOI: 10.1002/path.6153] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/10/2023] [Revised: 05/25/2023] [Accepted: 06/07/2023] [Indexed: 08/09/2023]
Abstract
In recent years, the application of advanced analytics, especially artificial intelligence (AI), to digital H&E images, and other histological image types, has begun to radically change how histological images are used in the clinic. Alongside the recognition that the tumour microenvironment (TME) has a profound impact on tumour phenotype, the technical development of highly multiplexed immunofluorescence platforms has enhanced the biological complexity that can be captured in the TME with high precision. AI has an increasingly powerful role in the recognition and quantitation of image features and the association of such features with clinically important outcomes, as occurs in distinct stages in conventional machine learning. Deep-learning algorithms are able to elucidate TME patterns inherent in the input data with minimum levels of human intelligence and, hence, have the potential to achieve clinically relevant predictions and discovery of important TME features. Furthermore, the diverse repertoire of deep-learning algorithms able to interrogate TME patterns extends beyond convolutional neural networks to include attention-based models, graph neural networks, and multimodal models. To date, AI models have largely been evaluated retrospectively, outside the well-established rigour of prospective clinical trials, in part because traditional clinical trial methodology may not always be suitable for the assessment of AI technology. However, to enable digital pathology-based advanced analytics to meaningfully impact clinical care, specific measures of 'added benefit' to the current standard of care and validation in a prospective setting are important. This will need to be accompanied by adequate measures of explainability and interpretability. Despite such challenges, the combination of expanding datasets, increased computational power, and the possibility of integration of pre-clinical experimental insights into model development means there is exciting potential for the future progress of these AI applications. © 2023 The Authors. The Journal of Pathology published by John Wiley & Sons Ltd on behalf of The Pathological Society of Great Britain and Ireland.
Collapse
Affiliation(s)
- Xiao Fu
- Tumour Cell Biology LaboratoryThe Francis Crick InstituteLondonUK
- Biomolecular Modelling LaboratoryThe Francis Crick InstituteLondonUK
| | - Erik Sahai
- Tumour Cell Biology LaboratoryThe Francis Crick InstituteLondonUK
| | - Anna Wilkins
- Tumour Cell Biology LaboratoryThe Francis Crick InstituteLondonUK
- Division of Radiotherapy and ImagingInstitute of Cancer ResearchLondonUK
- Royal Marsden Hospitals NHS TrustLondonUK
| |
Collapse
|
102
|
Reyes-Aldasoro CC. Modelling the Tumour Microenvironment, but What Exactly Do We Mean by "Model"? Cancers (Basel) 2023; 15:3796. [PMID: 37568612 PMCID: PMC10416922 DOI: 10.3390/cancers15153796] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/28/2023] [Revised: 07/19/2023] [Accepted: 07/25/2023] [Indexed: 08/13/2023] Open
Abstract
The Oxford English Dictionary includes 17 definitions for the word "model" as a noun and another 11 as a verb. Therefore, context is necessary to understand the meaning of the word model. For instance, "model railways" refer to replicas of railways and trains at a smaller scale and a "model student" refers to an exemplary individual. In some cases, a specific context, like cancer research, may not be sufficient to provide one specific meaning for model. Even if the context is narrowed, specifically, to research related to the tumour microenvironment, "model" can be understood in a wide variety of ways, from an animal model to a mathematical expression. This paper presents a review of different "models" of the tumour microenvironment, as grouped by different definitions of the word into four categories: model organisms, in vitro models, mathematical models and computational models. Then, the frequencies of different meanings of the word "model" related to the tumour microenvironment are measured from numbers of entries in the MEDLINE database of the United States National Library of Medicine at the National Institutes of Health. The frequencies of the main components of the microenvironment and the organ-related cancers modelled are also assessed quantitatively with specific keywords. Whilst animal models, particularly xenografts and mouse models, are the most commonly used "models", the number of these entries has been slowly decreasing. Mathematical models, as well as prognostic and risk models, follow in frequency, and these have been growing in use.
Collapse
|
103
|
Wu Y, Li Y, Xiong X, Liu X, Lin B, Xu B. Recent advances of pathomics in colorectal cancer diagnosis and prognosis. Front Oncol 2023; 13:1094869. [PMID: 37538112 PMCID: PMC10396402 DOI: 10.3389/fonc.2023.1094869] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/10/2022] [Accepted: 06/13/2023] [Indexed: 08/05/2023] Open
Abstract
Colorectal cancer (CRC) is one of the most common malignancies, with the third highest incidence and the second highest mortality in the world. To improve the therapeutic outcome, the risk stratification and prognosis predictions would help guide clinical treatment decisions. Achieving these goals have been facilitated by the fast development of artificial intelligence (AI) -based algorithms using radiological and pathological data, in combination with genomic information. Among them, features extracted from pathological images, termed pathomics, are able to reflect sub-visual characteristics linking to better stratification and prediction of therapeutic responses. In this paper, we review recent advances in pathological image-based algorithms in CRC, focusing on diagnosis of benign and malignant lesions, micro-satellite instability, as well as prediction of neoadjuvant chemoradiotherapy and the prognosis of CRC patients.
Collapse
Affiliation(s)
- Yihan Wu
- School of Medicine, Chongqing University, Chongqing, China
- Chongqing Key Laboratory of Intelligent Oncology for Breast Cancer, Chongqing University Cancer Hospital, Chongqing, China
| | - Yi Li
- Chongqing Key Laboratory of Intelligent Oncology for Breast Cancer, Chongqing University Cancer Hospital, Chongqing, China
- Bioengineering College, Chongqing University, Chongqing, China
| | - Xiaomin Xiong
- Chongqing Key Laboratory of Intelligent Oncology for Breast Cancer, Chongqing University Cancer Hospital, Chongqing, China
- Bioengineering College, Chongqing University, Chongqing, China
| | - Xiaohua Liu
- Bioengineering College, Chongqing University, Chongqing, China
| | - Bo Lin
- Chongqing Key Laboratory of Intelligent Oncology for Breast Cancer, Chongqing University Cancer Hospital, Chongqing, China
| | - Bo Xu
- School of Medicine, Chongqing University, Chongqing, China
- Chongqing Key Laboratory of Intelligent Oncology for Breast Cancer, Chongqing University Cancer Hospital, Chongqing, China
| |
Collapse
|
104
|
Lin JR, Chen YA, Campton D, Cooper J, Coy S, Yapp C, Tefft JB, McCarty E, Ligon KL, Rodig SJ, Reese S, George T, Santagata S, Sorger PK. High-plex immunofluorescence imaging and traditional histology of the same tissue section for discovering image-based biomarkers. NATURE CANCER 2023; 4:1036-1052. [PMID: 37349501 PMCID: PMC10368530 DOI: 10.1038/s43018-023-00576-1] [Citation(s) in RCA: 26] [Impact Index Per Article: 26.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/07/2022] [Accepted: 05/08/2023] [Indexed: 06/24/2023]
Abstract
Precision medicine is critically dependent on better methods for diagnosing and staging disease and predicting drug response. Histopathology using hematoxylin and eosin (H&E)-stained tissue (not genomics) remains the primary diagnostic method in cancer. Recently developed highly multiplexed tissue imaging methods promise to enhance research studies and clinical practice with precise, spatially resolved single-cell data. Here, we describe the 'Orion' platform for collecting H&E and high-plex immunofluorescence images from the same cells in a whole-slide format suitable for diagnosis. Using a retrospective cohort of 74 colorectal cancer resections, we show that immunofluorescence and H&E images provide human experts and machine learning algorithms with complementary information that can be used to generate interpretable, multiplexed image-based models predictive of progression-free survival. Combining models of immune infiltration and tumor-intrinsic features achieves a 10- to 20-fold discrimination between rapid and slow (or no) progression, demonstrating the ability of multimodal tissue imaging to generate high-performance biomarkers.
Collapse
Affiliation(s)
- Jia-Ren Lin
- Laboratory of Systems Pharmacology, Department of Systems Biology, Harvard Medical School, Boston, MA, USA
- Ludwig Center at Harvard, Harvard Medical School, Boston, MA, USA
| | - Yu-An Chen
- Laboratory of Systems Pharmacology, Department of Systems Biology, Harvard Medical School, Boston, MA, USA
- Ludwig Center at Harvard, Harvard Medical School, Boston, MA, USA
| | | | | | - Shannon Coy
- Laboratory of Systems Pharmacology, Department of Systems Biology, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
| | - Clarence Yapp
- Laboratory of Systems Pharmacology, Department of Systems Biology, Harvard Medical School, Boston, MA, USA
- Ludwig Center at Harvard, Harvard Medical School, Boston, MA, USA
| | - Juliann B Tefft
- Laboratory of Systems Pharmacology, Department of Systems Biology, Harvard Medical School, Boston, MA, USA
- Ludwig Center at Harvard, Harvard Medical School, Boston, MA, USA
| | | | - Keith L Ligon
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
| | - Scott J Rodig
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
| | | | | | - Sandro Santagata
- Laboratory of Systems Pharmacology, Department of Systems Biology, Harvard Medical School, Boston, MA, USA.
- Ludwig Center at Harvard, Harvard Medical School, Boston, MA, USA.
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA.
| | - Peter K Sorger
- Laboratory of Systems Pharmacology, Department of Systems Biology, Harvard Medical School, Boston, MA, USA.
- Ludwig Center at Harvard, Harvard Medical School, Boston, MA, USA.
| |
Collapse
|
105
|
Gui CP, Chen YH, Zhao HW, Cao JZ, Liu TJ, Xiong SW, Yu YF, Liao B, Cao Y, Li JY, Huang KB, Han H, Zhang ZL, Chen WF, Jiang ZY, Gao Y, Han GP, Tang Q, Ouyang K, Qu GM, Wu JT, Guo JP, Li CX, Li PX, Liu ZP, Hsieh JT, Cai MY, Li XS, Wei JH, Luo JH. Multimodal recurrence scoring system for prediction of clear cell renal cell carcinoma outcome: a discovery and validation study. Lancet Digit Health 2023:S2589-7500(23)00095-X. [PMID: 37393162 DOI: 10.1016/s2589-7500(23)00095-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/21/2022] [Revised: 04/02/2023] [Accepted: 05/03/2023] [Indexed: 07/03/2023]
Abstract
BACKGROUND Improved markers for predicting recurrence are needed to stratify patients with localised (stage I-III) renal cell carcinoma after surgery for selection of adjuvant therapy. We developed a novel assay integrating three modalities-clinical, genomic, and histopathological-to improve the predictive accuracy for localised renal cell carcinoma recurrence. METHODS In this retrospective analysis and validation study, we developed a histopathological whole-slide image (WSI)-based score using deep learning allied to digital scanning of conventional haematoxylin and eosin-stained tumour tissue sections, to predict tumour recurrence in a development dataset of 651 patients with distinctly good or poor disease outcome. The six single nucleotide polymorphism-based score, which was detected in paraffin-embedded tumour tissue samples, and the Leibovich score, which was established using clinicopathological risk factors, were combined with the WSI-based score to construct a multimodal recurrence score in the training dataset of 1125 patients. The multimodal recurrence score was validated in 1625 patients from the independent validation dataset and 418 patients from The Cancer Genome Atlas set. The primary outcome measured was the recurrence-free interval (RFI). FINDINGS The multimodal recurrence score had significantly higher predictive accuracy than the three single-modal scores and clinicopathological risk factors, and it precisely predicted the RFI of patients in the training and two validation datasets (areas under the curve at 5 years: 0·825-0·876 vs 0·608-0·793; p<0·05). The RFI of patients with low stage or grade is usually better than that of patients with high stage or grade; however, the RFI in the multimodal recurrence score-defined high-risk stage I and II group was shorter than in the low-risk stage III group (hazard ratio [HR] 4·57, 95% CI 2·49-8·40; p<0·0001), and the RFI of the high-risk grade 1 and 2 group was shorter than in the low-risk grade 3 and 4 group (HR 4·58, 3·19-6·59; p<0·0001). INTERPRETATION Our multimodal recurrence score is a practical and reliable predictor that can add value to the current staging system for predicting localised renal cell carcinoma recurrence after surgery, and this combined approach more precisely informs treatment decisions about adjuvant therapy. FUNDING National Natural Science Foundation of China, and National Key Research and Development Program of China.
Collapse
Affiliation(s)
- Cheng-Peng Gui
- Department of Urology, First Affiliated Hospital, Sun Yat-sen University, Guangzhou, China
| | - Yu-Hang Chen
- Department of Urology, First Affiliated Hospital, Sun Yat-sen University, Guangzhou, China
| | - Hong-Wei Zhao
- Department of Urology, Affiliated Yantai Yuhuangding Hospital, Qingdao University, Yantai, China
| | - Jia-Zheng Cao
- Department of Urology, Jiangmen Central Hospital, Jiangmen, China
| | - Tian-Jie Liu
- Department of Urology, The First Affiliated Hospital of Xi'an Jiaotong University, Xi'an, China
| | - Sheng-Wei Xiong
- Department of Urology, Peking University First Hospital, Institute of Urology, Peking University, National Urological Cancer Center, Beijing, China
| | - Yan-Fei Yu
- Department of Urology, Peking University First Hospital, Institute of Urology, Peking University, National Urological Cancer Center, Beijing, China
| | - Bing Liao
- Department of Pathology, First Affiliated Hospital, Sun Yat-sen University, Guangzhou, China
| | - Yun Cao
- Department of Pathology, Cancer Center, Sun Yat-sen University, Guangzhou, China
| | - Jia-Ying Li
- Department of Urology, First Affiliated Hospital, Sun Yat-sen University, Guangzhou, China
| | - Kang-Bo Huang
- Department of Urology, Cancer Center, Sun Yat-sen University, Guangzhou, China
| | - Hui Han
- Department of Urology, Cancer Center, Sun Yat-sen University, Guangzhou, China
| | - Zhi-Ling Zhang
- Department of Urology, Cancer Center, Sun Yat-sen University, Guangzhou, China
| | - Wen-Fang Chen
- Department of Pathology, First Affiliated Hospital, Sun Yat-sen University, Guangzhou, China
| | - Ze-Ying Jiang
- Department of Pathology, First Affiliated Hospital, Sun Yat-sen University, Guangzhou, China
| | - Ye Gao
- Department of Urology, The First Affiliated Hospital of Xi'an Jiaotong University, Xi'an, China
| | - Guan-Peng Han
- Department of Urology, Peking University First Hospital, Institute of Urology, Peking University, National Urological Cancer Center, Beijing, China
| | - Qi Tang
- Department of Urology, Peking University First Hospital, Institute of Urology, Peking University, National Urological Cancer Center, Beijing, China
| | - Kui Ouyang
- Department of Urology, Affiliated Yantai Yuhuangding Hospital, Qingdao University, Yantai, China
| | - Gui-Mei Qu
- Department of Pathology, Affiliated Yantai Yuhuangding Hospital, Qingdao University, Yantai, China
| | - Ji-Tao Wu
- Department of Urology, Affiliated Yantai Yuhuangding Hospital, Qingdao University, Yantai, China
| | - Jian-Ping Guo
- Institute of Precision Medicine, First Affiliated Hospital, Sun Yat-sen University, Guangzhou, China
| | - Cai-Xia Li
- School of Mathematics and Computational Science, Sun Yat-sen University, Guangzhou, China
| | - Pei-Xing Li
- School of Mathematics and Computational Science, Sun Yat-sen University, Guangzhou, China
| | - Zhi-Ping Liu
- Department of Internal Medicine and Department of Molecular Biology, University of Texas Southwestern Medical Center at Dallas, Dallas TX, USA
| | - Jer-Tsong Hsieh
- Department of Urology, University of Texas Southwestern Medical Center at Dallas, Dallas TX, USA
| | - Mu-Yan Cai
- Department of Pathology, Cancer Center, Sun Yat-sen University, Guangzhou, China
| | - Xue-Song Li
- Department of Urology, Peking University First Hospital, Institute of Urology, Peking University, National Urological Cancer Center, Beijing, China.
| | - Jin-Huan Wei
- Department of Urology, First Affiliated Hospital, Sun Yat-sen University, Guangzhou, China.
| | - Jun-Hang Luo
- Department of Urology, First Affiliated Hospital, Sun Yat-sen University, Guangzhou, China; Institute of Precision Medicine, First Affiliated Hospital, Sun Yat-sen University, Guangzhou, China.
| |
Collapse
|
106
|
Wu Y, Liu X, Liu F, Li Y, Xiong X, Sun H, Lin B, Li Y, Xu B. A multi-class classification algorithm based on hematoxylin-eosin staining for neoadjuvant therapy in rectal cancer: a retrospective study. PeerJ 2023; 11:e15408. [PMID: 37334122 PMCID: PMC10269576 DOI: 10.7717/peerj.15408] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/27/2022] [Accepted: 04/21/2023] [Indexed: 06/20/2023] Open
Abstract
Neoadjuvant therapy (NAT) is a major treatment option for locally advanced rectal cancer. With recent advancement of machine/deep learning algorithms, predicting the treatment response of NAT has become possible using radiological and/or pathological images. However, programs reported thus far are limited to binary classifications, and they can only distinguish the pathological complete response (pCR). In the clinical setting, the pathological NAT responses are classified as four classes: (TRG0-3), with 0 as pCR, 1 as moderate response, 2 as minimal response and 3 as poor response. Therefore, the actual clinical need for risk stratification remains unmet. By using ResNet (Residual Neural Network), we developed a multi-class classifier based on Hematoxylin-Eosin (HE) images to divide the response to three groups (TRG0, TRG1/2, and TRG3). Overall, the model achieved the AUC 0.97 at 40× magnification and AUC 0.89 at 10× magnification. For TRG0, the model under 40× magnification achieved a precision of 0.67, a sensitivity of 0.67, and a specificity of 0.95. For TRG1/2, a precision of 0.92, a sensitivity of 0.86, and a specificity of 0.89 were achieved. For TRG3, the model obtained a precision of 0.71, a sensitivity of 0.83, and a specificity of 0.88. To find the relationship between the treatment response and pathological images, we constructed a visual heat map of tiles using Class Activation Mapping (CAM). Notably, we found that tumor nuclei and tumor-infiltrating lymphocytes appeared to be potential features of the algorithm. Taken together, this multi-class classifier represents the first of its kind to predict different NAT responses in rectal cancer.
Collapse
Affiliation(s)
- Yihan Wu
- School of Medicine, Chongqing University, Chongqing, China
- Chongqing Key Laboratory of Intelligent Oncology for Breast Cancer, Chongqing University Cancer Hospital, Chongqing, China
| | - Xiaohua Liu
- Bioengineering College of Chongqing University, Chongqing, China
| | - Fang Liu
- Department of Pathology, Chongqing University Cancer Hospital, School of Medicine, Chongqing University, Chongqing, China
| | - Yi Li
- Chongqing Key Laboratory of Intelligent Oncology for Breast Cancer, Chongqing University Cancer Hospital, Chongqing, China
- Bioengineering College of Chongqing University, Chongqing, China
| | - Xiaomin Xiong
- Chongqing Key Laboratory of Intelligent Oncology for Breast Cancer, Chongqing University Cancer Hospital, Chongqing, China
- Bioengineering College of Chongqing University, Chongqing, China
| | - Hao Sun
- Department of Gastrointestinal Cancer Center, Chongqing University Cancer Hospital, Chongqing, China
| | - Bo Lin
- Chongqing Key Laboratory of Intelligent Oncology for Breast Cancer, Chongqing University Cancer Hospital, Chongqing, China
| | - Yu Li
- Department of Pathology, Chongqing University Cancer Hospital, School of Medicine, Chongqing University, Chongqing, China
| | - Bo Xu
- School of Medicine, Chongqing University, Chongqing, China
- Chongqing Key Laboratory of Intelligent Oncology for Breast Cancer, Chongqing University Cancer Hospital, Chongqing, China
| |
Collapse
|
107
|
Ke J, Shen Y, Lu Y, Guo Y, Shen D. Mine local homogeneous representation by interaction information clustering with unsupervised learning in histopathology images. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2023; 235:107520. [PMID: 37031665 DOI: 10.1016/j.cmpb.2023.107520] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/10/2022] [Revised: 03/13/2023] [Accepted: 03/28/2023] [Indexed: 05/08/2023]
Abstract
BACKGROUND AND OBJECTIVE The success of data-driven deep learning for histopathology images often depends on high-quality training sets and fine-grained annotations. However, as tumors are heterogeneous and annotations are expensive, unsupervised learning approaches are desirable to obtain full automation. METHODS In this paper, an Interaction Information Clustering (IIC) method is proposed to extract locally homogeneous features in mutually exclusive clusters. Trained in an unsupervised paradigm, the framework learns invariant information from multiple spatially adjacent regions for improved classification. Additionally, an adaptive Conditional Random Field (CRF) model is incorporated to detect spatially adjacent image patches of high morphological homogeneity in an offset-constraint free manner. RESULTS Empirically, the proposed model achieves an observable improvement of 11.4% on the downstream patch-level classification accuracy, compared with state-of-the-art unsupervised learning approaches. CONCLUSION Furthermore, evaluated with our clinically collected histopathology whole-slide images, the proposed model shows high consistency in tissue distribution compared with well-trained supervised learning, which is of important diagnostic significance in clinical practice.
Collapse
Affiliation(s)
- Jing Ke
- School of Electronic Information and Electrical Engineering, Shanghai 200240, China.
| | - Yiqing Shen
- Department of Computer Science, Johns Hopkins University, Baltimore, MD 21218, USA.
| | - Yizhou Lu
- Shanghai Advanced Research Institute, Chinese Academy of Sciences, Shanghai 201210, China
| | - Yi Guo
- School of Computer, Data and Mathematical Sciences, Western Sydney University, Penrith, NSW 2751, Australia
| | - Dinggang Shen
- School of Biomedical Engineering, ShanghaiTech University, Shanghai 201210, China; Shanghai United Imaging Intelligence Co., Ltd., Shanghai 200230, China; Shanghai Clinical Research and Trial Center, Shanghai, 201210, China
| |
Collapse
|
108
|
Stanciu SG, König K, Song YM, Wolf L, Charitidis CA, Bianchini P, Goetz M. Toward next-generation endoscopes integrating biomimetic video systems, nonlinear optical microscopy, and deep learning. BIOPHYSICS REVIEWS 2023; 4:021307. [PMID: 38510341 PMCID: PMC10903409 DOI: 10.1063/5.0133027] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/15/2022] [Accepted: 05/26/2023] [Indexed: 03/22/2024]
Abstract
According to the World Health Organization, the proportion of the world's population over 60 years will approximately double by 2050. This progressive increase in the elderly population will lead to a dramatic growth of age-related diseases, resulting in tremendous pressure on the sustainability of healthcare systems globally. In this context, finding more efficient ways to address cancers, a set of diseases whose incidence is correlated with age, is of utmost importance. Prevention of cancers to decrease morbidity relies on the identification of precursor lesions before the onset of the disease, or at least diagnosis at an early stage. In this article, after briefly discussing some of the most prominent endoscopic approaches for gastric cancer diagnostics, we review relevant progress in three emerging technologies that have significant potential to play pivotal roles in next-generation endoscopy systems: biomimetic vision (with special focus on compound eye cameras), non-linear optical microscopies, and Deep Learning. Such systems are urgently needed to enhance the three major steps required for the successful diagnostics of gastrointestinal cancers: detection, characterization, and confirmation of suspicious lesions. In the final part, we discuss challenges that lie en route to translating these technologies to next-generation endoscopes that could enhance gastrointestinal imaging, and depict a possible configuration of a system capable of (i) biomimetic endoscopic vision enabling easier detection of lesions, (ii) label-free in vivo tissue characterization, and (iii) intelligently automated gastrointestinal cancer diagnostic.
Collapse
Affiliation(s)
- Stefan G. Stanciu
- Center for Microscopy-Microanalysis and Information Processing, University Politehnica of Bucharest, Bucharest, Romania
| | | | | | - Lior Wolf
- School of Computer Science, Tel Aviv University, Tel-Aviv, Israel
| | - Costas A. Charitidis
- Research Lab of Advanced, Composite, Nano-Materials and Nanotechnology, School of Chemical Engineering, National Technical University of Athens, Athens, Greece
| | - Paolo Bianchini
- Nanoscopy and NIC@IIT, Italian Institute of Technology, Genoa, Italy
| | - Martin Goetz
- Medizinische Klinik IV-Gastroenterologie/Onkologie, Kliniken Böblingen, Klinikverbund Südwest, Böblingen, Germany
| |
Collapse
|
109
|
Altini N, Marvulli TM, Zito FA, Caputo M, Tommasi S, Azzariti A, Brunetti A, Prencipe B, Mattioli E, De Summa S, Bevilacqua V. The role of unpaired image-to-image translation for stain color normalization in colorectal cancer histology classification. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2023; 234:107511. [PMID: 37011426 DOI: 10.1016/j.cmpb.2023.107511] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/30/2022] [Revised: 03/14/2023] [Accepted: 03/25/2023] [Indexed: 06/19/2023]
Abstract
BACKGROUND Histological assessment of colorectal cancer (CRC) tissue is a crucial and demanding task for pathologists. Unfortunately, manual annotation by trained specialists is a burdensome operation, which suffers from problems like intra- and inter-pathologist variability. Computational models are revolutionizing the Digital Pathology field, offering reliable and fast approaches for challenges like tissue segmentation and classification. With this respect, an important obstacle to overcome consists in stain color variations among different laboratories, which can decrease the performance of classifiers. In this work, we investigated the role of Unpaired Image-to-Image Translation (UI2IT) models for stain color normalization in CRC histology and compared to classical normalization techniques for Hematoxylin-Eosin (H&E) images. METHODS Five Deep Learning normalization models based on Generative Adversarial Networks (GANs) belonging to the UI2IT paradigm have been thoroughly compared to realize a robust stain color normalization pipeline. To avoid the need for training a style transfer GAN between each pair of data domains, in this paper we introduce the concept of training by exploiting a meta-domain, which contains data coming from a wide variety of laboratories. The proposed framework enables a huge saving in terms of training time, by allowing to train a single image normalization model for a target laboratory. To prove the applicability of the proposed workflow in the clinical practice, we conceived a novel perceptive quality measure, which we defined as Pathologist Perceptive Quality (PPQ). The second stage involved the classification of tissue types in CRC histology, where deep features extracted from Convolutional Neural Networks have been exploited to realize a Computer-Aided Diagnosis system based on a Support Vector Machine (SVM). To prove the reliability of the system on new data, an external validation set composed of N = 15,857 tiles has been collected at IRCCS Istituto Tumori "Giovanni Paolo II". RESULTS The exploitation of a meta-domain consented to train normalization models that allowed achieving better classification results than normalization models explicitly trained on the source domain. PPQ metric has been found correlated to quality of distributions (Fréchet Inception Distance - FID) and to similarity of the transformed image to the original one (Learned Perceptual Image Patch Similarity - LPIPS), thus showing that GAN quality measures introduced in natural image processing tasks can be linked to pathologist evaluation of H&E images. Furthermore, FID has been found correlated to accuracies of the downstream classifiers. The SVM trained on DenseNet201 features allowed to obtain the highest classification results in all configurations. The normalization method based on the fast variant of CUT (Contrastive Unpaired Translation), FastCUT, trained with the meta-domain paradigm, allowed to achieve the best classification result for the downstream task and, correspondingly, showed the highest FID on the classification dataset. CONCLUSIONS Stain color normalization is a difficult but fundamental problem in the histopathological setting. Several measures should be considered for properly assessing normalization methods, so that they can be introduced in the clinical practice. UI2IT frameworks offer a powerful and effective way to perform the normalization process, providing realistic images with proper colorization, unlike traditional normalization methods that introduce color artifacts. By adopting the proposed meta-domain framework, the training time can be reduced, and the accuracy of downstream classifiers can be increased.
Collapse
Affiliation(s)
- Nicola Altini
- Department of Electrical and Information Engineering (DEI), Polytechnic University of Bari, Via Edoardo Orabona, 4, Bari 70126, Italy.
| | - Tommaso Maria Marvulli
- Laboratory of Experimental Pharmacology, IRCCS Istituto Tumori "Giovanni Paolo II", Via O. Flacco, 65, Bari 70124, Italy
| | - Francesco Alfredo Zito
- Pathology Department, IRCCS Istituto Tumori "Giovanni Paolo II", Via O. Flacco, 65, Bari 70124, Italy
| | - Mariapia Caputo
- Molecular Diagnostics and Pharmacogenetics Unit, IRCCS Istituto Tumori "Giovanni Paolo II", Via O. Flacco, 65, Bari 70124, Italy
| | - Stefania Tommasi
- Molecular Diagnostics and Pharmacogenetics Unit, IRCCS Istituto Tumori "Giovanni Paolo II", Via O. Flacco, 65, Bari 70124, Italy
| | - Amalia Azzariti
- Laboratory of Experimental Pharmacology, IRCCS Istituto Tumori "Giovanni Paolo II", Via O. Flacco, 65, Bari 70124, Italy
| | - Antonio Brunetti
- Department of Electrical and Information Engineering (DEI), Polytechnic University of Bari, Via Edoardo Orabona, 4, Bari 70126, Italy; Apulian Bioengineering srl, Via delle Violette, 14, Modugno 70026, Italy
| | - Berardino Prencipe
- Department of Electrical and Information Engineering (DEI), Polytechnic University of Bari, Via Edoardo Orabona, 4, Bari 70126, Italy
| | - Eliseo Mattioli
- Pathology Department, IRCCS Istituto Tumori "Giovanni Paolo II", Via O. Flacco, 65, Bari 70124, Italy
| | - Simona De Summa
- Molecular Diagnostics and Pharmacogenetics Unit, IRCCS Istituto Tumori "Giovanni Paolo II", Via O. Flacco, 65, Bari 70124, Italy
| | - Vitoantonio Bevilacqua
- Department of Electrical and Information Engineering (DEI), Polytechnic University of Bari, Via Edoardo Orabona, 4, Bari 70126, Italy; Apulian Bioengineering srl, Via delle Violette, 14, Modugno 70026, Italy
| |
Collapse
|
110
|
Khazaee Fadafen M, Rezaee K. Ensemble-based multi-tissue classification approach of colorectal cancer histology images using a novel hybrid deep learning framework. Sci Rep 2023; 13:8823. [PMID: 37258631 DOI: 10.1038/s41598-023-35431-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/04/2022] [Accepted: 05/17/2023] [Indexed: 06/02/2023] Open
Abstract
Colorectal cancer (CRC) is the second leading cause of cancer death in the world, so digital pathology is essential for assessing prognosis. Due to the increasing resolution and quantity of whole slide images (WSIs), as well as the lack of annotated information, previous methodologies cannot be generalized as effective decision-making systems. Since deep learning (DL) methods can handle large-scale applications, they can provide a viable alternative to histopathology image (HI) analysis. DL architectures, however, may not be sufficient to classify CRC tissues based on anatomical histopathology data. A dilated ResNet (dResNet) structure and attention module are used to generate deep feature maps in order to classify multiple tissues in HIs. In addition, neighborhood component analysis (NCA) overcomes the constraint of computational complexity. Data is fed into a deep support vector machine (SVM) based on an ensemble learning algorithm called DeepSVM after the features have been selected. CRC-5000 and NCT-CRC-HE-100 K datasets were analyzed to validate and test the hybrid procedure. We demonstrate that the hybrid model achieves 98.75% and 99.76% accuracy on CRC datasets. The results showed that only pathologists' labels could successfully classify unseen WSIs. Furthermore, the hybrid deep learning method outperforms state-of-the-art approaches in terms of computational efficiency and time. Using the proposed mechanism for tissue analysis, it will be possible to correctly predict CRC based on accurate pathology image classification.
Collapse
Affiliation(s)
- Masoud Khazaee Fadafen
- Department of Electrical Engineering, Technical and Vocational University (TVU), Tehran, Iran
| | - Khosro Rezaee
- Department of Biomedical Engineering, Meybod University, Meybod, Iran.
| |
Collapse
|
111
|
Petäinen L, Väyrynen JP, Ruusuvuori P, Pölönen I, Äyrämö S, Kuopio T. Domain-specific transfer learning in the automated scoring of tumor-stroma ratio from histopathological images of colorectal cancer. PLoS One 2023; 18:e0286270. [PMID: 37235626 DOI: 10.1371/journal.pone.0286270] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/03/2023] [Accepted: 05/11/2023] [Indexed: 05/28/2023] Open
Abstract
Tumor-stroma ratio (TSR) is a prognostic factor for many types of solid tumors. In this study, we propose a method for automated estimation of TSR from histopathological images of colorectal cancer. The method is based on convolutional neural networks which were trained to classify colorectal cancer tissue in hematoxylin-eosin stained samples into three classes: stroma, tumor and other. The models were trained using a data set that consists of 1343 whole slide images. Three different training setups were applied with a transfer learning approach using domain-specific data i.e. an external colorectal cancer histopathological data set. The three most accurate models were chosen as a classifier, TSR values were predicted and the results were compared to a visual TSR estimation made by a pathologist. The results suggest that classification accuracy does not improve when domain-specific data are used in the pre-training of the convolutional neural network models in the task at hand. Classification accuracy for stroma, tumor and other reached 96.1% on an independent test set. Among the three classes the best model gained the highest accuracy (99.3%) for class tumor. When TSR was predicted with the best model, the correlation between the predicted values and values estimated by an experienced pathologist was 0.57. Further research is needed to study associations between computationally predicted TSR values and other clinicopathological factors of colorectal cancer and the overall survival of the patients.
Collapse
Affiliation(s)
- Liisa Petäinen
- Faculty of Information Technology, University of Jyväskylä, Jyväskylä, Finland
| | - Juha P Väyrynen
- Cancer and Translational Medicine Research Unit, Medical Research Center, Oulu University Hospital, and University of Oulu, Oulu, Finland
| | - Pekka Ruusuvuori
- Faculty of Medicine and Health Technology, Tampere University, Tampere, Finland
- Cancer Research Unit, Institute of Biomedicine, University of Turku, Turku, Finland
- FICAN West Cancer Centre, Turku University Hospital, Turku, Finland
| | - Ilkka Pölönen
- Faculty of Information Technology, University of Jyväskylä, Jyväskylä, Finland
| | - Sami Äyrämö
- Faculty of Information Technology, University of Jyväskylä, Jyväskylä, Finland
| | - Teijo Kuopio
- Department of Education and Research, Hospital Nova of Central Finland, Jyväskylä, Finland
- Department of Biological and Environmental Science, University of Jyväskylä, Jyväskylä, Finland
- Department of Pathology, Hospital Nova of Central Finland, Jyväskylä, Finland
| |
Collapse
|
112
|
Zhang Y, Chen S, Wang Y, Li J, Xu K, Chen J, Zhao J. Deep learning-based methods for classification of microsatellite instability in endometrial cancer from HE-stained pathological images. J Cancer Res Clin Oncol 2023:10.1007/s00432-023-04838-4. [PMID: 37150803 DOI: 10.1007/s00432-023-04838-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/18/2023] [Accepted: 05/03/2023] [Indexed: 05/09/2023]
Abstract
BACKGROUND Microsatellite instability (MSI) is one of the essential tumor biomarkers for cancer treatment and prognosis. The presence of more significant PD-L1 expression on the surface of tumor cells in endometrial cancer with MSI suggests that MSI may be a promising biomarker for anti-PD-1/PD-L1 immunotherapy. However, the conventional testing methods are labor-intensive and expensive for patients. METHODS Inspired by classifiers for MSI based on fast and low-cost deep-learning methods in previous investigations, a new architecture for MSI classification based on an attention module is proposed to extract features from pathological images. Especially, slide-level microsatellite status will be obtained by the bag of words method to aggregate probabilities predicted by the proposed model. The H&E-stained whole slide images (WSIs) from The Cancer Genome Atlas endometrial cohort are collected as the dataset. The performances of the proposed model were primarily evaluated by the area under the receiver-operating characteristic curve, accuracy, sensitivity, and F1-Score. RESULTS On the randomly divided test dataset, the proposed model achieved an accuracy of 0.80, a sensitivity of 0.857, a F1-Score of 0.826, and an AUROC of 0.799. We then visualize the results of the microsatellite status classification to capture more specific morphological features, helping pathologists better understand how deep learning performs the classification. CONCLUSIONS This study implements the prediction of microsatellite status in endometrial cancer cases using deep-learning methods directly from H&E-stained WSIs. The proposed architecture can help the model capture more valuable features for classification. In contrast to current laboratory testing methods, the proposed model creates a more convenient screening tool for rapid automated testing for patients. This method can potentially be a clinical method for detecting the microsatellite status of endometrial cancer.
Collapse
Affiliation(s)
- Ying Zhang
- Xuzhou Medical University, Xuzhou, 221004, Jiangsu, China
| | - Shijie Chen
- Xuzhou Medical University, Xuzhou, 221004, Jiangsu, China
| | - Yuling Wang
- Xuzhou Medical University, Xuzhou, 221004, Jiangsu, China
| | - Jingjing Li
- Xuzhou Medical University, Xuzhou, 221004, Jiangsu, China
| | - Kai Xu
- Xuzhou Medical University, Xuzhou, 221004, Jiangsu, China
| | - Jyhcheng Chen
- Xuzhou Medical University, Xuzhou, 221004, Jiangsu, China
| | - Jie Zhao
- Xuzhou Medical University, Xuzhou, 221004, Jiangsu, China.
| |
Collapse
|
113
|
Fan L, Sowmya A, Meijering E, Song Y. Cancer Survival Prediction From Whole Slide Images With Self-Supervised Learning and Slide Consistency. IEEE TRANSACTIONS ON MEDICAL IMAGING 2023; 42:1401-1412. [PMID: 37015696 DOI: 10.1109/tmi.2022.3228275] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/04/2023]
Abstract
Histopathological Whole Slide Images (WSIs) at giga-pixel resolution are the gold standard for cancer analysis and prognosis. Due to the scarcity of pixel- or patch-level annotations of WSIs, many existing methods attempt to predict survival outcomes based on a three-stage strategy that includes patch selection, patch-level feature extraction and aggregation. However, the patch features are usually extracted by using truncated models (e.g. ResNet) pretrained on ImageNet without fine-tuning on WSI tasks, and the aggregation stage does not consider the many-to-one relationship between multiple WSIs and the patient. In this paper, we propose a novel survival prediction framework that consists of patch sampling, feature extraction and patient-level survival prediction. Specifically, we employ two kinds of self-supervised learning methods, i.e. colorization and cross-channel, as pretext tasks to train convnet-based models that are tailored for extracting features from WSIs. Then, at the patient-level survival prediction we explicitly aggregate features from multiple WSIs, using consistency and contrastive losses to normalize slide-level features at the patient level. We conduct extensive experiments on three large-scale datasets: TCGA-GBM, TCGA-LUSC and NLST. Experimental results demonstrate the effectiveness of our proposed framework, as it achieves state-of-the-art performance in comparison with previous studies, with concordance index of 0.670, 0.679 and 0.711 on TCGA-GBM, TCGA-LUSC and NLST, respectively.
Collapse
|
114
|
Morel LO, Derangère V, Arnould L, Ladoire S, Vinçon N. Preliminary evaluation of deep learning for first-line diagnostic prediction of tumor mutational status. Sci Rep 2023; 13:6927. [PMID: 37117277 PMCID: PMC10147624 DOI: 10.1038/s41598-023-34016-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/11/2022] [Accepted: 04/22/2023] [Indexed: 04/30/2023] Open
Abstract
The detection of tumour gene mutations by DNA or RNA sequencing is crucial for the prescription of effective targeted therapies. Recent developments showed promising results for tumoral mutational status prediction using new deep learning based methods on histopathological images. However, it is still unknown whether these methods can be useful aside from sequencing methods for efficient population diagnosis. In this retrospective study, we use a standard prediction pipeline based on a convolutional neural network for the detection of cancer driver genomic alterations in The Cancer Genome Atlas (TCGA) breast (BRCA, n = 719), lung (LUAD, n = 541) and colon (COAD, n = 459) cancer datasets. We propose 3 diagnostic strategies using deep learning methods as first-line diagnostic tools. Focusing on cancer driver genes such as KRAS, EGFR or TP53, we show that these methods help reduce DNA sequencing by up to 49.9% with a high sensitivity (95%). In a context of limited resources, these methods increase sensitivity up to 69.8% at a 30% capacity of DNA sequencing tests, up to 85.1% at a 50% capacity, and up to 91.8% at a 70% capacity. These methods can also be used to prioritize patients with a positive predictive value up to 90.6% in the 10% patient most at risk of being mutated. Limitations of this study include the lack of external validation on non-TCGA data, dependence on prevalence of mutations in datasets, and use of a standard DL method on a limited dataset. Future studies using state-of-the-art methods and larger datasets are needed for better evaluation and clinical implementation.
Collapse
|
115
|
Xiao Y, Wang S, Ling R, Song Y. Application of artificial neural network algorithm in pathological diagnosis and prognosis prediction of digestive tract malignant tumors. Zhejiang Da Xue Xue Bao Yi Xue Ban 2023; 52:243-248. [PMID: 37283110 DOI: 10.3724/zdxbyxb-2022-0569] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 06/08/2023]
Abstract
The application of artificial neural network algorithm in pathological diagnosis of gastrointestinal malignant tumors has become a research hotspot. In the previous studies, the algorithm research mainly focused on the model development based on convolutional neural networks, while only a few studies used the combination of convolutional neural networks and recurrent neural networks. The research contents included classical histopathological diagnosis and molecular typing of malignant tumors, and the prediction of patient prognosis by utilizing artificial neural networks. This article reviews the research progress on artificial neural network algorithm in the pathological diagnosis and prognosis prediction of digestive tract malignant tumors.
Collapse
Affiliation(s)
- Ya Xiao
- Health Science Center, Ningbo University, Ningbo 315211, Zhejiang Province, China.
| | - Shuyang Wang
- Department of Pathology, School of Basic Medical Sciences, Fudan University, Shanghai Key Laboratory of Medical Imaging Computing and Computer Assisted Intervention, Shanghai 200032, China
| | - Ren Ling
- Shanghai Laizi Software Technology Co. Ltd., Shanghai 201499, China
| | - Yufei Song
- Department of Gastroenterology, the Affiliated Lihuili Hospital, Ningbo University, Ningbo 315046, Zhejiang Province, China.
| |
Collapse
|
116
|
Roy M, Wang F, Teodoro G, Bhattarai S, Bhargava M, Rekha TS, Aneja R, Kong J. Deep learning based registration of serial whole-slide histopathology images in different stains. J Pathol Inform 2023; 14:100311. [PMID: 37214150 PMCID: PMC10193019 DOI: 10.1016/j.jpi.2023.100311] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/31/2022] [Revised: 04/11/2023] [Accepted: 04/12/2023] [Indexed: 05/24/2023] Open
Abstract
For routine pathology diagnosis and imaging-based biomedical research, Whole-slide image (WSI) analyses have been largely limited to a 2D tissue image space. For a more definitive tissue representation to support fine-resolution spatial and integrative analyses, it is critical to extend such tissue-based investigations to a 3D tissue space with spatially aligned serial tissue WSIs in different stains, such as Hematoxylin and Eosin (H&E) and Immunohistochemistry (IHC) biomarkers. However, such WSI registration is technically challenged by the overwhelming image scale, the complex histology structure change, and the significant difference in tissue appearances in different stains. The goal of this study is to register serial sections from multi-stain histopathology whole-slide image blocks. We propose a novel translation-based deep learning registration network CGNReg that spatially aligns serial WSIs stained in H&E and by IHC biomarkers without prior deformation information for the model training. First, synthetic IHC images are produced from H&E slides through a robust image synthesis algorithm. Next, the synthetic and the real IHC images are registered through a Fully Convolutional Network with multi-scaled deformable vector fields and a joint loss optimization. We perform the registration at the full image resolution, retaining the tissue details in the results. Evaluated with a dataset of 76 breast cancer patients with 1 H&E and 2 IHC serial WSIs for each patient, CGNReg presents promising performance as compared with multiple state-of-the-art systems in our evaluation. Our results suggest that CGNReg can produce promising registration results with serial WSIs in different stains, enabling integrative 3D tissue-based biomedical investigations.
Collapse
Affiliation(s)
- Mousumi Roy
- Department of Computer Science, Stony Brook University, NY 11794, USA
| | - Fusheng Wang
- Department of Computer Science, Stony Brook University, NY 11794, USA
- Department of Biomedical Informatics, Stony Brook University, NY 11794, USA
| | - George Teodoro
- Department of Computer Science, Federal University of Minas Gerais, Belo Horizonte 31270-901, Brazil
| | - Shristi Bhattarai
- Department of Clinical and Diagnostic Sciences, School of Health Profession, University of Alabama at Birmingham, Birmingham, AL 35233, USA
| | - Mahak Bhargava
- Department of Clinical and Diagnostic Sciences, School of Health Profession, University of Alabama at Birmingham, Birmingham, AL 35233, USA
| | - T. Subbanna Rekha
- Department of Pathology, JSS Medical College, JSS Academy of Higher Education and Research, Mysuru, Karnataka 570009, India
| | - Ritu Aneja
- Department of Clinical and Diagnostic Sciences, School of Health Profession, University of Alabama at Birmingham, Birmingham, AL 35233, USA
| | - Jun Kong
- Department of Mathematics and Statistics, Georgia State University, Atlanta, GA 30303, USA
- Department of Computer Science and Winship Cancer Institute, Emory University, Atlanta, GA 30322, USA
| |
Collapse
|
117
|
Tayebi Arasteh S, Isfort P, Saehn M, Mueller-Franzes G, Khader F, Kather JN, Kuhl C, Nebelung S, Truhn D. Collaborative training of medical artificial intelligence models with non-uniform labels. Sci Rep 2023; 13:6046. [PMID: 37055456 PMCID: PMC10102221 DOI: 10.1038/s41598-023-33303-y] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/11/2023] [Accepted: 04/11/2023] [Indexed: 04/15/2023] Open
Abstract
Due to the rapid advancements in recent years, medical image analysis is largely dominated by deep learning (DL). However, building powerful and robust DL models requires training with large multi-party datasets. While multiple stakeholders have provided publicly available datasets, the ways in which these data are labeled vary widely. For Instance, an institution might provide a dataset of chest radiographs containing labels denoting the presence of pneumonia, while another institution might have a focus on determining the presence of metastases in the lung. Training a single AI model utilizing all these data is not feasible with conventional federated learning (FL). This prompts us to propose an extension to the widespread FL process, namely flexible federated learning (FFL) for collaborative training on such data. Using 695,000 chest radiographs from five institutions from across the globe-each with differing labels-we demonstrate that having heterogeneously labeled datasets, FFL-based training leads to significant performance increase compared to conventional FL training, where only the uniformly annotated images are utilized. We believe that our proposed algorithm could accelerate the process of bringing collaborative training methods from research and simulation phase to the real-world applications in healthcare.
Collapse
Affiliation(s)
- Soroosh Tayebi Arasteh
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Pauwelsstr. 30, 52074, Aachen, Germany
| | - Peter Isfort
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Pauwelsstr. 30, 52074, Aachen, Germany
| | - Marwin Saehn
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Pauwelsstr. 30, 52074, Aachen, Germany
| | - Gustav Mueller-Franzes
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Pauwelsstr. 30, 52074, Aachen, Germany
| | - Firas Khader
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Pauwelsstr. 30, 52074, Aachen, Germany
| | - Jakob Nikolas Kather
- Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
- Medical Faculty Carl Gustav Carus, Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
- Division of Pathology and Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, UK
- Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany
| | - Christiane Kuhl
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Pauwelsstr. 30, 52074, Aachen, Germany
| | - Sven Nebelung
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Pauwelsstr. 30, 52074, Aachen, Germany
| | - Daniel Truhn
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Pauwelsstr. 30, 52074, Aachen, Germany.
| |
Collapse
|
118
|
Shi L, Zhang Y, Wang H. Prognostic prediction based on histopathologic features of tumor microenvironment in colorectal cancer. Front Med (Lausanne) 2023; 10:1154077. [PMID: 37089601 PMCID: PMC10117979 DOI: 10.3389/fmed.2023.1154077] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/30/2023] [Accepted: 03/20/2023] [Indexed: 04/09/2023] Open
Abstract
PurposeTo automatically quantify colorectal tumor microenvironment (TME) in hematoxylin and eosin stained whole slide images (WSIs), and to develop a TME signature for prognostic prediction in colorectal cancer (CRC).MethodsA deep learning model based on VGG19 architecture and transfer learning strategy was trained to recognize nine different tissue types in whole slide images of patients with CRC. Seven of the nine tissue types were defined as TME components besides background and debris. Then 13 TME features were calculated based on the areas of TME components. A total of 562 patients with gene expression data, survival information and WSIs were collected from The Cancer Genome Atlas project for further analysis. A TME signature for prognostic prediction was developed and validated using Cox regression method. A prognostic prediction model combined the TME signature and clinical variables was also established. At last, gene-set enrichment analysis was performed to identify the significant TME signature associated pathways by querying Gene Ontology database and Kyoto Encyclopedia of Genes and Genomes database.ResultsThe deep learning model achieved an accuracy of 94.2% for tissue type recognition. The developed TME signature was found significantly associated to progression-free survival. The clinical combined model achieved a concordance index of 0.714. Gene-set enrichment analysis revealed the TME signature associated genes were enriched in neuroactive ligand-receptor interaction pathway.ConclusionThe TME signature was proved to be a prognostic factor and the associated biologic pathways would be beneficial to a better understanding of TME in CRC patients.
Collapse
Affiliation(s)
- Liang Shi
- School of Clinical Medicine, Hebei University, Baoding, Hebei, China
- The First Department of General Surgery, Cangzhou Central Hospital of Hebei Province, Cangzhou, Hebei, China
| | - Yuhao Zhang
- Department of Neurosurgery, Zhejiang Provincial People's Hospital, Affiliated to Hangzhou Medical College, Hangzhou, Zhejiang, China
| | - Hong Wang
- School of Clinical Medicine, Hebei University, Baoding, Hebei, China
- *Correspondence: Hong Wang,
| |
Collapse
|
119
|
Lan J, Chen M, Wang J, Du M, Wu Z, Zhang H, Xue Y, Wang T, Chen L, Xu C, Han Z, Hu Z, Zhou Y, Zhou X, Tong T, Chen G. Using less annotation workload to establish a pathological auxiliary diagnosis system for gastric cancer. Cell Rep Med 2023; 4:101004. [PMID: 37044091 PMCID: PMC10140598 DOI: 10.1016/j.xcrm.2023.101004] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/29/2022] [Revised: 10/20/2022] [Accepted: 03/17/2023] [Indexed: 04/14/2023]
Abstract
Pathological diagnosis of gastric cancer requires pathologists to have extensive clinical experience. To help pathologists improve diagnostic accuracy and efficiency, we collected 1,514 cases of stomach H&E-stained specimens with complete diagnostic information to establish a pathological auxiliary diagnosis system based on deep learning. At the slide level, our system achieves a specificity of 0.8878 while maintaining a high sensitivity close to 1.0 on 269 biopsy specimens (147 malignancies) and 163 surgical specimens (80 malignancies). The classified accuracy of our system is 0.9034 at the slide level for 352 biopsy specimens (201 malignancies) from 50 medical centers. With the help of our system, the pathologists' average false-negative rate and average false-positive rate on 100 biopsy specimens (50 malignancies) are reduced to 1/5 and 1/2 of the original rates, respectively. At the same time, the average uncertainty rate and the average diagnosis time are reduced by approximately 22% and 20%, respectively.
Collapse
Affiliation(s)
- Junlin Lan
- College of Physics and Information Engineering, Fuzhou University, Fuzhou, Fujian 350108, China; Key Lab of Medical Instrumentation & Pharmaceutical Technology of Fujian Province, Fuzhou University, Fuzhou, Fujian 350108, China
| | - Musheng Chen
- Department of Pathology, Clinical Oncology School of Fujian Medical University, Fujian Cancer Hospital, Fuzhou, Fujian 350014, China; Fujian Key Laboratory of Translational Cancer Medicine, Fuzhou, Fujian 350014, China
| | - Jianchao Wang
- Department of Pathology, Clinical Oncology School of Fujian Medical University, Fujian Cancer Hospital, Fuzhou, Fujian 350014, China; Fujian Key Laboratory of Translational Cancer Medicine, Fuzhou, Fujian 350014, China
| | - Min Du
- College of Physics and Information Engineering, Fuzhou University, Fuzhou, Fujian 350108, China; Key Lab of Medical Instrumentation & Pharmaceutical Technology of Fujian Province, Fuzhou University, Fuzhou, Fujian 350108, China
| | - Zhida Wu
- Department of Pathology, Clinical Oncology School of Fujian Medical University, Fujian Cancer Hospital, Fuzhou, Fujian 350014, China; Fujian Key Laboratory of Translational Cancer Medicine, Fuzhou, Fujian 350014, China
| | - Hejun Zhang
- Department of Pathology, Clinical Oncology School of Fujian Medical University, Fujian Cancer Hospital, Fuzhou, Fujian 350014, China; Fujian Key Laboratory of Translational Cancer Medicine, Fuzhou, Fujian 350014, China
| | - Yuyang Xue
- School of Engineering, University of Edinburgh, Edinburgh EH8 9JU, UK
| | - Tao Wang
- College of Physics and Information Engineering, Fuzhou University, Fuzhou, Fujian 350108, China; Key Lab of Medical Instrumentation & Pharmaceutical Technology of Fujian Province, Fuzhou University, Fuzhou, Fujian 350108, China
| | - Lifan Chen
- Department of Pathology, Clinical Oncology School of Fujian Medical University, Fujian Cancer Hospital, Fuzhou, Fujian 350014, China; Fujian Key Laboratory of Translational Cancer Medicine, Fuzhou, Fujian 350014, China
| | - Chaohui Xu
- College of Physics and Information Engineering, Fuzhou University, Fuzhou, Fujian 350108, China; Key Lab of Medical Instrumentation & Pharmaceutical Technology of Fujian Province, Fuzhou University, Fuzhou, Fujian 350108, China
| | - Zixin Han
- College of Physics and Information Engineering, Fuzhou University, Fuzhou, Fujian 350108, China; Key Lab of Medical Instrumentation & Pharmaceutical Technology of Fujian Province, Fuzhou University, Fuzhou, Fujian 350108, China
| | - Ziwei Hu
- College of Physics and Information Engineering, Fuzhou University, Fuzhou, Fujian 350108, China; Key Lab of Medical Instrumentation & Pharmaceutical Technology of Fujian Province, Fuzhou University, Fuzhou, Fujian 350108, China
| | - Yuanbo Zhou
- College of Physics and Information Engineering, Fuzhou University, Fuzhou, Fujian 350108, China; Key Lab of Medical Instrumentation & Pharmaceutical Technology of Fujian Province, Fuzhou University, Fuzhou, Fujian 350108, China
| | - Xiaogen Zhou
- College of Physics and Information Engineering, Fuzhou University, Fuzhou, Fujian 350108, China; Key Lab of Medical Instrumentation & Pharmaceutical Technology of Fujian Province, Fuzhou University, Fuzhou, Fujian 350108, China
| | - Tong Tong
- College of Physics and Information Engineering, Fuzhou University, Fuzhou, Fujian 350108, China; Key Lab of Medical Instrumentation & Pharmaceutical Technology of Fujian Province, Fuzhou University, Fuzhou, Fujian 350108, China; Imperial Vision Technology, Fuzhou, Fujian 350100, China.
| | - Gang Chen
- Department of Pathology, Clinical Oncology School of Fujian Medical University, Fujian Cancer Hospital, Fuzhou, Fujian 350014, China; Fujian Key Laboratory of Translational Cancer Medicine, Fuzhou, Fujian 350014, China.
| |
Collapse
|
120
|
Liang J, Zhang W, Yang J, Wu M, Dai Q, Yin H, Xiao Y, Kong L. Deep learning supported discovery of biomarkers for clinical prognosis of liver cancer. NAT MACH INTELL 2023. [DOI: 10.1038/s42256-023-00635-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 04/05/2023]
|
121
|
Vu QD, Rajpoot K, Raza SEA, Rajpoot N. Handcrafted Histological Transformer (H2T): Unsupervised representation of whole slide images. Med Image Anal 2023; 85:102743. [PMID: 36702037 DOI: 10.1016/j.media.2023.102743] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/14/2022] [Revised: 11/30/2022] [Accepted: 01/05/2023] [Indexed: 01/20/2023]
Abstract
Diagnostic, prognostic and therapeutic decision-making of cancer in pathology clinics can now be carried out based on analysis of multi-gigapixel tissue images, also known as whole-slide images (WSIs). Recently, deep convolutional neural networks (CNNs) have been proposed to derive unsupervised WSI representations; these are attractive as they rely less on expert annotation which is cumbersome. However, a major trade-off is that higher predictive power generally comes at the cost of interpretability, posing a challenge to their clinical use where transparency in decision-making is generally expected. To address this challenge, we present a handcrafted framework based on deep CNN for constructing holistic WSI-level representations. Building on recent findings about the internal working of the Transformer in the domain of natural language processing, we break down its processes and handcraft them into a more transparent framework that we term as the Handcrafted Histological Transformer or H2T. Based on our experiments involving various datasets consisting of a total of 10,042 WSIs, the results demonstrate that H2T based holistic WSI-level representations offer competitive performance compared to recent state-of-the-art methods and can be readily utilized for various downstream analysis tasks. Finally, our results demonstrate that the H2T framework can be up to 14 times faster than the Transformer models.
Collapse
Affiliation(s)
- Quoc Dang Vu
- Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, UK
| | - Kashif Rajpoot
- School of Computer Science, University of Birmingham, UK
| | - Shan E Ahmed Raza
- Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, UK
| | - Nasir Rajpoot
- Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, UK; The Alan Turing Institute, London, UK; Department of Pathology, University Hospitals Coventry & Warwickshire, UK.
| |
Collapse
|
122
|
Li J, Chen J, Tang Y, Wang C, Landman BA, Zhou SK. Transforming medical imaging with Transformers? A comparative review of key properties, current progresses, and future perspectives. Med Image Anal 2023; 85:102762. [PMID: 36738650 PMCID: PMC10010286 DOI: 10.1016/j.media.2023.102762] [Citation(s) in RCA: 27] [Impact Index Per Article: 27.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/02/2022] [Revised: 01/18/2023] [Accepted: 01/27/2023] [Indexed: 02/01/2023]
Abstract
Transformer, one of the latest technological advances of deep learning, has gained prevalence in natural language processing or computer vision. Since medical imaging bear some resemblance to computer vision, it is natural to inquire about the status quo of Transformers in medical imaging and ask the question: can the Transformer models transform medical imaging? In this paper, we attempt to make a response to the inquiry. After a brief introduction of the fundamentals of Transformers, especially in comparison with convolutional neural networks (CNNs), and highlighting key defining properties that characterize the Transformers, we offer a comprehensive review of the state-of-the-art Transformer-based approaches for medical imaging and exhibit current research progresses made in the areas of medical image segmentation, recognition, detection, registration, reconstruction, enhancement, etc. In particular, what distinguishes our review lies in its organization based on the Transformer's key defining properties, which are mostly derived from comparing the Transformer and CNN, and its type of architecture, which specifies the manner in which the Transformer and CNN are combined, all helping the readers to best understand the rationale behind the reviewed approaches. We conclude with discussions of future perspectives.
Collapse
Affiliation(s)
- Jun Li
- Key Lab of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS, Beijing 100190, China
| | - Junyu Chen
- Russell H. Morgan Department of Radiology and Radiological Science, Johns Hopkins Medical Institutes, Baltimore, MD, USA
| | - Yucheng Tang
- Department of Electrical and Computer Engineering, Vanderbilt University, Nashville, TN, USA
| | - Ce Wang
- Key Lab of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS, Beijing 100190, China
| | - Bennett A Landman
- Department of Electrical and Computer Engineering, Vanderbilt University, Nashville, TN, USA
| | - S Kevin Zhou
- Key Lab of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS, Beijing 100190, China; School of Biomedical Engineering & Suzhou Institute for Advanced Research, Center for Medical Imaging, Robotics, and Analytic Computing & Learning (MIRACLE), University of Science and Technology of China, Suzhou 215123, China.
| |
Collapse
|
123
|
Sun K, Chen Y, Bai B, Gao Y, Xiao J, Yu G. Automatic Classification of Histopathology Images across Multiple Cancers Based on Heterogeneous Transfer Learning. Diagnostics (Basel) 2023; 13:diagnostics13071277. [PMID: 37046497 PMCID: PMC10093253 DOI: 10.3390/diagnostics13071277] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/10/2023] [Revised: 03/07/2023] [Accepted: 03/23/2023] [Indexed: 03/31/2023] Open
Abstract
Background: Current artificial intelligence (AI) in histopathology typically specializes on a single task, resulting in a heavy workload of collecting and labeling a sufficient number of images for each type of cancer. Heterogeneous transfer learning (HTL) is expected to alleviate the data bottlenecks and establish models with performance comparable to supervised learning (SL). Methods: An accurate source domain model was trained using 28,634 colorectal patches. Additionally, 1000 sentinel lymph node patches and 1008 breast patches were used to train two target domain models. The feature distribution difference between sentinel lymph node metastasis or breast cancer and CRC was reduced by heterogeneous domain adaptation, and the maximum mean difference between subdomains was used for knowledge transfer to achieve accurate classification across multiple cancers. Result: HTL on 1000 sentinel lymph node patches (L-HTL-1000) outperforms SL on 1000 sentinel lymph node patches (L-SL-1-1000) (average area under the curve (AUC) and standard deviation of L-HTL-1000 vs. L-SL-1-1000: 0.949 ± 0.004 vs. 0.931 ± 0.008, p value = 0.008). There is no significant difference between L-HTL-1000 and SL on 7104 patches (L-SL-2-7104) (0.949 ± 0.004 vs. 0.948 ± 0.008, p value = 0.742). Similar results are observed for breast cancer. B-HTL-1008 vs. B-SL-1-1008: 0.962 ± 0.017 vs. 0.943 ± 0.018, p value = 0.008; B-HTL-1008 vs. B-SL-2-5232: 0.962 ± 0.017 vs. 0.951 ± 0.023, p value = 0.148. Conclusions: HTL is capable of building accurate AI models for similar cancers using a small amount of data based on a large dataset for a certain type of cancer. HTL holds great promise for accelerating the development of AI in histopathology.
Collapse
|
124
|
Manzari ON, Ahmadabadi H, Kashiani H, Shokouhi SB, Ayatollahi A. MedViT: A robust vision transformer for generalized medical image classification. Comput Biol Med 2023; 157:106791. [PMID: 36958234 DOI: 10.1016/j.compbiomed.2023.106791] [Citation(s) in RCA: 11] [Impact Index Per Article: 11.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/06/2022] [Revised: 02/18/2023] [Accepted: 03/11/2023] [Indexed: 03/16/2023]
Abstract
Convolutional Neural Networks (CNNs) have advanced existing medical systems for automatic disease diagnosis. However, there are still concerns about the reliability of deep medical diagnosis systems against the potential threats of adversarial attacks since inaccurate diagnosis could lead to disastrous consequences in the safety realm. In this study, we propose a highly robust yet efficient CNN-Transformer hybrid model which is equipped with the locality of CNNs as well as the global connectivity of vision Transformers. To mitigate the high quadratic complexity of the self-attention mechanism while jointly attending to information in various representation subspaces, we construct our attention mechanism by means of an efficient convolution operation. Moreover, to alleviate the fragility of our Transformer model against adversarial attacks, we attempt to learn smoother decision boundaries. To this end, we augment the shape information of an image in the high-level feature space by permuting the feature mean and variance within mini-batches. With less computational complexity, our proposed hybrid model demonstrates its high robustness and generalization ability compared to the state-of-the-art studies on a large-scale collection of standardized MedMNIST-2D datasets.
Collapse
Affiliation(s)
- Omid Nejati Manzari
- School of Electrical Engineering, Iran University of Science and Technology, Tehran, Iran.
| | - Hamid Ahmadabadi
- School of Electrical Engineering, Iran University of Science and Technology, Tehran, Iran
| | - Hossein Kashiani
- Lane Department of Computer Science and Electrical Engineering, West Virginia University, Morgantown, USA
| | - Shahriar B Shokouhi
- School of Electrical Engineering, Iran University of Science and Technology, Tehran, Iran
| | - Ahmad Ayatollahi
- School of Electrical Engineering, Iran University of Science and Technology, Tehran, Iran
| |
Collapse
|
125
|
Basu A, Senapati P, Deb M, Rai R, Dhal KG. A survey on recent trends in deep learning for nucleus segmentation from histopathology images. EVOLVING SYSTEMS 2023; 15:1-46. [PMID: 38625364 PMCID: PMC9987406 DOI: 10.1007/s12530-023-09491-3] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/03/2022] [Accepted: 02/13/2023] [Indexed: 03/08/2023]
Abstract
Nucleus segmentation is an imperative step in the qualitative study of imaging datasets, considered as an intricate task in histopathology image analysis. Segmenting a nucleus is an important part of diagnosing, staging, and grading cancer, but overlapping regions make it hard to separate and tell apart independent nuclei. Deep Learning is swiftly paving its way in the arena of nucleus segmentation, attracting quite a few researchers with its numerous published research articles indicating its efficacy in the field. This paper presents a systematic survey on nucleus segmentation using deep learning in the last five years (2017-2021), highlighting various segmentation models (U-Net, SCPP-Net, Sharp U-Net, and LiverNet) and exploring their similarities, strengths, datasets utilized, and unfolding research areas.
Collapse
Affiliation(s)
- Anusua Basu
- Department of Computer Science and Application, Midnapore College (Autonomous), Paschim Medinipur, Midnapore, West Bengal India
| | - Pradip Senapati
- Department of Computer Science and Application, Midnapore College (Autonomous), Paschim Medinipur, Midnapore, West Bengal India
| | - Mainak Deb
- Wipro Technologies, Pune, Maharashtra India
| | - Rebika Rai
- Department of Computer Applications, Sikkim University, Sikkim, India
| | - Krishna Gopal Dhal
- Department of Computer Science and Application, Midnapore College (Autonomous), Paschim Medinipur, Midnapore, West Bengal India
| |
Collapse
|
126
|
Eloy C, Marques A, Pinto J, Pinheiro J, Campelos S, Curado M, Vale J, Polónia A. Artificial intelligence-assisted cancer diagnosis improves the efficiency of pathologists in prostatic biopsies. Virchows Arch 2023; 482:595-604. [PMID: 36809483 PMCID: PMC10033575 DOI: 10.1007/s00428-023-03518-5] [Citation(s) in RCA: 13] [Impact Index Per Article: 13.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/19/2023] [Revised: 02/07/2023] [Accepted: 02/09/2023] [Indexed: 02/23/2023]
Abstract
Paige Prostate is a clinical-grade artificial intelligence tool designed to assist the pathologist in detecting, grading, and quantifying prostate cancer. In this work, a cohort of 105 prostate core needle biopsies (CNBs) was evaluated through digital pathology. Then, we compared the diagnostic performance of four pathologists diagnosing prostatic CNB unaided and, in a second phase, assisted by Paige Prostate. In phase 1, pathologists had a diagnostic accuracy for prostate cancer of 95.00%, maintaining their performance in phase 2 (93.81%), with an intraobserver concordance rate between phases of 98.81%. In phase 2, pathologists reported atypical small acinar proliferation (ASAP) less often (about 30% less). Additionally, they requested significantly fewer immunohistochemistry (IHC) studies (about 20% less) and second opinions (about 40% less). The median time required for reading and reporting each slide was about 20% lower in phase 2, in both negative and cancer cases. Lastly, the average total agreement with the software performance was observed in about 70% of the cases, being significantly higher in negative cases (about 90%) than in cancer cases (about 30%). Most of the diagnostic discordances occurred in distinguishing negative cases with ASAP from small foci of well-differentiated (less than 1.5 mm) acinar adenocarcinoma. In conclusion, the synergic usage of Paige Prostate contributes to a significant decrease in IHC studies, second opinion requests, and time for reporting while maintaining highly accurate diagnostic standards.
Collapse
Affiliation(s)
- Catarina Eloy
- Pathology Laboratory, Institute of Molecular Pathology and Immunology of the University of Porto (Ipatimup), Porto, Portugal
- i3S - Instituto de Investigação E Inovação Em Saúde, Porto, Portugal
- Faculty of Medicine, University of Porto, Porto, Portugal
| | - Ana Marques
- Pathology Laboratory, Institute of Molecular Pathology and Immunology of the University of Porto (Ipatimup), Porto, Portugal
- Serviço de Anatomia Patológica, Centro Hospitalar Universitário de São João, Porto, Portugal
| | - João Pinto
- Pathology Laboratory, Institute of Molecular Pathology and Immunology of the University of Porto (Ipatimup), Porto, Portugal
- Serviço de Anatomia Patológica, Hospital Pedro Hispano - Unidade Local de Saúde de Matosinhos, Matosinhos, Portugal
| | - Jorge Pinheiro
- Pathology Laboratory, Institute of Molecular Pathology and Immunology of the University of Porto (Ipatimup), Porto, Portugal
- Serviço de Anatomia Patológica, Centro Hospitalar Universitário de São João, Porto, Portugal
| | - Sofia Campelos
- Pathology Laboratory, Institute of Molecular Pathology and Immunology of the University of Porto (Ipatimup), Porto, Portugal
| | - Mónica Curado
- Pathology Laboratory, Institute of Molecular Pathology and Immunology of the University of Porto (Ipatimup), Porto, Portugal
| | - João Vale
- Pathology Laboratory, Institute of Molecular Pathology and Immunology of the University of Porto (Ipatimup), Porto, Portugal
| | - António Polónia
- Pathology Laboratory, Institute of Molecular Pathology and Immunology of the University of Porto (Ipatimup), Porto, Portugal.
- i3S - Instituto de Investigação E Inovação Em Saúde, Porto, Portugal.
| |
Collapse
|
127
|
Hehr M, Sadafi A, Matek C, Lienemann P, Pohlkamp C, Haferlach T, Spiekermann K, Marr C. Explainable AI identifies diagnostic cells of genetic AML subtypes. PLOS DIGITAL HEALTH 2023; 2:e0000187. [PMID: 36921004 PMCID: PMC10016704 DOI: 10.1371/journal.pdig.0000187] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/26/2022] [Accepted: 12/19/2022] [Indexed: 03/16/2023]
Abstract
Explainable AI is deemed essential for clinical applications as it allows rationalizing model predictions, helping to build trust between clinicians and automated decision support tools. We developed an inherently explainable AI model for the classification of acute myeloid leukemia subtypes from blood smears and found that high-attention cells identified by the model coincide with those labeled as diagnostically relevant by human experts. Based on over 80,000 single white blood cell images from digitized blood smears of 129 patients diagnosed with one of four WHO-defined genetic AML subtypes and 60 healthy controls, we trained SCEMILA, a single-cell based explainable multiple instance learning algorithm. SCEMILA could perfectly discriminate between AML patients and healthy controls and detected the APL subtype with an F1 score of 0.86±0.05 (mean±s.d., 5-fold cross-validation). Analyzing a novel multi-attention module, we confirmed that our algorithm focused with high concordance on the same AML-specific cells as human experts do. Applied to classify single cells, it is able to highlight subtype specific cells and deconvolve the composition of a patient's blood smear without the need of single-cell annotation of the training data. Our large AML genetic subtype dataset is publicly available, and an interactive online tool facilitates the exploration of data and predictions. SCEMILA enables a comparison of algorithmic and expert decision criteria and can present a detailed analysis of individual patient data, paving the way to deploy AI in the routine diagnostics for identifying hematopoietic neoplasms.
Collapse
Affiliation(s)
- Matthias Hehr
- Institute of AI for Health, Helmholtz Zentrum München–German Research Center for Environmental Health, Neuherberg, Germany
- Institute of Computational Biology, Helmholtz Zentrum München–German Research Center for Environmental Health, Neuherberg, Germany
- Laboratory of Leukemia Diagnostics, Department of Medicine III, University Hospital, LMU Munich, Munich, Germany
| | - Ario Sadafi
- Institute of AI for Health, Helmholtz Zentrum München–German Research Center for Environmental Health, Neuherberg, Germany
- Institute of Computational Biology, Helmholtz Zentrum München–German Research Center for Environmental Health, Neuherberg, Germany
- Computer Aided Medical Procedures, Technical University of Munich, Munich, Germany
| | - Christian Matek
- Institute of AI for Health, Helmholtz Zentrum München–German Research Center for Environmental Health, Neuherberg, Germany
- Institute of Computational Biology, Helmholtz Zentrum München–German Research Center for Environmental Health, Neuherberg, Germany
- Laboratory of Leukemia Diagnostics, Department of Medicine III, University Hospital, LMU Munich, Munich, Germany
| | - Peter Lienemann
- Institute of AI for Health, Helmholtz Zentrum München–German Research Center for Environmental Health, Neuherberg, Germany
- Laboratory of Leukemia Diagnostics, Department of Medicine III, University Hospital, LMU Munich, Munich, Germany
| | | | | | - Karsten Spiekermann
- Laboratory of Leukemia Diagnostics, Department of Medicine III, University Hospital, LMU Munich, Munich, Germany
- German Cancer Consortium (DKTK), Heidelberg, Germany
- German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Carsten Marr
- Institute of AI for Health, Helmholtz Zentrum München–German Research Center for Environmental Health, Neuherberg, Germany
- Institute of Computational Biology, Helmholtz Zentrum München–German Research Center for Environmental Health, Neuherberg, Germany
- * E-mail:
| |
Collapse
|
128
|
Investigation of semi- and self-supervised learning methods in the histopathological domain. J Pathol Inform 2023; 14:100305. [PMID: 37025325 PMCID: PMC10070179 DOI: 10.1016/j.jpi.2023.100305] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/12/2023] [Revised: 03/03/2023] [Accepted: 03/03/2023] [Indexed: 03/14/2023] Open
Abstract
Training models with semi- or self-supervised learning methods is one way to reduce annotation effort since they rely on unlabeled or sparsely labeled datasets. Such approaches are particularly promising for domains with a time-consuming annotation process requiring specialized expertise and where high-quality labeled machine learning datasets are scarce, like in computational pathology. Even though some of these methods have been used in the histopathological domain, there is, so far, no comprehensive study comparing different approaches. Therefore, this work compares feature extractors models trained with state-of-the-art semi- or self-supervised learning methods PAWS, SimCLR, and SimSiam within a unified framework. We show that such models, across different architectures and network configurations, have a positive performance impact on histopathological classification tasks, even in low data regimes. Moreover, our observations suggest that features learned from a particular dataset, i.e., tissue type, are only in-domain transferable to a certain extent. Finally, we share our experience using each method in computational pathology and provide recommendations for its use.
Collapse
|
129
|
Eminaga O, Abbas M, Shen J, Laurie M, Brooks JD, Liao JC, Rubin DL. PlexusNet: A neural network architectural concept for medical image classification. Comput Biol Med 2023; 154:106594. [PMID: 36753979 DOI: 10.1016/j.compbiomed.2023.106594] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/10/2022] [Revised: 01/12/2023] [Accepted: 01/22/2023] [Indexed: 01/27/2023]
Abstract
State-of-the-art (SOTA) convolutional neural network models have been widely adapted in medical imaging and applied to address different clinical problems. However, the complexity and scale of such models may not be justified in medical imaging and subject to the available resource budget. Further increasing the number of representative feature maps for the classification task decreases the model explainability. The current data normalization practice is fixed prior to model development and discounting the specification of the data domain. Acknowledging these issues, the current work proposed a new scalable model family called PlexusNet; the block architecture and model scaling by the network's depth, width, and branch regulate PlexusNet's architecture. The efficient computation costs outlined the dimensions of PlexusNet scaling and design. PlexusNet includes a new learnable data normalization algorithm for better data generalization. We applied a simple yet effective neural architecture search to design PlexusNet tailored to five clinical classification problems that achieve a performance noninferior to the SOTA models ResNet-18 and EfficientNet B0/1. It also does so with lower parameter capacity and representative feature maps in ten-fold ranges than the smallest SOTA models with comparable performance. The visualization of representative features revealed distinguishable clusters associated with categories based on latent features generated by PlexusNet. The package and source code are at https://github.com/oeminaga/PlexusNet.git.
Collapse
Affiliation(s)
- Okyaz Eminaga
- Center for Artificial Intelligence in Medicine & Imaging and Department of Urology, Stanford School of Medicine, Stanford, CA, 94305, USA; Department of Urology, Stanford School of Medicine, Stanford, CA, 94305, USA
| | - Mahmoud Abbas
- Department of Pathology, University of Muenster, Muenster, Germany.
| | - Jeanne Shen
- Department of Pathology, Stanford School of Medicine, Stanford, CA, 94305, USA.
| | - Mark Laurie
- Department of Computer Science, Stanford University, Stanford, CA, 94305, USA.
| | - James D Brooks
- Department of Urology, Stanford School of Medicine, Stanford, CA, 94305, USA.
| | - Joseph C Liao
- Department of Urology, Stanford School of Medicine, Stanford, CA, 94305, USA.
| | - Daniel L Rubin
- Department of Biomedical Data Science, Stanford School of Medicine, Stanford, CA, 94305, USA.
| |
Collapse
|
130
|
Wen Z, Wang S, Yang DM, Xie Y, Chen M, Bishop J, Xiao G. Deep learning in digital pathology for personalized treatment plans of cancer patients. Semin Diagn Pathol 2023; 40:109-119. [PMID: 36890029 DOI: 10.1053/j.semdp.2023.02.003] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/12/2023] [Accepted: 02/22/2023] [Indexed: 02/27/2023]
Abstract
Over the past decade, many new cancer treatments have been developed and made available to patients. However, in most cases, these treatments only benefit a specific subgroup of patients, making the selection of treatment for a specific patient an essential but challenging task for oncologists. Although some biomarkers were found to associate with treatment response, manual assessment is time-consuming and subjective. With the rapid developments and expanded implementation of artificial intelligence (AI) in digital pathology, many biomarkers can be quantified automatically from histopathology images. This approach allows for a more efficient and objective assessment of biomarkers, aiding oncologists in formulating personalized treatment plans for cancer patients. This review presents an overview and summary of the recent studies on biomarker quantification and treatment response prediction using hematoxylin-eosin (H&E) stained pathology images. These studies have shown that an AI-based digital pathology approach can be practical and will become increasingly important in improving the selection of cancer treatments for patients.
Collapse
Affiliation(s)
- Zhuoyu Wen
- Quantitative Biomedical Research Center, Department of Population and Data Sciences, University of Texas Southwestern Medical Center, Dallas, TX, USA
| | - Shidan Wang
- Quantitative Biomedical Research Center, Department of Population and Data Sciences, University of Texas Southwestern Medical Center, Dallas, TX, USA
| | - Donghan M Yang
- Quantitative Biomedical Research Center, Department of Population and Data Sciences, University of Texas Southwestern Medical Center, Dallas, TX, USA
| | - Yang Xie
- Quantitative Biomedical Research Center, Department of Population and Data Sciences, University of Texas Southwestern Medical Center, Dallas, TX, USA; Simmons Comprehensive Cancer Center, UT Southwestern Medical Center, Dallas, TX, USA; Department of Pathology, University of Texas Southwestern Medical Center, Dallas, TX, USA
| | - Mingyi Chen
- Department of Bioinformatics, UT Southwestern Medical Center, Dallas, TX, USA
| | - Justin Bishop
- Department of Bioinformatics, UT Southwestern Medical Center, Dallas, TX, USA
| | - Guanghua Xiao
- Quantitative Biomedical Research Center, Department of Population and Data Sciences, University of Texas Southwestern Medical Center, Dallas, TX, USA; Simmons Comprehensive Cancer Center, UT Southwestern Medical Center, Dallas, TX, USA; Department of Pathology, University of Texas Southwestern Medical Center, Dallas, TX, USA.
| |
Collapse
|
131
|
Huss R, Raffler J, Märkl B. Artificial intelligence and digital biomarker in precision pathology guiding immune therapy selection and precision oncology. Cancer Rep (Hoboken) 2023:e1796. [PMID: 36813293 PMCID: PMC10363837 DOI: 10.1002/cnr2.1796] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/29/2022] [Revised: 01/15/2023] [Accepted: 02/09/2023] [Indexed: 02/24/2023] Open
Abstract
BACKGROUND The currently available immunotherapies already changed the strategy how many cancers are treated from first to last line. Understanding even the most complex heterogeneity in tumor tissue and mapping the spatial cartography of the tumor immunity allows the best and optimized selection of immune modulating agents to (re-)activate the patient's immune system and direct it against the individual cancer in the most effective way. RECENT FINDINGS Primary cancer and metastases maintain a high degree of plasticity to escape any immune surveillance and continue to evolve depending on many intrinsic and extrinsic factors In the field of immune-oncology (IO) immune modulating agents are recognized as practice changing therapeutic modalities. Recent studies have shown that an optimal and lasting efficacy of IO therapeutics depends on the understanding of the spatial communication network and functional context of immune and cancer cells within the tumor microenvironment. Artificial intelligence (AI) provides an insight into the immune-cancer-network through the visualization of very complex tumor and immune interactions in cancer tissue specimens and allows the computer-assisted development and clinical validation of such digital biomarker. CONCLUSIONS The successful implementation of AI-supported digital biomarker solutions guides the clinical selection of effective immune therapeutics based on the retrieval and visualization of spatial and contextual information from cancer tissue images and standardized data. As such, computational pathology (CP) turns into "precision pathology" delivering individual therapy response prediction. Precision Pathology does not only include digital and computational solutions but also high levels of standardized processes in the routine histopathology workflow and the use of mathematical tools to support clinical and diagnostic decisions as the basic principle of a "precision oncology".
Collapse
Affiliation(s)
- Ralf Huss
- Medical Faculty University Augsburg, Augsburg, Germany
- Institute for Digital Medicine, University Hospital Augsburg, Augsburg, Germany
| | - Johannes Raffler
- Institute for Digital Medicine, University Hospital Augsburg, Augsburg, Germany
| | - Bruno Märkl
- Medical Faculty University Augsburg, Augsburg, Germany
| |
Collapse
|
132
|
Whole-Slide Images and Patches of Clear Cell Renal Cell Carcinoma Tissue Sections Counterstained with Hoechst 33342, CD3, and CD8 Using Multiple Immunofluorescence. DATA 2023. [DOI: 10.3390/data8020040] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/18/2023] Open
Abstract
In recent years, there has been an increased effort to digitise whole-slide images of cancer tissue. This effort has opened up a range of new avenues for the application of deep learning in oncology. One such avenue is virtual staining, where a deep learning model is tasked with reproducing the appearance of stained tissue sections, conditioned on a different, often times less expensive, input stain. However, data to train such models in a supervised manner where the input and output stains are aligned on the same tissue sections are scarce. In this work, we introduce a dataset of ten whole-slide images of clear cell renal cell carcinoma tissue sections counterstained with Hoechst 33342, CD3, and CD8 using multiple immunofluorescence. We also provide a set of over 600,000 patches of size 256 × 256 pixels extracted from these images together with cell segmentation masks in a format amenable to training deep learning models. It is our hope that this dataset will be used to further the development of deep learning methods for digital pathology by serving as a dataset for comparing and benchmarking virtual staining models.
Collapse
|
133
|
Einhaus J, Rochwarger A, Mattern S, Gaudillière B, Schürch CM. High-multiplex tissue imaging in routine pathology-are we there yet? Virchows Arch 2023; 482:801-812. [PMID: 36757500 PMCID: PMC10156760 DOI: 10.1007/s00428-023-03509-6] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/02/2023] [Revised: 01/22/2023] [Accepted: 01/31/2023] [Indexed: 02/10/2023]
Abstract
High-multiplex tissue imaging (HMTI) approaches comprise several novel immunohistological methods that enable in-depth, spatial single-cell analysis. Over recent years, studies in tumor biology, infectious diseases, and autoimmune conditions have demonstrated the information gain accessible when mapping complex tissues with HMTI. Tumor biology has been a focus of innovative multiparametric approaches, as the tumor microenvironment (TME) contains great informative value for accurate diagnosis and targeted therapeutic approaches: unraveling the cellular composition and structural organization of the TME using sophisticated computational tools for spatial analysis has produced histopathologic biomarkers for outcomes in breast cancer, predictors of positive immunotherapy response in melanoma, and histological subgroups of colorectal carcinoma. Integration of HMTI technologies into existing clinical workflows such as molecular tumor boards will contribute to improve patient outcomes through personalized treatments tailored to the specific heterogeneous pathological fingerprint of cancer, autoimmunity, or infection. Here, we review the advantages and limitations of existing HMTI technologies and outline how spatial single-cell data can improve our understanding of pathological disease mechanisms and determinants of treatment success. We provide an overview of the analytic processing and interpretation and discuss how HMTI can improve future routine clinical diagnostic and therapeutic processes.
Collapse
Affiliation(s)
- Jakob Einhaus
- Department of Anaesthesiology, Perioperative & Pain Medicine, Stanford University School of Medicine, Stanford, CA, USA
- Department of Pathology and Neuropathology, University Hospital and Comprehensive Cancer Center Tübingen, Tübingen, Germany
| | - Alexander Rochwarger
- Department of Pathology and Neuropathology, University Hospital and Comprehensive Cancer Center Tübingen, Tübingen, Germany
| | - Sven Mattern
- Department of Pathology and Neuropathology, University Hospital and Comprehensive Cancer Center Tübingen, Tübingen, Germany
| | - Brice Gaudillière
- Department of Anaesthesiology, Perioperative & Pain Medicine, Stanford University School of Medicine, Stanford, CA, USA
| | - Christian M Schürch
- Department of Pathology and Neuropathology, University Hospital and Comprehensive Cancer Center Tübingen, Tübingen, Germany.
| |
Collapse
|
134
|
Zhang S, Mu W, Dong D, Wei J, Fang M, Shao L, Zhou Y, He B, Zhang S, Liu Z, Liu J, Tian J. The Applications of Artificial Intelligence in Digestive System Neoplasms: A Review. HEALTH DATA SCIENCE 2023; 3:0005. [PMID: 38487199 PMCID: PMC10877701 DOI: 10.34133/hds.0005] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 09/05/2022] [Accepted: 12/05/2022] [Indexed: 03/17/2024]
Abstract
Importance Digestive system neoplasms (DSNs) are the leading cause of cancer-related mortality with a 5-year survival rate of less than 20%. Subjective evaluation of medical images including endoscopic images, whole slide images, computed tomography images, and magnetic resonance images plays a vital role in the clinical practice of DSNs, but with limited performance and increased workload of radiologists or pathologists. The application of artificial intelligence (AI) in medical image analysis holds promise to augment the visual interpretation of medical images, which could not only automate the complicated evaluation process but also convert medical images into quantitative imaging features that associated with tumor heterogeneity. Highlights We briefly introduce the methodology of AI for medical image analysis and then review its clinical applications including clinical auxiliary diagnosis, assessment of treatment response, and prognosis prediction on 4 typical DSNs including esophageal cancer, gastric cancer, colorectal cancer, and hepatocellular carcinoma. Conclusion AI technology has great potential in supporting the clinical diagnosis and treatment decision-making of DSNs. Several technical issues should be overcome before its application into clinical practice of DSNs.
Collapse
Affiliation(s)
- Shuaitong Zhang
- School of Engineering Medicine, Beihang University, Beijing, China
- Key Laboratory of Big Data-Based Precision Medicine, Beihang University, Ministry of Industry and Information Technology, Beijing, China
| | - Wei Mu
- School of Engineering Medicine, Beihang University, Beijing, China
- Key Laboratory of Big Data-Based Precision Medicine, Beihang University, Ministry of Industry and Information Technology, Beijing, China
| | - Di Dong
- CAS Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing, China
| | - Jingwei Wei
- CAS Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing, China
| | - Mengjie Fang
- School of Engineering Medicine, Beihang University, Beijing, China
- Key Laboratory of Big Data-Based Precision Medicine, Beihang University, Ministry of Industry and Information Technology, Beijing, China
| | - Lizhi Shao
- CAS Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing, China
| | - Yu Zhou
- CAS Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing, China
| | - Bingxi He
- School of Engineering Medicine, Beihang University, Beijing, China
- Key Laboratory of Big Data-Based Precision Medicine, Beihang University, Ministry of Industry and Information Technology, Beijing, China
| | - Song Zhang
- CAS Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing, China
| | - Zhenyu Liu
- CAS Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing, China
| | - Jianhua Liu
- Department of Oncology, Guangdong Provincial People's Hospital/Second Clinical Medical College of Southern Medical University/Guangdong Academy of Medical Sciences, Guangzhou, Guangdong, China
| | - Jie Tian
- School of Engineering Medicine, Beihang University, Beijing, China
- Key Laboratory of Big Data-Based Precision Medicine, Beihang University, Ministry of Industry and Information Technology, Beijing, China
- CAS Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing, China
| |
Collapse
|
135
|
Amin MS, Ahn H. FabNet: A Features Agglomeration-Based Convolutional Neural Network for Multiscale Breast Cancer Histopathology Images Classification. Cancers (Basel) 2023; 15:cancers15041013. [PMID: 36831359 PMCID: PMC9954749 DOI: 10.3390/cancers15041013] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/12/2023] [Revised: 01/31/2023] [Accepted: 01/31/2023] [Indexed: 02/08/2023] Open
Abstract
The definitive diagnosis of histology specimen images is largely based on the radiologist's comprehensive experience; however, due to the fine to the coarse visual appearance of such images, experts often disagree with their assessments. Sophisticated deep learning approaches can help to automate the diagnosis process of the images and reduce the analysis duration. More efficient and accurate automated systems can also increase the diagnostic impartiality by reducing the difference between the operators. We propose a FabNet model that can learn the fine-to-coarse structural and textural features of multi-scale histopathological images by using accretive network architecture that agglomerate hierarchical feature maps to acquire significant classification accuracy. We expand on a contemporary design by incorporating deep and close integration to finely combine features across layers. Our deep layer accretive model structure combines the feature hierarchy in an iterative and hierarchically manner that infers higher accuracy and fewer parameters. The FabNet can identify malignant tumors from images and patches from histopathology images. We assessed the efficiency of our suggested model standard cancer datasets, which included breast cancer as well as colon cancer histopathology images. Our proposed avant garde model significantly outperforms existing state-of-the-art models in respect of the accuracy, F1 score, precision, and sensitivity, with fewer parameters.
Collapse
|
136
|
Khanal B, Hasan SMK, Khanal B, Linte CA. Investigating the impact of class-dependent label noise in medical image classification. PROCEEDINGS OF SPIE--THE INTERNATIONAL SOCIETY FOR OPTICAL ENGINEERING 2023; 12464:1246437. [PMID: 37123015 PMCID: PMC10134892 DOI: 10.1117/12.2654420] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/02/2023]
Abstract
Label noise is inevitable in medical image databases developed for deep learning due to the inter-observer variability caused by the different levels of expertise of the experts annotating the images, and, in some cases, the automated methods that generate labels from medical reports. It is known that incorrect annotations or label noise can degrade the actual performance of supervised deep learning models and can bias the model's evaluation. Existing literature show that noise in one class has minimal impact on the model's performance for another class in natural image classification problems where different target classes have a relatively distinct shape and share minimal visual cues for knowledge transfer among the classes. However, it is not clear how class-dependent label noise affects the model's performance when operating on medical images, for which different output classes can be difficult to distinguish even for experts, and there is a high possibility of knowledge transfer across classes during the training period. We hypothesize that for medical image classification tasks where the different classes share a very similar shape with differences only in texture, the noisy label for one class might affect the performance across other classes, unlike the case when the target classes have different shapes and are visually distinct. In this paper, we study this hypothesis using two publicly available datasets: a 2D organ classification dataset with target organ classes being visually distinct, and a histopathology image classification dataset where the target classes look very similar visually. Our results show that the label noise in one class has a much higher impact on the model's performance on other classes for the histopathology dataset compared to the organ dataset.
Collapse
Affiliation(s)
- Bidur Khanal
- Center for Imaging Science, Rochester Institute of Technology, NY, USA
- (✉)Further author information: Bidur Khanal (), S. M. Kamrul Hasan ()
| | - S. M. Kamrul Hasan
- Center for Imaging Science, Rochester Institute of Technology, NY, USA
- (✉)Further author information: Bidur Khanal (), S. M. Kamrul Hasan ()
| | - Bishesh Khanal
- NepAl Applied Mathematics and Informatics Institute for research (NAAMII), KTM, Nepal
| | - Cristian A. Linte
- Center for Imaging Science, Rochester Institute of Technology, NY, USA
- Biomedical Engineering, Rochester Institute of Technology, NY, USA
| |
Collapse
|
137
|
Kindler C, Elfwing S, Öhrvik J, Nikberg M. A Deep Neural Network-Based Decision Support Tool for the Detection of Lymph Node Metastases in Colorectal Cancer Specimens. Mod Pathol 2023; 36:100015. [PMID: 36853787 DOI: 10.1016/j.modpat.2022.100015] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/28/2022] [Revised: 09/13/2022] [Accepted: 09/16/2022] [Indexed: 01/11/2023]
Abstract
The identification of lymph node metastases in colorectal cancer (CRC) specimens is crucial for the planning of postoperative treatment and can be a time-consuming task for pathologists. In this study, we developed a deep neural network (DNN) algorithm for the detection of metastatic CRC in digitized histologic sections of lymph nodes and evaluated its performance as a diagnostic support tool. First, the DNN algorithm was trained using pixel-level annotations of cancerous areas on 758 whole slide images (360 with cancerous areas). The algorithm's performance was evaluated on 74 whole slide images (43 with cancerous areas). Second, the algorithm was evaluated as a decision support tool on 288 whole slide images covering 1517 lymph node sections, randomized in 16 batches. Two senior pathologists (C.K. and C.O.) assessed each batch with and without the help of the algorithm in a 2 × 2 crossover design, with a washout period of 1 month in between. The time needed for the evaluation of each node section was recorded. The DNN algorithm achieved a median pixel-level accuracy of 0.952 on slides with cancerous areas and 0.996 on slides with benign samples. N+ disease (metastases, micrometastases, or tumor deposits) was present in 103 of the 1517 sections. The algorithm highlighted cancerous areas in 102 of these sections, with a sensitivity of 0.990. Assisted by the algorithm, the median time needed for evaluation was significantly shortened for both pathologists (median time for pathologist 1, 26 vs 14 seconds; P < .001; 95% CI, 11.0-12.0; median time for pathologist 2, 25 vs 23 seconds; P < .001; 95% CI, 2.0-4.0). Our DNN showed high accuracy for detecting metastatic CRC in digitized histologic sections of lymph nodes. This decision support tool has the potential to improve the diagnostic workflow by shortening the time needed for the evaluation of lymph nodes in CRC specimens without impairing diagnostic accuracy.
Collapse
Affiliation(s)
- Csaba Kindler
- Department of Pathology, Laboratory Medicine, Västmanlands Hospital, Västerås, Sweden; Centre for Clinical Research, Uppsala University, Västerås, Sweden.
| | | | - John Öhrvik
- Centre for Clinical Research, Uppsala University, Västerås, Sweden
| | - Maziar Nikberg
- Centre for Clinical Research, Uppsala University, Västerås, Sweden; Department of Surgery, Västmanlands Hospital, Västerås, Sweden
| |
Collapse
|
138
|
Mansur A, Saleem Z, Elhakim T, Daye D. Role of artificial intelligence in risk prediction, prognostication, and therapy response assessment in colorectal cancer: current state and future directions. Front Oncol 2023; 13:1065402. [PMID: 36761957 PMCID: PMC9905815 DOI: 10.3389/fonc.2023.1065402] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/09/2022] [Accepted: 01/09/2023] [Indexed: 01/26/2023] Open
Abstract
Artificial Intelligence (AI) is a branch of computer science that utilizes optimization, probabilistic and statistical approaches to analyze and make predictions based on a vast amount of data. In recent years, AI has revolutionized the field of oncology and spearheaded novel approaches in the management of various cancers, including colorectal cancer (CRC). Notably, the applications of AI to diagnose, prognosticate, and predict response to therapy in CRC, is gaining traction and proving to be promising. There have also been several advancements in AI technologies to help predict metastases in CRC and in Computer-Aided Detection (CAD) Systems to improve miss rates for colorectal neoplasia. This article provides a comprehensive review of the role of AI in predicting risk, prognosis, and response to therapies among patients with CRC.
Collapse
Affiliation(s)
- Arian Mansur
- Harvard Medical School, Boston, MA, United States
| | | | - Tarig Elhakim
- Department of Radiology, Massachusetts General Hospital, Boston, MA, United States
| | - Dania Daye
- Department of Radiology, Massachusetts General Hospital, Boston, MA, United States,*Correspondence: Dania Daye,
| |
Collapse
|
139
|
Using deep learning to predict survival outcome in non-surgical cervical cancer patients based on pathological images. J Cancer Res Clin Oncol 2023:10.1007/s00432-022-04446-8. [PMID: 36653539 PMCID: PMC10356676 DOI: 10.1007/s00432-022-04446-8] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/31/2022] [Accepted: 10/19/2022] [Indexed: 01/20/2023]
Abstract
PURPOSE We analyzed clinical features and the representative HE-stained pathologic images to predict 5-year overall survival via the deep-learning approach in cervical cancer patients in order to assist oncologists in designing the optimal treatment strategies. METHODS The research retrospectively collected 238 non-surgical cervical cancer patients treated with radiochemotherapy from 2014 to 2017. These patients were randomly divided into the training set (n = 165) and test set (n = 73). Then, we extract deep features after segmenting the HE-stained image into patches of size 224 × 224. A Lasso-Cox model was constructed with clinical data to predict 5-year OS. C-index evaluated this model performance with 95% CI, calibration curve, and ROC. RESULTS Based on multivariate analysis, 2 of 11 clinical characteristics (C-index 0.68) and 2 of 2048 pathomic features (C-index 0.74) and clinical-pathomic model (C-index 0.83) of nomograms predict 5-year survival in the training set, respectively. In test set, compared with the pathomic and clinical characteristics used alone, the clinical-pathomic model had an AUC of 0.750 (95% CI 0.540-0.959), the clinical predictor model had an AUC of 0.729 (95% CI 0.551-0.909), and the pathomic model AUC was 0.703 (95% CI 0.487-0.919). Based on appropriate nomogram scores, we divided patients into high-risk and low-risk groups, and Kaplan-Meier survival probability curves for both groups showed statistical differences. CONCLUSION We built a clinical-pathomic model to predict 5-year OS in non-surgical cervical cancer patients, which may be a promising method to improve the precision of personalized therapy.
Collapse
|
140
|
Yang J, Shi R, Wei D, Liu Z, Zhao L, Ke B, Pfister H, Ni B. MedMNIST v2 - A large-scale lightweight benchmark for 2D and 3D biomedical image classification. Sci Data 2023; 10:41. [PMID: 36658144 PMCID: PMC9852451 DOI: 10.1038/s41597-022-01721-8] [Citation(s) in RCA: 14] [Impact Index Per Article: 14.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/29/2021] [Accepted: 09/26/2022] [Indexed: 01/20/2023] Open
Abstract
We introduce MedMNIST v2, a large-scale MNIST-like dataset collection of standardized biomedical images, including 12 datasets for 2D and 6 datasets for 3D. All images are pre-processed into a small size of 28 × 28 (2D) or 28 × 28 × 28 (3D) with the corresponding classification labels so that no background knowledge is required for users. Covering primary data modalities in biomedical images, MedMNIST v2 is designed to perform classification on lightweight 2D and 3D images with various dataset scales (from 100 to 100,000) and diverse tasks (binary/multi-class, ordinal regression, and multi-label). The resulting dataset, consisting of 708,069 2D images and 9,998 3D images in total, could support numerous research/educational purposes in biomedical image analysis, computer vision, and machine learning. We benchmark several baseline methods on MedMNIST v2, including 2D/3D neural networks and open-source/commercial AutoML tools. The data and code are publicly available at https://medmnist.com/ .
Collapse
Affiliation(s)
- Jiancheng Yang
- grid.16821.3c0000 0004 0368 8293Shanghai Jiao Tong University, Shanghai, China
| | - Rui Shi
- grid.16821.3c0000 0004 0368 8293Shanghai Jiao Tong University, Shanghai, China
| | - Donglai Wei
- grid.208226.c0000 0004 0444 7053Boston College, Chestnut Hill, MA USA
| | - Zequan Liu
- grid.1957.a0000 0001 0728 696XRWTH Aachen University, Aachen, Germany
| | - Lin Zhao
- grid.8547.e0000 0001 0125 2443Department of Endocrinology and Metabolism, Fudan Institute of Metabolic Diseases, Zhongshan Hospital, Fudan University, Shanghai, China
| | - Bilian Ke
- grid.16821.3c0000 0004 0368 8293Department of Ophthalmology, Shanghai General Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | | | - Bingbing Ni
- grid.16821.3c0000 0004 0368 8293Shanghai Jiao Tong University, Shanghai, China
| |
Collapse
|
141
|
Wang Z, Lu H, Wu Y, Ren S, Diaty DM, Fu Y, Zou Y, Zhang L, Wang Z, Wang F, Li S, Huo X, Yu W, Xu J, Ye Z. Predicting recurrence in osteosarcoma via a quantitative histological image classifier derived from tumour nuclear morphological features. CAAI TRANSACTIONS ON INTELLIGENCE TECHNOLOGY 2023. [DOI: 10.1049/cit2.12175] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/18/2023] Open
Affiliation(s)
- Zhan Wang
- Department of Orthopedic Surgery The Second Affiliated Hospital Zhejiang University School of Medicine Hangzhou China
- Orthopedics Research Institute of Zhejiang University Hangzhou China
- Key Laboratory of Motor System Disease Research and Precision Therapy of Zhejiang Province Hangzhou China
- Clinical Research Center of Motor System Disease of Zhejiang Province Hangzhou China
| | - Haoda Lu
- Institute for AI in Medicine School of Artificial Intelligence, Nanjing University of Information Science & Technology Nanjing China
| | - Yan Wu
- Department of Orthopedic Surgery The Second Affiliated Hospital Zhejiang University School of Medicine Hangzhou China
- Orthopedics Research Institute of Zhejiang University Hangzhou China
- Key Laboratory of Motor System Disease Research and Precision Therapy of Zhejiang Province Hangzhou China
- Clinical Research Center of Motor System Disease of Zhejiang Province Hangzhou China
| | - Shihong Ren
- Department of Orthopedic Surgery The Second Affiliated Hospital Zhejiang University School of Medicine Hangzhou China
| | - Diarra mohamed Diaty
- Department of Orthopedic Surgery The Second Affiliated Hospital Zhejiang University School of Medicine Hangzhou China
- Orthopedics Research Institute of Zhejiang University Hangzhou China
- Key Laboratory of Motor System Disease Research and Precision Therapy of Zhejiang Province Hangzhou China
- Clinical Research Center of Motor System Disease of Zhejiang Province Hangzhou China
| | - Yanbiao Fu
- Department of Pathology The Second Affiliated Hospital Zhejiang University School of Medicine Hangzhou China
| | - Yi Zou
- Department of Pathology The Second Affiliated Hospital Zhejiang University School of Medicine Hangzhou China
| | - Lingling Zhang
- Department of Orthopedic Surgery The Second Affiliated Hospital Zhejiang University School of Medicine Hangzhou China
- Orthopedics Research Institute of Zhejiang University Hangzhou China
- Key Laboratory of Motor System Disease Research and Precision Therapy of Zhejiang Province Hangzhou China
- Clinical Research Center of Motor System Disease of Zhejiang Province Hangzhou China
| | - Zenan Wang
- Department of Orthopedic Surgery The Second Affiliated Hospital Zhejiang University School of Medicine Hangzhou China
| | - Fangqian Wang
- Department of Orthopedic Surgery The Second Affiliated Hospital Zhejiang University School of Medicine Hangzhou China
| | - Shu Li
- Department of Hematology Shanghai General Hospital Shanghai Jiao Tong University School of Medicine Shanghai China
| | - Xinmi Huo
- Bioinformatics Institute (BII) Agency for Science, Technology and Research (A*STAR) Singapore Singapore
| | - Weimiao Yu
- Bioinformatics Institute (BII) Agency for Science, Technology and Research (A*STAR) Singapore Singapore
| | - Jun Xu
- Institute for AI in Medicine School of Artificial Intelligence, Nanjing University of Information Science & Technology Nanjing China
| | - Zhaoming Ye
- Department of Orthopedic Surgery The Second Affiliated Hospital Zhejiang University School of Medicine Hangzhou China
- Orthopedics Research Institute of Zhejiang University Hangzhou China
- Key Laboratory of Motor System Disease Research and Precision Therapy of Zhejiang Province Hangzhou China
- Clinical Research Center of Motor System Disease of Zhejiang Province Hangzhou China
| |
Collapse
|
142
|
Capobianco E, Dominietto M. Translating Data Science Results into Precision Oncology Decisions: A Mini Review. J Clin Med 2023; 12:438. [PMID: 36675367 PMCID: PMC9862106 DOI: 10.3390/jcm12020438] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/08/2022] [Revised: 12/29/2022] [Accepted: 01/03/2023] [Indexed: 01/06/2023] Open
Abstract
While reviewing and discussing the potential of data science in oncology, we emphasize medical imaging and radiomics as the leading contextual frameworks to measure the impacts of Artificial Intelligence (AI) and Machine Learning (ML) developments. We envision some domains and research directions in which radiomics should become more significant in view of current barriers and limitations.
Collapse
Affiliation(s)
- Enrico Capobianco
- The Jackson Laboratory, 10 Discovery Drive, Farmington, CT 06032, USA
| | - Marco Dominietto
- Paul Scherrer Institut, Forschungsstrasse 111, 5232 Villigen, Switzerland
| |
Collapse
|
143
|
Kumar A, Vishwakarma A, Bajaj V. CRCCN-Net: Automated framework for classification of colorectal tissue using histopathological images. Biomed Signal Process Control 2023. [DOI: 10.1016/j.bspc.2022.104172] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/14/2022]
|
144
|
Yin Z, Yao C, Zhang L, Qi S. Application of artificial intelligence in diagnosis and treatment of colorectal cancer: A novel Prospect. Front Med (Lausanne) 2023; 10:1128084. [PMID: 36968824 PMCID: PMC10030915 DOI: 10.3389/fmed.2023.1128084] [Citation(s) in RCA: 14] [Impact Index Per Article: 14.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/20/2022] [Accepted: 02/13/2023] [Indexed: 03/29/2023] Open
Abstract
In the past few decades, according to the rapid development of information technology, artificial intelligence (AI) has also made significant progress in the medical field. Colorectal cancer (CRC) is the third most diagnosed cancer worldwide, and its incidence and mortality rates are increasing yearly, especially in developing countries. This article reviews the latest progress in AI in diagnosing and treating CRC based on a systematic collection of previous literature. Most CRCs transform from polyp mutations. The computer-aided detection systems can significantly improve the polyp and adenoma detection rate by early colonoscopy screening, thereby lowering the possibility of mutating into CRC. Machine learning and bioinformatics analysis can help screen and identify more CRC biomarkers to provide the basis for non-invasive screening. The Convolutional neural networks can assist in reading histopathologic tissue images, reducing the experience difference among doctors. Various studies have shown that AI-based high-level auxiliary diagnostic systems can significantly improve the readability of medical images and help clinicians make more accurate diagnostic and therapeutic decisions. Moreover, Robotic surgery systems such as da Vinci have been more and more commonly used to treat CRC patients, according to their precise operating performance. The application of AI in neoadjuvant chemoradiotherapy has further improved the treatment and efficacy evaluation of CRC. In addition, AI represented by deep learning in gene sequencing research offers a new treatment option. All of these things have seen that AI has a promising prospect in the era of precision medicine.
Collapse
Affiliation(s)
- Zugang Yin
- Department of General Surgery, The First Affiliated Hospital of Dalian Medical University, Dalian, China
| | - Chenhui Yao
- Department of General Surgery, The First Affiliated Hospital of Dalian Medical University, Dalian, China
- *Correspondence: Chenhui Yao,
| | - Limin Zhang
- Department of Respiratory, The First Affiliated Hospital of Dalian Medical University, Dalian, China
| | - Shaohua Qi
- Institute of Laboratory Animal Science, Chinese Academy of Medical Sciences and Comparative Medicine Center, Peking Union Medical College, Beijing, China
| |
Collapse
|
145
|
Du J, Huang M, Liu L. AI-Aided Disease Prediction in Visualized Medicine. ADVANCES IN EXPERIMENTAL MEDICINE AND BIOLOGY 2023; 1199:107-126. [PMID: 37460729 DOI: 10.1007/978-981-32-9902-3_6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 07/20/2023]
Abstract
Artificial intelligence (AI) is playing a vitally important role in promoting the revolution of future technology. Healthcare is one of the promising applications in AI, which covers medical imaging, diagnosis, robotics, disease prediction, pharmacy, health management, and hospital management. Numbers of achievements that made in these fields overturn every aspect in traditional healthcare system. Therefore, to understand the state-of-art AI in healthcare, as well as the chances and obstacles in its development, the applications of AI in disease detection and outlook and the future trends of AI-aided disease prediction were discussed in this chapter.
Collapse
Affiliation(s)
- Juan Du
- Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin, China.
| | - Mengen Huang
- Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin, China
| | - Lin Liu
- Tianjin Key Laboratory of Retinal Functions and Diseases, Eye Institute and School of Optometry, Tianjin Medical University Eye Hospital, Tianjin, China
| |
Collapse
|
146
|
Celard P, Iglesias EL, Sorribes-Fdez JM, Romero R, Vieira AS, Borrajo L. A survey on deep learning applied to medical images: from simple artificial neural networks to generative models. Neural Comput Appl 2023; 35:2291-2323. [PMID: 36373133 PMCID: PMC9638354 DOI: 10.1007/s00521-022-07953-4] [Citation(s) in RCA: 9] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/23/2022] [Accepted: 10/12/2022] [Indexed: 11/06/2022]
Abstract
Deep learning techniques, in particular generative models, have taken on great importance in medical image analysis. This paper surveys fundamental deep learning concepts related to medical image generation. It provides concise overviews of studies which use some of the latest state-of-the-art models from last years applied to medical images of different injured body areas or organs that have a disease associated with (e.g., brain tumor and COVID-19 lungs pneumonia). The motivation for this study is to offer a comprehensive overview of artificial neural networks (NNs) and deep generative models in medical imaging, so more groups and authors that are not familiar with deep learning take into consideration its use in medicine works. We review the use of generative models, such as generative adversarial networks and variational autoencoders, as techniques to achieve semantic segmentation, data augmentation, and better classification algorithms, among other purposes. In addition, a collection of widely used public medical datasets containing magnetic resonance (MR) images, computed tomography (CT) scans, and common pictures is presented. Finally, we feature a summary of the current state of generative models in medical image including key features, current challenges, and future research paths.
Collapse
Affiliation(s)
- P. Celard
- Computer Science Department, Universidade de Vigo, Escuela Superior de Ingeniería Informática, Campus Universitario As Lagoas, 32004 Ourense, Spain ,CINBIO - Biomedical Research Centre, Universidade de Vigo, Campus Universitario Lagoas-Marcosende, 36310 Vigo, Spain ,SING Research Group, Galicia Sur Health Research Institute (IIS Galicia Sur), SERGAS-UVIGO, Vigo, Spain
| | - E. L. Iglesias
- Computer Science Department, Universidade de Vigo, Escuela Superior de Ingeniería Informática, Campus Universitario As Lagoas, 32004 Ourense, Spain ,CINBIO - Biomedical Research Centre, Universidade de Vigo, Campus Universitario Lagoas-Marcosende, 36310 Vigo, Spain ,SING Research Group, Galicia Sur Health Research Institute (IIS Galicia Sur), SERGAS-UVIGO, Vigo, Spain
| | - J. M. Sorribes-Fdez
- Computer Science Department, Universidade de Vigo, Escuela Superior de Ingeniería Informática, Campus Universitario As Lagoas, 32004 Ourense, Spain ,CINBIO - Biomedical Research Centre, Universidade de Vigo, Campus Universitario Lagoas-Marcosende, 36310 Vigo, Spain ,SING Research Group, Galicia Sur Health Research Institute (IIS Galicia Sur), SERGAS-UVIGO, Vigo, Spain
| | - R. Romero
- Computer Science Department, Universidade de Vigo, Escuela Superior de Ingeniería Informática, Campus Universitario As Lagoas, 32004 Ourense, Spain ,CINBIO - Biomedical Research Centre, Universidade de Vigo, Campus Universitario Lagoas-Marcosende, 36310 Vigo, Spain ,SING Research Group, Galicia Sur Health Research Institute (IIS Galicia Sur), SERGAS-UVIGO, Vigo, Spain
| | - A. Seara Vieira
- Computer Science Department, Universidade de Vigo, Escuela Superior de Ingeniería Informática, Campus Universitario As Lagoas, 32004 Ourense, Spain ,CINBIO - Biomedical Research Centre, Universidade de Vigo, Campus Universitario Lagoas-Marcosende, 36310 Vigo, Spain ,SING Research Group, Galicia Sur Health Research Institute (IIS Galicia Sur), SERGAS-UVIGO, Vigo, Spain
| | - L. Borrajo
- Computer Science Department, Universidade de Vigo, Escuela Superior de Ingeniería Informática, Campus Universitario As Lagoas, 32004 Ourense, Spain ,CINBIO - Biomedical Research Centre, Universidade de Vigo, Campus Universitario Lagoas-Marcosende, 36310 Vigo, Spain ,SING Research Group, Galicia Sur Health Research Institute (IIS Galicia Sur), SERGAS-UVIGO, Vigo, Spain
| |
Collapse
|
147
|
Graham S, Vu QD, Jahanifar M, Raza SEA, Minhas F, Snead D, Rajpoot N. One model is all you need: Multi-task learning enables simultaneous histology image segmentation and classification. Med Image Anal 2023; 83:102685. [PMID: 36410209 DOI: 10.1016/j.media.2022.102685] [Citation(s) in RCA: 7] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/28/2022] [Revised: 10/20/2022] [Accepted: 11/03/2022] [Indexed: 11/13/2022]
Abstract
The recent surge in performance for image analysis of digitised pathology slides can largely be attributed to the advances in deep learning. Deep models can be used to initially localise various structures in the tissue and hence facilitate the extraction of interpretable features for biomarker discovery. However, these models are typically trained for a single task and therefore scale poorly as we wish to adapt the model for an increasing number of different tasks. Also, supervised deep learning models are very data hungry and therefore rely on large amounts of training data to perform well. In this paper, we present a multi-task learning approach for segmentation and classification of nuclei, glands, lumina and different tissue regions that leverages data from multiple independent data sources. While ensuring that our tasks are aligned by the same tissue type and resolution, we enable meaningful simultaneous prediction with a single network. As a result of feature sharing, we also show that the learned representation can be used to improve the performance of additional tasks via transfer learning, including nuclear classification and signet ring cell detection. As part of this work, we train our developed Cerberus model on a huge amount of data, consisting of over 600 thousand objects for segmentation and 440 thousand patches for classification. We use our approach to process 599 colorectal whole-slide images from TCGA, where we localise 377 million, 900 thousand and 2.1 million nuclei, glands and lumina respectively. We make this resource available to remove a major barrier in the development of explainable models for computational pathology.
Collapse
Affiliation(s)
- Simon Graham
- Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, United Kingdom; Histofy Ltd, United Kingdom.
| | - Quoc Dang Vu
- Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, United Kingdom
| | - Mostafa Jahanifar
- Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, United Kingdom
| | - Shan E Ahmed Raza
- Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, United Kingdom
| | - Fayyaz Minhas
- Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, United Kingdom
| | - David Snead
- Histofy Ltd, United Kingdom; Department of Pathology, University Hospitals Coventry & Warwickshire, United Kingdom
| | - Nasir Rajpoot
- Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, United Kingdom; Histofy Ltd, United Kingdom; Department of Pathology, University Hospitals Coventry & Warwickshire, United Kingdom
| |
Collapse
|
148
|
Computer-aided detection and prognosis of colorectal cancer on whole slide images using dual resolution deep learning. J Cancer Res Clin Oncol 2023; 149:91-101. [PMID: 36331654 DOI: 10.1007/s00432-022-04435-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/14/2022] [Accepted: 10/18/2022] [Indexed: 11/06/2022]
Abstract
PURPOSE Rapid diagnosis and risk stratification can provide timely treatment for colorectal cancer (CRC) patients. Deep learning (DL) is not only used to identify tumor regions in histopathological images, but also applied to predict survival and achieve risk stratification. Whereas, most of methods dependent on regions of interest annotated by pathologist and ignore the global information in the image. METHODS A dual resolution DL network based on weakly supervised learning (WDRNet) was proposed for CRC identification and prognosis. The proposed method was trained and validated on the dataset from The Cancer Genome Atlas (TCGA) and tested on the external dataset from Affiliated Cancer Hospital and Institute of Guangzhou Medical University (ACHIGMU). RESULTS In identification task, WDRNet accurately identified tumor images with an accuracy of 0.977 in slide level and 0.953 in patch level. Besides, in prognosis task, WDRNet showed an excellent prediction performance in both datasets with the concordance index (C-index) of 0.716 ± 0.037 and 0.598 ± 0.024 respectively. Moreover, the results of risk stratification were statistically significant in univariate analysis (p < 0.001, HR = 7.892 in TCGA-CRC, and p = 0.009, HR = 1.718 in ACHIGMU) and multivariate analysis (p < 0.001, HR = 5.914 in TCGA-CRC, and p = 0.025, HR = 1.674 in ACHIGMU). CONCLUSIONS We developed a weakly supervised resolution DL network to achieve precise identification and prognosis of CRC patients, which will assist doctors in diagnosis on histopathological images and stratify patients to select appropriate therapeutic schedule.
Collapse
|
149
|
Yavuz A, Alpsoy A, Gedik EO, Celik MY, Bassorgun CI, Unal B, Elpek GO. Artificial intelligence applications in predicting the behavior of gastrointestinal cancers in pathology. Artif Intell Gastroenterol 2022; 3:142-162. [DOI: 10.35712/aig.v3.i5.142] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/16/2022] [Revised: 11/25/2022] [Accepted: 12/14/2022] [Indexed: 12/28/2022] Open
Abstract
Recent research has provided a wealth of data supporting the application of artificial intelligence (AI)-based applications in routine pathology practice. Indeed, it is clear that these methods can significantly support an accurate and rapid diagnosis by eliminating errors, increasing reliability, and improving workflow. In addition, the effectiveness of AI in the pathological evaluation of prognostic parameters associated with behavior, course, and treatment in many types of tumors has also been noted. Regarding gastrointestinal system (GIS) cancers, the contribution of AI methods to pathological diagnosis has been investigated in many studies. On the other hand, studies focusing on AI applications in evaluating parameters to determine tumor behavior are relatively few. For this purpose, the potential of AI models has been studied over a broad spectrum, from tumor subtyping to the identification of new digital biomarkers. The capacity of AI to infer genetic alterations of cancer tissues from digital slides has been demonstrated. Although current data suggest the merit of AI-based approaches in assessing tumor behavior in GIS cancers, a wide range of challenges still need to be solved, from laboratory infrastructure to improving the robustness of algorithms, before incorporating AI applications into real-life GIS pathology practice. This review aims to present data from AI applications in evaluating pathological parameters related to the behavior of GIS cancer with an overview of the opportunities and challenges encountered in implementing AI in pathology.
Collapse
Affiliation(s)
- Aysen Yavuz
- Department of Pathology, Akdeniz University Medical School, Antalya 07070, Turkey
| | - Anil Alpsoy
- Department of Pathology, Akdeniz University Medical School, Antalya 07070, Turkey
| | - Elif Ocak Gedik
- Department of Pathology, Akdeniz University Medical School, Antalya 07070, Turkey
| | | | | | - Betul Unal
- Department of Pathology, Akdeniz University Medical School, Antalya 07070, Turkey
| | - Gulsum Ozlem Elpek
- Department of Pathology, Akdeniz University Medical School, Antalya 07070, Turkey
| |
Collapse
|
150
|
Zhang H, Yang L, Han Q, Xu W. Antifibrotic effects of Quercetin on TGF-β1-induced vocal fold fibroblasts. Am J Transl Res 2022; 14:8552-8561. [PMID: 36628236 PMCID: PMC9827314] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/10/2022] [Accepted: 08/04/2022] [Indexed: 01/12/2023]
Abstract
OBJECTIVE To investigate the effects of Quercetin on vocal fold fibroblasts induced by TGF-β1 and vocal fold injury. METHODS The effects of Quercetin on collagen type I (COL-I), collagen type III (COL-III), and fibronectin (FN) expressions in transforming growth factor-β1 (TGF-β1)-induced human vocal fold fibroblasts were examined by the quantitative reverse transcription-polymerase chain reaction and the enzyme-linked immunosorbent assay. The Cell Counting Kit-8 was used to assess the influence of Quercetin on cell proliferation. A rat vocal fold injury model was developed. These rats were divided into a control group, a model group, and a Quercetin group. Pathorphological observations from Hematoxylin-eosin staining, the density of fibroblasts, and hypertrophic index were compared among the three groups. RESULTS Compared with cells untreated with TGF-β1, the mRNA and protein expression levels of COL-I, COL-II, and FN and proliferation ability in TGF-β1-induced human vocal fold fibroblasts were increased (all P<0.05). Compared with human vocal fold fibroblasts induced by TGF-β1, the mRNA and protein expression levels of COL-I, COL-II, and FN and proliferation ability in TGF-β1-induced human vocal fold fibroblasts following Quercetin stimulation were decreased (all P<0.05). There were no statistical differences between human vocal fold fibroblasts treated with or without Quercetin regarding cell proliferation and COL-I, COL-II, and FN expressions. Compared with the model group, Quercetin suppressed the fibrosis of the vocal fold following injury and the protein levels of COL-I, COL-II, and FN in tissue samples (all P<0.001). The density of fibroblasts and hypertrophic index in the vocal fold scar from the Quercetin group were lower than those in the model group (all P<0.05). CONCLUSIONS Quercetin inhibited TGF-β1-induced fibrotic changes and proliferation in human vocal fold fibroblasts and had an antifibrotic effect in vocal fold after injuries.
Collapse
Affiliation(s)
- Haiyan Zhang
- Department of Otorhinolaryngology Head and Neck Surgery, Shandong Provincial ENT Hospital, Shandong UniversityJinan, Shandong Province, China,Shandong Institute of OtorhinolaryngologyJinan, Shandong Province, China
| | - Linxue Yang
- Department of Otorhinolaryngology Head and Neck Surgery, Shandong Provincial ENT Hospital, Shandong UniversityJinan, Shandong Province, China,Shandong Institute of OtorhinolaryngologyJinan, Shandong Province, China
| | - Qianqian Han
- Department of Otorhinolaryngology Head and Neck Surgery, Shandong Provincial ENT Hospital, Shandong UniversityJinan, Shandong Province, China,Shandong Institute of OtorhinolaryngologyJinan, Shandong Province, China
| | - Wei Xu
- Department of Otorhinolaryngology Head and Neck Surgery, Shandong Provincial ENT Hospital, Shandong UniversityJinan, Shandong Province, China,Shandong Institute of OtorhinolaryngologyJinan, Shandong Province, China
| |
Collapse
|