1
|
Alsaafin A, Nejat P, Shafique A, Khan J, Alfasly S, Alabtah G, Tizhoosh HR. Sequential Patching Lattice for Image Classification and Enquiry: Streamlining Digital Pathology Image Processing. THE AMERICAN JOURNAL OF PATHOLOGY 2024; 194:1898-1912. [PMID: 39032601 DOI: 10.1016/j.ajpath.2024.06.007] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/19/2024] [Revised: 06/06/2024] [Accepted: 06/18/2024] [Indexed: 07/23/2024]
Abstract
Digital pathology and the integration of artificial intelligence (AI) models have revolutionized histopathology, opening new opportunities. With the increasing availability of whole-slide images (WSIs), demand is growing for efficient retrieval, processing, and analysis of relevant images from vast biomedical archives. However, processing WSIs presents challenges due to their large size and content complexity. Full computer digestion of WSIs is impractical, and processing all patches individually is prohibitively expensive. In this article, we propose an unsupervised patching algorithm, Sequential Patching Lattice for Image Classification and Enquiry (SPLICE). This novel approach condenses a histopathology WSI into a compact set of representative patches, forming a collage of WSI while minimizing redundancy. SPLICE prioritizes patch quality and uniqueness by sequentially analyzing a WSI and selecting nonredundant representative features. In search and match applications, SPLICE showed improved accuracy, reduced computation time, and storage requirements compared with existing state-of-the-art methods. As an unsupervised method, SPLICE effectively reduced storage requirements for representing tissue images by 50%. This reduction can enable numerous algorithms in computational pathology to operate much more efficiently, paving the way for accelerated adoption of digital pathology.
Collapse
Affiliation(s)
- Areej Alsaafin
- KIMIA Lab, Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, Minnesota
| | - Peyman Nejat
- KIMIA Lab, Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, Minnesota
| | - Abubakr Shafique
- KIMIA Lab, Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, Minnesota
| | - Jibran Khan
- KIMIA Lab, Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, Minnesota
| | - Saghir Alfasly
- KIMIA Lab, Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, Minnesota
| | - Ghazal Alabtah
- KIMIA Lab, Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, Minnesota
| | - Hamid R Tizhoosh
- KIMIA Lab, Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, Minnesota.
| |
Collapse
|
2
|
Iglesias G, Talavera E, Troya J, Díaz-Álvarez A, García-Remesal M. Artificial intelligence model for tumoral clinical decision support systems. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2024; 253:108228. [PMID: 38810378 DOI: 10.1016/j.cmpb.2024.108228] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/02/2023] [Revised: 04/21/2024] [Accepted: 05/13/2024] [Indexed: 05/31/2024]
Abstract
BACKGROUND AND OBJECTIVE Comparative diagnostic in brain tumor evaluation makes possible to use the available information of a medical center to compare similar cases when a new patient is evaluated. By leveraging Artificial Intelligence models, the proposed system is able of retrieving the most similar cases of brain tumors for a given query. The primary objective is to enhance the diagnostic process by generating more accurate representations of medical images, with a particular focus on patient-specific normal features and pathologies. A key distinction from previous models lies in its ability to produce enriched image descriptors solely from binary information, eliminating the need for costly and difficult to obtain tumor segmentation. METHODS The proposed model uses Artificial Intelligence to detect patient features to recommend the most similar cases from a database. The system not only suggests similar cases but also balances the representation of healthy and abnormal features in its design. This not only encourages the generalization of its use but also aids clinicians in their decision-making processes. This generalization makes possible for future research in different medical diagnosis areas with almost not any change in the system. RESULTS We conducted a comparative analysis of our approach in relation to similar studies. The proposed architecture obtains a Dice coefficient of 0.474 in both tumoral and healthy regions of the patients, which outperforms previous literature. Our proposed model excels at extracting and combining anatomical and pathological features from brain Magnetic Resonances (MRs), achieving state-of-the-art results while relying on less expensive label information. This substantially reduces the overall cost of the training process. Our findings highlight the significant potential for improving the efficiency and accuracy of comparative diagnostics and the treatment of tumoral pathologies. CONCLUSIONS This paper provides substantial grounds for further exploration of the broader applicability and optimization of the proposed architecture to enhance clinical decision-making. The novel approach presented in this work marks a significant advancement in the field of medical diagnosis, particularly in the context of Artificial Intelligence-assisted image retrieval, and promises to reduce costs and improve the quality of patient care using Artificial Intelligence as a support tool instead of a black box system.
Collapse
Affiliation(s)
- Guillermo Iglesias
- Departamento de Sistemas Informáticos, Escuela Técnica Superior de Ingeniería de Sistemas Informáticos, Universidad Politécnica de Madrid, Spain.
| | - Edgar Talavera
- Departamento de Sistemas Informáticos, Escuela Técnica Superior de Ingeniería de Sistemas Informáticos, Universidad Politécnica de Madrid, Spain.
| | - Jesús Troya
- Infanta Leonor University Hospital. Madrid., Spain
| | - Alberto Díaz-Álvarez
- Departamento de Sistemas Informáticos, Escuela Técnica Superior de Ingeniería de Sistemas Informáticos, Universidad Politécnica de Madrid, Spain.
| | - Miguel García-Remesal
- Biomedical Informatics Group, Departamento de Inteligencia Artificial, Escuela Técnica Superior de Ingenieros Informáticos, Universidad Politécnica de Madrid, Spain.
| |
Collapse
|
3
|
Tabatabaei Z, Wang Y, Colomer A, Oliver Moll J, Zhao Z, Naranjo V. WWFedCBMIR: World-Wide Federated Content-Based Medical Image Retrieval. Bioengineering (Basel) 2023; 10:1144. [PMID: 37892874 PMCID: PMC10604333 DOI: 10.3390/bioengineering10101144] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/15/2023] [Revised: 09/24/2023] [Accepted: 09/25/2023] [Indexed: 10/29/2023] Open
Abstract
The paper proposes a federated content-based medical image retrieval (FedCBMIR) tool that utilizes federated learning (FL) to address the challenges of acquiring a diverse medical data set for training CBMIR models. CBMIR is a tool to find the most similar cases in the data set to assist pathologists. Training such a tool necessitates a pool of whole-slide images (WSIs) to train the feature extractor (FE) to extract an optimal embedding vector. The strict regulations surrounding data sharing in hospitals makes it difficult to collect a rich data set. FedCBMIR distributes an unsupervised FE to collaborative centers for training without sharing the data set, resulting in shorter training times and higher performance. FedCBMIR was evaluated by mimicking two experiments, including two clients with two different breast cancer data sets, namely BreaKHis and Camelyon17 (CAM17), and four clients with the BreaKHis data set at four different magnifications. FedCBMIR increases the F1 score (F1S) of each client from 96% to 98.1% in CAM17 and from 95% to 98.4% in BreaKHis, with 11.44 fewer hours in training time. FedCBMIR provides 98%, 96%, 94%, and 97% F1S in the BreaKHis experiment with a generalized model and accomplishes this in 25.53 fewer hours of training.
Collapse
Affiliation(s)
- Zahra Tabatabaei
- Department of Artificial Intelligence, Tyris Tech S.L., 46021 Valencia, Spain
- Instituto Universitario de Investigación en Tecnología Centrada en el Ser Humano, HUMAN-Tech, Universitat Politècnica de València, 46021 Valencia, Spain
| | - Yuandou Wang
- Multiscale Networked Systems, Universiteit van Amsterdam, 1098XH Amsterdam, The Netherlands
| | - Adrián Colomer
- Instituto Universitario de Investigación en Tecnología Centrada en el Ser Humano, HUMAN-Tech, Universitat Politècnica de València, 46021 Valencia, Spain
- ValgrAI—Valencian Graduate School and Research Network for Artificial Intelligence, 46022 Valencia, Spain
| | - Javier Oliver Moll
- Department of Artificial Intelligence, Tyris Tech S.L., 46021 Valencia, Spain
| | - Zhiming Zhao
- Multiscale Networked Systems, Universiteit van Amsterdam, 1098XH Amsterdam, The Netherlands
| | - Valery Naranjo
- Instituto Universitario de Investigación en Tecnología Centrada en el Ser Humano, HUMAN-Tech, Universitat Politècnica de València, 46021 Valencia, Spain
| |
Collapse
|
4
|
Fast and scalable search of whole-slide images via self-supervised deep learning. Nat Biomed Eng 2022; 6:1420-1434. [PMID: 36217022 PMCID: PMC9792371 DOI: 10.1038/s41551-022-00929-8] [Citation(s) in RCA: 19] [Impact Index Per Article: 9.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/19/2021] [Accepted: 07/15/2022] [Indexed: 01/14/2023]
Abstract
The adoption of digital pathology has enabled the curation of large repositories of gigapixel whole-slide images (WSIs). Computationally identifying WSIs with similar morphologic features within large repositories without requiring supervised training can have significant applications. However, the retrieval speeds of algorithms for searching similar WSIs often scale with the repository size, which limits their clinical and research potential. Here we show that self-supervised deep learning can be leveraged to search for and retrieve WSIs at speeds that are independent of repository size. The algorithm, which we named SISH (for self-supervised image search for histology) and provide as an open-source package, requires only slide-level annotations for training, encodes WSIs into meaningful discrete latent representations and leverages a tree data structure for fast searching followed by an uncertainty-based ranking algorithm for WSI retrieval. We evaluated SISH on multiple tasks (including retrieval tasks based on tissue-patch queries) and on datasets spanning over 22,000 patient cases and 56 disease subtypes. SISH can also be used to aid the diagnosis of rare cancer types for which the number of available WSIs is often insufficient to train supervised deep-learning models.
Collapse
|
5
|
Rashmi R, Prasad K, Udupa CBK. Breast histopathological image analysis using image processing techniques for diagnostic puposes: A methodological review. J Med Syst 2021; 46:7. [PMID: 34860316 PMCID: PMC8642363 DOI: 10.1007/s10916-021-01786-9] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/30/2021] [Accepted: 10/21/2021] [Indexed: 12/24/2022]
Abstract
Breast cancer in women is the second most common cancer worldwide. Early detection of breast cancer can reduce the risk of human life. Non-invasive techniques such as mammograms and ultrasound imaging are popularly used to detect the tumour. However, histopathological analysis is necessary to determine the malignancy of the tumour as it analyses the image at the cellular level. Manual analysis of these slides is time consuming, tedious, subjective and are susceptible to human errors. Also, at times the interpretation of these images are inconsistent between laboratories. Hence, a Computer-Aided Diagnostic system that can act as a decision support system is need of the hour. Moreover, recent developments in computational power and memory capacity led to the application of computer tools and medical image processing techniques to process and analyze breast cancer histopathological images. This review paper summarizes various traditional and deep learning based methods developed to analyze breast cancer histopathological images. Initially, the characteristics of breast cancer histopathological images are discussed. A detailed discussion on the various potential regions of interest is presented which is crucial for the development of Computer-Aided Diagnostic systems. We summarize the recent trends and choices made during the selection of medical image processing techniques. Finally, a detailed discussion on the various challenges involved in the analysis of BCHI is presented along with the future scope.
Collapse
Affiliation(s)
- R Rashmi
- Manipal School of Information Sciences, Manipal Academy of Higher Education, Manipal, India
| | - Keerthana Prasad
- Manipal School of Information Sciences, Manipal Academy of Higher Education, Manipal, India
| | | |
Collapse
|
6
|
Zheng Y, Jiang Z, Shi J, Xie F, Zhang H, Luo W, Hu D, Sun S, Jiang Z, Xue C. Encoding histopathology whole slide images with location-aware graphs for diagnostically relevant regions retrieval. Med Image Anal 2021; 76:102308. [PMID: 34856455 DOI: 10.1016/j.media.2021.102308] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/19/2019] [Revised: 10/14/2021] [Accepted: 11/17/2021] [Indexed: 01/18/2023]
Abstract
Content-based histopathological image retrieval (CBHIR) has become popular in recent years in histopathological image analysis. CBHIR systems provide auxiliary diagnosis information for pathologists by searching for and returning regions that are contently similar to the region of interest (ROI) from a pre-established database. It is challenging and yet significant in clinical applications to retrieve diagnostically relevant regions from a database consisting of histopathological whole slide images (WSIs). In this paper, we propose a novel framework for regions retrieval from WSI database based on location-aware graphs and deep hash techniques. Compared to the present CBHIR framework, both structural information and global location information of ROIs in the WSI are preserved by graph convolution and self-attention operations, which makes the retrieval framework more sensitive to regions that are similar in tissue distribution. Moreover, benefited from the graph structure, the proposed framework has good scalability for both the size and shape variation of ROIs. It allows the pathologist to define query regions using free curves according to the appearance of tissue. Thirdly, the retrieval is achieved based on the hash technique, which ensures the framework is efficient and adequate for practical large-scale WSI database. The proposed method was evaluated on an in-house endometrium dataset with 2650 WSIs and the public ACDC-LungHP dataset. The experimental results have demonstrated that the proposed method achieved a mean average precision above 0.667 on the endometrium dataset and above 0.869 on the ACDC-LungHP dataset in the task of irregular region retrieval, which are superior to the state-of-the-art methods. The average retrieval time from a database containing 1855 WSIs is 0.752 ms. The source code is available at https://github.com/zhengyushan/lagenet.
Collapse
Affiliation(s)
- Yushan Zheng
- Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing 100191, China
| | - Zhiguo Jiang
- Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing 100191, China; Image Processing Center, School of Astronautics, Beihang University, Beijing 102206, China.
| | - Jun Shi
- School of Software, Hefei University of Technology, Hefei 230601, China.
| | - Fengying Xie
- Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing 100191, China; Image Processing Center, School of Astronautics, Beihang University, Beijing 102206, China
| | - Haopeng Zhang
- Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing 100191, China; Image Processing Center, School of Astronautics, Beihang University, Beijing 102206, China
| | - Wei Luo
- Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing 100191, China; Image Processing Center, School of Astronautics, Beihang University, Beijing 102206, China
| | - Dingyi Hu
- Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing 100191, China; Image Processing Center, School of Astronautics, Beihang University, Beijing 102206, China
| | - Shujiao Sun
- Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing 100191, China; Image Processing Center, School of Astronautics, Beihang University, Beijing 102206, China
| | - Zhongmin Jiang
- Department of Pathology, Tianjin Fifth Central Hospital, Tianjin 300450, China
| | - Chenghai Xue
- Wankangyuan Tianjin Gene Technology, Inc, Tianjin 300220, China; Tianjin Institute of Industrial Biotechnology, Chinese Academy of Sciences, Tianjin 300308, China
| |
Collapse
|
7
|
Zheng Y, Jiang Z, Xie F, Shi J, Zhang H, Huai J, Cao M, Yang X. Diagnostic Regions Attention Network (DRA-Net) for Histopathology WSI Recommendation and Retrieval. IEEE TRANSACTIONS ON MEDICAL IMAGING 2021; 40:1090-1103. [PMID: 33351756 DOI: 10.1109/tmi.2020.3046636] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
The development of whole slide imaging techniques and online digital pathology platforms have accelerated the popularization of telepathology for remote tumor diagnoses. During a diagnosis, the behavior information of the pathologist can be recorded by the platform and then archived with the digital case. The browsing path of the pathologist on the WSI is one of the valuable information in the digital database because the image content within the path is expected to be highly correlated with the diagnosis report of the pathologist. In this article, we proposed a novel approach for computer-assisted cancer diagnosis named session-based histopathology image recommendation (SHIR) based on the browsing paths on WSIs. To achieve the SHIR, we developed a novel diagnostic regions attention network (DRA-Net) to learn the pathology knowledge from the image content associated with the browsing paths. The DRA-Net does not rely on the pixel-level or region-level annotations of pathologists. All the data for training can be automatically collected by the digital pathology platform without interrupting the pathologists' diagnoses. The proposed approaches were evaluated on a gastric dataset containing 983 cases within 5 categories of gastric lesions. The quantitative and qualitative assessments on the dataset have demonstrated the proposed SHIR framework with the novel DRA-Net is effective in recommending diagnostically relevant cases for auxiliary diagnosis. The MRR and MAP for the recommendation are respectively 0.816 and 0.836 on the gastric dataset. The source code of the DRA-Net is available at https://github.com/zhengyushan/dpathnet.
Collapse
|
8
|
Gu Y, Vyas K, Shen M, Yang J, Yang GZ. Deep Graph-Based Multimodal Feature Embedding for Endomicroscopy Image Retrieval. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2021; 32:481-492. [PMID: 32310786 DOI: 10.1109/tnnls.2020.2980129] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
Representation learning is a critical task for medical image analysis in computer-aided diagnosis. However, it is challenging to learn discriminative features due to the limited size of the data set and the lack of labels. In this article, we propose a deep graph-based multimodal feature embedding (DGMFE) framework for medical image retrieval with application to breast tissue classification by learning discriminative features of probe-based confocal laser endomicroscopy (pCLE). We first build a multimodality graph model based on the visual similarity between pCLE data and reference histology images. The latent similar pCLE-histology pairs are extracted by walking with the cyclic path on the graph while the dissimilar pairs are extracted based on the geodesic distance. Given the similar and dissimilar pairs, the latent feature space is discovered by reconstructing the similarity between pCLE and histology images via deep Siamese neural networks. The proposed method is evaluated on a clinical database with 700 pCLE mosaics. The accuracy of image retrieval demonstrates that DGMFE can outperform previous works on feature learning. Especially, the top-1 accuracy in an eight-class retrieval task is 0.739, thus demonstrating a 10% improvement compared to the state-of-the-art method.
Collapse
|
9
|
Medical Image Retrieval Using Empirical Mode Decomposition with Deep Convolutional Neural Network. BIOMED RESEARCH INTERNATIONAL 2021; 2020:6687733. [PMID: 33426062 PMCID: PMC7781707 DOI: 10.1155/2020/6687733] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 11/08/2020] [Revised: 12/07/2020] [Accepted: 12/14/2020] [Indexed: 11/17/2022]
Abstract
Content-based medical image retrieval (CBMIR) systems attempt to search medical image database to narrow the semantic gap in medical image analysis. The efficacy of high-level medical information representation using features is a major challenge in CBMIR systems. Features play a vital role in the accuracy and speed of the search process. In this paper, we propose a deep convolutional neural network- (CNN-) based framework to learn concise feature vector for medical image retrieval. The medical images are decomposed into five components using empirical mode decomposition (EMD). The deep CNN is trained in a supervised way with multicomponent input, and the learned features are used to retrieve medical images. The IRMA dataset, containing 11,000 X-ray images, 116 classes, is used to validate the proposed method. We achieve a total IRMA error of 43.21 and a mean average precision of 0.86 for retrieval task and IRMA error of 68.48 and F1 measure of 0.66 on classification task, which is the best result compared with existing literature for this dataset.
Collapse
|
10
|
Kalra S, Tizhoosh HR, Choi C, Shah S, Diamandis P, Campbell CJV, Pantanowitz L. Yottixel - An Image Search Engine for Large Archives of Histopathology Whole Slide Images. Med Image Anal 2020; 65:101757. [PMID: 32623275 DOI: 10.1016/j.media.2020.101757] [Citation(s) in RCA: 46] [Impact Index Per Article: 11.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/07/2019] [Revised: 06/11/2020] [Accepted: 06/16/2020] [Indexed: 01/21/2023]
Abstract
With the emergence of digital pathology, searching for similar images in large archives has gained considerable attention. Image retrieval can provide pathologists with unprecedented access to the evidence embodied in already diagnosed and treated cases from the past. This paper proposes a search engine specialized for digital pathology, called Yottixel, a portmanteau for "one yotta pixel," alluding to the big-data nature of histopathology images. The most impressive characteristic of Yottixel is its ability to represent whole slide images (WSIs) in a compact manner. Yottixel can perform millions of searches in real-time with a high search accuracy and low storage profile. Yottixel uses an intelligent indexing algorithm capable of representing WSIs with a mosaic of patches which are then converted into barcodes, called "Bunch of Barcodes" (BoB), the most prominent performance enabler of Yottixel. The performance of the prototype platform is qualitatively tested using 300 WSIs from the University of Pittsburgh Medical Center (UPMC) and 2,020 WSIs from The Cancer Genome Atlas Program (TCGA) provided by the National Cancer Institute. Both datasets amount to more than 4,000,000 patches of 1000 × 1000 pixels. We report three sets of experiments that show that Yottixel can accurately retrieve organs and malignancies, and its semantic ordering shows good agreement with the subjective evaluation of human observers.
Collapse
Affiliation(s)
- Shivam Kalra
- Kimia Lab, University of Waterloo, Ontario, Canada; Huron Digital Pathology, St. Jacobs, ON, Canada
| | - H R Tizhoosh
- Kimia Lab, University of Waterloo, Ontario, Canada; Vector Institute, MaRS Centre, Toronto, Canada.
| | | | | | | | | | - Liron Pantanowitz
- University of Pittsburgh Medical Center, Department of Pathology, PA, USA
| |
Collapse
|
11
|
Gu Y, Yang J. Multi-level magnification correlation hashing for scalable histopathological image retrieval. Neurocomputing 2019. [DOI: 10.1016/j.neucom.2019.03.050] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/28/2022]
|
12
|
Sapkota M, Shi X, Xing F, Yang L. Deep Convolutional Hashing for Low-Dimensional Binary Embedding of Histopathological Images. IEEE J Biomed Health Inform 2019; 23:805-816. [PMID: 29993648 PMCID: PMC6429565 DOI: 10.1109/jbhi.2018.2827703] [Citation(s) in RCA: 15] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
Abstract
Compact binary representations of histopa-thology images using hashing methods provide efficient approximate nearest neighbor search for direct visual query in large-scale databases. They can be utilized to measure the probability of the abnormality of the query image based on the retrieved similar cases, thereby providing support for medical diagnosis. They also allow for efficient managing of large-scale image databases because of a low storage requirement. However, the effectiveness of binary representations heavily relies on the visual descriptors that represent the semantic information in the histopathological images. Traditional approaches with hand-crafted visual descriptors might fail due to significant variations in image appearance. Recently, deep learning architectures provide promising solutions to address this problem using effective semantic representations. In this paper, we propose a deep convolutional hashing method that can be trained "point-wise" to simultaneously learn both semantic and binary representations of histopathological images. Specifically, we propose a convolutional neural network that introduces a latent binary encoding (LBE) layer for low-dimensional feature embedding to learn binary codes. We design a joint optimization objective function that encourages the network to learn discriminative representations from the label information, and reduce the gap between the real-valued low-dimensional embedded features and desired binary values. The binary encoding for new images can be obtained by forward propagating through the network and quantizing the output of the LBE layer. Experimental results on a large-scale histopathological image dataset demonstrate the effectiveness of the proposed method.
Collapse
|
13
|
Gu Y, Yang J. Densely-Connected Multi-Magnification Hashing for Histopathological Image Retrieval. IEEE J Biomed Health Inform 2018; 23:1683-1691. [PMID: 30475737 DOI: 10.1109/jbhi.2018.2882647] [Citation(s) in RCA: 14] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Abstract
Content-based medical image retrieval is an important computer-aided diagnosis technique providing the clinicians with interpretative references based on visual similarity. In this paper, we focus on the tasks of histopathological image retrieval for breast cancer diagnosis. The densely-connected multi-magnification (DCMMH) framework is proposed to generate the discriminative binary codes by exploiting the histopathological images with multiple magnification factors. The low-magnification images are boosted by the accumulated similarity based on local patches that also regularize the feature learning of high-magnification images. In order to fully utilize the information across different magnification levels, a densely-connected architecture is finally deployed for high-low magnification pairs of datasets. Experiments on BreakHis dataset demonstrate that, DCMMH outperforms the previous hashing methods on histopathological image retrieval.
Collapse
|
14
|
Mining Big Neuron Morphological Data. COMPUTATIONAL INTELLIGENCE AND NEUROSCIENCE 2018; 2018:8234734. [PMID: 30034462 PMCID: PMC6035829 DOI: 10.1155/2018/8234734] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 02/24/2018] [Revised: 05/09/2018] [Accepted: 05/24/2018] [Indexed: 11/26/2022]
Abstract
The advent of automatic tracing and reconstruction technology has led to a surge in the number of neurons 3D reconstruction data and consequently the neuromorphology research. However, the lack of machine-driven annotation schema to automatically detect the types of the neurons based on their morphology still hinders the development of this branch of science. Neuromorphology is important because of the interplay between the shape and functionality of neurons and the far-reaching impact on the diagnostics and therapeutics in neurological disorders. This survey paper provides a comprehensive research in the field of automatic neurons classification and presents the existing challenges, methods, tools, and future directions for automatic neuromorphology analytics. We summarize the major automatic techniques applicable in the field and propose a systematic data processing pipeline for automatic neuron classification, covering data capturing, preprocessing, analyzing, classification, and retrieval. Various techniques and algorithms in machine learning are illustrated and compared to the same dataset to facilitate ongoing research in the field.
Collapse
|
15
|
Zheng Y, Jiang Z, Zhang H, Xie F, Ma Y, Shi H, Zhao Y. Histopathological Whole Slide Image Analysis Using Context-Based CBIR. IEEE TRANSACTIONS ON MEDICAL IMAGING 2018; 37:1641-1652. [PMID: 29969415 DOI: 10.1109/tmi.2018.2796130] [Citation(s) in RCA: 33] [Impact Index Per Article: 5.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/08/2023]
Abstract
Histopathological image classification (HIC) and content-based histopathological image retrieval (CBHIR) are two promising applications for the histopathological whole slide image (WSI) analysis. HIC can efficiently predict the type of lesion involved in a histopathological image. In general, HIC can aid pathologists in locating high-risk cancer regions from a WSI by providing a cancerous probability map for the WSI. In contrast, CBHIR was developed to allow searches for regions with similar content for a region of interest (ROI) from a database consisting of historical cases. Sets of cases with similar content are accessible to pathologists, which can provide more valuable references for diagnosis. A drawback of the recent CBHIR framework is that a query ROI needs to be manually selected from a WSI. An automatic CBHIR approach for a WSI-wise analysis needs to be developed. In this paper, we propose a novel aided-diagnosis framework of breast cancer using whole slide images, which shares the advantages of both HIC and CBHIR. In our framework, CBHIR is automatically processed throughout the WSI, based on which a probability map regarding the malignancy of breast tumors is calculated. Through the probability map, the malignant regions in WSIs can be easily recognized. Furthermore, the retrieval results corresponding to each sub-region of the WSIs are recorded during the automatic analysis and are available to pathologists during their diagnosis. Our method was validated on fully annotated WSI data sets of breast tumors. The experimental results certify the effectiveness of the proposed method.
Collapse
|
16
|
Ma Y, Jiang Z, Zhang H, Xie F, Zheng Y, Shi H, Zhao Y, Shi J. Generating region proposals for histopathological whole slide image retrieval. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2018; 159:1-10. [PMID: 29650303 DOI: 10.1016/j.cmpb.2018.02.020] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/01/2017] [Revised: 01/15/2018] [Accepted: 02/22/2018] [Indexed: 06/08/2023]
Abstract
BACKGROUND AND OBJECTIVE Content-based image retrieval is an effective method for histopathological image analysis. However, given a database of huge whole slide images (WSIs), acquiring appropriate region-of-interests (ROIs) for training is significant and difficult. Moreover, histopathological images can only be annotated by pathologists, resulting in the lack of labeling information. Therefore, it is an important and challenging task to generate ROIs from WSI and retrieve image with few labels. METHODS This paper presents a novel unsupervised region proposing method for histopathological WSI based on Selective Search. Specifically, the WSI is over-segmented into regions which are hierarchically merged until the WSI becomes a single region. Nucleus-oriented similarity measures for region mergence and Nucleus-Cytoplasm color space for histopathological image are specially defined to generate accurate region proposals. Additionally, we propose a new semi-supervised hashing method for image retrieval. The semantic features of images are extracted with Latent Dirichlet Allocation and transformed into binary hashing codes with Supervised Hashing. RESULTS The methods are tested on a large-scale multi-class database of breast histopathological WSIs. The results demonstrate that for one WSI, our region proposing method can generate 7.3 thousand contoured regions which fit well with 95.8% of the ROIs annotated by pathologists. The proposed hashing method can retrieve a query image among 136 thousand images in 0.29 s and reach precision of 91% with only 10% of images labeled. CONCLUSIONS The unsupervised region proposing method can generate regions as predictions of lesions in histopathological WSI. The region proposals can also serve as the training samples to train machine-learning models for image retrieval. The proposed hashing method can achieve fast and precise image retrieval with small amount of labels. Furthermore, the proposed methods can be potentially applied in online computer-aided-diagnosis systems.
Collapse
Affiliation(s)
- Yibing Ma
- Image Processing Center, School of Astronautics, Beihang University, Beijing 100191, China; Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing 100191, China; Beijing Key Laboratory of Digital Media, Beijing 100191, China.
| | - Zhiguo Jiang
- Image Processing Center, School of Astronautics, Beihang University, Beijing 100191, China; Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing 100191, China; Beijing Key Laboratory of Digital Media, Beijing 100191, China.
| | - Haopeng Zhang
- Image Processing Center, School of Astronautics, Beihang University, Beijing 100191, China; Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing 100191, China; Beijing Key Laboratory of Digital Media, Beijing 100191, China.
| | - Fengying Xie
- Image Processing Center, School of Astronautics, Beihang University, Beijing 100191, China; Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing 100191, China; Beijing Key Laboratory of Digital Media, Beijing 100191, China.
| | - Yushan Zheng
- Image Processing Center, School of Astronautics, Beihang University, Beijing 100191, China; Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing 100191, China; Beijing Key Laboratory of Digital Media, Beijing 100191, China.
| | - Huaqiang Shi
- Motic (Xiamen) Medical Diagnostic Systems Co. Ltd., Xiamen 361101, China; People's Liberation Army Air Force General Hospital, Beijing 100142, China.
| | - Yu Zhao
- Motic (Xiamen) Medical Diagnostic Systems Co. Ltd., Xiamen 361101, China.
| | - Jun Shi
- School of Software, Hefei University of Technology, Hefei 230601, China.
| |
Collapse
|
17
|
Khosravi P, Kazemi E, Imielinski M, Elemento O, Hajirasouliha I. Deep Convolutional Neural Networks Enable Discrimination of Heterogeneous Digital Pathology Images. EBioMedicine 2018; 27:317-328. [PMID: 29292031 PMCID: PMC5828543 DOI: 10.1016/j.ebiom.2017.12.026] [Citation(s) in RCA: 175] [Impact Index Per Article: 29.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/27/2017] [Revised: 12/21/2017] [Accepted: 12/21/2017] [Indexed: 12/18/2022] Open
Abstract
Pathological evaluation of tumor tissue is pivotal for diagnosis in cancer patients and automated image analysis approaches have great potential to increase precision of diagnosis and help reduce human error. In this study, we utilize several computational methods based on convolutional neural networks (CNN) and build a stand-alone pipeline to effectively classify different histopathology images across different types of cancer. In particular, we demonstrate the utility of our pipeline to discriminate between two subtypes of lung cancer, four biomarkers of bladder cancer, and five biomarkers of breast cancer. In addition, we apply our pipeline to discriminate among four immunohistochemistry (IHC) staining scores of bladder and breast cancers. Our classification pipeline includes a basic CNN architecture, Google's Inceptions with three training strategies, and an ensemble of two state-of-the-art algorithms, Inception and ResNet. Training strategies include training the last layer of Google's Inceptions, training the network from scratch, and fine-tunning the parameters for our data using two pre-trained version of Google's Inception architectures, Inception-V1 and Inception-V3. We demonstrate the power of deep learning approaches for identifying cancer subtypes, and the robustness of Google's Inceptions even in presence of extensive tumor heterogeneity. On average, our pipeline achieved accuracies of 100%, 92%, 95%, and 69% for discrimination of various cancer tissues, subtypes, biomarkers, and scores, respectively. Our pipeline and related documentation is freely available at https://github.com/ih-_lab/CNN_Smoothie.
Collapse
Affiliation(s)
- Pegah Khosravi
- Institute for Computational Biomedicine, Weill Cornell Medical College, NY, USA; Department of Physiology and Biophysics, Weill Cornell Medicine, New York, NY, USA
| | - Ehsan Kazemi
- Yale Institute for Network Science, Yale University, New Haven, CT, USA
| | - Marcin Imielinski
- Caryl and Israel Englander Institute for Precision Medicine, Weill Cornell Medical College, NY, USA; Department of Pathology and Laboratory Medicine, Weill Cornell Medical College, NY, USA; The New York Genome Center, NY, USA; The Meyer Cancer Center, Weill Cornell Medicine, New York, NY, USA
| | - Olivier Elemento
- Institute for Computational Biomedicine, Weill Cornell Medical College, NY, USA; Department of Physiology and Biophysics, Weill Cornell Medicine, New York, NY, USA; Caryl and Israel Englander Institute for Precision Medicine, Weill Cornell Medical College, NY, USA; The Meyer Cancer Center, Weill Cornell Medicine, New York, NY, USA
| | - Iman Hajirasouliha
- Institute for Computational Biomedicine, Weill Cornell Medical College, NY, USA; Department of Physiology and Biophysics, Weill Cornell Medicine, New York, NY, USA; Caryl and Israel Englander Institute for Precision Medicine, Weill Cornell Medical College, NY, USA; The Meyer Cancer Center, Weill Cornell Medicine, New York, NY, USA
| |
Collapse
|
18
|
Shi X, Xing F, Xu K, Xie Y, Su H, Yang L. Supervised graph hashing for histopathology image retrieval and classification. Med Image Anal 2017; 42:117-128. [DOI: 10.1016/j.media.2017.07.009] [Citation(s) in RCA: 19] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/04/2016] [Revised: 07/25/2017] [Accepted: 07/31/2017] [Indexed: 10/19/2022]
|
19
|
Li Z, Zhang X, Müller H, Zhang S. Large-scale retrieval for medical image analytics: A comprehensive review. Med Image Anal 2017; 43:66-84. [PMID: 29031831 DOI: 10.1016/j.media.2017.09.007] [Citation(s) in RCA: 75] [Impact Index Per Article: 10.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/01/2017] [Revised: 08/01/2017] [Accepted: 09/29/2017] [Indexed: 12/27/2022]
Abstract
Over the past decades, medical image analytics was greatly facilitated by the explosion of digital imaging techniques, where huge amounts of medical images were produced with ever-increasing quality and diversity. However, conventional methods for analyzing medical images have achieved limited success, as they are not capable to tackle the huge amount of image data. In this paper, we review state-of-the-art approaches for large-scale medical image analysis, which are mainly based on recent advances in computer vision, machine learning and information retrieval. Specifically, we first present the general pipeline of large-scale retrieval, summarize the challenges/opportunities of medical image analytics on a large-scale. Then, we provide a comprehensive review of algorithms and techniques relevant to major processes in the pipeline, including feature representation, feature indexing, searching, etc. On the basis of existing work, we introduce the evaluation protocols and multiple applications of large-scale medical image retrieval, with a variety of exploratory and diagnostic scenarios. Finally, we discuss future directions of large-scale retrieval, which can further improve the performance of medical image analysis.
Collapse
Affiliation(s)
- Zhongyu Li
- Department of Computer Science, University of North Carolina at Charlotte, Charlotte, NC 28223, USA
| | - Xiaofan Zhang
- Department of Computer Science, University of North Carolina at Charlotte, Charlotte, NC 28223, USA
| | - Henning Müller
- Information Systems Institute, HES-SO Valais, Sierre, Switzerland
| | - Shaoting Zhang
- Department of Computer Science, University of North Carolina at Charlotte, Charlotte, NC 28223, USA.
| |
Collapse
|
20
|
Gu Y, Vyas K, Yang J, Yang GZ. Unsupervised Feature Learning for Endomicroscopy Image Retrieval. ACTA ACUST UNITED AC 2017. [DOI: 10.1007/978-3-319-66179-7_8] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/23/2023]
|