1
|
Kobayashi K, Gu L, Hataya R, Mizuno T, Miyake M, Watanabe H, Takahashi M, Takamizawa Y, Yoshida Y, Nakamura S, Kouno N, Bolatkan A, Kurose Y, Harada T, Hamamoto R. Sketch-based semantic retrieval of medical images. Med Image Anal 2024; 92:103060. [PMID: 38104401 DOI: 10.1016/j.media.2023.103060] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/16/2022] [Revised: 08/31/2023] [Accepted: 12/05/2023] [Indexed: 12/19/2023]
Abstract
The volume of medical images stored in hospitals is rapidly increasing; however, the utilization of these accumulated medical images remains limited. Existing content-based medical image retrieval (CBMIR) systems typically require example images, leading to practical limitations, such as the lack of customizable, fine-grained image retrieval, the inability to search without example images, and difficulty in retrieving rare cases. In this paper, we introduce a sketch-based medical image retrieval (SBMIR) system that enables users to find images of interest without the need for example images. The key concept is feature decomposition of medical images, which allows the entire feature of a medical image to be decomposed into and reconstructed from normal and abnormal features. Building on this concept, our SBMIR system provides an easy-to-use two-step graphical user interface: users first select a template image to specify a normal feature and then draw a semantic sketch of the disease on the template image to represent an abnormal feature. The system integrates both types of input to construct a query vector and retrieves reference images. For evaluation, ten healthcare professionals participated in a user test using two datasets. Consequently, our SBMIR system enabled users to overcome previous challenges, including image retrieval based on fine-grained image characteristics, image retrieval without example images, and image retrieval for rare cases. Our SBMIR system provides on-demand, customizable medical image retrieval, thereby expanding the utility of medical image databases.
Collapse
Affiliation(s)
- Kazuma Kobayashi
- Division of Medical AI Research and Development, National Cancer Center Research Institute, 5-1-1 Tsukiji, Chuo-ku, Tokyo 104-0045, Japan; Cancer Translational Research Team, RIKEN Center for Advanced Intelligence Project, 1-4-1 Nihonbashi, Chuo-ku, Tokyo 103-0027, Japan.
| | - Lin Gu
- Machine Intelligence for Medical Engineering Team, RIKEN Center for Advanced Intelligence Project, 1-4-1 Nihonbashi, Chuo-ku, Tokyo 103-0027, Japan; Research Center for Advanced Science and Technology, The University of Tokyo, 4-6-1 Komaba, Meguro-ku, Tokyo 153-8904, Japan.
| | - Ryuichiro Hataya
- Cancer Translational Research Team, RIKEN Center for Advanced Intelligence Project, 1-4-1 Nihonbashi, Chuo-ku, Tokyo 103-0027, Japan.
| | - Takaaki Mizuno
- Department of Experimental Therapeutics, National Cancer Center Hospital, 5-1-1 Tsukiji, Chuo-ku, Tokyo 104-0045, Japan.
| | - Mototaka Miyake
- Department of Diagnostic Radiology, National Cancer Center Hospital, 5-1-1 Tsukiji, Chuo-ku, Tokyo 104-0045, Japan.
| | - Hirokazu Watanabe
- Department of Diagnostic Radiology, National Cancer Center Hospital, 5-1-1 Tsukiji, Chuo-ku, Tokyo 104-0045, Japan.
| | - Masamichi Takahashi
- Department of Neurosurgery and Neuro-Oncology, National Cancer Center Hospital, 5-1-1 Tsukiji, Chuo-ku, Tokyo 104-0045, Japan.
| | - Yasuyuki Takamizawa
- Department of Colorectal Surgery, National Cancer Center Hospital, 5-1-1 Tsukiji, Chuo-ku, Tokyo 104-0045, Japan.
| | - Yukihiro Yoshida
- Department of Thoracic Surgery, National Cancer Center Hospital, 5-1-1 Tsukiji, Chuo-ku, Tokyo 104-0045, Japan.
| | - Satoshi Nakamura
- Radiation Safety and Quality Assurance Division, National Cancer Center Hospital, 5-1-1 Tsukiji, Chuo-ku, Tokyo 104-0045, Japan; Division of Research and Development for Boron Neutron Capture Therapy, National Cancer Center, Exploratory Oncology Research & Clinical Trial Center, 5-1-1 Tsukiji, Chuo-ku, Tokyo 104-0045, Japan; Medical Physics Laboratory, Division of Health Science, Graduate School of Medicine, Osaka University, Yamadaoka 1-7, Suita-shi, Osaka 565-0871, Japan.
| | - Nobuji Kouno
- Division of Medical AI Research and Development, National Cancer Center Research Institute, 5-1-1 Tsukiji, Chuo-ku, Tokyo 104-0045, Japan; Department of Surgery, Kyoto University Graduate School of Medicine, 54 Shogoin Kawahara-cho, Sakyo-ku, Kyoto 606-8507, Japan.
| | - Amina Bolatkan
- Division of Medical AI Research and Development, National Cancer Center Research Institute, 5-1-1 Tsukiji, Chuo-ku, Tokyo 104-0045, Japan; Cancer Translational Research Team, RIKEN Center for Advanced Intelligence Project, 1-4-1 Nihonbashi, Chuo-ku, Tokyo 103-0027, Japan.
| | - Yusuke Kurose
- Machine Intelligence for Medical Engineering Team, RIKEN Center for Advanced Intelligence Project, 1-4-1 Nihonbashi, Chuo-ku, Tokyo 103-0027, Japan; Research Center for Advanced Science and Technology, The University of Tokyo, 4-6-1 Komaba, Meguro-ku, Tokyo 153-8904, Japan.
| | - Tatsuya Harada
- Machine Intelligence for Medical Engineering Team, RIKEN Center for Advanced Intelligence Project, 1-4-1 Nihonbashi, Chuo-ku, Tokyo 103-0027, Japan; Research Center for Advanced Science and Technology, The University of Tokyo, 4-6-1 Komaba, Meguro-ku, Tokyo 153-8904, Japan.
| | - Ryuji Hamamoto
- Division of Medical AI Research and Development, National Cancer Center Research Institute, 5-1-1 Tsukiji, Chuo-ku, Tokyo 104-0045, Japan; Cancer Translational Research Team, RIKEN Center for Advanced Intelligence Project, 1-4-1 Nihonbashi, Chuo-ku, Tokyo 103-0027, Japan.
| |
Collapse
|
2
|
Tu C, Du D, Zeng T, Zhang Y. Deep Multi-Dictionary Learning for Survival Prediction With Multi-Zoom Histopathological Whole Slide Images. IEEE/ACM TRANSACTIONS ON COMPUTATIONAL BIOLOGY AND BIOINFORMATICS 2024; 21:14-25. [PMID: 37788195 DOI: 10.1109/tcbb.2023.3321593] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 10/05/2023]
Abstract
Survival prediction based on histopathological whole slide images (WSIs) is of great significance for risk-benefit assessment and clinical decision. However, complex microenvironments and heterogeneous tissue structures in WSIs bring challenges to learning informative prognosis-related representations. Additionally, previous studies mainly focus on modeling using mono-scale WSIs, which commonly ignore useful subtle differences existed in multi-zoom WSIs. To this end, we propose a deep multi-dictionary learning framework for cancer survival prediction with multi-zoom histopathological WSIs. The framework can recognize and learn discriminative clusters (i.e., microenvironments) based on multi-scale deep representations for survival analysis. Specifically, we learn multi-scale features based on multi-zoom tiles from WSIs via stacked deep autoencoders network followed by grouping different microenvironments by cluster algorithm. Based on multi-scale deep features of clusters, a multi-dictionary learning method with a post-pruning strategy is devised to learn discriminative representations from selected prognosis-related clusters in a task-driven manner. Finally, a survival model (i.e., EN-Cox) is constructed to estimate the risk index of an individual patient. The proposed model is evaluated on three datasets derived from The Cancer Genome Atlas (TCGA), and the experimental results demonstrate that it outperforms several state-of-the-art survival analysis approaches.
Collapse
|
3
|
Gupta D, Loane R, Gayen S, Demner-Fushman D. Medical Image Retrieval via Nearest Neighbor Search on Pre-trained Image Features. Knowl Based Syst 2023; 278:110907. [PMID: 37780058 PMCID: PMC10540469 DOI: 10.1016/j.knosys.2023.110907] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/03/2023]
Abstract
Nearest neighbor search, also known as NNS, is a technique used to locate the points in a high-dimensional space closest to a given query point. This technique has multiple applications in medicine, such as searching large medical imaging databases, disease classification, and diagnosis. However, when the number of points is significantly large, the brute-force approach for finding the nearest neighbor becomes computationally infeasible. Therefore, various approaches have been developed to make the search faster and more efficient to support the applications. With a focus on medical imaging, this paper proposes DenseLinkSearch (DLS), an effective and efficient algorithm that searches and retrieves the relevant images from heterogeneous sources of medical images. Towards this, given a medical database, the proposed algorithm builds an index that consists of pre-computed links of each point in the database. The search algorithm utilizes the index to efficiently traverse the database in search of the nearest neighbor. We also explore the role of medical image feature representation in content-based medical image retrieval tasks. We propose a Transformer-based feature representation technique that outperformed the existing pre-trained Transformer-based approaches on benchmark medical image retrieval datasets. We extensively tested the proposed NNS approach and compared the performance with state-of-the-art NNS approaches on benchmark datasets and our created medical image datasets. The proposed approach outperformed the existing approaches in terms of retrieving accurate neighbors and retrieval speed. In comparison to the existing approximate NNS approaches, our proposed DLS approach outperformed them in terms of lower average time per query and ≥ 99% R@10 on 11 out of 13 benchmark datasets. We also found that the proposed medical feature representation approach is better for representing medical images compared to the existing pre-trained image models. The proposed feature extraction strategy obtained an improvement of 9.37%, 7.0%, and 13.33% in terms of P@5, P@10, and P@20, respectively, in comparison to the best-performing pre-trained image model. The source code and datasets of our experiments are available at https://github.com/deepaknlp/DLS.
Collapse
Affiliation(s)
- Deepak Gupta
- Lister Hill National Center for Biomedical Communications, National Library of Medicine, National Institutes of Health, Bethesda, MD, USA
| | - Russell Loane
- Lister Hill National Center for Biomedical Communications, National Library of Medicine, National Institutes of Health, Bethesda, MD, USA
| | - Soumya Gayen
- Lister Hill National Center for Biomedical Communications, National Library of Medicine, National Institutes of Health, Bethesda, MD, USA
| | - Dina Demner-Fushman
- Lister Hill National Center for Biomedical Communications, National Library of Medicine, National Institutes of Health, Bethesda, MD, USA
| |
Collapse
|
4
|
Zeng A, Wu C, Lin G, Xie W, Hong J, Huang M, Zhuang J, Bi S, Pan D, Ullah N, Khan KN, Wang T, Shi Y, Li X, Xu X. ImageCAS: A large-scale dataset and benchmark for coronary artery segmentation based on computed tomography angiography images. Comput Med Imaging Graph 2023; 109:102287. [PMID: 37634975 DOI: 10.1016/j.compmedimag.2023.102287] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/15/2023] [Revised: 05/03/2023] [Accepted: 08/03/2023] [Indexed: 08/29/2023]
Abstract
Cardiovascular disease (CVD) accounts for about half of non-communicable diseases. Vessel stenosis in the coronary artery is considered to be the major risk of CVD. Computed tomography angiography (CTA) is one of the widely used noninvasive imaging modalities in coronary artery diagnosis due to its superior image resolution. Clinically, segmentation of coronary arteries is essential for the diagnosis and quantification of coronary artery disease. Recently, a variety of works have been proposed to address this problem. However, on one hand, most works rely on in-house datasets, and only a few works published their datasets to the public which only contain tens of images. On the other hand, their source code have not been published, and most follow-up works have not made comparison with existing works, which makes it difficult to judge the effectiveness of the methods and hinders the further exploration of this challenging yet critical problem in the community. In this paper, we propose a large-scale dataset for coronary artery segmentation on CTA images. In addition, we have implemented a benchmark in which we have tried our best to implement several typical existing methods. Furthermore, we propose a strong baseline method which combines multi-scale patch fusion and two-stage processing to extract the details of vessels. Comprehensive experiments show that the proposed method achieves better performance than existing works on the proposed large-scale dataset. The benchmark and the dataset are published at https://github.com/XiaoweiXu/ImageCAS-A-Large-Scale-Dataset-and-Benchmark-for-Coronary-Artery-Segmentation-based-on-CT.
Collapse
Affiliation(s)
- An Zeng
- School of Computer Science, Guangdong University of Technology, Guangzhou, China
| | - Chunbiao Wu
- School of Computer Science, Guangdong University of Technology, Guangzhou, China
| | - Guisen Lin
- Department of Radiology, Shenzhen Children's Hospital, Shenzhen, China
| | - Wen Xie
- Guangdong Provincial Key Laboratory of South China Structural Heart Disease, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China
| | - Jin Hong
- Guangdong Provincial Key Laboratory of South China Structural Heart Disease, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China
| | - Meiping Huang
- Guangdong Provincial Key Laboratory of South China Structural Heart Disease, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China
| | - Jian Zhuang
- Guangdong Provincial Key Laboratory of South China Structural Heart Disease, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China
| | - Shanshan Bi
- Department of Computer Science and Engineering, Missouri University of Science and Technology, Rolla, MO, United States
| | - Dan Pan
- Department of Computer Science, Guangdong Polytechnic Normal University, Guangzhou, China
| | - Najeeb Ullah
- Department of Computer Science, University of Engineering and Technology, Mardan, KP, Pakistan
| | - Kaleem Nawaz Khan
- Department of Computer Science, University of Engineering and Technology, Mardan, KP, Pakistan
| | - Tianchen Wang
- Department of Computer Science and Engineering, University of Notre Dame, Indiana, United States
| | - Yiyu Shi
- Department of Computer Science and Engineering, University of Notre Dame, Indiana, United States
| | - Xiaomeng Li
- Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong Special Administrative Region, China
| | - Xiaowei Xu
- Guangdong Provincial Key Laboratory of South China Structural Heart Disease, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China.
| |
Collapse
|
5
|
Zhou W, Bai W, Ji J, Yi Y, Zhang N, Cui W. Dual-path multi-scale context dense aggregation network for retinal vessel segmentation. Comput Biol Med 2023; 164:107269. [PMID: 37562323 DOI: 10.1016/j.compbiomed.2023.107269] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/14/2023] [Revised: 06/22/2023] [Accepted: 07/16/2023] [Indexed: 08/12/2023]
Abstract
There has been steady progress in the field of deep learning-based blood vessel segmentation. However, several challenging issues still continue to limit its progress, including inadequate sample sizes, the neglect of contextual information, and the loss of microvascular details. To address these limitations, we propose a dual-path deep learning framework for blood vessel segmentation. In our framework, the fundus images are divided into concentric patches with different scales to alleviate the overfitting problem. Then, a Multi-scale Context Dense Aggregation Network (MCDAU-Net) is proposed to accurately extract the blood vessel boundaries from these patches. In MCDAU-Net, a Cascaded Dilated Spatial Pyramid Pooling (CDSPP) module is designed and incorporated into intermediate layers of the model, enhancing the receptive field and producing feature maps enriched with contextual information. To improve segmentation performance for low-contrast vessels, we propose an InceptionConv (IConv) module, which can explore deeper semantic features and suppress the propagation of non-vessel information. Furthermore, we design a Multi-scale Adaptive Feature Aggregation (MAFA) module to fuse the multi-scale feature by assigning adaptive weight coefficients to different feature maps through skip connections. Finally, to explore the complementary contextual information and enhance the continuity of microvascular structures, a fusion module is designed to combine the segmentation results obtained from patches of different sizes, achieving fine microvascular segmentation performance. In order to assess the effectiveness of our approach, we conducted evaluations on three widely-used public datasets: DRIVE, CHASE-DB1, and STARE. Our findings reveal a remarkable advancement over the current state-of-the-art (SOTA) techniques, with the mean values of Se and F1 scores being an increase of 7.9% and 4.7%, respectively. The code is available at https://github.com/bai101315/MCDAU-Net.
Collapse
Affiliation(s)
- Wei Zhou
- College of Computer Science, Shenyang Aerospace University, Shenyang, China
| | - Weiqi Bai
- College of Computer Science, Shenyang Aerospace University, Shenyang, China
| | - Jianhang Ji
- College of Computer Science, Shenyang Aerospace University, Shenyang, China
| | - Yugen Yi
- School of Software, Jiangxi Normal University, Nanchang, China.
| | - Ningyi Zhang
- School of Software, Jiangxi Normal University, Nanchang, China
| | - Wei Cui
- Institute for Infocomm Research, The Agency for Science, Technology and Research (A*STAR), Singapore.
| |
Collapse
|
6
|
Chen T, Hu L, Lu Q, Xiao F, Xu H, Li H, Lu L. A computer-aided diagnosis system for brain tumors based on artificial intelligence algorithms. Front Neurosci 2023; 17:1120781. [PMID: 37483342 PMCID: PMC10360168 DOI: 10.3389/fnins.2023.1120781] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/10/2022] [Accepted: 06/19/2023] [Indexed: 07/25/2023] Open
Abstract
The choice of treatment and prognosis evaluation depend on the accurate early diagnosis of brain tumors. Many brain tumors go undiagnosed or are overlooked by clinicians as a result of the challenges associated with manually evaluating magnetic resonance imaging (MRI) images in clinical practice. In this study, we built a computer-aided diagnosis (CAD) system for glioma detection, grading, segmentation, and knowledge discovery based on artificial intelligence algorithms. Neuroimages are specifically represented using a type of visual feature known as the histogram of gradients (HOG). Then, through a two-level classification framework, the HOG features are employed to distinguish between healthy controls and patients, or between different glioma grades. This CAD system also offers tumor visualization using a semi-automatic segmentation tool for better patient management and treatment monitoring. Finally, a knowledge base is created to offer additional advice for the diagnosis of brain tumors. Based on our proposed two-level classification framework, we train models for glioma detection and grading, achieving area under curve (AUC) of 0.921 and 0.806, respectively. Different from other systems, we integrate these diagnostic tools with a web-based interface, which provides the flexibility for system deployment.
Collapse
Affiliation(s)
- Tao Chen
- School of Information Technology, Shangqiu Normal University, Shangqiu, China
| | - Lianting Hu
- Medical Big Data Center, Guangdong Provincial People's Hospital, Guangdong Academy of Medical Sciences, Guangzhou, China
- Guangdong Cardiovascular Institute, Guangzhou, China
| | - Quan Lu
- School of Information Management, Wuhan University, Wuhan, China
| | - Feng Xiao
- Department of Radiology, Zhongnan Hospital of Wuhan University, Wuhan, China
| | - Haibo Xu
- Department of Radiology, Zhongnan Hospital of Wuhan University, Wuhan, China
| | - Hongjun Li
- Department of Radiology, Beijing Youan Hospital, Capital Medical University, Beijing, China
| | - Long Lu
- School of Information Management, Wuhan University, Wuhan, China
- Big Data Institute, Wuhan University, Wuhan, China
- School of Public Health, Wuhan University, Wuhan, China
- Institute of Pediatrics, Guangzhou Women and Children's Medical Center, Guangzhou Medical University, Guangzhou, China
| |
Collapse
|
7
|
Fu Y, Zhou F, Shi X, Wang L, Li Y, Wu J, Huang H. Classification of adenoid cystic carcinoma in whole slide images by using deep learning. Biomed Signal Process Control 2023. [DOI: 10.1016/j.bspc.2023.104789] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/11/2023]
|
8
|
Abid MH, Ashraf R, Mahmood T, Faisal CMN. Multi-modal medical image classification using deep residual network and genetic algorithm. PLoS One 2023; 18:e0287786. [PMID: 37384779 PMCID: PMC10309999 DOI: 10.1371/journal.pone.0287786] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/05/2023] [Accepted: 06/13/2023] [Indexed: 07/01/2023] Open
Abstract
Artificial intelligence (AI) development across the health sector has recently been the most crucial. Early medical information, identification, diagnosis, classification, then analysis, along with viable remedies, are always beneficial developments. Precise and consistent image classification has critical in diagnosing and tactical decisions for healthcare. The core issue with image classification has become the semantic gap. Conventional machine learning algorithms for classification rely mainly on low-level but rather high-level characteristics, employ some handmade features to close the gap, but force intense feature extraction as well as classification approaches. Deep learning is a powerful tool with considerable advances in recent years, with deep convolution neural networks (CNNs) succeeding in image classification. The main goal is to bridge the semantic gap and enhance the classification performance of multi-modal medical images based on the deep learning-based model ResNet50. The data set included 28378 multi-modal medical images to train and validate the model. Overall accuracy, precision, recall, and F1-score evaluation parameters have been calculated. The proposed model classifies medical images more accurately than other state-of-the-art methods. The intended research experiment attained an accuracy level of 98.61%. The suggested study directly benefits the health service.
Collapse
Affiliation(s)
- Muhammad Haris Abid
- Department of Computer Science, National Textile University, Faisalabad, Pakistan
| | - Rehan Ashraf
- Department of Computer Science, National Textile University, Faisalabad, Pakistan
| | - Toqeer Mahmood
- Department of Computer Science, National Textile University, Faisalabad, Pakistan
| | - C. M. Nadeem Faisal
- Department of Computer Science, National Textile University, Faisalabad, Pakistan
| |
Collapse
|
9
|
Wickstrøm KK, Østmo EA, Radiya K, Mikalsen KØ, Kampffmeyer MC, Jenssen R. A clinically motivated self-supervised approach for content-based image retrieval of CT liver images. Comput Med Imaging Graph 2023; 107:102239. [PMID: 37207397 DOI: 10.1016/j.compmedimag.2023.102239] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/06/2022] [Revised: 05/02/2023] [Accepted: 05/02/2023] [Indexed: 05/21/2023]
Abstract
Deep learning-based approaches for content-based image retrieval (CBIR) of computed tomography (CT) liver images is an active field of research, but suffer from some critical limitations. First, they are heavily reliant on labeled data, which can be challenging and costly to acquire. Second, they lack transparency and explainability, which limits the trustworthiness of deep CBIR systems. We address these limitations by: (1) Proposing a self-supervised learning framework that incorporates domain-knowledge into the training procedure, and, (2) by providing the first representation learning explainability analysis in the context of CBIR of CT liver images. Results demonstrate improved performance compared to the standard self-supervised approach across several metrics, as well as improved generalization across datasets. Further, we conduct the first representation learning explainability analysis in the context of CBIR, which reveals new insights into the feature extraction process. Lastly, we perform a case study with cross-examination CBIR that demonstrates the usability of our proposed framework. We believe that our proposed framework could play a vital role in creating trustworthy deep CBIR systems that can successfully take advantage of unlabeled data.
Collapse
Affiliation(s)
- Kristoffer Knutsen Wickstrøm
- Machine Learning Group at the Department of Physics and Technology, UiT the Arctic University of Norway, Tromsø NO-9037, Norway.
| | - Eirik Agnalt Østmo
- Machine Learning Group at the Department of Physics and Technology, UiT the Arctic University of Norway, Tromsø NO-9037, Norway
| | - Keyur Radiya
- Department of Gastrointestinal Surgery, University Hospital of North Norway (UNN), Tromsø, Norway
| | - Karl Øyvind Mikalsen
- Machine Learning Group at the Department of Physics and Technology, UiT the Arctic University of Norway, Tromsø NO-9037, Norway; Department of Gastrointestinal Surgery, University Hospital of North Norway (UNN), Tromsø, Norway
| | - Michael Christian Kampffmeyer
- Machine Learning Group at the Department of Physics and Technology, UiT the Arctic University of Norway, Tromsø NO-9037, Norway; Norwegian Computing Center, Department SAMBA, P.O. Box 114 Blindern, Oslo NO-0314, Norway
| | - Robert Jenssen
- Machine Learning Group at the Department of Physics and Technology, UiT the Arctic University of Norway, Tromsø NO-9037, Norway; Norwegian Computing Center, Department SAMBA, P.O. Box 114 Blindern, Oslo NO-0314, Norway; Department of Computer Science, University of Copenhagen, Universitetsparken 1, 2100 København Ø, Denmark
| |
Collapse
|
10
|
Fang J, Zeng M, Zhang X, Liu H, Zhao Y, Zhang P, Yang H, Liu J, Miao H, Hu Y, Liu J. Deep metric learning with mirror attention and fine triplet loss for fundus image retrieval in ophthalmology. Biomed Signal Process Control 2023. [DOI: 10.1016/j.bspc.2022.104277] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
|
11
|
Afriyie Y, Weyori BA, Opoku AA. A scaling up approach: a research agenda for medical imaging analysis with applications in deep learning. J EXP THEOR ARTIF IN 2023. [DOI: 10.1080/0952813x.2023.2165721] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/26/2023]
Affiliation(s)
- Yaw Afriyie
- Department of Computer Science and Informatics, University of Energy and Natural Resources, School of Sciences, Sunyani, Ghana
- Department of Computer Science, Faculty of Information and Communication Technology, SD Dombo University of Business and Integrated Development Studies, Wa, Ghana
| | - Benjamin A. Weyori
- Department of Computer Science and Informatics, University of Energy and Natural Resources, School of Sciences, Sunyani, Ghana
| | - Alex A. Opoku
- Department of Mathematics & Statistics, University of Energy and Natural Resources, School of Sciences, Sunyani, Ghana
| |
Collapse
|
12
|
Wang X, Du Y, Yang S, Zhang J, Wang M, Zhang J, Yang W, Huang J, Han X. RetCCL: Clustering-guided contrastive learning for whole-slide image retrieval. Med Image Anal 2023; 83:102645. [PMID: 36270093 DOI: 10.1016/j.media.2022.102645] [Citation(s) in RCA: 29] [Impact Index Per Article: 29.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/02/2021] [Revised: 07/21/2022] [Accepted: 09/27/2022] [Indexed: 02/07/2023]
Abstract
Benefiting from the large-scale archiving of digitized whole-slide images (WSIs), computer-aided diagnosis has been well developed to assist pathologists in decision-making. Content-based WSI retrieval can be a new approach to find highly correlated WSIs in a historically diagnosed WSI archive, which has the potential usages for assisted clinical diagnosis, medical research, and trainee education. During WSI retrieval, it is particularly challenging to encode the semantic content of histopathological images and to measure the similarity between images for interpretable results due to the gigapixel size of WSIs. In this work, we propose a Retrieval with Clustering-guided Contrastive Learning (RetCCL) framework for robust and accurate WSI-level image retrieval, which integrates a novel self-supervised feature learning method and a global ranking and aggregation algorithm for much improved performance. The proposed feature learning method makes use of existing large-scale unlabeled histopathological image data, which helps learn universal features that could be used directly for subsequent WSI retrieval tasks without extra fine-tuning. The proposed WSI retrieval method not only returns a set of WSIs similar to a query WSI, but also highlights patches or sub-regions of each WSI that share high similarity with patches of the query WSI, which helps pathologists interpret the searching results. Our WSI retrieval framework has been evaluated on the tasks of anatomical site retrieval and cancer subtype retrieval using over 22,000 slides, and the performance exceeds other state-of-the-art methods significantly (around 10% for the anatomic site retrieval in terms of average mMV@10). Besides, the patch retrieval using our learned feature representation offers a performance improvement of 24% on the TissueNet dataset in terms of mMV@5 compared with using ImageNet pre-trained features, which further demonstrates the effectiveness of the proposed CCL feature learning method.
Collapse
Affiliation(s)
- Xiyue Wang
- College of Biomedical Engineering, Sichuan University, Chengdu 610065, China; College of Computer Science, Sichuan University, Chengdu 610065, China
| | - Yuexi Du
- College of Engineering, University of Michigan, Ann Arbor, MI, 48109, United States
| | - Sen Yang
- Tencent AI Lab, Shenzhen 518057, China
| | - Jun Zhang
- Tencent AI Lab, Shenzhen 518057, China
| | - Minghui Wang
- College of Biomedical Engineering, Sichuan University, Chengdu 610065, China; College of Computer Science, Sichuan University, Chengdu 610065, China
| | - Jing Zhang
- College of Biomedical Engineering, Sichuan University, Chengdu 610065, China.
| | - Wei Yang
- Tencent AI Lab, Shenzhen 518057, China
| | | | - Xiao Han
- Tencent AI Lab, Shenzhen 518057, China.
| |
Collapse
|
13
|
Detecting the modality of a medical image using visual and textual features. Biomed Signal Process Control 2023. [DOI: 10.1016/j.bspc.2022.104035] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/22/2022]
|
14
|
Anand A, Flores AR, McDonald MF, Gadot R, Xu DS, Ropper AE. A computer vision approach to identifying the manufacturer of posterior thoracolumbar instrumentation systems. J Neurosurg Spine 2022; 38:417-424. [PMID: 36681945 DOI: 10.3171/2022.11.spine221009] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/06/2022] [Accepted: 11/17/2022] [Indexed: 12/28/2022]
Abstract
OBJECTIVE Knowledge of the manufacturer of the previously implanted pedicle screw systems prior to revision spinal surgery may facilitate faster and safer surgery. Often, this information is unavailable because patients are referred by other centers or because of missing information in the patients' records. Recently, machine learning and computer vision have gained wider use in clinical applications. The authors propose a computer vision approach to classify posterior thoracolumbar instrumentation systems. METHODS Lateral and anteroposterior (AP) radiographs obtained in patients undergoing posterior thoracolumbar pedicle screw implantation for any indication at the authors' institution (2015-2021) were obtained. DICOM images were cropped to include both the pedicle screws and rods. Images were labeled with the manufacturer according to the operative record. Multiple feature detection methods were tested (SURF, MESR, and Minimum Eigenvalues); however, the bag-of-visual-words technique with KAZE feature detection was ultimately used to construct a computer vision support vector machine (SVM) classifier for lateral, AP, and fused lateral and AP images. Accuracy was tested using an 80%/20% training/testing pseudorandom split over 100 iterations. Using a reader study, the authors compared the model performance with the current practice of surgeons and manufacturer representatives identifying spinal hardware by visual inspection. RESULTS Among the three image types, 355 lateral, 379 AP, and 338 fused radiographs were obtained. The five pedicle screw implants included in this study were the Globus Medical Creo, Medtronic Solera, NuVasive Reline, Stryker Xia, and DePuy Expedium. When the two most common manufacturers used at the authors' institution were binarily classified (Globus Medical and Medtronic), the accuracy rates for lateral, AP, and fused images were 93.15% ± 4.06%, 88.98% ± 4.08%, and 91.08% ± 5.30%, respectively. Classification accuracy decreased by approximately 10% with each additional manufacturer added. The multilevel five-way classification accuracy rates for lateral, AP, and fused images were 64.27% ± 5.13%, 60.95% ± 5.52%, and 65.90% ± 5.14%, respectively. In the reader study, the model performed five-way classification on 100 test images with 79% accuracy in 14 seconds, compared with an average of 44% accuracy in 20 minutes for two surgeons and three manufacturer representatives. CONCLUSIONS The authors developed a KAZE feature detector with an SVM classifier that successfully identified posterior thoracolumbar hardware at five-level classification. The model performed more accurately and efficiently than the method currently used in clinical practice. The relative computational simplicity of this model, from input to output, may facilitate future prospective studies in the clinical setting.
Collapse
Affiliation(s)
- Adrish Anand
- 1Department of Neurosurgery, Baylor College of Medicine, Houston
| | - Alex R Flores
- 1Department of Neurosurgery, Baylor College of Medicine, Houston
| | - Malcolm F McDonald
- 1Department of Neurosurgery, Baylor College of Medicine, Houston.,2Medical Scientist Training Program, Baylor College of Medicine, Houston, Texas; and
| | - Ron Gadot
- 1Department of Neurosurgery, Baylor College of Medicine, Houston
| | - David S Xu
- 3Department of Neurosurgery, The Ohio State University School of Medicine, Columbus, Ohio
| | | |
Collapse
|
15
|
Silva W, Gonçalves T, Härmä K, Schröder E, Obmann VC, Barroso MC, Poellinger A, Reyes M, Cardoso JS. Computer-aided diagnosis through medical image retrieval in radiology. Sci Rep 2022; 12:20732. [PMID: 36456605 PMCID: PMC9715673 DOI: 10.1038/s41598-022-25027-2] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/16/2022] [Accepted: 11/23/2022] [Indexed: 12/02/2022] Open
Abstract
Currently, radiologists face an excessive workload, which leads to high levels of fatigue, and consequently, to undesired diagnosis mistakes. Decision support systems can be used to prioritize and help radiologists making quicker decisions. In this sense, medical content-based image retrieval systems can be of extreme utility by providing well-curated similar examples. Nonetheless, most medical content-based image retrieval systems work by finding the most similar image, which is not equivalent to finding the most similar image in terms of disease and its severity. Here, we propose an interpretability-driven and an attention-driven medical image retrieval system. We conducted experiments in a large and publicly available dataset of chest radiographs with structured labels derived from free-text radiology reports (MIMIC-CXR-JPG). We evaluated the methods on two common conditions: pleural effusion and (potential) pneumonia. As ground-truth to perform the evaluation, query/test and catalogue images were classified and ordered by an experienced board-certified radiologist. For a profound and complete evaluation, additional radiologists also provided their rankings, which allowed us to infer inter-rater variability, and yield qualitative performance levels. Based on our ground-truth ranking, we also quantitatively evaluated the proposed approaches by computing the normalized Discounted Cumulative Gain (nDCG). We found that the Interpretability-guided approach outperforms the other state-of-the-art approaches and shows the best agreement with the most experienced radiologist. Furthermore, its performance lies within the observed inter-rater variability.
Collapse
Affiliation(s)
- Wilson Silva
- grid.20384.3d0000 0004 0500 6380INESC TEC, Porto, Portugal ,grid.5808.50000 0001 1503 7226Faculty of Engineering, University of Porto, Porto, Portugal
| | - Tiago Gonçalves
- grid.20384.3d0000 0004 0500 6380INESC TEC, Porto, Portugal ,grid.5808.50000 0001 1503 7226Faculty of Engineering, University of Porto, Porto, Portugal
| | - Kirsi Härmä
- grid.5734.50000 0001 0726 5157Department of Diagnostic, Interventional and Pediatric Radiology, Inselspital, Bern University Hospital, University of Bern, Bern, Switzerland
| | - Erich Schröder
- grid.5734.50000 0001 0726 5157Department of Diagnostic, Interventional and Pediatric Radiology, Inselspital, Bern University Hospital, University of Bern, Bern, Switzerland
| | - Verena Carola Obmann
- grid.5734.50000 0001 0726 5157Department of Diagnostic, Interventional and Pediatric Radiology, Inselspital, Bern University Hospital, University of Bern, Bern, Switzerland
| | - María Cecilia Barroso
- grid.5734.50000 0001 0726 5157Department of Diagnostic, Interventional and Pediatric Radiology, Inselspital, Bern University Hospital, University of Bern, Bern, Switzerland
| | - Alexander Poellinger
- grid.5734.50000 0001 0726 5157Department of Diagnostic, Interventional and Pediatric Radiology, Inselspital, Bern University Hospital, University of Bern, Bern, Switzerland
| | - Mauricio Reyes
- grid.5734.50000 0001 0726 5157ARTORG Center for Biomedical Engineering Research, University of Bern, Bern, Switzerland
| | - Jaime S. Cardoso
- grid.20384.3d0000 0004 0500 6380INESC TEC, Porto, Portugal ,grid.5808.50000 0001 1503 7226Faculty of Engineering, University of Porto, Porto, Portugal
| |
Collapse
|
16
|
Da Q, Huang X, Li Z, Zuo Y, Zhang C, Liu J, Chen W, Li J, Xu D, Hu Z, Yi H, Guo Y, Wang Z, Chen L, Zhang L, He X, Zhang X, Mei K, Zhu C, Lu W, Shen L, Shi J, Li J, S S, Krishnamurthi G, Yang J, Lin T, Song Q, Liu X, Graham S, Bashir RMS, Yang C, Qin S, Tian X, Yin B, Zhao J, Metaxas DN, Li H, Wang C, Zhang S. DigestPath: A benchmark dataset with challenge review for the pathological detection and segmentation of digestive-system. Med Image Anal 2022; 80:102485. [DOI: 10.1016/j.media.2022.102485] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/02/2021] [Revised: 04/08/2022] [Accepted: 05/20/2022] [Indexed: 12/19/2022]
|
17
|
Mustafa S, Iqbal MW, Rana TA, Jaffar A, Shiraz M, Arif M, Chelloug SA. Entropy and Gaussian Filter-Based Adaptive Active Contour for Segmentation of Skin Lesions. COMPUTATIONAL INTELLIGENCE AND NEUROSCIENCE 2022; 2022:4348235. [PMID: 35909861 PMCID: PMC9325593 DOI: 10.1155/2022/4348235] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 04/14/2022] [Revised: 06/13/2022] [Accepted: 06/28/2022] [Indexed: 11/18/2022]
Abstract
Malignant melanoma is considered one of the deadliest skin diseases if ignored without treatment. The mortality rate caused by melanoma is more than two times that of other skin malignancy diseases. These facts encourage computer scientists to find automated methods to discover skin cancers. Nowadays, the analysis of skin images is widely used by assistant physicians to discover the first stage of the disease automatically. One of the challenges the computer science researchers faced when developing such a system is the un-clarity of the existing images, such as noise like shadows, low contrast, hairs, and specular reflections, which complicates detecting the skin lesions in that images. This paper proposes the solution to the problem mentioned earlier using the active contour method. Still, seed selection in the dynamic contour method has the main drawback of where it should start the segmentation process. This paper uses Gaussian filter-based maximum entropy and morphological processing methods to find automatic seed points for active contour. By incorporating this, it can segment the lesion from dermoscopic images automatically. Our proposed methodology tested quantitative and qualitative measures on standard dataset dermis and used to test the proposed method's reliability which shows encouraging results.
Collapse
Affiliation(s)
- Saleem Mustafa
- Department of Computer Science, Superior University, Lahore 54600, Pakistan
| | | | - Toqir A. Rana
- Department of Computer Science and IT, The University of Lahore, Lahore 54000, Pakistan
- School of Computer Sciences, Universiti Sains Malaysia, Penang 11800, Malaysia
| | - Arfan Jaffar
- Department of Computer Science, Superior University, Lahore 54600, Pakistan
| | - Muhammad Shiraz
- Department of Computer Science, Federal Urdu University of Arts, Science & Technology, Islamabad 44000, Pakistan
| | - Muhammad Arif
- Department of Computer Science and IT, The University of Lahore, Lahore 54000, Pakistan
| | - Samia Allaoua Chelloug
- Department of Information Technology, College of Computer and Information Sciences, Princess Nourah bint Abdulrahman University, P. O. Box 84428, Riyadh 11671, Saudi Arabia
| |
Collapse
|
18
|
Hybrid Spatiotemporal Contrastive Representation Learning for Content-Based Surgical Video Retrieval. ELECTRONICS 2022. [DOI: 10.3390/electronics11091353] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/04/2023]
Abstract
In the medical field, due to their economic and clinical benefits, there is a growing interest in minimally invasive surgeries and microscopic surgeries. These types of surgeries are often recorded during operations, and these recordings have become a key resource for education, patient disease analysis, surgical error analysis, and surgical skill assessment. However, manual searching in this collection of long-term surgical videos is an extremely labor-intensive and long-term task, requiring an effective content-based video analysis system. In this regard, previous methods for surgical video retrieval are based on handcrafted features which do not represent the video effectively. On the other hand, deep learning-based solutions were found to be effective in both surgical image and video analysis, where CNN-, LSTM- and CNN-LSTM-based methods were proposed in most surgical video analysis tasks. In this paper, we propose a hybrid spatiotemporal embedding method to enhance spatiotemporal representations using an adaptive fusion layer on top of the LSTM and temporal causal convolutional modules. To learn surgical video representations, we propose exploring the supervised contrastive learning approach to leverage label information in addition to augmented versions. By validating our approach to a video retrieval task on two datasets, Surgical Actions 160 and Cataract-101, we significantly improve on previous results in terms of mean average precision, 30.012 ± 1.778 vs. 22.54 ± 1.557 for Surgical Actions 160 and 81.134 ± 1.28 vs. 33.18 ± 1.311 for Cataract-101. We also validate the proposed method’s suitability for surgical phase recognition task using the benchmark Cholec80 surgical dataset, where our approach outperforms (with 90.2% accuracy) the state of the art.
Collapse
|
19
|
Chu H, Zeng H, Lai H, Tang Y. Efficient modal-aware feature learning with application in multimodal hashing. INTELL DATA ANAL 2022. [DOI: 10.3233/ida-215780] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
Abstract
Many retrieval applications can benefit from multiple modalities, for which how to represent multimodal data is the critical component. Most deep multimodal learning methods typically involve two steps to construct the joint representations: 1) learning of multiple intermediate features, with each intermediate feature corresponding to a modality, using separate and independent deep models; 2) merging the intermediate features into a joint representation using a fusion strategy. However, in the first step, these intermediate features do not have previous knowledge of each other and cannot fully exploit the information contained in the other modalities. In this paper, we present a modal-aware operation as a generic building block to capture the non-linear dependencies among the heterogeneous intermediate features, which can learn the underlying correlation structures in other multimodal data as soon as possible. The modal-aware operation consists of a kernel network and an attention network. The kernel network is utilized to learn the non-linear relationships with other modalities. The attention network finds the informative regions of these modal-aware features that are favorable for retrieval. We verify the proposed modal-aware feature learning in the multimodal hashing task. The experiments conducted on three public benchmark datasets demonstrate significant improvements in the performance of our method relative to state-of-the-art methods.
Collapse
Affiliation(s)
- Hanlu Chu
- School of Computer Science, South China Normal University, Guangdong, China
- School of Computer Science, South China Normal University, Guangdong, China
| | - Haien Zeng
- School of Computer Science and Engineering, Sun Yat-Sen University, Guangdong, China
- School of Computer Science, South China Normal University, Guangdong, China
| | - Hanjiang Lai
- School of Computer Science and Engineering, Sun Yat-Sen University, Guangdong, China
| | - Yong Tang
- School of Computer Science, South China Normal University, Guangdong, China
| |
Collapse
|
20
|
A comprehensive review of computer-aided whole-slide image analysis: from datasets to feature extraction, segmentation, classification and detection approaches. Artif Intell Rev 2022. [DOI: 10.1007/s10462-021-10121-0] [Citation(s) in RCA: 10] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/06/2023]
|
21
|
Automatic selection algorithm for region of interest of acne face image compression. EVOLUTIONARY INTELLIGENCE 2022. [DOI: 10.1007/s12065-021-00692-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/19/2022]
|
22
|
Park C, You SC, Jeon H, Jeong CW, Choi JW, Park RW. Development and Validation of the Radiology Common Data Model (R-CDM) for the International Standardization of Medical Imaging Data. Yonsei Med J 2022; 63:S74-S83. [PMID: 35040608 PMCID: PMC8790584 DOI: 10.3349/ymj.2022.63.s74] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/08/2021] [Revised: 10/28/2021] [Accepted: 10/31/2021] [Indexed: 12/02/2022] Open
Abstract
PURPOSE Digital Imaging and Communications in Medicine (DICOM), a standard file format for medical imaging data, contains metadata describing each file. However, metadata are often incomplete, and there is no standardized format for recording metadata, leading to inefficiency during the metadata-based data retrieval process. Here, we propose a novel standardization method for DICOM metadata termed the Radiology Common Data Model (R-CDM). MATERIALS AND METHODS R-CDM was designed to be compatible with Health Level Seven International (HL7)/Fast Healthcare Interoperability Resources (FHIR) and linked with the Observational Medical Outcomes Partnership (OMOP)-CDM to achieve a seamless link between clinical data and medical imaging data. The terminology system was standardized using the RadLex playbook, a comprehensive lexicon of radiology. As a proof of concept, the R-CDM conversion process was conducted with 41.7 TB of data from the Ajou University Hospital. The R-CDM database visualizer was developed to visualize the main characteristics of the R-CDM database. RESULTS Information from 2801360 cases and 87203226 DICOM files was organized into two tables constituting the R-CDM. Information on imaging device and image resolution was recorded with more than 99.9% accuracy. Furthermore, OMOP-CDM and R-CDM were linked to efficiently extract specific types of images from specific patient cohorts. CONCLUSION R-CDM standardizes the structure and terminology for recording medical imaging data to eliminate incomplete and unstandardized information. Successful standardization was achieved by the extract, transform, and load process and image classifier. We hope that the R-CDM will contribute to deep learning research in the medical imaging field by enabling the securement of large-scale medical imaging data from multinational institutions.
Collapse
Affiliation(s)
- ChulHyoung Park
- Department of Biomedical Informatics, Ajou University School of Medicine, Suwon, Korea
| | - Seng Chan You
- Department of Preventive Medicine, Yonsei University College of Medicine, Seoul, Korea
| | - Hokyun Jeon
- Department of Biomedical Informatics, Ajou University School of Medicine, Suwon, Korea
| | - Chang Won Jeong
- Medical Convergence Research Center, Wonkwang University, Iksan, Korea
| | - Jin Wook Choi
- Department of Radiology, Ajou University Medical Center, Suwon, Korea
| | - Rae Woong Park
- Department of Biomedical Informatics, Ajou University School of Medicine, Suwon, Korea
- Department of Biomedical Sciences, Ajou University Graduate School of Medicine, Suwon, Korea.
| |
Collapse
|
23
|
Kaya E, Gunec HG, Aydin KC, Urkmez ES, Duranay R, Ates HF. A deep learning approach to permanent tooth germ detection on pediatric panoramic radiographs. Imaging Sci Dent 2022; 52:275-281. [PMID: 36238699 PMCID: PMC9530294 DOI: 10.5624/isd.20220050] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/10/2022] [Revised: 05/19/2022] [Accepted: 06/01/2022] [Indexed: 12/01/2022] Open
Abstract
Purpose The aim of this study was to assess the performance of a deep learning system for permanent tooth germ detection on pediatric panoramic radiographs. Materials and Methods In total, 4518 anonymized panoramic radiographs of children between 5 and 13 years of age were collected. YOLOv4, a convolutional neural network (CNN)-based object detection model, was used to automatically detect permanent tooth germs. Panoramic images of children processed in LabelImg were trained and tested in the YOLOv4 algorithm. True-positive, false-positive, and false-negative rates were calculated. A confusion matrix was used to evaluate the performance of the model. Results The YOLOv4 model, which detected permanent tooth germs on pediatric panoramic radiographs, provided an average precision value of 94.16% and an F1 value of 0.90, indicating a high level of significance. The average YOLOv4 inference time was 90 ms. Conclusion The detection of permanent tooth germs on pediatric panoramic X-rays using a deep learning-based approach may facilitate the early diagnosis of tooth deficiency or supernumerary teeth and help dental practitioners find more accurate treatment options while saving time and effort.
Collapse
Affiliation(s)
- Emine Kaya
- Department of Pediatric Dentistry, Faculty of Dentistry, Istanbul Okan University, Istanbul, Turkey
| | - Huseyin Gurkan Gunec
- Department of Endodontics, Faculty of Dentistry, Atlas University, Istanbul, Turkey
| | - Kader Cesur Aydin
- Department of Oral and Maxillofacial Radiology, Faculty of Dentistry, Istanbul Medipol University, Istanbul, Turkey
| | | | - Recep Duranay
- Department of Computer Engineering, Faculty of Engineering and Natural Sciences, Atlas University, Istanbul, Turkey
| | - Hasan Fehmi Ates
- Department of Computer Engineering, School of Engineering and Natural Sciences, Istanbul Medipol University, Istanbul, Turkey
| |
Collapse
|
24
|
Ahmedt-Aristizabal D, Armin MA, Denman S, Fookes C, Petersson L. A survey on graph-based deep learning for computational histopathology. Comput Med Imaging Graph 2022; 95:102027. [DOI: 10.1016/j.compmedimag.2021.102027] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/08/2021] [Revised: 11/25/2021] [Accepted: 12/04/2021] [Indexed: 12/21/2022]
|
25
|
Lesion-Based Chest Radiography Image Retrieval for Explainability in Pathology Detection. PATTERN RECOGNITION AND IMAGE ANALYSIS 2022. [DOI: 10.1007/978-3-031-04881-4_7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 10/18/2022]
|
26
|
A Survey on Big IoT Data Indexing: Potential Solutions, Recent Advancements, and Open Issues. FUTURE INTERNET 2021. [DOI: 10.3390/fi14010019] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022] Open
Abstract
The past decade has been characterized by the growing volumes of data due to the widespread use of the Internet of Things (IoT) applications, which introduced many challenges for efficient data storage and management. Thus, the efficient indexing and searching of large data collections is a very topical and urgent issue. Such solutions can provide users with valuable information about IoT data. However, efficient retrieval and management of such information in terms of index size and search time require optimization of indexing schemes which is rather difficult to implement. The purpose of this paper is to examine and review existing indexing techniques for large-scale data. A taxonomy of indexing techniques is proposed to enable researchers to understand and select the techniques that will serve as a basis for designing a new indexing scheme. The real-world applications of the existing indexing techniques in different areas, such as health, business, scientific experiments, and social networks, are presented. Open problems and research challenges, e.g., privacy and large-scale data mining, are also discussed.
Collapse
|
27
|
A Convolutional Neural Network for Automatic Tooth Numbering in Panoramic Images. BIOMED RESEARCH INTERNATIONAL 2021; 2021:3625386. [PMID: 34950732 PMCID: PMC8692013 DOI: 10.1155/2021/3625386] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 09/08/2021] [Revised: 11/02/2021] [Accepted: 11/24/2021] [Indexed: 12/30/2022]
Abstract
Analysis of dental radiographs and images is an important and common part of the diagnostic process in daily clinical practice. During the diagnostic process, the dentist must interpret, among others, tooth numbering. This study is aimed at proposing a convolutional neural network (CNN) that performs this task automatically for panoramic radiographs. A total of 8,000 panoramic images were categorized by two experts with more than three years of experience in general dentistry. The neural network consists of two main layers: object detection and classification, which is the support of the previous one and a transfer learning to improve computing time and precision. A Matterport Mask RCNN was employed in the object detection. A ResNet101 was employed in the classification layer. The neural model achieved a total loss of 6.17% (accuracy of 93.83%). The architecture of the model achieved an accuracy of 99.24% in tooth detection and 93.83% in numbering teeth with different oral health conditions.
Collapse
|
28
|
Zheng Y, Jiang Z, Shi J, Xie F, Zhang H, Luo W, Hu D, Sun S, Jiang Z, Xue C. Encoding histopathology whole slide images with location-aware graphs for diagnostically relevant regions retrieval. Med Image Anal 2021; 76:102308. [PMID: 34856455 DOI: 10.1016/j.media.2021.102308] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/19/2019] [Revised: 10/14/2021] [Accepted: 11/17/2021] [Indexed: 01/18/2023]
Abstract
Content-based histopathological image retrieval (CBHIR) has become popular in recent years in histopathological image analysis. CBHIR systems provide auxiliary diagnosis information for pathologists by searching for and returning regions that are contently similar to the region of interest (ROI) from a pre-established database. It is challenging and yet significant in clinical applications to retrieve diagnostically relevant regions from a database consisting of histopathological whole slide images (WSIs). In this paper, we propose a novel framework for regions retrieval from WSI database based on location-aware graphs and deep hash techniques. Compared to the present CBHIR framework, both structural information and global location information of ROIs in the WSI are preserved by graph convolution and self-attention operations, which makes the retrieval framework more sensitive to regions that are similar in tissue distribution. Moreover, benefited from the graph structure, the proposed framework has good scalability for both the size and shape variation of ROIs. It allows the pathologist to define query regions using free curves according to the appearance of tissue. Thirdly, the retrieval is achieved based on the hash technique, which ensures the framework is efficient and adequate for practical large-scale WSI database. The proposed method was evaluated on an in-house endometrium dataset with 2650 WSIs and the public ACDC-LungHP dataset. The experimental results have demonstrated that the proposed method achieved a mean average precision above 0.667 on the endometrium dataset and above 0.869 on the ACDC-LungHP dataset in the task of irregular region retrieval, which are superior to the state-of-the-art methods. The average retrieval time from a database containing 1855 WSIs is 0.752 ms. The source code is available at https://github.com/zhengyushan/lagenet.
Collapse
Affiliation(s)
- Yushan Zheng
- Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing 100191, China
| | - Zhiguo Jiang
- Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing 100191, China; Image Processing Center, School of Astronautics, Beihang University, Beijing 102206, China.
| | - Jun Shi
- School of Software, Hefei University of Technology, Hefei 230601, China.
| | - Fengying Xie
- Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing 100191, China; Image Processing Center, School of Astronautics, Beihang University, Beijing 102206, China
| | - Haopeng Zhang
- Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing 100191, China; Image Processing Center, School of Astronautics, Beihang University, Beijing 102206, China
| | - Wei Luo
- Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing 100191, China; Image Processing Center, School of Astronautics, Beihang University, Beijing 102206, China
| | - Dingyi Hu
- Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing 100191, China; Image Processing Center, School of Astronautics, Beihang University, Beijing 102206, China
| | - Shujiao Sun
- Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing 100191, China; Image Processing Center, School of Astronautics, Beihang University, Beijing 102206, China
| | - Zhongmin Jiang
- Department of Pathology, Tianjin Fifth Central Hospital, Tianjin 300450, China
| | - Chenghai Xue
- Wankangyuan Tianjin Gene Technology, Inc, Tianjin 300220, China; Tianjin Institute of Industrial Biotechnology, Chinese Academy of Sciences, Tianjin 300308, China
| |
Collapse
|
29
|
Unsupervised feature disentanglement for video retrieval in minimally invasive surgery. Med Image Anal 2021; 75:102296. [PMID: 34781159 DOI: 10.1016/j.media.2021.102296] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/16/2020] [Revised: 10/19/2021] [Accepted: 10/27/2021] [Indexed: 11/23/2022]
Abstract
In this paper, we propose a novel method of Unsupervised Disentanglement of Scene and Motion (UDSM) representations for minimally invasive surgery video retrieval within large databases, which has the potential to advance intelligent and efficient surgical teaching systems. To extract more discriminative video representations, two designed encoders with a triplet ranking loss and an adversarial learning mechanism are established to respectively capture the spatial and temporal information for achieving disentangled features from each frame with promising interpretability. In addition, the long-range temporal dependencies are improved in an integrated video level using a temporal aggregation module and then a set of compact binary codes that carries representative features is yielded to realize fast retrieval. The entire framework is trained in an unsupervised scheme, i.e., purely learning from raw surgical videos without using any annotation. We construct two large-scale minimally invasive surgery video datasets based on the public dataset Cholec80 and our in-house dataset of laparoscopic hysterectomy, to establish the learning process and validate the effectiveness of our proposed method qualitatively and quantitatively on the surgical video retrieval task. Extensive experiments show that our approach significantly outperforms the state-of-the-art video retrieval methods on both datasets, revealing a promising future for injecting intelligence in the next generation of surgical teaching systems.
Collapse
|
30
|
Pereira PMM, Thomaz LA, Tavora LMN, Assuncao PAA, Fonseca-Pinto RM, Paiva RP, Faria SMMD. Melanoma classification using light-Fields with morlet scattering transform and CNN: Surface depth as a valuable tool to increase detection rate. Med Image Anal 2021; 75:102254. [PMID: 34649195 DOI: 10.1016/j.media.2021.102254] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/31/2021] [Revised: 07/27/2021] [Accepted: 09/22/2021] [Indexed: 11/15/2022]
Abstract
Medical image classification through learning-based approaches has been increasingly used, namely in the discrimination of melanoma. However, for skin lesion classification in general, such methods commonly rely on dermoscopic or other 2D-macro RGB images. This work proposes to exploit beyond conventional 2D image characteristics, by considering a third dimension (depth) that characterises the skin surface rugosity, which can be obtained from light-field images, such as those available in the SKINL2 dataset. To achieve this goal, a processing pipeline was deployed using a morlet scattering transform and a CNN model, allowing to perform a comparison between using 2D information, only 3D information, or both. Results show that discrimination between Melanoma and Nevus reaches an accuracy of 84.00, 74.00 or 94.00% when using only 2D, only 3D, or both, respectively. An increase of 14.29pp in sensitivity and 8.33pp in specificity is achieved when expanding beyond conventional 2D information by also using depth. When discriminating between Melanoma and all other types of lesions (a further imbalanced setting), an increase of 28.57pp in sensitivity and decrease of 1.19pp in specificity is achieved for the same test conditions. Overall the results of this work demonstrate significant improvements over conventional approaches.
Collapse
Affiliation(s)
- Pedro M M Pereira
- Instituto de Telecomunicações, Morro do Lena - Alto do Vieiro, Leiria 2411-901, Portugal; University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Pinhal de Marrocos, Coimbra 3030-290, Portugal.
| | - Lucas A Thomaz
- Instituto de Telecomunicações, Morro do Lena - Alto do Vieiro, Leiria 2411-901, Portugal; ESTG, Polytechnic of Leiria, Morro do Lena - Alto do Vieiro, Leiria 2411-901, Portugal
| | - Luis M N Tavora
- ESTG, Polytechnic of Leiria, Morro do Lena - Alto do Vieiro, Leiria 2411-901, Portugal
| | - Pedro A A Assuncao
- Instituto de Telecomunicações, Morro do Lena - Alto do Vieiro, Leiria 2411-901, Portugal; ESTG, Polytechnic of Leiria, Morro do Lena - Alto do Vieiro, Leiria 2411-901, Portugal
| | - Rui M Fonseca-Pinto
- Instituto de Telecomunicações, Morro do Lena - Alto do Vieiro, Leiria 2411-901, Portugal; ESTG, Polytechnic of Leiria, Morro do Lena - Alto do Vieiro, Leiria 2411-901, Portugal
| | - Rui Pedro Paiva
- University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Pinhal de Marrocos, Coimbra 3030-290, Portugal
| | - Sergio M M de Faria
- Instituto de Telecomunicações, Morro do Lena - Alto do Vieiro, Leiria 2411-901, Portugal; ESTG, Polytechnic of Leiria, Morro do Lena - Alto do Vieiro, Leiria 2411-901, Portugal
| |
Collapse
|
31
|
He S, Wu J, Lian C, Gach H, Mutic S, Bosch W, Michalski J, Li H. An Adaptive Low-Rank Modeling-Based Active Learning Method for Medical Image Annotation. Ing Rech Biomed 2021; 42:334-344. [PMID: 34934476 PMCID: PMC8687126 DOI: 10.1016/j.irbm.2020.06.001] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/18/2022]
Abstract
Active learning is an effective solution to interactively select a limited number of informative examples and use them to train a learning algorithm that can achieve its optimal performance for specific tasks. It is suitable for medical image applications in which unlabeled data are abundant but manual annotation could be very time-consuming and expensive. However, designing an effective active learning strategy for informative example selection is a challenging task, due to the intrinsic presence of noise in medical images, the large number of images, and the variety of imaging modalities. In this study, a novel low-rank modeling-based multi-label active learning (LRMMAL) method is developed to address these challenges and select informative examples for training a classifier to achieve the optimal performance. The proposed method independently quantifies image noise and integrates it with other measures to guide a pool-based sampling process to determine the most informative examples for training a classifier. In addition, an automatic adaptive cross entropy-based parameter determination scheme is proposed for further optimizing the example sampling strategy. Experimental results on varied medical image datasets and comparisons with other state-of-the-art multi-label active learning methods illustrate the superior performance of the proposed method.
Collapse
Affiliation(s)
- S. He
- Department of Computer Science, Washington University, St. Louis, MO, USA
| | - J. Wu
- Department of Radiation Oncology, Washington University, St. Louis, MO, USA
| | - C. Lian
- Department of Radiology, The University of North Carolina at Chapel Hill, NC, USA
| | - H.M. Gach
- Department of Radiation Oncology, Washington University, St. Louis, MO, USA
| | - S. Mutic
- Department of Radiation Oncology, Washington University, St. Louis, MO, USA
| | - W. Bosch
- Department of Radiation Oncology, Washington University, St. Louis, MO, USA
| | - J. Michalski
- Department of Radiation Oncology, Washington University, St. Louis, MO, USA
| | - H. Li
- Department of BioEngineering, University of Illinois at Urbana-Champaign, Urbana, IL, USA
- Carle Cancer Center, Carle Foundation Hospital, Urbana, IL, USA
| |
Collapse
|
32
|
Barhoumi W, Khelifa A. Skin lesion image retrieval using transfer learning-based approach for query-driven distance recommendation. Comput Biol Med 2021; 137:104825. [PMID: 34507152 DOI: 10.1016/j.compbiomed.2021.104825] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/15/2021] [Revised: 08/11/2021] [Accepted: 08/29/2021] [Indexed: 12/27/2022]
Abstract
Content-Based Dermatological Lesion Retrieval (CBDLR) systems retrieve similar skin lesion images, with a pathology-confirmed diagnosis, for a given query image of a skin lesion. By producing an intuitive support to both inexperienced and experienced dermatologists, the early diagnosis through CBDLR screening can significantly enhance the patients' survival, while reducing the treatment cost. To deal with this issue, a CBDLR system is proposed in this study. This system integrates a similarity measure recommender which allows a dynamic selection of the adequate distance metric for each query image. The main contributions of this work reside in (i) the adoption of deep-learned features according to their performances for the classification of skin lesions into seven classes; and (ii) the automatic generation of ground truth that was investigated within the framework of transfer learning in order to recommend the most appropriate distance for any new query image. The proposed CBDLR system has been exhaustively evaluated using the challenging ISIC2018 and ISIC2019 datasets, and the obtained results show that the proposed system can provide a useful aided-decision while offering superior performances. Indeed, it outperforms similar CBDLR systems that adopt standard distances by at least 9% in terms of mAP@K.
Collapse
Affiliation(s)
- Walid Barhoumi
- Université de Tunis El Manar, Institut Supérieur d'Informatique, Research Team on Intelligent Systems in Imaging and Artificial Vision (SIIVA), LR16ES06 Laboratoire de recherche en Informatique, Modélisation et Traitement de l'Information et de la Connaissance (LIMTIC), 2 Rue Abou Rayhane Bayrouni, 2080, Ariana, Tunisia; Université de Carthage, Ecole Nationale d'Ingénieurs de Carthage, 45 Rue des Entrepreneurs, 2035, Tunis-Carthage, Tunisia.
| | - Afifa Khelifa
- Higher Institute of Technological Studies of Mahdia, 5111, Hiboun, Mahdia, Tunisia
| |
Collapse
|
33
|
Using DICOM Metadata for Radiological Image Series Categorization: a Feasibility Study on Large Clinical Brain MRI Datasets. J Digit Imaging 2021; 33:747-762. [PMID: 31950302 DOI: 10.1007/s10278-019-00308-x] [Citation(s) in RCA: 19] [Impact Index Per Article: 6.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/16/2023] Open
Abstract
The growing interest in machine learning (ML) in healthcare is driven by the promise of improved patient care. However, how many ML algorithms are currently being used in clinical practice? While the technology is present, as demonstrated in a variety of commercial products, clinical integration is hampered by a lack of infrastructure, processes, and tools. In particular, automating the selection of relevant series for a particular algorithm remains challenging. In this work, we propose a methodology to automate the identification of brain MRI sequences so that we can automatically route the relevant inputs for further image-related algorithms. The method relies on metadata required by the Digital Imaging and Communications in Medicine (DICOM) standard, resulting in generalizability and high efficiency (less than 0.4 ms/series). To support our claims, we test our approach on two large brain MRI datasets (40,000 studies in total) from two different institutions on two different continents. We demonstrate high levels of accuracy (ranging from 97.4 to 99.96%) and generalizability across the institutions. Given the complexity and variability of brain MRI protocols, we are confident that similar techniques could be applied to other forms of radiological imaging.
Collapse
|
34
|
Sotomayor CG, Mendoza M, Castañeda V, Farías H, Molina G, Pereira G, Härtel S, Solar M, Araya M. Content-Based Medical Image Retrieval and Intelligent Interactive Visual Browser for Medical Education, Research and Care. Diagnostics (Basel) 2021; 11:diagnostics11081470. [PMID: 34441404 PMCID: PMC8392084 DOI: 10.3390/diagnostics11081470] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/12/2021] [Revised: 08/03/2021] [Accepted: 08/09/2021] [Indexed: 01/17/2023] Open
Abstract
Medical imaging is essential nowadays throughout medical education, research, and care. Accordingly, international efforts have been made to set large-scale image repositories for these purposes. Yet, to date, browsing of large-scale medical image repositories has been troublesome, time-consuming, and generally limited by text search engines. A paradigm shift, by means of a query-by-example search engine, would alleviate these constraints and beneficially impact several practical demands throughout the medical field. The current project aims to address this gap in medical imaging consumption by developing a content-based image retrieval (CBIR) system, which combines two image processing architectures based on deep learning. Furthermore, a first-of-its-kind intelligent visual browser was designed that interactively displays a set of imaging examinations with similar visual content on a similarity map, making it possible to search for and efficiently navigate through a large-scale medical imaging repository, even if it has been set with incomplete and curated metadata. Users may, likewise, provide text keywords, in which case the system performs a content- and metadata-based search. The system was fashioned with an anonymizer service and designed to be fully interoperable according to international standards, to stimulate its integration within electronic healthcare systems and its adoption for medical education, research and care. Professionals of the healthcare sector, by means of a self-administered questionnaire, underscored that this CBIR system and intelligent interactive visual browser would be highly useful for these purposes. Further studies are warranted to complete a comprehensive assessment of the performance of the system through case description and protocolized evaluations by medical imaging specialists.
Collapse
Affiliation(s)
- Camilo G. Sotomayor
- Radiology Department, Clinical Hospital University of Chile, University of Chile, Santiago 8380453, Chile; (C.G.S.); (G.P.)
- Center for Medical Informatics and Telemedicine, Institute of Biomedical Sciences, Faculty of Medicine, University of Chile, Santiago 8380453, Chile; (V.C.); (S.H.)
- Department of Electronic Engineering, Federico Santa Maria Technical University, Valparaíso 2340000, Chile
| | - Marcelo Mendoza
- Department of Informatics, Federico Santa Maria Technical University, Santiago 8380453, Chile; (M.M.); (H.F.); (G.M.); (M.S.)
| | - Víctor Castañeda
- Center for Medical Informatics and Telemedicine, Institute of Biomedical Sciences, Faculty of Medicine, University of Chile, Santiago 8380453, Chile; (V.C.); (S.H.)
- Department of Medical Technology, Faculty of Medicine, University of Chile, Santiago 8380453, Chile
| | - Humberto Farías
- Department of Informatics, Federico Santa Maria Technical University, Santiago 8380453, Chile; (M.M.); (H.F.); (G.M.); (M.S.)
| | - Gabriel Molina
- Department of Informatics, Federico Santa Maria Technical University, Santiago 8380453, Chile; (M.M.); (H.F.); (G.M.); (M.S.)
| | - Gonzalo Pereira
- Radiology Department, Clinical Hospital University of Chile, University of Chile, Santiago 8380453, Chile; (C.G.S.); (G.P.)
| | - Steffen Härtel
- Center for Medical Informatics and Telemedicine, Institute of Biomedical Sciences, Faculty of Medicine, University of Chile, Santiago 8380453, Chile; (V.C.); (S.H.)
| | - Mauricio Solar
- Department of Informatics, Federico Santa Maria Technical University, Santiago 8380453, Chile; (M.M.); (H.F.); (G.M.); (M.S.)
| | - Mauricio Araya
- Department of Electronic Engineering, Federico Santa Maria Technical University, Valparaíso 2340000, Chile
- Correspondence:
| |
Collapse
|
35
|
Yu H, Zhang X, Song L, Jiang L, Huang X, Chen W, Zhang C, Li J, Yang J, Hu Z, Duan Q, Chen W, He X, Fan J, Jiang W, Zhang L, Qiu C, Gu M, Sun W, Zhang Y, Peng G, Shen W, Fu G. Large-scale gastric cancer screening and localization using multi-task deep neural network. Neurocomputing 2021. [DOI: 10.1016/j.neucom.2021.03.006] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/21/2022]
|
36
|
Lou M, Qi Y, Meng J, Xu C, Wang Y, Pi J, Ma Y. DCANet: Dual contextual affinity network for mass segmentation in whole mammograms. Med Phys 2021; 48:4291-4303. [PMID: 34061371 DOI: 10.1002/mp.15010] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/04/2021] [Revised: 04/27/2021] [Accepted: 05/25/2021] [Indexed: 12/26/2022] Open
Abstract
PURPOSE Breast mass segmentation in mammograms remains a crucial yet challenging topic in computer-aided diagnosis systems. Existing algorithms mainly used mass-centered patches to achieve mass segmentation, which is time-consuming and unstable in clinical diagnosis. Therefore, we aim to directly perform fully automated mass segmentation in whole mammograms with deep learning solutions. METHODS In this work, we propose a novel dual contextual affinity network (a.k.a., DCANet) for mass segmentation in whole mammograms. Based on the encoder-decoder structure, two lightweight yet effective contextual affinity modules including the global-guided affinity module (GAM) and the local-guided affinity module (LAM) are proposed. The former aggregates the features integrated by all positions and captures long-range contextual dependencies, aiming to enhance the feature representations of homogeneous regions. The latter emphasizes semantic information around each position and exploits contextual affinity based on the local field-of-view, aiming to improve the indistinction among heterogeneous regions. RESULTS The proposed DCANet is greatly demonstrated on two public mammographic databases including the DDSM and the INbreast, achieving the Dice similarity coefficient (DSC) of 85.95% and 84.65%, respectively. Both segmentation performance and computational efficiency outperform the current state-of-the-art methods. CONCLUSION According to extensive qualitative and quantitative analyses, we believe that the proposed fully automated approach has sufficient robustness to provide fast and accurate diagnoses for possible clinical breast mass segmentation.
Collapse
Affiliation(s)
- Meng Lou
- School of Information Science and Engineering, Lanzhou University, Lanzhou, Gansu, China
| | - Yunliang Qi
- School of Information Science and Engineering, Lanzhou University, Lanzhou, Gansu, China
| | - Jie Meng
- School of Information Science and Engineering, Lanzhou University, Lanzhou, Gansu, China
| | - Chunbo Xu
- School of Information Science and Engineering, Lanzhou University, Lanzhou, Gansu, China
| | - Yiming Wang
- School of Information Science and Engineering, Lanzhou University, Lanzhou, Gansu, China
| | - Jiande Pi
- School of Information Science and Engineering, Lanzhou University, Lanzhou, Gansu, China
| | - Yide Ma
- School of Information Science and Engineering, Lanzhou University, Lanzhou, Gansu, China
| |
Collapse
|
37
|
Pourasad Y, Cavallaro F. A Novel Image Processing Approach to Enhancement and Compression of X-ray Images. INTERNATIONAL JOURNAL OF ENVIRONMENTAL RESEARCH AND PUBLIC HEALTH 2021; 18:ijerph18136724. [PMID: 34206486 PMCID: PMC8297375 DOI: 10.3390/ijerph18136724] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 05/19/2021] [Revised: 06/17/2021] [Accepted: 06/19/2021] [Indexed: 11/28/2022]
Abstract
At present, there is an increase in the capacity of data generated and stored in the medical area. Thus, for the efficient handling of these extensive data, the compression methods need to be re-explored by considering the algorithm’s complexity. To reduce the redundancy of the contents of the image, thus increasing the ability to store or transfer information in optimal form, an image processing approach needs to be considered. So, in this study, two compression techniques, namely lossless compression and lossy compression, were applied for image compression, which preserves the image quality. Moreover, some enhancing techniques to increase the quality of a compressed image were employed. These methods were investigated, and several comparison results are demonstrated. Finally, the performance metrics were extracted and analyzed based on state-of-the-art methods. PSNR, MSE, and SSIM are three performance metrics that were used for the sample medical images. Detailed analysis of the measurement metrics demonstrates better efficiency than the other image processing techniques. This study helps to better understand these strategies and assists researchers in selecting a more appropriate technique for a given use case.
Collapse
Affiliation(s)
- Yaghoub Pourasad
- Department of Electrical Engineering, Urmia University of Technology, Urmia 17165-57166, Iran
- Correspondence:
| | - Fausto Cavallaro
- Department of Economics, University of Molise, Via De Sanctis, 86100 Campobasso, Italy;
| |
Collapse
|
38
|
Chan SW, Cheung YM. How to Make a Query in Image Retrieval with Partial Information Extracted from Multiple Image Samples? INT J PATTERN RECOGN 2021. [DOI: 10.1142/s0218001421540215] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/18/2022]
Abstract
The existing image retrieval methods generally require at least one complete image as a query sample. From the practical point of view, a user may not have an image sample in hand for query. Instead, partial information from multiple image samples would be available. This paper therefore attempts to deal with this problem by presenting a novel framework that allows a user to make an image query composed of several partial information extracted from multiple image samples via Boolean operations (i.e. AND, OR and NOT). Based on the request from the query, a Descriptor Cluster Label Table (DCLT) is designed to efficiently find out the result of Boolean operations on partial information. Experiments show the promising result of the proposed framework on commodity query and criminal investigation, respectively, although it is essentially applicable to different scenarios as well by changing descriptors.
Collapse
Affiliation(s)
- Sheung Wai Chan
- Department of Computer Science, Hong Kong Baptist University, Hong Kong SAR, P. R. China
| | - Yiu-Ming Cheung
- Department of Computer Science, Hong Kong Baptist University, Hong Kong SAR, P. R. China
| |
Collapse
|
39
|
Yasa Y, Çelik Ö, Bayrakdar IS, Pekince A, Orhan K, Akarsu S, Atasoy S, Bilgir E, Odabaş A, Aslan AF. An artificial intelligence proposal to automatic teeth detection and numbering in dental bite-wing radiographs. Acta Odontol Scand 2021; 79:275-281. [PMID: 33176533 DOI: 10.1080/00016357.2020.1840624] [Citation(s) in RCA: 18] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/26/2022]
Abstract
OBJECTIVES Radiological examination has an important place in dental practice, and it is frequently used in intraoral imaging. The correct numbering of teeth on radiographs is a routine practice that takes time for the dentist. This study aimed to propose an automatic detection system for the numbering of teeth in bitewing images using a faster Region-based Convolutional Neural Networks (R-CNN) method. METHODS The study included 1125 bite-wing radiographs of patients who attended the Faculty of Dentistry of Ordu University from 2018 to 2019. A faster R-CNN an advanced object identification method was used to identify the teeth. The confusion matrix was used as a metric and to evaluate the success of the model. RESULTS The deep CNN system (CranioCatch, Eskisehir, Turkey) was used to detect and number teeth in bitewing radiographs. Of 715 teeth in 109 bite-wing images, 697 were correctly numbered in the test data set. The F1 score, precision and sensitivity were 0.9515, 0.9293 and 0.9748, respectively. CONCLUSIONS A CNN approach for the analysis of bitewing images shows promise for detecting and numbering teeth. This method can save dentists time by automatically preparing dental charts.
Collapse
Affiliation(s)
- Yasin Yasa
- Department of Oral and Maxillofacial Radiology, Faculty of Dentistry, Ordu University, Ordu, Turkey
| | - Özer Çelik
- Department of Mathematics and Computer Science, Faculty of Science, Eskisehir Osmangazi University, Eskisehir, Turkey
| | - Ibrahim Sevki Bayrakdar
- Department of Oral and Maxillofacial Radiology, Faculty of Dentistry, Eskisehir Osmangazi University, Eskisehir, Turkey
| | - Adem Pekince
- Department of Oral and Maxillofacial Radiology, Faculty of Dentistry, Karabuk University, Karabuk, Turkey
| | - Kaan Orhan
- Department of Oral and Maxillofacial Radiology, Faculty of Dentistry, Ankara University, Ankara, Turkey
- Ankara University Medical Design Application and Research Center (MEDITAM), Ankara, Turkey
| | - Serdar Akarsu
- Department of Restorative Dentistry, Faculty of Dentistry, Ordu University, Ordu, Turkey
| | - Samet Atasoy
- Department of Restorative Dentistry, Faculty of Dentistry, Ordu University, Ordu, Turkey
| | - Elif Bilgir
- Department of Oral and Maxillofacial Radiology, Faculty of Dentistry, Eskisehir Osmangazi University, Eskisehir, Turkey
| | - Alper Odabaş
- Department of Mathematics and Computer Science, Faculty of Science, Eskisehir Osmangazi University, Eskisehir, Turkey
| | - Ahmet Faruk Aslan
- Department of Mathematics and Computer Science, Faculty of Science, Eskisehir Osmangazi University, Eskisehir, Turkey
| |
Collapse
|
40
|
Prados-Privado M, García Villalón J, Blázquez Torres A, Martínez-Martínez CH, Ivorra C. A Validation Employing Convolutional Neural Network for the Radiographic Detection of Absence or Presence of Teeth. J Clin Med 2021; 10:jcm10061186. [PMID: 33809045 PMCID: PMC8001963 DOI: 10.3390/jcm10061186] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/11/2021] [Revised: 03/09/2021] [Accepted: 03/10/2021] [Indexed: 11/22/2022] Open
Abstract
Dental radiography plays an important role in clinical diagnosis, treatment and making decisions. In recent years, efforts have been made on developing techniques to detect objects in images. The aim of this study was to detect the absence or presence of teeth using an effective convolutional neural network, which reduces calculation times and has success rates greater than 95%. A total of 8000 dental panoramic images were collected. Each image and each tooth was categorized, independently and manually, by two experts with more than three years of experience in general dentistry. The neural network used consists of two main layers: object detection and classification, which is the support of the previous one. A Matterport Mask RCNN was employed in the object detection. A ResNet (Atrous Convolution) was employed in the classification layer. The neural model achieved a total loss of 0.76% (accuracy of 99.24%). The architecture used in the present study returned an almost perfect accuracy in detecting teeth on images from different devices and different pathologies and ages.
Collapse
Affiliation(s)
- María Prados-Privado
- Asisa Dental, Research Department, C/José Abascal, 32, 28003 Madrid, Spain; (J.G.V.); (A.B.T.); (C.H.M.-M.); (C.I.)
- Department of Signal Theory and Communications, Higher Polytechnic School, Universidad de Alcalá de Henares, Ctra. Madrid-Barcelona, Km. 33,600, 28805 Alcala de Henares, Spain
- Department Continuum Mechanics and Structural Analysis, Higher Polytechnic School, Carlos III University, Avenida de la Universidad 30, Leganés, 28911 Madrid, Spain
- Correspondence:
| | - Javier García Villalón
- Asisa Dental, Research Department, C/José Abascal, 32, 28003 Madrid, Spain; (J.G.V.); (A.B.T.); (C.H.M.-M.); (C.I.)
| | - Antonio Blázquez Torres
- Asisa Dental, Research Department, C/José Abascal, 32, 28003 Madrid, Spain; (J.G.V.); (A.B.T.); (C.H.M.-M.); (C.I.)
- SysOnline, 30001 Murcia, Spain
| | - Carlos Hugo Martínez-Martínez
- Asisa Dental, Research Department, C/José Abascal, 32, 28003 Madrid, Spain; (J.G.V.); (A.B.T.); (C.H.M.-M.); (C.I.)
- Faculty of Medicine, Universidad Complutense de Madrid, Plaza de Ramón y Cajal, s/n, 28040 Madrid, Spain
| | - Carlos Ivorra
- Asisa Dental, Research Department, C/José Abascal, 32, 28003 Madrid, Spain; (J.G.V.); (A.B.T.); (C.H.M.-M.); (C.I.)
| |
Collapse
|
41
|
Wu H, Wang W, Zhong J, Lei B, Wen Z, Qin J. SCS-Net: A Scale and Context Sensitive Network for Retinal Vessel Segmentation. Med Image Anal 2021; 70:102025. [PMID: 33721692 DOI: 10.1016/j.media.2021.102025] [Citation(s) in RCA: 61] [Impact Index Per Article: 20.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/07/2020] [Revised: 02/24/2021] [Accepted: 02/25/2021] [Indexed: 02/01/2023]
Abstract
Accurately segmenting retinal vessel from retinal images is essential for the detection and diagnosis of many eye diseases. However, it remains a challenging task due to (1) the large variations of scale in the retinal vessels and (2) the complicated anatomical context of retinal vessels, including complex vasculature and morphology, the low contrast between some vessels and the background, and the existence of exudates and hemorrhage. It is difficult for a model to capture representative and distinguishing features for retinal vessels under such large scale and semantics variations. Limited training data also make this task even harder. In order to comprehensively tackle these challenges, we propose a novel scale and context sensitive network (a.k.a., SCS-Net) for retinal vessel segmentation. We first propose a scale-aware feature aggregation (SFA) module, aiming at dynamically adjusting the receptive fields to effectively extract multi-scale features. Then, an adaptive feature fusion (AFF) module is designed to guide efficient fusion between adjacent hierarchical features to capture more semantic information. Finally, a multi-level semantic supervision (MSS) module is employed to learn more distinctive semantic representation for refining the vessel maps. We conduct extensive experiments on the six mainstream retinal image databases (DRIVE, CHASEDB1, STARE, IOSTAR, HRF, and LES-AV). The experimental results demonstrate the effectiveness of the proposed SCS-Net, which is capable of achieving better segmentation performance than other state-of-the-art approaches, especially for the challenging cases with large scale variations and complex context environments.
Collapse
Affiliation(s)
- Huisi Wu
- College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China, 518060
| | - Wei Wang
- College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China, 518060
| | - Jiafu Zhong
- College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China, 518060
| | - Baiying Lei
- School of Biomedical Engineering, Health Science Centers, Shenzhen University, National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, Guangdong Key Laboratory for Biomedical Measurements and Ultrasound Imaging, Marshall Laboratory of Biomedical Engineering, AI Research Center for Medical Image Analysis and Diagnosis, Shenzhen, China, 518060.
| | - Zhenkun Wen
- College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China, 518060
| | - Jing Qin
- Center for Smart Health, School of Nursing, The Hong Kong Polytechnic University, Hong Kong
| |
Collapse
|
42
|
Zheng Y, Jiang Z, Xie F, Shi J, Zhang H, Huai J, Cao M, Yang X. Diagnostic Regions Attention Network (DRA-Net) for Histopathology WSI Recommendation and Retrieval. IEEE TRANSACTIONS ON MEDICAL IMAGING 2021; 40:1090-1103. [PMID: 33351756 DOI: 10.1109/tmi.2020.3046636] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
The development of whole slide imaging techniques and online digital pathology platforms have accelerated the popularization of telepathology for remote tumor diagnoses. During a diagnosis, the behavior information of the pathologist can be recorded by the platform and then archived with the digital case. The browsing path of the pathologist on the WSI is one of the valuable information in the digital database because the image content within the path is expected to be highly correlated with the diagnosis report of the pathologist. In this article, we proposed a novel approach for computer-assisted cancer diagnosis named session-based histopathology image recommendation (SHIR) based on the browsing paths on WSIs. To achieve the SHIR, we developed a novel diagnostic regions attention network (DRA-Net) to learn the pathology knowledge from the image content associated with the browsing paths. The DRA-Net does not rely on the pixel-level or region-level annotations of pathologists. All the data for training can be automatically collected by the digital pathology platform without interrupting the pathologists' diagnoses. The proposed approaches were evaluated on a gastric dataset containing 983 cases within 5 categories of gastric lesions. The quantitative and qualitative assessments on the dataset have demonstrated the proposed SHIR framework with the novel DRA-Net is effective in recommending diagnostically relevant cases for auxiliary diagnosis. The MRR and MAP for the recommendation are respectively 0.816 and 0.836 on the gastric dataset. The source code of the DRA-Net is available at https://github.com/zhengyushan/dpathnet.
Collapse
|
43
|
Abstract
STUDY DESIGN Retrospective observational study. OBJECTIVE To demonstrate the clinical usefulness of deep learning by identifying previous spinal implants through application of deep learning. SUMMARY OF BACKGROUND DATA Deep learning has recently been actively applied to medical images. However, despite many attempts to apply deep learning to medical images, the application has rarely been successful. We aimed to demonstrate the effectiveness and usefulness of deep learning in the medical field. The goal of this study was to demonstrate the clinical usefulness of deep learning by identifying previous spinal implants through application of deep learning. METHODS For deep learning algorithm development, radiographs were retrospectively obtained from clinical cases in which the patients had lumbar spine one-segment instrument surgery. A total of 2894 lumbar spine anteroposterior (AP: 1446 cases) and lateral (1448 cases) radiographs were collected. Labeling work was conducted for five different implants. We conducted experiments using three deep learning algorithms. The traditional deep neural network model built by coding the transfer learning algorithm, Google AutoML, and Apple Create ML. Recall (sensitivity) and precision (specificity) were measured after training. RESULTS Overall, each model performed well in identifying each pedicle screw implant. In conventional transfer learning, AP radiography showed 97.0% precision and 96.7% recall. Lateral radiography showed 98.7% precision and 98.2% recall. In Google AutoML, AP radiography showed 91.4% precision and 87.4% recall; lateral radiography showed 97.9% precision and 98.4% recall. In Apple Create ML, AP radiography showed 76.0% precision and 73.0% recall; lateral radiography showed 89.0% precision and 87.0% recall. In all deep learning algorithms, precision and recall were higher in lateral than in AP radiography. CONCLUSION The deep learning application is effective for spinal implant identification. This demonstrates that clinicians can use ML-based deep learning applications to improve clinical practice and patient care.Level of Evidence: 3.
Collapse
Affiliation(s)
- Hee-Seok Yang
- Department of Neurosurgery, Seoul Barunsesang Hospital, Seoul, South Korea
| | - Kwang-Ryeol Kim
- Department of Neurosurgery, International St. Mary's Hospital, Catholic Kwandong University, Incheon, South Korea
| | - Sungjun Kim
- Department of Radiology, Gangnam Severance Hospital, Yonsei University College of Medicine, Seoul, South Korea
| | - Jeong-Yoon Park
- Department of Neurosurgery, Spine and Spinal Cord Institute, Gangnam Severance Hospital, Yonsei University College of Medicine, Seoul, South Korea
| |
Collapse
|
44
|
Abstract
Histopathological images (HIs) are the gold standard for evaluating some types of tumors for cancer diagnosis. The analysis of such images is time and resource-consuming and very challenging even for experienced pathologists, resulting in inter-observer and intra-observer disagreements. One of the ways of accelerating such an analysis is to use computer-aided diagnosis (CAD) systems. This paper presents a review on machine learning methods for histopathological image analysis, including shallow and deep learning methods. We also cover the most common tasks in HI analysis, such as segmentation and feature extraction. Besides, we present a list of publicly available and private datasets that have been used in HI research.
Collapse
|
45
|
Zheng Y, Jiang Z, Zhang H, Xie F, Hu D, Sun S, Shi J, Xue C. Stain Standardization Capsule for Application-Driven Histopathological Image Normalization. IEEE J Biomed Health Inform 2021; 25:337-347. [PMID: 32248128 DOI: 10.1109/jbhi.2020.2983206] [Citation(s) in RCA: 13] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Abstract
Color consistency is crucial to developing robust deep learning methods for histopathological image analysis. With the increasing application of digital histopathological slides, the deep learning methods are probably developed based on the data from multiple medical centers. This requirement makes it a challenging task to normalize the color variance of histopathological images from different medical centers. In this paper, we propose a novel color standardization module named stain standardization capsule based on the capsule network and the corresponding dynamic routing algorithm. The proposed module can learn and generate uniform stain separation outputs for histopathological images in various color appearance without the reference to manually selected template images. The proposed module is light and can be jointly trained with the application-driven CNN model. The proposed method was validated on three histopathology datasets and a cytology dataset, and was compared with state-of-the-art methods. The experimental results have demonstrated that the SSC module is effective in improving the performance of histopathological image analysis and has achieved the best performance in the compared methods.
Collapse
|
46
|
Gu Y, Vyas K, Shen M, Yang J, Yang GZ. Deep Graph-Based Multimodal Feature Embedding for Endomicroscopy Image Retrieval. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2021; 32:481-492. [PMID: 32310786 DOI: 10.1109/tnnls.2020.2980129] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
Representation learning is a critical task for medical image analysis in computer-aided diagnosis. However, it is challenging to learn discriminative features due to the limited size of the data set and the lack of labels. In this article, we propose a deep graph-based multimodal feature embedding (DGMFE) framework for medical image retrieval with application to breast tissue classification by learning discriminative features of probe-based confocal laser endomicroscopy (pCLE). We first build a multimodality graph model based on the visual similarity between pCLE data and reference histology images. The latent similar pCLE-histology pairs are extracted by walking with the cyclic path on the graph while the dissimilar pairs are extracted based on the geodesic distance. Given the similar and dissimilar pairs, the latent feature space is discovered by reconstructing the similarity between pCLE and histology images via deep Siamese neural networks. The proposed method is evaluated on a clinical database with 700 pCLE mosaics. The accuracy of image retrieval demonstrates that DGMFE can outperform previous works on feature learning. Especially, the top-1 accuracy in an eight-class retrieval task is 0.739, thus demonstrating a 10% improvement compared to the state-of-the-art method.
Collapse
|
47
|
Yang E, Liu M, Yao D, Cao B, Lian C, Yap PT, Shen D. Deep Bayesian Hashing With Center Prior for Multi-Modal Neuroimage Retrieval. IEEE TRANSACTIONS ON MEDICAL IMAGING 2021; 40:503-513. [PMID: 33048672 PMCID: PMC7909752 DOI: 10.1109/tmi.2020.3030752] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/29/2023]
Abstract
Multi-modal neuroimage retrieval has greatly facilitated the efficiency and accuracy of decision making in clinical practice by providing physicians with previous cases (with visually similar neuroimages) and corresponding treatment records. However, existing methods for image retrieval usually fail when applied directly to multi-modal neuroimage databases, since neuroimages generally have smaller inter-class variation and larger inter-modal discrepancy compared to natural images. To this end, we propose a deep Bayesian hash learning framework, called CenterHash, which can map multi-modal data into a shared Hamming space and learn discriminative hash codes from imbalanced multi-modal neuroimages. The key idea to tackle the small inter-class variation and large inter-modal discrepancy is to learn a common center representation for similar neuroimages from different modalities and encourage hash codes to be explicitly close to their corresponding center representations. Specifically, we measure the similarity between hash codes and their corresponding center representations and treat it as a center prior in the proposed Bayesian learning framework. A weighted contrastive likelihood loss function is also developed to facilitate hash learning from imbalanced neuroimage pairs. Comprehensive empirical evidence shows that our method can generate effective hash codes and yield state-of-the-art performance in cross-modal retrieval on three multi-modal neuroimage datasets.
Collapse
|
48
|
Medical Image Retrieval Using Empirical Mode Decomposition with Deep Convolutional Neural Network. BIOMED RESEARCH INTERNATIONAL 2021; 2020:6687733. [PMID: 33426062 PMCID: PMC7781707 DOI: 10.1155/2020/6687733] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 11/08/2020] [Revised: 12/07/2020] [Accepted: 12/14/2020] [Indexed: 11/17/2022]
Abstract
Content-based medical image retrieval (CBMIR) systems attempt to search medical image database to narrow the semantic gap in medical image analysis. The efficacy of high-level medical information representation using features is a major challenge in CBMIR systems. Features play a vital role in the accuracy and speed of the search process. In this paper, we propose a deep convolutional neural network- (CNN-) based framework to learn concise feature vector for medical image retrieval. The medical images are decomposed into five components using empirical mode decomposition (EMD). The deep CNN is trained in a supervised way with multicomponent input, and the learned features are used to retrieve medical images. The IRMA dataset, containing 11,000 X-ray images, 116 classes, is used to validate the proposed method. We achieve a total IRMA error of 43.21 and a mean average precision of 0.86 for retrieval task and IRMA error of 68.48 and F1 measure of 0.66 on classification task, which is the best result compared with existing literature for this dataset.
Collapse
|
49
|
Zhao Y, Lai H, Yin J, Zhang Y, Yang S, Jia Z, Ma J. Zero-Shot Medical Image Retrieval for Emerging Infectious Diseases Based on Meta-Transfer Learning - Worldwide, 2020. China CDC Wkly 2020; 2:1004-1008. [PMID: 34594825 PMCID: PMC8422228 DOI: 10.46234/ccdcw2020.268] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/03/2020] [Accepted: 12/21/2020] [Indexed: 11/20/2022] Open
Affiliation(s)
- Yuying Zhao
- School of Computer Science and Engineering, Sun Yat-Sen University, Guangzhou, Guangdong, China
| | - Hanjiang Lai
- School of Computer Science and Engineering, Sun Yat-Sen University, Guangzhou, Guangdong, China
| | - Jian Yin
- School of Computer Science and Engineering, Sun Yat-Sen University, Guangzhou, Guangdong, China
| | - Yewu Zhang
- Center for Public Health Surveillance and Information Service, Chinese Center for Disease Control and Prevention, Beijing, China
| | - Shigui Yang
- State Key Laboratory for Diagnosis and Treatment of Infectious Diseases, Collaborative Innovation Center for Diagnosis and Treatment of Infectious Diseases, The First Affiliated Hospital, College of Medicine, Zhejiang University, Hangzhou, Zhejiang, China
| | - Zhongwei Jia
- School of Public Health, Peking University, Beijing, China
| | - Jiaqi Ma
- Center for Public Health Surveillance and Information Service, Chinese Center for Disease Control and Prevention, Beijing, China
| |
Collapse
|
50
|
Lv S, Chu Y, Zhang P, Ma S, Zhao M, Wang Z, Gu Y, Sun X. Improved efficiency of urine cell image segmentation using droplet microfluidics technology. Cytometry A 2020; 99:722-731. [PMID: 33342063 DOI: 10.1002/cyto.a.24296] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/11/2020] [Revised: 11/25/2020] [Accepted: 12/16/2020] [Indexed: 12/12/2022]
Abstract
Recent advances in the recognition of biological samples using machine vision have made this technology increasingly important in research and detection. Image segmentation is an important step in this process. This study focuses on how to reduce the interference factors such as the overlap between different types (or within the same type) of urine cells according to microfluidics and improve the machine vision segmentation accuracy for cell images. In this study, we demonstrate that the platform can realize this hypothesis using urine cell image segmentation as an example application. We first discuss the reported urine cell droplet microfluidic chip system, which can realize the test conditions in which urine cells are encapsulated in the droplet and isolated from salt crystallization and/or bacteria and other urine-formed elements. Then, based on the analysis conditions set in the aforementioned experiment, the proportions of red blood cells, white blood cells, and squamous epithelial cells covered by various formed elements in the total urine cells in the same urine sample are measured. We simultaneously analyze the percentage of urine cells covered by salt crystallization and the incidence of overlapping between urine cells. Finally, the Otsu algorithm is used to segment the urine cell images encapsulated by the droplet and the urine cell images not encapsulated by the droplet, and the Dice, Jaccard, precision, and recall values are calculated. The results suggest that the method of encapsulating single cells based on droplets can improve the image segmentation effect without optimizing the algorithm.
Collapse
Affiliation(s)
- Shuxing Lv
- School of Medical Laboratory, Tianjin Medical University, Tianjin, China
| | - Yuying Chu
- School of Medical Laboratory, Tianjin Medical University, Tianjin, China
| | - Panpan Zhang
- North China University of Science and Technology Affiliated Hospital, Tangshan, China
| | - Sike Ma
- Engineering Research Center of Learning-Based Intelligent System, Ministry of Education of China, Tianjin University of Technology, Tianjin, China
| | - Meng Zhao
- Engineering Research Center of Learning-Based Intelligent System, Ministry of Education of China, Tianjin University of Technology, Tianjin, China
| | - Zhexiang Wang
- School of Medical Laboratory, Tianjin Medical University, Tianjin, China
| | - Yajun Gu
- School of Medical Laboratory, Tianjin Medical University, Tianjin, China
| | - Xuguo Sun
- School of Medical Laboratory, Tianjin Medical University, Tianjin, China
| |
Collapse
|