51
|
Xie X, Huang Y, Ning W, Wu D, Li Z, Yang H. RDAD: A reconstructive and discriminative anomaly detection model based on transformer. INT J INTELL SYST 2022. [DOI: 10.1002/int.22974] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
Affiliation(s)
- Xin Xie
- School of Information Engineering East China Jiaotong University Nanchang China
| | - Yuhui Huang
- School of Information Engineering East China Jiaotong University Nanchang China
| | - Weiye Ning
- School of Information Engineering East China Jiaotong University Nanchang China
| | - Dengquan Wu
- School of Information Engineering East China Jiaotong University Nanchang China
| | - Zixi Li
- School of Information Engineering East China Jiaotong University Nanchang China
| | - Hao Yang
- State Grid Jiangxi Electric Power Co. Ltd., Electric Power Research Institute Nanchang China
| |
Collapse
|
52
|
Itoh H, Misawa M, Mori Y, Kudo SE, Oda M, Mori K. Positive-gradient-weighted object activation mapping: visual explanation of object detector towards precise colorectal-polyp localisation. Int J Comput Assist Radiol Surg 2022; 17:2051-2063. [PMID: 35939251 DOI: 10.1007/s11548-022-02696-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/11/2022] [Accepted: 05/31/2022] [Indexed: 11/29/2022]
Abstract
PURPOSE Precise polyp detection and localisation are essential for colonoscopy diagnosis. Statistical machine learning with a large-scale data set can contribute to the construction of a computer-aided diagnosis system for the prevention of overlooking and miss-localisation of a polyp in colonoscopy. We propose new visual explaining methods for a well-trained object detector, which achieves fast and accurate polyp detection with a bounding box towards a precise automated polyp localisation. METHOD We refine gradient-weighted class activation mapping for more accurate highlighting of important patterns in processing a convolutional neural network. Extending the refined mapping into multiscaled processing, we define object activation mapping that highlights important object patterns in an image for a detection task. Finally, we define polyp activation mapping to achieve precise polyp localisation by integrating adaptive local thresholding into object activation mapping. We experimentally evaluate the proposed visual explaining methods with four publicly available databases. RESULTS The refined mapping visualises important patterns in each convolutional layer more accurately than the original gradient-weighted class activation mapping. The object activation mapping clearly visualises important patterns in colonoscopic images for polyp detection. The polyp activation mapping localises the detected polyps in ETIS-Larib, CVC-Clinic and Kvasir-SEG database with mean Dice scores of 0.76, 0.72 and 0.72, respectively. CONCLUSIONS We developed new visual explaining methods for a convolutional neural network by refining and extending gradient-weighted class activation mapping. Experimental results demonstrated the validity of the proposed methods by showing that accurate visualisation of important patterns and localisation of polyps in a colonoscopic image. The proposed visual explaining methods are useful for the interpreting and applying a trained polyp detector.
Collapse
Affiliation(s)
- Hayato Itoh
- Graduate School of Informatics, Nagoya University, Furo-cho, Chikusa-ku, Nagoya, 464-8601, Japan.
| | - Masashi Misawa
- Digestive Disease Center, Showa University Northern Yokohama Hospital, Chigasaki-chuo 35-1, Tsuzuki-ku, Yokohama, 224-8503, Japan
| | - Yuichi Mori
- Digestive Disease Center, Showa University Northern Yokohama Hospital, Chigasaki-chuo 35-1, Tsuzuki-ku, Yokohama, 224-8503, Japan.,Clinical Effectiveness Research Group, University of Oslo, Gaustad Sykehus, Bygg 20, Sognsvannsveien 21, 0372, Oslo, Norway
| | - Shin-Ei Kudo
- Digestive Disease Center, Showa University Northern Yokohama Hospital, Chigasaki-chuo 35-1, Tsuzuki-ku, Yokohama, 224-8503, Japan
| | - Masahiro Oda
- Graduate School of Informatics, Nagoya University, Furo-cho, Chikusa-ku, Nagoya, 464-8601, Japan
| | - Kensaku Mori
- Graduate School of Informatics, Nagoya University, Furo-cho, Chikusa-ku, Nagoya, 464-8601, Japan
| |
Collapse
|
53
|
Cervical Cell Segmentation Method Based on Global Dependency and Local Attention. APPLIED SCIENCES-BASEL 2022. [DOI: 10.3390/app12157742] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/07/2022]
Abstract
The refined segmentation of nuclei and the cytoplasm is the most challenging task in the automation of cervical cell screening. The U-Shape network structure has demonstrated great superiority in the field of biomedical imaging. However, the classical U-Net network cannot effectively utilize mixed domain information and contextual information, and fails to achieve satisfactory results in this task. To address the above problems, a module based on global dependency and local attention (GDLA) for contextual information modeling and features refinement, is proposed in this study. It consists of three components computed in parallel, which are the global dependency module, the spatial attention module, and the channel attention module. The global dependency module models global contextual information to capture a priori knowledge of cervical cells, such as the positional dependence of the nuclei and cytoplasm, and the closure and uniqueness of the nuclei. The spatial attention module combines contextual information to extract cell boundary information and refine target boundaries. The channel and spatial attention modules are used to provide adaption of the input information, and make it easy to identify subtle but dominant differences of similar objects. Comparative and ablation experiments are conducted on the Herlev dataset, and the experimental results demonstrate the effectiveness of the proposed method, which surpasses the most popular existing channel attention, hybrid attention, and context networks in terms of the nuclei and cytoplasm segmentation metrics, achieving better segmentation performance than most previous advanced methods.
Collapse
|
54
|
Tavanapong W, Oh J, Riegler MA, Khaleel M, Mittal B, de Groen PC. Artificial Intelligence for Colonoscopy: Past, Present, and Future. IEEE J Biomed Health Inform 2022; 26:3950-3965. [PMID: 35316197 PMCID: PMC9478992 DOI: 10.1109/jbhi.2022.3160098] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/21/2022]
Abstract
During the past decades, many automated image analysis methods have been developed for colonoscopy. Real-time implementation of the most promising methods during colonoscopy has been tested in clinical trials, including several recent multi-center studies. All trials have shown results that may contribute to prevention of colorectal cancer. We summarize the past and present development of colonoscopy video analysis methods, focusing on two categories of artificial intelligence (AI) technologies used in clinical trials. These are (1) analysis and feedback for improving colonoscopy quality and (2) detection of abnormalities. Our survey includes methods that use traditional machine learning algorithms on carefully designed hand-crafted features as well as recent deep-learning methods. Lastly, we present the gap between current state-of-the-art technology and desirable clinical features and conclude with future directions of endoscopic AI technology development that will bridge the current gap.
Collapse
|
55
|
Matsubara K, Ibaraki M, Kinoshita T. DeepPVC: prediction of a partial volume-corrected map for brain positron emission tomography studies via a deep convolutional neural network. EJNMMI Phys 2022; 9:50. [PMID: 35907100 PMCID: PMC9339068 DOI: 10.1186/s40658-022-00478-8] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/21/2021] [Accepted: 07/20/2022] [Indexed: 12/04/2022] Open
Abstract
Background Partial volume correction with anatomical magnetic resonance (MR) images (MR-PVC) is useful for accurately quantifying tracer uptake on brain positron emission tomography (PET) images. However, MR segmentation processes for MR-PVC are time-consuming and prevent the widespread clinical use of MR-PVC. Here, we aimed to develop a deep learning model to directly predict PV-corrected maps from PET and MR images, ultimately improving the MR-PVC throughput. Methods We used MR T1-weighted and [11C]PiB PET images as input data from 192 participants from the Alzheimer’s Disease Neuroimaging Initiative database. We calculated PV-corrected maps as the training target using the region-based voxel-wise PVC method. Two-dimensional U-Net model was trained and validated by sixfold cross-validation with the dataset from the 156 participants, and then tested using MR T1-weighted and [11C]PiB PET images from 36 participants acquired at sites other than the training dataset. We calculated the structural similarity index (SSIM) of the PV-corrected maps and intraclass correlation (ICC) of the PV-corrected standardized uptake value between the region-based voxel-wise (RBV) PVC and deepPVC as indicators for validation and testing. Results A high SSIM (0.884 ± 0.021) and ICC (0.921 ± 0.042) were observed in the validation and test data (SSIM, 0.876 ± 0.028; ICC, 0.894 ± 0.051). The computation time required to predict a PV-corrected map for a participant (48 s without a graphics processing unit) was much shorter than that for the RBV PVC and MR segmentation processes. Conclusion These results suggest that the deepPVC model directly predicts PV-corrected maps from MR and PET images and improves the throughput of MR-PVC by skipping the MR segmentation processes. Supplementary Information The online version contains supplementary material available at 10.1186/s40658-022-00478-8.
Collapse
Affiliation(s)
- Keisuke Matsubara
- Department of Management Science and Engineering, Faculty of System Science and Technology, Akita Prefectural University, 84-4 Aza Ebinokuchi Tsuchiya, Yurihonjo, 015-0055, Japan. .,Department of Radiology and Nuclear Medicine, Research Institute for Brain and Blood Vessels, Akita Cerebrospinal and Cardiovascular Center, Akita, 010-0874, Japan.
| | - Masanobu Ibaraki
- Department of Radiology and Nuclear Medicine, Research Institute for Brain and Blood Vessels, Akita Cerebrospinal and Cardiovascular Center, Akita, 010-0874, Japan
| | - Toshibumi Kinoshita
- Department of Radiology and Nuclear Medicine, Research Institute for Brain and Blood Vessels, Akita Cerebrospinal and Cardiovascular Center, Akita, 010-0874, Japan
| | | |
Collapse
|
56
|
Ruiz L, Martinez F. Weakly Supervised Polyp Segmentation from an Attention Receptive Field Mechanism. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2022; 2022:3745-3748. [PMID: 36085632 DOI: 10.1109/embc48229.2022.9871158] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
Colorectal cancer is the third most incidence cancer world-around. Colonoscopies are the most effective resource to detect and segment abnormal polyp masses, considered as the main biomarker of this cancer. Nonetheless, some recent clinical studies have revealed a polyp miss rate up to 26% during the clinical routine. Also, the expert bias introduced during polyp shape characterization may induce to false-negative diagnosis. Current computational approaches have supported polyp segmentation but over controlled scenarios, where polyp frames have been labeled by an expert. These supervised representations are fully dependent of well-segmented polyps, in crop sequences that always report these masses. This work introduces an attention receptive field mechanism, that robustly recover the polyp shape, by learning non-local pixel relationship. Besides this deep representation is learning from a weakly supervised scheme that includes unlabeled background frames, to discriminate polyps from near structures like intestinal folds. The achieved results outperform state-of-the-art approaches achieving a 95.1% precision in the public CVC-Colon DB, with also competitive performance on other datasets. Clinical relevance-The work address a novel strategy to support segmentation tools in a clinical routine with redundant background over colonoscopy sequences.
Collapse
|
57
|
Colorectal polyp region extraction using saliency detection network with neutrosophic enhancement. Comput Biol Med 2022; 147:105760. [DOI: 10.1016/j.compbiomed.2022.105760] [Citation(s) in RCA: 17] [Impact Index Per Article: 8.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/27/2022] [Revised: 06/02/2022] [Accepted: 06/18/2022] [Indexed: 11/19/2022]
|
58
|
Caires Silveira E, Santos Corrêa CF, Madureira Silva L, Almeida Santos B, Mattos Pretti S, Freire de Melo F. Recognition of esophagitis in endoscopic images using transfer learning. World J Gastrointest Endosc 2022; 14:311-319. [PMID: 35719896 PMCID: PMC9157692 DOI: 10.4253/wjge.v14.i5.311] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/13/2021] [Revised: 07/15/2021] [Accepted: 04/26/2022] [Indexed: 02/06/2023] Open
Abstract
BACKGROUND Esophagitis is an inflammatory and damaging process of the esophageal mucosa, which is confirmed by endoscopic visualization and may, in extreme cases, result in stenosis, fistulization and esophageal perforation. The use of deep learning (a field of artificial intelligence) techniques can be considered to determine the presence of esophageal lesions compatible with esophagitis.
AIM To develop, using transfer learning, a deep neural network model to recognize the presence of esophagitis in endoscopic images.
METHODS Endoscopic images of 1932 patients with a diagnosis of esophagitis and 1663 patients without any pathological diagnosis provenient from the KSAVIR and HyperKSAVIR datasets were splitted in training (80%) and test (20%) and used to develop and evaluate a binary deep learning classifier built using the DenseNet-201 architecture, a densely connected convolutional network, with weights pretrained on the ImageNet image set and fine-tuned during training. The classifier model performance was evaluated in the test set according to accuracy, sensitivity, specificity and area under the receiver operating characteristic curve (AUC).
RESULTS The model was trained using Adam optimizer with a learning rate of 0.0001 and applying binary cross entropy loss function. In the test set (n = 719), the classifier achieved 93.32% accuracy, 93.18% sensitivity, 93.46% specificity and a 0.96 AUC. Heatmaps for spatial predictive relevance in esophagitis endoscopic images from the test set were also plotted. In face of the obtained results, the use of dense convolutional neural networks with pretrained and fine-tuned weights proves to be a good strategy for predictive modeling for esophagitis recognition in endoscopic images. In addition, adopting the classification approach combined with the subsequent plotting of heat maps associated with the classificatory decision gives greater explainability to the model.
CONCLUSION It is opportune to raise new studies involving transfer learning for the analysis of endoscopic images, aiming to improve, validate and disseminate its use for clinical practice.
Collapse
Affiliation(s)
- Elena Caires Silveira
- Multidisciplinary Institute of Health, Federal University of Bahia, Vitória da Conquista 45029-094, Bahia, Brazil
| | - Caio Fellipe Santos Corrêa
- Multidisciplinary Institute of Health, Federal University of Bahia, Vitória da Conquista 45029-094, Bahia, Brazil
| | - Leonardo Madureira Silva
- Multidisciplinary Institute of Health, Federal University of Bahia, Vitória da Conquista 45029-094, Bahia, Brazil
| | - Bruna Almeida Santos
- Multidisciplinary Institute of Health, Federal University of Bahia, Vitória da Conquista 45029-094, Bahia, Brazil
| | - Soraya Mattos Pretti
- Multidisciplinary Institute of Health, Federal University of Bahia, Vitória da Conquista 45029-094, Bahia, Brazil
| | - Fabrício Freire de Melo
- Multidisciplinary Institute of Health, Federal University of Bahia, Vitória da Conquista 45029-094, Bahia, Brazil
| |
Collapse
|
59
|
Yue G, Han W, Jiang B, Zhou T, Cong R, Wang T. Boundary Constraint Network with Cross Layer Feature Integration for Polyp Segmentation. IEEE J Biomed Health Inform 2022; 26:4090-4099. [PMID: 35536816 DOI: 10.1109/jbhi.2022.3173948] [Citation(s) in RCA: 13] [Impact Index Per Article: 6.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Abstract
Clinically, proper polyp localization in endoscopy images plays a vital role in the follow-up treatment (e.g., surgical planning). Deep convolutional neural networks (CNNs) provide a favoured prospect for automatic polyp segmentation and evade the limitations of visual inspection, e.g., subjectivity and overwork. However, most existing CNNs-based methods often provide unsatisfactory segmentation performance. In this paper, we propose a novel boundary constraint network, namely BCNet, for accurate polyp segmentation. The success of BCNet benefits from integrating cross-level context information and leveraging edge information. Specifically, to avoid the drawbacks caused by simple feature addition or concentration, BCNet applies a cross-layer feature integration strategy (CFIS) in fusing the features of the top-three highest layers, yielding a better performance. CFIS consists of three attention-driven cross-layer feature interaction modules (ACFIMs) and two global feature integration modules (GFIMs). ACFIM adaptively fuses the context information of the top-three highest layers via the self-attention mechanism instead of direct addition or concentration. GFIM integrates the fused information across layers with the guidance from global attention. To obtain accurate boundaries, BCNet introduces a bilateral boundary extraction module that explores the polyp and non-polyp information of the shallow layer collaboratively based on the high-level location information and boundary supervision. Through joint supervision of the polyp area and boundary, BCNet is able to get more accurate polyp masks. Experimental results on three public datasets show that the proposed BCNet outperforms seven state-of-the-art competing methods in terms of both effectiveness and generalization.
Collapse
|
60
|
Shamrat FMJM, Azam S, Karim A, Islam R, Tasnim Z, Ghosh P, De Boer F. LungNet22: A Fine-Tuned Model for Multiclass Classification and Prediction of Lung Disease Using X-ray Images. J Pers Med 2022; 12:jpm12050680. [PMID: 35629103 PMCID: PMC9143659 DOI: 10.3390/jpm12050680] [Citation(s) in RCA: 19] [Impact Index Per Article: 9.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/10/2022] [Revised: 04/01/2022] [Accepted: 04/20/2022] [Indexed: 12/29/2022] Open
Abstract
In recent years, lung disease has increased manyfold, causing millions of casualties annually. To combat the crisis, an efficient, reliable, and affordable lung disease diagnosis technique has become indispensable. In this study, a multiclass classification of lung disease from frontal chest X-ray imaging using a fine-tuned CNN model is proposed. The classification is conducted on 10 disease classes of the lungs, namely COVID-19, Effusion, Tuberculosis, Pneumonia, Lung Opacity, Mass, Nodule, Pneumothorax, and Pulmonary Fibrosis, along with the Normal class. The dataset is a collective dataset gathered from multiple sources. After pre-processing and balancing the dataset with eight augmentation techniques, a total of 80,000 X-ray images were fed to the model for classification purposes. Initially, eight pre-trained CNN models, AlexNet, GoogLeNet, InceptionV3, MobileNetV2, VGG16, ResNet 50, DenseNet121, and EfficientNetB7, were employed on the dataset. Among these, the VGG16 achieved the highest accuracy at 92.95%. To further improve the classification accuracy, LungNet22 was constructed upon the primary structure of the VGG16 model. An ablation study was used in the work to determine the different hyper-parameters. Using the Adam Optimizer, the proposed model achieved a commendable accuracy of 98.89%. To verify the performance of the model, several performance matrices, including the ROC curve and the AUC values, were computed as well.
Collapse
Affiliation(s)
- F. M. Javed Mehedi Shamrat
- Department of Software Engineering, Daffodil International University, Dhaka 1207, Bangladesh; (F.M.J.M.S.); (Z.T.)
| | - Sami Azam
- College of Engineering, IT and Environment, Charles Darwin University, Casuarina, NT 0909, Australia; (A.K.); (F.D.B.)
- Correspondence:
| | - Asif Karim
- College of Engineering, IT and Environment, Charles Darwin University, Casuarina, NT 0909, Australia; (A.K.); (F.D.B.)
| | - Rakibul Islam
- Department of Computer Science and Engineering, Daffodil International University, Dhaka 1207, Bangladesh;
| | - Zarrin Tasnim
- Department of Software Engineering, Daffodil International University, Dhaka 1207, Bangladesh; (F.M.J.M.S.); (Z.T.)
| | - Pronab Ghosh
- Department of Computer Science (CS), Lakehead University, 955 Oliver Rd, Thunder Bay, ON P7B 5E1, Canada;
| | - Friso De Boer
- College of Engineering, IT and Environment, Charles Darwin University, Casuarina, NT 0909, Australia; (A.K.); (F.D.B.)
| |
Collapse
|
61
|
Hicks SA, Strümke I, Thambawita V, Hammou M, Riegler MA, Halvorsen P, Parasa S. On evaluation metrics for medical applications of artificial intelligence. Sci Rep 2022; 12:5979. [PMID: 35395867 PMCID: PMC8993826 DOI: 10.1038/s41598-022-09954-8] [Citation(s) in RCA: 141] [Impact Index Per Article: 70.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/24/2021] [Accepted: 03/30/2022] [Indexed: 12/18/2022] Open
Abstract
Clinicians and software developers need to understand how proposed machine learning (ML) models could improve patient care. No single metric captures all the desirable properties of a model, which is why several metrics are typically reported to summarize a model’s performance. Unfortunately, these measures are not easily understandable by many clinicians. Moreover, comparison of models across studies in an objective manner is challenging, and no tool exists to compare models using the same performance metrics. This paper looks at previous ML studies done in gastroenterology, provides an explanation of what different metrics mean in the context of binary classification in the presented studies, and gives a thorough explanation of how different metrics should be interpreted. We also release an open source web-based tool that may be used to aid in calculating the most relevant metrics presented in this paper so that other researchers and clinicians may easily incorporate them into their research.
Collapse
Affiliation(s)
- Steven A Hicks
- SimulaMet, Oslo, Norway. .,Oslo Metropolitan University, Oslo, Norway.
| | | | | | | | | | - Pål Halvorsen
- SimulaMet, Oslo, Norway.,Oslo Metropolitan University, Oslo, Norway
| | | |
Collapse
|
62
|
A functional central limit theorem on non-stationary random fields with nested spatial structure. STATISTICAL INFERENCE FOR STOCHASTIC PROCESSES 2022. [DOI: 10.1007/s11203-022-09273-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 10/18/2022]
Abstract
AbstractIn this paper, we establish a functional central limit theorem on high dimensional random fields in the context of model-based survey analysis. For strongly-mixing non-stationary random fields, we provide an upper bound for the fourth moment of the finite population total. This inequality is the generalization of a key tool for proving functional central limit theorems in Rio (Asymptotic theory of weakly dependent random processes, Springer, Berlin, 2017). Under the nested sampling strategy, we introduce assumptions on strongly-mixing coefficients and quantile functions to show that a functional stochastic process asymptotically approaches to a Gaussian process.
Collapse
|
63
|
COMMA: Propagating Complementary Multi-Level Aggregation Network for Polyp Segmentation. APPLIED SCIENCES-BASEL 2022. [DOI: 10.3390/app12042114] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/04/2023]
Abstract
Colonoscopy is an effective method for detecting polyps to prevent colon cancer. Existing studies have achieved satisfactory polyp detection performance by aggregating low-level boundary and high-level region information in convolutional neural networks (CNNs) for precise polyp segmentation in colonoscopy images. However, multi-level aggregation provides limited polyp segmentation owing to the distribution discrepancy that occurs when integrating different layer representations. To address this problem, previous studies have employed complementary low- and high- level representations. In contrast to existing methods, we focus on propagating complementary information such that the complementary low-level explicit boundary with abstracted high-level representations diminishes the discrepancy. This study proposes COMMA, which propagates complementary multi-level aggregation to reduce distribution discrepancies. COMMA comprises a complementary masking module (CMM) and a boundary propagation module (BPM) as a multi-decoder. The CMM masks the low-level boundary noises through the abstracted high-level representation and leverages the masked information at both levels. Similarly, the BPM incorporates the lowest- and highest-level representations to obtain explicit boundary information and propagates the boundary to the CMMs to improve polyp detection. CMMs can discriminate polyps more elaborately than prior CMMs based on boundary and complementary representations. Moreover, we propose a hybrid loss function to mitigate class imbalance and noisy annotations in polyp segmentation. To evaluate the COMMA performance, we conducted experiments on five benchmark datasets using five metrics. The results proved that the proposed network outperforms state-of-the-art methods in terms of all datasets. Specifically, COMMA improved mIoU performance by 0.043 on average for all datasets compared to the existing state-of-the-art methods.
Collapse
|
64
|
Guo X, Chen Z, Liu J, Yuan Y. Non-equivalent images and pixels: confidence-aware resampling with meta-learning mixup for polyp segmentation. Med Image Anal 2022; 78:102394. [DOI: 10.1016/j.media.2022.102394] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/19/2021] [Revised: 12/07/2021] [Accepted: 02/11/2022] [Indexed: 01/27/2023]
|
65
|
Wang D, Chen S, Sun X, Chen Q, Cao Y, Liu B, Liu X. AFP-Mask: Anchor-free Polyp Instance Segmentation in Colonoscopy. IEEE J Biomed Health Inform 2022; 26:2995-3006. [PMID: 35104234 DOI: 10.1109/jbhi.2022.3147686] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
Abstract
Colorectal cancer (CRC) is a common and lethal disease. Globally, CRC is the third most commonly diagnosed cancer in males and the second in females. The most effective way to prevent CRC is through using colonoscopy to identify and remove precancerous growths at an early stage. During colonoscopy, a tiny camera at the tip of the endoscope captures a video of the intestinal mucosa of the colon, while a specialized physician examines the lining of the entire colon and checks for any precancerous growths (polyps) through the live feed. The detection and removal of colorectal polyps have been found to be associated with a reduction in mortality from colorectal cancer. However, the false negative rate of polyp detection during colonoscopy is often high even for experienced physicians, due to the high variance in polyp shape, size, texture, color, and illumination, which make them difficult to detect. With recent advances in deep learning based object detection techniques, automated polyp detection shows great potential in helping physicians reduce false positive rate during colonoscopy. In this paper, we propose a novel anchor-free instance segmentation framework that can localize polyps and produce the corresponding instance level masks without using predefined anchor boxes. Our framework consists of two branches: (a) an object detection branch that performs classification and localization, (b) a mask generation branch that produces instance level masks. Instead of predicting a two-dimensional mask directly, we encode it into a compact representation vector, which allows us to incorporate instance segmentation with one-stage bounding-box detectors in a simple yet effective way. Moreover, our proposed encoding method can be trained jointly with object detector. Our experiment results show that our framework achieves a precision of 99.36% and a recall of 96.44% on public datasets, outperforming existing anchor-free instance segmentation methods by at least 2.8% in mIoU on our private dataset.
Collapse
|
66
|
Srivastava A, Jha D, Chanda S, Pal U, Johansen H, Johansen D, Riegler M, Ali S, Halvorsen P. MSRF-Net: A Multi-Scale Residual Fusion Network for Biomedical Image Segmentation. IEEE J Biomed Health Inform 2021; 26:2252-2263. [PMID: 34941539 DOI: 10.1109/jbhi.2021.3138024] [Citation(s) in RCA: 57] [Impact Index Per Article: 19.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Abstract
Methods based on convolutional neural networks have improved the performance of biomedical image segmentation. However, most of these methods cannot efficiently segment objects of variable sizes and train on small and biased datasets, which are common for biomedical use cases. While methods exist that incorporate multi-scale fusion approaches to address the challenges arising with variable sizes, they usually use complex models that are more suitable for general semantic segmentation problems. In this paper, we propose a novel architecture called Multi-Scale Residual Fusion Network (MSRF-Net), which is specially designed for medical image segmentation. The proposed MSRF-Net is able to exchange multi-scale features of varying receptive fields using a Dual-Scale Dense Fusion (DSDF) block. Our DSDF block can exchange information rigorously across two different resolution scales, and our MSRF sub-network uses multiple DSDF blocks in sequence to perform multi-scale fusion. This allows the preservation of resolution, improved information flow and propagation of both high- and low-level features to obtain accurate segmentation maps. The proposed MSRF-Net allows to capture object variabilities and provides improved results on different biomedical datasets. Extensive experiments on MSRF-Net demonstrate that the proposed method outperforms the cutting edge medical image segmentation methods on four publicly available datasets. We achieve the Dice Coefficient (DSC) of 0.9217, 0.9420, and 0.9224, 0.8824 on Kvasir-SEG, CVC-ClinicDB, 2018 Data Science Bowl dataset, and ISIC-2018 skin lesion segmentation challenge dataset respectively. We further conducted generalizability tests that also achieved the highest DSC score with 0.7921 and 0.7575 on CVC-ClinicDB and Kvasir-SEG, respectively.
Collapse
|
67
|
Yeung M, Sala E, Schönlieb CB, Rundo L. Focus U-Net: A novel dual attention-gated CNN for polyp segmentation during colonoscopy. Comput Biol Med 2021; 137:104815. [PMID: 34507156 PMCID: PMC8505797 DOI: 10.1016/j.compbiomed.2021.104815] [Citation(s) in RCA: 45] [Impact Index Per Article: 15.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/21/2021] [Revised: 08/26/2021] [Accepted: 08/26/2021] [Indexed: 02/07/2023]
Abstract
BACKGROUND Colonoscopy remains the gold-standard screening for colorectal cancer. However, significant miss rates for polyps have been reported, particularly when there are multiple small adenomas. This presents an opportunity to leverage computer-aided systems to support clinicians and reduce the number of polyps missed. METHOD In this work we introduce the Focus U-Net, a novel dual attention-gated deep neural network, which combines efficient spatial and channel-based attention into a single Focus Gate module to encourage selective learning of polyp features. The Focus U-Net incorporates several further architectural modifications, including the addition of short-range skip connections and deep supervision. Furthermore, we introduce the Hybrid Focal loss, a new compound loss function based on the Focal loss and Focal Tversky loss, designed to handle class-imbalanced image segmentation. For our experiments, we selected five public datasets containing images of polyps obtained during optical colonoscopy: CVC-ClinicDB, Kvasir-SEG, CVC-ColonDB, ETIS-Larib PolypDB and EndoScene test set. We first perform a series of ablation studies and then evaluate the Focus U-Net on the CVC-ClinicDB and Kvasir-SEG datasets separately, and on a combined dataset of all five public datasets. To evaluate model performance, we use the Dice similarity coefficient (DSC) and Intersection over Union (IoU) metrics. RESULTS Our model achieves state-of-the-art results for both CVC-ClinicDB and Kvasir-SEG, with a mean DSC of 0.941 and 0.910, respectively. When evaluated on a combination of five public polyp datasets, our model similarly achieves state-of-the-art results with a mean DSC of 0.878 and mean IoU of 0.809, a 14% and 15% improvement over the previous state-of-the-art results of 0.768 and 0.702, respectively. CONCLUSIONS This study shows the potential for deep learning to provide fast and accurate polyp segmentation results for use during colonoscopy. The Focus U-Net may be adapted for future use in newer non-invasive colorectal cancer screening and more broadly to other biomedical image segmentation tasks similarly involving class imbalance and requiring efficiency.
Collapse
Affiliation(s)
- Michael Yeung
- Department of Radiology, University of Cambridge, Cambridge, CB2 0QQ, United Kingdom; School of Clinical Medicine, University of Cambridge, Cambridge, CB2 0SP, United Kingdom.
| | - Evis Sala
- Department of Radiology, University of Cambridge, Cambridge, CB2 0QQ, United Kingdom; Cancer Research UK Cambridge Centre, University of Cambridge, Cambridge, CB2 0RE, United Kingdom.
| | - Carola-Bibiane Schönlieb
- Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge, CB3 0WA, United Kingdom.
| | - Leonardo Rundo
- Department of Radiology, University of Cambridge, Cambridge, CB2 0QQ, United Kingdom; Cancer Research UK Cambridge Centre, University of Cambridge, Cambridge, CB2 0RE, United Kingdom.
| |
Collapse
|
68
|
Dulf EH, Bledea M, Mocan T, Mocan L. Automatic Detection of Colorectal Polyps Using Transfer Learning. SENSORS (BASEL, SWITZERLAND) 2021; 21:5704. [PMID: 34502594 PMCID: PMC8433882 DOI: 10.3390/s21175704] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 07/26/2021] [Revised: 08/18/2021] [Accepted: 08/19/2021] [Indexed: 12/25/2022]
Abstract
Colorectal cancer is the second leading cause of cancer death and ranks third worldwide in diagnosed malignant pathologies (1.36 million new cases annually). An increase in the diversity of treatment options as well as a rising population require novel diagnostic tools. Current diagnostics involve critical human thinking, but the decisional process loses accuracy due to the increased number of modulatory factors involved. The proposed computer-aided diagnosis system analyses each colonoscopy and provides predictions that will help the clinician to make the right decisions. Artificial intelligence is included in the system both offline and online image processing tools. Aiming to improve the diagnostic process of colon cancer patients, an application was built that allows the easiest and most intuitive interaction between medical staff and the proposed diagnosis system. The developed tool uses two networks. The first, a convolutional neural network, is capable of classifying eight classes of tissue with a sensitivity of 98.13% and an F1 score of 98.14%, while the second network, based on semantic segmentation, can identify the malignant areas with a Jaccard index of 75.18%. The results could have a direct impact on personalised medicine combining clinical knowledge with the computing power of intelligent algorithms.
Collapse
Affiliation(s)
- Eva-H. Dulf
- Department of Automation, Faculty of Automation and Computer Science, Technical University of Cluj-Napoca, Memorandumului Str. 28, 400014 Cluj-Napoca, Romania;
| | - Marius Bledea
- Department of Automation, Faculty of Automation and Computer Science, Technical University of Cluj-Napoca, Memorandumului Str. 28, 400014 Cluj-Napoca, Romania;
| | - Teodora Mocan
- Department of Physiology, Iuliu Hatieganu University of Medicine and Pharmacy, 400349 Cluj-Napoca, Romania;
- Nanomedicine Department, Regional Institute of Gatroenterology and Hepatology, 400000 Cluj-Napoca, Romania
| | - Lucian Mocan
- Department of Surgery, 3-rd Surgery Clinic, Iuliu Hatieganu University of Medicine and Pharmacy, 400349 Cluj-Napoca, Romania;
| |
Collapse
|
69
|
Jha D, Ali S, Tomar NK, Johansen HD, Johansen D, Rittscher J, Riegler MA, Halvorsen P. Real-Time Polyp Detection, Localization and Segmentation in Colonoscopy Using Deep Learning. IEEE ACCESS : PRACTICAL INNOVATIONS, OPEN SOLUTIONS 2021; 9:40496-40510. [PMID: 33747684 PMCID: PMC7968127 DOI: 10.1109/access.2021.3063716] [Citation(s) in RCA: 69] [Impact Index Per Article: 23.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/02/2021] [Accepted: 02/15/2021] [Indexed: 05/16/2023]
Abstract
Computer-aided detection, localisation, and segmentation methods can help improve colonoscopy procedures. Even though many methods have been built to tackle automatic detection and segmentation of polyps, benchmarking of state-of-the-art methods still remains an open problem. This is due to the increasing number of researched computer vision methods that can be applied to polyp datasets. Benchmarking of novel methods can provide a direction to the development of automated polyp detection and segmentation tasks. Furthermore, it ensures that the produced results in the community are reproducible and provide a fair comparison of developed methods. In this paper, we benchmark several recent state-of-the-art methods using Kvasir-SEG, an open-access dataset of colonoscopy images for polyp detection, localisation, and segmentation evaluating both method accuracy and speed. Whilst, most methods in literature have competitive performance over accuracy, we show that the proposed ColonSegNet achieved a better trade-off between an average precision of 0.8000 and mean IoU of 0.8100, and the fastest speed of 180 frames per second for the detection and localisation task. Likewise, the proposed ColonSegNet achieved a competitive dice coefficient of 0.8206 and the best average speed of 182.38 frames per second for the segmentation task. Our comprehensive comparison with various state-of-the-art methods reveals the importance of benchmarking the deep learning methods for automated real-time polyp identification and delineations that can potentially transform current clinical practices and minimise miss-detection rates.
Collapse
Affiliation(s)
- Debesh Jha
- SimulaMet0167OsloNorway
- Department of Engineering ScienceBig Data Institute, University of OxfordOxfordOX3 7XFU.K.
| | - Sharib Ali
- Department of Engineering ScienceBig Data Institute, University of OxfordOxfordOX3 7XFU.K.
- Oxford NIHR Biomedical Research CentreOxfordOX4 2PGvU.K.
| | | | - Håvard D. Johansen
- Department of Computer ScienceUiT–The Arctic University of Norway9037TromsøNorway
| | - Dag Johansen
- Department of Computer ScienceUiT–The Arctic University of Norway9037TromsøNorway
| | - Jens Rittscher
- Department of Engineering ScienceBig Data Institute, University of OxfordOxfordOX3 7XFU.K.
- Oxford NIHR Biomedical Research CentreOxfordOX4 2PGvU.K.
| | | | - Pål Halvorsen
- SimulaMet0167OsloNorway
- Department of Computer ScienceOslo Metropolitan University0167OsloNorway
| |
Collapse
|
70
|
Jha D, Ali S, Hicks S, Thambawita V, Borgli H, Smedsrud PH, de Lange T, Pogorelov K, Wang X, Harzig P, Tran MT, Meng W, Hoang TH, Dias D, Ko TH, Agrawal T, Ostroukhova O, Khan Z, Atif Tahir M, Liu Y, Chang Y, Kirkerød M, Johansen D, Lux M, Johansen HD, Riegler MA, Halvorsen P. A comprehensive analysis of classification methods in gastrointestinal endoscopy imaging. Med Image Anal 2021; 70:102007. [PMID: 33740740 DOI: 10.1016/j.media.2021.102007] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/18/2020] [Revised: 01/20/2021] [Accepted: 02/16/2021] [Indexed: 12/24/2022]
Abstract
Gastrointestinal (GI) endoscopy has been an active field of research motivated by the large number of highly lethal GI cancers. Early GI cancer precursors are often missed during the endoscopic surveillance. The high missed rate of such abnormalities during endoscopy is thus a critical bottleneck. Lack of attentiveness due to tiring procedures, and requirement of training are few contributing factors. An automatic GI disease classification system can help reduce such risks by flagging suspicious frames and lesions. GI endoscopy consists of several multi-organ surveillance, therefore, there is need to develop methods that can generalize to various endoscopic findings. In this realm, we present a comprehensive analysis of the Medico GI challenges: Medical Multimedia Task at MediaEval 2017, Medico Multimedia Task at MediaEval 2018, and BioMedia ACM MM Grand Challenge 2019. These challenges are initiative to set-up a benchmark for different computer vision methods applied to the multi-class endoscopic images and promote to build new approaches that could reliably be used in clinics. We report the performance of 21 participating teams over a period of three consecutive years and provide a detailed analysis of the methods used by the participants, highlighting the challenges and shortcomings of the current approaches and dissect their credibility for the use in clinical settings. Our analysis revealed that the participants achieved an improvement on maximum Mathew correlation coefficient (MCC) from 82.68% in 2017 to 93.98% in 2018 and 95.20% in 2019 challenges, and a significant increase in computational speed over consecutive years.
Collapse
Affiliation(s)
- Debesh Jha
- SimulaMet, Oslo, Norway; UiT The Arctic University of Norway, Tromsø, Norway.
| | - Sharib Ali
- Department of Engineering Science, University of Oxford, Oxford, UK; Oxford NIHR Biomedical Research Centre, Oxford, UK
| | - Steven Hicks
- SimulaMet, Oslo, Norway; Oslo Metropolitan University, Oslo, Norway
| | | | - Hanna Borgli
- SimulaMet, Oslo, Norway; University of Oslo, Oslo, Norway
| | - Pia H Smedsrud
- SimulaMet, Oslo, Norway; University of Oslo, Oslo, Norway; Augere Medical AS, Oslo, Norway
| | - Thomas de Lange
- SimulaMet, Oslo, Norway; Augere Medical AS, Oslo, Norway; Sahlgrenska University Hospital, Molndal, Sweden; Bærum Hospital, Vestre Viken, Oslo, Norway
| | | | | | | | | | | | | | | | | | | | - Olga Ostroukhova
- Research Institute of Multiprocessor Computation Systems, Russia
| | - Zeshan Khan
- School of Computer Science, National University of Computer and Emerging Sciences, Karachi Campus, Pakistan
| | - Muhammad Atif Tahir
- School of Computer Science, National University of Computer and Emerging Sciences, Karachi Campus, Pakistan
| | - Yang Liu
- Hong Kong Baptist University, Hong Kong
| | - Yuan Chang
- Beijing University of Posts and Telecom., China
| | | | - Dag Johansen
- UiT The Arctic University of Norway, Tromsø, Norway
| | - Mathias Lux
- Alpen-Adria-Universität Klagenfurt, Klagenfurt, Austria
| | | | | | - Pål Halvorsen
- SimulaMet, Oslo, Norway; Oslo Metropolitan University, Oslo, Norway
| |
Collapse
|