1
|
Xie L, Xu Y, Zheng M, Chen Y, Sun M, Archer MA, Mao W, Tong Y, Wan Y. An anthropomorphic diagnosis system of pulmonary nodules using weak annotation-based deep learning. Comput Med Imaging Graph 2024; 118:102438. [PMID: 39426342 PMCID: PMC11620937 DOI: 10.1016/j.compmedimag.2024.102438] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/25/2024] [Revised: 09/18/2024] [Accepted: 09/19/2024] [Indexed: 10/21/2024]
Abstract
The accurate categorization of lung nodules in CT scans is an essential aspect in the prompt detection and diagnosis of lung cancer. The categorization of grade and texture for nodules is particularly significant since it can aid radiologists and clinicians to make better-informed decisions concerning the management of nodules. However, currently existing nodule classification techniques have a singular function of nodule classification and rely on an extensive amount of high-quality annotation data, which does not meet the requirements of clinical practice. To address this issue, we develop an anthropomorphic diagnosis system of pulmonary nodules (PN) based on deep learning (DL) that is trained by weak annotation data and has comparable performance to full-annotation based diagnosis systems. The proposed system uses DL models to classify PNs (benign vs. malignant) with weak annotations, which eliminates the need for time-consuming and labor-intensive manual annotations of PNs. Moreover, the PN classification networks, augmented with handcrafted shape features acquired through the ball-scale transform technique, demonstrate capability to differentiate PNs with diverse labels, including pure ground-glass opacities, part-solid nodules, and solid nodules. Through 5-fold cross-validation on two datasets, the system achieved the following results: (1) an Area Under Curve (AUC) of 0.938 for PN localization and an AUC of 0.912 for PN differential diagnosis on the LIDC-IDRI dataset of 814 testing cases, (2) an AUC of 0.943 for PN localization and an AUC of 0.815 for PN differential diagnosis on the in-house dataset of 822 testing cases. In summary, our system demonstrates efficient localization and differential diagnosis of PNs in a resource limited environment, and thus could be translated into clinical use in the future.
Collapse
Affiliation(s)
- Lipeng Xie
- School of Cyber Science and Engineering, Zhengzhou University, Zhengzhou, China
| | - Yongrui Xu
- Department of Cardio-thoracic Surgery, Nanjing Medical University Affiliated Wuxi People's Hospital, Wuxi, Jiangsu, China; Nanjing Medical University, Nanjing, Jiangsu, China
| | - Mingfeng Zheng
- Department of Cardio-thoracic Surgery, Nanjing Medical University Affiliated Wuxi People's Hospital, Wuxi, Jiangsu, China; Nanjing Medical University, Nanjing, Jiangsu, China
| | - Yundi Chen
- Department of Biomedical Engineering, Binghamton University, Binghamton, NY, USA
| | - Min Sun
- Division of Oncology, University of Pittsburgh Medical Center Hillman Cancer Center at St. Margaret, Pittsburgh, PA, USA
| | - Michael A Archer
- Division of Thoracic Surgery, SUNY Upstate Medical University, USA
| | - Wenjun Mao
- Department of Cardio-thoracic Surgery, Nanjing Medical University Affiliated Wuxi People's Hospital, Wuxi, Jiangsu, China; Nanjing Medical University, Nanjing, Jiangsu, China.
| | - Yubing Tong
- Medical Image Processing Group, Department of Radiology, University of Pennsylvania, 602 Goddard building, 3710 Hamilton Walk, Philadelphia, PA 19104, USA.
| | - Yuan Wan
- Department of Biomedical Engineering, Binghamton University, Binghamton, NY, USA.
| |
Collapse
|
2
|
Takenaka K, Kondo K, Hasegawa T. Segment-Based Unsupervised Learning Method in Sensor-Based Human Activity Recognition. SENSORS (BASEL, SWITZERLAND) 2023; 23:8449. [PMID: 37896542 PMCID: PMC10610695 DOI: 10.3390/s23208449] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/24/2023] [Revised: 09/22/2023] [Accepted: 10/11/2023] [Indexed: 10/29/2023]
Abstract
Sensor-based human activity recognition (HAR) is a task to recognize human activities, and HAR has an important role in analyzing human behavior such as in the healthcare field. HAR is typically implemented using traditional machine learning methods. In contrast to traditional machine learning methods, deep learning models can be trained end-to-end with automatic feature extraction from raw sensor data. Therefore, deep learning models can adapt to various situations. However, deep learning models require substantial amounts of training data, and annotating activity labels to construct a training dataset is cost-intensive due to the need for human labor. In this study, we focused on the continuity of activities and propose a segment-based unsupervised deep learning method for HAR using accelerometer sensor data. We define segment data as sensor data measured at one time, and this includes only a single activity. To collect the segment data, we propose a measurement method where the users only need to annotate the starting, changing, and ending points of their activity rather than the activity label. We developed a new segment-based SimCLR, which uses pairs of segment data, and propose a method that combines segment-based SimCLR with SDFD. We investigated the effectiveness of feature representations obtained by training the linear layer with fixed weights obtained by unsupervised learning methods. As a result, we demonstrated that the proposed combined method acquires generalized feature representations. The results of transfer learning on different datasets suggest that the proposed method is robust to the sampling frequency of the sensor data, although it requires more training data than other methods.
Collapse
Affiliation(s)
- Koki Takenaka
- Graduate School of Engineering, University of Fukui, Fukui 910-8507, Japan
| | - Kei Kondo
- Graduate School of Engineering, University of Fukui, Fukui 910-8507, Japan
| | - Tatsuhito Hasegawa
- Graduate School of Engineering, University of Fukui, Fukui 910-8507, Japan
| |
Collapse
|
3
|
Bento N, Rebelo J, Carreiro AV, Ravache F, Barandas M. Exploring Regularization Methods for Domain Generalization in Accelerometer-Based Human Activity Recognition. SENSORS (BASEL, SWITZERLAND) 2023; 23:6511. [PMID: 37514805 PMCID: PMC10386236 DOI: 10.3390/s23146511] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/22/2023] [Revised: 07/03/2023] [Accepted: 07/17/2023] [Indexed: 07/30/2023]
Abstract
The study of Domain Generalization (DG) has gained considerable momentum in the Machine Learning (ML) field. Human Activity Recognition (HAR) inherently encompasses diverse domains (e.g., users, devices, or datasets), rendering it an ideal testbed for exploring Domain Generalization. Building upon recent work, this paper investigates the application of regularization methods to bridge the generalization gap between traditional models based on handcrafted features and deep neural networks. We apply various regularizers, including sparse training, Mixup, Distributionally Robust Optimization (DRO), and Sharpness-Aware Minimization (SAM), to deep learning models and assess their performance in Out-of-Distribution (OOD) settings across multiple domains using homogenized public datasets. Our results show that Mixup and SAM are the best-performing regularizers. However, they are unable to match the performance of models based on handcrafted features. This suggests that while regularization techniques can improve OOD robustness to some extent, handcrafted features remain superior for domain generalization in HAR tasks.
Collapse
Affiliation(s)
- Nuno Bento
- Associação Fraunhofer Portugal Research, Rua Alfredo Allen 455/461, 4200-135 Porto, Portugal
| | - Joana Rebelo
- Associação Fraunhofer Portugal Research, Rua Alfredo Allen 455/461, 4200-135 Porto, Portugal
| | - André V Carreiro
- Associação Fraunhofer Portugal Research, Rua Alfredo Allen 455/461, 4200-135 Porto, Portugal
| | - François Ravache
- ICOM France, 1 Rue Brindejonc des Moulinais, 31500 Toulouse, France
| | - Marília Barandas
- Associação Fraunhofer Portugal Research, Rua Alfredo Allen 455/461, 4200-135 Porto, Portugal
| |
Collapse
|
4
|
Akter M, Ansary S, Khan MAM, Kim D. Human Activity Recognition Using Attention-Mechanism-Based Deep Learning Feature Combination. SENSORS (BASEL, SWITZERLAND) 2023; 23:5715. [PMID: 37420881 DOI: 10.3390/s23125715] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/30/2023] [Revised: 06/14/2023] [Accepted: 06/15/2023] [Indexed: 07/09/2023]
Abstract
Human activity recognition (HAR) performs a vital function in various fields, including healthcare, rehabilitation, elder care, and monitoring. Researchers are using mobile sensor data (i.e., accelerometer, gyroscope) by adapting various machine learning (ML) or deep learning (DL) networks. The advent of DL has enabled automatic high-level feature extraction, which has been effectively leveraged to optimize the performance of HAR systems. In addition, the application of deep-learning techniques has demonstrated success in sensor-based HAR across diverse domains. In this study, a novel methodology for HAR was introduced, which utilizes convolutional neural networks (CNNs). The proposed approach combines features from multiple convolutional stages to generate a more comprehensive feature representation, and an attention mechanism was incorporated to extract more refined features, further enhancing the accuracy of the model. The novelty of this study lies in the integration of feature combinations from multiple stages as well as in proposing a generalized model structure with CBAM modules. This leads to a more informative and effective feature extraction technique by feeding the model with more information in every block operation. This research used spectrograms of the raw signals instead of extracting hand-crafted features through intricate signal processing techniques. The developed model has been assessed on three datasets, including KU-HAR, UCI-HAR, and WISDM datasets. The experimental findings showed that the classification accuracies of the suggested technique on the KU-HAR, UCI-HAR, and WISDM datasets were 96.86%, 93.48%, and 93.89%, respectively. The other evaluation criteria also demonstrate that the proposed methodology is comprehensive and competent compared to previous works.
Collapse
Affiliation(s)
- Morsheda Akter
- Department of Electronics Engineering, Dong-A University, Busan 49315, Republic of Korea
| | - Shafew Ansary
- Department of Electronics Engineering, Dong-A University, Busan 49315, Republic of Korea
| | - Md Al-Masrur Khan
- Department of ICT Integrated Ocean Smart and Cities Engineering, Dong-A University, Busan 49315, Republic of Korea
| | - Dongwan Kim
- Department of Electronics Engineering, Dong-A University, Busan 49315, Republic of Korea
| |
Collapse
|
5
|
Liu H, Gamboa H, Schultz T. Sensor-Based Human Activity and Behavior Research: Where Advanced Sensing and Recognition Technologies Meet. SENSORS (BASEL, SWITZERLAND) 2022; 23:s23010125. [PMID: 36616723 PMCID: PMC9823715 DOI: 10.3390/s23010125] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/10/2022] [Accepted: 12/13/2022] [Indexed: 06/01/2023]
Abstract
Human activity recognition (HAR) and human behavior recognition (HBR) have been playing increasingly important roles in the digital age [...].
Collapse
Affiliation(s)
- Hui Liu
- Cognitive Systems Lab, University of Bremen, Enrique-Schmidt-Str. 5, 28359 Bremen, Germany
- LIBPhys (Laboratory for Instrumentation, Biomedical Engineering and Radiation Physics), NOVA School of Science and Technology (Campus de Caparica), 2829-516 Caparica, Portugal
| | - Hugo Gamboa
- LIBPhys (Laboratory for Instrumentation, Biomedical Engineering and Radiation Physics), NOVA School of Science and Technology (Campus de Caparica), 2829-516 Caparica, Portugal
| | - Tanja Schultz
- Cognitive Systems Lab, University of Bremen, Enrique-Schmidt-Str. 5, 28359 Bremen, Germany
| |
Collapse
|