1
|
Ferrara E. Large Language Models for Wearable Sensor-Based Human Activity Recognition, Health Monitoring, and Behavioral Modeling: A Survey of Early Trends, Datasets, and Challenges. SENSORS (BASEL, SWITZERLAND) 2024; 24:5045. [PMID: 39124092 PMCID: PMC11314694 DOI: 10.3390/s24155045] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/10/2024] [Revised: 07/29/2024] [Accepted: 07/31/2024] [Indexed: 08/12/2024]
Abstract
The proliferation of wearable technology enables the generation of vast amounts of sensor data, offering significant opportunities for advancements in health monitoring, activity recognition, and personalized medicine. However, the complexity and volume of these data present substantial challenges in data modeling and analysis, which have been addressed with approaches spanning time series modeling to deep learning techniques. The latest frontier in this domain is the adoption of large language models (LLMs), such as GPT-4 and Llama, for data analysis, modeling, understanding, and human behavior monitoring through the lens of wearable sensor data. This survey explores the current trends and challenges in applying LLMs for sensor-based human activity recognition and behavior modeling. We discuss the nature of wearable sensor data, the capabilities and limitations of LLMs in modeling them, and their integration with traditional machine learning techniques. We also identify key challenges, including data quality, computational requirements, interpretability, and privacy concerns. By examining case studies and successful applications, we highlight the potential of LLMs in enhancing the analysis and interpretation of wearable sensor data. Finally, we propose future directions for research, emphasizing the need for improved preprocessing techniques, more efficient and scalable models, and interdisciplinary collaboration. This survey aims to provide a comprehensive overview of the intersection between wearable sensor data and LLMs, offering insights into the current state and future prospects of this emerging field.
Collapse
Affiliation(s)
- Emilio Ferrara
- Thomas Lord Department of Computer Science, University of Southern California, Los Angeles, CA 90007, USA;
- Information Sciences Institute, School of Advanced Computing, University of Southern California, Los Angeles, CA 90007, USA
| |
Collapse
|
2
|
Sopidis G, Haslgrübler M, Azadi B, Guiza O, Schobesberger M, Anzengruber-Tanase B, Ferscha A. System Design for Sensing in Manufacturing to Apply AI through Hierarchical Abstraction Levels. SENSORS (BASEL, SWITZERLAND) 2024; 24:4508. [PMID: 39065907 PMCID: PMC11280824 DOI: 10.3390/s24144508] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/04/2024] [Revised: 07/08/2024] [Accepted: 07/11/2024] [Indexed: 07/28/2024]
Abstract
Activity recognition combined with artificial intelligence is a vital area of research, ranging across diverse domains, from sports and healthcare to smart homes. In the industrial domain, and the manual assembly lines, the emphasis shifts to human-machine interaction and thus to human activity recognition (HAR) within complex operational environments. Developing models and methods that can reliably and efficiently identify human activities, traditionally just categorized as either simple or complex activities, remains a key challenge in the field. Limitations of the existing methods and approaches include their inability to consider the contextual complexities associated with the performed activities. Our approach to address this challenge is to create different levels of activity abstractions, which allow for a more nuanced comprehension of activities and define their underlying patterns. Specifically, we propose a new hierarchical taxonomy for human activity abstraction levels based on the context of the performed activities that can be used in HAR. The proposed hierarchy consists of five levels, namely atomic, micro, meso, macro, and mega. We compare this taxonomy with other approaches that divide activities into simple and complex categories as well as other similar classification schemes and provide real-world examples in different applications to demonstrate its efficacy. Regarding advanced technologies like artificial intelligence, our study aims to guide and optimize industrial assembly procedures, particularly in uncontrolled non-laboratory environments, by shaping workflows to enable structured data analysis and highlighting correlations across various levels throughout the assembly progression. In addition, it establishes effective communication and shared understanding between researchers and industry professionals while also providing them with the essential resources to facilitate the development of systems, sensors, and algorithms for custom industrial use cases that adapt to the level of abstraction.
Collapse
Affiliation(s)
- Georgios Sopidis
- Pro2Future GmbH, Altenberger Strasse 69, 4040 Linz, Austria; (M.H.); (B.A.); (O.G.); (B.A.-T.)
- Institute of Pervasive Computing, Johannes Kepler University, Altenberger Straße 69, 4040 Linz, Austria; (M.S.); (A.F.)
| | - Michael Haslgrübler
- Pro2Future GmbH, Altenberger Strasse 69, 4040 Linz, Austria; (M.H.); (B.A.); (O.G.); (B.A.-T.)
| | - Behrooz Azadi
- Pro2Future GmbH, Altenberger Strasse 69, 4040 Linz, Austria; (M.H.); (B.A.); (O.G.); (B.A.-T.)
| | - Ouijdane Guiza
- Pro2Future GmbH, Altenberger Strasse 69, 4040 Linz, Austria; (M.H.); (B.A.); (O.G.); (B.A.-T.)
| | - Martin Schobesberger
- Institute of Pervasive Computing, Johannes Kepler University, Altenberger Straße 69, 4040 Linz, Austria; (M.S.); (A.F.)
| | | | - Alois Ferscha
- Institute of Pervasive Computing, Johannes Kepler University, Altenberger Straße 69, 4040 Linz, Austria; (M.S.); (A.F.)
| |
Collapse
|
3
|
Papadakis A, Spyrou E. A Multi-Modal Egocentric Activity Recognition Approach towards Video Domain Generalization. SENSORS (BASEL, SWITZERLAND) 2024; 24:2491. [PMID: 38676108 PMCID: PMC11054491 DOI: 10.3390/s24082491] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/22/2024] [Revised: 04/08/2024] [Accepted: 04/10/2024] [Indexed: 04/28/2024]
Abstract
Egocentric activity recognition is a prominent computer vision task that is based on the use of wearable cameras. Since egocentric videos are captured through the perspective of the person wearing the camera, her/his body motions severely complicate the video content, imposing several challenges. In this work we propose a novel approach for domain-generalized egocentric human activity recognition. Typical approaches use a large amount of training data, aiming to cover all possible variants of each action. Moreover, several recent approaches have attempted to handle discrepancies between domains with a variety of costly and mostly unsupervised domain adaptation methods. In our approach we show that through simple manipulation of available source domain data and with minor involvement from the target domain, we are able to produce robust models, able to adequately predict human activity in egocentric video sequences. To this end, we introduce a novel three-stream deep neural network architecture combining elements of vision transformers and residual neural networks which are trained using multi-modal data. We evaluate the proposed approach using a challenging, egocentric video dataset and demonstrate its superiority over recent, state-of-the-art research works.
Collapse
Affiliation(s)
- Antonios Papadakis
- Department of Informatics and Telecommunications, National Kapodistrian University of Athens, 15772 Athens, Greece;
| | - Evaggelos Spyrou
- Department of Informatics and Telecommunications, University of Thessaly, 35100 Lamia, Greece
| |
Collapse
|
4
|
Marín-García D, Bienvenido-Huertas D, Moyano J, Rubio-Bellido C, Rodríguez-Jiménez CE. Detection of activities in bathrooms through deep learning and environmental data graphics images. Heliyon 2024; 10:e26942. [PMID: 38533014 PMCID: PMC10963196 DOI: 10.1016/j.heliyon.2024.e26942] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/30/2023] [Revised: 02/21/2024] [Accepted: 02/21/2024] [Indexed: 03/28/2024] Open
Abstract
Automatic detection activities in indoor spaces has been and is a matter of great interest. Thus, in the field of health surveillance, one of the spaces frequently studied is the bathroom of homes and specifically the behaviour of users in the said space, since certain pathologies can sometimes be deduced from it. That is why, the objective of this study is to know if it is possible to automatically classify the main activities that occur within the bathroom, using an innovative methodology with respect to the methods used to date, based on environmental parameters and the application of machine learning algorithms, thus allowing privacy to be preserved, which is a notable improvement in relation to other methods. For this, the methodology followed is based on the novel application of a pre-trained convolutional network for classifying graphs resulting from the monitoring of the environmental parameters of a bathroom. The results obtained allow us to conclude that, in addition to being able to check whether environmental data are adequate for health, it is possible to detect a high rate of true positives (around 80%) in some of the most frequent and important activities, thus facilitating its automation in a very simple and economical way.
Collapse
Affiliation(s)
- David Marín-García
- Department of Graphical Expression and Building Engineering, Higher Technical School of Building Engineering, University de Seville, 4A Reina Mercedes Avenue, Seville 41012, Spain
| | - David Bienvenido-Huertas
- Department of Building Construction, Higher Technical School of Building Engineering University of Granada, Severo Ochoa, Granada 18071, Spain
| | - Juan Moyano
- Department of Graphical Expression and Building Engineering, Higher Technical School of Building Engineering, University de Seville, 4A Reina Mercedes Avenue, Seville 41012, Spain
| | - Carlos Rubio-Bellido
- Department of Building Construction II, Higher Technical School of Building Engineering, University de Seville, 4A Reina Mercedes Avenue, Seville 41012, Spain
| | - Carlos E. Rodríguez-Jiménez
- Department of Building Construction II, Higher Technical School of Building Engineering, University de Seville, 4A Reina Mercedes Avenue, Seville 41012, Spain
| |
Collapse
|
5
|
Konak O, van de Water R, Döring V, Fiedler T, Liebe L, Masopust L, Postnov K, Sauerwald F, Treykorn F, Wischmann A, Gjoreski H, Luštrek M, Arnrich B. HARE: Unifying the Human Activity Recognition Engineering Workflow. SENSORS (BASEL, SWITZERLAND) 2023; 23:9571. [PMID: 38067946 PMCID: PMC10708727 DOI: 10.3390/s23239571] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 10/13/2023] [Revised: 11/21/2023] [Accepted: 11/29/2023] [Indexed: 12/18/2023]
Abstract
Sensor-based human activity recognition is becoming ever more prevalent. The increasing importance of distinguishing human movements, particularly in healthcare, coincides with the advent of increasingly compact sensors. A complex sequence of individual steps currently characterizes the activity recognition pipeline. It involves separate data collection, preparation, and processing steps, resulting in a heterogeneous and fragmented process. To address these challenges, we present a comprehensive framework, HARE, which seamlessly integrates all necessary steps. HARE offers synchronized data collection and labeling, integrated pose estimation for data anonymization, a multimodal classification approach, and a novel method for determining optimal sensor placement to enhance classification results. Additionally, our framework incorporates real-time activity recognition with on-device model adaptation capabilities. To validate the effectiveness of our framework, we conducted extensive evaluations using diverse datasets, including our own collected dataset focusing on nursing activities. Our results show that HARE's multimodal and on-device trained model outperforms conventional single-modal and offline variants. Furthermore, our vision-based approach for optimal sensor placement yields comparable results to the trained model. Our work advances the field of sensor-based human activity recognition by introducing a comprehensive framework that streamlines data collection and classification while offering a novel method for determining optimal sensor placement.
Collapse
Affiliation(s)
- Orhan Konak
- Hasso Plattner Institute, University of Potsdam, 14482 Potsdam, Germany; (R.v.d.W.); (V.D.); (T.F.); (L.L.); (L.M.); (K.P.); (F.S.); (F.T.); (A.W.); (B.A.)
| | - Robin van de Water
- Hasso Plattner Institute, University of Potsdam, 14482 Potsdam, Germany; (R.v.d.W.); (V.D.); (T.F.); (L.L.); (L.M.); (K.P.); (F.S.); (F.T.); (A.W.); (B.A.)
| | - Valentin Döring
- Hasso Plattner Institute, University of Potsdam, 14482 Potsdam, Germany; (R.v.d.W.); (V.D.); (T.F.); (L.L.); (L.M.); (K.P.); (F.S.); (F.T.); (A.W.); (B.A.)
| | - Tobias Fiedler
- Hasso Plattner Institute, University of Potsdam, 14482 Potsdam, Germany; (R.v.d.W.); (V.D.); (T.F.); (L.L.); (L.M.); (K.P.); (F.S.); (F.T.); (A.W.); (B.A.)
| | - Lucas Liebe
- Hasso Plattner Institute, University of Potsdam, 14482 Potsdam, Germany; (R.v.d.W.); (V.D.); (T.F.); (L.L.); (L.M.); (K.P.); (F.S.); (F.T.); (A.W.); (B.A.)
| | - Leander Masopust
- Hasso Plattner Institute, University of Potsdam, 14482 Potsdam, Germany; (R.v.d.W.); (V.D.); (T.F.); (L.L.); (L.M.); (K.P.); (F.S.); (F.T.); (A.W.); (B.A.)
| | - Kirill Postnov
- Hasso Plattner Institute, University of Potsdam, 14482 Potsdam, Germany; (R.v.d.W.); (V.D.); (T.F.); (L.L.); (L.M.); (K.P.); (F.S.); (F.T.); (A.W.); (B.A.)
| | - Franz Sauerwald
- Hasso Plattner Institute, University of Potsdam, 14482 Potsdam, Germany; (R.v.d.W.); (V.D.); (T.F.); (L.L.); (L.M.); (K.P.); (F.S.); (F.T.); (A.W.); (B.A.)
| | - Felix Treykorn
- Hasso Plattner Institute, University of Potsdam, 14482 Potsdam, Germany; (R.v.d.W.); (V.D.); (T.F.); (L.L.); (L.M.); (K.P.); (F.S.); (F.T.); (A.W.); (B.A.)
| | - Alexander Wischmann
- Hasso Plattner Institute, University of Potsdam, 14482 Potsdam, Germany; (R.v.d.W.); (V.D.); (T.F.); (L.L.); (L.M.); (K.P.); (F.S.); (F.T.); (A.W.); (B.A.)
| | - Hristijan Gjoreski
- Faculty of Electrical Engineering and Information Technologies, Ss. Cyril and Methodius University in Skopje, 1000 Skopje, North Macedonia;
| | - Mitja Luštrek
- Department of Intelligent Systems, Jožef Stefan Institute, 1000 Ljubljana, Slovenia;
| | - Bert Arnrich
- Hasso Plattner Institute, University of Potsdam, 14482 Potsdam, Germany; (R.v.d.W.); (V.D.); (T.F.); (L.L.); (L.M.); (K.P.); (F.S.); (F.T.); (A.W.); (B.A.)
| |
Collapse
|
6
|
Nakanishi K, Goto H. A New Index for the Quantitative Evaluation of Surgical Invasiveness Based on Perioperative Patients' Behavior Patterns: Machine Learning Approach Using Triaxial Acceleration. JMIR Perioper Med 2023; 6:e50188. [PMID: 37962919 PMCID: PMC10685283 DOI: 10.2196/50188] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/22/2023] [Revised: 09/12/2023] [Accepted: 10/11/2023] [Indexed: 11/15/2023] Open
Abstract
BACKGROUND The minimally invasive nature of thoracoscopic surgery is well recognized; however, the absence of a reliable evaluation method remains challenging. We hypothesized that the postoperative recovery speed is closely linked to surgical invasiveness, where recovery signifies the patient's behavior transition back to their preoperative state during the perioperative period. OBJECTIVE This study aims to determine whether machine learning using triaxial acceleration data can effectively capture perioperative behavior changes and establish a quantitative index for quantifying variations in surgical invasiveness. METHODS We trained 7 distinct machine learning models using a publicly available human acceleration data set as supervised data. The 3 top-performing models were selected to predict patient actions, as determined by the Matthews correlation coefficient scores. Two patients who underwent different levels of invasive thoracoscopic surgery were selected as participants. Acceleration data were collected via chest sensors for 8 hours during the preoperative and postoperative hospitalization days. These data were categorized into 4 actions (walking, standing, sitting, and lying down) using the selected models. The actions predicted by the model with intermediate results were adopted as the actions of the participants. The daily appearance probability was calculated for each action. The 2 differences between 2 appearance probabilities (sitting vs standing and lying down vs walking) were calculated using 2 coordinates on the x- and y-axes. A 2D vector composed of coordinate values was defined as the index of behavior pattern (iBP) for the day. All daily iBPs were graphed, and the enclosed area and distance between points were calculated and compared between participants to assess the relationship between changes in the indices and invasiveness. RESULTS Patients 1 and 2 underwent lung lobectomy and incisional tumor biopsy, respectively. The selected predictive model was a light-gradient boosting model (mean Matthews correlation coefficient 0.98, SD 0.0027; accuracy: 0.98). The acceleration data yielded 548,466 points for patient 1 and 466,407 points for patient 2. The iBPs of patient 1 were [(0.32, 0.19), (-0.098, 0.46), (-0.15, 0.13), (-0.049, 0.22)] and those of patient 2 were [(0.55, 0.30), (0.77, 0.21), (0.60, 0.25), (0.61, 0.31)]. The enclosed areas were 0.077 and 0.0036 for patients 1 and 2, respectively. Notably, the distances for patient 1 were greater than those for patient 2 ({0.44, 0.46, 0.37, 0.26} vs {0.23, 0.0065, 0.059}; P=.03 [Mann-Whitney U test]). CONCLUSIONS The selected machine learning model effectively predicted the actions of the surgical patients with high accuracy. The temporal distribution of action times revealed changes in behavior patterns during the perioperative phase. The proposed index may facilitate the recognition and visualization of perioperative changes in patients and differences in surgical invasiveness.
Collapse
Affiliation(s)
- Kozo Nakanishi
- Department of General Thoracic Surgery, National Hospital Organization Saitama Hospital, Wako Saitama, Japan
| | - Hidenori Goto
- Department of General Thoracic Surgery, National Hospital Organization Saitama Hospital, Wako Saitama, Japan
| |
Collapse
|
7
|
Cao K, Wang M. Human behavior recognition based on sparse transformer with channel attention mechanism. Front Physiol 2023; 14:1239453. [PMID: 38028781 PMCID: PMC10653302 DOI: 10.3389/fphys.2023.1239453] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/14/2023] [Accepted: 10/19/2023] [Indexed: 12/01/2023] Open
Abstract
Human activity recognition (HAR) has recently become a popular research field in the wearable sensor technology scene. By analyzing the human behavior data, some disease risks or potential health issues can be detected, and patients' rehabilitation progress can be evaluated. With the excellent performance of Transformer in natural language processing and visual tasks, researchers have begun to focus on its application in time series. The Transformer model models long-term dependencies between sequences through self-attention mechanisms, capturing contextual information over extended periods. In this paper, we propose a hybrid model based on the channel attention mechanism and Transformer model to improve the feature representation ability of sensor-based HAR tasks. Extensive experiments were conducted on three public HAR datasets, and the results show that our network achieved accuracies of 98.10%, 97.21%, and 98.82% on the HARTH, PAMAP2, and UCI-HAR datasets, respectively, The overall performance is at the level of the most advanced methods.
Collapse
Affiliation(s)
- Keyan Cao
- School of Computer Science and Engineering, Shenyang Jianzhu University, Shenyang, Liaoning, China
| | | |
Collapse
|
8
|
Menaka SR, Prakash M, Neelakandan S, Radhakrishnan A. A novel WGF-LN based edge driven intelligence for wearable devices in human activity recognition. Sci Rep 2023; 13:17822. [PMID: 37857665 PMCID: PMC10587088 DOI: 10.1038/s41598-023-44213-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/19/2023] [Accepted: 10/05/2023] [Indexed: 10/21/2023] Open
Abstract
Human activity recognition (HAR) is one of the key applications of health monitoring that requires continuous use of wearable devices to track daily activities. The most efficient supervised machine learning (ML)-based approaches for predicting human activity are based on a continuous stream of sensor data. Sensor data analysis for human activity recognition using conventional algorithms and deep learning (DL) models shows promising results, but evaluating their ambiguity in decision-making is still challenging. In order to solve these issues, the paper proposes a novel Wasserstein gradient flow legonet WGF-LN-based human activity recognition system. At first, the input data is pre-processed. From the pre-processed data, the features are extracted using Haar Wavelet mother- Symlet wavelet coefficient scattering feature extraction (HS-WSFE). After that, the interest features are selected from the extracted features using (Binomial Distribution integrated-Golden Eagle Optimization) BD-GEO. The important features are then post-processed using the scatter plot matrix method. Obtained post-processing features are finally given into the WGF-LN for classifying human activities. From these experiments, the results can be obtained and showed the efficacy of the proposed model.
Collapse
Affiliation(s)
- S R Menaka
- Department of Information Technology, KSR College of Engineering, Tiruchengode, India
| | - M Prakash
- School of Computer Science and Engineering, Vellore Institute of Technology, Vellore, India
| | - S Neelakandan
- Department of Computer Science and Engineering, R.M.K. Engineering College, Kavaraipettai, Tamil Nadu, India
| | - Arun Radhakrishnan
- Faculty of Electrical and Computer Engineering, Jimma Institute of Technology, Jimma University, Jimma, Ethiopia.
| |
Collapse
|
9
|
Martínez-Zarzuela M, González-Alonso J, Antón-Rodríguez M, Díaz-Pernas FJ, Müller H, Simón-Martínez C. Multimodal video and IMU kinematic dataset on daily life activities using affordable devices. Sci Data 2023; 10:648. [PMID: 37737210 PMCID: PMC10516922 DOI: 10.1038/s41597-023-02554-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/06/2023] [Accepted: 09/08/2023] [Indexed: 09/23/2023] Open
Abstract
Human activity recognition and clinical biomechanics are challenging problems in physical telerehabilitation medicine. However, most publicly available datasets on human body movements cannot be used to study both problems in an out-of-the-lab movement acquisition setting. The objective of the VIDIMU dataset is to pave the way towards affordable patient gross motor tracking solutions for daily life activities recognition and kinematic analysis. The dataset includes 13 activities registered using a commodity camera and five inertial sensors. The video recordings were acquired in 54 subjects, of which 16 also had simultaneous recordings of inertial sensors. The novelty of dataset lies in: (i) the clinical relevance of the chosen movements, (ii) the combined utilization of affordable video and custom sensors, and (iii) the implementation of state-of-the-art tools for multimodal data processing of 3D body pose tracking and motion reconstruction in a musculoskeletal model from inertial data. The validation confirms that a minimally disturbing acquisition protocol, performed according to real-life conditions can provide a comprehensive picture of human joint angles during daily life activities.
Collapse
Affiliation(s)
| | | | | | | | - Henning Müller
- University of Applied Sciences and Arts Western Switzerland (HES-SO) Valais-Wallis, Sierre, Switzerland
- Medical faculty, University of Geneva, Geneva, Switzerland
| | - Cristina Simón-Martínez
- University of Applied Sciences and Arts Western Switzerland (HES-SO) Valais-Wallis, Sierre, Switzerland
| |
Collapse
|
10
|
Zha Q, Xu Z, Cai X, Zhang G, Shen X. Wearable rehabilitation wristband for distal radius fractures. Front Neurosci 2023; 17:1238176. [PMID: 37781255 PMCID: PMC10536142 DOI: 10.3389/fnins.2023.1238176] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/11/2023] [Accepted: 08/07/2023] [Indexed: 10/03/2023] Open
Abstract
Background Distal radius fractures are a common type of fracture. For patients treated with closed reduction with splinting, a period of rehabilitation is still required after the removal of the splint. However, there is a general lack of attention and low compliance to rehabilitation training during this period, so it is necessary to build a rehabilitation training monitoring system to improve the efficiency of patients' rehabilitation. Methods A wearable rehabilitation training wristband was proposed, which could be used in the patient's daily rehabilitation training scenario and could recognize four common wrist rehabilitation actions in real-time by using three thin film pressure sensors to detect the pressure change curve at three points on the wrist. An algorithmic framework for classifying rehabilitation training actions was proposed. In our framework, an action pre-detection strategy was designed to exclude false detections caused by switching initial gestures during rehabilitation training and wait for the arrival of the complete signal. To classify the action signals into four categories, firstly an autoencoder was used to downscale the original signal. Six SVMs were then used for evaluation and voting, and the final action with the highest number of votes would be used as the prediction result. Results Experimental results showed that the proposed algorithmic framework achieved an average recognition accuracy of 89.62%, an average recognition recall of 88.93%, and an f1 score of 89.27% on the four rehabilitation training actions. Conclusion The developed device has the advantages of being small size and easy to wear, which can quickly and accurately identify and classify four common rehabilitation training actions. It can easily be combined with peripheral devices and technologies (e.g., cell phones, computers, Internet) to build different rehabilitation training scenarios, making it worthwhile to use and promote in clinical settings.
Collapse
Affiliation(s)
- Qing Zha
- School of Biomedical Engineering (Suzhou), Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, China
- Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Science, Suzhou, China
| | - Zeou Xu
- School of Biomedical Engineering (Suzhou), Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, China
- Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Science, Suzhou, China
| | - Xuefeng Cai
- Suzhou TCM Hospital Affiliated to Nanjing University of Chinese Medicine, Suzhou, China
| | - Guodong Zhang
- Suzhou TCM Hospital Affiliated to Nanjing University of Chinese Medicine, Suzhou, China
| | - Xiaofeng Shen
- Suzhou TCM Hospital Affiliated to Nanjing University of Chinese Medicine, Suzhou, China
| |
Collapse
|
11
|
Jaramillo IE, Chola C, Jeong JG, Oh JH, Jung H, Lee JH, Lee WH, Kim TS. Human Activity Prediction Based on Forecasted IMU Activity Signals by Sequence-to-Sequence Deep Neural Networks. SENSORS (BASEL, SWITZERLAND) 2023; 23:6491. [PMID: 37514789 PMCID: PMC10385571 DOI: 10.3390/s23146491] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/10/2023] [Revised: 07/10/2023] [Accepted: 07/14/2023] [Indexed: 07/30/2023]
Abstract
Human Activity Recognition (HAR) has gained significant attention due to its broad range of applications, such as healthcare, industrial work safety, activity assistance, and driver monitoring. Most prior HAR systems are based on recorded sensor data (i.e., past information) recognizing human activities. In fact, HAR works based on future sensor data to predict human activities are rare. Human Activity Prediction (HAP) can benefit in multiple applications, such as fall detection or exercise routines, to prevent injuries. This work presents a novel HAP system based on forecasted activity data of Inertial Measurement Units (IMU). Our HAP system consists of a deep learning forecaster of IMU activity signals and a deep learning classifier to recognize future activities. Our deep learning forecaster model is based on a Sequence-to-Sequence structure with attention and positional encoding layers. Then, a pre-trained deep learning Bi-LSTM classifier is used to classify future activities based on the forecasted IMU data. We have tested our HAP system for five daily activities with two tri-axial IMU sensors. The forecasted signals show an average correlation of 91.6% to the actual measured signals of the five activities. The proposed HAP system achieves an average accuracy of 97.96% in predicting future activities.
Collapse
Affiliation(s)
- Ismael Espinoza Jaramillo
- Department of Electronics and Information Convergence Engineering, Kyung Hee University, Yongin 17104, Republic of Korea
| | - Channabasava Chola
- Department of Electronics and Information Convergence Engineering, Kyung Hee University, Yongin 17104, Republic of Korea
| | - Jin-Gyun Jeong
- Department of Electronics and Information Convergence Engineering, Kyung Hee University, Yongin 17104, Republic of Korea
| | - Ji-Heon Oh
- Department of Electronics and Information Convergence Engineering, Kyung Hee University, Yongin 17104, Republic of Korea
| | - Hwanseok Jung
- Department of Electronics and Information Convergence Engineering, Kyung Hee University, Yongin 17104, Republic of Korea
| | - Jin-Hyuk Lee
- Department of Electronics and Information Convergence Engineering, Kyung Hee University, Yongin 17104, Republic of Korea
| | - Won Hee Lee
- Department of Software Convergence, Kyung Hee University, Yongin 17104, Republic of Korea
| | - Tae-Seong Kim
- Department of Electronics and Information Convergence Engineering, Kyung Hee University, Yongin 17104, Republic of Korea
| |
Collapse
|
12
|
Yuan L, He Z, Wang Q, Xu L, Ma X. Improving Small-Scale Human Action Recognition Performance Using a 3D Heatmap Volume. SENSORS (BASEL, SWITZERLAND) 2023; 23:6364. [PMID: 37514658 PMCID: PMC10383990 DOI: 10.3390/s23146364] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/11/2023] [Revised: 06/18/2023] [Accepted: 07/10/2023] [Indexed: 07/30/2023]
Abstract
In recent years, skeleton-based human action recognition has garnered significant research attention, with proposed recognition or segmentation methods typically validated on large-scale coarse-grained action datasets. However, there remains a lack of research on the recognition of small-scale fine-grained human actions using deep learning methods, which have greater practical significance. To address this gap, we propose a novel approach based on heatmap-based pseudo videos and a unified, general model applicable to all modality datasets. Leveraging anthropometric kinematics as prior information, we extract common human motion features among datasets through an ad hoc pre-trained model. To overcome joint mismatch issues, we partition the human skeleton into five parts, a simple yet effective technique for information sharing. Our approach is evaluated on two datasets, including the public Nursing Activities and our self-built Tai Chi Action dataset. Results from linear evaluation protocol and fine-tuned evaluation demonstrate that our pre-trained model effectively captures common motion features among human actions and achieves steady and precise accuracy across all training settings, while mitigating network overfitting. Notably, our model outperforms state-of-the-art models in recognition accuracy when fusing joint and limb modality features along the channel dimension.
Collapse
Affiliation(s)
- Lin Yuan
- Department of Control Science and Engineering, Harbin Institute of Technology, Harbin 150001, China
| | - Zhen He
- Department of Control Science and Engineering, Harbin Institute of Technology, Harbin 150001, China
| | - Qiang Wang
- Department of Control Science and Engineering, Harbin Institute of Technology, Harbin 150001, China
| | - Leiyang Xu
- Department of Control Science and Engineering, Harbin Institute of Technology, Harbin 150001, China
| | - Xiang Ma
- Department of Control Science and Engineering, Harbin Institute of Technology, Harbin 150001, China
| |
Collapse
|
13
|
Koo B, Nguyen NT, Kim J. Identification and Classification of Human Body Exercises on Smart Textile Bands by Combining Decision Tree and Convolutional Neural Networks. SENSORS (BASEL, SWITZERLAND) 2023; 23:6223. [PMID: 37448070 DOI: 10.3390/s23136223] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/30/2023] [Revised: 06/27/2023] [Accepted: 06/27/2023] [Indexed: 07/15/2023]
Abstract
In recent years, human activity recognition (HAR) has gained significant interest from researchers in the sports and fitness industries. In this study, the authors have proposed a cascaded method including two classifying stages to classify fitness exercises, utilizing a decision tree as the first stage and a one-dimension convolutional neural network as the second stage. The data acquisition was carried out by five participants performing exercises while wearing an inertial measurement unit sensor attached to a wristband on their wrists. However, only data acquired along the z-axis of the IMU accelerator was used as input to train and test the proposed model, to simplify the model and optimize the training time while still achieving good performance. To examine the efficiency of the proposed method, the authors compared the performance of the cascaded model and the conventional 1D-CNN model. The obtained results showed an overall improvement in the accuracy of exercise classification by the proposed model, which was approximately 92%, compared to 82.4% for the 1D-CNN model. In addition, the authors suggested and evaluated two methods to optimize the clustering outcome of the first stage in the cascaded model. This research demonstrates that the proposed model, with advantages in terms of training time and computational cost, is able to classify fitness workouts with high performance. Therefore, with further development, it can be applied in various real-time HAR applications.
Collapse
Affiliation(s)
- Bonhak Koo
- Department of Materials Science and Engineering, Soongsil University, Seoul 156-743, Republic of Korea
| | - Ngoc Tram Nguyen
- Department of Smart Wearable Engineering, Soongsil University, Seoul 156-743, Republic of Korea
| | - Jooyong Kim
- Department of Materials Science and Engineering, Soongsil University, Seoul 156-743, Republic of Korea
| |
Collapse
|
14
|
Agac S, Durmaz Incel O. On the Use of a Convolutional Block Attention Module in Deep Learning-Based Human Activity Recognition with Motion Sensors. Diagnostics (Basel) 2023; 13:diagnostics13111861. [PMID: 37296713 DOI: 10.3390/diagnostics13111861] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/05/2023] [Revised: 05/21/2023] [Accepted: 05/23/2023] [Indexed: 06/12/2023] Open
Abstract
Sensor-based human activity recognition with wearable devices has captured the attention of researchers in the last decade. The possibility of collecting large sets of data from various sensors in different body parts, automatic feature extraction, and aiming to recognize more complex activities have led to a rapid increase in the use of deep learning models in the field. More recently, using attention-based models for dynamically fine-tuning the model features and, in turn, improving the model performance has been investigated. However, the impact of using channel, spatial, or combined attention methods of the convolutional block attention module (CBAM) on the high-performing DeepConvLSTM model, a hybrid model proposed for sensor-based human activity recognition, has yet to be studied. Additionally, since wearables have limited resources, analysing the parameter requirements of attention modules can serve as an indicator for optimizing resource consumption. In this study, we explored the performance of CBAM on the DeepConvLSTM architecture both in terms of recognition performance and the number of additional parameters required by attention modules. In this direction, the effect of channel and spatial attention, individually and in combination, were examined. To evaluate the model performance, the Pamap2 dataset containing 12 daily activities and the Opportunity dataset with its 18 micro activities were utilized. The results showed that the performance for Opportunity increased from 0.74 to 0.77 in the macro f1-score owing to spatial attention, while for Pamap2, the performance increased from 0.95 to 0.96 owing to the channel attention applied to DeepConvLSTM with a negligible number of additional parameters. Moreover, when the activity-based results were analysed, it was observed that the attention mechanism increased the performance of the activities with the worst performance in the baseline model without attention. We present a comparison with related studies that use the same datasets and show that we could achieve higher scores on both datasets by combining CBAM and DeepConvLSTM.
Collapse
Affiliation(s)
- Sumeyye Agac
- Department of Computer Engineering, Bogazici University, Istanbul 34342, Turkey
| | - Ozlem Durmaz Incel
- Department of Computer Engineering, Bogazici University, Istanbul 34342, Turkey
| |
Collapse
|
15
|
Bhola G, Vishwakarma DK. A review of vision-based indoor HAR: state-of-the-art, challenges, and future prospects. MULTIMEDIA TOOLS AND APPLICATIONS 2023:1-41. [PMID: 37362688 PMCID: PMC10173923 DOI: 10.1007/s11042-023-15443-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 12/28/2021] [Revised: 02/10/2023] [Accepted: 04/18/2023] [Indexed: 06/28/2023]
Abstract
With the advent of technology, we are getting more comfortable with the use of gadgets, cameras, etc., and find Artificial Intelligence as an integral part of most of the tasks we perform throughout the day. In such a scenario, the use of cameras and vision-based sensors comes as an escape from many real-time problems and challenges. One major application of these vision-based systems is Indoor Human Activity Recognition (HAR) which serves in a variety of scenarios ranging from smart homes, elderly care, assisted living, and human behavior pattern analysis for identifying any abnormal behavior to abnormal activity recognition like falling, slipping, domestic violence, etc. The effect of HAR in real time has made the area of indoor activity recognition a more explored zone by the industrial segment to attract users with their products in multiple domains. Hence, considering these aspects of HAR, this work proposes a detailed survey on indoor HAR. Through this work, we have highlighted the recent methodologies and their performance in the field of indoor activity recognition. We have also discussed- the challenges, detailed study of approaches with real-world applications of indoor-HAR, datasets available for indoor activity, and their technical details in this work. We have proposed a taxonomy for indoor HAR and highlighted the state-of-the-art and future prospects by mentioning the research gaps and the shortcomings of recent surveys with respect to our work.
Collapse
Affiliation(s)
- Geetanjali Bhola
- Biometric Research Laboratory, Department of Information Technology, Delhi Technological University, Bawana Road, Delhi, 11042 India
| | - Dinesh Kumar Vishwakarma
- Biometric Research Laboratory, Department of Information Technology, Delhi Technological University, Bawana Road, Delhi, 11042 India
| |
Collapse
|
16
|
Su H, Qi W, Chen J, Yang C, Sandoval J, Laribi MA. Recent advancements in multimodal human-robot interaction. Front Neurorobot 2023; 17:1084000. [PMID: 37250671 PMCID: PMC10210148 DOI: 10.3389/fnbot.2023.1084000] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/29/2022] [Accepted: 04/20/2023] [Indexed: 05/31/2023] Open
Abstract
Robotics have advanced significantly over the years, and human-robot interaction (HRI) is now playing an important role in delivering the best user experience, cutting down on laborious tasks, and raising public acceptance of robots. New HRI approaches are necessary to promote the evolution of robots, with a more natural and flexible interaction manner clearly the most crucial. As a newly emerging approach to HRI, multimodal HRI is a method for individuals to communicate with a robot using various modalities, including voice, image, text, eye movement, and touch, as well as bio-signals like EEG and ECG. It is a broad field closely related to cognitive science, ergonomics, multimedia technology, and virtual reality, with numerous applications springing up each year. However, little research has been done to summarize the current development and future trend of HRI. To this end, this paper systematically reviews the state of the art of multimodal HRI on its applications by summing up the latest research articles relevant to this field. Moreover, the research development in terms of the input signal and the output signal is also covered in this manuscript.
Collapse
Affiliation(s)
- Hang Su
- Department of Electronics, Information and Bioengineering, Politecnico di Milano, Milan, Italy
| | - Wen Qi
- School of Future Technology, South China University of Technology, Guangzhou, China
| | - Jiahao Chen
- State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China
| | - Chenguang Yang
- Bristol Robotics Laboratory, University of the West of England, Bristol, United Kingdom
| | - Juan Sandoval
- Department of GMSC, Pprime Institute, CNRS, ENSMA, University of Poitiers, Poitiers, France
| | - Med Amine Laribi
- Department of GMSC, Pprime Institute, CNRS, ENSMA, University of Poitiers, Poitiers, France
| |
Collapse
|
17
|
Sardari S, Sharifzadeh S, Daneshkhah A, Nakisa B, Loke SW, Palade V, Duncan MJ. Artificial Intelligence for skeleton-based physical rehabilitation action evaluation: A systematic review. Comput Biol Med 2023; 158:106835. [PMID: 37019012 DOI: 10.1016/j.compbiomed.2023.106835] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/02/2022] [Revised: 03/09/2023] [Accepted: 03/26/2023] [Indexed: 04/03/2023]
Abstract
Performing prescribed physical exercises during home-based rehabilitation programs plays an important role in regaining muscle strength and improving balance for people with different physical disabilities. However, patients attending these programs are not able to assess their action performance in the absence of a medical expert. Recently, vision-based sensors have been deployed in the activity monitoring domain. They are capable of capturing accurate skeleton data. Furthermore, there have been significant advancements in Computer Vision (CV) and Deep Learning (DL) methodologies. These factors have promoted the solutions for designing automatic patient's activity monitoring models. Then, improving such systems' performance to assist patients and physiotherapists has attracted wide interest of the research community. This paper provides a comprehensive and up-to-date literature review on different stages of skeleton data acquisition processes for the aim of physio exercise monitoring. Then, the previously reported Artificial Intelligence (AI) - based methodologies for skeleton data analysis will be reviewed. In particular, feature learning from skeleton data, evaluation, and feedback generation for the purpose of rehabilitation monitoring will be studied. Furthermore, the associated challenges to these processes will be reviewed. Finally, the paper puts forward several suggestions for future research directions in this area.
Collapse
|
18
|
Mo L, Zhu Y, Zeng L. A Multi-Label Based Physical Activity Recognition via Cascade Classifier. SENSORS (BASEL, SWITZERLAND) 2023; 23:s23052593. [PMID: 36904797 PMCID: PMC10006903 DOI: 10.3390/s23052593] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/09/2023] [Revised: 02/21/2023] [Accepted: 02/24/2023] [Indexed: 05/14/2023]
Abstract
Physical activity recognition is a field that infers human activities used in machine learning techniques through wearable devices and embedded inertial sensors of smartphones. It has gained much research significance and promising prospects in the fields of medical rehabilitation and fitness management. Generally, datasets with different wearable sensors and activity labels are used to train machine learning models, and most research has achieved satisfactory performance for these datasets. However, most of the methods are incapable of recognizing the complex physical activity of free living. To address the issue, we propose a cascade classifier structure for sensor-based physical activity recognition from a multi-dimensional perspective, with two types of labels that work together to represent an exact type of activity. This approach employed the cascade classifier structure based on a multi-label system (Cascade Classifier on Multi-label, CCM). The labels reflecting the activity intensity would be classified first. Then, the data flow is divided into the corresponding activity type classifier according to the output of the pre-layer prediction. The dataset of 110 participants has been collected for the experiment on PA recognition. Compared with the typical machine learning algorithms of Random Forest (RF), Sequential Minimal Optimization (SMO) and K Nearest Neighbors (KNN), the proposed method greatly improves the overall recognition accuracy of ten physical activities. The results show that the RF-CCM classifier has achieved 93.94% higher accuracy than the 87.93% obtained from the non-CCM system, which could obtain better generalization performance. The comparison results reveal that the novel CCM system proposed is more effective and stable in physical activity recognition than the conventional classification methods.
Collapse
|
19
|
Manouchehri N, Bouguila N. Human Activity Recognition with an HMM-Based Generative Model. SENSORS (BASEL, SWITZERLAND) 2023; 23:1390. [PMID: 36772428 PMCID: PMC9920173 DOI: 10.3390/s23031390] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/24/2022] [Revised: 01/11/2023] [Accepted: 01/20/2023] [Indexed: 06/18/2023]
Abstract
Human activity recognition (HAR) has become an interesting topic in healthcare. This application is important in various domains, such as health monitoring, supporting elders, and disease diagnosis. Considering the increasing improvements in smart devices, large amounts of data are generated in our daily lives. In this work, we propose unsupervised, scaled, Dirichlet-based hidden Markov models to analyze human activities. Our motivation is that human activities have sequential patterns and hidden Markov models (HMMs) are some of the strongest statistical models used for modeling data with continuous flow. In this paper, we assume that emission probabilities in HMM follow a bounded-scaled Dirichlet distribution, which is a proper choice in modeling proportional data. To learn our model, we applied the variational inference approach. We used a publicly available dataset to evaluate the performance of our proposed model.
Collapse
Affiliation(s)
- Narges Manouchehri
- Algorithmic Dynamics Lab, Unit of Computational Medicine, Karolinska Institute, 171 77 Stockholm, Sweden
- Concordia Institute for Information Systems Engineering, Concordia University, Montreal, QC H3G1T7, Canada
| | - Nizar Bouguila
- Concordia Institute for Information Systems Engineering, Concordia University, Montreal, QC H3G1T7, Canada
| |
Collapse
|
20
|
Kiani S, Rezaei I, Abasi S, Zakerabasali S, Yazdani A. Technical aspects of virtual augmented reality-based rehabilitation systems for musculoskeletal disorders of the lower limbs: a systematic review. BMC Musculoskelet Disord 2023; 24:4. [PMID: 36597077 PMCID: PMC9808732 DOI: 10.1186/s12891-022-06062-6] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/29/2021] [Accepted: 12/06/2022] [Indexed: 01/04/2023] Open
Abstract
INTRODUCTION Musculoskeletal disorders are one of the most common causes of physical disability. The rehabilitation process after musculoskeletal disorders is long and tedious, and patients are not motivated to follow rehabilitation protocols. Therefore, new systems must be used to increase patient motivation. Virtual reality (VR) and augmented reality (AR) technologies can be used in this regard. In developing such systems, various technologies and methods of movement recognition are used; therefore, this study aims to summarize the technical aspects of using VR/AR in rehabilitation and evaluate and discuss efficient methods of investigating studies using the Statement of Standards for Reporting Implementation Studies (StaRI). METHODS Search in four scientific databases was done systematically based on PRISMA through online search engines from inception to June 2021. These databases include Medline (PubMed), Scopus, IEEE, and Web of Science. An updated search was also conducted on 17 December 2021. The research used keywords and MeSH terms associated with VR/AR, musculoskeletal disorder, and rehabilitation. Selected articles were evaluated qualitatively using the Standards for Reporting Implementation Studies (StaRI) statement. RESULTS A total of 2343 articles were found, and 20 studies were included. We found that 11 (55%) studies used Kinect technology as input tools, and 15 (75%) studies have described the techniques used to analyze human movements, such as dynamic time warping (DTW) and support vector machines (SVM). In 10 (50%) studies, the Unity game engine was used for visualization. In 8 studies (40%), usability was assessed, and high usability was reported. Similarly, the results of the review of studies according to the StaRI checklist showed poor reporting in the title and discussion of the studies. CONCLUSIONS We found that academic studies did not describe the technical aspects of rehabilitation systems. Therefore, a good description of the technical aspects of the system in such studies should be considered to provide repeatability and generalizability of these systems for investigations by other researchers.
Collapse
Affiliation(s)
- Shamim Kiani
- grid.412571.40000 0000 8819 4698Student Research Committee, Shiraz University of Medical Sciences, Shiraz, Iran
| | - Iman Rezaei
- grid.412571.40000 0000 8819 4698Physical Therapy Department, School of Rehabilitation Sciences, Shiraz University of Medical Sciences, Shiraz, Iran ,grid.412571.40000 0000 8819 4698Rehabilitation Sciences Research Center, Shiraz University of Medical Sciences, Shiraz, Iran
| | - Sanaz Abasi
- grid.412571.40000 0000 8819 4698Student Research Committee, Shiraz University of Medical Sciences, Shiraz, Iran
| | - Somayyeh Zakerabasali
- grid.412571.40000 0000 8819 4698Health Human Resources Research Center, Shiraz University of Medical Sciences, Shiraz, Iran ,grid.412571.40000 0000 8819 4698Health Information Management Department, Shiraz University of Medical Sciences, Shiraz, Iran ,grid.412571.40000 0000 8819 4698 Clinical Education Research Center, Shiraz University of Medical Sciences, Shiraz, Iran
| | - Azita Yazdani
- grid.412571.40000 0000 8819 4698Health Human Resources Research Center, Shiraz University of Medical Sciences, Shiraz, Iran ,grid.412571.40000 0000 8819 4698Health Information Management Department, Shiraz University of Medical Sciences, Shiraz, Iran ,grid.412571.40000 0000 8819 4698 Clinical Education Research Center, Shiraz University of Medical Sciences, Shiraz, Iran
| |
Collapse
|
21
|
Bernaś M, Płaczek B, Lewandowski M. Ensemble of RNN Classifiers for Activity Detection Using a Smartphone and Supporting Nodes. SENSORS (BASEL, SWITZERLAND) 2022; 22:9451. [PMID: 36502154 PMCID: PMC9739648 DOI: 10.3390/s22239451] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 11/01/2022] [Revised: 11/27/2022] [Accepted: 12/01/2022] [Indexed: 06/17/2023]
Abstract
Nowadays, sensor-equipped mobile devices allow us to detect basic daily activities accurately. However, the accuracy of the existing activity recognition methods decreases rapidly if the set of activities is extended and includes training routines, such as squats, jumps, or arm swings. Thus, this paper proposes a model of a personal area network with a smartphone (as a main node) and supporting sensor nodes that deliver additional data to increase activity-recognition accuracy. The introduced personal area sensor network takes advantage of the information from multiple sensor nodes attached to different parts of the human body. In this scheme, nodes process their sensor readings locally with the use of recurrent neural networks (RNNs) to categorize the activities. Then, the main node collects results from supporting sensor nodes and performs a final activity recognition run based on a weighted voting procedure. In order to save energy and extend the network's lifetime, sensor nodes report their local results only for specific types of recognized activity. The presented method was evaluated during experiments with sensor nodes attached to the waist, chest, leg, and arm. The results obtained for a set of eight activities show that the proposed approach achieves higher recognition accuracy when compared with the existing methods. Based on the experimental results, the optimal configuration of the sensor nodes was determined to maximize the activity-recognition accuracy and reduce the number of transmissions from supporting sensor nodes.
Collapse
Affiliation(s)
- Marcin Bernaś
- Department of Computer Science and Automatics, University of Bielsko-Biała, Willowa 2, 43-309 Bielsko-Biała, Poland
| | - Bartłomiej Płaczek
- Institute of Computer Science, University of Silesia, Będzińska 39, 41-200 Sosnowiec, Poland
| | - Marcin Lewandowski
- Institute of Computer Science, University of Silesia, Będzińska 39, 41-200 Sosnowiec, Poland
| |
Collapse
|
22
|
The HA4M dataset: Multi-Modal Monitoring of an assembly task for Human Action recognition in Manufacturing. Sci Data 2022; 9:745. [PMID: 36460662 PMCID: PMC9718853 DOI: 10.1038/s41597-022-01843-z] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/18/2022] [Accepted: 11/11/2022] [Indexed: 12/03/2022] Open
Abstract
This paper introduces the Human Action Multi-Modal Monitoring in Manufacturing (HA4M) dataset, a collection of multi-modal data relative to actions performed by different subjects building an Epicyclic Gear Train (EGT). In particular, 41 subjects executed several trials of the assembly task, which consists of 12 actions. Data were collected in a laboratory scenario using a Microsoft® Azure Kinect which integrates a depth camera, an RGB camera, and InfraRed (IR) emitters. To the best of authors' knowledge, the HA4M dataset is the first multi-modal dataset about an assembly task containing six types of data: RGB images, Depth maps, IR images, RGB-to-Depth-Aligned images, Point Clouds and Skeleton data. These data represent a good foundation to develop and test advanced action recognition systems in several fields, including Computer Vision and Machine Learning, and application domains such as smart manufacturing and human-robot collaboration.
Collapse
|
23
|
Clark BK, Brakenridge CL, Healy GN. The Importance of Research on Occupational Sedentary Behaviour and Activity Right Now. INTERNATIONAL JOURNAL OF ENVIRONMENTAL RESEARCH AND PUBLIC HEALTH 2022; 19:15816. [PMID: 36497891 PMCID: PMC9738861 DOI: 10.3390/ijerph192315816] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/22/2022] [Accepted: 11/24/2022] [Indexed: 06/17/2023]
Abstract
The workplace has been identified as a key setting for public health interventions to 'promote and maintain the highest degree of physical, mental and social well-being of workers in all occupations' by the World Health Organisation [...].
Collapse
Affiliation(s)
- Bronwyn K. Clark
- School of Human Movement and Nutrition Sciences, The University of Queensland, St Lucia 4072, Australia
| | - Charlotte L. Brakenridge
- School of Human Movement and Nutrition Sciences, The University of Queensland, St Lucia 4072, Australia
| | - Genevieve N. Healy
- School of Human Movement and Nutrition Sciences, The University of Queensland, St Lucia 4072, Australia
- Baker Heart and Diabetes Institute, Melbourne 3004, Australia
- School of Physiotherapy and Exercise Science, Curtin University, Perth 6845, Australia
| |
Collapse
|
24
|
Shafiqul IM, Jannat MKA, Kim JW, Lee SW, Yang SH. HHI-AttentionNet: An Enhanced Human-Human Interaction Recognition Method Based on a Lightweight Deep Learning Model with Attention Network from CSI. SENSORS (BASEL, SWITZERLAND) 2022; 22:6018. [PMID: 36015776 PMCID: PMC9414797 DOI: 10.3390/s22166018] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 07/15/2022] [Revised: 08/05/2022] [Accepted: 08/10/2022] [Indexed: 06/15/2023]
Abstract
Nowadays WiFi based human activity recognition (WiFi-HAR) has gained much attraction in an indoor environment due to its various benefits, including privacy and security, device free sensing, and cost-effectiveness. Recognition of human-human interactions (HHIs) using channel state information (CSI) signals is still challenging. Although some deep learning (DL) based architectures have been proposed in this regard, most of them suffer from limited recognition accuracy and are unable to support low computation resource devices due to having a large number of model parameters. To address these issues, we propose a dynamic method using a lightweight DL model (HHI-AttentionNet) to automatically recognize HHIs, which significantly reduces the parameters with increased recognition accuracy. In addition, we present an Antenna-Frame-Subcarrier Attention Mechanism (AFSAM) in our model that enhances the representational capability to recognize HHIs correctly. As a result, the HHI-AttentionNet model focuses on the most significant features, ignoring the irrelevant features, and reduces the impact of the complexity on the CSI signal. We evaluated the performance of the proposed HHI-AttentionNet model on a publicly available CSI-based HHI dataset collected from 40 individual pairs of subjects who performed 13 different HHIs. Its performance is also compared with other existing methods. These proved that the HHI-AttentionNet is the best model providing an average accuracy, F1 score, Cohen's Kappa, and Matthews correlation coefficient of 95.47%, 95.45%, 0.951%, and 0.950%, respectively, for recognition of 13 HHIs. It outperforms the best existing model's accuracy by more than 4%.
Collapse
Affiliation(s)
- Islam Md Shafiqul
- Department of Electronic Engineering, Kwangwoon University, Seoul 01897, Korea
| | | | - Jin-Woo Kim
- Department of Electronic Engineering, Kwangwoon University, Seoul 01897, Korea
| | - Soo-Wook Lee
- Kwangwoon Academy, Kwangwoon University, Seoul 01897, Korea
| | - Sung-Hyun Yang
- Department of Electronic Engineering, Kwangwoon University, Seoul 01897, Korea
| |
Collapse
|
25
|
Yuan L, Andrews J, Mu H, Vakil A, Ewing R, Blasch E, Li J. Interpretable Passive Multi-Modal Sensor Fusion for Human Identification and Activity Recognition. SENSORS (BASEL, SWITZERLAND) 2022; 22:5787. [PMID: 35957343 PMCID: PMC9371208 DOI: 10.3390/s22155787] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 06/17/2022] [Revised: 07/05/2022] [Accepted: 07/08/2022] [Indexed: 06/15/2023]
Abstract
Human monitoring applications in indoor environments depend on accurate human identification and activity recognition (HIAR). Single modality sensor systems have shown to be accurate for HIAR, but there are some shortcomings to these systems, such as privacy, intrusion, and costs. To combat these shortcomings for a long-term monitoring solution, an interpretable, passive, multi-modal, sensor fusion system PRF-PIR is proposed in this work. PRF-PIR is composed of one software-defined radio (SDR) device and one novel passive infrared (PIR) sensor system. A recurrent neural network (RNN) is built as the HIAR model for this proposed solution to handle the temporal dependence of passive information captured by both modalities. We validate our proposed PRF-PIR system for a potential human monitoring system through the data collection of eleven activities from twelve human subjects in an academic office environment. From our data collection, the efficacy of the sensor fusion system is proven via an accuracy of 0.9866 for human identification and an accuracy of 0.9623 for activity recognition. The results of the system are supported with explainable artificial intelligence (XAI) methodologies to serve as a validation for sensor fusion over the deployment of single sensor solutions. PRF-PIR provides a passive, non-intrusive, and highly accurate system that allows for robustness in uncertain, highly similar, and complex at-home activities performed by a variety of human subjects.
Collapse
Affiliation(s)
- Liangqi Yuan
- Department of Electrical and Computer Engineering, Oakland University, Rochester, MI 48309, USA; (L.Y.); (J.A.); (H.M.); (A.V.)
| | - Jack Andrews
- Department of Electrical and Computer Engineering, Oakland University, Rochester, MI 48309, USA; (L.Y.); (J.A.); (H.M.); (A.V.)
| | - Huaizheng Mu
- Department of Electrical and Computer Engineering, Oakland University, Rochester, MI 48309, USA; (L.Y.); (J.A.); (H.M.); (A.V.)
| | - Asad Vakil
- Department of Electrical and Computer Engineering, Oakland University, Rochester, MI 48309, USA; (L.Y.); (J.A.); (H.M.); (A.V.)
| | - Robert Ewing
- Sensors Directorate, Air Force Research Laboratory, WPAFB, Dayton, OH 45433, USA;
| | - Erik Blasch
- Information Directorate, Air Force Research Laboratory, Rome, NY 13441, USA;
| | - Jia Li
- Department of Electrical and Computer Engineering, Oakland University, Rochester, MI 48309, USA; (L.Y.); (J.A.); (H.M.); (A.V.)
| |
Collapse
|
26
|
Yadav SK, Agarwal A, Kumar A, Tiwari K, Pandey HM, Akbar SA. YogNet: A two-stream network for realtime multiperson yoga action recognition and posture correction. Knowl Based Syst 2022. [DOI: 10.1016/j.knosys.2022.109097] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/29/2022]
|
27
|
Dehkordi HA, Nezhad AS, Kashiani H, Shokouhi SB, Ayatollahi A. Multi-expert human action recognition with hierarchical super-class learning. Knowl Based Syst 2022. [DOI: 10.1016/j.knosys.2022.109091] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
|
28
|
Wang Y, Song Q, Ma T, Chen Y, Li H, Liu R. Transformation classification of human squat/sit-to-stand based on multichannel information fusion. INT J ADV ROBOT SYST 2022. [DOI: 10.1177/17298806221103708] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022] Open
Abstract
In existing rehabilitation training, research on the accuracy of recognizing completed actions has achieved good results; however, the reduction in the misjudgment rate in the action conversion process needs further research. This article proposes a multichannel information fusion method for the movement conversion process of squat/sit-to-stand, which can help online movement conversion classification during rehabilitation training. We collected a training dataset from a total of eight subjects performing three different motions, including half squat, full squat, and sitting, equipped with plantar pressure sensors, RGB cameras, and five inertial measurement units. Our evaluation includes the misjudgment rate for each action and the time needed for classification. The experimental results show that, compared with the recognition of a single sensor, the accuracy after fusion can reach 96.6% in the case of no occlusion and 86.7% in the case of occlusion. Compared with the complete time window, the classification time window is shortened by approximately 25%.
Collapse
Affiliation(s)
- Yu Wang
- Institute of Intelligent Machines, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei, Anhui, China
- Department of Science Island, University of Science and Technology of China, Hefei, Anhui, China
| | - Quanjun Song
- Institute of Intelligent Machines, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei, Anhui, China
| | - Tingting Ma
- Institute of Intelligent Machines, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei, Anhui, China
| | - Yong Chen
- Institute of Intelligent Machines, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei, Anhui, China
| | - Hao Li
- Institute of Intelligent Machines, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei, Anhui, China
| | - Rongkai Liu
- Institute of Intelligent Machines, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei, Anhui, China
- Department of Science Island, University of Science and Technology of China, Hefei, Anhui, China
| |
Collapse
|
29
|
Construction of Knowledge Graph of 3D Clothing Design Resources Based on Multimodal Clustering Network. COMPUTATIONAL INTELLIGENCE AND NEUROSCIENCE 2022; 2022:1168012. [PMID: 35694580 PMCID: PMC9184191 DOI: 10.1155/2022/1168012] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 03/31/2022] [Revised: 04/24/2022] [Accepted: 05/17/2022] [Indexed: 11/18/2022]
Abstract
The construction of 3D design model is a hotspot of applied research in the fields of clothing functional design system teaching and display. The simple 3D clothing visualization postprocessing lacks interactive functions, which is a hot issue that needs to be solved urgently at present. Based on analyzing the existing clothing modeling technology, template technology, and fusion technology, and based on the multimodal clustering network theory, this paper proposes a 3D clothing design resource knowledge graph modeling method with multiple fusion of features and templates. The position of each joint point is converted into the coordinate system centered on the torso point in advance and normalized to avoid the problem that the relative position of the camera and the collector cannot be determined, and the shape of different collectors is different. The paper provides a multimodal clustering network intelligence method, illustrates the interoperability of users switching between different design networks in the seamless connection movement, and combines the hybrid intelligence algorithm with the fuzzy logic interpretation algorithm to solve the problems in the field of 3D clothing design service quality. During the simulation process, the research scheme builds a logical multimodal clustering network framework, which integrates compatibility access and global access partition fusion of style templates to achieve information extraction of clothing parts. The experimental results show that the realistic 3D clothing modeling can be achieved by layering the 3D clothing map, contour features, clothing size features, and color texture features with the modeling template. The developed ActiveX control is mounted on MSN, and the system is compatible. The performance and integration rate reached 77.1% and 89.7%, respectively, which effectively strengthened the practical role of the 3D clothing design system.
Collapse
|
30
|
Jin Z, Li Z, Gan T, Fu Z, Zhang C, He Z, Zhang H, Wang P, Liu J, Ye X. A Novel Central Camera Calibration Method Recording Point-to-Point Distortion for Vision-Based Human Activity Recognition. SENSORS (BASEL, SWITZERLAND) 2022; 22:s22093524. [PMID: 35591215 PMCID: PMC9105339 DOI: 10.3390/s22093524] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/23/2022] [Revised: 04/28/2022] [Accepted: 04/29/2022] [Indexed: 06/02/2023]
Abstract
The camera is the main sensor of vison-based human activity recognition, and its high-precision calibration of distortion is an important prerequisite of the task. Current studies have shown that multi-parameter model methods achieve higher accuracy than traditional methods in the process of camera calibration. However, these methods need hundreds or even thousands of images to optimize the camera model, which limits their practical use. Here, we propose a novel point-to-point camera distortion calibration method that requires only dozens of images to get a dense distortion rectification map. We have designed an objective function based on deformation between the original images and the projection of reference images, which can eliminate the effect of distortion when optimizing camera parameters. Dense features between the original images and the projection of the reference images are calculated by digital image correlation (DIC). Experiments indicate that our method obtains a comparable result with the multi-parameter model method using a large number of pictures, and contributes a 28.5% improvement to the reprojection error over the polynomial distortion model.
Collapse
Affiliation(s)
- Ziyi Jin
- Biosensor National Special Laboratory, Key Laboratory of Biomedical Engineering of Ministry of Education, Zhejiang University, Hangzhou 310027, China; (Z.J.); (T.G.); (Z.F.); (C.Z.); (Z.H.); (H.Z.); (P.W.)
- College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou 310027, China;
| | - Zhixue Li
- Independent Researcher, 181 Gaojiao Road, Yuhang District, Hangzhou 311122, China;
| | - Tianyuan Gan
- Biosensor National Special Laboratory, Key Laboratory of Biomedical Engineering of Ministry of Education, Zhejiang University, Hangzhou 310027, China; (Z.J.); (T.G.); (Z.F.); (C.Z.); (Z.H.); (H.Z.); (P.W.)
- College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou 310027, China;
| | - Zuoming Fu
- Biosensor National Special Laboratory, Key Laboratory of Biomedical Engineering of Ministry of Education, Zhejiang University, Hangzhou 310027, China; (Z.J.); (T.G.); (Z.F.); (C.Z.); (Z.H.); (H.Z.); (P.W.)
- College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou 310027, China;
| | - Chongan Zhang
- Biosensor National Special Laboratory, Key Laboratory of Biomedical Engineering of Ministry of Education, Zhejiang University, Hangzhou 310027, China; (Z.J.); (T.G.); (Z.F.); (C.Z.); (Z.H.); (H.Z.); (P.W.)
- College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou 310027, China;
| | - Zhongyu He
- Biosensor National Special Laboratory, Key Laboratory of Biomedical Engineering of Ministry of Education, Zhejiang University, Hangzhou 310027, China; (Z.J.); (T.G.); (Z.F.); (C.Z.); (Z.H.); (H.Z.); (P.W.)
- College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou 310027, China;
| | - Hong Zhang
- Biosensor National Special Laboratory, Key Laboratory of Biomedical Engineering of Ministry of Education, Zhejiang University, Hangzhou 310027, China; (Z.J.); (T.G.); (Z.F.); (C.Z.); (Z.H.); (H.Z.); (P.W.)
- College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou 310027, China;
| | - Peng Wang
- Biosensor National Special Laboratory, Key Laboratory of Biomedical Engineering of Ministry of Education, Zhejiang University, Hangzhou 310027, China; (Z.J.); (T.G.); (Z.F.); (C.Z.); (Z.H.); (H.Z.); (P.W.)
- College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou 310027, China;
| | - Jiquan Liu
- College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou 310027, China;
| | - Xuesong Ye
- Biosensor National Special Laboratory, Key Laboratory of Biomedical Engineering of Ministry of Education, Zhejiang University, Hangzhou 310027, China; (Z.J.); (T.G.); (Z.F.); (C.Z.); (Z.H.); (H.Z.); (P.W.)
- College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou 310027, China;
| |
Collapse
|
31
|
STA-TSN: Spatial-Temporal Attention Temporal Segment Network for action recognition in video. PLoS One 2022; 17:e0265115. [PMID: 35298497 PMCID: PMC8929560 DOI: 10.1371/journal.pone.0265115] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/06/2021] [Accepted: 02/24/2022] [Indexed: 11/19/2022] Open
Abstract
Most deep learning-based action recognition models focus only on short-term motions, so the model often causes misjudgments of actions that are combined by multiple processes, such as long jump, high jump, etc. The proposal of Temporal Segment Networks (TSN) enables the network to capture long-term information in the video, but ignores that some unrelated frames or areas in the video can also cause great interference to action recognition. To solve this problem, a soft attention mechanism is introduced in TSN and a Spatial-Temporal Attention Temporal Segment Networks (STA-TSN), which retains the ability to capture long-term information and enables the network to adaptively focus on key features in space and time, is proposed. First, a multi-scale spatial focus feature enhancement strategy is proposed to fuse original convolution features with multi-scale spatial focus features obtained through a soft attention mechanism with spatial pyramid pooling. Second, a deep learning-based key frames exploration module, which utilizes a soft attention mechanism based on Long-Short Term Memory (LSTM) to adaptively learn temporal attention weights, is designed. Third, a temporal-attention regularization is developed to guide our STA-TSN to better realize the exploration of key frames. Finally, the experimental results show that our proposed STA-TSN outperforms TSN in the four public datasets UCF101, HMDB51, JHMDB and THUMOS14, as well as achieves state-of-the-art results.
Collapse
|
32
|
Dirgová Luptáková I, Kubovčík M, Pospíchal J. Wearable Sensor-Based Human Activity Recognition with Transformer Model. SENSORS (BASEL, SWITZERLAND) 2022; 22:1911. [PMID: 35271058 PMCID: PMC8914677 DOI: 10.3390/s22051911] [Citation(s) in RCA: 16] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 02/04/2022] [Revised: 02/22/2022] [Accepted: 02/25/2022] [Indexed: 06/14/2023]
Abstract
Computing devices that can recognize various human activities or movements can be used to assist people in healthcare, sports, or human-robot interaction. Readily available data for this purpose can be obtained from the accelerometer and the gyroscope built into everyday smartphones. Effective classification of real-time activity data is, therefore, actively pursued using various machine learning methods. In this study, the transformer model, a deep learning neural network model developed primarily for the natural language processing and vision tasks, was adapted for a time-series analysis of motion signals. The self-attention mechanism inherent in the transformer, which expresses individual dependencies between signal values within a time series, can match the performance of state-of-the-art convolutional neural networks with long short-term memory. The performance of the proposed adapted transformer method was tested on the largest available public dataset of smartphone motion sensor data covering a wide range of activities, and obtained an average identification accuracy of 99.2% as compared with 89.67% achieved on the same data by a conventional machine learning method. The results suggest the expected future relevance of the transformer model for human activity recognition.
Collapse
|
33
|
Li F, Chen CH, Lee CH, Feng S. Artificial intelligence-enabled non-intrusive vigilance assessment approach to reducing traffic controller’s human errors. Knowl Based Syst 2022. [DOI: 10.1016/j.knosys.2021.108047] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/25/2022]
|
34
|
Hu C, Chen Y, Hu L, Yu H, Lu D. Disagreement-based class incremental random forest for sensor-based activity recognition. Knowl Based Syst 2022. [DOI: 10.1016/j.knosys.2021.108044] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/22/2022]
|
35
|
Chan VCH, Ross GB, Clouthier AL, Fischer SL, Graham RB. The role of machine learning in the primary prevention of work-related musculoskeletal disorders: A scoping review. APPLIED ERGONOMICS 2022; 98:103574. [PMID: 34547578 DOI: 10.1016/j.apergo.2021.103574] [Citation(s) in RCA: 8] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/20/2021] [Revised: 08/22/2021] [Accepted: 08/24/2021] [Indexed: 06/13/2023]
Abstract
To determine the applications of machine learning (ML) techniques used for the primary prevention of work-related musculoskeletal disorders (WMSDs), a scoping review was conducted using seven literature databases. Of the 4,639 initial results, 130 primary research studies were deemed relevant for inclusion. Studies were reviewed and classified as a contribution to one of six steps within the primary WMSD prevention research framework by van der Beek et al. (2017). ML techniques provided the greatest contributions to the development of interventions (48 studies), followed by risk factor identification (33 studies), underlying mechanisms (29 studies), incidence of WMSDs (14 studies), evaluation of interventions (6 studies), and implementation of effective interventions (0 studies). Nearly a quarter (23.8%) of all included studies were published in 2020. These findings provide insight into the breadth of ML techniques used for primary WMSD prevention and can help identify areas for future research and development.
Collapse
Affiliation(s)
- Victor C H Chan
- School of Human Kinetics, Faculty of Health Sciences, University of Ottawa, 200 Lees Avenue, Ottawa, Ontario, K1N 6N5, Canada
| | - Gwyneth B Ross
- School of Human Kinetics, Faculty of Health Sciences, University of Ottawa, 200 Lees Avenue, Ottawa, Ontario, K1N 6N5, Canada
| | - Allison L Clouthier
- School of Human Kinetics, Faculty of Health Sciences, University of Ottawa, 200 Lees Avenue, Ottawa, Ontario, K1N 6N5, Canada
| | - Steven L Fischer
- Department of Kinesiology, University of Waterloo, Waterloo, ON, Canada
| | - Ryan B Graham
- School of Human Kinetics, Faculty of Health Sciences, University of Ottawa, 200 Lees Avenue, Ottawa, Ontario, K1N 6N5, Canada; Department of Kinesiology, University of Waterloo, Waterloo, ON, Canada.
| |
Collapse
|
36
|
Yadav SK, Tiwari K, Pandey HM, Akbar SA. Skeleton-based human activity recognition using ConvLSTM and guided feature learning. Soft comput 2022. [DOI: 10.1007/s00500-021-06238-7] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
Abstract
AbstractHuman activity recognition aims to determine actions performed by a human in an image or video. Examples of human activity include standing, running, sitting, sleeping, etc. These activities may involve intricate motion patterns and undesired events such as falling. This paper proposes a novel deep convolutional long short-term memory (ConvLSTM) network for skeletal-based activity recognition and fall detection. The proposed ConvLSTM network is a sequential fusion of convolutional neural networks (CNNs), long short-term memory (LSTM) networks, and fully connected layers. The acquisition system applies human detection and pose estimation to pre-calculate skeleton coordinates from the image/video sequence. The ConvLSTM model uses the raw skeleton coordinates along with their characteristic geometrical and kinematic features to construct the novel guided features. The geometrical and kinematic features are built upon raw skeleton coordinates using relative joint position values, differences between joints, spherical joint angles between selected joints, and their angular velocities. The novel spatiotemporal-guided features are obtained using a trained multi-player CNN-LSTM combination. Classification head including fully connected layers is subsequently applied. The proposed model has been evaluated on the KinectHAR dataset having 130,000 samples with 81 attribute values, collected with the help of a Kinect (v2) sensor. Experimental results are compared against the performance of isolated CNNs and LSTM networks. Proposed ConvLSTM have achieved an accuracy of 98.89% that is better than CNNs and LSTMs having an accuracy of 93.89 and 92.75%, respectively. The proposed system has been tested in realtime and is found to be independent of the pose, facing of the camera, individuals, clothing, etc. The code and dataset will be made publicly available.
Collapse
|
37
|
CSITime: Privacy-preserving human activity recognition using WiFi channel state information. Neural Netw 2021; 146:11-21. [PMID: 34839089 DOI: 10.1016/j.neunet.2021.11.011] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/25/2021] [Revised: 11/07/2021] [Accepted: 11/09/2021] [Indexed: 11/21/2022]
Abstract
Human activity recognition (HAR) is an important task in many applications such as smart homes, sports analysis, healthcare services, etc. Popular modalities for human activity recognition involving computer vision and inertial sensors are in the literature for solving HAR, however, they face serious limitations with respect to different illumination, background, clutter, obtrusiveness, and other factors. In recent years, WiFi channel state information (CSI) based activity recognition is gaining momentum due to its many advantages including easy deployability, and cost-effectiveness. This work proposes CSITime, a modified InceptionTime network architecture, a generic architecture for CSI-based human activity recognition. We perceive CSI activity recognition as a multi-variate time series problem. The methodology of CSITime is threefold. First, we pre-process CSI signals followed by data augmentation using two label-mixing strategies - mixup and cutmix to enhance the neural network's learning. Second, in the basic block of CSITime, features from multiple convolutional kernels are concatenated and passed through a self-attention layer followed by a fully connected layer with Mish activation. CSITime network consists of six such blocks followed by a global average pooling layer and a final fully connected layer for the final classification. Third, in the training of the neural network, instead of adopting general training procedures such as early stopping, we use one-cycle policy and cosine annealing to monitor the learning rate. The proposed model has been tested on publicly available benchmark datasets, i.e., ARIL, StanWiFi, and SignFi datasets. The proposed CSITime has achieved accuracy of 98.20%, 98%, and 95.42% on ARIL, StanWiFi, and SignFi datasets, respectively, for WiFi-based activity recognition. This is an improvement on state-of-the-art accuracies by 3.3%, 0.67%, and 0.82% on ARIL, StanWiFi, and SignFi datasets, respectively. In lab-5 users' scenario of the SignFi dataset, which has the training and testing data from different distributions, our model achieved accuracy was 2.17% higher than state-of-the-art, which shows the comparative robustness of our model.
Collapse
|