1
|
Alarfaj M, Al Madini A, Alsafran A, Farag M, Chtourou S, Afifi A, Ahmad A, Al Rubayyi O, Al Harbi A, Al Thunaian M. Wearable sensors based on artificial intelligence models for human activity recognition. Front Artif Intell 2024; 7:1424190. [PMID: 39015365 PMCID: PMC11250658 DOI: 10.3389/frai.2024.1424190] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/27/2024] [Accepted: 06/17/2024] [Indexed: 07/18/2024] Open
Abstract
Human motion detection technology holds significant potential in medicine, health care, and physical exercise. This study introduces a novel approach to human activity recognition (HAR) using convolutional neural networks (CNNs) designed for individual sensor types to enhance the accuracy and address the challenge of diverse data shapes from accelerometers, gyroscopes, and barometers. Specific CNN models are constructed for each sensor type, enabling them to capture the characteristics of their respective sensors. These adapted CNNs are designed to effectively process varying data shapes and sensor-specific characteristics to accurately classify a wide range of human activities. The late-fusion technique is employed to combine predictions from various models to obtain comprehensive estimates of human activity. The proposed CNN-based approach is compared to a standard support vector machine (SVM) classifier using the one-vs-rest methodology. The late-fusion CNN model showed significantly improved performance, with validation and final test accuracies of 99.35 and 94.83% compared to the conventional SVM classifier at 87.07 and 83.10%, respectively. These findings provide strong evidence that combining multiple sensors and a barometer and utilizing an additional filter algorithm greatly improves the accuracy of identifying different human movement patterns.
Collapse
Affiliation(s)
- Mohammed Alarfaj
- Department of Electrical Engineering, College of Engineering, King Faisal University, Al-Ahsa, Saudi Arabia
| | - Azzam Al Madini
- Department of Electrical Engineering, College of Engineering, King Faisal University, Al-Ahsa, Saudi Arabia
| | - Ahmed Alsafran
- Department of Electrical Engineering, College of Engineering, King Faisal University, Al-Ahsa, Saudi Arabia
| | - Mohammed Farag
- Department of Electrical Engineering, College of Engineering, King Faisal University, Al-Ahsa, Saudi Arabia
| | - Slim Chtourou
- Department of Electrical Engineering, College of Engineering, King Faisal University, Al-Ahsa, Saudi Arabia
| | - Ahmed Afifi
- Department of Computer Science, College of Computer Science and Information Technology, King Faisal University, Al-Ahsa, Saudi Arabia
| | - Ayaz Ahmad
- Department of Electrical Engineering, College of Engineering, King Faisal University, Al-Ahsa, Saudi Arabia
| | - Osama Al Rubayyi
- Department of Electrical Engineering, College of Engineering, King Faisal University, Al-Ahsa, Saudi Arabia
| | - Ali Al Harbi
- Department of Electrical Engineering, College of Engineering, King Faisal University, Al-Ahsa, Saudi Arabia
| | - Mustafa Al Thunaian
- Department of Electrical Engineering, College of Engineering, King Faisal University, Al-Ahsa, Saudi Arabia
| |
Collapse
|
2
|
Xiao L, Luo K, Liu J, Foroughi A. A hybrid deep approach to recognizing student activity and monitoring health physique based on accelerometer data from smartphones. Sci Rep 2024; 14:14006. [PMID: 38890409 PMCID: PMC11189493 DOI: 10.1038/s41598-024-63934-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/03/2024] [Accepted: 05/31/2024] [Indexed: 06/20/2024] Open
Abstract
Smartphone sensors have gained considerable traction in Human Activity Recognition (HAR), drawing attention for their diverse applications. Accelerometer data monitoring holds promise in understanding students' physical activities, fostering healthier lifestyles. This technology tracks exercise routines, sedentary behavior, and overall fitness levels, potentially encouraging better habits, preempting health issues, and bolstering students' well-being. Traditionally, HAR involved analyzing signals linked to physical activities using handcrafted features. However, recent years have witnessed the integration of deep learning into HAR tasks, leveraging digital physiological signals from smartwatches and learning features automatically from raw sensory data. The Long Short-Term Memory (LSTM) network stands out as a potent algorithm for analyzing physiological signals, promising improved accuracy and scalability in automated signal analysis. In this article, we propose a feature analysis framework for recognizing student activity and monitoring health based on smartphone accelerometer data through an edge computing platform. Our objective is to boost HAR performance by accounting for the dynamic nature of human behavior. Nonetheless, the current LSTM network's presetting of hidden units and initial learning rate relies on prior knowledge, potentially leading to suboptimal states. To counter this, we employ Bidirectional LSTM (BiLSTM), enhancing sequence processing models. Furthermore, Bayesian optimization aids in fine-tuning the BiLSTM model architecture. Through fivefold cross-validation on training and testing datasets, our model showcases a classification accuracy of 97.5% on the tested dataset. Moreover, edge computing offers real-time processing, reduced latency, enhanced privacy, bandwidth efficiency, offline capabilities, energy efficiency, personalization, and scalability. Extensive experimental results validate that our proposed approach surpasses state-of-the-art methodologies in recognizing human activities and monitoring health based on smartphone accelerometer data.
Collapse
Affiliation(s)
- Lei Xiao
- Chengdu Technological University, Chengdu, 610000, China
- Graduate School of Business Faculty, Malaysia SEGi University, 47810, Petaling Jaya, Malaysia
| | - Kangrong Luo
- Chengdu Technological University, Chengdu, 610000, China
| | - Juntong Liu
- Chengdu University of Information Technology, Chengdu, 610000, China
| | - Andia Foroughi
- Department of Biomedical Engineering, Central Tehran Branch, Islamic Azad University, Tehran, Iran.
| |
Collapse
|
3
|
Raj R, Kos A. An improved human activity recognition technique based on convolutional neural network. Sci Rep 2023; 13:22581. [PMID: 38114574 PMCID: PMC10730728 DOI: 10.1038/s41598-023-49739-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/23/2023] [Accepted: 12/11/2023] [Indexed: 12/21/2023] Open
Abstract
A convolutional neural network (CNN) is an important and widely utilized part of the artificial neural network (ANN) for computer vision, mostly used in the pattern recognition system. The most important applications of CNN are medical image analysis, image classification, object recognition from videos, recommender systems, financial time series analysis, natural language processing, and human-computer interfaces. However, after the technological advancement in the power of computing ability and the emergence of huge quantities of labeled data provided through enhanced algorithms, nowadays, CNN is widely used in almost every area of study. One of the main uses of wearable technology and CNN within medical surveillance is human activity recognition (HAR), which must require constant tracking of everyday activities. This paper provides a comprehensive study of the application of CNNs in the classification of HAR tasks. We describe their enhancement, from their antecedents up to the current state-of-the-art systems of deep learning (DL). We have provided a comprehensive working principle of CNN for HAR tasks, and a CNN-based model is presented to perform the classification of human activities. The proposed technique interprets data from sensor sequences of inputs by using a multi-layered CNN that gathers temporal and spatial data related to human activities. The publicly available WISDM dataset for HAR has been used to perform this study. This proposed study uses the two-dimensional CNN approach to make a model for the classification of different human activities. A recent version of Python software has been used to perform the study. The rate of accuracy for HAR through the proposed model in this experiment is 97.20%, which is better than the previously estimated state-of-the-art technique. The findings of the study imply that using DL methods for activity recognition might greatly increase accuracy and increase the range of applications where HAR can be used successfully. We have also described the future research trends in the field of HAR in this article.
Collapse
Affiliation(s)
- Ravi Raj
- Faculty of Computer Science, Electronics, and Telecommunications, AGH University of Science and Technology, Aleja Adama Mickiewicza 30, 30-059, Krakow, Poland.
| | - Andrzej Kos
- Faculty of Computer Science, Electronics, and Telecommunications, AGH University of Science and Technology, Aleja Adama Mickiewicza 30, 30-059, Krakow, Poland
| |
Collapse
|
4
|
Guerra BMV, Torti E, Marenzi E, Schmid M, Ramat S, Leporati F, Danese G. Ambient assisted living for frail people through human activity recognition: state-of-the-art, challenges and future directions. Front Neurosci 2023; 17:1256682. [PMID: 37849892 PMCID: PMC10577184 DOI: 10.3389/fnins.2023.1256682] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/11/2023] [Accepted: 09/18/2023] [Indexed: 10/19/2023] Open
Abstract
Ambient Assisted Living is a concept that focuses on using technology to support and enhance the quality of life and well-being of frail or elderly individuals in both indoor and outdoor environments. It aims at empowering individuals to maintain their independence and autonomy while ensuring their safety and providing assistance when needed. Human Activity Recognition is widely regarded as the most popular methodology within the field of Ambient Assisted Living. Human Activity Recognition involves automatically detecting and classifying the activities performed by individuals using sensor-based systems. Researchers have employed various methodologies, utilizing wearable and/or non-wearable sensors, and employing algorithms ranging from simple threshold-based techniques to more advanced deep learning approaches. In this review, literature from the past decade is critically examined, specifically exploring the technological aspects of Human Activity Recognition in Ambient Assisted Living. An exhaustive analysis of the methodologies adopted, highlighting their strengths and weaknesses is provided. Finally, challenges encountered in the field of Human Activity Recognition for Ambient Assisted Living are thoroughly discussed. These challenges encompass issues related to data collection, model training, real-time performance, generalizability, and user acceptance. Miniaturization, unobtrusiveness, energy harvesting and communication efficiency will be the crucial factors for new wearable solutions.
Collapse
Affiliation(s)
- Bruna Maria Vittoria Guerra
- Bioengineering Laboratory, Department of Electrical, Computer and Biomedical Engineering, University of Pavia, Pavia, Italy
| | - Emanuele Torti
- Custom Computing and Programmable Systems Laboratory, Department of Electrical, Computer and Biomedical Engineering, University of Pavia, Pavia, Italy
| | - Elisa Marenzi
- Custom Computing and Programmable Systems Laboratory, Department of Electrical, Computer and Biomedical Engineering, University of Pavia, Pavia, Italy
| | - Micaela Schmid
- Bioengineering Laboratory, Department of Electrical, Computer and Biomedical Engineering, University of Pavia, Pavia, Italy
| | - Stefano Ramat
- Bioengineering Laboratory, Department of Electrical, Computer and Biomedical Engineering, University of Pavia, Pavia, Italy
| | - Francesco Leporati
- Custom Computing and Programmable Systems Laboratory, Department of Electrical, Computer and Biomedical Engineering, University of Pavia, Pavia, Italy
| | - Giovanni Danese
- Custom Computing and Programmable Systems Laboratory, Department of Electrical, Computer and Biomedical Engineering, University of Pavia, Pavia, Italy
| |
Collapse
|
5
|
Jaramillo IE, Chola C, Jeong JG, Oh JH, Jung H, Lee JH, Lee WH, Kim TS. Human Activity Prediction Based on Forecasted IMU Activity Signals by Sequence-to-Sequence Deep Neural Networks. SENSORS (BASEL, SWITZERLAND) 2023; 23:6491. [PMID: 37514789 PMCID: PMC10385571 DOI: 10.3390/s23146491] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/10/2023] [Revised: 07/10/2023] [Accepted: 07/14/2023] [Indexed: 07/30/2023]
Abstract
Human Activity Recognition (HAR) has gained significant attention due to its broad range of applications, such as healthcare, industrial work safety, activity assistance, and driver monitoring. Most prior HAR systems are based on recorded sensor data (i.e., past information) recognizing human activities. In fact, HAR works based on future sensor data to predict human activities are rare. Human Activity Prediction (HAP) can benefit in multiple applications, such as fall detection or exercise routines, to prevent injuries. This work presents a novel HAP system based on forecasted activity data of Inertial Measurement Units (IMU). Our HAP system consists of a deep learning forecaster of IMU activity signals and a deep learning classifier to recognize future activities. Our deep learning forecaster model is based on a Sequence-to-Sequence structure with attention and positional encoding layers. Then, a pre-trained deep learning Bi-LSTM classifier is used to classify future activities based on the forecasted IMU data. We have tested our HAP system for five daily activities with two tri-axial IMU sensors. The forecasted signals show an average correlation of 91.6% to the actual measured signals of the five activities. The proposed HAP system achieves an average accuracy of 97.96% in predicting future activities.
Collapse
Affiliation(s)
- Ismael Espinoza Jaramillo
- Department of Electronics and Information Convergence Engineering, Kyung Hee University, Yongin 17104, Republic of Korea
| | - Channabasava Chola
- Department of Electronics and Information Convergence Engineering, Kyung Hee University, Yongin 17104, Republic of Korea
| | - Jin-Gyun Jeong
- Department of Electronics and Information Convergence Engineering, Kyung Hee University, Yongin 17104, Republic of Korea
| | - Ji-Heon Oh
- Department of Electronics and Information Convergence Engineering, Kyung Hee University, Yongin 17104, Republic of Korea
| | - Hwanseok Jung
- Department of Electronics and Information Convergence Engineering, Kyung Hee University, Yongin 17104, Republic of Korea
| | - Jin-Hyuk Lee
- Department of Electronics and Information Convergence Engineering, Kyung Hee University, Yongin 17104, Republic of Korea
| | - Won Hee Lee
- Department of Software Convergence, Kyung Hee University, Yongin 17104, Republic of Korea
| | - Tae-Seong Kim
- Department of Electronics and Information Convergence Engineering, Kyung Hee University, Yongin 17104, Republic of Korea
| |
Collapse
|
6
|
Khan YA, Imaduddin S, Singh YP, Wajid M, Usman M, Abbas M. Artificial Intelligence Based Approach for Classification of Human Activities Using MEMS Sensors Data. SENSORS (BASEL, SWITZERLAND) 2023; 23:1275. [PMID: 36772315 PMCID: PMC9919731 DOI: 10.3390/s23031275] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 12/27/2022] [Revised: 01/15/2023] [Accepted: 01/18/2023] [Indexed: 06/18/2023]
Abstract
The integration of Micro Electronic Mechanical Systems (MEMS) sensor technology in smartphones has greatly improved the capability for Human Activity Recognition (HAR). By utilizing Machine Learning (ML) techniques and data from these sensors, various human motion activities can be classified. This study performed experiments and compiled a large dataset of nine daily activities, including Laying Down, Stationary, Walking, Brisk Walking, Running, Stairs-Up, Stairs-Down, Squatting, and Cycling. Several ML models, such as Decision Tree Classifier, Random Forest Classifier, K Neighbors Classifier, Multinomial Logistic Regression, Gaussian Naive Bayes, and Support Vector Machine, were trained on sensor data collected from accelerometer, gyroscope, and magnetometer embedded in smartphones and wearable devices. The highest test accuracy of 95% was achieved using the random forest algorithm. Additionally, a custom-built Bidirectional Long-Short-Term Memory (Bi-LSTM) model, a type of Recurrent Neural Network (RNN), was proposed and yielded an improved test accuracy of 98.1%. This approach differs from traditional algorithmic-based human activity detection used in current wearable technologies, resulting in improved accuracy.
Collapse
Affiliation(s)
- Yusuf Ahmed Khan
- Department of Electronics Engineering, ZHCET, Aligarh Muslim University, Aligarh 202002, India
| | - Syed Imaduddin
- Department of Electronics Engineering, ZHCET, Aligarh Muslim University, Aligarh 202002, India
| | - Yash Pratap Singh
- Department of Electronics Engineering, ZHCET, Aligarh Muslim University, Aligarh 202002, India
| | - Mohd Wajid
- Department of Electronics Engineering, ZHCET, Aligarh Muslim University, Aligarh 202002, India
| | - Mohammed Usman
- Department of Electrical Engineering, King Khalid University, Abha 61411, Saudi Arabia
| | - Mohamed Abbas
- Electrical Engineering Department, College of Engineering, King Khalid University, Abha 61421, Saudi Arabia
- Electronics and Communication Department, College of Engineering, Delta University for Science and Technology, Gamasa 35712, Egypt
| |
Collapse
|
7
|
Farag MM. Matched Filter Interpretation of CNN Classifiers with Application to HAR. SENSORS (BASEL, SWITZERLAND) 2022; 22:8060. [PMID: 36298408 PMCID: PMC9607232 DOI: 10.3390/s22208060] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 10/02/2022] [Revised: 10/16/2022] [Accepted: 10/18/2022] [Indexed: 06/16/2023]
Abstract
Time series classification is an active research topic due to its wide range of applications and the proliferation of sensory data. Convolutional neural networks (CNNs) are ubiquitous in modern machine learning (ML) models. In this work, we present a matched filter (MF) interpretation of CNN classifiers accompanied by an experimental proof of concept using a carefully developed synthetic dataset. We exploit this interpretation to develop an MF CNN model for time series classification comprising a stack of a Conv1D layer followed by a GlobalMaxPooling layer acting as a typical MF for automated feature extraction and a fully connected layer with softmax activation for computing class probabilities. The presented interpretation enables developing superlight highly accurate classifier models that meet the tight requirements of edge inference. Edge inference is emerging research that addresses the latency, availability, privacy, and connectivity concerns of the commonly deployed cloud inference. The MF-based CNN model has been applied to the sensor-based human activity recognition (HAR) problem due to its significant importance in a broad range of applications. The UCI-HAR, WISDM-AR, and MotionSense datasets are used for model training and testing. The proposed classifier is tested and benchmarked on an android smartphone with average accuracy and F1 scores of 98% and 97%, respectively, which outperforms state-of-the-art HAR methods in terms of classification accuracy and run-time performance. The proposed model size is less than 150 KB, and the average inference time is less than 1 ms. The presented interpretation helps develop a better understanding of CNN operation and decision mechanisms. The proposed model is distinguished from related work by jointly featuring interpretability, high accuracy, and low computational cost, enabling its ready deployment on a wide set of mobile devices for a broad range of applications.
Collapse
Affiliation(s)
- Mohammed M. Farag
- Electrical Engineering Department, College of Engineering, King Faisal University, Al-Ahsa 31982, Saudi Arabia;
- Electrical Engineering Department, Faculty of Engineering, Alexandria University, Alexandria 5424041, Egypt;
| |
Collapse
|
8
|
Arshad MH, Bilal M, Gani A. Human Activity Recognition: Review, Taxonomy and Open Challenges. SENSORS (BASEL, SWITZERLAND) 2022; 22:s22176463. [PMID: 36080922 PMCID: PMC9460866 DOI: 10.3390/s22176463] [Citation(s) in RCA: 8] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/28/2022] [Revised: 08/23/2022] [Accepted: 08/24/2022] [Indexed: 06/12/2023]
Abstract
Nowadays, Human Activity Recognition (HAR) is being widely used in a variety of domains, and vision and sensor-based data enable cutting-edge technologies to detect, recognize, and monitor human activities. Several reviews and surveys on HAR have already been published, but due to the constantly growing literature, the status of HAR literature needed to be updated. Hence, this review aims to provide insights on the current state of the literature on HAR published since 2018. The ninety-five articles reviewed in this study are classified to highlight application areas, data sources, techniques, and open research challenges in HAR. The majority of existing research appears to have concentrated on daily living activities, followed by user activities based on individual and group-based activities. However, there is little literature on detecting real-time activities such as suspicious activity, surveillance, and healthcare. A major portion of existing studies has used Closed-Circuit Television (CCTV) videos and Mobile Sensors data. Convolutional Neural Network (CNN), Long short-term memory (LSTM), and Support Vector Machine (SVM) are the most prominent techniques in the literature reviewed that are being utilized for the task of HAR. Lastly, the limitations and open challenges that needed to be addressed are discussed.
Collapse
Affiliation(s)
- Muhammad Haseeb Arshad
- Department of Computer Science, National University of Computer and Emerging Sciences, Chiniot-Faisalabad Campus, Chiniot 35400, Pakistan
| | - Muhammad Bilal
- Department of Software Engineering, National University of Computer and Emerging Sciences, Chiniot-Faisalabad Campus, Chiniot 35400, Pakistan
| | - Abdullah Gani
- Faculty of Computing and Informatics, University Malaysia Sabah, Kota Kinabalu 88400, Sabah, Malaysia
| |
Collapse
|