Human Activity Recognition: Review, Taxonomy and Open Challenges

BodyFlow: An Open-Source Library for Multimodal Human Activity Recognition

Human activity recognition is a critical task for various applications across healthcare, sports, security, gaming, and other fields. This paper presents BodyFlow, a comprehensive library that seamlessly integrates human pose estimation and multiple-person estimation and tracking, along with activity recognition modules. BodyFlow enables users to effortlessly identify common activities and 2D/3D body joints from input sources such as videos, image sets, or webcams. Additionally, the library can simultaneously process inertial sensor data, offering users the flexibility to choose their preferred input, thus facilitating multimodal human activity recognition. BodyFlow incorporates state-of-the-art algorithms for 2D and 3D pose estimation and three distinct models for human activity recognition.

Detection of Lowering in Sport Climbing Using Orientation-Based Sensor-Enhanced Quickdraws: A Preliminary Investigation

Climbing gyms aim to continuously improve their offerings and make the best use of their infrastructure to provide a unique experience for their clients, the climbers. One approach to achieve this goal is to track and analyze climbing sessions from the beginning of the ascent until the climber's descent. Detecting the climber's descent is crucial because it indicates when the ascent has ended. This paper discusses an approach that preserves climber privacy (e.g., not using cameras) while considering the convenience of climbers and the costs to the gyms. To this aim, a hardware prototype has been developed to collect data using accelerometer sensors attached to a piece of climbing equipment mounted on the wall, called a quickdraw, which connects the climbing rope to the bolt anchors. The sensors are configured to be energy-efficient, making them practical in terms of expenses and time required for replacement when used in large quantities in a climbing gym. This paper describes the hardware specifications, studies data measured by the sensors in ultra-low power mode, detects sensors' orientation patterns during descent on different routes, and develops a supervised approach to identify lowering. Additionally, the study emphasizes the benefits of multidisciplinary feature engineering, combining domain-specific knowledge with machine learning to enhance performance and simplify implementation.

A Robust Deep Feature Extraction Method for Human Activity Recognition Using a Wavelet Based Spectral Visualisation Technique

Human Activity Recognition (HAR), alongside Ambient Assisted Living (AAL), are integral components of smart homes, sports, surveillance, and investigation activities. To recognize daily activities, researchers are focusing on lightweight, cost-effective, wearable sensor-based technologies as traditional vision-based technologies lack elderly privacy, a fundamental right of every human. However, it is challenging to extract potential features from 1D multi-sensor data. Thus, this research focuses on extracting distinguishable patterns and deep features from spectral images by time-frequency-domain analysis of 1D multi-sensor data. Wearable sensor data, particularly accelerator and gyroscope data, act as input signals of different daily activities, and provide potential information using time-frequency analysis. This potential time series information is mapped into spectral images through a process called use of 'scalograms', derived from the continuous wavelet transform. The deep activity features are extracted from the activity image using deep learning models such as CNN, MobileNetV3, ResNet, and GoogleNet and subsequently classified using a conventional classifier. To validate the proposed model, SisFall and PAMAP2 benchmark datasets are used. Based on the experimental results, this proposed model shows the optimal performance for activity recognition obtaining an accuracy of 98.4% for SisFall and 98.1% for PAMAP2, using Morlet as the mother wavelet with ResNet-101 and a softmax classifier, and outperforms state-of-the-art algorithms.

Biosensor-Driven IoT Wearables for Accurate Body Motion Tracking and Localization

The domain of human locomotion identification through smartphone sensors is witnessing rapid expansion within the realm of research. This domain boasts significant potential across various sectors, including healthcare, sports, security systems, home automation, and real-time location tracking. Despite the considerable volume of existing research, the greater portion of it has primarily concentrated on locomotion activities. Comparatively less emphasis has been placed on the recognition of human localization patterns. In the current study, we introduce a system by facilitating the recognition of both human physical and location-based patterns. This system utilizes the capabilities of smartphone sensors to achieve its objectives. Our goal is to develop a system that can accurately identify different human physical and localization activities, such as walking, running, jumping, indoor, and outdoor activities. To achieve this, we perform preprocessing on the raw sensor data using a Butterworth filter for inertial sensors and a Median Filter for Global Positioning System (GPS) and then applying Hamming windowing techniques to segment the filtered data. We then extract features from the raw inertial and GPS sensors and select relevant features using the variance threshold feature selection method. The extrasensory dataset exhibits an imbalanced number of samples for certain activities. To address this issue, the permutation-based data augmentation technique is employed. The augmented features are optimized using the Yeo-Johnson power transformation algorithm before being sent to a multi-layer perceptron for classification. We evaluate our system using the K-fold cross-validation technique. The datasets used in this study are the Extrasensory and Sussex Huawei Locomotion (SHL), which contain both physical and localization activities. Our experiments demonstrate that our system achieves high accuracy with 96% and 94% over Extrasensory and SHL in physical activities and 94% and 91% over Extrasensory and SHL in the location-based activities, outperforming previous state-of-the-art methods in recognizing both types of activities.

Human Activity Recognition in a Free-Living Environment Using an Ear-Worn Motion Sensor

Human activity recognition (HAR) technology enables continuous behavior monitoring, which is particularly valuable in healthcare. This study investigates the viability of using an ear-worn motion sensor for classifying daily activities, including lying, sitting/standing, walking, ascending stairs, descending stairs, and running. Fifty healthy participants (between 20 and 47 years old) engaged in these activities while under monitoring. Various machine learning algorithms, ranging from interpretable shallow models to state-of-the-art deep learning approaches designed for HAR (i.e., DeepConvLSTM and ConvTransformer), were employed for classification. The results demonstrate the ear sensor's efficacy, with deep learning models achieving a 98% accuracy rate of classification. The obtained classification models are agnostic regarding which ear the sensor is worn and robust against moderate variations in sensor orientation (e.g., due to differences in auricle anatomy), meaning no initial calibration of the sensor orientation is required. The study underscores the ear's efficacy as a suitable site for monitoring human daily activity and suggests its potential for combining HAR with in-ear vital sign monitoring. This approach offers a practical method for comprehensive health monitoring by integrating sensors in a single anatomical location. This integration facilitates individualized health assessments, with potential applications in tele-monitoring, personalized health insights, and optimizing athletic training regimes.

Detection of activities in bathrooms through deep learning and environmental data graphics images

Automatic detection activities in indoor spaces has been and is a matter of great interest. Thus, in the field of health surveillance, one of the spaces frequently studied is the bathroom of homes and specifically the behaviour of users in the said space, since certain pathologies can sometimes be deduced from it. That is why, the objective of this study is to know if it is possible to automatically classify the main activities that occur within the bathroom, using an innovative methodology with respect to the methods used to date, based on environmental parameters and the application of machine learning algorithms, thus allowing privacy to be preserved, which is a notable improvement in relation to other methods. For this, the methodology followed is based on the novel application of a pre-trained convolutional network for classifying graphs resulting from the monitoring of the environmental parameters of a bathroom. The results obtained allow us to conclude that, in addition to being able to check whether environmental data are adequate for health, it is possible to detect a high rate of true positives (around 80%) in some of the most frequent and important activities, thus facilitating its automation in a very simple and economical way.

HARE: Unifying the Human Activity Recognition Engineering Workflow

Sensor-based human activity recognition is becoming ever more prevalent. The increasing importance of distinguishing human movements, particularly in healthcare, coincides with the advent of increasingly compact sensors. A complex sequence of individual steps currently characterizes the activity recognition pipeline. It involves separate data collection, preparation, and processing steps, resulting in a heterogeneous and fragmented process. To address these challenges, we present a comprehensive framework, HARE, which seamlessly integrates all necessary steps. HARE offers synchronized data collection and labeling, integrated pose estimation for data anonymization, a multimodal classification approach, and a novel method for determining optimal sensor placement to enhance classification results. Additionally, our framework incorporates real-time activity recognition with on-device model adaptation capabilities. To validate the effectiveness of our framework, we conducted extensive evaluations using diverse datasets, including our own collected dataset focusing on nursing activities. Our results show that HARE's multimodal and on-device trained model outperforms conventional single-modal and offline variants. Furthermore, our vision-based approach for optimal sensor placement yields comparable results to the trained model. Our work advances the field of sensor-based human activity recognition by introducing a comprehensive framework that streamlines data collection and classification while offering a novel method for determining optimal sensor placement.

Machine Learning-Based Human Posture Identification from Point Cloud Data Acquisitioned by FMCW Millimetre-Wave Radar

Human posture recognition technology is widely used in the fields of healthcare, human-computer interaction, and sports. The use of a Frequency-Modulated Continuous Wave (FMCW) millimetre-wave (MMW) radar sensor in measuring human posture characteristics data is of great significance because of its robust and strong recognition capabilities. This paper demonstrates how human posture characteristics data are measured, classified, and identified using FMCW techniques. First of all, the characteristics data of human posture is measured with the MMW radar sensors. Secondly, the point cloud data for human posture is generated, considering both the dynamic and static features of the reflected signal from the human body, which not only greatly reduces the environmental noise but also strengthens the reflection of the detected target. Lastly, six different machine learning models are applied for posture classification based on the generated point cloud data. To comparatively evaluate the proper model for point cloud data classification procedure-in addition to using the traditional index-the Kappa index was introduced to eliminate the effect due to the uncontrollable imbalance of the sampling data. These results support our conclusion that among the six machine learning algorithms implemented in this paper, the multi-layer perceptron (MLP) method is regarded as the most promising classifier.

Human Action Recognition in Smart Living Services and Applications: Context Awareness, Data Availability, Personalization, and Privacy

Smart living, an increasingly prominent concept, entails incorporating sophisticated technologies in homes and urban environments to elevate the quality of life for citizens. A critical success factor for smart living services and applications, from energy management to healthcare and transportation, is the efficacy of human action recognition (HAR). HAR, rooted in computer vision, seeks to identify human actions and activities using visual data and various sensor modalities. This paper extensively reviews the literature on HAR in smart living services and applications, amalgamating key contributions and challenges while providing insights into future research directions. The review delves into the essential aspects of smart living, the state of the art in HAR, and the potential societal implications of this technology. Moreover, the paper meticulously examines the primary application sectors in smart living that stand to gain from HAR, such as smart homes, smart healthcare, and smart cities. By underscoring the significance of the four dimensions of context awareness, data availability, personalization, and privacy in HAR, this paper offers a comprehensive resource for researchers and practitioners striving to advance smart living services and applications. The methodology for this literature review involved conducting targeted Scopus queries to ensure a comprehensive coverage of relevant publications in the field. Efforts have been made to thoroughly evaluate the existing literature, identify research gaps, and propose future research directions. The comparative advantages of this review lie in its comprehensive coverage of the dimensions essential for smart living services and applications, addressing the limitations of previous reviews and offering valuable insights for researchers and practitioners in the field.

Review on Human Action Recognition in Smart Living: Sensing Technology, Multimodality, Real-Time Processing, Interoperability, and Resource-Constrained Processing

Smart living, a concept that has gained increasing attention in recent years, revolves around integrating advanced technologies in homes and cities to enhance the quality of life for citizens. Sensing and human action recognition are crucial aspects of this concept. Smart living applications span various domains, such as energy consumption, healthcare, transportation, and education, which greatly benefit from effective human action recognition. This field, originating from computer vision, seeks to recognize human actions and activities using not only visual data but also many other sensor modalities. This paper comprehensively reviews the literature on human action recognition in smart living environments, synthesizing the main contributions, challenges, and future research directions. This review selects five key domains, i.e., Sensing Technology, Multimodality, Real-time Processing, Interoperability, and Resource-Constrained Processing, as they encompass the critical aspects required for successfully deploying human action recognition in smart living. These domains highlight the essential role that sensing and human action recognition play in successfully developing and implementing smart living solutions. This paper serves as a valuable resource for researchers and practitioners seeking to further explore and advance the field of human action recognition in smart living.

Deep Learning for Human Activity Recognition on 3D Human Skeleton: Survey and Comparative Study

Human activity recognition (HAR) is an important research problem in computer vision. This problem is widely applied to building applications in human-machine interactions, monitoring, etc. Especially, HAR based on the human skeleton creates intuitive applications. Therefore, determining the current results of these studies is very important in selecting solutions and developing commercial products. In this paper, we perform a full survey on using deep learning to recognize human activity based on three-dimensional (3D) human skeleton data as input. Our research is based on four types of deep learning networks for activity recognition based on extracted feature vectors: Recurrent Neural Network (RNN) using extracted activity sequence features; Convolutional Neural Network (CNN) uses feature vectors extracted based on the projection of the skeleton into the image space; Graph Convolution Network (GCN) uses features extracted from the skeleton graph and the temporal-spatial function of the skeleton; Hybrid Deep Neural Network (Hybrid-DNN) uses many other types of features in combination. Our survey research is fully implemented from models, databases, metrics, and results from 2019 to March 2023, and they are presented in ascending order of time. In particular, we also carried out a comparative study on HAR based on a 3D human skeleton on the KLHA3D 102 and KLYOGA3D datasets. At the same time, we performed analysis and discussed the obtained results when applying CNN-based, GCN-based, and Hybrid-DNN-based deep learning networks.

Effect of Equipment on the Accuracy of Accelerometer-Based Human Activity Recognition in Extreme Environments

A little explored area of human activity recognition (HAR) is in people operating in relation to extreme environments, e.g., mountaineers. In these contexts, the ability to accurately identify activities, alongside other data streams, has the potential to prevent death and serious negative health events to the operators. This study aimed to address this user group and investigate factors associated with the placement, number, and combination of accelerometer sensors. Eight participants (age = 25.0 ± 7 years) wore 17 accelerometers simultaneously during lab-based simulated mountaineering activities, under a range of equipment and loading conditions. Initially, a selection of machine learning techniques was tested. Secondly, a comprehensive analysis of all possible combinations of the 17 accelerometers was performed to identify the optimum number of sensors, and their respective body locations. Finally, the impact of activity-specific equipment on the classifier accuracy was explored. The results demonstrated that the support vector machine (SVM) provided the most accurate classifications of the five machine learning algorithms tested. It was found that two sensors provided the optimum balance between complexity, performance, and user compliance. Sensors located on the hip and right tibia produced the most accurate classification of the simulated activities (96.29%). A significant effect associated with the use of mountaineering boots and a 12 kg rucksack was established.

Human Activity Recognition with an HMM-Based Generative Model

Human activity recognition (HAR) has become an interesting topic in healthcare. This application is important in various domains, such as health monitoring, supporting elders, and disease diagnosis. Considering the increasing improvements in smart devices, large amounts of data are generated in our daily lives. In this work, we propose unsupervised, scaled, Dirichlet-based hidden Markov models to analyze human activities. Our motivation is that human activities have sequential patterns and hidden Markov models (HMMs) are some of the strongest statistical models used for modeling data with continuous flow. In this paper, we assume that emission probabilities in HMM follow a bounded-scaled Dirichlet distribution, which is a proper choice in modeling proportional data. To learn our model, we applied the variational inference approach. We used a publicly available dataset to evaluate the performance of our proposed model.

Human Activity Recognition for AI-Enabled Healthcare Using Low-Resolution Infrared Sensor Data

This paper explores the feasibility of using low-resolution infrared (LRIR) image streams for human activity recognition (HAR) with potential application in e-healthcare. Two datasets based on synchronized multichannel LRIR sensors systems are considered for a comprehensive study about optimal data acquisition. A novel noise reduction technique is proposed for alleviating the effects of horizontal and vertical periodic noise in the 2D spatiotemporal activity profiles created by vectorizing and concatenating the LRIR frames. Two main analysis strategies are explored for HAR, including (1) manual feature extraction using texture-based and orthogonal-transformation-based techniques, followed by classification using support vector machine (SVM), random forest (RF), k-nearest neighbor (k-NN), and logistic regression (LR), and (2) deep neural network (DNN) strategy based on a convolutional long short-term memory (LSTM). The proposed periodic noise reduction technique showcases an increase of up to 14.15% using different models. In addition, for the first time, the optimum number of sensors, sensor layout, and distance to subjects are studied, indicating the optimum results based on a single side sensor at a close distance. Reasonable accuracies are achieved in the case of sensor displacement and robustness in detection of multiple subjects. Furthermore, the models show suitability for data collected in different environments.