Exploration of Human Activity Recognition Using a Single Sensor for Stroke Survivors and Able-Bodied People

Human Activity Recognition Algorithm with Physiological and Inertial Signals Fusion: Photoplethysmography, Electrodermal Activity, and Accelerometry

Inertial signals are the most widely used signals in human activity recognition (HAR) applications, and extensive research has been performed on developing HAR classifiers using accelerometer and gyroscope data. This study aimed to investigate the potential enhancement of HAR models through the fusion of biological signals with inertial signals. The classification of eight common low-, medium-, and high-intensity activities was assessed using machine learning (ML) algorithms, trained on accelerometer (ACC), blood volume pulse (BVP), and electrodermal activity (EDA) data obtained from a wrist-worn sensor. Two types of ML algorithms were employed: a random forest (RF) trained on features; and a pre-trained deep learning (DL) network (ResNet-18) trained on spectrogram images. Evaluation was conducted on both individual activities and more generalized activity groups, based on similar intensity. Results indicated that RF classifiers outperformed corresponding DL classifiers at both individual and grouped levels. However, the fusion of EDA and BVP signals with ACC data improved DL classifier performance compared to a baseline DL model with ACC-only data. The best performance was achieved by a classifier trained on a combination of ACC, EDA, and BVP images, yielding F1-scores of 69 and 87 for individual and grouped activity classifications, respectively. For DL models trained with additional biological signals, almost all individual activity classifications showed improvement (p-value < 0.05). In grouped activity classifications, DL model performance was enhanced for low- and medium-intensity activities. Exploring the classification of two specific activities, ascending/descending stairs and cycling, revealed significantly improved results using a DL model trained on combined ACC, BVP, and EDA spectrogram images (p-value < 0.05).

Data Augmentation Techniques for Accurate Action Classification in Stroke Patients with Hemiparesis

Stroke survivors with hemiparesis require extensive home-based rehabilitation. Deep learning-based classifiers can detect actions and provide feedback based on patient data; however, this is difficult owing to data sparsity and heterogeneity. In this study, we investigate data augmentation and model training strategies to address this problem. Three transformations are tested with varying data volumes to analyze the changes in the classification performance of individual data. Moreover, the impact of transfer learning relative to a pre-trained one-dimensional convolutional neural network (Conv1D) and training with an advanced InceptionTime model are estimated with data augmentation. In Conv1D, the joint training data of non-disabled (ND) participants and double rotationally augmented data of stroke patients is observed to outperform the baseline in terms of F1-score (60.9% vs. 47.3%). Transfer learning pre-trained with ND data exhibits 60.3% accuracy, whereas joint training with InceptionTime exhibits 67.2% accuracy under the same conditions. Our results indicate that rotational augmentation is more effective for individual data with initially lower performance and subset data with smaller numbers of participants than other techniques, suggesting that joint training on rotationally augmented ND and stroke data enhances classification performance, particularly in cases with sparse data and lower initial performance.

A deep learning wearable-based solution for continuous at-home monitoring of upper limb goal-directed movements

Introduction

Monitoring upper limb function is crucial for tracking progress, assessing treatment effectiveness, and identifying potential problems or complications. Hand goal-directed movements (GDMs) are a crucial aspect of daily life, reflecting planned motor commands with hand trajectories towards specific target locations. Previous studies have shown that GDM tasks can detect early changes in upper limb function in neurodegenerative diseases and can be used to track disease progression over time.

Methods

In this study, we used accelerometer data from stroke survivor participants and controls doing activities of daily living to develop an automated deep learning approach to detect GDMs. The model performance for detecting GDM or non-GDM from windowed data achieved an AUC of 0.9, accuracy 0.83, sensitivity 0.81, specificity 0.84 and F1 0.82.

Results

We further validated the utility of detecting GDM by extracting features from GDM periods and using these features to classify whether the measurements are collected from a stroke survivor or a control participant, and to predict the Fugl-Meyer assessment score from stroke survivors.

Discussion

This study presents a promising and reliable tool for monitoring upper limb function in a real-world setting, and assessing biomarkers related to upper limb health in neurological, neuromuscular and muscles disorders.

A Robust Gaze Estimation Approach via Exploring Relevant Electrooculogram Features and Optimal Electrodes Placements

Gaze estimation, as a technique that reflects individual attention, can be used for disability assistance and assisting physicians in diagnosing diseases such as autism spectrum disorder (ASD), Parkinson's disease, and attention deficit hyperactivity disorder (ADHD). Various techniques have been proposed for gaze estimation and achieved high resolution. Among these approaches, electrooculography (EOG)-based gaze estimation, as an economical and effective method, offers a promising solution for practical applications.

OBJECTIVE

In this paper, we systematically investigated the possible EOG electrode locations which are spatially distributed around the orbital cavity. Afterward, quantities of informative features to characterize physiological information of eye movement from the temporal-spectral domain are extracted from the seven differential channels.

METHODS AND PROCEDURES

To select the optimum channels and relevant features, and eliminate irrelevant information, a heuristical search algorithm (i.e., forward stepwise strategy) is applied. Subsequently, a comparative analysis of the impacts of electrode placement and feature contributions on gaze estimation is evaluated via 6 classic models with 18 subjects.

RESULTS

Experimental results showed that the promising performance was achieved both in the Mean Absolute Error (MAE) and Root Mean Square Error (RMSE) within a wide gaze that ranges from -50° to +50°. The MAE and RMSE can be improved to 2.80° and 3.74° ultimately, while only using 10 features extracted from 2 channels. Compared with the prevailing EOG-based techniques, the performance improvement of MAE and RMSE range from 0.70° to 5.48° and 0.66° to 5.42°, respectively.

CONCLUSION

We proposed a robust EOG-based gaze estimation approach by systematically investigating the optimal channel/feature combination. The experimental results indicated not only the superiority of the proposed approach but also its potential for clinical application. Clinical and translational impact statement: Accurate gaze estimation is a key step for assisting disabilities and accurate diagnosis of various diseases including ASD, Parkinson's disease, and ADHD. The proposed approach can accurately estimate the points of gaze via EOG signals, and thus has the potential for various related medical applications.

Review of adaptive control for stroke lower limb exoskeleton rehabilitation robot based on motion intention recognition

Stroke is a significant cause of disability worldwide, and stroke survivors often experience severe motor impairments. Lower limb rehabilitation exoskeleton robots provide support and balance for stroke survivors and assist them in performing rehabilitation training tasks, which can effectively improve their quality of life during the later stages of stroke recovery. Lower limb rehabilitation exoskeleton robots have become a hot topic in rehabilitation therapy research. This review introduces traditional rehabilitation assessment methods, explores the possibility of lower limb exoskeleton robots combining sensors and electrophysiological signals to assess stroke survivors' rehabilitation objectively, summarizes standard human-robot coupling models of lower limb rehabilitation exoskeleton robots in recent years, and critically introduces adaptive control models based on motion intent recognition for lower limb exoskeleton robots. This provides new design ideas for the future combination of lower limb rehabilitation exoskeleton robots with rehabilitation assessment, motion assistance, rehabilitation treatment, and adaptive control, making the rehabilitation assessment process more objective and addressing the shortage of rehabilitation therapists to some extent. Finally, the article discusses the current limitations of adaptive control of lower limb rehabilitation exoskeleton robots for stroke survivors and proposes new research directions.

Automatic Post-Stroke Severity Assessment Using Novel Unsupervised Consensus Learning for Wearable and Camera-Based Sensor Datasets

Stroke survivors often suffer from movement impairments that significantly affect their daily activities. The advancements in sensor technology and IoT have provided opportunities to automate the assessment and rehabilitation process for stroke survivors. This paper aims to provide a smart post-stroke severity assessment using AI-driven models. With the absence of labelled data and expert assessment, there is a research gap in providing virtual assessment, especially for unlabeled data. Inspired by the advances in consensus learning, in this paper, we propose a consensus clustering algorithm, PSA-NMF, that combines various clusterings into one united clustering, i.e., cluster consensus, to produce more stable and robust results compared to individual clustering. This paper is the first to investigate severity level using unsupervised learning and trunk displacement features in the frequency domain for post-stroke smart assessment. Two different methods of data collection from the U-limb datasets-the camera-based method (Vicon) and wearable sensor-based technology (Xsens)-were used. The trunk displacement method labelled each cluster based on the compensatory movements that stroke survivors employed for their daily activities. The proposed method uses the position and acceleration data in the frequency domain. Experimental results have demonstrated that the proposed clustering method that uses the post-stroke assessment approach increased the evaluation metrics such as accuracy and F-score. These findings can lead to a more effective and automated stroke rehabilitation process that is suitable for clinical settings, thus improving the quality of life for stroke survivors.

Automatic Upper-Limb Brunnstrom Recovery Stage Evaluation via Daily Activity Monitoring

Motor function assessment is crucial for post-stroke rehabilitation. Conventional evaluation methods are subjective, heavily depending on the experience of therapists. In light of the strong correlation between the stroke severity level and the performance of activities of daily living (ADLs), we explored the possibility of automatically evaluating the upper-limb Brunnstrom Recovery Stage (BRS) via three typical ADLs (tooth brushing, face washing and drinking). Multimodal data (acceleration, angular velocity, surface electromyography) were synchronously collected from 5 upper-limb-worn sensor modules. The performance of BRS evaluation system is known to be variable with different system parameters (e.g., number of sensor modules, feature types and classifiers). We systematically searched for the optimal parameters from different data segmentation strategies (five window lengths and four overlaps), 42 types of features, 12 feature optimization techniques and 9 classifiers with the leave-one-subject-out cross-validation. To achieve reliable and low-cost monitoring, we further explored whether it was possible to obtain a satisfactory result using a relatively small number of sensor modules. As a result, the proposed approach can correctly recognize the stages of all 27 participants using only three sensor modules with the optimized data segmentation parameters (window length: 7s, overlap: 50%), extracted features (simple square integral, slope sign change, modified mean absolute value 1 and modified mean absolute value 2), the feature optimization method (principal component analysis) and the logistic regression classifier. According to the literature, this is the first study to comprehensively optimize sensor configuration and parameters in each stage of the BRS classification framework. The proposed approach can serve as a factor-screening tool towards the automatic BRS classification and is promising to be further used at home.

Exploring human activity recognition using feature level fusion of inertial and electromyography data

Wearables are objective tools for human activity recognition (HAR). Advances in wearables enable synchronized multi-sensing within a single device. This has resulted in studies investigating the use of single or multiple wearable sensor modalities for HAR. Some studies use inertial data, others use surface electromyography (sEMG) from multiple muscles and different post-processing approaches. Yet, questions remain about accuracies relating to e.g., multi-modal approaches, and sEMG post-processing. Here, we explored how inertial and sEMG could be efficiently combined with machine learning and used with post-processing methods for better HAR. This study aims recognition of four basic daily life activities; walking, standing, stair ascent and descent. Firstly, we created a new feature vector based on the domain knowledge gained from previous mobility studies. Then, a feature level data fusion approach was used to combine inertial and sEMG data. Finally, two supervised learning classifiers (Support Vector Machine, SVM, and the k-Nearest Neighbors, kNN) were tested with 5-fold cross-validation. Results show the use of inertial data with sEMG increased overall accuracy by 3.5% (SVM) and 6.3% (kNN). Extracting features from linear envelopes instead of bandpass filtered sEMG improves overall HAR accuracy in both classifiers. Clinical Relevance- Post-processing on sEMG signals can improve the performance of multimodal HAR.

Automatic upper-extremity Brunnstrom Classification for Stroke Survivors with a Minimum Number of Tasks

Motor function evaluation plays an important role in post-stroke rehabilitation. However, the traditional evaluation is subjective and laborious, which may bring a heavy burden to both physicians and stroke survivors. Therefore, an automatic and objective rehabilitation evaluation is needed to minimize the burden of physician, so as to achieve a simplified and objective evaluation process. The main purpose of this study is to investigate the minimum number of tasks for upper-extremity actions in objective assessment of stroke survivors with a Brunnstrom stage (BS) based on wearable sensing device, which can achieve a satisfactory result to reduce the burden of stroke survivors. In this study, we employed 20 stroke survivors and 7 healthy participants, performing three types of daily living activities (drinking, teeth brushing, face washing). The acceleration, angular velocity and surface Electromyography signals on five parts of the forearm were simultaneously acquired. Then, we compared the effects of each action combination under multiple classifiers. The results show that the use of a single action can achieve competitive results compared with multiple action combination classifications, and the use of K nearest neighbor (KNN) algorithm for the average recognition accuracy of face washing action shows better performance, with the highest accuracy reaching 85.65±6.21% (mean ± standard error), 23 of the 27 subjects were accurately classified. These findings indicate that the predominant qualitative assessment after stroke can be supplemented by corresponding quantitative solutions, and that stroke rehabilitation can be automated with less professional therapist involvement.