A Data Set of Human Body Movements for Physical Rehabilitation Exercises

Viewpoint-invariant exercise repetition counting

Counting the repetition of human exercise and physical rehabilitation is common in rehabilitation and exercise training. The existing vision-based repetition counting methods less emphasize the concurrent motions in the same video, and counting skeleton in different view angles. This work analyzed the spectrogram of the pose estimation cosine similarity to count the repetition. Besides the public datasets. This work also collected exercise videos from 11 adults to verify that the proposed method can handle concurrent motion and different view angles. The presented method was validated on the University of Idaho Physical Rehabilitation Movements Data Set (UI-PRMD) and MM-fit dataset. The overall mean absolute error (MAE) for MM-fit was 0.06 with off-by-one Accuracy (OBOA) of 0.94. As for the UI-PRMD dataset, MAE was 0.06 with OBOA 0.95. We have also tested the performance in various camera locations and concurrent motions with 57 skeleton time-series videos with an overall MAE of 0.07 and OBOA of 0.91. The proposed method provides a view-angle and motion agnostic concurrent motion counting. This method can potentially use in large-scale remote rehabilitation and exercise training with only one camera.

Supplementary Information

The online version contains supplementary material available at 10.1007/s13755-023-00258-3.

Automatic two-dimensional & three-dimensional video analysis with deep learning for movement disorders: A systematic review

The advent of computer vision technology and increased usage of video cameras in clinical settings have facilitated advancements in movement disorder analysis. This review investigated these advancements in terms of providing practical, low-cost solutions for the diagnosis and analysis of movement disorders, such as Parkinson's disease, ataxia, dyskinesia, and Tourette syndrome. Traditional diagnostic methods for movement disorders are typically reliant on the subjective assessment of motor symptoms, which poses inherent challenges. Furthermore, early symptoms are often overlooked, and overlapping symptoms across diseases can complicate early diagnosis. Consequently, deep learning has been used for the objective video-based analysis of movement disorders. This study systematically reviewed the latest advancements in automatic two-dimensional & three-dimensional video analysis using deep learning for movement disorders. We comprehensively analyzed the literature published until September 2023 by searching the Web of Science, PubMed, Scopus, and Embase databases. We identified 68 relevant studies and extracted information on their objectives, datasets, modalities, and methodologies. The study aimed to identify, catalogue, and present the most significant advancements, offering a consolidated knowledge base on the role of video analysis and deep learning in movement disorder analysis. First, the objectives, including specific PD symptom quantification, ataxia assessment, cerebral palsy assessment, gait disorder analysis, tremor assessment, tic detection (in the context of Tourette syndrome), dystonia assessment, and abnormal movement recognition were discussed. Thereafter, the datasets used in the study were examined. Subsequently, video modalities and deep learning methodologies related to the topic were investigated. Finally, the challenges and opportunities in terms of datasets, interpretability, evaluation methods, and home/remote monitoring were discussed.

Promoting fairness in activity recognition algorithms for patient's monitoring and evaluation systems in healthcare

Researchers face the challenge of defining subject selection criteria when training algorithms for human activity recognition tasks. The ongoing uncertainty revolves around which characteristics should be considered to ensure algorithmic robustness across diverse populations. This study aims to address this challenge by conducting an analysis of heterogeneity in the training data to assess the impact of physical characteristics and soft-biometric attributes on activity recognition performance. The performance of various state-of-the-art deep neural network architectures (tCNN, hybrid-LSTM, Transformer model) processing time-series data using the IntelliRehab (IRDS) dataset was evaluated. By intentionally introducing bias into the training data based on human characteristics, the objective is to identify the characteristics that influence algorithms in motion analysis. Experimental findings reveal that the CNN-LSTM model achieved the highest accuracy, reaching 88%. Moreover, models trained on heterogeneous distributions of disability attributes exhibited notably higher accuracy, reaching 51%, compared to those not considering such factors, which scored an average of 33%. These evaluations underscore the significant influence of subjects' characteristics on activity recognition performance, providing valuable insights into the algorithm's robustness across diverse populations. This study represents a significant step forward in promoting fairness and trustworthiness in artificial intelligence by quantifying representation bias in multi-channel time-series activity recognition data within the healthcare domain.

EGCN++: A New Fusion Strategy for Ensemble Learning in Skeleton-Based Rehabilitation Exercise Assessment

Skeleton-based exercise assessment focuses on evaluating the correctness or quality of an exercise performed by a subject. Skeleton data provide two groups of features (i.e., position and orientation), which existing methods have not fully harnessed. We previously proposed an ensemble-based graph convolutional network (EGCN) that considers both position and orientation features to construct a model-based approach. Integrating these types of features achieved better performance than available methods. However, EGCN lacked a fusion strategy across the data, feature, decision, and model levels. In this paper, we present an advanced framework, EGCN++, for rehabilitation exercise assessment. Based on EGCN, a new fusion strategy called MLE-PO is proposed for EGCN++; this technique considers fusion at the data and model levels. We conduct extensive cross-validation experiments and investigate the consistency between machine and human evaluations on three datasets: UI-PRMD, KIMORE, and EHE. Results demonstrate that MLE-PO outperforms other EGCN ensemble strategies and representative baselines. Furthermore, the MLE-PO's model evaluation scores are more quantitatively consistent with clinical evaluations than other ensemble strategies.

Automatic rehabilitation exercise task assessment of stroke patients based on wearable sensors with a lightweight multichannel 1D-CNN model

Approximately 75% of stroke survivors have movement dysfunction. Rehabilitation exercises are capable of improving physical coordination. They are mostly conducted in the home environment without guidance from therapists. It is impossible to provide timely feedback on exercises without suitable devices or therapists. Human action quality assessment in the home setting is a challenging topic for current research. In this paper, a low-cost HREA system in which wearable sensors are used to collect upper limb exercise data and a multichannel 1D-CNN framework is used to automatically assess action quality. The proposed 1D-CNN model is first pretrained on the UCI-HAR dataset, and it achieves a performance of 91.96%. Then, five typical actions were selected from the Fugl-Meyer Assessment Scale for the experiment, wearable sensors were used to collect the participants' exercise data, and experienced therapists were employed to assess participants' exercise at the same time. Following the above process, a dataset was built based on the Fugl-Meyer scale. Based on the 1D-CNN model, a multichannel 1D-CNN model was built, and the model using the Naive Bayes fusion had the best performance (precision: 97.26%, recall: 97.22%, F1-score: 97.23%) on the dataset. This shows that the HREA system provides accurate and timely assessment, which can provide real-time feedback for stroke survivors' home rehabilitation.

LightPRA: A Lightweight Temporal Convolutional Network for Automatic Physical Rehabilitation Exercise Assessment

Research evidence shows that physical rehabilitation exercises prescribed by medical experts can assist in restoring physical function, improving life quality, and promoting independence for physically disabled individuals. In response to the absence of immediate expert feedback on performed actions, developing a Human Action Evaluation (HAE) system emerges as a valuable automated solution, addressing the need for accurate assessment of exercises and guidance during physical rehabilitation. Previous HAE systems developed for the rehabilitation exercises have focused on developing models that utilize skeleton data as input to compute a quality score for each action performed by the patient. However, existing studies have focused on improving scoring performance while often overlooking computational efficiency. In this research, we propose LightPRA (Light Physical Rehabilitation Assessment) system, an innovative architectural solution based on a Temporal Convolutional Network (TCN), which harnesses the capabilities of dilated causal Convolutional Neural Networks (CNNs). This approach efficiently captures complex temporal features and characteristics of the skeleton data with lower computational complexity, making it suitable for real-time feedback provided on resource-constrained devices such as Internet of Things (IoT) devices and Edge computing frameworks. Through empirical analysis performed on the University of Idaho-Physical Rehabilitation Movement Data (UI-PRMD) and KInematic assessment of MOvement for remote monitoring of physical REhabilitation (KIMORE) datasets, our proposed LightPRA model demonstrates superior performance over several state-of-the-art approaches such as Spatial-Temporal Graph Convolutional Network (STGCN) and Long Short-Term Memory (LSTM)-based models in scoring human activity performance, while exhibiting lower computational cost and complexity.

Technological advances in lower-limb tele-rehabilitation: A review of literature

Tele-rehabilitation is a healthcare practice that leverages technology to provide rehabilitation services remotely to individuals in their own homes or other locations. With advancements in remote monitoring and Artificial Intelligence, automatic tele-rehabilitation systems that can measure joint angles, recognize exercises, and provide feedback based on movement analysis are being developed. Such platforms can offer valuable information to clinicians for improved care planning. However, with various methods and sensors being used, understanding their pros, cons, and performance is important. This paper reviews and compares the performance of recent vision-based, wearable, and pressure-sensing technologies used in lower limb tele-rehabilitation systems over the past 10 years (from 2014 to 2023). We selected studies that were published in English and focused on joint angle estimation, activity recognition, and exercise assessment. Vision-based approaches were the most common, accounting for 42% of studies. Wearable technology followed at approximately 37%, and pressure-sensing technology appeared in 21% of studies. Identified gaps include a lack of uniformity in reported performance metrics and evaluation methods, a need for cross-subject validation, inadequate testing with patients and older adults, restricted sets of exercises evaluated, and a scarcity of comprehensive datasets on lower limb exercises, especially those involving movements while lying down.

D-STGCNT: A Dense Spatio-Temporal Graph Conv-GRU Network based on transformer for assessment of patient physical rehabilitation

This paper tackles the challenge of automatically assessing physical rehabilitation exercises for patients who perform the exercises without clinician supervision. The objective is to provide a quality score to ensure correct performance and achieve desired results. To achieve this goal, a new graph-based model, the Dense Spatio-Temporal Graph Conv-GRU Network with Transformer, is introduced. This model combines a modified version of STGCN and transformer architectures for efficient handling of spatio-temporal data. The key idea is to consider skeleton data respecting its non-linear structure as a graph and detecting joints playing the main role in each rehabilitation exercise. Dense connections and GRU mechanisms are used to rapidly process large 3D skeleton inputs and effectively model temporal dynamics. The transformer encoder's attention mechanism focuses on relevant parts of the input sequence, making it useful for evaluating rehabilitation exercises. The evaluation of our proposed approach on the KIMORE and UI-PRMD datasets highlighted its potential, surpassing state-of-the-art methods in terms of accuracy and computational time. This resulted in faster and more accurate learning and assessment of rehabilitation exercises. Additionally, our model provides valuable feedback through qualitative illustrations, effectively highlighting the significance of joints in specific exercises.

A Contrastive Learning Network for Performance Metric and Assessment of Physical Rehabilitation Exercises

Human activity analysis in the legal monitoring environment plays an important role in the physical rehabilitation field, as it helps patients with physical injuries improve their postoperative conditions and reduce their medical costs. Recently, several deep learning-based action quality assessment (AQA) frameworks have been proposed to evaluate physical rehabilitation exercises. However, most of them treat this problem as a simple regression task, which requires both the action instance and its score label as input. This approach is limited by the fact that the annotations in this field usually consist of healthy or unhealthy labels rather than quality scores provided by professional physicians. Additionally, most of these methods cannot provide informative feedback on a patient's motion defects, which weakens their practical application. To address these problems, we propose a multi-task contrastive learning framework to learn subtle and critical differences from skeleton sequences to deal with the performance metric and AQA problems of physical rehabilitation exercises. Specifically, we propose a performance metric network that takes triplets of training samples as input for score generation. For the AQA task, the same contrast learning strategy is used, but pairwise training samples are fed into the action quality assessment network for score prediction. Notably, we propose quantifying the deviation of the joint attention matrix between different skeleton sequences and introducing it into the loss function of our learning network. It is proven that considering both score prediction loss and joint attention deviation loss improves physical exercises AQA performance. Furthermore, it helps to obtain informative feedback for patients to improve their motion defects by visualizing the joint attention matrix's difference. The proposed method is verified on the UI-PRMD and KIMORE datasets. Experimental results show that the proposed method achieves state-of-the-art performance.

Artificial Intelligence for skeleton-based physical rehabilitation action evaluation: A systematic review

Performing prescribed physical exercises during home-based rehabilitation programs plays an important role in regaining muscle strength and improving balance for people with different physical disabilities. However, patients attending these programs are not able to assess their action performance in the absence of a medical expert. Recently, vision-based sensors have been deployed in the activity monitoring domain. They are capable of capturing accurate skeleton data. Furthermore, there have been significant advancements in Computer Vision (CV) and Deep Learning (DL) methodologies. These factors have promoted the solutions for designing automatic patient's activity monitoring models. Then, improving such systems' performance to assist patients and physiotherapists has attracted wide interest of the research community. This paper provides a comprehensive and up-to-date literature review on different stages of skeleton data acquisition processes for the aim of physio exercise monitoring. Then, the previously reported Artificial Intelligence (AI) - based methodologies for skeleton data analysis will be reviewed. In particular, feature learning from skeleton data, evaluation, and feedback generation for the purpose of rehabilitation monitoring will be studied. Furthermore, the associated challenges to these processes will be reviewed. Finally, the paper puts forward several suggestions for future research directions in this area.

A database of physical therapy exercises with variability of execution collected by wearable sensors

This document introduces the PHYTMO database, which contains data from physical therapies recorded with inertial sensors, including information from an optical reference system. PHYTMO includes the recording of 30 volunteers, aged between 20 and 70 years old. A total amount of 6 exercises and 3 gait variations were recorded. The volunteers performed two series with a minimum of 8 repetitions in each one. PHYTMO includes magneto-inertial data, together with a highly accurate location and orientation in the 3D space provided by the optical system. The files were stored in CSV format to ensure its usability. The aim of this dataset is the availability of data for two main purposes: the analysis of techniques for the identification and evaluation of exercises using inertial sensors and the validation of inertial sensor-based algorithms for human motion monitoring. Furthermore, the database stores enough data to apply Machine Learning-based algorithms. The participants' age range is large enough to establish age-based metrics for the exercises evaluation or the study of differences in motions between different groups.

Human-Robot Interaction Torque Estimation Methods for a Lower Limb Rehabilitation Robotic System with Uncertainties

Lower limb rehabilitation robot (LLRR) users, to successfully conduct isotonic exercises, require real-time feedback on the torque they exert on the robot to meet the goal of the treatment. Still, direct torque measuring is expensive, and indirect encoder-based estimation strategies, such as inverse dynamics (ID) and Nonlinear Disturbance Observers (NDO), are sensitive to Body Segment Inertial Parameters (BSIPs) uncertainties. We envision a way to minimize such parametric uncertainties. This paper proposes two human–robot interaction torque estimation methods: the Identified ID-based method (IID) and the Identified NDO-based method (INDO). Evaluating in simulation the proposal to apply, in each rehabilitation session, a sequential two-phase method: (1) An initial calibration phase will use an online parameter estimation to reduce sensitivity to BSIPs uncertainties. (2) The torque estimation phase uses the estimated parameters to obtain a better result. We conducted simulations under signal-to-noise ratio (SNR) = 40 dB and 20% BSIPs uncertainties. In addition, we compared the effectiveness with two of the best methods reported in the literature via simulation. Both proposed methods obtained the best Coefficient of Correlation, Mean Absolute Error, and Root Mean Squared Error compared to the benchmarks. Moreover, the IID and INDO fulfilled more than 72.2% and 88.9% of the requirements, respectively. In contrast, both methods reported in the literature only accomplish 27.8% and 33.3% of the requirements when using simulations under noise and BSIPs uncertainties. Therefore, this paper extends two methods reported in the literature and copes with BSIPs uncertainties without using additional sensors.

Real-Time Stylized Humanoid Behavior Control through Interaction and Synchronization

Restricted by the diversity and complexity of human behaviors, simulating a character to achieve human-level perception and motion control is still an active as well as a challenging area. We present a style-based teleoperation framework with the help of human perceptions and analyses to understand the tasks being handled and the unknown environment to control the character. In this framework, the motion optimization and body controller with center-of-mass and root virtual control (CR-VC) method are designed to achieve motion synchronization and style mimicking while maintaining the balance of the character. The motion optimization synthesizes the human high-level style features with the balance strategy to create a feasible, stylized, and stable pose for the character. The CR-VC method including the model-based torque compensation synchronizes the motion rhythm of the human and character. Without any inverse dynamics knowledge or offline preprocessing, our framework is generalized to various scenarios and robust to human behavior changes in real-time. We demonstrate the effectiveness of this framework through the teleoperation experiments with different tasks, motion styles, and operators. This study is a step toward building a human-robot interaction that uses humans to help characters understand and achieve the tasks.

Graph Convolutional Networks for Assessment of Physical Rehabilitation Exercises

Health professionals often prescribe patients to perform specific exercises for rehabilitation of several diseases (e.g., stroke, Parkinson, backpain). When patients perform those exercises in the absence of an expert (e.g., physicians/therapists), they cannot assess the correctness of the performance. Automatic assessment of physical rehabilitation exercises aims to assign a quality score given an RGBD video of the body movement as input. Recent deep learning approaches address this problem by extracting CNN features from co-ordinate grids of skeleton data (body-joints) obtained from videos. However, they could not extract rich spatio-temporal features from variable-length inputs. To address this issue, we investigate Graph Convolutional Networks (GCNs) for this task. We adapt spatio-temporal GCN to predict continuous scores(assessment) instead of discrete class labels. Our model can process variable-length inputs so that users can perform any number of repetitions of the prescribed exercise. Moreover, our novel design also provides self-attention of body-joints, indicating their role in predicting assessment scores. It guides the user to achieve a better score in future trials by matching the same attention weights of expert users. Our model successfully outperforms existing exercise assessment methods on KIMORE and UI-PRMD datasets.

A real-time algorithm for the detection of compensatory movements during reaching

Introduction: Interactive game systems can motivate stroke survivors to engage with their rehabilitation exercises. However, it is crucial that systems are in place to detect if exercises are performed correctly as stroke survivors often perform compensatory movements which can be detrimental to recovery. Very few game systems integrate motion tracking algorithms to monitor performance and detect such movements. This paper describes the development of algorithms which monitor for compensatory movements during upper limb reaching movements in real-time and provides quantitative metrics for health professionals to monitor performance and progress over time.

Methods: A real-time algorithm was developed to analyse reaching motions in real-time through a low-cost depth camera. The algorithm segments cyclical reaching motions into component parts, including compensatory movement, and provides a graphical representation of task performance. Healthy participants (n = 10) performed reaching motions facing the camera. The real-time accuracy of the algorithm was assessed by comparing offline analysis to real-time collection of data.

Results: The algorithm’s ability to segment cyclical reaching motions and detect the component parts in real-time was assessed. Results show that movement types can be detected in real time with accuracy, showing a maximum error of 1.71%.

Conclusions: Using the methods outlined, the real-time detection and quantification of compensatory movements is feasible for integration within home-based, repetitive task practice game systems for people with stroke.

Descripción del movimiento humano basado en el marco de Frenet Serret y datos tipo MOCAP

Clasificar el movimiento humano se ha convertido en una necesidad tecnológica, en donde para definir la posición de un sujeto requiere identificar el recorrido de las extremidades y el tronco del cuerpo, y tener la capacidad de diferenciar esta posición respecto a otros sujetos o movimientos, generándose la necesidad tener datos y algoritmos que faciliten su clasificación. Es así, como en este trabajo, se evalúa la capacidad discriminante de datos de captura de movimiento en rehabilitación física, donde la posición de los sujetos es adquirida con el Kinect de Microsoft y marcadores ópticos, y atributos del movimiento generados con el marco de Frenet Serret, evaluando su capacidad discriminante con los algoritmos máquinas de soporte vectorial, redes neuronales y k vecinos más cercanos. Los resultados presentan porcentajes de acierto del 93.5% en la clasificación con datos obtenidos del Kinect, y un éxito del 100% para los movimientos con marcadores ópticos. Classify human movement has become a technological necessity, where defining the position of a subject requires identifying the trajectory of the limbs and trunk of the body, having the ability to differentiate this position from other subjects or movements, which generates the need to have data and algorithms that help their classification. Therefore, the discriminant capacity of motion capture data in physical rehabilitation is evaluated, where the position of the subjects is acquired with the Microsoft Kinect and optical markers. Attributes of the movement generated with the Frenet Serret framework. Evaluating their discriminant capacity by means of support vector machines, neural networks, and k nearest neighbors algorithms. The obtained results present an accuracy of 93.5% in the classification with data obtained from the Kinect, and success of 100% for movements where the position is defined with optical markers.

Assessing Physical Rehabilitation Exercises using Graph Convolutional Network with Self-supervised regularization

Computer-vision techniques provide a way to conduct low-cost, portable, and real-time evaluations of exercises performed as a part of physical rehabilitation. Recent data-driven methods have explored using deep learning on 3D body-landmark sequences for automatic assessment of physical rehabilitation exercises. However, existing deep learning methods using convolutional neural networks (CNN) fail to utilize the spatial connection information of the human body, which limits the accuracy of these assessments. To overcome these limitations and provide a more accurate method to assess physical rehabilitation exercises, we propose a deep learning framework using a graph convolutional network (GCN) with self-supervised regularization. The experimental results on an existing benchmark dataset validate that the proposed method achieves state-of-the-art performance with lower error than other CNN methods, and the self-supervised learning improves the prediction accuracy.Clinical relevance-This work established a supervised learning method to automatically assess physical rehabilitation exercises in the home environment using computer vision. This low-cost, portable, and real-time evaluation may provide clinicians with a way to provide feedback to patients about their exercise performance without having to provide in-person supervision.

IntelliRehabDS (IRDS)—A Dataset of Physical Rehabilitation Movements

In this article, we present a dataset that comprises different physical rehabilitation movements. The dataset was captured as part of a research project intended to provide automatic feedback on the execution of rehabilitation exercises, even in the absence of a physiotherapist. A Kinect motion sensor camera was used to record gestures. The dataset contains repetitions of nine gestures performed by 29 subjects, out of which 15 were patients and 14 were healthy controls. The data are presented in an easily accessible format, provided as 3D coordinates of 25 body joints along with the corresponding depth map for each frame. Each movement was annotated with the gesture type, the position of the person performing the gesture (sitting or standing) as well as a correctness label. The data are publicly available and were released with to provide a comprehensive dataset that can be used for assessing the performance of different patients while performing simple movements in a rehabilitation setting and for comparing these movements with a control group of healthy individuals.

A review of the evolution of scientific literature on technology-assisted approaches using RGB-D sensors for musculoskeletal health monitoring

The human musculoskeletal (MSK) system (also known as the locomotor system) provides strength and assistance to perform functional tasks and daily life activities. The MSK health monitoring plays a vital role in maintaining the body mobility and quality of life. Manual approaches for musculoskeletal health monitoring are subjective and require a clinician's intervention. The evolution in motion tracking technology enables us to capture the fine details of body movements. The research community has proposed various approaches to help clinicians in diagnosis and monitor treatment sessions. This paper succinctly reviews the evolution of technology-assisted approaches for musculoskeletal health monitoring, using motion capture sensors. To streamline the search through the literature database, the PICOS framework and PRISMA method have been incorporated. The present study reviews methods to transform motion capture data into kinematics variables and factors that affect the tracking performance of RGB-D sensors. Furthermore, widely utilized time-series filters for skeletal data denoising and smoothing for kinematics analysis, stochastic models for movement modeling, rule-based and template-based approaches for rehabilitation exercises assessment, and telerehabilitation sessions for remote health monitoring are explored. This article analyzes skeletal tracking methods by providing advantages and drawbacks of the state of the art rehabilitation sessions assessment, skeletal joint kinematics analysis, and MSK Telerehabilitation approaches. It also discusses the possible future research avenues to improve musculoskeletal disorder diagnosis and treatment monitoring. Our review signifies that RGB-D sensor-based approaches are inexpensive and portable for disorder diagnosis and treatment monitoring. It can also be a viable option for clinicians to provide contactless healthcare access to patients in the current scenario of the COVID-19 pandemic.

VI-Net-View-Invariant Quality of Human Movement Assessment

We propose a view-invariant method towards the assessment of the quality of human movements which does not rely on skeleton data. Our end-to-end convolutional neural network consists of two stages, where at first a view-invariant trajectory descriptor for each body joint is generated from RGB images, and then the collection of trajectories for all joints are processed by an adapted, pre-trained 2D convolutional neural network (CNN) (e.g., VGG-19 or ResNeXt-50) to learn the relationship amongst the different body parts and deliver a score for the movement quality. We release the only publicly-available, multi-view, non-skeleton, non-mocap, rehabilitation movement dataset (QMAR), and provide results for both cross-subject and cross-view scenarios on this dataset. We show that VI-Net achieves average rank correlation of 0.66 on cross-subject and 0.65 on unseen views when trained on only two views. We also evaluate the proposed method on the single-view rehabilitation dataset KIMORE and obtain 0.66 rank correlation against a baseline of 0.62.

LARa: Creating a Dataset for Human Activity Recognition in Logistics Using Semantic Attributes

Optimizations in logistics require recognition and analysis of human activities. The potential of sensor-based human activity recognition (HAR) in logistics is not yet well explored. Despite a significant increase in HAR datasets in the past twenty years, no available dataset depicts activities in logistics. This contribution presents the first freely accessible logistics-dataset. In the ’Innovationlab Hybrid Services in Logistics’ at TU Dortmund University, two picking and one packing scenarios were recreated. Fourteen subjects were recorded individually when performing warehousing activities using Optical marker-based Motion Capture (OMoCap), inertial measurement units (IMUs), and an RGB camera. A total of 758 min of recordings were labeled by 12 annotators in 474 person-h. All the given data have been labeled and categorized into 8 activity classes and 19 binary coarse-semantic descriptions, also called attributes. The dataset is deployed for solving HAR using deep networks.

A review of computational approaches for evaluation of rehabilitation exercises

Recent advances in data analytics and computer-aided diagnostics stimulate the vision of patient-centric precision healthcare, where treatment plans are customized based on the health records and needs of every patient. In physical rehabilitation, the progress in machine learning and the advent of affordable and reliable motion capture sensors have been conducive to the development of approaches for automated assessment of patient performance and progress toward functional recovery. The presented study reviews computational approaches for evaluating patient performance in rehabilitation programs using motion capture systems. Such approaches will play an important role in supplementing traditional rehabilitation assessment performed by trained clinicians, and in assisting patients participating in home-based rehabilitation. The reviewed computational methods for exercise evaluation are grouped into three main categories: discrete movement score, rule-based, and template-based approaches. The review places an emphasis on the application of machine learning methods for movement evaluation in rehabilitation. Related work in the literature on data representation, feature engineering, movement segmentation, and scoring functions is presented. The study also reviews existing sensors for capturing rehabilitation movements and provides an informative listing of pertinent benchmark datasets. The significance of this paper is in being the first to provide a comprehensive review of computational methods for evaluation of patient performance in rehabilitation programs.

A Deep Learning Framework for Assessing Physical Rehabilitation Exercises

Computer-aided assessment of physical rehabilitation entails evaluation of patient performance in completing prescribed rehabilitation exercises, based on processing movement data captured with a sensory system. Despite the essential role of rehabilitation assessment toward improved patient outcomes and reduced healthcare costs, existing approaches lack versatility, robustness, and practical relevance. In this paper, we propose a deep learning-based framework for automated assessment of the quality of physical rehabilitation exercises. The main components of the framework are metrics for quantifying movement performance, scoring functions for mapping the performance metrics into numerical scores of movement quality, and deep neural network models for generating quality scores of input movements via supervised learning. The proposed performance metric is defined based on the log-likelihood of a Gaussian mixture model, and encodes low-dimensional data representation obtained with a deep autoencoder network. The proposed deep spatio-temporal neural network arranges data into temporal pyramids, and exploits the spatial characteristics of human movements by using sub-networks to process joint displacements of individual body parts. The presented framework is validated using a dataset of ten rehabilitation exercises. The significance of this work is that it is the first that implements deep neural networks for assessment of rehabilitation performance.

Assessment of physical rehabilitation movements through dimensionality reduction and statistical modeling

The article proposes a method for evaluation of the consistency of human movements within the context of physical therapy and rehabilitation. Captured movement data in the form of joint angular displacements in a skeletal human model is considered in this work. The proposed approach employs an autoencoder neural network to project the high-dimensional motion trajectories into a low-dimensional manifold. Afterwards, a Gaussian mixture model is used to derive a parametric probabilistic model of the density of the movements. The resulting probabilistic model is employed for evaluation of the consistency of unseen motion sequences based on the likelihood of the data being drawn from the model. The approach is validated on two physical rehabilitation movements.

A Closed-Form Expression of the Instantaneous Rotational Lurch Index to Evaluate Its Numerical Approximation

The lurch index has recently been introduced in applied kinematics as an integral descriptor of the fluency of the motion of a rigid body in space. It may be defined in different versions, according to the component of motion under investigation. In the present paper, we analyze a rotational lurch index, which describes the fluency of the spin component of motion and whose value depends, through involved relations, on the dynamics of three canonical descriptors of the orientation of a rigid body in space. The aim of the present paper is to offer a closed-form expression of the instantaneous component of the rotational lurch, which leads to the namesake index upon integration and normalization. The closed form of the index is, then, used to evaluate its practical calculation, based on numerical approximations on a number of data sets.

A Survey of Vision-Based Human Action Evaluation Methods

The fields of human activity analysis have recently begun to diversify. Many researchers have taken much interest in developing action recognition or action prediction methods. The research on human action evaluation differs by aiming to design computation models and evaluation approaches for automatically assessing the quality of human actions. This line of study has become popular because of its explosively emerging real-world applications, such as physical rehabilitation, assistive living for elderly people, skill training on self-learning platforms, and sports activity scoring. This paper presents a comprehensive survey of approaches and techniques in action evaluation research, including motion detection and preprocessing using skeleton data, handcrafted feature representation methods, and deep learning-based feature representation methods. The benchmark datasets from this research field and some evaluation criteria employed to validate the algorithms' performance are introduced. Finally, the authors present several promising future directions for further studies.

Support Vector Machine-Based Classifier for the Assessment of Finger Movement of Stroke Patients Undergoing Rehabilitation

Purpose

Traditionally, clinical evaluation of motor paralysis following stroke has been of value to physicians and therapists because it allows for immediate pathophysiological assessment without the need for specialized tools. However, current clinical methods do not provide objective quantification of movement; therefore, they are of limited use to physicians and therapists when assessing responses to rehabilitation. The present study aimed to create a support vector machine (SVM)-based classifier to analyze and validate finger kinematics using the leap motion controller. Results were compared with those of 24 stroke patients assessed by therapists.

Methods

A non-linear SVM was used to classify data according to the Brunnstrom recovery stages of finger movements by focusing on peak angle and peak velocity patterns during finger flexion and extension. One thousand bootstrap data values were generated by randomly drawing a series of sample data from the actual normalized kinematics-related data. Bootstrap data values were randomly classified into training (940) and testing (60) datasets. After establishing an SVM classification model by training with the normalized kinematics-related parameters of peak angle and peak velocity, the testing dataset was assigned to predict classification of paralytic movements.

Results

High separation accuracy was obtained (mean 0.863; 95% confidence interval 0.857–0.869; p = 0.006).

Conclusion

This study highlights the ability of artificial intelligence to assist physicians and therapists evaluating hand movement recovery of stroke patients.

The KIMORE Dataset: KInematic Assessment of MOvement and Clinical Scores for Remote Monitoring of Physical REhabilitation

This paper proposes a free dataset, available at the following link,¹named KIMORE, regarding different rehabilitation exercises collected by a RGB-D sensor. Three data inputs including RGB, depth videos, and skeleton joint positions were recorded during five physical exercises, specific for low back pain and accurately selected by physicians. For each exercise, the dataset also provides a set of features, specifically defined by the physicians, and relevant to describe its scope. These features, validated with respect to a stereophotogrammetric system, can be analyzed to compute a score for the subject's performance. The dataset also contains an evaluation of the same performance provided by the clinicians, through a clinical questionnaire. The impact of KIMORE has been analyzed by comparing the output obtained by an example of rule and template-based approaches and the clinical score. The dataset presented is intended to be used as a benchmark for human movement assessment in a rehabilitation scenario in order to test the effectiveness and the reliability of different computational approaches. Unlike other existing datasets, the KIMORE merges a large heterogeneous population of 78 subjects, divided into 2 groups with 44 healthy subjects and 34 with motor dysfunctions. It provides the most clinically-relevant features and the clinical score for each exercise.¹https://univpm-my.sharepoint.com/:f:/g/personal/p008099_staff_univpm_it/EiwbKIzk6N9NoJQx4J8aubIBx0o7tIa1XwclWp1NmRkA-w?e=F3jtBk.

Statistical Modeling of Trivariate Static Systems: Isotonic Models

This paper presents an improved version of a statistical trivariate modeling algorithm introduced in a short Letter by the first author. This paper recalls the fundamental concepts behind the proposed algorithm, evidences its criticalities and illustrates a number of improvements which lead to a functioning modeling algorithm. The present paper also illustrates the features of the improved statistical modeling algorithm through a comprehensive set of numerical experiments performed on four synthetic and five natural datasets. The obtained results confirm that the proposed algorithm is able to model the considered synthetic and the natural datasets faithfully.

Generative Adversarial Networks for Generation and Classification of Physical Rehabilitation Movement Episodes

This article proposes a method for mathematical modeling of human movements related to patient exercise episodes performed during physical therapy sessions by using artificial neural networks. The generative adversarial network structure is adopted, whereby a discriminative and a generative model are trained concurrently in an adversarial manner. Different network architectures are examined, with the discriminative and generative models structured as deep subnetworks of hidden layers comprised of convolutional or recurrent computational units. The models are validated on a data set of human movements recorded with an optical motion tracker. The results demonstrate an ability of the networks for classification of new instances of motions, and for generation of motion examples that resemble the recorded motion sequences.