Smart Annotation of Cyclic Data Using Hierarchical Hidden Markov Models

A Systematic Review of Time Series Classification Techniques Used in Biomedical Applications

Background: Digital clinical measures collected via various digital sensing technologies such as smartphones, smartwatches, wearables, and ingestible and implantable sensors are increasingly used by individuals and clinicians to capture the health outcomes or behavioral and physiological characteristics of individuals. Time series classification (TSC) is very commonly used for modeling digital clinical measures. While deep learning models for TSC are very common and powerful, there exist some fundamental challenges. This review presents the non-deep learning models that are commonly used for time series classification in biomedical applications that can achieve high performance. Objective: We performed a systematic review to characterize the techniques that are used in time series classification of digital clinical measures throughout all the stages of data processing and model building. Methods: We conducted a literature search on PubMed, as well as the Institute of Electrical and Electronics Engineers (IEEE), Web of Science, and SCOPUS databases using a range of search terms to retrieve peer-reviewed articles that report on the academic research about digital clinical measures from a five-year period between June 2016 and June 2021. We identified and categorized the research studies based on the types of classification algorithms and sensor input types. Results: We found 452 papers in total from four different databases: PubMed, IEEE, Web of Science Database, and SCOPUS. After removing duplicates and irrelevant papers, 135 articles remained for detailed review and data extraction. Among these, engineered features using time series methods that were subsequently fed into widely used machine learning classifiers were the most commonly used technique, and also most frequently achieved the best performance metrics (77 out of 135 articles). Statistical modeling (24 out of 135 articles) algorithms were the second most common and also the second-best classification technique. Conclusions: In this review paper, summaries of the time series classification models and interpretation methods for biomedical applications are summarized and categorized. While high time series classification performance has been achieved in digital clinical, physiological, or biomedical measures, no standard benchmark datasets, modeling methods, or reporting methodology exist. There is no single widely used method for time series model development or feature interpretation, however many different methods have proven successful.

A data analytic end-to-end framework for the automated quantification of ergonomic risk factors across multiple tasks using a single wearable sensor

Existing ergonomic risk assessment tools require monitoring of multiple risk factors. To eliminate the direct observation, we investigated the effectiveness of an end-to-end framework that works with the data from a single wearable sensor. The framework is used to identify the performed task as the major contextual risk factor, and then estimate the task duration and number of repetitions as two main indicators of task intensity. For evaluation of the framework, we recruited 37 participants to complete 10 simulated work tasks in a laboratory setting. In testing, we achieved an average accuracy of 92% for task identification, 7.3% error in estimation of task duration, and 7.1% error for counting the number of task repetitions. Moreover, we showed the utility of the framework outputs in two ergonomic tools to estimate the risk of injury. Overall, we indicated the feasibility of using data from wearable sensors to automate the ergonomic risk assessment in workplaces.

The Diverse Gait Dataset: Gait Segmentation Using Inertial Sensors for Pedestrian Localization with Different Genders, Heights and Walking Speeds

Stride length estimation is one of the most crucial aspects of Pedestrian Dead Reckoning (PDR). Due to the measurement noise of inertial sensors, individual variances of pedestrians, and the uncertainty in pedestrians walking, there is a substantial error in the assessment of stride length, which causes the accumulated deviation of Pedestrian Dead Reckoning (PDR). With the help of multi-gait analysis, which decomposes strides in time and space with greater detail and accuracy, a novel and revolutionary stride estimating model or scheme could improve the performance of PDR on different users. This paper presents a diverse stride gait dataset by using inertial sensors that collect foot movement data from people of different genders, heights, and walking speeds. The dataset contains 4690 walking strides data and 19,083 gait labels. Based on the dataset, we propose a threshold-independent stride segmentation algorithm called SDATW and achieve an F-measure of 0.835. We also provide the detailed results of recognizing four gaits under different walking speeds, demonstrating the utility of our dataset for helping train stride segmentation algorithms and gait detection algorithms.

Sensor positioning for a human activity recognition system using a double layer classifier

This paper describes the development of a human gait activity recognition system. A multi-sensor recognition system, which has been developed for this purpose, was reduced to a single sensor-based recognition system. A sensor election method was devised based on the maximum relevance minimum redundancy feature selector to determine the sensor's optimum position regarding activity recognition. The election method proved that the thigh has the highest contribution to recognize walking, stairs and ramp ascending, and descending activities. A recognition algorithm (which depends mainly on features that are classified by random forest, and selected by a combined feature selector using the maximum relevance minimum redundancy and genetic algorithm) has been modified to compensate the degradation that occurs in the prediction accuracy due to the reduction in the number of sensors. The first modification was implementing a double layer classifier in order to discriminate between the interfered activities. The second modification was adding physical features to the features dictionary used. These modifications succeeded to improve the prediction accuracy to allow a single sensor recognition system to behave in the same manner as a multi-sensor activity recognition system.

Possible predictive formulas for quantitative and time-based estimation of muscle strength during motion

[Purpose] To examine the validity of the predictive formulas based on the angle information of the segment center of mass and moments of inertia, and to propose a joint moment estimation method. [Participants and Methods] Twenty nine young healthy adults were divided into two groups: the Creation group (20 adults) was needed to create the prediction formulas, and the Verification group (9 adults) was needed to verify the formulas. By monitoring the Creation group, the angular information from inertial motion sensors and moments of inertia of each limb were used to estimate actual ankle joint moment and knee joint moment. Thereafter, the actual joint moments was derived from the Verification group and compared to the predicted values via Pearson correlations. [Results] Good to excellent correlations were obtained between the actual joint moments of the two groups for most of the motions. [Conclusion] It is suggested that the predictive formulas created from the angle information of the segment center of mass and moments of inertia can be used for an approximate estimation of the lower limb joint moments in the sagittal plane and more clinically useful tools need to be considered in the future.

Hidden Markov Model-Based Smart Annotation for Benchmark Cyclic Activity Recognition Database Using Wearables

Activity monitoring using wearables is becoming ubiquitous, although accurate cycle level analysis, such as step-counting and gait analysis, are limited by a lack of realistic and labeled datasets. The effort required to obtain and annotate such datasets is massive, therefore we propose a smart annotation pipeline which reduces the number of events needing manual adjustment to 14%. For scenarios dominated by walking, this annotation effort is as low as 8%. The pipeline consists of three smart annotation approaches, namely edge detection of the pressure data, local cyclicity estimation, and iteratively trained hierarchical hidden Markov models. Using this pipeline, we have collected and labeled a dataset with over 150,000 labeled cycles, each with 2 phases, from 80 subjects, which we have made publicly available. The dataset consists of 12 different task-driven activities, 10 of which are cyclic. These activities include not only straight and steady-state motions, but also transitions, different ranges of bouts, and changing directions. Each participant wore 5 synchronized inertial measurement units (IMUs) on the wrists, shoes, and in a pocket, as well as pressure insoles and video. We believe that this dataset and smart annotation pipeline are a good basis for creating a benchmark dataset for validation of other semi- and unsupervised algorithms.

Exploring Semi-Supervised Methods for Labeling Support in Multimodal Datasets

Working with multimodal datasets is a challenging task as it requires annotations which often are time consuming and difficult to acquire. This includes in particular video recordings which often need to be watched as a whole before they can be labeled. Additionally, other modalities like acceleration data are often recorded alongside a video. For that purpose, we created an annotation tool that enables to annotate datasets of video and inertial sensor data. In contrast to most existing approaches, we focus on semi-supervised labeling support to infer labels for the whole dataset. This means, after labeling a small set of instances our system is able to provide labeling recommendations. We aim to rely on the acceleration data of a wrist-worn sensor to support the labeling of a video recording. For that purpose, we apply template matching to identify time intervals of certain activities. We test our approach on three datasets, one containing warehouse picking activities, one consisting of activities of daily living and one about meal preparations. Our results show that the presented method is able to give hints to annotators about possible label candidates.

Automatic Annotation of Unlabeled Data from Smartphone-Based Motion and Location Sensors

Automatic data annotation eliminates most of the challenges we faced due to the manual methods of annotating sensor data. It significantly improves users’ experience during sensing activities since their active involvement in the labeling process is reduced. An unsupervised learning technique such as clustering can be used to automatically annotate sensor data. However, the lingering issue with clustering is the validation of generated clusters. In this paper, we adopted the k-means clustering algorithm for annotating unlabeled sensor data for the purpose of detecting sensitive location information of mobile crowd sensing users. Furthermore, we proposed a cluster validation index for the k-means algorithm, which is based on Multiple Pair-Frequency. Thereafter, we trained three classifiers (Support Vector Machine, K-Nearest Neighbor, and Naïve Bayes) using cluster labels generated from the k-means clustering algorithm. The accuracy, precision, and recall of these classifiers were evaluated during the classification of “non-sensitive” and “sensitive” data from motion and location sensors. Very high accuracy scores were recorded from Support Vector Machine and K-Nearest Neighbor classifiers while a fairly high accuracy score was recorded from the Naïve Bayes classifier. With the hybridized machine learning (unsupervised and supervised) technique presented in this paper, unlabeled sensor data was automatically annotated and then classified.