Automated freezing of gait assessment with marker-based motion capture and multi-stage spatial-temporal graph convolutional neural networks

A data-driven approach to detect upper limb functional use during daily life in breast cancer survivors using wrist-worn sensors

To gain insights into the impact of upper limb (UL) dysfunctions after breast cancer treatment, this study aimed to develop a temporal convolutional neural network (TCN) to detect functional daily UL use in breast cancer survivors using data from a wrist-worn accelerometer. A pre-existing dataset of 10 breast cancer survivors was used that contained raw 3-axis acceleration data and simultaneously recorded video data, captured during four daily life activities. The input of our TCN consists of a 3-axis acceleration sequence with a receptive field of 243 samples. The 4 ResNet TCN blocks perform dilated temporal convolutions with a kernel of size 3 and a dilation rate that increases by a factor of 3 after each iteration. Outcomes of interest were functional UL use (minutes) and percentage UL use. We found strong agreement between the video and predicted data for functional UL use (ICC = 0.975) and moderately strong agreement for %UL use (ICC = 0.794). The TCN model overestimated the functional UL use by 0.71 min and 3.06%. Model performance showed good accuracy, f1, and AUPRC scores (0.875, 0.909, 0.954, respectively). In conclusion, using wrist-worn accelerometer data, the TCN model effectively identified functional UL use in daily life among breast cancer survivors.

A study of animal action segmentation algorithms across supervised, unsupervised, and semi-supervised learning paradigms

Action segmentation of behavioral videos is the process of labeling each frame as belonging to one or more discrete classes, and is a crucial component of many studies that investigate animal behavior. A wide range of algorithms exist to automatically parse discrete animal behavior, encompassing supervised, unsupervised, and semi-supervised learning paradigms. These algorithms - which include tree-based models, deep neural networks, and graphical models - differ widely in their structure and assumptions on the data. Using four datasets spanning multiple species - fly, mouse, and human - we systematically study how the outputs of these various algorithms align with manually annotated behaviors of interest. Along the way, we introduce a semi-supervised action segmentation model that bridges the gap between supervised deep neural networks and unsupervised graphical models. We find that fully supervised temporal convolutional networks with the addition of temporal information in the observations perform the best on our supervised metrics across all datasets.

Video-Based Detection of Freezing of Gait in Daily Clinical Practice in Patients With Parkinsonism

Freezing of gait (FoG) is a prevalent symptom among individuals with Parkinson's disease and related disorders. FoG detection from videos has been developed recently; however, the process requires using videos filmed within a controlled environment. We attempted to establish an automatic FoG detection method from videos taken in uncontrolled environments such as in daily clinical practices. Motion features of 16 patients were extracted from timed-up-and-go test in 109 video data points, through object tracking and three-dimension pose estimation. These motion features were utilized to form the FoG detection model, which combined rule-based and machine learning-based models. The rule-based model distinguished the frames in which the patient was walking from those when the patient has stopped, using the pelvic position coordinates; the machine learning-based model distinguished between FoG and stop using a combined one-dimensional convolutional neural network and long short-term memory (1dCNN-LSTM). The model achieved a high intraclass correlation coefficient of 0.75-0.94 with a manually-annotated duration of FoG and %FoG. This method is novel as it combines object tracking, 3D pose estimation, and expert-guided feature selection in the preprocessing and modeling phases, enabling FoG detection even from videos captured in uncontrolled environments.

Freezing of gait assessment with inertial measurement units and deep learning: effect of tasks, medication states, and stops

BACKGROUND

Freezing of gait (FOG) is an episodic and highly disabling symptom of Parkinson's Disease (PD). Traditionally, FOG assessment relies on time-consuming visual inspection of camera footage. Therefore, previous studies have proposed portable and automated solutions to annotate FOG. However, automated FOG assessment is challenging due to gait variability caused by medication effects and varying FOG-provoking tasks. Moreover, whether automated approaches can differentiate FOG from typical everyday movements, such as volitional stops, remains to be determined. To address these questions, we evaluated an automated FOG assessment model with deep learning (DL) based on inertial measurement units (IMUs). We assessed its performance trained on all standardized FOG-provoking tasks and medication states, as well as on specific tasks and medication states. Furthermore, we examined the effect of adding stopping periods on FOG detection performance.

METHODS

Twelve PD patients with self-reported FOG (mean age 69.33 ± 6.02 years) completed a FOG-provoking protocol, including timed-up-and-go and 360-degree turning-in-place tasks in On/Off dopaminergic medication states with/without volitional stopping. IMUs were attached to the pelvis and both sides of the tibia and talus. A temporal convolutional network (TCN) was used to detect FOG episodes. FOG severity was quantified by the percentage of time frozen (%TF) and the number of freezing episodes (#FOG). The agreement between the model-generated outcomes and the gold standard experts' video annotation was assessed by the intra-class correlation coefficient (ICC).

RESULTS

For FOG assessment in trials without stopping, the agreement of our model was strong (ICC (%TF) = 0.92 [0.68, 0.98]; ICC(#FOG) = 0.95 [0.72, 0.99]). Models trained on a specific FOG-provoking task could not generalize to unseen tasks, while models trained on a specific medication state could generalize to unseen states. For assessment in trials with stopping, the agreement of our model was moderately strong (ICC (%TF) = 0.95 [0.73, 0.99]; ICC (#FOG) = 0.79 [0.46, 0.94]), but only when stopping was included in the training data.

CONCLUSION

A TCN trained on IMU signals allows valid FOG assessment in trials with/without stops containing different medication states and FOG-provoking tasks. These results are encouraging and enable future work investigating automated FOG assessment during everyday life.

Eat-Radar: Continuous Fine-Grained Intake Gesture Detection Using FMCW Radar and 3D Temporal Convolutional Network With Attention

Unhealthy dietary habits are considered as the primary cause of various chronic diseases, including obesity and diabetes. The automatic food intake monitoring system has the potential to improve the quality of life (QoL) of people with diet-related diseases through dietary assessment. In this work, we propose a novel contactless radar-based approach for food intake monitoring. Specifically, a Frequency Modulated Continuous Wave (FMCW) radar sensor is employed to recognize fine-grained eating and drinking gestures. The fine-grained eating/drinking gesture contains a series of movements from raising the hand to the mouth until putting away the hand from the mouth. A 3D temporal convolutional network with self-attention (3D-TCN-Att) is developed to detect and segment eating and drinking gestures in meal sessions by processing the Range-Doppler Cube (RD Cube). Unlike previous radar-based research, this work collects data in continuous meal sessions (more realistic scenarios). We create a public dataset comprising 70 meal sessions (4,132 eating gestures and 893 drinking gestures) from 70 participants with a total duration of 1,155 minutes. Four eating styles (fork & knife, chopsticks, spoon, hand) are included in this dataset. To validate the performance of the proposed approach, seven-fold cross-validation method is applied. The 3D-TCN-Att model achieves a segmental F1-score of 0.896 and 0.868 for eating and drinking gestures, respectively. The results of the proposed approach indicate the feasibility of using radar for fine-grained eating and drinking gesture detection and segmentation in meal sessions.

Overview on wearable sensors for the management of Parkinson's disease

Parkinson's disease (PD) is affecting about 1.2 million patients in Europe with a prevalence that is expected to have an exponential increment, in the next decades. This epidemiological evolution will be challenged by the low number of neurologists able to deliver expert care for PD. As PD is better recognized, there is an increasing demand from patients for rigorous control of their symptoms and for therapeutic education. In addition, the highly variable nature of symtoms between patients and the fluctuations within the same patient requires innovative tools to help doctors and patients monitor the disease in their usual living environment and adapt treatment in a more relevant way. Nowadays, there are various body-worn sensors (BWS) proposed to monitor parkinsonian clinical features, such as motor fluctuations, dyskinesia, tremor, bradykinesia, freezing of gait (FoG) or gait disturbances. BWS have been used as add-on tool for patients' management or research purpose. Here, we propose a practical anthology, summarizing the characteristics of the most used BWS for PD patients in Europe, focusing on their role as tools to improve treatment management. Consideration regarding the use of technology to monitor non-motor features is also included. BWS obviously offer new opportunities for improving management strategy in PD but their precise scope of use in daily routine care should be clarified.

Classifying Tremor Dominant and Postural Instability and Gait Difficulty Subtypes of Parkinson's Disease from Full-Body Kinematics

Characterizing motor subtypes of Parkinson's disease (PD) is an important aspect of clinical care that is useful for prognosis and medical management. Although all PD cases involve the loss of dopaminergic neurons in the brain, individual cases may present with different combinations of motor signs, which may indicate differences in underlying pathology and potential response to treatment. However, the conventional method for distinguishing PD motor subtypes involves resource-intensive physical examination by a movement disorders specialist. Moreover, the standardized rating scales for PD rely on subjective observation, which requires specialized training and unavoidable inter-rater variability. In this work, we propose a system that uses machine learning models to automatically and objectively identify some PD motor subtypes, specifically Tremor-Dominant (TD) and Postural Instability and Gait Difficulty (PIGD), from 3D kinematic data recorded during walking tasks for patients with PD (MDS-UPDRS-III Score, 34.7 ± 10.5, average disease duration 7.5 ± 4.5 years). This study demonstrates a machine learning model utilizing kinematic data that identifies PD motor subtypes with a 79.6% F1 score (N = 55 patients with parkinsonism). This significantly outperformed a comparison model using classification based on gait features (19.8% F1 score). Variants of our model trained to individual patients achieved a 95.4% F1 score. This analysis revealed that both temporal, spectral, and statistical features from lower body movements are helpful in distinguishing motor subtypes. Automatically assessing PD motor subtypes simply from walking may reduce the time and resources required from specialists, thereby improving patient care for PD treatments. Furthermore, this system can provide objective assessments to track the changes in PD motor subtypes over time to implement and modify appropriate treatment plans for individual patients as needed.

Multi-Level Adversarial Spatio-Temporal Learning for Footstep Pressure Based FoG Detection

Freezing of gait (FoG) is one of the most common symptoms of Parkinson's disease, which is a neurodegenerative disorder of the central nervous system impacting millions of people around the world. To address the pressing need to improve the quality of treatment for FoG, devising a computer-aided detection and quantification tool for FoG has been increasingly important. As a non-invasive technique for collecting motion patterns, the footstep pressure sequences obtained from pressure sensitive gait mats provide a great opportunity for evaluating FoG in the clinic and potentially in the home environment. In this study, FoG detection is formulated as a sequential modelling task and a novel deep learning architecture, namely Adversarial Spatio-temporal Network (ASTN), is proposed to learn FoG patterns across multiple levels. ASTN introduces a novel adversarial training scheme with a multi-level subject discriminator to obtain subject-independent FoG representations, which helps to reduce the over-fitting risk due to the high inter-subject variance. As a result, robust FoG detection can be achieved for unseen subjects. The proposed scheme also sheds light on improving subject-level clinical studies from other scenarios as it can be integrated with many existing deep architectures. To the best of our knowledge, this is one of the first studies of footstep pressure-based FoG detection and the approach of utilizing ASTN is the first deep neural network architecture in pursuit of subject-independent representations. In our experiments on 393 trials collected from 21 subjects, the proposed ASTN achieved an AUC 0.85, clearly outperforming conventional learning methods.

FoG-Finder: Real-time Freezing of Gait Detection and Treatment

Freezing of gait is a serious symptom of Parkinson's disease that increases the risk of injury through falling, and reduces quality of life. Current clinical freezing of gait treatments fail to adequately address the fall risk posed by freezing of gait symptoms, and current real-time treatment systems have high false positive rates. To address this problem, we designed a closed-loop, non-intrusive, and real-time freezing of gait detection and treatment system, FoG-Finder, that automatically detects and treats freezing of gait. To evaluate FoG-Finder, we first collected 716 freezing of gait events from 11 patients. We then compared FoG-Finder against other real-time systems with our dataset. Our system was able to achieve a 13.4% higher F1 score and a 10.7% higher overall accuracy while achieving a reduction of 85.8% in the false positive treatment rate compared with other validated real-time freezing of gait detection and treatment systems. Additionally, FoG-Finder achieved an average treatment latency of 427ms and 615ms for subject-dependent and leave-one-subject-out settings, respectively, making it a viable system to treat freezing of gait in the real-world.