Objective measurement of limb bradykinesia using a marker-less tracking algorithm with 2D-video in PD patients

Automatic two-dimensional & three-dimensional video analysis with deep learning for movement disorders: A systematic review

The advent of computer vision technology and increased usage of video cameras in clinical settings have facilitated advancements in movement disorder analysis. This review investigated these advancements in terms of providing practical, low-cost solutions for the diagnosis and analysis of movement disorders, such as Parkinson's disease, ataxia, dyskinesia, and Tourette syndrome. Traditional diagnostic methods for movement disorders are typically reliant on the subjective assessment of motor symptoms, which poses inherent challenges. Furthermore, early symptoms are often overlooked, and overlapping symptoms across diseases can complicate early diagnosis. Consequently, deep learning has been used for the objective video-based analysis of movement disorders. This study systematically reviewed the latest advancements in automatic two-dimensional & three-dimensional video analysis using deep learning for movement disorders. We comprehensively analyzed the literature published until September 2023 by searching the Web of Science, PubMed, Scopus, and Embase databases. We identified 68 relevant studies and extracted information on their objectives, datasets, modalities, and methodologies. The study aimed to identify, catalogue, and present the most significant advancements, offering a consolidated knowledge base on the role of video analysis and deep learning in movement disorder analysis. First, the objectives, including specific PD symptom quantification, ataxia assessment, cerebral palsy assessment, gait disorder analysis, tremor assessment, tic detection (in the context of Tourette syndrome), dystonia assessment, and abnormal movement recognition were discussed. Thereafter, the datasets used in the study were examined. Subsequently, video modalities and deep learning methodologies related to the topic were investigated. Finally, the challenges and opportunities in terms of datasets, interpretability, evaluation methods, and home/remote monitoring were discussed.

Feasibility of differentiating gait in Parkinson's disease and spinocerebellar degeneration using a pose estimation algorithm in two-dimensional video

BACKGROUND

Although pose estimation algorithms have been used to analyze videos of patients with Parkinson's disease (PD) to assess symptoms, their feasibility for differentiating PD from other neurological disorders that cause gait disturbances has not been evaluated yet. We aimed to determine whether it was possible to differentiate between PD and spinocerebellar degeneration (SCD) by analyzing video recordings of patient gait using a pose estimation algorithm.

METHODS

We videotaped 82 patients with PD and 61 patients with SCD performing the timed up-and-go test. A pose estimation algorithm was used to extract the coordinates of 25 key points of the participants from these videos. A transformer-based deep neural network (DNN) model was trained to predict PD or SCD using the extracted coordinate data. We employed a leave-one-participant-out cross-validation method to evaluate the predictive performance of the trained model using accuracy, sensitivity, and specificity. As there were significant differences in age, weight, and body mass index between the PD and SCD groups, propensity score matching was used to perform the same experiment in a population that did not differ in these clinical characteristics.

RESULTS

The accuracy, sensitivity, and specificity of the trained model were 0.86, 0.94, and 0.75 for all participants and 0.83, 0.88, and 0.78 for the participants extracted by propensity score matching.

CONCLUSION

The differentiation of PD and SCD using key point coordinates extracted from gait videos and the DNN model was feasible and could be used as a collaborative tool in clinical practice and telemedicine.

Deep learning of Parkinson's movement from video, without human-defined measures

BACKGROUND

The core clinical sign of Parkinson's disease (PD) is bradykinesia, for which a standard test is finger tapping: the clinician observes a person repetitively tap finger and thumb together. That requires an expert eye, a scarce resource, and even experts show variability and inaccuracy. Existing applications of technology to finger tapping reduce the tapping signal to one-dimensional measures, with researcher-defined features derived from those measures.

OBJECTIVES

(1) To apply a deep learning neural network directly to video of finger tapping, without human-defined measures/features, and determine classification accuracy for idiopathic PD versus controls. (2) To visualise the features learned by the model.

METHODS

152 smartphone videos of 10s finger tapping were collected from 40 people with PD and 37 controls. We down-sampled pixel dimensions and videos were split into 1 s clips. A 3D convolutional neural network was trained on these clips.

RESULTS

For discriminating PD from controls, our model showed training accuracy 0.91, and test accuracy 0.69, with test precision 0.73, test recall 0.76 and test AUROC 0.76. We also report class activation maps for the five most predictive features. These show the spatial and temporal sections of video upon which the network focuses attention to make a prediction, including an apparent dropping thumb movement distinct for the PD group.

CONCLUSIONS

A deep learning neural network can be applied directly to standard video of finger tapping, to distinguish PD from controls, without a requirement to extract a one-dimensional signal from the video, or pre-define tapping features.

Deep Learning for hand tracking in Parkinson's Disease video-based assessment: Current and future perspectives

BACKGROUND

Parkinson's Disease (PD) demands early diagnosis and frequent assessment of symptoms. In particular, analysing hand movements is pivotal to understand disease progression. Advancements in hand tracking using Deep Learning (DL) allow for the automatic and objective disease evaluation from video recordings of standardised motor tasks, which are the foundation of neurological examinations. In view of this scenario, this narrative review aims to describe the state of the art and the future perspective of DL frameworks for hand tracking in video-based PD assessment.

METHODS

A rigorous search of PubMed, Web of Science, IEEE Explorer, and Scopus until October 2023 using primary keywords such as parkinson, hand tracking, and deep learning was performed to select eligible by focusing on video-based PD assessment through DL-driven hand tracking frameworks RESULTS:: After accurate screening, 23 publications met the selection criteria. These studies used various solutions, from well-established pose estimation frameworks, like OpenPose and MediaPipe, to custom deep architectures designed to accurately track hand and finger movements and extract relevant disease features. Estimated hand tracking data were then used to differentiate PD patients from healthy individuals, characterise symptoms such as tremors and bradykinesia, or regress the Movement Disorder Society-Unified Parkinson's Disease Rating Scale (MDS-UPDRS) by automatically assessing clinical tasks such as finger tapping, hand movements, and pronation-supination.

CONCLUSIONS

DL-driven hand tracking holds promise for PD assessment, offering precise, objective measurements for early diagnosis and monitoring, especially in a telemedicine scenario. However, to ensure clinical acceptance, standardisation and validation are crucial. Future research should prioritise large open datasets, rigorous validation on patients, and the investigation of new frontiers such as tracking hand-hand and hand-object interactions for daily-life tasks assessment.

MocapMe: DeepLabCut-Enhanced Neural Network for Enhanced Markerless Stability in Sit-to-Stand Motion Capture

This study examined the efficacy of an optimized DeepLabCut (DLC) model in motion capture, with a particular focus on the sit-to-stand (STS) movement, which is crucial for assessing the functional capacity in elderly and postoperative patients. This research uniquely compared the performance of this optimized DLC model, which was trained using 'filtered' estimates from the widely used OpenPose (OP) model, thereby emphasizing computational effectiveness, motion-tracking precision, and enhanced stability in data capture. Utilizing a combination of smartphone-captured videos and specifically curated datasets, our methodological approach included data preparation, keypoint annotation, and extensive model training, with an emphasis on the flow of the optimized model. The findings demonstrate the superiority of the optimized DLC model in various aspects. It exhibited not only higher computational efficiency, with reduced processing times, but also greater precision and consistency in motion tracking thanks to the stability brought about by the meticulous selection of the OP data. This precision is vital for developing accurate biomechanical models for clinical interventions. Moreover, this study revealed that the optimized DLC maintained higher average confidence levels across datasets, indicating more reliable and accurate detection capabilities compared with standalone OP. The clinical relevance of these findings is profound. The optimized DLC model's efficiency and enhanced point estimation stability make it an invaluable tool in rehabilitation monitoring and patient assessments, potentially streamlining clinical workflows. This study suggests future research directions, including integrating the optimized DLC model with virtual reality environments for enhanced patient engagement and leveraging its improved data quality for predictive analytics in healthcare. Overall, the optimized DLC model emerged as a transformative tool for biomechanical analysis and physical rehabilitation, promising to enhance the quality of patient care and healthcare delivery efficiency.

Sensor-Based Quantification of MDS-UPDRS III Subitems in Parkinson's Disease Using Machine Learning

Wearable sensors could be beneficial for the continuous quantification of upper limb motor symptoms in people with Parkinson's disease (PD). This work evaluates the use of two inertial measurement units combined with supervised machine learning models to classify and predict a subset of MDS-UPDRS III subitems in PD. We attached the two compact wearable sensors on the dorsal part of each hand of 33 people with PD and 12 controls. Each participant performed six clinical movement tasks in parallel with an assessment of the MDS-UPDRS III. Random forest (RF) models were trained on the sensor data and motor scores. An overall accuracy of 94% was achieved in classifying the movement tasks. When employed for classifying the motor scores, the averaged area under the receiver operating characteristic values ranged from 68% to 92%. Motor scores were additionally predicted using an RF regression model. In a comparative analysis, trained support vector machine models outperformed the RF models for specific tasks. Furthermore, our results surpass the literature in certain cases. The methods developed in this work serve as a base for future studies, where home-based assessments of pharmacological effects on motor function could complement regular clinical assessments.

Gait video-based prediction of unified Parkinson's disease rating scale score: a retrospective study

BACKGROUND

The diagnosis of Parkinson's disease (PD) and evaluation of its symptoms require in-person clinical examination. Remote evaluation of PD symptoms is desirable, especially during a pandemic such as the coronavirus disease 2019 pandemic. One potential method to remotely evaluate PD motor impairments is video-based analysis. In this study, we aimed to assess the feasibility of predicting the Unified Parkinson's Disease Rating Scale (UPDRS) score from gait videos using a convolutional neural network (CNN) model.

METHODS

We retrospectively obtained 737 consecutive gait videos of 74 patients with PD and their corresponding neurologist-rated UPDRS scores. We utilized a CNN model for predicting the total UPDRS part III score and four subscores of axial symptoms (items 27, 28, 29, and 30), bradykinesia (items 23, 24, 25, 26, and 31), rigidity (item 22) and tremor (items 20 and 21). We trained the model on 80% of the gait videos and used 10% of the videos as a validation dataset. We evaluated the predictive performance of the trained model by comparing the model-predicted score with the neurologist-rated score for the remaining 10% of videos (test dataset). We calculated the coefficient of determination (R2) between those scores to evaluate the model's goodness of fit.

RESULTS

In the test dataset, the R2 values between the model-predicted and neurologist-rated values for the total UPDRS part III score and subscores of axial symptoms, bradykinesia, rigidity, and tremor were 0.59, 0.77, 0.56, 0.46, and 0.0, respectively. The performance was relatively low for videos from patients with severe symptoms.

CONCLUSIONS

Despite the low predictive performance of the model for the total UPDRS part III score, it demonstrated relatively high performance in predicting subscores of axial symptoms. The model approximately predicted the total UPDRS part III scores of patients with moderate symptoms, but the performance was low for patients with severe symptoms owing to limited data. A larger dataset is needed to improve the model's performance in clinical settings.

A First Methodological Development and Validation of ReTap: An Open-Source UPDRS Finger Tapping Assessment Tool Based on Accelerometer-Data

Bradykinesia is a cardinal hallmark of Parkinson's disease (PD). Improvement in bradykinesia is an important signature of effective treatment. Finger tapping is commonly used to index bradykinesia, albeit these approaches largely rely on subjective clinical evaluations. Moreover, recently developed automated bradykinesia scoring tools are proprietary and are not suitable for capturing intraday symptom fluctuation. We assessed finger tapping (i.e., Unified Parkinson's Disease Rating Scale (UPDRS) item 3.4) in 37 people with Parkinson's disease (PwP) during routine treatment follow ups and analyzed their 350 sessions of 10-s tapping using index finger accelerometry. Herein, we developed and validated ReTap, an open-source tool for the automated prediction of finger tapping scores. ReTap successfully detected tapping blocks in over 94% of cases and extracted clinically relevant kinematic features per tap. Importantly, based on the kinematic features, ReTap predicted expert-rated UPDRS scores significantly better than chance in a hold out validation sample (n = 102). Moreover, ReTap-predicted UPDRS scores correlated positively with expert ratings in over 70% of the individual subjects in the holdout dataset. ReTap has the potential to provide accessible and reliable finger tapping scores, either in the clinic or at home, and may contribute to open-source and detailed analyses of bradykinesia.

Deep Clinical Phenotyping of Parkinson's Disease: Towards a New Era of Research and Clinical Care

Despite recent advances in technology, clinical phenotyping of Parkinson's disease (PD) has remained relatively limited as current assessments are mainly based on empirical observation and subjective categorical judgment at the clinic. A lack of comprehensive, objective, and quantifiable clinical phenotyping data has hindered our capacity to diagnose, assess patients' conditions, discover pathogenesis, identify preclinical stages and clinical subtypes, and evaluate new therapies. Therefore, deep clinical phenotyping of PD patients is a necessary step towards understanding PD pathology and improving clinical care. In this review, we present a growing community consensus and perspective on how to clinically phenotype this disease, that is, to phenotype the entire course of disease progression by integrating capacity, performance, and perception approaches with state-of-the-art technology. We also explore the most studied aspects of PD deep clinical phenotypes, namely, bradykinesia, tremor, dyskinesia and motor fluctuation, gait impairment, speech impairment, and non-motor phenotypes.

Vision-based Finger Tapping Test in Patients with Parkinson's Disease via Spatial-temporal 3D Hand Pose Estimation

Finger tapping test is crucial for diagnosing Parkinson's Disease (PD), but manual visual evaluations can result in score discrepancy due to clinicians' subjectivity. Moreover, applying wearable sensors requires making physical contact and may hinder PD patient's raw movement patterns. Accordingly, a novel computer-vision approach is proposed using depth camera and spatial-temporal 3D hand pose estimation to capture and evaluate PD patients' 3D hand movement. Within this approach, a temporal encoding module is leveraged to extend A2J's deep learning framework to counter the pose jittering problem, and a pose refinement process is utilized to alleviate dependency on massive data. Additionally, the first vision-based 3D PD hand dataset of 112 hand samples from 48 PD patients and 11 control subjects is constructed, fully annotated by qualified physicians under clinical settings. Testing on this real-world data, this new model achieves 81.2% classification accuracy, even surpassing that of individual clinicians in comparison, fully demonstrating this proposition's effectiveness. The demo video can be ac-cessed at https://github.com/ZhilinGuo/ST-A2J.

Quantitative Gait Analysis Using a Pose-Estimation Algorithm with a Single 2D-Video of Parkinson's Disease Patients

BACKGROUND

Clinician-based rating scales or questionnaires for gait in Parkinson's disease (PD) are subjective and sensor-based analysis is limited in accessibility.

OBJECTIVE

To develop an easily accessible and objective tool to evaluate gait in PD patients, we analyzed gait from a single 2-dimensional (2D) video.

METHODS

We prospectively recorded 2D videos of PD patients (n = 16) and healthy controls (n = 15) performing the timed up and go test (TUG). The gait was simultaneously evaluated with a pressure-sensor (GAITRite). We estimated the 3D position of toes and heels with a deep-learning based pose-estimation algorithm and calculated gait parameters including step length, step length variability, gait velocity and step cadence which was validated with the result from the GAITRite. We further calculated the time and steps required for turning. Then, we applied the algorithm to previously recorded and archived videos of PD patients (n = 32) performing the TUG.

RESULTS

From the validation experiment, gait parameters derived from video tracking were in excellent agreement with the parameters obtained with the GAITRite. (Intraclass correlation coefficient > 0.9). From the analysis with the archived videos, step length, gait velocity, number of steps, and the time required for turning were significantly correlated (Absolute R > 0.4, p < 0.005) with the Freezing of gait questionnaire, Unified PD Rating scale part III total score, HY stage and postural instability. Furthermore, the video-based tracking objectively measured significant improvement of step length, gait velocity, steps and the time required for turning with antiparkinsonian medication.

CONCLUSION

2D video-based tracking could objectively evaluate gait in PD patients.

Developing and assessing a new web-based tapping test for measuring distal movement in Parkinson's disease: a Distal Finger Tapping test

Disability in Parkinson's disease (PD) is measured by standardised scales including the MDS-UPDRS, which are subject to high inter and intra-rater variability and fail to capture subtle motor impairment. The BRadykinesia Akinesia INcoordination (BRAIN) test is a validated keyboard tapping test, evaluating proximal upper-limb motor impairment. Here, a new Distal Finger Tapping (DFT) test was developed to assess distal upper-limb function. Kinetic parameters of the test include kinesia score (KS20, key taps over 20 s), akinesia time (AT20, mean dwell-time on each key) and incoordination score (IS20, variance of travelling time between key taps). To develop and evaluate a new keyboard-tapping test for objective and remote distal motor function in PD patients. The DFT and BRAIN tests were assessed in 55 PD patients and 65 controls. Test scores were compared between groups and correlated with the MDS-UPDRS-III finger tapping sub-scores. Nine additional PD patients were recruited for monitoring motor fluctuations. All three parameters discriminated effectively between PD patients and controls, with KS20 performing best, yielding 79% sensitivity for 85% specificity; area under the receiver operating characteristic curve (AUC) = 0.90. A combination of DFT and BRAIN tests improved discrimination (AUC = 0.95). Among three parameters, KS20 showed a moderate correlation with the MDS-UPDRS finger-tapping sub-score (Pearson's r = - 0.40, p = 0.002). Further, the DFT test detected subtle changes in motor fluctuation states which were not reflected clearly by the MDS-UPDRS-III finger tapping sub-scores. The DFT test is an online tool for assessing distal movements in PD, with future scope for longitudinal monitoring of motor complications.

Video-based quantification of human movement frequency using pose estimation: A pilot study

Assessment of repetitive movements (e.g., finger tapping) is a hallmark of motor examinations in several neurologic populations. These assessments are traditionally performed by a human rater via visual inspection; however, advances in computer vision offer potential for remote, quantitative assessment using simple video recordings. Here, we evaluated a pose estimation approach for measurement of human movement frequency from smartphone videos. Ten healthy young participants provided videos of themselves performing five repetitive movement tasks (finger tapping, hand open/close, hand pronation/supination, toe tapping, leg agility) at four target frequencies (1–4 Hz). We assessed the ability of a workflow that incorporated OpenPose (a freely available whole-body pose estimation algorithm) to estimate movement frequencies by comparing against manual frame-by-frame (i.e., ground-truth) measurements for all tasks and target frequencies using repeated measures ANOVA, Pearson’s correlations, and intraclass correlations. Our workflow produced largely accurate estimates of movement frequencies; only the hand open/close task showed a significant difference in the frequencies estimated by pose estimation and manual measurement (while statistically significant, these differences were small in magnitude). All other tasks and frequencies showed no significant differences between pose estimation and manual measurement. Pose estimation-based detections of individual events (e.g., finger taps, hand closures) showed strong correlations (all r>0.99) with manual detections for all tasks and frequencies. In summary, our pose estimation-based workflow accurately tracked repetitive movements in healthy adults across a range of tasks and movement frequencies. Future work will test this approach as a fast, quantitative, video-based approach to assessment of repetitive movements in clinical populations.

Slow Motion Analysis of Repetitive Tapping (SMART) Test: Measuring Bradykinesia in Recently Diagnosed Parkinson's Disease and Idiopathic Anosmia

BACKGROUND

Bradykinesia is the defining motor feature of Parkinson's disease (PD). There are limitations to its assessment using standard clinical rating scales, especially in the early stages of PD when a floor effect may be observed.

OBJECTIVE

To develop a quantitative method to track repetitive tapping movements and to compare people in the early stages of PD, healthy controls, and individuals with idiopathic anosmia.

METHODS

This was a cross-sectional study of 99 participants (early-stage PD = 26, controls = 64, idiopathic anosmia = 9). For each participant, repetitive finger tapping was recorded over 20 seconds using a smartphone at 240 frames per second. From each video, amplitude between fingers, frequency (number of taps per second), and velocity (distance travelled per second) was extracted. Clinical assessment was based on the motor section of the MDS-UPDRS.

RESULTS

People in the early stage of PD performed the task with slower velocity (p <  0.001) and with greater frequency slope than controls (p = 0.003). The combination of reduced velocity and greater frequency slope obtained the best accuracy to separate early-stage PD from controls based on metric thresholds alone (AUC = 0.88). Individuals with anosmia exhibited slower velocity (p = 0.001) and smaller amplitude (p <  0.001) compared with controls.

CONCLUSION

We present a simple, proof-of-concept method to detect early motor dysfunction in PD. Mean tap velocity appeared to be the best parameter to differentiate patients with PD from controls. Patients with anosmia also showed detectable differences in motor performance compared with controls which may suggest that some were in the prodromal phase of PD.