Video-Based Hand Movement Analysis of Parkinson Patients before and after Medication Using High-Frame-Rate Videos and MediaPipe

Automatic two-dimensional & three-dimensional video analysis with deep learning for movement disorders: A systematic review

The advent of computer vision technology and increased usage of video cameras in clinical settings have facilitated advancements in movement disorder analysis. This review investigated these advancements in terms of providing practical, low-cost solutions for the diagnosis and analysis of movement disorders, such as Parkinson's disease, ataxia, dyskinesia, and Tourette syndrome. Traditional diagnostic methods for movement disorders are typically reliant on the subjective assessment of motor symptoms, which poses inherent challenges. Furthermore, early symptoms are often overlooked, and overlapping symptoms across diseases can complicate early diagnosis. Consequently, deep learning has been used for the objective video-based analysis of movement disorders. This study systematically reviewed the latest advancements in automatic two-dimensional & three-dimensional video analysis using deep learning for movement disorders. We comprehensively analyzed the literature published until September 2023 by searching the Web of Science, PubMed, Scopus, and Embase databases. We identified 68 relevant studies and extracted information on their objectives, datasets, modalities, and methodologies. The study aimed to identify, catalogue, and present the most significant advancements, offering a consolidated knowledge base on the role of video analysis and deep learning in movement disorder analysis. First, the objectives, including specific PD symptom quantification, ataxia assessment, cerebral palsy assessment, gait disorder analysis, tremor assessment, tic detection (in the context of Tourette syndrome), dystonia assessment, and abnormal movement recognition were discussed. Thereafter, the datasets used in the study were examined. Subsequently, video modalities and deep learning methodologies related to the topic were investigated. Finally, the challenges and opportunities in terms of datasets, interpretability, evaluation methods, and home/remote monitoring were discussed.

Deep Learning for hand tracking in Parkinson's Disease video-based assessment: Current and future perspectives

BACKGROUND

Parkinson's Disease (PD) demands early diagnosis and frequent assessment of symptoms. In particular, analysing hand movements is pivotal to understand disease progression. Advancements in hand tracking using Deep Learning (DL) allow for the automatic and objective disease evaluation from video recordings of standardised motor tasks, which are the foundation of neurological examinations. In view of this scenario, this narrative review aims to describe the state of the art and the future perspective of DL frameworks for hand tracking in video-based PD assessment.

METHODS

A rigorous search of PubMed, Web of Science, IEEE Explorer, and Scopus until October 2023 using primary keywords such as parkinson, hand tracking, and deep learning was performed to select eligible by focusing on video-based PD assessment through DL-driven hand tracking frameworks RESULTS:: After accurate screening, 23 publications met the selection criteria. These studies used various solutions, from well-established pose estimation frameworks, like OpenPose and MediaPipe, to custom deep architectures designed to accurately track hand and finger movements and extract relevant disease features. Estimated hand tracking data were then used to differentiate PD patients from healthy individuals, characterise symptoms such as tremors and bradykinesia, or regress the Movement Disorder Society-Unified Parkinson's Disease Rating Scale (MDS-UPDRS) by automatically assessing clinical tasks such as finger tapping, hand movements, and pronation-supination.

CONCLUSIONS

DL-driven hand tracking holds promise for PD assessment, offering precise, objective measurements for early diagnosis and monitoring, especially in a telemedicine scenario. However, to ensure clinical acceptance, standardisation and validation are crucial. Future research should prioritise large open datasets, rigorous validation on patients, and the investigation of new frontiers such as tracking hand-hand and hand-object interactions for daily-life tasks assessment.

Development of a Tremor Detection Algorithm for Use in an Academic Movement Disorders Center

Tremor, defined as an "involuntary, rhythmic, oscillatory movement of a body part", is a key feature of many neurological conditions including Parkinson's disease and essential tremor. Clinical assessment continues to be performed by visual observation with quantification on clinical scales. Methodologies for objectively quantifying tremor are promising but remain non-standardized across centers. Our center performs full-body behavioral testing with 3D motion capture for clinical and research purposes in patients with Parkinson's disease, essential tremor, and other conditions. The objective of this study was to assess the ability of several candidate processing pipelines to identify the presence or absence of tremor in kinematic data from patients with confirmed movement disorders and compare them to expert ratings from movement disorders specialists. We curated a database of 2272 separate kinematic data recordings from our center, each of which was contemporaneously annotated as tremor present or absent by a movement physician. We compared the ability of six separate processing pipelines to recreate clinician ratings based on F1 score, in addition to accuracy, precision, and recall. The performance across algorithms was generally comparable. The average F1 score was 0.84±0.02 (mean ± SD; range 0.81-0.87). The second highest performing algorithm (cross-validated F1=0.87) was a hybrid that used engineered features adapted from an algorithm in longstanding clinical use with a modern Support Vector Machine classifier. Taken together, our results suggest the potential to update legacy clinical decision support systems to incorporate modern machine learning classifiers to create better-performing tools.

Interpretable video-based tracking and quantification of parkinsonism clinical motor states

Quantification of motor symptom progression in Parkinson's disease (PD) patients is crucial for assessing disease progression and for optimizing therapeutic interventions, such as dopaminergic medications and deep brain stimulation. Cumulative and heuristic clinical experience has identified various clinical signs associated with PD severity, but these are neither objectively quantifiable nor robustly validated. Video-based objective symptom quantification enabled by machine learning (ML) introduces a potential solution. However, video-based diagnostic tools often have implementation challenges due to expensive and inaccessible technology, and typical "black-box" ML implementations are not tailored to be clinically interpretable. Here, we address these needs by releasing a comprehensive kinematic dataset and developing an interpretable video-based framework that predicts high versus low PD motor symptom severity according to MDS-UPDRS Part III metrics. This data driven approach validated and robustly quantified canonical movement features and identified new clinical insights, not previously appreciated as related to clinical severity, including pinkie finger movements and lower limb and axial features of gait. Our framework is enabled by retrospective, single-view, seconds-long videos recorded on consumer-grade devices such as smartphones, tablets, and digital cameras, thereby eliminating the requirement for specialized equipment. Following interpretable ML principles, our framework enforces robustness and interpretability by integrating (1) automatic, data-driven kinematic metric evaluation guided by pre-defined digital features of movement, (2) combination of bi-domain (body and hand) kinematic features, and (3) sparsity-inducing and stability-driven ML analysis with simple-to-interpret models. These elements ensure that the proposed framework quantifies clinically meaningful motor features useful for both ML predictions and clinical analysis.

Validation and application of computer vision algorithms for video-based tremor analysis

Tremor is one of the most common neurological symptoms. Its clinical and neurobiological complexity necessitates novel approaches for granular phenotyping. Instrumented neurophysiological analyses have proven useful, but are highly resource-intensive and lack broad accessibility. In contrast, bedside scores are simple to administer, but lack the granularity to capture subtle but relevant tremor features. We utilise the open-source computer vision pose tracking algorithm Mediapipe to track hands in clinical video recordings and use the resulting time series to compute canonical tremor features. This approach is compared to marker-based 3D motion capture, wrist-worn accelerometry, clinical scoring and a second, specifically trained tremor-specific algorithm in two independent clinical cohorts. These cohorts consisted of 66 patients diagnosed with essential tremor, assessed in different task conditions and states of deep brain stimulation therapy. We find that Mediapipe-derived tremor metrics exhibit high convergent clinical validity to scores (Spearman's ρ = 0.55-0.86, p≤ .01) as well as an accuracy of up to 2.60 mm (95% CI [-3.13, 8.23]) and ≤0.21 Hz (95% CI [-0.05, 0.46]) for tremor amplitude and frequency measurements, matching gold-standard equipment. Mediapipe, but not the disease-specific algorithm, was capable of analysing videos involving complex configurational changes of the hands. Moreover, it enabled the extraction of tremor features with diagnostic and prognostic relevance, a dimension which conventional tremor scores were unable to provide. Collectively, this demonstrates that current computer vision algorithms can be transformed into an accurate and highly accessible tool for video-based tremor analysis, yielding comparable results to gold standard tremor recordings.

Biomechanical Posture Analysis in Healthy Adults with Machine Learning: Applicability and Reliability

Posture analysis is important in musculoskeletal disorder prevention but relies on subjective assessment. This study investigates the applicability and reliability of a machine learning (ML) pose estimation model for the human posture assessment, while also exploring the underlying structure of the data through principal component and cluster analyses. A cohort of 200 healthy individuals with a mean age of 24.4 ± 4.2 years was photographed from the frontal, dorsal, and lateral views. We used Student's t-test and Cohen's effect size (d) to identify gender-specific postural differences and used the Intraclass Correlation Coefficient (ICC) to assess the reliability of this method. Our findings demonstrate distinct sex differences in shoulder adduction angle (men: 16.1° ± 1.9°, women: 14.1° ± 1.5°, d = 1.14) and hip adduction angle (men: 9.9° ± 2.2°, women: 6.7° ± 1.5°, d = 1.67), with no significant differences in horizontal inclinations. ICC analysis, with the highest value of 0.95, confirms the reliability of the approach. Principal component and clustering analyses revealed potential new patterns in postural analysis such as significant differences in shoulder-hip distance, highlighting the potential of unsupervised ML for objective posture analysis, offering a promising non-invasive method for rapid, reliable screening in physical therapy, ergonomics, and sports.

Video-based analysis of the blink reflex in Parkinson's disease patients

We developed a video-based tool to quantitatively assess the Glabellar Tap Reflex (GTR) in patients with idiopathic Parkinson's disease (iPD) as well as healthy age-matched participants. We also video-graphically assessed the effect of dopaminergic medication on the GTR in iPD patients, as well as the frequency and blinking duration of reflex and non-reflex blinks. The Glabellar Tap Reflex is a clinical sign seen in patients e.g. suffering from iPD. Reliable tools to quantify this sign are lacking.

METHODS

We recorded the GTR in 11 iPD patients and 12 healthy controls (HC) with a consumer-grade camera at a framerate of at least 180 images/s. In these videos, reflex and non-reflex blinks were analyzed for blink count and blinking duration in an automated fashion.

RESULTS

With our setup, the GTR can be extracted from high-framerate cameras using landmarks of the MediaPipe face algorithm. iPD patients did not habituate to the GTR; dopaminergic medication did not alter that response. iPD patients' non-reflex blinks were higher in frequency and higher in blinking duration (width at half prominence); dopaminergic medication decreased the median frequency (Before medication-HC: p < 0.001, After medication-HC: p = 0.0026) and decreased the median blinking duration (Before medication-HC: p = 0.8594, After medication-HC: p = 0.6943)-both in the direction of HC.

CONCLUSION

We developed a quantitative, video-based tool to assess the GTR and other blinking-specific parameters in HC and iPD patients. Further studies could compare the video data to electromyogram (EMG) data for accuracy and comparability, as well as evaluate the specificity of the GTR in patients with other neurodegenerative disorders, in whom the GTR can also be present.

SIGNIFICANCE

The video-based detection of the blinking parameters allows for unobtrusive measurement in patients, a safer and more comfortable option.

Sensor-Based Quantification of MDS-UPDRS III Subitems in Parkinson's Disease Using Machine Learning

Wearable sensors could be beneficial for the continuous quantification of upper limb motor symptoms in people with Parkinson's disease (PD). This work evaluates the use of two inertial measurement units combined with supervised machine learning models to classify and predict a subset of MDS-UPDRS III subitems in PD. We attached the two compact wearable sensors on the dorsal part of each hand of 33 people with PD and 12 controls. Each participant performed six clinical movement tasks in parallel with an assessment of the MDS-UPDRS III. Random forest (RF) models were trained on the sensor data and motor scores. An overall accuracy of 94% was achieved in classifying the movement tasks. When employed for classifying the motor scores, the averaged area under the receiver operating characteristic values ranged from 68% to 92%. Motor scores were additionally predicted using an RF regression model. In a comparative analysis, trained support vector machine models outperformed the RF models for specific tasks. Furthermore, our results surpass the literature in certain cases. The methods developed in this work serve as a base for future studies, where home-based assessments of pharmacological effects on motor function could complement regular clinical assessments.

An automated ICU agitation monitoring system for video streaming using deep learning classification

OBJECTIVE

To address the challenge of assessing sedation status in critically ill patients in the intensive care unit (ICU), we aimed to develop a non-contact automatic classifier of agitation using artificial intelligence and deep learning.

METHODS

We collected the video recordings of ICU patients and cut them into 30-second (30-s) and 2-second (2-s) segments. All of the segments were annotated with the status of agitation as "Attention" and "Non-attention". After transforming the video segments into movement quantification, we constructed the models of agitation classifiers with Threshold, Random Forest, and LSTM and evaluated their performances.

RESULTS

The video recording segmentation yielded 427 30-s and 6405 2-s segments from 61 patients for model construction. The LSTM model achieved remarkable accuracy (ACC 0.92, AUC 0.91), outperforming other methods.

CONCLUSION

Our study proposes an advanced monitoring system combining LSTM and image processing to ensure mild patient sedation in ICU care. LSTM proves to be the optimal choice for accurate monitoring. Future efforts should prioritize expanding data collection and enhancing system integration for practical application.

Computer Vision for Parkinson's Disease Evaluation: A Survey on Finger Tapping

Parkinson's disease (PD) is a progressive neurodegenerative disorder whose prevalence has steadily been rising over the years. Specialist neurologists across the world assess and diagnose patients with PD, although the diagnostic process is time-consuming and various symptoms take years to appear, which means that the diagnosis is prone to human error. The partial automatization of PD assessment and diagnosis through computational processes has therefore been considered for some time. One well-known tool for PD assessment is finger tapping (FT), which can now be assessed through computer vision (CV). Artificial intelligence and related advances over recent decades, more specifically in the area of CV, have made it possible to develop computer systems that can help specialists assess and diagnose PD. The aim of this study is to review some advances related to CV techniques and FT so as to offer insight into future research lines that technological advances are now opening up.

Improving Gait Analysis Techniques with Markerless Pose Estimation Based on Smartphone Location

Marker-based 3D motion capture systems, widely used for gait analysis, are accurate but have disadvantages such as cost and accessibility. Whereas markerless pose estimation has emerged as a convenient and cost-effective alternative for gait analysis, challenges remain in achieving optimal accuracy. Given the limited research on the effects of camera location and orientation on data collection accuracy, this study investigates how camera placement affects gait assessment accuracy utilizing five smartphones. This study aimed to explore the differences in data collection accuracy between marker-based systems and pose estimation, as well as to assess the impact of camera location and orientation on accuracy in pose estimation. The results showed that the differences in joint angles between pose estimation and marker-based systems are below 5°, an acceptable level for gait analysis, with a strong correlation between the two datasets supporting the effectiveness of pose estimation in gait analysis. In addition, hip and knee angles were accurately measured at the front diagonal of the subject and ankle angle at the lateral side. This research highlights the significance of careful camera placement for reliable gait analysis using pose estimation, serving as a concise reference to guide future efforts in enhancing the quantitative accuracy of gait analysis.

Bioinspired Photoreceptors with Neural Network for Recognition and Classification of Sign Language Gesture

This work addresses the design and implementation of a novel PhotoBiological Filter Classifier (PhBFC) to improve the accuracy of a static sign language translation system. The captured images are preprocessed by a contrast enhancement algorithm inspired by the capacity of retinal photoreceptor cells from mammals, which are responsible for capturing light and transforming it into electric signals that the brain can interpret as images. This sign translation system not only supports the effective communication between an agent and an operator but also between a community with hearing disabilities and other people. Additionally, this technology could be integrated into diverse devices and applications, further broadening its scope, and extending its benefits for the community in general. The bioinspired photoreceptor model is evaluated under different conditions. To validate the advantages of applying photoreceptors cells, 100 tests were conducted per letter to be recognized, on three different models (V1, V2, and V3), obtaining an average of 91.1% of accuracy on V3, compared to 63.4% obtained on V1, and an average of 55.5 Frames Per Second (FPS) in each letter classification iteration for V1, V2, and V3, demonstrating that the use of photoreceptor cells does not affect the processing time while also improving the accuracy. The great application potential of this system is underscored, as it can be employed, for example, in Deep Learning (DL) for pattern recognition or agent decision-making trained by reinforcement learning, etc.

Feasibility of virtual low-cost quantitative continuous measurement of movements in the extremities of people with Parkinson's disease

A low-cost quantitative continuous measurement of movements in the extremities of people with Parkinson's disease, a structured motor assessment administered by a trained examiner to a patient physically present in the same room, utilizes sensors to generate output to facilitate the evaluation of the patient. However, motor assessments with the patient and the examiner in the same room may not be feasible due to distances between the patient and the examiner and the risk of transmission of infections between the patient and the examiner. Therefore, we propose a protocol for the remote assessment by examiners in different locations of both (A) videos of patients recorded during in-person motor assessments and (B) live virtual assessments of patients in different locations from examiners. The proposed procedure provides a framework for providers, investigators, and patients in vastly diverse locations to conduct optimal motor assessments required to develop treatment plans utilizing precision medicine tailored to the specific needs of each individual patient. The proposed protocol generates the foundation for providers to remotely perform structured motor assessments necessary for optimal diagnosis and treatment of people with Parkinson's disease and related conditions.

Objective measurement versus clinician-based assessment for Parkinson's disease

INTRODUCTION

Although clinician-based assessment through standardized clinical rating scales is currently the gold standard for quantifying motor impairment in Parkinson's disease (PD), it is not without limitations, including intra- and inter-rater variability and a degree of approximation. There is increasing evidence supporting the use of objective motion analyses to complement clinician-based assessment. Objective measurement tools hold significant potential for improving the accuracy of clinical and research-based evaluations of patients.

AREAS COVERED

The authors provide several examples from the literature demonstrating how different motion measurement tools, including optoelectronics, contactless and wearable systems allow for both the objective quantification and monitoring of key motor symptoms (such as bradykinesia, rigidity, tremor, and gait disturbances), and the identification of motor fluctuations in PD patients. Furthermore, they discuss how, from a clinician's perspective, objective measurements can help in various stages of PD management.

EXPERT OPINION

In our opinion, sufficient evidence supports the assertion that objective monitoring systems enable accurate evaluation of motor symptoms and complications in PD. A range of devices can be utilized not only to support diagnosis but also to monitor motor symptom during the disease progression and can become relevant in the therapeutic decision-making process.

Recent Advancements in Sensor Technologies for Healthcare and Biomedical Applications

Biomedical sensors are the key units of medical and healthcare systems [...].

Smartphone video nystagmography using convolutional neural networks: ConVNG

Background

Eye movement abnormalities are commonplace in neurological disorders. However, unaided eye movement assessments lack granularity. Although videooculography (VOG) improves diagnostic accuracy, resource intensiveness precludes its broad use. To bridge this care gap, we here validate a framework for smartphone video-based nystagmography capitalizing on recent computer vision advances.

Methods

A convolutional neural network was fine-tuned for pupil tracking using > 550 annotated frames: ConVNG. In a cross-sectional approach, slow-phase velocity of optokinetic nystagmus was calculated in 10 subjects using ConVNG and VOG. Equivalence of accuracy and precision was assessed using the “two one-sample t-test” (TOST) and Bayesian interval-null approaches. ConVNG was systematically compared to OpenFace and MediaPipe as computer vision (CV) benchmarks for gaze estimation.

Results

ConVNG tracking accuracy reached 9–15% of an average pupil diameter. In a fully independent clinical video dataset, ConVNG robustly detected pupil keypoints (median prediction confidence 0.85). SPV measurement accuracy was equivalent to VOG (TOST p < 0.017; Bayes factors (BF) > 24). ConVNG, but not MediaPipe, achieved equivalence to VOG in all SPV calculations. Median precision was 0.30°/s for ConVNG, 0.7°/s for MediaPipe and 0.12°/s for VOG. ConVNG precision was significantly higher than MediaPipe in vertical planes, but both algorithms’ precision was inferior to VOG.

Conclusions

ConVNG enables offline smartphone video nystagmography with an accuracy comparable to VOG and significantly higher precision than MediaPipe, a benchmark computer vision application for gaze estimation. This serves as a blueprint for highly accessible tools with potential to accelerate progress toward precise and personalized Medicine.