Assessment of Parkinsonian gait in older adults with dementia via human pose tracking in video data

Multimodal Technologies for Remote Assessment of Neurological and Mental Health

PURPOSE

Automated remote assessment and monitoring of patients' neurological and mental health is increasingly becoming an essential component of the digital clinic and telehealth ecosystem, especially after the COVID-19 pandemic. This review article reviews various modalities of health information that are useful for developing such remote clinical assessments in the real world at scale.

APPROACH

We first present an overview of the various modalities of health information-speech acoustics, natural language, conversational dynamics, orofacial or full body movement, eye gaze, respiration, cardiopulmonary, and neural-which can each be extracted from various signal sources-audio, video, text, or sensors. We further motivate their clinical utility with examples of how information from each modality can help us characterize how different disorders affect different aspects of patients' spoken communication. We then elucidate the advantages of combining one or more of these modalities toward a more holistic, informative, and robust assessment.

FINDINGS

We find that combining multiple modalities of health information allows for improved scientific interpretability, improved performance on downstream health applications such as early detection and progress monitoring, improved technological robustness, and improved user experience. We illustrate how these principles can be leveraged for remote clinical assessment at scale using a real-world case study of the Modality assessment platform.

CONCLUSION

This review article motivates the combination of human-centric information from multiple modalities to measure various aspects of patients' health, arguing that remote clinical assessment that integrates this complementary information can be more effective and lead to better clinical outcomes than using any one data stream in isolation.

Vision-based motion capture for the gait analysis of neurodegenerative diseases: A review

BACKGROUND

Developments in vision-based systems and human pose estimation algorithms have the potential to detect, monitor and intervene early on neurodegenerative diseases through gait analysis. However, the gap between the technology available and actual clinical practice is evident as most clinicians still rely on subjective observational gait analysis or objective marker-based analysis that is time-consuming.

RESEARCH QUESTION

This paper aims to examine the main developments of vision-based motion capture and how such advances may be integrated into clinical practice.

METHODS

The literature review was conducted in six online databases using Boolean search terms. A commercial system search was also included. A predetermined methodological criterion was then used to assess the quality of the selected articles.

RESULTS

A total of seventeen studies were evaluated, with thirteen studies focusing on gait classification systems and four studies on gait measurement systems. Of the gait classification systems, nine studies utilized artificial intelligence-assisted techniques, while four studies employed statistical techniques. The results revealed high correlations of gait features identified by classifier models with existing clinical rating scales. These systems demonstrated generally high classification accuracies and were effective in diagnosing disease severity levels. Gait measurement systems that extract spatiotemporal and kinematic joint information from video data generally found accurate measurements of gait parameters with low mean absolute errors, high intra- and inter-rater reliability.

SIGNIFICANCE

Low cost, portable vision-based systems can provide proof of concept for the quantification of gait, expansion of gait assessment tools, remote gait analysis of neurodegenerative diseases and a point of care system for orthotic evaluation. However, certain challenges, including small sample sizes, occlusion risks, and selection bias in training models, need to be addressed. Nevertheless, these systems can serve as complementary tools, equipping clinicians with essential gait information to objectively assess disease severity and tailor personalized treatment for enhanced patient care.

Causality-Enhanced Multiple Instance Learning With Graph Convolutional Networks for Parkinsonian Freezing-of-Gait Assessment

Freezing of gait (FoG) is a common disabling symptom of Parkinson's disease (PD). It is clinically characterized by sudden and transient walking interruptions for specific human body parts, and it presents the localization in time and space. Due to the difficulty in extracting global fine-grained features from lengthy videos, developing an automated five-point FoG scoring system is quite challenging. Therefore, we propose a novel video-based automated five-classification FoG assessment method with a causality-enhanced multiple-instance-learning graph convolutional network (GCN). This method involves developing a temporal segmentation GCN to segment each video into three motion stages for stage-level feature modeling, followed by a multiple-instance-learning framework to divide each stage into short clips for instance-level feature extraction. Subsequently, an uncertainty-driven multiple-instance-learning GCN is developed to capture spatial and temporal fine-grained features through GCN scheme and uncertainty learning, respectively, for acquiring global representations. Finally, a causality-enhanced graph generation strategy is proposed to exploit causal inference for mining and enhancing human structures causally related to clinical assessment, thereby extracting spatial causal features. Extensive experimental results demonstrate the excellent performance of the proposed method on five-classification FoG assessment with an accuracy of 62.72% and an acceptable accuracy of 91.32%, which is confirmed by independent testing. Additionally, it enables temporal and spatial localization of FoG events to a certain extent, facilitating reasonable clinical interpretations. In conclusion, our method provides a valuable tool for automated FoG assessment in PD, and the proposed causality-related component exhibits promising potential for extension to other general and medical fine-grained action recognition tasks.

Parkinson's severity diagnosis explainable model based on 3D multi-head attention residual network

The severity evaluation of Parkinson's disease (PD) is of great significance for the treatment of PD. However, existing methods either have limitations based on prior knowledge or are invasive methods. To propose a more generalized severity evaluation model, this paper proposes an explainable 3D multi-head attention residual convolution network. First, we introduce the 3D attention-based convolution layer to extract video features. Second, features will be fed into LSTM and residual backbone networks, which can be used to capture the contextual information of the video. Finally, we design a feature compression module to condense the learned contextual features. We develop some interpretable experiments to better explain this black-box model so that it can be better generalized. Experiments show that our model can achieve state-of-the-art diagnosis performance. The proposed lightweight but effective model is expected to serve as a suitable end-to-end deep learning baseline in future research on PD video-based severity evaluation and has the potential for large-scale application in PD telemedicine. The source code is available at https://github.com/JackAILab/MARNet.

Comprehensive Geriatric Assessment in the era of telemedicine

The aging global population poses significant medical and social challenges, necessitating efforts to promote healthy aging. Comprehensive Geriatric Assessment (CGA) is a multidimensional diagnostic approach for older adults that aims to improve overall health. Remote CGA, facilitated by technological advancements, offers convenience and other potential advantages. It enables early disease detection, monitors chronic disease progression, delivers personalized care, and optimizes healthcare resources for better health outcomes in older individuals. However, remote CGA also has limitations, including technological requirements, data security, and the need for comprehensive evaluation and simplicity. Collaborative efforts are essential to developing a digital home-based CGA platform that addresses accessibility issues and tailors the assessment process to meet the needs of older adults. Continuous optimization of remote CGA can become a pivotal tool for advancing geriatric care and ensuring the well-being of the aging population. Geriatr Gerontol Int 2024; 24: 67-73.

Cross-Spatiotemporal Graph Convolution Networks for Skeleton-Based Parkinsonian Gait MDS-UPDRS Score Estimation

Gait impairment in Parkinson's Disease (PD) is quantitatively assessed using the Movement Disorder Society Unified Parkinson's Disease Rating Scale (MDS-UPDRS), a well-established clinical tool. Objective and efficient PD gait assessment is crucial for developing interventions to slow or halt its advancement. Skeleton-based PD gait MDS-UPDRS score estimation has attracted increasing interest in improving diagnostic efficiency and objectivity. However, previous works ignore the important cross-spacetime dependencies between joints in PD gait. Moreover, existing PD gait skeleton datasets are very small, which is a big issue in deep learning-based gait studies. In this work, we collect a sizable PD gait skeleton dataset by multi-view Azure Kinect sensors. The collected dataset contains 102 PD patients and 30 healthy older adults. In addition, gait data from 16 young adults (aged 24-50 years) are collected to further examine the effect of age on PD gait assessment. For skeleton-based automatic PD gait analysis, we propose a novel cross-spatiotemporal graph convolution network (CST-GCN) to learn complex features of gait patterns. Specifically, a gait graph labeling strategy is designed to assemble and group cross-spacetime neighbors of the root node according to the spatiotemporal semantics of the gait skeleton. Based on this strategy, the CST-GCN module explicitly models the cross-spacetime dependencies among joints. Finally, a dual-path model is presented to realize the modeling and fusion of spatial, temporal, and cross-spacetime gait features. Extensive experiments validate the effectiveness of our method on the collected dataset.

Automated Quantification of Eye Tics Using Computer Vision and Deep Learning Techniques

BACKGROUND

Tourette syndrome (TS) tics are typically quantified using "paper and pencil" rating scales that are susceptible to factors that adversely impact validity. Video-based methods to more objectively quantify tics have been developed but are challenged by reliance on human raters and procedures that are resource intensive. Computer vision approaches that automate detection of atypical movements may be useful to apply to tic quantification.

OBJECTIVE

The current proof-of-concept study applied a computer vision approach to train a supervised deep learning algorithm to detect eye tics in video, the most common tic type in patients with TS.

METHODS

Videos (N = 54) of 11 adolescent patients with TS were rigorously coded by trained human raters to identify 1.5-second clips depicting "eye tic events" (N = 1775) and "non-tic events" (N = 3680). Clips were encoded into three-dimensional facial landmarks. Supervised deep learning was applied to processed data using random split and disjoint split regimens to simulate model validity under different conditions.

RESULTS

Area under receiver operating characteristic curve was 0.89 for the random split regimen, indicating high accuracy in the algorithm's ability to properly classify eye tic vs. non-eye tic movements. Area under receiver operating characteristic curve was 0.74 for the disjoint split regimen, suggesting that algorithm generalizability is more limited when trained on a small patient sample.

CONCLUSIONS

The algorithm was successful in detecting eye tics in unseen validation sets. Automated tic detection from video is a promising approach for tic quantification that may have future utility in TS screening, diagnostics, and treatment outcome measurement. © 2023 The Authors. Movement Disorders published by Wiley Periodicals LLC on behalf of International Parkinson and Movement Disorder Society.

Automatic labeling of Parkinson's Disease gait videos with weak supervision

Motor dysfunction in Parkinson's Disease (PD) patients is typically assessed by clinicians employing the Movement Disorder Society's Unified Parkinson's Disease Rating Scale (MDS-UPDRS). Such comprehensive clinical assessments are time-consuming, expensive, semi-subjective, and may potentially result in conflicting labels across different raters. To address this problem, we propose an automatic, objective, and weakly-supervised method for labeling PD patients' gait videos. The proposed method accepts videos of patients and classifies their gait scores as normal (Gait score in MDS-UPDRS = 0) or PD (MDS-UPDRS ≥ 1). Unlike previous work, the proposed method does not require a priori MDS-UPDRS ratings for training, utilizing only domain-specific knowledge obtained from neurologists. We propose several labeling functions that classify patients' gait and use a generative model to learn the accuracy of each labeling function in a self-supervised manner. Since results depended upon the estimated values of the patients' 3D poses, and existing pre-trained 3D pose estimators did not yield accurate results, we propose a weakly-supervised 3D human pose estimation method for fine-tuning pre-trained models in a clinical setting. Using leave-one-out evaluations, the proposed method obtains an accuracy of 89% on a dataset of 29 PD subjects - a significant improvement compared to previous work by 7%-10% depending upon the dataset. The method obtained state-of-the-art results on the Human3.6M dataset. Our results suggest that the use of labeling functions may provide a robust means to interpret and classify patient-oriented videos involving motor tasks.

Motion Capture Technologies for Ergonomics: A Systematic Literature Review

Muscular skeletal disorder is a difficult challenge faced by the working population. Motion capture (MoCap) is used for recording the movement of people for clinical, ergonomic and rehabilitation solutions. However, knowledge barriers about these MoCap systems have made them difficult to use for many people. Despite this, no state-of-the-art literature review on MoCap systems for human clinical, rehabilitation and ergonomic analysis has been conducted. A medical diagnosis using AI applies machine learning algorithms and motion capture technologies to analyze patient data, enhancing diagnostic accuracy, enabling early disease detection and facilitating personalized treatment plans. It revolutionizes healthcare by harnessing the power of data-driven insights for improved patient outcomes and efficient clinical decision-making. The current review aimed to investigate: (i) the most used MoCap systems for clinical use, ergonomics and rehabilitation, (ii) their application and (iii) the target population. We used preferred reporting items for systematic reviews and meta-analysis guidelines for the review. Google Scholar, PubMed, Scopus and Web of Science were used to search for relevant published articles. The articles obtained were scrutinized by reading the abstracts and titles to determine their inclusion eligibility. Accordingly, articles with insufficient or irrelevant information were excluded from the screening. The search included studies published between 2013 and 2023 (including additional criteria). A total of 40 articles were eligible for review. The selected articles were further categorized in terms of the types of MoCap used, their application and the domain of the experiments. This review will serve as a guide for researchers and organizational management.

A systematic review of the applications of markerless motion capture (MMC) technology for clinical measurement in rehabilitation

BACKGROUND

Markerless motion capture (MMC) technology has been developed to avoid the need for body marker placement during motion tracking and analysis of human movement. Although researchers have long proposed the use of MMC technology in clinical measurement-identification and measurement of movement kinematics in a clinical population, its actual application is still in its preliminary stages. The benefits of MMC technology are also inconclusive with regard to its use in assessing patients' conditions. In this review we put a minor focus on the method's engineering components and sought primarily to determine the current application of MMC as a clinical measurement tool in rehabilitation.

METHODS

A systematic computerized literature search was conducted in PubMed, Medline, CINAHL, CENTRAL, EMBASE, and IEEE. The search keywords used in each database were "Markerless Motion Capture OR Motion Capture OR Motion Capture Technology OR Markerless Motion Capture Technology OR Computer Vision OR Video-based OR Pose Estimation AND Assessment OR Clinical Assessment OR Clinical Measurement OR Assess." Only peer-reviewed articles that applied MMC technology for clinical measurement were included. The last search took place on March 6, 2023. Details regarding the application of MMC technology for different types of patients and body parts, as well as the assessment results, were summarized.

RESULTS

A total of 65 studies were included. The MMC systems used for measurement were most frequently used to identify symptoms or to detect differences in movement patterns between disease populations and their healthy counterparts. Patients with Parkinson's disease (PD) who demonstrated obvious and well-defined physical signs were the largest patient group to which MMC assessment had been applied. Microsoft Kinect was the most frequently used MMC system, although there was a recent trend of motion analysis using video captured with a smartphone camera.

CONCLUSIONS

This review explored the current uses of MMC technology for clinical measurement. MMC technology has the potential to be used as an assessment tool as well as to assist in the detection and identification of symptoms, which might further contribute to the use of an artificial intelligence method for early screening for diseases. Further studies are warranted to develop and integrate MMC system in a platform that can be user-friendly and accurately analyzed by clinicians to extend the use of MMC technology in the disease populations.

Vision-based estimation of MDS-UPDRS scores for quantifying Parkinson's disease tremor severity

Parkinson's disease (PD) is a common neurodegenerative movement disorder among older individuals. As one of the typical symptoms of PD, tremor is a critical reference in the PD assessment. A widely accepted clinical approach to assessing tremors in PD is based on part III of the Movement Disorder Society-Unified Parkinson's Disease Rating Scale (MDS-UPDRS). However, expert assessment of tremor is a time-consuming and laborious process that poses considerable challenges to the medical evaluation of PD. In this paper, we proposed a novel model, Global Temporal-difference Shift Network (GTSN), to estimate the MDS-UPDRS score of PD tremors based on video. The PD tremor videos were scored according to the majority vote of multiple raters. We used Eulerian Video Magnification (EVM) pre-processing to enhance the representations of subtle PD tremors in the videos. To make the model better focus on the tremors in the video, we proposed a special temporal difference module, which stacks the current optical flow to the result of inter-frame difference. The prediction scores were obtained from the Residual Networks (ResNet) embedded with a novel module, the Global Shift Module (GSM), which allowed the features of the current segment to include the global segment features. We carried out independent experiments using PD tremor videos of different body parts based on the scoring content of the MDS-UPDRS. On a fairly large dataset, our method achieved an accuracy of 90.6% for hands with rest tremors, 85.9% for tremors in the leg, and 89.0% for the jaw. An accuracy of 84.9% was obtained for postural tremors. Our study demonstrated the effectiveness of computer-assisted assessment for PD tremors based on video analysis. The latest version of the code is available at https://github.com/199507284711/PD-GTSN.

Multi-speed transformer network for neurodegenerative disease assessment and activity recognition

BACKGROUND AND OBJECTIVE

Neurodegenerative diseases are the most frequent age-related diseases. This type of disease, if not discovered in the initial stage, will compromise the quality of life of the affected subject. Thus, a timely diagnosis is of paramount importance. One of the most used tasks from neurologists to detect and determine the severity of the disease is analysing human gait. This work presents the dataset named "Beside Gait" containing timeseries of coordinates of extracted body joints of people with neurodegenerative diseases in various stages of the disease as well as control subjects. In addition, the novel Multi-Speed transformer technique will be presented and benchmarked against several other techniques making use of deep learning and Shallow Learning. The objective is to recognize subjects affected by some form of neurodegenerative disease in early stage using a computer vision technique making use of deep learning that can be integrated into a smartphone app for offline inference with the aim of promptly initiate investigations and treatment to improve the patient's quality of life.

METHODS

The recorded videos were processed, and the skeleton of the person in the video was extracted using pose estimation. The raw time-series coordinates of the joints extracted by the pose estimation algorithm were tested against novel deep neural network architectures and Shallow Learning techniques. In this work, the proposed Multi-Speed Transformer is benchmarked against other deep neural networks such as Temporal Convolutional Neural Networks, Transformers, as well as Shallow Learning techniques making use of feature extraction and different classifiers such as Random Forests, K Nearest Neighbours, Ada Boost, Linear and RBF SVM. The proposed Multi-Speed Transformer architecture has been developed to learn short and long-term patterns to model the various pathological gaits.

RESULTS

The Multi-Speed Transformer outperformed all other existing models reaching an accuracy of 96.9%, a sensitivity of 96.9%, a precision of 97.7%, and a specificity of 97.1% in binary classification. The accuracy in multi-class classification for detecting the presence of the disease in various stages is 71.6%, the sensitivity is 67.7%, and the specificity is 71.8%. In addition, tests have also been conducted against two other different activity recognition datasets, namely SHREC and JHMDB, in the exact same conditions. Multi-Speed Transformer has demonstrated to beat always all other tested techniques as well as the techniques reviewed in the state-of-the-art with respectively of accuracy 91.8% and 74%. Having those datasets more than two classes, specificity was not computed.

CONCLUSIONS

The Multi-Speed Transformer is a valuable technique for neurodegenerative disease assessment through computer vision. In addition, the novel dataset "Beside Gait" here presented is an important starting point for future research work on automatic recognition of neurodegenerative diseases using gait analysis.

Enabling Gait Analysis in the Telemedicine Practice through Portable and Accurate 3D Human Pose Estimation

Human pose estimation (HPE) through deep learning-based software applications is a trend topic for markerless motion analysis. Thanks to the accuracy of the state-of-the-art technology, HPE could enable gait analysis in the telemedicine practice. On the other hand, delivering such a service at a distance requires the system to satisfy multiple and different constraints like accuracy, portability, real-time, and privacy compliance at the same time. Existing solutions either guarantee accuracy and real-time (e.g., the widespread OpenPose software on well-equipped computing platforms) or portability and data privacy (e.g., light convolutional neural networks on mobile phones). We propose a portable and low-cost platform that implements real-time and accurate 3D HPE through an embedded software on a low-power off-the-shelf computing device that guarantees privacy by default and by design. We present an extended evaluation of both accuracy and performance of the proposed solution conducted with a marker-based motion capture system (i.e., Vicon) as ground truth. The results show that the platform achieves real-time performance and high-accuracy with a deviation below the error tolerance when compared to the marker-based motion capture system (e.g., less than an error of 5^∘ on the estimated knee flexion difference on the entire gait cycle and correlation 0.91<ρ<0.99). We provide a proof-of-concept study, showing that such portable technology, considering the limited discrepancies with respect to the marker-based motion capture system and its working tolerance, could be used for gait analysis at a distance without leading to different clinical interpretation.

Detection and assessment of Parkinson's disease based on gait analysis: A survey

Neurological disorders represent one of the leading causes of disability and mortality in the world. Parkinson's Disease (PD), for example, affecting millions of people worldwide is often manifested as impaired posture and gait. These impairments have been used as a clinical sign for the early detection of PD, as well as an objective index for pervasive monitoring of the PD patients in daily life. This review presents the evidence that demonstrates the relationship between human gait and PD, and illustrates the role of different gait analysis systems based on vision or wearable sensors. It also provides a comprehensive overview of the available automatic recognition systems for the detection and management of PD. The intervening measures for improving gait performance are summarized, in which the smart devices for gait intervention are emphasized. Finally, this review highlights some of the new opportunities in detecting, monitoring, and treating of PD based on gait, which could facilitate the development of objective gait-based biomarkers for personalized support and treatment of PD.

The Toronto older adults gait archive: video and 3D inertial motion capture data of older adults' walking

We introduce the Toronto Older Adults Gait Archive, a gait dataset of 14 older adults containing 2D video recordings, and 2D (video pose tracking algorithms) and 3D (inertial motion capture) joint locations of the lower body. Participants walked for 60 seconds. We also collected participants’ scores on four clinical assessments of gait and balance, namely the Tinneti performance-oriented mobility assessment (POMA-gait and -balance), the Berg balance scale (BBS), and the timed-up-and-go (TUG). Three human pose tracking models (Alphapose, OpenPose, and Detectron) were used to detect body joint positions in 2D video frames and a number of gait parameters were computed using 2D video-based and 3D motion capture data. To show an example usage of our datasets, we performed a correlation analysis between the gait variables and the clinical scores. Our findings revealed that the temporal but not the spatial or variability gait variables from both systems had high correlations to clinical scores. This dataset can be used to evaluate, or to enhance vision-based pose-tracking models to the specifics of older adults’ walking.

Measurement(s)	Lower Extremity Gait, CTCAE
Technology Type(s)	digital camera • Accelerometer
Sample Characteristic - Organism	Homo
Sample Characteristic - Environment	nursing home
Sample Characteristic - Location	City of Toronto

Using a Video Device and a Deep Learning-Based Pose Estimator to Assess Gait Impairment in Neurodegenerative Related Disorders: A Pilot Study

As the world’s population is living longer, age-related neurodegenerative diseases are becoming a more significant global issue. Neurodegenerative diseases cause worsening motor, cognitive and autonomic dysfunction over time and reduce functional abilities required for daily living. Compromised motor performance is one of the first and most evident manifestations. In the case of Parkinson’s disease, these impairments are currently evaluated by experts through the use of rating scales. Although this method is widely used by experts worldwide, it includes subjective and error-prone motor examinations that also fail in the characterization of symptoms’ fluctuations. The aim of this study is to evaluate whether artificial intelligence techniques can be used to objectively assess gait impairment in subjects with Parkinson’s disease. This paper presents the results of a cohort of ten subjects, five with a Parkinson’s disease diagnosis at different degrees of severity. We experimentally demonstrate good effectiveness of the proposed system in extracting the main features concerning people’s gait during the standard tests that clinicians use to assess the burden of disease. This system can offer neurologists, through accurate and objective data, a second opinion or a suggestion to reconsider score assignment. Thanks to its simplicity, tactful and non-intrusive approach and clinical-grade accuracy, it can be adopted on an ongoing basis even in environments where people usually live and work.

Automatic Classification and Severity Estimation of Ataxia From Finger Tapping Videos

Digital assessments enable objective measurements of ataxia severity and provide informative features that expand upon the information obtained during a clinical examination. In this study, we demonstrate the feasibility of using finger tapping videos to distinguish participants with Ataxia (N = 169) from participants with parkinsonism (N = 78) and from controls (N = 58), and predict their upper extremity and overall disease severity. Features were extracted from the time series representing the distance between the index and thumb and its derivatives. Classification models in ataxia archived areas under the receiver-operating curve of around 0.91, and regression models estimating disease severity obtained correlation coefficients around r = 0.64. Classification and prediction model coefficients were examined and they not only were in accordance, but were in line with clinical observations of ataxia phenotypes where rate and rhythm are altered during upper extremity motor movement.

Review-Emerging Portable Technologies for Gait Analysis in Neurological Disorders

The understanding of locomotion in neurological disorders requires technologies for quantitative gait analysis. Numerous modalities are available today to objectively capture spatiotemporal gait and postural control features. Nevertheless, many obstacles prevent the application of these technologies to their full potential in neurological research and especially clinical practice. These include the required expert knowledge, time for data collection, and missing standards for data analysis and reporting. Here, we provide a technological review of wearable and vision-based portable motion analysis tools that emerged in the last decade with recent applications in neurological disorders such as Parkinson's disease and Multiple Sclerosis. The goal is to enable the reader to understand the available technologies with their individual strengths and limitations in order to make an informed decision for own investigations and clinical applications. We foresee that ongoing developments toward user-friendly automated devices will allow for closed-loop applications, long-term monitoring, and telemedical consulting in real-life environments.

Concurrent Validity of Zeno Instrumented Walkway and Video-Based Gait Features in Adults With Parkinson’s Disease

Background: Parkinson’s disease (PD) presents with motor symptoms such as bradykinesia, rigidity, and tremor that can affect gait. To monitor changes associated with disease progression or medication use, quantitative gait assessment is often performed during clinical visits. Conversely, vision-based solutions have been proposed for monitoring gait quality in non-clinical settings. Methods: We use three 2D human pose-estimation libraries (AlphaPose, Detectron, OpenPose) and one 3D library (ROMP) to calculate gait features from color video, and correlate them with those extracted by a Zeno instrumented walkway in older adults with PD. We calculate video-based gait features using a manual and automated heel-strike detection algorithm, and compare the correlations when the participants walk towards and away from the camera separately. Results: Based on analysis of 67 bidirectional walking bouts from 25 adults with PD, moderate to strong positive correlations were identified between the number of steps, cadence, as well as the mean and coefficient of variation of step width calculated from Zeno and video using 2D pose-estimation libraries. We noted that our automated heel-strike annotation method struggled to identify short steps. Conclusion: Gait features calculated from 2D joint trajectories are more strongly correlated with the Zeno than analogous gait features calculated from ROMP. Based on our analysis, videos processed with 2D pose-estimation libraries can be used for longitudinal gait monitoring in individuals with PD. Future work will seek to improve the prediction of gait features using a comprehensive machine learning model to predict gait features directly from color video without relying on intermediate extraction of joint trajectories.

Prediction of Parkinsonian Gait in Older Adults with Dementia using Joint Trajectories and Gait Features from 2D Video. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2021;2021:5700-5703. [PMID: 34892415 DOI: 10.1109/embc46164.2021.9630563] [Citation(s) in RCA: 0] [Impact Index Per Article: 0]

Older adults with dementia have a high risk of developing drug-induced parkinsonism; however, formal clinical gait assessments are too infrequent to capture fluctuations in their gait. Camera-based human pose estimation and tracking provides a means to frequently monitor gait in nonclinical settings. In this study, 2160 walking bouts from 49 participants were recorded using a ceiling-mounted camera. Recorded color videos were processed using AlphaPose to obtain 2D joint trajectories of the participant as they were walking down a hallway of the unit. A subset of 324 walking bouts from 14 participants were annotated with clinical scores of parkinsonism on the Unified Parkinson's Disease Rating Scale (UPDRS)-gait scale. Linear, random forest, and ordinal logistic regression models were evaluated for regression to UPDRS-gait scores using engineered 2D gait features calculated from the AlphaPose joint trajectories. Additionally, spatial temporal graph convolutional networks (ST-GCNs) were trained to predict UPDRS-gait scores from joint trajectories and gait features using a two-stage training scheme (self-supervised pretraining stage on all walks followed by a finetuning stage on labelled walks). All models were trained using leave-one-subject-out cross-validation to simulate testing on previously unseen participants. The macro-averaged F1-score was 0.333 for the best model operating on only gait features and 0.372 for the top ST-GCN model that used both joint trajectories and gait features as input. When accepting predicted scores that were only off by at most 1 point on the UPDRS-gait scale, the accuracy of the model that only used gait features was 82.8%, while the model that also used joint trajectories had an accuracy of 94.2%.Clinical Relevance- The combination of gait features and joint trajectories capture parkinsonian qualities in gait better than either group of data individually.

Early Detection of Low Cognitive Scores from Dual-task Performance Data Using a Spatio-temporal Graph Convolutional Neural Network

Detecting low cognitive scores at an early stage is important for delaying the progress of dementia. Investigations of early-stage detection have employed automatic assessment using dual-task (i.e., performing two different tasks simultaneously). However, current approaches to dual-task-based detection are based on either simple features or limited motion information, which degrades the detection accuracy. To address this problem, we proposed a framework that uses graph convolutional networks to extract spatio-temporal features from dual-task performance data. Moreover, to make the proposed method robust against data imbalance, we devised a loss function that directly optimizes the summation of the sensitivity and specificity of the detection of low cognitive scores (i.e., score≤ 23 or score≤ 27). Our evaluation is based on 171 subjects from 6 different senior citizens' facilities. Our experimental results demonstrated that the proposed algorithm considerably outperforms the previous standard with respect to both the sensitivity and specificity of the detection of low cognitive scores.

Quantifying Parkinson's disease motor severity under uncertainty using MDS-UPDRS videos

Parkinson's disease (PD) is a brain disorder that primarily affects motor function, leading to slow movement, tremor, and stiffness, as well as postural instability and difficulty with walking/balance. The severity of PD motor impairments is clinically assessed by part III of the Movement Disorder Society Unified Parkinson's Disease Rating Scale (MDS-UPDRS), a universally-accepted rating scale. However, experts often disagree on the exact scoring of individuals. In the presence of label noise, training a machine learning model using only scores from a single rater may introduce bias, while training models with multiple noisy ratings is a challenging task due to the inter-rater variabilities. In this paper, we introduce an ordinal focal neural network to estimate the MDS-UPDRS scores from input videos, to leverage the ordinal nature of MDS-UPDRS scores and combat class imbalance. To handle multiple noisy labels per exam, the training of the network is regularized via rater confusion estimation (RCE), which encodes the rating habits and skills of raters via a confusion matrix. We apply our pipeline to estimate MDS-UPDRS test scores from their video recordings including gait (with multiple Raters, R=3) and finger tapping scores (single rater). On a sizable clinical dataset for the gait test (N=55), we obtained a classification accuracy of 72% with majority vote as ground-truth, and an accuracy of ∼84% of our model predicting at least one of the raters' scores. Our work demonstrates how computer-assisted technologies can be used to track patients and their motor impairments, even when there is uncertainty in the clinical ratings. The latest version of the code will be available at https://github.com/mlu355/PD-Motor-Severity-Estimation.

Concurrent validity of human pose tracking in video for measuring gait parameters in older adults: a preliminary analysis with multiple trackers, viewing angles, and walking directions

Background

Many of the available gait monitoring technologies are expensive, require specialized expertise, are time consuming to use, and are not widely available for clinical use. The advent of video-based pose tracking provides an opportunity for inexpensive automated analysis of human walking in older adults using video cameras. However, there is a need to validate gait parameters calculated by these algorithms against gold standard methods for measuring human gait data in this population.

Methods

We compared quantitative gait variables of 11 older adults (mean age = 85.2) calculated from video recordings using three pose trackers (AlphaPose, OpenPose, Detectron) to those calculated from a 3D motion capture system. We performed comparisons for videos captured by two cameras at two different viewing angles, and viewed from the front or back. We also analyzed the data when including gait variables of individual steps of each participant or each participant’s averaged gait variables.

Results

Our findings revealed that, i) temporal (cadence and step time), but not spatial and variability gait measures (step width, estimated margin of stability, coefficient of variation of step time and width), calculated from the video pose tracking algorithms correlate significantly to that of motion capture system, and ii) there are minimal differences between the two camera heights, and walks viewed from the front or back in terms of correlation of gait variables, and iii) gait variables extracted from AlphaPose and Detectron had the highest agreement while OpenPose had the lowest agreement.

Conclusions

There are important opportunities to evaluate models capable of 3D pose estimation in video data, improve the training of pose-tracking algorithms for older adult and clinical populations, and develop video-based 3D pose trackers specifically optimized for quantitative gait measurement.

A Clinically Interpretable Computer-Vision Based Method for Quantifying Gait in Parkinson's Disease

Gait is a core motor function and is impaired in numerous neurological diseases, including Parkinson's disease (PD). Treatment changes in PD are frequently driven by gait assessments in the clinic, commonly rated as part of the Movement Disorder Society (MDS) Unified PD Rating Scale (UPDRS) assessment (item 3.10). We proposed and evaluated a novel approach for estimating severity of gait impairment in Parkinson's disease using a computer vision-based methodology. The system we developed can be used to obtain an estimate for a rating to catch potential errors, or to gain an initial rating in the absence of a trained clinician-for example, during remote home assessments. Videos (n=729) were collected as part of routine MDS-UPDRS gait assessments of Parkinson's patients, and a deep learning library was used to extract body key-point coordinates for each frame. Data were recorded at five clinical sites using commercially available mobile phones or tablets, and had an associated severity rating from a trained clinician. Six features were calculated from time-series signals of the extracted key-points. These features characterized key aspects of the movement including speed (step frequency, estimated using a novel Gamma-Poisson Bayesian model), arm swing, postural control and smoothness (or roughness) of movement. An ordinal random forest classification model (with one class for each of the possible ratings) was trained and evaluated using 10-fold cross validation. Step frequency point estimates from the Bayesian model were highly correlated with manually labelled step frequencies of 606 video clips showing patients walking towards or away from the camera (Pearson's r=0.80, p<0.001). Our classifier achieved a balanced accuracy of 50% (chance = 25%). Estimated UPDRS ratings were within one of the clinicians' ratings in 95% of cases. There was a significant correlation between clinician labels and model estimates (Spearman's ρ=0.52, p<0.001). We show how the interpretability of the feature values could be used by clinicians to support their decision-making and provide insight into the model's objective UPDRS rating estimation. The severity of gait impairment in Parkinson's disease can be estimated using a single patient video, recorded using a consumer mobile device and within standard clinical settings; i.e., videos were recorded in various hospital hallways and offices rather than gait laboratories. This approach can support clinicians during routine assessments by providing an objective rating (or second opinion), and has the potential to be used for remote home assessments, which would allow for more frequent monitoring.

Sparse Adaptive Graph Convolutional Network for Leg Agility Assessment in Parkinson's Disease

Motor disorder is a typical symptom of Parkinson's disease (PD). Neurologists assess the severity of PD motor symptoms using the clinical rating scale, i.e., MDS-UPDRS. However, this assessment method is time-consuming and easily affected by the perception difference of assessors. In the recent outbreak of coronavirus disease 2019, telemedicine for PD has become extremely urgent for clinical practice. To solve these problems, we developed an automated and objective assessment method of the leg agility task in the MDS-UPDRS using videos and a graph neural network. In this study, a sparse adaptive graph convolutional network (SA-GCN) was proposed to achieve fine-grained quantitative assessment of skeleton sequences extracted from videos. Specifically, the sparse adaptive graph convolutional unit with a prior knowledge constraint was proposed to perform adaptive spatial modeling of physical and logical dependency for skeleton sequences, thus achieving the sparse modeling of the discriminative spatial relationships. Subsequently, a temporal context module was introduced to construct the remote context dependency in the temporal dimension, hence determining the global changes of the task. A multi-domain attention learning module was also developed to integrate the static spatial features and dynamic temporal features, and then to emphasize the salient feature selection in the channel domain, thereby capturing the multi-domain fine-grained information. Finally, the evaluation results using a dataset with 148 patients and 870 samples confirmed the effectiveness and reliability of our scheme, and the method outperformed other related state-of-the-art methods. Our contactless method provides a new potential tool for automated PD assessment and telemedicine.

Video-Based Analyses of Parkinson's Disease Severity: A Brief Review

Remote and objective assessment of the motor symptoms of Parkinson's disease is an area of great interest particularly since the COVID-19 crisis emerged. In this paper, we focus on a) the challenges of assessing motor severity via videos and b) the use of emerging video-based Artificial Intelligence (AI)/Machine Learning techniques to quantitate human movement and its potential utility in assessing motor severity in patients with Parkinson's disease. While we conclude that video-based assessment may be an accessible and useful way of monitoring motor severity of Parkinson's disease, the potential of video-based AI to diagnose and quantify disease severity in the clinical context is dependent on research with large, diverse samples, and further validation using carefully considered performance standards.