Video-Based Facial Movement Analysis in the Assessment of Bulbar Amyotrophic Lateral Sclerosis: Clinical Validation

Multimodal speech biomarkers for remote monitoring of ALS disease progression

Amyotrophic lateral sclerosis (ALS) is a progressive neurodegenerative disease that severely impacts affected persons' speech and motor functions, yet early detection and tracking of disease progression remain challenging. The current gold standard for monitoring ALS progression, the ALS functional rating scale - revised (ALSFRS-R), is based on subjective ratings of symptom severity, and may not capture subtle but clinically meaningful changes due to a lack of granularity. Multimodal speech measures which can be automatically collected from patients in a remote fashion allow us to bridge this gap because they are continuous-valued and therefore, potentially more granular at capturing disease progression. Here we investigate the responsiveness and sensitivity of multimodal speech measures in persons with ALS (pALS) collected via a remote patient monitoring platform in an effort to quantify how long it takes to detect a clinically-meaningful change associated with disease progression. We recorded audio and video from 278 participants and automatically extracted multimodal speech biomarkers (acoustic, orofacial, linguistic) from the data. We find that the timing alignment of pALS speech relative to a canonical elicitation of the same prompt and the number of words used to describe a picture are the most responsive measures at detecting such change in both pALS with bulbar (n = 36) and non-bulbar onset (n = 107). Interestingly, the responsiveness of these measures is stable even at small sample sizes. We further found that certain speech measures are sensitive enough to track bulbar decline even when there is no patient-reported clinical change, i.e. the ALSFRS-R speech score remains unchanged at 3 out of a total possible score of 4. The findings of this study have the potential to facilitate improved, accelerated and cost-effective clinical trials and care.

What the trained eye cannot see: Quantitative kinematics and machine learning detect movement deficits in early-stage Parkinson's disease from videos

BACKGROUND

Evaluation of disease severity in Parkinson's disease (PD) relies on motor symptoms quantification. However, during early-stage PD, these symptoms are subtle and difficult to quantify by experts, which might result in delayed diagnosis and suboptimal disease management.

OBJECTIVE

To evaluate the use of videos and machine learning (ML) for automatic quantification of motor symptoms in early-stage PD.

METHODS

We analyzed videos of three movement tasks-Finger Tapping, Hand Movement, and Leg Agility- from 26 aged-matched healthy controls and 31 early-stage PD patients. Utilizing ML algorithms for pose estimation we extracted kinematic features from these videos and trained three classification models based on left and right-side movements, and right/left symmetry. The models were trained to differentiate healthy controls from early-stage PD from videos.

RESULTS

Combining left side, right side, and symmetry features resulted in a PD detection accuracy of 79 % from Finger Tap videos, 75 % from Hand Movement videos, 79 % from Leg Agility videos, and 86 % when combining the three tasks using a soft voting approach. In contrast, the classification accuracy varied between 40 % and 72 % when the movement side or symmetry were not considered.

CONCLUSIONS

Our methodology effectively differentiated between early-stage PD and healthy controls using videos of standardized motor tasks by integrating kinematic analyses of left-side, right-side, and bilateral symmetry movements. These results demonstrate that ML can detect movement deficits in early-stage PD from videos. This technology is easy-to-use, highly scalable, and has the potential to improve the management and quantification of motor symptoms in early-stage PD.

Multimodal Technologies for Remote Assessment of Neurological and Mental Health

PURPOSE

Automated remote assessment and monitoring of patients' neurological and mental health is increasingly becoming an essential component of the digital clinic and telehealth ecosystem, especially after the COVID-19 pandemic. This review article reviews various modalities of health information that are useful for developing such remote clinical assessments in the real world at scale.

APPROACH

We first present an overview of the various modalities of health information-speech acoustics, natural language, conversational dynamics, orofacial or full body movement, eye gaze, respiration, cardiopulmonary, and neural-which can each be extracted from various signal sources-audio, video, text, or sensors. We further motivate their clinical utility with examples of how information from each modality can help us characterize how different disorders affect different aspects of patients' spoken communication. We then elucidate the advantages of combining one or more of these modalities toward a more holistic, informative, and robust assessment.

FINDINGS

We find that combining multiple modalities of health information allows for improved scientific interpretability, improved performance on downstream health applications such as early detection and progress monitoring, improved technological robustness, and improved user experience. We illustrate how these principles can be leveraged for remote clinical assessment at scale using a real-world case study of the Modality assessment platform.

CONCLUSION

This review article motivates the combination of human-centric information from multiple modalities to measure various aspects of patients' health, arguing that remote clinical assessment that integrates this complementary information can be more effective and lead to better clinical outcomes than using any one data stream in isolation.

Multimodal Speech Biomarkers for Remote Monitoring of ALS Disease Progression

Amyotrophic lateral sclerosis (ALS) is a progressive neurodegenerative disease that severely impacts affected persons' speech and motor functions, yet early detection and tracking of disease progression remain challenging. The current gold standard for monitoring ALS progression, the ALS functional rating scale - revised (ALSFRS-R), is based on subjective ratings of symptom severity, and may not capture subtle but clinically meaningful changes due to a lack of granularity. Multimodal speech measures which can be automatically collected from patients in a remote fashion allow us to bridge this gap because they are continuous-valued and therefore, potentially more granular at capturing disease progression. Here we investigate the responsiveness and sensitivity of multimodal speech measures in persons with ALS (pALS) collected via a remote patient monitoring platform in an effort to quantify how long it takes to detect a clinically-meaningful change associated with disease progression. We recorded audio and video from 278 participants and automatically extracted multimodal speech biomarkers (acoustic, orofacial, linguistic) from the data. We find that the timing alignment of pALS speech relative to a canonical elicitation of the same prompt and the number of words used to describe a picture are the most responsive measures at detecting such change in both pALS with bulbar (n = 36) and non-bulbar onset (n = 107). Interestingly, the responsiveness of these measures is stable even at small sample sizes. We further found that certain speech measures are sensitive enough to track bulbar decline even when there is no patient-reported clinical change, i.e. the ALSFRS-R speech score remains unchanged at 3 out of a total possible score of 4. The findings of this study have the potential to facilitate improved, accelerated and cost-effective clinical trials and care.

Characterizing Disease Progression in Parkinson's Disease from Videos of the Finger Tapping Test

INTRODUCTION

Parkinson's disease (PD) is characterized by motor symptoms whose progression is typically assessed using clinical scales, namely the Movement Disorder Society-Unified Parkinson's Disease Rating Scale (MDS-UPDRS). Despite its reliability, the scale is bounded by a 5-point scale that limits its ability to track subtle changes in disease progression and is prone to subjective interpretations. We aimed to develop an automated system to objectively quantify motor symptoms in PD using Machine Learning (ML) algorithms to analyze videos and capture nuanced features of disease progression.

METHODS

We analyzed videos of the Finger Tapping test, a component of the MDS-UPDRS, from 24 healthy controls and 66 PD patients using ML algorithms for hand pose estimation. We computed multiple movement features related to bradykinesia from videos and employed a novel tiered classification approach to predict disease severity that employed different features according to severity. We compared our video-based disease severity prediction approach against other approaches recently introduced in the literature.

RESULTS

Traditional kinematics features such as amplitude and velocity changed linearly with disease severity, while other non-traditional features displayed non-linear trends. The proposed disease severity prediction approach demonstrated superior accuracy in detecting PD and distinguishing between different levels of disease severity when compared to existing approaches.

A multimodal approach to automated hierarchical assessment of bulbar involvement in amyotrophic lateral sclerosis

Introduction

As a hallmark feature of amyotrophic lateral sclerosis (ALS), bulbar involvement leads to progressive declines of speech and swallowing functions, significantly impacting social, emotional, and physical health, and quality of life. Standard clinical tools for bulbar assessment focus primarily on clinical symptoms and functional outcomes. However, ALS is known to have a long, clinically silent prodromal stage characterized by complex subclinical changes at various levels of the bulbar motor system. These changes accumulate over time and eventually culminate in clinical symptoms and functional declines. Detection of these subclinical changes is critical, both for mechanistic understanding of bulbar neuromuscular pathology and for optimal clinical management of bulbar dysfunction in ALS. To this end, we developed a novel multimodal measurement tool based on two clinically readily available, noninvasive instruments-facial surface electromyography (sEMG) and acoustic techniques-to hierarchically assess seven constructs of bulbar/speech motor control at the neuromuscular and acoustic levels. These constructs, including prosody, pause, functional connectivity, amplitude, rhythm, complexity, and regularity, are both mechanically and clinically relevant to bulbar involvement.

Methods

Using a custom-developed, fully automated data analytic algorithm, a variety of features were extracted from the sEMG and acoustic recordings of a speech task performed by 13 individuals with ALS and 10 neurologically healthy controls. These features were then factorized into 10 composite outcome measures using confirmatory factor analysis. Statistical and machine learning techniques were applied to these composite outcome measures to evaluate their reliability (internal consistency), validity (concurrent and construct), and efficacy for early detection and progress monitoring of bulbar involvement in ALS.

Results

The composite outcome measures were demonstrated to (1) be internally consistent and structurally valid in measuring the targeted constructs; (2) hold concurrent validity with the existing clinical and functional criteria for bulbar assessment; and (3) outperform the outcome measures obtained from each constituent modality in differentiating individuals with ALS from healthy controls. Moreover, the composite outcome measures combined demonstrated high efficacy for detecting subclinical changes in the targeted constructs, both during the prodromal stage and during the transition from prodromal to symptomatic stages.

Discussion

The findings provided compelling initial evidence for the utility of the multimodal measurement tool for improving early detection and progress monitoring of bulbar involvement in ALS, which have important implications in facilitating timely access to and delivery of optimal clinical care of bulbar dysfunction.

Improving Dysarthric Speech Segmentation With Emulated and Synthetic Augmentation

Acoustic features extracted from speech can help with the diagnosis of neurological diseases and monitoring of symptoms over time. Temporal segmentation of audio signals into individual words is an important pre-processing step needed prior to extracting acoustic features. Machine learning techniques could be used to automate speech segmentation via automatic speech recognition (ASR) and sequence to sequence alignment. While state-of-the-art ASR models achieve good performance on healthy speech, their performance significantly drops when evaluated on dysarthric speech. Fine-tuning ASR models on impaired speech can improve performance in dysarthric individuals, but it requires representative clinical data, which is difficult to collect and may raise privacy concerns. This study explores the feasibility of using two augmentation methods to increase ASR performance on dysarthric speech: 1) healthy individuals varying their speaking rate and loudness (as is often used in assessments of pathological speech); 2) synthetic speech with variations in speaking rate and accent (to ensure more diverse vocal representations and fairness). Experimental evaluations showed that fine-tuning a pre-trained ASR model with data from these two sources outperformed a model fine-tuned only on real clinical data and matched the performance of a model fine-tuned on the combination of real clinical data and synthetic speech. When evaluated on held-out acoustic data from 24 individuals with various neurological diseases, the best performing model achieved an average word error rate of 5.7% and a mean correct count accuracy of 94.4%. In segmenting the data into individual words, a mean intersection-over-union of 89.2% was obtained against manual parsing (ground truth). It can be concluded that emulated and synthetic augmentations can significantly reduce the need for real clinical data of dysarthric speech when fine-tuning ASR models and, in turn, for speech segmentation.

A preliminary study on self-care telemonitoring of dysarthria in spinal muscular atrophy

Spinal muscular atrophy (SMA) is a rare neuromuscular disease which may cause impairments in oro-facial musculature. Most of the individuals with SMA present bulbar signs such as flaccid dysarthria which mines their abilities to speak and, as consequence, their psychic balance. To support clinicians, recent work has demonstrated the feasibility of video-based techniques for assessing the oro-facial functions in patients with neurological disorders such as amyotrophic lateral sclerosis. However, no work has so far focused on automatic and quantitative monitoring of dysarthria in SMA. To overcome limitations this work's aim is to propose a cloud-based store-and-forward telemonitoring system for automatic and quantitative evaluation of oro-facial muscles in individuals with SMA. The system integrates a convolutional neural network (CNN) aimed at identifying the position of facial landmarks from video recordings acquired via a web application by an SMA patient.Clinical relevance- The proposed work is in the preliminary stage, but it represents the first step towards a better understanding of the bulbar-functions' evolution in patients with SMA.