Human performance in three-hands tasks

Smartphone automated motor and speech analysis for early detection of Alzheimer's disease and Parkinson's disease: Validation of TapTalk across 20 different devices

INTRODUCTION

Smartphones are proving useful in assessing movement and speech function in Alzheimer's disease and other neurodegenerative conditions. Valid outcomes across different smartphones are needed before population-level tests are deployed. This study introduces the TapTalk protocol, a novel app designed to capture hand and speech function and validate it in smartphones against gold-standard measures.

METHODS

Twenty different smartphones collected video data from motor tests and audio data from speech tests. Features were extracted using Google Mediapipe (movement) and Python audio analysis packages (speech). Electromagnetic sensors (60 Hz) and a microphone acquired simultaneous movement and voice data, respectively.

RESULTS

TapTalk video and audio outcomes were comparable to gold-standard data: 90.3% of video, and 98.3% of audio, data recorded tapping/speech frequencies within ± 1 Hz of the gold-standard measures.

DISCUSSION

Validation of TapTalk across a range of devices is an important step in the development of smartphone-based telemedicine and was achieved in this study.

Highlights

TapTalk evaluates hand motor and speech functions across a wide range of smartphones.Data showed 90.3% motor and 98.3% speech accuracy within +/-1 Hz of gold standards.Validation advances smartphone-based telemedicine for neurodegenerative diseases.

Development of a smartphone screening test for preclinical Alzheimer's disease and validation across the dementia continuum

BACKGROUND

Dementia prevalence is predicted to triple to 152 million globally by 2050. Alzheimer's disease (AD) constitutes 70% of cases. There is an urgent need to identify individuals with preclinical AD, a 10-20-year period of progressive brain pathology without noticeable cognitive symptoms, for targeted risk reduction. Current tests of AD pathology are either too invasive, specialised or expensive for population-level assessments. Cognitive tests are normal in preclinical AD. Emerging evidence demonstrates that movement analysis is sensitive to AD across the disease continuum, including preclinical AD. Our new smartphone test, TapTalk, combines analysis of hand and speech-like movements to detect AD risk. This study aims to [1] determine which combinations of hand-speech movement data most accurately predict preclinical AD [2], determine usability, reliability, and validity of TapTalk in cognitively asymptomatic older adults and [3], prospectively validate TapTalk in older adults who have cognitive symptoms against cognitive tests and clinical diagnoses of Mild Cognitive Impairment and AD dementia.

METHODS

Aim 1 will be addressed in a cross-sectional study of at least 500 cognitively asymptomatic older adults who will complete computerised tests comprising measures of hand motor control (finger tapping) and oro-motor control (syllabic diadochokinesis). So far, 1382 adults, mean (SD) age 66.20 (7.65) years, range 50-92 (72.07% female) have been recruited. Motor measures will be compared to a blood-based AD biomarker, phosphorylated tau 181 to develop an algorithm that classifies preclinical AD risk. Aim 2 comprises three sub-studies in cognitively asymptomatic adults: (i) a cross-sectional study of 30-40 adults to determine the validity of data collection from different types of smartphones, (ii) a prospective cohort study of 50-100 adults ≥ 50 years old to determine usability and test-retest reliability, and (iii) a prospective cohort study of ~1,000 adults ≥ 50 years old to validate against cognitive measures. Aim 3 will be addressed in a cross-sectional study of ~200 participants with cognitive symptoms to validate TapTalk against Montreal Cognitive Assessment and interdisciplinary consensus diagnosis.

DISCUSSION

This study will establish the precision of TapTalk to identify preclinical AD and estimate risk of cognitive decline. If accurate, this innovative smartphone app will enable low-cost, accessible screening of individuals for AD risk. This will have wide applications in public health initiatives and clinical trials.

TRIAL REGISTRATION

ClinicalTrials.gov identifier: NCT06114914, 29 October 2023. Retrospectively registered.

Human motor augmentation with an extra robotic arm without functional interference

Extra robotic arms (XRAs) are gaining interest in neuroscience and robotics, offering potential tools for daily activities. However, this compelling opportunity poses new challenges for sensorimotor control strategies and human-machine interfaces (HMIs). A key unsolved challenge is allowing users to proficiently control XRAs without hindering their existing functions. To address this, we propose a pipeline to identify suitable HMIs given a defined task to accomplish with the XRA. Following such a scheme, we assessed a multimodal motor HMI based on gaze detection and diaphragmatic respiration in a purposely designed modular neurorobotic platform integrating virtual reality and a bilateral upper limb exoskeleton. Our results show that the proposed HMI does not interfere with speaking or visual exploration and that it can be used to control an extra virtual arm independently from the biological ones or in coordination with them. Participants showed significant improvements in performance with daily training and retention of learning, with no further improvements when artificial haptic feedback was provided. As a final proof of concept, naïve and experienced participants used a simplified version of the HMI to control a wearable XRA. Our analysis indicates how the presented HMI can be effectively used to control XRAs. The observation that experienced users achieved a success rate 22.2% higher than that of naïve users, combined with the result that naïve users showed average success rates of 74% when they first engaged with the system, endorses the viability of both the virtual reality-based testing and training and the proposed pipeline.

Can Training Make Three Arms Better Than Two Heads for Trimanual Coordination?

Supernumerary effectors have been proposed to enable users to perform tasks alone that normally require assistance from a partner. While various supernumerary robotic limbs have been developed in the last decade, the capability of users to operate them effectively has not yet been proven. Here we tested whether users (i) can complete a task that requires simultaneous and fine control of three effectors, and (ii) can be trained to do so with similar or superior performance as through collaboration with a human partner. As in previous studies, initial augmented capability was less than that of working with a partner. However, one hour of dedicated solo trimanual training across three days significantly increased task performance, so that participants became able to perform trimanual control alone as well as or better than they could with a new partner. This shows the viability of augmentation systems for applications such as in robotic surgery or industrial assembly, which can be further validated on real tasks with physical systems.

A virtual reality platform to evaluate the effects of supernumerary limbs' appearance

Supernumerary robot limbs (SL) can expand the ability of users by increasing the number of degrees of freedom that they control. While several SLs have been designed and tested on human participants, the effect of the limb's appearance on the user's acceptance, embodiment and device usage is not yet understood. We developed a virtual reality platform with a three-arm avatar that enabled us to systematically investigate the effect of the supernumerary limb's appearance on their perception and motion control performance. A pilot study with 14 participants exhibited similar performance, workload and preference in human-like or robot-like appearance with a trend of preference for the robotic appearance.

Redundancy Resolution in Trimanual vs. Bimanual Tracking Tasks

Supernumerary limbs promise to allow users to perform complex tasks that would otherwise require the actions of teams. However, how the user's capability for multimanual coordination compares to bimanual coordination, and how the motor system decides to configure its limb contributions given task redundancy is unclear. We conducted bimanual and trimanual (with the foot as a third-hand controller) virtual reality visuomotor tracking experiments to study how 32 healthy participants changed their limb coordination in response to uninstructed cursor mapping changes. This used a shared cursor mapped to the average limbs' position for different limb combinations. The results show that most participants correctly identified the different mappings during bimanual tracking, and accordingly minimized task-irrelevant motion. Instead during trimanual coordination, participants consistently moved all three limbs concurrently, showing weaker ipsilateral hand-foot coordination. These findings show how redundancy resolution and the resulting coordination patterns differ between similar bimanual and trimanual tasks. Further research is needed to consider the effect of learning on coordination behaviour.

Comparing end-effector position and joint angle feedback for online robotic limb tracking

Somatosensation greatly increases the ability to control our natural body. This suggests that supplementing vision with haptic sensory feedback would also be helpful when a user aims at controlling a robotic arm proficiently. However, whether the position of the robot and its continuous update should be coded in a extrinsic or intrinsic reference frame is not known. Here we compared two different supplementary feedback contents concerning the status of a robotic limb in 2-DoFs configuration: one encoding the Cartesian coordinates of the end-effector of the robotic arm (i.e., Task-space feedback) and another and encoding the robot joints angles (i.e., Joint-space feedback). Feedback was delivered to blindfolded participants through vibrotactile stimulation applied on participants' leg. After a 1.5-hour training with both feedbacks, participants were significantly more accurate with Task compared to Joint-space feedback, as shown by lower position and aiming errors, albeit not faster (i.e., similar onset delay). However, learning index during training was significantly higher in Joint space feedback compared to Task-space feedback. These results suggest that Task-space feedback is probably more intuitive and more suited for activities which require short training sessions, while Joint space feedback showed potential for long-term improvement. We speculate that the latter, despite performing worse in the present work, might be ultimately more suited for applications requiring long training, such as the control of supernumerary robotic limbs for surgical robotics, heavy industrial manufacturing, or more generally, in the context of human movement augmentation.

Human Performance of Three Hands in Unimanual, Bimanual and Trimanual Tasks

Trimanual operation using a robotic supernumerary limb is a new and challenging mechanism for human operators that could enable a single user to perform tasks requiring more than two hands. Foot-controlled interfaces have previously proven able to be intuitively controlled, enabling simple tasks to be performed. However, the effect of going from unimanual to bimanual and then to trimanual tasks on subjects performance and coordination is not well understood. In this paper, unimanual, bimanual and trimanual teleoperation tasks were performed in a virtual reality scene to evaluate the impact of extending to trimanual actions. 15 participants were required to move their limbs together in a coordinated reaching activity. The results show that the addition of another hand resulted in an increase in operating time, where the time increased in going from unimanual to bimanual operation and then increased further when going from bimanual to trimanual. Moreover, the success rate for performing bimanual and trimanual tasks was strongly influenced by the subject's performance in ipsilateral hand-foot activities, where the ipsilateral combination had a lower success rate than contralateral limbs. The addition of a hand did not affect any two-hand coordination rate and even in some cases reduced coordination deviations. Clinical relevance - This work can contribute to build efficient training and learning framework on human multiple limbs motion control and coordination for both rehabilitation and augmentation.

Principles of human movement augmentation and the challenges in making it a reality

Augmenting the body with artificial limbs controlled concurrently to one's natural limbs has long appeared in science fiction, but recent technological and neuroscientific advances have begun to make this possible. By allowing individuals to achieve otherwise impossible actions, movement augmentation could revolutionize medical and industrial applications and profoundly change the way humans interact with the environment. Here, we construct a movement augmentation taxonomy through what is augmented and how it is achieved. With this framework, we analyze augmentation that extends the number of degrees-of-freedom, discuss critical features of effective augmentation such as physiological control signals, sensory feedback and learning as well as application scenarios, and propose a vision for the field.

Simultaneous control of natural and extra degrees of freedom by isometric force and electromyographic activity in the muscle-to-force null space

OBJECTIVE

Muscle activation patterns in the muscle-to-force null space, i.e., patterns that do not generate task-relevant forces, may provide an opportunity for motor augmentation by allowing to control additional end-effectors simultaneously to natural limbs. Here we tested the feasibility of muscular null space control for augmentation by assessing simultaneous control of natural and extra degrees of freedom.

APPROACH

We instructed eight participants to control translation and rotation of a virtual 3D end-effector by simultaneous generation of isometric force at the hand and null space activity extracted in real-time from the electromyographic signals recorded from 15 shoulder and arm muscles. First, we identified the null space components that each participant could control more naturally by voluntary co-contraction. Then, participants performed several blocks of a reaching and holding task. They displaced an ellipsoidal cursor to reach one of nine targets by generating force, and simultaneously rotated the cursor to match the target orientation by activating null space components. We developed an information-theoretic metric, an index of difficulty defined as the sum of a spatial and a temporal term, to assess individual null space control ability for both reaching and holding.

MAIN RESULTS

On average, participants could reach the targets in most trials already in the first block (72%) and they improved with practice (maximum 93%) but holding performance remained lower (maximum 43%). As there was a high inter-individual variability in performance, we performed a simulation with different spatial and temporal task conditions to estimate those for which each individual participants would have performed best.

SIGNIFICANCE

Muscular null space control is feasible and may be used to control additional virtual or robotics end-effectors. However, decoding of motor commands must be optimized according to individual null space control ability.

Perception and Performance of Electrical Stimulation for Proprioception

Proprioception, yielding awareness of the body's position and motion in space, is typically lacking in prostheses and supernumerary limbs. Electrical stimulation is one technique that may provide these devices with proprioception. This paper first investigates how the modalities of electrotactile cues, such as frequency and intensity, are perceived. Using the results, we designed and compared several comfortable and perceptible feedback mappings for spatial cues. Two experiments were conducted using a 16-electrode bracelet worn above the elbow to provide electrical stimuli. We found that subjects could localize the stimulating electrode with a precision of ±1 electrode (110 mm) in all feedback conditions. Moreover, within the range of pulse intensities perceived as comfortable, the participants' performance was more sensitive to changes in frequency than in intensity. The highest performance was obtained for the condition which increased both intensity and frequency with radial distance. These results suggest that electrical stimulation can be used for artificial proprioceptive feedback, which can ensure a comfortable and intuitive interaction and provides high spatial accuracy.