Recent Developments in Prosthesis Sensors, Texture Recognition, and Sensory Stimulation for Upper Limb Prostheses

A Biomimetic Nociceptor Using Centrosymmetric Crystals for Machine Intelligence

Pain sensation is a crucial aspect of perception in the body. Force-activated nociceptors encode electrochemical signals and yield multilevel information of pain, thus enabling smart feedback. Inspired by the natural template, multi-dimensional mechano-sensing materials provide promising approaches for biomimetic nociceptors in intelligent terminals. However, the reliance on non-centrosymmetric crystals has narrowed the range of these materials. Here centrosymmetric crystal Cr3+ -doped zinc gallogermanate (ZGGO:Cr) with multi-dimensional mechano-sensing is reported, eliminating the limitation of crystal structure. Under forces, ZGGO:Cr generates electrical signals imitating those of neuronal systems, and produces luminescence for spatial mapping of mechanical stimuli, suggesting a path toward bionic pain perception. On that basis, a wireless biomimetic nociceptor system is developed and a smart pain reflex in a robotic hand and robot-assisted biopsy surgery of rat and dog is achieved.

Integration of proprioception in upper limb prostheses through non-invasive strategies: a review

Proprioception plays a key role in moving our body dexterously and effortlessly. Nevertheless, the majority of investigations evaluating the benefits of providing supplemental feedback to prosthetics users focus on delivering touch restitution. These studies evaluate the influence of touch sensation in an attempt to improve the controllability of current robotic devices. Contrarily, investigations evaluating the capabilities of proprioceptive supplemental feedback have yet to be comprehensively analyzed to the same extent, marking a major gap in knowledge within the current research climate. The non-invasive strategies employed so far to restitute proprioception are reviewed in this work. In the absence of a clearly superior strategy, approaches employing vibrotactile, electrotactile and skin-stretch stimulation achieved better and more consistent results, considering both kinesthetic and grip force information, compared with other strategies or any incidental feedback. Although emulating the richness of the physiological sensory return through artificial feedback is the primary hurdle, measuring its effects to eventually support the integration of cumbersome and energy intensive hardware into commercial prosthetic devices could represent an even greater challenge. Thus, we analyze the strengths and limitations of previous studies and discuss the possible benefits of coupling objective measures, like neurophysiological parameters, as well as measures of prosthesis embodiment and cognitive load with behavioral measures of performance. Such insights aim to provide additional and collateral outcomes to be considered in the experimental design of future investigations of proprioception restitution that could, in the end, allow researchers to gain a more detailed understanding of possibly similar behavioral results and, thus, support one strategy over another.

Recent advancements in multimodal human-robot interaction

Robotics have advanced significantly over the years, and human-robot interaction (HRI) is now playing an important role in delivering the best user experience, cutting down on laborious tasks, and raising public acceptance of robots. New HRI approaches are necessary to promote the evolution of robots, with a more natural and flexible interaction manner clearly the most crucial. As a newly emerging approach to HRI, multimodal HRI is a method for individuals to communicate with a robot using various modalities, including voice, image, text, eye movement, and touch, as well as bio-signals like EEG and ECG. It is a broad field closely related to cognitive science, ergonomics, multimedia technology, and virtual reality, with numerous applications springing up each year. However, little research has been done to summarize the current development and future trend of HRI. To this end, this paper systematically reviews the state of the art of multimodal HRI on its applications by summing up the latest research articles relevant to this field. Moreover, the research development in terms of the input signal and the output signal is also covered in this manuscript.

Object stiffness recognition and vibratory feedback without ad-hoc sensing on the Hannes prosthesis: A machine learning approach

Introduction

In recent years, hand prostheses achieved relevant improvements in term of both motor and functional recovery. However, the rate of devices abandonment, also due to their poor embodiment, is still high. The embodiment defines the integration of an external object - in this case a prosthetic device - into the body scheme of an individual. One of the limiting factors causing lack of embodiment is the absence of a direct interaction between user and environment. Many studies focused on the extraction of tactile information via custom electronic skin technologies coupled with dedicated haptic feedback, though increasing the complexity of the prosthetic system. Contrary wise, this paper stems from the authors' preliminary works on multi-body prosthetic hand modeling and the identification of possible intrinsic information to assess object stiffness during interaction.

Methods

Based on these initial findings, this work presents the design, implementation and clinical validation of a novel real-time stiffness detection strategy, without ad-hoc sensing, based on a Non-linear Logistic Regression (NLR) classifier. This exploits the minimum grasp information available from an under-sensorized and under-actuated myoelectric prosthetic hand, Hannes. The NLR algorithm takes as input motor-side current, encoder position, and reference position of the hand and provides as output a classification of the grasped object (no-object, rigid object, and soft object). This information is then transmitted to the user via vibratory feedback to close the loop between user control and prosthesis interaction. This implementation was validated through a user study conducted both on able bodied subjects and amputees.

Results

The classifier achieved excellent performance in terms of F1Score (94.93%). Further, the able-bodied subjects and amputees were able to successfully detect the objects' stiffness with a F1Score of 94.08% and 86.41%, respectively, by using our proposed feedback strategy. This strategy allowed amputees to quickly recognize the objects' stiffness (response time of 2.82 s), indicating high intuitiveness, and it was overall appreciated as demonstrated by the questionnaire. Furthermore, an embodiment improvement was also obtained as highlighted by the proprioceptive drift toward the prosthesis (0.7 cm).

FeetBack–Redirecting touch sensation from a prosthetic hand to the human foot

Introduction

Adding sensory feedback to myoelectric prosthetic hands was shown to enhance the user experience in terms of controllability and device embodiment. Often this is realized non-invasively by adding devices, such as actuators or electrodes, within the prosthetic shaft to deliver the desired feedback. However, adding a feedback system in the socket adds more weight, steals valuable space, and may interfere with myoelectric signals. To circumvent said drawbacks we tested for the first time if force feedback from a prosthetic hand could be redirected to another similarly sensitive part of the body: the foot.

Methods

We developed a vibrotactile insole that vibrates depending on the sensed force on the prosthetic fingers. This self-controlled clinical pilot trial included four experienced users of myoelectric prostheses. The participants solved two types of tasks with the artificial hands: 1) sorting objects depending on their plasticity with the feedback insole but without audio-visual feedback, and 2) manipulating fragile, heavy, and delicate objects with and without the feedback insole. The sorting task was evaluated with Goodman-Kruskal's gamma for ranked correlation. The manipulation tasks were assessed by the success rate.

Results

The results from the sorting task with vibrotactile feedback showed a substantial positive effect. The success rates for manipulation tasks with fragile and heavy objects were high under both conditions (feedback on or off, respectively). The manipulation task with delicate objects revealed inferior success with feedback in three of four participants.

Conclusion

We introduced a novel approach to touch sensation in myoelectric prostheses. The results for the sorting task and the manipulation tasks diverged. This is likely linked to the availability of various feedback sources. Our results for redirected feedback to the feet fall in line with previous similar studies that applied feedback to the residual arm.

Clinical trial registration

Name: Sensor Glove and Non-Invasive Vibrotactile Feedback Insole to Improve Hand Prostheses Functions and Embodiment (FeetBack). Date of registration: 23 April 2019. Date the first participant was enrolled: 3 September 2021. ClinicalTrials.gov Identifier: NCT03924310.

Physics-Informed Machine Learning Improves Detection of Head Impacts

In this work we present a new physics-informed machine learning model that can be used to analyze kinematic data from an instrumented mouthguard and detect impacts to the head. Monitoring player impacts is vitally important to understanding and protecting from injuries like concussion. Typically, to analyze this data, a combination of video analysis and sensor data is used to ascertain the recorded events are true impacts and not false positives. In fact, due to the nature of using wearable devices in sports, false positives vastly outnumber the true positives. Yet, manual video analysis is time-consuming. This imbalance leads traditional machine learning approaches to exhibit poor performance in both detecting true positives and preventing false negatives. Here, we show that by simulating head impacts numerically using a standard Finite Element head-neck model, a large dataset of synthetic impacts can be created to augment the gathered, verified, impact data from mouthguards. This combined physics-informed machine learning impact detector reported improved performance on test datasets compared to traditional impact detectors with negative predictive value and positive predictive values of 88 and 87% respectively. Consequently, this model reported the best results to date for an impact detection algorithm for American football, achieving an F₁ score of 0.95. In addition, this physics-informed machine learning impact detector was able to accurately detect true and false impacts from a test dataset at a rate of 90% and 100% relative to a purely manual video analysis workflow. Saving over 12 h of manual video analysis for a modest dataset, at an overall accuracy of 92%, these results indicate that this model could be used in place of, or alongside, traditional video analysis to allow for larger scale and more efficient impact detection in sports such as American Football.

Effective Evaluation of Finger Sensation Evoking by Non-invasive Stimulation for Sensory Function Recovery in Transradial Amputees

Synergetic recovery of both somatosensory and motor functions is highly desired by limb amputees to fully regain their lost limb abilities. The commercially available prostheses can restore the lost motor function in amputees but lack intuitive sensory feedback. The previous studies showed that electrical stimulation on the arm stump would be a promising approach to induce sensory information into the nervous system, enabling the possibility of realizing sensory feedback in limb prostheses. However, there are currently limited studies on the effective evaluation of the sensations evoked by transcutaneous electrical nerve stimulation (TENS). In this paper, a multichannel TENS platform was developed and the different stimulus patterns were designed to evoke stable finger sensations for a transradial amputee. Electroencephalogram (EEG) was recorded simultaneously during TENS on the arm stump, which was utilized to evaluate the evoked sensations. The experimental results revealed that different types of sensations on three phantom fingers could be stably evoked for the amputee by properly selecting TENS patterns. The analysis of the event-related potential (ERP) of EEG recordings further confirmed the evoked sensations, and ERP latencies and curve characteristics for different phantom fingers showed significant differences. This work may provide insight for an in-depth understanding of how somatosensation could be restored in limb amputees and offer technical support for the applications of non-invasive sensory feedback systems.

Development of a Low-Power Instrumented Mouthpiece for Directly Measuring Head Acceleration in American Football

Instrumented mouthpieces (IM) offer a means of measuring head impacts that occur in sport. Direct measurement of angular head kinematics is preferential for accuracy; however, existing IMs measure angular velocity and differentiate the measurement to calculate angular acceleration, which can limit bandwidth and consume more power. This study presents the development and validation of an IM that uses new, low-power accelerometers for direct measurement of linear and angular acceleration over a broad range of head impact conditions in American football. IM sensor accuracy for measuring six-degree-of-freedom head kinematics was assessed using two helmeted headforms instrumented with a custom-fit IM and reference sensor instrumentation. Head impacts were performed at 10 locations and 6 speeds representative of the on-field conditions associated with injurious and non-injurious impacts in American football. Sensor measurements from the IM were highly correlated with those from the reference instrumentation located at the maxilla and skull center of gravity. Based on pooled data across headform and impact location, R² ≥ 0.94, mean absolute error (AE) ≤ 7%, and mean relative impact angle ≤ 11° for peak linear and angular acceleration and angular velocity while R² ≥ 0.90 and mean AE ≤ 7% for kinematic-based injury metrics used in helmet tests.

Tactile Interaction Sensor with Millimeter Sensing Acuity

In this article we report on a 3 × 3 mm tactile interaction sensor that is able to simultaneously detect pressure level, pressure distribution, and shear force direction. The sensor consists of multiple mechanical switches under a conducting diaphragm. An external stimulus is measured by the deflection of the diaphragm and the arrangement of mechanical switches, resulting in low noise, high reliability, and high uniformity. Our sensor is able to detect tactile forces as small as ~50 mgf along with the direction of the shear force. It also distinguishes whether there is a normal pressure during slip motion. We also succeed in detecting the contact shape and the contact motion, demonstrating potential applications in robotics and remote input interfaces. Since our sensor has a simple structure and its function depends only on sensor dimensions, not on an active sensing material, in comparison with previous tactile sensors, our sensor shows high uniformity and reliability for an array-type integration.

Electrotactile Feedback for the Discrimination of Different Surface Textures Using a Microphone

Most commercial prosthetic hands lack closed-loop feedback, thus, a lot of research has been focusing on implementing sensory feedback systems to provide the user with sensory information during activities of daily living. This study evaluates the possibilities of using a microphone and electrotactile feedback to identify different textures. A condenser microphone was used as a sensor to detect the friction sound generated from the contact between different textures and the microphone. The generated signal was processed to provide a characteristic electrical stimulation presented to the participants. The main goal of the processing was to derive a continuous and intuitive transfer function between the microphone signal and stimulation frequency. Twelve able-bodied volunteers participated in the study, in which they were asked to identify the stroked texture (among four used in this study: Felt, sponge, silicone rubber, and string mesh) using only electrotactile feedback. The experiments were done in three phases: (1) Training, (2) with-feedback, (3) without-feedback. Each texture was stroked 20 times each during all three phases. The results show that the participants were able to differentiate between different textures, with a median accuracy of 85%, by using only electrotactile feedback with the stimulation frequency being the only variable parameter.

High Energy Side and Rear American Football Head Impacts Cause Obvious Performance Decrement on Video

The objective of this study was to compare head impact data acquired with an impact monitoring mouthguard (IMM) to the video-observed behavior of athletes' post-collision relative to their pre-collision behaviors. A total of n = 83 college and high school American football players wore the IMM and were video-recorded over 260 athlete-exposures. Ex-athletes and clinicians reviewed the video in a two-step process and categorized abnormal post-collision behaviors according to previously published Obvious Performance Decrement (OPD) definitions. Engineers qualitatively reviewed datasets to check head impact and non-head impact signal frequency and magnitude. The ex-athlete reviewers identified 2305 head impacts and 16 potential OPD impacts, 13 of which were separately categorized as Likely-OPD impacts by the clinical reviewers. All 13 Likely-OPD impacts were in the top 1% of impacts measured by the IMM (ranges 40-100 g, 3.3-7.0 m/s and 35-118 J) and 12 of the 13 impacts (92%) were to the side or rear of the head. These findings require confirmation in a larger data set before proposing any type of OPD impact magnitude or direction threshold exists. However, OPD cases in this study compare favorably with previously published impact monitoring studies in high school and college American football players that looked for OPD signs, impact magnitude and direction. Our OPD findings also compare well with NFL reconstruction studies for ranges of concussion and sub-concussive impact magnitudes in side/rear collisions, as well as prior theory, analytical models and empirical research that suggest a directional sensitivity to brain injury exists for single high-energy impacts.