Haptic-assistive technologies for audition and vision sensory disabilities

Static and Dynamic Hand Gestures: A Review of Techniques of Virtual Reality Manipulation

This review explores the historical and current significance of gestures as a universal form of communication with a focus on hand gestures in virtual reality applications. It highlights the evolution of gesture detection systems from the 1990s, which used computer algorithms to find patterns in static images, to the present day where advances in sensor technology, artificial intelligence, and computing power have enabled real-time gesture recognition. The paper emphasizes the role of hand gestures in virtual reality (VR), a field that creates immersive digital experiences through the Ma blending of 3D modeling, sound effects, and sensing technology. This review presents state-of-the-art hardware and software techniques used in hand gesture detection, primarily for VR applications. It discusses the challenges in hand gesture detection, classifies gestures as static and dynamic, and grades their detection difficulty. This paper also reviews the haptic devices used in VR and their advantages and challenges. It provides an overview of the process used in hand gesture acquisition, from inputs and pre-processing to pose detection, for both static and dynamic gestures.

Proportional sway-based electrotactile feedback improves lateral standing balance

Introduction

Plantar cutaneous augmentation is a promising approach in balance rehabilitation by enhancing motion-dependent sensory feedback. The effect of plantar cutaneous augmentation on balance has been mainly investigated in its passive form (e.g., textured insole) or on lower-limb amputees. In this study, we tested the effect of plantar cutaneous augmentation on balance in its active form (i.e., electrical stimulation) for individuals with intact limbs.

Methods

Ten healthy subjects participated in the study and were instructed to maintain their balance as long as possible on the balance board, with or without electrotactile feedback evoked on the medial side of the heel, synched with the lateral board sway. Electrotactile feedback was given in two different modes: 1) Discrete-mode E-stim as the stimulation on/off by a predefined threshold of lateral board sway and 2) Proportional-mode E-stim as the stimulation frequency proportional to the amount of lateral board sway. All subjects were distracted from the balancing task by the n-back counting task, to test subjects' balancing capability with minimal cognitive involvement.

Results

Proportional-mode E-stim, along with the n-back counting task, increased the balance time from 1.86 ± 0.03 s to 1.98 ± 0.04 s (p = 0.010). However, discrete-mode E-stim did not change the balance time (p = 0.669). Proportional-mode E-stim also increased the time duration per each swayed state (p = 0.035) while discrete-mode E-stim did not (p = 0.053).

Discussion

These results suggest that proportional-mode E-stim is more effective than discrete-mode E-stim on improving standing balance. It is perhaps because the proportional electrotactile feedback better mimics the natural tactile sensation of foot pressure than its discrete counterpart.

Enhancing human-human musical interaction through kinesthetic haptic feedback using wearable exoskeletons: theoretical foundations, validation scenarios, and limitations

In this perspective paper, we explore the use of haptic feedback to enhance human-human interaction during musical tasks. We start by providing an overview of the theoretical foundation that underpins our approach, which is rooted in the embodied music cognition framework, and by briefly presenting the concepts of action-perception loop, sensorimotor coupling and entrainment. Thereafter, we focus on the role of haptic information in music playing and we discuss the use of wearable technologies, namely lightweight exoskeletons, for the exchange of haptic information between humans. We present two experimental scenarios in which the effectiveness of this technology for enhancing musical interaction and learning might be validated. Finally, we briefly discuss some of the theoretical and pedagogical implications of the use of technologies for haptic communication in musical contexts, while also addressing the potential barriers to the widespread adoption of exoskeletons in such contexts.

Promoting Universal Equitable Accessibility: An Overview on the Impact of Assistive Technology in the UN, UNICEF, and WHO Web Portals

The number of people with disabilities and frailties who need support and assistance is increasing. Assistive technologies (ATs) are increasingly playing a central role in supporting people with disabilities and frailties. The study investigated the impact of the ATs on the websites of the UN, UNICEF, and WHO in terms of proposed activities and actions. The methodology proposed was based on two points of view: (1) A formal process to directly select elements in the institutional webs of the UN, UNICEF, and WHO. (2) A formal process for a complementary literature narrative review based on an umbrella review of Pubmed and Scopus. A standard checklist and a qualification process were applied. The outcome reported 35 documents from the direct search on the web and 19 systematic reviews for the complimentary literature overview. The direct search returned documents related to initiatives focused on the following: The tailoring of the ATs to a person based on international guidelines and specific monitoring initiatives of the AT introduction/access based on surveys both at the population and system/government level with the publication of the data/metadata in an observatory. Dissemination initiatives of both the culture of ATs (e.g., catalog, guidelines, reports, congresses) and of recommendations. The literature overview contributed more specifically to the use and effectiveness of categories of ATs. Both direct research and the literature overview have shown a consistent growth in interest in ATs. The initiatives of the UN, UNICEF, and WHO have been consistent with the institutional role and aimed at improving the diffusion of ATs through capillary monitoring, which is not free from obstacles, and a diffusion of the culture and rational use of ATs. The narrative review shows also the important role of research in monitoring the development, use, and effectiveness of devices, strategies, and support of international institutional initiatives. Important initiatives have been launched internationally on AT in terms of monitoring, dissemination, and improvement in access. However, it is necessary to consider and face the obstacles that limit these initiatives.

Human–Machine Interaction through Advanced Haptic Sensors: A Piezoelectric Sensory Glove with Edge Machine Learning for Gesture and Object Recognition

Human–machine interaction (HMI) refers to systems enabling communication between machines and humans. Systems for human–machine interfaces have advanced significantly in terms of materials, device design, and production methods. Energy supply units, logic circuits, sensors, and data storage units must be flexible, stretchable, undetectable, biocompatible, and self-healing to act as human–machine interfaces. This paper discusses the technologies for providing different haptic feedback of different natures. Notably, the physiological mechanisms behind touch perception are reported, along with a classification of the main haptic interfaces. Afterward, a comprehensive overview of wearable haptic interfaces is presented, comparing them in terms of cost, the number of integrated actuators and sensors, their main haptic feedback typology, and their future application. Additionally, a review of sensing systems that use haptic feedback technologies—specifically, smart gloves—is given by going through their fundamental technological specifications and key design requirements. Furthermore, useful insights related to the design of the next-generation HMI devices are reported. Lastly, a novel smart glove based on thin and conformable AlN (aluminum nitride) piezoelectric sensors is demonstrated. Specifically, the device acquires and processes the signal from the piezo sensors to classify performed gestures through an onboard machine learning (ML) algorithm. Then, the design and testing of the electronic conditioning section of AlN-based sensors integrated into the smart glove are shown. Finally, the architecture of a wearable visual-tactile recognition system is presented, combining visual data acquired by a micro-camera mounted on the user’s glass with the haptic ones provided by the piezoelectric sensors.

Controlling fine touch sensations with polymer tacticity and crystallinity

The friction generated between a finger and an object forms the mechanical stimuli behind fine touch perception. To control friction, and therefore tactile perception, current haptic devices typically rely on physical features like bumps or pins, but chemical and microscale morphology of surfaces could be harnessed to recreate a wider variety of tactile sensations. Here, we sought to develop a new way to create tactile sensations by relying on differences in microstructure as quantified by the degree of crystallinity in polymer films. To isolate crystallinity, we used polystyrene films with the same chemical formula and number averaged molecular weights, but which differed in tacticity and annealing conditions. These films were also sufficiently thin as to be rigid which minimized effects from bulk stiffness and had variations in roughness lower than detectable by humans. To connect crystallinity to human perception, we performed mechanical testing with a mock finger to form predictions about the degree of crystallinity necessary to result in successful discrimination by human subjects. Psychophysical testing verified that humans could discriminate surfaces which differed only in the degree of crystallinity. Although related, human performance was not strongly correlated with a straightforward difference in the degree of crystallinity. Rather, human performance was better explained by quantifying transitions in steady to unsteady sliding and the generation of slow frictional waves (r² = 79.6%). Tuning fine touch with polymer crystallinity may lead to better engineering of existing haptic interfaces or lead to new classes of actuators based on changes in microstructure.

Adaptability of Assistive Mobility Devices and the Role of the Internet of Medical Things: Comprehensive Review

Background

With the projected upsurge in the percentage of people with some form of disability, there has been a significant increase in the need for assistive mobility devices. However, for mobility aids to be effective, such devices should be adapted to the user’s needs. This can be achieved by improving the confidence of the acquired information (interaction between the user, the environment, and the device) following design specifications. Therefore, there is a need for literature review on the adaptability of assistive mobility devices.

Objective

In this study, we aim to review the adaptability of assistive mobility devices and the role of the internet of medical things in terms of the acquired information for assistive mobility devices. We review internet-enabled assistive mobility technologies and non–internet of things (IoT) assistive mobility devices. These technologies will provide awareness of the status of adaptive mobility technology and serve as a source and reference regarding information to health care professionals and researchers.

Methods

We performed a literature review search on the following databases of academic references and journals: Google Scholar, ScienceDirect, Institute of Electrical and Electronics Engineers, Springer, and websites of assistive mobility and foundations presenting studies on assistive mobility found through a generic Google search (including the World Health Organization website). The following keywords were used: assistive mobility OR assistive robots, assistive mobility devices, internet-enabled assistive mobility technologies, IoT Framework OR IoT Architecture AND for Healthcare, assisted navigation OR autonomous navigation, mobility AND aids OR devices, adaptability of assistive technology, adaptive mobility devices, pattern recognition, autonomous navigational systems, human-robot interfaces, motor rehabilitation devices, perception, and ambient assisted living.

Results

We identified 13,286 results (excluding titles that were not relevant to this study). Then, through a narrative review, we selected 189 potential studies (189/13,286, 1.42%) from the existing literature on the adaptability of assistive mobility devices and IoT frameworks for assistive mobility and conducted a critical analysis. Of the 189 potential studies, 82 (43.4%) were selected for analysis after meeting the inclusion criteria. On the basis of the type of technologies presented in the reviewed articles, we proposed a categorization of the adaptability of smart assistive mobility devices in terms of their interaction with the user (user system interface), perception techniques, and communication and sensing frameworks.

Conclusions

We discussed notable limitations of the reviewed literature studies. The findings revealed that an improvement in the adaptation of assistive mobility systems would require a reduction in training time and avoidance of cognitive overload. Furthermore, sensor fusion and classification accuracy are critical for achieving real-world testing requirements. Finally, the trade-off between cost and performance should be considered in the commercialization of these devices.

Speaking Haptically: From Phonemes to Phrases With a Mobile Haptic Communication System

In this article, we present three studies involving WhatsHap, a mobile system designed to deliver speech as vibrations on the forearm with minimal hardware demands and practice time. After only 4.2 h of training on a 24-haptic phoneme vocabulary and on how to combine these to form words, participants were able to generalize their phoneme identification skills to the understanding of untrained English words, correctly identifying 65% of words in phrases rendered with a user-controlled interval between words, and up to 59% with a fixed interval. Ultimately, participants were able to complete 88% of simple communicative tasks that elicited spontaneous speech and semi-structured bidirectional conversation using the apparatus. We conclude by providing insights as to how such a system may ultimately be used for communication under more natural conditions.

Tacsac: A Wearable Haptic Device with Capacitive Touch-Sensing Capability for Tactile Display

This paper presents a dual-function wearable device (Tacsac) with capacitive tactile sensing and integrated tactile feedback capability to enable communication among deafblind people. Tacsac has a skin contactor which enhances localized vibrotactile stimulation of the skin as a means of feedback to the user. It comprises two main modules—the touch-sensing module and the vibrotactile module; both stacked and integrated as a single device. The vibrotactile module is an electromagnetic actuator that employs a flexible coil and a permanent magnet assembled in soft poly (dimethylsiloxane) (PDMS), while the touch-sensing module is a planar capacitive metal-insulator-metal (MIM) structure. The flexible coil was fabricated on a 50 µm polyimide (PI) sheet using Lithographie Galvanoformung Abformung (LIGA) micromoulding technique. The Tacsac device has been tested for independent sensing and actuation as well as dual sensing-actuation mode. The measured vibration profiles of the actuator showed a synchronous response to external stimulus for a wide range of frequencies (10 Hz to 200 Hz) within the perceivable tactile frequency thresholds of the human hand. The resonance vibration frequency of the actuator is in the range of 60–70 Hz with an observed maximum off-plane displacement of 0.377 mm at coil current of 180 mA. The capacitive touch-sensitive layer was able to respond to touch with minimal noise both when actuator vibration is ON and OFF. A mobile application was also developed to demonstrate the application of Tacsac for communication between deafblind person wearing the device and a mobile phone user who is not deafblind. This advances existing tactile displays by providing efficient two-way communication through the use of a single device for both localized haptic feedback and touch-sensing.

Assistive technologies for severe and profound hearing loss: Beyond hearing aids and implants

Assistive technologies (ATs) offer capabilities that were previously inaccessible to individuals with severe and profound hearing loss who have no or limited access to hearing aids and implants. This literature review aims to explore existing ATs and identify what still needs to be done. It is found that there is a lack of focus on the overall objectives of ATs. In addition, several other issues are identified, i.e. only a very small number of ATs developed within a research context have led to commercial devices, and there is a predisposition to use the latest expensive technologies and a tendency to avoid designing products universally. Finally, the further development of plug-ins that translate the text content of a website to various sign languages is needed to make information on the internet more accessible.

Wearable Assistive Tactile Communication Interface Based on Integrated Touch Sensors and Actuators

This paper presents the design and fabrication of a wearable tactile communication interface with vibrotactile feedback for assistive communication. The interface is based on finger Braille, which is a simple and efficient tactile communication method used by deafblind people. It consists of a flexible piezoresistive sensor and a vibrotactile actuator integrated together and positioned at the index, middle and ring fingers of both hands to represent the six dots of Braille. The sensors were made using flexible piezoresistive material whereas the actuator utilizes electromagnetic principle by means of a flexible coil and a tiny NdFeB permanent magnet. Both were integrated to realize a Bluetooth-enabled tactile communication glove which enables deafblind people to communicate using Braille codes. The evaluation with 20 end-users (10 deafblind and 10 sighted and hearing person) of the tactile interface under standardized conditions demonstrated that users can feel and distinguish the vibration at frequencies ranging from 10Hz to 200Hz which is within the perceivable frequency range for the FA-II receptors. The results show that it took non-experts in Braille within 25s and 55s to send and receive words like "BEST" and "JOURNAL", with an accuracy of ~75% and 68% respectively.

Smartphone-based computer vision travelling aids for blind and visually impaired individuals: A systematic review

Given the growth in the numbers of visually impaired (VI) people in low-income countries, the development of affordable electronic travel aid (ETA) systems employing devices, sensors, and apps embedded in ordinary smartphones becomes a potentially cost-effective and reasonable all-in-one solution of utmost importance for the VI. This paper offers an overview of recent ETA research prototypes that employ smartphones for assisted orientation and navigation in indoor and outdoor spaces by providing additional information about the surrounding objects. Scientific achievements in the field were systematically reviewed using PRISMA methodology. Comparative meta-analysis showed how various smartphone-based ETA prototypes could assist with better orientation, navigation, and wayfinding in indoor and outdoor environments. The analysis found limited interest among researchers in combining haptic interfaces and computer vision capabilities in smartphone-based ETAs for the blind, few attempts to employ novel state-of-the-art computer vision methods based on deep neural networks, and no evaluations of existing off-the-shelf navigation solutions. These results were contrasted with findings from a survey of blind expert users on their problems in navigating in indoor and outdoor environments. This revealed a major mismatch between user needs and academic development in the field.

Audio motor training improves mobility and spatial cognition in visually impaired children

Since it has been demonstrated that spatial cognition can be affected in visually impaired children, training strategies that exploit the plasticity of the human brain should be early adopted. Here we developed and tested a new training protocol based on the reinforcement of audio-motor associations and thus supporting spatial development in visually impaired children. The study involved forty-four visually impaired children aged 6–17 years old assigned to an experimental (ABBI training) or a control (classical training) rehabilitation conditions. The experimental training group followed an intensive but entertaining rehabilitation for twelve weeks during which they performed ad-hoc developed audio-spatial exercises with the Audio Bracelet for Blind Interaction (ABBI). A battery of spatial tests administered before and after the training indicated that children significantly improved in almost all the spatial aspects considered, while the control group didn’t show any improvement. These results confirm that perceptual development in the case of blindness can be enhanced with naturally associated auditory feedbacks to body movements. Therefore the early introduction of a tailored audio-motor training could potentially prevent spatial developmental delays in visually impaired children.

Neuromorphic Vibrotactile Stimulation of Fingertips for Encoding Object Stiffness in Telepresence Sensory Substitution and Augmentation Applications

We present a tactile telepresence system for real-time transmission of information about object stiffness to the human fingertips. Experimental tests were performed across two laboratories (Italy and Ireland). In the Italian laboratory, a mechatronic sensing platform indented different rubber samples. Information about rubber stiffness was converted into on-off events using a neuronal spiking model and sent to a vibrotactile glove in the Irish laboratory. Participants discriminated the variation of the stiffness of stimuli according to a two-alternative forced choice protocol. Stiffness discrimination was based on the variation of the temporal pattern of spikes generated during the indentation of the rubber samples. The results suggest that vibrotactile stimulation can effectively simulate surface stiffness when using neuronal spiking models to trigger vibrations in the haptic interface. Specifically, fractional variations of stiffness down to 0.67 were significantly discriminated with the developed neuromorphic haptic interface. This is a performance comparable, though slightly worse, to the threshold obtained in a benchmark experiment evaluating the same set of stimuli naturally with the own hand. Our paper presents a bioinspired method for delivering sensory feedback about object properties to human skin based on contingency-mimetic neuronal models, and can be useful for the design of high performance haptic devices.