Simultaneous Multimodal PC Access for People With Disabilities by Integrating Head Tracking, Speech Recognition, and Tongue Motion

Low-Cost Human-Machine Interface for Computer Control with Facial Landmark Detection and Voice Commands

Nowadays, daily life involves the extensive use of computers, since human beings are immersed in a technological society. Therefore, it is mandatory to interact with computers, which represents a true disadvantage for people with upper limb disabilities. In this context, this work aims to develop an interface for emulating mouse and keyboard functions (EMKEY) by applying concepts of artificial vision and voice recognition to replace the use of hands. Pointer control is achieved by head movement, whereas voice recognition is used to perform interface functionalities, including speech-to-text transcription. To evaluate the interface's usability and usefulness, two studies were carried out. The first study was performed with 30 participants without physical disabilities. Throughout this study, there were significant correlations found between the emulator's usability and aspects such as adaptability, execution time, and the participant's age. In the second study, the use of the emulator was analyzed by four participants with motor disabilities. It was found that the interface was best used by the participant with cerebral palsy, followed by the participants with upper limb paralysis, spina bifida, and muscular dystrophy. In general, the results show that the proposed interface is easy to use, practical, fairly accurate, and works on a wide range of computers.

Exploration of multimodal alternative access for individuals with severe motor impairments: Proof of concept

Many individuals with complex communication needs and severe motor impairments are unable to control technologies through conventional means and require alternative access techniques to achieve accurate and efficient access. With current alternative access techniques, individuals with severe motor impairments are limited in that they can only use one access technique at a time. The purpose of this project was to test proof of concept of a new multimodal access technique which integrated eye gaze and single switch scanning selection techniques. The aims were to investigate the learning patterns of two adults with severe cerebral palsy when first introduced to the multimodal access technique and then to compare the accuracy and efficiency of multimodal to single-modality access when selecting targets on an AAC visual scene display. The participants learned to use the multimodal access technique; they demonstrated improvements in their accuracy of selection across sessions and attained at least 80% accuracy within 3-15 training sessions. Both participants achieved similar accuracy with multimodal access compared to single-modality, but took longer to select targets with multimodal access compared to single-modality. The potential utility of multimodal access is explored as well as important avenues for future research.

Evaluation of a Head-Tongue Controller for Power Wheelchair Driving By People With Quadriplegia

The head-tongue controller (HTC) is a multimodal alternative controller designed for people with quadriplegia to access complex control capabilities by combining tongue and head tracking to offer both discrete and proportional controls in a single controller. In this human study, 17 patients with quadriplegia and current users of alternative controllers were asked to perform four trials of either simple driving tasks or advanced maneuvers in a custom-designed course. Completion time and accuracy were compared between their personal alternative controller (PAC) and various combinations of driving modalities with the HTC. Out of 8 subjects assigned to simple driving, the best HTC trial of 3 subjects was completed faster than their PAC for the tasks of rolling forward and turning around cones, and 5 subjects in rolling backward. Across all these subjects, the average completion time of their best HTC modality is 23s for rolling forward, 15s for rolling backward, and 70s for turning around cones as compared to 19s, 17s, and 45s with their PAC. For advanced driving, the course was completed faster with the HTC by 1 out of 9 subjects, while the best HTC trials of all subjects are less than 1.3 times of their best PAC completion time with an average of 170s for the HTC and 140s for their PAC. The qualitative feedback provided by all subjects to a post-study questionnaire scored to an average of 7.5 out of 10 which shows their interests in the HTC and acknowledgement of its usefulness for this population.

A Stand-Alone Intraoral Tongue-Controlled Computer Interface for People With Tetraplegia

The intraoral Tongue Drive System (iTDS) is an embedded wireless tongue-operated assistive technology developed for people with tetraplegia to provide them a higher level of independence in performing daily living tasks, such as accessing computers, smartphones, and driving wheelchairs. The iTDS was built as an arch-shaped dental retainer hermetically sealed and placed in the buccal shelf area of the mouth, completely hidden from sight. To provide high level of comfort, the iTDS is customized based on the users' oral anatomy to stably fix onto the lower teeth. We have presented a standalone version of the iTDS, capable of recognizing tongue gestures/commands by processing raw magnetic sensor data with a built-in pattern recognition algorithm in real time. The iTDS then sends the commands out in 10-b packets through a custom-designed high-gain intraoral antenna at 2.4 GHz to an external receiver. To evaluate the standalone iTDS performance, four subjects performed a computer access task by issuing random tongue commands over five sessions. Subjects completed 99.2% of the commands, and achieved an information transfer rate of 150.1 b/min. Moreover, a new typing method, designed specifically for the iTDS, resulted in typing at a rate of 3.76 words/min and error rate of 2.23%.

Eyelid Drive System: An Assistive Technology Employing Inductive Sensing of Eyelid Movement

This paper presents the design, development, and validation of the eyelid drive system (EDS), an assistive technology comprising a specialized pair of glasses and millimeter-sized passive resonators, attached to the user's eyelids, that transduce eyelid movement (blinking and winking) through inductive sensing. The theory of operation and design optimization with simulations are presented. A proof-of-concept prototype EDS was constructed using a pair of nonprescription glasses and commercial-off-the-shelf components. In benchtop tests with model eyelids, the EDS demonstrated basic functionality. Initial trials were performed involving six human subjects interacting with custom designed graphical user interfaces on a computer. A group mean accuracy of 96.3% was achieved using a set of four different commands at a response rate of 3 s. A mean information transfer rate (ITR) of 56.1 b/min over all subjects was achieved with a set of six different commands at a response rate of 1.5 s. This proof-of-concept device consumes 51.6 mW of power. The EDS compares favorably with related eye-interfacing assistive technologies and provides a unique combination of advantages, including high accuracy and ITR, wearability, insensitivity to lighting and noise conditions, obviation of facial electrodes, and the use of nonexaggerated gestures.

New and emerging access technologies for adults with complex communication needs and severe motor impairments: State of the science

Individuals with complex communication needs often use alternative access technologies to control their augmentative and alternative communication (AAC) devices, their computers, and mobile technologies. While a range of access devices is available, many challenges continue to exist, particularly for those with severe motor-control limitations. For some, access options may not be readily available or access itself may be inaccurate and frustrating. For others, access may be available but only under optimal conditions and support. There is an urgent need to develop new options for individuals with severe motor impairments and to leverage existing technology to improve efficiency, increase accuracy, and decrease fatigue of access. This paper describes person-centred research and development activities related to new and emerging access technologies, with a particular focus on adults with acquired neurological conditions.

Adaptive Matching Transmitter With Dual-Band Antenna for Intraoral Tongue Drive System

The intraoral Tongue Drive System (iTDS) is a wireless assistive technology that detects users' voluntary tongue gestures, and converts them to user-defined commands, enabling them to access computers and navigate powered wheelchairs. In this paper, we presented a transmitter (Tx) with adaptive matching and three bands (27, 433, and 915 MHz) to create a robust wireless link between iTDS and an external receiver (Rx) by addressing the effects of external RF interference and impedance variations of the Tx antenna in the dynamic mouth environment. The upper two Tx bands share a dual-band antenna, while the lower band drives a coil. The Tx antenna is simulated in a simplified human mouth model in HFSS as well as a real human head model. The adaptive triple-band Tx chip was fabricated in a 0.35-μm 4P2M standard CMOS process. The Tx chip and antenna have been characterized in a human subject as part of an iTDS prototype under open-and closed-mouth scenarios, which present the peak gain of -24.4 and -15.63 dBi at 433 and 915 MHz, respectively. Two adaptive matching networks for these bands compensate variations of the Tx antenna impedance via a feedback mechanism. The measured S₁₁ tuning range of the proposed network can cover up to 60 and 75 jΩ at 433 and 915 MHz, respectively.

Comparing the Use of Single vs. Multiple Combined Abilities in Conducting Complex Computer Tasks Hands-free

OBJECTIVE

Assistive technologies often focus on a remaining ability of their users, particularly those with physical disabilities, e.g. tetraplegia, to facilitate their computer access. We hypothesized that by combining multiple remaining abilities of the end users in an intuitive fashion, it is possible to improve the quality of computer access. In this study, 15 able-bodied subjects completed four computer access tasks without using their hands: center-out tapping, on-screen maze navigation, playing a game, and sending an email. They used the multimodal Tongue Drive System (mTDS), which offers proportional cursor control via head motion, discrete clicks via tongue gestures, and typing via speech recognition simultaneously. Their performances were compared against unimodal tongue gestures (TDS), and Keyboard & Mouse combination (KnM), as the gold standard.

RESULTS

Center-out tapping task average throughputs using mTDS and TDS were 0.84 bps and 0.94 bps, which were 21% and 22.4% of the throughput using mouse, respectively, while the average error rate and missed targets using mTDS were 4.1% and 25.5% less than TDS. Maze navigation throughputs using mTDS and TDS were 0.35 bps and 0.46 bps, which were 16.6% and 21.8% of the throughput using mouse, respectively. Participants achieved 72.32% higher score using mTDS than TDS when playing a simple game. Average email generating time with mTDS was ~2x longer than KnM with a mean typing accuracy of 78.1%.

CONCLUSION

Engaging multimodal abilities helped participants perform considerably better in complex tasks, such as sending an email, compared to a unimodal system (TDS). Their performances were similar for simpler task, while multimodal inputs improved interaction accuracy. Cursor navigation with head motion led to higher score in less constrained tasks, such as game, than a highly constrained maze task.

A Multimodal Adaptive Wireless Control Interface for People With Upper-Body Disabilities

This paper describes a multimodal body-machine interface (BoMI) to help individuals with upper-limb disabilities using advanced assistive technologies, such as robotic arms. The proposed system uses a wearable and wireless body sensor network (WBSN) supporting up to six sensor nodes to measure the natural upper-body gesture of the users and translate it into control commands. Natural gesture of the head and upper-body parts, as well as muscular activity, are measured using inertial measurement units (IMUs) and surface electromyography (sEMG) using custom-designed multimodal wireless sensor nodes. An IMU sensing node is attached to a headset worn by the user. It has a size of 2.9 cm 2.9 cm, a maximum power consumption of 31 mW, and provides angular precision of 1. Multimodal patch sensor nodes, including both IMU and sEMG sensing modalities are placed over the user able-body parts to measure the motion and muscular activity. These nodes have a size of 2.5 cm 4.0 cm and a maximum power consumption of 11 mW. The proposed BoMI runs on a Raspberry Pi. It can adapt to several types of users through different control scenarios using the head and shoulder motion, as well as muscular activity, and provides a power autonomy of up to 24 h. JACO, a 6-DoF assistive robotic arm, is used as a testbed to evaluate the performance of the proposed BoMI. Ten able-bodied subjects performed ADLs while operating the AT device, using the Test d'Évaluation des Membres Supérieurs de Personnes Âgées to evaluate and compare the proposed BoMI with the conventional joystick controller. It is shown that the users can perform all tasks with the proposed BoMI, almost as fast as with the joystick controller, with only 30% time overhead on average, while being potentially more accessible to the upper-body disabled who cannot use the conventional joystick controller. Tests show that control performance with the proposed BoMI improved by up to 17% on average, after three trials.