Quantitative and comparative assessment of learning in a tongue-operated computer input device

An Innovative Device Based on Human-Machine Interface (HMI) for Powered Wheelchair Control for Neurodegenerative Disease: A Proof-of-Concept

In the global context, advancements in technology and science have rendered virtual, augmented, and mixed-reality technologies capable of transforming clinical care and medical environments by offering enhanced features and improved healthcare services. This paper aims to present a mixed reality-based system to control a robotic wheelchair for people with limited mobility. The test group comprised 11 healthy subjects (six male, five female, mean age 35.2 ± 11.7 years). A novel platform that integrates a smart wheelchair and an eye-tracking-enabled head-mounted display was proposed to reduce the cognitive requirements needed for wheelchair movement and control. The approach's effectiveness was demonstrated by evaluating our system in realistic scenarios. The demonstration of the proposed AR head-mounted display user interface for controlling a smart wheelchair and the results provided in this paper could highlight the potential of the HoloLens 2-based innovative solutions and bring focus to emerging research topics, such as remote control, cognitive rehabilitation, the implementation of patient autonomy with severe disabilities, and telemedicine.

Assistive Technologies and Quadriplegia: A Map Point on the Development and Spread of the Tongue Barbell Piercing

The barbell piercing can be used as an assistive device that allows people with severe disabilities, such as tetraplegia, to control their environments using the movement of the tongue. The human tongue can move rapidly and accurately, such that the tip can touch every tooth. Lingual control systems allow people with disabilities to take advantage of their residual skills for easier communication and to improve the control of mobility and the surrounding environment. The aim of this study was to conduct a narrative review of the development and dissemination of the assistive technologies based on tongue control by means of the barbell piercing. The design of the study was based on: (I) an overview of Pubmed complemented with other databases and Web searches (also institutional); (II) an organization according to a standardized checklist for narrative reviews; (III) an arrangement with four different perspectives: the trends in the scientific literature, technological evolution and categorization, dominant approaches, issues of incorporation into the health domain-such as acceptance, safety, and regulations. The results have highlighted: (1) that the volume of scientific productions, which started in this sector before the smartphone expansion, has not increased; (2) that it is possible to make a map point of the technological evolution and categorization; (3) that these assistive technologies have a high degree of acceptance and performance, especially when integrated with aid tools with mechatronics; (4) and the complexity of the regulatory framework in this area. The study, from a general point of view, highlighted the high potential of these systems and we suggest investing the energy into agreement tools for assistive technologies (AT)s, such as health technology assessment studies, comparative assessment analysis, or consensus conferences that could allow a better diffusion and use of ATs, including these systems.

Comparing the Use of Single vs. Multiple Combined Abilities in Conducting Complex Computer Tasks Hands-free

OBJECTIVE

Assistive technologies often focus on a remaining ability of their users, particularly those with physical disabilities, e.g. tetraplegia, to facilitate their computer access. We hypothesized that by combining multiple remaining abilities of the end users in an intuitive fashion, it is possible to improve the quality of computer access. In this study, 15 able-bodied subjects completed four computer access tasks without using their hands: center-out tapping, on-screen maze navigation, playing a game, and sending an email. They used the multimodal Tongue Drive System (mTDS), which offers proportional cursor control via head motion, discrete clicks via tongue gestures, and typing via speech recognition simultaneously. Their performances were compared against unimodal tongue gestures (TDS), and Keyboard & Mouse combination (KnM), as the gold standard.

RESULTS

Center-out tapping task average throughputs using mTDS and TDS were 0.84 bps and 0.94 bps, which were 21% and 22.4% of the throughput using mouse, respectively, while the average error rate and missed targets using mTDS were 4.1% and 25.5% less than TDS. Maze navigation throughputs using mTDS and TDS were 0.35 bps and 0.46 bps, which were 16.6% and 21.8% of the throughput using mouse, respectively. Participants achieved 72.32% higher score using mTDS than TDS when playing a simple game. Average email generating time with mTDS was ~2x longer than KnM with a mean typing accuracy of 78.1%.

CONCLUSION

Engaging multimodal abilities helped participants perform considerably better in complex tasks, such as sending an email, compared to a unimodal system (TDS). Their performances were similar for simpler task, while multimodal inputs improved interaction accuracy. Cursor navigation with head motion led to higher score in less constrained tasks, such as game, than a highly constrained maze task.

Simultaneous Multimodal PC Access for People With Disabilities by Integrating Head Tracking, Speech Recognition, and Tongue Motion

Multimodal Tongue Drive System (mTDS) is a highly integrated wireless assistive technology (AT) in the form of a lightweight wearable headset that utilizes three remaining key control and communication abilities in people with severe physical disabilities, such as tetraplegia, to provide them with effective access to computers: 1) tongue motion for discrete/switch-based control (e.g., clicking), 2) head tracking for proportional control (e.g., mouse pointer movements), and 3) speech recognition for typing, all available simultaneously. The mTDS architecture is presented here with new sensor signal processing algorithm for head tracking. To evaluate the device performance, it was compared against keyboard-and-mouse (KnM) combination, the gold standard in computer input methods, by 15 able-bodied participants, who used both mTDS and KnM to generate and sent an email with randomly selected content, under a 5-minute time constraint. In four repetitions, in the last trial, it took participants only 1.8 times longer to complete the email task, on average, using the mTDS versus KnM at 82.4% typing accuracy. Mean task completion time and typing accuracy improved 24.6% and 18.8% from first to fourth trial using mTDS. Multimodal simultaneous discrete and proportional control input options of mTDS, plus rapid typing, is expected to provide more effective computer access to people with severe physical disabilities.

Non-Model-Based Control of a Wheeled Vehicle Pulling Two Trailers to Provide Early Powered Mobility and Driving Experiences

Non-model-based control of a wheeled vehicle pulling two trailers is proposed. It is a fun train for disabled children consisting of a locomotive and two carriages. The fun train has afforded opportunities for both disabled and able bodied young people to share an activity and has provided early driving experiences for disabled children; it has introduced them to assistive and powered mobility. The train is a nonlinear system and subject to nonholonomic kinematic constraints, so that position and state depend on the path taken to get there. The train is described, and then, a robust control algorithm using proportional-derivative filtered errors is proposed to control the locomotive. The controller was not dependent on an accurate model of the train, because the mass of the vehicle and two carriages changed depending on the number, size, and shape of children and wheelchair seats on the train. The controller was robust and stable in uncertainty. Results are presented to show the effectiveness of the approach, and the suggested control algorithm is shown to be acceptable without knowing the exact plant dynamics.

Qualitative assessment of tongue drive system by people with high-level spinal cord injury

The Tongue Drive System (TDS) is a minimally invasive, wireless, and wearable assistive technology (AT) that enables people with severe disabilities to control their environments using tongue motion. TDS translates specific tongue gestures into commands by sensing the magnetic field created by a small magnetic tracer applied to the user's tongue. We have previously quantitatively evaluated the TDS for accessing computers and powered wheelchairs, demonstrating its usability. In this study, we focused on its qualitative evaluation by people with high-level spinal cord injury who each received a magnetic tongue piercing and used the TDS for 6 wk. We used two questionnaires, an after-scenario and a poststudy, designed to evaluate the tongue-piercing experience and the TDS usability compared with that of the sip-and-puff and the users' current ATs. After study completion, 73% of the participants were positive about keeping the magnetic tongue-barbell in order to use the TDS. All were satisfied with the TDS performance and most said that they were able to do more things using TDS than their current ATs (4.22/5).

Safety and efficacy of medically performed tongue piercing in people with tetraplegia for use with tongue-operated assistive technology

BACKGROUND

Individuals with high-level spinal cord injuries need effective ways to perform activities.

OBJECTIVES

To develop and test a medically supervised tongue-piercing protocol and the wearing of a magnet-containing tongue barbell for use with the Tongue Drive System (TDS) in persons with tetraplegia.

METHODS

Volunteers with tetraplegia underwent initial screening sessions using a magnet glued on the tongue to activate and use the TDS. This was followed by tongue piercing, insertion of a standard barbell, a 4-week healing period, and an exchange of the standard barbell for a magnet-containing barbell. This was then used twice weekly for 6 to 8 weeks to perform computer tasks, drive a powered wheelchair, accomplish in-chair weight shifts, and dial a phone. Symptoms of intraoral dysfunction, change in tongue size following piercing, and subjective assessment of receiving and wearing a magnet-containing tongue barbell and its usability with the TDS were evaluated.

RESULTS

Twenty-one volunteers underwent initial trial sessions. Thirteen had their tongues pierced. One individual's barbell dislodged during healing resulting in tongue-tract closure. Twelve had the barbell exchanged for a magnet-containing barbell. One subject withdrew for unrelated issues. Eleven completed the TDS testing sessions and were able to complete the assigned tasks. No serious adverse events occurred related to wearing or using a tongue barbell to operate the TDS.

CONCLUSIONS

Using careful selection criteria and a medically supervised piercing protocol, no excess risk was associated with tongue piercing and wearing a tongue barbell in people with tetraplegia. Participants were able to operate the TDS.

Facial Position and Expression-Based Human-Computer Interface for Persons With Tetraplegia

A human-computer interface (namely Facial position and expression Mouse system, FM) for the persons with tetraplegia based on a monocular infrared depth camera is presented in this paper. The nose position along with the mouth status (close/open) is detected by the proposed algorithm to control and navigate the cursor as computer user input. The algorithm is based on an improved Randomized Decision Tree, which is capable of detecting the facial information efficiently and accurately. A more comfortable user experience is achieved by mapping the nose motion to the cursor motion via a nonlinear function. The infrared depth camera enables the system to be independent of illumination and color changes both from the background and on human face, which is a critical advantage over RGB camera-based options. Extensive experimental results show that the proposed system outperforms existing assistive technologies in terms of quantitative and qualitative assessments.

Assessment of the Tongue-Drive System Using a Computer, a Smartphone, and a Powered-Wheelchair by People With Tetraplegia

Tongue-Drive System (TDS) is a wireless and wearable assistive technology that enables people with severe disabilities to control their computers, wheelchairs, and smartphones using voluntary tongue motion. To evaluate the efficacy of the TDS, several experiments were conducted, in which the performance of nine able-bodied (AB) participants using a mouse, a keypad, and the TDS, as well as a cohort of 11 participants with tetraplegia (TP) using the TDS, were observed and compared. Experiments included the Fitts' law tapping, wheelchair driving, phone-dialing, and weight-shifting tasks over five to six consecutive sessions. All participants received a tongue piercing, wore a magnetic tongue stud, and completed the trials as evaluable participants. Although AB participants were already familiar with the keypad, throughputs of their tapping tasks using the keypad were only 1.4 times better than those using the TDS. The completion times of wheelchair driving task using the TDS for AB and TP participants were between 157 s and 180 s with three different control strategies. Participants with TP completed phone-dialing and weight-shifting tasks in 81.9 s and 71.5 s, respectively, using tongue motions. Results showed statistically significant improvement or trending to improvement in performance status over the sessions. Most of the learning occurred between the first and second sessions, but trends did suggest that more practice would lead to increased improvement in performance using the TDS.

Human-computer interface controlled by the lip

Lip control system is an innovative human-computer interface specially designed for people with tetraplegia. This paper presents an evaluation of the lower lip potential to control an input device, according to Fitts' law (ISO/TS 9241-411:2012 standard). The results show that the lower lip throughput is comparable with the thumb throughput using the same input device under the same conditions. These results establish the baseline for future research studies about the lower lip capacity to operate a computer input device.

An arch-shaped intraoral tongue drive system with built-in tongue-computer interfacing SoC

We present a new arch-shaped intraoral Tongue Drive System (iTDS) designed to occupy the buccal shelf in the user's mouth. The new arch-shaped iTDS, which will be referred to as the iTDS-2, incorporates a system-on-a-chip (SoC) that amplifies and digitizes the raw magnetic sensor data and sends it wirelessly to an external TDS universal interface (TDS-UI) via an inductive coil or a planar inverted-F antenna. A built-in transmitter (Tx) employs a dual-band radio that operates at either 27 MHz or 432 MHz band, according to the wireless link quality. A built-in super-regenerative receiver (SR-Rx) monitors the wireless link quality and switches the band if the link quality is below a predetermined threshold. An accompanying ultra-low power FPGA generates data packets for the Tx and handles digital control functions. The custom-designed TDS-UI receives raw magnetic sensor data from the iTDS-2, recognizes the intended user commands by the sensor signal processing (SSP) algorithm running in a smartphone, and delivers the classified commands to the target devices, such as a personal computer or a powered wheelchair. We evaluated the iTDS-2 prototype using center-out and maze navigation tasks on two human subjects, which proved its functionality. The subjects' performance with the iTDS-2 was improved by 22% over its predecessor, reported in our earlier publication.

The tongue enables computer and wheelchair control for people with spinal cord injury

The Tongue Drive System (TDS) is a wireless and wearable assistive technology, designed to allow individuals with severe motor impairments such as tetraplegia to access their environment using voluntary tongue motion. Previous TDS trials used a magnetic tracer temporarily attached to the top surface of the tongue with tissue adhesive. We investigated TDS efficacy for controlling a computer and driving a powered wheelchair in two groups of able-bodied subjects and a group of volunteers with spinal cord injury (SCI) at C6 or above. All participants received a magnetic tongue barbell and used the TDS for five to six consecutive sessions. The performance of the group was compared for TDS versus keypad and TDS versus a sip-and-puff device (SnP) using accepted measures of speed and accuracy. All performance measures improved over the course of the trial. The gap between keypad and TDS performance narrowed for able-bodied subjects. Despite participants with SCI already having familiarity with the SnP, their performance measures were up to three times better with the TDS than with the SnP and continued to improve. TDS flexibility and the inherent characteristics of the human tongue enabled individuals with high-level motor impairments to access computers and drive wheelchairs at speeds that were faster than traditional assistive technologies but with comparable accuracy.

A dual-mode human computer interface combining speech and tongue motion for people with severe disabilities

We are presenting a new wireless and wearable human computer interface called the dual-mode Tongue Drive System (dTDS), which is designed to allow people with severe disabilities to use computers more effectively with increased speed, flexibility, usability, and independence through their tongue motion and speech. The dTDS detects users' tongue motion using a magnetic tracer and an array of magnetic sensors embedded in a compact and ergonomic wireless headset. It also captures the users' voice wirelessly using a small microphone embedded in the same headset. Preliminary evaluation results based on 14 able-bodied subjects and three individuals with high level spinal cord injuries at level C3-C5 indicated that the dTDS headset, combined with a commercially available speech recognition (SR) software, can provide end users with significantly higher performance than either unimodal forms based on the tongue motion or speech alone, particularly in completing tasks that require both pointing and text entry.

Quantitative assessment of magnetic sensor signal processing algorithms in a wireless tongue-operated assistive technology

In this paper, we evaluate the overall performance of various magnetic-sensor signal processing (mSSP) algorithms for the Tongue Drive System based on a comprehensive dataset collected from trials with a total of eight able-bodied subjects. More specifically, we measure the performance of nine classifiers on the two-stage classification used by the mSSP algorithm, in order to learn how to improve the current algorithm. Results show that is it possible to reduce misclassification error from 5.95% and 20.13% to 3.98% and 5.63%, from the two assessed datasets, respectively, without sacrificing correctness. Furthermore, since the mSSP algorithm must run in real time, the results show where to focus the computational resources when they are constrained by the platforms with limited resources, such as smartphones.

Development and preliminary evaluation of an intraoral Tongue Drive System

Tongue Drive System (TDS) is a wireless tongue-operated assistive technology (AT), developed for people with severe physical impediments to control their environments using their tongue motion. We have developed a new intraoral TDS (iTDS) in a form of a dental retainer, which can tightly clasp onto the upper teeth, completely hidden inside the mouth, using commercial off-the-shelf components (COTS). The iTDS retainer was tested by two healthy subjects and their performance was compared with that of an external TDS (eTDS) implemented in the form of a headset. The iTDS retainer showed comparable performance with the eTDS headset. The iTDS is expected to improve the stability and robustness of the TDS, while giving users a certain degree of privacy.

Tongue-operated assistive technology with access to common smartphone applications via Bluetooth link

Tongue Drive System (TDS) is a wireless and wearable assistive technology (AT) that enables people with severe disabilities to control their computers, wheelchairs, and electronic gadgets using their tongue motion. We developed the TDS to control smartphone's (iPhone/iPod Touch) built-in and downloadable apps with a customized Bluetooth mouse module by emulating finger taps on the touchscreen. The TDS-iPhone Bluetooth mouse interface was evaluated by four able-bodied subjects to complete a scenario consisting of seven tasks, which were randomly ordered by using touch on the iPhone screen with index finger, a computer mouse on iPhone, and TDS-iPhone Bluetooth mouse interface with tongue motion. Preliminary results show that the average completion times of a scenario with touch, mouse, and TDS are 165.6 ± 14.50 s, 186.1 ± 15.37 s, and 651.6 ± 113.4 s, respectively, showing that the TDS is 84.37% and 81.16% slower than touch and mouse for speed of typing with negligible errors. Overall, considering the limited number of commands and unfamiliarity of the subjects with the TDS, we achieved acceptable results for hands-free functionality.

A wireless magnetoresistive sensing system for an intraoral tongue-computer interface

Tongue drive system (TDS) is a tongue-operated, minimally invasive, unobtrusive, and wireless assistive technology (AT) that infers users' intentions by detecting their voluntary tongue motion and translating them into user-defined commands. Here we present the new intraoral version of the TDS (iTDS), which has been implemented in the form of a dental retainer. The iTDS system-on-a-chip (SoC) features a configurable analog front-end (AFE) that reads the magnetic field variations inside the mouth from four 3-axial magnetoresistive sensors located at four corners of the iTDS printed circuit board (PCB). A dual-band transmitter (Tx) on the same chip operates at 27 and 432 MHz in the Industrial/Scientific/Medical (ISM) band to allow users to switch in the presence of external interference. The Tx streams the digitized samples to a custom-designed TDS universal interface, built from commercial off-the-shelf (COTS) components, which delivers the iTDS data to other devices such as smartphones, personal computers (PC), and powered wheelchairs (PWC). Another key block on the iTDS SoC is the power management integrated circuit (PMIC), which provides individually regulated and duty-cycled 1.8 V supplies for sensors, AFE, Tx, and digital control blocks. The PMIC also charges a 50 mAh Li-ion battery with constant current up to 4.2 V, and recovers data and clock to update its configuration register through a 13.56 MHz inductive link. The iTDS SoC has been implemented in a 0.5-μm standard CMOS process and consumes 3.7 mW on average.

Quantitative and comparative assessment of learning in a tongue-operated computer input device--part II: navigation tasks

Tongue drive system (TDS) is a novel tongue-operated assistive technology (AT) for the mobility impaired, to empower them to access computers and drive powered wheelchairs (PWC) using their free voluntary tongue motion. We have evaluated the TDS performance in five sessions over 5-8 weeks to study the learning process in different tasks of computer access and PWC navigation on nine able-bodied subjects who already had tongue piercing and used our magnetic tongue studs throughout the trial. Computer access tasks included on-screen maze navigation and issuing random commands to measure the TDS information transfer rate. PWC navigation included driving through a ~50-m obstacle course using three control strategies. Some of the qualitative aspects of using the TDS were also evaluated based on the two Likert scale questionnaires, one of which was short (eight questions) and asked at the end of each session and the other one (46 questions) at the end of the trial. Included in this study was also a task to measure the tongue fatigue as a result of using the TDS continuously for a few hours. All performance measures showed significant improvement from the first to the second session as well as further gradual improvements throughout the rest of the sessions, suggesting a rapid learning process.

Evaluation of a smartphone platform as a wireless interface between tongue drive system and electric-powered wheelchairs

Tongue drive system (TDS) is a new wireless assistive technology (AT) for the mobility impaired population. It provides users with the ability to drive powered wheelchairs (PWC) and access computers using their unconstrained tongue motion. Migration of the TDS processing unit and user interface platform from a bulky personal computer to a smartphone (iPhone) has significantly facilitated its usage by turning it into a true wireless and wearable AT. After implementation of the necessary interfacing hardware and software to allow the smartphone to act as a bridge between the TDS and PWC, the wheelchair navigation performance and associated learning was evaluated in nine able-bodied subjects in five sessions over a 5-week period. Subjects wore magnetic tongue studs over the duration of the study and drove the PWC in an obstacle course with their tongue using three different navigation strategies; namely unlatched, latched, and semiproportional. Qualitative aspects of using the TDS-iPhone-PWC interface were also evaluated via a five-point Likert scale questionnaire. Subjects showed more than 20% improvement in the overall completion time between the first and second sessions, and maintained a modest improvement of ∼9% per session over the following three sessions.

Using speech recognition to enhance the Tongue Drive System functionality in computer access

Tongue Drive System (TDS) is a wireless tongue operated assistive technology (AT), which can enable people with severe physical disabilities to access computers and drive powered wheelchairs using their volitional tongue movements. TDS offers six discrete commands, simultaneously available to the users, for pointing and typing as a substitute for mouse and keyboard in computer access, respectively. To enhance the TDS performance in typing, we have added a microphone, an audio codec, and a wireless audio link to its readily available 3-axial magnetic sensor array, and combined it with a commercially available speech recognition software, the Dragon Naturally Speaking, which is regarded as one of the most efficient ways for text entry. Our preliminary evaluations indicate that the combined TDS and speech recognition technologies can provide end users with significantly higher performance than using each technology alone, particularly in completing tasks that require both pointing and text entry, such as web surfing.