Fusion Models for Generalized Classification of Multi-Axial Human Movement: Validation in Sport Performance

We introduce a set of input models for fusing information from ensembles of wearable sensors supporting human performance and telemedicine. Veracity is demonstrated in action classification related to sport, specifically strikes in boxing and taekwondo. Four input models, formulated to be compatible with a broad range of classifiers, are introduced and two diverse classifiers, dynamic time warping (DTW) and convolutional neural networks (CNNs) are implemented in conjunction with the input models. Seven classification models fusing information at the input-level, output-level, and a combination of both are formulated. Action classification for 18 boxing punches and 24 taekwondo kicks demonstrate our fusion classifiers outperform the best DTW and CNN uni-axial classifiers. Furthermore, although DTW is ostensibly an ideal choice for human movements experiencing non-linear variations, our results demonstrate deep learning fusion classifiers outperform DTW. This is a novel finding given that CNNs are normally designed for multi-dimensional data and do not specifically compensate for non-linear variations within signal classes. The generalized formulation enables subject-specific movement classification in a feature-blind fashion with trivial computational expense for trained CNNs. A commercial boxing system, 'Corner', has been produced for real-world mass-market use based on this investigation providing a basis for future telemedicine translation.

Earable TEMPO: A Novel, Hands-Free Input Device that Uses the Movement of the Tongue Measured with a Wearable Ear Sensor

In this study, an earphone-type interface named “earable TEMPO” was developed for hands-free operation, wherein the user can control the device by simply pushing the tongue against the roof of the mouth for about one second. This interface can be used to start and stop the music from a portable audio player. The earable TEMPO uses an earphone-type sensor equipped with a light emitting diode (LED) and a phototransistor to optically measure shape variations that occur in the external auditory meatus when the tongue is pressed against the roof of the mouth. To evaluate the operation of the earable TEMPO, experiments were performed on five subjects (men and women aged 22–58) while resting, chewing gum (representing mastication), and walking. The average accuracy was 100% while resting and chewing and 99% while walking. The precision was 100% under all conditions. The average recall value of the five subjects was 92%, 90%, and 48% while resting, masticating, and walking, respectively. All subjects were reliably able to perform the action of pressing the tongue against the roof of the mouth. The measured shape variations in the ear canal were highly reproducible, indicating that this method is suitable for various applications such as controlling a portable audio player.

Design of a wearable interface for lightweight robotic arm for people with mobility impairments

Many common activities of daily living like open a door or fill a glass of water, which most of us take for granted, could be an insuperable problem for people who have limited mobility or impairments. For years the unique alternative to overcame this limitation was asking for human help. Nowadays thanks to recent studies and technology developments, having an assistive devices to compensate the loss of mobility is becoming a real opportunity. Off-the-shelf assistive robotic manipulators have the capability to improve the life of people with motor impairments. Robotic lightweight arms represent one of the most spread solution, in particular some of them are designed specifically to be mounted on wheelchairs to assist users in performing manipulation tasks. On the other hand, usually their control interface relies on joystick and buttons, making the use very challenging for people affected by impaired motor abilities. In this paper, we present a novel wearable control interface for users with limb mobility impairments. We make use of muscles residual motion capabilities, captured through a Body-Machine Interface based on a combination of head tilt estimation and electromyography signals. The proposed BMI is completely wearable, wireless and does not require frequently long calibrations. Preliminary experiments showed the effectiveness of the proposed system for subjects with motor impairments, allowing them to easily control a robotic arm for activities of daily living.

A Dual Mode Human-Robot Teleoperation Interface Based on Airflow in the Aural Cavity

Robot teleoperation systems have been limited in their utility due to the need for operator motion, lack of portability and limitation to singular input modalities. In this article, the design and construction of a dual-mode human—machine interface system for robot teleoperation addressing all these issues is presented. The interface is capable of directing robotic devices in response to tongue movement and/or speech without insertion of any device in the vicinity of the oral cavity. The interface is centered on the unique properties of the human ear as an acoustic output device. Specifically, we present: (1) an analysis of the sensitivity of human ear canals as acoustic output device; (2) the design of a new sensor for monitoring airflow in the aural canal; (3) pattern recognition procedures for recognition of both speech and tongue movement by monitoring aural flow across several human test subjects; and (4) a conceptual design and simulation of the machine interface system. We believe this work will lay the foundation for a new generation of human machine interface systems for all manner of robotic applications.

Noninvasive tongue-motion controlled computer mouse for the disabled

BACKGROUND

A patient whose spinal cord was damaged due to accident may result in Tetraplegia or lose the ability to control his/her daily living environment. Currently, patients must use an invasive tool tongue movement, to help the patient communicate with the external environment.

OBJECTIVE

This study designed a non-invasive tongue movement computer mouse system that allows the patient to use tongue movement to control a computer to communicate with the external environment.

METHODS

Via a pressure sensor and assistive holder designed in this study, the pressure sensor can be moved using the assistive holder close to the mylohyoid muscle of the patient's lower jaw. The changes in pressure from the mylohyoid muscle are converted into computer mouse control signals to control a computer to communicate with the external environment.

RESULTS

This study is based on ISO9241-Part 9 to design four kinds of training modes with varying difficulties. The data were collected from five able persons participating in the test over 7 days. The data includes throughput, path efficiency, test completion time and reaction time. The data verifies that the proposed system is stable and practical for persons with disabilities.

CONCLUSION

The non-invasive computer mouse system for sensing tongue movement can completely breakthrough the limitations of the invasive tongue movement sensing system. This study uses non-invasive, simple tongue movements that correspond to the stretching and shrinking of the lower jaw mylohyoid muscle to control the computer mouse.

Integrated vision-based robotic arm interface for operators with upper limb mobility impairments

An integrated, computer vision-based system was developed to operate a commercial wheelchair-mounted robotic manipulator (WMRM). In this paper, a gesture recognition interface system developed specifically for individuals with upper-level spinal cord injuries (SCIs) was combined with object tracking and face recognition systems to be an efficient, hands-free WMRM controller. In this test system, two Kinect cameras were used synergistically to perform a variety of simple object retrieval tasks. One camera was used to interpret the hand gestures to send as commands to control the WMRM and locate the operator's face for object positioning. The other sensor was used to automatically recognize different daily living objects for test subjects to select. The gesture recognition interface incorporated hand detection, tracking and recognition algorithms to obtain a high recognition accuracy of 97.5% for an eight-gesture lexicon. An object recognition module employing Speeded Up Robust Features (SURF) algorithm was performed and recognition results were sent as a command for "coarse positioning" of the robotic arm near the selected daily living object. Automatic face detection was also provided as a shortcut for the subjects to position the objects to the face by using a WMRM. Completion time tasks were conducted to compare manual (gestures only) and semi-manual (gestures, automatic face detection and object recognition) WMRM control modes. The use of automatic face and object detection significantly increased the completion times for retrieving a variety of daily living objects.

The tongue enables computer and wheelchair control for people with spinal cord injury

The Tongue Drive System (TDS) is a wireless and wearable assistive technology, designed to allow individuals with severe motor impairments such as tetraplegia to access their environment using voluntary tongue motion. Previous TDS trials used a magnetic tracer temporarily attached to the top surface of the tongue with tissue adhesive. We investigated TDS efficacy for controlling a computer and driving a powered wheelchair in two groups of able-bodied subjects and a group of volunteers with spinal cord injury (SCI) at C6 or above. All participants received a magnetic tongue barbell and used the TDS for five to six consecutive sessions. The performance of the group was compared for TDS versus keypad and TDS versus a sip-and-puff device (SnP) using accepted measures of speed and accuracy. All performance measures improved over the course of the trial. The gap between keypad and TDS performance narrowed for able-bodied subjects. Despite participants with SCI already having familiarity with the SnP, their performance measures were up to three times better with the TDS than with the SnP and continued to improve. TDS flexibility and the inherent characteristics of the human tongue enabled individuals with high-level motor impairments to access computers and drive wheelchairs at speeds that were faster than traditional assistive technologies but with comparable accuracy.

Tongue motion-based operation of support system for paralyzed patients

This paper proposes a new control device based on tongue motions to control and communicate with a support system for a paralyzed patient. We focus on the tongue movements as one of output of human intentions, because the tongue has one of capable parts for the motions and it does not affected by spinal cord damage. The tongue motion is easily observed from his/her mouse inside, it is, however, hard to observe them from outside. We therefore propose a detection algorithm of the tongue motions by using multiple array electrodes attached on a skin surface around a neck. The tongue motions are detected based on the center position of distributions of muscle elctric potentials that are measured by the electrodes. We investigated the precisions of the detection algorithm through some experiments and then confirmed that almost accucracy of discrimination is more than 70 as for six tongue movements such as left, right, forward, back, up, and down. Additionally, we evaluated operability of the proposed algorithm quantitatively using Fitts' law based test bed GUI, and the performance of the proposed interface was compared with that of other available interfaces.

Ensemble classification for robust discrimination of multi-channel, multi-class tongue-movement ear pressure signals

In this paper we introduce a robust classification framework for tongue-movement ear pressure signals based around an ensemble voting methodology. The ensemble members are comprised of different combinations of sensor inputs i.e. two in-ear microphones and an acoustic gel sensor positioned under the chin of the individual and classification using three different base models. It is shown that by using all nine ensemble members when compared to the individual (base) models, the average misclassification rate can be reduced from 23% to 2.8% when using the majority voting strategy. The correct classification rate is improved from 76% to 92.4% when utilizing either the borda count or condorcet methods. This is achieved through a combination of rejection based on ambiguity in the ensemble and diversity in the misclassified instances across the ensemble members.

Evaluation of a wireless wearable tongue-computer interface by individuals with high-level spinal cord injuries

The tongue drive system (TDS) is an unobtrusive, minimally invasive, wearable and wireless tongue-computer interface (TCI), which can infer its users' intentions, represented in their volitional tongue movements, by detecting the position of a small permanent magnetic tracer attached to the users' tongues. Any specific tongue movements can be translated into user-defined commands and used to access and control various devices in the users' environments. The latest external TDS (eTDS) prototype is built on a wireless headphone and interfaced to a laptop PC and a powered wheelchair. Using customized sensor signal processing algorithms and graphical user interface, the eTDS performance was evaluated by 13 naive subjects with high-level spinal cord injuries (C2-C5) at the Shepherd Center in Atlanta, GA. Results of the human trial show that an average information transfer rate of 95 bits/min was achieved for computer access with 82% accuracy. This information transfer rate is about two times higher than the EEG-based BCIs that are tested on human subjects. It was also demonstrated that the subjects had immediate and full control over the powered wheelchair to the extent that they were able to perform complex wheelchair navigation tasks, such as driving through an obstacle course.

A wavelet denoising approach for signal action isolation in the ear canal

The goal of this work was to develop and implement a new filtering strategy to denoise acoustic signals in the ear canal resulting from voluntary movement of the tongue (as a method of generating control input), as well as from other active actions, (speech, eating, drinking, smoking), and passive actions (swallowing, adjusting the jaw, physiological activity). The strategy is based on a denoising wavelet shrinkage approach that separates rhythmic bursting activity and white noise representing sustained tonic activity. While past work has addressed the discrimination of voluntary TMEP signals from one-another, no work has addressed acoustic artefact rejection within the ear. The results described here, combined with our past work in isolating critical components of tongue movement ear pressure (TMEP) signals, provide a basis for discriminating voluntary and involuntary actions of the tongue by monitoring pressure in the ear. At this time, the system has worked in real-time for assistive device control.