1
|
Fusion Models for Generalized Classification of Multi-Axial Human Movement: Validation in Sport Performance. SENSORS 2021; 21:s21248409. [PMID: 34960500 PMCID: PMC8706912 DOI: 10.3390/s21248409] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 10/16/2021] [Revised: 12/06/2021] [Accepted: 12/10/2021] [Indexed: 11/17/2022]
Abstract
We introduce a set of input models for fusing information from ensembles of wearable sensors supporting human performance and telemedicine. Veracity is demonstrated in action classification related to sport, specifically strikes in boxing and taekwondo. Four input models, formulated to be compatible with a broad range of classifiers, are introduced and two diverse classifiers, dynamic time warping (DTW) and convolutional neural networks (CNNs) are implemented in conjunction with the input models. Seven classification models fusing information at the input-level, output-level, and a combination of both are formulated. Action classification for 18 boxing punches and 24 taekwondo kicks demonstrate our fusion classifiers outperform the best DTW and CNN uni-axial classifiers. Furthermore, although DTW is ostensibly an ideal choice for human movements experiencing non-linear variations, our results demonstrate deep learning fusion classifiers outperform DTW. This is a novel finding given that CNNs are normally designed for multi-dimensional data and do not specifically compensate for non-linear variations within signal classes. The generalized formulation enables subject-specific movement classification in a feature-blind fashion with trivial computational expense for trained CNNs. A commercial boxing system, 'Corner', has been produced for real-world mass-market use based on this investigation providing a basis for future telemedicine translation.
Collapse
|
2
|
Glossokinetic potential based tongue–machine interface for 1-D extraction. AUSTRALASIAN PHYSICAL & ENGINEERING SCIENCES IN MEDICINE 2018; 41:379-391. [PMID: 29633174 DOI: 10.1007/s13246-018-0635-x] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/26/2017] [Accepted: 04/04/2018] [Indexed: 10/17/2022]
|
3
|
Earable TEMPO: A Novel, Hands-Free Input Device that Uses the Movement of the Tongue Measured with a Wearable Ear Sensor. SENSORS 2018; 18:s18030733. [PMID: 29494482 PMCID: PMC5876610 DOI: 10.3390/s18030733] [Citation(s) in RCA: 10] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 12/23/2017] [Revised: 02/12/2018] [Accepted: 02/27/2018] [Indexed: 11/30/2022]
Abstract
In this study, an earphone-type interface named “earable TEMPO” was developed for hands-free operation, wherein the user can control the device by simply pushing the tongue against the roof of the mouth for about one second. This interface can be used to start and stop the music from a portable audio player. The earable TEMPO uses an earphone-type sensor equipped with a light emitting diode (LED) and a phototransistor to optically measure shape variations that occur in the external auditory meatus when the tongue is pressed against the roof of the mouth. To evaluate the operation of the earable TEMPO, experiments were performed on five subjects (men and women aged 22–58) while resting, chewing gum (representing mastication), and walking. The average accuracy was 100% while resting and chewing and 99% while walking. The precision was 100% under all conditions. The average recall value of the five subjects was 92%, 90%, and 48% while resting, masticating, and walking, respectively. All subjects were reliably able to perform the action of pressing the tongue against the roof of the mouth. The measured shape variations in the ear canal were highly reproducible, indicating that this method is suitable for various applications such as controlling a portable audio player.
Collapse
|
4
|
Gorur K, Bozkurt MR, Bascil MS, Temurtas F. Glossokinetic potential based tongue–machine interface for 1-D extraction using neural networks. Biocybern Biomed Eng 2018. [DOI: 10.1016/j.bbe.2018.06.004] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/28/2022]
|
5
|
Baldi TL, Spagnoletti G, Dragusanu M, Prattichizzo D. Design of a wearable interface for lightweight robotic arm for people with mobility impairments. IEEE Int Conf Rehabil Robot 2017; 2017:1567-1573. [PMID: 28814043 DOI: 10.1109/icorr.2017.8009471] [Citation(s) in RCA: 13] [Impact Index Per Article: 1.9] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Abstract
Many common activities of daily living like open a door or fill a glass of water, which most of us take for granted, could be an insuperable problem for people who have limited mobility or impairments. For years the unique alternative to overcame this limitation was asking for human help. Nowadays thanks to recent studies and technology developments, having an assistive devices to compensate the loss of mobility is becoming a real opportunity. Off-the-shelf assistive robotic manipulators have the capability to improve the life of people with motor impairments. Robotic lightweight arms represent one of the most spread solution, in particular some of them are designed specifically to be mounted on wheelchairs to assist users in performing manipulation tasks. On the other hand, usually their control interface relies on joystick and buttons, making the use very challenging for people affected by impaired motor abilities. In this paper, we present a novel wearable control interface for users with limb mobility impairments. We make use of muscles residual motion capabilities, captured through a Body-Machine Interface based on a combination of head tilt estimation and electromyography signals. The proposed BMI is completely wearable, wireless and does not require frequently long calibrations. Preliminary experiments showed the effectiveness of the proposed system for subjects with motor impairments, allowing them to easily control a robotic arm for activities of daily living.
Collapse
|
6
|
Vaidyanathan R, Fargues MP, Serdar Kurcan R, Gupta L, Kota S, Quinn RD, Lin D. A Dual Mode Human-Robot Teleoperation Interface Based on Airflow in the Aural Cavity. Int J Rob Res 2016. [DOI: 10.1177/0278364907082612] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
Abstract
Robot teleoperation systems have been limited in their utility due to the need for operator motion, lack of portability and limitation to singular input modalities. In this article, the design and construction of a dual-mode human—machine interface system for robot teleoperation addressing all these issues is presented. The interface is capable of directing robotic devices in response to tongue movement and/or speech without insertion of any device in the vicinity of the oral cavity. The interface is centered on the unique properties of the human ear as an acoustic output device. Specifically, we present: (1) an analysis of the sensitivity of human ear canals as acoustic output device; (2) the design of a new sensor for monitoring airflow in the aural canal; (3) pattern recognition procedures for recognition of both speech and tongue movement by monitoring aural flow across several human test subjects; and (4) a conceptual design and simulation of the machine interface system. We believe this work will lay the foundation for a new generation of human machine interface systems for all manner of robotic applications.
Collapse
Affiliation(s)
- Ravi Vaidyanathan
- University of Southampton, Southampton, UK Naval Postgraduate School, Monterey, CA, USA Case Western Reserve University, OH, USA
| | | | | | - Lalit Gupta
- Southern Illinois University, Carbondale, IL, USA
| | | | | | - Dong Lin
- Think-A-Move, Ltd, Beachwood, OH, USA
| |
Collapse
|
7
|
Chou CH, Hwang YS, Chen CC, Chen SC, Chou SW, Chen YL. Noninvasive tongue-motion controlled computer mouse for the disabled. Technol Health Care 2016; 24:401-8. [PMID: 26835730 DOI: 10.3233/thc-161137] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
Abstract
BACKGROUND A patient whose spinal cord was damaged due to accident may result in Tetraplegia or lose the ability to control his/her daily living environment. Currently, patients must use an invasive tool tongue movement, to help the patient communicate with the external environment. OBJECTIVE This study designed a non-invasive tongue movement computer mouse system that allows the patient to use tongue movement to control a computer to communicate with the external environment. METHODS Via a pressure sensor and assistive holder designed in this study, the pressure sensor can be moved using the assistive holder close to the mylohyoid muscle of the patient's lower jaw. The changes in pressure from the mylohyoid muscle are converted into computer mouse control signals to control a computer to communicate with the external environment. RESULTS This study is based on ISO9241-Part 9 to design four kinds of training modes with varying difficulties. The data were collected from five able persons participating in the test over 7 days. The data includes throughput, path efficiency, test completion time and reaction time. The data verifies that the proposed system is stable and practical for persons with disabilities. CONCLUSION The non-invasive computer mouse system for sensing tongue movement can completely breakthrough the limitations of the invasive tongue movement sensing system. This study uses non-invasive, simple tongue movements that correspond to the stretching and shrinking of the lower jaw mylohyoid muscle to control the computer mouse.
Collapse
Affiliation(s)
- C-H Chou
- Department of Electronic Engineering and Graduate Institute of Computer and Communication Engineering, National Taipei University of Technology, Taipei, Taiwan
| | - Y-S Hwang
- Department of Electronic Engineering and Graduate Institute of Computer and Communication Engineering, National Taipei University of Technology, Taipei, Taiwan
| | - C-C Chen
- Department of Management Information Systems, Hwa Hsia University of Technology, New Taipei City, Taiwan
| | - S-C Chen
- Department of Physical Medicine and Rehabilitation, School of Medicine, College of Medicine, Taipei Medical University, Taipei, Taiwan
| | - S-W Chou
- Department of Applied Mathematics, Chinese Culture University, Taipei, Taiwan
| | - Y-L Chen
- Department of Digital Technology Design, National Taipei University of Education, Taipei, Taiwan
| |
Collapse
|
8
|
Jiang H, Wachs JP, Duerstock BS. Integrated vision-based robotic arm interface for operators with upper limb mobility impairments. IEEE Int Conf Rehabil Robot 2014; 2013:6650447. [PMID: 24187264 DOI: 10.1109/icorr.2013.6650447] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
Abstract
An integrated, computer vision-based system was developed to operate a commercial wheelchair-mounted robotic manipulator (WMRM). In this paper, a gesture recognition interface system developed specifically for individuals with upper-level spinal cord injuries (SCIs) was combined with object tracking and face recognition systems to be an efficient, hands-free WMRM controller. In this test system, two Kinect cameras were used synergistically to perform a variety of simple object retrieval tasks. One camera was used to interpret the hand gestures to send as commands to control the WMRM and locate the operator's face for object positioning. The other sensor was used to automatically recognize different daily living objects for test subjects to select. The gesture recognition interface incorporated hand detection, tracking and recognition algorithms to obtain a high recognition accuracy of 97.5% for an eight-gesture lexicon. An object recognition module employing Speeded Up Robust Features (SURF) algorithm was performed and recognition results were sent as a command for "coarse positioning" of the robotic arm near the selected daily living object. Automatic face detection was also provided as a shortcut for the subjects to position the objects to the face by using a WMRM. Completion time tasks were conducted to compare manual (gestures only) and semi-manual (gestures, automatic face detection and object recognition) WMRM control modes. The use of automatic face and object detection significantly increased the completion times for retrieving a variety of daily living objects.
Collapse
|
9
|
Kim J, Park H, Bruce J, Sutton E, Rowles D, Pucci D, Holbrook J, Minocha J, Nardone B, West D, Laumann A, Roth E, Jones M, Veledar E, Ghovanloo M. The tongue enables computer and wheelchair control for people with spinal cord injury. Sci Transl Med 2014; 5:213ra166. [PMID: 24285485 DOI: 10.1126/scitranslmed.3006296] [Citation(s) in RCA: 45] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/02/2022]
Abstract
The Tongue Drive System (TDS) is a wireless and wearable assistive technology, designed to allow individuals with severe motor impairments such as tetraplegia to access their environment using voluntary tongue motion. Previous TDS trials used a magnetic tracer temporarily attached to the top surface of the tongue with tissue adhesive. We investigated TDS efficacy for controlling a computer and driving a powered wheelchair in two groups of able-bodied subjects and a group of volunteers with spinal cord injury (SCI) at C6 or above. All participants received a magnetic tongue barbell and used the TDS for five to six consecutive sessions. The performance of the group was compared for TDS versus keypad and TDS versus a sip-and-puff device (SnP) using accepted measures of speed and accuracy. All performance measures improved over the course of the trial. The gap between keypad and TDS performance narrowed for able-bodied subjects. Despite participants with SCI already having familiarity with the SnP, their performance measures were up to three times better with the TDS than with the SnP and continued to improve. TDS flexibility and the inherent characteristics of the human tongue enabled individuals with high-level motor impairments to access computers and drive wheelchairs at speeds that were faster than traditional assistive technologies but with comparable accuracy.
Collapse
Affiliation(s)
- Jeonghee Kim
- GT-Bionics Lab, School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA 30308, USA
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
Collapse
|
10
|
Mamun KA, Mace M, Gupta L, Verschuur CA, Lutman ME, Stokes M, Vaidyanathan R, Wang S. Robust real-time identification of tongue movement commands from interferences. Neurocomputing 2012. [DOI: 10.1016/j.neucom.2011.09.018] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/15/2022]
|
11
|
Takahashi J, Suezawa S, Hasegawa Y, Sankai Y. Tongue motion-based operation of support system for paralyzed patients. IEEE Int Conf Rehabil Robot 2012; 2011:5975359. [PMID: 22275563 DOI: 10.1109/icorr.2011.5975359] [Citation(s) in RCA: 8] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
Abstract
This paper proposes a new control device based on tongue motions to control and communicate with a support system for a paralyzed patient. We focus on the tongue movements as one of output of human intentions, because the tongue has one of capable parts for the motions and it does not affected by spinal cord damage. The tongue motion is easily observed from his/her mouse inside, it is, however, hard to observe them from outside. We therefore propose a detection algorithm of the tongue motions by using multiple array electrodes attached on a skin surface around a neck. The tongue motions are detected based on the center position of distributions of muscle elctric potentials that are measured by the electrodes. We investigated the precisions of the detection algorithm through some experiments and then confirmed that almost accucracy of discrimination is more than 70 as for six tongue movements such as left, right, forward, back, up, and down. Additionally, we evaluated operability of the proposed algorithm quantitatively using Fitts' law based test bed GUI, and the performance of the proposed interface was compared with that of other available interfaces.
Collapse
Affiliation(s)
- Junji Takahashi
- Graduate School of Systems and Information Engineering, University of Tsukuba, Ibaraki, 305-8573, Japan.
| | | | | | | |
Collapse
|
12
|
Mace M, Abdullah-Al-Mamun K, Wang S, Gupta L, Vaidyanathan R. Ensemble classification for robust discrimination of multi-channel, multi-class tongue-movement ear pressure signals. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2012; 2011:1733-6. [PMID: 22254661 DOI: 10.1109/iembs.2011.6090496] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
Abstract
In this paper we introduce a robust classification framework for tongue-movement ear pressure signals based around an ensemble voting methodology. The ensemble members are comprised of different combinations of sensor inputs i.e. two in-ear microphones and an acoustic gel sensor positioned under the chin of the individual and classification using three different base models. It is shown that by using all nine ensemble members when compared to the individual (base) models, the average misclassification rate can be reduced from 23% to 2.8% when using the majority voting strategy. The correct classification rate is improved from 76% to 92.4% when utilizing either the borda count or condorcet methods. This is achieved through a combination of rejection based on ambiguity in the ensemble and diversity in the misclassified instances across the ensemble members.
Collapse
Affiliation(s)
- Michael Mace
- Department of Mechanical Engineering, Imperial College London, London, UK.
| | | | | | | | | |
Collapse
|
13
|
Kim DJ, Hazlett-Knudsen R, Culver-Godfrey H, Rucks G, Cunningham T, Portee D, Bricout J, Wang Z, Behal A. How Autonomy Impacts Performance and Satisfaction: Results From a Study With Spinal Cord Injured Subjects Using an Assistive Robot. ACTA ACUST UNITED AC 2012. [DOI: 10.1109/tsmca.2011.2159589] [Citation(s) in RCA: 87] [Impact Index Per Article: 7.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
|
14
|
Huo X, Ghovanloo M. Evaluation of a wireless wearable tongue-computer interface by individuals with high-level spinal cord injuries. J Neural Eng 2010; 7:26008. [PMID: 20332552 DOI: 10.1088/1741-2560/7/2/026008] [Citation(s) in RCA: 80] [Impact Index Per Article: 5.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/11/2022]
Abstract
The tongue drive system (TDS) is an unobtrusive, minimally invasive, wearable and wireless tongue-computer interface (TCI), which can infer its users' intentions, represented in their volitional tongue movements, by detecting the position of a small permanent magnetic tracer attached to the users' tongues. Any specific tongue movements can be translated into user-defined commands and used to access and control various devices in the users' environments. The latest external TDS (eTDS) prototype is built on a wireless headphone and interfaced to a laptop PC and a powered wheelchair. Using customized sensor signal processing algorithms and graphical user interface, the eTDS performance was evaluated by 13 naive subjects with high-level spinal cord injuries (C2-C5) at the Shepherd Center in Atlanta, GA. Results of the human trial show that an average information transfer rate of 95 bits/min was achieved for computer access with 82% accuracy. This information transfer rate is about two times higher than the EEG-based BCIs that are tested on human subjects. It was also demonstrated that the subjects had immediate and full control over the powered wheelchair to the extent that they were able to perform complex wheelchair navigation tasks, such as driving through an obstacle course.
Collapse
Affiliation(s)
- Xueliang Huo
- GT-Bionics Lab, School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA, USA
| | | |
Collapse
|
15
|
Gupta L, Kota S, Murali S, Molfese D, Vaidyanathan R. A Feature Ranking Strategy to Facilitate Multivariate Signal Classification. ACTA ACUST UNITED AC 2010. [DOI: 10.1109/tsmcc.2009.2024648] [Citation(s) in RCA: 14] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
|
16
|
Magee J, Betke M, Gips J, Scott M, Waber B. A Human–Computer Interface Using Symmetry Between Eyes to Detect Gaze Direction. ACTA ACUST UNITED AC 2008. [DOI: 10.1109/tsmca.2008.2003466] [Citation(s) in RCA: 40] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
|
17
|
Vaidyanathan R, Wang S, Gupta L. A wavelet denoising approach for signal action isolation in the ear canal. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2008; 2008:2677-2680. [PMID: 19163256 DOI: 10.1109/iembs.2008.4649753] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/27/2023]
Abstract
The goal of this work was to develop and implement a new filtering strategy to denoise acoustic signals in the ear canal resulting from voluntary movement of the tongue (as a method of generating control input), as well as from other active actions, (speech, eating, drinking, smoking), and passive actions (swallowing, adjusting the jaw, physiological activity). The strategy is based on a denoising wavelet shrinkage approach that separates rhythmic bursting activity and white noise representing sustained tonic activity. While past work has addressed the discrimination of voluntary TMEP signals from one-another, no work has addressed acoustic artefact rejection within the ear. The results described here, combined with our past work in isolating critical components of tongue movement ear pressure (TMEP) signals, provide a basis for discriminating voluntary and involuntary actions of the tongue by monitoring pressure in the ear. At this time, the system has worked in real-time for assistive device control.
Collapse
|