Design and Evaluation of a Hands-Free Video Game Controller for Individuals With Motor Impairments

Over the past few decades, video gaming has evolved at a tremendous rate although game input methods have been slower to change. Game input methods continue to rely on two-handed control of the joystick and D-pad or the keyboard and mouse for simultaneously controlling player movement and camera actions. Bi-manual input poses a significant play impediment to those with severe motor impairments. In this work, we propose and evaluate a hands-free game input control method that uses real-time facial expression recognition. Through our novel input method, our goal is to enable and empower individuals with neurological and neuromuscular diseases, who may lack hand muscle control, to be able to independently play video games. To evaluate the usability and acceptance of our system, we conducted a remote user study with eight severely motor-impaired individuals. Our results indicate high user satisfaction and greater preference for our input system with participants rating the input system as easy to learn. With this work, we aim to highlight that facial expression recognition can be a valuable input method.

A vision-free brain-computer interface (BCI) paradigm based on auditory selective attention

Majority of the recently developed brain computer interface (BCI) systems have been using visual stimuli or visual feedbacks. However, the BCI paradigms based on visual perception might not be applicable to severe locked-in patients who have lost their ability to control their eye movement or even their vision. In the present study, we investigated the feasibility of a vision-free BCI paradigm based on auditory selective attention. We used the power difference of auditory steady-state responses (ASSRs) when the participant modulates his/her attention to the target auditory stimulus. The auditory stimuli were constructed as two pure-tone burst trains with different beat frequencies (37 and 43 Hz) which were generated simultaneously from two speakers located at different positions (left and right). Our experimental results showed high classification accuracies (64.67%, 30 commands/min, information transfer rate (ITR) = 1.89 bits/min; 74.00%, 12 commands/min, ITR = 2.08 bits/min; 82.00%, 6 commands/min, ITR = 1.92 bits/min; 84.33%, 3 commands/min, ITR = 1.12 bits/min; without any artifact rejection, inter-trial interval = 6 sec), enough to be used for a binary decision. Based on the suggested paradigm, we implemented a first online ASSR-based BCI system that demonstrated the possibility of materializing a totally vision-free BCI system.

Classification of selective attention to auditory stimuli: toward vision-free brain-computer interfacing

Brain-computer interface (BCI) is a developing, novel mode of communication for individuals with severe motor impairments or those who have no other options for communication aside from their brain signals. However, the majority of current BCI systems are based on visual stimuli or visual feedback, which may not be applicable for severe locked-in patients that have lost their eyesight or the ability to control their eye movements. In the present study, we investigated the feasibility of using auditory steady-state responses (ASSRs), elicited by selective attention to a specific sound source, as an electroencephalography (EEG)-based BCI paradigm. In our experiment, two pure tone burst trains with different beat frequencies (37 and 43 Hz) were generated simultaneously from two speakers located at different positions (left and right). Six participants were instructed to close their eyes and concentrate their attention on either auditory stimulus according to the instructions provided randomly through the speakers during the inter-stimulus interval. EEG signals were recorded at multiple electrodes mounted over the temporal, occipital, and parietal cortices. We then extracted feature vectors by combining spectral power densities evaluated at the two beat frequencies. Our experimental results showed high classification accuracies (64.67%, 30 commands/min, information transfer rate (ITR) = 1.89 bits/min; 74.00%, 12 commands/min, ITR = 2.08 bits/min; 82.00%, 6 commands/min, ITR = 1.92 bits/min; 84.33%, 3 commands/min, ITR = 1.12 bits/min; without any artifact rejection, inter-trial interval = 6s), enough to be used for a binary decision. Based on the suggested paradigm, we implemented a first online ASSR-based BCI system that demonstrated the possibility of materializing a totally vision-free BCI system.

Research on the relation of EEG signal chaos characteristics with high-level intelligence activity of human brain

Using phase space reconstruct technique from one-dimensional and multi-dimensional time series and the quantitative criterion rule of system chaos, and combining the neural network; analyses, computations and sort are conducted on electroencephalogram (EEG) signals of five kinds of human consciousness activities (relaxation, mental arithmetic of multiplication, mental composition of a letter, visualizing a 3-dimensional object being revolved about an axis, and visualizing numbers being written or erased on a blackboard). Through comparative studies on the determinacy, the phase graph, the power spectra, the approximate entropy, the correlation dimension and the Lyapunov exponent of EEG signals of 5 kinds of consciousness activities, the following conclusions are shown: (1) The statistic results of the deterministic computation indicate that chaos characteristic may lie in human consciousness activities, and central tendency measure (CTM) is consistent with phase graph, so it can be used as a division way of EEG attractor. (2) The analyses of power spectra show that ideology of single subject is almost identical but the frequency channels of different consciousness activities have slight difference. (3) The approximate entropy between different subjects exist discrepancy. Under the same conditions, the larger the approximate entropy of subject is, the better the subject's innovation is. (4) The results of the correlation dimension and the Lyapunov exponent indicate that activities of human brain exist in attractors with fractional dimensions. (5) Nonlinear quantitative criterion rule, which unites the neural network, can classify different kinds of consciousness activities well. In this paper, the results of classification indicate that the consciousness activity of arithmetic has better differentiation degree than that of abstract.

Infrared-based communication augmentation system for people with multiple disabilities

PURPOSE

This study describes an eyeglass-type infrared-based communication board for the nonspeaking with quadriplegia.

METHOD

This system is composed of four major components: a headset, an infrared transmitting module, an infrared receiving/signal-processing module, and a main controller, the Intel-8951 microprocessor. This design concept was based on the use of an infrared remote module fastened to the eyeglasses which could allow the convenient control of the input motion on the keys of a communication board, which are all modified with infrared receiving/signal-processing modules. For system evaluation, 12 subjects (all men, 21-45 years old, six normal subjects as the control group and six nonspeaking with quadriplegia as the experimental group) were recruited.

RESULTS

The average accuracy of the control group and the experimental group were 93.1 +/- 4.3% and 89.7 +/- 5.5%, respectively. The average time cost of the control group and the experimental group were 78.3 +/- 8.7 s and 89.9 +/- 10.2 s, respectively. An independent t-test revealed that the differences in the average accuracy and the average time cost of the control group and the experimental group were not significant (p>0.05).

CONCLUSIONS

The increase of opportunity to communicate using the infrared-based communication board would help people with multiple disabilities to socialize actively.

Plausibility assessment of a 2-state self-paced mental task-based BCI using the no-control performance analysis

The feasibility of having a self-paced brain-computer interface (BCI) based on mental tasks is investigated. The EEG signals of four subjects performing five mental tasks each are used in the design of a 2-state self-paced BCI. The output of the BCI should only be activated when the subject performs a specific mental task and should remain inactive otherwise. For each subject and each task, the feature coefficient and the classifier that yield the best performance are selected, using the autoregressive coefficients as the features. The classifier with a zero false positive rate and the highest true positive rate is selected as the best classifier. The classifiers tested include: linear discriminant analysis, quadratic discriminant analysis, Mahalanobis discriminant analysis, support vector machine, and radial basis function neural network. The results show that: (1) some classifiers obtained the desired zero false positive rate; (2) the linear discriminant analysis classifier does not yield acceptable performance; (3) the quadratic discriminant analysis classifier outperforms the Mahalanobis discriminant analysis classifier and performs almost as well as the radial basis function neural network; and (4) the support vector machine classifier has the highest true positive rates but unfortunately has nonzero false positive rates in most cases.

Feature extraction of mental task in BCI based on the method of approximate entropy

Brain computer interface (BCI) is based on processing brain signals recorded from the scalp or the surface of the cortex in order to identify the different brain states and covert to corresponded control command. The key problems in BCI research are feature extraction and classification. In this paper, two experiments were performed, and the EEG data were recording during each experiment. One experiment contains five mental tasks, including "baseline", "rotation", "multiplication", "counting" and "letter-composing", the other contains two mental tasks which are left hand imagery movement and right hand imagery movement. EEG data recorded from both experiments are analyzed by approximate entropy (Apen), which is used to extract the characteristic feature of different mental tasks. A three-layer BP Neural Network classifier was structured for pattern classification. Different results were gained from the mental task experiment and imagery movement experiment. The results show that Apen is an effective method to extract the feature of different brain states.

Preparing computers for affective communication: a psychophysiological concept and preliminary results

Despite rapid advances in technology, computers remain incapable of responding to human emotions. An exploratory study was conducted to find out what physiological parameters might be useful to differentiate among 4 emotional states, based on 2 dimensions: pleasantness versus unpleasantness and arousal versus relaxation. The 4 emotions were induced by exposing 26 undergraduate students to different combinations of olfactory and auditory stimuli, selected in a pretest from 12 stimuli by subjective ratings of arousal and valence. Changes in electroencephalographic (EEG), heart rate variability, and electrodermal measures were used to differentiate the 4 emotions. EEG activity separates pleasantness from unpleasantness only in the aroused but not in the relaxed domain, where electrodermal parameters are the differentiating ones. All three classes of parameters contribute to a separation between arousal and relaxation in the positive valence domain, whereas the latency of the electrodermal response is the only differentiating parameter in the negative domain. We discuss how such a psychophysiological approach may be incorporated into a systemic model of a computer responsive to affective communication from the user.

The camera mouse: visual tracking of body features to provide computer access for people with severe disabilities

The "Camera Mouse" system has been developed to provide computer access for people with severe disabilities. The system tracks the computer user's movements with a video camera and translates them into the movements of the mouse pointer on the screen. Body features such as the tip of the user's nose or finger can be tracked. The visual tracking algorithm is based on cropping an online template of the tracked feature from the current image frame and testing where this template correlates in the subsequent frame. The location of the highest correlation is interpreted as the new location of the feature in the subsequent frame. Various body features are examined for tracking robustness and user convenience. A group of 20 people without disabilities tested the Camera Mouse and quickly learned how to use it to spell out messages or play games. Twelve people with severe cerebral palsy or traumatic brain injury have tried the system, nine of whom have shown success. They interacted with their environment by spelling out messages and exploring the Internet.

Application of tilt sensors in human-computer mouse interface for people with disabilities

This study describes the motivation and the design considerations of an economical head-operated computer mouse. In addition, it focuses on the invention of a head-operated computer mouse that employs two tilt sensors placed in the headset to determine head position and to function as simple head-operated computer mouse. One tilt sensor detects the lateral head-motion to drive the left/right displacement of the mouse. The other one detects the head's vertical motion to move up and down with respect to the displacement of the mouse. A touch switch device was designed to contact gently with operator's cheek. Operator may puff his cheek to trigger the device to perform single click, double clicks, and drag commands. This system was invented to assist people with disabilities to live an independent professional life.

Brain-computer communication: unlocking the locked in

With the increasing efficiency of life-support systems and better intensive care, more patients survive severe injuries of the brain and spinal cord. Many of these patients experience locked-in syndrome: The active mind is locked in a paralyzed body. Consequently, communication is extremely restricted or impossible. A muscle-independent communication channel overcomes this problem and is realized through a brain-computer interface, a direct connection between brain and computer. The number of technically elaborated brain-computer interfaces is in contrast with the number of systems used in the daily life of locked-in patients. It is hypothesized that a profound knowledge and consideration of psychological principles are necessary to make brain-computer interfaces feasible for locked-in patients.

The new design of an infrared-controlled human-computer interface for the disabled

This paper reports on the development of an eyeglass- type infrared (IR)-controlled computer interface for the disabled. This system may serve to assist those who suffer from spinal cord injuries or other handicaps to operate a computer. This system is comprised of three major components: 1) an infrared transmitting module, 2) an infrared receiving/signal-processing module, and 3) a main controller, the Intel-8951 microprocessor. The infrared transmitting module utilizes tongue-touch circuitry which is converted to an infrared beam and a low power laser (<0.1 mW) beam. The infrared receiving/signal-processing module, receives the infrared beam and fine tunes the unstable infrared beam into standard pulses which are used as control signals. The main controller is responsible for detecting the input signals from the infrared receiving/signal-processing module and verifying these signals with the mapping table in its memory. After the signal is verified, it is released to control the keys of the computer keyboard and mouse interface. This design concept was mainly based on the idea that the use of an infrared remote module fastened to the eyeglasses could allow the convenient control of the input motion on the keys of a computer keyboard and mouse which are all modified with infrared receiving/signal-processing modules. The system is designed for individuals with spinal cord injuries and disabled in which the subjects' movement are severely restricted. The infrared transmitting module can be easily mounted on eyeglasses or artificial limbs.

Frequency component selection for an EEG-based brain to computer interface

A new communication channel for severely handicapped people could be opened with a direct brain to computer interface (BCI). Such a system classifies electrical brain signals online. In a series of training sessions, where electroencephalograph (EEG) signals are recorded on the intact scalp, a classifier is trained to discriminate a limited number of different brain states. In a subsequent series of feedback sessions, where the subject is confronted with the classification results, the subject tries to reduce the number of misclassifications. In this study the relevance of different spectral components is analyzed: 1) on the training sessions to select optimal frequency bands for the feedback sessions and 2) on the feedback sessions to monitor changes.

On the intercorrelation of some frequency and amplitude parameters of the human EEG and its functional significance. Communication. I: Multidimensional neurodynamic organization of functional states of the brain during intellectual, perceptive and motor activity in normal subjects

In 95 normal subjects, a separate evaluation of the amplitude and frequency parameters of EEG by period analysis made it possible to reveal, using factor analysis, four independent groups of parameters--the EEG factors, two of which being independent of the amplitude fluctuations. They were considered as integral EEG characteristics of qualitatively different neurophysiological processes. Decrease of Factor I values during mental activity (called 'general activation') reflected an intercorrelated desynchronization of the wave amplitudes in all the bands, a decrease of alpha-index (percentage presence in epoch) and regularity together with parallel increase of the indices and mean periods of delta- and theta-waves. This generalized reaction has shown 'non-specific' dependence upon novelty and difficulty of the tasks and stimuli with certain task-specific topographical distribution. An increase of values of regional Factor Ia in the anterior areas was caused by delta- and theta-amplitude synchronization, more pronounced during matching the rhymes (MR) than in mental multiplication (MM). An increase of Factor II values (related to increase of the index, frequency and regularity of beta-activity and called 'cortical excitation', CE) was more expressed during MR, whereas an increase of Factor III values (an increase of mean alpha-period and theta-index called 'active selective inhibition', ASI) was characteristic of MM, the latter reaction being evident in the right hemisphere. During analysis of external sound stimuli and rhythmical clenching of a fist, an increase of Factor III values was accompanied by decrease of Factor II values [corrected]; in the motor activity, such reciprocal reaction being localized in the central areas contralateral to the hand moved . Neuropsychological analysis suggests that CE correlates with associative and successively organized mental operations involving search for memory traces and ASI presumably relates to different aspects of mental selectivity such as simultaneous mental operations, voluntary attention and mental automation, the latter two cases being supported by parallel reduction of CE.

EEG activation patterns during the performance of tasks involving different components of mental calculation

In this study we demonstrate the existence of different patterns of EEG activation during the performance of 4 different tasks involving different components of mental calculation in normal subjects. The EEG was recorded in all monopolar leads of the 10/20 system using linked ear lobes as reference. Absolute and relative power were calculated in the delta (1.5-3.5 Hz), theta (3.5-7.5 Hz), alpha (7.5-12.5 Hz) and beta (12.5-19 Hz) bands. The tasks were presented randomly and the EEG segments preceding presentation of the stimulus were considered as the rest corresponding to the task requested by the stimulus. Tasks were of 4 different types, involving number comprehension, recognition of mathematical symbols, the calculation process and the spatial component. ANOVAs between the rest periods showed no differences in any band. Neither did ANOVAs between tasks. However, other variables (task minus rest), which were calculated as the differences in power between task and rest respectively, showed significant differences between tasks in the delta and beta bands in the frontal lobes. In addition, new variables were calculated as the difference between tasks, since many factors were common across several tasks. These variables correspond to the EEG change due to a specific component of mental calculation. Significant differences were obtained in delta and theta bands in right posterior areas and in the beta band in frontal areas. We concluded that the EEG differences observed during different components of mental calculation suggest the participation of different networks.

Multichannel EEG-based brain-computer communication

Individuals who are paralyzed or have other severe movement disorders often need alternative means for communicating with and controlling their environments. In this study, human subjects learned to use two channels of bipolar EEG activity to control 2-dimensional movement of a cursor on a computer screen. Amplitudes of 8-12 Hz activity in the EEG recorded from the scalp across right and left central sulci were determined by fast Fourier transform and combined to control vertical and horizontal cursor movements simultaneously. This independent control of two separate EEG channels cannot be attributed to a non-specific change in brain activity and appeared to be specific to the mu rhythm frequency range. With further development, multichannel EEG-based communication may prove of significant value to those with severe motor disabilities.