Feasibility and safety of shared EEG/EOG and vision-guided autonomous whole-arm exoskeleton control to perform activities of daily living

A muscle synergies-based controller to drive a powered upper-limb exoskeleton in reaching tasks

This work introduces a real-time intention decoding algorithm grounded in muscle synergies (Syn-ID). The algorithm detects the electromyographic (EMG) onset and infers the direction of the movement during reaching tasks to control a powered shoulder-elbow exoskeleton. Features related to muscle synergies are used in a Gaussian Mixture Model and probability accumulation-based logic to infer the user's movement direction. The performance of the algorithm was verified by a feasibility study including eight healthy participants. The experiments comprised a transparent session, during which the exoskeleton did not provide any assistance, and an assistive session in which the Syn-ID strategy was employed. Participants were asked to reach eight targets equally spaced on a circumference of 25 cm radius (adjusted chance level: 18.1%). The results showed an average accuracy of 48.7% after 0.6 s from the EMG onset. Most of the confusion of the estimate was found along directions adjacent to the actual one (type 1 error: 33.4%). Effects of the assistance were observed in a statistically significant reduction in the activation of Posterior Deltoid and Triceps Brachii. The final positions of the movements during the assistive session were on average 1.42 cm far from the expected ones, both when the directions were estimated correctly and when type 1 errors occurred. Therefore, combining accurate estimates with type 1 errors, we computed a modified accuracy of 82.10±6.34%. Results were benchmarked with respect to a purely kinematics-based approach. The Syn-ID showed better performance in the first portion of the movement (0.14 s after EMG onset).

Emerging Medical Technologies and Their Use in Bionic Repair and Human Augmentation

As both the proportion of older people and the length of life increases globally, a rise in age-related degenerative diseases, disability, and prolonged dependency is projected. However, more sophisticated biomedical materials, as well as an improved understanding of human disease, is forecast to revolutionize the diagnosis and treatment of conditions ranging from osteoarthritis to Alzheimer's disease as well as impact disease prevention. Another, albeit quieter, revolution is also taking place within society: human augmentation. In this context, humans seek to improve themselves, metamorphosing through self-discipline or more recently, through use of emerging medical technologies, with the goal of transcending aging and mortality. In this review, and in the pursuit of improved medical care following aging, disease, disability, or injury, we first highlight cutting-edge and emerging materials-based neuroprosthetic technologies designed to restore limb or organ function. We highlight the potential for these technologies to be utilized to augment human performance beyond the range of natural performance. We discuss and explore the growing social movement of human augmentation and the idea that it is possible and desirable to use emerging technologies to push the boundaries of what it means to be a healthy human into the realm of superhuman performance and intelligence. This potential future capability is contrasted with limitations in the right-to-repair legislation, which may create challenges for patients. Now is the time for continued discussion of the ethical strategies for research, implementation, and long-term device sustainability or repair.

Biosignal-integrated robotic systems with emerging trends in visual interfaces: A systematic review

Human-machine interfaces (HMI) are currently a trendy and rapidly expanding area of research. Interestingly, the human user does not readily observe the interface between humans and machines. Instead, interactions between the machine and electrical signals from the user's body are obscured by complex control algorithms. The result is effectively a one-way street, wherein data is only transmitted from human to machine. Thus, a gap remains in the literature: how can information be effectively conveyed to the user to enable mutual understanding between humans and machines? Here, this paper reviews recent advancements in biosignal-integrated wearable robotics, with a particular emphasis on "visualization"-the presentation of relevant data, statistics, and visual feedback to the user. This review article covers various signals of interest, such as electroencephalograms and electromyograms, and explores novel sensor architectures and key materials. Recent developments in wearable robotics are examined from control and mechanical design perspectives. Additionally, we discuss current visualization methods and outline the field's future direction. While much of the HMI field focuses on biomedical and healthcare applications, such as rehabilitation of spinal cord injury and stroke patients, this paper also covers less common applications in manufacturing, defense, and other domains.

A scoping review of gaze and eye tracking-based control methods for assistive robotic arms

Background: Assistive Robotic Arms are designed to assist physically disabled people with daily activities. Existing joysticks and head controls are not applicable for severely disabled people such as people with Locked-in Syndrome. Therefore, eye tracking control is part of ongoing research. The related literature spans many disciplines, creating a heterogeneous field that makes it difficult to gain an overview. Objectives: This work focuses on ARAs that are controlled by gaze and eye movements. By answering the research questions, this paper provides details on the design of the systems, a comparison of input modalities, methods for measuring the performance of these controls, and an outlook on research areas that gained interest in recent years. Methods: This review was conducted as outlined in the PRISMA 2020 Statement. After identifying a wide range of approaches in use the authors decided to use the PRISMA-ScR extension for a scoping review to present the results. The identification process was carried out by screening three databases. After the screening process, a snowball search was conducted. Results: 39 articles and 6 reviews were included in this article. Characteristics related to the system and study design were extracted and presented divided into three groups based on the use of eye tracking. Conclusion: This paper aims to provide an overview for researchers new to the field by offering insight into eye tracking based robot controllers. We have identified open questions that need to be answered in order to provide people with severe motor function loss with systems that are highly useable and accessible.

Hand-worn devices for assessment and rehabilitation of motor function and their potential use in BCI protocols: a review

Introduction

Various neurological conditions can impair hand function. Affected individuals cannot fully participate in activities of daily living due to the lack of fine motor control. Neurorehabilitation emphasizes repetitive movement and subjective clinical assessments that require clinical experience to administer.

Methods

Here, we perform a review of literature focused on the use of hand-worn devices for rehabilitation and assessment of hand function. We paid particular attention to protocols that involve brain-computer interfaces (BCIs) since BCIs are gaining ground as a means for detecting volitional signals as the basis for interactive motor training protocols to augment recovery. All devices reviewed either monitor, assist, stimulate, or support hand and finger movement.

Results

A majority of studies reviewed here test or validate devices through clinical trials, especially for stroke. Even though sensor gloves are the most commonly employed type of device in this domain, they have certain limitations. Many such gloves use bend or inertial sensors to monitor the movement of individual digits, but few monitor both movement and applied pressure. The use of such devices in BCI protocols is also uncommon.

Discussion

We conclude that hand-worn devices that monitor both flexion and grip will benefit both clinical diagnostic assessment of function during treatment and closed-loop BCI protocols aimed at rehabilitation.

Hybrid brain/neural interface and autonomous vision-guided whole-arm exoskeleton control to perform activities of daily living (ADLs)

BACKGROUND

The aging of the population and the progressive increase of life expectancy in developed countries is leading to a high incidence of age-related cerebrovascular diseases, which affect people's motor and cognitive capabilities and might result in the loss of arm and hand functions. Such conditions have a detrimental impact on people's quality of life. Assistive robots have been developed to help people with motor or cognitive disabilities to perform activities of daily living (ADLs) independently. Most of the robotic systems for assisting on ADLs proposed in the state of the art are mainly external manipulators and exoskeletal devices. The main objective of this study is to compare the performance of an hybrid EEG/EOG interface to perform ADLs when the user is controlling an exoskeleton rather than using an external manipulator.

METHODS

Ten impaired participants (5 males and 5 females, mean age 52 ± 16 years) were instructed to use both systems to perform a drinking task and a pouring task comprising multiple subtasks. For each device, two modes of operation were studied: synchronous mode (the user received a visual cue indicating the sub-tasks to be performed at each time) and asynchronous mode (the user started and finished each of the sub-tasks independently). Fluent control was assumed when the time for successful initializations ranged below 3 s and a reliable control in case it remained below 5 s. NASA-TLX questionnaire was used to evaluate the task workload. For the trials involving the use of the exoskeleton, a custom Likert-Scale questionnaire was used to evaluate the user's experience in terms of perceived comfort, safety, and reliability.

RESULTS

All participants were able to control both systems fluently and reliably. However, results suggest better performances of the exoskeleton over the external manipulator (75% successful initializations remain below 3 s in case of the exoskeleton and bellow 5s in case of the external manipulator).

CONCLUSIONS

Although the results of our study in terms of fluency and reliability of EEG control suggest better performances of the exoskeleton over the external manipulator, such results cannot be considered conclusive, due to the heterogeneity of the population under test and the relatively limited number of participants.

A flexible speller based on time-space frequency conversion SSVEP stimulation paradigm under dry electrode

Introduction

Speller is the best way to express the performance of the brain-computer interface (BCI) paradigm. Due to its advantages of short analysis time and high accuracy, the SSVEP paradigm has been widely used in the BCI speller system based on the wet electrode. It is widely known that the wet electrode operation is cumbersome and that the subjects have a poor experience. In addition, in the asynchronous SSVEP system based on threshold analysis, the system flickers continuously from the beginning to the end of the experiment, which leads to visual fatigue. The dry electrode has a simple operation and provides a comfortable experience for subjects. The EOG signal can avoid the stimulation of SSVEP for a long time, thus reducing fatigue.

Methods

This study first designed the brain-controlled switch based on continuous blinking EOG signal and SSVEP signal to improve the flexibility of the BCI speller. Second, in order to increase the number of speller instructions, we designed the time-space frequency conversion (TSFC) SSVEP stimulus paradigm by constantly changing the time and space frequency of SSVEP sub-stimulus blocks, and designed a speller in a dry electrode environment.

Results

Seven subjects participated and completed the experiments. The results showed that the accuracy of the brain-controlled switch designed in this study was up to 94.64%, and all the subjects could use the speller flexibly. The designed 60-character speller based on the TSFC-SSVEP stimulus paradigm has an accuracy rate of 90.18% and an information transmission rate (ITR) of 117.05 bits/min. All subjects can output the specified characters in a short time.

Discussion

This study designed and implemented a multi-instruction SSVEP speller based on dry electrode. Through the combination of EOG and SSVEP signals, the speller can be flexibly controlled. The frequency of SSVEP stimulation sub-block is recoded in time and space by TSFC-SSVEP stimulation paradigm, which greatly improves the number of output instructions of BCI system in dry electrode environment. This work only uses FBCCA algorithm to test the stimulus paradigm, which requires a long stimulus time. In the future, we will use trained algorithms to study stimulus paradigm to improve its overall performance.

User Based Development and Test of the EXOTIC Exoskeleton: Empowering Individuals with Tetraplegia Using a Compact, Versatile, 5-DoF Upper Limb Exoskeleton Controlled through Intelligent Semi-Automated Shared Tongue Control

This paper presents the EXOTIC- a novel assistive upper limb exoskeleton for individuals with complete functional tetraplegia that provides an unprecedented level of versatility and control. The current literature on exoskeletons mainly focuses on the basic technical aspects of exoskeleton design and control while the context in which these exoskeletons should function is less or not prioritized even though it poses important technical requirements. We considered all sources of design requirements, from the basic technical functions to the real-world practical application. The EXOTIC features: (1) a compact, safe, wheelchair-mountable, easy to don and doff exoskeleton capable of facilitating multiple highly desired activities of daily living for individuals with tetraplegia; (2) a semi-automated computer vision guidance system that can be enabled by the user when relevant; (3) a tongue control interface allowing for full, volitional, and continuous control over all possible motions of the exoskeleton. The EXOTIC was tested on ten able-bodied individuals and three users with tetraplegia caused by spinal cord injury. During the tests the EXOTIC succeeded in fully assisting tasks such as drinking and picking up snacks, even for users with complete functional tetraplegia and the need for a ventilator. The users confirmed the usability of the EXOTIC.

The impact of interdisciplinarity and user involvement on the design and usability of an assistive upper limb exoskeleton - a case study on the EXOTIC

This study describes an interdisciplinary approach to develop a 5 degrees of freedom assistive upper limb exoskeleton (ULE) for users with severe to complete functional tetraplegia. Four different application levels were identified for the ULE ranging from basic technical application to interaction with users, interaction with caregivers and interaction with the society, each level posing requirements for the design and functionality of the ULE. These requirements were addressed through an interdisciplinary collaboration involving users, clinicians and researchers within social sciences and humanities, mechanical engineering, control engineering media technology and biomedical engineering. The results showed that the developed ULE, the EXOTIC, had a high level of usability, safety and adoptability. Further, the results showed that several topics are important to explicitly address in relation to the facilitation of interdisciplinary collaboration including, defining a common language, a joint visualization of the end goal and a physical frame for the collaboration, such as a shared laboratory. The study underlined the importance of interdisciplinarity and we believe that future collaboration amongst interdisciplinary researchers and centres, also at an international level, can strongly facilitate the usefulness and adoption of assistive exoskeletons and similar technologies.

EOG-Based Human-Computer Interface: 2000-2020 Review

Electro-oculography (EOG)-based brain-computer interface (BCI) is a relevant technology influencing physical medicine, daily life, gaming and even the aeronautics field. EOG-based BCI systems record activity related to users' intention, perception and motor decisions. It converts the bio-physiological signals into commands for external hardware, and it executes the operation expected by the user through the output device. EOG signal is used for identifying and classifying eye movements through active or passive interaction. Both types of interaction have the potential for controlling the output device by performing the user's communication with the environment. In the aeronautical field, investigations of EOG-BCI systems are being explored as a relevant tool to replace the manual command and as a communicative tool dedicated to accelerating the user's intention. This paper reviews the last two decades of EOG-based BCI studies and provides a structured design space with a large set of representative papers. Our purpose is to introduce the existing BCI systems based on EOG signals and to inspire the design of new ones. First, we highlight the basic components of EOG-based BCI studies, including EOG signal acquisition, EOG device particularity, extracted features, translation algorithms, and interaction commands. Second, we provide an overview of EOG-based BCI applications in the real and virtual environment along with the aeronautical application. We conclude with a discussion of the actual limits of EOG devices regarding existing systems. Finally, we provide suggestions to gain insight for future design inquiries.

Computer Vision-Based Adaptive Semi-Autonomous Control of an Upper Limb Exoskeleton for Individuals with Tetraplegia

We propose the use of computer vision for adaptive semi-autonomous control of an upper limb exoskeleton for assisting users with severe tetraplegia to increase independence and quality of life. A tongue-based interface was used together with the semi-autonomous control such that individuals with complete tetraplegia were able to use it despite being paralyzed from the neck down. The semi-autonomous control uses computer vision to detect nearby objects and estimate how to grasp them to assist the user in controlling the exoskeleton. Three control schemes were tested: non-autonomous (i.e., manual control using the tongue) control, semi-autonomous control with a fixed level of autonomy, and a semi-autonomous control with a confidence-based adaptive level of autonomy. Studies on experimental participants with and without tetraplegia were carried out. The control schemes were evaluated both in terms of their performance, such as the time and number of commands needed to complete a given task, as well as ratings from the users. The studies showed a clear and significant improvement in both performance and user ratings when using either of the semi-autonomous control schemes. The adaptive semi-autonomous control outperformed the fixed version in some scenarios, namely, in the more complex tasks and with users with more training in using the system.

A Hybrid Arm-Hand Rehabilitation Robot With EMG-Based Admittance Controller

Reach-and-grasp is one of the most fundamental activities in daily life, while few rehabilitation robots provide integrated and active training of the arm and hand for patients after stroke to improve their mobility. In this study, a novel hybrid arm-hand rehabilitation robot (HAHRR) was built for the reach-and-grasp task. This hybrid structure consisted of a cable-driven module for three-dimensional arm motion and an exoskeleton for hand motion, which enabled assistance of the arm and hand simultaneously. To implement active compliance control, an EMG-based admittance controller was applied to the HAHRR. Experimental results showed that the HAHRR with the EMG-based admittance controller could not only assist the subject in fulfilling the reach-and-grasp task, but also generate smoother trajectories compared with the force-sensing-based admittance controller. These findings also suggested that the proposed approach might be applicable to post-stroke arm-hand rehabilitation training.

Movement-Related EEG Oscillations of Contralesional Hemisphere Discloses Compensation Mechanisms of Severely Affected Motor Chronic Stroke Patients

Conventional rehabilitation strategies for stroke survivors become difficult when voluntary movements are severely disturbed. Combining passive limb mobilization, robotic devices and EEG-based brain-computer interfaces (BCI) systems might improve treatment and clinical follow-up of these patients, but detailed knowledge of neurophysiological mechanisms involved in functional recovery, which might help for tailoring stroke treatment strategies, is lacking. Movement-related EEG changes (EEG event-related desynchronization (ERD) in [Formula: see text] and [Formula: see text] bands, an indicator of motor cortex activation traditionally used for BCI systems), were evaluated in a group of 23 paralyzed chronic stroke patients in two unilateral motor tasks alternating paretic and healthy hands ((i) passive movement, using a hand exoskeleton, and (ii) voluntary movement), and compared to nine healthy subjects. In tasks using unaffected hand, we observed an increase of contralesional hemisphere activation for stroke patients group. Unexpectedly, when using paralyzed hand, motor cortex activation was reduced or absent in severely affected group of patients, while patients with moderate motor deficit showed an activation greater than control group. Cortical activation was reduced or absent in damaged hemisphere of all the patients in both tasks. Significant differences related to severity of motor deficit were found in the time course of [Formula: see text] bands power ratio in EEG of contralesional hemisphere while moving affected hand. These findings suggest the presence of different compensation mechanisms in contralesional hemisphere of stroke patients related to the grade of motor disability, that might turn quantitative EEG during a movement task, obtained from a BCI system controlling a robotic device included in a rehabilitation task, into a valuable tool for monitoring clinical progression, evaluating recovery, and tailoring treatment of stroke patients.

A Modular Mobile Robotic Platform to Assist People with Different Degrees of Disability

Robotics to support elderly people in living independently and to assist disabled people in carrying out the activities of daily living independently have demonstrated good results. Basically, there are two approaches: one of them is based on mobile robot assistants, such as Care-O-bot, PR2, and Tiago, among others; the other one is the use of an external robotic arm or a robotic exoskeleton fixed or mounted on a wheelchair. In this paper, a modular mobile robotic platform to assist moderately and severely impaired people based on an upper limb robotic exoskeleton mounted on a robotized wheel chair is presented. This mobile robotic platform can be customized for each user’s needs by exploiting its modularity. Finally, experimental results in a simulated home environment with a living room and a kitchen area, in order to simulate the interaction of the user with different elements of a home, are presented. In this experiment, a subject suffering from multiple sclerosis performed different activities of daily living (ADLs) using the platform in front of a group of clinicians composed of nurses, doctors, and occupational therapists. After that, the subject and the clinicians replied to a usability questionnaire. The results were quite good, but two key factors arose that need to be improved: the complexity and the cumbersome aspect of the platform.

Optical brain imaging and its application to neurofeedback

Besides passive recording of brain electric or magnetic activity, also non-ionizing electromagnetic or optical radiation can be used for real-time brain imaging. Here, changes in the radiation's absorption or scattering allow for continuous in vivo assessment of regional neurometabolic and neurovascular activity. Besides magnetic resonance imaging (MRI), over the last years, also functional near-infrared spectroscopy (fNIRS) was successfully established in real-time metabolic brain imaging. In contrast to MRI, fNIRS is portable and can be applied at bedside or in everyday life environments, e.g., to restore communication and movement. Here we provide a comprehensive overview of the history and state-of-the-art of real-time optical brain imaging with a special emphasis on its clinical use towards neurofeedback and brain-computer interface (BCI) applications. Besides pointing to the most critical challenges in clinical use, also novel approaches that combine real-time optical neuroimaging with other recording modalities (e.g. electro- or magnetoencephalography) are described, and their use in the context of neuroergonomics, neuroenhancement or neuroadaptive systems discussed.

A Roadmap Towards Standards for Neurally Controlled End Effectors

The control and manipulation of various types of end effectors such as powered exoskeletons, prostheses, and ‘neural’ cursors by brain-machine interface (BMI) systems has been the target of many research projects. A seamless “plug and play” interface between any BMI and end effector is desired, wherein similar user's intent cause similar end effectors to behave identically. This report is based on the outcomes of an IEEE Standards Association Industry Connections working group on End Effectors for Brain-Machine Interfacing that convened to identify and address gaps in the existing standards for BMI-based solutions with a focus on the end-effector component. A roadmap towards standardization of end effectors for BMI systems is discussed by identifying current device standards that are applicable for end effectors. While current standards address basic electrical and mechanical safety, and to some extent, performance requirements, several gaps exist pertaining to unified terminologies, data communication protocols, patient safety and risk mitigation.

Electro-Encephalography and Electro-Oculography in Aeronautics: A Review Over the Last Decade (2010-2020)

Electro-encephalography (EEG) and electro-oculography (EOG) are methods of electrophysiological monitoring that have potentially fruitful applications in neuroscience, clinical exploration, the aeronautical industry, and other sectors. These methods are often the most straightforward way of evaluating brain oscillations and eye movements, as they use standard laboratory or mobile techniques. This review describes the potential of EEG and EOG systems and the application of these methods in aeronautics. For example, EEG and EOG signals can be used to design brain-computer interfaces (BCI) and to interpret brain activity, such as monitoring the mental state of a pilot in determining their workload. The main objectives of this review are to, (i) offer an in-depth review of literature on the basics of EEG and EOG and their application in aeronautics; (ii) to explore the methodology and trends of research in combined EEG-EOG studies over the last decade; and (iii) to provide methodological guidelines for beginners and experts when applying these methods in environments outside the laboratory, with a particular focus on human factors and aeronautics. The study used databases from scientific, clinical, and neural engineering fields. The review first introduces the characteristics and the application of both EEG and EOG in aeronautics, undertaking a large review of relevant literature, from early to more recent studies. We then built a novel taxonomy model that includes 150 combined EEG-EOG papers published in peer-reviewed scientific journals and conferences from January 2010 to March 2020. Several data elements were reviewed for each study (e.g., pre-processing, extracted features and performance metrics), which were then examined to uncover trends in aeronautics and summarize interesting methods from this important body of literature. Finally, the review considers the advantages and limitations of these methods as well as future challenges.

Feasibility and Safety of Bilateral Hybrid EEG/EOG Brain/Neural-Machine Interaction

Cervical spinal cord injuries (SCIs) often lead to loss of motor function in both hands and legs, limiting autonomy and quality of life. While it was shown that unilateral hand function can be restored after SCI using a hybrid electroencephalography/electrooculography (EEG/EOG) brain/neural hand exoskeleton (B/NHE), it remained unclear whether such hybrid paradigm also could be used for operating two hand exoskeletons, e.g., in the context of bimanual tasks such as eating with fork and knife. To test whether EEG/EOG signals allow for fluent and reliable as well as safe and user-friendly bilateral B/NHE control, eight healthy participants (six females, mean age 24.1 ± 3.2 years) as well as four chronic tetraplegics (four males, mean age 51.8 ± 15.2 years) performed a complex sequence of EEG-controlled bilateral grasping and EOG-controlled releasing motions of two exoskeletons visually presented on a screen. A novel EOG command performed by prolonged horizontal eye movements (>1 s) to the left or right was introduced as a reliable switch to activate either the left or right exoskeleton. Fluent EEG control was defined as average “time to initialize” (TTI) grasping motions below 3 s. Reliable EEG control was assumed when classification accuracy exceeded 80%. Safety was defined as “time to stop” (TTS) all unintended grasping motions within 2 s. After the experiment, tetraplegics were asked to rate the user-friendliness of bilateral B/NHE control using Likert scales. Average TTI and accuracy of EEG-controlled operations ranged at 2.14 ± 0.66 s and 85.89 ± 15.81% across healthy participants and at 1.90 ± 0.97 s and 81.25 ± 16.99% across tetraplegics. Except for one tetraplegic, all participants met the safety requirements. With 88 ± 11% of the maximum achievable score, tetraplegics rated the control paradigm as user-friendly and reliable. These results suggest that hybrid EEG/EOG B/NHE control of two assistive devices is feasible and safe, paving the way to test this paradigm in larger clinical trials performing bimanual tasks in everyday life environments.

Perturbation-evoked potentials can be classified from single-trial EEG

OBJECTIVE

Loss of balance control can have serious consequences on interaction between humans and machines as well as the general well-being of humans. Perceived balance perturbations are always accompanied by a specific cortical activation, the so-called perturbation-evoked potential (PEP). In this study, we investigate the possibility to classify PEPs from ongoing EEG.

APPROACH

Fifteen healthy subjects were exposed to seated whole-body perturbations. Each participant performed 120 trials; they were rapidly tilted to the right and left, 60 times respectively.

MAIN RESULTS

We achieved classification accuracies of more than 85% between PEPs and rest EEG using a window-based classification approach. Different window lengths and electrode layouts were compared. We were able to achieve excellent classification performance (87.6 ± 8.0% accuracy) by using a short window length of 200 ms and a minimal electrode layout consisting of only the Cz electrode. The peak classification accuracy coincides in time with the strongest component of PEPs, called N1.

SIGNIFICANCE

We showed that PEPs can be discriminated against ongoing EEG with high accuracy. These findings can contribute to the development of a system that can detect balance perturbations online.

EEG-EOG based Virtual Keyboard: Toward Hybrid Brain Computer Interface

The past twenty years have ignited a new spark in the research of Electroencephalogram (EEG), which was pursued to develop innovative Brain Computer Interfaces (BCIs) in order to help severely disabled people live a better life with a high degree of independence. Current BCIs are more theoretical than practical and are suffering from numerous challenges. New trends of research propose combining EEG to other simple and efficient bioelectric inputs such as Electro-oculography (EOG) resulting from eye movements, to produce more practical and robust Hybrid Brain Computer Interface systems (hBCI) or Brain/Neuronal Computer Interface (BNCI). Working towards this purpose, existing research in EOG based Human Computer Interaction (HCI) applications, must be organized and surveyed in order to develop a vision on the potential benefits of combining both input modalities and give rise to new designs that maximize these benefits. Our aim is to support and inspire the design of new hBCI systems based on both EEG and EOG signals, in doing so; first the current EOG based HCI systems were surveyed with a particular focus on EOG based systems for communication using virtual keyboard. Then, a survey of the current EEG-EOG virtual keyboard was performed highlighting the design protocols employed. We concluded with a discussion of the potential advantages of combining both systems with recommendations to give deep insight for future design issues for all EEG-EOG hBCI systems. Finally, a general architecture was proposed for a new EEG-EOG hBCI system. The proposed hybrid system completely alters the traditional view of the eye movement features present in EEG signal as artifacts that should be removed; instead EOG traces are extracted from EEG in our proposed hybrid architecture and are considered as an additional input modality sharing control according to the chosen design protocol.

Physiological Responses During Hybrid BNCI Control of an Upper-Limb Exoskeleton

When combined with assistive robotic devices, such as wearable robotics, brain/neural-computer interfaces (BNCI) have the potential to restore the capabilities of handicapped people to carry out activities of daily living. To improve applicability of such systems, workload and stress should be reduced to a minimal level. Here, we investigated the user's physiological reactions during the exhaustive use of the interfaces of a hybrid control interface. Eleven BNCI-naive healthy volunteers participated in the experiments. All participants sat in a comfortable chair in front of a desk and wore a whole-arm exoskeleton as well as wearable devices for monitoring physiological, electroencephalographic (EEG) and electrooculographic (EoG) signals. The experimental protocol consisted of three phases: (i) Set-up, calibration and BNCI training; (ii) Familiarization phase; and (iii) Experimental phase during which each subject had to perform EEG and EoG tasks. After completing each task, the NASA-TLX questionnaire and self-assessment manikin (SAM) were completed by the user. We found significant differences (p-value < 0.05) in heart rate variability (HRV) and skin conductance level (SCL) between participants during the use of the two different biosignal modalities (EEG, EoG) of the BNCI. This indicates that EEG control is associated with a higher level of stress (associated with a decrease in HRV) and mental work load (associated with a higher level of SCL) when compared to EoG control. In addition, HRV and SCL modulations correlated with the subject's workload perception and emotional responses assessed through NASA-TLX questionnaires and SAM.

Unimanual and Bimanual Reach-and-Grasp Actions Can Be Decoded From Human EEG

While most tasks of daily life can be handled through a small number of different grasps, many tasks require the action of both hands. In these bimanual tasks, the second hand has either a supporting role (e.g. for fixating a jar) or a more active role (e.g. grasping a pot on both handles). In this study we attempt to discriminate the neural correlates of unimanual (performed with left and right hand) from bimanual reach-and-grasp actions using the low-frequency time-domain electroencephalogram (EEG). In a self-initiated movement task, 15 healthy participants were asked to perform unimanual (palmar and lateral grasps with left and right hand) and bimanual (double lateral, mixed palmar/lateral) reach-and-grasps on objects of daily life. Using EEG time-domain features in the frequency range of 0.3-3 Hz, we achieved multiclass-classification accuracies of 38.6 ± 6.6% (7 classes, 17.1% chance level) for a combination of 6 movements and 1 rest condition. The grand average confusion matrix shows highest true positive rates (TPR) for the rest (63%) condition while TPR for the movement classes varied between 33 to 41%. The underlying movement-related cortical potentials (MRCPs) show significant differences between unimanual (e.g left hand vs. right hand grasps) as well unimanual vs. bimanual conditions which both can be attributed to lateralization effects. We believe that these findings can be exploited and further used for attempts in providing persons with spinal cord injury a form of natural control for bimanual neuroprostheses.

Multi-connection load compensation and load information calculation for an upper-limb exoskeleton based on a six-axis force/torque sensor

In this article, a method of multi-connection load compensation and load information calculation for an upper-limb exoskeleton is proposed based on a six-axis force/torque sensor installed between the exoskeleton and the end effector. The proposed load compensation method uses a mounted sensor to measure the force and torque between the exoskeleton and load of different connections and adds a compensator to the controller to compensate the component caused by the load in the human–robot interaction force, so that the human–robot interaction force is only used to operate the exoskeleton. Therefore, the operator can manipulate the exoskeleton with the same interaction force to lift loads of different weights with a passive or fixed connection, and the human–robot interaction force is minimized. Moreover, the proposed load information calculation method can calculate the weight of the load and the position of its center of gravity relative to the exoskeleton and end effector accurately, which is necessary for acquiring the upper-limb exoskeleton center of gravity and stability control of whole-body exoskeleton. In order to verify the effectiveness of the proposed method, we performed load handling and operational stability experiments. The experimental results showed that the proposed method realized the expected function.

Decoding electroencephalographic signals for direction in brain-computer interface using echo state network and Gaussian readouts

BACKGROUND

Noninvasive brain-computer interfaces (BCI) for movement control via an electroencephalogram (EEG) have been extensively investigated. However, most previous studies decoded user intention for movement directions based on sensorimotor rhythms during motor imagery. BCI systems based on mapping imagery movement of body parts (e.g., left or right hands) to movement directions (left or right directional movement of a machine or cursor) are less intuitive and less convenient due to the complex training procedures. Thus, direct decoding methods for detecting user intention about movement directions are urgently needed.

METHODS

Here, we describe a novel direct decoding method for user intention about the movement directions using the echo state network and Gaussian readouts. Importantly parameters in the network were optimized using the genetic algorithm method to achieve better decoding performance. We tested the decoding performance of this method with four healthy subjects and an inexpensive wireless EEG system containing 14 channels and then compared the performance outcome with that of a conventional machine learning method.

RESULTS

We showed that this decoding method successfully classified eight directions of intended movement (approximately 95% of an accuracy).

CONCLUSIONS

We suggest that the echo state network and Gaussian readouts can be a useful decoding method to directly read user intention of movement directions even using an inexpensive and portable EEG system.

Suit-type Wearable Robot Powered by Shape-memory-alloy-based Fabric Muscle

A suit-type wearable robot (STWR) is a new type of soft wearable robot (SWR) that can be worn easily anywhere and anytime to assist the muscular strength of wearers because it can be worn like normal clothes and is comfortable to wear even with no power supply. This paper proposes an STWR, in which a shape-memory-alloy-based fabric muscle (SFM) is used as the actuator. The STWR, which weighs less than 1 kg, has a simple structure, with the following components: SFMs, wire encoders for measuring the contraction length of the SFMs, and BOA that fix the actuators on the forearms. In this study, a position controller for the SFM using the wire encoder was developed, and a prototype STWR was fabricated using this position controller. Moreover, by putting the STWR on a mannequin, step-response experiments were performed in which the arms of the mannequin lifted barbells weighing 2 kg and 4 kg to a certain target position. A fast response of moving to the target position in less than 1 s was observed in all steps except for the initial heating step for the 2 kg barbell. The response speed of the SFM was noticeably slower for the 4 kg barbell compared to that for the 2 kg barbell; it moved to the target position in approximately 3 s in all the steps except for the initial heating step. The SFM-applied STWR could overcome the limitations of conventional robots in terms of weight and inconvenience, thereby demonstrating the application potential of STWRs.

Detection of movement onset using EMG signals for upper-limb exoskeletons in reaching tasks

BACKGROUND

To assist people with disabilities, exoskeletons must be provided with human-robot interfaces and smart algorithms capable to identify the user's movement intentions. Surface electromyographic (sEMG) signals could be suitable for this purpose, but their applicability in shared control schemes for real-time operation of assistive devices in daily-life activities is limited due to high inter-subject variability, which requires custom calibrations and training. Here, we developed a machine-learning-based algorithm for detecting the user's motion intention based on electromyographic signals, and discussed its applicability for controlling an upper-limb exoskeleton for people with severe arm disabilities.

METHODS

Ten healthy participants, sitting in front of a screen while wearing the exoskeleton, were asked to perform several reaching movements toward three LEDs, presented in a random order. EMG signals from seven upper-limb muscles were recorded. Data were analyzed offline and used to develop an algorithm that identifies the onset of the movement across two different events: moving from a resting position toward the LED (Go-forward), and going back to resting position (Go-backward). A set of subject-independent time-domain EMG features was selected according to information theory and their probability distributions corresponding to rest and movement phases were modeled by means of a two-component Gaussian Mixture Model (GMM). The detection of movement onset by two types of detectors was tested: the first type based on features extracted from single muscles, whereas the second from multiple muscles. Their performances in terms of sensitivity, specificity and latency were assessed for the two events with a leave one-subject out test method.

RESULTS

The onset of movement was detected with a maximum sensitivity of 89.3% for Go-forward and 60.9% for Go-backward events. Best performances in terms of specificity were 96.2 and 94.3% respectively. For both events the algorithm was able to detect the onset before the actual movement, while computational load was compatible with real-time applications.

CONCLUSIONS

The detection performances and the low computational load make the proposed algorithm promising for the control of upper-limb exoskeletons in real-time applications. Fast initial calibration makes it also suitable for helping people with severe arm disabilities in performing assisted functional tasks.

A Novel Generation of Ergonomic Upper-Limb Wearable Robots: Design Challenges and Solutions

In this work we present NEUROExos, a novel generation of upper-limb exoskeletons developed in recent years at The BioRobotics Institute of Scuola Superiore Sant’Anna (Italy). Specifically, we present our attempts to progressively (i) improve the ergonomics and safety (ii) reduce the encumbrance and weight, and (iii) develop more intuitive human–robot cognitive interfaces. Our latest prototype, described here for the first time, extends the field of application to assistance in activities of daily living, thanks to its compact and portable design. The experimental studies carried out on these devices are summarized, and a perspective on future developments is presented.

Development, Dynamic Modeling, and Multi-Modal Control of a Therapeutic Exoskeleton for Upper Limb Rehabilitation Training

Robot-assisted training is a promising technology in clinical rehabilitation providing effective treatment to the patients with motor disability. In this paper, a multi-modal control strategy for a therapeutic upper limb exoskeleton is proposed to assist the disabled persons perform patient-passive training and patient-cooperative training. A comprehensive overview of the exoskeleton with seven actuated degrees of freedom is introduced. The dynamic modeling and parameters identification strategies of the human-robot interaction system are analyzed. Moreover, an adaptive sliding mode controller with disturbance observer (ASMCDO) is developed to ensure the position control accuracy in patient-passive training. A cascade-proportional-integral-derivative (CPID)-based impedance controller with graphical game-like interface is designed to improve interaction compliance and motivate the active participation of patients in patient-cooperative training. Three typical experiments are conducted to verify the feasibility of the proposed control strategy, including the trajectory tracking experiments, the trajectory tracking experiments with impedance adjustment, and the intention-based training experiments. The experimental results suggest that the tracking error of ASMCDO controller is smaller than that of terminal sliding mode controller. By optimally changing the impedance parameters of CPID-based impedance controller, the training intensity can be adjusted to meet the requirement of different patients.