BMI control of a third arm for multitasking

BandFocusNet: A Lightweight Model for Motor Imagery Classification of a Supernumerary Thumb in Virtual Reality

Objective: Human movement augmentation through supernumerary effectors is an emerging field of research. However, controlling these effectors remains challenging due to issues with agency, control, and synchronizing movements with natural limbs. A promising control strategy for supernumerary effectors involves utilizing electroencephalography (EEG) through motor imagery (MI) functions. In this work, we investigate whether MI activity associated with a supernumerary effector could be reliably differentiated from that of a natural one, thus addressing the concern of concurrency. Twenty subjects were recruited to participate in a two-fold experiment in which they observed movements of natural and supernumerary thumbs, then engaged in MI of the observed movements, conducted in a virtual reality setting. Results: A lightweight deep-learning model that accounts for the temporal, spatial and spectral nature of the EEG data is proposed and called BandFocusNet, achieving an average classification accuracy of 70.9% using the leave-one-subject-out cross validation method. The trustworthiness of the model is examined through explainability analysis, and influential regions-of-interests are cross-validated through event-related-spectral-perturbation (ERSPs) analysis. Explainability results showed the importance of the right and left frontal cortical regions, and ERSPs analysis showed an increase in the delta and theta powers in these regions during the MI of the natural thumb but not during the MI of the supernumerary thumb. Conclusion: Evidence in the literature indicates that such activation is observed during the MI of natural effectors, and its absence could be interpreted as a lack of embodiment of the supernumerary thumb.

Error-related potentials during multitasking involving sensorimotor control: an ERP and offline decoding study for brain-computer interface

Humans achieve efficient behaviors by perceiving and responding to errors. Error-related potentials (ErrPs) are electrophysiological responses that occur upon perceiving errors. Leveraging ErrPs to improve the accuracy of brain-computer interfaces (BCIs), utilizing the brain's natural error-detection processes to enhance system performance, has been proposed. However, the influence of external and contextual factors on the detectability of ErrPs remains poorly understood, especially in multitasking scenarios involving both BCI operations and sensorimotor control. Herein, we hypothesized that the difficulty in sensorimotor control would lead to the dispersion of neural resources in multitasking, resulting in a reduction in ErrP features. To examine this, we conducted an experiment in which participants were instructed to keep a ball within a designated area on a board, while simultaneously attempting to control a cursor on a display through motor imagery. The BCI provided error feedback with a random probability of 30%. Three scenarios-without a ball (single-task), lightweight ball (easy-task), and heavyweight ball (hard-task)-were used for the characterization of ErrPs based on the difficulty of sensorimotor control. In addition, to examine the impact of multitasking on ErrP-BCI performance, we analyzed single-trial classification accuracy offline. Contrary to our hypothesis, varying the difficulty of sensorimotor control did not result in significant changes in ErrP features. However, multitasking significantly affected ErrP classification accuracy. Post-hoc analyses revealed that the classifier trained on single-task ErrPs exhibited reduced accuracy under hard-task scenarios. To our knowledge, this study is the first to investigate how ErrPs are modulated in a multitasking environment involving both sensorimotor control and BCI operation in an offline framework. Although the ErrP features remained unchanged, the observed variation in accuracy suggests the need to design classifiers that account for task load even before implementing a real-time ErrP-based BCI.

Shared Control of Supernumerary Robotic Limbs Using Mixed Realityand Mouth-and-Tongue Interfaces

Supernumerary Robotic Limbs (SRLs) are designed to collaborate with the wearer, enhancing operational capabilities. When human limbs are occupied with primary tasks, controlling SRLs flexibly and naturally becomes a challenge. Existing methods such as electromyography (EMG) control and redundant limb control partially address SRL control issues. However, they still face limitations like restricted degrees of freedom and complex data requirements, which hinder their applicability in real-world scenarios. Additionally, fully autonomous control methods, while efficient, often lack the flexibility needed for complex tasks, as they do not allow for real-time user adjustments. In contrast, shared control combines machine autonomy with human input, enabling finer control and more intuitive task completion. Building on our previous work with the mouth-and-tongue interface, this paper integrates a mixed reality (MR) device to form an interactive system that enables shared control of the SRL. The system allows users to dynamically switch between voluntary and autonomous control, providing both flexibility and efficiency. A random forest model classifies 14 distinct tongue and mouth operations, mapping them to six-degree-of-freedom SRL control. In comparative experiments involving ten healthy subjects performing assembly tasks under three control modes (shared control, autonomous control, and voluntary control), shared control demonstrates a balance between machine autonomy and human input. While autonomous control offers higher task efficiency, shared control achieves greater task success rates and improves user experience by combining the advantages of both autonomous operation and voluntary control. This study validates the feasibility of shared control and highlights its advantages in providing flexible switching between autonomy and user intervention, offering new insights into SRL control.

An fMRI study on the generalization of motor learning after brain actuated supernumerary robot training

Generalization is central to motor learning. However, few studies are on the learning generalization of BCI-actuated supernumerary robotic finger (BCI-SRF) for human-machine interaction training, and no studies have explored its longitudinal neuroplasticity mechanisms. Here, 20 healthy right-handed participants were recruited and randomly assigned to BCI-SRF group or inborn finger group (Finger) for 4-week training and measured by novel SRF-finger opposition sequences and multimodal MRI. After training, the BCI-SRF group showed 350% times compared to the Finger group in the improvement of sequence opposition accuracy before and after training, and accompanied by significant functional connectivity increases in the sensorimotor region and prefrontal cortex, as well as in the intra- and inter-hemisphere of the sensorimotor network. Moreover, Granger Causality Analysis identified causal effect main transfer within the sensorimotor cortex-cerebellar-thalamus loop and frontal-parietal loop. The findings suggest that BCI-SRF training enhances motor sequence learning ability by influencing the functional reorganization of sensorimotor network.

A Systematic Review of Bimanual Motor Coordination in Brain-Computer Interface

Advancements in neuroscience and artificial intelligence are propelling rapid progress in brain-computer interfaces (BCIs). These developments hold significant potential for decoding motion intentions from brain signals, enabling direct control commands without reliance on conventional neural pathways. Growing interest exists in decoding bimanual motor tasks, crucial for activities of daily living. This stems from the need to restore motor function, especially in individuals with deficits. This review aims to summarize neurological advancements in bimanual BCIs, encompassing neuroimaging techniques, experimental paradigms, and analysis algorithms. Thirty-six articles were reviewed, adhering to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. The literature search result revealed diverse experimental paradigms, protocols, and research directions, including enhancing the decoding accuracy, advancing versatile prosthesis robots, and enabling real-time applications. Notably, within BCI studies on bimanual movement coordination, a shared objective is to achieve naturalistic movement and practical applications with neurorehabilitation potential.

EEG Characteristic Comparison of Motor Imagery Between Supernumerary and Inherent Limb: Sixth-Finger MI Enhances the ERD Pattern and Classification Performance

Adding supernumerary robotic limbs (SRLs) to humans and controlling them directly through the brain are main goals for movement augmentation. However, it remains uncertain whether neural patterns different from the traditional inherent limbs motor imagery (MI) can be extracted, which is essential for high-dimensional control of external devices. In this work, we established a MI neo-framework consisting of novel supernumerary robotic sixth-finger MI (SRF-MI) and traditional right-hand MI (RH-MI) paradigms and validated the distinctness of EEG response patterns between two MI tasks for the first time. Twenty-four subjects were recruited for this experiment involving three mental tasks. Event-related spectral perturbation was adopted to supply details about event-related desynchronization (ERD). Activation region, intensity and response time (RT) of ERD were compared between SRF-MI and RH-MI tasks. Three classical classification algorithms were utilized to verify the separability between different mental tasks. And genetic algorithm aims to select optimal combination of channels for neo-framework. A bilateral sensorimotor and prefrontal modulation was found during the SRF-MI task, whereas in RH-MI only contralateral sensorimotor modulation was exhibited. The novel SRF-MI paradigm enhanced ERD intensity by a maximum of 117% in prefrontal area and 188% in the ipsilateral somatosensory-association cortex. And, a global decrease of RT was exhibited during SRF-MI tasks compared to RH-MI. Classification results indicate well separable performance among different mental tasks (88.1% maximum for 2-class and 88.2% maximum for 3-class). This work demonstrated the difference between the SRF-MI and RH-MI paradigms, widening the control bandwidth of the BCI system.

Enhancing ERD Activation and Functional Connectivity via the Sixth-Finger Motor Imagery in Stroke Patients

Motor imagery (MI) is widely employed in stroke rehabilitation due to the event-related desynchronization (ERD) phenomenon in sensorimotor cortex induced by MI is similar to actual movement. However, the traditional BCI paradigm, in which the patient imagines the movement of affected hand (AH-MI) with a weak ERD caused by the damaged brain regions, retards motor relearning process. In this work, we applied a novel MI paradigm based on the "sixth-finger" (SF-MI) in stroke patients and systematically uncovered the ERD pattern enhancement of novel MI paradigm compared to traditional MI paradigm. Twenty stroke patients were recruited for this experiment. Event-related spectral perturbation was adopted to supply details about ERD. Brain activation region, intensity and functional connectivity were compared between SF-MI and AH-MI to reveal the ERD enhancement performance of novel MI paradigm. A "wider range, stronger intensity, greater connection" ERD activation pattern was induced in stroke patients by novel SF-MI paradigm compared to traditional AH-MI paradigm. The bilateral sensorimotor and prefrontal modulation was found in SF-MI, which was different in AH-MI only weak sensorimotor modulation was exhibited. The ERD enhancement is mainly concentrated in mu rhythm. More synchronized and intimate neural activity between different brain regions was found during SF-MI tasks compared to AH-MI tasks. Classification results (>80% in SF-MI vs. REST) also indicated the feasibility of applying novel MI paradigm to clinical stroke rehabilitation. This work provides a novel MI paradigm and demonstrates its neural activation-enhancing performance, helping to develop more effective MI-based BCI system for stroke rehabilitation.

Neuro-motor controlled wearable augmentations: current research and emerging trends

Wearable augmentations (WAs) designed for movement and manipulation, such as exoskeletons and supernumerary robotic limbs, are used to enhance the physical abilities of healthy individuals and substitute or restore lost functionality for impaired individuals. Non-invasive neuro-motor (NM) technologies, including electroencephalography (EEG) and sufrace electromyography (sEMG), promise direct and intuitive communication between the brain and the WA. After presenting a historical perspective, this review proposes a conceptual model for NM-controlled WAs, analyzes key design aspects, such as hardware design, mounting methods, control paradigms, and sensory feedback, that have direct implications on the user experience, and in the long term, on the embodiment of WAs. The literature is surveyed and categorized into three main areas: hand WAs, upper body WAs, and lower body WAs. The review concludes by highlighting the primary findings, challenges, and trends in NM-controlled WAs. This review motivates researchers and practitioners to further explore and evaluate the development of WAs, ensuring a better quality of life.

Iteratively Calibratable Network for Reliable EEG-Based Robotic Arm Control

Robotic arms are increasingly being utilized in shared workspaces, which necessitates the accurate interpretation of human intentions for both efficiency and safety. Electroencephalogram (EEG) signals, commonly employed to measure brain activity, offer a direct communication channel between humans and robotic arms. However, the ambiguous and unstable characteristics of EEG signals, coupled with their widespread distribution, make it challenging to collect sufficient data and hinder the calibration performance for new signals, thereby reducing the reliability of EEG-based applications. To address these issues, this study proposes an iteratively calibratable network aimed at enhancing the reliability and efficiency of EEG-based robotic arm control systems. The proposed method integrates feature inputs with network expansion techniques. This integration allows a network trained on an extensive initial dataset to adapt effectively to new users during calibration. Additionally, our approach combines motor imagery and speech imagery datasets to increase not only its intuitiveness but also the number of command classes. The evaluation is conducted in a pseudo-online manner, with a robotic arm operating in real-time to collect data, which is then analyzed offline. The evaluation results demonstrated that the proposed method outperformed the comparison group in 10 sessions and demonstrated competitive results when the two paradigms were combined. Therefore, it was confirmed that the network can be calibrated and personalized using only the new data from new users.

EEG Pattern Comparison and Classification Performance of Motor Imagery Between Supernumerary and Inherent Limbs

Adding supernumerary robotic limbs (SRLs) to humans and controlling them directly through the brain are main goals for movement augmentation. However, whether neural patterns that are distinct from the traditional inherent limbs motor imagery (MI) paradigm can be extracted, which is essential for the high-dimensional control of external equipment. In this study, a novel type of MI paradigm based on SRLs was proposed, consisting of "the sixth-finger", "the third-arm" and "the third-leg", and validated the distinctness of EEG response patterns between the novel and the traditional (hand, arm and leg) MI paradigm. The results showed that imagining extra limbs induced more obvious event-related desynchronization (ERD) phenomenon in sensorimotor areas compared to imagining inherent limbs. Classification results indicate well separable performance among different mental tasks (all above 86%, with a maximum of 90.5%). This work proposed a novel type of MI paradigm, and offered new way for widening the control bandwidth of the BCI system.

Enhancing Brain-Computer Interface Performance by Incorporating Brain-to-Brain Coupling

Human cooperation relies on key features of social interaction in order to reach desirable outcomes. Similarly, human-robot interaction may benefit from integration with human-human interaction factors. In this paper, we aim to investigate brain-to-brain coupling during motor imagery (MI)-based brain-computer interface (BCI) training using eye-contact and hand-touch interaction. Twelve pairs of friends (experimental group) and 10 pairs of strangers (control group) were recruited for MI-based BCI tests concurrent with electroencephalography (EEG) hyperscanning. Event-related desynchronization (ERD) was estimated to measure cortical activation, and interbrain functional connectivity was assessed using multilevel statistical analysis. Furthermore, we compared BCI classification performance under different social interaction conditions. In the experimental group, greater ERD was found around the contralateral sensorimotor cortex under social interaction conditions compared with MI without any social interaction. Notably, EEG channels with decreased power were mainly distributed around the frontal, central, and occipital regions. A significant increase in interbrain coupling was also found under social interaction conditions. BCI decoding accuracies were significantly improved in the eye contact condition and eye and hand contact condition compared with the no-interaction condition. However, for the strangers' group, no positive effects were observed in comparisons of cortical activations between interaction and no-interaction conditions. These findings indicate that social interaction can improve the neural synchronization between familiar partners with enhanced brain activations and brain-to-brain coupling. This study may provide a novel method for enhancing MI-based BCI performance in conjunction with neural synchronization between users.

A Mouth and Tongue Interactive Device to Control Wearable Robotic Limbs in Tasks where Human Limbs Are Occupied

The Wearable Robotic Limb (WRL) is a type of robotic arm worn on the human body, aiming to enhance the wearer's operational capabilities. However, proposing additional methods to control and perceive the WRL when human limbs are heavily occupied with primary tasks presents a challenge. Existing interactive methods, such as voice, gaze, and electromyography (EMG), have limitations in control precision and convenience. To address this, we have developed an interactive device that utilizes the mouth and tongue. This device is lightweight and compact, allowing wearers to achieve continuous motion and contact force control of the WRL. By using a tongue controller and mouth gas pressure sensor, wearers can control the WRL while also receiving sensitive contact feedback through changes in mouth pressure. To facilitate bidirectional interaction between the wearer and the WRL, we have devised an algorithm that divides WRL control into motion and force-position hybrid modes. In order to evaluate the performance of the device, we conducted an experiment with ten participants tasked with completing a pin-hole assembly task with the assistance of the WRL system. The results show that the device enables continuous control of the position and contact force of the WRL, with users perceiving feedback through mouth airflow resistance. However, the experiment also revealed some shortcomings of the device, including user fatigue and its impact on breathing. After experimental investigation, it was observed that fatigue levels can decrease with training. Experimental studies have revealed that fatigue levels can decrease with training. Furthermore, the limitations of the device have shown potential for improvement through structural enhancements. Overall, our mouth and tongue interactive device shows promising potential in controlling the WRL during tasks where human limbs are occupied.

Human motor augmentation with an extra robotic arm without functional interference

Extra robotic arms (XRAs) are gaining interest in neuroscience and robotics, offering potential tools for daily activities. However, this compelling opportunity poses new challenges for sensorimotor control strategies and human-machine interfaces (HMIs). A key unsolved challenge is allowing users to proficiently control XRAs without hindering their existing functions. To address this, we propose a pipeline to identify suitable HMIs given a defined task to accomplish with the XRA. Following such a scheme, we assessed a multimodal motor HMI based on gaze detection and diaphragmatic respiration in a purposely designed modular neurorobotic platform integrating virtual reality and a bilateral upper limb exoskeleton. Our results show that the proposed HMI does not interfere with speaking or visual exploration and that it can be used to control an extra virtual arm independently from the biological ones or in coordination with them. Participants showed significant improvements in performance with daily training and retention of learning, with no further improvements when artificial haptic feedback was provided. As a final proof of concept, naïve and experienced participants used a simplified version of the HMI to control a wearable XRA. Our analysis indicates how the presented HMI can be effectively used to control XRAs. The observation that experienced users achieved a success rate 22.2% higher than that of naïve users, combined with the result that naïve users showed average success rates of 74% when they first engaged with the system, endorses the viability of both the virtual reality-based testing and training and the proposed pipeline.

Real-Time Deep Neurolinguistic Learning Enhances Noninvasive Neural Language Decoding for Brain-Machine Interaction

Electroencephalogram (EEG)-based brain-machine interface (BMI) has been utilized to help patients regain motor function and has recently been validated for its use in healthy people because of its ability to directly decipher human intentions. In particular, neurolinguistic research using EEGs has been investigated as an intuitive and naturalistic communication tool between humans and machines. In this study, the human mind directly decoded the neural languages based on speech imagery using the proposed deep neurolinguistic learning. Through real-time experiments, we evaluated whether BMI-based cooperative tasks between multiple users could be accomplished using a variety of neural languages. We successfully demonstrated a BMI system that allows a variety of scenarios, such as essential activity, collaborative play, and emotional interaction. This outcome presents a novel BMI frontier that can interact at the level of human-like intelligence in real time and extends the boundaries of the communication paradigm.

Explainable cross-task adaptive transfer learning for motor imagery EEG classification

Objective. In the field of motor imagery (MI) electroencephalography (EEG)-based brain-computer interfaces, deep transfer learning (TL) has proven to be an effective tool for solving the problem of limited availability in subject-specific data for the training of robust deep learning (DL) models. Although considerable progress has been made in the cross-subject/session and cross-device scenarios, the more challenging problem of cross-task deep TL remains largely unexplored.Approach. We propose a novel explainable cross-task adaptive TL method for MI EEG decoding. Firstly, similarity analysis and data alignment are performed for EEG data of motor execution (ME) and MI tasks. Afterwards, the MI EEG decoding model is obtained via pre-training with extensive ME EEG data and fine-tuning with partial MI EEG data. Finally, expected gradient-based post-hoc explainability analysis is conducted for the visualization of important temporal-spatial features.Main results. Extensive experiments are conducted on one large ME EEG High-Gamma dataset and two large MI EEG datasets (openBMI and GIST). The best average classification accuracy of our method reaches 80.00% and 72.73% for OpenBMI and GIST respectively, which outperforms several state-of-the-art algorithms. In addition, the results of the explainability analysis further validate the correlation between ME and MI EEG data and the effectiveness of ME/MI cross-task adaptation.Significance. This paper confirms that the decoding of MI EEG can be well facilitated by pre-existing ME EEG data, which largely relaxes the constraint of training samples for MI EEG decoding and is important in a practical sense.

EEG-Based Target Detection Using an RSVP Paradigm under Five Levels of Weak Hidden Conditions

Although target detection based on electroencephalogram (EEG) signals has been extensively investigated recently, EEG-based target detection under weak hidden conditions remains a problem. In this paper, we proposed a rapid serial visual presentation (RSVP) paradigm for target detection corresponding to five levels of weak hidden conditions quantitively based on the RGB color space. Eighteen subjects participated in the experiment, and the neural signatures, including P300 amplitude and latency, were investigated. Detection performance was evaluated under five levels of weak hidden conditions using the linear discrimination analysis and support vector machine classifiers on different channel sets. The experimental results showed that, compared with the benchmark condition, (1) the P300 amplitude significantly decreased (8.92 ± 1.24 μV versus 7.84 ± 1.40 μV, p = 0.021) and latency was significantly prolonged (582.39 ± 25.02 ms versus 643.83 ± 26.16 ms, p = 0.028) only under the weakest hidden condition, and (2) the detection accuracy decreased by less than 2% (75.04 ± 3.24% versus 73.35 ± 3.15%, p = 0.029) with a more than 90% reduction in channel number (62 channels versus 6 channels), determined using the proposed channel selection method under the weakest hidden condition. Our study can provide new insights into target detection under weak hidden conditions based on EEG signals with a rapid serial visual presentation paradigm. In addition, it may expand the application of brain-computer interfaces in EEG-based target detection areas.

Favoring the cognitive-motor process in the closed-loop of BCI mediated post stroke motor function recovery: challenges and approaches

The brain-computer interface (BCI)-mediated rehabilitation is emerging as a solution to restore motor skills in paretic patients after stroke. In the human brain, cortical motor neurons not only fire when actions are carried out but are also activated in a wired manner through many cognitive processes related to movement such as imagining, perceiving, and observing the actions. Moreover, the recruitment of motor cortexes can usually be regulated by environmental conditions, forming a closed-loop through neurofeedback. However, this cognitive-motor control loop is often interrupted by the impairment of stroke. The requirement to bridge the stroke-induced gap in the motor control loop is promoting the evolution of the BCI-based motor rehabilitation system and, notably posing many challenges regarding the disease-specific process of post stroke motor function recovery. This review aimed to map the current literature surrounding the new progress in BCI-mediated post stroke motor function recovery involved with cognitive aspect, particularly in how it refired and rewired the neural circuit of motor control through motor learning along with the BCI-centric closed-loop.

Human Operation Augmentation through Wearable Robotic Limb Integrated with Mixed Reality Device

Mixed reality technology can give humans an intuitive visual experience, and combined with the multi-source information of the human body, it can provide a comfortable human-robot interaction experience. This paper applies a mixed reality device (Hololens2) to provide interactive communication between the wearer and the wearable robotic limb (supernumerary robotic limb, SRL). Hololens2 can obtain human body information, including eye gaze, hand gestures, voice input, etc. It can also provide feedback information to the wearer through augmented reality and audio output, which is the communication bridge needed in human-robot interaction. Implementing a wearable robotic arm integrated with HoloLens2 is proposed to augment the wearer's capabilities. Taking two typical practical tasks of cable installation and electrical connector soldering in aircraft manufacturing as examples, the task models and interaction scheme are designed. Finally, human augmentation is evaluated in terms of task completion time statistics.

3D Convolution neural network with multiscale spatial and temporal cues for motor imagery EEG classification

Recently, deep learning-based methods have achieved meaningful results in the Motor imagery electroencephalogram (MI EEG) classification. However, because of the low signal-to-noise ratio and the various characteristics of brain activities among subjects, these methods lack a subject adaptive feature extraction mechanism. Another issue is that they neglect important spatial topological information and the global temporal variation trend of MI EEG signals. These issues limit the classification accuracy. Here, we propose an end-to-end 3D CNN to extract multiscale spatial and temporal dependent features for improving the accuracy performance of 4-class MI EEG classification. The proposed method adaptively assigns higher weights to motor-related spatial channels and temporal sampling cues than the motor-unrelated ones across all brain regions, which can prevent influences caused by biological and environmental artifacts. Experimental evaluation reveals that the proposed method achieved an average classification accuracy of 93.06% and 97.05% on two commonly used datasets, demonstrating excellent performance and robustness for different subjects compared to other state-of-the-art methods.In order to verify the real-time performance in actual applications, the proposed method is applied to control the robot based on MI EEG signals. The proposed approach effectively addresses the issues of existing methods, improves the classification accuracy and the performance of BCI system, and has great application prospects.

Impact of supplementary sensory feedback on the control and embodiment in human movement augmentation

In human movement augmentation, the number of controlled degrees of freedom could be enhanced by the simultaneous and independent use of supernumerary robotic limbs (SRL) and natural ones. However, this poses several challenges, that could be mitigated by encoding and relaying the SRL status. Here, we review the impact of supplementary sensory feedback on the control and embodiment of SRLs. We classify the main feedback features and analyse how they improve control performance. We report the feasibility of pushing body representation beyond natural human morphology and suggest that gradual SRL embodiment could make multisensory incongruencies less disruptive. We also highlight shared computational bases between SRL motor control and embodiment and suggest contextualizing them within the same theoretical framework. Finally, we argue that a shift towards long term experimental paradigms is necessary for successfully integrating motor control and embodiment.

Supernumerary robotic limbs are robotic devices providing additional limbs to the user. Mattia Pinardi and colleagues review the impact of supplementary sensory feedback on the control performance and embodiment of supernumerary robotic limbs.

A shallow mirror transformer for subject-independent motor imagery BCI

OBJECTIVE

Motor imagery BCI plays an increasingly important role in motor disorders rehabilitation. However, the position and duration of the discriminative segment in an EEG trial vary from subject to subject and even trial to trial, and this leads to poor performance of subject-independent motor imagery classification. Thus, determining how to detect and utilize the discriminative signal segments is crucial for improving the performance of subject-independent motor imagery BCI.

APPROACH

In this paper, a shallow mirror transformer is proposed for subject-independent motor imagery EEG classification. Specifically, a multihead self-attention layer with a global receptive field is employed to detect and utilize the discriminative segment from the entire input EEG trial. Furthermore, the mirror EEG signal and the mirror network structure are constructed to improve the classification precision based on ensemble learning. Finally, the subject-independent setup was used to evaluate the shallow mirror transformer on motor imagery EEG signals from subjects existing in the training set and new subjects.

MAIN RESULTS

The experiments results on BCI Competition IV datasets 2a and 2b and the OpenBMI dataset demonstrated the promising effectiveness of the proposed shallow mirror transformer. The shallow mirror transformer obtained average accuracies of 74.48% and 76.1% for new subjects and existing subjects, respectively, which were highest among the compared state-of-the-art methods. In addition, visualization of the attention score showed the ability of discriminative EEG segment detection. This paper demonstrated that multihead self-attention is effective in capturing global EEG signal information in motor imagery classification.

SIGNIFICANCE

This study provides an effective model based on a multihead self-attention layer for subject-independent motor imagery-based BCIs. To the best of our knowledge, this is the shallowest transformer model available, in which a small number of parameters promotes the performance in motor imagery EEG classification for such a small sample problem.

Shared Three-Dimensional Robotic Arm Control Based on Asynchronous BCI and Computer Vision

OBJECTIVE

A brain-computer interface (BCI) can be used to translate neuronal activity into commands to control external devices. However, using noninvasive BCI to control a robotic arm for movements in three-dimensional (3D) environments and accomplish complicated daily tasks, such as grasping and drinking, remains a challenge.

APPROACH

In this study, a shared robotic arm control system based on hybrid asynchronous BCI and computer vision was presented. The BCI model, which combines steady-state visual evoked potentials (SSVEPs) and blink-related electrooculography (EOG) signals, allows users to freely choose from fifteen commands in an asynchronous mode corresponding to robot actions in a 3D workspace and reach targets with a wide movement range, while computer vision can identify objects and assist a robotic arm in completing more precise tasks, such as grasping a target automatically.

RESULTS

Ten subjects participated in the experiments and achieved an average accuracy of more than 92% and a high trajectory efficiency for robot movement. All subjects were able to perform the reach-grasp-drink tasks successfully using the proposed shared control method, with fewer error commands and shorter completion time than with direct BCI control.

SIGNIFICANCE

Our results demonstrated the feasibility and efficiency of generating practical multidimensional control of an intuitive robotic arm by merging hybrid asynchronous BCI and computer vision-based recognition.

Improving single-hand open/close motor imagery classification by error-related potentials correction

Objective

The ability of a brain-computer interface (BCI) to classify brain activity in electroencephalograms (EEG) during motor imagery (MI) tasks is an important performance indicator. Because the cortical regions that drive the single-handed open and closed tasks overlap, it is difficult to classify the EEG signals during executing both tasks.

Approach

The addition of special EEG features can improve the accuracy of classifying single-hand open and closed tasks. In this work, we designed a hybrid BCI paradigm based on error-related potentials (ErrP) and motor imagery (MI) and proposed a strategy to correct the classification results of MI by using ErrP information. The ErrP and MI features of EEG data from 11 subjects were superimposed.

Main results

The corrected strategy improved the classification accuracy of single-hand open/close MI tasks from 52.3% to 73.7%, an increase of approximately 21%.

Significance

Our hybrid BCI paradigm improves the classification accuracy of single-hand MI by adding ErrP information, which provides a new approach for improving the classification performance of BCI.

Conformal in-ear bioelectronics for visual and auditory brain-computer interfaces

Brain-computer interfaces (BCIs) have attracted considerable attention in motor and language rehabilitation. Most devices use cap-based non-invasive, headband-based commercial products or microneedle-based invasive approaches, which are constrained for inconvenience, limited applications, inflammation risks and even irreversible damage to soft tissues. Here, we propose in-ear visual and auditory BCIs based on in-ear bioelectronics, named as SpiralE, which can adaptively expand and spiral along the auditory meatus under electrothermal actuation to ensure conformal contact. Participants achieve offline accuracies of 95% in 9-target steady state visual evoked potential (SSVEP) BCI classification and type target phrases successfully in a calibration-free 40-target online SSVEP speller experiment. Interestingly, in-ear SSVEPs exhibit significant 2nd harmonic tendencies, indicating that in-ear sensing may be complementary for studying harmonic spatial distributions in SSVEP studies. Moreover, natural speech auditory classification accuracy can reach 84% in cocktail party experiments. The SpiralE provides innovative concepts for designing 3D flexible bioelectronics and assists the development of biomedical engineering and neural monitoring.

Comparing end-effector position and joint angle feedback for online robotic limb tracking

Somatosensation greatly increases the ability to control our natural body. This suggests that supplementing vision with haptic sensory feedback would also be helpful when a user aims at controlling a robotic arm proficiently. However, whether the position of the robot and its continuous update should be coded in a extrinsic or intrinsic reference frame is not known. Here we compared two different supplementary feedback contents concerning the status of a robotic limb in 2-DoFs configuration: one encoding the Cartesian coordinates of the end-effector of the robotic arm (i.e., Task-space feedback) and another and encoding the robot joints angles (i.e., Joint-space feedback). Feedback was delivered to blindfolded participants through vibrotactile stimulation applied on participants' leg. After a 1.5-hour training with both feedbacks, participants were significantly more accurate with Task compared to Joint-space feedback, as shown by lower position and aiming errors, albeit not faster (i.e., similar onset delay). However, learning index during training was significantly higher in Joint space feedback compared to Task-space feedback. These results suggest that Task-space feedback is probably more intuitive and more suited for activities which require short training sessions, while Joint space feedback showed potential for long-term improvement. We speculate that the latter, despite performing worse in the present work, might be ultimately more suited for applications requiring long training, such as the control of supernumerary robotic limbs for surgical robotics, heavy industrial manufacturing, or more generally, in the context of human movement augmentation.

Sci-fi imagines how good brain-machine interfaces will amplify bad choices

Machine Man and The Andromeda Evolution explore personal and societal ramifications of brain-machine interfaces.

Hairy-Skin-Adaptive Viscoelastic Dry Electrodes for Long-Term Electrophysiological Monitoring

Long-term epidermal electrophysiological (EP) monitoring is crucial for disease diagnosis and human-machine synergy. The human skin is covered with hair that grows at an average rate of 0.3 mm per day. This impedes a stable contact between the skin and dry epidermal electrodes, resulting in motion artifacts during ultralong-term EP monitoring. Therefore, accurate and high-quality EP signal detection remains challenging. To address this issue, a new solution-the hairy-skin-adaptive viscoelastic dry electrode (VDE) is reported. This innovative technology is capable of bypassing hair and filling into the skin wrinkles, leading to long-lasting and stable interface impedance. The VDE maintains a stable interface impedance for a remarkable period of 48 days and 100 cycles. The VDE is highly effective in shielding against hair disturbances in electrocardiography (ECG) monitoring, even during intense chest expansion, and in electromyography (EMG) monitoring during large strain. Furthermore, the VDE is easily attachable to the skull without requiring any electroencephalogram (EEG) cap or bandage, making it an ideal solution for EEG monitoring. This work represents a substantial breakthrough in the field of EP monitoring, providing a solution for the previously challenging issue of monitoring human EP signals on hairy skin.

Functional Two-Dimensional Materials for Bioelectronic Neural Interfacing

Realizing the neurological information processing by analyzing the complex data transferring behavior of populations and individual neurons is one of the fast-growing fields of neuroscience and bioelectronic technologies. This field is anticipated to cover a wide range of advanced applications, including neural dynamic monitoring, understanding the neurological disorders, human brain-machine communications and even ambitious mind-controlled prosthetic implant systems. To fulfill the requirements of high spatial and temporal resolution recording of neural activities, electrical, optical and biosensing technologies are combined to develop multifunctional bioelectronic and neuro-signal probes. Advanced two-dimensional (2D) layered materials such as graphene, graphene oxide, transition metal dichalcogenides and MXenes with their atomic-layer thickness and multifunctional capabilities show bio-stimulation and multiple sensing properties. These characteristics are beneficial factors for development of ultrathin-film electrodes for flexible neural interfacing with minimum invasive chronic interfaces to the brain cells and cortex. The combination of incredible properties of 2D nanostructure places them in a unique position, as the main materials of choice, for multifunctional reception of neural activities. The current review highlights the recent achievements in 2D-based bioelectronic systems for monitoring of biophysiological indicators and biosignals at neural interfaces.

Recognition of single upper limb motor imagery tasks from EEG using multi-branch fusion convolutional neural network

Motor imagery-based brain-computer interfaces (MI-BCI) have important application values in the field of neurorehabilitation and robot control. At present, MI-BCI mostly use bilateral upper limb motor tasks, but there are relatively few studies on single upper limb MI tasks. In this work, we conducted studies on the recognition of motor imagery EEG signals of the right upper limb and proposed a multi-branch fusion convolutional neural network (MF-CNN) for learning the features of the raw EEG signals as well as the two-dimensional time-frequency maps at the same time. The dataset used in this study contained three types of motor imagery tasks: extending the arm, rotating the wrist, and grasping the object, 25 subjects were included. In the binary classification experiment between the grasping object and the arm-extending tasks, MF-CNN achieved an average classification accuracy of 78.52% and kappa value of 0.57. When all three tasks were used for classification, the accuracy and kappa value were 57.06% and 0.36, respectively. The comparison results showed that the classification performance of MF-CNN is higher than that of single CNN branch algorithms in both binary-class and three-class classification. In conclusion, MF-CNN makes full use of the time-domain and frequency-domain features of EEG, can improve the decoding accuracy of single limb motor imagery tasks, and it contributes to the application of MI-BCI in motor function rehabilitation training after stroke.

NeuroGrasp: Real-Time EEG Classification of High-Level Motor Imagery Tasks Using a Dual-Stage Deep Learning Framework

Brain-computer interfaces (BCIs) have been widely employed to identify and estimate a user's intention to trigger a robotic device by decoding motor imagery (MI) from an electroencephalogram (EEG). However, developing a BCI system driven by MI related to natural hand-grasp tasks is challenging due to its high complexity. Although numerous BCI studies have successfully decoded large body parts, such as the movement intention of both hands, arms, or legs, research on MI decoding of high-level behaviors such as hand grasping is essential to further expand the versatility of MI-based BCIs. In this study, we propose NeuroGrasp, a dual-stage deep learning framework that decodes multiple hand grasping from EEG signals under the MI paradigm. The proposed method effectively uses an EEG and electromyography (EMG)-based learning, such that EEG-based inference at test phase becomes possible. The EMG guidance during model training allows BCIs to predict hand grasp types from EEG signals accurately. Consequently, NeuroGrasp improved classification performance offline, and demonstrated a stable classification performance online. Across 12 subjects, we obtained an average offline classification accuracy of 0.68 (±0.09) in four-grasp-type classifications and 0.86 (±0.04) in two-grasp category classifications. In addition, we obtained an average online classification accuracy of 0.65 (±0.09) and 0.79 (±0.09) across six high-performance subjects. Because the proposed method has demonstrated a stable classification performance when evaluated either online or offline, in the future, we expect that the proposed method could contribute to different BCI applications, including robotic hands or neuroprosthetics for handling everyday objects.

The sense of agency in emerging technologies for human–computer integration: A review

Human–computer integration is an emerging area in which the boundary between humans and technology is blurred as users and computers work collaboratively and share agency to execute tasks. The sense of agency (SoA) is an experience that arises by a combination of a voluntary motor action and sensory evidence whether the corresponding body movements have somehow influenced the course of external events. The SoA is not only a key part of our experiences in daily life but also in our interaction with technology as it gives us the feeling of “I did that” as opposed to “the system did that,” thus supporting a feeling of being in control. This feeling becomes critical with human–computer integration, wherein emerging technology directly influences people’s body, their actions, and the resulting outcomes. In this review, we analyse and classify current integration technologies based on what we currently know about agency in the literature, and propose a distinction between body augmentation, action augmentation, and outcome augmentation. For each category, we describe agency considerations and markers of differentiation that illustrate a relationship between assistance level (low, high), agency delegation (human, technology), and integration type (fusion, symbiosis). We conclude with a reflection on the opportunities and challenges of integrating humans with computers, and finalise with an expanded definition of human–computer integration including agency aspects which we consider to be particularly relevant. The aim this review is to provide researchers and practitioners with guidelines to situate their work within the integration research agenda and consider the implications of any technologies on SoA, and thus overall user experience when designing future technology.

Performance Variation of a Somatosensory BCI Based on Imagined Sensation: A Large Population Study

A proportion of users cannot achieve adequate brain-computer interface (BCI) control. The diversity of BCI modalities provides a way to solve this emerging issue. Here, we investigate the accuracy of a somatosensory BCI based on sensory imagery (SI). During the SI tasks, subjects were instructed to imagine a tactile sensation and to maintain the attention on the corresponding hand, as if there was tactile stimulus on the skin of the wrist. The performance across 106 healthy subjects in left- and right-hand SI discrimination was 78.9±13.2%. In 70.7% of the subjects the performance was above 70%. The SI task induced a contralateral cortical activation, and high-density EEG source localization showed that the real tactile stimulation and imagined tactile stimulation shared similar cortical activations within the somatosensory cortex. The somatosensory BCI based on SI provides a new signal modality for independent BCI development. Moreover, a combination of SI and other BCI modalities, such as motor imagery, may provide new avenues for further improving BCI usage and applicability, especially in those subjects unable to attain adequate BCI control with conventional BCI modalities.

Control of a Wheelchair-Mounted 6DOF Assistive Robot With Chin and Finger Joysticks

Throughout the last decade, many assistive robots for people with disabilities have been developed; however, researchers have not fully utilized these robotic technologies to entirely create independent living conditions for people with disabilities, particularly in relation to activities of daily living (ADLs). An assistive system can help satisfy the demands of regular ADLs for people with disabilities. With an increasing shortage of caregivers and a growing number of individuals with impairments and the elderly, assistive robots can help meet future healthcare demands. One of the critical aspects of designing these assistive devices is to improve functional independence while providing an excellent human–machine interface. People with limited upper limb function due to stroke, spinal cord injury, cerebral palsy, amyotrophic lateral sclerosis, and other conditions find the controls of assistive devices such as power wheelchairs difficult to use. Thus, the objective of this research was to design a multimodal control method for robotic self-assistance that could assist individuals with disabilities in performing self-care tasks on a daily basis. In this research, a control framework for two interchangeable operating modes with a finger joystick and a chin joystick is developed where joysticks seamlessly control a wheelchair and a wheelchair-mounted robotic arm. Custom circuitry was developed to complete the control architecture. A user study was conducted to test the robotic system. Ten healthy individuals agreed to perform three tasks using both (chin and finger) joysticks for a total of six tasks with 10 repetitions each. The control method has been tested rigorously, maneuvering the robot at different velocities and under varying payload (1–3.5 lb) conditions. The absolute position accuracy was experimentally found to be approximately 5 mm. The round-trip delay we observed between the commands while controlling the xArm was 4 ms. Tests performed showed that the proposed control system allowed individuals to perform some ADLs such as picking up and placing items with a completion time of less than 1 min for each task and 100% success.

2020 International brain-computer interface competition: A review

The brain-computer interface (BCI) has been investigated as a form of communication tool between the brain and external devices. BCIs have been extended beyond communication and control over the years. The 2020 international BCI competition aimed to provide high-quality neuroscientific data for open access that could be used to evaluate the current degree of technical advances in BCI. Although there are a variety of remaining challenges for future BCI advances, we discuss some of more recent application directions: (i) few-shot EEG learning, (ii) micro-sleep detection (iii) imagined speech decoding, (iv) cross-session classification, and (v) EEG(+ear-EEG) detection in an ambulatory environment. Not only did scientists from the BCI field compete, but scholars with a broad variety of backgrounds and nationalities participated in the competition to address these challenges. Each dataset was prepared and separated into three data that were released to the competitors in the form of training and validation sets followed by a test set. Remarkable BCI advances were identified through the 2020 competition and indicated some trends of interest to BCI researchers.

Endless forms most beautiful 2.0: teleonomy and the bioengineering of chimaeric and synthetic organisms

The rich variety of biological forms and behaviours results from one evolutionary history on Earth, via frozen accidents and selection in specific environments. This ubiquitous baggage in natural, familiar model species obscures the plasticity and swarm intelligence of cellular collectives. Significant gaps exist in our understanding of the origin of anatomical novelty, of the relationship between genome and form, and of strategies for control of large-scale structure and function in regenerative medicine and bioengineering. Analysis of living forms that have never existed before is necessary to reveal deep design principles of life as it can be. We briefly review existing examples of chimaeras, cyborgs, hybrots and other beings along the spectrum containing evolved and designed systems. To drive experimental progress in multicellular synthetic morphology, we propose teleonomic (goal-seeking, problem-solving) behaviour in diverse problem spaces as a powerful invariant across possible beings regardless of composition or origin. Cybernetic perspectives on chimaeric morphogenesis erase artificial distinctions established by past limitations of technology and imagination. We suggest that a multi-scale competency architecture facilitates evolution of robust problem-solving, living machines. Creation and analysis of novel living forms will be an essential testbed for the emerging field of diverse intelligence, with numerous implications across regenerative medicine, robotics and ethics.

Flexible Electronics and Devices as Human-Machine Interfaces for Medical Robotics

Medical robots are invaluable players in non-pharmaceutical treatment of disabilities. Particularly, using prosthetic and rehabilitation devices with human-machine interfaces can greatly improve the quality of life for impaired patients. In recent years, flexible electronic interfaces and soft robotics have attracted tremendous attention in this field due to their high biocompatibility, functionality, conformability, and low-cost. Flexible human-machine interfaces on soft robotics will make a promising alternative to conventional rigid devices, which can potentially revolutionize the paradigm and future direction of medical robotics in terms of rehabilitation feedback and user experience. In this review, the fundamental components of the materials, structures, and mechanisms in flexible human-machine interfaces are summarized by recent and renowned applications in five primary areas: physical and chemical sensing, physiological recording, information processing and communication, soft robotic actuation, and feedback stimulation. This review further concludes by discussing the outlook and current challenges of these technologies as a human-machine interface in medical robotics.

Principles of human movement augmentation and the challenges in making it a reality

Augmenting the body with artificial limbs controlled concurrently to one's natural limbs has long appeared in science fiction, but recent technological and neuroscientific advances have begun to make this possible. By allowing individuals to achieve otherwise impossible actions, movement augmentation could revolutionize medical and industrial applications and profoundly change the way humans interact with the environment. Here, we construct a movement augmentation taxonomy through what is augmented and how it is achieved. With this framework, we analyze augmentation that extends the number of degrees-of-freedom, discuss critical features of effective augmentation such as physiological control signals, sensory feedback and learning as well as application scenarios, and propose a vision for the field.

Functional Reorganization After Four-Week Brain-Computer Interface-Controlled Supernumerary Robotic Finger Training: A Pilot Study of Longitudinal Resting-State fMRI

Humans have long been fascinated by the opportunities afforded through motor augmentation provided by the supernumerary robotic fingers (SRFs) and limbs (SRLs). However, the neuroplasticity mechanism induced by the motor augmentation equipment still needs further investigation. This study focused on the resting-state brain functional reorganization during longitudinal brain-computer interface (BCI)-controlled SRF training in using the fractional amplitude of low-frequency fluctuation (fALFF), regional homogeneity (ReHo), and degree centrality (DC) metrics. Ten right-handed subjects were enrolled for 4 weeks of BCI-controlled SRF training. The behavioral data and the neurological changes were recorded at baseline, training for 2 weeks, training for 4 weeks immediately after, and 2 weeks after the end of training. One-way repeated-measure ANOVA was used to investigate long-term motor improvement [F(2.805,25.24) = 43.94, p < 0.0001] and neurological changes. The fALFF values were significantly modulated in Cerebelum_6_R and correlated with motor function improvement (r = 0.6887, p < 0.0402) from t0 to t2. Besides, Cerebelum_9_R and Vermis_3 were also significantly modulated and showed different trends in longitudinal SRF training in using ReHo metric. At the same time, ReHo values that changed from t0 to t1 in Vermis_3 was significantly correlated with motor function improvement (r = 0.7038, p < 0.0344). We conclude that the compensation and suppression mechanism of the cerebellum existed during BCI-controlled SRF training, and this current result provided evidence to the neuroplasticity mechanism brought by the BCI-controlled motor-augmentation devices.

Motor Imagery Classification Using Inter-Task Transfer Learning via A Channel-Wise Variational Autoencoder-based Convolutional Neural Network

Highly sophisticated control based on a brain-computer interface (BCI) requires decoding kinematic information from brain signals. The forearm is a region of the upper limb that is often used in everyday life, but intuitive movements within the same limb have rarely been investigated in previous BCI studies. In this study, we focused on various forearm movement decoding from electroencephalography (EEG) signals using a small number of samples. Ten healthy participants took part in an experiment and performed motor execution (ME) and motor imagery (MI) of the intuitive movement tasks (Dataset I). We propose a convolutional neural network using a channel-wise variational autoencoder (CVNet) based on inter-task transfer learning. We approached that training the reconstructed ME-EEG signals together will also achieve more sufficient classification performance with only a small amount of MI-EEG signals. The proposed CVNet was validated on our own Dataset I and a public dataset, BNCI Horizon 2020 (Dataset II). The classification accuracies of various movements are confirmed to be 0.83 (±0.04) and 0.69 (±0.04) for Dataset I and II, respectively. The results show that the proposed method exhibits performance increases of approximately 0.09~0.27 and 0.08~0.24 compared with the conventional models for Dataset I and II, respectively. The outcomes suggest that the training model for decoding imagined movements can be performed using data from ME and a small number of data samples from MI. Hence, it is presented the feasibility of BCI learning strategies that can sufficiently learn deep learning with a few amount of calibration dataset and time only, with stable performance.

EEG characteristic investigation of the sixth-finger motor imagery and optimal channel selection for classification

OBJECTIVE

Supernumerary Robotic Limbs (SRL) are body augmentation robotic devices by adding extra limbs or fingers to the human body different from the traditional wearable robotic devices such as prosthesis and exoskeleton. We proposed a novel MI (Motor imagery)-based BCI paradigm based on the sixth-finger which imagines controlling the extra finger movements. The goal of this work is to investigate the EEG characteristics and the application potential of MI-based BCI systems based on the new imagination paradigm (the sixth finger MI).

APPROACH

14 subjects participated in the experiment involving the sixth finger MI tasks and rest state. Event-related spectral perturbation (ERSP) was adopted to analyse EEG spatial features and key-channel time-frequency features. Common spatial patterns (CSP) were used for feature extraction and classification was implemented by support vector machine (SVM). A genetic algorithm (GA) was used to select combinations of EEG channels that maximized classification accuracy and verified EEG patterns based on the sixth finger MI. And we conducted a longitudinal 4-week EEG control experiment based on the new paradigm.

MAIN RESULTS

ERD (event-related desynchronization) was found in the supplementary motor area (SMA) and primary motor area (M1) with a faint contralateral dominance. Unlike traditional MI based on the human hand, ERD was also found in frontal lobe. GA results showed that the distribution of the optimal 8-channel is similar to EEG topographical distributions, nearing parietal and frontal lobe. And the classification accuracy based on the optimal 8-channel (the highest accuracy of 80% and mean accuracy of 70%) was significantly better than that based on the random 8-channel (p<0.01).

SIGNIFICANCE

This work provided a new paradigm for MI-based MI system and verified its feasibility, widened the control bandwidth of the BCI system.

Simultaneous control of natural and extra degrees of freedom by isometric force and electromyographic activity in the muscle-to-force null space

OBJECTIVE

Muscle activation patterns in the muscle-to-force null space, i.e., patterns that do not generate task-relevant forces, may provide an opportunity for motor augmentation by allowing to control additional end-effectors simultaneously to natural limbs. Here we tested the feasibility of muscular null space control for augmentation by assessing simultaneous control of natural and extra degrees of freedom.

APPROACH

We instructed eight participants to control translation and rotation of a virtual 3D end-effector by simultaneous generation of isometric force at the hand and null space activity extracted in real-time from the electromyographic signals recorded from 15 shoulder and arm muscles. First, we identified the null space components that each participant could control more naturally by voluntary co-contraction. Then, participants performed several blocks of a reaching and holding task. They displaced an ellipsoidal cursor to reach one of nine targets by generating force, and simultaneously rotated the cursor to match the target orientation by activating null space components. We developed an information-theoretic metric, an index of difficulty defined as the sum of a spatial and a temporal term, to assess individual null space control ability for both reaching and holding.

MAIN RESULTS

On average, participants could reach the targets in most trials already in the first block (72%) and they improved with practice (maximum 93%) but holding performance remained lower (maximum 43%). As there was a high inter-individual variability in performance, we performed a simulation with different spatial and temporal task conditions to estimate those for which each individual participants would have performed best.

SIGNIFICANCE

Muscular null space control is feasible and may be used to control additional virtual or robotics end-effectors. However, decoding of motor commands must be optimized according to individual null space control ability.

Quantitative Investigation of Hand Grasp Functionality: Thumb Grasping Behavior Adapting to Different Object Shapes, Sizes, and Relative Positions

This paper is the first in the two-part series quantitatively modelling human grasp functionality and understanding the way human grasp objects. The aim is to investigate the thumb movement behavior influenced by object shapes, sizes, and relative positions. Ten subjects were requested to grasp six objects (3 shapes × 2 sizes) in 27 different relative positions (3 X deviation × 3 Y deviation × 3 Z deviation). Thumb postures were investigated to each specific joint. The relative position (X, Y, and Z deviation) significantly affects thumb opposition rotation (Rot) and flexion (interphalangeal (IP) and metacarpo-phalangeal (MCP)), while the object property (object shape and size) significantly affects thumb abduction/adduction (ABD) motion. Based on the F value, the Y deviation has the primary effects on thumb motion. When the Y deviation changing from proximal to distal, thumb opposition rotation (Rot) and flexion (IP and MCP joint) angles were increased and decreased, respectively. For principal component analysis (PCA) results, thumb grasp behavior can be accurately reconstructed by first two principal components (PCs) which variance explanation ratio reached 93.8% and described by the inverse and homodromous coordination movement between thumb opposition and IP flexion. This paper provides a more comprehensive understanding of thumb grasp behavior. The postural synergies can reproduce the anthropomorphic motion, reduce the robot hardware, and control dimensionality. All of these provide a more accurate and general basis for the design and control of the bionic thumb and novel wearable assistant robot, thumb function assessment, and rehabilitation.

Practical real-time MEG-based neural interfacing with optically pumped magnetometers

BACKGROUND

Brain-computer interfaces decode intentions directly from the human brain with the aim to restore lost functionality, control external devices or augment daily experiences. To combine optimal performance with wide applicability, high-quality brain signals should be captured non-invasively. Magnetoencephalography (MEG) is a potent candidate but currently requires costly and confining recording hardware. The recently developed optically pumped magnetometers (OPMs) promise to overcome this limitation, but are currently untested in the context of neural interfacing.

RESULTS

In this work, we show that OPM-MEG allows robust single-trial analysis which we exploited in a real-time 'mind-spelling' application yielding an average accuracy of 97.7%.

CONCLUSIONS

This shows that OPM-MEG can be used to exploit neuro-magnetic brain responses in a practical and flexible manner, and opens up new avenues for a wide range of new neural interface applications in the future.

Robotic hand augmentation drives changes in neural body representation

Humans have long been fascinated by the opportunities afforded through augmentation. This vision not only depends on technological innovations but also critically relies on our brain's ability to learn, adapt, and interface with augmentation devices. Here, we investigated whether successful motor augmentation with an extra robotic thumb can be achieved and what its implications are on the neural representation and function of the biological hand. Able-bodied participants were trained to use an extra robotic thumb (called the Third Thumb) over 5 days, including both lab-based and unstructured daily use. We challenged participants to complete normally bimanual tasks using only the augmented hand and examined their ability to develop hand-robot interactions. Participants were tested on a variety of behavioral and brain imaging tests, designed to interrogate the augmented hand's representation before and after the training. Training improved Third Thumb motor control, dexterity, and hand-robot coordination, even when cognitive load was increased or when vision was occluded. It also resulted in increased sense of embodiment over the Third Thumb. Consequently, augmentation influenced key aspects of hand representation and motor control. Third Thumb usage weakened natural kinematic synergies of the biological hand. Furthermore, brain decoding revealed a mild collapse of the augmented hand's motor representation after training, even while the Third Thumb was not worn. Together, our findings demonstrate that motor augmentation can be readily achieved, with potential for flexible use, reduced cognitive reliance, and increased sense of embodiment. Yet, augmentation may incur changes to the biological hand representation. Such neurocognitive consequences are crucial for successful implementation of future augmentation technologies.

Towards human motor augmentation by voluntary decoupling beta activity in the neural drive to muscle and force production

Objective.Effective human motor augmentation should rely on biological signals that can be volitionally modulated without compromising natural motor control.Approach.We provided human subjects with real-time information on the power of two separate spectral bands of the spiking activity of motor neurons innervating the tibialis anterior muscle: the low-frequency band (<7 Hz), which is directly translated into natural force control, and the beta band (13-30 Hz), which is outside the dynamics of the neuromuscular system.Main Results.Subjects could gain control over the powers in these two bands to navigate a cursor towards specific targets in a 2D space (experiment 1) and to up- and down-modulate beta activity while keeping steady force contractions (experiment 2).Significance.Results indicate that beta projections to the spinal motor neuron pool can be voluntarily controlled partially decoupled from natural muscle contractions and, therefore, they could be valid control signals for implementing effective human motor augmentation platforms.

Improving the performance of multisubject motor imagery-based BCIs using twin cascaded softmax CNNs

OBJECTIVE

Motor imagery (MI) EEG signals vary greatly among subjects, so scholarly research on motor imagery-based brain-computer interfaces (BCIs) has mainly focused on single-subject systems or subject-dependent systems. However, the single-subject model is applicable only to the target subject, and the small sample number greatly limits the performance of the model. This paper aims to study a convolutional neural network to achieve an adaptable MI-BCI that is applicable to multiple subjects.

APPROACH

In this paper, a twin cascaded softmax convolutional neural network (TCSCNN) is proposed for multisubject MI-BCIs. The proposed TCSCNN is independent and can be applied to any single-subject MI classification CNN model. First, to reduce the influence of individual differences, subject recognition and MI recognition are accomplished simultaneously. A cascaded softmax structure consisting of two softmax layers, related to subject recognition and MI recognition, is subsequently applied. Second, to improve the MI classification precision, a twin network structure is proposed on the basis of ensemble learning. TCSCNN is built by combining a cascaded softmax structure and twin network structure.

MAIN RESULTS

Experiments were conducted on three popular CNN models (EEGNet and Shallow ConvNet and Deep ConvNet from EEGDecoding) and three public datasets (BCI Competition IV datasets 2a and 2b and the High-Gamma dataset) to verify the performance of the proposed TCSCNN. The results show that compared with the state-of-the-art CNN model, the proposed TCSCNN obviously improves the precision and convergence of multisubject MI recognition.

SIGNIFICANCE

This study provides a promising scheme for multisubject MI-BCI, reflecting the progress made in the development and application of MI-BCIs.