Effective Synchronization of EEG and EMG for Mobile Brain/Body Imaging in Clinical Settings

A method for synchronized use of EEG and eye tracking in fully immersive VR

This study explores the synchronization of multimodal physiological data streams, in particular, the integration of electroencephalography (EEG) with a virtual reality (VR) headset featuring eye-tracking capabilities. A potential use case for the synchronized data streams is demonstrated by implementing a hybrid steady-state visually evoked potential (SSVEP) based brain-computer interface (BCI) speller within a fully immersive VR environment. The hardware latency analysis reveals an average offset of 36 ms between EEG and eye-tracking data streams and a mean jitter of 5.76 ms. The study further presents a proof of concept brain-computer interface (BCI) speller in VR, showcasing its potential for real-world applications. The findings highlight the feasibility of combining commercial EEG and VR technologies for neuroscientific research and open new avenues for studying brain activity in ecologically valid VR environments. Future research could focus on refining the synchronization methods and exploring applications in various contexts, such as learning and social interactions.

Dual-layer electroencephalography data during real-world table tennis

Real-world settings are necessary to improve the ecological validity of neuroscience research, and electroencephalography (EEG) facilitates mobile electrocortical recordings because of its easy portability and high temporal resolution. Table tennis is a whole-body, goal-directed sport that requires constant visuomotor feedback, anticipation, strategic decision-making, object interception, and performance monitoring - making it an interesting testbed for a variety of neuroscience studies. Although traditionally plagued by artifact contamination, recent advances in signal processing and hardware approaches, such as the dual-layer approach, have allowed high fidelity EEG recordings during whole-body maneuvers. Here, we present a dual-layer EEG dataset from 25 healthy human participants playing table tennis with a human opponent and a ball machine. Our dataset includes synchronized, multivariate time series recordings from 120 scalp electrodes, 120 noise electrodes, 8 neck electromyography electrodes, and inertial measurement units on the participant, paddles, and ball machine to record hit events. We also include de-identified T1 anatomical MR images and digitized electrode locations to create subject-specific head models for source localization. In addition, we provide anonymized video recordings and Adobe Premiere project files with hit events labeled (originally used to mark successful/missed hits). Researchers could use the videos to mark their own events of interest. We formatted our dataset in the Brain Imaging Data Structure (BIDS) format to facilitate data reuse and to adhere to the scientific community's new organization standard.

A review of combined functional neuroimaging and motion capture for motor rehabilitation

BACKGROUND

Technological advancements in functional neuroimaging and motion capture have led to the development of novel methods that facilitate the diagnosis and rehabilitation of motor deficits. These advancements allow for the synchronous acquisition and analysis of complex signal streams of neurophysiological data (e.g., EEG, fNIRS) and behavioral data (e.g., motion capture). The fusion of those data streams has the potential to provide new insights into cortical mechanisms during movement, guide the development of rehabilitation practices, and become a tool for assessment and therapy in neurorehabilitation.

RESEARCH OBJECTIVE

This paper aims to review the existing literature on the combined use of motion capture and functional neuroimaging in motor rehabilitation. The objective is to understand the diversity and maturity of technological solutions employed and explore the clinical advantages of this multimodal approach.

METHODS

This paper reviews literature related to the combined use of functional neuroimaging and motion capture for motor rehabilitation following the PRISMA guidelines. Besides study and participant characteristics, technological aspects of the used systems, signal processing methods, and the nature of multimodal feature synchronization and fusion were extracted.

RESULTS

Out of 908 publications, 19 were included in the final review. Basic or translation studies were mainly represented and based predominantly on healthy participants or stroke patients. EEG and mechanical motion capture technologies were most used for biomechanical data acquisition, and their subsequent processing is based mainly on traditional methods. The system synchronization techniques at large were underreported. The fusion of multimodal features mainly supported the identification of movement-related cortical activity, and statistical methods were occasionally employed to examine cortico-kinematic relationships.

CONCLUSION

The fusion of motion capture and functional neuroimaging might offer advantages for motor rehabilitation in the future. Besides facilitating the assessment of cognitive processes in real-world settings, it could also improve rehabilitative devices' usability in clinical environments. Further, by better understanding cortico-peripheral coupling, new neuro-rehabilitation methods can be developed, such as personalized proprioceptive training. However, further research is needed to advance our knowledge of cortical-peripheral coupling, evaluate the validity and reliability of multimodal parameters, and enhance user-friendly technologies for clinical adaptation.

Cortico-muscular connectivity is modulated by passive and active Lokomat-assisted Gait

The effects of robotic-assisted gait (RAG) training, besides conventional therapy, on neuroplasticity mechanisms and cortical integration in locomotion are still uncertain. To advance our knowledge on the matter, we determined the involvement of motor cortical areas in the control of muscle activity in healthy subjects, during RAG with Lokomat, both with maximal guidance force (100 GF-passive RAG) and without guidance force (0 GF-active RAG) as customary in rehabilitation treatments. We applied a novel cortico-muscular connectivity estimation procedure, based on Partial Directed Coherence, to jointly study source localized EEG and EMG activity during rest (standing) and active/passive RAG. We found greater cortico-cortical connectivity, with higher path length and tendency toward segregation during rest than in both RAG conditions, for all frequency bands except for delta. We also found higher cortico-muscular connectivity in distal muscles during swing (0 GF), and stance (100 GF), highlighting the importance of direct supraspinal control to maintain balance, even when gait is supported by a robotic exoskeleton. Source-localized connectivity shows that this control is driven mainly by the parietal and frontal lobes. The involvement of many cortical areas also in passive RAG (100 GF) justifies the use of the 100 GF RAG training for neurorehabilitation, with the aim of enhancing cortical-muscle connections and driving neural plasticity in neurological patients.

The truth Hertz-synchronization of electroencephalogram signals with physiological waveforms recorded in an intensive care unit

Objective.The ability to synchronize continuous electroencephalogram (cEEG) signals with physiological waveforms such as electrocardiogram (ECG), invasive pressures, photoplethysmography and other signals can provide meaningful insights regarding coupling between brain activity and other physiological subsystems. Aligning these datasets is a particularly challenging problem because device clocks handle time differently and synchronization protocols may be undocumented or proprietary.Approach.We used an ensemble-based model to detect the timestamps of heartbeat artefacts from ECG waveforms recorded from inpatient bedside monitors and from cEEG signals acquired using a different device. Vectors of inter-beat intervals were matched between both datasets and robust linear regression was applied to measure the relative time offset between the two datasets as a function of time.Main Results.The timing error between the two unsynchronized datasets ranged between -84 s and +33 s (mean 0.77 s, median 4.31 s, IQR25-4.79 s, IQR75 11.38s). Application of our method improved the relative alignment to within ± 5ms for more than 61% of the dataset. The mean clock drift between the two datasets was 418.3 parts per million (ppm) (median 414.6 ppm, IQR25 411.0 ppm, IQR75 425.6 ppm). A signal quality index was generated that described the quality of alignment for each cEEG study as a function of time.Significance.We developed and tested a method to retrospectively time-align two clinical waveform datasets acquired from different devices using a common signal. The method was applied to 33,911h of signals collected in a paediatric critical care unit over six years, demonstrating that the method can be applied to long-term recordings collected under clinical conditions. The method can account for unknown clock drift rates and the presence of discontinuities caused by clock resynchronization events.

A wearable group-synchronized EEG system for multi-subject brain-computer interfaces

Objective

The multi-subject brain-computer interface (mBCI) is becoming a key tool for the analysis of group behaviors. It is necessary to adopt a neural recording system for collaborative brain signal acquisition, which is usually in the form of a fixed wire.

Approach

In this study, we designed a wireless group-synchronized neural recording system that supports real-time mBCI and event-related potential (ERP) analysis. This system uses a wireless synchronizer to broadcast events to multiple wearable EEG amplifiers. The simultaneously received broadcast signals are marked in data packets to achieve real-time event correlation analysis of multiple targets in a group.

Main results

To evaluate the performance of the proposed real-time group-synchronized neural recording system, we conducted collaborative signal sampling on 10 wireless mBCI devices. The average signal correlation reached 99.8%, the amplitude of average noise was 0.87 μV, and the average common mode rejection ratio (CMRR) reached 109.02 dB. The minimum synchronization error is 237 μs. We also tested the system in real-time processing of the steady-state visual-evoked potential (SSVEP) ranging from 8 to 15.8 Hz. Under 40 target stimulators, with 2 s data length, the average information transfer rate (ITR) reached 150 ± 20 bits/min, and the highest reached 260 bits/min, which was comparable to the marketing leading EEG system (the average: 150 ± 15 bits/min; the highest: 280 bits/min). The accuracy of target recognition in 2 s was 98%, similar to that of the Synamps2 (99%), but a higher signal-to-noise ratio (SNR) of 5.08 dB was achieved. We designed a group EEG cognitive experiment; to verify, this system can be used in noisy settings.

Significance

The evaluation results revealed that the proposed real-time group-synchronized neural recording system is a high-performance tool for real-time mBCI research. It is an enabler for a wide range of future applications in collaborative intelligence, cognitive neurology, and rehabilitation.

Outdoor walking: Mobile EEG dataset from walking during oddball task and walking synchronization task

This article describes a dataset from one standing and two outdoor walking tasks. Each task was performed by the same 18 participants twice, using foot accelerometers and two different EEG system configurations. The first task was a brief eyes open/eyes closed task. The second task was a six minute auditory oddball task performed in three conditions: Standing, walking alone and walking next to an experimenter. In the third task, the participants walked with the experimenter in three conditions: With their view of the experimenter blocked, walking naturally, and trying to synchronize their steps with the experimenter. During all walking conditions which included the experimenter, the experimenter walked following a headphone metronome to keep their steps consistent, also wearing a foot accelerometer. All tasks were performed twice on two separate days, using active electrode and passive electrode EEG configurations (Brain Products, GmbH). Data was used for Scanlon et al. (2021) and Scanlon et al. (2022), and could be used for learning about attention, walking mechanisms and social neuroscience. Scanlon, J. E., Jacobsen, N. S. J., Maack, M. C., & Debener, S. (2021). Does the electrode amplification style matter? A comparison of active and passive EEG system configurations during standing and walking. European Journal of Neuroscience, 54(12), 8381-8395. Scanlon, J. E. M., Jacobsen, N. S. J., Maack, M. C., & Debener, S. (2022). Stepping in time: Alpha-mu and beta oscillations during a walking synchronization task. NeuroImage, 253, 119099.

Timing errors and temporal uncertainty in clinical databases-A narrative review

A firm concept of time is essential for establishing causality in a clinical setting. Review of critical incidents and generation of study hypotheses require a robust understanding of the sequence of events but conducting such work can be problematic when timestamps are recorded by independent and unsynchronized clocks. Most clinical models implicitly assume that timestamps have been measured accurately and precisely, but this custom will need to be re-evaluated if our algorithms and models are to make meaningful use of higher frequency physiological data sources. In this narrative review we explore factors that can result in timestamps being erroneously recorded in a clinical setting, with particular focus on systems that may be present in a critical care unit. We discuss how clocks, medical devices, data storage systems, algorithmic effects, human factors, and other external systems may affect the accuracy and precision of recorded timestamps. The concept of temporal uncertainty is introduced, and a holistic approach to timing accuracy, precision, and uncertainty is proposed. This quantitative approach to modeling temporal uncertainty provides a basis to achieve enhanced model generalizability and improved analytical outcomes.

Synchronization of wearable motion capture and EMG measurement systems

Synchronization of motion capture systems with other modalities in out-of-the-lab settings is not trivial. Various synchronization methods exist, such as using servers or transistor-transistor-logic pulses. However, not all measurement set-ups allow for such synchronization methods. Therefore, we have developed and validated an acceleration based post-measurement method to synchronize an IMU based motion capture system and an EMG measurement device. On top of the thigh IMU an additional accelerometer was placed which was connected to the analog input of the EMG device. By applying cross-correlation continuously, the similarities in the measured acceleration by the two measurement systems can be used for synchronization. We performed a validation measurement on seven able-bodied subjects and tested various correlation window sizes in hour long measurements in an out of the lab setting. It can be concluded that the developed method works for different activities when a suitable window length is chosen for cross-correlation. If no other options are available for synchronization, this correlation based method using an additional accelerometer is a viable option.

A framework for quantifying the effects of transcranial magnetic stimulation on motor recovery from hemiparesis: Corticomuscular Network

OBJECTIVE

Transcranial magnetic stimulation (TMS) is an experimental therapy for promoting motor recovery from hemiparesis. At present, hemiparesis patients' responses to TMS are variable. To maximize its therapeutic potential, we need an approach that relates the electrophysiology of motor recovery and TMS. To this end, we propose Corticomuscular Network (CMN) representing the holistic motor system, including the cortico-cortical pathway, corticospinal tract, and muscle co-activation.

METHODS

CMN is made up of coherence between pairs of electrode signals and spatial locations of the electrodes. We associated coherence and graph features of CMN with Fugl-Meyer Assessment (FMA) for the upper extremity. Besides, we compared CMN between 8 patients with hemiparesis and 6 healthy controls and contrasted CMN of patients before and after a 1Hz TMS.

MAIN RESULTS

Corticomuscular coherence (CMC) correlated positively with FMA. The regression model between FMA and CMC between 5 pairs of channels had 0.99 adjusted R^2 and a p-value less than 0.01. Compared to healthy controls, CMN of patients tended to be a small-world network and was more interconnected with higher CMC. CMC between cortex and triceps brachii long head was higher in patients. 15-minute 1Hz TMS protocol induced coherence changes beyond the stimulation side and had a limited impact on CMN parameters that are related to motor recovery.

SIGNIFICANCE

CMN is a potential clinical approach to quantify rehabilitating progress. It also sheds light on the desirable electrophysiological effects of TMS based on which rehabilitating strategies can be optimized.

Stepping in time: Alpha-mu and beta oscillations during a walking synchronization task

Interpersonal behavioral synchrony is referred to as temporal coordination of action between two or more individuals. Humans tend to synchronize their movements during repetitive movement tasks such as walking. Mobile EEG technology now allows us to examine how this happens during gait. 18 participants equipped with foot accelerometers and mobile EEG walked with an experimenter in three conditions: With their view of the experimenter blocked, walking naturally, and trying to synchronize their steps with the experimenter. The experimenter walked following a headphone metronome to keep their steps consistent for all conditions. Step behavior and synchronization between the experimenter and participant were compared between conditions. Additionally, event-related spectral perturbations (ERSPs) were time-warped to the gait cycle in order to analyze alpha-mu (7.5-12.5 Hz) and beta (16-32 Hz) rhythms over the whole gait cycle. Step synchronization was significantly higher in the synchrony condition than in the natural condition. Likewise regarding ERSPs, right parietal channel (C4, C6, CP4, CP6) alpha-mu and central channel (C1, Cz, C2) beta power were suppressed from baseline in the walking synchrony condition compared to the natural walking condition. The natural and blocked conditions were not found to be significantly different in behavioral or spectral comparisons. Our results are compatible with the view that intentional synchronization employs systems associated with social interaction as well as the central motor system.

Pocketable Labs for Everyone: Synchronized Multi-Sensor Data Streaming and Recording on Smartphones with the Lab Streaming Layer

The streaming and recording of smartphone sensor signals is desirable for mHealth, telemedicine, environmental monitoring and other applications. Time series data gathered in these fields typically benefit from the time-synchronized integration of different sensor signals. However, solutions required for this synchronization are mostly available for stationary setups. We hope to contribute to the important emerging field of portable data acquisition by presenting open-source Android applications both for the synchronized streaming (Send-a) and recording (Record-a) of multiple sensor data streams. We validate the applications in terms of functionality, flexibility and precision in fully mobile setups and in hybrid setups combining mobile and desktop hardware. Our results show that the fully mobile solution is equivalent to well-established desktop versions. With the streaming application Send-a and the recording application Record-a, purely smartphone-based setups for mobile research and personal health settings can be realized on off-the-shelf Android devices.

Near-zero phase-lag hyperscanning in a novel wireless EEG system

Objective. Hyperscanning is an emerging technology that concurrently scans the neural dynamics of multiple individuals to study interpersonal interactions. In particular, hyperscanning with electroencephalography (EEG) is increasingly popular owing to its mobility and its ability to allow studying social interactions in naturalistic settings at the millisecond scale.Approach.To align multiple EEG time series with sophisticated event markers in a single time domain, a precise and unified timestamp is required for stream synchronization. This study proposes a clock-synchronized method that uses a custom-made RJ45 cable to coordinate the sampling between wireless EEG amplifiers to prevent incorrect estimation of interbrain connectivity due to asynchronous sampling. In this method, analog-to-digital converters are driven by the same sampling clock. Additionally, two clock-synchronized amplifiers leverage additional radio frequency channels to keep the counter of their receiving dongles updated, which guarantees that binding event markers received by the dongle with the EEG time series have the correct timestamp.Main results.The results of two simulation experiments and one video gaming experiment reveal that the proposed method ensures synchronous sampling in a system with multiple EEG devices, achieving near-zero phase lag and negligible amplitude difference between the signals.Significance.According to all of the signal-similarity metrics, the suggested method is a promising option for wireless EEG hyperscanning and can be utilized to precisely assess the interbrain couplings underlying social-interaction behaviors.

Wireless Multi-sensor Physio-Motion Measurement and Synchronization System and Method for HRI Research

There is a strong demand for acquisition, processing and understanding of a variety of physiological and behavioral signals from the measurements in human-robot interface (HRI). However, multiple data streams from these measurements bring considerable challenges for their synchronizations, either for offline analysis or for online HRI applications, especially when the sensors are wirelessly connected, without synchronization mechanisms, such as a network-time-protocol. In this paper, we presented a full wireless multi-modality sensor system comprising biopotential measurements such as EEG, EMG and inertial parameter data of articulated body-limb motions. In the paper, we propose two methods to synchronize and calibrate the transmission latencies from different wireless channels. The first method employs the traditional artificial electrical timing signal. The other one employs the force-acceleration relationship governed by Newton's Second Law to facilitate reconstruction of the sample-to-sample alignment between the two wireless sensors. The measured latencies are investigated and the result show that they could be determined consistently and accurately by the devised techniques.

Hybrid Human-Machine Interface for Gait Decoding Through Bayesian Fusion of EEG and EMG Classifiers

Despite the advances in the field of brain computer interfaces (BCI), the use of the sole electroencephalography (EEG) signal to control walking rehabilitation devices is currently not viable in clinical settings, due to its unreliability. Hybrid interfaces (hHMIs) represent a very recent solution to enhance the performance of single-signal approaches. These are classification approaches that combine multiple human-machine interfaces, normally including at least one BCI with other biosignals, such as the electromyography (EMG). However, their use for the decoding of gait activity is still limited. In this work, we propose and evaluate a hybrid human-machine interface (hHMI) to decode walking phases of both legs from the Bayesian fusion of EEG and EMG signals. The proposed hHMI significantly outperforms its single-signal counterparts, by providing high and stable performance even when the reliability of the muscular activity is compromised temporarily (e.g., fatigue) or permanently (e.g., weakness). Indeed, the hybrid approach shows a smooth degradation of classification performance after temporary EMG alteration, with more than 75% of accuracy at 30% of EMG amplitude, with respect to the EMG classifier whose performance decreases below 60% of accuracy. Moreover, the fusion of EEG and EMG information helps keeping a stable recognition rate of each gait phase of more than 80% independently on the permanent level of EMG degradation. From our study and findings from the literature, we suggest that the use of hybrid interfaces may be the key to enhance the usability of technologies restoring or assisting the locomotion on a wider population of patients in clinical applications and outside the laboratory environment.

ESB: A low-cost EEG Synchronization Box

Electroencephalography (EEG) is a neuroimaging technique with a temporal resolution in the millisecond scale. Popular ERPs and ERD/ERS functions, as well as EEG-fRMI data and hyperscanning methods requires a proper temporal alignment (namely, synchronization) with stimulus onsets and other devices. Hardware-based synchronization, based on a SYNC signal injected into the device, ensures a reliable timing. In this paper we describe the design, test and validation of an EEG Synchronization Box (ESB), able to condition and distribute a SYNC signal (analog and digital) to different devices simultaneously. ESB can be easily built by individuals with basic soldering skills and represents a cost-effective solution to the available commercial synchronization boxes, while preserving similar electrical and functional features.

Gumpy: a Python toolbox suitable for hybrid brain-computer interfaces

OBJECTIVE

The objective of this work is to present gumpy, a new free and open source Python toolbox designed for hybrid brain-computer interface (BCI).

APPROACH

Gumpy provides state-of-the-art algorithms and includes a rich selection of signal processing methods that have been employed by the BCI community over the last 20 years. In addition, a wide range of classification methods that span from classical machine learning algorithms to deep neural network models are provided. Gumpy can be used for both EEG and EMG biosignal analysis, visualization, real-time streaming and decoding.

RESULTS

The usage of the toolbox was demonstrated through two different offline example studies, namely movement prediction from EEG motor imagery, and the decoding of natural grasp movements with the applied finger forces from surface EMG (sEMG) signals. Additionally, gumpy was used for real-time control of a robot arm using steady-state visually evoked potentials (SSVEP) as well as for real-time prosthetic hand control using sEMG. Overall, obtained results with the gumpy toolbox are comparable or better than previously reported results on the same datasets.

SIGNIFICANCE

Gumpy is a free and open source software, which allows end-users to perform online hybrid BCIs and provides different techniques for processing and decoding of EEG and EMG signals. More importantly, the achieved results reveal that gumpy's deep learning toolbox can match or outperform the state-of-the-art in terms of accuracy. This can therefore enable BCI researchers to develop more robust decoding algorithms using novel techniques and hence chart a route ahead for new BCI improvements.