1
|
Larsen OFP, Tresselt WG, Lorenz EA, Holt T, Sandstrak G, Hansen TI, Su X, Holt A. A method for synchronized use of EEG and eye tracking in fully immersive VR. Front Hum Neurosci 2024; 18:1347974. [PMID: 38468815 PMCID: PMC10925625 DOI: 10.3389/fnhum.2024.1347974] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/01/2023] [Accepted: 02/06/2024] [Indexed: 03/13/2024] Open
Abstract
This study explores the synchronization of multimodal physiological data streams, in particular, the integration of electroencephalography (EEG) with a virtual reality (VR) headset featuring eye-tracking capabilities. A potential use case for the synchronized data streams is demonstrated by implementing a hybrid steady-state visually evoked potential (SSVEP) based brain-computer interface (BCI) speller within a fully immersive VR environment. The hardware latency analysis reveals an average offset of 36 ms between EEG and eye-tracking data streams and a mean jitter of 5.76 ms. The study further presents a proof of concept brain-computer interface (BCI) speller in VR, showcasing its potential for real-world applications. The findings highlight the feasibility of combining commercial EEG and VR technologies for neuroscientific research and open new avenues for studying brain activity in ecologically valid VR environments. Future research could focus on refining the synchronization methods and exploring applications in various contexts, such as learning and social interactions.
Collapse
Affiliation(s)
- Olav F. P. Larsen
- Motion Capture and Visualization Laboratory, Department of Computer Science, Faculty of Information Technology and Electrical Engineering, Norwegian University of Science and Technology, Trondheim, Norway
| | - William G. Tresselt
- Motion Capture and Visualization Laboratory, Department of Computer Science, Faculty of Information Technology and Electrical Engineering, Norwegian University of Science and Technology, Trondheim, Norway
| | - Emanuel A. Lorenz
- Motion Capture and Visualization Laboratory, Department of Computer Science, Faculty of Information Technology and Electrical Engineering, Norwegian University of Science and Technology, Trondheim, Norway
| | - Tomas Holt
- Motion Capture and Visualization Laboratory, Department of Computer Science, Faculty of Information Technology and Electrical Engineering, Norwegian University of Science and Technology, Trondheim, Norway
| | - Grethe Sandstrak
- Motion Capture and Visualization Laboratory, Department of Computer Science, Faculty of Information Technology and Electrical Engineering, Norwegian University of Science and Technology, Trondheim, Norway
| | - Tor I. Hansen
- Department of Neuromedicine and Movement Science, Faculty of Medicine and Health Sciences, Norwegian University of Science and Technology, Trondheim, Norway
- Department of Acquired Brain Injury, St. Olav's University Hospital, Trondheim, Norway
| | - Xiaomeng Su
- Motion Capture and Visualization Laboratory, Department of Computer Science, Faculty of Information Technology and Electrical Engineering, Norwegian University of Science and Technology, Trondheim, Norway
| | - Alexander Holt
- Motion Capture and Visualization Laboratory, Department of Computer Science, Faculty of Information Technology and Electrical Engineering, Norwegian University of Science and Technology, Trondheim, Norway
| |
Collapse
|
2
|
Studnicki A, Ferris DP. Dual-layer electroencephalography data during real-world table tennis. Data Brief 2024; 52:110024. [PMID: 38287945 PMCID: PMC10823104 DOI: 10.1016/j.dib.2023.110024] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/28/2023] [Revised: 12/21/2023] [Accepted: 12/28/2023] [Indexed: 01/31/2024] Open
Abstract
Real-world settings are necessary to improve the ecological validity of neuroscience research, and electroencephalography (EEG) facilitates mobile electrocortical recordings because of its easy portability and high temporal resolution. Table tennis is a whole-body, goal-directed sport that requires constant visuomotor feedback, anticipation, strategic decision-making, object interception, and performance monitoring - making it an interesting testbed for a variety of neuroscience studies. Although traditionally plagued by artifact contamination, recent advances in signal processing and hardware approaches, such as the dual-layer approach, have allowed high fidelity EEG recordings during whole-body maneuvers. Here, we present a dual-layer EEG dataset from 25 healthy human participants playing table tennis with a human opponent and a ball machine. Our dataset includes synchronized, multivariate time series recordings from 120 scalp electrodes, 120 noise electrodes, 8 neck electromyography electrodes, and inertial measurement units on the participant, paddles, and ball machine to record hit events. We also include de-identified T1 anatomical MR images and digitized electrode locations to create subject-specific head models for source localization. In addition, we provide anonymized video recordings and Adobe Premiere project files with hit events labeled (originally used to mark successful/missed hits). Researchers could use the videos to mark their own events of interest. We formatted our dataset in the Brain Imaging Data Structure (BIDS) format to facilitate data reuse and to adhere to the scientific community's new organization standard.
Collapse
Affiliation(s)
- Amanda Studnicki
- J. Crayton Pruitt Family Department of Biomedical Engineering, University of Florida, Gainesville, FL 32611, United States
| | - Daniel P. Ferris
- J. Crayton Pruitt Family Department of Biomedical Engineering, University of Florida, Gainesville, FL 32611, United States
| |
Collapse
|
3
|
Lorenz EA, Su X, Skjæret-Maroni N. A review of combined functional neuroimaging and motion capture for motor rehabilitation. J Neuroeng Rehabil 2024; 21:3. [PMID: 38172799 PMCID: PMC10765727 DOI: 10.1186/s12984-023-01294-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/23/2023] [Accepted: 12/11/2023] [Indexed: 01/05/2024] Open
Abstract
BACKGROUND Technological advancements in functional neuroimaging and motion capture have led to the development of novel methods that facilitate the diagnosis and rehabilitation of motor deficits. These advancements allow for the synchronous acquisition and analysis of complex signal streams of neurophysiological data (e.g., EEG, fNIRS) and behavioral data (e.g., motion capture). The fusion of those data streams has the potential to provide new insights into cortical mechanisms during movement, guide the development of rehabilitation practices, and become a tool for assessment and therapy in neurorehabilitation. RESEARCH OBJECTIVE This paper aims to review the existing literature on the combined use of motion capture and functional neuroimaging in motor rehabilitation. The objective is to understand the diversity and maturity of technological solutions employed and explore the clinical advantages of this multimodal approach. METHODS This paper reviews literature related to the combined use of functional neuroimaging and motion capture for motor rehabilitation following the PRISMA guidelines. Besides study and participant characteristics, technological aspects of the used systems, signal processing methods, and the nature of multimodal feature synchronization and fusion were extracted. RESULTS Out of 908 publications, 19 were included in the final review. Basic or translation studies were mainly represented and based predominantly on healthy participants or stroke patients. EEG and mechanical motion capture technologies were most used for biomechanical data acquisition, and their subsequent processing is based mainly on traditional methods. The system synchronization techniques at large were underreported. The fusion of multimodal features mainly supported the identification of movement-related cortical activity, and statistical methods were occasionally employed to examine cortico-kinematic relationships. CONCLUSION The fusion of motion capture and functional neuroimaging might offer advantages for motor rehabilitation in the future. Besides facilitating the assessment of cognitive processes in real-world settings, it could also improve rehabilitative devices' usability in clinical environments. Further, by better understanding cortico-peripheral coupling, new neuro-rehabilitation methods can be developed, such as personalized proprioceptive training. However, further research is needed to advance our knowledge of cortical-peripheral coupling, evaluate the validity and reliability of multimodal parameters, and enhance user-friendly technologies for clinical adaptation.
Collapse
Affiliation(s)
- Emanuel A Lorenz
- Department of Computer Science, Norwegian University of Science and Technology, Trondheim, Norway.
| | - Xiaomeng Su
- Department of Computer Science, Norwegian University of Science and Technology, Trondheim, Norway
| | - Nina Skjæret-Maroni
- Department of Neuromedicine and Movement Science, Norwegian University of Science and Technology, Trondheim, Norway
| |
Collapse
|
4
|
Artoni F, Cometa A, Dalise S, Azzollini V, Micera S, Chisari C. Cortico-muscular connectivity is modulated by passive and active Lokomat-assisted Gait. Sci Rep 2023; 13:21618. [PMID: 38062035 PMCID: PMC10703891 DOI: 10.1038/s41598-023-48072-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/06/2023] [Accepted: 11/22/2023] [Indexed: 12/18/2023] Open
Abstract
The effects of robotic-assisted gait (RAG) training, besides conventional therapy, on neuroplasticity mechanisms and cortical integration in locomotion are still uncertain. To advance our knowledge on the matter, we determined the involvement of motor cortical areas in the control of muscle activity in healthy subjects, during RAG with Lokomat, both with maximal guidance force (100 GF-passive RAG) and without guidance force (0 GF-active RAG) as customary in rehabilitation treatments. We applied a novel cortico-muscular connectivity estimation procedure, based on Partial Directed Coherence, to jointly study source localized EEG and EMG activity during rest (standing) and active/passive RAG. We found greater cortico-cortical connectivity, with higher path length and tendency toward segregation during rest than in both RAG conditions, for all frequency bands except for delta. We also found higher cortico-muscular connectivity in distal muscles during swing (0 GF), and stance (100 GF), highlighting the importance of direct supraspinal control to maintain balance, even when gait is supported by a robotic exoskeleton. Source-localized connectivity shows that this control is driven mainly by the parietal and frontal lobes. The involvement of many cortical areas also in passive RAG (100 GF) justifies the use of the 100 GF RAG training for neurorehabilitation, with the aim of enhancing cortical-muscle connections and driving neural plasticity in neurological patients.
Collapse
Affiliation(s)
- Fiorenzo Artoni
- Department of Clinical Neurosciences, University of Genève, Faculty of Medicine, 1211, Geneva, Switzerland.
- Ago Neurotechnologies Sàrl, 1201, Geneva, Switzerland.
| | - Andrea Cometa
- The BioRobotics Institute and Department of Excellence in Robotics and AI, Scuola Superiore Sant'Anna, Viale Rinaldo Piaggio 34, 56025, Pontedera, Italy
- University School for Advanced Studies IUSS Pavia, 27100, Pavia, Italy
| | - Stefania Dalise
- Unit of Neurorehabilitation, Pisa University Hospital, Pisa, Italy
| | - Valentina Azzollini
- Department of Translational Research and New Technologies in Medicine and Surgery, University of Pisa, Pisa, Italy
| | - Silvestro Micera
- The BioRobotics Institute and Department of Excellence in Robotics and AI, Scuola Superiore Sant'Anna, Viale Rinaldo Piaggio 34, 56025, Pontedera, Italy
- Translational Neural Engineering Laboratory (TNE), École Polytechnique Fédérale de Lausanne, Campus Biotech, Geneva, Switzerland
| | - Carmelo Chisari
- Unit of Neurorehabilitation, Pisa University Hospital, Pisa, Italy
- Department of Translational Research and New Technologies in Medicine and Surgery, University of Pisa, Pisa, Italy
| |
Collapse
|
5
|
Goodwin AJ, Dixon W, Mazwi M, Hahn CD, Meir T, Goodfellow SD, Kazazian V, Greer RW, McEwan A, Laussen PC, Eytan D. The truth Hertz-synchronization of electroencephalogram signals with physiological waveforms recorded in an intensive care unit. Physiol Meas 2023; 44:085002. [PMID: 37406636 DOI: 10.1088/1361-6579/ace49e] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/09/2023] [Accepted: 07/05/2023] [Indexed: 07/07/2023]
Abstract
Objective.The ability to synchronize continuous electroencephalogram (cEEG) signals with physiological waveforms such as electrocardiogram (ECG), invasive pressures, photoplethysmography and other signals can provide meaningful insights regarding coupling between brain activity and other physiological subsystems. Aligning these datasets is a particularly challenging problem because device clocks handle time differently and synchronization protocols may be undocumented or proprietary.Approach.We used an ensemble-based model to detect the timestamps of heartbeat artefacts from ECG waveforms recorded from inpatient bedside monitors and from cEEG signals acquired using a different device. Vectors of inter-beat intervals were matched between both datasets and robust linear regression was applied to measure the relative time offset between the two datasets as a function of time.Main Results.The timing error between the two unsynchronized datasets ranged between -84 s and +33 s (mean 0.77 s, median 4.31 s, IQR25-4.79 s, IQR75 11.38s). Application of our method improved the relative alignment to within ± 5ms for more than 61% of the dataset. The mean clock drift between the two datasets was 418.3 parts per million (ppm) (median 414.6 ppm, IQR25 411.0 ppm, IQR75 425.6 ppm). A signal quality index was generated that described the quality of alignment for each cEEG study as a function of time.Significance.We developed and tested a method to retrospectively time-align two clinical waveform datasets acquired from different devices using a common signal. The method was applied to 33,911h of signals collected in a paediatric critical care unit over six years, demonstrating that the method can be applied to long-term recordings collected under clinical conditions. The method can account for unknown clock drift rates and the presence of discontinuities caused by clock resynchronization events.
Collapse
Affiliation(s)
- Andrew J Goodwin
- School of Biomedical Engineering, University of Sydney, Sydney, New South Wales, Australia
- Department of Critical Care Medicine, The Hospital for Sick Children, Toronto, Ontario, Canada
| | - William Dixon
- Department of Critical Care Medicine, The Hospital for Sick Children, Toronto, Ontario, Canada
| | - Mjaye Mazwi
- Department of Critical Care Medicine, The Hospital for Sick Children, Toronto, Ontario, Canada
- Division of Cardiology, Department of Paediatrics, The Hospital for Sick Children, Toronto, Ontario, Canada
| | - Cecil D Hahn
- Department of Paediatrics, University of Toronto, Toronto, Ontario, Canada
- Division of Neurology, The Hospital for Sick Children, Toronto, ON, Canada
| | - Tomer Meir
- Technion - Israel Institute of Technology, Haifa, Israel
| | - Sebastian D Goodfellow
- Department of Critical Care Medicine, The Hospital for Sick Children, Toronto, Ontario, Canada
- Department of Civil and Mineral Engineering, University of Toronto, Toronto, Ontario, Canada
| | - Vanna Kazazian
- Division of Cardiology, Department of Paediatrics, The Hospital for Sick Children, Toronto, Ontario, Canada
| | - Robert W Greer
- Department of Critical Care Medicine, The Hospital for Sick Children, Toronto, Ontario, Canada
| | - Alistair McEwan
- School of Biomedical Engineering, University of Sydney, Sydney, New South Wales, Australia
| | - Peter C Laussen
- Department of Anesthesia, Boston Children's Hospital, Boston, MA, United States of America
- Department of Cardiology, Boston Children's Hospital, Boston, MA, United States of America
- Department of Anaesthesia, Harvard Medical School, Boston MA, United States of America
| | - Danny Eytan
- Department of Critical Care Medicine, The Hospital for Sick Children, Toronto, Ontario, Canada
- Department of Medicine, Technion, Haifa, Israel
| |
Collapse
|
6
|
Huang Y, Huan Y, Zou Z, Pei W, Gao X, Wang Y, Zheng L. A wearable group-synchronized EEG system for multi-subject brain-computer interfaces. Front Neurosci 2023; 17:1176344. [PMID: 37539380 PMCID: PMC10396297 DOI: 10.3389/fnins.2023.1176344] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/28/2023] [Accepted: 06/26/2023] [Indexed: 08/05/2023] Open
Abstract
Objective The multi-subject brain-computer interface (mBCI) is becoming a key tool for the analysis of group behaviors. It is necessary to adopt a neural recording system for collaborative brain signal acquisition, which is usually in the form of a fixed wire. Approach In this study, we designed a wireless group-synchronized neural recording system that supports real-time mBCI and event-related potential (ERP) analysis. This system uses a wireless synchronizer to broadcast events to multiple wearable EEG amplifiers. The simultaneously received broadcast signals are marked in data packets to achieve real-time event correlation analysis of multiple targets in a group. Main results To evaluate the performance of the proposed real-time group-synchronized neural recording system, we conducted collaborative signal sampling on 10 wireless mBCI devices. The average signal correlation reached 99.8%, the amplitude of average noise was 0.87 μV, and the average common mode rejection ratio (CMRR) reached 109.02 dB. The minimum synchronization error is 237 μs. We also tested the system in real-time processing of the steady-state visual-evoked potential (SSVEP) ranging from 8 to 15.8 Hz. Under 40 target stimulators, with 2 s data length, the average information transfer rate (ITR) reached 150 ± 20 bits/min, and the highest reached 260 bits/min, which was comparable to the marketing leading EEG system (the average: 150 ± 15 bits/min; the highest: 280 bits/min). The accuracy of target recognition in 2 s was 98%, similar to that of the Synamps2 (99%), but a higher signal-to-noise ratio (SNR) of 5.08 dB was achieved. We designed a group EEG cognitive experiment; to verify, this system can be used in noisy settings. Significance The evaluation results revealed that the proposed real-time group-synchronized neural recording system is a high-performance tool for real-time mBCI research. It is an enabler for a wide range of future applications in collaborative intelligence, cognitive neurology, and rehabilitation.
Collapse
Affiliation(s)
- Yong Huang
- School of Biomedical Engineering, Southern Medical University, Guangzhou, China
- Brain-Inspired Computing Laboratory, Guangdong Institute of Intelligence Science and Technology, Hengqin, China
| | - Yuxiang Huan
- Brain-Inspired Computing Laboratory, Guangdong Institute of Intelligence Science and Technology, Hengqin, China
| | - Zhuo Zou
- School of Information Science and Technology, Fudan University, Shanghai, China
| | - Weihua Pei
- Institute of Semiconductors, Chinese Academy of Sciences (CAS), Beijing, China
| | - Xiaorong Gao
- Department of Biomedical Engineering, Tsinghua University, Beijing, China
| | - Yijun Wang
- Institute of Semiconductors, Chinese Academy of Sciences (CAS), Beijing, China
| | - Lirong Zheng
- School of Biomedical Engineering, Southern Medical University, Guangzhou, China
- Brain-Inspired Computing Laboratory, Guangdong Institute of Intelligence Science and Technology, Hengqin, China
| |
Collapse
|
7
|
Scanlon JE, Jacobsen NS, Maack MC, Debener S. Outdoor walking: Mobile EEG dataset from walking during oddball task and walking synchronization task. Data Brief 2022; 46:108847. [PMID: 36687153 PMCID: PMC9852920 DOI: 10.1016/j.dib.2022.108847] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/24/2022] [Revised: 12/01/2022] [Accepted: 12/19/2022] [Indexed: 12/24/2022] Open
Abstract
This article describes a dataset from one standing and two outdoor walking tasks. Each task was performed by the same 18 participants twice, using foot accelerometers and two different EEG system configurations. The first task was a brief eyes open/eyes closed task. The second task was a six minute auditory oddball task performed in three conditions: Standing, walking alone and walking next to an experimenter. In the third task, the participants walked with the experimenter in three conditions: With their view of the experimenter blocked, walking naturally, and trying to synchronize their steps with the experimenter. During all walking conditions which included the experimenter, the experimenter walked following a headphone metronome to keep their steps consistent, also wearing a foot accelerometer. All tasks were performed twice on two separate days, using active electrode and passive electrode EEG configurations (Brain Products, GmbH). Data was used for Scanlon et al. (2021) and Scanlon et al. (2022), and could be used for learning about attention, walking mechanisms and social neuroscience. Scanlon, J. E., Jacobsen, N. S. J., Maack, M. C., & Debener, S. (2021). Does the electrode amplification style matter? A comparison of active and passive EEG system configurations during standing and walking. European Journal of Neuroscience, 54(12), 8381-8395. Scanlon, J. E. M., Jacobsen, N. S. J., Maack, M. C., & Debener, S. (2022). Stepping in time: Alpha-mu and beta oscillations during a walking synchronization task. NeuroImage, 253, 119099.
Collapse
Affiliation(s)
- Joanna E.M. Scanlon
- Neuropsychology Lab, Department of Psychology, University of Oldenburg, Germany,Branch for Hearing, Speech and Audio Technology HSA, Fraunhofer Institute for Digital Media Technology IDMT, Oldenburg, Germany,Corresponding author at: Neuropsychology Lab, Department of Psychology, University of Oldenburg, Germany. @joanna_scanlon
| | | | - Marike C. Maack
- Neuropsychology Lab, Department of Psychology, University of Oldenburg, Germany
| | - Stefan Debener
- Neuropsychology Lab, Department of Psychology, University of Oldenburg, Germany,Cluster of Excellence Hearing4all, University of Oldenburg, Oldenburg, Germany,Center for Neurosensory Science and Systems, University of Oldenburg, Oldenburg, Germany,Branch for Hearing, Speech and Audio Technology HSA, Fraunhofer Institute for Digital Media Technology IDMT, Oldenburg, Germany
| |
Collapse
|
8
|
Goodwin AJ, Eytan D, Dixon W, Goodfellow SD, Doherty Z, Greer RW, McEwan A, Tracy M, Laussen PC, Assadi A, Mazwi M. Timing errors and temporal uncertainty in clinical databases-A narrative review. Front Digit Health 2022; 4:932599. [PMID: 36060541 PMCID: PMC9433547 DOI: 10.3389/fdgth.2022.932599] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/30/2022] [Accepted: 07/11/2022] [Indexed: 11/28/2022] Open
Abstract
A firm concept of time is essential for establishing causality in a clinical setting. Review of critical incidents and generation of study hypotheses require a robust understanding of the sequence of events but conducting such work can be problematic when timestamps are recorded by independent and unsynchronized clocks. Most clinical models implicitly assume that timestamps have been measured accurately and precisely, but this custom will need to be re-evaluated if our algorithms and models are to make meaningful use of higher frequency physiological data sources. In this narrative review we explore factors that can result in timestamps being erroneously recorded in a clinical setting, with particular focus on systems that may be present in a critical care unit. We discuss how clocks, medical devices, data storage systems, algorithmic effects, human factors, and other external systems may affect the accuracy and precision of recorded timestamps. The concept of temporal uncertainty is introduced, and a holistic approach to timing accuracy, precision, and uncertainty is proposed. This quantitative approach to modeling temporal uncertainty provides a basis to achieve enhanced model generalizability and improved analytical outcomes.
Collapse
Affiliation(s)
- Andrew J. Goodwin
- Department of Critical Care Medicine, The Hospital for Sick Children, Toronto, ON, Canada
- School of Biomedical Engineering, University of Sydney, Sydney, NSW, Australia
| | - Danny Eytan
- Department of Critical Care Medicine, The Hospital for Sick Children, Toronto, ON, Canada
- Department of Medicine, Technion - Israel Institute of Technology, Haifa, Israel
| | - William Dixon
- Department of Critical Care Medicine, The Hospital for Sick Children, Toronto, ON, Canada
| | - Sebastian D. Goodfellow
- Department of Critical Care Medicine, The Hospital for Sick Children, Toronto, ON, Canada
- Department of Civil and Mineral Engineering, University of Toronto, Toronto, ON, Canada
| | - Zakary Doherty
- Research Fellow, School of Rural Health, Monash University, Melbourne, VIC, Australia
| | - Robert W. Greer
- Department of Critical Care Medicine, The Hospital for Sick Children, Toronto, ON, Canada
| | - Alistair McEwan
- School of Biomedical Engineering, University of Sydney, Sydney, NSW, Australia
| | - Mark Tracy
- Neonatal Intensive Care Unit, Westmead Hospital, Sydney, NSW, Australia
- Department of Paediatrics and Child Health, The University of Sydney, Sydney, NSW, Australia
| | - Peter C. Laussen
- Department of Anesthesia, Boston Children's Hospital, Boston, MA, United States
| | - Azadeh Assadi
- Department of Critical Care Medicine, The Hospital for Sick Children, Toronto, ON, Canada
- Department of Engineering and Applied Sciences, Institute of Biomedical Engineering, University of Toronto, Toronto, ON, Canada
| | - Mjaye Mazwi
- Department of Critical Care Medicine, The Hospital for Sick Children, Toronto, ON, Canada
- Department of Paediatrics, University of Toronto, Toronto, ON, Canada
| |
Collapse
|
9
|
Schulte RV, Prinsen EC, Schaake L, Buurke JH. Synchronization of wearable motion capture and EMG measurement systems. IEEE Int Conf Rehabil Robot 2022; 2022:1-6. [PMID: 36176165 DOI: 10.1109/icorr55369.2022.9896587] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 06/16/2023]
Abstract
Synchronization of motion capture systems with other modalities in out-of-the-lab settings is not trivial. Various synchronization methods exist, such as using servers or transistor-transistor-logic pulses. However, not all measurement set-ups allow for such synchronization methods. Therefore, we have developed and validated an acceleration based post-measurement method to synchronize an IMU based motion capture system and an EMG measurement device. On top of the thigh IMU an additional accelerometer was placed which was connected to the analog input of the EMG device. By applying cross-correlation continuously, the similarities in the measured acceleration by the two measurement systems can be used for synchronization. We performed a validation measurement on seven able-bodied subjects and tested various correlation window sizes in hour long measurements in an out of the lab setting. It can be concluded that the developed method works for different activities when a suitable window length is chosen for cross-correlation. If no other options are available for synchronization, this correlation based method using an additional accelerometer is a viable option.
Collapse
|
10
|
Tan G, Wang J, Liu J, Sheng Y, Xie Q, Liu H. A framework for quantifying the effects of transcranial magnetic stimulation on motor recovery from hemiparesis: Corticomuscular Network. J Neural Eng 2022; 19. [PMID: 35366651 DOI: 10.1088/1741-2552/ac636b] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/11/2022] [Accepted: 04/01/2022] [Indexed: 11/12/2022]
Abstract
OBJECTIVE Transcranial magnetic stimulation (TMS) is an experimental therapy for promoting motor recovery from hemiparesis. At present, hemiparesis patients' responses to TMS are variable. To maximize its therapeutic potential, we need an approach that relates the electrophysiology of motor recovery and TMS. To this end, we propose Corticomuscular Network (CMN) representing the holistic motor system, including the cortico-cortical pathway, corticospinal tract, and muscle co-activation. METHODS CMN is made up of coherence between pairs of electrode signals and spatial locations of the electrodes. We associated coherence and graph features of CMN with Fugl-Meyer Assessment (FMA) for the upper extremity. Besides, we compared CMN between 8 patients with hemiparesis and 6 healthy controls and contrasted CMN of patients before and after a 1Hz TMS. MAIN RESULTS Corticomuscular coherence (CMC) correlated positively with FMA. The regression model between FMA and CMC between 5 pairs of channels had 0.99 adjusted R^2 and a p-value less than 0.01. Compared to healthy controls, CMN of patients tended to be a small-world network and was more interconnected with higher CMC. CMC between cortex and triceps brachii long head was higher in patients. 15-minute 1Hz TMS protocol induced coherence changes beyond the stimulation side and had a limited impact on CMN parameters that are related to motor recovery. SIGNIFICANCE CMN is a potential clinical approach to quantify rehabilitating progress. It also sheds light on the desirable electrophysiological effects of TMS based on which rehabilitating strategies can be optimized.
Collapse
Affiliation(s)
- Gansheng Tan
- Washington University in St Louis, 520 S Euclid Ave, St. Louis, MO 63110, St Louis, Missouri, 63130-4899, UNITED STATES
| | - Jixian Wang
- Shanghai Jiao Tong University Medical School Affiliated Ruijin Hospital, 800 Dongchuan Rd, Shanghai, 200025, CHINA
| | - Jinbiao Liu
- Shanghai Jiao Tong University, 800 Dongchuan Rd, Shanghai, 200240, CHINA
| | - Yixuan Sheng
- Shanghai Jiao Tong University, 800 Dongchuan Rd, Shanghai, 200240, CHINA
| | - Qing Xie
- Ruijin Hospital, 800 Dongchuan Rd, Shanghai, 200025, CHINA
| | - Honghai Liu
- Harbin Institute of Technology Shenzhen, Pingshan 1 Rd, Nanshan, Shenzhen, Guangdong, 518055, CHINA
| |
Collapse
|
11
|
Scanlon JEM, Jacobsen NSJ, Maack MC, Debener S. Stepping in time: Alpha-mu and beta oscillations during a walking synchronization task. Neuroimage 2022; 253:119099. [PMID: 35301131 DOI: 10.1016/j.neuroimage.2022.119099] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/05/2021] [Revised: 02/18/2022] [Accepted: 03/13/2022] [Indexed: 11/25/2022] Open
Abstract
Interpersonal behavioral synchrony is referred to as temporal coordination of action between two or more individuals. Humans tend to synchronize their movements during repetitive movement tasks such as walking. Mobile EEG technology now allows us to examine how this happens during gait. 18 participants equipped with foot accelerometers and mobile EEG walked with an experimenter in three conditions: With their view of the experimenter blocked, walking naturally, and trying to synchronize their steps with the experimenter. The experimenter walked following a headphone metronome to keep their steps consistent for all conditions. Step behavior and synchronization between the experimenter and participant were compared between conditions. Additionally, event-related spectral perturbations (ERSPs) were time-warped to the gait cycle in order to analyze alpha-mu (7.5-12.5 Hz) and beta (16-32 Hz) rhythms over the whole gait cycle. Step synchronization was significantly higher in the synchrony condition than in the natural condition. Likewise regarding ERSPs, right parietal channel (C4, C6, CP4, CP6) alpha-mu and central channel (C1, Cz, C2) beta power were suppressed from baseline in the walking synchrony condition compared to the natural walking condition. The natural and blocked conditions were not found to be significantly different in behavioral or spectral comparisons. Our results are compatible with the view that intentional synchronization employs systems associated with social interaction as well as the central motor system.
Collapse
Affiliation(s)
- J E M Scanlon
- Neuropsychology Lab, Department of Psychology, University of Oldenburg, Oldenburg, Germany.
| | - N S J Jacobsen
- Neuropsychology Lab, Department of Psychology, University of Oldenburg, Oldenburg, Germany
| | - M C Maack
- Neuropsychology Lab, Department of Psychology, University of Oldenburg, Oldenburg, Germany
| | - S Debener
- Neuropsychology Lab, Department of Psychology, University of Oldenburg, Oldenburg, Germany; Cluster of Excellence Hearing4all, University of Oldenburg, Oldenburg, Germany; Center for Neurosensory Science and Systems, University of Oldenburg, Oldenburg, Germany
| |
Collapse
|
12
|
Blum S, Hölle D, Bleichner MG, Debener S. Pocketable Labs for Everyone: Synchronized Multi-Sensor Data Streaming and Recording on Smartphones with the Lab Streaming Layer. SENSORS (BASEL, SWITZERLAND) 2021; 21:8135. [PMID: 34884139 PMCID: PMC8662410 DOI: 10.3390/s21238135] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 10/15/2021] [Revised: 11/16/2021] [Accepted: 12/01/2021] [Indexed: 12/14/2022]
Abstract
The streaming and recording of smartphone sensor signals is desirable for mHealth, telemedicine, environmental monitoring and other applications. Time series data gathered in these fields typically benefit from the time-synchronized integration of different sensor signals. However, solutions required for this synchronization are mostly available for stationary setups. We hope to contribute to the important emerging field of portable data acquisition by presenting open-source Android applications both for the synchronized streaming (Send-a) and recording (Record-a) of multiple sensor data streams. We validate the applications in terms of functionality, flexibility and precision in fully mobile setups and in hybrid setups combining mobile and desktop hardware. Our results show that the fully mobile solution is equivalent to well-established desktop versions. With the streaming application Send-a and the recording application Record-a, purely smartphone-based setups for mobile research and personal health settings can be realized on off-the-shelf Android devices.
Collapse
Affiliation(s)
- Sarah Blum
- Neuropsychology Lab, Department of Psychology, University of Oldenburg, 26111 Oldenburg, Germany;
- Cluster of Excellence Hearing4all, 26111 Oldenburg, Germany
| | - Daniel Hölle
- Neurophysiology of Everyday Life Group, Department of Psychology, University of Oldenburg, 26111 Oldenburg, Germany; (D.H.); (M.G.B.)
| | - Martin Georg Bleichner
- Neurophysiology of Everyday Life Group, Department of Psychology, University of Oldenburg, 26111 Oldenburg, Germany; (D.H.); (M.G.B.)
| | - Stefan Debener
- Neuropsychology Lab, Department of Psychology, University of Oldenburg, 26111 Oldenburg, Germany;
- Cluster of Excellence Hearing4all, 26111 Oldenburg, Germany
| |
Collapse
|
13
|
Chuang CH, Lu SW, Chao YP, Peng PH, Hsu HC, Hung CC, Chang CL, Jung TP. Near-zero phase-lag hyperscanning in a novel wireless EEG system. J Neural Eng 2021; 18. [PMID: 34706357 DOI: 10.1088/1741-2552/ac33e6] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/03/2021] [Accepted: 10/27/2021] [Indexed: 11/12/2022]
Abstract
Objective. Hyperscanning is an emerging technology that concurrently scans the neural dynamics of multiple individuals to study interpersonal interactions. In particular, hyperscanning with electroencephalography (EEG) is increasingly popular owing to its mobility and its ability to allow studying social interactions in naturalistic settings at the millisecond scale.Approach.To align multiple EEG time series with sophisticated event markers in a single time domain, a precise and unified timestamp is required for stream synchronization. This study proposes a clock-synchronized method that uses a custom-made RJ45 cable to coordinate the sampling between wireless EEG amplifiers to prevent incorrect estimation of interbrain connectivity due to asynchronous sampling. In this method, analog-to-digital converters are driven by the same sampling clock. Additionally, two clock-synchronized amplifiers leverage additional radio frequency channels to keep the counter of their receiving dongles updated, which guarantees that binding event markers received by the dongle with the EEG time series have the correct timestamp.Main results.The results of two simulation experiments and one video gaming experiment reveal that the proposed method ensures synchronous sampling in a system with multiple EEG devices, achieving near-zero phase lag and negligible amplitude difference between the signals.Significance.According to all of the signal-similarity metrics, the suggested method is a promising option for wireless EEG hyperscanning and can be utilized to precisely assess the interbrain couplings underlying social-interaction behaviors.
Collapse
Affiliation(s)
- Chun-Hsiang Chuang
- Research Center for Education and Mind Sciences, College of Education, National Tsing Hua University, Hsinchu, Taiwan.,Institute of Information Systems and Applications, College of Electrical Engineering and Computer Science, National Tsing Hua University, Hsinchu, Taiwan
| | - Shao-Wei Lu
- Artise Biomedical Co., Ltd, Hsinchu, Taiwan.,Department of Biomedical Engineering, National Yang Ming Chiao Tung University, Hsinchu, Taiwan
| | - Yi-Ping Chao
- Graduate Institute of Biomedical Engineering, Chang Gung University, Taoyuan, Taiwan
| | - Po-Hsun Peng
- Department of Computer Science and Engineering, National Taiwan Ocean University, Keelung, Taiwan
| | - Hao-Che Hsu
- Department of Computer Science and Engineering, National Taiwan Ocean University, Keelung, Taiwan
| | - Cheng-Chieh Hung
- Research Center for Education and Mind Sciences, College of Education, National Tsing Hua University, Hsinchu, Taiwan
| | | | - Tzyy-Ping Jung
- Institute for Neural Computation and Institute of Engineering in Medicine, University of California, San Diego, La Jolla, United States of America
| |
Collapse
|
14
|
Wang C, Zhang H, Ng SH, Zhu X, Ang KK. Wireless Multi-sensor Physio-Motion Measurement and Synchronization System and Method for HRI Research. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2021; 2021:7328-7331. [PMID: 34892790 DOI: 10.1109/embc46164.2021.9629500] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
There is a strong demand for acquisition, processing and understanding of a variety of physiological and behavioral signals from the measurements in human-robot interface (HRI). However, multiple data streams from these measurements bring considerable challenges for their synchronizations, either for offline analysis or for online HRI applications, especially when the sensors are wirelessly connected, without synchronization mechanisms, such as a network-time-protocol. In this paper, we presented a full wireless multi-modality sensor system comprising biopotential measurements such as EEG, EMG and inertial parameter data of articulated body-limb motions. In the paper, we propose two methods to synchronize and calibrate the transmission latencies from different wireless channels. The first method employs the traditional artificial electrical timing signal. The other one employs the force-acceleration relationship governed by Newton's Second Law to facilitate reconstruction of the sample-to-sample alignment between the two wireless sensors. The measured latencies are investigated and the result show that they could be determined consistently and accurately by the devised techniques.
Collapse
|
15
|
Tortora S, Tonin L, Chisari C, Micera S, Menegatti E, Artoni F. Hybrid Human-Machine Interface for Gait Decoding Through Bayesian Fusion of EEG and EMG Classifiers. Front Neurorobot 2020; 14:582728. [PMID: 33281593 PMCID: PMC7705173 DOI: 10.3389/fnbot.2020.582728] [Citation(s) in RCA: 17] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/13/2020] [Accepted: 09/30/2020] [Indexed: 01/25/2023] Open
Abstract
Despite the advances in the field of brain computer interfaces (BCI), the use of the sole electroencephalography (EEG) signal to control walking rehabilitation devices is currently not viable in clinical settings, due to its unreliability. Hybrid interfaces (hHMIs) represent a very recent solution to enhance the performance of single-signal approaches. These are classification approaches that combine multiple human-machine interfaces, normally including at least one BCI with other biosignals, such as the electromyography (EMG). However, their use for the decoding of gait activity is still limited. In this work, we propose and evaluate a hybrid human-machine interface (hHMI) to decode walking phases of both legs from the Bayesian fusion of EEG and EMG signals. The proposed hHMI significantly outperforms its single-signal counterparts, by providing high and stable performance even when the reliability of the muscular activity is compromised temporarily (e.g., fatigue) or permanently (e.g., weakness). Indeed, the hybrid approach shows a smooth degradation of classification performance after temporary EMG alteration, with more than 75% of accuracy at 30% of EMG amplitude, with respect to the EMG classifier whose performance decreases below 60% of accuracy. Moreover, the fusion of EEG and EMG information helps keeping a stable recognition rate of each gait phase of more than 80% independently on the permanent level of EMG degradation. From our study and findings from the literature, we suggest that the use of hybrid interfaces may be the key to enhance the usability of technologies restoring or assisting the locomotion on a wider population of patients in clinical applications and outside the laboratory environment.
Collapse
Affiliation(s)
- Stefano Tortora
- Department of Information Engineering, University of Padova, Padova, Italy
| | - Luca Tonin
- Department of Information Engineering, University of Padova, Padova, Italy
| | - Carmelo Chisari
- Unit of Neurorehabilitation, Department of Medical Specialties, University Hospital of Pisa, Pisa, Italy
| | - Silvestro Micera
- Department of Excellence in Robotics and AI Scuola Superiore Sant'Anna, The Biorobotics Institute, Pisa, Italy.,Bertarelli Foundation Chair in Translational Neuroengineering, Center for Neuroprosthetics and Institute of Bioengineering, Lausanne, Switzerland
| | - Emanuele Menegatti
- Department of Information Engineering, University of Padova, Padova, Italy
| | - Fiorenzo Artoni
- Bertarelli Foundation Chair in Translational Neuroengineering, Center for Neuroprosthetics and Institute of Bioengineering, Lausanne, Switzerland.,Functional Brain Mapping Laboratory, Department of Basic Neuroscience, Faculty of Medicine, University of Geneva, Geneva, Switzerland
| |
Collapse
|
16
|
Bilucaglia M, Masi R, Stanislao GD, Laureanti R, Fici A, Circi R, Zito M, Russo V. ESB: A low-cost EEG Synchronization Box. HARDWAREX 2020; 8:e00125. [PMID: 35498268 PMCID: PMC9041257 DOI: 10.1016/j.ohx.2020.e00125] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
Electroencephalography (EEG) is a neuroimaging technique with a temporal resolution in the millisecond scale. Popular ERPs and ERD/ERS functions, as well as EEG-fRMI data and hyperscanning methods requires a proper temporal alignment (namely, synchronization) with stimulus onsets and other devices. Hardware-based synchronization, based on a SYNC signal injected into the device, ensures a reliable timing. In this paper we describe the design, test and validation of an EEG Synchronization Box (ESB), able to condition and distribute a SYNC signal (analog and digital) to different devices simultaneously. ESB can be easily built by individuals with basic soldering skills and represents a cost-effective solution to the available commercial synchronization boxes, while preserving similar electrical and functional features.
Collapse
Affiliation(s)
| | - Riccardo Masi
- Behavior and Brain Lab, Università IULM, Milan, Italy
| | | | - Rita Laureanti
- Department of Electronic, Information and Bioengineering, Politecnico di Milano, Milan, Italy
| | | | | | - Margherita Zito
- Behavior and Brain Lab, Università IULM, Milan, Italy
- Department of Business, Law, Economics and Consumer Behaviour “Carlo A. Ricciardi”, Università IULM, Milan, Italy
| | - Vincenzo Russo
- Behavior and Brain Lab, Università IULM, Milan, Italy
- Department of Business, Law, Economics and Consumer Behaviour “Carlo A. Ricciardi”, Università IULM, Milan, Italy
| |
Collapse
|
17
|
Tortora S, Ghidoni S, Chisari C, Micera S, Artoni F. Deep learning-based BCI for gait decoding from EEG with LSTM recurrent neural network. J Neural Eng 2020; 17:046011. [DOI: 10.1088/1741-2552/ab9842] [Citation(s) in RCA: 30] [Impact Index Per Article: 7.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/01/2023]
|
18
|
Tayeb Z, Waniek N, Fedjaev J, Ghaboosi N, Rychly L, Widderich C, Richter C, Braun J, Saveriano M, Cheng G, Conradt J. Gumpy: a Python toolbox suitable for hybrid brain-computer interfaces. J Neural Eng 2018; 15:065003. [PMID: 30215610 DOI: 10.1088/1741-2552/aae186] [Citation(s) in RCA: 27] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/03/2023]
Abstract
OBJECTIVE The objective of this work is to present gumpy, a new free and open source Python toolbox designed for hybrid brain-computer interface (BCI). APPROACH Gumpy provides state-of-the-art algorithms and includes a rich selection of signal processing methods that have been employed by the BCI community over the last 20 years. In addition, a wide range of classification methods that span from classical machine learning algorithms to deep neural network models are provided. Gumpy can be used for both EEG and EMG biosignal analysis, visualization, real-time streaming and decoding. RESULTS The usage of the toolbox was demonstrated through two different offline example studies, namely movement prediction from EEG motor imagery, and the decoding of natural grasp movements with the applied finger forces from surface EMG (sEMG) signals. Additionally, gumpy was used for real-time control of a robot arm using steady-state visually evoked potentials (SSVEP) as well as for real-time prosthetic hand control using sEMG. Overall, obtained results with the gumpy toolbox are comparable or better than previously reported results on the same datasets. SIGNIFICANCE Gumpy is a free and open source software, which allows end-users to perform online hybrid BCIs and provides different techniques for processing and decoding of EEG and EMG signals. More importantly, the achieved results reveal that gumpy's deep learning toolbox can match or outperform the state-of-the-art in terms of accuracy. This can therefore enable BCI researchers to develop more robust decoding algorithms using novel techniques and hence chart a route ahead for new BCI improvements.
Collapse
Affiliation(s)
- Zied Tayeb
- Department of Electrical and Computer Engineering, Neuroscientific System Theory, Technical University of Munich, Munich, Germany. Institute for Cognitive Systems, Technical University of Munich, Munich, Germany
| | | | | | | | | | | | | | | | | | | | | |
Collapse
|