A Wearable Multi-Modal Bio-Sensing System Towards Real-World Applications

Can cognitive neuroscience solve the lab-dilemma by going wild?

Reproducibility, measurability, and refutability are the foundation of the scientific method applied to empirical work. In the study of animal and human behavior, experimental protocols conducted in the lab are the most reliable means by which scientists can operationalize behaviors using controlled and parameterized setups. However, whether observations in the lab fully generalize in the real world remain legitimately disputed. The notion of "experimental design" was originally intended to ensure the generalizability of experimental findings to real-world situations. Experiments in the wild are more frequently explored and significant technological advances have been made allowing mobile neuroimaging. Yet some methodological limitations remain when testing scientific hypotheses in ecological conditions. Herein, we discuss the limitations of inferential processes derive from empirical observations in the wild. The multi-causal property of an ecological situation often lacks controls, and this major concern may prevent the replication and the reliability of behavioral observations. We discuss the epistemological and historical grounds of the induction process for behavioral and cognitive neurosciences and provide some possible heuristics for In situ experimental designs compatible with psychophysics in the wild.

A Versatile In-Ear Biosensing System and Body-Area Network for Unobtrusive Continuous Health Monitoring

To enable continuous, mobile health monitoring, body-worn sensors need to offer comparable performance to clinical devices in a lightweight, unobtrusive package. This work presents a complete versatile wireless electrophysiology data acquisition system (weDAQ) that is demonstrated for in-ear electroencephalography (EEG) and other on-body electrophysiology with user-generic dry-contact electrodes made from standard printed circuit boards (PCBs). Each weDAQ device provides 16 recording channels, driven right leg (DRL), a 3-axis accelerometer, local data storage, and adaptable data transmission modes. The weDAQ wireless interface supports deployment of a body area network (BAN) capable of aggregating various biosignal streams over multiple worn devices simultaneously, on the 802.11n WiFi protocol. Each channel resolves biopotentials ranging over 5 orders of magnitude with a noise level of 0.52 μVrms over a 1000-Hz bandwidth, and a peak SNDR of 119 dB and CMRR of 111 dB at 2 ksps. The device leverages in-band impedance scanning and an input multiplexer to dynamically select good skin contacting electrodes for reference and sensing channels. In-ear and forehead EEG measurements taken from subjects captured modulation of alpha brain activity, electrooculogram (EOG) characteristic eye movements, and electromyogram (EMG) from jaw muscles. Simultaneous ECG and EMG measurements were demonstrated on multiple, freely-moving subjects in their natural office environment during periods of rest and exercise. The small footprint, performance, and configurability of the open-source weDAQ platform and scalable PCB electrodes presented, aim to provide the biosensing community greater experimental flexibility and lower the barrier to entry for new health monitoring research.

Timing errors and temporal uncertainty in clinical databases-A narrative review

A firm concept of time is essential for establishing causality in a clinical setting. Review of critical incidents and generation of study hypotheses require a robust understanding of the sequence of events but conducting such work can be problematic when timestamps are recorded by independent and unsynchronized clocks. Most clinical models implicitly assume that timestamps have been measured accurately and precisely, but this custom will need to be re-evaluated if our algorithms and models are to make meaningful use of higher frequency physiological data sources. In this narrative review we explore factors that can result in timestamps being erroneously recorded in a clinical setting, with particular focus on systems that may be present in a critical care unit. We discuss how clocks, medical devices, data storage systems, algorithmic effects, human factors, and other external systems may affect the accuracy and precision of recorded timestamps. The concept of temporal uncertainty is introduced, and a holistic approach to timing accuracy, precision, and uncertainty is proposed. This quantitative approach to modeling temporal uncertainty provides a basis to achieve enhanced model generalizability and improved analytical outcomes.

Efficient on-site confirmatory testing for atrial fibrillation with derived 12-lead ECG in a wireless body area network

Smartphones that can support and assist the screening of various cardiovascular diseases are gaining popularity in recent years. The timely detection, diagnosis, and treatment of atrial fibrillation (AF) are critical, especially for those who are at risk of stroke. AF detection via screening with wearable devices should always be confirmed by a standard 12-lead electrocardiogram (ECG). However, the inability to perform on-site AF confirmatory testing results in increased patient anxiety, followed by unnecessary diagnostic procedures and treatments. Also, the delay in confirmation procedure may conclude the condition as non-AF while it was indeed present at the time of screening. To overcome these challenges, we propose an efficient on-site confirmatory testing for AF with 12-lead ECG derived from the reduced lead set (RLS) in a wireless body area network (WBAN) environment. The reduction in the number of leads enhances the comfort level of patients as well as minimizes the hurdles associated with continuous telemonitoring applications such as data transmission, storage, and bandwidth of the overall system. The proposed method is characterized by segment-wise regression and a lead selection algorithm, facilitating improved P-wave reconstruction. Further, an efficient AF detection algorithm is proposed by incorporating a novel three-level P-wave evidence score with an RR irregularity evidence score. The proposed on-site AF confirmation test reduces false positives and false negatives by 88% and 53% respectively, compared to single lead screening. In addition, the proposed lead derivation method improves accuracy, F 1 -score, and Matthews correlation coefficient (MCC) for the on-site AF detection compared to existing related methods.

EEG-Based Brain-Computer Interfaces (BCIs): A Survey of Recent Studies on Signal Sensing Technologies and Computational Intelligence Approaches and Their Applications

Brain-Computer interfaces (BCIs) enhance the capability of human brain activities to interact with the environment. Recent advancements in technology and machine learning algorithms have increased interest in electroencephalographic (EEG)-based BCI applications. EEG-based intelligent BCI systems can facilitate continuous monitoring of fluctuations in human cognitive states under monotonous tasks, which is both beneficial for people in need of healthcare support and general researchers in different domain areas. In this review, we survey the recent literature on EEG signal sensing technologies and computational intelligence approaches in BCI applications, compensating for the gaps in the systematic summary of the past five years. Specifically, we first review the current status of BCI and signal sensing technologies for collecting reliable EEG signals. Then, we demonstrate state-of-the-art computational intelligence techniques, including fuzzy models and transfer learning in machine learning and deep learning algorithms, to detect, monitor, and maintain human cognitive states and task performance in prevalent applications. Finally, we present a couple of innovative BCI-inspired healthcare applications and discuss future research directions in EEG-based BCI research.

A dataset of daily ambulatory psychological and physiological recording for emotion research

To better understand the psychological and physiological basis of human emotion, increasing interest has been drawn towards ambulatory recordings of emotion-related data beyond the laboratories. By employing smartphones-based ambulatory assessment and wrist-worn physiological recording devices, the Daily Ambulatory Psychological and Physiological recording for Emotion Research (DAPPER) dataset provides momentary self-reports and physiological data of people's emotional experiences in their daily life. The dataset consists of ambulatory psychological recordings from 142 participants and physiological recordings from 88 of them over five days. Both the experience sampling method (ESM) and the day reconstruction method (DRM) were employed to have a comprehensive description of the participants' daily emotional experiences. Heart rate, galvanic skin response, and three-axis acceleration were recorded during the day time. By including multiple types of physiological and self-report data at a scale of five days with 100+ participants, the present dataset is expected to promote emotion researches in real-life, daily settings.

The Cognitive-Emotional Design and Study of Architectural Space: A Scoping Review of Neuroarchitecture and Its Precursor Approaches

Humans respond cognitively and emotionally to the built environment. The modern possibility of recording the neural activity of subjects during exposure to environmental situations, using neuroscientific techniques and virtual reality, provides a promising framework for future design and studies of the built environment. The discipline derived is termed "neuroarchitecture". Given neuroarchitecture's transdisciplinary nature, it progresses needs to be reviewed in a contextualised way, together with its precursor approaches. The present article presents a scoping review, which maps out the broad areas on which the new discipline is based. The limitations, controversies, benefits, impact on the professional sectors involved, and potential of neuroarchitecture and its precursors' approaches are critically addressed.

The AudioMaze: An EEG and motion capture study of human spatial navigation in sparse augmented reality

Spatial navigation is one of the fundamental cognitive functions central to survival in most animals. Studies in humans investigating the neural foundations of spatial navigation traditionally use stationary, desk-top protocols revealing the hippocampus, parahippocampal place area (PPA), and retrosplenial complex to be involved in navigation. However, brain dynamics, while freely navigating the real world remain poorly understood. To address this issue, we developed a novel paradigm, the AudioMaze, in which participants freely explore a room-sized virtual maze, while EEG is recorded synchronized to motion capture. Participants (n = 16) were blindfolded and explored different mazes, each in three successive trials, using their right hand as a probe to "feel" for virtual maze walls. When their hand "neared" a virtual wall, they received directional noise feedback. Evidence for spatial learning include shortening of time spent and an increase of movement velocity as the same maze was repeatedly explored. Theta-band EEG power in or near the right lingual gyrus, the posterior portion of the PPA, decreased across trials, potentially reflecting the spatial learning. Effective connectivity analysis revealed directed information flow from the lingual gyrus to the midcingulate cortex, which may indicate an updating process that integrates spatial information with future action. To conclude, we found behavioral evidence of navigational learning in a sparse-AR environment, and a neural correlate of navigational learning was found near the lingual gyrus.

A Predictive Multimodal Framework to Alert Caregivers of Problem Behaviors for Children with ASD (PreMAC)

Autism Spectrum Disorder (ASD) impacts 1 in 54 children in the US. Two-thirds of children with ASD display problem behavior. If a caregiver can predict that a child is likely to engage in problem behavior, they may be able to take action to minimize that risk. Although experts in Applied Behavior Analysis can offer caregivers recognition and remediation strategies, there are limitations to the extent to which human prediction of problem behavior is possible without the assistance of technology. In this paper, we propose a machine learning-based predictive framework, PreMAC, that uses multimodal signals from precursors of problem behaviors to alert caregivers of impending problem behavior for children with ASD. A multimodal data capture platform, M2P3, was designed to collect multimodal training data for PreMAC. The development of PreMAC integrated a rapid functional analysis, the interview-informed synthesized contingency analysis (IISCA), for collection of training data. A feasibility study with seven 4 to 15-year-old children with ASD was conducted to investigate the tolerability and feasibility of the M2P3 platform and the accuracy of PreMAC. Results indicate that the M2P3 platform was well tolerated by the children and PreMAC could predict precursors of problem behaviors with high prediction accuracies.

The Current Research of Combining Multi-Modal Brain-Computer Interfaces With Virtual Reality

Combing brain-computer interfaces (BCI) and virtual reality (VR) is a novel technique in the field of medical rehabilitation and game entertainment. However, the limitations of BCI such as a limited number of action commands and low accuracy hinder the widespread use of BCI-VR. Recent studies have used hybrid BCIs that combine multiple BCI paradigms and/or the multi-modal biosensors to alleviate these issues, which may become the mainstream of BCIs in the future. The main purpose of this review is to discuss the current status of multi-modal BCI-VR. This study first reviewed the development of the BCI-VR, and explored the advantages and disadvantages of incorporating eye tracking, motor capture, and myoelectric sensing into the BCI-VR system. Then, this study discussed the development trend of the multi-modal BCI-VR, hoping to provide a pathway for further research in this field.

Individual differences in trait creativity moderate the state-level mood-creativity relationship

The relationship between mood states and state creativity has long been investigated. Exploring individual differences may provide additional important information to further our understanding of the complex mood-creativity relationship. The present study explored the state-level mood-creativity relationship from the perspective of trait creativity. We employed the experience sampling method (ESM) in a cohort of 56 college students over five consecutive days. The participants reported their state creativity on originality and usefulness dimensions at six random points between 9:00 a.m. and 11:00 p.m., along with a 10-item concurrent mood state report. Their trait creativity was measured by the Guildford Alternative Uses Test (AUT) and the Remote Associates Test (RAT). We found moderating effects of the participants' trait creativity on their state-level mood-creativity relationship. Specifically, whereas the positive correlation between positive mood state and originality of state creativity was stronger for the participants with higher AUT flexibility scores, stronger positive correlations between negative mood state and originality of state creativity were observed for individuals with higher AUT originality scores. Our findings provide evidence in support of introducing individual differences to achieve a more comprehensive understanding of the mood-creativity link. The results could be of practical value, in developing individualized mood state regulation strategies for promoting state creativity.

On Assessing Driver Awareness of Situational Criticalities: Multi-modal Bio-Sensing and Vision-Based Analysis, Evaluations, and Insights

Automobiles for our roadways are increasingly using advanced driver assistance systems. The adoption of such new technologies requires us to develop novel perception systems not only for accurately understanding the situational context of these vehicles, but also to infer the driver's awareness in differentiating between safe and critical situations. This manuscript focuses on the specific problem of inferring driver awareness in the context of attention analysis and hazardous incident activity. Even after the development of wearable and compact multi-modal bio-sensing systems in recent years, their application in driver awareness context has been scarcely explored. The capability of simultaneously recording different kinds of bio-sensing data in addition to traditionally employed computer vision systems provides exciting opportunities to explore the limitations of these sensor modalities. In this work, we explore the applications of three different bio-sensing modalities namely electroencephalogram (EEG), photoplethysmogram (PPG) and galvanic skin response (GSR) along with a camera-based vision system in driver awareness context. We assess the information from these sensors independently and together using both signal processing- and deep learning-based tools. We show that our methods outperform previously reported studies to classify driver attention and detecting hazardous/non-hazardous situations for short time scales of two seconds. We use EEG and vision data for high resolution temporal classification (two seconds) while additionally also employing PPG and GSR over longer time periods. We evaluate our methods by collecting user data on twelve subjects for two real-world driving datasets among which one is publicly available (KITTI dataset) while the other was collected by us (LISA dataset) with the vehicle being driven in an autonomous mode. This work presents an exhaustive evaluation of multiple sensor modalities on two different datasets for attention monitoring and hazardous events classification.

Impact of Affective Multimedia Content on the Electroencephalogram and Facial Expressions

Most of the research in the field of affective computing has focused on detecting and classifying human emotions through electroencephalogram (EEG) or facial expressions. Designing multimedia content to evoke certain emotions has been largely motivated by manual rating provided by users. Here we present insights from the correlation of affective features between three modalities namely, affective multimedia content, EEG, and facial expressions. Interestingly, low-level Audio-visual features such as contrast and homogeneity of the video and tone of the audio in the movie clips are most correlated with changes in facial expressions and EEG. We also detect the regions associated with the human face and the brain (in addition to the EEG frequency bands) that are most representative of affective responses. The computational modeling between the three modalities showed a high correlation between features from these regions and user-reported affective labels. Finally, the correlation between different layers of convolutional neural networks with EEG and Face images as input provides insights into human affection. Together, these findings will assist in (1) designing more effective multimedia contents to engage or influence the viewers, (2) understanding the brain/body bio-markers of affection, and (3) developing newer brain-computer interfaces as well as facial-expression-based algorithms to read emotional responses of the viewers.

Cost-efficient and Custom Electrode-holder Assembly Infrastructure for EEG Recordings

Mobile electroencephalogram (EEG)-sensing technologies have rapidly progressed and made the access of neuroelectrical brain activity outside the laboratory in everyday life more realistic. However, most existing EEG headsets exhibit a fixed design, whereby its immobile montage in terms of electrode density and coverage inevitably poses a great challenge with applicability and generalizability to the fundamental study and application of the brain-computer interface (BCI). In this study, a cost-efficient, custom EEG-electrode holder infrastructure was designed through the assembly of primary components, including the sensor-positioning ring, inter-ring bridge, and bridge shield. It allows a user to (re)assemble a compact holder grid to accommodate a desired number of electrodes only to the regions of interest of the brain and iteratively adapt it to a given head size for optimal electrode-scalp contact and signal quality. This study empirically demonstrated its easy-to-fabricate nature by a low-end fused deposition modeling (FDM) 3D printer and proved its practicability of capturing event-related potential (ERP) and steady-state visual-evoked potential (SSVEP) signatures over 15 subjects. This paper highlights the possibilities for a cost-efficient electrode-holder assembly infrastructure with replaceable montage, flexibly retrofitted in an unlimited fashion, for an individual for distinctive fundamental EEG studies and BCI applications.