Neuromorphic Computing via Fission-based Broadband Frequency Generation

The performance limitations of traditional computer architectures have led to the rise of brain-inspired hardware, with optical solutions gaining popularity due to the energy efficiency, high speed, and scalability of linear operations. However, the use of optics to emulate the synaptic activity of neurons has remained a challenge since the integration of nonlinear nodes is power-hungry and, thus, hard to scale. Neuromorphic wave computing offers a new paradigm for energy-efficient information processing, building upon transient and passively nonlinear interactions between optical modes in a waveguide. Here, an implementation of this concept is presented using broadband frequency conversion by coherent higher-order soliton fission in a single-mode fiber. It is shown that phase encoding on femtosecond pulses at the input, alongside frequency selection and weighting at the system output, makes transient spectro-temporal system states interpretable and allows for the energy-efficient emulation of various digital neural networks. The experiments in a compact, fully fiber-integrated setup substantiate an anticipated enhancement in computational performance with increasing system nonlinearity. The findings suggest that broadband frequency generation, accessible on-chip and in-fiber with off-the-shelf components, may challenge the traditional approach to node-based brain-inspired hardware design, ultimately leading to energy-efficient, scalable, and dependable computing with minimal optical hardware requirements.

EEG Amplitude Modulation Analysis across Mental Tasks: Towards Improved Active BCIs

Brain-computer interface (BCI) technology has emerged as an influential communication tool with extensive applications across numerous fields, including entertainment, marketing, mental state monitoring, and particularly medical neurorehabilitation. Despite its immense potential, the reliability of BCI systems is challenged by the intricacies of data collection, environmental factors, and noisy interferences, making the interpretation of high-dimensional electroencephalogram (EEG) data a pressing issue. While the current trends in research have leant towards improving classification using deep learning-based models, our study proposes the use of new features based on EEG amplitude modulation (AM) dynamics. Experiments on an active BCI dataset comprised seven mental tasks to show the importance of the proposed features, as well as their complementarity to conventional power spectral features. Through combining the seven mental tasks, 21 binary classification tests were explored. In 17 of these 21 tests, the addition of the proposed features significantly improved classifier performance relative to using power spectral density (PSD) features only. Specifically, the average kappa score for these classifications increased from 0.57 to 0.62 using the combined feature set. An examination of the top-selected features showed the predominance of the AM-based measures, comprising over 77% of the top-ranked features. We conclude this paper with an in-depth analysis of these top-ranked features and discuss their potential for use in neurophysiology.

Quantifying time perception during virtual reality gameplay using a multimodal biosensor-instrumented headset: a feasibility study

We have all experienced the sense of time slowing down when we are bored or speeding up when we are focused, engaged, or excited about a task. In virtual reality (VR), perception of time can be a key aspect related to flow, immersion, engagement, and ultimately, to overall quality of experience. While several studies have explored changes in time perception using questionnaires, limited studies have attempted to characterize them objectively. In this paper, we propose the use of a multimodal biosensor-embedded VR headset capable of measuring electroencephalography (EEG), electrooculography (EOG), electrocardiography (ECG), and head movement data while the user is immersed in a virtual environment. Eight gamers were recruited to play a commercial action game comprised of puzzle-solving tasks and first-person shooting and combat. After gameplay, ratings were given across multiple dimensions, including (1) the perception of time flowing differently than usual and (2) the gamers losing sense of time. Several features were extracted from the biosignals, ranked based on a two-step feature selection procedure, and then mapped to a predicted time perception rating using a Gaussian process regressor. Top features were found to come from the four signal modalities and the two regressors, one for each time perception scale, were shown to achieve results significantly better than chance. An in-depth analysis of the top features is presented with the hope that the insights can be used to inform the design of more engaging and immersive VR experiences.

Automated Multi-Wavelength Quality Assessment of Photoplethysmography Signals Using Modulation Spectrum Shape Features

Photoplethysmography (PPG) is used to measure blood volume changes in the microvascular bed of tissue. Information about these changes along time can be used for estimation of various physiological parameters, such as heart rate variability, arterial stiffness, and blood pressure, to name a few. As a result, PPG has become a popular biological modality and is widely used in wearable health devices. However, accurate measurement of various physiological parameters requires good-quality PPG signals. Therefore, various signal quality indexes (SQIs) for PPG signals have been proposed. These metrics have usually been based on statistical, frequency, and/or template analyses. The modulation spectrogram representation, however, captures the second-order periodicities of a signal and has been shown to provide useful quality cues for electrocardiograms and speech signals. In this work, we propose a new PPG quality metric based on properties of the modulation spectrum. The proposed metric is tested using data collected from subjects while they performed various activity tasks contaminating the PPG signals. Experiments on this multi-wavelength PPG dataset show the combination of proposed and benchmark measures significantly outperforming several benchmark SQIs with improvements of 21.3% BACC (balanced accuracy) for green, 21.6% BACC for red, and 19.0% BACC for infrared wavelengths, respectively, for PPG quality detection tasks. The proposed metrics also generalize for cross-wavelength PPG quality detection tasks.

Enhancing motor imagery detection efficacy using multisensory virtual reality priming

Brain-computer interfaces (BCI) have been developed to allow users to communicate with the external world by translating brain activity into control signals. Motor imagery (MI) has been a popular paradigm in BCI control where the user imagines movements of e.g., their left and right limbs and classifiers are then trained to detect such intent directly from electroencephalography (EEG) signals. For some users, however, it is difficult to elicit patterns in the EEG signal that can be detected with existing features and classifiers. As such, new user control strategies and training paradigms have been highly sought-after to help improve motor imagery performance. Virtual reality (VR) has emerged as one potential tool where improvements in user engagement and level of immersion have shown to improve BCI accuracy. Motor priming in VR, in turn, has shown to further enhance BCI accuracy. In this pilot study, we take the first steps to explore if multisensory VR motor priming, where haptic and olfactory stimuli are present, can improve motor imagery detection efficacy in terms of both improved accuracy and faster detection. Experiments with 10 participants equipped with a biosensor-embedded VR headset, an off-the-shelf scent diffusion device, and a haptic glove with force feedback showed that significant improvements in motor imagery detection could be achieved. Increased activity in the six common spatial pattern filters used were also observed and peak accuracy could be achieved with analysis windows that were 2 s shorter. Combined, the results suggest that multisensory motor priming prior to motor imagery could improve detection efficacy.

Task-specific speech enhancement and data augmentation for improved multimodal emotion recognition under noisy conditions

Automatic emotion recognition (AER) systems are burgeoning and systems based on either audio, video, text, or physiological signals have emerged. Multimodal systems, in turn, have shown to improve overall AER accuracy and to also provide some robustness against artifacts and missing data. Collecting multiple signal modalities, however, can be very intrusive, time consuming, and expensive. Recent advances in deep learning based speech-to-text and natural language processing systems, however, have enabled the development of reliable multimodal systems based on speech and text while only requiring the collection of audio data. Audio data, however, is extremely sensitive to environmental disturbances, such as additive noise, thus faces some challenges when deployed “in the wild.” To overcome this issue, speech enhancement algorithms have been deployed at the input signal level to improve testing accuracy in noisy conditions. Speech enhancement algorithms can come in different flavors and can be optimized for different tasks (e.g., for human perception vs. machine performance). Data augmentation, in turn, has also been deployed at the model level during training time to improve accuracy in noisy testing conditions. In this paper, we explore the combination of task-specific speech enhancement and data augmentation as a strategy to improve overall multimodal emotion recognition in noisy conditions. We show that AER accuracy under noisy conditions can be improved to levels close to those seen in clean conditions. When compared against a system without speech enhancement or data augmentation, an increase in AER accuracy of 40% was seen in a cross-corpus test, thus showing promising results for “in the wild” AER.

Estimating distribution shifts for predicting cross-subject generalization in electroencephalography-based mental workload assessment

Assessment of mental workload in real-world conditions is key to ensuring the performance of workers executing tasks that demand sustained attention. Previous literature has employed electroencephalography (EEG) to this end despite having observed that EEG correlates of mental workload vary across subjects and physical strain, thus making it difficult to devise models capable of simultaneously presenting reliable performance across users. Domain adaptation consists of a set of strategies that aim at allowing for improving machine learning systems performance on unseen data at training time. Such methods, however, might rely on assumptions over the considered data distributions, which typically do not hold for applications of EEG data. Motivated by this observation, in this work we propose a strategy to estimate two types of discrepancies between multiple data distributions, namely marginal and conditional shifts, observed on data collected from different subjects. Besides shedding light on the assumptions that hold for a particular dataset, the estimates of statistical shifts obtained with the proposed approach can be used for investigating other aspects of a machine learning pipeline, such as quantitatively assessing the effectiveness of domain adaptation strategies. In particular, we consider EEG data collected from individuals performing mental tasks while running on a treadmill and pedaling on a stationary bike and explore the effects of different normalization strategies commonly used to mitigate cross-subject variability. We show the effects that different normalization schemes have on statistical shifts and their relationship with the accuracy of mental workload prediction as assessed on unseen participants at training time.

Multimodal resting-state connectivity predicts affective neurofeedback performance

Neurofeedback has been suggested as a potential complementary therapy to different psychiatric disorders. Of interest for this approach is the prediction of individual performance and outcomes. In this study, we applied functional connectivity-based modeling using electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) modalities to (i) investigate whether resting-state connectivity predicts performance during an affective neurofeedback task and (ii) evaluate the extent to which predictive connectivity profiles are correlated across EEG and fNIRS techniques. The fNIRS oxyhemoglobin and deoxyhemoglobin concentrations and the EEG beta and gamma bands modulated by the alpha frequency band (beta-m-alpha and gamma-m-alpha, respectively) recorded over the frontal cortex of healthy subjects were used to estimate functional connectivity from each neuroimaging modality. For each connectivity matrix, relevant edges were selected in a leave-one-subject-out procedure, summed into "connectivity summary scores" (CSS), and submitted as inputs to a support vector regressor (SVR). Then, the performance of the left-out-subject was predicted using the trained SVR model. Linear relationships between the CSS across both modalities were evaluated using Pearson's correlation. The predictive model showed a mean absolute error smaller than 20%, and the fNIRS oxyhemoglobin CSS was significantly correlated with the EEG gamma-m-alpha CSS (r = -0.456, p = 0.030). These results support that pre-task electrophysiological and hemodynamic resting-state connectivity are potential predictors of neurofeedback performance and are meaningfully coupled. This investigation motivates the use of joint EEG-fNIRS connectivity as outcome predictors, as well as a tool for functional connectivity coupling investigation.

Cross-Language Speech Emotion Recognition Using Bag-of-Word Representations, Domain Adaptation, and Data Augmentation

To date, several methods have been explored for the challenging task of cross-language speech emotion recognition, including the bag-of-words (BoW) methodology for feature processing, domain adaptation for feature distribution "normalization", and data augmentation to make machine learning algorithms more robust across testing conditions. Their combined use, however, has yet to be explored. In this paper, we aim to fill this gap and compare the benefits achieved by combining different domain adaptation strategies with the BoW method, as well as with data augmentation. Moreover, while domain adaptation strategies, such as the correlation alignment (CORAL) method, require knowledge of the test data language, we propose a variant that we term N-CORAL, in which test languages (in our case, Chinese) are mapped to a common distribution in an unsupervised manner. Experiments with German, French, and Hungarian language datasets were performed, and the proposed N-CORAL method, combined with BoW and data augmentation, was shown to achieve the best arousal and valence prediction accuracy, highlighting the usefulness of the proposed method for "in the wild" speech emotion recognition. In fact, N-CORAL combined with BoW was shown to provide robustness across languages, whereas data augmentation provided additional robustness against cross-corpus nuance factors.

Modulation Spectral Signal Representation for Quality Measurement and Enhancement of Wearable Device Data: A Technical Note

Wearable devices are burgeoning, and applications across numerous verticals are emerging, including human performance monitoring, at-home patient monitoring, and health tracking, to name a few. Off-the-shelf wearables have been developed with focus on portability, usability, and low-cost. As such, when deployed in highly ecological settings, wearable data can be corrupted by artifacts and by missing data, thus severely hampering performance. In this technical note, we overview a signal processing representation called the modulation spectrum. The representation quantifies the rate-of-change of different spectral magnitude components and is shown to separate signal from noise, thus allowing for improved quality measurement, quality enhancement, and noise-robust feature extraction, as well as for disease characterization. We provide an overview of numerous applications developed by the authors over the last decade spanning different wearable modalities and list the results obtained from experimental results alongside comparisons with various state-of-the-art benchmark methods. Open-source software is showcased with the hope that new applications can be developed. We conclude with a discussion on possible future research directions, such as context awareness, signal compression, and improved input representations for deep learning algorithms.

Immersive media experience: a survey of existing methods and tools for human influential factors assessment

Virtual reality (VR) applications, especially those where the user is untethered to a computer, are becoming more prevalent as new hardware is developed, computational power and artificial intelligence algorithms are available, and wireless communication networks are becoming more reliable, fast, and providing higher reliability. In fact, recent projections show that by 2022 the number of VR users will double, suggesting the sector was not negatively affected by the worldwide COVID-19 pandemic. The success of any immersive communication system is heavily dependent on the user experience it delivers, thus now more than ever has it become crucial to develop reliable models of immersive media experience (IMEx). In this paper, we survey the literature for existing methods and tools to assess human influential factors (HIFs) related to IMEx. In particular, subjective, behavioural, and psycho-physiological methods are covered. We describe tools available to monitor these HIFs, including the user's sense of presence and immersion, cybersickness, and mental/affective states, as well as their role in overall experience. Special focus is placed on psycho-physiological methods, as it was found that such in-depth evaluation was lacking from the existing literature. We conclude by touching on emerging applications involving multiple-sensorial immersive media and provide suggestions for future research directions to fill existing gaps. It is hoped that this survey will be useful for researchers interested in building new immersive (adaptive) applications that maximize user experience.

Predicting Driver Stress Levels with a Sensor-Equipped Steering Wheel and a Quality-Aware Heart Rate Measurement Algorithm

Unobtrusive monitoring of driver mental states has been regarded as an important element in improving the safety of existing transportation systems. While many solutions exist relying on camera-based systems for e.g., drowsiness detection, these can be sensitive to varying lighting conditions and to driver facial accessories, such as eye/sunglasses. In this work, we evaluate the use of physiological signals derived from sensors embedded directly into the steering wheel. In particular, we are interested in monitoring driver stress levels. To achieve this goal, we first propose a modulation spectral signal representation to reliably extract electrocardiogram (ECG) signals from the steering wheel sensors, thus allowing for heart rate and heart rate variability features to be computed. When input to a simple logistic regression classifier, we show that up to 72% accuracy can be achieved when discriminating between stressful and non-stressful driving conditions. In particular, the proposed modulation spectral signal representation allows for direct quality assessment of the obtained heart rate information, thus can provide additional intelligence to autonomous driver monitoring systems.

Remote COPD Severity and Exacerbation Detection Using Heart Rate and Activity Data Measured from a Wearable Device

Chronic obstructive pulmonary disease (COPD) is one of the leading causes of human mortality worldwide. Traditionally, estimating COPD severity has been done in controlled clinical conditions using cough sounds, respiration, and heart rate variability, with the latter reporting insights on the autonomic dysfunction caused by the disease. Advancements in remote monitoring and wearable device technologies, in turn, have allowed for remote COPD monitoring in daily life conditions. In this study, we explore the potential for predicting COPD severity and exacerbation using a low-cost wearable device that measures heart rate and activity data. We collected smartwatch sensor data from 35 COPD patients over a period of three months. Our evaluation shows that future trajectory of the disease can be predicted using only the first few days of continuous unobtrusive wearable data collected from COPD patients. Using features extracted from wearable device an Isolation Forest was able to predict exacerbation with an area under curve (AUC) 0.69 thus showing improvement over a random choice classifier.

Multimodal Prediction of Alzheimer's Disease Severity Level Based on Resting-State EEG and Structural MRI

While several biomarkers have been developed for the detection of Alzheimer's disease (AD), not many are available for the prediction of disease severity, particularly for patients in the mild stages of AD. In this paper, we explore the multimodal prediction of Mini-Mental State Examination (MMSE) scores using resting-state electroencephalography (EEG) and structural magnetic resonance imaging (MRI) scans. Analyses were carried out on a dataset comprised of EEG and MRI data collected from 89 patients diagnosed with minimal-mild AD. Three feature selection algorithms were assessed alongside four machine learning algorithms. Results showed that while MRI features alone outperformed EEG features, when both modalities were combined, improved results were achieved. The top-selected EEG features conveyed information about amplitude modulation rate-of-change, whereas top-MRI features comprised information about cortical area and white matter volume. Overall, a root mean square error between predicted MMSE values and true MMSE scores of 1.682 was achieved with a multimodal system and a random forest regression model.

Neurofeedback and the Aging Brain: A Systematic Review of Training Protocols for Dementia and Mild Cognitive Impairment

Dementia describes a set of symptoms that occur in neurodegenerative disorders and that is characterized by gradual loss of cognitive and behavioral functions. Recently, non-invasive neurofeedback training has been explored as a potential complementary treatment for patients suffering from dementia or mild cognitive impairment. Here we systematically reviewed studies that explored neurofeedback training protocols based on electroencephalography or functional magnetic resonance imaging for these groups of patients. From a total of 1,912 screened studies, 10 were included in our final sample (N = 208 independent participants in experimental and N = 81 in the control groups completing the primary endpoint). We compared the clinical efficacy across studies, and evaluated their experimental designs and reporting quality. In most studies, patients showed improved scores in different cognitive tests. However, data from randomized controlled trials remains scarce, and clinical evidence based on standardized metrics is still inconclusive. In light of recent meta-research developments in the neurofeedback field and beyond, quality and reporting practices of individual studies are reviewed. We conclude with recommendations on best practices for future studies that investigate the effects of neurofeedback training in dementia and cognitive impairment.

Affective Neurofeedback Under Naturalistic Conditions: A Mini-Review of Current Achievements and Open Challenges

Affective neurofeedback training allows for the self-regulation of the putative circuits of emotion regulation. This approach has recently been studied as a possible additional treatment for psychiatric disorders, presenting positive effects in symptoms and behaviors. After neurofeedback training, a critical aspect is the transference of the learned self-regulation strategies to outside the laboratory and how to continue reinforcing these strategies in non-controlled environments. In this mini-review, we discuss the current achievements of affective neurofeedback under naturalistic setups. For this, we first provide a brief overview of the state-of-the-art for affective neurofeedback protocols. We then discuss virtual reality as a transitional step toward the final goal of "in-the-wild" protocols and current advances using mobile neurotechnology. Finally, we provide a discussion of open challenges for affective neurofeedback protocols in-the-wild, including topics such as convenience and reliability, environmental effects in attention and workload, among others.

Adaptive Filtering for Improved EEG-Based Mental Workload Assessment of Ambulant Users

Recently, due to the emergence of mobile electroencephalography (EEG) devices, assessment of mental workload in highly ecological settings has gained popularity. In such settings, however, motion and other common artifacts have been shown to severely hamper signal quality and to degrade mental workload assessment performance. Here, we show that classical EEG enhancement algorithms, conventionally developed to remove ocular and muscle artifacts, are not optimal in settings where participant movement (e.g., walking or running) is expected. As such, an adaptive filter is proposed that relies on an accelerometer-based referential signal. We show that when combined with classical algorithms, accurate mental workload assessment is achieved. To test the proposed algorithm, data from 48 participants was collected as they performed the Revised Multi-Attribute Task Battery-II (MATB-II) under a low and a high workload setting, either while walking/jogging on a treadmill, or using a stationary exercise bicycle. Accuracy as high as 95% could be achieved with a random forest based mental workload classifier with ambulant users. Moreover, an increase in gamma activity was found in the parietal cortex, suggesting a connection between sensorimotor integration, attention, and workload in ambulant users.

PASS: A Multimodal Database of Physical Activity and Stress for Mobile Passive Body/ Brain-Computer Interface Research

With the burgeoning of wearable devices and passive body/brain-computer interfaces (B/BCIs), automated stress monitoring in everyday settings has gained significant attention recently, with applications ranging from serious games to clinical monitoring. With mobile users, however, challenges arise due to other overlapping (and potentially confounding) physiological responses (e.g., due to physical activity) that may mask the effects of stress, as well as movement artifacts that can be introduced in the measured signals. For example, the classical increase in heart rate can no longer be attributed solely to stress and could be caused by the activity itself. This makes the development of mobile passive B/BCIs challenging. In this paper, we introduce PASS, a multimodal database of Physical Activity and StresS collected from 48 participants. Participants performed tasks of varying stress levels at three different activity levels and provided quantitative ratings of their perceived stress and fatigue levels. To manipulate stress, two video games (i.e., a calm exploration game and a survival game) were used. Peripheral physical activity (electrocardiography, electrodermal activity, breathing, skin temperature) as well as cerebral activity (electroencephalography) were measured throughout the experiment. A complete description of the experimental protocol is provided and preliminary analyses are performed to investigate the physiological reactions to stress in the presence of physical activity. The PASS database, including raw data and subjective ratings has been made available to the research community at http://musaelab.ca/pass-database/. It is hoped that this database will help advance mobile passive B/BCIs for use in everyday settings.

WAUC: A Multi-Modal Database for Mental Workload Assessment Under Physical Activity

Assessment of mental workload is crucial for applications that require sustained attention and where conditions such as mental fatigue and drowsiness must be avoided. Previous work that attempted to devise objective methods to model mental workload were mainly based on neurological or physiological data collected when the participants performed tasks that did not involve physical activity. While such models may be useful for scenarios that involve static operators, they may not apply in real-world situations where operators are performing tasks under varying levels of physical activity, such as those faced by first responders, firefighters, and police officers. Here, we describe WAUC, a multimodal database of mental Workload Assessment Under physical aCtivity. The study involved 48 participants who performed the NASA Revised Multi-Attribute Task Battery II under three different activity level conditions. Physical activity was manipulated by changing the speed of a stationary bike or a treadmill. During data collection, six neural and physiological modalities were recorded, namely: electroencephalography, electrocardiography, breathing rate, skin temperature, galvanic skin response, and blood volume pulse, in addition to 3-axis accelerometry. Moreover, participants were asked to answer the NASA Task Load Index questionnaire after each experimental section, as well as rate their physical fatigue level on the Borg fatigue scale. In order to bring our experimental setup closer to real-world situations, all signals were monitored using wearable, off-the-shelf devices. In this paper, we describe the adopted experimental protocol, as well as validate the subjective, neural, and physiological data collected. The WAUC database, including the raw data and features, subjective ratings, and scripts to reproduce the experiments reported herein will be made available at: http://musaelab.ca/resources/.

Virtual reality and non-invasive brain stimulation for rehabilitation applications: a systematic review

The present article reports the results of a systematic review on the potential benefits of the combined use of virtual reality (VR) and non-invasive brain stimulation (NIBS) as a novel approach for rehabilitation. VR and NIBS are two rehabilitation techniques that have been consistently explored by health professionals, and in recent years there is strong evidence of the therapeutic benefits of their combined use. In this work, we reviewed research articles that report the combined use of VR and two common NIBS techniques, namely transcranial direct current stimulation (tDCS) and transcranial magnetic stimulation (TMS). Relevant queries to six major bibliographic databases were performed to retrieve original research articles that reported the use of the combination VR-NIBS for rehabilitation applications. A total of 16 articles were identified and reviewed. The reviewed studies have significant differences in the goals, materials, methods, and outcomes. These differences are likely caused by the lack of guidelines and best practices on how to combine VR and NIBS techniques. Five therapeutic applications were identified: stroke, neuropathic pain, cerebral palsy, phobia and post-traumatic stress disorder, and multiple sclerosis rehabilitation. The majority of the reviewed studies reported positive effects of the use of VR-NIBS. However, further research is still needed to validate existing results on larger sample sizes and across different clinical conditions. For these reasons, in this review recommendations for future studies exploring the combined use of VR and NIBS are presented to facilitate the comparison among works.

TILES-2018, a longitudinal physiologic and behavioral data set of hospital workers

We present a novel longitudinal multimodal corpus of physiological and behavioral data collected from direct clinical providers in a hospital workplace. We designed the study to investigate the use of off-the-shelf wearable and environmental sensors to understand individual-specific constructs such as job performance, interpersonal interaction, and well-being of hospital workers over time in their natural day-to-day job settings. We collected behavioral and physiological data from n = 212 participants through Internet-of-Things Bluetooth data hubs, wearable sensors (including a wristband, a biometrics-tracking garment, a smartphone, and an audio-feature recorder), together with a battery of surveys to assess personality traits, behavioral states, job performance, and well-being over time. Besides the default use of the data set, we envision several novel research opportunities and potential applications, including multi-modal and multi-task behavioral modeling, authentication through biometrics, and privacy-aware and privacy-preserving machine learning.

Measurement(s)	Overall Sleep Quality Rating • Step Unit of Distance • Speech • Mean Heart Rate • Proximity • Electrocardiogram Sequence • heart rate variability measurement • Respiratory Rate • physical activity measurement • light • door motion • Changes in Ambient Temperature in Medical Device Environment • humidity • Overall Emotional Well-Being • Stress • psychological flexibility • work-related acceptance • work engagement • psychological capital • intelligence • job performance • organizational citizenship behavior • counter-productive work behavior • personality trait measurement • Negative affectivity • positive affectivity • anxiety-related behavior trait • Alcohol Use History • Overall Health Rating During Past Week
Technology Type(s)	photoplethysmography • Accelerometer • Microphone Device • Bluetooth-enabled Activity Monitor • electrocardiogram • Sensor Device • Photodetector Device • Temperature Sensor Device • questionnaire • Multidimensional Psychological Flexibility Inventory (MPFI) • Utrecht work engagement scale • survey method • individual task proficiency • Search Results Web results Organizational Citizenship Behavior Checklist • big five inventory • Positive and Negative Affect Schedule (PANAS-X) • State-Trait Anxiety Inventory
Sample Characteristic - Organism	Homo sapiens
Sample Characteristic - Environment	hospital

Machine-accessible metadata file describing the reported data: 10.6084/m9.figshare.12465101

Optimal filter characterization for photoplethysmography-based pulse rate and pulse power spectrum estimation

Photoplethysmography (PPG) is a non-invasive, low-cost optical technique used to assess the cardiovascular system. In recent years, PPG-based heart rate measurement has gained significant attention due to its popularity in wearable devices, as well as its practicality relative to electrocardiography (ECG). Studies comparing the dynamics of ECG- and PPG-based heart rate measures have found small differences between these two modalities; differences related to the physiological processes behind each technique. In this work, we analyzed the spectral coherence and the signal-to-noise ratio between isolated PPG pulses and the raw PPG signal in order to: (i) determine the optimal filter to enhance pulse detection from raw PPG for improved heart rate estimation, and (ii) characterize the spectral content of the PPG pulse. The proposed methods were evaluated on 27000 pulses from a PPG database acquired from 42 participants (adults and children). The results showed that the optimal bandpass filter to enhance PPG from the adult group was 0.6-3.3 Hz, while for the children group it was 1.0-2.7 Hz. The spectral analysis on the pulse signal showed that similar bandwidths were found for the adult (0.8-2.4 Hz) and children (0.9-2.7 Hz) groups. We hope that the results presented herein serve as a baseline for pulse detection algorithms and assist with the development of more sophisticated PPG processing algorithms.

Prediction of Stress and Mental Workload during Police Academy Training Using Ultra-Short-Term Heart Rate Variability and Breathing Analysis

Heart rate variability (HRV) has been studied in the context of human behavior analysis and many features have been extracted from the inter-beat interval (RR) time series and tested as correlates of constructs such as mental workload, stress and anxiety. Such constructs are crucial in assessing quality-of-life of individuals, as well as their overall performance when doing critical tasks. Most studies, however, have been conducted in controlled laboratory environments with artificially-induced psychological responses. While this assures that high quality data are collected, the amount of data is limited and the transferability of the findings to more ecologically-appropriate settings remains unknown. Additionally, it is desirable for such mental state monitoring systems to have high temporal resolution, thus allowing for quick feedback and adaptive decision making. In this article, we explore the use of features computed from time windows much shorter than typically reported in the literature. More specifically, we evaluate the potential of HRV and breathing features computed over so-called ultra-short-term segments (i.e., < 5 minutes) for stress and mental workload prediction. Experiments with 27 police academy trainees show that short time windows as low as 60 seconds can provide useful insights, in particular for mental workload assessment. Moreover, the fusion of HRV and breathing features showed to be an important aspect for reliable behavioural assessment in highly ecological settings.

EEG spectro-temporal amplitude modulation as a measurement of cortical hemodynamics: an EEG-fNIRS study

Neurovascular coupling provides valuable descriptive information about neural function and communication. In this work, we propose to objectively characterize EEG sub-band modulation in an attempt to compare with local variations of fNIRS hemoglobin concentration. First, full-band EEG signals are decomposed into five well-known frequency sub-bands: delta, theta, alpha, beta, and gamma. The temporal amplitude envelope of each sub-band is then computed via Hilbert transformation. The proposed EEG 'spectro-temporal amplitude modulation' (EEG-AM) feature measures the rate at which each sub-band is modulated. Similarities between EEG-AM features and fNIRS hemoglobin concentration are computed for four neighboring channels over the occipital area during resting-state. Experiments with a database of 29 participants show statistically significant similarities between the total hemoglobin concentration and the alpha band modulating the alpha, beta, and gamma frequencies. These results support the idea that the EEG-AM can carry hemodynamic properties.Clinical relevance- This shows that the EEG spectro-temporal amplitude modulation present similarities with the hemoglobin concentration in co-placed channels.

Stress and Anxiety Measurement "In-the-Wild" Using Quality-aware Multi-scale HRV Features

Heart rate variability (HRV) has been studied in the context of human behavior analysis and many features have been extracted from the inter-beat interval (RR) time series and tested as correlates of constructs such as mental workload, stress and anxiety. Most studies, however, have been conducted in controlled laboratory environments with artificially-induced psychological responses. While this assures that high quality data are collected, the amount of data is limited and the transferability of the findings to more ecologically-appropriate settings (i.e., "in-the-wild") remains unknown. In this paper, we explore the use of motif-based multi-scale HRV features to predict anxiety and stress in-the-wild. To further improve their robustness to artifacts, we propose a quality-aware feature aggregation method. The new quality-aware features are tested on a dataset collected using a wearable biometric sensor from over 200 hospital workers (nurses and staff) during their work shifts. Results show improved stress/anxiety measurement over using conventional time- and frequency-domain HRV measures.

A Comparative Study of Stress and Anxiety Estimation in Ecological Settings Using a Smart-shirt and a Smart-bracelet

In recent years, consumer wearable devices focused on health assessment have gained popularity. Of these devices, a large number target monitoring heart rate; a few among them include additional biometrics such as breathing rate, galvanic skin response, and skin temperature. Heart rate, and more specifically, heart rate variability (HRV) measures have proven useful in monitoring user psychological states, such as mental workload, stress and anxiety. Most studies, however, have been conducted in controlled laboratory environments with artificially-induced psychological responses. While these conditions assure high quality in the collected data, the amount of data are limited and the generalization of the findings to more ecologically-appropriate settings remains unknown. To this end, in this paper we compare the accuracy of two wearable devices, namely a smart-shirt measuring electrocardiograms and a smart-bracelet measuring photoplethysmograms. Several HRV features are extracted and tested as correlates of stress and anxiety. Data were collected from 196 participants during their normal work shifts for a period of 10 weeks. The complementarity of the two devices is also explored and the advantages of each method are discussed.

Alzheimer's Disease Diagnosis and Severity Level Detection Based on Electroencephalography Modulation Spectral "Patch" Features

Over the last two decades, electroencephalography (EEG) has emerged as a reliable tool for the diagnosis of cortical disorders such as Alzheimer's disease (AD). Typically, resting-state EEG (rsEEG) signals have been used, and traditional frequency bands (delta, theta, alpha, beta and gamma) have been explored. Recent studies, however, have suggested that non-conventional bands may lead to improved diagnostic performance. In this work, we propose a new type of features derived from the 2-dimensional modulation spectral domain representation of the rsEEG signal in order to characterize the neuromodulatory deficit emergent with AD. The proposed features are computed as the power in specific "patches" or regions of interest in the power modulation spectrogram, which are shown to be highly discriminant of AD severity levels. The proposed features were compared with traditional features used in the rsEEG AD monitoring literature. Results showed the proposed features not only achieving improved performance at discriminating between healthy normal elderly controls (Nold) and AD patients with varying severity levels, but also at monitoring severity levels (i.e., mild AD versus moderate AD). Moreover, the proposed features were shown to outperform traditional rsEEG features. Finally, we validated the biological origin of the proposed features by using source localization and comparing the obtained results with ones reported in the AD literature.

Brain-computer interfaces and personhood: interdisciplinary deliberations on neural technology

OBJECTIVE

Scientists, engineers, and healthcare professionals are currently developing a variety of new devices under the category of brain-computer interfaces (BCIs). Current and future applications are both medical/assistive (e.g. for communication) and non-medical (e.g. for gaming). This array of possibilities has been met with both enthusiasm and ethical concern in various media, with no clear resolution of these conflicting sentiments.

APPROACH

To better understand how BCIs may either harm or help the user, and to investigate whether ethical guidance is required, a meeting entitled 'BCIs and Personhood: A Deliberative Workshop' was held in May 2018.

MAIN RESULTS

We argue that the hopes and fears associated with BCIs can be productively understood in terms of personhood, specifically the impact of BCIs on what it means to be a person and to be recognized as such by others.

SIGNIFICANCE

Our findings suggest that the development of neural technologies raises important questions about the concept of personhood and its role in society. Accordingly, we propose recommendations for BCI development and governance.

Multimodal Human and Environmental Sensing for Longitudinal Behavioral Studies in Naturalistic Settings: Framework for Sensor Selection, Deployment, and Management

Background

Recent advances in mobile technologies for sensing human biosignals are empowering researchers to collect real-world data outside of the laboratory, in natural settings where participants can perform their daily activities with minimal disruption. These new sensing opportunities usher a host of challenges and constraints for both researchers and participants.

Objective

This viewpoint paper aims to provide a comprehensive guide to aid research teams in the selection and management of sensors before beginning and while conducting human behavior studies in the wild. The guide aims to help researchers achieve satisfactory participant compliance and minimize the number of unexpected procedural outcomes.

Methods

This paper presents a collection of challenges, consideration criteria, and potential solutions for enabling researchers to select and manage appropriate sensors for their research studies. It explains a general data collection framework suitable for use with modern consumer sensors, enabling researchers to address many of the described challenges. In addition, it provides a description of the criteria affecting sensor selection, management, and integration that researchers should consider before beginning human behavior studies involving sensors. On the basis of a survey conducted in mid-2018, this paper further illustrates an organized snapshot of consumer-grade human sensing technologies that can be used for human behavior research in natural settings.

Results

The research team applied the collection of methods and criteria to a case study aimed at predicting the well-being of nurses and other staff in a hospital. Average daily compliance for sensor usage measured by the presence of data exceeding half the total possible hours each day was about 65%, yielding over 355,000 hours of usable sensor data across 212 participants. A total of 6 notable unexpected events occurred during the data collection period, all of which had minimal impact on the research project.

Conclusions

The satisfactory compliance rates and minimal impact of unexpected events during the case study suggest that the challenges, criteria, methods, and mitigation strategies presented as a guide for researchers are helpful for sensor selection and management in longitudinal human behavior studies in the wild.

Deep learning-based electroencephalography analysis: a systematic review

CONTEXT

Electroencephalography (EEG) is a complex signal and can require several years of training, as well as advanced signal processing and feature extraction methodologies to be correctly interpreted. Recently, deep learning (DL) has shown great promise in helping make sense of EEG signals due to its capacity to learn good feature representations from raw data. Whether DL truly presents advantages as compared to more traditional EEG processing approaches, however, remains an open question.

OBJECTIVE

In this work, we review 154 papers that apply DL to EEG, published between January 2010 and July 2018, and spanning different application domains such as epilepsy, sleep, brain-computer interfacing, and cognitive and affective monitoring. We extract trends and highlight interesting approaches from this large body of literature in order to inform future research and formulate recommendations.

METHODS

Major databases spanning the fields of science and engineering were queried to identify relevant studies published in scientific journals, conferences, and electronic preprint repositories. Various data items were extracted for each study pertaining to (1) the data, (2) the preprocessing methodology, (3) the DL design choices, (4) the results, and (5) the reproducibility of the experiments. These items were then analyzed one by one to uncover trends.

RESULTS

Our analysis reveals that the amount of EEG data used across studies varies from less than ten minutes to thousands of hours, while the number of samples seen during training by a network varies from a few dozens to several millions, depending on how epochs are extracted. Interestingly, we saw that more than half the studies used publicly available data and that there has also been a clear shift from intra-subject to inter-subject approaches over the last few years. About [Formula: see text] of the studies used convolutional neural networks (CNNs), while [Formula: see text] used recurrent neural networks (RNNs), most often with a total of 3-10 layers. Moreover, almost one-half of the studies trained their models on raw or preprocessed EEG time series. Finally, the median gain in accuracy of DL approaches over traditional baselines was [Formula: see text] across all relevant studies. More importantly, however, we noticed studies often suffer from poor reproducibility: a majority of papers would be hard or impossible to reproduce given the unavailability of their data and code.

SIGNIFICANCE

To help the community progress and share work more effectively, we provide a list of recommendations for future studies and emphasize the need for more reproducible research. We also make our summary table of DL and EEG papers available and invite authors of published work to contribute to it directly. A planned follow-up to this work will be an online public benchmarking portal listing reproducible results.

Multi-Scale Heart Beat Entropy Measures for Mental Workload Assessment of Ambulant Users

Mental workload assessment is crucial in many real life applications which require constant attention and where imbalance of mental workload resources may cause safety hazards. As such, mental workload and its relationship with heart rate variability (HRV) have been well studied in the literature. However, the majority of the developed models have assumed individuals are not ambulant, thus bypassing the issue of movement-related electrocardiography (ECG) artifacts and changing heart beat dynamics due to physical activity. In this work, multi-scale features for mental workload assessment of ambulatory users is explored. ECG data was sampled from users while they performed different types and levels of physical activity while performing the multi-attribute test battery (MATB-II) task at varying difficulty levels. Proposed features are shown to outperform benchmark ones and further exhibit complementarity when used in combination. Indeed, results show gains over the benchmark HRV measures of 24.41% in accuracy and of 27.97% in F1 score can be achieved even at high activity levels.

Systematic Review on Resting-State EEG for Alzheimer's Disease Diagnosis and Progression Assessment

Alzheimer's disease (AD) is a neurodegenerative disorder that accounts for nearly 70% of the more than 46 million dementia cases estimated worldwide. Although there is no cure for AD, early diagnosis and an accurate characterization of the disease progression can improve the quality of life of AD patients and their caregivers. Currently, AD diagnosis is carried out using standardized mental status examinations, which are commonly assisted by expensive neuroimaging scans and invasive laboratory tests, thus rendering the diagnosis time consuming and costly. Notwithstanding, over the last decade, electroencephalography (EEG) has emerged as a noninvasive alternative technique for the study of AD, competing with more expensive neuroimaging tools, such as MRI and PET. This paper reports on the results of a systematic review on the utilization of resting-state EEG signals for AD diagnosis and progression assessment. Recent journal articles obtained from four major bibliographic databases were analyzed. A total of 112 journal articles published from January 2010 to February 2018 were meticulously reviewed, and relevant aspects of these papers were compared across articles to provide a general overview of the research on this noninvasive AD diagnosis technique. Finally, recommendations for future studies with resting-state EEG were presented to improve and facilitate the knowledge transfer among research groups.

Early diagnosis of mild cognitive impairment and Alzheimer's with event-related potentials and event-related desynchronization in N-back working memory tasks

BACKGROUND AND OBJECTIVE

In this study we investigate whether or not event-related potentials (ERP) and/or event-related (de)synchronization (ERD/ERS) can be used to differentiate between 27 healthy elderly (HE), 21 subjects diagnosed with mild cognitive impairment (MCI) and 15 mild Alzheimer's disease (AD) patients.

METHODS

Using 32-channel EEG recordings, we measured ERP responses to a three-level (N-back, N = 0,1,2) visual working memory task. We also performed ERD analysis over the same EEG data, dividing the full-band signal into the well-known delta, theta, alpha, beta and gamma bands. Both ERP and ERD analyses were followed by cluster analysis with correction for multicomparisons whenever significant differences were found between groups.

RESULTS

Regarding ERP (full-band analysis), our findings have shown both patient groups (MCI and AD) with reduced P450 amplitude (compared to HE controls) in the execution of the non-match 1-back task at many scalp electrodes, chiefly at parietal and centro-parietal areas. However, no significant differences were found between MCI and AD in ERP analysis whatever was the task. As for sub-band analyses, ERD/ERS measures revealed that HE subjects elicited consistently greater alpha ERD responses than MCI and AD patients during the 1-back task in the match condition, with all differences located at frontal, central and occipital regions. Moreover, in the non-match condition, it was possible to distinguish between MCI and AD patients when they were performing the 0-back task, with MCI presenting more desynchronization than AD on the theta band at temporal and fronto-temporal areas. In summary, ERD analyses have revealed themselves more valuable than ERP, since they showed significant differences in all three group comparisons: HE vs. MCI, HE vs. AD, and MCI vs. AD.

CONCLUSIONS

Based on these findings, we conclude that ERD responses to working memory (N-back) tasks could be useful not only for early MCI diagnosis or for improved AD diagnosis, but probably also for assessing the likelihood of MCI progression to AD, after further validated by a longitudinal study.

Functional near-infrared spectroscopy-based affective neurofeedback: feedback effect, illiteracy phenomena, and whole-connectivity profiles

Background: Affective neurofeedback constitutes a suitable approach to control abnormal neural activities associated with psychiatric disorders and might consequently relief symptom severity. However, different aspects of neurofeedback remain unclear, such as its neural basis, the performance variation, the feedback effect, among others. Aim: First, we aimed to propose a functional near-infrared spectroscopy (fNIRS)-based affective neurofeedback based on the self-regulation of frontal and occipital networks. Second, we evaluated three different feedback approaches on performance: real, fixed, and random feedback. Third, we investigated different demographic, psychological, and physiological predictors of performance. Approach: Thirty-three healthy participants performed a task whereby an amorphous figure changed its shape according to the elicited affect (positive or neutral). During the task, the participants randomly received three different feedback approaches: real feedback, with no change of the classifier output; fixed feedback, keeping the feedback figure unmodified; and random feedback, where the classifier output was multiplied by an arbitrary value, causing a feedback different than expected by the subject. Then, we applied a multivariate comparison of the whole-connectivity profiles according to the affective states and feedback approaches, as well as during a pretask resting-state block, to predict performance. Results: Participants were able to control this feedback system with 70.00 % ± 24.43 % ( p < 0.01 ) of performance during the real feedback trials. No significant differences were found when comparing the average performances of the feedback approaches. However, the whole functional connectivity profiles presented significant Mahalanobis distances ( p ≪ 0.001 ) when comparing both affective states and all feedback approaches. Finally, task performance was positively correlated to the pretask resting-state whole functional connectivity ( r = 0.512 , p = 0.009 ). Conclusions: Our results suggest that fNIRS might be a feasible tool to develop a neurofeedback system based on the self-regulation of affective networks. This finding enables future investigations using an fNIRS-based affective neurofeedback in psychiatric populations. Furthermore, functional connectivity profiles proved to be a good predictor of performance and suggested an increased effort to maintain task control in the presence of feedback distractors.

Electroencephalography Amplitude Modulation Analysis for Automated Affective Tagging of Music Video Clips

The quantity of music content is rapidly increasing and automated affective tagging of music video clips can enable the development of intelligent retrieval, music recommendation, automatic playlist generators, and music browsing interfaces tuned to the users' current desires, preferences, or affective states. To achieve this goal, the field of affective computing has emerged, in particular the development of so-called affective brain-computer interfaces, which measure the user's affective state directly from measured brain waves using non-invasive tools, such as electroencephalography (EEG). Typically, conventional features extracted from the EEG signal have been used, such as frequency subband powers and/or inter-hemispheric power asymmetry indices. More recently, the coupling between EEG and peripheral physiological signals, such as the galvanic skin response (GSR), have also been proposed. Here, we show the importance of EEG amplitude modulations and propose several new features that measure the amplitude-amplitude cross-frequency coupling per EEG electrode, as well as linear and non-linear connections between multiple electrode pairs. When tested on a publicly available dataset of music video clips tagged with subjective affective ratings, support vector classifiers trained on the proposed features were shown to outperform those trained on conventional benchmark EEG features by as much as 6, 20, 8, and 7% for arousal, valence, dominance and liking, respectively. Moreover, fusion of the proposed features with EEG-GSR coupling features showed to be particularly useful for arousal (feature-level fusion) and liking (decision-level fusion) prediction. Together, these findings show the importance of the proposed features to characterize human affective states during music clip watching.

Modeling Speech Level as a Function of Background Noise Level and Talker-to-Listener Distance for Talkers Wearing Hearing Protection Devices

PURPOSE

Studying the variations in speech levels with changing background noise level and talker-to-listener distance for talkers wearing hearing protection devices (HPDs) can aid in understanding communication in background noise.

METHOD

Speech was recorded using an intra-aural HPD from 12 different talkers at 5 different distances in 3 different noise conditions and 2 quiet conditions.

RESULTS

This article proposes models that can predict the difference in speech level as a function of background noise level and talker-to-listener distance for occluded talkers. The proposed model complements the existing model presented by Pelegrín-García, Smits, Brunskog, and Jeong (2011) and expands on it by taking into account the effects of occlusion and background noise level on changes in speech sound level.

CONCLUSIONS

Three models of the relationship between vocal effort, background noise level, and talker-to-listener distance for talkers wearing HPDs are presented. The model with the best prediction intervals is a talker-dependent model that requires the users' unoccluded speech level at 10 m as a reference. A model describing the relationship between speech level, talker-to-listener distance, and background noise level for occluded talkers could eventually be incorporated with radio protocols to transmit verbal communication only to an intended set of listeners within a given spatial range-this range being dependent on the changes in speech level and background noise level.

Spectro-Temporal Electrocardiogram Analysis for Noise-Robust Heart Rate and Heart Rate Variability Measurement

The last few years has seen a proliferation of wearable electrocardiogram (ECG) devices in the market with applications in fitness tracking, patient monitoring, athletic performance assessment, stress and fatigue detection, and biometrics, to name a few. The majority of these applications rely on the computation of the heart rate (HR) and the so-called heart rate variability (HRV) index via time-, frequency-, or non-linear-domain approaches. Wearable/portable devices, however, are highly susceptible to artifacts, particularly those resultant from movement. These artifacts can hamper HR/HRV measurement, thus pose a serious threat to cardiac monitoring applications. While current solutions rely on ECG enhancement as a pre-processing step prior to HR/HRV calculation, existing artifact removal algorithms still perform poorly under extremely noisy scenarios. To overcome this limitation, we take an alternate approach and propose the use of a spectro-temporal ECG signal representation that we show separates cardiac components from artifacts. More specifically, by quantifying the rate-of-change of ECG spectral components over time, we show that heart rate estimates can be reliably obtained even in extremely noisy signals, thus bypassing the need for ECG enhancement. With such HR measurements in hands, we then propose a new noise-robust HRV index termed MD-HRV (modulation-domain HRV) computed as the standard deviation of the obtained HR values. Experiments with synthetic ECG signals corrupted at various different signal-to-noise levels, as well as recorded noisy signals show the proposed measure outperforming several HRV benchmark parameters computed post wavelet-based enhancement. These findings suggest that the proposed HR measures and derived MD-HRV metric are well-suited for ambulant cardiac monitoring applications, particularly those involving intense movement (e.g., elite athletic training).

In-ear microphone speech quality enhancement via adaptive filtering and artificial bandwidth extension

Bone and tissue conducted speech has been used in noisy environments to provide a relatively high signal-to-noise ratio signal. However, the limited bandwidth of bone and tissue conducted speech degrades the quality of the speech signal. Moreover in very noisy conditions, bandwidth extension of the bone and tissue conducted speech becomes problematic. In this paper, speech generated from bone and tissue conduction captured using an in-ear microphone is enhanced using adaptive filtering and a non-linear bandwidth extension method. Objective and subjective tests are used to evaluate the performance of the proposed techniques. Both evaluations show a statistically significant quality enhancement of the noisy in-ear microphone speech with ρ<0.0001 after denoising and ρ<0.01 after bandwidth extension.