Predicting affective valence using cortical hemodynamic signals

A pediatric near-infrared spectroscopy brain-computer interface based on the detection of emotional valence

Brain-computer interfaces (BCIs) are being investigated as an access pathway to communication for individuals with physical disabilities, as the technology obviates the need for voluntary motor control. However, to date, minimal research has investigated the use of BCIs for children. Traditional BCI communication paradigms may be suboptimal given that children with physical disabilities may face delays in cognitive development and acquisition of literacy skills. Instead, in this study we explored emotional state as an alternative access pathway to communication. We developed a pediatric BCI to identify positive and negative emotional states from changes in hemodynamic activity of the prefrontal cortex (PFC). To train and test the BCI, 10 neurotypical children aged 8–14 underwent a series of emotion-induction trials over four experimental sessions (one offline, three online) while their brain activity was measured with functional near-infrared spectroscopy (fNIRS). Visual neurofeedback was used to assist participants in regulating their emotional states and modulating their hemodynamic activity in response to the affective stimuli. Child-specific linear discriminant classifiers were trained on cumulatively available data from previous sessions and adaptively updated throughout each session. Average online valence classification exceeded chance across participants by the last two online sessions (with 7 and 8 of the 10 participants performing better than chance, respectively, in Sessions 3 and 4). There was a small significant positive correlation with online BCI performance and age, suggesting older participants were more successful at regulating their emotional state and/or brain activity. Variability was seen across participants in regards to BCI performance, hemodynamic response, and discriminatory features and channels. Retrospective offline analyses yielded accuracies comparable to those reported in adult affective BCI studies using fNIRS. Affective fNIRS-BCIs appear to be feasible for school-aged children, but to further gauge the practical potential of this type of BCI, replication with more training sessions, larger sample sizes, and end-users with disabilities is necessary.

Affective Neurofeedback Under Naturalistic Conditions: A Mini-Review of Current Achievements and Open Challenges

Affective neurofeedback training allows for the self-regulation of the putative circuits of emotion regulation. This approach has recently been studied as a possible additional treatment for psychiatric disorders, presenting positive effects in symptoms and behaviors. After neurofeedback training, a critical aspect is the transference of the learned self-regulation strategies to outside the laboratory and how to continue reinforcing these strategies in non-controlled environments. In this mini-review, we discuss the current achievements of affective neurofeedback under naturalistic setups. For this, we first provide a brief overview of the state-of-the-art for affective neurofeedback protocols. We then discuss virtual reality as a transitional step toward the final goal of "in-the-wild" protocols and current advances using mobile neurotechnology. Finally, we provide a discussion of open challenges for affective neurofeedback protocols in-the-wild, including topics such as convenience and reliability, environmental effects in attention and workload, among others.

Closed-loop neurostimulation for affective symptoms and disorders: An overview

Affective and anxiety disorders are the most prevalent and incident psychiatric disorders worldwide. Therapeutic approaches to these disorders using non-invasive brain stimulation (NIBS) and analogous techniques have been extensively investigated. In this paper, we discuss the combination of NIBS and neurofeedback in closed-loop setups and its application for affective symptoms and disorders. For this, we first provide a rationale for this combination by presenting some of the main original findings of NIBS, with a primary focus on transcranial magnetic stimulation (TMS), and neurofeedback, including protocols based on electroencephalography (EEG) and functional magnetic resonance imaging (fMRI). Then, we provide a scope review of studies combining real-time neurofeedback with NIBS protocols in the so-called closed-loop brain state-dependent neuromodulation (BSDS). Finally, we discuss the concomitant use of TMS and real-time functional near-infrared spectroscopy (fNIRS) as a possible solution to the current limitations of BSDS-based protocols for affective and anxiety disorders.

Subject-independent decoding of affective states using functional near-infrared spectroscopy

Affective decoding is the inference of human emotional states using brain signal measurements. This approach is crucial to develop new therapeutic approaches for psychiatric rehabilitation, such as affective neurofeedback protocols. To reduce the training duration and optimize the clinical outputs, an ideal clinical neurofeedback could be trained using data from an independent group of volunteers before being used by new patients. Here, we investigated if this subject-independent design of affective decoding can be achieved using functional near-infrared spectroscopy (fNIRS) signals from frontal and occipital areas. For this purpose, a linear discriminant analysis classifier was first trained in a dataset (49 participants, 24.65±3.23 years) and then tested in a completely independent one (20 participants, 24.00±3.92 years). Significant balanced accuracies between classes were found for positive vs. negative (64.50 ± 12.03%, p<0.01) and negative vs. neutral (68.25 ± 12.97%, p<0.01) affective states discrimination during a reactive block consisting in viewing affective-loaded images. For an active block, in which volunteers were instructed to recollect personal affective experiences, significant accuracy was found for positive vs. neutral affect classification (71.25 ± 18.02%, p<0.01). In this last case, only three fNIRS channels were enough to discriminate between neutral and positive affective states. Although more research is needed, for example focusing on better combinations of features and classifiers, our results highlight fNIRS as a possible technique for subject-independent affective decoding, reaching significant classification accuracies of emotional states using only a few but biologically relevant features.

Surface Acting or Deep Acting, Who Need More Effortful? A Study on Emotional Labor Using Functional Near-Infrared Spectroscopy

Emotional labor is characterized by two main regulation strategies: surface acting and deep acting. However, which strategy consumes more energy? To explore this, we used functional near-infrared spectroscopy (fNIRS) to measure changes in hemoglobin density while participants performed a task requiring them to make the opposite emotional facial expression of that presented in a picture. We found that (1) neither surface nor deep acting led to a significant change in hemoglobin concentration in the prefrontal cortex; (2) making negative and positive facial expressions activated the same left front and middle areas of the prefrontal cortex; and (3) making positive facial expressions activated the rear portion of the prefrontal cortex, but making negative facial expressions did not. Based on these findings and past work, we can infer that deep and surface acting may not significantly differ in terms of the activity in the prefrontal cortex energy consumed. Furthermore, engaging in positive and negative emotional labor appear to utilize some of the same neurological mechanisms, although they differ in others.

fNIRS Evidence for Recognizably Different Positive Emotions

The behavioral differentiation of positive emotions has recently been studied in terms of their discrete adaptive functions or appraising profiles. Some preliminary neurophysiological evidences have been found with electroencephalography or autonomic nervous system measurements such as heart rate, skin conductance, etc. However, the brain's hemodynamic responses to different positive emotions remain largely unknown. In the present study, the functional near-infrared spectroscopy (fNIRS) technique was employed. With this tool, we for the first time reported recognizable discrete positive emotions using fNIRS signals. Thirteen participants watched 30 emotional video clips to elicit 10 typical kinds of positive emotions (joy, gratitude, serenity, interest, hope, pride, amusement, inspiration, awe, and love), and their frontal neural activities were simultaneously recorded with a 24-channel fNIRS system. The multidimensional scaling analysis of participants' subjective ratings on these 10 positive emotions revealed three distinct clusters, which could be interpreted as "playfulness" for amusement, joy, interest, "encouragement" for awe, gratitude, hope, inspiration, pride, and "harmony" for love, serenity. Hemodynamic responses to these three positive emotion clusters showed distinct patterns, and HbO-based individual-level binary classifications between them achieved an averaged accuracy of 73.79 ± 11.49% (77.56 ± 7.39% for encouragement vs. harmony, 73.29 ± 11.87% for playfulness vs. harmony, 70.51 ± 13.96% for encouragement vs. harmony). Benefited from fNIRS's high portability, low running cost and the relative robustness against motion and electrical artifacts, our findings provided support for implementing a more fine-grained emotion recognition system with subdivided positive emotion categories.

Functional near-infrared spectroscopy-based affective neurofeedback: feedback effect, illiteracy phenomena, and whole-connectivity profiles

Background: Affective neurofeedback constitutes a suitable approach to control abnormal neural activities associated with psychiatric disorders and might consequently relief symptom severity. However, different aspects of neurofeedback remain unclear, such as its neural basis, the performance variation, the feedback effect, among others. Aim: First, we aimed to propose a functional near-infrared spectroscopy (fNIRS)-based affective neurofeedback based on the self-regulation of frontal and occipital networks. Second, we evaluated three different feedback approaches on performance: real, fixed, and random feedback. Third, we investigated different demographic, psychological, and physiological predictors of performance. Approach: Thirty-three healthy participants performed a task whereby an amorphous figure changed its shape according to the elicited affect (positive or neutral). During the task, the participants randomly received three different feedback approaches: real feedback, with no change of the classifier output; fixed feedback, keeping the feedback figure unmodified; and random feedback, where the classifier output was multiplied by an arbitrary value, causing a feedback different than expected by the subject. Then, we applied a multivariate comparison of the whole-connectivity profiles according to the affective states and feedback approaches, as well as during a pretask resting-state block, to predict performance. Results: Participants were able to control this feedback system with 70.00 % ± 24.43 % ( p < 0.01 ) of performance during the real feedback trials. No significant differences were found when comparing the average performances of the feedback approaches. However, the whole functional connectivity profiles presented significant Mahalanobis distances ( p ≪ 0.001 ) when comparing both affective states and all feedback approaches. Finally, task performance was positively correlated to the pretask resting-state whole functional connectivity ( r = 0.512 , p = 0.009 ). Conclusions: Our results suggest that fNIRS might be a feasible tool to develop a neurofeedback system based on the self-regulation of affective networks. This finding enables future investigations using an fNIRS-based affective neurofeedback in psychiatric populations. Furthermore, functional connectivity profiles proved to be a good predictor of performance and suggested an increased effort to maintain task control in the presence of feedback distractors.