Affective computing in virtual reality: emotion recognition from brain and heartbeat dynamics using wearable sensors

Site Experience Enhancement and Perspective in Cultural Heritage Fruition—A Survey on New Technologies and Methodologies Based on a “Four-Pillars” Approach

This paper deals with innovative fruition modalities of cultural heritage sites. Based on two ongoing experiments, four pillars are considered, that is, User Localization, Multimodal Interaction, User Understanding and Gamification. A survey of the existing literature regarding one or more issues related to the four pillars is proposed. It aims to put in evidence the exploitation of these contributions to cultural heritage. It is discussed how a cultural site can be enriched, extended and transformed into an intelligent multimodal environment in this perspective. This new augmented environment can focus on the visitor, analyze his activity and behavior, and make his experience more satisfying, fulfilling and unique. After an in-depth overview of the existing technologies and methodologies for the fruition of cultural interest sites, the two experiments are described in detail and the authors’ vision of the future is proposed.

The Cognitive-Emotional Design and Study of Architectural Space: A Scoping Review of Neuroarchitecture and Its Precursor Approaches

Humans respond cognitively and emotionally to the built environment. The modern possibility of recording the neural activity of subjects during exposure to environmental situations, using neuroscientific techniques and virtual reality, provides a promising framework for future design and studies of the built environment. The discipline derived is termed "neuroarchitecture". Given neuroarchitecture's transdisciplinary nature, it progresses needs to be reviewed in a contextualised way, together with its precursor approaches. The present article presents a scoping review, which maps out the broad areas on which the new discipline is based. The limitations, controversies, benefits, impact on the professional sectors involved, and potential of neuroarchitecture and its precursors' approaches are critically addressed.

Exploring EEG Characteristics to Identify Emotional Reactions under Videogame Scenarios

In this article we present the study of electroencephalography (EEG) traits for emotion recognition process using a videogame as a stimuli tool, and considering two different kind of information related to emotions: arousal–valence self-assesses answers from participants, and game events that represented positive and negative emotional experiences under the videogame context. We performed a statistical analysis using Spearman’s correlation between the EEG traits and the emotional information. We found that EEG traits had strong correlation with arousal and valence scores; also, common EEG traits with strong correlations, belonged to the theta band of the central channels. Then, we implemented a regression algorithm with feature selection to predict arousal and valence scores using EEG traits. We achieved better result for arousal regression, than for valence regression. EEG traits selected for arousal and valence regression belonged to time domain (standard deviation, complexity, mobility, kurtosis, skewness), and frequency domain (power spectral density—PDS, and differential entropy—DE from theta, alpha, beta, gamma, and all EEG frequency spectrum). Addressing game events, we found that EEG traits related with the theta, alpha and beta band had strong correlations. In addition, distinctive event-related potentials where identified in the presence of both types of game events. Finally, we implemented a classification algorithm to discriminate between positive and negative events using EEG traits to identify emotional information. We obtained good classification performance using only two traits related with frequency domain on the theta band and on the full EEG spectrum.

Ultra-Short Window Length and Feature Importance Analysis for Cognitive Load Detection from Wearable Sensors

Human cognitive capabilities are under constant pressure in the modern information society. Cognitive load detection would be beneficial in several applications of human–computer interaction, including attention management and user interface adaptation. However, current research into accurate and real-time biosignal-based cognitive load detection lacks understanding of the optimal and minimal window length in data segmentation which would allow for more timely, continuous state detection. This study presents a comparative analysis of ultra-short (30 s or less) window lengths in cognitive load detection with a wearable device. Heart rate, heart rate variability, galvanic skin response, and skin temperature features are extracted at six different window lengths and used to train an Extreme Gradient Boosting classifier to detect between cognitive load and rest. A 25 s window showed the highest accury (67.6%), which is similar to earlier studies using the same dataset. Overall, model accuracy tended to decrease as the window length decreased, and lowest performance (60.0%) was observed with a 5 s window. The contribution of different physiological features to the classification performance and the most useful features that react in short windows are also discussed. The analysis provides a promising basis for future real-time applications with wearable sensors.

An Intra-Subject Approach Based on the Application of HMM to Predict Concentration in Educational Contexts from Nonintrusive Physiological Signals in Real-World Situations

Previous research has proven the strong influence of emotions on student engagement and motivation. Therefore, emotion recognition is becoming very relevant in educational scenarios, but there is no standard method for predicting students’ affects. However, physiological signals have been widely used in educational contexts. Some physiological signals have shown a high accuracy in detecting emotions because they reflect spontaneous affect-related information, which is fresh and does not require additional control or interpretation. Most proposed works use measuring equipment for which applicability in real-world scenarios is limited because of its high cost and intrusiveness. To tackle this problem, in this work, we analyse the feasibility of developing low-cost and nonintrusive devices to obtain a high detection accuracy from easy-to-capture signals. By using both inter-subject and intra-subject models, we present an experimental study that aims to explore the potential application of Hidden Markov Models (HMM) to predict the concentration state from 4 commonly used physiological signals, namely heart rate, breath rate, skin conductance and skin temperature. We also study the effect of combining these four signals and analyse their potential use in an educational context in terms of intrusiveness, cost and accuracy. The results show that a high accuracy can be achieved with three of the signals when using HMM-based intra-subject models. However, inter-subject models, which are meant to obtain subject-independent approaches for affect detection, fail at the same task.

Virtual Reality as a Tool to Study the Influence of the Eating Environment on Eating Behavior: A Feasibility Study

In this manuscript, we describe a new approach to study the effect of the eating environment on food intake and eating behavior using virtual reality technology. Fifteen adults consumed pizza rolls in two virtual reality (VR) environments: a restaurant and a table in an empty room. The participants' food intake, eating parameters (e.g., masticatory parameters and eating rate), and their sensory evaluation of the test food was measured. The participants' sense of presence (the feeling of being in the virtual environment) and markers of arousal were also measured. There was no statistical significant difference in food intake or the sensory evaluation of the test food. In the restaurant condition, participants used fewer masticatory cycles before swallowing but there was no effect on eating rate or maximum bite force. Participants experienced a greater sense of presence when they were in the pizza restaurant scene. Moreover, their heart rate and skin temperature were higher in the restaurant condition. This study suggests that VR could be developed as a new tool to study the effect of the eating environment on food intake and eating behavior.

A Review of Extended Reality (XR) Technologies for Manufacturing Training

Recently, the use of extended reality (XR) systems has been on the rise, to tackle various domains such as training, education, safety, etc. With the recent advances in augmented reality (AR), virtual reality (VR) and mixed reality (MR) technologies and ease of availability of high-end, commercially available hardware, the manufacturing industry has seen a rise in the use of advanced XR technologies to train its workforce. While several research publications exist on applications of XR in manufacturing training, a comprehensive review of recent works and applications is lacking to present a clear progress in using such advance technologies. To this end, we present a review of the current state-of-the-art of use of XR technologies in training personnel in the field of manufacturing. First, we put forth the need of XR in manufacturing. We then present several key application domains where XR is being currently applied, notably in maintenance training and in performing assembly task. We also reviewed the applications of XR in other vocational domains and how they can be leveraged in the manufacturing industry. We finally present some current barriers to XR adoption in manufacturing training and highlight the current limitations that should be considered when looking to develop and apply practical applications of XR.

A Review of Hyperscanning and Its Use in Virtual Environments

Hyperscanning is a technique which simultaneously records the neural activity of two or more people. This is done using one of several neuroimaging methods, such as electroencephalography (EEG), functional magnetic resonance imaging (fMRI), and functional near-infrared spectroscopy (fNIRS). The use of hyperscanning has seen a dramatic rise in recent years to monitor social interactions between two or more people. Similarly, there has been an increase in the use of virtual reality (VR) for collaboration, and an increase in the frequency of social interactions being carried out in virtual environments (VE). In light of this, it is important to understand how interactions function within VEs, and how they can be enhanced to improve their quality in a VE. In this paper, we present some of the work that has been undertaken in the field of social neuroscience, with a special emphasis on hyperscanning. We also cover the literature detailing the work that has been carried out in the human–computer interaction domain that addresses remote collaboration. Finally, we present a way forward where these two research domains can be combined to explore how monitoring the neural activity of a group of participants in VE could enhance collaboration among them.

Extended Reality for the Clinical, Affective, and Social Neurosciences

Brain science research often involves the use of low-dimensional tools and stimuli that lack several of the potentially valuable features of everyday activities and interactions. Although this research has provided important information about cognitive, affective, and social processes for both clinical and nonclinical populations, there is growing interest in high-dimensional simulations that extend reality. These high-dimensional simulations involve dynamic stimuli presented serially or concurrently to permit the assessment and training of perceivers' integrative processes over time. Moreover, high-dimensional simulation platforms can contextually restrain interpretations of cues about a target's internal states. Extended reality environments extend assessment and training platforms that balance experimental control with emotionally engaging background narratives aimed at extending the affective experience and social interactions. Herein, we highlight the promise of extended reality platforms for greater ecological validity in the clinical, affective, and social neurosciences.

Feature Selection Model based on EEG Signals for Assessing the Cognitive Workload in Drivers

In recent years, research has focused on generating mechanisms to assess the levels of subjects’ cognitive workload when performing various activities that demand high concentration levels, such as driving a vehicle. These mechanisms have implemented several tools for analyzing the cognitive workload, and electroencephalographic (EEG) signals have been most frequently used due to their high precision. However, one of the main challenges in implementing the EEG signals is finding appropriate information for identifying cognitive states. Here, we present a new feature selection model for pattern recognition using information from EEG signals based on machine learning techniques called GALoRIS. GALoRIS combines Genetic Algorithms and Logistic Regression to create a new fitness function that identifies and selects the critical EEG features that contribute to recognizing high and low cognitive workloads and structures a new dataset capable of optimizing the model’s predictive process. We found that GALoRIS identifies data related to high and low cognitive workloads of subjects while driving a vehicle using information extracted from multiple EEG signals, reducing the original dataset by more than 50% and maximizing the model’s predictive capacity, achieving a precision rate greater than 90%.

Built Environment Evaluation in Virtual Reality Environments—A Cognitive Neuroscience Approach

To date, the predominant tools for the evaluation of built environment quality and impact have been surveys, scorecards, or verbal comments—approaches that rely upon user-reported responses. The goal of this research project is to develop, test, and validate a data-driven approach for built environment quality evaluation/validation based upon measurement of real-time emotional responses to simulated environments. This paper presents an experiment that was conducted by combining an immersive virtual environment (virtual reality) and electroencephalogram (EEG) as a tool to evaluate Pre and Post Purple Line development. More precisely, the objective was to (a) develop a data-driven approach for built environment quality evaluation and (b) understand the correlation between the built environment characters and emotional state. The preliminary validation of the proposed evaluation method identified discrepancies between traditional evaluation results and emotion response indications through EEG signals. The validation and findings have laid a foundation for further investigation of relations between people’s general cognitive and emotional responses in evaluating built environment quality and characters.

EEG-Based Emotion Recognition: A State-of-the-Art Review of Current Trends and Opportunities

Emotions are fundamental for human beings and play an important role in human cognition. Emotion is commonly associated with logical decision making, perception, human interaction, and to a certain extent, human intelligence itself. With the growing interest of the research community towards establishing some meaningful "emotional" interactions between humans and computers, the need for reliable and deployable solutions for the identification of human emotional states is required. Recent developments in using electroencephalography (EEG) for emotion recognition have garnered strong interest from the research community as the latest developments in consumer-grade wearable EEG solutions can provide a cheap, portable, and simple solution for identifying emotions. Since the last comprehensive review was conducted back from the years 2009 to 2016, this paper will update on the current progress of emotion recognition using EEG signals from 2016 to 2019. The focus on this state-of-the-art review focuses on the elements of emotion stimuli type and presentation approach, study size, EEG hardware, machine learning classifiers, and classification approach. From this state-of-the-art review, we suggest several future research opportunities including proposing a different approach in presenting the stimuli in the form of virtual reality (VR). To this end, an additional section devoted specifically to reviewing only VR studies within this research domain is presented as the motivation for this proposed new approach using VR as the stimuli presentation device. This review paper is intended to be useful for the research community working on emotion recognition using EEG signals as well as for those who are venturing into this field of research.

Emotion Recognition in Immersive Virtual Reality: From Statistics to Affective Computing

Emotions play a critical role in our daily lives, so the understanding and recognition of emotional responses is crucial for human research. Affective computing research has mostly used non-immersive two-dimensional (2D) images or videos to elicit emotional states. However, immersive virtual reality, which allows researchers to simulate environments in controlled laboratory conditions with high levels of sense of presence and interactivity, is becoming more popular in emotion research. Moreover, its synergy with implicit measurements and machine-learning techniques has the potential to impact transversely in many research areas, opening new opportunities for the scientific community. This paper presents a systematic review of the emotion recognition research undertaken with physiological and behavioural measures using head-mounted displays as elicitation devices. The results highlight the evolution of the field, give a clear perspective using aggregated analysis, reveal the current open issues and provide guidelines for future research.

Immediate Attention Enhancement and Restoration From Interactive and Immersive Technologies: A Scoping Review

Interactive and immersive technologies such as video games, exergames, and virtual reality are typically regarded as entertainment mediums. They also offer a multitude of health and well-being benefits. They have the capacity to incorporate established well-being techniques (e.g., mindfulness, exercise, and play) and expose users to beneficial environment settings with greater ease, improved access, and a broader appeal. The authors conducted a scoping review to explore whether these technologies could be used to benefit attention in healthy adults, that is, in a regulatory sense such as during periods of cognitive fatigue or attention-critical tasks. Research efforts have typically focused on long-term practice methods for attention enhancement with these technologies. Instead, this review provides the first attempt to unify a broad range of investigations concerned with their immediate impact on attention through state-change mechanisms. This applies the concept of attention state training and a growing evidence base, which suggests that meditative practices, exercise bouts, and nature exposures can provide short-term improvements in attentional performance following brief interactions. A systematic search of MEDLINE, Embase, and PsycINFO databases resulted in 11 peer-reviewed articles (13 experimental trials) each including at least one objective measure of attention directly following the use of an interactive or immersive technology. Most studies involved interactive technologies (i.e., video games and exergames), whereas there were three immersive interventions in the form of virtual reality. The comparisons between baseline and postintervention showed mostly no effect on attention, although there were five cases of improved attention. There were no instances of negative effects on attention. The results are significant considering mounting concerns that technology use could be detrimental for cognitive functioning. The positive effects reported here indicate a need to specify the type of technology in question and bring attention to positive vs. negative technology interactions. Implications for the literature concerning attention state training are discussed considering promising effects of technology exposures geared toward flow state induction. Significant gaps in the literature are identified regarding the implementation of traditional attention state training practices.

Virtual Reality Potentiates Emotion and Task Effects of Alpha/Beta Brain Oscillations

The progress of technology has increased research on neuropsychological emotion and attention with virtual reality (VR). However, direct comparisons between conventional two-dimensional (2D) and VR stimulations are lacking. Thus, the present study compared electroencephalography (EEG) correlates of explicit task and implicit emotional attention between 2D and VR stimulation. Participants (n = 16) viewed angry and neutral faces with equal size and distance in both 2D and VR, while they were asked to count one of the two facial expressions. For the main effects of emotion (angry vs. neutral) and task (target vs. nontarget), established event related potentials (ERP), namely the late positive potential (LPP) and the target P300, were replicated. VR stimulation compared to 2D led to overall bigger ERPs but did not interact with emotion or task effects. In the frequency domain, alpha/beta-activity was larger in VR compared to 2D stimulation already in the baseline period. Of note, while alpha/beta event related desynchronization (ERD) for emotion and task conditions were seen in both VR and 2D stimulation, these effects were significantly stronger in VR than in 2D. These results suggest that enhanced immersion with the stimulus materials enabled by VR technology can potentiate induced brain oscillation effects to implicit emotion and explicit task effects.

Physiological State and Learning Ability of Students in Normal and Virtual Reality Conditions: Complexity-Based Analysis

Background

Education and learning are the most important goals of all universities. For this purpose, lecturers use various tools to grab the attention of students and improve their learning ability. Virtual reality refers to the subjective sensory experience of being immersed in a computer-mediated world, and has recently been implemented in learning environments.

Objective

The aim of this study was to analyze the effect of a virtual reality condition on students’ learning ability and physiological state.

Methods

Students were shown 6 sets of videos (3 videos in a two-dimensional condition and 3 videos in a three-dimensional condition), and their learning ability was analyzed based on a subsequent questionnaire. In addition, we analyzed the reaction of the brain and facial muscles of the students during both the two-dimensional and three-dimensional viewing conditions and used fractal theory to investigate their attention to the videos.

Results

The learning ability of students was increased in the three-dimensional condition compared to that in the two-dimensional condition. In addition, analysis of physiological signals showed that students paid more attention to the three-dimensional videos.

Conclusions

A virtual reality condition has a greater effect on enhancing the learning ability of students. The analytical approach of this study can be further extended to evaluate other physiological signals of subjects in a virtual reality condition.

Virtual Reality Experiments on Emotional Face Recognition Find No Evidence of Mood-Congruent Effects

Mood-congruent effects have been demonstrated many times, but few studies have managed to replicate the effect with natural moods. Additionally, the ecological validity of mood induction and real-time observation deficiency remain unresolved. Using a newly developed, virtual-reality-based eye-tracking technique, the present study conducted real-time observations of mood effects on emotional face recognition with simulated “real-life” pleasant and grisly scenes. In experiment 1, participants performed an emotional face recognition task in both positive and negative virtual reality scenes. The recognition tests and gaze tracking results failed to support mood-congruent effects but did show a mood effect independent of a strong emotional face effect. In experiment 2, participants performed a neutral face recognition task in pleasant and grisly scenes that were matched for arousal levels, and the mood effect disappeared. The results also revealed a robust negativity bias in emotional face recognition, which was found to accompany a mood repair effect.

The Design of Memristive Circuit for Affective Multi-Associative Learning

In this work, a memristive circuit with affective multi-associative learning function is proposed, which mimics the process of human affective formation. It mainly contains three modules: affective associative learning, affective formation, affective expression. The first module is composed of several affective single-associative learning circuits consisting of memristive neurons and synapses. Memristive neuron will be activated and output pulses if its input exceeds the threshold. After it is activated, memristive neuron can automatically return to the inactive state. Memristive synapse can realize learning and forgetting functions based on the signals from pre- and post-neurons. The learning rule is pre-neuron activated lags behind post-neuron for a short time; the forgetting rule is to repeatedly activate pre-neuron after the emotion is learned. The process of learning or forgetting corresponds to facilitating or inhibiting synaptic weight, that is, decreasing or increasing memristance continuously. Different voltage signals applied to memristors and different parameters of memristors would lead to different synaptic weights which indicate different affective association. The second module can convert affective signals to corresponding emotions. The formed emotions can be shown in a face by the third module. The simulation results in PSPICE show that the proposed circuit system can learn, forget and form emotions like human. If the proposed circuit is further used on a humanoid robot platform through further research, the robot will have the ability of affective interaction with human so that it can be effectively used in affective company and other aspects.

Human Emotion Recognition: Review of Sensors and Methods

Automated emotion recognition (AEE) is an important issue in various fields of activities which use human emotional reactions as a signal for marketing, technical equipment, or human-robot interaction. This paper analyzes scientific research and technical papers for sensor use analysis, among various methods implemented or researched. This paper covers a few classes of sensors, using contactless methods as well as contact and skin-penetrating electrodes for human emotion detection and the measurement of their intensity. The results of the analysis performed in this paper present applicable methods for each type of emotion and their intensity and propose their classification. The classification of emotion sensors is presented to reveal area of application and expected outcomes from each method, as well as their limitations. This paper should be relevant for researchers using human emotion evaluation and analysis, when there is a need to choose a proper method for their purposes or to find alternative decisions. Based on the analyzed human emotion recognition sensors and methods, we developed some practical applications for humanizing the Internet of Things (IoT) and affective computing systems.

Emotion Classification Based on Biophysical Signals and Machine Learning Techniques

Emotions constitute an indispensable component of our everyday life. They consist of conscious mental reactions towards objects or situations and are associated with various physiological, behavioral, and cognitive changes. In this paper, we propose a comparative analysis between different machine learning and deep learning techniques, with and without feature selection, for binarily classifying the six basic emotions, namely anger, disgust, fear, joy, sadness, and surprise, into two symmetrical categorical classes (emotion and no emotion), using the physiological recordings and subjective ratings of valence, arousal, and dominance from the DEAP (Dataset for Emotion Analysis using EEG, Physiological and Video Signals) database. The results showed that the maximum classification accuracies for each emotion were: anger: 98.02%, joy:100%, surprise: 96%, disgust: 95%, fear: 90.75%, and sadness: 90.08%. In the case of four emotions (anger, disgust, fear, and sadness), the classification accuracies were higher without feature selection. Our approach to emotion classification has future applicability in the field of affective computing, which includes all the methods used for the automatic assessment of emotions and their applications in healthcare, education, marketing, website personalization, recommender systems, video games, and social media.

Real vs. immersive-virtual emotional experience: Analysis of psycho-physiological patterns in a free exploration of an art museum

Virtual reality is a powerful tool in human behaviour research. However, few studies compare its capacity to evoke the same emotional responses as in real scenarios. This study investigates psycho-physiological patterns evoked during the free exploration of an art museum and the museum virtualized through a 3D immersive virtual environment (IVE). An exploratory study involving 60 participants was performed, recording electroencephalographic and electrocardiographic signals using wearable devices. The real vs. virtual psychological comparison was performed using self-assessment emotional response tests, whereas the physiological comparison was performed through Support Vector Machine algorithms, endowed with an effective feature selection procedure for a set of state-of-the-art metrics quantifying cardiovascular and brain linear and nonlinear dynamics. We included an initial calibration phase, using standardized 2D and 360° emotional stimuli, to increase the accuracy of the model. The self-assessments of the physical and virtual museum support the use of IVEs in emotion research. The 2-class (high/low) system accuracy was 71.52% and 77.08% along the arousal and valence dimension, respectively, in the physical museum, and 75.00% and 71.08% in the virtual museum. The previously presented 360° stimuli contributed to increasing the accuracy in the virtual museum. Also, the real vs. virtual classifier accuracy was 95.27%, using only EEG mean phase coherency features, which demonstrates the high involvement of brain synchronization in emotional virtual reality processes. These findings provide an important contribution at a methodological level and to scientific knowledge, which will effectively guide future emotion elicitation and recognition systems using virtual reality.

Wearable-Based Affect Recognition-A Review

Affect recognition is an interdisciplinary research field bringing together researchers from natural and social sciences. Affect recognition research aims to detect the affective state of a person based on observables, with the goal to, for example, provide reasoning for the person's decision making or to support mental wellbeing (e.g., stress monitoring). Recently, beside of approaches based on audio, visual or text information, solutions relying on wearable sensors as observables, recording mainly physiological and inertial parameters, have received increasing attention. Wearable systems enable an ideal platform for long-term affect recognition applications due to their rich functionality and form factor, while providing valuable insights during everyday life through integrated sensors. However, existing literature surveys lack a comprehensive overview of state-of-the-art research in wearable-based affect recognition. Therefore, the aim of this paper is to provide a broad overview and in-depth understanding of the theoretical background, methods and best practices of wearable affect and stress recognition. Following a summary of different psychological models, we detail the influence of affective states on the human physiology and the sensors commonly employed to measure physiological changes. Then, we outline lab protocols eliciting affective states and provide guidelines for ground truth generation in field studies. We also describe the standard data processing chain and review common approaches related to the preprocessing, feature extraction and classification steps. By providing a comprehensive summary of the state-of-the-art and guidelines to various aspects, we would like to enable other researchers in the field to conduct and evaluate user studies and develop wearable systems.

Self-Powered Inhomogeneous Strain Sensor Enabled Joint Motion and Three-Dimensional Muscle Sensing

Wearable electronics containing different functional sensors with abilities to meet people's daily needs are highly desirable. Here, a stretchable triboelectric nanogenerator (TENG) for wearable electronics is demonstrated. By stacking two layers of silicone rubbers embedded with silver nanowires (AgNWs) and a Ni foam as electrodes, respectively, the fabricated TENG can serve as a new type of sensor that is wearable, stretchable, skin-friendly, noninvasive, and durable. It can convert mechanical deformation into electric signals. Deformation like stretching and extruding of the TENG results in interlayer rubbing because of inhomogeneous strain, producing triboelectric charges that can induce voltage signals in the electrodes in response to the deformation. On the basis of the principle, a joint sensor based on the TENG is demonstrated, which can generate different output voltages according to the bending degrees of the joint. Furthermore, a three-dimensional sensor integrating three TENGs is fabricated to depict the deformations of different muscle areas. The output voltages of the three TENGs can be simultaneously monitored to reflect the deformation degrees of different muscle areas.

Virtual Reality in Marketing: A Framework, Review, and Research Agenda

Marketing scholars and practitioners are showing increasing interest in Extended Reality (XR) technologies (XRs), such as virtual reality (VR), augmented reality (AR), and mixed reality (MR), as very promising technological tools for producing satisfactory consumer experiences that mirror those experienced in physical stores. However, most of the studies published to date lack a certain measure of methodological rigor in their characterization of XR technologies and in the assessment techniques used to characterize the consumer experience, which limits the generalization of the results. We argue that it is necessary to define a rigorous methodological framework for the use of XRs in marketing. This article reviews the literature on XRs in marketing, and provides a conceptual framework to organize this disparate body of work.

Information Dynamics of the Brain, Cardiovascular and Respiratory Network during Different Levels of Mental Stress

In this study, an analysis of brain, cardiovascular and respiratory dynamics was conducted combining information-theoretic measures with the Network Physiology paradigm during different levels of mental stress. Starting from low invasive recordings of electroencephalographic, electrocardiographic, respiratory, and blood volume pulse signals, the dynamical activity of seven physiological systems was probed with one-second time resolution measuring the time series of the δ, θ, α and β brain wave amplitudes, the cardiac period (RR interval), the respiratory amplitude, and the duration of blood pressure wave propagation (pulse arrival time, PAT). Synchronous 5-min windows of these time series, obtained from 18 subjects during resting wakefulness (REST), mental stress induced by mental arithmetic (MA) and sustained attention induced by serious game (SG), were taken to describe the dynamics of the nodes composing the observed physiological network. Network activity and connectivity were then assessed in the framework of information dynamics computing the new information generated by each node, the information dynamically stored in it, and the information transferred to it from the other network nodes. Moreover, the network topology was investigated using directed measures of conditional information transfer and assessing their statistical significance. We found that all network nodes dynamically produce and store significant amounts of information, with the new information being prevalent in the brain systems and the information storage being prevalent in the peripheral systems. The transition from REST to MA was associated with an increase of the new information produced by the respiratory signal time series (RESP), and that from MA to SG with a decrease of the new information produced by PAT. Each network node received a significant amount of information from the other nodes, with the highest amount transferred to RR and the lowest transferred to δ, θ, α and β. The topology of the physiological network underlying such information transfer was node- and state-dependent, with the peripheral subnetwork showing interactions from RR to PAT and between RESP and RR, PAT consistently across states, the brain subnetwork resulting more connected during MA, and the subnetwork of brain–peripheral interactions involving different brain rhythms in the three states and resulting primarily activated during MA. These results have both physiological relevance as regards the interpretation of central and autonomic effects on cardiovascular and respiratory variability, and practical relevance as regards the identification of features useful for the automatic distinction of different mental states.

User-Centered Virtual Reality for Promoting Relaxation: An Innovative Approach

Virtual reality has been used effectively to promote relaxation and reduce stress. It is possible to find two main approaches to achieve such aims across the literature. The first one is focused on generic environments filled with relaxing "narratives" to induce control over one's own body and physiological response, while the second one engages the user in virtual reality-mediated activities to empower his/her own abilities to regulate emotion. The scope of the present contribution is to extend the discourse on VR use to promote relaxation, by proposing a third approach. This would be based on VR with personalized content, based on user research to identify important life events. As a second step, distinctive features of such events may be rendered with symbols, activities or other virtual environments contents. According to literature, it is possible that such an approach would obtain more sophisticated and long-lasting relaxation in users. The present contribution explores this innovative theoretical proposal and its potential applications within future research and interventions.

Virtual Reality as a New Approach for Risk Taking Assessment

Understanding how people behave when facing hazardous situations, how intrinsic and extrinsic factors influence the risk taking (RT) decision making process and to what extent it is possible to modify their reactions externally, are questions that have long interested academics and society in general. In the spheres, among others, of Occupational Safety and Health (OSH), the military, finance and sociology, this topic has multidisciplinary implications because we all constantly face RT situations. Researchers have hitherto assessed RT profiles by conducting questionnaires prior to and after the presentation of stimuli; however, this can lead to the production of biased, non-realistic, RT profiles. This is due to the reflexive nature of choosing an answer in a questionnaire, which is remote from the reactive, emotional and impulsive decision making processes inherent to real, risky situations. One way to address this question is to exploit VR capabilities to generate immersive environments that recreate realistic seeming but simulated hazardous situations. We propose VR as the next-generation tool to study RT processes, taking advantage of the big four families of metrics which can provide objective assessment methods with high ecological validity: the real-world risks approach (high presence VR environments triggering real-world reactions), embodied interactions (more natural interactions eliciting more natural behaviors), stealth assessment (unnoticed real-time assessments offering efficient behavioral metrics) and physiological real-time measurement (physiological signals avoiding subjective bias). Additionally, VR can provide an invaluable tool, after the assessment phase, to train in skills related to RT due to its transferability to real-world situations.

Toward Emotionally Adaptive Virtual Reality for Mental Health Applications

Here, we introduce the design and preliminary validation of a general-purpose architecture for affective-driven procedural content generation in virtual reality (VR) applications in mental health and wellbeing. The architecture supports seven commercial physiological sensing technologies and can be deployed in immersive and non-immersive VR systems. To demonstrate the concept, we developed the "The Emotional Labyrinth," a non-linear scenario in which navigation in a procedurally generated three-dimensional maze is entirely decided by the user, and whose features are dynamically adapted according to a set of emotional states. During navigation, affective states are dynamically represented through pictures, music, and animated visual metaphors chosen to represent and induce affective states. The underlying hypothesis is that exposing users to multimodal representations of their affective states can create a feedback loop that supports emotional self-awareness and fosters more effective emotional regulation strategies. We carried out a first study to, first, assess the effectiveness of the selected metaphors in inducing target emotions, and second, identify relevant psycho-physiological markers of the emotional experience generated by the labyrinth. Results show that the Emotional Labyrinth is overall a pleasant experience in which the proposed procedural content generation can induce distinctive psycho-physiological patterns, generally coherent with the meaning of the metaphors used in the labyrinth design. Furthermore, collected psycho-physiological responses such as electrocardiography, respiration, electrodermal activity, and electromyography are used to generate computational models of users' reported experience. These models enable the future implementation of the closed loop mechanism to adapt the Labyrinth procedurally to the users' affective state.

Inter and Intra-Rater Reliability of Short-Term Measurement of Heart Rate Variability on Rest in Diabetic Type 2 Patients

Heart rate variability (HRV) among other methods can be used to assess diabetic cardiac autonomic neuropathy by cardiac intervals were recorded. However, the amount of error depending on this measurement methodology is unclear. To evaluate the intra- and inter-rater reliability to calculate HRV indices, comparing different times and by different trained examiners in patients with type 2 diabetes mellitus (T2DM). Thirty individuals of both genders, aged between 18 and 45 years, with T2DM. The RR interval (RRi) were recorded during a 10 min period on supine position using a portable heart rate monitor (Polar® S810i model). HRV indices were calculated by the software Kubios® HRV analysis (version 2.2). Linear (Mean RRi; STD RR; Mean HR; rMSSD; RR Tri; TINN LF; HF; total power) and non-linear (SD1; SD2; DFα1; DFα2, ApEn and, SampEn) indices were calculated by two examiners with an interval of one week between them. Substantial to excellent was found for reliability of the intra-examiner, with intraclass correlation coefficient (ICC) values ranging from 0.79 to 0.99, standard error of measurement (SEM) between 0.02 and 123.49 (in percentage: 1.83 and 16.67), and minimum detectable change (MDC) between 0.07 and 342.30. Regarding the inter-examiner reliability, substantial to excellent reliability was found, with ICC values ranging from 0.73 to 0.97, SEM between 0.04 and 178.13 (in percentage: 3.26 and 24.18), and MDC between 0.11 and 493.77. The use of the portable heart rate monitor to measure HRV showed acceptable intra and inter reliability in individuals with T2DM, supporting the use of this method of evaluation in research and clinical practice.