Neural correlate of spatial presence in an arousing and noninteractive virtual reality: an EEG and psychophysiology study

Human Factors at Play: Understanding the Impact of Conditioning on Presence and Reaction Time in Mixed Reality

A prerequisite to improving the presence of a user in mixed reality (MR) is the ability to measure and quantify presence. Traditionally, subjective questionnaires have been used to assess the level of presence. However, recent studies have shown that presence is correlated with objective and systemic human performance measures such as reaction time. These studies analyze the correlation between presence and reaction time when technical factors such as object realism and plausibility of the object's behavior change. However, additional psychological and physiological human factors can also impact presence. It is unclear if presence can be mapped to and correlated with reaction time when human factors such as conditioning are involved. To answer this question, we conducted an exploratory study ($N=60$) where the relationship between presence and reaction time was assessed under three different conditioning scenarios: control, positive, and negative. We demonstrated that human factors impact presence. We found that presence scores and reaction times are significantly correlated (correlation coefficient of -0.64), suggesting that the impact of human factors on reaction time correlates with its effect on presence. In demonstrating that, our study takes another important step toward using objective and systemic measures like reaction time as a presence measure.

The convergence of naturalistic paradigms and cognitive neuroscience methods to investigate memory and its development

Studies that involve lab-based stimuli (e.g., words, pictures) are fundamental in the memory literature. At the same time, there is growing acknowledgment that memory processes assessed in the lab may not be analogous to how memory operates in the real world. Naturalistic paradigms can bridge this gap and over the decades a growing proportion of memory research has involved more naturalistic events. However, there is significant variation in the types of naturalistic studies used to study memory and its development, each with various advantages and limitations. Further, there are notable gaps in how often different types of naturalistic approaches have been combined with cognitive neuroscience methods (e.g., fMRI, EEG) to elucidate the neural processes and substrates involved in memory encoding and retrieval in the real world. Here we summarize and discuss what we identify as progressively more naturalistic methodologies used in the memory literature (movie, virtual reality, staged-events inside and outside of the lab, photo-taking, and naturally occurring event studies). Our goal is to describe each approach's benefits (e.g., naturalistic quality, feasibility), limitations (e.g., viability of neuroimaging method for event encoding versus event retrieval), and discuss possible future directions with each approach. We focus on child studies, when available, but also highlight past adult studies. Although there is a growing body of child memory research, naturalistic approaches combined with cognitive neuroscience methodologies in this domain remain sparse. Overall, this viewpoint article reviews how we can study memory through the lens of developmental cognitive neuroscience, while utilizing naturalistic and real-world events.

High-dimensional Metaverse Platforms and the Virtually Extended Self

The study of cognition has traditionally used low-dimensional measures and stimulus presentations that emphasize laboratory control over high-dimensional (i.e., ecologically valid) tools that reflect the activities and interactions in everyday living. Although controlled experimental presentations in laboratories have enhanced our understanding of cognition for both healthy and clinical cohorts, high dimensionality may extend reality and cognition. High-dimensional Metaverse approaches use extended reality (XR) platforms with dynamic stimulus presentations that couple humans and simulation technologies to extend cognition. The plan for this paper is as follows: The "Extending from low to high-dimensional studies of cognition" section discusses current needs for high-dimensional stimulus presentations that reflect everyday cognitive activities. In the "Algorithmic devices and digital extension of cognition" section, technologies of the extended mind are introduced with the Metaverse as a candidate cognitive process for extension. Next, in the "A neurocognitive framework for understanding technologies of the extended mind" section, a framework and model are proposed for understanding the neural correlates of human technology couplings in terms of automatic algorithmic processes (limbic-ventral striatal loop); reflective cognition (prefrontal-dorsal striatal loop); and algorithmic processing (insular cortex). The algorithmic processes of human-technology interactions can, over time, become an automated and algorithmic coupling of brain and technology. The manuscript ends with a brief summary and discussion of the ways in which the Metaverse can be used for studying how persons respond to high-dimensional stimuli in simulations that approximate real-world activities and interactions.

Neural correlates of virtual reality-based attention training: An fMRI study

THEORETICAL BACKGROUND

Virtual Reality technology is increasingly used in attention rehabilitation for functional training purposes. However, the neural mechanisms by which Virtual Reality can affect attentional functioning are still unclear. The current study's objective is to examine the effects of stereoscopic vs. monoscopic presentation on neural processing during a visual attention task.

METHOD

Thirty-two healthy participants performed a visual attention task in an immersive virtual environment that was displayed via MR-compatible video goggles in an MRI scanner. The paradigm altered between trials that required active engagement with the task and mere observation trials. Furthermore, the form of binocular presentation switched between monoscopic and stereoscopic presentation.

RESULTS

Analyses yielded evidence for increased activation in stereoscopic compared to monoscopic trials in the tertiary visual cortex area V3A as well as elevated activation in the dorsal attention network when engaging in the attention task. An additional ROI analysis of area V3A revealed significantly lower attentional engagement costs in stereoscopic conditions.

DISCUSSION

Results support previous findings suggesting that V3A is involved in binocular depth perception. Furthermore, heightened activation in V3A following stereoscopic presentation seemed to facilitate attentional engagement with the task. Considering that V3A is the origin of the dorso-dorsal, ventro-dorsal, and ventral visual processing pathways, we regard it as a gating area that decides which kind of visual perception is processed.

Alpha-band oscillations and emotion: A review of studies on picture perception

Although alpha-band activity has long been a focus of psychophysiological research, its modulation by emotional value during picture perception has only recently been studied systematically. Here, we review these studies and report that the most consistent alpha oscillatory pattern indexing emotional processing is an enhanced desynchronization (ERD) over posterior sensors when viewing emotional compared with neutral pictures. This enhanced alpha ERD is not specific to unpleasant picture content, as previously proposed for other measures of affective response, but has also been observed for pleasant stimuli. Evidence suggests that this effect is not confined to the alpha band but that it also involves a desynchronization of the lower beta frequencies (8-20 Hz). The emotional modulation of alpha ERD occurs even after massive stimulus repetition and when emotional cues serve as task-irrelevant distractors, consistent with the hypothesis that evaluative processes are mandatory in emotional picture processing. A similar enhanced ERD has been observed for other significant cues (e.g., conditioned aversive stimuli, or in anticipation of a potential threat), suggesting that it reflects cortical excitability associated with the engagement of the motivational systems.

Recovery time from VR sickness due to susceptibility: Objective and quantitative evaluation using electroencephalography

With the increasing use of virtual reality (VR) devices, interest in reducing their negative effects, such as VR sickness, is also increasing. This study used electroencephalography (EEG) to investigate participants' VR sickness recovery time after watching a VR video. We tested 40 participants in advance using a motion sickness susceptibility questionnaire (MSSQ). We classified the participants into two groups (sensitive group/non-sensitive group) depending on their MSSQ scores. We used a simulator sickness questionnaire (SSQ) and EEG to evaluate VR sickness. The SSQ score increased significantly after watching the VR sickness-inducing video (VR video) in both groups (p < 0.001). The recovery time based on the SSQ was 11.3 ± 6.6 min for the sensitive group and 9.1 ± 5.2 min for the non-sensitive group. The difference in recovery time between the two groups was not significant (p > 0.05). EEG results showed that recovery time took an average of 11.5 ± 7.1 min in both groups. The EEG data showed that the delta wave increased significantly across all brain areas (p < 0.01). There was no statistical difference between groups in recovering VR sickness depending on individual characteristics. However, we confirmed that subjective and objective VR recovery required at least 11.5 min. This finding can inform recommendations regarding the VR sickness recovery times.

EEG guided electrical stimulation parameters generation from texture force profiles

Objective.Our aim is to enhance sensory perception and spatial presence in artificial interfaces guided by EEG. This is done by developing a closed-loop electro-tactile system guided by EEG that adaptively update the electrical stimulation parameters to achieve EEG responses similar to the EEG responses generated from touching textured surface.Approach.In this work, we introduce a model that defines the relationship between the contact force profiles and the electrical stimulation parameters. This is done by using the EEG and force data collected from two experiments. The first was conducted by moving a set of textured surfaces against the subjects' fingertip, while collecting both EEG and force data. Whereas the second was carried out by applying a set of different pulse and amplitude modulated electrical stimuli to the subjects' index finger while recording EEG.Main results.We were able to develop a model which could generate electrical stimulation parameters corresponding to different textured surfaces. We showed by offline testing and validation analysis that the average error between the EEG generated from the estimated electrical stimulation parameters and the actual EEG generated from touching textured surfaces is around 7%.Significance.Haptic feedback plays a vital role in our daily life, as it allows us to become aware of our environment. Even though a number of methods have been developed to measure perception of spatial presence and provide sensory feedback in virtual reality environments, there is currently no closed-loop control of sensory stimulation. The proposed model provides an initial step towards developing a closed loop electro-tactile haptic feedback model that delivers more realistic touch sensation through electrical stimulation.

Neurophysiological and Subjective Analysis of VR Emotion Induction Paradigm

The ecological validity of emotion-inducing scenarios is essential for emotion research. In contrast to the classical passive induction paradigm, immersive VR fully engages the psychological and physiological components of the subject, which is considered an ecologically valid paradigm for studying emotion. Several studies investigate the emotional responses to different VR tasks or games using subjective scales. However, little research regards VR as an eliciting material, especially when systematically analyzing emotional processes in VR from a neurophysiological perspective. To fill this gap and scientifically evaluate VR's ability to be used as an active method for emotion elicitation, we investigate the dynamic relationship between explicit information (subjective evaluations) and implicit information (objective neurophysiological data). A total of 28 participants are enlisted to watch eight VR videos while their SAM/IPQ scores and EEG data are recorded simultaneously. In ecologically valid scenarios, the subjective results demonstrate that VR has significant advantages for evoking emotion in arousal-valence. This conclusion is backed by our examination of objective neurophysiological evidence that VR videos effectively induce high-arousal emotions. In addition, we obtain features of critical channels and frequency oscillations associated with emotional valence, thereby validating previous research in more lifelike circumstances. In particular, we discover hemispheric asymmetry in the occipital region under high and low emotional arousal, which adds to our understanding of neural features and the dynamics of emotional arousal. As a result, we successfully integrate EEG and VR to demonstrate that VR is more pragmatic for evoking natural feelings and is beneficial for emotional research. Our research has set a precedent for new methodologies of using VR induction paradigms to acquire a more reliable explanation of affective computing.

Identifying neural correlates of multidimensional, subjective gaming experiences during active gameplay

Studying how gaming experiences are encoded is important to understand the effects of gaming on the brain. Although studies have investigated neural correlates of gaming experiences, the brain patterns related to the full range of subjective experiences across different types of games are yet to be identified. The present study used three custom-made, immersive driving games with different input dynamics (controlling a car, a boat, or a spaceship) and different mechanics to assess subjective gaming experiences in a magnetic resonance imaging scanner. A correlational analysis identified several brain networks associated with different subjective gaming experiences, including visual and attentional processing networks. The contributions of these networks were further validated using meta-analysis-based functional term decoding. The results of the present study point to a range of perceptual, motivational, and control networks that are engaged during active gameplay.

Estimating objective (EEG) and subjective (SSQ) cybersickness in people with susceptibility to motion sickness

Cybersickness refers to the uncomfortable side effects, such as headaches, dizziness, and nausea, felt while experiencing virtual reality (VR). This study investigated cybersickness in people with sensitivity to motion sickness using electroencephalography (EEG), the Simulator Sickness Questionnaire (SSQ), and simple VR content. Based on the scores from the Motion Sickness Susceptibility Questionnaire (MSSQ), 40 males in their twenties were selected as the sensitive group (n = 20) and non-sensitive group (n = 20). The experiment contained two conditions: a baseline condition representing a resting state and a cybersickness condition in which watching VR content induced cybersickness. The SSQ score increased significantly after watching the VR content in both groups. The sensitive group showed significantly lower absolute power in the beta and gamma bands than the non-sensitive group. The cybersickness condition showed significantly increased delta and decreased alpha compared to the baseline condition. We evaluated EEG and SSQ to identify subjective symptoms and objective physiological changes associated with cybersickness.

Immersive media experience: a survey of existing methods and tools for human influential factors assessment

Virtual reality (VR) applications, especially those where the user is untethered to a computer, are becoming more prevalent as new hardware is developed, computational power and artificial intelligence algorithms are available, and wireless communication networks are becoming more reliable, fast, and providing higher reliability. In fact, recent projections show that by 2022 the number of VR users will double, suggesting the sector was not negatively affected by the worldwide COVID-19 pandemic. The success of any immersive communication system is heavily dependent on the user experience it delivers, thus now more than ever has it become crucial to develop reliable models of immersive media experience (IMEx). In this paper, we survey the literature for existing methods and tools to assess human influential factors (HIFs) related to IMEx. In particular, subjective, behavioural, and psycho-physiological methods are covered. We describe tools available to monitor these HIFs, including the user's sense of presence and immersion, cybersickness, and mental/affective states, as well as their role in overall experience. Special focus is placed on psycho-physiological methods, as it was found that such in-depth evaluation was lacking from the existing literature. We conclude by touching on emerging applications involving multiple-sensorial immersive media and provide suggestions for future research directions to fill existing gaps. It is hoped that this survey will be useful for researchers interested in building new immersive (adaptive) applications that maximize user experience.

Ability Orientation or Good Character? Moderated Mediation Mechanism to Determine the Impact of Telepresence on Consumer Purchasing Intention in Cross-Border E-Commerce

A research model is proposed based on the telepresence theory to determine the long-term mechanism for generating consumer purchasing intention and explore potential information haze caused by information overload. A total of 406 usable samples were collected from the consumers of several cross-border e-commerce (CBEC) platforms in China. The results showed that telepresence has a positive effect on benevolence trust, integrity trust, and ability trust. As expected, benevolence trust and ability trust were found to exhibit significant mediation effects, while integrity trust did not have a significant mediation role. The moderated mediation mechanism shows that product information (description, display, and content) restrain mediation paths. The findings provide new perspectives on the information haze caused by information overload. The results suggest that promoting an ability-oriented (ability trust) business philosophy and instilling good corporate character (benevolence trust) are vital in achieving sustainable development in CBEC platforms. Eliminating information haze could also help strengthen the activation effect of telepresence and promote the guiding role of customer trust on purchasing intention. This study's theoretical and management contributions extend our knowledge of optimizing management strategies for CBEC platforms.

The effects of virtual soccer game on balance, gait function, and kick speed in chronic incomplete spinal cord injury: a randomized controlled trial

STUDY DESIGN

Participant-blinded comparative pre-post randomized controlled trial.

OBJECTIVE

To evaluate the effects of virtual soccer game on balance, gait function, and kick speed in individuals with spinal cord injuries.

SETTING

Forty participants with incomplete spinal cord injury (SCI) were included.

METHODS

The participants were randomized into either an experimental group (EG) or a control group for treatment 3 days/week over 4 weeks. The clinical outcomes included the results of the chair stand test (CST), timed up-and-go (TUG) test, and 10 m walking test (10MWT). The virtual reality (VR) content outcome measure was kick speed.

RESULTS

The independent t-test results indicated that the participants in the EG exhibited superior performances in the CST, TUG test, 10MWT, and kick speed test (all P < 0.05).

CONCLUSIONS

Our results provide novel, promising clinical evidence that VR rehabilitation improves both dynamic and static balance and reduces the risk of falls in patients with incomplete SCI of Asia impairment scale grades C-D.

Using electrophysiological measures to evaluate the sense of presence in immersive virtual environments: An event-related potential study

Sense of presence has been often explored in the context of virtual reality (VR) and immersive visual technologies; however, standardized and objective measures of the sense of presence have been difficult to find. Studies attempting to find physiological correlates of sense presence using electroencephalography (EEG) have reported mixed results. In the present study, we used brain event-related potentials (ERPs) elicited by auditory stimuli to identify an objective physiological index of sense of presence during VR, attempting to replicate the findings of previous studies and explain the heterogeneity of results reported in the literature. Participants in our experiment were asked to experience an immersive virtual environment using a modern head-mounted display while passively hearing task-irrelevant frequent standard and infrequent deviant tones as in a classic auditory oddball paradigm. Subsequently, they were asked to complete a battery of questionnaires aimed to estimate their sense of presence during the VR. EEG and questionnaire data from three-seventh participants were analyzed. ERP components evoked by the auditory stimuli were then analyzed. Late ERP components (after 450 ms from stimulus onset) registered over central brain areas were associated with the sense of presence as measured with questionnaires, while earlier components were not associated with presence. The use of different questionnaires and the content of the VR environment may both be a plausible explanation for heterogeneous results as reported in previous studies. The present study showed that late ERP components recorded over the central brain may represent good electrophysiological correlates of the subjective sense of presence.

Towards a Design Space of Haptics in Everyday Virtual Reality across Different Spatial Scales

Virtual Reality (VR) has become a consumer-grade technology, especially with the advent of standalone headsets working independently from a powerful computer. Domestic VR mainly uses the visual and auditory senses since VR headsets make this accessible. Haptic feedback, however, has the potential to increase immersion substantially. So far, it is mostly used in laboratory settings with specialized haptic devices. Especially for domestic VR, there is underexplored potential in exploiting physical elements of the often confined space in which it is used. In a literature review (n = 20), we analyzed VR interaction using haptic feedback with or without physical limitations. From this, we derive a design space for VR haptics across three spatial scales (seated, standing, and walking). In our narrow selection of papers, we found inspirations for future work and will discuss two example scenarios. Our work gives a current overview of haptic VR solutions and highlights strategies for adapting laboratory solutions to an everyday context.

Effectiveness of Virtual Reality Interventions for Adolescent Patients in Hospital Settings: Systematic Review

Background

Given the high level of interest and increasing familiarity with virtual reality among adolescents, there is great potential to use virtual reality to address adolescents’ unique health care delivery needs while in hospital. While there have been reviews on the use of virtual reality for specific health conditions and procedures, none to date have reviewed the full scope of virtual reality hospital interventions for adolescents who are often combined with children as a homogenous group, despite the fact that adolescents experience virtual environments different from children.

Objective

The aim of this review was to systematically identify available evidence regarding the use of virtual reality interventions for adolescent patients in hospital settings to evaluate effectiveness, suitability, and safety and identify opportunities for future research.

Methods

PubMed, PsycINFO, Medline, and Scopus databases were searched using keywords and phrases. Retrieved abstracts (n=1525) were double screened, yielding 276 articles for full-text screening. Of these, 8 articles met inclusion criteria. Data were extracted to a standardized coding sheet, and a narrative synthesis was performed due to the heterogeneity of the studies.

Results

Four RCTs and 4 single-case reports were identified for inclusion, all of which aimed to reduce pain or anxiety. The scenarios targeted were burn pain, venipuncture, chemotherapy, preoperative anxiety, and palliative care. Three out of 4 RCTs found significant reductions in pain or anxiety outcomes measures when using virtual reality compared to standard care or other distraction techniques; however, only 1 study combined self-reported experiences of pain or anxiety with any physiological measures. Single-case reports relied primarily upon qualitative feedback, with patients reporting reduced pain or anxiety and a preference for virtual reality to no virtual reality.

Conclusions

Virtual reality can provide a safe and engaging way to reduce pain and anxiety in adolescents while in hospital, particularly when virtual reality software is highly immersive and specifically designed for therapeutic purposes. As VR becomes more accessible and affordable for use in hospitals, larger and more diverse studies that capitalize on adolescents’ interest in and aptitude for virtual reality, and on the full range of capabilities of this emerging technology, are needed to build on these promising results.

Trial Registration

PROSPERO International Prospective Register of Systematic Reviews CRD42020198760; https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42020198760

A Methodological Framework for Assessing Social Presence in Music Interactions in Virtual Reality

Virtual reality (VR) brings radical new possibilities to the empirical study of social music cognition and interaction. In the present article, we consider the role of VR as a research tool, based on its potential to create a sense of "social presence": the illusory feeling of being, and socially interacting, inside a virtual environment. This makes VR promising for bridging ecological validity ("research in the wild") and experimental control ("research in the lab") in empirical music research. A critical assumption however is the actual ability of VR to simulate real-life social interactions, either via human-embodied avatars or computer-controlled agents. The mediation of social musical interactions via VR is particularly challenging due to their embodied, complex, and emotionally delicate nature. In this article, we introduce a methodological framework to operationalize social presence by a combination of factors across interrelated layers, relating to the performance output, embodied co-regulation, and subjective experiences. This framework provides the basis for the proposal of a pragmatic approach to determine the level of social presence in virtual musical interactions, by comparing the outcomes across the multiple layers with the outcomes of corresponding real-life musical interactions. We applied and tested this pragmatic approach via a case-study of piano duet performances of the piece Piano Phase composed by Steve Reich. This case-study indicated that a piano duet performed in VR, in which the real-time interaction between pianists is mediated by embodied avatars, might lead to a strong feeling of social presence, as reflected in the measures of performance output, embodied co-regulation, and subjective experience. In contrast, although a piano duet in VR between an actual pianist and a computer-controlled agent led to a relatively successful performance output, it was inadequate in terms of both embodied co-regulation and subjective experience.

Memory for a virtual reality experience in children and adults according to image quality, emotion, and sense of presence

Numerous studies have explored the effects of virtual reality (VR) on adults' cognition. Little is known, however, of these effects in children. The aim of this study was to explore, in both children and adults, the respective roles of the specific factors of VR, such as immersion, sense of presence and emotion, on memory performance. To do so, we used a head-mounted display to present a VR experience in which we manipulated immersion by varying 3D asset quality (High and Low) and emotion by presenting negative, neutral and positive stimuli. 48 adults (M _age = 20.65) and 40 children (M _age = 11.63) were both divided into two experimental groups (High vs. Low 3D model quality). Valence, arousal, and sense of presence were self-assessed by means of questionnaires, while memory of the presented stimuli was assessed using a free recall task. We also performed physiological measurements to provide objective support for our data. Results showed that memory performance was better for emotional than for neutral stimuli regardless of age group, even though children seemed to avoid looking at negative stimuli compared to neutral ones. Memory was predicted by arousal and presence in adults and only by arousal in children. Memory was not impaired by using poor image quality when highly arousing content was displayed. This study revealed that, contrary to adults, the use of poor image quality did not protect children from strong emotional experiences in VR. The roles of familiarity and arousal are discussed to help explain these results.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1007/s10055-021-00537-y.

EEG Cortical Activities and Networks Altered by Watching 2D/3D Virtual Reality Videos

Abstract. Virtual reality (VR), which can represent real-life events and situations, is being increasingly applied to many fields, such as education, entertainment, and medical rehabilitation. Correspondingly, the neural information processing of VR has attracted attention. However, the underlying neural mechanisms of VR environments have not yet been fully revealed. The purpose of this study was to examine the possible differences in brain activities and networks between the less immersive 2D and the fully immersive 3D VR environments. 3D VR videos and the same 2D scenes were presented to the participants and the scalp electroencephalogram (EEG) was recorded, respectively. Power spectral density (PSD) and the functional connectivity of these EEG signals were analyzed. The results showed that 3D VR videos significantly enhanced the PSD of θ rhythm (4–7 Hz) in the frontal lobe; decreased the PSD of α rhythm (8–13 Hz) in the parietal and the occipital lobes; increased the PSD of β rhythm (14–30 Hz) in the frontal, the parietal, the temporal, and the occipital lobes, relative to 2D VR watching. Furthermore, 3D versus 2D VR-induced alterations in the patterns of brain networks were similar to the patterns of PSD. Specifically, for the θ rhythm, 3D VR significantly enhanced the frontal and the temporal brain functional connectivity; for the α rhythm, 3D VR increased the parietal and the occipital networks; for the β rhythm, 3D VR remarkably increased the frontal, the occipital, the frontal-temporal and the frontal-occipital brain functional connectivity, relative to 2D VR. These significant differences between 3D and 2D VR video-watching suggest that the neural information processing of cortical activities and networks is correlated to the degree of immersion. The present results, collected with previous researches, implicate that some visual-related information processes, such as visual attention, visual perception, and visual immersion are more robust in 3D VR environments.

Neurophysiological Correlates of Cognition as Revealed by Virtual Reality: Delving the Brain with a Synergistic Approach

The synergy of perceptual psychology, technology, and neuroscience can be used to comprehend how virtual reality affects cognition of human brain. Numerous studies have used neuroimaging modalities to assess the cognitive state and response of the brain with various external stimulations. The virtual reality-based devices are well known to incur visual, auditory, and haptic induced perceptions. Neurophysiological recordings together with virtual stimulations can assist in correlating humans’ physiological perception with response in the environment designed virtually. The effective combination of these two has been utilized to study human behavior, spatial navigation performance, and spatial presence, to name a few. Moreover, virtual reality-based devices can be evaluated for the neurophysiological correlates of cognition through neurophysiological recordings. Challenges exist in the integration of real-time neuronal signals with virtual reality-based devices, and enhancing the experience together with real-time feedback and control through neuronal signals. This article provides an overview of neurophysiological correlates of cognition as revealed by virtual reality experience, together with a description of perception and virtual reality-based neuromodulation, various applications, and existing challenges in this field of research.

Smoking-related cue reactivity in a virtual reality setting: association between craving and EEG measures

BACKGROUND

Cue-reactivity is the array of responses that smokers exhibit when exposed to conditioned and contextual stimuli previously associated to substance use. The difficulty to experimentally recreate the complexity of smokers' spatial experience and context requires more ecological models. Virtual reality (VR) creates a state of immersion close to reality allowing controlled assessments of behavioral responses. To date, no studies investigated brain activation associated to smoking cue-reactivity in VR using electroencephalography (EEG).

AIMS

To investigate whether a VR cue-reactivity paradigm (a) may increase smoking craving, (b) is feasible with EEG recording, and (c) induces craving levels associated to EEG desynchronization.

METHODS

Smokers (N = 20) and non-smokers (N = 20) were exposed to neutral and smoking-related VR scenarios, without and with smoking conditioned stimuli, respectively. EEG was recorded from occipital and parietal leads throughout the sessions to assess alpha band desynchronization. Smoking and food craving and presence visual analogue scales (VAS) were assessed during the session.

RESULTS

To be smoker, but not non-smoker, significantly influenced smoking craving VAS induced by smoking cue VR but not by neutral VR. No significant food craving changes was observed during the VR sessions. The new finding was that EEG alpha band power in posterior leads was significantly increased by the smoking context scenario only in smokers, and that the degree of smoking (i.e., heavy vs. light) was significantly associated to this neurophysiological measure.

CONCLUSIONS

This study demonstrated, for the first time, the feasibility of EEG recording in a VR setting, suggesting that EEG desynchronization may be a neurophysiological marker of smoking cue-reactivity.

Decision Making, Stress Assessed by Physiological Response and Virtual Reality Stimuli

Many decisions must be made under stress; therefore, stress and decision-making are intrinsically related not only at the behavioral level but also at the neural level. Additionally, virtual reality tools have been proposed as a method to induce stress in the laboratory. This review focuses on answering the following research question: Does stress assessed by physiological variables of a subject under virtual reality stimuli increase the chances of error in decision-making? The reviewed studies were consulted in the following databases: PubMed, IEEE Xplore, and Science Direct. The analysis of the consulted literature indicates that the stress induced in the laboratory using virtual reality tools and the physiological response of the central and autonomous nervous system are complementary subjects and allow the design of training and support systems for the decision-making process

Questionnaire Measures and Physiological Correlates of Presence: A Systematic Review

The published literature has produced several definitions for the sense of presence in a simulated environment, as well as various methods for measuring it. The variety of conceptualizations makes it difficult for researchers to interpret, compare, and evaluate the presence ratings obtained from individual studies. Presence has been measured by employing questionnaires, physiological indices, behavioral feedbacks, and interviews. A systematic literature review was conducted to provide insight into the definitions and measurements of presence in studies from 2002 to 2019, with a focus on questionnaires and physiological measures. The review showed that scholars had introduced various definitions of presence that often originate from different theoretical standpoints and that this has produced a multitude of different questionnaires that aim to measure presence. At the same time, physiological studies that investigate the physiological correlates of the sense of presence have often shown ambiguous results or have not been replicated. Most of the scholars have preferred the use of questionnaires, with Witmer and Singer's Presence Questionnaire being the most prevalent. Among the physiological measures, electroencephalography was the most frequently used. The conclusions of the present review aim to stimulate future structured efforts to standardize the use of the construct of presence, as well as inspire the replication of the findings reported in the published literature.

Too Real to Be Virtual: Autonomic and EEG Responses to Extreme Stress Scenarios in Virtual Reality

The evolution of virtual reality (VR) technologies requires setting boundaries of its use. In this study, 3 female participants were experiencing VR scenarios with stressful content and their activity of the autonomic nervous system and EEG were recorded. It has been discovered that virtual reality can evoke acute stress reactions accompanied by activation of the sympathetic nervous system and a decrease in the activity of the parasympathetic nervous system. The high-stress response is accompanied by a decrease in the power of the EEG, and, on the contrary, the activation of the avoidance reaction is accompanied by an increase in the power of the EEG alpha waves. Therefore, the use of stressful VR content can cause high emotional stress to a user and restrictions should be considered.

Audiohaptic Feedback Enhances Motor Performance in a Low-Fidelity Simulated Drilling Task

When used in educational settings, simulations utilizing virtual reality (VR) technologies can reduce training costs while providing a safe and effective learning environment. Tasks can be easily modified to maximize learning objectives of different levels of trainees (e.g., novice, intermediate, expert), and can be repeated for the development of psychomotor skills. VR offers a multisensory experience, providing visual, auditory, and haptic sensations with varying levels of fidelity. While simulating visual and auditory stimuli is relatively easy and cost-effective, similar representations of haptic sensation still require further development. Evidence suggests that mixing high- and low-fidelity realistic sensations (e.g., audition and haptic) can improve the overall perception of realism, however, whether this also leads to improved performance has not been examined. The current study examined whether audiohaptic stimuli presented in a virtual drilling task can lead to improved motor performance and subjective realism, compared to auditory stimuli alone. Right-handed participants (n = 16) completed 100 drilling trials of each stimulus type. Performance measures indicated that participants overshot the target during auditory trials, and undershot the target during audiohaptic trials. Undershooting is thought to be indicative of improved performance, optimizing both time and energy requirements.

Stereoscopic Rendering via Goggles Elicits Higher Functional Connectivity During Virtual Reality Gaming

Virtual reality (VR) simulates real-world scenarios by creating a sense of presence in its users. Such immersive scenarios lead to behavior that is more similar to that displayed in real world settings, which may facilitate the transfer of knowledge and skills acquired in VR to similar real world situations. VR has already been used in education, psychotherapy, rehabilitation and it comes as an appealing choice for training intervention purposes. The aim of the present study was to investigate to what extent VR technology for games presented via goggles can be used in a magnetic resonance imaging scanner (MRI), addressing the question of whether brain connectivity differs between VR stimulation via goggles and a presentation from a screen via mirror projection. Moreover, we wanted to investigate whether stereoscopic goggle stimulation, where both eyes receive different visual input, would elicit stronger brain connectivity than a stimulation in which both eyes receive the same visual input (monoscopic). To our knowledge, there is no previous research using games and functional connectivity (FC) in MRI to address this question. Multiple analyses approaches were taken so that different aspects of brain connectivity could be covered: fractional low-frequency fluctuation, independent component analysis (ICA), seed-based FC (SeedFC) and graph analysis. In goggle presentation (mono and stereoscopic) as contrasted to screen, we found differences in brain activation in left cerebellum and postcentral gyrus as well as differences in connectivity in the visual cortex and frontal inferior cortex [when focusing on the visual and default mode network (DMN)]. When considering connectivity in specific areas of interest, we found higher connectivity between bilateral superior frontal cortex and the temporal lobe, as well as bilateral inferior parietal cortex with right calcarine and right lingual cortex. Furthermore, we found superior frontal cortex and insula/putamen to be more strongly connected in goggle stereoscopic vs. goggle monoscopic, in line with our hypothesis. We assume that the condition that elicits higher brain connectivity values should be most suited for long-term brain training interventions given that, extended training under these conditions could permanently improve brain connectivity on a functional as well as on a structural level.

Real vs. immersive-virtual emotional experience: Analysis of psycho-physiological patterns in a free exploration of an art museum

Virtual reality is a powerful tool in human behaviour research. However, few studies compare its capacity to evoke the same emotional responses as in real scenarios. This study investigates psycho-physiological patterns evoked during the free exploration of an art museum and the museum virtualized through a 3D immersive virtual environment (IVE). An exploratory study involving 60 participants was performed, recording electroencephalographic and electrocardiographic signals using wearable devices. The real vs. virtual psychological comparison was performed using self-assessment emotional response tests, whereas the physiological comparison was performed through Support Vector Machine algorithms, endowed with an effective feature selection procedure for a set of state-of-the-art metrics quantifying cardiovascular and brain linear and nonlinear dynamics. We included an initial calibration phase, using standardized 2D and 360° emotional stimuli, to increase the accuracy of the model. The self-assessments of the physical and virtual museum support the use of IVEs in emotion research. The 2-class (high/low) system accuracy was 71.52% and 77.08% along the arousal and valence dimension, respectively, in the physical museum, and 75.00% and 71.08% in the virtual museum. The previously presented 360° stimuli contributed to increasing the accuracy in the virtual museum. Also, the real vs. virtual classifier accuracy was 95.27%, using only EEG mean phase coherency features, which demonstrates the high involvement of brain synchronization in emotional virtual reality processes. These findings provide an important contribution at a methodological level and to scientific knowledge, which will effectively guide future emotion elicitation and recognition systems using virtual reality.

A Mixed-Methods Approach Using Self-Report, Observational Time Series Data, and Content Analysis for Process Analysis of a Media Reception Phenomenon

Due to the complexityof research objects, theoretical concepts, and stimuli in media research, researchers in psychology and communications presumably need sophisticated measures beyond self-report scales to answer research questions on media use processes. The present study evaluates stimulus-dependent structure in spontaneous eye-blink behavior as an objective, corroborative measure for the media use phenomenon of spatial presence. To this end, a mixed methods approach is used in an experimental setting to collect, combine, analyze, and interpret data from standardized participant self-report, observation of participant behavior, and content analysis of the media stimulus. T-pattern detection is used to analyze stimulus-dependent blinking behavior, and this structural data is then contrasted with self-report data. The combined results show that behavioral indicators yield the predicted results, while self-report data shows unpredicted results that are not predicted by the underlying theory. The use of a mixed methods approach offered insights that support further theory development and theory testing beyond a traditional, mono-method experimental approach.

Toward understanding the neurophysiological basis of peripersonal space: An EEG study on healthy individuals

The subcortical mechanisms subtending the sensorimotor processes related to the peripersonal space (PPS) have been well characterized, whereas less evidence is available concerning the cortical mechanisms. We investigated the theta, alpha and beta event-related spectral perturbations (ERSP) while holding the forearm in different positions into the PPS of the face. Fifty healthy individuals were subjected to EEG recording while being provided with median nerve electric stimulation at the wrist of the right hand held at different hand-to-face distances. Theta and beta rhythms were significantly perturbed depending on the hand-to-face distance, whereas alpha oscillations reflected a more general, non-specific oscillatory response to the motor task. The perturbation of theta and beta frequency bands may reflect the processes of top-down modulation overseeing the conscious spatiotemporal encoding of sensory-motor information within the PPS. In other words, such perturbation reflects the continuous update of the conscious internal representations of the PPS to build up a purposeful and reflexive motor response.

Psychophysiological Alteration After Virtual Reality Experiences Using Smartphone-Assisted Head Mount Displays: An EEG-Based Source Localization Study

Brain functional changes could be observed in people after an experience of virtual reality (VR). The present study investigated cyber sickness and changes of brain regional activity using electroencephalogram (EEG)-based source localization, before and after a VR experience involving a smartphone-assisted head mount display. Thirty participants (mean age = 25 years old) were recruited. All were physically healthy and had no ophthalmological diseases. Their corrected vision was better than 20/20. Resting state EEG and the simulator sickness questionnaire (SSQ) were measured before and after the VR experience. Source activity of each frequency band was calculated using the sLORETA program. After the VR experience, the SSQ total score and sub scores (nausea, oculomotor symptoms, and disorientation) were significantly increased, and brain source activations were significantly increased: alpha1 activity in the cuneus and alpha2 activity in the cuneus and posterior cingulate gyrus (PCG). The change of SSQ score (after–before) showed significant negative correlation with the change of PCG activation (after–before) in the alpha2 band. The study demonstrated increased cyber sickness and increased alpha band power in the cuneus and PCG after the VR experience. Reduced PCG activation in alpha band may be associated with the symptom severity of cyber sickness.

Attention allocation to 2D and 3D emotion-inducing scenes: A neurophysiological study

Past research on emotions was mostly supported on emotion inducing slides, but studies in the field are moving towards more ecological stimuli, including 3D stimuli and virtual reality scenes. The present study aims to compare the effects of two-dimensional (2D) and three-dimensional (3D) emotion-inducing scenes in the modulation of attention, as indexed by the amplitude of the P3 event-related component in a dual-task paradigm. We recorded the EEG signal of 31 healthy male participants performing an active auditory oddball task, while simultaneously observing pleasant, unpleasant and neutral scenes in 2D or 3D. We analyzed the effects of emotional valence and visualization mode in the mean amplitude of the P3 component at Pz. The results revealed than revealed that 2D scenarios were as effective as 3D stimuli in capturing attention resources and this effect was consistently observed in all emotional scenes.

The combined use of virtual reality and EEG to study language processing in naturalistic environments

When we comprehend language, we often do this in rich settings where we can use many cues to understand what someone is saying. However, it has traditionally been difficult to design experiments with rich three-dimensional contexts that resemble our everyday environments, while maintaining control over the linguistic and nonlinguistic information that is available. Here we test the validity of combining electroencephalography (EEG) and virtual reality (VR) to overcome this problem. We recorded electrophysiological brain activity during language processing in a well-controlled three-dimensional virtual audiovisual environment. Participants were immersed in a virtual restaurant while wearing EEG equipment. In the restaurant, participants encountered virtual restaurant guests. Each guest was seated at a separate table with an object on it (e.g., a plate with salmon). The restaurant guest would then produce a sentence (e.g., “I just ordered this salmon.”). The noun in the spoken sentence could either match (“salmon”) or mismatch (“pasta”) the object on the table, creating a situation in which the auditory information was either appropriate or inappropriate in the visual context. We observed a reliable N400 effect as a consequence of the mismatch. This finding validates the combined use of VR and EEG as a tool to study the neurophysiological mechanisms of everyday language comprehension in rich, ecologically valid settings.

Presence and Cybersickness in Virtual Reality Are Negatively Related: A Review

In order to take advantage of the potential offered by the medium of virtual reality (VR), it will be essential to develop an understanding of how to maximize the desirable experience of "presence" in a virtual space ("being there"), and how to minimize the undesirable feeling of "cybersickness" (a constellation of discomfort symptoms experienced in VR). Although there have been frequent reports of a possible link between the observer's sense of presence and the experience of bodily discomfort in VR, the amount of literature that discusses the nature of the relationship is limited. Recent research has underlined the possibility that these variables have shared causes, and that both factors may be manipulated with a single approach. This review paper summarizes the concepts of presence and cybersickness and highlights the strengths and gaps in our understanding about their relationship. We review studies that have measured the association between presence and cybersickness, and conclude that the balance of evidence favors a negative relationship between the two factors which is driven principally by sensory integration processes. We also discuss how system immersiveness might play a role in modulating both presence and cybersickness. However, we identify a serious absence of high-powered studies that aim to reveal the nature of this relationship. Based on this evidence we propose recommendations for future studies investigating presence, cybersickness, and other related factors.

Flow and Immersion in Video Games: The Aftermath of a Conceptual Challenge

One of the most pleasurable aspects of video games is their ability to induce immersive experiences. However, there appears to be a tentative conceptualization of what an immersive experience is. In this short review, we specifically focus on the terms of flow and immersion, as they are the most widely used and applied definitions in the video game literature, whilst their differences remain disputable. We critically review the concepts separately and proceed with a comparison on their proposed differences. We conclude that immersion and flow do not substantially differ in current studies and that more evidence is needed to justify their separation.

Virtual Reality for Research in Social Neuroscience

The emergence of social neuroscience has significantly advanced our understanding of the relationship that exists between social processes and their neurobiological underpinnings. Social neuroscience research often involves the use of simple and static stimuli lacking many of the potentially important aspects of real world activities and social interactions. Whilst this research has merit, there is a growing interest in the presentation of dynamic stimuli in a manner that allows researchers to assess the integrative processes carried out by perceivers over time. Herein, we discuss the potential of virtual reality for enhancing ecological validity while maintaining experimental control in social neuroscience research. Virtual reality is a technology that allows for the creation of fully interactive, three-dimensional computerized models of social situations that can be fully controlled by the experimenter. Furthermore, the introduction of interactive virtual characters—either driven by a human or by a computer—allows the researcher to test, in a systematic and independent manner, the effects of various social cues. We first introduce key technical features and concepts related to virtual reality. Next, we discuss the potential of this technology for enhancing social neuroscience protocols, drawing on illustrative experiments from the literature.

Modeling behavior dynamics using computational psychometrics within virtual worlds

In case of fire in a building, how will people behave in the crowd? The behavior of each individual affects the behavior of others and, conversely, each one behaves considering the crowd as a whole and the individual others. In this article, I propose a three-step method to explore a brand new way to study behavior dynamics. The first step relies on the creation of specific situations with standard techniques (such as mental imagery, text, video, and audio) and an advanced technique [Virtual Reality (VR)] to manipulate experimental settings. The second step concerns the measurement of behavior in one, two, or many individuals focusing on parameters extractions to provide information about the behavior dynamics. Finally, the third step, which uses the parameters collected and measured in the previous two steps in order to simulate possible scenarios to forecast through computational models, understand, and explain behavior dynamics at the social level. An experimental study was also included to demonstrate the three-step method and a possible scenario.

Towards immersive virtual reality (iVR): a route to surgical expertise

Surgery is characterized by complex tasks performed in stressful environments. To enhance patient safety and reduce errors, surgeons must be trained in environments that mimic the actual clinical setting. Rasmussen’s model of human behavior indicates that errors in surgical procedures may be skill-, rule-, or knowledge-based. While skill-based behavior and some rule-based behavior may be taught using box trainers and ex vivo or in vivo animal models, we posit that multimodal immersive virtual reality (iVR) that includes high-fidelity visual as well as other sensory feedback in a seamless fashion provides the only means of achieving true surgical expertise by addressing all three levels of human behavior. While the field of virtual reality is not new, realization of the goals of complete immersion is challenging and has been recognized as a Grand Challenge by the National Academy of Engineering. Recent technological advances in both interface and computational hardware have generated significant enthusiasm in this field. In this paper, we discuss convergence of some of these technologies and possible evolution of the field in the near term.

Neuroelectric adaptations to cognitive processing in virtual environments: an exercise-related approach

Recently, virtual environments (VEs) are suggested to encourage users to exercise regularly. The benefits of chronic exercise on cognitive performance are well documented in non-VE neurophysiological and behavioural studies. Based on event-related potentials (ERP) such as the N200 and P300, cognitive processing may be interpreted on a neuronal level. However, exercise-related neuroelectric adaptation in VE remains widely unclear and thus characterizes the primary aim of the present study. Twenty-two healthy participants performed active (moderate cycling exercise) and passive (no exercise) sessions in three VEs (control, front, surround), each generating a different sense of presence. Within sessions, conditions were randomly assigned, each lasting 5 min and including a choice reaction-time task to assess cognitive performance. According to the international 10:20 system, EEG with real-time triggered stimulus onset was recorded, and peaks of N200 and P300 components (amplitude, latency) were exported for analysis. Heart rate was recorded, and sense of presence assessed prior to and following each session and condition. Results revealed an increase in ERP amplitudes (N200: p < 0.001; P300: p < 0.001) and latencies (N200: p < 0.001) that were most pronounced over fronto-central and occipital electrode sites relative to an increased sense of presence (p < 0.001); however, ERP were not modulated by exercise (each p > 0.05). Hypothesized to mirror cognitive processing, decreases of cognitive performance's accuracy and reaction time failed significance. With respect to previous research, the present neuroelectric adaptation gives reason to believe in compensative neuronal resources that balance demanding cognitive processing in VE to avoid behavioural inefficiency.

Natural stimuli from three coherent modalities enhance behavioral responses and electrophysiological cortical activity in humans

Cues that involve a number of sensory modalities are processed in the brain in an interactive multimodal manner rather than independently for each modality. We studied multimodal integration in a natural, yet fully controlled scene, implemented as an interactive game in an auditory-haptic-visual virtual environment. In this imitation of a natural scene, the targets of perception were ecologically valid uni-, bi- and tri-modal manifestations of a simple event-a ball hitting a wall. Subjects were engaged in the game while their behavioral and early cortical electrophysiological responses were measured. Behavioral results confirmed that tri-modal cues were detected faster and more accurately than bi-modal cues, which, likewise, showed advantages over unimodal responses. Event-Related Potentials (ERPs) were recorded, and the first 200 ms following stimulus onset was analyzed to reveal the latencies of cortical multimodal interactions as estimated by sLORETA. These electrophysiological findings indicated bi-modal as well as tri-modal interactions beginning very early (~30 ms), uniquely for each multimodal combination. The results suggest that early cortical multimodal integration accelerates cortical activity and, in turn, enhances performance measures. This acceleration registers on the scalp as sub-additive cortical activation.

Perspective and agency during video gaming influences spatial presence experience and brain activation patterns

Background

The experience of spatial presence (SP), i.e., the sense of being present in a virtual environment, emerges if an individual perceives himself as 1) if he were actually located (self-location) and 2) able to act in the virtual environment (possible actions). In this study, two main media factors (perspective and agency) were investigated while participants played a commercially available video game.

Methods

The differences in SP experience and associated brain activation were compared between the conditions of game play in first person perspective (1PP) and third person perspective (3PP) as well as between agency, i.e., active navigation of the video game character (active), and non-agency, i.e., mere passive observation (passive). SP was assessed using standard questionnaires, and brain activation was measured using electroencephalography (EEG) and sLORETA source localisation (standard low-resolution brain electromagnetic tomography).

Results

Higher SP ratings were obtained in the 1PP compared with the 3PP condition and in the active compared with the passive condition. On a neural level, we observed in the 1PP compared with the 3PP condition significantly less alpha band power in the parietal, the occipital and the limbic cortex. In the active compared with the passive condition, we uncovered significantly more theta band power in frontal brain regions.

Conclusion

We propose that manipulating the factors perspective and agency influences SP formation by either directly or indirectly modulating the ego-centric visual processing in a fronto-parietal network. The neuroscientific results are discussed in terms of the theoretical concepts of SP.

Psychophysiology to Assess Impact of Varying Levels of Simulation Fidelity in a Threat Environment

There are many virtual environments found in the serious game community that simulate real world scenarios. There is a broad range of fidelity and experimental controls among these serious games. An important component to most evaluations is the extent to which level of fidelity impacts the persons immersed in the serious game. While a great deal of virtual environment and serious game research has assessed the subjective state or feeling of the participant (e.g., the participant’s sense of presence) through the use of questionnaires, the current study examines participant experience by examining psychophysiological responses of participants to their surroundings. The primary goal in this study was evaluative: will a virtual environment with arousing contents result in increased sensory arousal if it is presented in a highly immersive configuration? A secondary goal of this study was to investigate the utility of our environment to offer varying levels of stimulus threat to impact the user’s experience of the virtual environment. Increased simulation fidelity in an arousing environment resulted in faster heart rates and increased startle eyeblink amplitudes, suggesting that higher fidelity scenarios had great efficacy related to sensory arousal.

An exploratory fNIRS study with immersive virtual reality: a new method for technical implementation

For over two decades Virtual Reality (VR) has been used as a useful tool in several fields, from medical and psychological treatments, to industrial and military applications. Only in recent years researchers have begun to study the neural correlates that subtend VR experiences. Even if the functional Magnetic Resonance Imaging (fMRI) is the most common and used technique, it suffers several limitations and problems. Here we present a methodology that involves the use of a new and growing brain imaging technique, functional Near-infrared Spectroscopy (fNIRS), while participants experience immersive VR. In order to allow a proper fNIRS probe application, a custom-made VR helmet was created. To test the adapted helmet, a virtual version of the line bisection task was used. Participants could bisect the lines in a virtual peripersonal or extrapersonal space, through the manipulation of a Nintendo Wiimote ® controller in order for the participants to move a virtual laser pointer. Although no neural correlates of the dissociation between peripersonal and extrapersonal space were found, a significant hemodynamic activity with respect to the baseline was present in the right parietal and occipital areas. Both advantages and disadvantages of the presented methodology are discussed.