Why and how to use virtual reality to study human social interaction: The challenges of exploring a new research landscape

An immersive virtual reality communication skills training for dietitians: A feasibility study

Objective

The training of dietitians emphasizes the development of strong communication skills. Immersive virtual reality (IVR) has been successfully employed for various healthcare scenarios; however, it is yet understudied in dietetics education. Therefore, we conducted a feasibility study to investigate the usefulness of IVR for the purpose of communication skills training.

Methods

We designed a multi-user virtual hospital environment that enabled the training of professional conversations between dietitians and patients. Divided into groups of three, 30 dietetics students participated in a role-play task. Taking a qualitative approach to inquiry, we evaluated how participants assessed the benefits and limitations of the IVR training.

Results

Participants appreciated the authenticity of the training environment and mentioned various advantages of IVR (e.g., sense of privacy, better focus on conversation) over traditional modes of instruction. On the other side, participants frequently mentioned that the lack of avatars' facial expressions might present an obstacle for effective communication skills training.

Conclusion

IVR enables authentic communication skills trainings for dietitians. Special consideration should be given to providing ample social cues during training.

Innovation

The study demonstrates that the promising results from other healthcare professions regarding the usefulness of IVR training also apply to dietetics.

Gender Swap in Virtual Reality for Supporting Inclusion and Implications in the Workplace

OCCUPATIONAL APPLICATIONSWe explored the potential impacts of a virtual gender swap on perceptions toward sexual harassment, which is a harmful behavior that lacks respect and inclusivity. Given that perceptions of harassing behaviors can vary, and that gender may influence one's interpretation of such behaviors, we implemented gender swap in virtual reality (VR) to examine changes in sensitivity to harassment across genders. Participants reported harassing behaviors as more inappropriate when embodying female avatars, regardless of their own gender. Our results suggest that gender swap in VR may raise awareness and narrow the gender gap in harassment perceptions, showing potential of VR-based interventions for immersive workplace training to effectively address biases and promote inclusivity among diversity, equity, and inclusion training. Our study also shows the potential of VR to simulate diverse scenarios and perspectives for tailored training experiences that cater to the specific needs and challenges of different occupational settings.

Towards an ecologically valid naturalistic cognitive neuroscience of memory and event cognition

The landscape of human memory and event cognition research has witnessed a transformative journey toward the use of naturalistic contexts and tasks. In this review, we track this progression from abrupt, artificial stimuli used in extensively controlled laboratory experiments to more naturalistic tasks and stimuli that present a more faithful representation of the real world. We argue that in order to improve ecological validity, naturalistic study designs must consider the complexity of the cognitive phenomenon being studied. Then, we review the current state of "naturalistic" event segmentation studies and critically assess frequently employed movie stimuli. We evaluate recently developed tools like lifelogging and other extended reality technologies to help address the challenges we identified with existing naturalistic approaches. We conclude by offering some guidelines that can be used to design ecologically valid cognitive neuroscience studies of memory and event cognition.

Tools and methods to study and replicate experiments addressing human social cognition in interactive scenarios

In the last decade, scientists investigating human social cognition have started bringing traditional laboratory paradigms more "into the wild" to examine how socio-cognitive mechanisms of the human brain work in real-life settings. As this implies transferring 2D observational paradigms to 3D interactive environments, there is a risk of compromising experimental control. In this context, we propose a methodological approach which uses humanoid robots as proxies of social interaction partners and embeds them in experimental protocols that adapt classical paradigms of cognitive psychology to interactive scenarios. This allows for a relatively high degree of "naturalness" of interaction and excellent experimental control at the same time. Here, we present two case studies where our methods and tools were applied and replicated across two different laboratories, namely the Italian Institute of Technology in Genova (Italy) and the Agency for Science, Technology and Research in Singapore. In the first case study, we present a replication of an interactive version of a gaze-cueing paradigm reported in Kompatsiari et al. (J Exp Psychol Gen 151(1):121-136, 2022). The second case study presents a replication of a "shared experience" paradigm reported in Marchesi et al. (Technol Mind Behav 3(3):11, 2022). As both studies replicate results across labs and different cultures, we argue that our methods allow for reliable and replicable setups, even though the protocols are complex and involve social interaction. We conclude that our approach can be of benefit to the research field of social cognition and grant higher replicability, for example, in cross-cultural comparisons of social cognition mechanisms.

Sense of Agency during Encoding Predicts Subjective Reliving

Autonoetic consciousness (ANC), the ability to re-experience personal past events links episodic memory and self-consciousness by bridging awareness of oneself in a past event (i.e., during its encoding) with awareness of oneself in the present (i.e., during the reliving of a past event). Recent neuroscience research revealed a bodily form of self-consciousness, including the sense of agency (SoA) and the sense of body ownership (SoO) that are based on the integration of multisensory bodily inputs and motor signals. However, the relation between SoA and/or SoO with ANC is not known. Here, we used immersive virtual reality technology and motion tracking and investigated the potential association of SoA/SoO with ANC. For this, we exposed participants to different levels of visuomotor and perspectival congruency, known to modulate SoA and SoO, during the encoding of virtual scenes and collected ANC ratings 1 week after the encoding session. In a total of 74 healthy participants, we successfully induced systematic changes in SoA and SoO during encoding and found that ANC depended on the level of SoA experienced during encoding. Moreover, ANC was positively associated with SoA, but only for the scene encoded with preserved visuomotor and perspectival congruency, and such SoA-ANC coupling was absent for SoO and control questions. Collectively, these data provide behavioral evidence in a novel paradigm that links a key subjective component of bodily self-consciousness during encoding, SoA, to the subjective reliving of those encoded events from one's past, ANC.

Current opportunities and challenges of immersive virtual reality for psychological research and application

Immersive virtual reality (iVR), that is, digital stereoscopic 360° scenarios usually presented in head-mounted displays, has gained much popularity in medical, educational, and consumer contexts in the last years. Recently, psychological research started to utilize the theoretical and methodological advantages of iVR. Furthermore, understanding cognitive, emotional, and behavioral processes in iVR similar to real-life is a genuinely psychological, currently understudied topic. This article briefly reviews the current application of iVR in psychological research and related disciplines. The review presents empirical evidence for opportunities and strengths (e.g., realism, experimental control, effectiveness of therapeutic and educational interventions) as well as challenges and weaknesses (e.g., differences in experiencing presence, interacting with VR content including avatars, i.e., graphical representation of a person). The main part discusses areas requiring additional basic research, such as cognitive processes, socio-emotional processes during social interactions in iVR, and possible societal implications (e.g., fraud, VR-addiction). For both research and application, iVR offers a contemporary extension of the psychological toolkit, offering new avenues to investigate and enhance core phenomena of psychology such as cognition, affect, motivation, and behavior. Still, it is crucial to exercise caution in its application as excessive and careless use of iVR can pose risks to individuals' mental and physical well-being.

Observers predict actions from facial emotional expressions during real-time social interactions

In face-to-face social interactions, emotional expressions provide insights into the mental state of an interactive partner. This information can be crucial to infer action intentions and react towards another person's actions. Here we investigate how facial emotional expressions impact subjective experience and physiological and behavioral responses to social actions during real-time interactions. Thirty-two participants interacted with virtual agents while fully immersed in Virtual Reality. Agents displayed an angry or happy facial expression before they directed an appetitive (fist bump) or aversive (punch) social action towards the participant. Participants responded to these actions, either by reciprocating the fist bump or by defending the punch. For all interactions, subjective experience was measured using ratings. In addition, physiological responses (electrodermal activity, electrocardiogram) and participants' response times were recorded. Aversive actions were judged to be more arousing and less pleasant relative to appetitive actions. In addition, angry expressions increased heart rate relative to happy expressions. Crucially, interaction effects between facial emotional expression and action were observed. Angry expressions reduced pleasantness stronger for appetitive compared to aversive actions. Furthermore, skin conductance responses to aversive actions were increased for happy compared to angry expressions and reaction times were faster to aversive compared to appetitive actions when agents showed an angry expression. These results indicate that observers used facial emotional expression to generate expectations for particular actions. Consequently, the present study demonstrates that observers integrate information from facial emotional expressions with actions during social interactions.

Feasibility of virtual reality and machine learning to assess personality traits in an organizational environment

Introduction

Personality plays a crucial role in shaping an individual's interactions with the world. The Big Five personality traits are widely used frameworks that help describe people's psychological behaviours. These traits predict how individuals behave within an organizational setting.

Methods

In this article, we introduce a virtual reality (VR) strategy for relatively scoring an individual's personality to evaluate the feasibility of predicting personality traits from implicit measures captured from users interacting in VR simulations of different organizational situations. Specifically, eye-tracking and decision-making patterns were used to classify individuals according to their level in each of the Big Five dimensions using statistical machine learning (ML) methods. The virtual environment was designed using an evidence-centered design approach.

Results

The dimensions were assessed using NEO-FFI inventory. A random forest ML model provided 83% accuracy in predicting agreeableness. A k-nearest neighbour ML model provided 75%, 75%, and 77% accuracy in predicting openness, neuroticism, and conscientiousness, respectively. A support vector machine model provided 85% accuracy for predicting extraversion. These analyses indicated that the dimensions could be differentiated by eye-gaze patterns and behaviours during immersive VR.

Discussion

Eye-tracking measures contributed more significantly to this differentiation than the behavioural metrics. Currently, we have obtained promising results with our group of participants, but to ensure the robustness and generalizability of our findings, it is imperative to replicate the study with a considerably larger sample. This study demonstrates the potential of VR and ML to recognize personality traits.

Combining EEG and eye-tracking in virtual reality: Obtaining fixation-onset event-related potentials and event-related spectral perturbations

Extensive research conducted in controlled laboratory settings has prompted an inquiry into how results can be generalized to real-world situations influenced by the subjects' actions. Virtual reality lends itself ideally to investigating complex situations but requires accurate classification of eye movements, especially when combining it with time-sensitive data such as EEG. We recorded eye-tracking data in virtual reality and classified it into gazes and saccades using a velocity-based classification algorithm, and we cut the continuous data into smaller segments to deal with varying noise levels, as introduced in the REMoDNav algorithm. Furthermore, we corrected for participants' translational movement in virtual reality. Various measures, including visual inspection, event durations, and the velocity and dispersion distributions before and after gaze onset, indicate that we can accurately classify the continuous, free-exploration data. Combining the classified eye-tracking with the EEG data, we generated fixation-onset event-related potentials (ERPs) and event-related spectral perturbations (ERSPs), providing further evidence for the quality of the eye-movement classification and timing of the onset of events. Finally, investigating the correlation between single trials and the average ERP and ERSP identified that fixation-onset ERSPs are less time sensitive, require fewer repetitions of the same behavior, and are potentially better suited to study EEG signatures in naturalistic settings. We modified, designed, and tested an algorithm that allows the combination of EEG and eye-tracking data recorded in virtual reality.

Age-related differences in subjective and physiological emotion evoked by immersion in natural and social virtual environments

Age-related changes in emotional processing are complex, with a bias toward positive information. However, the impact of aging on emotional responses in positive everyday situations remains unclear. Virtual Reality (VR) has emerged as a promising tool for investigating emotional processing, offering a unique balance between ecological validity and experimental control. Yet, limited evidence exists regarding its efficacy to elicit positive emotions in older adults. Our study aimed to explore age-related differences in positive emotional responses to immersion in both social and nonsocial virtual emotional environments. We exposed 34 younger adults and 24 older adults to natural and social 360-degree video content through a low immersive computer screen and a highly immersive Head-Mounted Display, while recording participants' physiological reactions. Participants also provided self-report of their emotions and sense of presence. The findings support VR's efficacy in eliciting positive emotions in both younger and older adults, with age-related differences in emotional responses influenced by the specific video content rather than immersion level. These findings underscore the potential of VR as a valuable tool for examining age-related differences in emotional responses and developing VR applications to enhance emotional wellbeing across diverse user populations.

Experimental Study on Emergency Psychophysiological and Behavioral Reactions to Coal Mining Accidents

Effective emergency responses are crucial for preventing coal mine accidents and mitigating injuries. This paper aims to investigate the characteristics of emergency psychophysiological reactions to coal mine accidents and to explore the potential of key indicators for identifying emergency behavioral patterns. Initially, virtual reality technology facilitated a simulation experiment for emergency escape during coal mine accidents. Subsequently, the characteristics of emergency reactions were analyzed through correlation analysis, hypothesis testing, and analysis of variance. The significant changes in physiological indicators were then taken as input features and fed into the three classifiers of machine learning algorithms. These classifications ultimately led to the identification of behavioral patterns, including agility, defensiveness, panic, and rigidity, that individuals may exhibit during a coal mine accident emergency. The study results revealed an intricate relationship between the mental activities induced by accident stimuli and the resulting physiological changes and behavioral performances. During the virtual reality simulation of a coal mine accident, subjects were observed to experience significant physiological changes in electrodermal activity, heart rate variability, electromyogram, respiration, and skin temperature. The random forest classification model, based on SCR + RANGE + IBI + SDNN + LF/HF, outperformed all other models, achieving accuracies of up to 92%. These findings hold promising implications for early warning systems targeting abnormal psychophysiological and behavioral reactions to emergency accidents, potentially serving as a life-saving measure in perilous situations and fostering the sustainable growth of the coal mining industry.

Expanding the empirical study of virtual reality beyond empathy to compassion, moral reasoning, and moral foundations

This study utilizes a controlled experimental design to investigate the influence of a virtual reality experience on empathy, compassion, moral reasoning, and moral foundations. With continued debate and mixed results from previous studies attempting to show relationships between virtual reality and empathy, this study takes advantage of the technology for its ability to provide a consistent, repeatable experience, broadening the scope of analysis beyond empathy. A systematic literature review identified the most widely used and validated moral psychology assessments for the constructs, and these assessments were administered before and after the virtual reality experience. The study is comprised of two pre-post experiments with student participants from a university in the United States. The first experiment investigated change in empathy and moral foundations among 44 participants, and the second investigated change in compassion and moral reasoning among 69 participants. The results showed no significant change in empathy nor compassion, but significant change in moral reasoning from personal interest to post-conventional stages, and significant increase in the Care/harm factor of moral foundations. By testing four of the primary constructs of moral psychology with the most widely used and validated assessments in controlled experiments, this study attempts to advance our understanding of virtual reality and its potential to influence human morality. It also raises questions about our self-reported assessment tools and provides possible new insights for the constructs examined.

Brain waves and landscape settings: emotional responses to attractiveness

Neuro-architecture is a specific branch of architecture that studies how the physical environment can change our mental processes and influence our behaviors. One of the main purposes of this field is to use changes in brain activities as a measure to quantify attractiveness of the landscapes. In this study, we investigated how changes in elements of attractiveness influence ones' emotional perception and present the related pattern of changes in brain activities. Therefore, we implied five elements of attractiveness including mystery, visual openness, landscape or greenness, walkability, and social interaction using the Delphi method. Then, we made changes in each element separately to make the landscape more attractive and assessed their effects on a group of young adults. We used the self-assessment manikin questionnaire to measure the participants' emotional perception while the participants' brain activities were recorded using a 32-channel EEG while exposed to the landscape images. The results showed that changes in attractive elements of the landscape could significantly improve ones' emotional perception of the landscape. In addition, these changes are perceived by changing the oscillatory pattern of brain activities. We hope these findings could shed a light to use of neural markers in measurement of place attractiveness.

Taking consciousness for real: Increasing the ecological validity of the study of conscious vs. unconscious processes

The study of consciousness has developed well-controlled, rigorous methods for manipulating and measuring consciousness. Yet, in the process, experimental paradigms grew farther away from everyday conscious and unconscious processes, which raises the concern of ecological validity. In this review, we suggest that the field can benefit from adopting a more ecological approach, akin to other fields of cognitive science. There, this approach challenged some existing hypotheses, yielded stronger effects, and enabled new research questions. We argue that such a move is critical for studying consciousness, where experimental paradigms tend to be artificial and small effect sizes are relatively prevalent. We identify three paths for doing so-changing the stimuli and experimental settings, changing the measures, and changing the research questions themselves-and review works that have already started implementing such approaches. While acknowledging the inherent challenges, we call for increasing ecological validity in consciousness studies.

Electrophysiological correlates of face and object perception: A comparative analysis of 2D laboratory and virtual reality conditions

Human face perception is a specialized visual process with inherent social significance. The neural mechanisms reflecting this intricate cognitive process have evolved in spatially complex and emotionally rich environments. Previous research using VR to transfer an established face perception paradigm to realistic conditions has shown that the functional properties of face-sensitive neural correlates typically observed in the laboratory are attenuated outside the original modality. The present study builds on these results by comparing the perception of persons and objects under conventional laboratory (PC) and realistic conditions in VR. Adhering to established paradigms, the PC- and VR modalities both featured images of persons and cars alongside standard control images. To investigate the individual stages of realistic face processing, response times, the typical face-sensitive N170 component, and relevant subsequent components (L1, L2; pre-, post-response) were analyzed within and between modalities. The between-modality comparison of response times and component latencies revealed generally faster processing under realistic conditions. However, the obtained N170 latency and amplitude differences showed reduced discriminative capacity under realistic conditions during this early stage. These findings suggest that the effects commonly observed in the lab are specific to monitor-based presentations. Analyses of later and response-locked components showed specific neural mechanisms for identification and evaluation are employed when perceiving the stimuli under realistic conditions, reflected in discernible amplitude differences in response to faces and objects beyond the basic perceptual features. Conversely, the results do not provide evidence for comparable stimulus-specific perceptual processing pathways when viewing pictures of the stimuli under conventional laboratory conditions.

Measurement of Empathy in Virtual Reality with Head-Mounted Displays: A Systematic Review

We present a systematic review of 111 papers that measure the impact of virtual experiences created through head-mounted displays (HMDs) on empathy. Our goal was to analyze the conditions and the extent to which virtual reality (VR) enhances empathy. To achieve this, we categorized the relevant literature according to measurement methods, correlated human factors, viewing experiences, topics, and participants. Meta-analysis was performed based on categorized themes, and under specified conditions, we found that VR can improve empathy. Emotional empathy increased temporarily after the VR experience and returned to its original level over time, whereas cognitive empathy remained enhanced. Furthermore, while VR did not surpass 2D video in improving emotional empathy, it did enhance cognitive empathy, which is associated with embodiment. Our results are consistent with existing research suggesting differentiation between cognitive empathy (influenced by environmental factors and learnable) and emotional empathy (highly heritable and less variable). Interactivity, target of empathy, and point of view were not found to significantly affect empathy, but participants' age and nationality were found to influence empathy levels. It can be concluded that VR enhances cognitive empathy by immersing individuals in the perspective of others and that storytelling and personal characteristics are more important than the composition of the VR scene. Our findings provide guiding information for creating empathy content in VR and designing experiments to measure empathy.

Examining the Effects of Gender Transfer in Virtual Reality on Implicit Gender Bias

OBJECTIVE

To investigate the effect of gender transfer in virtual reality on implicit gender bias.

BACKGROUND

Gender bias is a type of discrimination based on gender, which can lead to increased self-doubt and decreased self-esteem. Sexual harassment is a hostile form of gender bias that can cause anxiety, depression, and significant mental health issues. Virtual reality (VR) has been employed to help make people become aware of their biases and change their attitudes regarding gender, race, and age.

METHODS

Forty participants were embodied in avatars of different genders and experienced sexual harassment scenarios in VR. A gender Implicit Association Test (IAT) was administered before and after the experience.

RESULTS

There was a statistically significant main effect of participant gender (F (1,36) = 10.67, p = .002, partial η2 = .23) on ΔIAT, where males and females reported a decrease (M = -.12, SD = .24) and an increase (M = .10, SD = .25) in IAT scores, respectively. A statistically significant two-way interaction between gender transfer and participant gender was revealed (F (1,36) = 6.32, p = .02, partial η2 = .15). There was a significant simple effect of gender transfer for male participants (F (1,36) = 8.70, p = .006, partial η2 = .19).

CONCLUSIONS

Implicit gender bias can be modified, at least temporarily, through embodiment in VR. Gender transfer through embodiment while encountering different sexual harassment scenarios helped reduce implicit gender bias. There was a tendency for individuals to increase bias for the gender of the avatar in which they embodied.

APPLICATIONS

The current research provided promising evidence that a virtual environment system may be used as a potential training tool to improve implicit gender bias.

Induced oscillatory brain responses under virtual reality conditions in the context of repetition priming

In the human electroencephalogram (EEG), induced oscillatory responses in various frequency bands are regarded as valuable indices to examine the neural mechanisms underlying human memory. While the advent of virtual reality (VR) drives the investigation of mnemonic processing under more lifelike settings, the joint application of VR and EEG methods is still in its infancy (e.g., due to technical limitations impeding the signal acquisition). The objective of the present EEG study was twofold. First, we examined whether the investigation of induced oscillations under VR conditions yields equivalent results compared to standard paradigms. Second, we aimed at obtaining further insights into basic memory-related brain mechanisms in VR. To these ends, we relied on a standard implicit memory design, namely repetition priming, for which the to-be-expected effects are well-documented for conventional studies. Congruently, we replicated a suppression of the evoked potential after stimulus onset. Regarding the induced responses, we observed a modulation of induced alphaband in response to a repeated stimulus. Importantly, our results revealed a repetition-related suppression of the high-frequency induced gammaband response (>30 Hz), indicating the sharpening of a cortical object representation fostering behavioral priming effects. Noteworthy, the analysis of the induced gammaband responses required a number of measures to minimize the influence of external and internal sources of artefacts (i.e., the electrical shielding of the technical equipment and the control for miniature eye movements). In conclusion, joint VR-EEG studies with a particular focus on induced oscillatory responses offer a promising advanced understanding of mnemonic processing under lifelike conditions.

Virtual reality roleplays for patients with depression: A user experience evaluation

Background

Virtual reality (VR) has been used successfully and effectively in psychotherapy for a variety of disorders. In the field of depression, there are only a few VR interventions and approaches. Although simple social interactions have been successfully modeled in VR for several mental disorders, there has been no transfer to the field of depression therapy. VR may be employed for psychodynamic psychotherapy to work on interpersonal conflict patterns. In this study, we developed and evaluated a VR intervention for the simulation of roleplay situations in the context of supportive-expressive therapy.

Methods

We conducted a clinical user experience (UX) study at a psychotherapeutic clinic in Düsseldorf, Germany. Eight inpatients with depression and four therapists were included. Semi-structured interviews and qualitative content analysis were used to identify UX issues of the developed VR intervention. Usability questionnaires and technical usage data were also considered. The VR intervention consisted of two therapist-controlled roleplay scenarios designed to support work on the core conflictual relationship theme by allowing patients to interact in typical problematic social situations. Recorded VR roleplays allow for therapeutic debriefing with a change of perspective. Therapists were given the option of using the roleplay in multiple sessions.

Results

All therapists conducted one session per patient with the VR intervention. From the patient interviews, 26 UX issues were extracted, of which one technical malfunction and two unclarities in the interaction with the VR agent were rated as major problems. From the therapist interviews, 14 UX issues were extracted, of which five were rated as major problems related to the interface in the dialog control or the complex system setup.

Conclusion

The main problem was designing a dialog structure that allows both complex conversational flows and a clear control interface. In principle, VR roleplays could be integrated well and safely into therapy. The VR intervention shows promise for providing an emotional experience of interpersonal conflict patterns in the context of psychotherapy. Additionally, other roleplay situations involving various social problem areas must be created and evaluated in terms of the fit to the patients' core conflictual relationship themes.

An attention-based approach for assessing the effectiveness of emotion-evoking in immersive environment

Visual stimuli within an immersive virtual environment impact human perception and behavior in notably different ways compared to the real world. Previous studies have presented evidence indicating that individuals in various emotional states exhibit an unconscious attentional bias toward either positive or negative stimuli. However, whether these findings can be replicated within an immersive virtual environment remains uncertain. In this study, we devised an attention-based experiment to explore whether the correlation between participants' emotional states and the valence of visual stimuli influences their attentional bias. Participants (n=28) viewed 360-degree videos with varying valence levels (positive and negative) to evoke emotions. Subsequently, we utilized standard emotional human faces as stimuli to assess how the consistency in video valence and emotional faces affects reaction time (RT) in Go tasks and error rates in No-go tasks. We employed the Ex-Gaussian approach to analyze the RT data. The parameters-mu (μ), sigma (σ), and tau (τ)-were computed to denote response speed and attentional lapses, respectively. Our findings revealed a significant increase in tau (τ) when the valence of the video and emotional faces aligned. This suggests that the Go/No-go paradigm is effective in evaluating the impact of emotion-evoking stimuli within an immersive environment.

Sensory conflicts through short, discrete visual input manipulations: Identification of balance responses to varied input characteristics

Human balance control relies on various sensory modalities, and conflict of sensory input may result in postural instability. Virtual reality (VR) technology allows to train balance under conflicting sensory information by decoupling visual from somatosensory and vestibular systems, creating additional demands on sensory reweighting for balance control. However, there is no metric for the design of visual input manipulations that can induce persistent sensory conflicts to perturb balance. This limits the possibilities to generate sustained sensory reweighting processes and design well-defined training approaches. This study aimed to investigate the effects that different onset characteristics, amplitudes and velocities of visual input manipulations may have on balance control and their ability to create persistent balance responses. Twenty-four young adults were recruited for the study. The VR was provided using a state-of-the-art head-mounted display and balance was challenged in two experiments by rotations of the visual scene in the frontal plane with scaled constellations of trajectories, amplitudes and velocities. Mean center of pressure speed was recorded and revealed to be greater when the visual input manipulation had an abrupt onset compared to a smooth onset. Furthermore, the balance response was greatest and most persistent when stimulus velocity was low and stimulus amplitude was large. These findings show clear dissociation in the state of the postural system for abrupt and smooth visual manipulation onsets with no indication of short-term adaption to abrupt manipulations with slow stimulus velocity. This augments our understanding of how conflicting visual information affect balance responses and could help to optimize the conceptualization of training and rehabilitation interventions.

Multifactorial stress reactivity to virtual TSST-C in healthy children and adolescents-It works, but not as well as a real TSST-C

A virtual version of the Trier Social Stress Test (TSST) has been increasingly used in stress research. Benefits of the virtual TSST include that it is more economical, and offers improved control of context and enhanced flexibility of use. Many studies have confirmed the applicability of virtual TSSTs in stress research, but only in adulthood. In the present study, we aimed to experimentally verify the transferability of a virtual TSST to adolescence. A total of 73 healthy adolescents (aged 11-17 years) of both sexes completed either a real (IV-TSST-C) or virtual (VR-TSST-C) TSST for children. The surveyed stress parameters included salivary cortisol and alpha amylase concentrations, heart rate, heart rate variability, and subjective stress ratings across test sessions, as well as a pre- to post-TSST-C comparison. All parameters revealed significant stress responses over time. We observed significant effects of group on cortisol and subjective stress ratings, with the VR-TSST-C inducing less stress than the IV-TSST-C. In alpha amylase, heart rate, and heart rate variability, we found no significant difference between TSST versions, but only significant equivalence for alpha amylase. Some parameters may have been influenced by an expectancy effect and the participant's sex. In conclusion, among adolescents, a virtual TSST-C effectively elicits stress at multiple levels (endocrinological, autonomic, and subjective); however, the magnitude is not always comparable to with the real TSST-C, which should be considered when applied.

Social identification and risk dynamics: How perceptions of (inter)personal and collective risk impact the adoption of COVID-19 preventative behaviors

Public adoption of preventative behaviors to reduce the transmission of COVID-19 is crucial to managing the pandemic, and so it is vital to determine what factors influence the uptake of those behaviors. Previous studies have identified COVID-19 risk perceptions as a key factor, but this work has typically been limited both in assuming that risk means risk to the personal self, and in being reliant on self-reported data. Drawing on the social identity approach, we conducted two online studies in which we investigated the effects of two different types of risk on preventative measure taking: risk to the personal self and risk to the collective self (i.e., members of a group with which one identifies). Both studies involved behavioral measures using innovative interactive tasks. In Study 1 (n = 199; data collected 27 May 2021), we investigated the effects of (inter)personal and collective risk on physical distancing. In Study 2 (n = 553; data collected 20 September 2021), we investigated the effects of (inter)personal and collective risk on the speed at which tests are booked as COVID-19 symptoms develop. In both studies, we find that perceptions of collective risk, but not perceptions of (inter)personal risk, influence the extent to which preventative measures are adopted. We discuss the implications both conceptually (as they relate to both the conceptualization of risk and social identity processes) and also practically (in terms of the implications for public health communications).

Conversational facial signals combine into compositional meanings that change the interpretation of speaker intentions

Human language is extremely versatile, combining a limited set of signals in an unlimited number of ways. However, it is unknown whether conversational visual signals feed into the composite utterances with which speakers communicate their intentions. We assessed whether different combinations of visual signals lead to different intent interpretations of the same spoken utterance. Participants viewed a virtual avatar uttering spoken questions while producing single visual signals (i.e., head turn, head tilt, eyebrow raise) or combinations of these signals. After each video, participants classified the communicative intention behind the question. We found that composite utterances combining several visual signals conveyed different meaning compared to utterances accompanied by the single visual signals. However, responses to combinations of signals were more similar to the responses to related, rather than unrelated, individual signals, indicating a consistent influence of the individual visual signals on the whole. This study therefore provides first evidence for compositional, non-additive (i.e., Gestalt-like) perception of multimodal language.

Reliability of repeated exposure to the human elevated plus-maze in virtual reality: Behavioral, emotional, and autonomic responses

Approach-avoidance conflicts are a hallmark of anxiety-related behaviors. A gold standard for assessing anxiety-related behaviors in rodents is the elevated plus-maze (EPM), which was recently translated to humans using immersive virtual reality. Repeated behavioral testing is particularly interesting for clinical and pharmacological research in humans but could be limited by habituation effects. Here, we tested whether comparable strategies that are used in rodents (different environments and inter-trial interval of 28 days) are sufficient to avoid habituation or sensitization effects on the EPM, making it possible to perform repeated measurement of anxiety-related behavior in humans. Moreover, we developed two novel virtual environments for repeated testing to explore whether a scenario resembling the real world is superior to a video game-like EPM in terms of lifelike physiological, emotional, and behavioral responses. On a behavioral level, no significant differences but a high correlation between first and repeated exposure to the human EPM independent of EPM version were found. On a psychophysiological level, salivary alpha-amylase, skin-conductance, and respiratory frequency increased at first and second exposure independent of EPM version. However, at repeated exposure, skin-conductance and heart rate showed indicators for anticipatory anxiety and a small sensitization effect, while no effect of real-world resemblance on these physiological measures was found. This was also reflected in slightly higher subjective anxiety levels at second exposure, although subjective anxiety still correlated strongly between first and second exposure. In conclusion, the human EPM can be used for longitudinal assessments of human anxiety-related behavior when strategies to avoid habituation and sensitization are considered.

The Jena Eyewitness Research Stimuli (JERS): A database of mock theft videos involving two perpetrators, presented in 2D and VR formats with corresponding 2D and 3D lineup images

Empirical investigations into eyewitness identification accuracy typically necessitate the creation of novel stimulus materials, which can be a challenging and time-consuming task. To facilitate this process and promote further research in this domain, we introduce the new Jena Eyewitness Research Stimuli (JERS). They comprise six video sequences depicting a mock theft committed by two different perpetrators, available in both two-dimensional (2D) and 360° format, combined with the corresponding lineup images presented in 2D or three-dimensional (3D) format. Images of one suspect and eight fillers are available for each lineup. We evaluated lineup fairness by using mock eyewitness paradigm and noted a Tredoux's E of 4.687 for Perpetrator 1 and 5.406 for Perpetrator 2. Moreover, no bias towards the perpetrators was observed in the lineups. We incorporated 360° videos and 3D lineup images to encourage the adoption of innovative data formats in experimental investigations of eyewitness accuracy. In particular, compatibility with Virtual Reality (VR) makes JERS a promising tool for advancing eyewitness research by enabling researchers to construct controlled environments that offer observers an immersive experience. JERS is freely accessible for the use of academic purposes via the Open Science Framework (OSF).

Virtual reality-assisted assessment of paranoid ideation in forensic psychiatric inpatients: A mixed-methods pilot study

Background

Reliable and valid assessment of paranoia is important in forensic psychiatry for providing adequate care. VR technology may add to current assessment procedures, as it enables observation within realistic (social) situations resembling the complexity of everyday life. VR constitutes a promising tool within forensics, due to the restricted nature of forensic psychiatric hospitals and ethical challenges arising from observing potentially dangerous behaviors in real life.

Objective

To investigate the feasibility of VR assessment for paranoid ideation in forensic psychiatric inpatients qualitatively by assessing the experiences of patients and a clinician, and to explore how the VR measures relate to established clinical measures.

Methods

One clinician (experienced psychiatrist) and 10 forensic psychiatric inpatients with a history or suspicion of paranoid ideation were included. Patients participated in two immersive VR scenarios (bus and supermarket) during which paranoia was assessed by the clinician. Qualitative interviews were performed with patients and the clinician performing the assessment to investigate experiences and feasibility. Further, measures of paranoia, social anxiety, and positive symptoms were obtained.

Results

Nine out of 10 participants with varying levels of paranoid ideation completed the assessment. Manifest inductive content analyses of the interviews revealed general experiences, advantages such as enabling observing participants from a different perspective, and challenges of the VR assessment, such as a lack of objectivity and the laboriousness of the assessment for the clinician. Although more paranoia was experienced during the supermarket scenario, correlates with classical measures were only significant for the bus scenario.

Discussion

The VR assessment was appreciated by most patients and the clinician. Based on our results short, standardized VR assessment scenarios are feasible, however, they do not appear reliable or objective for assessing paranoia. The clinical usefulness is most likely as a collaborative tool and add-on measure to existing methods.

Conversational Eyebrow Frowns Facilitate Question Identification: An Online Study Using Virtual Avatars

Conversation is a time-pressured environment. Recognizing a social action (the ''speech act,'' such as a question requesting information) early is crucial in conversation to quickly understand the intended message and plan a timely response. Fast turns between interlocutors are especially relevant for responses to questions since a long gap may be meaningful by itself. Human language is multimodal, involving speech as well as visual signals from the body, including the face. But little is known about how conversational facial signals contribute to the communication of social actions. Some of the most prominent facial signals in conversation are eyebrow movements. Previous studies found links between eyebrow movements and questions, suggesting that these facial signals could contribute to the rapid recognition of questions. Therefore, we aimed to investigate whether early eyebrow movements (eyebrow frown or raise vs. no eyebrow movement) facilitate question identification. Participants were instructed to view videos of avatars where the presence of eyebrow movements accompanying questions was manipulated. Their task was to indicate whether the utterance was a question or a statement as accurately and quickly as possible. Data were collected using the online testing platform Gorilla. Results showed higher accuracies and faster response times for questions with eyebrow frowns, suggesting a facilitative role of eyebrow frowns for question identification. This means that facial signals can critically contribute to the communication of social actions in conversation by signaling social action-specific visual information and providing visual cues to speakers' intentions.

Situational social influence leading to non-compliance with conservation rules

It is well established that the decisions that we make can be strongly influenced by the behaviour of others. However, testing how social influence can lead to non-compliance with conservation rules during an individual's decision-making process has received little research attention. We synthesise advances in understanding of conformity and rule-breaking in individuals and in groups, and take a situational approach to studying the social dynamics and ensuing social identity changes that can lead to non-compliant decision-making. We focus on situational social influence contagion that are copresent (i.e., same space and same time) or trace-based (i.e., behavioural traces in the same space). We then suggest approaches for testing how situational social influence can lead to certain behaviours in non-compliance with conservation rules.

Measuring trust with the Wayfinding Task: Implementing a novel task in immersive virtual reality and desktop setups across remote and in-person test environments

Trust is a key feature of social relationships. Common measures of trust, questionnaires and economic games, lack ecological validity. Hence, we sought to introduce an immersive, virtual reality (VR) measure for the behavioral assessment of trust across remote and in-person settings, building on the maze task of Hale et al. (2018). Our 'Wayfinding Task' consists of an interconnected urban environment for participants to navigate on the advice of two characters of differing trustworthiness. We present four studies implementing the Wayfinding Task in remote and in-person testing environments and comparing performance across head-mounted display (HMD)-based VR and desktop setups. In each study, the trustworthiness of two virtual characters was manipulated, through either a fact sheet providing trustworthiness information, or a behavior-based trustworthiness manipulation task termed the Door Game, based on Van der Biest et al., 2020. Participants then completed the Wayfinding Task. Overall, we found that participant behavior in the Wayfinding Task reflected the relative trustworthiness of the two characters; in particular, the trustworthy character was approached more often for advice, reflecting data from our Door Game. We found mostly null results for our novel outcome measure, interpersonal distance. Remote testing successfully achieved these effects. While HMD-based VR and desktop setups both showed these effects, there was a stronger effect of trustworthiness in the HMD VR version of the task. These results have implications for the measurement of trust in behavioral settings and the use of remote and VR-based testing in social experiments.

Toward parallel intelligence: An interdisciplinary solution for complex systems

The growing complexity of real-world systems necessitates interdisciplinary solutions to confront myriad challenges in modeling, analysis, management, and control. To meet these demands, the parallel systems method rooted in the artificial systems, computational experiments, and parallel execution (ACP) approach has been developed. The method cultivates a cycle termed parallel intelligence, which iteratively creates data, acquires knowledge, and refines the actual system. Over the past two decades, the parallel systems method has continuously woven advanced knowledge and technologies from various disciplines, offering versatile interdisciplinary solutions for complex systems across diverse fields. This review explores the origins and fundamental concepts of the parallel systems method, showcasing its accomplishments as a diverse array of parallel technologies and applications while also prognosticating potential challenges. We posit that this method will considerably augment sustainable development while enhancing interdisciplinary communication and cooperation.

What's your point? Insights from virtual reality on the relation between intention and action in the production of pointing gestures

Human communication involves the process of translating intentions into communicative actions. But how exactly do our intentions surface in the visible communicative behavior we display? Here we focus on pointing gestures, a fundamental building block of everyday communication, and investigate whether and how different types of underlying intent modulate the kinematics of the pointing hand and the brain activity preceding the gestural movement. In a dynamic virtual reality environment, participants pointed at a referent to either share attention with their addressee, inform their addressee, or get their addressee to perform an action. Behaviorally, it was observed that these different underlying intentions modulated how long participants kept their arm and finger still, both prior to starting the movement and when keeping their pointing hand in apex position. In early planning stages, a neurophysiological distinction was observed between a gesture that is used to share attitudes and knowledge with another person versus a gesture that mainly uses that person as a means to perform an action. Together, these findings suggest that our intentions influence our actions from the earliest neurophysiological planning stages to the kinematic endpoint of the movement itself.

Keeping track of reality: embedding visual memory in natural behaviour

Since immersive virtual reality (IVR) emerged as a research method in the 1980s, the focus has been on the similarities between IVR and actual reality. In this vein, it has been suggested that IVR methodology might fill the gap between laboratory studies and real life. IVR allows for high internal validity (i.e., a high degree of experimental control and experimental replicability), as well as high external validity by letting participants engage with the environment in an almost natural manner. Despite internal validity being crucial to experimental designs, external validity also matters in terms of the generalizability of results. In this paper, we first highlight and summarise the similarities and differences between IVR, desktop situations (both non-immersive VR and computer experiments), and reality. In the second step, we propose that IVR is a promising tool for visual memory research in terms of investigating the representation of visual information embedded in natural behaviour. We encourage researchers to carry out experiments on both two-dimensional computer screens and in immersive virtual environments to investigate visual memory and validate and replicate the findings. IVR is valuable because of its potential to improve theoretical understanding and increase the psychological relevance of the findings.

Differences in configural processing for human versus android dynamic facial expressions

Humanlike androids can function as social agents in social situations and in experimental research. While some androids can imitate facial emotion expressions, it is unclear whether their expressions tap the same processing mechanisms utilized in human expression processing, for example configural processing. In this study, the effects of global inversion and asynchrony between facial features as configuration manipulations were compared in android and human dynamic emotion expressions. Seventy-five participants rated (1) angry and happy emotion recognition and (2) arousal and valence ratings of upright or inverted, synchronous or asynchronous, android or human agent dynamic emotion expressions. Asynchrony in dynamic expressions significantly decreased all ratings (except valence in angry expressions) in all human expressions, but did not affect android expressions. Inversion did not affect any measures regardless of agent type. These results suggest that dynamic facial expressions are processed in a synchrony-based configural manner for humans, but not for androids.

A modularity design approach to behavioral research with immersive virtual reality: A SkyrimVR-based behavioral experimental framework

Virtual reality (VR) has been shown to be a potential research tool, yet the gap between traditional and VR behavioral experiment systems poses a challenge for many behavioral researchers. To address the challenge posed, the present study first adopted a modularity design strategy and proposed a five-module architectural framework for a VR behavioral experiment system that aimed to reduce complexity and costs of development. Applying the five-module architectural framework, the present study developed the SkyrimVR-based behavioral experimental framework (SkyBXF) module, a basic experimental framework module that adopted and integrated the classic human behavior experiment structure (i.e., session-block-trial model) with the modifiable VR massive gaming franchise The Elder Scrolls V: Skyrim VR. A modified version of previous behavioral research to investigate the effects of masked peripheral vision on visually-induced motion sickness in an immersive virtual environment was conducted as a proof of concept to showcase the feasibility of the proposed five-module architectural framework and the SkyBXF module developed. Behavioral data acquired through the case study were consistent with those from previous behavioral research. This indicates the viability of the proposed five-module architectural framework and the SkyBXF module developed, and provides proof that future behavioral researchers with minimal programming proficiency, 3D environment development expertise, time, personnel, and resources may reuse ready-to-go resources and behavioral experiment templates offered by SkyBXF to swiftly establish realistic virtual worlds that can be further customized for experimental need on the go.

These Aren't The Droids You Are Looking for: Promises and Challenges for the Intersection of Affective Science and Robotics/AI

AI research focused on interactions with humans, particularly in the form of robots or virtual agents, has expanded in the last two decades to include concepts related to affective processes. Affective computing is an emerging field that deals with issues such as how the diagnosis of affective states of users can be used to improve such interactions, also with a view to demonstrate affective behavior towards the user. This type of research often is based on two beliefs: (1) artificial emotional intelligence will improve human computer interaction (or more specifically human robot interaction), and (2) we understand the role of affective behavior in human interaction sufficiently to tell artificial systems what to do. However, within affective science the focus of research is often to test a particular assumption, such as "smiles affect liking." Such focus does not provide the information necessary to synthesize affective behavior in long dynamic and real-time interactions. In consequence, theories do not play a large role in the development of artificial affective systems by engineers, but self-learning systems develop their behavior out of large corpora of recorded interactions. The status quo is characterized by measurement issues, theoretical lacunae regarding prevalence and functions of affective behavior in interaction, and underpowered studies that cannot provide the solid empirical foundation for further theoretical developments. This contribution will highlight some of these challenges and point towards next steps to create a rapprochement between engineers and affective scientists with a view to improving theory and solid applications.

A Randomized Clinical Trial to Evaluate a Digital Therapeutic to Enhance Gait Function in Individuals With Parkinson's Disease

BACKGROUND

Postural instability and gait dysfunction (PIGD) is a cardinal symptom of Parkinson's disease (PD) and is exacerbated under dual-task conditions. Dual-task training (DTT), enhances gait performance, however it is time and cost intensive. Digitizing DTT via the Dual-task Augmented Reality Treatment (DART) platform can expand the availability of an effective intervention to address PIGD.

OBJECTIVE

The aim of this project was to evaluate DART in the treatment of PIGD in people with PD compared to a Traditional DTT intervention. It was hypothesized that both groups would exhibit significant improvements in gait, and the improvements for the DART group would be non-inferior to Traditional DTT.

METHODS

A single-blind randomized controlled trial was conducted with 47 PD participants with PIGD. Both groups completed 16 therapeutic sessions over 8 weeks; the DART platform delivered DTT via the Microsoft HoloLens2. Primary outcomes included clinical ratings and single- and dual-task gait biomechanical outcomes.

RESULTS

Clinical measures of PD symptoms remained stable for DART and Traditional DTT groups. However, both groups exhibited a significant increase in gait velocity, cadence, and step length during single- and multiple dual-task conditions following the interventions. Improvements in gait velocity in the DART group were non-inferior to Traditional DTT under the majority of conditions.

CONCLUSION

Non-inferior improvements in gait parameters across groups provides evidence of the DART platform being an effective digital therapeutic capable of improving PIGD. Effective digital delivery of DTT has the potential to increase use and accessibility to a promising, yet underutilized and difficult to administer, intervention for PIGD.

CLINICAL TRIAL REGISTRATION

ClinicalTrials.gov Dual-task Augmented Reality Treatment for Parkinson's Disease (DART) NCT04634331; posted November 18, 2020.

The inversion effect on the cubic humanness-uncanniness relation in humanlike agents

The uncanny valley describes the typically nonlinear relation between the esthetic appeal of artificial entities and their human likeness. The effect has been attributed to specialized (configural) processing that increases sensitivity to deviations from human norms. We investigate this effect in computer-generated, humanlike android and human faces using dynamic facial expressions. Angry and happy expressions with varying degrees of synchrony were presented upright and inverted and rated on their eeriness, strangeness, and human likeness. A sigmoidal function of human likeness and uncanniness ("uncanny slope") was found for upright expressions and a linear relation for inverted faces. While the function is not indicative of an uncanny valley, the results support the view that configural processing moderates the effect of human likeness on uncanniness and extend its role to dynamic facial expressions.

Biological postural oscillations during facial expression of pain in virtual characters modulate early and late ERP components associated with empathy: A pilot study

There is a surge in the use of virtual characters in cognitive sciences. However, their behavioural realism remains to be perfected in order to trigger more spontaneous and socially expected reactions in users. It was recently shown that biological postural oscillations (idle motion) were a key ingredient to enhance the empathic response to its facial pain expression. The objective of this study was to examine, using electroencephalography, whether idle motion would modulate the neural response associated with empathy when viewing a pain-expressing virtual character. Twenty healthy young adults were shown video clips of a virtual character displaying a facial expression of pain while its body was either static (Still condition) or animated with pre-recorded human postural oscillations (Idle condition). Participants rated the virtual human's facial expression of pain as significantly more intense in the Idle condition compared to the Still condition. Both the early (N2-N3) and the late (rLPP) event-related potentials (ERPs) associated with distinct dimensions of empathy, affective resonance and perspective-taking, respectively, were greater in the Idle condition compared to the Still condition. These findings confirm the potential of idle motion to increase empathy for pain expressed by virtual characters. They are discussed in line with contemporary empathy models in relation to human-machine interactions.

Ecological and Convergent Validity of Experimentally and Dynamically Assessed Capacity for Social Contingency Detection Using the Perceptual Crossing Experiment in Adolescence

The Perceptual Crossing Experiment (PCE) captures the capacity for social contingency detection using real-time social interaction dynamics but has not been externally validated. We tested ecological and convergent validity of the PCE in a sample of 208 adolescents from the general population, aged 11 to 19 years. We expected associations between PCE performance and (a) quantity and quality of social interaction in daily life, using Experience Sampling Methodology (ESM; ecological validity) and (b) self-reported social skills using a questionnaire (convergent validity). We also expected PCE performance to better explain variance in ESM social measures than self-reported social skills. Multilevel analyses showed that only self-reported social skills were positively associated with social experience of company in daily life. These initial results do not support ecological and convergent validity of the PCE. However, fueled by novel insights regarding the complexity of capturing social dynamics, we identified promising methodological advances for future validation efforts.

Learning Experience Design of Project PHoENIX: Addressing the Lack of Autistic Representation in Extended Reality Design and Development

This paper presents Project PHoENIX, which stands for Participatory, Human-centered, Equitable, Neurodiverse, Inclusive, eXtended reality. The project aims to co-produce research with autistic users to create a virtual reality (VR) environment that is highly usable, accessible, and sensitive to the needs and preferences of these individuals. Project PHoENIX utilizes participatory design within a learning experience design (LXD) frame to locate autistic people, their caregivers, and providers centrally in the processes of immersive technology design and development, as well as research design and execution. An overarching literature review on VR and autism and issues of limited design precedent of VR environments with autistic participants is provided, as well as details on the Project PHoENIX design framework, project description, and project design outcomes. Details are provided on how the online VR environment was co-designed and co-developed through collaborative research with autistic stakeholders while being sensitive to their needs and preferences. Research findings and implications are discussed regarding the design process, constraints, principles, and insights. The paper concludes by discussing lessons learned and how this project can provide much-needed design precedent for advancing the field towards a more inclusive, human-centered, and neurodiverse VR research and development paradigms.

The promise and peril of interactive embodied agents for studying non-verbal communication: a machine learning perspective

In face-to-face interactions, parties rapidly react and adapt to each other's words, movements and expressions. Any science of face-to-face interaction must develop approaches to hypothesize and rigorously test mechanisms that explain such interdependent behaviour. Yet conventional experimental designs often sacrifice interactivity to establish experimental control. Interactive virtual and robotic agents have been offered as a way to study true interactivity while enforcing a measure of experimental control by allowing participants to interact with realistic but carefully controlled partners. But as researchers increasingly turn to machine learning to add realism to such agents, they may unintentionally distort the very interactivity they seek to illuminate, particularly when investigating the role of non-verbal signals such as emotion or active-listening behaviours. Here I discuss some of the methodological challenges that may arise when machine learning is used to model the behaviour of interaction partners. By articulating and explicitly considering these commitments, researchers can transform 'unintentional distortions' into valuable methodological tools that yield new insights and better contextualize existing experimental findings that rely on learning technology. This article is part of a discussion meeting issue 'Face2face: advancing the science of social interaction'.

Real-life relevant face perception is not captured by the N170 but reflected in later potentials: A comparison of 2D and virtual reality stimuli

The perception of faces is one of the most specialized visual processes in the human brain and has been investigated by means of the early event-related potential component N170. However, face perception has mostly been studied in the conventional laboratory, i.e., monitor setups, offering rather distal presentation of faces as planar 2D-images. Increasing spatial proximity through Virtual Reality (VR) allows to present 3D, real-life-sized persons at personal distance to participants, thus creating a feeling of social involvement and adding a self-relevant value to the presented faces. The present study compared the perception of persons under conventional laboratory conditions (PC) with realistic conditions in VR. Paralleling standard designs, pictures of unknown persons and standard control images were presented in a PC- and a VR-modality. To investigate how the mechanisms of face perception differ under realistic conditions from those under conventional laboratory conditions, the typical face-specific N170 and subsequent components were analyzed in both modalities. Consistent with previous laboratory research, the N170 lost discriminatory power when translated to realistic conditions, as it only discriminated faces and controls under laboratory conditions. Most interestingly, analysis of the later component [230–420 ms] revealed more differentiated face-specific processing in VR, as indicated by distinctive, stimulus-specific topographies. Complemented by source analysis, the results on later latencies show that face-specific neural mechanisms are applied only under realistic conditions (A video abstract is available in the Supplementary material and via YouTube: https://youtu.be/TF8wiPUrpSY).

A predictive model for understanding the role of emotion for the formation of presence in virtual reality

Users' emotions may influence the formation of presence in virtual reality (VR). Users' expectations, state of arousal and personality may also moderate the relationship between emotions and presence. An interoceptive predictive coding model of conscious presence (IPCM) considers presence as a product of the match between predictions of interoceptive emotional states and the actual states evoked by an experience (Seth et al. 2012). The present paper aims to test this model's applicability to VR for both high-arousal and low-arousal emotions. The moderating effect of personality traits on the creation of presence is also investigated. Results show that user expectations about emotional states in VR have an impact on presence, however, expression of this relationship is moderated by the intensity of an emotion, with only high-arousal emotions showing an effect. Additionally, users' personality traits moderated the relationship between emotions and presence. A refined model is proposed that predicts presence in VR by weighting emotions according to their level of arousal and by considering the impact of personality traits.

Using 360-degree immersive videos to assess multiple transdiagnostic symptoms: A study focusing on fear of negative evaluation, paranoid thoughts, negative automatic thoughts, and craving

Over the last 20 years, virtual reality (VR) has gained a great interest for both assessment and treatment of various psychopathologies. However, due to high costs and material specificity, VR remains disadvantageous for clinicians. Adopting a multiple transdiagnostic approach, this study aims at testing the validity of a 360-degree immersive video (360IV) for the assessment of five common psychological symptoms (fear of negative evaluation, paranoid thoughts, negative automatic thoughts, craving for alcohol and for nicotine). A 360IV was constructed in the Darius Café and included actors behaving naturally. One hundred and fifty-eight adults from the general population were assessed in terms of their proneness towards the five symptoms, were then exposed to the 360IV and completed measures for the five state symptoms, four dimensions of presence (place, plausibility, copresence and social presence illusions) and cybersickness. Results revealed that the five symptoms occurred during the immersion and were predicted by the participants' proneness towards these symptoms. The 360IV was also able to elicit various levels of the four dimensions of presence while producing few cybersickness. The present study provides evidence supporting the use of the 360IV as a new accessible, ecological, and standardized tool to assess multiple transdiagnostic symptoms.

Supplementary Information

The online version contains supplementary material available at 10.1007/s10055-023-00779-y.

Psychedelic replications in virtual reality and their potential as a therapeutic instrument: an open-label feasibility study

Background

Recent research has shown promising results for the therapeutic benefits of psychedelics. One popular view claims that these benefits are mediated by the subjective experiences induced by these substances. Based on this, we designed a virtual reality experience, Psyrreal, that mimics the phenomenological components of psychedelic experiences.

Aims

We aimed to investigate the therapeutic efficacy of Psyrreal and psychedelic VR experiences in treating depressive symptoms as well as explore the effect of Psyrreal on subjective factors which have been suggested to mediate the therapeutic benefits of psychedelics.

Methods

In this open-label feasibility study, thirteen participants with mild-to-moderate depression underwent a 2-day therapeutic intervention implementing Psyrreal. Depressive symptoms were evaluated by the Emotional State Questionnaire (EST-Q2) at the start of the intervention and 2 weeks after. A thematic analysis of semi-structured interviews after Psyrreal was also conducted as an additional assessment of the method.

Results

A 2-day intervention implementing Psyrreal led to significant decreases in depressive symptoms at the 2-week follow-up (n = 10, p = 0.007, Hedges' g = 1.046) measured by the Emotional State Questionnaire (EST-Q2). The analysis of semi-structured interviews suggests that Psyrreal could lead to insight and alterations in the sense of self in some people.

Conclusion

This work proposes a novel method using virtual reality to augment the treatment of psychological disorders as well as to precisely investigate the mediating subjective factors of the therapeutic effects of psychedelic substances. Our preliminary results suggest that VR experiences combined with psychological support show potential in treating depressive symptoms and further research into similar methods is warranted.

Examining the impact of experiencing auditory verbal hallucinations from a first-person perspective on the degree of empathy and stigmatization in a group of psychology students: A study using 360° immersive videos

BACKGROUND

Stigmatization toward psychosis is persistent among mental health professionals (MHPs) and negatively impacts the patients' outcomes. One suggested way of reducing stigmatization is to expose MHPs to simulations of psychotic symptoms. This approach has been associated with an increase in empathy, but also with an increase in the desire of social distance. The addition of an empathic task (ET) has been suggested to neutralize this effect on social distance. The present study aims to (1) examine the effect of a remotely administered 360° immersive video (360IV) simulation on empathy and stigma among psychology students and (2) replicate the neutralizing effect of an ET on social distance. Finally, the potential role of immersive properties on changes will also be explored.

METHODS

A 360IV simulating auditory hallucinations was constructed in collaboration with patient partners. 121 psychology students were allocated to one of three conditions: (i) exposure to the 360IV, (ii) exposure to the 360IV and to an ET (360IV + ET), and (iii) no exposure (control). Measures of empathy and stigma (stereotypes and social distance) were collected before and after the interventions.

RESULTS

An increase of empathy was observed in the 360IV and 360IV + ET conditions compared to the control condition. There was an increase of stereotypes in all conditions and no effect on social distance.

CONCLUSIONS

This study confirms the effectiveness of a 360IV simulation intervention in increasing empathy in psychology students but questions its efficacy in reducing stigma.

Action recognition based on multimode fusion for VR online platform

The current popular online communication platforms can convey information only in the form of text, voice, pictures, and other electronic means. The richness and reliability of information is not comparable to traditional face-to-face communication. The use of virtual reality (VR) technology for online communication is a viable alternative to face-to-face communication. In the current VR online communication platform, users are in a virtual world in the form of avatars, which can achieve "face-to-face" communication to a certain extent. However, the actions of the avatar do not follow the user, which makes the communication process less realistic. Decision-makers need to make decisions based on the behavior of VR users, but there are no effective methods for action data collection in VR environments. In our work, three modalities of nine actions from VR users are collected using a virtual reality head-mounted display (VR HMD) built-in sensors, RGB cameras and human pose estimation. Using these data and advanced multimodal fusion action recognition networks, we obtained a high accuracy action recognition model. In addition, we take advantage of the VR HMD to collect 3D position data and design a 2D key point augmentation scheme for VR users. Using the augmented 2D key point data and VR HMD sensor data, we can train action recognition models with high accuracy and strong stability. In data collection and experimental work, we focus our research on classroom scenes, and the results can be extended to other scenes.

The limitations and challenges in the assessment of executive dysfunction associated with real-world functioning: The opportunity of serious games

Nowadays, there is a broad range of methods for detecting and evaluating executive dysfunction ranging from clinical interview to neuropsychological evaluation. Nevertheless, a critical issue of these assessments is the lack of correspondence of the neuropsychological test's results with real-world functioning. This paper proposes serious games as a new framework to improve the neuropsychological assessment of real-world functioning. We briefly discuss the contribution and limitations of current methods of evaluation of executive dysfunction (paper-and-pencil tests, naturalistic observation methods, and Information and Communications Technologies) to inform on daily life functioning. Then, we analyze what are the limitations of these methods to predict real-world performance: (1) A lack of appropriate instruments to investigate the complexity of real-world functioning, (2) the vast majority of neuropsychological tests assess well-structured tasks, and (3) measurement of behaviors are based on simplistic data collection and statistical analysis. This work shows how serious games offer an opportunity to develop more efficient tools to detect executive dysfunction in everyday life contexts. Serious games provide meaningful narrative stories and virtual or real environments that immerse the user in natural and social environments with social interactions. In those highly interactive game environments, the player needs to adapt his/her behavioral performance to novel and ill-structured tasks which are suited for collecting user interaction evidence. Serious games offer a novel opportunity to develop better tools to improve diagnosis of the executive dysfunction in everyday life contexts. However, more research is still needed to implement serious games in everyday clinical practice.