Eye Tracking in Virtual Reality

A tutorial: Analyzing eye and head movements in virtual reality

This tutorial provides instruction on how to use the eye tracking technology built into virtual reality (VR) headsets, emphasizing the analysis of head and eye movement data when an observer is situated in the center of an omnidirectional environment. We begin with a brief description of how VR eye movement research differs from previous forms of eye movement research, as well as identifying some outstanding gaps in the current literature. We then introduce the basic methodology used to collect VR eye movement data both in general and with regard to the specific data that we collected to illustrate different analytical approaches. We continue with an introduction of the foundational ideas regarding data analysis in VR, including frames of reference, how to map eye and head position, and event detection. In the next part, we introduce core head and eye data analyses focusing on determining where the head and eyes are directed. We then expand on what has been presented, introducing several novel spatial, spatio-temporal, and temporal head-eye data analysis techniques. We conclude with a reflection on what has been presented, and how the techniques introduced in this tutorial provide the scaffolding for extensions to more complex and dynamic VR environments.

Effects of driver's braking behavior by the real-time pedestrian scale warning system

A driver warning system can improve pedestrian safety by providing drivers with alerts about potential hazards. Most driver warning systems have primarily focused on detecting the presence of pedestrians, without considering other factors, such as the pedestrian's gender and speed, and whether pedestrians are carrying luggage, that can affect driver braking behavior. Therefore, this study aims to investigate how driver braking behavior changes based on the information about the number of pedestrians in a crowd and examine if a developed warning system based on this information can induce safe braking behavior. For this purpose, an experiment scenario was conducted using a virtual reality-based driving simulator and an eye tracker. The collected driver data were analyzed using mixed ANOVA to derive meaningful conclusions. The research findings indicate that providing information about the number of pedestrians in a crowd has a positive impact on driver braking behavior, including deceleration, yielding intention, and attention. Particularly, It was found that in scenarios with a larger number of pedestrians, the Time to Collision (TTC) and distance to the crosswalk were increased by 12%, and the pupil diameter was increased by 9%. This research also verified the applicability of the proposed warning system in complex road environments, especially under conditions with poor visibility such as nighttime. The system was able to induce safe braking behavior even at night and exhibited consistent performance regardless of gender. In conclusion, considering various factors that influence driver behavior, this research provides a comprehensive understanding of the potential and effectiveness of a driver warning system based on information about the number of pedestrians in a crowd in complex road environments.

Eye-Tracking in Physical Human-Robot Interaction: Mental Workload and Performance Prediction

BACKGROUND

In Physical Human-Robot Interaction (pHRI), the need to learn the robot's motor-control dynamics is associated with increased cognitive load. Eye-tracking metrics can help understand the dynamics of fluctuating mental workload over the course of learning.

OBJECTIVE

The aim of this study was to test eye-tracking measures' sensitivity and reliability to variations in task difficulty, as well as their performance-prediction capability, in physical human-robot collaboration tasks involving an industrial robot for object comanipulation.

METHODS

Participants (9M, 9F) learned to coperform a virtual pick-and-place task with a bimanual robot over multiple trials. Joint stiffness of the robot was manipulated to increase motor-coordination demands. The psychometric properties of eye-tracking measures and their ability to predict performance was investigated.

RESULTS

Stationary Gaze Entropy and pupil diameter were the most reliable and sensitive measures of workload associated with changes in task difficulty and learning. Increased task difficulty was more likely to result in a robot-monitoring strategy. Eye-tracking measures were able to predict the occurrence of success or failure in each trial with 70% sensitivity and 71% accuracy.

CONCLUSION

The sensitivity and reliability of eye-tracking measures was acceptable, although values were lower than those observed in cognitive domains. Measures of gaze behaviors indicative of visual monitoring strategies were most sensitive to task difficulty manipulations, and should be explored further for the pHRI domain where motor-control and internal-model formation will likely be strong contributors to workload.

APPLICATION

Future collaborative robots can adapt to human cognitive state and skill-level measured using eye-tracking measures of workload and visual attention.

Wearable Near-Eye Tracking Technologies for Health: A Review

With the rapid advancement of computer vision, machine learning, and consumer electronics, eye tracking has emerged as a topic of increasing interest in recent years. It plays a key role across diverse domains including human-computer interaction, virtual reality, and clinical and healthcare applications. Near-eye tracking (NET) has recently been developed to possess encouraging features such as wearability, affordability, and interactivity. These features have drawn considerable attention in the health domain, as NET provides accessible solutions for long-term and continuous health monitoring and a comfortable and interactive user interface. Herein, this work offers an inaugural concise review of NET for health, encompassing approximately 70 related articles published over the past two decades and supplemented by an in-depth examination of 30 literatures from the preceding five years. This paper provides a concise analysis of health-related NET technologies from aspects of technical specifications, data processing workflows, and the practical advantages and limitations. In addition, the specific applications of NET are introduced and compared, revealing that NET is fairly influencing our lives and providing significant convenience in daily routines. Lastly, we summarize the current outcomes of NET and highlight the limitations.

Immersive virtual reality for interdisciplinary trauma management - initial evaluation of a training tool prototype

INTRODUCTION

Emergency care of critically ill patients in the trauma room is an integral part of interdisciplinary work in hospitals. Live threatening injuries require swift diagnosis, prioritization, and treatment; thus, different medical specialties need to work together closely for optimal patient care. Training is essential to facilitate smooth performance. This study presents a training tool for familiarization with trauma room algorithms in immersive virtual reality (VR), and a first qualitative assessment.

MATERIALS AND METHODS

An interdisciplinary team conceptualized two scenarios and filmed these in the trauma room of the University Medical Center Mainz, Germany in 3D-360°. This video content was used to create an immersive VR experience. Participants of the Department of Anesthesiology were included in the study, questionnaires were obtained and eye movement was recorded.

RESULTS

31 volunteers participated in the study, of which 10 (32,2%) had completed specialist training in anesthesiology. Participants reported a high rate of immersion (immersion(mean) = 6 out of 7) and low Visually Induced Motion Sickness (VIMS(mean) = 1,74 out of 20). Participants agreed that VR is a useful tool for medical education (mean = 1,26; 1 very useful, 7 not useful at all). Residents felt significantly more secure in the matter after training (p < 0,05), specialist showed no significant difference.

DISCUSSION

This study presents a novel tool for familiarization with trauma room procedures, which is especially helpful for less experienced residents. Training in VR was well accepted and may be a solution to enhance training in times of low resources for in person training.

Animated VR and 360-degree VR to assess and train team sports decision-making: a scoping review

Introduction

In team sports, athletes' ability to make quick decisions plays a crucial role. Decision-making proficiency relies on the intricate balance of athletes' perceptual and cognitive abilities, enabling them to assess the competitive environment swiftly and select the most appropriate actions from various options. Virtual reality (VR) technology is emerging as a valuable tool for evaluating and refining athletes' decision-making skills. This study systematically examined the integration of VR technology into decision-making processes in team sports, aiming to identify more effective methods for presenting and interacting with virtual decision-making systems, thus enhancing the evaluation and refinement of athletes' decision making abilities.

Methods

Following the preferred reporting items for systematic reviews and meta-analyses (PRISMA) guidelines, a thorough search of respected research databases, including Web of Science, PubMed, SPORTDiscus, ScienceDirect, PsycINFO, and IEEE, was conducted using carefully selected keywords.

Results

Twenty research papers meeting predefined inclusion criteria were included after careful evaluation. These papers were systematically analyzed to delineate the attributes of virtual decision-making task environments, the interactive dynamics inherent in motor decision-making tasks, and the significant findings.

Discussion

This review indicate that (1) the effectiveness of VR technology in assessing and improving athletes' decision-making skills in team sports; (2) the construction of virtual environments using the Head-Mounted Display (HMD) system, characterized by enhanced ease and efficiency; (3) the potential for future investigations to explore computer simulations to create more expansive virtual motion scenarios, thus efficiently generating substantial task scenario material, diverging from the constraints posed by 360-degree panoramic videos; and (4) the integration of motion capture technology for identifying and monitoring athletes' decision-making behaviors, which not only enhances ecological validity but also augments the transfer validity of virtual sports decision-making systems. Future research endeavors could explore integrating eye-tracking technology with virtual reality to gain insights into the intrinsic cognitive-action associations exhibited by athletes.

Combining EEG and eye-tracking in virtual reality: Obtaining fixation-onset event-related potentials and event-related spectral perturbations

Extensive research conducted in controlled laboratory settings has prompted an inquiry into how results can be generalized to real-world situations influenced by the subjects' actions. Virtual reality lends itself ideally to investigating complex situations but requires accurate classification of eye movements, especially when combining it with time-sensitive data such as EEG. We recorded eye-tracking data in virtual reality and classified it into gazes and saccades using a velocity-based classification algorithm, and we cut the continuous data into smaller segments to deal with varying noise levels, as introduced in the REMoDNav algorithm. Furthermore, we corrected for participants' translational movement in virtual reality. Various measures, including visual inspection, event durations, and the velocity and dispersion distributions before and after gaze onset, indicate that we can accurately classify the continuous, free-exploration data. Combining the classified eye-tracking with the EEG data, we generated fixation-onset event-related potentials (ERPs) and event-related spectral perturbations (ERSPs), providing further evidence for the quality of the eye-movement classification and timing of the onset of events. Finally, investigating the correlation between single trials and the average ERP and ERSP identified that fixation-onset ERSPs are less time sensitive, require fewer repetitions of the same behavior, and are potentially better suited to study EEG signatures in naturalistic settings. We modified, designed, and tested an algorithm that allows the combination of EEG and eye-tracking data recorded in virtual reality.

Application of immersive virtual reality (IVR) for the care of common diseases of older adults course: A mixed methods study

OBJECTIVE

This study aimed to enhance understanding, engagement, and learning efficiency in the course "The Care of Common Diseases of Older Adults" using a developed Immersive Virtual Reality(IVR) system.

METHODS

A mixed-methods study with 32 students was conducted. The quantitative part involved a randomized controlled trial, and the qualitative part included thematic interviews with students and teachers.

RESULTS

The intervention group using the IVR system showed significant improvements in positivity and performance evaluation scores (P < 0.05) compared to the control group. Negative affect scores also decreased significantly (P < 0.05). Qualitative data from interviews supported the quantitative findings, highlighting increased curiosity, learning enthusiasm, and academic performance.

CONCLUSION

IVR significantly enhances learning by stimulating curiosity and active participation, making education more accessible and improving student performance. Future IVR enhancements should focus on user-friendliness and empathetic feedback in adult care.

Introduction to the Special Thematic Issue "Virtual Reality and Eye Tracking"

Technological advancements have made it possible to integrate eye tracking in virtual reality (VR) and augmented reality (AR). Many new VR/AR headsets already include eye tracking as a standard feature. While its application previously has been mostly limited to research, we now see installations of eye tracking into consumer level VR products in entertainment, training, and therapy.  The combination of eye tracking and VR creates new opportunities for end users, creators, and researchers alike: The high level of immersion – while shielded from visual distractions of the physical environment – leads to natural behavior inside the virtual environment. This enables researchers to study how humans perceive and interact with three-dimensional environments in experimentally controlled, ecologically valid settings. Simultaneously, eye tracking in VR poses new challenges to gaze analyses and requires the establishment of new tools and best practices in gaze interaction and psychological research from controlling influence factors, such as simulator sickness, to adaptations of algorithms in various situations. This thematic special issue introduces and discusses novel applications, challenges and possibilities of eye tracking and gaze interaction in VR from an interdisciplinary perspective, including contributions from the fields of psychology, human-computer interaction, human factors, engineering, neuroscience, and education. It addresses a variety of issues and topics, such as practical guidelines for VR-based eye tracking technologies, exploring new research avenues, evaluation of gaze-based assessments, and training interventions. 

Gaze-based attention network analysis in a virtual reality classroom

This article provides a step-by-step guideline for measuring and analyzing visual attention in 3D virtual reality (VR) environments based on eye-tracking data. We propose a solution to the challenges of obtaining relevant eye-tracking information in a dynamic 3D virtual environment and calculating interpretable indicators of learning and social behavior. With a method called "gaze-ray casting," we simulated 3D-gaze movements to obtain information about the gazed objects. This information was used to create graphical models of visual attention, establishing attention networks. These networks represented participants' gaze transitions between different entities in the VR environment over time. Measures of centrality, distribution, and interconnectedness of the networks were calculated to describe the network structure. The measures, derived from graph theory, allowed for statistical inference testing and the interpretation of participants' visual attention in 3D VR environments. Our method provides useful insights when analyzing students' learning in a VR classroom, as reported in a corresponding evaluation article with N = 274 participants. •Guidelines on implementing gaze-ray casting in VR using the Unreal Engine and the HTC VIVE Pro Eye.•Creating gaze-based attention networks and analyzing their network structure.•Implementation tutorials and the Open Source software code are provided via OSF: https://osf.io/pxjrc/?view_only=1b6da45eb93e4f9eb7a138697b941198.

The impact of presentation modes on mental rotation processing: a comparative analysis of eye movements and performance

Mental rotation is the ability to rotate mental representations of objects in space. Shepard and Metzler's shape-matching tasks, frequently used to test mental rotation, involve presenting pictorial representations of 3D objects. This stimulus material has raised questions regarding the ecological validity of the test for mental rotation with actual visual 3D objects. To systematically investigate differences in mental rotation with pictorial and visual stimuli, we compared data of N = 54 university students from a virtual reality experiment. Comparing both conditions within subjects, we found higher accuracy and faster reaction times for 3D visual figures. We expected eye tracking to reveal differences in participants' stimulus processing and mental rotation strategies induced by the visual differences. We statistically compared fixations (locations), saccades (directions), pupil changes, and head movements. Supplementary Shapley values of a Gradient Boosting Decision Tree algorithm were analyzed, which correctly classified the two conditions using eye and head movements. The results indicated that with visual 3D figures, the encoding of spatial information was less demanding, and participants may have used egocentric transformations and perspective changes. Moreover, participants showed eye movements associated with more holistic processing for visual 3D figures and more piecemeal processing for pictorial 2D figures.

Virtual Reality Could Help Assess Sexual Aversion Disorder

Virtual reality (VR) may improve our understanding of sexual dysfunctions' manifestations, although research in this area remains limited. This study assessed the potential use of a VR Behavior Avoidance Test (VR-BAT) as a tool for examining the clinical features of Sexual Aversion Disorder (SAD): the experience of fear, disgust, and avoidance when facing sexual cues/contexts. A sample of 55 adults (≥ 18y) with (n = 27) and without SAD (n = 28) completed a self-report measure of sexual avoidance. Their anxiety, disgust, electrodermal activity, heart rate, and visual and behavioral avoidance were then examined during two VR-BATs involving sexual or non-sexual stimuli. Mixed repeated measures ANOVAs, t-tests, and correlational analyses were performed. Results showed that individuals in the SAD group reported greater anxiety and disgust compared to their non-SAD counterparts during the sexual stimuli condition. Sexual avoidance scores were largely positively related to anxiety and disgust during the VR sexual condition, and moderately negatively related to the time spent touching the virtual character's genitals. This study is important given the prevalence of sexual difficulties, such as SAD, and the new research avenues offered by emerging technologies, like VR.

Oculomotor routines for perceptual judgments

In everyday life we frequently make simple visual judgments about object properties, for example, how big or wide is a certain object? Our goal is to test whether there are also task-specific oculomotor routines that support perceptual judgments, similar to the well-established exploratory routines for haptic perception. In a first study, observers saw different scenes with two objects presented in a photorealistic virtual reality environment. Observers were asked to judge which of two objects was taller or wider while gaze was tracked. All tasks were performed with the same set of virtual objects in the same scenes, so that we can compare spatial characteristics of exploratory gaze behavior to quantify oculomotor routines for each task. Width judgments showed fixations around the center of the objects with larger horizontal spread. In contrast, for height judgments, gaze was shifted toward the top of the objects with larger vertical spread. These results suggest specific strategies in gaze behavior that presumably are used for perceptual judgments. To test the causal link between oculomotor behavior and perception, in a second study, observers could freely gaze at the object or we introduced a gaze-contingent setup forcing observers to fixate specific positions on the object. Discrimination performance was similar between free-gaze and the gaze-contingent conditions for width and height judgments. These results suggest that although gaze is adapted for different tasks, performance seems to be based on a perceptual strategy, independent of potential cues that can be provided by the oculomotor system.

Effects of cannabidiol on psychosocial stress, situational anxiety and nausea in a virtual reality environment: a protocol for a single-centre randomised clinical trial

INTRODUCTION

The non-intoxicating plant-derived cannabinoid, cannabidiol (CBD), has demonstrated therapeutic potential in a number of clinical conditions. Most successful clinical trials have used relatively high (≥300 mg) oral doses of CBD. Relatively few studies have investigated the efficacy of lower (<300 mg) oral doses, typical of those available in over-the-counter CBD products.

METHODS

We present a protocol for a randomised, double-blind, placebo-controlled, parallel-group clinical trial investigating the effects of a low oral dose (150 mg) of CBD on acute psychosocial stress, situational anxiety, motion sickness and cybersickness in healthy individuals. Participants (n=74) will receive 150 mg of CBD or a matched placebo 90 min before completing three virtual reality (VR) challenges (tasks) designed to induce transient stress and motion sickness: (a) a 15 min 'Public Speaking' task; (b) a 5 min 'Walk the Plank' task (above a sheer drop); and (c) a 5 min 'Rollercoaster Ride' task. The primary outcomes will be self-reported stress and nausea measured on 100 mm Visual Analogue Scales. Secondary outcomes will include salivary cortisol concentrations, skin conductance, heart rate and vomiting episodes (if any). Statistical analyses will test the hypothesis that CBD reduces nausea and attenuates subjective, endocrine and physiological responses to stress compared with placebo. This study will indicate whether low-dose oral CBD has positive effects in reducing acute psychosocial stress, situational anxiety, motion sickness and cybersickness.

ETHICS AND DISSEMINATION

The University of Sydney Human Research Ethics Committee has granted approval (2023/307, version 1.6, 16 February 2024). Study findings will be disseminated in a peer-reviewed journal and at academic conferences.

TRIAL REGISTRATION NUMBER

Australian New Zealand Clinical Trials Registry (ACTRN12623000872639).

Dynamics of Eye Dominance Behavior in Virtual Reality

Prior research has shown that sighting eye dominance is a dynamic behavior and dependent on horizontal viewing angle. Virtual reality (VR) offers high flexibility and control for studying eye movement and human behavior, yet eye dominance has not been given significant attention within this domain. In this work, we replicate Khan and Crawford's (2001) original study in VR to confirm their findings within this specific context. Additionally, this study extends its scope to study alignment with objects presented at greater depth in the visual field. Our results align with previous results, remaining consistent when targets are presented at greater distances in the virtual scene. Using greater target distances presents opportunities to investigate alignment with objects at varying depths, providing greater flexibility for the design of methods that infer eye dominance from interaction in VR.

Virtual reality tasks with eye tracking for mild spatial neglect assessment: a pilot study with acute stroke patients

Objective

Increasing evidence shows that traditional neuropsychological tests are insensitive for detecting mild unilateral spatial neglect (USN), lack ecological validity, and are unable to clarify USN in all different spatial domains. Here we present a new, fully immersive virtual reality (VR) task battery with integrated eye tracking for mild visual USN and extinction assessment in the acute state of stroke to overthrow these limitations.

Methods

We included 11 right-sided stroke patients and 10 healthy controls aged 18-75 years. Three VR tasks named the Extinction, the Storage and the Shoot the target tasks were developed to assess USN. Furthermore, neuropsychological assessment examining various parts of cognitive functioning was conducted to measure general abilities. We compared VR and neuropsychological task performance in stroke patients - those with (USN+, n = 5) and without USN (USN-, n = 6) - to healthy controls (n = 10) and tentatively reported the usability of VR system in the acute state of stroke.

Results

Patients had mostly mild neurological and USN symptoms. Nonetheless, we found several differences between the USN+ and healthy control groups in VR task performance. Compared to controls, USN+ patients showed visual extinction and asymmetry in gaze behavior and detection times in distinct spatial locations. Extinction was most evident in the extrapersonal space and delayed detection times on the extreme left and on the left upper parts. Also, USN+ patients needed more time to complete TMT A compared with USN- patients and TMT B compared with controls. VR system usability and acceptance were rated high; no relevant adverse effects occurred.

Conclusion

New VR technology with eye tracking enables ecologically valid and objective assessment methods with various exact measures for mild USN and thus could potentially improve future clinical assessments.

Evaluation of Gaze-to-Object Mapping Algorithms for Use in "Real-World" Translatable Neuropsychological Paradigms

Objective

Eye-tracking technology is commonly used for identifying objects of visual attention. However, applying this technology to virtual reality (VR) applications is challenging. This report analyzes the performance of two gaze-to-object mapping (GTOM) algorithms applied to eye-gaze data acquired during a "real-world" VR cue-reactivity paradigm.

Methods

Two groups of participants completed a VR paradigm using an HTC Vive Pro Eye. The gazed objects were determined by the reported gaze rays and one of two GTOM algorithms - naïve ray-casting (n=18) or a combination of ray-casting and Tobii's G2OM algorithm (n=18). Percent gaze duration was calculated from 1-second intervals before each object interaction to estimate gaze accuracy. The object volume of maximal divergence between algorithms was determined by maximizing the difference in Hedge's G effect sizes between small and large percent gaze duration distributions. Differences in percent gaze duration based on algorithm and target object size were tested with a mixed ANOVA.

Results

The maximum Hedge's G effect sizes differentiating large and small target objects was observed at an 800cm3 threshold. The combination algorithm performed better than the naïve ray-casting algorithm (p=.003, ηp2=.23), and large objects (>800cm3) were associated with a higher gaze duration percentage than small objects (≤800cm3; p<.001, ηp2=.76). No significant interaction between algorithm and size was observed.

Conclusions

Results demonstrated that Tobii's G2OM method outperformed naïve ray-casting in this "real-world" paradigm. As both algorithms show a clear decrease in performance for detecting objects with volumes <800cm3, we recommend using gaze-interactable objects >800cm3 for future HTC Vive Pro Eye applications.

How robust are wearable eye trackers to slow and fast head and body movements?

How well can modern wearable eye trackers cope with head and body movement? To investigate this question, we asked four participants to stand still, walk, skip, and jump while fixating a static physical target in space. We did this for six different eye trackers. All the eye trackers were capable of recording gaze during the most dynamic episodes (skipping and jumping). The accuracy became worse as movement got wilder. During skipping and jumping, the biggest error was 5.8∘. However, most errors were smaller than 3∘. We discuss the implications of decreased accuracy in the context of different research scenarios.

The Virtual Navigation Toolbox: Providing tools for virtual navigation experiments

Spatial navigation research in humans increasingly relies on experiments using virtual reality (VR) tools, which allow for the creation of highly flexible, and immersive study environments, that can react to participant interaction in real time. Despite the popularity of VR, tools simplifying the creation and data management of such experiments are rare and often restricted to a specific scope-limiting usability and comparability. To overcome those limitations, we introduce the Virtual Navigation Toolbox (VNT), a collection of interchangeable and independent tools for the development of spatial navigation VR experiments using the popular Unity game engine. The VNT's features are packaged in loosely coupled and reusable modules, facilitating convenient implementation of diverse experimental designs. Here, we depict how the VNT fulfils feature requirements of different VR environments and experiments, guiding through the implementation and execution of a showcase study using the toolbox. The presented showcase study reveals that homing performance in a classic triangle completion task is invariant to translation velocity of the participant's avatar, but highly sensitive to the number of landmarks. The VNT is freely available under a creative commons license, and we invite researchers to contribute, extending and improving tools using the provided repository.

A Framework for the Continuous Evaluation of 3D Motion Perception in Virtual Reality

OBJECTIVE

We present a novel framework for the detection and continuous evaluation of 3D motion perception by deploying a virtual reality environment with built-in eye tracking.

METHODS

We created a biologically-motivated virtual scene that involved a ball moving in a restricted Gaussian random walk against a background of 1/f noise. Sixteen visually healthy participants were asked to follow the moving ball while their eye movements were monitored binocularly using the eye tracker. We calculated the convergence positions of their gaze in 3D using their fronto-parallel coordinates and linear least-squares optimization. Subsequently, to quantify 3D pursuit performance, we employed a first-order linear kernel analysis known as the Eye Movement Correlogram technique to separately analyze the horizontal, vertical and depth components of the eye movements. Finally, we checked the robustness of our method by adding systematic and variable noise to the gaze directions and re-evaluating 3D pursuit performance.

RESULTS

We found that the pursuit performance in the motion-through depth component was reduced significantly compared to that for fronto-parallel motion components. We found that our technique was robust in evaluating 3D motion perception, even when systematic and variable noise was added to the gaze directions.

CONCLUSION

The proposed framework enables the assessment of 3D Motion perception by evaluating continuous pursuit performance through eye-tracking.

SIGNIFICANCE

Our framework paves the way for a rapid, standardized and intuitive assessment of 3D motion perception in patients with various eye disorders.

Examination of gaze behaviour in social anxiety disorder using a virtual reality eye-tracking paradigm: protocol for a casecontrol study

INTRODUCTION

Social anxiety disorder (SAD) has an early onset, a high lifetime prevalence, and may be a risk factor for developing other mental disorders. Gaze behaviour is considered an aberrant feature of SAD. Eye-tracking, a novel technology device, enables recording eye movements in real time, making it a direct and objective measure of gaze behaviour. Virtual reality (VR) is a promising tool for assessment and diagnostic purposes. Developing an objective screening tool based on examination of gaze behaviour in SAD may potentially aid early detection. The objective of this current study is, therefore to examine gaze behaviour in SAD utilising VR.

METHODS AND ANALYSIS

A case-control study design is employed in which a clinical sample of 29 individuals with SAD will be compared with a matched healthy control group of 29 individuals. In the VR-based eye-tracking paradigm, participants will be presented to stimuli consisting of high-res 360° 3D stereoscopic videos of three social-evaluative tasks designed to elicit social anxiety. The study will investigate between-group gaze behaviour differences during stimuli presentation.

ETHICS AND DISSEMINATION

The study has been approved by the National Committee on Health Research Ethics for the Capital Region of Denmark (H-22041443). The study has been preregistered on OSF registries: https://doi.org/10.17605/OSF.IO/XCTAKAll participants will be provided with written and oral information. Informed consent is required for all the participants. Participation is voluntarily, and the participants can at any time terminate their participation without any consequences. Study results; positive, negative or inconclusive will be published in relevant scientific journals.

Virtual reality for assessing stereopsis performance and eye characteristics in Post-COVID

In 2019, we faced a pandemic due to the coronavirus disease (COVID-19), with millions of confirmed cases and reported deaths. Even in recovered patients, symptoms can be persistent over weeks, termed Post-COVID. In addition to common symptoms of fatigue, muscle weakness, and cognitive impairments, visual impairments have been reported. Automatic classification of COVID and Post-COVID is researched based on blood samples and radiation-based procedures, among others. However, a symptom-oriented assessment for visual impairments is still missing. Thus, we propose a Virtual Reality environment in which stereoscopic stimuli are displayed to test the patient's stereopsis performance. While performing the visual tasks, the eyes' gaze and pupil diameter are recorded. We collected data from 15 controls and 20 Post-COVID patients in a study. Therefrom, we extracted features of three main data groups, stereopsis performance, pupil diameter, and gaze behavior, and trained various classifiers. The Random Forest classifier achieved the best result with 71% accuracy. The recorded data support the classification result showing worse stereopsis performance and eye movement alterations in Post-COVID. There are limitations in the study design, comprising a small sample size and the use of an eye tracking system.

A Novel Sensor Fusion Approach for Precise Hand Tracking in Virtual Reality-Based Human-Computer Interaction

The rapidly evolving field of Virtual Reality (VR)-based Human-Computer Interaction (HCI) presents a significant demand for robust and accurate hand tracking solutions. Current technologies, predominantly based on single-sensing modalities, fall short in providing comprehensive information capture due to susceptibility to occlusions and environmental factors. In this paper, we introduce a novel sensor fusion approach combined with a Long Short-Term Memory (LSTM)-based algorithm for enhanced hand tracking in VR-based HCI. Our system employs six Leap Motion controllers, two RealSense depth cameras, and two Myo armbands to yield a multi-modal data capture. This rich data set is then processed using LSTM, ensuring the accurate real-time tracking of complex hand movements. The proposed system provides a powerful tool for intuitive and immersive interactions in VR environments.

Let's get it started: Eye Tracking in VR with the Pupil Labs Eye Tracking Add-On for the HTC Vive

Combining eye tracking and virtual reality (VR) is a promising approach to tackle various applied research questions. As this approach is relatively new, routines are not established yet and the first steps can be full of potential pitfalls. The present paper gives a practice example to lower the boundaries for getting started. More specifically, I focus on an affordable add-on technology, the Pupil Labs eye tracking add-on for the HTC Vive. As add-on technology with all relevant source code available on GitHub, a high degree of freedom in preprocessing, visualizing, and analyzing eye tracking data in VR can be achieved. At the same time, some extra preparatory steps for the setup of hardware and software are necessary. Therefore, specifics of eye tracking in VR from unboxing, software integration, and procedures to analyzing the data and maintaining the hardware will be addressed. The Pupil Labs eye tracking add-on for the HTC Vive represents a highly transparent approach to existing alternatives. Characteristics of eye tracking in VR in contrast to other headmounded and remote eye trackers applied in the physical world will be discussed. In conclusion, the paper contributes to the idea of open science in two ways: First, by making the necessary routines transparent and therefore reproducible. Second, by stressing the benefits of using open source software.

Design guidelines for limiting and eliminating virtual reality-induced symptoms and effects at work: a comprehensive, factor-oriented review

Virtual reality (VR) can induce side effects known as virtual reality-induced symptoms and effects (VRISE). To address this concern, we identify a literature-based listing of these factors thought to influence VRISE with a focus on office work use. Using those, we recommend guidelines for VRISE amelioration intended for virtual environment creators and users. We identify five VRISE risks, focusing on short-term symptoms with their short-term effects. Three overall factor categories are considered: individual, hardware, and software. Over 90 factors may influence VRISE frequency and severity. We identify guidelines for each factor to help reduce VR side effects. To better reflect our confidence in those guidelines, we graded each with a level of evidence rating. Common factors occasionally influence different forms of VRISE. This can lead to confusion in the literature. General guidelines for using VR at work involve worker adaptation, such as limiting immersion times to between 20 and 30 min. These regimens involve taking regular breaks. Extra care is required for workers with special needs, neurodiversity, and gerontechnological concerns. In addition to following our guidelines, stakeholders should be aware that current head-mounted displays and virtual environments can continue to induce VRISE. While no single existing method fully alleviates VRISE, workers' health and safety must be monitored and safeguarded when VR is used at work.

A systematic performance comparison of two Smooth Pursuit detection algorithms in Virtual Reality depending on target number, distance, and movement patterns

We compared the performance of two smooth-pursuit-based object selection algorithms in Virtual Reality (VR). To assess the best algorithm for a range of configurations, we systematically varied the number of targets to choose from, their distance, and their movement pattern (linear and circular). Performance was operationalized as the ratio of hits, misses and non-detections. Averaged over all distances, the correlation-based algorithm performed better for circular movement patterns compared to linear ones (F(1,11) = 24.27, p < .001, η² = .29). This was not found for the difference-based algorithm (F(1,11) = 0.98, p = .344, η² = .01). Both algorithms performed better in close distances compared to larger ones (F(1,11) = 190.77, p < .001, η² = .75 correlation-based, and F(1,11) = 148.20, p < .001, η² = .42, difference-based). An interaction effect for distance x movement emerged. After systematically varying the number of targets, these results could be replicated, with a slightly smaller effect. Based on performance levels, we introduce the concept of an optimal threshold algorithm, suggesting the best detection algorithm for the individual target configuration. Learnings of adding the third dimension to the detection algorithms and the role of distractors are discussed and suggestions for future research added.

Contact lens embedded holographic pointer

In this paper we present an infrared laser pointer, consisting of a vertical-cavity surface-emitting laser (VCSEL) and a diffractive optical element (DOE), encapsulated into a scleral contact lens (SCL). The VCSEL is powered remotely by inductive coupling from a primary antenna embedded into an eyewear frame. The DOE is used either to collimate the laser beam or to project a pattern image at a chosen distance in front of the eye. We detail the different SCL constitutive blocks, how they are manufactured and assembled. We particularly emphasize the various technological challenges related to their encapsulation in the reduced volume of the SCL, while keeping the pupil free. Finally, we describe how the laser pointer operates, what are its performances (e.g. collimation, image formation) and how it can be used efficiently in various application fields such as visual assistance and augmented reality.

An eye tracking study of the application of gestalt theory in photography

Photography is an art form where integration of the human visual perception and psychological experiences result in aesthetic pleasure. This research utilizes eye tracking to explore the impact of the properties of Gestalt in photography on people's visual cognitive process in order to understand the psychological processes and patterns of photography appreciation. This study found that images with Gestalt qualities can significantly affect fixation, sightline distribution, and subjective evaluation of aesthetics and complexity. Closure composition images seem to make cognition simpler, resulting in the least number of fixation and saccades, longer fixation duration, and more concentrated sightline indicating stronger feeling of beauty, while images which portray similarity results in the greatest fixation and saccades, longest saccade duration, and greater scattering of sightline, indicating feelings of complexity and unsightliness. The results of this research are closely related to the theories of art and design, and have reference value for photography theory and application.

Eye movement characteristics in a mental rotation task presented in virtual reality

Introduction

Eye-tracking technology provides a reliable and cost-effective approach to characterize mental representation according to specific patterns. Mental rotation tasks, referring to the mental representation and transformation of visual information, have been widely used to examine visuospatial ability. In these tasks, participants visually perceive three-dimensional (3D) objects and mentally rotate them until they identify whether the paired objects are identical or mirrored. In most studies, 3D objects are presented using two-dimensional (2D) images on a computer screen. Currently, visual neuroscience tends to investigate visual behavior responding to naturalistic stimuli rather than image stimuli. Virtual reality (VR) is an emerging technology used to provide naturalistic stimuli, allowing the investigation of behavioral features in an immersive environment similar to the real world. However, mental rotation tasks using 3D objects in immersive VR have been rarely reported.

Methods

Here, we designed a VR mental rotation task using 3D stimuli presented in a head-mounted display (HMD). An eye tracker incorporated into the HMD was used to examine eye movement characteristics during the task synchronically. The stimuli were virtual paired objects oriented at specific angular disparities (0, 60, 120, and 180°). We recruited thirty-three participants who were required to determine whether the paired 3D objects were identical or mirrored.

Results

Behavioral results demonstrated that the response times when comparing mirrored objects were longer than identical objects. Eye-movement results showed that the percent fixation time, the number of within-object fixations, and the number of saccades for the mirrored objects were significantly lower than that for the identical objects, providing further explanations for the behavioral results.

Discussion

In the present work, we examined behavioral and eye movement characteristics during a VR mental rotation task using 3D stimuli. Significant differences were observed in response times and eye movement metrics between identical and mirrored objects. The eye movement data provided further explanation for the behavioral results in the VR mental rotation task.

Extended reality for biomedicine

Extended reality (XR) refers to an umbrella of methods that allows users to be immersed in a three-dimensional (3D) or a 4D (spatial + temporal) virtual environment to different extents, including virtual reality (VR), augmented reality (AR), and mixed reality (MR). While VR allows a user to be fully immersed in a virtual environment, AR and MR overlay virtual objects over the real physical world. The immersion and interaction of XR provide unparalleled opportunities to extend our world beyond conventional lifestyles. While XR has extensive applications in fields such as entertainment and education, its numerous applications in biomedicine create transformative opportunities in both fundamental research and healthcare. This Primer outlines XR technology from instrumentation to software computation methods, delineating the biomedical applications that have been advanced by state-of-the-art techniques. We further describe the technical advances overcoming current limitations in XR and its applications, providing an entry point for professionals and trainees to thrive in this emerging field.

Immersive medium for early clinical exposure - knowledge acquisition, spatial orientation and the unexpected role of annotation in 360° VR photos

AIM

360° VR photos could be a low-threshold possibility to increase early clinical exposure. Apart from granting insights into local routines and premises, the medium should facilitate knowledge acquisition and spatial orientation depending on its design. This assumption, however, is not yet substantiated empirically. Thus, three hypotheses were tested in consideration of Mayer's modality principle: 1) Providing 360° VR photos as visual reference improves retention and comprehension of information. 2) The annotation of text boxes in 360° VR photos compromises spatial orientation and presence. 3) Annotated audio commentary is superior to annotated text boxes in terms of cognitive load and knowledge acquisition.

METHODS

Using head-mounted displays, students of human (N=53) and dental medicine (N=8) completed one of three virtual tours through a surgical unit, which were created with 360° VR photos. In the first two variants, information about the facilities, medical devices and clinical procedures was annotated either as text boxes or audio commentary comprising 67 words on average (SD=6.67). In the third variant, the same information was given separately on a printed handout before the virtual tour. Taking user experience and individual learner characteristics into account, differences between conditions were measured regarding retention, comprehension, spatial orientation, cognitive load, and presence.

RESULTS

Concerning retention and comprehension of information, annotated text boxes outperformed annotated audio commentary and the handout condition. Although annotated audio commentary exhibited the lowest knowledge test scores, students preferred listening over reading. Students with an interest in VR and 360° media reported higher levels of enjoyment and presence. Regarding spatial orientation and presence, no significant group differences were found.

CONCLUSIONS

360° VR photos can convey information and a sense of spatial orientation effectively in the same learning scenario. For students, their use is both enjoyable and instructive. Unexpectedly, the ideal mode of annotation is not dictated by Mayer's modality principle. For information like in this study, annotated text boxes are better for knowledge acquisition than the subjectively preferred audio commentary. This finding is probably contingent on the length and the quality of the annotated text. To identify boundary conditions and to validate the findings, more research is required on the design and educational use of 360° VR photos.

vexptoolbox: A software toolbox for human behavior studies using the Vizard virtual reality platform

Virtual reality (VR) is a powerful tool for researchers due to its potential to study dynamic human behavior in highly naturalistic environments while retaining full control over the presented stimuli. Due to advancements in consumer hardware, VR devices are now very affordable and have also started to include technologies such as eye tracking, further extending potential research applications. Rendering engines such as Unity, Unreal, or Vizard now enable researchers to easily create complex VR environments. However, implementing the experimental design can still pose a challenge, and these packages do not provide out-of-the-box support for trial-based behavioral experiments. Here, we present a Python toolbox, designed to facilitate common tasks when developing experiments using the Vizard VR platform. It includes functionality for common tasks like creating, randomizing, and presenting trial-based experimental designs or saving results to standardized file formats. Moreover, the toolbox greatly simplifies continuous recording of eye and body movements using any hardware supported in Vizard. We further implement and describe a simple goal-directed reaching task in VR and show sample data recorded from five volunteers. The toolbox, example code, and data are all available on GitHub under an open-source license. We hope that our toolbox can simplify VR experiment development, reduce code duplication, and aid reproducibility and open-science efforts.

Eye tracking: empirical foundations for a minimal reporting guideline

In this paper, we present a review of how the various aspects of any study using an eye tracker (such as the instrument, methodology, environment, participant, etc.) affect the quality of the recorded eye-tracking data and the obtained eye-movement and gaze measures. We take this review to represent the empirical foundation for reporting guidelines of any study involving an eye tracker. We compare this empirical foundation to five existing reporting guidelines and to a database of 207 published eye-tracking studies. We find that reporting guidelines vary substantially and do not match with actual reporting practices. We end by deriving a minimal, flexible reporting guideline based on empirical research (Section "An empirically based minimal reporting guideline").

Virtual Reality for Awe and Imagination

Empirical research has explored the potential of the emotion of awe to shape creativity, while theoretical work has sought to understand the link between this emotion and transformation in terms of imagining new possible worlds. This branch of study relies on the transformative potential of virtual reality (VR) to examine and invite cognitive and emotional components of transformative experiences (TEs) within the interdisciplinary model of Transformative Experience Design (TED) and the Appraisal-Tendency Framework (ATF). TED suggests using the epistemic and emotional affordances of interactive technologies, such as VR, to invite TEs. The ATF can provide insight into the nature of these affordances and their relationship. This line of research draws on empirical evidence of the awe-creativity link to broaden the discourse and consider the potential impact of this emotion on core beliefs about the world. The combination of VR with these theoretical and design-oriented approaches may enable a new generation of potentially transformative experiences that remind people that they can aspire to more and inspire them to work toward imagining and creating a new possible world.

Eyes on me: Investigating the role and influence of eye-tracking data on user modeling in virtual reality

Research has shown that sensor data generated by a user during a VR experience is closely related to the user's behavior or state, meaning that the VR user can be quantitatively understood and modeled. Eye-tracking as a sensor signal has been studied in prior research, but its usefulness in a VR context has been less examined, and most extant studies have dealt with eye-tracking within a single environment. Our goal is to expand the understanding of the relationship between eye-tracking data and user modeling in VR. In this paper, we examined the role and influence of eye-tracking data in predicting a level of cybersickness and types of locomotion. We developed and applied the same structure of a deep learning model to the multi-sensory data collected from two different studies (cybersickness and locomotion) with a total of 50 participants. The experiment results highlight not only a high applicability of our model to sensor data in a VR context, but also a significant relevance of eye-tracking data as a potential supplement to improving the model's performance and the importance of eye-tracking data in learning processes overall. We conclude by discussing the relevance of these results to potential future studies on this topic.

EyeT4Empathy: Dataset of foraging for visual information, gaze typing and empathy assessment

We present a dataset of eye-movement recordings collected from 60 participants, along with their empathy levels, towards people with movement impairments. During each round of gaze recording, participants were divided into two groups, each one completing one task. One group performed a task of free exploration of structureless images, and a second group performed a task consisting of gaze typing, i.e. writing sentences using eye-gaze movements on a card board. The eye-tracking data recorded from both tasks is stored in two datasets, which, besides gaze position, also include pupil diameter measurements. The empathy levels of participants towards non-verbal movement-impaired people were assessed twice through a questionnaire, before and after each task. The questionnaire is composed of forty questions, extending a established questionnaire of cognitive and affective empathy. Finally, our dataset presents an opportunity for analysing and evaluating, among other, the statistical features of eye-gaze trajectories in free-viewing as well as how empathy is reflected in eye features.

Advances in Materials, Sensors, and Integrated Systems for Monitoring Eye Movements

Eye movements show primary responses that reflect humans' voluntary intention and conscious selection. Because visual perception is one of the fundamental sensory interactions in the brain, eye movements contain critical information regarding physical/psychological health, perception, intention, and preference. With the advancement of wearable device technologies, the performance of monitoring eye tracking has been significantly improved. It also has led to myriad applications for assisting and augmenting human activities. Among them, electrooculograms, measured by skin-mounted electrodes, have been widely used to track eye motions accurately. In addition, eye trackers that detect reflected optical signals offer alternative ways without using wearable sensors. This paper outlines a systematic summary of the latest research on various materials, sensors, and integrated systems for monitoring eye movements and enabling human-machine interfaces. Specifically, we summarize recent developments in soft materials, biocompatible materials, manufacturing methods, sensor functions, systems' performances, and their applications in eye tracking. Finally, we discuss the remaining challenges and suggest research directions for future studies.

Eye Tracking in Virtual Reality: Vive Pro Eye Spatial Accuracy, Precision, and Calibration Reliability

A growing number of virtual reality devices now include eye tracking technology, which can facilitate oculomotor and cognitive research in VR and enable use cases like foveated rendering. These applications require different tracking performance, often measured as spatial accuracy and precision. While manufacturers report data quality estimates for their devices, these typically represent ideal performance and may not reflect real-world data quality. Additionally, it is unclear how accuracy and precision change across sessions within the same participant or between devices, and how performance is influenced by vision correction. Here, we measured spatial accuracy and precision of the Vive Pro Eye built-in eye tracker across a range of 30 visual degrees horizontally and vertically. Participants completed ten measurement sessions over multiple days, allowing to evaluate calibration reliability. Accuracy and precision were highest for central gaze and decreased with greater eccentricity in both axes. Calibration was successful in all participants, including those wearing contacts or glasses, but glasses yielded significantly lower performance. We further found differences in accuracy (but not precision) between two Vive Pro Eye headsets, and estimated participants' inter-pupillary distance. Our metrics suggest high calibration reliability and can serve as a baseline for expected eye tracking performance in VR experiments.

Terrestrial health applications of visual assessment technology and machine learning in spaceflight associated neuro-ocular syndrome

The neuro-ocular effects of long-duration spaceflight have been termed Spaceflight Associated Neuro-Ocular Syndrome (SANS) and are a potential challenge for future, human space exploration. The underlying pathogenesis of SANS remains ill-defined, but several emerging translational applications of terrestrial head-mounted, visual assessment technology and machine learning frameworks are being studied for potential use in SANS. To develop such technology requires close consideration of the spaceflight environment which is limited in medical resources and imaging modalities. This austere environment necessitates the utilization of low mass, low footprint technology to build a visual assessment system that is comprehensive, accessible, and efficient. In this paper, we discuss the unique considerations for developing this technology for SANS and translational applications on Earth. Several key limitations observed in the austere spaceflight environment share similarities to barriers to care for underserved areas on Earth. We discuss common terrestrial ophthalmic diseases and how machine learning and visual assessment technology for SANS can help increase screening for early intervention. The foundational developments with this novel system may help protect the visual health of both astronauts and individuals on Earth.

Body Gaze as a Marker of Sexual Objectification: A New Scale for Pervasive Gaze and Gaze Provocation Behaviors in Heterosexual Women and Men

Body gaze behavior is assumed to be a key feature of sexual objectification. However, there are few self-report gaze measures available and none capturing behavior which seeks to invite body gaze from others. Across two studies, we used existing self-report instruments and measurement of eye movements to validate a new self-report scale to measure pervasive body gaze behavior and body gaze provocation behavior in heterosexual women and men. In Study 1, participants (N = 1021) completed a survey with newly created items related to pervasive body gaze and body gaze provocation behavior. Participants also completed preexisting measures of body attitudes, sexual assault attitudes, pornography use, and relationship status. Exploratory and confirmatory factor analyses across independent samples suggested a 12-item scale for men and women to separately measure pervasive body gaze (5 items) and body gaze provocation (7 items) toward the opposite sex. The two scales yielded excellent internal consistency estimates (.86-.89) and promising convergent validity via positive correlations with body and sexual attitudes. In Study 2, a subsample (N = 167) of participants from Study 1 completed an eye-tracking task to capture their gaze behavior toward matched images of partially and fully dressed female and male subjects. Men exhibited body-biased gaze behavior toward all the female imagery, whereas women exhibited head-biased gaze behavior toward fully clothed male imagery. Importantly, self-reported body gaze correlated positively with some aspects of objectively measured body gaze behavior. Both scales showed good test-retest reliability and were positively correlated with sexual assault attitudes.

A Study on Attention Attracting Elements of 360-Degree Videos Based on VR Eye-Tracking System

In 360-degree virtual reality (VR) videos, users possess increased freedom in terms of gaze movement. As a result, the users’ attention may not move according to the narrative intended by the director and miss out on important parts of the narrative of the 360-degree video. Therefore, it is necessary to study a directing technique that can attract user attention in 360-degree VR videos. In this study, we analyzed the directing elements that can attract users’ attention in a 360-degree VR video and developed a 360 VR eye-tracking system to investigate the effect of the attention-attracting elements on the user. Elements that can attract user attention were classified into five categories: object movement, hand gesture, GUI insertion, camera movement, and gaze angle variation. We developed a 360 VR eye-tracking system to analyze whether five attention-attracting elements influence the user’s attention. Based on the eye tracking system, an experiment was conducted to analyze whether the user’s attention moves according to the five attention-attracting elements. Based on the experimental results, it can be seen that ‘hand gesture’ attracted the second most attention shift of the subjects, and ‘GUI insertion’ induced the smallest shift of attention of the subjects.

Employing Eye Tracking to Study Visual Attention to Live Streaming: A Case Study of Facebook Live

In recent years, the COVID-19 pandemic has led to the development of a new business model, “Live Streaming + Ecommerce”, which is a new method for commercial sales that shares the goal of sustainable economic growth (SDG 8). As information technology finds its way into the digital lives of internet users, the real-time and interactive nature of live streaming has overturned the traditional entertainment experience of audio and video content, moving towards a more nuanced division of labor with multiple applications. This study used a portable eye tracker to collect eye movement information from participants watching Facebook Live, with 31 participants who had experience using the live streaming platform. The four eye movement indicators, namely, latency of first fixation (LFF), duration of first fixation (DFF), total fixation durations (TFD), and the number of fixations (NOF), were used to analyze the distribution of the visual attention in each region of interest (ROI) and explore the study questions based on the ROIs. The findings of this study were as follows: (1) the fixation order of the ROIs in the live ecommerce platform differed between participants of different sexes; (2) the DFF of the ROIs in the live ecommerce platform differed among participants of different sexes; and (3) regarding the ROIs of participants on the live ecommerce platform, participants of different sexes showed the same attention to the live products according to the TFD and NOF eye movement indicators. This study explored the visual search behaviors of existing consumers watching live ecommerce and provides the results as a reference for operators and researchers of live streaming platforms.

Virtual reality for the assessment and rehabilitation of neglect: where are we now? A 6-year review update

Unilateral spatial neglect (USN) is a frequent repercussion of a cerebrovascular accident, typically a stroke. USN patients fail to orient their attention to the contralesional side to detect auditory, visual, and somatosensory stimuli, as well as to collect and purposely use this information. Traditional methods for USN assessment and rehabilitation include paper-and-pencil procedures, which address cognitive functions as isolated from other aspects of patients' functioning within a real-life context. This might compromise the ecological validity of these procedures and limit their generalizability; moreover, USN evaluation and treatment currently lacks a gold standard. The field of technology has provided several promising tools that have been integrated within the clinical practice; over the years, a "first wave" has promoted computerized methods, which cannot provide an ecological and realistic environment and tasks. Thus, a "second wave" has fostered the implementation of virtual reality (VR) devices that, with different degrees of immersiveness, induce a sense of presence and allow patients to actively interact within the life-like setting. The present paper provides an updated, comprehensive picture of VR devices in the assessment and rehabilitation of USN, building on the review of Pedroli et al. (2015). The present paper analyzes the methodological and technological aspects of the studies selected, considering the issue of usability and ecological validity of virtual environments and tasks. Despite the technological advancement, the studies in this field lack methodological rigor as well as a proper evaluation of VR usability and should improve the ecological validity of VR-based assessment and rehabilitation of USN.

A Virtual Reality Based System for the Screening and Classification of Autism

Autism - also known as Autism Spectrum Disorders or Autism Spectrum Conditions - is a neurodevelopmental condition characterized by repetitive behaviours and differences in communication and social interaction. As a consequence, many autistic individuals may struggle in everyday life, which sometimes manifests in depression, unemployment, or addiction. One crucial problem in patient support and treatment is the long waiting time to diagnosis, which was approximated to thirteen months on average. Yet, the earlier an intervention can take place the better the patient can be supported, which was identified as a crucial factor. We propose a system to support the screening of Autism Spectrum Disorders based on a virtual reality social interaction, namely a shopping experience, with an embodied agent. During this everyday interaction, behavioral responses are tracked and recorded. We analyze this behavior with machine learning approaches to classify participants from an autistic participant sample in comparison to a typically developed individuals control sample with high accuracy, demonstrating the feasibility of the approach. We believe that such tools can strongly impact the way mental disorders are assessed and may help to further find objective criteria and categorization.

Applying machine learning to dissociate between stroke patients and healthy controls using eye movement features obtained from a virtual reality task

Conventional neuropsychological tests do not represent the complex and dynamic situations encountered in daily life. Immersive virtual reality simulations can be used to simulate dynamic and interactive situations in a controlled setting. Adding eye tracking to such simulations may provide highly detailed outcome measures, and has great potential for neuropsychological assessment. Here, participants (83 stroke patients and 103 healthy controls) we instructed to find either 3 or 7 items from a shopping list in a virtual super market environment while eye movements were being recorded. Using Logistic Regression and Support Vector Machine models, we aimed to predict the task of the participant and whether they belonged to the stroke or the control group. With a limited number of eye movement features, our models achieved an average Area Under the Curve (AUC) of .76 in predicting whether each participant was assigned a short or long shopping list (3 or 7 items). Identifying participant as either stroke patients and controls led to an AUC of .64. In both classification tasks, the frequency with which aisles were revisited was the most dissociating feature. As such, eye movement data obtained from a virtual reality simulation contain a rich set of signatures for detecting cognitive deficits, opening the door to potential clinical applications.

A Perspective Review on Integrating VR/AR with Haptics into STEM Education for Multi-Sensory Learning

As a result of several governments closing educational facilities in reaction to the COVID-19 pandemic in 2020, almost 80% of the world’s students were not in school for several weeks. Schools and universities are thus increasing their efforts to leverage educational resources and provide possibilities for remote learning. A variety of educational programs, platforms, and technologies are now accessible to support student learning; while these tools are important for society, they are primarily concerned with the dissemination of theoretical material. There is a lack of support for hands-on laboratory work and practical experience. This is particularly important for all disciplines related to science, technology, engineering, and mathematics (STEM), where labs and pedagogical assets must be continuously enhanced in order to provide effective study programs. In this study, we describe a unique perspective to achieving multi-sensory learning through the integration of virtual and augmented reality (VR/AR) with haptic wearables in STEM education. We address the implications of a novel viewpoint on established pedagogical notions. We want to encourage worldwide efforts to make fully immersive, open, and remote laboratory learning a reality.

Using Artificial Intelligence for Assistance Systems to Bring Motor Learning Principles into Real World Motor Tasks

Humans learn movements naturally, but it takes a lot of time and training to achieve expert performance in motor skills. In this review, we show how modern technologies can support people in learning new motor skills. First, we introduce important concepts in motor control, motor learning and motor skill learning. We also give an overview about the rapid expansion of machine learning algorithms and sensor technologies for human motion analysis. The integration between motor learning principles, machine learning algorithms and recent sensor technologies has the potential to develop AI-guided assistance systems for motor skill training. We give our perspective on this integration of different fields to transition from motor learning research in laboratory settings to real world environments and real world motor tasks and propose a stepwise approach to facilitate this transition.

The Static and Dynamic Analyses of Drivers’ Gaze Movement Using VR Driving Simulator

Drivers collect information of road and traffic conditions through a visual search while driving to avoid any potential hazards they perceive. Novice drivers with lack of driving experience may be involved in a car accident as they misjudge the information obtained by insufficient visual search with a narrower field of vision than experienced drivers do. In this regard, the current study compared and identified the gap between novice and experienced drivers in regard to the information they obtained in a visual search of gaze movement and visual attention. A combination of a static analysis, based on the dwell time, fixation duration, the number of fixations and stationary gaze entropy in visual search, and a dynamic analysis using gaze transition entropy was applied. The static analysis on gaze indicated that the group of novice drivers showed a longer dwell time on the traffic lights, pedestrians, and passing vehicles, and a longer fixation duration on the navigation system and the dashboard than the experienced ones. Also, the novice had their eyes fixed on the area of interests straight ahead more frequently while driving at an intersection. In addition, the novice group demonstrated less information at 2.60 bits out of the maximum stationary gaze entropy of 3.32 bits that a driver can exhibit, which indicated that their gaze fixations were concentrated. Meanwhile, the experienced group displayed approx. 3.09 bits, showing that their gaze was not narrowed on a certain area of interests, but was relatively evenly distributed. The dynamic analysis results showed that the novice group conducted the most gaze transitions between traffic lights, pedestrians and passing vehicles, whereas experienced drivers displayed the most transitions between the right- and left-side mirrors, passing vehicles, pedestrians, and traffic lights to find more out about the surrounding traffic conditions. In addition, the experienced group (3.04 bits) showed a higher gaze transition entropy than the novice group (2.21 bits). This indicated that a larger entropy was required to understand the visual search data because visual search strategies changed depending on the situations.

Eye tracking applied to tobacco smoking: current directions and future perspectives

Over the years the general awareness of the health costs associated with tobacco smoking has motivated scientists to apply the measurement of eye movements to this form of addiction. On one hand they have investigated whether smokers attend and look preferentially at smoking related scenes and objects. In parallel, on the other hand eye tracking has been used to test how smokers and nonsmokers interact with the different types of health warning that policymakers have mandated in tobacco advertisements and packages. Here we provide an overview of the main findings from the different lines of research, such as the evidence related to the attentional bias for smoking cues in smokers and the evidence that graphic warning labels and plain packages measurably increase the salience of the warning labels. We point to some open questions, such as the conditions that determine whether heavy smokers exhibit a tendency to actively avoid looking at graphic warning labels. Finally we argue that the research applied to gaze exploration of warning labels would benefit from a more widespread use of the more naturalistic testing conditions (e.g. mobile eye tracking or virtual reality) that have been introduced to study the smokers' attentional bias for tobacco-related objects when freely exploring the surrounding environment.

Eye-Tracking in Interactive Virtual Environments: Implementation and Evaluation

Not all eye-tracking methodology and data processing are equal. While the use of eye-tracking is intricate because of its grounding in visual physiology, traditional 2D eye-tracking methods are supported by software, tools, and reference studies. This is not so true for eye-tracking methods applied in virtual reality (imaginary 3D environments). Previous research regarded the domain of eye-tracking in 3D virtual reality as an untamed realm with unaddressed issues. The present paper explores these issues, discusses possible solutions at a theoretical level, and offers example implementations. The paper also proposes a workflow and software architecture that encompasses an entire experimental scenario, including virtual scene preparation and operationalization of visual stimuli, experimental data collection and considerations for ambiguous visual stimuli, post-hoc data correction, data aggregation, and visualization. The paper is accompanied by examples of eye-tracking data collection and evaluation based on ongoing research of indoor evacuation behavior.