Accuracy and precision of visual and auditory stimulus presentation in virtual reality in Python 2 and 3 environments for human behavior research

PTVR - A software in Python to make virtual reality experiments easier to build and more reproducible

Researchers increasingly use virtual reality (VR) to perform behavioral experiments, especially in vision science. These experiments are usually programmed directly in so-called game engines that are extremely powerful. However, this process is tricky and time-consuming as it requires solid knowledge of game engines. Consequently, the anticipated prohibitive effort discourages many researchers who want to engage in VR. This paper introduces the Perception Toolbox for Virtual Reality (PTVR) library, allowing visual perception studies in VR to be created using high-level Python script programming. A crucial consequence of using a script is that an experiment can be described by a single, easy-to-read piece of code, thus improving VR studies' transparency, reproducibility, and reusability. We built our library upon a seminal open-source library released in 2018 that we have considerably developed since then. This paper aims to provide a comprehensive overview of the PTVR software for the first time. We introduce the main objects and features of PTVR and some general concepts related to the three-dimensional (3D) world. This new library should dramatically reduce the difficulty of programming experiments in VR and elicit a whole new set of visual perception studies with high ecological validity.

Temporal precision and accuracy of audio-visual stimuli in mixed reality systems

Mixed Reality (MR) techniques, such as Virtual (VR) and Augmented Reality (AR), are gaining popularity as a new methodology for neuroscience and psychology research. In studies involving audiovisual stimuli, it is crucial to have MR systems that can deliver these bimodal stimuli with controlled timing between the onset of each modality. However, the extent to which modern MR setups can achieve the necessary precision and accuracy of audiovisual stimulus onset asynchronies (SOAs) remains largely unknown. The objective of this study is to systematically evaluate the lag and variability between the auditory and visual onset of audiovisual stimuli produced on popular modern MR head-mounted displays (HMDs) from Meta, Microsoft, HTC, and Varjo in conjunction with commonly used development environments such as Unity and the Unreal Engine. To accomplish this, we developed a low-cost measurement system that enabled us to measure the actual SOA and its associated jitter. Our findings revealed that certain MR systems exhibited significant SOAs, with one case averaging 156.63 ms, along with jitter of up to ±11.82 ms. Using our methodology, we successfully conducted experimental calibration of a headset, achieving SOAs of -3.89 ± 1.56 ms. This paper aims to raise awareness among neuroscience researchers regarding the limitations of MR systems in delivering audiovisual stimuli without prior calibration. Furthermore, we present cost-effective methods to calibrate these systems, thereby facilitating the replication of future results.

Perceptual decisions interfere more with eye movements than with reach movements

Perceptual judgements are formed through invisible cognitive processes. Reading out these judgements is essential for advancing our understanding of decision making and requires inferring covert cognitive states based on overt motor actions. Although intuition suggests that these actions must be related to the formation of decisions about where to move body parts, actions have been reported to be influenced by perceptual judgements even when the action is irrelevant to the perceptual judgement. However, despite performing multiple actions in our daily lives, how perceptual judgements influence multiple judgement-irrelevant actions is unknown. Here we show that perceptual judgements affect only saccadic eye movements when simultaneous judgement-irrelevant saccades and reaches are made, demonstrating that perceptual judgement-related signals continuously flow into the oculomotor system alone when multiple judgement-irrelevant actions are performed. This suggests that saccades are useful for making inferences about covert perceptual decisions, even when the actions are not tied to decision making.

Multiple representations of the body schema for the same body part

Accurate motor control depends on maps of the body in the brain, called the body schema. Disorders of the body schema cause motor deficits. Although we often execute actions with different motor systems such as the eye and hand, how the body schema operates during such actions is unknown. In this study, participants simultaneously directed eye and hand movements to the same body part. These two movements were found to be guided by different body maps. This finding demonstrates multiple motor system–specific representations of the body schema, suggesting that the choice of motor system toward one’s body can determine which of the brain’s body maps is observed. This may offer a new way to visualize patients’ body schema.

Purposeful motor actions depend on the brain’s representation of the body, called the body schema, and disorders of the body schema have been reported to show motor deficits. The body schema has been assumed for almost a century to be a common body representation supporting all types of motor actions, and previous studies have considered only a single motor action. Although we often execute multiple motor actions, how the body schema operates during such actions is unknown. To address this issue, I developed a technique to measure the body schema during multiple motor actions. Participants made simultaneous eye and reach movements to the same location of 10 landmarks on their hand. By analyzing the internal configuration of the locations of these points for each of the eye and reach movements, I produced maps of the mental representation of hand shape. Despite these two movements being simultaneously directed to the same bodily location, the resulting hand map (i.e., a part of the body schema) was much more distorted for reach movements than for eye movements. Furthermore, the weighting of visual and proprioceptive bodily cues to build up this part of the body schema differed for each effector. These results demonstrate that the body schema is organized as multiple effector-specific body representations. I propose that the choice of effector toward one’s body can determine which body representation in the brain is observed and that this visualization approach may offer a new way to understand patients’ body schema.