The impact of multisensory integration and perceptual load in virtual reality settings on performance, workload and presence

Modulation of thermal perception by VR-based visual stimulation to the embodied virtual body

Visual stimulation to the embodied virtual body could modulate human perception, however the associated neurophysiological mechanisms have not elucidated yet. The present study aimed to reveal the underlying neurophysiological mechanisms from a neurophysiological viewpoint. Fifteen healthy participants were subjected to three visual conditions (i.e., fire, water, and non-visual effect conditions) and psychological pain stimulation (thermal grill stimulation). Oscillatory neural activities during stimulation were measured with electroencephalogram. The association between accessory visual stimulation applied to the embodied virtual body, induced by virtual reality, and perception was examined through neuronal oscillatory analysis using electroencephalogram data. Regression analysis was performed to obtain data on brain regions contributing to sensory modulation with body illusion. The results of subjective measures under the fire and water conditions showed that thermal perception were modulated by a visual stimulus to the virtual hand. Furthermore, we found that the insula was commonly associated with thermal perception under the fire and water conditions. This result indicate that the insula may control sensory information as a gatekeeper as well as facilitate the access to human attention and cognition as a hub, suggesting the influence on perception and cognition.

Immersive interfaces for clinical applications: current status and future perspective

Digital immersive technologies have become increasingly prominent in clinical research and practice, including medical communication and technical education, serious games for health, psychotherapy, and interfaces for neurorehabilitation. The worldwide enthusiasm for digital health and digital therapeutics has prompted the development and testing of numerous applications and interaction methods. Nevertheless, the lack of consistency in the approaches and the peculiarity of the constructed environments contribute to an increasing disparity between the eagerness for new immersive designs and the long-term clinical adoption of these technologies. Several challenges emerge in aligning the different priorities of virtual environment designers and clinicians. This article seeks to examine the utilization and mechanics of medical immersive interfaces based on extended reality and highlight specific design challenges. The transfer of skills from virtual to clinical environments is often confounded by perceptual and attractiveness factors. We argue that a multidisciplinary approach to development and testing, along with a comprehensive acknowledgement of the shared mechanisms that underlie immersive training, are essential for the sustainable integration of extended reality into clinical settings. The present review discusses the application of a multilevel sensory framework to extended reality design, with the aim of developing brain-centered immersive interfaces tailored for therapeutic and educational purposes. Such a framework must include broader design questions, such as the integration of digital technologies into psychosocial care models, clinical validation, and related ethical concerns. We propose that efforts to bridge the virtual gap should include mixed methodologies and neurodesign approaches, integrating user behavioral and physiological feedback into iterative design phases.

The impact of virtual reality and distractors on attentional processes: insights from EEG

Virtual reality (VR) allows to create controlled scenarios in which the quantity of stimuli can be modulated, as happen in real-life, where humans are subjected to various multisensory-often overlapping-stimuli. The present research aimed to study changes in attentional processes within an auditory oddball paradigm during a virtual exploration, while varying the amount of distractors. Twenty healthy volunteers underwent electroencephalography (EEG) during three different experimental conditions: an auditory oddball without VR (No-VR condition), an auditory oddball during VR exploration without distractors (VR-Empty condition), and an auditory oddball during VR exploration with a high level of distractors (VR-Full condition). Event-related potentials (ERPs) were computed averaging epochs of EEGs and analyzing peaks at 100 ms (N100) and 300 ms (P300) latencies. Results showed modulation of N100 amplitude in Fz and of P300 amplitude in Pz. Statistically significant differences in latency were observed only for P300 where the latency results delayed from the No-VR to VR-Full. The scalp topography revealed for P100 no significant differences between frequent and rare stimuli in either the No-VR and VR-Empty conditions. However, significant results were found in N100 in VR-Full condition. For P300, results showed differences between frequent and rare stimuli, in every condition. However, this difference is gradually less widespread from No-VR condition to the VR-Full. The emerging integration of VR with EEG may have important implications for studying brain attentional processing.

Reducing flight upset risk and startle response: A study of the wake vortex alert with licensed commercial pilots

The study aimed at investigating the impact of an innovative Wake Vortex Alert (WVA) avionics on pilots' operation and mental states, intending to improve aviation safety by mitigating the risks associated with wake vortex encounters (WVEs). Wake vortices, generated by jet aircraft, pose a significant hazard to trailing or crossing aircrafts. Despite existing separation rules, incidents involving WVEs continue to occur, especially affecting smaller aircrafts like business jets, resulting in aircraft upsets and occasional cabin injuries. To address these challenges, the study focused on developing and validating an alert system that can be presented to air traffic controllers, enabling them to warn flight crews. This empowers the flight crews to either avoid the wake vortex or secure the cabin to prevent injuries. The research employed a multidimensional approach including an analysis of human performance and human factors (HF) issues to determine the potential impact of the alert on pilots' roles, tasks, and mental states. It also utilizes Human Assurance Levels (HALs) to evaluate the necessary human factors support based on the safety criticality of the new system. Realistic flight simulations were conducted to collect data of pilots' behavioural, subjective and neurophysiological responses during WVEs. The data allowed for an objective evaluation of the WVA impact on pilots' operation, behaviour and mental states (mental workload, stress levels and arousal). In particular, the results highlighted the effectiveness of the alert system in facilitating pilots' preparation, awareness and crew resource management (CRM). The results also highlighted the importance of avionics able to enhance aviation safety and reducing risks associated with wake vortex encounters. In particular, we demonstrated how providing timely information and improving situational awareness, the WVA will minimize the occurrence of WVEs and contribute to safer aviation operations.

Towards electrophysiological measurement of presence in virtual reality through auditory oddball stimuli

Objective.Presence is an important aspect of user experience in virtual reality (VR). It corresponds to the illusion of being physically located in a virtual environment (VE). This feeling is usually measured through questionnaires that disrupt presence, are subjective and do not allow for real-time measurement. Electroencephalography (EEG), which measures brain activity, is increasingly used to monitor the state of users, especially while immersed in VR.Approach.In this paper, we present a way of evaluating presence, through the measure of the attention dedicated to the real environment via an EEG oddball paradigm. Using breaks in presence, this experimental protocol constitutes an ecological method for the study of presence, as different levels of presence are experienced in an identical VE.Main results.Through analysing the EEG data of 18 participants, a significant increase in the neurophysiological reaction to the oddball, i.e. the P300 amplitude, was found in low presence condition compared to high presence condition. This amplitude was significantly correlated with the self-reported measure of presence. Using Riemannian geometry to perform single-trial classification, we present a classification algorithm with 79% accuracy in detecting between two presence conditions.Significance.Taken together our results promote the use of EEG and oddball stimuli to monitor presence offline or in real-time without interrupting the user in the VE.

Unsupervised Characterization of Prediction Error Markers in Unisensory and Multisensory Streams Reveal the Spatiotemporal Hierarchy of Cortical Information Processing

Elicited upon violation of regularity in stimulus presentation, mismatch negativity (MMN) reflects the brain's ability to perform automatic comparisons between consecutive stimuli and provides an electrophysiological index of sensory error detection whereas P300 is associated with cognitive processes such as updating of the working memory. To date, there has been extensive research on the roles of MMN and P300 individually, because of their potential to be used as clinical markers of consciousness and attention, respectively. Here, we intend to explore with an unsupervised and rigorous source estimation approach, the underlying cortical generators of MMN and P300, in the context of prediction error propagation along the hierarchies of brain information processing in healthy human participants. The existing methods of characterizing the two ERPs involve only approximate estimations of their amplitudes and latencies based on specific sensors of interest. Our objective is twofold: first, we introduce a novel data-driven unsupervised approach to compute latencies and amplitude of ERP components accurately on an individual-subject basis and reconfirm earlier findings. Second, we demonstrate that in multisensory environments, MMN generators seem to reflect a significant overlap of "modality-specific" and "modality-independent" information processing while P300 generators mark a shift toward completely "modality-independent" processing. Advancing earlier understanding that multisensory contexts speed up early sensory processing, our study reveals that temporal facilitation extends to even the later components of prediction error processing, using EEG experiments. Such knowledge can be of value to clinical research for characterizing the key developmental stages of lifespan aging, schizophrenia, and depression.

Privacy-Preserving Gaze Data Streaming in Immersive Interactive Virtual Reality: Robustness and User Experience

Eye tracking is routinely being incorporated into virtual reality (VR) systems. Prior research has shown that eye tracking data, if exposed, can be used for re-identification attacks [14]. The state of our knowledge about currently existing privacy mechanisms is limited to privacy-utility trade-off curves based on data-centric metrics of utility, such as prediction error, and black-box threat models. We propose that for interactive VR applications, it is essential to consider user-centric notions of utility and a variety of threat models. We develop a methodology to evaluate real-time privacy mechanisms for interactive VR applications that incorporate subjective user experience and task performance metrics. We evaluate selected privacy mechanisms using this methodology and find that re-identification accuracy can be decreased to as low as 14% while maintaining a high usability score and reasonable task performance. Finally, we elucidate three threat scenarios (black-box, black-box with exemplars, and white-box) and assess how well the different privacy mechanisms hold up to these adversarial scenarios. This work advances the state of the art in VR privacy by providing a methodology for end-to-end assessment of the risk of re-identification attacks and potential mitigating solutions. f.

The role of attention in immersion: The two-competitor model

Currently, we face an exponentially increasing interest in immersion, especially sensory-driven immersion, mainly due to the rapid development of ideas and business models centered around a digital virtual universe as well as the increasing availability of affordable immersive technologies for education, communication, and entertainment. However, a clear definition of 'immersion', in terms of established neurocognitive concepts and measurable properties, remains elusive, slowing research on the human side of immersive interfaces. To address this problem, we propose a conceptual, taxonomic model of attention in immersion. We argue (a) modeling immersion theoretically as well as studying immersion experimentally requires a detailed characterization of the role of attention in immersion, even though (b) attention, while necessary, cannot be a sufficient condition for defining immersion. Our broader goal is to characterize immersion in terms that will be compatible with established psychophysiolgical measures that could then in principle be used for the assessment and eventually the optimization of an immersive experience. We start from the perspective that immersion requires the projection of attention to an induced reality, and build on accepted taxonomies of different modes of attention for the development of our two-competitor model. The two-competitor model allows for a quantitative implementation and has an easy graphical interpretation. It helps to highlight the important link between different modes of attention and affect in studying immersion.

Cortical cellular encoding of thermotactile integration

Recent evidence suggests that primary sensory cortical regions play a role in the integration of information from multiple sensory modalities. How primary cortical neurons integrate different sources of sensory information is unclear, partly because non-primary sensory input to a cortical sensory region is often weak or modulatory. To address this question, we take advantage of the robust representation of thermal (cooling) and tactile stimuli in mouse forelimb primary somatosensory cortex (fS1). Using a thermotactile detection task, we show that the perception of threshold-level cool or tactile information is enhanced when they are presented simultaneously, compared with presentation alone. To investigate the cortical cellular correlates of thermotactile integration, we performed in vivo extracellular recordings from fS1 in awake resting and anesthetized mice during unimodal and bimodal stimulation of the forepaw. Unimodal stimulation evoked thermal- or tactile- specific excitatory and inhibitory responses of fS1 neurons. The most prominent features of combined thermotactile stimulation are the recruitment of unimodally silent fS1 neurons, non-linear integration features, and response dynamics that favor longer response durations with additional spikes. Together, we identify quantitative and qualitative changes in cortical encoding that may underlie the improvement in perception of thermotactile surfaces during haptic exploration.

iVR-fNIRS: studying brain functions in a fully immersive virtual environment

Immersive virtual reality (iVR) employs head-mounted displays or cave-like environments to create a sensory-rich virtual experience that simulates the physical presence of a user in a digital space. The technology holds immense promise in neuroscience research and therapy. In particular, virtual reality (VR) technologies facilitate the development of diverse tasks and scenarios closely mirroring real-life situations to stimulate the brain within a controlled and secure setting. It also offers a cost-effective solution in providing a similar sense of interaction to users when conventional stimulation methods are limited or unfeasible. Although combining iVR with traditional brain imaging techniques may be difficult due to signal interference or instrumental issues, recent work has proposed the use of functional near infrared spectroscopy (fNIRS) in conjunction with iVR for versatile brain stimulation paradigms and flexible examination of brain responses. We present a comprehensive review of current research studies employing an iVR-fNIRS setup, covering device types, stimulation approaches, data analysis methods, and major scientific findings. The literature demonstrates a high potential for iVR-fNIRS to explore various types of cognitive, behavioral, and motor functions in a fully immersive VR (iVR) environment. Such studies should set a foundation for adaptive iVR programs for both training (e.g., in novel environments) and clinical therapeutics (e.g., pain, motor and sensory disorders and other psychiatric conditions).

Virtual reality exposure with vibrotactile stimulation for the treatment of fear of flying: A pilot study

BACKGROUND AND OBJECTIVES

Virtual reality (VR) interventions are becoming more prevalent in treating fear of flying (FoF). Since multisensory stimulation can enhance the sense of presence in a virtual environment, the present study compared virtual reality exposure with and without vibrotactile cues to determine its contribution to the realism of the virtual experience.

METHODS

A repeated measures design was used. Thirty-one participants were exposed to two experimental conditions with a minimum of a one-week interval between them: one in which participants were exposed to the virtual environment with vibrotactile cues (smart chair, SC), and another in which participants were exposed to the virtual environment without vibrotactile cues (ordinary chair, OC). The administration order of both conditions was counterbalanced to avoid possible order effects.

RESULTS

Participants felt higher levels of sense of presence when using the SC than the OC. However, the addition of vibrotactile stimulation partially influenced experienced anxiety. Some personality traits were also associated with participants' sense of presence and anxiety responses during the exposure.

LIMITATIONS

The sample size was smaller than required. Moreover, only self-reported measures were used. Finally, a roller coaster instead of an airplane scenario was used for the exposure, which might not have been suitable enough for provoking anxiety in participants with FoF.

CONCLUSIONS

Vibrotactile cues enhanced the sense of presence. However, the addition of vibrotactile stimulation did not have a consistent effect on anxiety experienced during exposure. Therefore, the benefits of incorporating vibrotactile cues in virtual reality environments for exposure therapy are not clear.

Multisensory integration deficits in Schizophrenia and Autism evidenced in behaviour but not event related potentials

The process of integrating information from different sensory channels, known as multisensory integration (MSI) was assessed in two disorders, Autism Spectrum Disorder (ASD) and Schizophrenia (SCZ). 32 healthy controls (HC), 35 SCZ patients, and 23 ASD patients performed an audiovisual (AV) synchronous target detection task while reaction time (RT) and scalp recorded electrophysiological (EEG) activity were measured. MSI in the AV condition resulted in faster and less variable RTs compared to the unimodal conditions. Using our novel bootstrap method, MSI gain was observed in 78 % of HC, 26 % of ASD, and 48 % of SCZ patients. At the neural level, MSI in the AV condition resulted in larger amplitude of sensory evoked responses and cognitive P3 response compared to the corresponding unimodal conditions. These neural effects of MSI were not related to the behavioural MSI gain identified at the individual level and could not explain the deficits in behavioural MSI of patient groups. In conclusion, a robust MSI gain deficit in RT was observed in both patient groups that was not reflected in early perceptual and cognitive electro-cortical responses, suggesting that behavioural MSI deficits in ASD and SCZ may arise at late processing stages such as response selection.

Distinguishing Laparoscopic Surgery Experts from Novices Using EEG Topographic Features

The study aimed to differentiate experts from novices in laparoscopic surgery tasks using electroencephalogram (EEG) topographic features. A microstate-based common spatial pattern (CSP) analysis with linear discriminant analysis (LDA) was compared to a topography-preserving convolutional neural network (CNN) approach. Expert surgeons (N = 10) and novice medical residents (N = 13) performed laparoscopic suturing tasks, and EEG data from 8 experts and 13 novices were analysed. Microstate-based CSP with LDA revealed distinct spatial patterns in the frontal and parietal cortices for experts, while novices showed frontal cortex involvement. The 3D CNN model (ESNet) demonstrated a superior classification performance (accuracy > 98%, sensitivity 99.30%, specificity 99.70%, F1 score 98.51%, MCC 97.56%) compared to the microstate based CSP analysis with LDA (accuracy ~90%). Combining spatial and temporal information in the 3D CNN model enhanced classifier accuracy and highlighted the importance of the parietal-temporal-occipital association region in differentiating experts and novices.

Audiovisual biofeedback amplifies plantarflexor adaptation during walking among children with cerebral palsy

BACKGROUND

Biofeedback is a promising noninvasive strategy to enhance gait training among individuals with cerebral palsy (CP). Commonly, biofeedback systems are designed to guide movement correction using audio, visual, or sensorimotor (i.e., tactile or proprioceptive) cues, each of which has demonstrated measurable success in CP. However, it is currently unclear how the modality of biofeedback may influence user response which has significant implications if systems are to be consistently adopted into clinical care.

METHODS

In this study, we evaluated the extent to which adolescents with CP (7M/1F; 14 [12.5,15.5] years) adapted their gait patterns during treadmill walking (6 min/modality) with audiovisual (AV), sensorimotor (SM), and combined AV + SM biofeedback before and after four acclimation sessions (20 min/session) and at a two-week follow-up. Both biofeedback systems were designed to target plantarflexor activity on the more-affected limb, as these muscles are commonly impaired in CP and impact walking function. SM biofeedback was administered using a resistive ankle exoskeleton and AV biofeedback displayed soleus activity from electromyography recordings during gait. At every visit, we measured the time-course response to each biofeedback modality to understand how the rate and magnitude of gait adaptation differed between modalities and following acclimation.

RESULTS

Participants significantly increased soleus activity from baseline using AV + SM (42.8% [15.1, 59.6]), AV (28.5% [19.2, 58.5]), and SM (10.3% [3.2, 15.2]) biofeedback, but the rate of soleus adaptation was faster using AV + SM biofeedback than either modality alone. Further, SM-only biofeedback produced small initial increases in plantarflexor activity, but these responses were transient within and across sessions (p > 0.11). Following multi-session acclimation and at the two-week follow-up, responses to AV and AV + SM biofeedback were maintained.

CONCLUSIONS

This study demonstrated that AV biofeedback was critical to increase plantarflexor engagement during walking, but that combining AV and SM modalities further amplified the rate of gait adaptation. Beyond improving our understanding of how individuals may differentially prioritize distinct forms of afferent information, outcomes from this study may inform the design and selection of biofeedback systems for use in clinical care.

A Neuroergonomic Approach Fostered by Wearable EEG for the Multimodal Assessment of Drivers Trainees

When assessing trainees' progresses during a driving training program, instructors can only rely on the evaluation of a trainee's explicit behavior and their performance, without having any insight about the training effects at a cognitive level. However, being able to drive does not imply knowing how to drive safely in a complex scenario such as the road traffic. Indeed, the latter point involves mental aspects, such as the ability to manage and allocate one's mental effort appropriately, which are difficult to assess objectively. In this scenario, this study investigates the validity of deploying an electroencephalographic neurometric of mental effort, obtained through a wearable electroencephalographic device, to improve the assessment of the trainee. The study engaged 22 young people, without or with limited driving experience. They were asked to drive along five different but similar urban routes, while their brain activity was recorded through electroencephalography. Moreover, driving performance, subjective and reaction times measures were collected for a multimodal analysis. In terms of subjective and performance measures, no driving improvement could be detected either through the driver's subjective measures or through their driving performance. On the other side, through the electroencephalographic neurometric of mental effort, it was possible to catch their improvement in terms of mental performance, with a decrease in experienced mental demand after three repetitions of the driving training tasks. These results were confirmed by the analysis of reaction times, that significantly improved from the third repetition as well. Therefore, being able to measure when a task is less mentally demanding, and so more automatic, allows to deduce the degree of users training, becoming capable of handling additional tasks and reacting to unexpected events.

No pain, no gain? The effects of adding a pain stimulus in virtual training for police officers

Virtual training systems provide highly realistic training environments for police. This study assesses whether a pain stimulus can enhance the training responses and sense of the presence of these systems. Police officers (n = 219) were trained either with or without a pain stimulus in a 2D simulator (VirTra V-300) and a 3D virtual reality (VR) system. Two (training simulator) × 2 (pain stimulus) ANOVAs revealed a significant interaction effect for perceived stress (p = .010, ηp2 = .039). Post-hoc pairwise comparisons showed that VR provokes significantly higher levels of perceived stress compared to VirTra when no pain stimulus is used (p = .009). With a pain stimulus, VirTra training provokes significantly higher levels of perceived stress compared to VirTra training without a pain stimulus (p < .001). Sense of presence was unaffected by the pain stimulus in both training systems. Our results indicate that VR training appears sufficiently realistic without adding a pain stimulus. Practitioner summary: Virtual police training benefits from highly realistic training environments. This study found that adding a pain stimulus heightened perceived stress in a 2D simulator, whereas it influenced neither training responses nor sense of presence in a VR system. VR training appears sufficiently realistic without adding a pain stimulus.

Beyond performance: the role of task demand, effort, and individual differences in ab initio pilots

Aviation safety depends on the skill and expertise of pilots to meet the task demands of flying an aircraft in an effective and efficient manner. During flight training, students may respond differently to imposed task demands based on individual differences in capacity, physiological arousal, and effort. To ensure that pilots achieve a common desired level of expertise, training programs should account for individual differences to optimize pilot performance. This study investigates the relationship between task performance and physiological correlates of effort in ab initio pilots. Twenty-four participants conducted a flight simulator task with three difficulty levels and were asked to rate their perceived demand and effort using the NASA TLX. We recorded heart rate, EEG brain activity, and pupil size to assess changes in the participants' mental and physiological states across different task demands. We found that, despite group-level correlations between performance error and physiological responses, individual differences in physiological responses to task demands reflected different levels of participant effort and task efficiency. These findings suggest that physiological monitoring of student pilots might provide beneficial insights to flight instructors to optimize pilot training at the individual level.

sEMG-controlled forearm bracelet and serious game-based rehabilitation for training manual dexterity in people with multiple sclerosis: a randomised controlled trial

BACKGROUND

Muscle strength and dexterity impairments are common among patients with multiple sclerosis (MS) producing limitations in activities of daily living related to the upper limb (UL). This study aimed to evaluate the effectiveness of serious games specifically developed for the MYO Armband® capture sensor in improving forearm and wrist mobility, UL muscle strength, dexterity, fatigue, functionality, quality of life, satisfaction, adverse effects and compliance.

METHODS

A double-blinded (allocation concealment was performed by a blinded investigator and by blinding for assessors) randomised controlled trial was conducted. The sample was randomised into two groups: an experimental group that received treatment based on UL serious games designed by the research team and controlled by the MYO Armband® gesture capture sensor, along with conventional rehabilitation and a control group that received the same conventional rehabilitation for the UL. Both groups received two 60-min sessions per week over an eight-week period. Wrist range of motion (goniometry), grip muscle strength (Jamar® dynamometer), coordination and gross UL dexterity (Box and Block Test), fatigue (Fatigue Severity Scale), functionality (ABILHAND), quality of life (Multiple Sclerosis Impact Scale-29), adverse effects (Simulator Sickness Questionnaire, SSQ), perceived workload (NASA-Task load index), satisfaction (Client Satisfaction Questionnaire-8 (CSQ-8), Satisfaction with Technology Scale, System Usability Scale (SUS) and QUEST 2.0) and compliance (attendance) were assessed in both groups pre-treatment, post-treatment and during a follow-up period of 2 weeks without receiving any treatment.

RESULTS

Significant differences were observed in the experimental group compared to the control group in the assessment of forearm supination (p = .004) and grip strength (p = .004). Adverse effects were minimal (SSQ: 7/100 points) and perceived workload was low (NASA-Task Load Index: 25/100 points) in the experimental group. The MYO Armband® technology proved to be useful for the participants (SUS: 80.66/100) and the satisfaction scales received high scores (QUEST 2.0: 59.4/70 points; Satisfaction with Technology: 84.36/100 points). There were significant differences between the groups in terms of attendance percentage (p = .029).

CONCLUSIONS

An experimental protocol using MYO Armband®-based serious games designed for UL rehabilitation showed improvements in active wrist range of motion and handgrip strength in patients with MS, with high satisfaction, minimal adverse effects and workload and excellent compliance.

TRIAL REGISTRATION NUMBER

This randomised controlled trial has been registered at ClinicalTrials.gov Identifier: NCT04171908.

Attention Load Regulates the Facilitation of Audio-Visual Information on Landing Perception in Badminton

Based on the role of the high temporal sensitivity of the auditory modality and the advantage of audio-visual integration in motion perception and anticipation, we investigated the effect of audio-visual information on landing perception in badminton through two experiments; and we explored the regulatory role of attention load. In this study, experienced badminton players were asked to predict the landing position of the shuttle under the conditions of video (visual) or audio-video (audio-visual) presentation. We manipulated flight information or attention load. The results of Experiment 1 showed that, whether the visual information was rich or not, that is, whether or not it contained the early flight trajectory, the addition of auditory information played a promoting role. The results of Experiment 2 showed that attention load regulated the facilitation of multi-modal integration on landing perception. The facilitation of audio-visual information was impaired under high load, meaning that audio-visual integration tended to be guided by attention from top to bottom. The results support the superiority effect of multi-modal integration, suggesting that adding auditory perception training to sports training could significantly improve athletes' performance.

A novel multisensory device for the assessment and rehabilitation of perceptual and attentional competencies

The present study aims to assess a novel technological device suitable for investigating perceptual and attentional competencies in people with or without sensory impairment. The TechPAD is a cabled system including embedded sensors and actuators to enable visual, auditory, and tactile interactions and a capacitive surface receiving inputs from the user. The system is conceived to create multisensory environments, using multiple units controlled separately and simultaneously. We assessed the device by adapting a spatial attention task comparing performances in different cognitive load conditions (high or low) and stimulation (unimodal, bimodal, or trimodal). 28 sighted adults were asked to monitor both the central and peripheral parts of the device and to tap a target stimulus (either visual, auditory, haptic, or multimodal) as fast as they could. Our results suggest that this new device can provide congruent and incongruent multimodal stimuli and quantitatively measure parameters such as reaction time and accuracy, allowing to investigate perceptual mechanisms in multisensory environments.Clinical Relevance-The TechPad is a reliable tool for the assessment of spatial attention during interactive tasks. its application in clinical trials will pave the way to its role in multisensory rehabilitation.

Sound Improves Neuronal Encoding of Visual Stimuli in Mouse Primary Visual Cortex

In everyday life, we integrate visual and auditory information in routine tasks such as navigation and communication. While concurrent sound can improve visual perception, the neuronal correlates of audiovisual integration are not fully understood. Specifically, it remains unclear whether neuronal firing patters in the primary visual cortex (V1) of awake animals demonstrate similar sound-induced improvement in visual discriminability. Furthermore, presentation of sound is associated with movement in the subjects, but little is understood about whether and how sound-associated movement affects audiovisual integration in V1. Here, we investigated how sound and movement interact to modulate V1 visual responses in awake, head-fixed mice and whether this interaction improves neuronal encoding of the visual stimulus. We presented visual drifting gratings with and without simultaneous auditory white noise to awake mice while recording mouse movement and V1 neuronal activity. Sound modulated activity of 80% of light-responsive neurons, with 95% of neurons increasing activity when the auditory stimulus was present. A generalized linear model (GLM) revealed that sound and movement had distinct and complementary effects of the neuronal visual responses. Furthermore, decoding of the visual stimulus from the neuronal activity was improved with sound, an effect that persisted even when controlling for movement. These results demonstrate that sound and movement modulate visual responses in complementary ways, improving neuronal representation of the visual stimulus. This study clarifies the role of movement as a potential confound in neuronal audiovisual responses and expands our knowledge of how multimodal processing is mediated at a neuronal level in the awake brain.SIGNIFICANCE STATEMENT Sound and movement are both known to modulate visual responses in the primary visual cortex; however, sound-induced movement has largely remained unaccounted for as a potential confound in audiovisual studies in awake animals. Here, authors found that sound and movement both modulate visual responses in an important visual brain area, the primary visual cortex, in distinct, yet complementary ways. Furthermore, sound improved encoding of the visual stimulus even when accounting for movement. This study reconciles contrasting theories on the mechanism underlying audiovisual integration and asserts the primary visual cortex as a key brain region participating in tripartite sensory interactions.

Health Implications of Virtual Architecture: An Interdisciplinary Exploration of the Transferability of Findings from Neuroarchitecture

Virtual architecture has been increasingly relied on to evaluate the health impacts of physical architecture. In this health research, exposure to virtual architecture has been used as a proxy for exposure to physical architecture. Despite the growing body of research on the health implications of physical architecture, there is a paucity of research examining the long-term health impacts of prolonged exposure to virtual architecture. In response, this paper considers: what can proxy studies, which use virtual architecture to assess the physiological response to physical architecture, tell us about the impact of extended exposure to virtual architecture on human health? The paper goes on to suggest that the applicability of these findings to virtual architecture may be limited by certain confounding variables when virtual architecture is experienced for a prolonged period of time. This paper explores the potential impact of two of these confounding variables: multisensory integration and gravitational perception. This paper advises that these confounding variables are unique to extended virtual architecture exposure and may not be captured by proxy studies that aim to capture the impact of physical architecture on human health through acute exposure to virtual architecture. While proxy studies may be suitable for measuring some aspects of the impact of both physical and virtual architecture on human health, this paper argues that they may be insufficient to fully capture the unintended consequences of extended exposure to virtual architecture on human health. Therefore, in the face of the increasing use of virtual architectural environments, the author calls for the establishment of a subfield of neuroarchitectural health research that empirically examines the physiological impacts of extended exposure to virtual architecture in its own right.

Ranking Crossing Scenario Complexity for eHMIs Testing: A Virtual Reality Study

External human–machine interfaces (eHMIs) have the potential to benefit AV–pedestrian interactions. The majority of studies investigating eHMIs have used relatively simple traffic environments, i.e., a single pedestrian crossing in front of a single eHMI on a one-lane straight road. While this approach has proved to be efficient in providing an initial understanding of how pedestrians respond to eHMIs, it over-simplifies interactions which will be substantially more complex in real-life circumstances. A process is illustrated in a small-scale study (N = 10) to rank different crossing scenarios by level of complexity. Traffic scenarios were first developed for varying traffic density, visual complexity of the road scene, road geometry, weather and visibility conditions, and presence of distractions. These factors have been previously shown to increase difficulty and riskiness of the crossing task. The scenarios were then tested in a motion-based, virtual reality environment. Pedestrians’ perceived workload and objective crossing behaviour were measured as indirect indicators of the level of complexity of the crossing scenario. Sense of presence and simulator sickness were also recorded as a measure of the ecological validity of the virtual environment. The results indicated that some crossing scenarios were more taxing for pedestrians than others, such as those with road geometries where traffic approached from multiple directions. Further, the presence scores showed that the virtual environments experienced were found to be realistic. This paper concludes by proposing a “complex” environment to test eHMIs under more challenging crossing circumstances.

Evoked sensory stimulation of the eating environment, impacts feeling of presence and food desires in an online environment

Online food choices are often made outside a regular food environment and suffer from sensory deprivation. The present study investigated if evoked multi-sensory stimulation can drive context specific food desires in an online environment. In a randomised between subject design, participants expressed their food desire on a visual analogue scale and feeling of presence (e.g., did you feel present on a beach) on a Likert scale, whilst looking online at a picture and reading a neutral description of a sensory laboratory (control condition), looking at a photo of a beach and reading a neutral description (beach condition), or looking at a photo of a beach and reading a sensory based description (beach + ). Participants (n = 725 participants, 622 females) who saw the beach photo increased their desire for cold, but not neutral foods (p < 0.05), those who were exposed to the sensory description in addition to the photo showed a higher desire for cold foods compared to those who just saw the beach photo (p < 0.001). These effects were modulated by an increased feeling of presence and how often participants visited the beach. Participants with a higher feeling of presence showed a higher desire for cold foods (p < 0.05). Food desires of those who visited the beach often were more impacted by the evoked sensory stimulation than food desires of those who visited the beach rarely. Food desires created in an online environment can be influenced by visual, and text based evoked sensory stimulation as long as consumers' feeling of presence in is high. The results can inform public health professionals how to impact healthy food choices in an online environment.

Directed information flow during laparoscopic surgical skill acquisition dissociated skill level and medical simulation technology

Virtual reality (VR) simulator has emerged as a laparoscopic surgical skill training tool that needs validation using brain-behavior analysis. Therefore, brain network and skilled behavior relationship were evaluated using functional near-infrared spectroscopy (fNIRS) from seven experienced right-handed surgeons and six right-handed medical students during the performance of Fundamentals of Laparoscopic Surgery (FLS) pattern of cutting tasks in a physical and a VR simulator. Multiple regression and path analysis (MRPA) found that the FLS performance score was statistically significantly related to the interregional directed functional connectivity from the right prefrontal cortex to the supplementary motor area with F (2, 114) = 9, p < 0.001, and R² = 0.136. Additionally, a two-way multivariate analysis of variance (MANOVA) found a statistically significant effect of the simulator technology on the interregional directed functional connectivity from the right prefrontal cortex to the left primary motor cortex (F (1, 15) = 6.002, p = 0.027; partial η² = 0.286) that can be related to differential right-lateralized executive control of attention. Then, MRPA found that the coefficient of variation (CoV) of the FLS performance score was statistically significantly associated with the CoV of the interregionally directed functional connectivity from the right primary motor cortex to the left primary motor cortex and the left primary motor cortex to the left prefrontal cortex with F (2, 22) = 3.912, p = 0.035, and R² = 0.262. This highlighted the importance of the efference copy information from the motor cortices to the prefrontal cortex for postulated left-lateralized perceptual decision-making to reduce behavioral variability.

EEG-Based Index for Timely Detecting User's Drowsiness Occurrence in Automotive Applications

Human errors are widely considered among the major causes of road accidents. Furthermore, it is estimated that more than 90% of vehicle crashes causing fatal and permanent injuries are directly related to mental tiredness, fatigue, and drowsiness of the drivers. In particular, driving drowsiness is recognized as a crucial aspect in the context of road safety, since drowsy drivers can suddenly lose control of the car. Moreover, the driving drowsiness episodes mostly appear suddenly without any prior behavioral evidence. The present study aimed at characterizing the onset of drowsiness in car drivers by means of a multimodal neurophysiological approach to develop a synthetic electroencephalographic (EEG)-based index, able to detect drowsy events. The study involved 19 participants in a simulated scenario structured in a sequence of driving tasks under different situations and traffic conditions. The experimental conditions were designed to induce prominent mental drowsiness in the final part. The EEG-based index, so-called “MDrow index”, was developed and validated to detect the driving drowsiness of the participants. The MDrow index was derived from the Global Field Power calculated in the Alpha EEG frequency band over the parietal brain sites. The results demonstrated the reliability of the proposed MDrow index in detecting the driving drowsiness experienced by the participants, resulting also more sensitive and timely sensible with respect to more conventional autonomic parameters, such as the EyeBlinks Rate and the Heart Rate Variability, and to subjective measurements (self-reports).

Influence of Auditory Cues on the Neuronal Response to Naturalistic Visual Stimuli in a Virtual Reality Setting

Virtual reality environments offer great opportunities to study the performance of brain-computer interfaces (BCIs) in real-world contexts. As real-world stimuli are typically multimodal, their neuronal integration elicits complex response patterns. To investigate the effect of additional auditory cues on the processing of visual information, we used virtual reality to mimic safety-related events in an industrial environment while we concomitantly recorded electroencephalography (EEG) signals. We simulated a box traveling on a conveyor belt system where two types of stimuli – an exploding and a burning box – interrupt regular operation. The recordings from 16 subjects were divided into two subsets, a visual-only and an audio-visual experiment. In the visual-only experiment, the response patterns for both stimuli elicited a similar pattern – a visual evoked potential (VEP) followed by an event-related potential (ERP) over the occipital-parietal lobe. Moreover, we found the perceived severity of the event to be reflected in the signal amplitude. Interestingly, the additional auditory cues had a twofold effect on the previous findings: The P1 component was significantly suppressed in the case of the exploding box stimulus, whereas the N2c showed an enhancement for the burning box stimulus. This result highlights the impact of multisensory integration on the performance of realistic BCI applications. Indeed, we observed alterations in the offline classification accuracy for a detection task based on a mixed feature extraction (variance, power spectral density, and discrete wavelet transform) and a support vector machine classifier. In the case of the explosion, the accuracy slightly decreased by –1.64% p. in an audio-visual experiment compared to the visual-only. Contrarily, the classification accuracy for the burning box increased by 5.58% p. when additional auditory cues were present. Hence, we conclude, that especially in challenging detection tasks, it is favorable to consider the potential of multisensory integration when BCIs are supposed to operate under (multimodal) real-world conditions.

Developing the Rationale for Including Virtual Reality in Cognitive Rehabilitation and Exercise Training Approaches for Managing Cognitive Dysfunction in MS

Cognitive impairment is a common and detrimental consequence of multiple sclerosis (MS) and current rehabilitation methods are insufficient. Cognitive rehabilitation (CR) and exercise training (ET) are the most promising behavioral approaches to mitigate cognitive deficits, but effects are small and do not effectively translate to improvements in everyday function. This article presents a conceptual framework supporting the use of virtual reality (VR) as an ideal, common adjuvant traditional CR and ET in MS. VR could strengthen the effects of CR and ET by increasing sensory input and promoting multisensory integration and processing during rehabilitation. For ET specifically, VR can also help incorporate components of CR into exercise sessions. In addition, VR can enhance the transfer of cognitive improvements to everyday functioning by providing a more ecologically valid training environment. There is a clear interest in adding VR to traditional rehabilitation techniques for neurological populations, a stronger body of evidence of this unique approach is needed in MS. Finally, to better understand how to best utilize VR in rehabilitation for cognitive deficits in MS, more systematic research is needed to better understand the mechanism(s) of action of VR with CR and ET.

Design of Proactive Interaction for In-Vehicle Robots Based on Transparency

Based on the transparency theory, this study investigates the appropriate amount of transparency information expressed by the in-vehicle robot under two channels of voice and visual in a proactive interaction scenario. The experiments are to test and evaluate different transparency levels and combinations of information in different channels of the in-vehicle robot, based on a driving simulator to collect subjective and objective data, which focuses on users' safety, usability, trust, and emotion dimensions under driving conditions. The results show that appropriate transparency expression is able to improve drivers' driving control and subjective evaluation and that drivers need a different amount of transparency information in different types of tasks.

Studying the Effects of Congruence of Auditory and Visual Stimuli on Virtual Reality Experiences

Studies in virtual reality (VR) have introduced numerous multisensory simulation techniques for more immersive VR experiences. However, although they primarily focus on expanding sensory types or increasing individual sensory quality, they lack consensus in designing appropriate interactions between different sensory stimuli. This paper explores how the congruence between auditory and visual (AV) stimuli, which are the sensory stimuli typically provided by VR devices, affects the cognition and experience of VR users as a critical interaction factor in promoting multisensory integration. We defined the types of (in)congruence between AV stimuli, and then designed 12 virtual spaces with different types or degrees of congruence between AV stimuli. We then evaluated the presence, immersion, motion sickness, and cognition changes in each space. We observed the following key findings: 1) there is a limit to the degree of temporal or spatial incongruence that can be tolerated, with few negative effects on user experience until that point is exceeded; 2) users are tolerant of semantic incongruence; 3) a simulation that considers synesthetic congruence contributes to the user's sense of immersion and presence. Based on these insights, we identified the essential considerations for designing sensory simulations in VR and proposed future research directions.

Validation of a Light EEG-Based Measure for Real-Time Stress Monitoring during Realistic Driving

Driver’s stress affects decision-making and the probability of risk occurrence, and it is therefore a key factor in road safety. This suggests the need for continuous stress monitoring. This work aims at validating a stress neurophysiological measure—a Neurometric—for out-of-the-lab use obtained from lightweight EEG relying on two wet sensors, in real-time, and without calibration. The Neurometric was tested during a multitasking experiment and validated with a realistic driving simulator. Twenty subjects participated in the experiment, and the resulting stress Neurometric was compared with the Random Forest (RF) model, calibrated by using EEG features and both intra-subject and cross-task approaches. The Neurometric was also compared with a measure based on skin conductance level (SCL), representing one of the physiological parameters investigated in the literature mostly correlated with stress variations. We found that during both multitasking and realistic driving experiments, the Neurometric was able to discriminate between low and high levels of stress with an average Area Under Curve (AUC) value higher than 0.9. Furthermore, the stress Neurometric showed higher AUC and stability than both the SCL measure and the RF calibrated with a cross-task approach. In conclusion, the Neurometric proposed in this work proved to be suitable for out-of-the-lab monitoring of stress levels.

Hiding Assistive Robots During Training in Immersive VR Does not Affect Users' Motivation, Presence, Embodiment, Performance, nor Visual Attention

Combining immersive virtual reality (VR) using head-mounted displays (HMDs) with assisting robotic devices might be a promising procedure to enhance neurorehabilitation. However, it is still an open question how immersive virtual environments (VE) should be designed when interacting with rehabilitation robots. In conventional training, the robot is usually not visually represented in the VE, resulting in a visuo-haptic sensory conflict between what users see and feel. This study aimed to investigate how motivation, embodiment, and presence are affected by this visuo-haptic sensory conflict. Using an HMD and a rehabilitation robot, 28 healthy participants performed a path-tracing task, while the robot was either visually reproduced in the VE or not and while the robot either assisted the movements or not. Participants' performance and visual attention were measured during the tasks, and after each visibility/assistance condition, they reported their motivation, presence, and embodiment with questionnaires. We found that, independently of the assistance, the robot visibility did not affect participants' motivation, presence, embodiment, nor task performance. We only found a greater effort/importance reported when the robot was visible. The visual attention was also slightly affected by the robot's visibility. Importantly, we found that the robotic assistance hampered presence and embodiment, but improved motivation. Our results indicate no disadvantage of not reproducing robotic devices in VEs when using HMDs. However, caution must be put when developing assisting controllers, as they might hamper users' affect.

The Sample Size Matters: To What Extent the Participant Reduction Affects the Outcomes of a Neuroscientific Research. A Case-Study in Neuromarketing Field

The sample size is a crucial concern in scientific research and even more in behavioural neurosciences, where besides the best practice it is not always possible to reach large experimental samples. In this study we investigated how the outcomes of research change in response to sample size reduction. Three indices computed during a task involving the observations of four videos were considered in the analysis, two related to the brain electroencephalographic (EEG) activity and one to autonomic physiological measures, i.e., heart rate and skin conductance. The modifications of these indices were investigated considering five subgroups of sample size (32, 28, 24, 20, 16), each subgroup consisting of 630 different combinations made by bootstrapping n (n = sample size) out of 36 subjects, with respect to the total population (i.e., 36 subjects). The correlation analysis, the mean squared error (MSE), and the standard deviation (STD) of the indexes were studied at the participant reduction and three factors of influence were considered in the analysis: the type of index, the task, and its duration (time length). The findings showed a significant decrease of the correlation associated to the participant reduction as well as a significant increase of MSE and STD (p < 0.05). A threshold of subjects for which the outcomes remained significant and comparable was pointed out. The effects were to some extents sensitive to all the investigated variables, but the main effect was due to the task length. Therefore, the minimum threshold of subjects for which the outcomes were comparable increased at the reduction of the spot duration.

Effect of lighting illuminance and colour temperature on mental workload in an office setting

The mental workload of subjects was tested under different lighting conditions, with colour temperatures ranging from 3000 to 6500 K and illuminance ranging from 300 to 1000 lx. We used both psychological and physiological responses for evaluation. The former was based on NASA Task Load Index (NASA-TLX, NASA), and the latter was based on the electroencephalogram (EEG) P3b analysis of event-related potentials using the “oddball” paradigm experimental task. The results show that as illuminance increases, and the response time becomes longer with a colour temperature of 3000 K (P < 0.01). However, when the colour temperature is set at 6500 K, the response time becomes shorter as the illuminance increases (P < 0.01). P3b amplitudes were significantly affected by colour temperature (P = 0.009) and illuminance (P = 0.038) levels. The highest amplitudes occurred at 3000 K and 750 lx, which is consistent with the trend shown by the subjective scale. The data analysis of error rates is not significant. These results suggest that an office environment with a colour temperature of 3000 K and illumination of 750 lx, which exerts the lowest mental workload, is the most suitable for working. However, the interaction between colour temperature and illuminance in affecting the mental workload of participants is not clear. This work provides more appropriate lighting choices with colour temperature and illuminance to reduce people’s mental workload in office settings.