The influence of visual distance on the room-acoustic experience of auralizations

The Influence of Multisensory Input On Voice Perception and Production Using Immersive Virtual Reality

OBJECTIVES

The purpose was to examine the influence of auditory vs visual vs combined audiovisual input on perception and production of one's own voice, using immersive virtual reality technology.

METHODS

Thirty-one vocally healthy men and women were investigated under 18 sensory input conditions, using immersive virtual reality technology. Conditions included two auditory rooms with varying reverberation times, two visual rooms with varying volumes, and the combination of audiovisual conditions. All conditions were repeated with and without background noise. Speech tasks included counting, sustained vowel phonation, an all-voiced sentence from the Consensus Auditory-Perceptual Evaluation of Voice, and the first sentence from the Rainbow Passage, randomly ordered. Perception outcome measures were participants' self-reported perceptions of their vocal loudness, vocal effort, and vocal comfort in speech. Production outcome measures were sound pressure level (SPL) and spectral moments (spectral mean and standard deviation in Hz, skewness, and kurtosis). Statistical analyses used self-reported vocal effort, vocal loudness, and vocal comfort in percent (0 = "not at all," 100 = extremely), SPL in dB, and spectral moments in Hz. The reference level was a baseline audiovisual deprivation condition.

RESULTS

Results suggested (i) increased self-perceived vocal loudness and effort, and decreased comfort, with increasing room volume, speaker-to-listener distance, audiovisual input, and background noise, and (ii) increased SPL and fluctuations in spectral moments across conditions.

CONCLUSIONS

Not only auditory, but also visual and audiovisual input influenced voice perception and production in ways that have not been previously documented. Findings contribute to the basic science understanding the role of visual, audiovisual and auditory input in voice perception and production, and also to models of voice training and therapy. The findings also set the foundation for the use of virtual reality in voice and speech training, as a potentially power solution to the generalization problem.

Separate effects of auditory and visual room size on auditorium seat preference: a virtual reality study

Audiovisual integrations and interactions happen everywhere, including in music concerts, where combined visual and auditory perception contributes to overall enjoyment. Thirty-three participants evaluated their overall subjective preference at various seats in four virtual auditoria, which comprised congruent and incongruent auditory and visual renders of two auditoria that differ only in size. Results show no significant difference between participants who completed the experiment in a fully calibrated and standardized laboratory environment and participants who completed remotely using various VR equipment in various environments. Both visual and auditory auditorium size have significant main effects, but no interaction. The larger hall is preferred for both conditions. Audiovisual congruency does not significantly affect preference.

Auditory spatial analysis in reverberant multi-talker environments with congruent and incongruent audio-visual room information

In a multi-talker situation, listeners have the challenge of identifying a target speech source out of a mixture of interfering background noises. In the current study, it was investigated how listeners analyze audio-visual scenes with varying complexity in terms of number of talkers and reverberation. The visual information of the room was either congruent with the acoustic room or incongruent. The listeners' task was to locate an ongoing speech source in a mixture of other speech sources. The three-dimensional audio-visual scenarios were presented using a loudspeaker array and virtual reality glasses. It was shown that room reverberation, as well as the number of talkers in a scene, influence the ability to analyze an auditory scene in terms of accuracy and response time. Incongruent visual information of the room did not affect this ability. When few talkers were presented simultaneously, listeners were able to detect a target talker quickly and accurately even in adverse room acoustical conditions. Reverberation started to affect the response time when four or more talkers were presented. The number of talkers became a significant factor for five or more simultaneous talkers.

Best Distance Perception in Virtual Audiovisual Environment

Auditory distance perception is important to virtual sound production and is affected by many factors. In order to investigate the effect of the visual cue on auditory distance perception and the best auditory distance perception in the existence of virtual sound source, experiments on the effect of visual cue on distance perception and the best auditory distance perception were conducted, respectively. The results of the first experiment showed that there was no obvious difference between the auditory distance perception in the existence and absence of virtual sound source, but visual cue can decrease the fluctuation of the perception. From the best auditory distance perception experiment, the attenuating SPL of the initial sound signal that made the subjects perceive the best auditory distance in the existence of virtual sound source at the distance of 4m, 6m, 8m, 10m, and 12m was measured, respectively, and compared the difference of attenuating SPL between experiment measurement and theoretical calculation.

Spatial Knowledge via Auditory Information for Blind Individuals: Spatial Cognition Studies and the Use of Audio-VR

Spatial cognition is a daily life ability, developed in order to be able to understand and interact with our environment. Even if all the senses are involved in mental representation of space elaboration, the lack of vision makes it more difficult, especially because of the importance of peripheral information in updating the relative positions of surrounding landmarks when one is moving. Spatial audio technology has long been used for studies of human perception, particularly in the area of auditory source localisation. The ability to reproduce individual sounds at desired positions, or complex spatial audio scenes, without the need to manipulate physical devices has provided researchers with many benefits. We present a review of several studies employing the power of spatial audio virtual reality for research in spatial cognition with blind individuals. These include studies investigating simple spatial configurations, architectural navigation, reaching to sounds, and sound design for improved acceptability. Prospects for future research, including those currently underway, are also discussed.

Perceptual Matching of Room Acoustics for Auditory Augmented Reality in Small Rooms - Literature Review and Theoretical Framework

For the realization of auditory augmented reality (AAR), it is important that the room acoustical properties of the virtual elements are perceived in agreement with the acoustics of the actual environment. This perceptual matching of room acoustics is the subject reviewed in this paper. Realizations of AAR that fulfill the listeners' expectations were achieved based on pre-characterization of the room acoustics, for example, by measuring acoustic impulse responses or creating detailed room models for acoustic simulations. For future applications, the goal is to realize an online adaptation in (close to) real-time. Perfect physical matching is hard to achieve with these practical constraints. For this reason, an understanding of the essential psychoacoustic cues is of interest and will help to explore options for simplifications. This paper reviews a broad selection of previous studies and derives a theoretical framework to examine possibilities for psychoacoustical optimization of room acoustical matching.

Auditory-visual interactions in egocentric distance perception: Ventriloquism effect and aftereffect

This study describes data on auditory-visual integration and visually-guided adaptation of auditory distance perception using the ventriloquism effect (VE) and ventriloquism aftereffect (VAE). In an experiment, participants judged egocentric distance of interleaved auditory or auditory-visual stimuli with the auditory component located from 0.7 to 2.04 m in front of listeners in a real reverberant environment. The visual component of auditory-visual stimuli was displaced 30% closer (V-closer), 30% farther (V-farther), or aligned (V-aligned) with respect to the auditory component. The VE and VAE were measured in auditory and auditory-visual trials, respectively. Both effects were approximately independent of target distance when expressed in logarithmic units. The VE strength, defined as a difference of V-misaligned and V-aligned response bias, was approximately 72% of the auditory-visual disparity regardless of the visual-displacement direction, while the VAE was stronger in the V-farther (44%) than the V-closer (31%) condition. The VAE persisted to post-adaptation auditory-only blocks of trials, although it was diminished. The rates of build-up/break-down of the VAE were asymmetrical, with slower adaptation in the V-closer condition. These results suggest that auditory-visual distance integration is independent of the direction of induced shift, while the re-calibration is stronger and faster when evoked by more distant visual stimuli.

Ecological Validity of Immersive Virtual Reality (IVR) Techniques for the Perception of Urban Sound Environments

Immersive Virtual Reality (IVR) is a simulated technology used to deliver multisensory information to people under different environmental conditions. When IVR is generally applied in urban planning and soundscape research, it reveals attractive possibilities for the assessment of urban sound environments with higher immersion for human participation. In virtual sound environments, various topics and measures are designed to collect subjective responses from participants under simulated laboratory conditions. Soundscape or noise assessment studies during virtual experiences adopt an evaluation approach similar to in situ methods. This paper aims to review the approaches that are utilized to assess the ecological validity of IVR for the perception of urban sound environments and the necessary technologies during audio–visual reproduction to establish a dynamic IVR experience that ensures ecological validity. The review shows that, through the use of laboratory tests including subjective response surveys, cognitive performance tests and physiological responses, the ecological validity of IVR can be assessed for the perception of urban sound environments. The reproduction system with head-tracking functions synchronizing spatial audio and visual stimuli (e.g., head-mounted displays (HMDs) with first-order Ambisonics (FOA)-tracked binaural playback) represents the prevailing trend to achieve high ecological validity. These studies potentially contribute to the outcomes of a normalized evaluation framework for subjective soundscape and noise assessments in virtual environments.

The percept of reverberation is not affected by visual room impression in virtual environments

Humans possess mechanisms to suppress distracting early sound reflections, summarized as the precedence effect. Recent work shows that precedence is affected by visual stimulation. This paper investigates possible effects of visual stimulation on the perception of later reflections, i.e., reverberation. In a highly immersive audio-visual virtual reality environment, subjects were asked to quantify reverberation in conditions where simultaneously presented auditory and visual stimuli either match in room identity, sound source azimuth, and sound source distance, or diverge in one of these aspects. While subjects reliably judged reverberation across acoustic environments, the visual room impression did not affect reverberation estimates.