The Dynamics and Neural Correlates of Audio-Visual Integration Capacity as Determined by Temporal Unpredictability, Proactive Interference, and SOA

Attentional modulations of audiovisual interactions in apparent motion: Temporal ventriloquism effects on perceived visual speed

The timing of brief stationary sounds has been shown to alter different aspects of visual motion, such as speed estimation. These effects of auditory timing have been explained by temporal ventriloquism and auditory dominance over visual information in the temporal domain. Although previous studies provide unprecedented evidence for the multisensory nature of speed estimation, how attention is involved in these audiovisual interactions remains unclear. Here, we aimed to understand the effects of spatial attention on these audiovisual interactions in time. We utilized a set of audiovisual stimuli that elicit temporal ventriloquism in visual apparent motion and asked participants to perform a speed comparison task. We manipulated attention either in the visual or auditory domain and systematically changed the number of moving objects in the visual field. When attention was diverted to a stationary object in the visual field via a secondary task, the temporal ventriloquism effects on perceived speed decreased. On the other hand, focusing attention on the auditory stimuli facilitated these effects consistently across different difficulty levels of secondary auditory task. Moreover, the effects of auditory timing on perceived speed did not change with the number of moving objects and existed in all the experimental conditions. Taken together, our findings revealed differential effects of allocating attentional resources in the visual and auditory domains. These behavioral results also demonstrate that reliable temporal ventriloquism effects on visual motion can be induced even in the presence of multiple moving objects in the visual field and under different perceptual load conditions.

Individual differences in multiple object tracking, attentional cueing, and age account for variability in the capacity of audiovisual integration

There has been a recent increase in individual differences research within the field of audiovisual perception (Spence & Squire, 2003, Current Biology, 13(13), R519-R521), and furthering the understanding of audiovisual integration capacity with an individual differences approach is an important facet within this line of research. Across four experiments, participants were asked to complete an audiovisual integration capacity task (cf. Van der Burg, Awh, & Olivers, 2013, Psychological Science, 24(3), 345-351; Wilbiks & Dyson, 2016, PLOS ONE 11(12), e0168304; 2018, Journal of Experimental Psychology: Human Perception and Performance, 44(6), 871-884), along with differing combinations of additional perceptual tasks. Experiment 1 employed a multiple object tracking task and a visual working memory task. Experiment 2 compared performance on the capacity task with that of the Attention Network Test. Experiment 3 examined participants' focus in space through a Navon task and vigilance through time. Having completed this exploratory work, in Experiment 4 we collected data again from the tasks that were found to correlate significantly across the first three experiments and entered them into a regression model to predict capacity. The current research provides a preliminary explanation of the vast individual differences seen in audiovisual integration capacity in previous research, showing that by considering an individual's multiple object tracking span, focus in space, and attentional factors, we can account for up to 34.3% of the observed variation in capacity. Future research should seek to examine higher-level differences between individuals that may contribute to audiovisual integration capacity, including neurodevelopmental and mental health differences.

Musical Training Improves Audiovisual Integration Capacity under Conditions of High Perceptual Load

In considering capacity measures of audiovisual integration, it has become apparent that there is a wide degree of variation both within (based on unimodal and multimodal stimulus characteristics) and between participants. Recent work has discussed performance on a number of cognitive tasks that can form a regression model accounting for nearly a quarter of the variation in audiovisual integration capacity. The current study involves an investigation of whether different elements of musicality in participants can contribute to additional variation in capacity. Participants were presented with a series of rapidly changing visual displays and asked to note which elements of that display changed in synchrony with a tone. Results were fitted to a previously used model to establish capacity estimates, and these estimates were included in correlational analyses with musical training, musical perceptual abilities, and active engagement in music. We found that audiovisual integration capacity was positively correlated with amount of musical training, and that this correlation was statistically significant under the most difficult perceptual conditions. Results are discussed in the context of the boosting of perceptual abilities due to musical training, even under conditions that have been previously found to be overly demanding for participants.

Audiovisual integration capacity modulates as a function of illusory visual contours, visual display circumference, and sound type

Research into the capacity of audiovisual integration has previously assessed whether capacity is strictly limited to a single item, or whether it can exceed one item under certain environmental conditions. More recently, investigations have turned to examining the effects of various stimulus factors on capacity. Across two experiments, we looked at a number of factors that were expected to play a modulatory role on capacity. Experiment 1 deployed a manipulation of illusory polygons, revealing an increase in audiovisual capacity, even in an absence of visual connections. This demonstrates that exceeding the capacity of 1 does not only represent a functional increase in the binding of a singular, complex visual object, but that it can also represent binding of multiple simpler objects. Findings also support the hypothesis that capacity modulates quantitatively, but not qualitatively, with respect to speed of presentation. Experiment 2 examined the effects of different sound types (sine tones or white noise) and of different spatial visual field sizes on the capacity of audiovisual integration. The results indicate that capacity is maximized when stimuli are presented in a smaller circle (7.5°) if alongside a sine tone, and when presented in a larger circle (18.5°) alongside white noise. These results suggest that audiovisual integration capacity is dependent on the combination of sound type and visual spatial field size. The combination of these results reveal additional phenomenological features of the capacity of audiovisual integration, and provides impetus for further research into applications of the findings.