Listen to Your Heart: Examining Modality Dominance Using Cross-Modal Oddball Tasks

Increases in pre-stimulus theta and alpha oscillations precede successful encoding of crossmodal associations

A central aspect of episodic memory is the formation of associations between stimuli from different modalities. Current theoretical approaches assume a functional role of ongoing oscillatory power and phase in the theta band (3-7 Hz) for the encoding of crossmodal associations. Furthermore, ongoing activity in the theta range as well as alpha (8-12 Hz) and low beta activity (13-20 Hz) before the presentation of a stimulus is thought to modulate subsequent cognitive processing, including processes that are related to memory. In this study, we tested the hypothesis that pre-stimulus characteristics of low frequency activity are relevant for the successful formation of crossmodal memory. The experimental design that was used specifically allowed for the investigation of associative memory independent from individual item memory. Participants (n = 51) were required to memorize associations between audiovisual stimulus pairs and distinguish them from newly arranged ones consisting of the same single stimuli in the subsequent recognition task. Our results show significant differences in the state of pre-stimulus theta and alpha power between remembered and not remembered crossmodal associations, clearly relating increased power to successful recognition. These differences were positively correlated with memory performance, suggesting functional relevance for behavioral measures of associative memory. Further analysis revealed similar effects in the low beta frequency ranges, indicating the involvement of different pre-stimulus-related cognitive processes. Phase-based connectivity measures in the theta band did not differ between remembered and not remembered stimulus pairs. The findings support the assumed functional relevance of theta band oscillations for the formation of associative memory and demonstrate that an increase of theta as well as alpha band oscillations in the pre-stimulus period is beneficial for the establishment of crossmodal memory.

Stimulus temporal uncertainty balances intersensory dominance

Vision is typically dominant over audition or touch in human adults. One classic example is the Colavita visual dominance effect: The presentation of a tone sometimes goes undetected when it is paired with a flash even though it is well detected when presented alone. We investigated whether the Colavita visual dominance effect is modulated by stimulus uncertainty in the temporal and spatial domains. In a simple discrimination task, participants were asked to press a predesignated key when detecting a flash, another key when detecting a tone, and both keys when detecting both a flash and a tone. Temporal uncertainty was increased by introducing temporal jitter between trials (Experiment 1), and spatial uncertainty was increased by shifting the flash to different locations (Experiment 2). The Colavita visual dominance effect was reduced when temporal uncertainty was increased, while it remained similar when spatial uncertainty was increased. We therefore demonstrate a novel consideration, where increasing temporal uncertainty balances the competition between vision and audition, suggesting that people's degree of sensory dominance is malleable. Our result therefore highlights the concept that intersensory competition is susceptible to the temporal predictability of the stimulus, which provides critical insights into the design of effective warning systems in the field of ergonomics.