Colavita dominance effect revisited: the effect of semantic congruity

Long-term memory representations for audio-visual scenes

In this study, we investigated the nature of long-term memory representations for naturalistic audio-visual scenes. Whereas previous research has shown that audio-visual scenes are recognized more accurately than their unimodal counterparts, it remains unclear whether this benefit stems from audio-visually integrated long-term memory representations or a summation of independent retrieval cues. We tested two predictions for audio-visually integrated memory representations. First, we used a modeling approach to test whether recognition performance for audio-visual scenes is more accurate than would be expected from independent retrieval cues. This analysis shows that audio-visual integration is not necessary to explain the benefit of audio-visual scenes relative to purely auditory or purely visual scenes. Second, we report a series of experiments investigating the occurrence of study-test congruency effects for unimodal and audio-visual scenes. Most importantly, visually encoded information was immune to additional auditory information presented during testing, whereas auditory encoded information was susceptible to additional visual information presented during testing. This renders a true integration of visual and auditory information in long-term memory representations unlikely. In sum, our results instead provide evidence for visual dominance in long-term memory. Whereas associative auditory information is capable of enhancing memory performance, the long-term memory representations appear to be primarily visual.

What was that object? On the role of identity information in the formation of object files and conscious object perception

Object files are a psychological representation that allows the human brain to keep track of objects, as they move and change across time. The question regarding what information is used to individuate versus update object files has been the focus of considerable scientific debate. Historically, the role of an object's spatiotemporal history was emphasised, whereas more recent work has demonstrated a key contribution from surface features, such as colour. The purpose of the present study was to investigate the role of identity-level information in the formation and individuation of object files, and how it compares to the contribution of featural information. Using a modified spatial repetition-blindness paradigm, across four experiments, there was convergent evidence that surface features contribute to the formation of object files, whereas the role of identity information was at best much smaller and less reliable than the clear contribution from surface features, and the most parsimonious explanation is that it was not present at all.

The effect of synesthetic associations between the visual and auditory modalities on the Colavita effect

The Colavita effect refers to the phenomenon that when confronted with an audiovisual stimulus, observers report more often to have perceived the visual than the auditory component. The Colavita effect depends on low-level stimulus factors such as spatial and temporal proximity between the unimodal signals. Here, we examined whether the Colavita effect is modulated by synesthetic congruency between visual size and auditory pitch. If the Colavita effect depends on synesthetic congruency, we expect a larger Colavita effect for synesthetically congruent size/pitch (large visual stimulus/low-pitched tone; small visual stimulus/high-pitched tone) than synesthetically incongruent (large visual stimulus/high-pitched tone; small visual stimulus/low-pitched tone) combinations. Participants had to identify stimulus type (visual, auditory or audiovisual). The study replicated the Colavita effect because participants reported more often the visual than auditory component of the audiovisual stimuli. Synesthetic congruency had, however, no effect on the magnitude of the Colavita effect. EEG recordings to congruent and incongruent audiovisual pairings showed a late frontal congruency effect at 400-550 ms and an occipitoparietal effect at 690-800 ms with neural sources in the anterior cingulate and premotor cortex for the 400- to 550-ms window and premotor cortex, inferior parietal lobule and the posterior middle temporal gyrus for the 690- to 800-ms window. The electrophysiological data show that synesthetic congruency was probably detected in a processing stage subsequent to the Colavita effect. We conclude that-in a modality detection task-the Colavita effect can be modulated by low-level structural factors but not by higher-order associations between auditory and visual inputs.

Conceptual priming with pictures and environmental sounds

A series of experiments was conducted to examine conceptual priming within and across modalities with pictures and environmental sounds. In Experiment 1, we developed a new multimodal stimulus set consisting of two picture and sound exemplars that represented 80 object items. In Experiments 2, we investigated whether categorization of the stimulus items would be facilitated by picture and environmental sound primes that were derived from different exemplars of the target items; and in Experiments 3 and 4, we tested the additional influence on priming when trials were consolidated within a target modality and the inter stimulus interval was lengthened. The results demonstrated that target categorization was facilitated by the advanced presentation of conceptually related exemplars, but there were differences in effectiveness when pictures and sounds appeared as primes.