Cortical processes underlying sound-induced flash fusion

Cross-modal recruitment of primary visual cortex by auditory stimuli in the nonhuman primate brain: a molecular mapping study

Recent studies suggest that exposure to only one component of audiovisual events can lead to cross-modal cortical activation. However, it is not certain whether such crossmodal recruitment can occur in the absence of explicit conditioning, semantic factors, or long-term associations. A recent study demonstrated that crossmodal cortical recruitment can occur even after a brief exposure to bimodal stimuli without semantic association. In addition, the authors showed that the primary visual cortex is under such crossmodal influence. In the present study, we used molecular activity mapping of the immediate early gene zif268. We found that animals, which had previously been exposed to a combination of auditory and visual stimuli, showed increased number of active neurons in the primary visual cortex when presented with sounds alone. As previously implied, this crossmodal activation appears to be the result of implicit associations of the two stimuli, likely driven by their spatiotemporal characteristics; it was observed after a relatively short period of exposure (~45 min) and lasted for a relatively long period after the initial exposure (~1 day). These results suggest that the previously reported findings may be directly rooted in the increased activity of the neurons occupying the primary visual cortex.

Neural correlates of multisensory perceptual learning

The brain's ability to bind incoming auditory and visual stimuli depends critically on the temporal structure of this information. Specifically, there exists a temporal window of audiovisual integration within which stimuli are highly likely to be perceived as part of the same environmental event. Several studies have described the temporal bounds of this window, but few have investigated its malleability. Recently, our laboratory has demonstrated that a perceptual training paradigm is capable of eliciting a 40% narrowing in the width of this window that is stable for at least 1 week after cessation of training. In the current study, we sought to reveal the neural substrates of these changes. Eleven human subjects completed an audiovisual simultaneity judgment training paradigm, immediately before and after which they performed the same task during an event-related 3T fMRI session. The posterior superior temporal sulcus (pSTS) and areas of auditory and visual cortex exhibited robust BOLD decreases following training, and resting state and effective connectivity analyses revealed significant increases in coupling among these cortices after training. These results provide the first evidence of the neural correlates underlying changes in multisensory temporal binding likely representing the substrate for a multisensory temporal binding window.

Neuromagnetic responses reveal the cortical timing of audiovisual synchrony

Multisensory processing involving visual and auditory inputs is modulated by their relative temporal offsets. In order to assess whether multisensory integration alters the activation timing of primary visual and auditory cortices as a function of the temporal offsets between auditory and visual stimuli, a task was designed in which subjects had to judge the perceptual simultaneity of the onset of visual stimuli and brief acoustic tones. These were presented repeatedly with three different inter-stimulus intervals that were chosen to meet three perceptual conditions: (1) physical synchrony perceived as synchrony by subjects (SYNC); (2) physical asynchrony perceived as asynchrony (ASYNC); (3) physical asynchrony perceived ambiguously (AMB, i.e. 50% perceived as synchrony, 50% as asynchrony). Magnetoencephalographic activity was recorded during crossmodal sessions and unimodal control sessions. The activation of primary visual and auditory cortices peaked at a longer latency for the crossmodal conditions as compared to the unimodal conditions. Moreover, the latency in the auditory cortex was longer in the SYNC than in the ASYNC condition, whereas in the visual cortex the latency in the AMB condition was longer than in the ASYNC condition. These findings suggest that multisensory processing affects temporal dynamics already in primary cortices, that such activity can differ regionally and can be sensitive to the temporal offsets of multisensory inputs. In addition, in the AMB condition the conscious awareness of asynchrony might be associated to a later activation of the primary auditory cortex.

Grapheme-color synesthetes show enhanced crossmodal processing between auditory and visual modalities

Synesthesia is an involuntary experience in which stimulation of one sensory modality triggers additional, atypical sensory experiences. Strong multisensory processes are present in the general population, but the relationship between these 'normal' sensory interactions and synesthesia is currently unknown. Neuroimaging research suggests that some forms of synesthesia are caused by enhanced cross-activation between brain areas specialized for the processing of different sensory attributes, and finds evidence of increased white matter connections among regions known to be involved in typical crossmodal processes. Using two classic crossmodal integration tasks we show that grapheme-color synesthetes exhibit enhanced crossmodal interactions between auditory and visual modalities, suggesting that the experience of synesthesia in one modality generalizes to enhanced crossmodal processes with other modalities. This finding supports our conjecture that the atypical sensory experiences of synesthetes represent a selective expression of a more diffuse propensity toward 'typical' crossmodality interactions.

Is inefficient multisensory processing associated with falls in older people?

Although falling is a significant problem for older persons, little is understood about its underlying causes. Spatial cognition and balance maintenance rely on the efficient integration of information across the main senses. We investigated general multisensory efficiency in older persons with a history of falls compared to age- and sensory acuity-matched controls and younger adults using a sound-induced flash illusion. Older fallers were as susceptible to the illusion as age-matched, non-fallers or younger adults at a short delay of 70 ms between the auditory and visual stimuli. Both older adult groups were more susceptible to the illusion at longer SOAs than younger adults. However, with increasing delays between the visual and auditory stimuli, older fallers did not show a decline in the frequency at which the illusion was experienced even with delays of up to 270 ms. We argue that this relatively higher susceptibility to the illusion reflects inefficient audio-visual processing in the central nervous system and has important implications for the diagnosis and rehabilitation of falling in older persons.

The audiovisual tau effect in infancy

BACKGROUND

Perceived spatial intervals between successive flashes can be distorted by varying the temporal intervals between them (the "tau effect"). A previous study showed that a tau effect for visual flashes could be induced when they were accompanied by auditory beeps with varied temporal intervals (an audiovisual tau effect).

METHODOLOGY/PRINCIPAL FINDINGS

We conducted two experiments to investigate whether the audiovisual tau effect occurs in infancy. Forty-eight infants aged 5-8 months took part in this study. In Experiment 1, infants were familiarized with audiovisual stimuli consisting of three pairs of two flashes and three beeps. The onsets of the first and third pairs of flashes were respectively matched to those of the first and third beeps. The onset of the second pair of flashes was separated from that of the second beep by 150 ms. Following the familiarization phase, infants were exposed to a test stimulus composed of two vertical arrays of three static flashes with different spatial intervals. We hypothesized that if the audiovisual tau effect occurred in infancy then infants would preferentially look at the flash array with spatial intervals that would be expected to be different from the perceived spatial intervals between flashes they were exposed to in the familiarization phase. The results of Experiment 1 supported this hypothesis. In Experiment 2, the first and third beeps were removed from the familiarization stimuli, resulting in the disappearance of the audiovisual tau effect. This indicates that the modulation of temporal intervals among flashes by beeps was essential for the audiovisual tau effect to occur (Experiment 2).

CONCLUSIONS/SIGNIFICANCE

These results suggest that the cross-modal processing that underlies the audiovisual tau effect occurs even in early infancy. In particular, the results indicate that audiovisual modulation of temporal intervals emerges by 5-8 months of age.

Crossmodal recruitment of primary visual cortex following brief exposure to bimodal audiovisual stimuli

Several lines of evidence suggest that exposure to only one component of typically audiovisual events can lead to crossmodal cortical activation. These effects are likely explained by long-term associations formed between the auditory and visual components of such events. It is not certain whether such crossmodal recruitment can occur in the absence of explicit conditioning, semantic factors, or long-term association; nor is it clear whether primary sensory cortices can be recruited in such paradigms. In the present study we tested the hypothesis that crossmodal cortical recruitment would occur even after a brief exposure to bimodal stimuli without semantic association. We used positron emission tomography, and an apparatus allowing presentation of spatially and temporally congruous audiovisual stimuli (noise bursts and light flashes). When presented with only the auditory or visual components of the bimodal stimuli, naïve subjects showed only modality-specific cortical activation, as expected. However, subjects who had previously been exposed to the audiovisual stimuli showed increased cerebral blood flow in the primary visual cortex when presented with sounds alone. Functional connectivity analysis suggested that the auditory cortex was the source of visual cortex activity. This crossmodal activation appears to be the result of implicit associations of the two stimuli, likely driven by their spatiotemporal characteristics; it was observed after a relatively short period of exposure (approximately 45 min), and lasted for a relatively long period after the initial exposure (approximately 1 day). The findings indicate that auditory and visual cortices interact with one another to a larger degree than typically assumed.

The impact of spatial incongruence on an auditory-visual illusion

Background

The sound-induced flash illusion is an auditory-visual illusion – when a single flash is presented along with two or more beeps, observers report seeing two or more flashes. Previous research has shown that the illusion gradually disappears as the temporal delay between auditory and visual stimuli increases, suggesting that the illusion is consistent with existing temporal rules of neural activation in the superior colliculus to multisensory stimuli. However little is known about the effect of spatial incongruence, and whether the illusion follows the corresponding spatial rule. If the illusion occurs less strongly when auditory and visual stimuli are separated, then integrative processes supporting the illusion must be strongly dependant on spatial congruence. In this case, the illusion would be consistent with both the spatial and temporal rules describing response properties of multisensory neurons in the superior colliculus.

Methodology/Principal Findings

The main aim of this study was to investigate the importance of spatial congruence in the flash-beep illusion. Selected combinations of one to four short flashes and zero to four short 3.5 KHz tones were presented. Observers were asked to count the number of flashes they saw. After replication of the basic illusion using centrally-presented stimuli, the auditory and visual components of the illusion stimuli were presented either both 10 degrees to the left or right of fixation (spatially congruent) or on opposite (spatially incongruent) sides, for a total separation of 20 degrees.

Conclusions/Significance

The sound-induced flash fission illusion was successfully replicated. However, when the sources of the auditory and visual stimuli were spatially separated, perception of the illusion was unaffected, suggesting that the “spatial rule” does not extend to describing behavioural responses in this illusion. We also find no evidence for an associated “fusion” illusion reportedly occurring when multiple flashes are accompanied by a single beep.

Visual stimulus locking of EEG is modulated by temporal congruency of auditory stimuli

Disparate sensory streams originating from a common underlying event share similar dynamics, and this plays an important part in multisensory integration. Here we investigate audiovisual binding by presenting continuously changing, temporally congruent and incongruent stimuli. Recorded EEG signals are used to quantify spectrotemporal and waveform locking of neural activity to stimulus dynamics. Spectrotemporal analysis reveals locking to visual stimulus dynamics in both a broad alpha and the beta band. The properties of these effects suggest they are a correlate of bottom-up processing in the visual system. Waveform locking reveals two cortically distinct processes that lock to visual stimulus dynamics with differing topographies and time lags relative to the stimuli. Most importantly, these are modulated in strength by the congruency of an accompanying auditory stream. In addition, the waveform locking found at occipital electrodes shows an increase over stimulus duration for visual and congruent audiovisual stimuli. Hence we argue that these effects reflect audiovisual interaction. We thus propose that spectrotemporal and waveform locking reflect different mechanisms involved in the processing of dynamic audiovisual stimuli.

Audiovisual temporal capture underlies flash fusion

When sequential visual flashes are accompanied by a lower number of sequential auditory pulses, the perceived number of visual flashes is lower than the actual number, an illusion termed 'flash fusion'. We examined whether temporal capture of flashes by pulses underlay flash fusion. One of the visual flashes was given a luminance increment, and observers reported which flash had the luminance increment. Results showed that the pulse strongly captured the flashes in its temporal vicinity, resulting in flash fusion. Moreover, when one of the successive pulses was given a higher frequency than others, the luminance increment was perceptually paired with the pulse with the higher frequency. The pairing of audiovisual features disappeared when the temporal pattern of the pulse frequency was difficult for the observer to anticipate. These data indicate that flash fusion is caused by temporal capture of flashes by the pulse, and that feature matching between auditory and visual signals also contributes to the modulation of perceived temporal structure of flashes during flash fusion.