Electrical activity in visual cortex associated with combined auditory and visual stimulation in temporal sequences known to be associated with a visual illusion

A Fast Detection Method of Break Points in Effective Connectivity Networks

There is increasing interest in identifying changes in the underlying states of brain networks. The availability of large scale neuroimaging data creates a strong need to develop fast, scalable methods for detecting and localizing in time such changes and also identify their drivers, thus enabling neuroscientists to hypothesize about potential mechanisms. This paper presents a fast method for detecting break points in exceedingly long time series neurogimaging data, based on vector autoregressive (Granger causal) models. It uses a multi-step strategy based on a regularized objective function that leads to fast identification of candidate break points, followed by clustering steps to select the final set of break points and subsequent estimation with false positives control of the underlying Granger causal networks. The latter provide insights into key changes in network connectivity that led to the presence of break points. The proposed methodology is illustrated on synthetic data varying in their length, dimensionality, number of break points, strength of signal and also applied to EEG data related to visual tasks.

The Influence of Auditory Information on Visual Size Adaptation

Size perception can be influenced by several visual cues, such as spatial (e.g., depth or vergence) and temporal contextual cues (e.g., adaptation to steady visual stimulation). Nevertheless, perception is generally multisensory and other sensory modalities, such as auditory, can contribute to the functional estimation of the size of objects. In this study, we investigate whether auditory stimuli at different sound pitches can influence visual size perception after visual adaptation. To this aim, we used an adaptation paradigm (Pooresmaeili et al., 2013) in three experimental conditions: visual-only, visual-sound at 100 Hz and visual-sound at 9,000 Hz. We asked participants to judge the size of a test stimulus in a size discrimination task. First, we obtained a baseline for all conditions. In the visual-sound conditions, the auditory stimulus was concurrent to the test stimulus. Secondly, we repeated the task by presenting an adapter (twice as big as the reference stimulus) before the test stimulus. We replicated the size aftereffect in the visual-only condition: the test stimulus was perceived smaller than its physical size. The new finding is that we found the auditory stimuli have an effect on the perceived size of the test stimulus after visual adaptation: low frequency sound decreased the effect of visual adaptation, making the stimulus perceived bigger compared to the visual-only condition, and contrarily, the high frequency sound had the opposite effect, making the test size perceived even smaller.

Multisensory integration in hemianopia and unilateral spatial neglect: Evidence from the sound induced flash illusion

Recent neuropsychological evidence suggests that acquired brain lesions can, in some instances, abolish the ability to integrate inputs from different sensory modalities, disrupting multisensory perception. We explored the ability to perceive multisensory events, in particular the integrity of audio-visual processing in the temporal domain, in brain-damaged patients with visual field defects (VFD), or with unilateral spatial neglect (USN), by assessing their sensitivity to the 'Sound-Induced Flash Illusion' (SIFI). The study yielded two key findings. Firstly, the 'fission' illusion (namely, seeing multiple flashes when a single flash is paired with multiple sounds) is reduced in both left- and right-brain-damaged patients with VFD, but not in right-brain-damaged patients with left USN. The disruption of the fission illusion is proportional to the extent of the occipital damage. Secondly, a reliable 'fusion' illusion (namely, seeing less flashes when a single sound is paired with multiple flashes) is evoked in USN patients, but neither in VFD patients nor in healthy participants. A control experiment showed that the fusion, but not the fission, illusion is lost in older participants (>50 year-old), as compared with younger healthy participants (<30 year-old). This evidence indicates that the fission and fusion illusions are dissociable multisensory phenomena, altered differently by impairments of visual perception (i.e. VFD) and spatial attention (i.e. USN). The occipital cortex represents a key cortical site for binding auditory and visual stimuli in the SIFI, while damage to right-hemisphere areas mediating spatial attention and awareness does not prevent the integration of audio-visual inputs in the temporal domain.

Audition influences color processing in the sound-induced visual flash illusion

Multisensory interactions can lead to illusory percepts, as exemplified by the sound-induced extra flash illusion (SIFI: Shams, Kamitani, & Shimojo, 2000, 2002). In this illusion, an audio-visual stimulus sequence consisting of two pulsed sounds and a light flash presented within a 100 ms time window generates the visual percept of two flashes. Here, we used colored visual stimuli to investigate whether concurrent auditory stimuli can affect the perceived features of the illusory flash. Zero, one or two pulsed sounds were presented concurrently with either a red or green flash or with two flashes of different colors (red followed by green) in rapid sequence. By querying both the number and color of the participants' visual percepts, we found that the double flash illusion is stimulus specific: i.e., two sounds paired with one red or one green flash generated the percept of two red or two green flashes, respectively. This implies that the illusory second flash is induced at a level of visual processing after perceived color has been encoded. In addition, we found that the presence of two sounds influenced the integration of color information from two successive flashes. In the absence of any sounds, a red and a green flash presented in rapid succession fused to form a single orange percept, but when accompanied by two sounds, this integrated orange percept was perceived to flash twice on a significant proportion of trials. In addition, the number of concurrent auditory stimuli modified the degree to which the successive flashes were integrated to an orange percept vs. maintained as separate red-green percepts. Overall, these findings show that concurrent auditory input can affect both the temporal and featural properties of visual percepts.

Gone in a flash: manipulation of audiovisual temporal integration using transcranial magnetic stimulation

While converging evidence implicates the right inferior parietal lobule in audiovisual integration, its role has not been fully elucidated by direct manipulation of cortical activity. Replicating and extending an experiment initially reported by Kamke et al. (2012), we employed the sound-induced flash illusion, in which a single visual flash, when accompanied by two auditory tones, is misperceived as multiple flashes (Wilson, 1987; Shams et al., 2000). Slow repetitive (1 Hz) TMS administered to the right angular gyrus, but not the right supramarginal gyrus, induced a transient decrease in the Peak Perceived Flashes (PPF), reflecting reduced susceptibility to the illusion. This finding independently confirms that perturbation of networks involved in multisensory integration can result in a more veridical representation of asynchronous auditory and visual events and that cross-modal integration is an active process in which the objective is the identification of a meaningful constellation of inputs, at times at the expense of accuracy.

Effect of Attention on Early Cortical Processes Associated with the Sound-induced Extra Flash Illusion

When a single flash of light is presented interposed between two brief auditory stimuli separated by 60–100 msec, subjects typically report perceiving two flashes [Shams, L., Kamitani, Y., & Shimojo, S. Visual illusion induced by sound. Brain Research, Cognitive Brain Research, 14, 147–152, 2002; Shams, L., Kamitani, Y., & Shimojo, S. Illusions. What you see is what you hear. Nature, 408, 788, 2000]. Using ERP recordings, we previously found that perception of the illusory extra flash was accompanied by a rapid dynamic interplay between auditory and visual cortical areas that was triggered by the second sound [Mishra, J., Martínez, A., Sejnowski, T. J., & Hillyard, S. A. Early cross-modal interactions in auditory and visual cortex underlie a sound-induced visual illusion. Journal of Neuroscience, 27, 4120–4131, 2007]. In the current study, we investigated the effect of attention on the ERP components associated with the illusory extra flash in 15 individuals who perceived this cross-modal illusion frequently. All early ERP components in the cross-modal difference wave associated with the extra flash illusion were significantly enhanced by selective spatial attention. The earliest attention-related modulation was an amplitude increase of the positive-going PD110/PD120 component, which was previously shown to be correlated with an individual's propensity to perceive the illusory second flash [Mishra, J., Martínez, A., Sejnowski, T. J., & Hillyard, S. A. Early cross-modal interactions in auditory and visual cortex underlie a sound-induced visual illusion. Journal of Neuroscience, 27, 4120–4131, 2007]. The polarity of the early PD110/PD120 component did not differ as a function of the visual field (upper vs. lower) of stimulus presentation. This, along with the source localization of the component, suggested that its principal generator lies in extrastriate visual cortex. These results indicate that neural processes previously shown to be associated with the extra flash illusion can be modulated by attention, and thus are not the result of a wholly automatic cross-modal integration process.

Sound-induced flash illusion is resistant to feedback training

A single flash accompanied by two auditory beeps tends to be perceived as two flashes (Shams et al. Nature 408:788, 2000, Cogn Brain Res 14:147-152, 2002). This phenomenon is known as 'sound-induced flash illusion.' Previous neuroimaging studies have shown that this illusion is correlated with modulation of activity in early visual cortical areas (Arden et al. Vision Res 43(23):2469-2478, 2003; Bhattacharya et al. NeuroReport 13:1727-1730, 2002; Shams et al. NeuroReport 12(17):3849-3852, 2001, Neurosci Lett 378(2):76-81, 2005; Watkins et al. Neuroimage 31:1247-1256, 2006, Neuroimage 37:572-578, 2007; Mishra et al. J Neurosci 27(15):4120-4131, 2007). We examined how robust the illusion is by testing whether the frequency of the illusion can be reduced by providing feedback. We found that the sound-induced flash illusion was resistant to feedback training, except when the amount of monetary reward was made dependent on accuracy in performance. However, even in the latter case the participants reported that they still perceived illusory two flashes even though they correctly reported single flash. Moreover, the feedback training effect seemed to disappear once the participants were no longer provided with feedback suggesting a short-lived refinement of discrimination between illusory and physical double flashes rather than vanishing of the illusory percept. These findings indicate that the effect of sound on the perceptual representation of visual stimuli is strong and robust to feedback training, and provide further evidence against decision factors accounting for the sound-induced flash illusion.

Cortical processes underlying sound-induced flash fusion

When two brief flashes presented in rapid succession (<100 ms apart) are paired with a single auditory stimulus, subjects often report perceiving only a single flash [Andersen, T.S., Tiippana, K., Sams, M., 2004. Factors influencing audiovisual fission and fusion illusions. Brain Res. Cogn. Brain Res. 21, 301-308; Shams, L., Iwaki, S., Chawla, A., Bhattacharya, J., 2005a. Early modulation of visual cortex by sound: an MEG study. Neurosci. Lett. 378, 76-81, Shams, L., Ma, W.J., Beierholm, U., 2005b. Sound-induced flash illusion as an optimal percept. Neuroreport 16, 1923-1927]. We used event-related potentials (ERPs) to investigate the timing and localization of the cortical processes that underlie this sound induced flash fusion, which is complementary to the sound-induced extra flash illusion that we analyzed previously [Mishra, J., Martinez, A., Sejnowski, T.J. and Hillyard, S.A., Early cross-modal interactions in auditory and visual cortex underlie a sound-induced visual illusion. J. Neurosci. 27 (2007) 4120-4131]. The difference ERP that represented the cross-modal interaction between the visual (two flashes) and auditory (one sound) constituents of the bimodal stimulus revealed a positive component elicited 160-190 ms after stimulus onset, which was markedly attenuated in subjects who did not perceive the second flash. This component, previously designated as PD180 [Mishra, J., Martinez, A., Sejnowski, T.J. and Hillyard, S.A., Early cross-modal interactions in auditory and visual cortex underlie a sound-induced visual illusion. J. Neurosci. 27 (2007) 4120-4131], was localized by dipole modeling to polysensory superior temporal cortex. PD180 was found to covary in amplitude across subjects with the visual evoked N1 component (148-184 ms), suggesting that inter-individual differences in perceiving the illusion are based at least in part on differences in visual processing. A trial-by-trial analysis found that the PD180 as well as a subsequent modulation in visual cortex at 228-248 ms was diminished on trials when the two flashes were perceived as one relative to trials when two flashes were correctly reported. These results suggest that the sound induced flash fusion is based on an interaction between polysensory and visual cortical areas.

Multisensory interplay reveals crossmodal influences on 'sensory-specific' brain regions, neural responses, and judgments

Although much traditional sensory research has studied each sensory modality in isolation, there has been a recent explosion of interest in causal interplay between different senses. Various techniques have now identified numerous multisensory convergence zones in the brain. Some convergence may arise surprisingly close to low-level sensory-specific cortex, and some direct connections may exist even between primary sensory cortices. A variety of multisensory phenomena have now been reported in which sensory-specific brain responses and perceptual judgments concerning one sense can be affected by relations with other senses. We survey recent progress in this multisensory field, foregrounding human studies against the background of invasive animal work and highlighting possible underlying mechanisms. These include rapid feedforward integration, possible thalamic influences, and/or feedback from multisensory regions to sensory-specific brain areas. Multisensory interplay is more prevalent than classic modular approaches assumed, and new methods are now available to determine the underlying circuits.

Early cross-modal interactions in auditory and visual cortex underlie a sound-induced visual illusion

When a single flash of light is presented interposed between two brief auditory stimuli separated by 60-100 ms, subjects typically report perceiving two flashes (Shams et al., 2000, 2002). We investigated the timing and localization of the cortical processes that underlie this illusory flash effect in 34 subjects by means of 64-channel recordings of event-related potentials (ERPs). A difference ERP calculated to isolate neural activity associated with the illusory second flash revealed an early modulation of visual cortex activity at 30-60 ms after the second sound, which was larger in amplitude in subjects who saw the illusory flash more frequently. These subjects also showed this early modulation in response to other combinations of auditory and visual stimuli, thus pointing to consistent individual differences in the neural connectivity that underlies cross-modal integration. The overall pattern of cortical activity associated with the cross-modally induced illusory flash, however, differed markedly from that evoked by a real second flash. A trial-by-trial analysis showed that short-latency ERP activity localized to auditory cortex and polymodal cortex of the temporal lobe, concurrent with gamma bursts in visual cortex, were associated with perception of the double-flash illusion. These results provide evidence that perception of the illusory second flash is based on a very rapid dynamic interplay between auditory and visual cortical areas that is triggered by the second sound.

Brief report: can you see what is not there? low-level auditory-visual integration in autism spectrum disorder

Patients diagnosed with Autism Spectrum Disorder, show impaired integration of information across different senses. The processing-level from which this impairment originates, however, remains unclear. We investigated low-level integration of auditory and visual stimuli in subjects with Autism Spectrum Disorder. High-functioning adult subjects with Autism Spectrum Disorder as well as age- and IQ-matched adults were tested using a task that evokes illusory visual stimuli, by presenting sounds concurrently with visual flashes. In both groups the number of sounds presented significantly affected the number of flashes perceived, yet there was no difference between groups. This finding implicates that any problems arising from integrating auditory and visual information must stem from higher processing stages in high-functioning adults with Autism Spectrum Disorder.

Auditory-visual multisensory interactions attenuate subsequent visual responses in humans

Effects of multisensory interactions on how subsequent sensory inputs are processed remain poorly understood. We investigated whether multisensory interactions between rudimentary visual and auditory stimuli (flashes and beeps) affect later visual processing. A 2 x 3 design varied the number of flashes (1 or 2) with the number of beeps (0, 1, or 2) presented on each trial, such that '2F1B' refers to the presentation of 2 flashes with 1 beep. Beeps, when present, were synchronous with the first flash, and pairs of stimuli within a trial were separated by 52 ms ISI. Subjects indicated the number of flashes presented. Electrical neuroimaging of 128-channel event-related potentials assessed both the electric field strength and topography. Isolation of responses a visual stimulus that was preceded by a multisensory event was achieved by calculating the difference between the 2F1B and 1F1B conditions, and responses to a visual stimulus preceded by a unisensory event were isolated by calculating the difference between the 2F0B and 1F0B conditions (MUL and VIS, respectively). Comparison of MUL and VIS revealed that the treatment of visual information was significantly attenuated approximately 160 ms after the onset of the second flash when it was preceded by a multisensory event. Source estimations further indicated that this attenuation occurred within low-level visual cortices. Multisensory interactions are ongoing in low-level visual cortices and affect incoming sensory processing. These data provide evidence that multisensory interactions are not restricted in time and can dramatically influence the treatment of subsequent stimuli, opening new lines of multisensory research.

Sound alters activity in human V1 in association with illusory visual perception

When a single brief visual flash is accompanied by two auditory bleeps, it is frequently perceived incorrectly as two flashes. Here, we used high field functional MRI in humans to examine the neural basis of this multisensory perceptual illusion. We show that activity in retinotopic visual cortex is increased by the presence of concurrent auditory stimulation, irrespective of any illusory perception. However, when concurrent auditory stimulation gave rise to illusory visual perception, activity in V1 was enhanced, despite auditory and visual stimulation being unchanged. These findings confirm that responses in human V1 can be altered by sound and show that they reflect subjective perception rather than the physically present visual stimulus. Moreover, as the right superior temporal sulcus and superior colliculus were also activated by illusory visual perception, together with V1, they provide a potential neural substrate for the generation of this multisensory illusion.

Crossmodal audio-visual interactions in the primary visual cortex of the visually deprived cat: a physiological and anatomical study

Blind individuals often demonstrate enhanced non-visual perceptual abilities. Neuroimaging and transcranial magnetic stimulation experiments have suggested that computations carried out in the occipital cortex may underlie these enhanced somatosensory or auditory performances. Thus, cortical areas that are dedicated to the analysis of the visual scene may, in the blind, acquire the capacity to participate in other sensory processing. However, the neural substrate that underlies this transfer of function is not fully characterized. Here we studied the synaptic and anatomical basis of this phenomenon in cats that were visually deprived by dark rearing, either early visually deprived after birth (EVD), or late visually deprived after the end of the critical period (LVD); data were compared with those obtained in normally reared cats (controls). The presence of synaptic and spike responses to auditory stimulation was examined by means of intracellular recordings in area 17 and the border between areas 17 and 18. While none of the cells recorded in control and LVD cats showed responses to sound, 14% of the cells recorded in EVD cats showed both subthreshold synaptic responses and suprathreshold spike responses to auditory stimuli. Synaptic responses were of small amplitude, but well time-locked to the stimuli and had an average latency of 30+/-12ms. In an attempt to identify the origin of the inputs carrying auditory information to the visual cortex, wheat germ agglutinin-horseradish peroxidase (WGA-HRP) was injected in the visual cortex and retrograde labeling examined in the cortex and thalamus. No significant retrograde labeling was found in auditory cortical areas. However, the proportion of neurons projecting from supragranular layers of the posteromedial and posterolateral parts of the lateral suprasylvian region to V1 was higher than that in control cats. Retrograde labeling in the lateral geniculate nucleus showed no difference in the total number of neurons between control and visually deprived cats, but there was a higher proportion of labeling in C-laminae in deprived cats. Labeled cells were not found in the medial geniculate nucleus, a thalamic relay for auditory information, in either control or visually deprived cats. Finally, immunohistochemistry of the visual cortex of deprived cats revealed a striking decrease in pavalbumin- and calretinin-positive neurons, the functional implications of which we discuss.

Early modulation of visual cortex by sound: an MEG study

Sound can alter visual perception. This has been recently demonstrated by a strong illusion in which a single flash is perceived as multiple flashes when accompanied by multiple brief sounds. While psychophysical findings on this sound-induced flash illusion indicate that the modulations of visual percept by sound occur at a perceptual processing level, it remains unclear at what level of perceptual processing these interactions occur and what mechanisms mediate them. Here we investigated these questions using MEG. We found modulation of activity in occipital and parietal scalp locations, when comparing illusion trials with visual-alone and auditory-alone trials. This modulation occurred as early as 35-65 ms from the onset of the visual stimulus. Activity was also modulated in the occipital and parietal areas as well as anterior areas at a later ( approximately 150 ms post-stimulus) onset. No significant interactions were observed in occipital and parietal areas in trials in which illusion was not perceived. These results indicate that the auditory alteration of visual perception as reflected by the illusion is associated with modulation of activity in visual cortex. The early onset of these modulations suggests that a feed-forward or lateral circuitry is at least partially involved in these interactions.