Sound-induced illusory flash perception: role of gamma band responses

Crossmodal plasticity following short-term monocular deprivation

A brief period of monocular deprivation (MD) induces short-term plasticity of the adult visual system. Whether MD elicits neural changes beyond visual processing is yet unclear. Here, we assessed the specific impact of MD on neural correlates of multisensory processes. Neural oscillations associated with visual and audio-visual processing were measured for both the deprived and the non-deprived eye. Results revealed that MD changed neural activities associated with visual and multisensory processes in an eye-specific manner. Selectively for the deprived eye, alpha synchronization was reduced within the first 150 ms of visual processing. Conversely, gamma activity was enhanced in response to audio-visual events only for the non-deprived eye within 100-300 ms after stimulus onset. The analysis of gamma responses to unisensory auditory events revealed that MD elicited a crossmodal upweight for the non-deprived eye. Distributed source modeling suggested that the right parietal cortex played a major role in neural effects induced by MD. Finally, visual and audio-visual processing alterations emerged for the induced component of the neural oscillations, indicating a prominent role of feedback connectivity. Results reveal the causal impact of MD on both unisensory (visual and auditory) and multisensory (audio-visual) processes and, their frequency-specific profiles. These findings support a model in which MD increases excitability to visual events for the deprived eye and audio-visual and auditory input for the non-deprived eye.

Neurofeedback Modulation of the Sound-induced Flash Illusion Using Parietal Cortex Alpha Oscillations Reveals Dependency on Prior Multisensory Congruency

Spontaneous neural oscillations are key predictors of perceptual decisions to bind multisensory signals into a unified percept. Research links decreased alpha power in the posterior cortices to attention and audiovisual binding in the sound-induced flash illusion (SIFI) paradigm. This suggests that controlling alpha oscillations would be a way of controlling audiovisual binding. In the present feasibility study we used MEG-neurofeedback to train one group of subjects to increase left/right and another to increase right/left alpha power ratios in the parietal cortex. We tested for changes in audiovisual binding in a SIFI paradigm where flashes appeared in both hemifields. Results showed that the neurofeedback induced a significant asymmetry in alpha power for the left/right group, not seen for the right/left group. Corresponding asymmetry changes in audiovisual binding in illusion trials (with 2, 3, and 4 beeps paired with 1 flash) were not apparent. Exploratory analyses showed that neurofeedback training effects were present for illusion trials with the lowest numeric disparity (i.e., 2 beeps and 1 flash trials) only if the previous trial had high congruency (2 beeps and 2 flashes). Our data suggest that the relation between parietal alpha power (an index of attention) and its effect on audiovisual binding is dependent on the learned causal structure in the previous stimulus. The present results suggests that low alpha power biases observers towards audiovisual binding when they have learned that audiovisual signals originate from a common origin, consistent with a Bayesian causal inference account of multisensory perception.

Impoverished Inhibitory Control Exacerbates Multisensory Impairments in Older Fallers

Impaired temporal perception of multisensory cues is a common phenomenon observed in older adults that can lead to unreliable percepts of the external world. For instance, the sound induced flash illusion (SIFI) can induce an illusory percept of a second flash by presenting a beep close in time to an initial flash-beep pair. Older adults that have enhanced susceptibility to a fall demonstrate significantly stronger illusion percepts during the SIFI task compared to those older adults without any history of falling. We hypothesize that a global inhibitory deficit may be driving the impairments across both postural stability and multisensory function in older adults with a fall history (FH). We investigated oscillatory activity and perceptual performance during the SIFI task, to understand how active sensory processing, measured by gamma (30–80 Hz) power, was regulated by alpha activity (8–13 Hz), oscillations that reflect inhibitory control. Compared to young adults (YA), the FH and non-faller (NF) groups demonstrated enhanced susceptibility to the SIFI. Further, the FH group had significantly greater illusion strength compared to the NF group. The FH group also showed significantly impaired performance relative to YA during congruent trials (2 flash-beep pairs resulting in veridical perception of 2 flashes). In illusion compared to non-illusion trials, the NF group demonstrated reduced alpha power (or diminished inhibitory control). Relative to YA and NF, the FH group showed reduced phase-amplitude coupling between alpha and gamma activity in non-illusion trials. This loss of inhibitory capacity over sensory processing in FH compared to NF suggests a more severe change than that consequent of natural aging.

Causally linking neural dominance to perceptual dominance in a multisensory conflict

When different senses are in conflict, one sense may dominate the perception of other sense, but it is not known whether the sensory cortex associated with the dominant modality exerts directional influence, at the functional brain level, over the sensory cortex associated with the dominated modality; in short, the link between sensory dominance and neuronal dominance is not established. In a task involving audio-visual conflict, using magnetoencephalography recordings in humans, we first demonstrated that the neuronal dominance - auditory cortex functionally influencing visual cortex - was associated with the sensory dominance - sound qualitatively altering visual perception. Further, we found that prestimulus auditory-to-visual connectivity could predict the perceptual outcome on a trial-by-trial basis. Subsequently, we performed an effective connectivity-guided neurofeedback electroencephalography experiment and showed that participants who were briefly trained to increase the neuronal dominance from auditory to visual cortex showed higher sensory, that is auditory, dominance during the conflict task immediately after the training. These results shed new light into the interactive neuronal nature of multisensory integration and open up exciting opportunities by enhancing or suppressing targeted mental functions subserved by effective connectivity.

Multi-spectral oscillatory dynamics serving directed and divided attention

Attention-related amplification of neural representations of external stimuli has been well documented in the visual domain, however, research concerning the oscillatory dynamics of such directed attention is relatively sparse in humans. Specifically, it is unknown which spectrally-specific neural responses are mainly impacted by the direction and division of attention, as well as whether the effects of attention on these oscillations are spatially disparate. In this study, we use magnetoencephalography and a visual-somatosensory oddball task to investigate the whole-brain oscillatory dynamics of directed (Experiment 1; N = 26) and divided (Experiment 2; N = 34) visual attention. Sensor-level data were transformed into the time-frequency domain and significant responses from baseline were imaged using a frequency-resolved beamformer. We found that multi-spectral cortical oscillations were stronger when attention was sustained in the visual space and that these effects exhibited informative spatial distributions that differed by frequency. More specifically, we found stronger frontal theta (4–8 Hz), frontal and occipital alpha (8–14 Hz), occipital beta (16–22 Hz), and frontal gamma (74–84 Hz) responses when visual attention was sustained than when it was directed away from the visual domain. Similarly, in the divided attention condition, we observed stronger fronto-parietal theta activity and temporo-parietal alpha and beta oscillations when visual attention was sustained toward the visual stimuli than divided between the visual and somatosensory domains. Investigating how attentional gain is implemented in the human brain is essential for better understanding how this process is degraded in disease, and may provide useful targets for future therapies.

Double Flash Illusions: Current Findings and Future Directions

Twenty years ago, the first report on the sound-induced double flash illusion, a visual illusion induced by sound, was published. In this paradigm, participants are presented with different numbers of auditory and visual stimuli. In case of an incongruent number of auditory and visual stimuli, the influence of auditory information on visual perception can lead to the perception of the illusion. Thus, combining two auditory stimuli with one visual stimulus can induce the perception of two visual stimuli, the so-called fission illusion. Alternatively, combining one auditory stimulus with two visual stimuli can induce the perception of one visual stimulus, the so-called fusion illusion. Overall, current research shows that the illusion is a reliable indicator of multisensory integration. It has also been replicated using different stimulus combinations, such as visual and tactile stimuli. Importantly, the robustness of the illusion allows the widespread use for assessing multisensory integration across different groups of healthy participants and clinical populations and in various task setting. This review will give an overview of the experimental evidence supporting the illusion, the current state of research concerning the influence of cognitive processes on the illusion, the neural mechanisms underlying the illusion, and future research directions. Moreover, an exemplary experimental setup will be described with different options to examine perception, alongside code to test and replicate the illusion online or in the laboratory.

Detecting cognitive hacking in visual inspection with physiological measurements

Cyber threats are targeting vulnerabilities of human workers performing tasks in manufacturing processes, including visual inspection to bias their decision-making, thereby sabotaging product quality. This article examines the use of priming as a form of "cognitive hacking" to adversely affect quality inspection decisions in manufacturing, and investigates physiological measurements as means to detect such intrusion. In a within-subject design experiment, twenty participants inspected surface roughness of a manufactured component with and without exposure to priming on the display of an inspection logging system. The results show that the presence of primes impacted accuracy on surface roughness, cortical activities at parietal lobe P4, and eye gaze for inspecting components. The experiment provides supporting evidence that basic hacking of a worker display can be an effective method to alter decision making in inspection. The findings also illustrate that cortical activities and eye gaze can be useful indicators of cognitive hacking. A major implication of the study results is that physiological indicators can be effective at revealing unconscious cognitive influence in visual inspection.

Emergence of β and γ networks following multisensory training

Our perceptual reality relies on inferences about the causal structure of the world given by multiple sensory inputs. In ecological settings, multisensory events that cohere in time and space benefit inferential processes: hearing and seeing a speaker enhances speech comprehension, and the acoustic changes of flapping wings naturally pace the motion of a flock of birds. Here, we asked how a few minutes of (multi)sensory training could shape cortical interactions in a subsequent unisensory perceptual task. For this, we investigated oscillatory activity and functional connectivity as a function of individuals' sensory history during training. Human participants performed a visual motion coherence discrimination task while being recorded with magnetoencephalography. Three groups of participants performed the same task with visual stimuli only, while listening to acoustic textures temporally comodulated with the strength of visual motion coherence, or with auditory noise uncorrelated with visual motion. The functional connectivity patterns before and after training were contrasted to resting-state networks to assess the variability of common task-relevant networks, and the emergence of new functional interactions as a function of sensory history. One major finding is the emergence of a large-scale synchronization in the high γ (gamma: 60-120Hz) and β (beta: 15-30Hz) bands for individuals who underwent comodulated multisensory training. The post-training network involved prefrontal, parietal, and visual cortices. Our results suggest that the integration of evidence and decision-making strategies become more efficient following congruent multisensory training through plasticity in network routing and oscillatory regimes.

Single trial prestimulus oscillations predict perception of the sound-induced flash illusion

In the sound-induced flash illusion, auditory input affects the perception of visual stimuli with a large inter- and intraindividual variability. Crossmodal influence in this illusion has been shown to be associated with activity in visual and temporal areas. In this electroencephalography study, we investigated the relationship between oscillatory brain activity prior to stimulus presentation and subsequent perception of the illusion on the level of single trials. Using logistic regression, we modeled the perceptual outcome dependent on oscillatory power. We found that 25 Hz to 41 Hz activity over occipital electrodes from 0.17 s to 0.05 s prior to stimulus onset predicted the perception of the illusion. A t-test of power values, averaged over the significant cluster, between illusion and no-illusion trials showed higher power in illusion trials, corroborating the modeling result. We conclude that the observed power modulation predisposes the integration of audiovisual signals, providing further evidence for the governing role of prestimulus brain oscillations in multisensory perception.

Neural Oscillations Orchestrate Multisensory Processing

At any given moment, we receive input through our different sensory systems, and this information needs to be processed and integrated. Multisensory processing requires the coordinated activity of distinct cortical areas. Key mechanisms implicated in these processes include local neural oscillations and functional connectivity between distant cortical areas. Evidence is now emerging that neural oscillations in distinct frequency bands reflect different mechanisms of multisensory processing. Moreover, studies suggest that aberrant neural oscillations contribute to multisensory processing deficits in clinical populations, such as schizophrenia. In this article, we review recent literature on the neural mechanisms underlying multisensory processing, focusing on neural oscillations. We derive a framework that summarizes findings on (1) stimulus-driven multisensory processing, (2) the influence of top-down information on multisensory processing, and (3) the role of predictions for the formation of multisensory perception. We propose that different frequency band oscillations subserve complementary mechanisms of multisensory processing. These processes can act in parallel and are essential for multisensory processing.

U-shaped Relation between Prestimulus Alpha-band and Poststimulus Gamma-band Power in Temporal Tactile Perception in the Human Somatosensory Cortex

Neuronal oscillations are a ubiquitous phenomenon in the human nervous system. Alpha-band oscillations (8–12 Hz) have been shown to correlate negatively with attention and performance, whereas gamma-band oscillations (40–150 Hz) correlate positively. Here, we studied the relation between prestimulus alpha-band power and poststimulus gamma-band power in a suprathreshold tactile discrimination task. Participants received two electrical stimuli to their left index finger with different SOAs (0 msec, 100 msec, intermediate SOA, intermediate SOA ± 10 msec). The intermediate SOA was individually determined so that stimulation was bistable, and participants perceived one stimulus in half of the trials and two stimuli in the other half. We measured neuronal activity with magnetoencephalography (MEG). In trials with intermediate SOAs, behavioral performance correlated inversely with prestimulus alpha-band power but did not correlate with poststimulus gamma-band power. Poststimulus gamma-band power was high in trials with low and high prestimulus alpha-band power and low for intermediate prestimulus alpha-band power (i.e., U-shaped). We suggest that prestimulus alpha activity modulates poststimulus gamma activity and subsequent perception: (1) low prestimulus alpha-band power leads to high poststimulus gamma-band power, biasing perception such that two stimuli were perceived; (2) intermediate prestimulus alpha-band power leads to low gamma-band power (interpreted as inefficient stimulus processing), consequently, perception was not biased in either direction; and (3) high prestimulus alpha-band power leads to high poststimulus gamma-band power, biasing perception such that only one stimulus was perceived.

Transcranial Alternating Current Stimulation (tACS) Mechanisms and Protocols

Perception, cognition and consciousness can be modulated as a function of oscillating neural activity, while ongoing neuronal dynamics are influenced by synaptic activity and membrane potential. Consequently, transcranial alternating current stimulation (tACS) may be used for neurological intervention. The advantageous features of tACS include the biphasic and sinusoidal tACS currents, the ability to entrain large neuronal populations, and subtle control over somatic effects. Through neuromodulation of phasic, neural activity, tACS is a powerful tool to investigate the neural correlates of cognition. The rapid development in this area requires clarity about best practices. Here we briefly introduce tACS and review the most compelling findings in the literature to provide a starting point for using tACS. We suggest that tACS protocols be based on functional brain mechanisms and appropriate control experiments, including active sham and condition blinding.

Self-Grounded Vision: Hand Ownership Modulates Visual Location through Cortical β and γ Oscillations

Vision is known to be shaped by context, defined by environmental and bodily signals. In the Taylor illusion, the size of an afterimage projected on one's hand changes according to proprioceptive signals conveying hand position. Here, we assessed whether the Taylor illusion does not just depend on the physical hand position, but also on bodily self-consciousness as quantified through illusory hand ownership. Relying on the somatic rubber hand illusion, we manipulated hand ownership, such that participants embodied a rubber hand placed next to their own hand. We found that an afterimage projected on the participant's hand drifted depending on illusory ownership between the participants' two hands, showing an implication of self-representation during the Taylor illusion. Oscillatory power analysis of electroencephalographic signals showed that illusory hand ownership was stronger in participants with stronger α suppression over left sensorimotor cortex, whereas the Taylor illusion correlated with higher β/γ power over frontotemporal regions. Higher γ connectivity between left sensorimotor and inferior parietal cortex was also found during illusory hand ownership. These data show that afterimage drifts in the Taylor illusion do not only depend on the physical hand position but also on subjective ownership, which itself is based on the synchrony of somatosensory signals from the two hands. The effect of ownership on afterimage drifts is associated with β/γ power and γ connectivity between frontoparietal regions and the visual cortex. Together, our results suggest that visual percepts are not only influenced by bodily context but are self-grounded, mapped on a self-referential frame.

SIGNIFICANCE STATEMENT

Vision is influenced by the body: in the Taylor illusion, the size of an afterimage projected on one's hand changes according to tactile and proprioceptive signals conveying hand position. Here, we report a new phenomenon revealing that the perception of afterimages depends not only on bodily signals, but also on the sense of self. Relying on the rubber hand illusion, we manipulated hand ownership, so that participants embodied a rubber hand placed next to their own hand. We found that visual afterimages projected on the participant's hand drifted laterally, only when the rubber hand was embodied. Electroencephalography revealed spectral dissociations between somatic and visual effects, and higher γ connectivity along the dorsal visual pathways when the rubber hand was embodied.

Multisensory temporal function and EEG complexity in patients with epilepsy and psychogenic nonepileptic events

Cognitive and perceptual comorbidities frequently accompany epilepsy and psychogenic nonepileptic events (PNEE). However, and despite the fact that perceptual function is built upon a multisensory foundation, little knowledge exists concerning multisensory function in these populations. Here, we characterized facets of multisensory processing abilities in patients with epilepsy and PNEE, and probed the relationship between individual resting-state EEG complexity and these psychophysical measures in each patient. We prospectively studied a cohort of patients with epilepsy (N=18) and PNEE (N=20) patients who were admitted to Vanderbilt's Epilepsy Monitoring Unit (EMU) and weaned off of anticonvulsant drugs. Unaffected age-matched persons staying with the patients in the EMU (N=15) were also recruited as controls. All participants performed two tests of multisensory function: an audio-visual simultaneity judgment and an audio-visual redundant target task. Further, in the cohort of patients with epilepsy and PNEE we quantified resting state EEG gamma power and complexity. Compared with both patients with epilepsy and control subjects, patients with PNEE exhibited significantly poorer acuity in audiovisual temporal function as evidenced in significantly larger temporal binding windows (i.e., they perceived larger stimulus asynchronies as being presented simultaneously). These differences appeared to be specific for temporal function, as there was no difference among the three groups in a non-temporally based measure of multisensory function - the redundant target task. Further, patients with PNEE exhibited more complex resting state EEG patterns as compared to their patients with epilepsy, and EEG complexity correlated with multisensory temporal performance on a subject-by-subject manner. Taken together, findings seem to indicate that patients with PNEE bind information from audition and vision over larger temporal intervals when compared with control subjects as well as patients with epilepsy. This difference in multisensory function appears to be specific to the temporal domain, and may be a contributing factor to the behavioral and perceptual alterations seen in this population.

Beta/Gamma Oscillations and Event-Related Potentials Indicate Aberrant Multisensory Processing in Schizophrenia

Recent behavioral and neuroimaging studies have suggested multisensory processing deficits in patients with schizophrenia (SCZ). Thus far, the neural mechanisms underlying these deficits are not well understood. Previous studies with unisensory stimulation have shown altered neural oscillations in SCZ. As such, altered oscillations could contribute to aberrant multisensory processing in this patient group. To test this assumption, we conducted an electroencephalography (EEG) study in 15 SCZ and 15 control participants in whom we examined neural oscillations and event-related potentials (ERPs) in the sound-induced flash illusion (SIFI). In the SIFI multiple auditory stimuli that are presented alongside a single visual stimulus can induce the illusory percept of multiple visual stimuli. In SCZ and control participants we compared ERPs and neural oscillations between trials that induced an illusion and trials that did not induce an illusion. On the behavioral level, SCZ (55.7%) and control participants (55.4%) did not significantly differ in illusion rates. The analysis of ERPs revealed diminished amplitudes and altered multisensory processing in SCZ compared to controls around 135 ms after stimulus onset. Moreover, the analysis of neural oscillations revealed altered 25–35 Hz power after 100 to 150 ms over occipital scalp for SCZ compared to controls. Our findings extend previous observations of aberrant neural oscillations in unisensory perception paradigms. They suggest that altered ERPs and altered occipital beta/gamma band power reflect aberrant multisensory processing in SCZ.

Induced Gamma-Band Activity and Human Brain Function

Oscillatory activity in the gamma-band range has been related both to gestalt perception and to cognitive functions such as attention, learning, and memory. After giving a brief account of recent findings from electroencephalography and intracortical recordings, the present review will focus on spectral activity in the magnetoencephalogram. Here, gamma-band effects are topographically more local and involve higher frequencies than in the electroencephalogram. Bottom-up-driven auditory spatial mismatch detection elicits gamma-band activity over posterior parietal cortex, whereas auditory pattern mismatch processing leads to gamma-band enhancements over anterior temporal and inferior frontal regions. These topographies support representations of auditory spatial and pattern information in the putative dual auditory “where” and “what” pathways, respectively. During top-down-guided auditory spatial and pattern-working memory tasks, prefrontal gamma-band increases are observed in addition to activations over putative auditory stream areas. Moreover, stimulus maintenance in working memory is accompanied by coherence increases between sensory and prefrontal regions. Gamma-band topographies in magnetoencephalogram are highly comparable with hemodynamic brain imaging studies but yield additional information on the temporal dynamics of activations and connectivity patterns. In summary, magnetoencephalographic gammaband activity revealed both local synchronization patterns and cortico-cortical interactions accompanying cognitive processes at a good spatial and high temporal resolution.

Fusion and fission in the visual pathways

Inconsistent information from different modalities can be delusive for perception. This phenomenon can be observed with simultaneously presented inconsistent numbers of brief flashes and short tones. The conflict of bimodal information is reflected in double flash or fission, and flash fusion illusions, respectively. The temporal resolution of the vision system plays a fundamental role in the development of these illusions. As the parallel, dorsal and ventral pathways have different temporal resolution we presume that these pathways play different roles in the illusions. We used pathway-optimized stimuli to induce the illusions on separately driven visual streams. Our results show that both pathways support the double flash illusion, while the presence of the fusion illusion depends on the activated pathway. The dorsal pathway, which has better temporal resolution, does not support fusion, while the ventral pathway which has worse temporal resolution shows fusion strongly.

Pre- and post-stimulus EEG patterns associated with the touch-induced illusory flash

Pairing two brief auditory beeps with a single flash can evoke the percept of a second, illusory, flash. Investigations of the underlying neural mechanisms are limited to post-stimulus effects of this sound-induced illusory flash. We investigated whether touch modulates the visual evoked potential in a similar vein, and also looked at pre-stimulus activity. Electroencephalogram (EEG) was recorded over occipital and parieto-occipital areas of 12 observers. We compared bimodal EEG to its unimodal constituents (i.e., the difference waves) and found significant positive deflections around 110 ms and 200 ms and negative deflections around 330 ms and 390 ms from stimulus onset. These results are similar to those reported for the sound-induced illusion, albeit somewhat later. Furthermore, comparison of the EEG activity between those trials in which the illusion was perceived and those in which it was absent revealed that the phase of pre-stimulus alpha was linked to perceiving the illusion or not. We conclude that touch can modulate activity in the visual cortex and that similar neural mechanisms underlie perception of the sound- and touch-induced illusory flash and that the phase of the alpha wave at the moment of presentation that affects perception.

Neural responses in parietal and occipital areas in response to visual events are modulated by prior multisensory stimuli

The effect of multi-modal vs uni-modal prior stimuli on the subsequent processing of a simple flash stimulus was studied in the context of the audio-visual 'flash-beep' illusion, in which the number of flashes a person sees is influenced by accompanying beep stimuli. EEG recordings were made while combinations of simple visual and audio-visual stimuli were presented. The experiments found that the electric field strength related to a flash stimulus was stronger when it was preceded by a multi-modal flash/beep stimulus, compared to when it was preceded by another uni-modal flash stimulus. This difference was found to be significant in two distinct timeframes--an early timeframe, from 130-160 ms, and a late timeframe, from 300-320 ms. Source localisation analysis found that the increased activity in the early interval was localised to an area centred on the inferior and superior parietal lobes, whereas the later increase was associated with stronger activity in an area centred on primary and secondary visual cortex, in the occipital lobe. The results suggest that processing of a visual stimulus can be affected by the presence of an immediately prior multisensory event. Relatively long-lasting interactions generated by the initial auditory and visual stimuli altered the processing of a subsequent visual stimulus.

A brain basis for musical hallucinations

The physiological basis for musical hallucinations (MH) is not understood. One obstacle to understanding has been the lack of a method to manipulate the intensity of hallucination during the course of experiment. Residual inhibition, transient suppression of a phantom percept after the offset of a masking stimulus, has been used in the study of tinnitus. We report here a human subject whose MH were residually inhibited by short periods of music. Magnetoencephalography (MEG) allowed us to examine variation in the underlying oscillatory brain activity in different states. Source-space analysis capable of single-subject inference defined left-lateralised power increases, associated with stronger hallucinations, in the gamma band in left anterior superior temporal gyrus, and in the beta band in motor cortex and posteromedial cortex. The data indicate that these areas form a crucial network in the generation of MH, and are consistent with a model in which MH are generated by persistent reciprocal communication in a predictive coding hierarchy.

Audition influences color processing in the sound-induced visual flash illusion

Multisensory interactions can lead to illusory percepts, as exemplified by the sound-induced extra flash illusion (SIFI: Shams, Kamitani, & Shimojo, 2000, 2002). In this illusion, an audio-visual stimulus sequence consisting of two pulsed sounds and a light flash presented within a 100 ms time window generates the visual percept of two flashes. Here, we used colored visual stimuli to investigate whether concurrent auditory stimuli can affect the perceived features of the illusory flash. Zero, one or two pulsed sounds were presented concurrently with either a red or green flash or with two flashes of different colors (red followed by green) in rapid sequence. By querying both the number and color of the participants' visual percepts, we found that the double flash illusion is stimulus specific: i.e., two sounds paired with one red or one green flash generated the percept of two red or two green flashes, respectively. This implies that the illusory second flash is induced at a level of visual processing after perceived color has been encoded. In addition, we found that the presence of two sounds influenced the integration of color information from two successive flashes. In the absence of any sounds, a red and a green flash presented in rapid succession fused to form a single orange percept, but when accompanied by two sounds, this integrated orange percept was perceived to flash twice on a significant proportion of trials. In addition, the number of concurrent auditory stimuli modified the degree to which the successive flashes were integrated to an orange percept vs. maintained as separate red-green percepts. Overall, these findings show that concurrent auditory input can affect both the temporal and featural properties of visual percepts.

Parietal connectivity mediates multisensory facilitation

Our senses interact in daily life through multisensory integration, facilitating perceptual processes and behavioral responses. The neural mechanisms proposed to underlie this multisensory facilitation include anatomical connections directly linking early sensory areas, indirect connections to higher-order multisensory regions, as well as thalamic connections. Here we examine the relationship between white matter connectivity, as assessed with diffusion tensor imaging, and individual differences in multisensory facilitation and provide the first demonstration of a relationship between anatomical connectivity and multisensory processing in typically developed individuals. Using a whole-brain analysis and contrasting anatomical models of multisensory processing we found that increased connectivity between parietal regions and early sensory areas was associated with the facilitation of reaction times to multisensory (auditory-visual) stimuli. Furthermore, building on prior animal work suggesting the involvement of the superior colliculus in this process, using probabilistic tractography we determined that the strongest cortical projection area connected with the superior colliculus includes the region of connectivity implicated in our independent whole-brain analysis.

Multisensory integration and neuroplasticity in the human cerebral cortex

There is a strong interaction between multisensory processing and the neuroplasticity of the human brain. On one hand, recent research demonstrates that experience and training in various domains modifies how information from the different senses is integrated; and, on the other hand multisensory training paradigms seem to be particularly effective in driving functional and structural plasticity. Multisensory training affects early sensory processing within separate sensory domains, as well as the functional and structural connectivity between uni- and multisensory brain regions. In this review, we discuss the evidence for interactions of multisensory processes and brain plasticity and give an outlook on promising clinical applications and open questions.

Nothing is irrelevant in a noisy world: sensory illusions reveal obligatory within-and across-modality integration

In the "flash-beep illusion," a single light flash is perceived as multiple flashes when presented in close temporal proximity to multiple auditory beeps. Accounts of this illusion argue that temporal auditory information interferes with visual information because temporal acuity is better in audition than vision. However, it may also be that whenever there are multiple sensory inputs, the interference caused by a to-be-ignored stimulus on an attended stimulus depends on the likelihood that the stimuli are perceived as coming from a single distal source. Here we explore, in human observers, perceptual interactions between competing auditory and visual inputs while varying spatial proximity, which affects object formation. When two spatially separated streams are presented in the same (visual or auditory) modality, temporal judgments about a target stream from one direction are biased by the content of the competing distractor stream. Cross-modally, auditory streams from both target and distractor directions bias the perceived number of events in a target visual stream; however, importantly, the auditory stream from the target direction influences visual judgments more than does the auditory stream from the opposite hemifield. As in the original flash-beep illusion, visual streams weakly influence auditory judgments, regardless of spatial proximity. We also find that perceptual interference in the flash-beep illusion is similar to within-modality interference from a competing same-modality stream. Results reveal imperfect and obligatory within- and across-modality integration of information, and hint that the strength of these interactions depends on object binding.

Neurophysiological investigation of idiopathic acquired auditory-visual synesthesia

We present a case of acquired auditory-visual synesthesia and its neurophysiological investigation in a healthy 42-year-old woman. She started experiencing persistent positive and intermittent negative visual phenomena at age 37 followed by auditory-visual synesthesia. Her neurophysiological investigation included video-EEG, fMRI, and MEG. Auditory stimuli (700 Hz, 50 ms duration, 0.5 s ISI) were presented binaurally at 60 db above the hearing threshold in a dark room. The patient had bilateral symmetrical auditory-evoked neuromagnetic responses followed by an occipital-evoked field 16.3 ms later. The activation of occipital cortex following auditory stimuli may represent recruitment of existing cross-modal sensory pathways.

Grapheme-color synesthetes show enhanced crossmodal processing between auditory and visual modalities

Synesthesia is an involuntary experience in which stimulation of one sensory modality triggers additional, atypical sensory experiences. Strong multisensory processes are present in the general population, but the relationship between these 'normal' sensory interactions and synesthesia is currently unknown. Neuroimaging research suggests that some forms of synesthesia are caused by enhanced cross-activation between brain areas specialized for the processing of different sensory attributes, and finds evidence of increased white matter connections among regions known to be involved in typical crossmodal processes. Using two classic crossmodal integration tasks we show that grapheme-color synesthetes exhibit enhanced crossmodal interactions between auditory and visual modalities, suggesting that the experience of synesthesia in one modality generalizes to enhanced crossmodal processes with other modalities. This finding supports our conjecture that the atypical sensory experiences of synesthetes represent a selective expression of a more diffuse propensity toward 'typical' crossmodality interactions.

Perception of the touch-induced visual double-flash illusion correlates with changes of rhythmic neuronal activity in human visual and somatosensory areas

A single brief visual stimulus accompanied by two brief tactile stimuli is frequently perceived incorrectly as two flashes, a phenomenon called double-flash illusion (DFI). We investigated whether the DFI is accompanied by changes in rhythmic neuronal activity, using magnetoencephalography in human subjects. Twenty-two subjects received visuo-tactile stimulation and reported the number of perceived visual stimuli. We sorted trials with identical physical stimulation according to the reported subjective percept and assessed differences in spectral power in somatosensory and occipital sensors. In DFI trials, occipital sensors displayed a contralateral enhancement of gamma-band (80-140 Hz) activity in response to stimulation. In somatosensory sensors, the DFI was associated with an increase of spectral power for low frequencies (5-17.5 Hz) around stimulation and a decrease of spectral power in the 22.5-30 Hz range between 450 and 750 ms post-stimulation. In summary, several components of rhythmic activity predicted variable subjective experience for constant physical stimulation. Notably, the enhanced occipital gamma-band activity during DFI was similar in time and frequency extent to the somatosensory gamma-band response to tactile stimulation. We speculate that the DFI might therefore occur when the somatosensory gamma-response is transmitted to visual cortex. This transmission might be supported by the observed modulations in low-frequency activity.

Effect of Attention on Early Cortical Processes Associated with the Sound-induced Extra Flash Illusion

When a single flash of light is presented interposed between two brief auditory stimuli separated by 60–100 msec, subjects typically report perceiving two flashes [Shams, L., Kamitani, Y., & Shimojo, S. Visual illusion induced by sound. Brain Research, Cognitive Brain Research, 14, 147–152, 2002; Shams, L., Kamitani, Y., & Shimojo, S. Illusions. What you see is what you hear. Nature, 408, 788, 2000]. Using ERP recordings, we previously found that perception of the illusory extra flash was accompanied by a rapid dynamic interplay between auditory and visual cortical areas that was triggered by the second sound [Mishra, J., Martínez, A., Sejnowski, T. J., & Hillyard, S. A. Early cross-modal interactions in auditory and visual cortex underlie a sound-induced visual illusion. Journal of Neuroscience, 27, 4120–4131, 2007]. In the current study, we investigated the effect of attention on the ERP components associated with the illusory extra flash in 15 individuals who perceived this cross-modal illusion frequently. All early ERP components in the cross-modal difference wave associated with the extra flash illusion were significantly enhanced by selective spatial attention. The earliest attention-related modulation was an amplitude increase of the positive-going PD110/PD120 component, which was previously shown to be correlated with an individual's propensity to perceive the illusory second flash [Mishra, J., Martínez, A., Sejnowski, T. J., & Hillyard, S. A. Early cross-modal interactions in auditory and visual cortex underlie a sound-induced visual illusion. Journal of Neuroscience, 27, 4120–4131, 2007]. The polarity of the early PD110/PD120 component did not differ as a function of the visual field (upper vs. lower) of stimulus presentation. This, along with the source localization of the component, suggested that its principal generator lies in extrastriate visual cortex. These results indicate that neural processes previously shown to be associated with the extra flash illusion can be modulated by attention, and thus are not the result of a wholly automatic cross-modal integration process.

Integration of auditory and vibrotactile stimuli: effects of phase and stimulus-onset asynchrony

The perceptual integration of 250 Hz, 500 ms vibrotactile and auditory tones was studied in detection experiments as a function of (1) relative phase and (2) temporal asynchrony of the tone pulses. Vibrotactile stimuli were delivered through a single-channel vibrator to the left middle fingertip and auditory stimuli were presented diotically through headphones in a background of 50 dB sound pressure level broadband noise. The vibrotactile and auditory stimulus levels used each yielded 63%-77%-correct unimodal detection performance in a 2-I, 2-AFC task. Results for combined vibrotactile and auditory detection indicated that (1) performance improved for synchronous presentation, (2) performance was not affected by the relative phase of the auditory and tactile sinusoidal stimuli, and (3) performance for non-overlapping stimuli improved only if the tactile stimulus preceded the auditory. The results are generally more consistent with a "Pythagorean Sum" model than with either an "Algebraic Sum" or an "Optimal Single-Channel" Model of perceptual integration. Thus, certain combinations of auditory and tactile signals result in significant integrative effects. The lack of phase effect suggests an envelope rather than fine-structure operation for integration. The effects of asynchronous presentation of the auditory and tactile stimuli are consistent with time constants deduced from single-modality masking experiments.

Investigating neuromagnetic brain responses against chromatic flickering stimuli by wavelet entropies

BACKGROUND

Photosensitive epilepsy is a type of reflexive epilepsy triggered by various visual stimuli including colourful ones. Despite the ubiquitous presence of colorful displays, brain responses against different colour combinations are not properly studied.

METHODOLOGY/PRINCIPAL FINDINGS

Here, we studied the photosensitivity of the human brain against three types of chromatic flickering stimuli by recording neuromagnetic brain responses (magnetoencephalogram, MEG) from nine adult controls, an unmedicated patient, a medicated patient, and two controls age-matched with patients. Dynamical complexities of MEG signals were investigated by a family of wavelet entropies. Wavelet entropy is a newly proposed measure to characterize large scale brain responses, which quantifies the degree of order/disorder associated with a multi-frequency signal response. In particular, we found that as compared to the unmedicated patient, controls showed significantly larger wavelet entropy values. We also found that Renyi entropy is the most powerful feature for the participant classification. Finally, we also demonstrated the effect of combinational chromatic sensitivity on the underlying order/disorder in MEG signals.

CONCLUSIONS/SIGNIFICANCE

Our results suggest that when perturbed by potentially epileptic-triggering stimulus, healthy human brain manages to maintain a non-deterministic, possibly nonlinear state, with high degree of disorder, but an epileptic brain represents a highly ordered state which making it prone to hyper-excitation. Further, certain colour combination was found to be more threatening than other combinations.

The impact of spatial incongruence on an auditory-visual illusion

Background

The sound-induced flash illusion is an auditory-visual illusion – when a single flash is presented along with two or more beeps, observers report seeing two or more flashes. Previous research has shown that the illusion gradually disappears as the temporal delay between auditory and visual stimuli increases, suggesting that the illusion is consistent with existing temporal rules of neural activation in the superior colliculus to multisensory stimuli. However little is known about the effect of spatial incongruence, and whether the illusion follows the corresponding spatial rule. If the illusion occurs less strongly when auditory and visual stimuli are separated, then integrative processes supporting the illusion must be strongly dependant on spatial congruence. In this case, the illusion would be consistent with both the spatial and temporal rules describing response properties of multisensory neurons in the superior colliculus.

Methodology/Principal Findings

The main aim of this study was to investigate the importance of spatial congruence in the flash-beep illusion. Selected combinations of one to four short flashes and zero to four short 3.5 KHz tones were presented. Observers were asked to count the number of flashes they saw. After replication of the basic illusion using centrally-presented stimuli, the auditory and visual components of the illusion stimuli were presented either both 10 degrees to the left or right of fixation (spatially congruent) or on opposite (spatially incongruent) sides, for a total separation of 20 degrees.

Conclusions/Significance

The sound-induced flash fission illusion was successfully replicated. However, when the sources of the auditory and visual stimuli were spatially separated, perception of the illusion was unaffected, suggesting that the “spatial rule” does not extend to describing behavioural responses in this illusion. We also find no evidence for an associated “fusion” illusion reportedly occurring when multiple flashes are accompanied by a single beep.

Audiovisual temporal capture underlies flash fusion

When sequential visual flashes are accompanied by a lower number of sequential auditory pulses, the perceived number of visual flashes is lower than the actual number, an illusion termed 'flash fusion'. We examined whether temporal capture of flashes by pulses underlay flash fusion. One of the visual flashes was given a luminance increment, and observers reported which flash had the luminance increment. Results showed that the pulse strongly captured the flashes in its temporal vicinity, resulting in flash fusion. Moreover, when one of the successive pulses was given a higher frequency than others, the luminance increment was perceptually paired with the pulse with the higher frequency. The pairing of audiovisual features disappeared when the temporal pattern of the pulse frequency was difficult for the observer to anticipate. These data indicate that flash fusion is caused by temporal capture of flashes by the pulse, and that feature matching between auditory and visual signals also contributes to the modulation of perceived temporal structure of flashes during flash fusion.

Sound-induced flash illusion is resistant to feedback training

A single flash accompanied by two auditory beeps tends to be perceived as two flashes (Shams et al. Nature 408:788, 2000, Cogn Brain Res 14:147-152, 2002). This phenomenon is known as 'sound-induced flash illusion.' Previous neuroimaging studies have shown that this illusion is correlated with modulation of activity in early visual cortical areas (Arden et al. Vision Res 43(23):2469-2478, 2003; Bhattacharya et al. NeuroReport 13:1727-1730, 2002; Shams et al. NeuroReport 12(17):3849-3852, 2001, Neurosci Lett 378(2):76-81, 2005; Watkins et al. Neuroimage 31:1247-1256, 2006, Neuroimage 37:572-578, 2007; Mishra et al. J Neurosci 27(15):4120-4131, 2007). We examined how robust the illusion is by testing whether the frequency of the illusion can be reduced by providing feedback. We found that the sound-induced flash illusion was resistant to feedback training, except when the amount of monetary reward was made dependent on accuracy in performance. However, even in the latter case the participants reported that they still perceived illusory two flashes even though they correctly reported single flash. Moreover, the feedback training effect seemed to disappear once the participants were no longer provided with feedback suggesting a short-lived refinement of discrimination between illusory and physical double flashes rather than vanishing of the illusory percept. These findings indicate that the effect of sound on the perceptual representation of visual stimuli is strong and robust to feedback training, and provide further evidence against decision factors accounting for the sound-induced flash illusion.