Grey matter volume in early human visual cortex predicts proneness to the sound-induced flash illusion

Prior multisensory learning can facilitate auditory-only voice-identity and speech recognition in noise

Seeing the visual articulatory movements of a speaker, while hearing their voice, helps with understanding what is said. This multisensory enhancement is particularly evident in noisy listening conditions. Multisensory enhancement also occurs even in auditory-only conditions: auditory-only speech and voice-identity recognition are superior for speakers previously learned with their face, compared to control learning; an effect termed the "face-benefit." Whether the face-benefit can assist in maintaining robust perception in increasingly noisy listening conditions, similar to concurrent multisensory input, is unknown. Here, in two behavioural experiments, we examined this hypothesis. In each experiment, participants learned a series of speakers' voices together with their dynamic face or control image. Following learning, participants listened to auditory-only sentences spoken by the same speakers and recognised the content of the sentences (speech recognition, Experiment 1) or the voice-identity of the speaker (Experiment 2) in increasing levels of auditory noise. For speech recognition, we observed that 14 of 30 participants (47%) showed a face-benefit. 19 of 25 participants (76%) showed a face-benefit for voice-identity recognition. For those participants who demonstrated a face-benefit, the face-benefit increased with auditory noise levels. Taken together, the results support an audio-visual model of auditory communication and suggest that the brain can develop a flexible system in which learned facial characteristics are used to deal with varying auditory uncertainty.

Metacognitive awareness in the sound-induced flash illusion

Hundreds (if not thousands) of multisensory studies provide evidence that the human brain can integrate temporally and spatially discrepant stimuli from distinct modalities into a singular event. This process of multisensory integration is usually portrayed in the scientific literature as contributing to our integrated, coherent perceptual reality. However, missing from this account is an answer to a simple question: how do confidence judgements compare between multisensory information that is integrated across multiple sources, and multisensory information that comes from a single, congruent source in the environment? In this paper, we use the sound-induced flash illusion to investigate if confidence judgements are similar across multisensory conditions when the numbers of auditory and visual events are the same, and the numbers of auditory and visual events are different. Results showed that congruent audiovisual stimuli produced higher confidence than incongruent audiovisual stimuli, even when the perceptual report was matched across the two conditions. Integrating these behavioural findings with recent neuroimaging and theoretical work, we discuss the role that prefrontal cortex may play in metacognition, multisensory causal inference and sensory source monitoring in general. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

Rhythmic TMS as a Feasible Tool to Uncover the Oscillatory Signatures of Audiovisual Integration

Multisensory integration is quintessential to adaptive behavior, with clinical populations showing significant impairments in this domain, most notably hallucinatory reports. Interestingly, altered cross-modal interactions have also been reported in healthy individuals when engaged in tasks such as the Sound-Induced Flash-Illusion (SIFI). The temporal dynamics of the SIFI have been recently tied to the speed of occipital alpha rhythms (IAF), with faster oscillations entailing reduced temporal windows within which the illusion is experienced. In this regard, entrainment-based protocols have not yet implemented rhythmic transcranial magnetic stimulation (rhTMS) to causally test for this relationship. It thus remains to be evaluated whether rhTMS-induced acoustic and somatosensory sensations may not specifically interfere with the illusion. Here, we addressed this issue by asking 27 volunteers to perform a SIFI paradigm under different Sham and active rhTMS protocols, delivered over the occipital pole at the IAF. Although TMS has been proven to act upon brain tissues excitability, results show that the SIFI occurred for both Sham and active rhTMS, with the illusory rate not being significantly different between baseline and stimulation conditions. This aligns with the discrete sampling hypothesis, for which alpha amplitude modulation, known to reflect changes in cortical excitability, should not account for changes in the illusory rate. Moreover, these findings highlight the viability of rhTMS-based interventions as a means to probe the neuroelectric signatures of illusory and hallucinatory audiovisual experiences, in healthy and neuropsychiatric populations.

The effects of estradiol levels on crossmodal perception: a study on the sound induced flash illusion in healthy and menstrually related migraine individuals

OBJECTIVE

The sound-induced flash illusion (SIFI) is a valid paradigm to study multisensorial perception. In the "fission" SIFI, multiple flashes are perceived when observing a single flash paired with two or more beeps. SIFI is largely dependent on visual and acoustic cortex excitability; in migraine, dysfunctional cortical excitability affects SIFI perception. Since estrogen peak occurring during ovulation can increase neuronal excitability, the present study aims to verify whether cortical excitability shifts linked to the menstrual cycle could influence SIFI.

METHODS

In a comparative prospective study, we tested the effect of estrogens on crossmodal perception using the SIFI. We recruited 27 females in reproductive age, including 16 healthy and 11 menstrually related migraine females, testing their proneness to SIFI on day 14 (high estradiol) and day 27 (low estradiol) of menstrual cycle.

RESULTS

Women on day 14 reported less flashes than on day 27 (p = 0.02) in the fission illusion, suggesting a pro-excitatory effect of estradiol on visual cortex excitability during ovulation. Moreover, we confirmed that migraine women perceived less flashes (p = 0.001) than controls, independently from cycle phase. Non-migraineurs women significantly reported more flashes on day 27 than on day 14 (p = 0.04).

CONCLUSIONS

This study suggests that estradiol may influence the multisensory perception due to changes of visual cortex excitability, with high estradiol peak leading to increased visual cortical sensitivity during ovulation in non-migraineurs. Visual cortex hyperresponsiveness, here reflected by reduced SIFI, is not influenced by estradiol fluctuations in migraine women, as shown by reduced fission effects on day 14 and 27.

Sound-induced flash illusions at different spatial locations were affected by personality traits

Sound-induced flash illusion (SiFI) is an auditory-dominated effect in which observers will misperceive the number of flashes due to simultaneously presented beeps, which includes fission and fusion illusions. Although several individual differences have been found in SiFI, little is known about the effect of personality traits. In the present study, we presented flashes in near space and beeps in far space (Vnear_Afar) and flashes in far space and beeps in near space (Vfar_Anear) to better approximate the real world. We collected 103 participants' Big Five questionnaire results and their SiFI task performance to investigate the difference in trait level on the SiFI in the performance of accuracy, d' and c. The results show that all five personality traits had certain effects on the SiFI to different degrees, and different personality traits played different roles in the fission illusion and fusion illusion. The high agreeableness group was more prone to the fission illusion, and the report criteria were less strict. The report criteria of the low neuroticism group were stricter for the fusion illusion. The extraversion, conscientiousness and low openness groups were more prone to the fusion illusion in the Vnear_Afar condition than in the Vfar_Anear condition. The study indicated that personality traits were important but easily overlooked factors in multisensory illusion, which might make a difference between the fission illusion and the fusion illusion.

Differences in eccentricity for sound-induced flash illusion in four visual fields

A sound-induced flash illusion (SiFI) is a multisensory illusion dominated by auditory stimuli, in which the individual perceives that the number of visual flashes is equal to the number of auditory stimuli when visual flashes are presented along with an unequal number of auditory stimuli. Although the mechanisms underlying fission and fusion illusions have been documented, there is not yet a consensus on how they vary according to the different eccentricities. In the present study, by incorporating the classic SiFI paradigm into four different eccentricities, we aimed to investigate whether the SiFI varies under the different eccentricities. The results showed that the fission illusion varied significantly across the four eccentricities, with the perifovea (7°) and peripheral (11°) illusions being greater than the fovea and parafovea (3°) illusions. In contrast, the fusion illusion did not vary significantly across the four eccentricities. Our findings revealed that SiFI was affected by different visual fields and that there were differences between the fission and the fusion illusions. Furthermore, by examining the SiFI of eccentricity across visual fields, this study also suggests that bottom-up factors affect the SiFI.

Compensation Mechanisms May Not Always Account for Enhanced Multisensory Illusion in Older Adults: Evidence from Sound-Induced Flash Illusion

Sound-induced flash illusion (SiFI) is typical auditory dominance phenomenon in multisensory illusion. Although a number of studies have explored the SiFI in terms of age-related effects, the reasons for the enhanced SiFI in older adults are still controversial. In the present study, older and younger adults with equal visual discrimination were selected to explore age differences in SiFI effects, and to explore the neural indicators by resting-state functional magnetic resonance imaging (rs-fMRI) signals. A correlation analysis was calculated to examine the relationship between regional homogeneity (ReHo) and the SiFI. The results showed that both younger and older adults experienced significant fission and fusion illusions, and fission illusions of older adults were greater than that of younger adults. In addition, our results showed ReHo values of the left middle frontal gyrus (MFG), the right inferior frontal gyrus (IFG) and right superior frontal gyrus (SFG) were significantly positively correlated with the SiFI in older adults. More importantly, the comparison between older and younger adults showed that ReHo values of the right superior temporal gyrus (STG) decreased in older adults, and this was independent of the SiFI. The results indicated that when there was no difference in unisensory ability, the enhancement of multisensory illusion in older adults may not always be explained by compensation mechanisms.

Reward reduces the fission illusion in the sound-induced flash illusion

Pairing a single visual stimulus with multiple auditory stimuli will lead to the illusory perception of multiple visual stimuli, which is known as sound-induced flash illusion (SIFI). The present study adopted the classic SIFI paradigm to investigate whether value-associated tasks could affect the SIFI. By adjusting the sequence of reward and nonreward conditions, we also examined the effect of reward history on SIFI. The results showed that the fission illusion was reduced when associated with momentary reward, demonstrating significantly higher accuracy and discriminability than the nonreward condition. However, the fusion illusion was not affected by the momentary reward, and the explanation was that the fusion illusion was not as stable as the fission illusion and disappeared across different trials and conditions. Moreover, the robustness of reward history in the present study was not as strong as previous studies have suggested, indicating that the effect of sound on the perceptual representation of visual stimuli is strong and robust to reward history. These findings demonstrated that the reward could reduce the SIFI and broaden the existing dichotomy of SIFI. New evidence for the operation of value-driven attention mechanisms is also provided, suggesting that the underlying value-driven attention operates across multiple sensory systems.

Treatment of Age-Related Hearing Loss Alters Audiovisual Integration and Resting-State Functional Connectivity: A Randomized Controlled Pilot Trial

Untreated age-related hearing loss increases audiovisual integration and impacts resting state functional brain connectivity. Further, there is a relation between crossmodal plasticity and audiovisual integration strength in cochlear implant patients. However, it is currently unclear whether amplification of the auditory input by hearing aids influences audiovisual integration and resting state functional brain connectivity. We conducted a randomized controlled pilot study to investigate how the McGurk illusion, a common measure for audiovisual integration, and resting state functional brain connectivity of the auditory cortex are altered by six-month hearing aid use. Thirty-two older participants with slight-to-moderate, symmetric, age-related hearing loss were allocated to a treatment or waiting control group and measured one week before and six months after hearing aid fitting with functional magnetic resonance imaging. Our results showed a statistical trend for an increased McGurk illusion after six months of hearing aid use. We further demonstrated that an increase in McGurk susceptibility is related to a decreased hearing aid benefit for auditory speech intelligibility in noise. No significant interaction between group and time point was obtained in the whole-brain resting state analysis. However, a region of interest (ROI)-to-ROI analysis indicated that hearing aid use of six months was associated with a decrease in resting state functional connectivity between the auditory cortex and the fusiform gyrus and that this decrease was related to an increase of perceived McGurk illusions. Our study, therefore, suggests that even short-term hearing aid use alters audiovisual integration and functional brain connectivity between auditory and visual cortices.

Long-term training reduces the responses to the sound-induced flash illusion

The sound-induced flash illusion (SiFI) is a robust auditory-dominated multisensory integration phenomenon that is used as a reliable indicator to assess multisensory integration. Previous studies have indicated that the SiFI effect is correlated with perceptual sensitivity. However, to date, there is no consensus regarding how it corresponds to sensitivity with long-term training. The present study adopted the classic SiFI paradigm with feedback training to investigate the effect of a week of long-term training on the SiFI effect. Both the training group and control group completed a pretest and a posttest before and after the perceptual training; however, only the training group was required to complete 7-day behavioral training. The results showed that (1) long-term training could reduce the response of fission and fusion illusions by improving perceptual sensitivity and that (2) there was a "plateau effect" that emerged during the training stage, which tended to stabilize by the fifth day. These findings demonstrated that the SiFI effect could be modified with long-term training by ameliorating perceptual sensitivity, especially in terms of the fission illusion. Therefore, the present study supplements perceptual training in SiFI domains and provides evidence that the SiFI could be used as an assessment intervention to improve the efficiency of multisensory integration.

Gray matter volume in the right angular gyrus is associated with differential patterns of multisensory integration with aging

Multisensory perception might provide an important marker of brain function in aging. However, the cortical structures supporting multisensory perception in aging are poorly understood. In this study, we compared regional gray matter volume in a group of middle-aged (n = 101; 49-64 years) and older (n = 116; 71-87 years) adults from The Irish Longitudinal Study on Aging using voxel-based morphometry. Participants completed a measure of multisensory integration, the sound-induced flash illusion, and were grouped as per their illusion susceptibility. A significant interaction was observed in the right angular gyrus; in the middle-aged group, larger gray matter volume corresponded to stronger illusion perception while in older adults larger gray matter corresponded to less illusion susceptibility. This interaction remained significant even when controlling for a range of demographic, sensory, cognitive, and health variables. These findings show that multisensory integration is associated with specific structural differences in the aging brain and highlight the angular gyrus as a possible "cross-modal hub" associated with age-related change in multisensory perception.

Exposure to first-person shooter videogames is associated with multisensory temporal precision and migraine incidence

Adaptive interactions with the environment require optimal integration and segregation of sensory information. Yet, temporal misalignments in the presentation of visual and auditory stimuli may generate illusory phenomena such as the sound-induced flash illusion, in which a single flash paired with multiple auditory stimuli induces the perception of multiple illusory flashes. This phenomenon has been shown to be robust and resistant to feedback training. According to a Bayesian account, this is due to a statistically optimal combination of the signals operated by the nervous system. From this perspective, individual susceptibility to the illusion might be moulded through prolonged experience. For example, repeated exposure to the illusion and prolonged training sessions partially impact on the reported illusion. Therefore, extensive and immersive audio-visual experience, such as first-person shooter videogames, should sharpen individual capacity to correctly integrate multisensory information over time, leading to more veridical perception. We tested this hypothesis by comparing the temporal profile of the sound-induced illusion in a group of expert first-person shooter gamers and a non-players group. In line with the hypotheses, gamers experience significantly narrower windows of illusion (~87 ms) relative to non-players (~105 ms), leading to higher veridical reports in gamers (~68%) relative to non-players (~59%). Moreover, according to recent literature, we tested whether audio-visual intensive training in gamers could be related to the incidence of migraine, and found that its severity may be directly proportioned to the time spent on videogames. Overall, these results suggest that continued training within audio-visual environments such as first-person shooter videogames improves temporal discrimination and sensory integration. This finding may pave the way for future therapeutic strategies based on self-administered multisensory training. On the other hand, the impact of intensive training on visual-related stress disorders, such as migraine incidence, should be taken into account as a risk factor during therapeutic planning.

Everything has Its Time: Narrow Temporal Windows are Associated with High Levels of Autistic Traits Via Weaknesses in Multisensory Integration

The present study examined whether fundamental sensory functions such as temporal processing and multisensory integration are related to autistic traits in the general population. Both a narrower temporal window (TW) for simultaneous perception, as measured by a temporal order judgement task, and a reduced ability to engage in multisensory integration during the sound-induced flash illusion task were related to higher levels of autistic traits. Additionally, a narrow TW is associated with high levels of autistic traits due to a deficiency in multisensory integration. Taken together, these findings suggest that alterations in fundamental functions produce a cascading effect on higher-order social and cognitive functions, such as those experienced by people with autism spectrum disorder.

Double Flash Illusions: Current Findings and Future Directions

Twenty years ago, the first report on the sound-induced double flash illusion, a visual illusion induced by sound, was published. In this paradigm, participants are presented with different numbers of auditory and visual stimuli. In case of an incongruent number of auditory and visual stimuli, the influence of auditory information on visual perception can lead to the perception of the illusion. Thus, combining two auditory stimuli with one visual stimulus can induce the perception of two visual stimuli, the so-called fission illusion. Alternatively, combining one auditory stimulus with two visual stimuli can induce the perception of one visual stimulus, the so-called fusion illusion. Overall, current research shows that the illusion is a reliable indicator of multisensory integration. It has also been replicated using different stimulus combinations, such as visual and tactile stimuli. Importantly, the robustness of the illusion allows the widespread use for assessing multisensory integration across different groups of healthy participants and clinical populations and in various task setting. This review will give an overview of the experimental evidence supporting the illusion, the current state of research concerning the influence of cognitive processes on the illusion, the neural mechanisms underlying the illusion, and future research directions. Moreover, an exemplary experimental setup will be described with different options to examine perception, alongside code to test and replicate the illusion online or in the laboratory.

Age-related sensory decline mediates the Sound-Induced Flash Illusion: Evidence for reliability weighting models of multisensory perception

Perception of our world is proposed to arise from combining multiple sensory inputs according to their relative reliability. We tested multisensory processes in a large sample of 2920 older adults to assess whether sensory ability mediates age-related changes in perception. Participants completed a test of audio-visual integration, the Sound Induced Flash Illusion (SIFI), alongside measures of visual (acuity, contrast sensitivity, self-reported vision and visual temporal discrimination (VTD)) and auditory (self-reported hearing and auditory temporal discrimination (ATD)) function. Structural equation modelling showed that SIFI susceptibility increased with age. This was mediated by visual acuity and self-reported hearing: better scores on these measures predicted reduced and stronger SIFI susceptibility, respectively. Unexpectedly, VTD improved with age and predicted increased SIFI susceptibility. Importantly, the relationship between age and SIFI susceptibility remained significant, even when considering mediators. A second model showed that, with age, visual 'gain' (the benefit of congruent auditory information on visual judgements) was predicted by ATD: better ATD predicted stronger visual gain. However, neither age nor SIFI susceptibility were directly associated with visual gain. Our findings illustrate, in the largest sample of older adults to date, how multisensory perception is influenced, but not fully accounted for, by age-related changes in unisensory abilities.

Cognitive Impairment in Multiple Sclerosis Is Reflected by Increased Susceptibility to the Sound-Induced Flash Illusion

Objective: To determine whether the performance of multiple sclerosis (MS) patients in the sound-induced flash illusion (SiFi), a multisensory perceptual illusion, would reflect their cognitive impairment.

Methods: We performed the SiFi task as well as an extensive neuropsychological testing in 95 subjects [39 patients with relapse-remitting MS (RRMS), 16 subjects with progressive multiple sclerosis (PMS) and 40 healthy control subjects (HC)].

Results: MS patients reported more frequently the multisensory SiFi than HC. In contrast, there were no group differences in the control conditions. Essentially, patients with progressive type of MS continued to perceive the illusion at stimulus onset asynchronies (SOA) that were more than three times longer than the SOA at which the illusion was already disrupted for healthy controls. Furthermore, MS patients' degree of cognitive impairment measured with a broad neuropsychological battery encompassing tests for memory, attention, executive functions, and fluency was predicted by their performance in the SiFi task for the longest SOA of 500 ms.

Conclusions: These findings support the notion that MS patients exhibit an altered multisensory perception in the SiFi task and that their susceptibility to the perceptual illusion is negatively correlated with their neuropsychological test performance. Since MS lesions affect white matter tracts and cortical regions which seem to be involved in the transfer and processing of both crossmodal and cognitive information, this might be one possible explanation for our findings. SiFi might be considered as a brief, non-expensive, language- and education-independent screening test for cognitive deficits in MS patients.

How to Enhance the Power to Detect Brain-Behavior Correlations With Limited Resources

Neuroscience has been diagnosed with a pervasive lack of statistical power and, in turn, reliability. One remedy proposed is a massive increase of typical sample sizes. Parts of the neuroimaging community have embraced this recommendation and actively push for a reallocation of resources toward fewer but larger studies. This is especially true for neuroimaging studies focusing on individual differences to test brain-behavior correlations. Here, I argue for a more efficient solution. Ad hoc simulations show that statistical power crucially depends on the choice of behavioral and neural measures, as well as on sampling strategy. Specifically, behavioral prescreening and the selection of extreme groups can ascertain a high degree of robust in-sample variance. Due to the low cost of behavioral testing compared to neuroimaging, this is a more efficient way of increasing power. For example, prescreening can achieve the power boost afforded by an increase of sample sizes from n = 30 to n = 100 at ∼5% of the cost. This perspective article briefly presents simulations yielding these results, discusses the strengths and limitations of prescreening and addresses some potential counter-arguments. Researchers can use the accompanying online code to simulate the expected power boost of prescreening for their own studies.

Does hearing aid use affect audiovisual integration in mild hearing impairment?

There is converging evidence for altered audiovisual integration abilities in hearing-impaired individuals and those with profound hearing loss who are provided with cochlear implants, compared to normal-hearing adults. Still, little is known on the effects of hearing aid use on audiovisual integration in mild hearing loss, although this constitutes one of the most prevalent conditions in the elderly and, yet, often remains untreated in its early stages. This study investigated differences in the strength of audiovisual integration between elderly hearing aid users and those with the same degree of mild hearing loss who were not using hearing aids, the non-users, by measuring their susceptibility to the sound-induced flash illusion. We also explored the corresponding window of integration by varying the stimulus onset asynchronies. To examine general group differences that are not attributable to specific hearing aid settings but rather reflect overall changes associated with habitual hearing aid use, the group of hearing aid users was tested unaided while individually controlling for audibility. We found greater audiovisual integration together with a wider window of integration in hearing aid users compared to their age-matched untreated peers. Signal detection analyses indicate that a change in perceptual sensitivity as well as in bias may underlie the observed effects. Our results and comparisons with other studies in normal-hearing older adults suggest that both mild hearing impairment and hearing aid use seem to affect audiovisual integration, possibly in the sense that hearing aid use may reverse the effects of hearing loss on audiovisual integration. We suggest that these findings may be particularly important for auditory rehabilitation and call for a longitudinal study.

A simple and efficient method to enhance audiovisual binding tendencies

Individuals vary in their tendency to bind signals from multiple senses. For the same set of sights and sounds, one individual may frequently integrate multisensory signals and experience a unified percept, whereas another individual may rarely bind them and often experience two distinct sensations. Thus, while this binding/integration tendency is specific to each individual, it is not clear how plastic this tendency is in adulthood, and how sensory experiences may cause it to change. Here, we conducted an exploratory investigation which provides evidence that (1) the brain’s tendency to bind in spatial perception is plastic, (2) that it can change following brief exposure to simple audiovisual stimuli, and (3) that exposure to temporally synchronous, spatially discrepant stimuli provides the most effective method to modify it. These results can inform current theories about how the brain updates its internal model of the surrounding sensory world, as well as future investigations seeking to increase integration tendencies.

Perceptual similarity and the neural correlates of geometrical illusions in human brain structure

Geometrical visual illusions are an intriguing phenomenon, in which subjective perception consistently misjudges the objective, physical properties of the visual stimulus. Prominent theoretical proposals have been advanced attempting to find common mechanisms across illusions. But empirically testing the similarity between illusions has been notoriously difficult because illusions have very different visual appearances. Here we overcome this difficulty by capitalizing on the variability of the illusory magnitude across participants. Fifty-nine healthy volunteers participated in the study that included measurement of individual illusion magnitude and structural MRI scanning. We tested the Muller-Lyer, Ebbinghaus, Ponzo, and vertical-horizontal geometrical illusions as well as a non-geometrical, contrast illusion. We found some degree of similarity in behavioral judgments of all tested geometrical illusions, but not between geometrical illusions and non-geometrical, contrast illusion. The highest similarity was found between Ebbinghaus and Muller-Lyer geometrical illusions. Furthermore, the magnitude of all geometrical illusions, and particularly the Ebbinghaus and Muller-Lyer illusions, correlated with local gray matter density in the parahippocampal cortex, but not in other brain areas. Our findings suggest that visuospatial integration and scene construction processes might partly mediate individual differences in geometric illusory perception. Overall, these findings contribute to a better understanding of the mechanisms behind geometrical illusions.

Contextual modulation of primary visual cortex by auditory signals

Early visual cortex receives non-feedforward input from lateral and top-down connections (Muckli & Petro 2013 Curr. Opin. Neurobiol.23, 195–201. (doi:10.1016/j.conb.2013.01.020)), including long-range projections from auditory areas. Early visual cortex can code for high-level auditory information, with neural patterns representing natural sound stimulation (Vetter et al. 2014 Curr. Biol.24, 1256–1262. (doi:10.1016/j.cub.2014.04.020)). We discuss a number of questions arising from these findings. What is the adaptive function of bimodal representations in visual cortex? What type of information projects from auditory to visual cortex? What are the anatomical constraints of auditory information in V1, for example, periphery versus fovea, superficial versus deep cortical layers? Is there a putative neural mechanism we can infer from human neuroimaging data and recent theoretical accounts of cortex? We also present data showing we can read out high-level auditory information from the activation patterns of early visual cortex even when visual cortex receives simple visual stimulation, suggesting independent channels for visual and auditory signals in V1. We speculate which cellular mechanisms allow V1 to be contextually modulated by auditory input to facilitate perception, cognition and behaviour. Beyond cortical feedback that facilitates perception, we argue that there is also feedback serving counterfactual processing during imagery, dreaming and mind wandering, which is not relevant for immediate perception but for behaviour and cognition over a longer time frame.

This article is part of the themed issue ‘Auditory and visual scene analysis’.

Attention and multisensory modulation argue against total encapsulation

Firestone & Scholl (F&S) postulate that vision proceeds without any direct interference from cognition. We argue that this view is extreme and not in line with the available evidence. Specifically, we discuss two well-established counterexamples: Attention directly affects core aspects of visual processing, and multisensory modulations of vision originate on multiple levels, some of which are unlikely to fall "within perception."

The sound-induced flash illusion reveals dissociable age-related effects in multisensory integration

While aging can lead to significant declines in perceptual and cognitive function, the effects of age on multisensory integration, the process in which the brain combines information across the senses, are less clear. Recent reports suggest that older adults are susceptible to the sound-induced flash illusion (Shams et al., 2000) across a much wider range of temporal asynchronies than younger adults (Setti et al., 2011). To assess whether this cost for multisensory integration is a general phenomenon of combining asynchronous audiovisual input, we compared the time courses of two variants of the sound-induced flash illusion in young and older adults: the fission illusion, where one flash accompanied by two beeps appears as two flashes, and the fusion illusion, where two flashes accompanied by one beep appear as one flash. Twenty-five younger (18–30 years) and older (65+ years) adults were required to report whether they perceived one or two flashes, whilst ignoring irrelevant auditory beeps, in bimodal trials where auditory and visual stimuli were separated by one of six stimulus onset asynchronies (SOAs). There was a marked difference in the pattern of results for the two variants of the illusion. In conditions known to produce the fission illusion, older adults were significantly more susceptible to the illusion at longer SOAs compared to younger participants. In contrast, the performance of the younger and older groups was almost identical in conditions known to produce the fusion illusion. This surprising difference between sound-induced fission and fusion in older adults suggests dissociable age-related effects in multisensory integration, consistent with the idea that these illusions are mediated by distinct neural mechanisms.

What can volumes reveal about human brain evolution? A framework for bridging behavioral, histometric, and volumetric perspectives

An overall relationship between brain size and cognitive ability exists across primates. Can more specific information about neural function be gleaned from cortical area volumes? Numerous studies have found significant relationships between brain structures and behaviors. However, few studies have speculated about brain structure-function relationships from the microanatomical to the macroanatomical level. Here we address this problem in comparative neuroanatomy, where the functional relevance of overall brain size and the sizes of cortical regions have been poorly understood, by considering comparative psychology, with measures of visual acuity and the perception of visual illusions. We outline a model where the macroscopic size (volume or surface area) of a cortical region (such as the primary visual cortex, V1) is related to the microstructure of discrete brain regions. The hypothesis developed here is that an absolutely larger V1 can process more information with greater fidelity due to having more neurons to represent a field of space. This is the first time that the necessary comparative neuroanatomical research at the microstructural level has been brought to bear on the issue. The evidence suggests that as the size of V1 increases: the number of neurons increases, the neuron density decreases, and the density of neuronal connections increases. Thus, we describe how information about gross neuromorphology, using V1 as a model for the study of other cortical areas, may permit interpretations of cortical function.

Neural responses in parietal and occipital areas in response to visual events are modulated by prior multisensory stimuli

The effect of multi-modal vs uni-modal prior stimuli on the subsequent processing of a simple flash stimulus was studied in the context of the audio-visual 'flash-beep' illusion, in which the number of flashes a person sees is influenced by accompanying beep stimuli. EEG recordings were made while combinations of simple visual and audio-visual stimuli were presented. The experiments found that the electric field strength related to a flash stimulus was stronger when it was preceded by a multi-modal flash/beep stimulus, compared to when it was preceded by another uni-modal flash stimulus. This difference was found to be significant in two distinct timeframes--an early timeframe, from 130-160 ms, and a late timeframe, from 300-320 ms. Source localisation analysis found that the increased activity in the early interval was localised to an area centred on the inferior and superior parietal lobes, whereas the later increase was associated with stronger activity in an area centred on primary and secondary visual cortex, in the occipital lobe. The results suggest that processing of a visual stimulus can be affected by the presence of an immediately prior multisensory event. Relatively long-lasting interactions generated by the initial auditory and visual stimuli altered the processing of a subsequent visual stimulus.

Gone in a flash: manipulation of audiovisual temporal integration using transcranial magnetic stimulation

While converging evidence implicates the right inferior parietal lobule in audiovisual integration, its role has not been fully elucidated by direct manipulation of cortical activity. Replicating and extending an experiment initially reported by Kamke et al. (2012), we employed the sound-induced flash illusion, in which a single visual flash, when accompanied by two auditory tones, is misperceived as multiple flashes (Wilson, 1987; Shams et al., 2000). Slow repetitive (1 Hz) TMS administered to the right angular gyrus, but not the right supramarginal gyrus, induced a transient decrease in the Peak Perceived Flashes (PPF), reflecting reduced susceptibility to the illusion. This finding independently confirms that perturbation of networks involved in multisensory integration can result in a more veridical representation of asynchronous auditory and visual events and that cross-modal integration is an active process in which the objective is the identification of a meaningful constellation of inputs, at times at the expense of accuracy.

The duration of a co-occurring sound modulates visual detection performance in humans

Background

The duration of sounds can affect the perceived duration of co-occurring visual stimuli. However, it is unclear whether this is limited to amodal processes of duration perception or affects other non-temporal qualities of visual perception.

Methodology/Principal Findings

Here, we tested the hypothesis that visual sensitivity - rather than only the perceived duration of visual stimuli - can be affected by the duration of co-occurring sounds. We found that visual detection sensitivity (d’) for unimodal stimuli was higher for stimuli of longer duration. Crucially, in a cross-modal condition, we replicated previous unimodal findings, observing that visual sensitivity was shaped by the duration of co-occurring sounds. When short visual stimuli (∼24 ms) were accompanied by sounds of matching duration, visual sensitivity was decreased relative to the unimodal visual condition. However, when the same visual stimuli were accompanied by longer auditory stimuli (∼60–96 ms), visual sensitivity was increased relative to the performance for ∼24 ms auditory stimuli. Across participants, this sensitivity enhancement was observed within a critical time window of ∼60–96 ms. Moreover, the amplitude of this effect correlated with visual sensitivity enhancement found for longer lasting visual stimuli across participants.

Conclusions/Significance

Our findings show that the duration of co-occurring sounds affects visual perception; it changes visual sensitivity in a similar way as altering the (actual) duration of the visual stimuli does.