Visual sensitivity is a stronger determinant of illusory processes than auditory cue parameters in the sound-induced flash illusion

Visual adaptation changes the susceptibility to the fission illusion

Sound-induced flash illusion (SiFI) is the illusion that participants perceive incorrectly that the number of visual flashes is equal to the number of auditory beeps when presented within 100 ms. Although previous studies found that repetition suppression can reduce an individual's perceptual sensitivity to the SiFI, there is not yet a consensus as to how visual adaptation affects the SiFI. In the present study, we added prolonged adapting visual stimuli prior to the presentation of audiovisual stimuli to investigate whether the bottom-up factor of adaptation affects the SiFI. The adapting visual stimuli consisted of one or two of the same visual stimuli that lasted for 2 minutes in succession, followed by the audiovisual stimuli. Both adaptation conditions showed SiFI effects. The accuracy of adapting double-flashes was significantly lower than that of in adapting a single flash for the fission illusion. Our analyses indicated that such a pattern could be attributed to a lower d' in adapting double-flashes than in adapting a single flash. However, the accuracy, discriminability and criterion were not significantly different between the two adaptation conditions because of the instability of the fusion illusion. Thus, the present study indicated that the reduced perceptual sensitivity based on visual adaptation could enhance the fission illusion in multisensory integration.

The temporal precision of audiovisual integration is associated with longitudinal fall incidents but not sensorimotor fall risk in older adults

Sustained multisensory integration over long inter-stimulus time delays is typically found in older adults, particularly those with a history of falls. However, the extent to which the temporal precision of audio-visual integration is associated with longitudinal fall or fall risk trajectories is unknown. A large sample of older adults (N = 2319) were grouped into longitudinal trajectories of self-reported fall incidents (i.e., decrease, stable, or increase in number) and, separately, their performance on a standard, objective measure of fall risk, Timed Up and Go (TUG; stable, moderate decline, severe decline). Multisensory integration was measured once as susceptibility to the Sound-Induced Flash Illusion (SIFI) across three stimulus onset asynchronies (SOAs): 70 ms, 150 ms and 230 ms. Older adults with an increasing fall number showed a significantly different pattern of performance on the SIFI than non-fallers, depending on age: For adults with increasing incidents of falls, those aged 53-59 years showed a much smaller difference in illusion susceptibility at 70 ms versus 150 ms than those aged 70 + years. In contrast, non-fallers showed a more comparable difference between these SOA conditions across age groups. There was no association between TUG performance trajectories and SIFI susceptibility. These findings suggests that a fall event is associated with distinct temporal patterns of multisensory integration in ageing and have implications for our understanding of the mechanisms underpinning brain health in older age.

Audiovisual illusion training improves multisensory temporal integration

When we perceive external physical stimuli from the environment, the brain must remain somewhat flexible to unaligned stimuli within a specific range, as multisensory signals are subject to different transmission and processing delays. Recent studies have shown that the width of the 'temporal binding window (TBW)' can be reduced by perceptual learning. However, to date, the vast majority of studies examining the mechanisms of perceptual learning have focused on experience-dependent effects, failing to reach a consensus on its relationship with the underlying perception influenced by audiovisual illusion. The sound-induced flash illusion (SiFI) training is a reliable function for improving perceptual sensitivity. The present study utilized the classic auditory-dominated SiFI paradigm with feedback training to investigate the effect of a 5-day SiFI training on multisensory temporal integration, as evaluated by a simultaneity judgment (SJ) task and temporal order judgment (TOJ) task. We demonstrate that audiovisual illusion training enhances multisensory temporal integration precision in the form of (i) the point of subjective simultaneity (PSS) shifts to reality (0 ms) and (ii) a narrowing TBW. The results are consistent with a Bayesian model of causal inference, suggesting that perception learning reduce the susceptibility to SiFI, whilst improving the precision of audiovisual temporal estimation.

Sound-induced flash illusions at different spatial locations were affected by personality traits

Sound-induced flash illusion (SiFI) is an auditory-dominated effect in which observers will misperceive the number of flashes due to simultaneously presented beeps, which includes fission and fusion illusions. Although several individual differences have been found in SiFI, little is known about the effect of personality traits. In the present study, we presented flashes in near space and beeps in far space (Vnear_Afar) and flashes in far space and beeps in near space (Vfar_Anear) to better approximate the real world. We collected 103 participants' Big Five questionnaire results and their SiFI task performance to investigate the difference in trait level on the SiFI in the performance of accuracy, d' and c. The results show that all five personality traits had certain effects on the SiFI to different degrees, and different personality traits played different roles in the fission illusion and fusion illusion. The high agreeableness group was more prone to the fission illusion, and the report criteria were less strict. The report criteria of the low neuroticism group were stricter for the fusion illusion. The extraversion, conscientiousness and low openness groups were more prone to the fusion illusion in the Vnear_Afar condition than in the Vfar_Anear condition. The study indicated that personality traits were important but easily overlooked factors in multisensory illusion, which might make a difference between the fission illusion and the fusion illusion.

Differences in eccentricity for sound-induced flash illusion in four visual fields

A sound-induced flash illusion (SiFI) is a multisensory illusion dominated by auditory stimuli, in which the individual perceives that the number of visual flashes is equal to the number of auditory stimuli when visual flashes are presented along with an unequal number of auditory stimuli. Although the mechanisms underlying fission and fusion illusions have been documented, there is not yet a consensus on how they vary according to the different eccentricities. In the present study, by incorporating the classic SiFI paradigm into four different eccentricities, we aimed to investigate whether the SiFI varies under the different eccentricities. The results showed that the fission illusion varied significantly across the four eccentricities, with the perifovea (7°) and peripheral (11°) illusions being greater than the fovea and parafovea (3°) illusions. In contrast, the fusion illusion did not vary significantly across the four eccentricities. Our findings revealed that SiFI was affected by different visual fields and that there were differences between the fission and the fusion illusions. Furthermore, by examining the SiFI of eccentricity across visual fields, this study also suggests that bottom-up factors affect the SiFI.

The magnitude of the sound-induced flash illusion does not increase monotonically as a function of visual stimulus eccentricity

The sound-induced flash illusion (SIFI) occurs when a rapidly presented visual stimulus is accompanied by two auditory stimuli, creating the illusory percept of two visual stimuli. While much research has focused on how the temporal proximity of the audiovisual stimuli impacts susceptibility to the illusion, comparatively less research has focused on the impact of spatial manipulations. Here, we aimed to assess whether manipulating the eccentricity of visual flash stimuli altered the properties of the temporal binding window associated with the SIFI. Twenty participants were required to report whether they perceived one or two flashes that were concurrently presented with one or two beeps. Visual stimuli were presented at one of four different retinal eccentricities (2.5, 5, 7.5, or 10 degrees below fixation) and audiovisual stimuli were separated by one of eight stimulus-onset asynchronies. In keeping with previous findings, increasing stimulus-onset asynchrony between the auditory and visual stimuli led to a marked decrease in susceptibility to the illusion allowing us to estimate the width and amplitude of the temporal binding window. However, varying the eccentricity of the visual stimulus had no effect on either the width or the peak amplitude of the temporal binding window, with a similar pattern of results observed for both the “fission” and “fusion” variants of the illusion. Thus, spatial manipulations of the audiovisual stimuli used to elicit the SIFI appear to have a weaker effect on the integration of sensory signals than temporal manipulations, a finding which has implications for neuroanatomical models of multisensory integration.

Reward reduces the fission illusion in the sound-induced flash illusion

Pairing a single visual stimulus with multiple auditory stimuli will lead to the illusory perception of multiple visual stimuli, which is known as sound-induced flash illusion (SIFI). The present study adopted the classic SIFI paradigm to investigate whether value-associated tasks could affect the SIFI. By adjusting the sequence of reward and nonreward conditions, we also examined the effect of reward history on SIFI. The results showed that the fission illusion was reduced when associated with momentary reward, demonstrating significantly higher accuracy and discriminability than the nonreward condition. However, the fusion illusion was not affected by the momentary reward, and the explanation was that the fusion illusion was not as stable as the fission illusion and disappeared across different trials and conditions. Moreover, the robustness of reward history in the present study was not as strong as previous studies have suggested, indicating that the effect of sound on the perceptual representation of visual stimuli is strong and robust to reward history. These findings demonstrated that the reward could reduce the SIFI and broaden the existing dichotomy of SIFI. New evidence for the operation of value-driven attention mechanisms is also provided, suggesting that the underlying value-driven attention operates across multiple sensory systems.

Long-term training reduces the responses to the sound-induced flash illusion

The sound-induced flash illusion (SiFI) is a robust auditory-dominated multisensory integration phenomenon that is used as a reliable indicator to assess multisensory integration. Previous studies have indicated that the SiFI effect is correlated with perceptual sensitivity. However, to date, there is no consensus regarding how it corresponds to sensitivity with long-term training. The present study adopted the classic SiFI paradigm with feedback training to investigate the effect of a week of long-term training on the SiFI effect. Both the training group and control group completed a pretest and a posttest before and after the perceptual training; however, only the training group was required to complete 7-day behavioral training. The results showed that (1) long-term training could reduce the response of fission and fusion illusions by improving perceptual sensitivity and that (2) there was a "plateau effect" that emerged during the training stage, which tended to stabilize by the fifth day. These findings demonstrated that the SiFI effect could be modified with long-term training by ameliorating perceptual sensitivity, especially in terms of the fission illusion. Therefore, the present study supplements perceptual training in SiFI domains and provides evidence that the SiFI could be used as an assessment intervention to improve the efficiency of multisensory integration.

Repetition Suppression in Visual and Auditory Modalities Affects the Sound-Induced Flash Illusion

Sound-induced flash illusion (SiFI) refers to the illusion that the number of visual flashes is equal to the number of auditory sounds when the visual flashes are accompanied by an unequal number of auditory sounds presented within 100 ms. The effect of repetition suppression (RS), an adaptive effect caused by stimulus repetition, upon the SiFI has not been investigated. Based on the classic SiFI paradigm, the present study investigated whether RS would affect the SiFI differently by adding preceding stimuli in visual and auditory modalities prior to the appearance of audiovisual stimuli. The results showed the auditory RS effect on the SiFI varied with the number of preceding auditory stimuli. The hit rate was higher with two preceding auditory stimuli than one preceding auditory stimulus in fission illusion, but it did not affect the size of the fusion illusion. However, the visual RS had no effect on the size of the fission and fusion illusions. The present study suggested that RS could affect the SiFI, indicating that the RS effect in different modalities would differentially affect the magnitude of the SiFI. In the process of multisensory integration, the visual and auditory modalities had asymmetrical RS effects.

Perceptual Inference, Learning, and Attention in a Multisensory World

Adaptive behavior in a complex, dynamic, and multisensory world poses some of the most fundamental computational challenges for the brain, notably inference, decision-making, learning, binding, and attention. We first discuss how the brain integrates sensory signals from the same source to support perceptual inference and decision-making by weighting them according to their momentary sensory uncertainties. We then show how observers solve the binding or causal inference problem-deciding whether signals come from common causes and should hence be integrated or else be treated independently. Next, we describe the multifarious interplay between multisensory processing and attention. We argue that attentional mechanisms are crucial to compute approximate solutions to the binding problem in naturalistic environments when complex time-varying signals arise from myriad causes. Finally, we review how the brain dynamically adapts multisensory processing to a changing world across multiple timescales.

Incongruent Audiovisual Inducer Information and Fission/Fusion Illusions

In research studies on how people perceive simultaneously presented audiovisual information, researchers have often shown that the number of visual flashes participants perceive on a computer screen can be altered by varying the number of accompanying auditory, visual, or combined audiovisual cues or inducers. In the present study, we examined the effects of number-incongruent audiovisual inducer stimuli on the participants' perceived number of target flashes. We instructed 16 participants (eight males and eight females; M_age = 21.56; SD_age = 1.93) to report their perceived number of target flashes while ignoring the visual and auditory inducers. Across 18 different experimental conditions, we presented one or two target flashes in association with varied numbers (0, 1, 2) of auditory and visual inducer stimuli. In the condition with one target flash paired with one visual and two auditory inducers, the number of visual inducers (i.e., one) had a greater influence on the number of perceived target flashes than did the number of auditory inducers (i.e., two). Under all other number incongruent audiovisual inducer conditions, the participants' perceived number of target flashes was influenced more by the number of auditory than the number of visual inducers. We discuss these findings in the context of perceptual grouping and perceptual temporal uncertainty.

The Dynamic Double Flash Illusion: Auditory Triggered Replay of Illusory Visual Expansion

In the original double flash illusion, a visual flash (e.g., a sharp-edged disk, or uniformly filled circle) presented with two short auditory tones (beeps) is often followed by an illusory flash. The illusory flash has been previously shown to be triggered by the second auditory beep. The current study extends the double flash illusion by showing that this paradigm can not only create the illusory repeat of an on-off flash, but also trigger an illusory expansion (and in some cases a subsequent contraction) that is induced by the flash of a circular brightness gradient (gradient disk) to replay as well. The perception of the dynamic double flash illusion further supports the interpretation of the illusory flash (in the double flash illusion) as similar in its spatial and temporal properties to the perception of the real visual flash, likely by replicating the neural processes underlying the illusory expansion of the real flash. We show further that if a gradient disk (generating an illusory expansion) and a sharp-edged disk are presented simultaneously side by side with two sequential beeps, often only one visual stimulus or the other will be perceived to double flash. This indicates selectivity in auditory-visual binding, suggesting the usefulness of this paradigm as a psychophysical tool for investigating crossmodal binding phenomena.

Effects of Repetition Suppression on Sound Induced Flash Illusion With Aging

The sound-induced flash illusion (SiFI) is a classical auditory-dominated multisensory integration phenomenon in which the observer misperceives the number of visual flashes due to the simultaneous presentation of a different number of auditory beeps. Although the SiFI has been documented to correlate with perceptual sensitivity, to date there is no consensus as to how it corresponds to sensitivity with aging. The present study was based on the SiFI paradigm (Shams et al., 2000), adding repeated auditory stimuli prior to the appearance of audiovisual stimuli to investigate the effects of repetition suppression (RS) on the SiFI with aging. The repeated auditory stimuli consisted of one or two of the same auditory stimuli presented twice in succession, which were then followed by the audiovisual stimuli. By comparing the illusions in old and young adults, we aimed to explore the influence of aging on the RS of auditory stimuli on the SiFI. The results showed that both age groups showed SiFI effects, however, the RS performance of the two age groups had different effects on the fusion and fission illusions. The illusion effect in old adults was weaker than in young adults. Specifically, RS only affected fission illusions in the old adults but both fission and fusion illusions in young adults. Thus, the present study indicated that the decreased perceptual sensitivity based on auditory RS could weaken the SiFI effect in multisensory integration and that old adults are more susceptible to RS, showing that old adults perceived the SiFI effect weakly under auditory RS.

Age-related sensory decline mediates the Sound-Induced Flash Illusion: Evidence for reliability weighting models of multisensory perception

Perception of our world is proposed to arise from combining multiple sensory inputs according to their relative reliability. We tested multisensory processes in a large sample of 2920 older adults to assess whether sensory ability mediates age-related changes in perception. Participants completed a test of audio-visual integration, the Sound Induced Flash Illusion (SIFI), alongside measures of visual (acuity, contrast sensitivity, self-reported vision and visual temporal discrimination (VTD)) and auditory (self-reported hearing and auditory temporal discrimination (ATD)) function. Structural equation modelling showed that SIFI susceptibility increased with age. This was mediated by visual acuity and self-reported hearing: better scores on these measures predicted reduced and stronger SIFI susceptibility, respectively. Unexpectedly, VTD improved with age and predicted increased SIFI susceptibility. Importantly, the relationship between age and SIFI susceptibility remained significant, even when considering mediators. A second model showed that, with age, visual 'gain' (the benefit of congruent auditory information on visual judgements) was predicted by ATD: better ATD predicted stronger visual gain. However, neither age nor SIFI susceptibility were directly associated with visual gain. Our findings illustrate, in the largest sample of older adults to date, how multisensory perception is influenced, but not fully accounted for, by age-related changes in unisensory abilities.

Effects of Cognitive Expectation on Sound-Induced Flash Illusion

Sound-induced flash illusion (SIFI) is an auditory-dominated multisensory integration phenomenon in which flashes presented in conjunction with an unequal number of auditory sounds are illusorily perceived as equal in number to the auditory sounds. Previous studies on the factors that impact SIFI have mainly focused on top-down and bottom-up factors. This study aimed to explore the effects of top-down cognitive expectations on the SIFI by manipulating the proportion of trial types. The results showed that the accuracy of judgment was improved and reaction times were shortened when the instructions were consistent with the actual proportion of trial type. When the instructions were not consistent with the actual proportion of trial types, the instructions could still regulate the accuracy and reaction times in judging the fission illusion (i.e., a brief flash accompanied by two auditory stimuli tends to be perceived as two flashes) regardless of the actual proportion of trial types. The results indicated that top-down cognitive expectations could significantly reduce the fission illusion and accelerate the judgment, but the effect was not significant in the fusion illusion (i.e., two brief flashes accompanied by single auditory stimuli tend to be perceived as a single flash) due to the instability of the illusion.

Re-weighting of Sound Localization Cues by Audiovisual Training

Sound localization requires the integration in the brain of auditory spatial cues generated by interactions with the external ears, head and body. Perceptual learning studies have shown that the relative weighting of these cues can change in a context-dependent fashion if their relative reliability is altered. One factor that may influence this process is vision, which tends to dominate localization judgments when both modalities are present and induces a recalibration of auditory space if they become misaligned. It is not known, however, whether vision can alter the weighting of individual auditory localization cues. Using virtual acoustic space stimuli, we measured changes in subjects’ sound localization biases and binaural localization cue weights after ∼50 min of training on audiovisual tasks in which visual stimuli were either informative or not about the location of broadband sounds. Four different spatial configurations were used in which we varied the relative reliability of the binaural cues: interaural time differences (ITDs) and frequency-dependent interaural level differences (ILDs). In most subjects and experiments, ILDs were weighted more highly than ITDs before training. When visual cues were spatially uninformative, some subjects showed a reduction in auditory localization bias and the relative weighting of ILDs increased after training with congruent binaural cues. ILDs were also upweighted if they were paired with spatially-congruent visual cues, and the largest group-level improvements in sound localization accuracy occurred when both binaural cues were matched to visual stimuli. These data suggest that binaural cue reweighting reflects baseline differences in the relative weights of ILDs and ITDs, but is also shaped by the availability of congruent visual stimuli. Training subjects with consistently misaligned binaural and visual cues produced the ventriloquism aftereffect, i.e., a corresponding shift in auditory localization bias, without affecting the inter-subject variability in sound localization judgments or their binaural cue weights. Our results show that the relative weighting of different auditory localization cues can be changed by training in ways that depend on their reliability as well as the availability of visual spatial information, with the largest improvements in sound localization likely to result from training with fully congruent audiovisual information.

Audiovisual Temporal Perception in Aging: The Role of Multisensory Integration and Age-Related Sensory Loss

Within each sensory modality, age-related deficits in temporal perception contribute to the difficulties older adults experience when performing everyday tasks. Since perceptual experience is inherently multisensory, older adults also face the added challenge of appropriately integrating or segregating the auditory and visual cues present in our dynamic environment into coherent representations of distinct objects. As such, many studies have investigated how older adults perform when integrating temporal information across audition and vision. This review covers both direct judgments about temporal information (the sound-induced flash illusion, temporal order, perceived synchrony, and temporal rate discrimination) and judgments regarding stimuli containing temporal information (the audiovisual bounce effect and speech perception). Although an age-related increase in integration has been demonstrated on a variety of tasks, research specifically investigating the ability of older adults to integrate temporal auditory and visual cues has produced disparate results. In this short review, we explore what factors could underlie these divergent findings. We conclude that both task-specific differences and age-related sensory loss play a role in the reported disparity in age-related effects on the integration of auditory and visual temporal information.

Does hearing aid use affect audiovisual integration in mild hearing impairment?

There is converging evidence for altered audiovisual integration abilities in hearing-impaired individuals and those with profound hearing loss who are provided with cochlear implants, compared to normal-hearing adults. Still, little is known on the effects of hearing aid use on audiovisual integration in mild hearing loss, although this constitutes one of the most prevalent conditions in the elderly and, yet, often remains untreated in its early stages. This study investigated differences in the strength of audiovisual integration between elderly hearing aid users and those with the same degree of mild hearing loss who were not using hearing aids, the non-users, by measuring their susceptibility to the sound-induced flash illusion. We also explored the corresponding window of integration by varying the stimulus onset asynchronies. To examine general group differences that are not attributable to specific hearing aid settings but rather reflect overall changes associated with habitual hearing aid use, the group of hearing aid users was tested unaided while individually controlling for audibility. We found greater audiovisual integration together with a wider window of integration in hearing aid users compared to their age-matched untreated peers. Signal detection analyses indicate that a change in perceptual sensitivity as well as in bias may underlie the observed effects. Our results and comparisons with other studies in normal-hearing older adults suggest that both mild hearing impairment and hearing aid use seem to affect audiovisual integration, possibly in the sense that hearing aid use may reverse the effects of hearing loss on audiovisual integration. We suggest that these findings may be particularly important for auditory rehabilitation and call for a longitudinal study.

Grouping by feature of cross-modal flankers in temporal ventriloquism

Signals in one sensory modality can influence perception of another, for example the bias of visual timing by audition: temporal ventriloquism. Strong accounts of temporal ventriloquism hold that the sensory representation of visual signal timing changes to that of the nearby sound. Alternatively, underlying sensory representations do not change. Rather, perceptual grouping processes based on spatial, temporal, and featural information produce best-estimates of global event properties. In support of this interpretation, when feature-based perceptual grouping conflicts with temporal information-based in scenarios that reveal temporal ventriloquism, the effect is abolished. However, previous demonstrations of this disruption used long-range visual apparent-motion stimuli. We investigated whether similar manipulations of feature grouping could also disrupt the classical temporal ventriloquism demonstration, which occurs over a short temporal range. We estimated the precision of participants' reports of which of two visual bars occurred first. The bars were accompanied by different cross-modal signals that onset synchronously or asynchronously with each bar. Participants' performance improved with asynchronous presentation relative to synchronous - temporal ventriloquism - however, unlike the long-range apparent motion paradigm, this was unaffected by different combinations of cross-modal feature, suggesting that featural similarity of cross-modal signals may not modulate cross-modal temporal influences in short time scales.

Defining Auditory-Visual Objects: Behavioral Tests and Physiological Mechanisms

Crossmodal integration is a term applicable to many phenomena in which one sensory modality influences task performance or perception in another sensory modality. We distinguish the term binding as one that should be reserved specifically for the process that underpins perceptual object formation. To unambiguously differentiate binding form other types of integration, behavioral and neural studies must investigate perception of a feature orthogonal to the features that link the auditory and visual stimuli. We argue that supporting true perceptual binding (as opposed to other processes such as decision-making) is one role for cross-sensory influences in early sensory cortex. These early multisensory interactions may therefore form a physiological substrate for the bottom-up grouping of auditory and visual stimuli into auditory-visual (AV) objects.

Crossmodal integration and binding have been treated as synonymous in the literature, with no clear delineation between perceptual changes and other interactions such as decision-making.

Crossmodal binding is proposed as a distinct form of integration leading to multisensory object formation.

Multisensory stimuli are most beneficial in noisy situations, but few studies use stimulus competition to investigate the processes underpinning multisensory integration.

Evidence suggests that both visual and auditory attention is object-based – all features within an object are enhanced and there is a cost to attending features across versus within objects.

Multisensory interactions can be observed throughout the brain, including early sensory cortex.

The role of early sensory cortex in multisensory integration is unknown, but may underlie crossmodal binding.