Auditory induced bounce perception persists as the probability of a motion reversal is reduced

The characteristics of audiovisual temporal integration in streaming-bouncing bistable motion perception: considering both implicit and explicit processing perspectives

This study explored the behavioral and neural activity characteristics of audiovisual temporal integration in motion perception from both implicit and explicit perspectives. The streaming-bouncing bistable paradigm (SB task) was employed to investigate implicit temporal integration, while the corresponding simultaneity judgment task (SJ task) was used to examine explicit temporal integration. The behavioral results revealed a negative correlation between implicit and explicit temporal processing. In the ERP results of both tasks, three neural phases (PD100, ND180, and PD290) in the fronto-central region were identified as reflecting integration effects and the auditory-evoked multisensory N1 component may serve as a primary component responsible for cross-modal temporal processing. However, there were significant differences between the VA ERPs in the SB and SJ tasks and the influence of speed on implicit and explicit integration effects also varied. The aforementioned results, building upon the validation of previous temporal renormalization theory, suggest that implicit and explicit temporal integration operate under distinct processing modes within a shared neural network. This underscores the brain's flexibility and adaptability in cross-modal temporal processing.

Perceptions of Audio-Visual Impact Events in Younger and Older Adults

Previous studies have examined whether audio-visual integration changes in older age, with some studies reporting age-related differences and others reporting no differences. Most studies have either used very basic and ambiguous stimuli (e.g., flash/beep) or highly contextualized, causally related stimuli (e.g., speech). However, few have used tasks that fall somewhere between the extremes of this continuum, such as those that include contextualized, causally related stimuli that are not speech-based; for example, audio-visual impact events. The present study used a paradigm requiring duration estimates and temporal order judgements (TOJ) of audio-visual impact events. Specifically, the Schutz-Lipscomb illusion, in which the perceived duration of a percussive tone is influenced by the length of the visual striking gesture, was examined in younger and older adults. Twenty-one younger and 21 older adult participants were presented with a visual point-light representation of a percussive impact event (i.e., a marimbist striking their instrument with a long or short gesture) combined with a percussive auditory tone. Participants completed a tone duration judgement task and a TOJ task. Five audio-visual temporal offsets (-400 to +400 ms) and five spatial offsets (from -90 to +90°) were randomly introduced. Results demonstrated that the strength of the illusion did not differ between older and younger adults and was not influenced by spatial or temporal offsets. Older adults showed an 'auditory first bias' when making TOJs. The current findings expand what is known about age-related differences in audio-visual integration by considering them in the context of impact-related events.

Temporal dynamics of the flash-induced bouncing effect

Two identical visual disks moving toward each other on a two‐dimensional (2D) display are more likely to be perceived as “streaming through” than “bouncing off” each other after their coincidence. However, either a brief auditory tone or visual flash presented at the coincident moment of the disks can strikingly increase the incidence of the bouncing percept. Despite the neural substrates underlying the sound‐induced bouncing effect have been widely investigated, little is known about the neural mechanisms underlying the flash‐induced bouncing effect. The present study used event‐related potential recordings to explore the temporal dynamics of the flash‐induced bouncing effect. The results showed that the amplitude of the postcoincidence parietooccipital P2 component (190–230 ms after coincidence) elicited by the visual motion was significantly smaller on bouncing relative to streaming trials only when the flash was presented but not when absent. In addition, the parietal P3 component (330–430 ms) was found to be larger on bouncing than streaming trials when the flash was presented, but the opposite was true when no flash was presented. These electrophysiological findings suggest that the flash‐induced bouncing effect may occur at both perceptual and postperceptual stages of processing.

Multiple phases of cross-sensory interactions associated with the audiovisual bounce-inducing effect

Using event-related potential (ERP) recordings, the present study investigated the cross-modal neural activities underlying the audiovisual bounce-inducing effect (ABE) via a novel experimental design wherein the audiovisual bouncing trials were induced solely by the ABE. The within-subject (percept-based) analysis showed that early cross-modal interactions within 100-200 ms after sound onset over fronto-central and occipital regions were associated with the occurrence of the ABE, but the cross-modal interaction at a later latency (ND250, 220-280 ms) over fronto-central region did not differ between ABE trials and non-ABE trials. The between-subject analysis indicated that the cross-modal interaction revealed by ND250 was larger for subjects who perceived the ABE more frequently. These findings suggest that the ABE is generated as a consequence of the rapid interplay between the variations of early cross-modal interactions and the general multisensory binding predisposition at an individual level.

The Effect of Transient Location on the Resolution of Bistable Visual and Audiovisual Motion Sequences

We examined the attention and inference accounts of audiovisual perception using the stream/bounce display, a visual stimulus wherein two identical objects move toward each other, completely superimpose, then move apart. This display has two candidate percepts: stream past each other or bounce off each other. Presented without additional visual or auditory transients, the motion sequence tends to yield the streaming percept, but when coupled with a tone or flash at the point of coincidence, the response bias flips toward bouncing. We explored two competing accounts of this effect: the attentional hypothesis and the inference hypothesis. Participants watched a series of motion sequences where a transient, when present, occurred at the moment of coincidence either colocalised with the motion sequence (congruent presentation) or on the opposite side of the display (incongruent presentation). Assuming the spotlight or zoom lens metaphor, an attentional account predicts that incongruent presentations should be associated with a higher percentage of bouncing responses than congruent presentations, while the inferential account predicts the opposite effect. No effect was found for tone-only trials. However, in trials containing a visual transient, results showed higher proportions of bounce responses within congruent over incongruent presentations, favouring the inference hypothesis over a spotlight or zoom lens attentional account.

Early cross-modal interactions underlie the audiovisual bounce-inducing effect

Two identical visual disks moving towards one another on a two-dimensional display can be perceived as either "streaming through" or "bouncing off" each other after their coincidence/overlapping. A brief sound presented at the moment of the coincidence of the disks could strikingly bias the percept towards bouncing, which was termed the audiovisual bounce-inducing effect (ABE). Although the ABE has been studied intensively since its discovery, the debate about its origin is still unresolved so far. The present study used event-related potential (ERP) recordings to investigate whether or not early neural activities associated with cross-modal interactions play a role on the ABE. The results showed that the fronto-central P2 component ∼200 ms before the coincidence of the disks was predictive of subsequent streaming or bouncing percept in the unimodal visual display but not in auditory-visual display. More importantly, the cross-modal interactions revealed by the fronto-central positivity PD170 (125-175 ms after sound onset), as well as the occipital positivity PD190 (180-200 ms), were substantially enhanced on bouncing trials compared to streaming trials in the auditory-visual display. These findings provide direct electrophysiological evidence that early cross-modal interactions contribute to the origin of ABE phenomenon at the perceptual stage of processing.

Tactile stimulation disambiguates the perception of visual motion paths

Although visual perception traditionally has been considered to be impenetrable by non-visual information, there are a rising number of reports discussing cross-modal influences on visual perception. In two experiments, we investigated how coinciding vibrotactile stimulation affects the perception of two discs that move toward each other, superimpose in the center of the screen, and then move apart. Whereas two discs streaming past each other was the dominant impression when the visual event was presented in isolation, a brief coinciding vibrotactile stimulation at the moment of overlap biased the visual impression toward two discs bouncing off each other (Experiment 1). Further, the vibrotactile stimulation actually changed perceptual processing by reducing the amount of perceived overlap between the discs (Experiment 2), which has been demonstrated to be associated with a higher proportion of bouncing impressions. We propose that tactile-induced quantitative changes in the visual percept might alter the quality of the visual percept (from streaming to bouncing), thereby adding to the understanding of how cross-modal information interacts with early visual perception and how this interaction influences subsequent visual impressions.

Disambiguating the Stream/Bounce Illusion With Inference

The ‘stream/bounce’ illusion refers to the perception of an ambiguous visual display in which two discs approach each other on a collision course. The display can be seen as two discs streaming through each other or bouncing off each other. Which perception dominates, may be influenced by a brief transient, usually a sound, presented around the time of simulated contact. Several theories have been proposed to account for the switching in dominance based on sensory processing, attention and cognitive inference, but a universally applicable, parsimonious explanation has not emerged. We hypothesized that only cognitive inference would be influenced by the perceptual history of the display. We rendered the display technically unambiguous by vertically offsetting the targets’ trajectories and manipulated their history by allowing the objects to switch from one trajectory to the other up to four times before the potential collision point. As the number of switches increased, the number of ‘bounce’ responses also increased. These observations show that expectancy is a critical factor in determining whether a bounce or streaming is perceived and may form the basis for a universal explanation of instances of the stream/bounce illusion.

Auditory-induced bouncing is a perceptual (rather than a cognitive) phenomenon: Evidence from illusory crescents

A central task for vision is to identify objects as the same persisting individuals over time and motion. The need for such processing is made especially clear in ambiguous situations such as the bouncing/streaming display: two discs move toward each other, superimpose, and then continue along their trajectories. Did the discs stream past each other, or bounce off each other? When people are likely to perceive streaming, playing a brief tone at the moment of overlap can readily cause them to see bouncing instead. Recent research has attributed this effect to decisional (rather than perceptual) processes by showing that auditory tones alter response biases but not the underlying sensitivity for detecting objective bounces. Here we explore the nature of this phenomenon using 'illusory causal crescents': when people perceive bouncing (or causal 'launching'), they also perceive the second disc to begin moving before being fully overlapped with the first disc (i.e. leaving an uncovered crescent). Here we demonstrate that merely playing a sound coincident to the moment of overlap can also reliably induce the perception of such illusory crescents. Moreover, this effect is due to the coincidence of the tone, per se, since the effect disappears when the tone is embedded in a larger regular tone sequence. Because observers never have to explicitly categorize their percept (e.g. as streaming)-and because the effect involves a subtle quantitative influence on another clearly visual property (i.e. the crescent's width)-we conclude that this audiovisual influence on the perception of identity over time reflects perceptual processing rather than higher-level decisions.

Pre-coincidence brain activity predicts the perceptual outcome of streaming/bouncing motion display

When two identical visual discs move toward each other on a two-dimensional visual display, they can be perceived as either "streaming through" or "bouncing off" each other after their coincidence. Previous studies have observed a strong bias toward the streaming percept. Additionally, the incidence of the bouncing percept in this ambiguous display could be increased by various factors, such as a brief sound at the moment of coincidence and a momentary pause of the two discs. The streaming/bouncing bistable motion phenomenon has been studied intensively since its discovery. However, little is known regarding the neural basis underling the perceptual ambiguity in the classic version of the streaming/bouncing motion display. The present study investigated the neural basis of the perception disambiguating underling the processing of the streaming/bouncing bistable motion display using event-related potential (ERP) recordings. Surprisingly, the amplitude of frontal central P2 (220-260 ms) that was elicited by the moving discs ~200 ms before the coincidence of the two discs was observed to be predictive of subsequent streaming or bouncing percept. A larger P2 amplitude was observed for streaming percept than the bouncing percept. These findings suggest that the streaming/bouncing bistable perception may have been disambiguated unconsciously ~200 ms before the coincidence of the two discs.

Sensitivity and Bias in the Resolution of Stream-Bounce Stimuli

The audiovisual stream-bounce effect refers to the resolution of ambiguous motion sequences as streaming or bouncing depending on the presence or absence of a sound. We used a novel experimental design and signal detection theory (SDT) to determine its sensory or decisional origins. To account for issues raised by Witt et al. on the interpretation of SDT results, we devised a pure signal detection (as opposed to signal discrimination) paradigm and measured participants' sensitivity and criterion when detecting a weak tone concurrent with objectively streaming or bouncing visual displays. We observed no change in sensitivity but a significant change in criterion with participants' criterion more liberal with bouncing targets than for streaming targets with. In a second experiment, we tasked participants with detecting a weak tone in noise while viewing an ambiguous motion sequence. They also indicated whether the targets appeared to stream or bounce. Participants' reported equivalent, mostly bouncing responses for hit and false alarm trials, and equivalent, mostly streaming responses for correct rejection and miss trials. Further, differences in participants' sensitivity and criterion measures for detecting tones in subjectively streaming compared to subjectively bouncing targets were inconsistent with sensory factors. These results support a decisional account of the sound-induced switch from mostly streaming to mostly bouncing responses in audiovisual stream-bounce displays.

Aging and audio-visual and multi-cue integration in motion

The perception of naturalistic events relies on the ability to integrate information from multiple sensory systems, an ability that may change with healthy aging. When two objects move toward and then past one another, their trajectories are perceptually ambiguous: the objects may seem to stream past one another or bounce off one another. Previous research showed that auditory or visual events that occur at the time of disks' coincidence could bias the percept toward bouncing or streaming. We exploited this malleable percept to assay age-related changes in the integration of multiple inter- and intra-modal cues. The disks' relative luminances were manipulated to produce stimuli strongly favoring either bouncing or streaming, or to produce ambiguous motion (equal luminances). A sharp sound coincident with the disks' overlap increased both groups' perception of bouncing, but did so significantly less for older subjects. An occluder's impact on motion perception varied with its duration: a long duration occluder promoted streaming in both groups; a brief occluder promoted bouncing in younger subjects, but not older ones. Control experiments demonstrated that the observed differences between younger and older subjects resulted from neither age-related changes in retinal illuminance nor age-related changes in hearing, pointing to weakened inter- and intra-modal integration with aging. These changes could contribute to previously demonstrated age-related perceptual and memory deficits.

The origin of the audiovisual bounce inducing effect: a TMS study

The audiovisual bounce inducing effect (ABE) is a bouncing percept induced by the presence of a sound at the moment of two moving objects intercepting in a motion display otherwise perceived as streaming. The origin of the ABE is still debated: the effect could arise from the subtraction of attentional resources caused by the sound (needed to favor the perception of streaming), and/or from the cross-modal integration (binding) of visual and auditory information: indeed bouncing-like sounds are best in inducing the ABE. The neural mechanism responsible for the ABE is still unknown. Here, by using offline TMS, we investigated the role of the posterior parietal cortex (PPC), thought to be involved in both attentional and binding processes, in the generation of the ABE. Results show that disrupting the functional integrity of the right (but not the left) PPC has the effect of weakening the binding of cross-modal information, which reduces the magnitude of the ABE. Indeed, if the effect of parietal stimulation was merely to disrupt attention, we would expect an increase (not a decrease) of bouncing percepts. The present study not only shows the involvement of the right PPC in the ABE, but also support the notion that cross-modal binding (and not attention) is at the origin of the ABE.

The Stream/Bounce Effect Occurs for Luminance- and Disparity-Defined Motion Targets

We generalised the stream/bounce effect to dynamic random element displays containing luminance- or disparity-defined targets. Previous studies investigating audio-visual interactions in this context have exclusively employed motion sequences with luminance-defined disks or squares and have focused on properties of the accompanying brief stimuli rather than the visual properties of the motion targets. We found that the presence of a brief sound temporally close to coincidence, or a visual flash at coincidence significantly promote bounce perception for motion targets defined by either luminance contrast or disparity contrast. A brief tone significantly promoted bouncing of luminance-defined targets above a no-sound baseline when it was presented at least 250 ms before coincidence and 100 ms after coincidence. A similar pattern was observed for disparity-defined targets, though the tone promoted bouncing above the no-sound baseline when presented at least 350 ms before and 300 ms after coincidence. We further explored the temporal properties of audio-visual interactions for these two display types and found that bounce perception saturated at similar durations after coincidence. The stream/bounce illusion manifests itself in dynamic random-element displays and is similar for luminance- and disparity-defined motion targets.