Attentional modulation in perception of visual motion events

Sound-modulations of visual motion perception implicate the cortico-vestibular brain

A widely used example of the intricate (yet poorly understood) intertwining of multisensory signals in the brain is the audiovisual bounce inducing effect (ABE). This effect presents two identical objects moving along the azimuth with uniform motion and towards opposite directions. The perceptual interpretation of the motion is ambiguous and is modulated if a transient (sound) is presented in coincidence with the point of overlap of the two objects' motion trajectories. This phenomenon has long been written-off to simple attentional or decision-making mechanisms, although the neurological underpinnings for the effect are not well understood. Using behavioural metrics concurrently with event-related fMRI, we show that sound-induced modulations of motion perception can be further modulated by changing motion dynamics of the visual targets. The phenomenon engages the posterior parietal cortex and the parieto-insular-vestibular cortical complex, with a close correspondence of activity in these regions with behaviour. These findings suggest that the insular cortex is engaged in deriving a probabilistic perceptual solution through the integration of multisensory data.

Exploring the effectiveness of auditory, visual, and audio-visual sensory cues in a multiple object tracking environment

Maintaining object correspondence among multiple moving objects is an essential task of the perceptual system in many everyday life activities. A substantial body of research has confirmed that observers are able to track multiple target objects amongst identical distractors based only on their spatiotemporal information. However, naturalistic tasks typically involve the integration of information from more than one modality, and there is limited research investigating whether auditory and audio-visual cues improve tracking. In two experiments, we asked participants to track either five target objects or three versus five target objects amongst similarly indistinguishable distractor objects for 14 s. During the tracking interval, the target objects bounced occasionally against the boundary of a centralised orange circle. A visual cue, an auditory cue, neither or both coincided with these collisions. Following the motion interval, the participants were asked to indicate all target objects. Across both experiments and both set sizes, our results indicated that visual and auditory cues increased tracking accuracy although visual cues were more effective than auditory cues. Audio-visual cues, however, did not increase tracking performance beyond the level of purely visual cues for both high and low load conditions. We discuss the theoretical implications of our findings for multiple object tracking as well as for the principles of multisensory integration.

The multisensory cocktail party problem in adults: Perceptual segregation of talking faces on the basis of audiovisual temporal synchrony

Social interactions often involve a cluttered multisensory scene consisting of multiple talking faces. We investigated whether audiovisual temporal synchrony can facilitate perceptual segregation of talking faces. Participants either saw four identical or four different talking faces producing temporally jittered versions of the same visible speech utterance and heard the audible version of the same speech utterance. The audible utterance was either synchronized with the visible utterance produced by one of the talking faces or not synchronized with any of them. Eye tracking indicated that participants exhibited a marked preference for the synchronized talking face, that they gazed more at the mouth than the eyes overall, that they gazed more at the eyes of an audiovisually synchronized than a desynchronized talking face, and that they gazed more at the mouth when all talking faces were audiovisually desynchronized. These findings demonstrate that audiovisual temporal synchrony plays a major role in perceptual segregation of multisensory clutter and that adults rely on differential scanning strategies of a talker's eyes and mouth to discover sources of multisensory coherence.

Temporal dynamics of the flash-induced bouncing effect

Two identical visual disks moving toward each other on a two‐dimensional (2D) display are more likely to be perceived as “streaming through” than “bouncing off” each other after their coincidence. However, either a brief auditory tone or visual flash presented at the coincident moment of the disks can strikingly increase the incidence of the bouncing percept. Despite the neural substrates underlying the sound‐induced bouncing effect have been widely investigated, little is known about the neural mechanisms underlying the flash‐induced bouncing effect. The present study used event‐related potential recordings to explore the temporal dynamics of the flash‐induced bouncing effect. The results showed that the amplitude of the postcoincidence parietooccipital P2 component (190–230 ms after coincidence) elicited by the visual motion was significantly smaller on bouncing relative to streaming trials only when the flash was presented but not when absent. In addition, the parietal P3 component (330–430 ms) was found to be larger on bouncing than streaming trials when the flash was presented, but the opposite was true when no flash was presented. These electrophysiological findings suggest that the flash‐induced bouncing effect may occur at both perceptual and postperceptual stages of processing.

Language-Based Modulation of the Stream/Bounce Judgment

When two identical objects on a screen move toward each other, coincide at the center of the screen, and then continue to move along their original trajectories to the opposite starting points, observers perceive these visual stimuli as showing one of the two possible scenarios: streaming through or bouncing off each other (stream/bounce perception). Previous research has shown that when a high-arousal face is presented along with the two moving objects, the bouncing percept was predominant, as compared with when a middle- or low-arousal face is presented. In this study, however, such a modulatory effect of the emotional face was eliminated when participants did not judge stream or bounce and the terms "bouncing/streaming" were not used in the experiments. These results suggest that the modulatory effect of emotional stimuli on the stream/bounce judgment cannot be explained solely by the emotional processing per se but, rather, can be modulated by language-based processing.

Multiple phases of cross-sensory interactions associated with the audiovisual bounce-inducing effect

Using event-related potential (ERP) recordings, the present study investigated the cross-modal neural activities underlying the audiovisual bounce-inducing effect (ABE) via a novel experimental design wherein the audiovisual bouncing trials were induced solely by the ABE. The within-subject (percept-based) analysis showed that early cross-modal interactions within 100-200 ms after sound onset over fronto-central and occipital regions were associated with the occurrence of the ABE, but the cross-modal interaction at a later latency (ND250, 220-280 ms) over fronto-central region did not differ between ABE trials and non-ABE trials. The between-subject analysis indicated that the cross-modal interaction revealed by ND250 was larger for subjects who perceived the ABE more frequently. These findings suggest that the ABE is generated as a consequence of the rapid interplay between the variations of early cross-modal interactions and the general multisensory binding predisposition at an individual level.

The Effect of Transient Location on the Resolution of Bistable Visual and Audiovisual Motion Sequences

We examined the attention and inference accounts of audiovisual perception using the stream/bounce display, a visual stimulus wherein two identical objects move toward each other, completely superimpose, then move apart. This display has two candidate percepts: stream past each other or bounce off each other. Presented without additional visual or auditory transients, the motion sequence tends to yield the streaming percept, but when coupled with a tone or flash at the point of coincidence, the response bias flips toward bouncing. We explored two competing accounts of this effect: the attentional hypothesis and the inference hypothesis. Participants watched a series of motion sequences where a transient, when present, occurred at the moment of coincidence either colocalised with the motion sequence (congruent presentation) or on the opposite side of the display (incongruent presentation). Assuming the spotlight or zoom lens metaphor, an attentional account predicts that incongruent presentations should be associated with a higher percentage of bouncing responses than congruent presentations, while the inferential account predicts the opposite effect. No effect was found for tone-only trials. However, in trials containing a visual transient, results showed higher proportions of bounce responses within congruent over incongruent presentations, favouring the inference hypothesis over a spotlight or zoom lens attentional account.

Early cross-modal interactions underlie the audiovisual bounce-inducing effect

Two identical visual disks moving towards one another on a two-dimensional display can be perceived as either "streaming through" or "bouncing off" each other after their coincidence/overlapping. A brief sound presented at the moment of the coincidence of the disks could strikingly bias the percept towards bouncing, which was termed the audiovisual bounce-inducing effect (ABE). Although the ABE has been studied intensively since its discovery, the debate about its origin is still unresolved so far. The present study used event-related potential (ERP) recordings to investigate whether or not early neural activities associated with cross-modal interactions play a role on the ABE. The results showed that the fronto-central P2 component ∼200 ms before the coincidence of the disks was predictive of subsequent streaming or bouncing percept in the unimodal visual display but not in auditory-visual display. More importantly, the cross-modal interactions revealed by the fronto-central positivity PD170 (125-175 ms after sound onset), as well as the occipital positivity PD190 (180-200 ms), were substantially enhanced on bouncing trials compared to streaming trials in the auditory-visual display. These findings provide direct electrophysiological evidence that early cross-modal interactions contribute to the origin of ABE phenomenon at the perceptual stage of processing.

Tactile stimulation disambiguates the perception of visual motion paths

Although visual perception traditionally has been considered to be impenetrable by non-visual information, there are a rising number of reports discussing cross-modal influences on visual perception. In two experiments, we investigated how coinciding vibrotactile stimulation affects the perception of two discs that move toward each other, superimpose in the center of the screen, and then move apart. Whereas two discs streaming past each other was the dominant impression when the visual event was presented in isolation, a brief coinciding vibrotactile stimulation at the moment of overlap biased the visual impression toward two discs bouncing off each other (Experiment 1). Further, the vibrotactile stimulation actually changed perceptual processing by reducing the amount of perceived overlap between the discs (Experiment 2), which has been demonstrated to be associated with a higher proportion of bouncing impressions. We propose that tactile-induced quantitative changes in the visual percept might alter the quality of the visual percept (from streaming to bouncing), thereby adding to the understanding of how cross-modal information interacts with early visual perception and how this interaction influences subsequent visual impressions.

Arousing emoticons edit stream/bounce perception of objects moving past each other

When two identical objects move toward each other, overlap completely, and continue toward opposite ends of a space, observers might perceive them as streaming through or bouncing off each other. This phenomenon is known as ‘stream/bounce perception’. In this study, we investigated the effect of the presentation of emoticons on stream/bounce perception in five experiments. In Experiment 1, we used emoticons representing anger (‘(‘∧’)’), a smile (‘(^_^)’), and a sober face (‘(°_°)’, as a control), and observers were asked to judge whether two objects unrelated to the emoticon had streamed through or bounced off each other. The anger emoticon biased perception toward bouncing when compared with the smile or sober face emoticon. In Experiments 2 and 3, we controlled for the valence and arousal of emoticons, and found that arousal influenced stream/bounce perception but valence did not. Experiments 4 and 5 ruled out the possibility of attentional capture and response bias for the emoticon with higher arousal. Taken together, the findings indicate that emoticons with higher arousal evoke a mental image of a ‘collision’ in observers, thereby eliciting the bounce perception.

Disambiguating the Stream/Bounce Illusion With Inference

The ‘stream/bounce’ illusion refers to the perception of an ambiguous visual display in which two discs approach each other on a collision course. The display can be seen as two discs streaming through each other or bouncing off each other. Which perception dominates, may be influenced by a brief transient, usually a sound, presented around the time of simulated contact. Several theories have been proposed to account for the switching in dominance based on sensory processing, attention and cognitive inference, but a universally applicable, parsimonious explanation has not emerged. We hypothesized that only cognitive inference would be influenced by the perceptual history of the display. We rendered the display technically unambiguous by vertically offsetting the targets’ trajectories and manipulated their history by allowing the objects to switch from one trajectory to the other up to four times before the potential collision point. As the number of switches increased, the number of ‘bounce’ responses also increased. These observations show that expectancy is a critical factor in determining whether a bounce or streaming is perceived and may form the basis for a universal explanation of instances of the stream/bounce illusion.

The Posture of Putting One's Palms Together Modulates Visual Motion Event Perception

We investigated the effect of an observer's hand postures on visual motion perception using the stream/bounce display. When two identical visual objects move across collinear horizontal trajectories toward each other in a two-dimensional display, observers perceive them as either streaming or bouncing. In our previous study, we found that when observers put their palms together just below the coincidence point of the two objects, the percentage of bouncing responses increased, mainly depending on the proprioceptive information from their own hands. However, it remains unclear if the tactile or haptic (force) information produced by the postures mostly influences the stream/bounce perception. We solved this problem by changing the tactile and haptic information on the palms of the hands. Experiment 1 showed that the promotion of bouncing perception was observed only when the posture of directly putting one's palms together was used, while there was no effect when a brick was sandwiched between the participant's palms. Experiment 2 demonstrated that the strength of force used when putting the palms together had no effect on increasing bounce perception. Our findings indicate that the hands-induced bounce effect derives from the tactile information produced by the direct contact between both palms.

Auditory-induced bouncing is a perceptual (rather than a cognitive) phenomenon: Evidence from illusory crescents

A central task for vision is to identify objects as the same persisting individuals over time and motion. The need for such processing is made especially clear in ambiguous situations such as the bouncing/streaming display: two discs move toward each other, superimpose, and then continue along their trajectories. Did the discs stream past each other, or bounce off each other? When people are likely to perceive streaming, playing a brief tone at the moment of overlap can readily cause them to see bouncing instead. Recent research has attributed this effect to decisional (rather than perceptual) processes by showing that auditory tones alter response biases but not the underlying sensitivity for detecting objective bounces. Here we explore the nature of this phenomenon using 'illusory causal crescents': when people perceive bouncing (or causal 'launching'), they also perceive the second disc to begin moving before being fully overlapped with the first disc (i.e. leaving an uncovered crescent). Here we demonstrate that merely playing a sound coincident to the moment of overlap can also reliably induce the perception of such illusory crescents. Moreover, this effect is due to the coincidence of the tone, per se, since the effect disappears when the tone is embedded in a larger regular tone sequence. Because observers never have to explicitly categorize their percept (e.g. as streaming)-and because the effect involves a subtle quantitative influence on another clearly visual property (i.e. the crescent's width)-we conclude that this audiovisual influence on the perception of identity over time reflects perceptual processing rather than higher-level decisions.

Pre-coincidence brain activity predicts the perceptual outcome of streaming/bouncing motion display

When two identical visual discs move toward each other on a two-dimensional visual display, they can be perceived as either "streaming through" or "bouncing off" each other after their coincidence. Previous studies have observed a strong bias toward the streaming percept. Additionally, the incidence of the bouncing percept in this ambiguous display could be increased by various factors, such as a brief sound at the moment of coincidence and a momentary pause of the two discs. The streaming/bouncing bistable motion phenomenon has been studied intensively since its discovery. However, little is known regarding the neural basis underling the perceptual ambiguity in the classic version of the streaming/bouncing motion display. The present study investigated the neural basis of the perception disambiguating underling the processing of the streaming/bouncing bistable motion display using event-related potential (ERP) recordings. Surprisingly, the amplitude of frontal central P2 (220-260 ms) that was elicited by the moving discs ~200 ms before the coincidence of the two discs was observed to be predictive of subsequent streaming or bouncing percept. A larger P2 amplitude was observed for streaming percept than the bouncing percept. These findings suggest that the streaming/bouncing bistable perception may have been disambiguated unconsciously ~200 ms before the coincidence of the two discs.

Noise, multisensory integration, and previous response in perceptual disambiguation

Sensory information about the state of the world is generally ambiguous. Understanding how the nervous system resolves such ambiguities to infer the actual state of the world is a central quest for sensory neuroscience. However, the computational principles of perceptual disambiguation are still poorly understood: What drives perceptual decision-making between multiple equally valid solutions? Here we investigate how humans gather and combine sensory information–within and across modalities–to disambiguate motion perception in an ambiguous audiovisual display, where two moving stimuli could appear as either streaming through, or bouncing off each other. By combining psychophysical classification tasks with reverse correlation analyses, we identified the particular spatiotemporal stimulus patterns that elicit a stream or a bounce percept, respectively. From that, we developed and tested a computational model for uni- and multi-sensory perceptual disambiguation that tightly replicates human performance. Specifically, disambiguation relies on knowledge of prototypical bouncing events that contain characteristic patterns of motion energy in the dynamic visual display. Next, the visual information is linearly integrated with auditory cues and prior knowledge about the history of recent perceptual interpretations. What is more, we demonstrate that perceptual decision-making with ambiguous displays is systematically driven by noise, whose random patterns not only promote alternation, but also provide signal-like information that biases perception in highly predictable fashion.

Sensory information is generally ambiguous, and a single sensory modality most often cannot provide enough information to univocally specify the actual state of the world. A primary task for the brain is therefore to resolve perceptual ambiguity. Here we use a dynamic audiovisual ambiguous display embedded in noise to investigate the computational mechanisms of perceptual disambiguation. Results demonstrate that the brain first extracts visual information for perceptual disambiguation through motion detectors. Such information is next combined with auditory information–and memory of recent perceptual history–through weighted averaging to determine the final percept. This study revealed the particular spatiotemporal stimulus patterns that elicit a stream or a bounce percept, respectively, and it demonstrates that perceptual disambiguation is majorly affected by noise, whose random spatiotemporal patterns provide signal-like information that bias perception in a very systematic fashion.

Sensitivity and Bias in the Resolution of Stream-Bounce Stimuli

The audiovisual stream-bounce effect refers to the resolution of ambiguous motion sequences as streaming or bouncing depending on the presence or absence of a sound. We used a novel experimental design and signal detection theory (SDT) to determine its sensory or decisional origins. To account for issues raised by Witt et al. on the interpretation of SDT results, we devised a pure signal detection (as opposed to signal discrimination) paradigm and measured participants' sensitivity and criterion when detecting a weak tone concurrent with objectively streaming or bouncing visual displays. We observed no change in sensitivity but a significant change in criterion with participants' criterion more liberal with bouncing targets than for streaming targets with. In a second experiment, we tasked participants with detecting a weak tone in noise while viewing an ambiguous motion sequence. They also indicated whether the targets appeared to stream or bounce. Participants' reported equivalent, mostly bouncing responses for hit and false alarm trials, and equivalent, mostly streaming responses for correct rejection and miss trials. Further, differences in participants' sensitivity and criterion measures for detecting tones in subjectively streaming compared to subjectively bouncing targets were inconsistent with sensory factors. These results support a decisional account of the sound-induced switch from mostly streaming to mostly bouncing responses in audiovisual stream-bounce displays.

Low-Level Motion Characteristics Do Not Account for Perceptions of Stream-Bounce Stimuli

The stream-bounce effect refers to a bistable motion stimulus that is interpreted as two targets either “streaming” past or “bouncing” off one another, and the manipulations that bias responses. Directional bias, according to Bertenthal et al., is an account of the effect proposing that low-level motion integration promotes streaming, and its disruption leads to bouncing, and it is sometimes cited either directly in a bottom-up fashion or indirectly under top-down control despite Sekuler and Sekuler finding evidence inconsistent with it. We tested two key aspects of the hypothesis: (a) comparable changes in speed should produce comparable disruptions and lead to similar effects; and (b) speed changes alone should disrupt integration without the need for additional more complex changes of motion. We found that target motion influences stream-bounce perception, but not as directional bias predicts. Our results support Sekuler and Sekuler and argue against the low-level motion signals driving perceptual outcomes in stream-bounce displays (directly or indirectly) and point to higher level inferential processes involving perceptual history and expectation. Directional bias as a mechanism should be abandoned and either another specific bottom-up process must be proposed and tested or consideration should be given to top-down factors alone driving the effect.

Hand Positions Alter Bistable Visual Motion Perception

We found that a hand posture with the palms together located just below the stream/bounce display could increase the proportion of bouncing perception. This effect, called the hands-induced bounce (HIB) effect, did not occur in the hands-cross condition or in the one-hand condition. By using rubber hands or covering the participants’ hands with a cloth, we demonstrated that the visual information of the hand shapes was not a critical factor in producing the HIB effect, whereas proprioceptive information seemed to be important. We also found that the HIB effect did not occur when the participants’ hands were far from the coincidence point, suggesting that the HIB effect might be produced within a limited spatial area around the hands.

Multi- and Unisensory Visual Flash Illusions

The role of stimulus structure in multisensory and unisensory interactions was examined. When a flash (17 ms) was accompanied by multiple tones (each 7 ms, SOA < 100 ms) multiple flashes were reported, and this effect has been suggested to reflect the role of stimulus continuity in multisensory interactions. In experiments 1 and 2 we examined if stimulus continuity would affect concurrently presented stimuli. When a relatively longer flash (317 ms) was accompanied by multiple tones (each 7 ms), observers reported perceiving multiple flashes. In experiment 3 we tested whether a flash presented near fixation would induce an illusory flash further in the periphery. One flash (17 ms) presented 5° below fixation was reported as multiple flashes if presented with two flashes (each 17 ms, SOA = 100 ms) 2° above fixation. The extent to which these data support a phenomenological continuity principle and whether this principle applies to unisensory perception is discussed.

The effects of attention on the temporal integration of multisensory stimuli

In unisensory contexts, spatially-focused attention tends to enhance perceptual processing. How attention influences the processing of multisensory stimuli, however, has been of much debate. In some cases, attention has been shown to be important for processes related to the integration of audio-visual stimuli, but in other cases such processes have been reported to occur independently of attention. To address these conflicting results, we performed three experiments to examine how attention interacts with a key facet of multisensory processing: the temporal window of integration (TWI). The first two experiments used a novel cued-spatial-attention version of the bounce/stream illusion, wherein two moving visual stimuli with intersecting paths tend to be perceived as bouncing off rather than streaming through each other when a brief sound occurs near in time. When the task was to report whether the visual stimuli appeared to bounce or stream, attention served to narrow this measure of the TWI and bias perception toward “streaming”. When the participants’ task was to explicitly judge the simultaneity of the sound with the intersection of the moving visual stimuli, however, the results were quite different. Specifically, attention served to mainly widen the TWI, increasing the likelihood of simultaneity perception, while also substantially increasing the simultaneity judgment accuracy when the stimuli were actually physically simultaneous. Finally, in Experiment 3, where the task was to judge the simultaneity of a simple, temporally discrete, flashed visual stimulus and the same brief tone pip, attention had no effect on the measured TWI. These results highlight the flexibility of attention in enhancing multisensory perception and show that the effects of attention on multisensory processing are highly dependent on the task demands and observer goals.

Feature- and Face-Exchange illusions: new insights and applications for the study of the binding problem

The binding problem is a longstanding issue in vision science: i.e., how are humans able to maintain a relatively stable representation of objects and features even though the visual system processes many aspects of the world separately and in parallel? We previously investigated this issue with a variant of the bounce-pass paradigm, which consists of two rectangular bars moving in opposite directions; if the bars are identical and never overlap, the motion could equally be interpreted as bouncing or passing. Although bars of different colors should be seen as passing each other (since the colors provide more information about the bars' paths), we found “Feature Exchange”: observers reported the paradoxical perception that the bars appear to bounce off of each other and exchange colors. Here we extend our previous findings with three demonstrations. “Peripheral Feature-Exchange” consists of two colored bars that physically bounce (they continually meet in the middle of the monitor and return to the sides). When viewed in the periphery, the bars appear to stream past each other even though this percept relies on the exchange of features and contradicts the information provided by the color of the bars. In “Face-Exchange” two different faces physically pass each other. When fixating centrally, observers typically report the perception of bouncing faces that swap features, indicating that the Feature Exchange effect can occur even with complex objects. In “Face-Go-Round,” one face repeatedly moves from left to right on the top of the monitor, and the other from right to left at the bottom of the monitor. Observers typically perceive the faces moving in a circle—a percept that contradicts information provided by the identity of the faces. We suggest that Feature Exchange and the paradigms used to elicit it can be useful for the investigation of the binding problem as well as other contemporary issues of interest to vision science.

Global-motion aftereffect does not depend on awareness of the adapting motion direction

It has been shown that humans cannot perceive more than three directions from a multidirectional motion stimulus. However, it remains unknown whether adapting to such imperceptible motion directions could generate motion aftereffects (MAEs). A series of psychophysical experiments were conducted to address this issue. Using a display consisting of randomly oriented Gabors, we replicated previous findings that observers were unable to perceive the global directions embedded in a five-direction motion pattern. However, adapting to this multidirectional pattern induced both static and dynamic MAEs, despite the fact that observers were unaware of any global motion directions during adaptation. Furthermore, by comparing the strengths of the dynamic MAEs induced at different levels of motion processing, we found that spatial integration of local illusory signals per se was sufficient to produce a significant global MAE. These psychophysical results show that the generation of a directional global MAE does not require conscious perception of the global motion during adaptation.

Pattern dot quantity affects auditory facilitation effects on visual object representations

Auditory stimuli often facilitate visual perception. Audiovisual integration requires spatial and/or temporal proximity between visual and auditory stimuli; additionally, sensory processing speed affects the audiovisual integration process. In the present study we examined the relationship between processing speed and the auditory facilitation effect on visual representations by manipulating dot quantity patterns. We hypothesized that the auditory facilitation effect would be observed in longer interstimulus interval conditions with more dot quantities. This is because more processing time would be required to integrate visual and auditory stimuli. During a backward masking paradigm used in experiment 1, the auditory facilitation effect depended on dot quantity among patterns and the interval between visual stimuli and masks. Moreover, differences in processing time required to integrate visual and auditory stimuli between dot quantities was confirmed from a same-different discrimination task in experiment 2. Therefore, dot quantity affects sensory processing time, and a longer processing time is required for integrating visual and auditory stimuli when visual dot quantity is high.

The spinning dancer illusion and spontaneous brain fluctuations: an fMRI study

The brain activation associated with the Spinning Dancer Illusion, a cognitive visual illusion, is not entirely known. Inferences from other study modalities point to the involvement of the dorso-parieto-occipital areas in the spontaneous switchings of perception in other bistable non-kinetic illusions. fMRI is a mature technique used to investigate the brain responses associated with mental changes. Resting-state fMRI is a novel technique that may help ascertain the effects of spontaneous brain changes in the top-down regulation of visual perception. The purpose of this report is to describe the brain activation associated with the subjective illusory changes of perception of a kinetic bistable stimulus. We hypothesize that there is a relationship between the perception phases with the very slow cortical spontaneous fluctuations, recently described. A single normal subject who was trained to produce voluntarily perception phase switches underwent a series of fMRI studies whose blocks were either defined post-hoc or accordingly with a predefined timeline to assess spontaneous and voluntarily evoked visual perception switches, respectively. Correlation of findings with resting-state fMRI and independent component analysis of the task series was sought. Phases of the rotation direction were found associated with right parietal activity. Independent component analysis of the task series and their comparison with basal resting-state components suggest that this activity is related to one of the very slow spontaneous brain fluctuations. The spontaneous fluctuations of the cortical activity may explain the subjective changes in perception of direction of the Spinning Dancer Illusion. This observation is a proof-of-principle, suggesting that the spontaneous brain oscillations may influence top-down sensory regulation.

Aging and audio-visual and multi-cue integration in motion

The perception of naturalistic events relies on the ability to integrate information from multiple sensory systems, an ability that may change with healthy aging. When two objects move toward and then past one another, their trajectories are perceptually ambiguous: the objects may seem to stream past one another or bounce off one another. Previous research showed that auditory or visual events that occur at the time of disks' coincidence could bias the percept toward bouncing or streaming. We exploited this malleable percept to assay age-related changes in the integration of multiple inter- and intra-modal cues. The disks' relative luminances were manipulated to produce stimuli strongly favoring either bouncing or streaming, or to produce ambiguous motion (equal luminances). A sharp sound coincident with the disks' overlap increased both groups' perception of bouncing, but did so significantly less for older subjects. An occluder's impact on motion perception varied with its duration: a long duration occluder promoted streaming in both groups; a brief occluder promoted bouncing in younger subjects, but not older ones. Control experiments demonstrated that the observed differences between younger and older subjects resulted from neither age-related changes in retinal illuminance nor age-related changes in hearing, pointing to weakened inter- and intra-modal integration with aging. These changes could contribute to previously demonstrated age-related perceptual and memory deficits.

Effect of flanking sounds on the auditory continuity illusion

Background

The auditory continuity illusion or the perceptual restoration of a target sound briefly interrupted by an extraneous sound has been shown to depend on masking. However, little is known about factors other than masking.

Methodology/Principal Findings

We examined whether a sequence of flanking transient sounds affects the apparent continuity of a target tone alternated with a bandpass noise at regular intervals. The flanking sounds significantly increased the limit of perceiving apparent continuity in terms of the maximum target level at a fixed noise level, irrespective of the frequency separation between the target and flanking sounds: the flanking sounds enhanced the continuity illusion. This effect was dependent on the temporal relationship between the flanking sounds and noise bursts.

Conclusions/Significance

The spectrotemporal characteristics of the enhancement effect suggest that a mechanism to compensate for exogenous attentional distraction may contribute to the continuity illusion.

An odorant congruent with a colour cue is selectively perceived in an odour mixture

Odour identification can be influenced by colour cues. This study examined the mechanism underlying this colour context effect. We hypothesised that a specific odour component congruent with a colour would be selectively perceived in preference to another odour component in a binary odour mixture. We used a ratio estimation method under two colour conditions, a binary odour mixture (experiment 1) and single chemicals presented individually (experiment 2). Each colour was congruent with one of the odour components. Participants judged the perceived mixture ratio in each odour container on which a colour patch was pasted. An influence of colour was not observed when the odour stimulus did not contain the odour component congruent with the colour (experiment 2); however, the odour component congruent with the colour was perceived as more dominant when the odour stimulus did contain the colour-congruent odorant (experiment 1). This pattern indicates that a colour-congruent odour component is selectively perceived in an odour mixture. This finding suggests that colours can enhance the perceptual representation of the colour-associated component in an odour mixture.

The origin of the audiovisual bounce inducing effect: a TMS study

The audiovisual bounce inducing effect (ABE) is a bouncing percept induced by the presence of a sound at the moment of two moving objects intercepting in a motion display otherwise perceived as streaming. The origin of the ABE is still debated: the effect could arise from the subtraction of attentional resources caused by the sound (needed to favor the perception of streaming), and/or from the cross-modal integration (binding) of visual and auditory information: indeed bouncing-like sounds are best in inducing the ABE. The neural mechanism responsible for the ABE is still unknown. Here, by using offline TMS, we investigated the role of the posterior parietal cortex (PPC), thought to be involved in both attentional and binding processes, in the generation of the ABE. Results show that disrupting the functional integrity of the right (but not the left) PPC has the effect of weakening the binding of cross-modal information, which reduces the magnitude of the ABE. Indeed, if the effect of parietal stimulation was merely to disrupt attention, we would expect an increase (not a decrease) of bouncing percepts. The present study not only shows the involvement of the right PPC in the ABE, but also support the notion that cross-modal binding (and not attention) is at the origin of the ABE.

The Stream/Bounce Effect Occurs for Luminance- and Disparity-Defined Motion Targets

We generalised the stream/bounce effect to dynamic random element displays containing luminance- or disparity-defined targets. Previous studies investigating audio-visual interactions in this context have exclusively employed motion sequences with luminance-defined disks or squares and have focused on properties of the accompanying brief stimuli rather than the visual properties of the motion targets. We found that the presence of a brief sound temporally close to coincidence, or a visual flash at coincidence significantly promote bounce perception for motion targets defined by either luminance contrast or disparity contrast. A brief tone significantly promoted bouncing of luminance-defined targets above a no-sound baseline when it was presented at least 250 ms before coincidence and 100 ms after coincidence. A similar pattern was observed for disparity-defined targets, though the tone promoted bouncing above the no-sound baseline when presented at least 350 ms before and 300 ms after coincidence. We further explored the temporal properties of audio-visual interactions for these two display types and found that bounce perception saturated at similar durations after coincidence. The stream/bounce illusion manifests itself in dynamic random-element displays and is similar for luminance- and disparity-defined motion targets.

Cognitive binding in schizophrenia: weakened integration of temporal intersensory information

Cognitive functioning is based on binding processes, by which different features and elements of neurocognition are integrated and coordinated. Binding is an essential ingredient of, for instance, Gestalt perception. We have implemented a paradigm of causality perception based on the work of Albert Michotte, in which 2 identical discs move from opposite sides of a monitor, steadily toward, and then past one another. Their coincidence generates an ambiguous percept of either "streaming" or "bouncing," which the subjects (34 schizophrenia spectrum patients and 34 controls with mean age 27.9 y) were instructed to report. The latter perception is a marker of the binding processes underlying perceived causality (type I binding). In addition to this visual task, acoustic stimuli were presented at different times during the task (150 ms before and after visual coincidence), which can modulate perceived causality. This modulation by intersensory and temporally delayed stimuli is viewed as a different type of binding (type II). We show here, using a mixed-effects hierarchical analysis, that type II binding distinguishes schizophrenia spectrum patients from healthy controls, whereas type I binding does not. Type I binding may even be excessive in some patients, especially those with positive symptoms; Type II binding, however, was generally attenuated in patients. The present findings point to ways in which the disconnection (or Gestalt) hypothesis of schizophrenia can be refined, suggesting more specific markers of neurocognitive functioning and potential targets of treatment.

Stream/bounce event perception reveals a temporal limit of motion correspondence based on surface feature over space and time

We examined how stream/bounce event perception is affected by motion correspondence based on the surface features of moving objects passing behind an occlusion. In the stream/bounce display two identical objects moving across each other in a two-dimensional display can be perceived as either streaming through or bouncing off each other at coincidence. Here, surface features such as colour (Experiments 1 and 2) or luminance (Experiment 3) were switched between the two objects at coincidence. The moment of coincidence was invisible to observers due to an occluder. Additionally, the presentation of the moving objects was manipulated in duration after the feature switch at coincidence. The results revealed that a postcoincidence duration of approximately 200 ms was required for the visual system to stabilize judgments of stream/bounce events by determining motion correspondence between the objects across the occlusion on the basis of the surface feature. The critical duration was similar across motion speeds of objects and types of surface features. Moreover, controls (Experiments 4a–4c) showed that cognitive bias based on feature (colour/luminance) congruency across the occlusion could not fully account for the effects of surface features on the stream/bounce judgments. We discuss the roles of motion correspondence, visual feature processing, and attentive tracking in the stream/bounce judgments.

The development of multisensory speech perception continues into the late childhood years

Observing a speaker's articulations substantially improves the intelligibility of spoken speech, especially under noisy listening conditions. This multisensory integration of speech inputs is crucial to effective communication. Appropriate development of this ability has major implications for children in classroom and social settings, and deficits in it have been linked to a number of neurodevelopmental disorders, especially autism. It is clear from structural imaging studies that there is a prolonged maturational course within regions of the perisylvian cortex that persists into late childhood, and these regions have been firmly established as being crucial to speech and language functions. Given this protracted maturational timeframe, we reasoned that multisensory speech processing might well show a similarly protracted developmental course. Previous work in adults has shown that audiovisual enhancement in word recognition is most apparent within a restricted range of signal-to-noise ratios (SNRs). Here, we investigated when these properties emerge during childhood by testing multisensory speech recognition abilities in typically developing children aged between 5 and 14 years, and comparing them with those of adults. By parametrically varying SNRs, we found that children benefited significantly less from observing visual articulations, displaying considerably less audiovisual enhancement. The findings suggest that improvement in the ability to recognize speech-in-noise and in audiovisual integration during speech perception continues quite late into the childhood years. The implication is that a considerable amount of multisensory learning remains to be achieved during the later schooling years, and that explicit efforts to accommodate this learning may well be warranted.

The development of multisensory speech perception continues into the late childhood years

Observing a speaker's articulations substantially improves the intelligibility of spoken speech, especially under noisy listening conditions. This multisensory integration of speech inputs is crucial to effective communication. Appropriate development of this ability has major implications for children in classroom and social settings, and deficits in it have been linked to a number of neurodevelopmental disorders, especially autism. It is clear from structural imaging studies that there is a prolonged maturational course within regions of the perisylvian cortex that persists into late childhood, and these regions have been firmly established as being crucial to speech and language functions. Given this protracted maturational timeframe, we reasoned that multisensory speech processing might well show a similarly protracted developmental course. Previous work in adults has shown that audiovisual enhancement in word recognition is most apparent within a restricted range of signal-to-noise ratios (SNRs). Here, we investigated when these properties emerge during childhood by testing multisensory speech recognition abilities in typically developing children aged between 5 and 14 years, and comparing them with those of adults. By parametrically varying SNRs, we found that children benefited significantly less from observing visual articulations, displaying considerably less audiovisual enhancement. The findings suggest that improvement in the ability to recognize speech-in-noise and in audiovisual integration during speech perception continues quite late into the childhood years. The implication is that a considerable amount of multisensory learning remains to be achieved during the later schooling years, and that explicit efforts to accommodate this learning may well be warranted.

Action and attentional load can influence aperture effects on motion perception

When a moving featureless contour is viewed through a stationary circular aperture that occludes the contour's endpoints and the contour moves in a direction non-parallel to its orientation, observers report the contour's direction of motion as perpendicular to the contour's orientation regardless of its actual direction. In typical studies of this aperture effect on motion perception, observers made perceptual judgments of the line's motion. The aperture effect was not measured when observers actively controlled the line's motion. In addition, effects of attentional load on the aperture effect were not measured. Here, we demonstrated that attentional load influenced the aperture effect. Active control reduced the aperture effect, but did not eliminate it. Results have theoretical implications for motion perception and practical implications for the design of technologies that limit an observer's field-of-view such as surgical cameras.

Voluntary action affects perception of bistable motion display

Hand action can bias the perceived direction of ambiguously moving objects (Wohlschlager, 2000 Vision Research 40 925-930). A common coding framework has been proposed to account for action-perception interaction, assuming goal-based feature matching between action and perception [Prinz, 2003, in Agency and Self-Awareness (New York: Oxford University Press)]. I investigated whether and how movement representation and a distal action goal are implicated in biased motion perception using a stream-bounce display as stimulus. By moving a mouse, participants controlled the progression of disks, which were initially separated and then moved toward each other. Mouse movement was found to bias perception toward the stream when the directional motion of the mouse and disk were consistent; however, this biased perception was not observed when the directions were orthogonal. Subsequent experiments revealed that this induced stream percept could not be attributed to the processes of attention, and the action goal of disk control was critical to the occurrence of biased motion perception. These results suggest that feature matching occurs at the level of movement representation, and the action goal is important in linking movement representation to perceptual representation.

Auditory induced bounce perception persists as the probability of a motion reversal is reduced

When two identical targets move toward one another from opposite sides of a display and continue past one another along collinear trajectories, they can be perceived to either stream past or bounce off of one another. Streaming is the dominant perception in motion displays free of additional transients, while bouncing predominates when a transient (eg auditory or visual) is presented at the point of coincidence. We investigated whether the auditory induced bias towards bouncing would persist as the probability of a motion reversal was reduced by introducing a spatial offset either vertically in a 2-D display or in depth in a 3-D display. Offset conditions were combined with two auditory conditions (tone or no-tone at the point of coincidence) in the presence or absence of a central occluder. In conditions with no sound, streaming was reported on a clear majority of trials, regardless of spatial offset. When a transient tone was presented, reported motion reversals dominated and persisted for increasing verbal offsets up to 17.9 min of arc and for 3-D trajectory offsets up to 25.6 min of arc. The bounce-promoting effect of an auditory tone at the point of coincidence in stream/bounce displays persists in spite of rendering the visual motion sequence unambiguous and more consistent with streaming.

Retinotopically defined primary visual cortex in Williams syndrome

Williams syndrome, caused by a hemizygous microdeletion on chromosome 7q11.23, is characterized by severe impairment in visuospatial construction. To examine potential contributions of early visual processing to this cognitive problem, we functionally mapped the size and neuroanatomical variability of primary visual cortex (V1) in high-functioning adults with Williams syndrome and age- and IQ-matched control participants from the general population by using fMRI-based retinotopic mapping and cortical surface models generated from high-resolution structural MRI. Visual stimulation, consisting of rotating hemicircles and expanding rings, was used to retinotopically define early visual processing areas. V1 boundaries based on computed phase and field sign maps were used to calculate the functional area of V1. Neuroanatomical variability was assessed by computing overlap maps of V1 location for each group on standardized cortical surfaces, and non-parametric permutation test methods were used for statistical inference. V1 did not differ in size between groups, although its anatomical boundaries were more variable in the group with Williams syndrome. V1 overlap maps showed that the average centres of gravity for the two groups were similarly located near the fundus of the calcarine fissure, approximately 25 mm away from the most posterior aspect of the occipital lobe. In summary, our functional definition of V1 size and location indicates that recruitment of primary visual cortex is grossly normal in Williams syndrome, consistent with the notion that neural abnormalities underlying visuospatial construction arise at later stages in the visual processing hierarchy.

Asymmetric cross-modal effects in time perception

It is common to judge the duration of an audiovisual event, and yet it remains controversial how the judgment of duration is affected by signals from other modalities. We used an oddball paradigm to examine the effect of sound on the judgment of visual duration and that of a visual object on the judgment of an auditory duration. In a series of standards and oddballs, the participants compared the duration of the oddballs to that of the standards. Results showed asymmetric cross-modal effects, supporting the auditory dominance hypothesis: a sound extends the perceived visual duration, whereas a visual object has no effect on perceived auditory duration. The possible mechanisms (pacemaker or mode switch) proposed in the Scalar Expectancy Theory [Gibbon, J., Church, R. M., & Meck, W. H. (1984). Scalar timing in memory. In J. Gibbon & L. Allan (Eds.), Annals of the New York Academy of Sciences: Vol. 423. Timing and time perception (pp. 52-77). New York: New York Academy of Sciences] were examined using different standard durations. We conclude that sound increases the perceived visual duration by accelerating the pulse rate in the visual pacemaker.

BOLD response during visual perception of biological motion in obsessive-compulsive disorder : an fMRI study using the dynamic point-light animation paradigm

OBJECTIVE

Although research has shown that deficits in various cognitive functions may underlie obsessive-compulsive disorder (OCD), studies have not yet clarified the specificity and etiology of perception processing, particularly the perception of biological motion that is correlated with social cognition. We used functional magnetic resonance imaging (fMRI) to investigate neural activity associated with the perception of biological motion in OCD patients.

METHODS

The subjects were 15 patients with OCD and 15 age- and IQ-matched healthy volunteers. All subjects participated in a biological motion task in which they performed a one-back task signaling a repeated stimulus with a key press in each block condition to obligate attention to both types of stimuli.

RESULTS

The biological motion versus scrambled motion contrast revealed that both OCD patients and healthy controls exhibited increased activation of the superior and middle temporal gyrus, the regions implicated in processing of biological motion, which is consistent with previous studies. However, direct comparison between OCD subjects and healthy controls indicated that patients with OCD exhibited increased activation in the right superior and middle temporal gyrus and the left inferior temporal and fusiform gyrus, and reduced activation in the right postcentral gyrus (BA 40) compared to healthy subjects. OCD patients exhibited increased activation in the ventral visual system, including the inferior temporal and fusiform gyrus.

DISCUSSION

We observed a differential pattern of activity between OCD patients and healthy controls, indicating that OCD patients have functional differences related to the perception of biological motion. The differential activation between OCD patients and healthy subjects might contribute to the pathophysiological understanding of obsessive compulsive disorder.

Semantic congruency and the Colavita visual dominance effect

Participants presented with auditory, visual, or bimodal audiovisual stimuli in a speeded discrimination task, fail to respond to the auditory component of bimodal targets significantly more often than to the visual component, a phenomenon known as the Colavita visual dominance effect. Given that spatial and temporal factors have recently been shown to modulate the Colavita effect, the aim of the present study, was to investigate whether semantic congruency also modulates the effect. In the three experiments reported here, participants were presented with a version of the Colavita task in which the stimulus congruency between the auditory and visual components of the bimodal targets was manipulated. That is, the auditory and visual stimuli could refer to the same or different object (in Experiments 1 and 2) or audiovisual speech event (Experiment 3). Surprisingly, semantic/stimulus congruency had no effect on the magnitude of the Colavita effect in any of the experiments, although it exerted a significant effect on certain other aspects of participants' performance. This finding contrasts with the results of other recent studies showing that semantic/stimulus congruency can affect certain multisensory interactions.

The effect of a flashing visual stimulus on the auditory continuity illusion

The effect of a visual stimulus on the auditory continuity illusion was examined. Observers judged whether a tone that was repeatedly alternated with a band-pass noise was continuous or discontinuous. In most observers, a transient visual stimulus that was synchronized with the onset of the noise increased the limit of illusory continuity in terms of maximum noise duration and maximum tone level. The smaller the asynchrony between the noise onset and the visual stimulus onset, the larger the visual effect on this illusion. On the other hand, detection of a tone added to the noise was not enhanced by the visual stimulus. These results cannot be fully explained by the conventional theory that illusory continuity is created by the decomposition of peripheral excitation produced by the occluding sound.

Shared attentional resources for global and local motion processing

One of the most important aspects of visual attention is its flexibility; our attentional "window" can be tuned to different spatial scales, allowing us to perceive large-scale global patterns and local features effortlessly. We investigated whether the perception of global and local motion competes for a common attentional resource. Subjects viewed arrays of individual moving Gabors that group to produce a global motion percept when subjects attended globally. When subjects attended locally, on the other hand, they could identify the direction of individual uncrowded Gabors. Subjects were required to devote their attention toward either scale of motion or divide it between global and local scales. We measured direction discrimination as a function of the validity of a precue, which was varied in opposite directions for global and local motion such that when the precue was valid for global motion, it was invalid for local motion and vice versa. There was a trade-off between global and local motion thresholds, such that increasing the validity of precues at one spatial scale simultaneously reduced thresholds at that spatial scale but increased thresholds at the other spatial scale. In a second experiment, we found a similar pattern of results for static-oriented Gabors: Attending to local orientation information impaired the subjects' ability to perceive globally defined orientation and vice versa. Thresholds were higher for orientation compared to motion, however, suggesting that motion discrimination in the first experiment was not driven by orientation information alone but by motion-specific processing. The results of these experiments demonstrate that a shared attentional resource flexibly moves between different spatial scales and allows for the perception of both local and global image features, whether these features are defined by motion or orientation.

Presentation of a visual nearby moving object alters stream/bounce event perception

Two identical visual objects moving across each other in a two-dimensional display can be perceived as either streaming through or bouncing off each other. The bouncing event percept is promoted by the presentation of a brief sound at the point of coincidence of the two objects. In this study, we examined the effect of the presence of a moving object near the two objects as well as the brief sound on the stream/bounce event perception. When both the nearby moving object and brief sound were presented, a streaming event, not a bouncing event, was robustly perceived (experiment 1). The percentage of the streaming percept was also systematically affected by the proximity of the nearby object (experiment 2). These results suggest that the processing of intramodal grouping between a nearby moving object and either of the two objects in the stream/bounce display interferes with crossmodal (audiovisual) processing. Moreover, we demonstrated that, depending on the trajectory of the nearby moving object, the processing of intramodal grouping can promote the bouncing percept, just as crossmodal processing does (experiment 3).

Perceptual completion in a dynamic scene: an investigation with an ambiguous motion paradigm

In this study we employed the streaming-bouncing stimulus to investigate aspects of dynamic occlusion, e.g., of objects that temporarily move under occlusion while covertly being tracked. Two occluders, both either luminance-defined or invisible (virtual), were placed on the trajectories of the moving objects in the streaming-bouncing stimulus. We found that the bouncing percept was dominant when the objects moved under luminance-defined occluders but not when they moved under virtual occluders. Perceived motion direction thus varied with occluder visibility. The results seem to suggest that perceptual completion of a moving object interferes with constant motion processing of the same object.

Seeing the light: exploring the Colavita visual dominance effect

The Colavita visual dominance effect refers to the phenomenon whereby participants presented with unimodal auditory, unimodal visual, or bimodal audiovisual stimuli in a speeded discrimination task, fail to respond to the auditory component of bimodal targets significantly more often than they fail to respond to the visual component. The Colavita effect was demonstrated in this study when participants were presented with unimodal auditory, unimodal visual, or bimodal stimuli (in the ratios 40:40:20, Experiment 1; or 33:33:33, Experiment 2), to which they had to respond by pressing an auditory response key, a visual response key, or both response keys. The Colavita effect was also demonstrated when participants had to respond to the bimodal targets using a dedicated third (bimodal) response key (Experiment 3). These results therefore suggest that stimulus probability and the response demands of the task do not contribute significantly to the Colavita effect. In Experiment 4, we investigated what role exogenous attention toward a sensory modality plays in the Colavita effect. A significantly larger Colavita effect was observed when a visual cue preceded the bimodal target than when an auditory cue preceded it. This result suggests that the Colavita visual dominance effect can be partially explained in terms of the greater exogenous attention-capturing qualities of visual versus auditory stimuli.