Sound induces perceptual reorganization of an ambiguous motion display in human infants

Infant attention to rhythmic audiovisual synchrony is modulated by stimulus properties

Musical interactions are a common and multimodal part of an infant's daily experiences. Infants hear their parents sing while watching their lips move and see their older siblings dance along to music playing over the radio. Here, we explore whether 8- to 12-month-old infants associate musical rhythms they hear with synchronous visual displays by tracking their dynamic visual attention to matched and mismatched displays. Visual attention was measured using eye-tracking while they attended to a screen displaying two videos of a finger tapping at different speeds. These videos were presented side by side while infants listened to an auditory rhythm (high or low pitch) synchronized with one of the two videos. Infants attended more to the low-pitch trials than to the high-pitch trials but did not display a preference for attending to the synchronous hand over the asynchronous hand within trials. Exploratory evidence, however, suggests that tempo, pitch, and rhythmic complexity interactively engage infants' visual attention to a tapping hand, especially when that hand is aligned with the auditory stimulus. For example, when the rhythm was complex and the auditory stimulus was low in pitch, infants attended to the fast hand more when it aligned with the auditory stream than to misaligned trials. These results suggest that the audiovisual integration in rhythmic non-speech contexts is influenced by stimulus properties.

Synchrony perception across senses: A systematic review of temporal binding window changes from infancy to adolescence in typical and atypical development

Sensory integration is increasingly acknowledged as being crucial for the development of cognitive and social abilities. However, its developmental trajectory is still little understood. This systematic review delves into the topic by investigating the literature about the developmental changes from infancy through adolescence of the Temporal Binding Window (TBW) - the epoch of time within which sensory inputs are perceived as simultaneous and therefore integrated. Following comprehensive searches across PubMed, Elsevier, and PsycInfo databases, only experimental, behavioral, English-language, peer-reviewed studies on multisensory temporal processing in 0-17-year-olds have been included. Non-behavioral, non-multisensory, and non-human studies have been excluded as those that did not directly focus on the TBW. The selection process was independently performed by two Authors. The 39 selected studies involved 2859 participants in total. Findings indicate a predisposition towards cross-modal asynchrony sensitivity and a composite, still unclear, developmental trajectory, with atypical development associated to increased asynchrony tolerance. These results highlight the need for consistent and thorough research into TBW development to inform potential interventions.

Multisensory integration in the mammalian brain: diversity and flexibility in health and disease

Multisensory integration (MSI) occurs in a variety of brain areas, spanning cortical and subcortical regions. In traditional studies on sensory processing, the sensory cortices have been considered for processing sensory information in a modality-specific manner. The sensory cortices, however, send the information to other cortical and subcortical areas, including the higher association cortices and the other sensory cortices, where the multiple modality inputs converge and integrate to generate a meaningful percept. This integration process is neither simple nor fixed because these brain areas interact with each other via complicated circuits, which can be modulated by numerous internal and external conditions. As a result, dynamic MSI makes multisensory decisions flexible and adaptive in behaving animals. Impairments in MSI occur in many psychiatric disorders, which may result in an altered perception of the multisensory stimuli and an abnormal reaction to them. This review discusses the diversity and flexibility of MSI in mammals, including humans, primates and rodents, as well as the brain areas involved. It further explains how such flexibility influences perceptual experiences in behaving animals in both health and disease. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

The multisensory cocktail party problem in children: Synchrony-based segregation of multiple talking faces improves in early childhood

Extraction of meaningful information from multiple talkers relies on perceptual segregation. The temporal synchrony statistics inherent in everyday audiovisual (AV) speech offer a powerful basis for perceptual segregation. We investigated the developmental emergence of synchrony-based perceptual segregation of multiple talkers in 3-7-year-old children. Children either saw four identical or four different faces articulating temporally jittered versions of the same utterance and heard the audible version of the same utterance either synchronized with one of the talkers or desynchronized with all of them. Eye tracking revealed that selective attention to the temporally synchronized talking face increased while attention to the desynchronized faces decreased with age and that attention to the talkers' mouth primarily drove responsiveness. These findings demonstrate that the temporal synchrony statistics inherent in fluent AV speech assume an increasingly greater role in perceptual segregation of the multisensory clutter created by multiple talking faces in early childhood.

Development of multisensory integration following prolonged early-onset visual deprivation

Adult humans make effortless use of multisensory signals and typically integrate them in an optimal fashion.¹ This remarkable ability takes many years for normally sighted children to develop.^2,3 Would individuals born blind or with extremely low vision still be able to develop multisensory integration later in life when surgically treated for sight restoration? Late acquisition of such capability would be a vivid example of the brain's ability to retain high levels of plasticity. We studied the development of multisensory integration in individuals suffering from congenital dense bilateral cataract, surgically treated years after birth. We assessed cataract-treated individuals' reliance on their restored visual abilities when estimating the size of an object simultaneously explored by touch. Within weeks to months after surgery, when combining information from vision and touch, they developed a multisensory weighting behavior similar to matched typically sighted controls. Next, we tested whether cataract-treated individuals benefited from integrating vision with touch by increasing the precision of size estimates, as it occurs when integrating signals in a statistically optimal fashion.¹ For participants retested multiple times, such a benefit developed within months after surgery to levels of precision indistinguishable from optimal behavior. To summarize, the development of multisensory integration does not merely depend on age, but requires extensive multisensory experience with the world, rendered possible by the improved post-surgical visual acuity. We conclude that early exposure to multisensory signals is not essential for the development of multisensory integration, which can still be acquired even after many years of visual deprivation.

The multisensory cocktail party problem in adults: Perceptual segregation of talking faces on the basis of audiovisual temporal synchrony

Social interactions often involve a cluttered multisensory scene consisting of multiple talking faces. We investigated whether audiovisual temporal synchrony can facilitate perceptual segregation of talking faces. Participants either saw four identical or four different talking faces producing temporally jittered versions of the same visible speech utterance and heard the audible version of the same speech utterance. The audible utterance was either synchronized with the visible utterance produced by one of the talking faces or not synchronized with any of them. Eye tracking indicated that participants exhibited a marked preference for the synchronized talking face, that they gazed more at the mouth than the eyes overall, that they gazed more at the eyes of an audiovisually synchronized than a desynchronized talking face, and that they gazed more at the mouth when all talking faces were audiovisually desynchronized. These findings demonstrate that audiovisual temporal synchrony plays a major role in perceptual segregation of multisensory clutter and that adults rely on differential scanning strategies of a talker's eyes and mouth to discover sources of multisensory coherence.

Cross-Talk of Low-Level Sensory and High-Level Cognitive Processing: Development, Mechanisms, and Relevance for Cross-Modal Abilities of the Brain

The emergence of cross-modal learning capabilities requires the interaction of neural areas accounting for sensory and cognitive processing. Convergence of multiple sensory inputs is observed in low-level sensory cortices including primary somatosensory (S1), visual (V1), and auditory cortex (A1), as well as in high-level areas such as prefrontal cortex (PFC). Evidence shows that local neural activity and functional connectivity between sensory cortices participate in cross-modal processing. However, little is known about the functional interplay between neural areas underlying sensory and cognitive processing required for cross-modal learning capabilities across life. Here we review our current knowledge on the interdependence of low- and high-level cortices for the emergence of cross-modal processing in rodents. First, we summarize the mechanisms underlying the integration of multiple senses and how cross-modal processing in primary sensory cortices might be modified by top-down modulation of the PFC. Second, we examine the critical factors and developmental mechanisms that account for the interaction between neuronal networks involved in sensory and cognitive processing. Finally, we discuss the applicability and relevance of cross-modal processing for brain-inspired intelligent robotics. An in-depth understanding of the factors and mechanisms controlling cross-modal processing might inspire the refinement of robotic systems by better mimicking neural computations.

Multiple phases of cross-sensory interactions associated with the audiovisual bounce-inducing effect

Using event-related potential (ERP) recordings, the present study investigated the cross-modal neural activities underlying the audiovisual bounce-inducing effect (ABE) via a novel experimental design wherein the audiovisual bouncing trials were induced solely by the ABE. The within-subject (percept-based) analysis showed that early cross-modal interactions within 100-200 ms after sound onset over fronto-central and occipital regions were associated with the occurrence of the ABE, but the cross-modal interaction at a later latency (ND250, 220-280 ms) over fronto-central region did not differ between ABE trials and non-ABE trials. The between-subject analysis indicated that the cross-modal interaction revealed by ND250 was larger for subjects who perceived the ABE more frequently. These findings suggest that the ABE is generated as a consequence of the rapid interplay between the variations of early cross-modal interactions and the general multisensory binding predisposition at an individual level.

Infants are superior in implicit crossmodal learning and use other learning mechanisms than adults

During development internal models of the sensory world must be acquired which have to be continuously adapted later. We used event-related potentials (ERP) to test the hypothesis that infants extract crossmodal statistics implicitly while adults learn them when task relevant. Participants were passively exposed to frequent standard audio-visual combinations (A1V1, A2V2, p=0.35 each), rare recombinations of these standard stimuli (A1V2, A2V1, p=0.10 each), and a rare audio-visual deviant with infrequent auditory and visual elements (A3V3, p=0.10). While both six-month-old infants and adults differentiated between rare deviants and standards involving early neural processing stages only infants were sensitive to crossmodal statistics as indicated by a late ERP difference between standard and recombined stimuli. A second experiment revealed that adults differentiated recombined and standard combinations when crossmodal combinations were task relevant. These results demonstrate a heightened sensitivity for crossmodal statistics in infants and a change in learning mode from infancy to adulthood.

Overcoming the other-race effect in infancy with multisensory redundancy: 10-12-month-olds discriminate dynamic other-race faces producing speech

We tested 4-6- and 10-12-month-old infants to investigate whether the often-reported decline in infant sensitivity to other-race faces may reflect responsiveness to static or dynamic/silent faces rather than a general process of perceptual narrowing. Across three experiments, we tested discrimination of either dynamic own-race or other-race faces which were either accompanied by a speech syllable, no sound, or a non-speech sound. Results indicated that 4-6- and 10-12-month-old infants discriminated own-race as well as other-race faces accompanied by a speech syllable, that only the 10-12-month-olds discriminated silent own-race faces, and that 4-6-month-old infants discriminated own-race and other-race faces accompanied by a non-speech sound but that 10-12-month-old infants only discriminated own-race faces accompanied by a non-speech sound. Overall, the results suggest that the ORE reported to date reflects infant responsiveness to static or dynamic/silent faces rather than a general process of perceptual narrowing.

Pre-coincidence brain activity predicts the perceptual outcome of streaming/bouncing motion display

When two identical visual discs move toward each other on a two-dimensional visual display, they can be perceived as either "streaming through" or "bouncing off" each other after their coincidence. Previous studies have observed a strong bias toward the streaming percept. Additionally, the incidence of the bouncing percept in this ambiguous display could be increased by various factors, such as a brief sound at the moment of coincidence and a momentary pause of the two discs. The streaming/bouncing bistable motion phenomenon has been studied intensively since its discovery. However, little is known regarding the neural basis underling the perceptual ambiguity in the classic version of the streaming/bouncing motion display. The present study investigated the neural basis of the perception disambiguating underling the processing of the streaming/bouncing bistable motion display using event-related potential (ERP) recordings. Surprisingly, the amplitude of frontal central P2 (220-260 ms) that was elicited by the moving discs ~200 ms before the coincidence of the two discs was observed to be predictive of subsequent streaming or bouncing percept. A larger P2 amplitude was observed for streaming percept than the bouncing percept. These findings suggest that the streaming/bouncing bistable perception may have been disambiguated unconsciously ~200 ms before the coincidence of the two discs.

The impact of semantically congruent and incongruent visual information on auditory object recognition across development

The ability to use different sensory signals in conjunction confers numerous advantages on perception. Multisensory perception in adults is influenced by factors beyond low-level stimulus properties such as semantic congruency. Sensitivity to semantic relations has been shown to emerge early in development; however, less is known about whether implementation of these associations changes with development or whether development in the representations themselves might modulate their influence. Here, we used a Stroop-like paradigm that requires participants to identify an auditory stimulus while ignoring a visual stimulus. Prior research shows that in adults visual distractors have more impact on processing of auditory objects than vice versa; however, this pattern appears to be inverted early in development. We found that children from 8years of age (and adults) gain a speed advantage from semantically congruent visual information and are disadvantaged by semantically incongruent visual information. At 6years of age, children gain a speed advantage for semantically congruent visual information but are not disadvantaged by semantically incongruent visual information (as compared with semantically unrelated visual information). Both children and adults were influenced by associations between auditory and visual stimuli, which they had been exposed to on only 12 occasions during the learning phase of the study. Adults showed a significant speed advantage over children for well-established associations but showed no such advantage for newly acquired pairings. This suggests that the influence of semantic associations on multisensory processing does not change with age but rather these associations become more robust and, in turn, more influential.

Selective attention to a talker's mouth in infancy: role of audiovisual temporal synchrony and linguistic experience

Previous studies have found that infants shift their attention from the eyes to the mouth of a talker when they enter the canonical babbling phase after 6 months of age. Here, we investigated whether this increased attentional focus on the mouth is mediated by audio-visual synchrony and linguistic experience. To do so, we tracked eye gaze in 4-, 6-, 8-, 10-, and 12-month-old infants while they were exposed either to desynchronized native or desynchronized non-native audiovisual fluent speech. Results indicated that, regardless of language, desynchronization disrupted the usual pattern of relative attention to the eyes and mouth found in response to synchronized speech at 10 months but not at any other age. These findings show that audio-visual synchrony mediates selective attention to a talker's mouth just prior to the emergence of initial language expertise and that it declines in importance once infants become native-language experts.

Multisensory Processes: A Balancing Act across the Lifespan

Multisensory processes are fundamental in scaffolding perception, cognition, learning, and behavior. How and when stimuli from different sensory modalities are integrated rather than treated as separate entities is poorly understood. We review how the relative reliance on stimulus characteristics versus learned associations dynamically shapes multisensory processes. We illustrate the dynamism in multisensory function across two timescales: one long term that operates across the lifespan and one short term that operates during the learning of new multisensory relations. In addition, we highlight the importance of task contingencies. We conclude that these highly dynamic multisensory processes, based on the relative weighting of stimulus characteristics and learned associations, provide both stability and flexibility to brain functions over a wide range of temporal scales.

Multisensory integration and child neurodevelopment

A considerable number of cognitive processes depend on the integration of multisensory information. The brain integrates this information, providing a complete representation of our surrounding world and giving us the ability to react optimally to the environment. Infancy is a period of great changes in brain structure and function that are reflected by the increase of processing capacities of the developing child. However, it is unclear if the optimal use of multisensory information is present early in childhood or develops only later, with experience. The first part of this review has focused on the typical development of multisensory integration (MSI). We have described the two hypotheses on the developmental process of MSI in neurotypical infants and children, and have introduced MSI and its neuroanatomic correlates. The second section has discussed the neurodevelopmental trajectory of MSI in cognitively-challenged infants and children. A few studies have brought to light various difficulties to integrate sensory information in children with a neurodevelopmental disorder. Consequently, we have exposed certain possible neurophysiological relationships between MSI deficits and neurodevelopmental disorders, especially dyslexia and attention deficit disorder with/without hyperactivity.

The audiovisual temporal binding window narrows in early childhood

Binding is key in multisensory perception. This study investigated the audio-visual (A-V) temporal binding window in 4-, 5-, and 6-year-old children (total N = 120). Children watched a person uttering a syllable whose auditory and visual components were either temporally synchronized or desynchronized by 366, 500, or 666 ms. They were asked whether the voice and face went together (Experiment 1) or whether the desynchronized videos differed from the synchronized one (Experiment 2). Four-year-olds detected the 666-ms asynchrony, 5-year-olds detected the 666- and 500-ms asynchrony, and 6-year-olds detected all asynchronies. These results show that the A-V temporal binding window narrows slowly during early childhood and that it is still wider at 6 years of age than in older children and adults.

Early experience and multisensory perceptual narrowing

Perceptual narrowing reflects the effects of early experience and contributes in key ways to perceptual and cognitive development. Previous studies have found that unisensory perceptual sensitivity in young infants is broadly tuned such that they can discriminate native as well as non-native sensory inputs but that it is more narrowly tuned in older infants such that they only respond to native inputs. Recently, my coworkers and I discovered that multisensory perceptual sensitivity narrows as well. The present article reviews this new evidence in the general context of multisensory perceptual development and the effects of early experience. Together, the evidence on unisensory and multisensory narrowing shows that early experience shapes the emergence of perceptual specialization and expertise.

Experiences of activity and causality in schizophrenia: when predictive deficits lead to a retrospective over-binding

In this paper I discuss an intriguing and relatively little studied symptomatic expression of schizophrenia known as experiences of activity in which patients form the delusion that they can control some external events by the sole means of their mind. I argue that experiences of activity result from patients being prone to aberrantly infer causal relations between unrelated events in a retrospective way owing to widespread predictive deficits. Moreover, I suggest that such deficits may, in addition, lead to an aberrant intentional binding effect i.e., the subjective compression of the temporal interval between an intentional action and its external effects (Haggard, Clark, & Kalogeras, 2002). In particular, it might be that patient's thoughts are bound to the external events they aimed to control producing, arguably, a temporal contiguity between these two components. Such temporal contiguity would reinforce or sustain the (causal) feeling that the patient mind is directly causally efficient.

Development of ordinal sequence perception in infancy

Perception of the ordinal position of a sequence element is critical to many cognitive and motor functions. Here, the prediction that this ability is based on a domain-general perceptual mechanism and, thus, that it emerges prior to the emergence of language was tested. Infants were habituated with sequences of moving/sounding objects and then tested for the ability to perceive the invariant ordinal position of a single element (Experiment 1) or the invariant relative ordinal position of two adjacent elements (Experiment 2). Experiment 1 tested 4- and 6-month-old infants and showed that 4-month-old infants focused on conflicting low-level sequence statistics and, therefore, failed to detect the ordinal position information, but that 6-month-old infants ignored the statistics and detected the ordinal position information. Experiment 2 tested 6-, 8-, and 10-month-old infants and showed that only 10-month-old infants detected relative ordinal position information and that they could only accomplish this with the aid of concurrent statistical cues. Together, these results indicate that a domain-general ability to detect ordinal position information emerges during infancy and that its initial emergence is preceded and facilitated by the earlier emergence of the ability to detect statistical cues.

Recognition of Amodal Language Identity Emerges in Infancy

Audiovisual speech consists of overlapping and invariant patterns of dynamic acoustic and optic articulatory information. Research has shown that infants can perceive a variety of basic audio-visual (A-V) relations but no studies have investigated whether and when infants begin to perceive higher order A-V relations inherent in speech. Here, we asked whether and when infants become capable of recognizing amodal language identity, a critical perceptual skill that is necessary for the development of multisensory communication. Because, at a minimum, such a skill requires the ability to perceive suprasegmental auditory and visual linguistic information, we predicted that this skill would not emerge before higher-level speech processing and multisensory integration skills emerge. Consistent with this prediction, we found that recognition of the amodal identity of language emerges at 10-12 months of age but that when it emerges it is restricted to infants' native language.

Effect of auditory input on activations in infant diverse cortical regions during audiovisual processing

A fundamental question with regard to perceptual development is how multisensory information is processed in the brain during the early stages of development. Although a growing body of evidence has shown the early emergence of modality-specific functional differentiation of the cortical regions, the interplay between sensory inputs from different modalities in the developing brain is not well understood. To study the effects of auditory input during audio-visual processing in 3-month-old infants, we evaluated the spatiotemporal cortical hemodynamic responses of 50 infants while they perceived visual objects with or without accompanying sounds. The responses were measured using 94-channel near-infrared spectroscopy over the occipital, temporal, and frontal cortices. The effects of sound manipulation were pervasive throughout the diverse cortical regions and were specific to each cortical region. Visual stimuli co-occurring with sound induced the early-onset activation of the early auditory region, followed by activation of the other regions. Removal of the sound stimulus resulted in focal deactivation in the auditory regions and reduced activation in the early visual region, the association region of the temporal and parietal cortices, and the anterior prefrontal regions, suggesting multisensory interplay. In contrast, equivalent activations were observed in the lateral occipital and lateral prefrontal regions, regardless of sound manipulation. Our findings indicate that auditory input did not generally enhance overall activation in relation to visual perception, but rather induced specific changes in each cortical region. The present study implies that 3-month-old infants may perceive audio-visual multisensory inputs by using the global network of functionally differentiated cortical regions.

Cognitive binding in schizophrenia: weakened integration of temporal intersensory information

Cognitive functioning is based on binding processes, by which different features and elements of neurocognition are integrated and coordinated. Binding is an essential ingredient of, for instance, Gestalt perception. We have implemented a paradigm of causality perception based on the work of Albert Michotte, in which 2 identical discs move from opposite sides of a monitor, steadily toward, and then past one another. Their coincidence generates an ambiguous percept of either "streaming" or "bouncing," which the subjects (34 schizophrenia spectrum patients and 34 controls with mean age 27.9 y) were instructed to report. The latter perception is a marker of the binding processes underlying perceived causality (type I binding). In addition to this visual task, acoustic stimuli were presented at different times during the task (150 ms before and after visual coincidence), which can modulate perceived causality. This modulation by intersensory and temporally delayed stimuli is viewed as a different type of binding (type II). We show here, using a mixed-effects hierarchical analysis, that type II binding distinguishes schizophrenia spectrum patients from healthy controls, whereas type I binding does not. Type I binding may even be excessive in some patients, especially those with positive symptoms; Type II binding, however, was generally attenuated in patients. The present findings point to ways in which the disconnection (or Gestalt) hypothesis of schizophrenia can be refined, suggesting more specific markers of neurocognitive functioning and potential targets of treatment.

Two- to eight-month-old infants' perception of dynamic auditory-visual spatial colocation

From birth, infants detect associations between the locations of static visual objects and sounds they emit, but there is limited evidence regarding their sensitivity to the dynamic equivalent when a sound-emitting object moves. In 4 experiments involving thirty-six 2-month-olds, forty-eight 5-month-olds, and forty-eight 8-month-olds, we investigated infants' ability to process this form of spatial colocation. Whereas there was no evidence of spontaneous sensitivity, all age groups detected a dynamic colocation during habituation and looked longer at test trials in which sound and sight were dislocated. Only 2-month-olds showed clear sensitivity to the dislocation relation, although 8-month-olds did so following additional habituation. These results are discussed relative to the intersensory redundancy hypothesis and work suggesting increasing specificity in processing with age.

Auditory induced bounce perception persists as the probability of a motion reversal is reduced

When two identical targets move toward one another from opposite sides of a display and continue past one another along collinear trajectories, they can be perceived to either stream past or bounce off of one another. Streaming is the dominant perception in motion displays free of additional transients, while bouncing predominates when a transient (eg auditory or visual) is presented at the point of coincidence. We investigated whether the auditory induced bias towards bouncing would persist as the probability of a motion reversal was reduced by introducing a spatial offset either vertically in a 2-D display or in depth in a 3-D display. Offset conditions were combined with two auditory conditions (tone or no-tone at the point of coincidence) in the presence or absence of a central occluder. In conditions with no sound, streaming was reported on a clear majority of trials, regardless of spatial offset. When a transient tone was presented, reported motion reversals dominated and persisted for increasing verbal offsets up to 17.9 min of arc and for 3-D trajectory offsets up to 25.6 min of arc. The bounce-promoting effect of an auditory tone at the point of coincidence in stream/bounce displays persists in spite of rendering the visual motion sequence unambiguous and more consistent with streaming.

Sound enhances detection of visual target during infancy: a study using illusory contours

In adults, a salient tone embedded in a sequence of nonsalient tones improves detection of a synchronously and briefly presented visual target in a rapid, visually distracting sequence. This phenomenon indicates that perception from one sensory modality can be influenced by another one even when the latter modality provides no information about the judged property itself. However, no study has revealed the age-related development of this kind of cross-modal enhancement. Here we tested the effect of concurrent and unique sounds on detection of illusory contours during infancy. We used a preferential looking technique to investigate whether audio-visual enhancement of the detection of illusory contours could be observed at 5, 6, and 7 months of age. A significant enhancement, induced by sound, of the preference for illusory contours was observed only in the 7-month-olds. These results suggest that audio-visual enhancement in visual target detection emerges at 7 months of age.

Speech and non-speech audio-visual illusions: a developmental study

It is well known that simultaneous presentation of incongruent audio and visual stimuli can lead to illusory percepts. Recent data suggest that distinct processes underlie non-specific intersensory speech as opposed to non-speech perception. However, the development of both speech and non-speech intersensory perception across childhood and adolescence remains poorly defined. Thirty-eight observers aged 5 to 19 were tested on the McGurk effect (an audio-visual illusion involving speech), the Illusory Flash effect and the Fusion effect (two audio-visual illusions not involving speech) to investigate the development of audio-visual interactions and contrast speech vs. non-speech developmental patterns. Whereas the strength of audio-visual speech illusions varied as a direct function of maturational level, performance on non-speech illusory tasks appeared to be homogeneous across all ages. These data support the existence of independent maturational processes underlying speech and non-speech audio-visual illusory effects.

Multi-sensory integration of spatio-temporal segmentation cues: one plus one does not always equal two

How are multiple, multi-sensory stimuli combined for use in segmenting spatio-temporal events? For an answer, we measured the effect of various auditory or visual stimuli, in isolation or in combination, on a bistable percept of visual motion ("bouncing" vs. "streaming"). To minimize individual differences, the physical properties of stimuli were adjusted to reflect individual subjects' sensitivity to each cue in isolation. When put into combination, perceptual influences that had been equipotent in isolation were substantially altered. Specifically, auditory cues that had been strong when presented alone were greatly reduced in combination. Evaluation of alternative models of sensory integration showed that the state of the visual bistable percept could not be accounted for by probability summation among cues, as might occur at the level of decision processes. Instead, the state of the bistable percept was well predicted from a weighted sum of cues, with visual cues strongly dominating auditory cues. Finally, when cue weights were compared for individual subjects, it was found that subjects differ somewhat in the strategy they use for integrating multi-sensory information.

Effects of the orientation of moving objects on the perception of streaming/bouncing motion displays

In this study, we examined the contribution of the orientation of moving objects to perception of a streaming/bouncing motion display. In three experiments, participants reported which of the two types of motion, streaming or bouncing, they perceived. The following independent variables were used: orientation differences between Gabor micropatterns (Gabors) and their path of motion (all the experiments) and the presence/absence of a transient tone (Experiment 1), transient visual flash (Experiment 2), or concurrent secondary task (Experiment 3) at the coincidence of Gabors. The results showed that the events at coincidence generally biased responses toward the perception of bouncing. On the other hand, alignment of Gabors with their motion axes significantly reduced the frequency of bounce perception. The results also indicated that an object whose orientation was parallel to its motion path strengthened the spatiotemporal integration of local motion signals along a straight motion path, resulting in the perception of streaming. We suggest that the effect of collinearity between Gabors and their motion path is relatively free from the effect of attention distraction.

Development of multisensory spatial integration and perception in humans

Previous studies have shown that adults respond faster and more reliably to bimodal compared to unimodal localization cues. The current study investigated for the first time the development of audiovisual (A-V) integration in spatial localization behavior in infants between 1 and 10 months of age. We observed infants' head and eye movements in response to auditory, visual, or both kinds of stimuli presented either 25 degrees or 45 degrees to the right or left of midline. Infants under 8 months of age intermittently showed response latencies significantly faster toward audiovisual targets than toward either auditory or visual targets alone They did so, however, without exhibiting a reliable violation of the Race Model, suggesting that probability summation alone could explain the faster bimodal response. In contrast, infants between 8 and 10 months of age exhibited bimodal response latencies significantly faster than unimodal latencies for both eccentricity conditions and their latencies violated the Race Model at 25 degrees eccentricity. In addition to this main finding, we found age-dependent eccentricity and modality effects on response latencies. Together, these findings suggest that audiovisual integration emerges late in the first year of life and are consistent with neurophysiological findings from multisensory sites in the superior colliculus of infant monkeys showing that multisensory enhancement of responsiveness is not present at birth but emerges later in life.

Perception of audiovisual rhythm and its invariance in 4- to 10-month-old infants

This study investigated the perception of complex audiovisual rhythmic patterns in 4-, 6-, 8-, and 10-month-old human infants. In Experiment 1, we first habituated infants to an event in which an object could be seen and heard bouncing in a rhythmic fashion. We then tested them to determine if they would detect a relative temporal pattern change produced by rearranging the intrapattern intervals. Regardless of age, infants successfully detected the pattern change. In Experiment 2, we asked whether infants also can extract rhythmic pattern invariance amid tempo variations. Thus, we first habituated infants to a particular rhythmic pattern but this time varying in its tempo of presentation across trials. We then administered one test trial in which a novel rhythm was presented at a familiar tempo and another test trial in which a familiar rhythm was presented at a novel tempo. Infants detected both types of changes indicating that they perceived the invariant rhythm and that they did so despite the fact that they also detected the varying tempo. Overall, the findings demonstrate that infants between 4 and 10 months of age can perceive and discriminate complex audiovisual temporal patterns on the basis of relative temporal differences and that they also can learn the invariant nature of such patterns.

The relationship between object files and conscious perception

Object files (OFs) are hypothesized mid-level representations which mediate our conscious perception of persisting objects-e.g. telling us 'which went where'. Despite the appeal of the OF framework, not previous research has directly explored whether OFs do indeed correspond to conscious percepts. Here we present at least one case wherein conscious percepts of 'which went where' in dynamic ambiguous displays diverge from the analogous correspondence computed by the OF system. Observers viewed a 'bouncing/streaming' display in which two identical objects moved such that they could have either bounced off or streamed past each other. We measured two dependent variables: (1) an explicit report of perceived bouncing or streaming; and (2) an implicit 'object-specific preview benefit' (OSPB), wherein a 'preview' of information on a specific object speeds the recognition of that information at a later point when it appears again on the same object (compared to when it reappears on a different object), beyond display-wide priming. When the displays were manipulated such that observers had a strong bias to perceive streaming (on over 95% of the trials), there was nevertheless a strong OSPB in the opposite direction-such that the object files appeared to have 'bounced' even though the percept 'streamed'. Given that OSPBs have been taken as a hallmark of the operation of object files, the five experiments reported here suggest that in at least some specialized (and perhaps ecologically invalid) cases, conscious percepts of 'which went where' in dynamic ambiguous displays can diverge from the mapping computed by the object-file system.

Perception of serial order in infants

Serial order is fundamental to perception, cognition and behavioral action. Three experiments investigated infants' perception, learning and discrimination of serial order. Four- and 8-month-old infants were habituated to three sequentially moving objects making visible and audible impacts and then were tested on separate test trials for their ability to detect auditory, visual or auditory-visual changes in their ordering. The 4-month-old infants did not respond to any order changes and instead appeared to attend to the 'local' audio-visual synchrony part of the event. When this local part of the event was blocked from view, the 4-month-olds did perceive the serial order feature of the event but only when it was specified multimodally. In contrast, the 8-month-old infants perceived all three kinds of order changes regardless of whether the synchrony part of the event was visible or not. The findings show that perception of spatiotemporal serial order emerges early in infancy and that its perception is initially facilitated by multimodal specification.

Multimodal Events and Moving Locations: Eye Movements of Adults and 6-Month-Olds Reveal Dynamic Spatial Indexing

The ability to keep track of locations in a dynamic, multimodal environment is crucial for successful interactions with other people and objects. The authors investigated the existence and flexibility of spatial indexing in adults and 6-month-old infants by adapting an eye-tracking paradigm from D. C. Richardson and M. J. Spivey (2000). Multimodal events were presented in specific locations, and eye movements were measured when the auditory portion of the stimulus was presented without its visual counterpart. Experiment 1 showed that adults spatially index auditory information even when the original associated locations move. Experiments 2 and 3 showed that infants are capable of both binding multimodal events to locations and tracking those locations when they move.

Crossmodal integration for perception and action

The integration of information from different sensory modalities has many advantages for human observers, including increase of salience, resolution of perceptual ambiguities, and unified perception of objects and surroundings. Several behavioral, electrophysiological and neuroimaging data collected in various tasks, including localization and detection of spatial events, crossmodal perception of object properties and scene analysis are reviewed here. All the results highlight the multiple faces of crossmodal interactions and provide converging evidence that the brain takes advantages of spatial and temporal coincidence between spatial events in the crossmodal binding of spatial features gathered through different modalities. Furthermore, the elaboration of a multimodal percept appears to be based on an adaptive combination of the contribution of each modality, according to the intrinsic reliability of sensory cue, which itself depends on the task at hand and the kind of perceptual cues involved in sensory processing. Computational models based on bayesian sensory estimation provide valuable explanations of the way perceptual system could perform such crossmodal integration. Recent anatomical evidence suggest that crossmodal interactions affect early stages of sensory processing, and could be mediated through a dynamic recurrent network involving backprojections from multimodal areas as well as lateral connections that can modulate the activity of primary sensory cortices, though future behavioral and neurophysiological studies should allow a better understanding of the underlying mechanisms.

Heterogeneity and heterochrony in the development of intersensory perception

It is now well established that a variety of intersensory perceptual skills emerge in early human development. Empirical evidence from studies in the author's as well as other laboratories charting the developmental emergence of these abilities is reviewed. The evidence is considered in terms of the currently dominant theoretical view of intersensory development that assigns the detection of amodal invariants a primary and foundational role. It is argued that this view is inadequate because the detection of amodal invariants is only one of three distinct intersensory integration processes. It is noted that the other two processes, namely, intersensory association of modality-specific cues and non-specific effects of stimulation in one modality on responsiveness to stimulation in another modality, are equally important and that the operation of all three and, in particular, the relation between them, must be studied to attain a complete understanding of intersensory perceptual development. It is suggested that the theoretical approach to the development of intersensory perception should be broadened to include all three types of processes and that developmental studies must respect basic facts and principles of development. To this end, a developmental systems approach is proposed that holds that the development of intersensory integration consists of the heterochronous emergence of heterogeneous perceptual skills.