Infants' response to temporally based intersensory equivalence: The effect of synchronous sounds on visual preferences for moving stimuli

Synchrony perception across senses: A systematic review of temporal binding window changes from infancy to adolescence in typical and atypical development

Sensory integration is increasingly acknowledged as being crucial for the development of cognitive and social abilities. However, its developmental trajectory is still little understood. This systematic review delves into the topic by investigating the literature about the developmental changes from infancy through adolescence of the Temporal Binding Window (TBW) - the epoch of time within which sensory inputs are perceived as simultaneous and therefore integrated. Following comprehensive searches across PubMed, Elsevier, and PsycInfo databases, only experimental, behavioral, English-language, peer-reviewed studies on multisensory temporal processing in 0-17-year-olds have been included. Non-behavioral, non-multisensory, and non-human studies have been excluded as those that did not directly focus on the TBW. The selection process was independently performed by two Authors. The 39 selected studies involved 2859 participants in total. Findings indicate a predisposition towards cross-modal asynchrony sensitivity and a composite, still unclear, developmental trajectory, with atypical development associated to increased asynchrony tolerance. These results highlight the need for consistent and thorough research into TBW development to inform potential interventions.

The multisensory cocktail party problem in children: Synchrony-based segregation of multiple talking faces improves in early childhood

Extraction of meaningful information from multiple talkers relies on perceptual segregation. The temporal synchrony statistics inherent in everyday audiovisual (AV) speech offer a powerful basis for perceptual segregation. We investigated the developmental emergence of synchrony-based perceptual segregation of multiple talkers in 3-7-year-old children. Children either saw four identical or four different faces articulating temporally jittered versions of the same utterance and heard the audible version of the same utterance either synchronized with one of the talkers or desynchronized with all of them. Eye tracking revealed that selective attention to the temporally synchronized talking face increased while attention to the desynchronized faces decreased with age and that attention to the talkers' mouth primarily drove responsiveness. These findings demonstrate that the temporal synchrony statistics inherent in fluent AV speech assume an increasingly greater role in perceptual segregation of the multisensory clutter created by multiple talking faces in early childhood.

Infants and Adults Use Visual Cues to Improve Detection and Discrimination of Speech in Noise

Purpose This study assessed the extent to which 6- to 8.5-month-old infants and 18- to 30-year-old adults detect and discriminate auditory syllables in noise better in the presence of visual speech than in auditory-only conditions. In addition, we examined whether visual cues to the onset and offset of the auditory signal account for this benefit. Method Sixty infants and 24 adults were randomly assigned to speech detection or discrimination tasks and were tested using a modified observer-based psychoacoustic procedure. Each participant completed 1-3 conditions: auditory-only, with visual speech, and with a visual signal that only cued the onset and offset of the auditory syllable. Results Mixed linear modeling indicated that infants and adults benefited from visual speech on both tasks. Adults relied on the onset-offset cue for detection, but the same cue did not improve their discrimination. The onset-offset cue benefited infants for both detection and discrimination. Whereas the onset-offset cue improved detection similarly for infants and adults, the full visual speech signal benefited infants to a lesser extent than adults on the discrimination task. Conclusions These results suggest that infants' use of visual onset-offset cues is mature, but their ability to use more complex visual speech cues is still developing. Additional research is needed to explore differences in audiovisual enhancement (a) of speech discrimination across speech targets and (b) with increasingly complex tasks and stimuli.

Selective attention to a talker's mouth in infancy: role of audiovisual temporal synchrony and linguistic experience

Previous studies have found that infants shift their attention from the eyes to the mouth of a talker when they enter the canonical babbling phase after 6 months of age. Here, we investigated whether this increased attentional focus on the mouth is mediated by audio-visual synchrony and linguistic experience. To do so, we tracked eye gaze in 4-, 6-, 8-, 10-, and 12-month-old infants while they were exposed either to desynchronized native or desynchronized non-native audiovisual fluent speech. Results indicated that, regardless of language, desynchronization disrupted the usual pattern of relative attention to the eyes and mouth found in response to synchronized speech at 10 months but not at any other age. These findings show that audio-visual synchrony mediates selective attention to a talker's mouth just prior to the emergence of initial language expertise and that it declines in importance once infants become native-language experts.

Babies know bad dancing when they see it: Older but not younger infants discriminate between synchronous and asynchronous audiovisual musical displays

Movement to music is a universal human behavior, yet little is known about how observers perceive audiovisual synchrony in complex musical displays such as a person dancing to music, particularly during infancy and childhood. In the current study, we investigated how perception of musical audiovisual synchrony develops over the first year of life. We habituated infants to a video of a person dancing to music and subsequently presented videos in which the visual track was matched (synchronous) or mismatched (asynchronous) with the audio track. In a visual-only control condition, we presented the same visual stimuli with no sound. In Experiment 1, we found that older infants (8-12months) exhibited a novelty preference for the mismatched movie when both auditory information and visual information were available and showed no preference when only visual information was available. By contrast, younger infants (5-8months) in Experiment 2 did not discriminate matching stimuli from mismatching stimuli. This suggests that the ability to perceive musical audiovisual synchrony may develop during the second half of the first year of infancy.

Effects of movement distance, duration, velocity, and type on action prediction in 12-month-olds

The goal of the present study was to test the influence of the spatial and temporal dynamics of observed manual actions on infants' action prediction. Twelve-month-old infants were presented with reach-and-transport actions performed by a human agent. Movement distance, duration, and - resulting from the two - movement velocity were systematically varied. Action prediction was measured via the latency of gaze arrival at target in relation to agent's hand. The results showed a general effect of all parameters on the infants' perception of goal-directed actions: Infants were more likely to predict the action goal the longer the movement distance was, the longer the movement duration was, and the slower the movement velocity was. In addition, they were more likely to predict the goal of a reaching than a transport action. The present findings extent previous findings by showing that infants are not only sensitive to differences in distances, durations, and velocities at early age but that these factors have a strong impact on the prediction of the goal of observed actions.

Perception of the multisensory coherence of fluent audiovisual speech in infancy: its emergence and the role of experience

To investigate the developmental emergence of the perception of the multisensory coherence of native and non-native audiovisual fluent speech, we tested 4-, 8- to 10-, and 12- to 14-month-old English-learning infants. Infants first viewed two identical female faces articulating two different monologues in silence and then in the presence of an audible monologue that matched the visible articulations of one of the faces. Neither the 4-month-old nor 8- to 10-month-old infants exhibited audiovisual matching in that they did not look longer at the matching monologue. In contrast, the 12- to 14-month-old infants exhibited matching and, consistent with the emergence of perceptual expertise for the native language, perceived the multisensory coherence of native-language monologues earlier in the test trials than that of non-native language monologues. Moreover, the matching of native audible and visible speech streams observed in the 12- to 14-month-olds did not depend on audiovisual synchrony, whereas the matching of non-native audible and visible speech streams did depend on synchrony. Overall, the current findings indicate that the perception of the multisensory coherence of fluent audiovisual speech emerges late in infancy, that audiovisual synchrony cues are more important in the perception of the multisensory coherence of non-native speech than that of native audiovisual speech, and that the emergence of this skill most likely is affected by perceptual narrowing.

Audiovisual temporal fusion in 6-month-old infants

The aim of this study was to investigate neural dynamics of audiovisual temporal fusion processes in 6-month-old infants using event-related brain potentials (ERPs). In a habituation-test paradigm, infants did not show any behavioral signs of discrimination of an audiovisual asynchrony of 200 ms, indicating perceptual fusion. In a subsequent EEG experiment, audiovisual synchronous stimuli and stimuli with a visual delay of 200 ms were presented in random order. In contrast to the behavioral data, brain activity differed significantly between the two conditions. Critically, N1 and P2 latency delays were not observed between synchronous and fused items, contrary to previously observed N1 and P2 latency delays between synchrony and perceived asynchrony. Hence, temporal interaction processes in the infant brain between the two sensory modalities varied as a function of perceptual fusion versus asynchrony perception. The visual recognition components Pb and Nc were modulated prior to sound onset, emphasizing the importance of anticipatory visual events for the prediction of auditory signals. Results suggest mechanisms by which young infants predictively adjust their ongoing neural activity to the temporal synchrony relations to be expected between vision and audition.

Early experience and multisensory perceptual narrowing

Perceptual narrowing reflects the effects of early experience and contributes in key ways to perceptual and cognitive development. Previous studies have found that unisensory perceptual sensitivity in young infants is broadly tuned such that they can discriminate native as well as non-native sensory inputs but that it is more narrowly tuned in older infants such that they only respond to native inputs. Recently, my coworkers and I discovered that multisensory perceptual sensitivity narrows as well. The present article reviews this new evidence in the general context of multisensory perceptual development and the effects of early experience. Together, the evidence on unisensory and multisensory narrowing shows that early experience shapes the emergence of perceptual specialization and expertise.

Development of ordinal sequence perception in infancy

Perception of the ordinal position of a sequence element is critical to many cognitive and motor functions. Here, the prediction that this ability is based on a domain-general perceptual mechanism and, thus, that it emerges prior to the emergence of language was tested. Infants were habituated with sequences of moving/sounding objects and then tested for the ability to perceive the invariant ordinal position of a single element (Experiment 1) or the invariant relative ordinal position of two adjacent elements (Experiment 2). Experiment 1 tested 4- and 6-month-old infants and showed that 4-month-old infants focused on conflicting low-level sequence statistics and, therefore, failed to detect the ordinal position information, but that 6-month-old infants ignored the statistics and detected the ordinal position information. Experiment 2 tested 6-, 8-, and 10-month-old infants and showed that only 10-month-old infants detected relative ordinal position information and that they could only accomplish this with the aid of concurrent statistical cues. Together, these results indicate that a domain-general ability to detect ordinal position information emerges during infancy and that its initial emergence is preceded and facilitated by the earlier emergence of the ability to detect statistical cues.

Does visual information influence infants' movement to music?

Humans are often exposed to music beginning at birth (or even before birth), yet the study of the development of musical abilities during infancy has only recently gained momentum. The goals of the present study were to determine whether young infants (ages four to seven months) spontaneously moved rhythmically in the presence of music, and whether the presence of visual information in addition to music would increase or decrease infants' movement. While nearly all infants moved in the presence of music, very few infants demonstrated rhythmic movement. Results revealed that, when visual information was present, and particularly when infants appeared to show focused attention toward the visual information, infants moved less than when only auditory information was present. The latter result is in agreement with most studies of sensory dominance in adults in which visual stimuli are dominant over auditory stimuli.

Speed discrimination in 6- and 10-month-old infants follows Weber's law

The speed of a moving object is a critical variable that factors into actions such as crossing a street and catching a ball. However, it is not clear when the ability to discriminate between different speeds develops. Here, we investigated speed discrimination in 6- and 10-month-old infants using a habituation paradigm showing infants events of a ball rolling at different speeds. The 6-month-olds looked longer at novel speeds that differed by a 1:2 ratio than at the familiar ones but showed no difference in looking time to speeds that differed by a 2:3 ratio. In contrast, the 10-month-olds succeeded at discriminating a 2:3 ratio. For both age groups, discrimination was modulated by the ratio between novel and familiar speeds, suggesting that speed discrimination is subject to Weber's law. These findings show striking parallels to previous results in infants' discrimination of duration, size, and number and suggest a shared system for processing different magnitudes.

Effect of auditory input on activations in infant diverse cortical regions during audiovisual processing

A fundamental question with regard to perceptual development is how multisensory information is processed in the brain during the early stages of development. Although a growing body of evidence has shown the early emergence of modality-specific functional differentiation of the cortical regions, the interplay between sensory inputs from different modalities in the developing brain is not well understood. To study the effects of auditory input during audio-visual processing in 3-month-old infants, we evaluated the spatiotemporal cortical hemodynamic responses of 50 infants while they perceived visual objects with or without accompanying sounds. The responses were measured using 94-channel near-infrared spectroscopy over the occipital, temporal, and frontal cortices. The effects of sound manipulation were pervasive throughout the diverse cortical regions and were specific to each cortical region. Visual stimuli co-occurring with sound induced the early-onset activation of the early auditory region, followed by activation of the other regions. Removal of the sound stimulus resulted in focal deactivation in the auditory regions and reduced activation in the early visual region, the association region of the temporal and parietal cortices, and the anterior prefrontal regions, suggesting multisensory interplay. In contrast, equivalent activations were observed in the lateral occipital and lateral prefrontal regions, regardless of sound manipulation. Our findings indicate that auditory input did not generally enhance overall activation in relation to visual perception, but rather induced specific changes in each cortical region. The present study implies that 3-month-old infants may perceive audio-visual multisensory inputs by using the global network of functionally differentiated cortical regions.

The Biological Implausibility of the Nature-Nurture Dichotomy & What It Means for the Study of Infancy

Since the time of the Greeks, philosophers and scientists have wondered about the origins of structure and function. Plato proposed that the origins of structure and function lie in the organism's nature whereas Aristotle proposed that they lie in its nurture. This nature/nurture dichotomy and the emphasis on the origins question has had a powerful effect on our thinking about development right into modern times. Despite this, empirical findings from various branches of developmental science have made a compelling case that the nature/nurture dichotomy is biologically implausible and, thus, that a search for developmental origins must be replaced by research into developmental processes. This change in focus recognizes that development is an immensely complex, dynamic, embedded, interdependent, and probabilistic process and, therefore, renders simplistic questions such as whether a particular behavioral capacity is innate or acquired scientifically uninteresting.

Binding of sights and sounds: age-related changes in multisensory temporal processing

We live in a multisensory world and one of the challenges the brain is faced with is deciding what information belongs together. Our ability to make assumptions about the relatedness of multisensory stimuli is partly based on their temporal and spatial relationships. Stimuli that are proximal in time and space are likely to be bound together by the brain and ascribed to a common external event. Using this framework we can describe multisensory processes in the context of spatial and temporal filters or windows that compute the probability of the relatedness of stimuli. Whereas numerous studies have examined the characteristics of these multisensory filters in adults and discrepancies in window size have been reported between infants and adults, virtually nothing is known about multisensory temporal processing in childhood. To examine this, we compared the ability of 10 and 11 year olds and adults to detect audiovisual temporal asynchrony. Findings revealed striking and asymmetric age-related differences. Whereas children were able to identify asynchrony as readily as adults when visual stimuli preceded auditory cues, significant group differences were identified at moderately long stimulus onset asynchronies (150-350 ms) where the auditory stimulus was first. Results suggest that changes in audiovisual temporal perception extend beyond the first decade of life. In addition to furthering our understanding of basic multisensory developmental processes, these findings have implications on disorders (e.g., autism, dyslexia) in which emerging evidence suggests alterations in multisensory temporal function.

Visual stimulation enhances auditory processing in 3-month-old infants and adults

Behavioral work demonstrates human infants are sensitive to a host of intersensory properties and this sensitivity promotes early learning and memory. However, little is known regarding the neural basis of this ability in infants. Using event-related potentials (ERPs) with infants and adults, we show that during passive viewing auditory evoked brain responses are increased with the presence of simultaneous visual stimulation. Results converge with previous adult neuroimaging studies, single-cell recordings in nonhuman animals, and behavioral studies with human infants to provide evidence for an elevated status of multisensory stimulation in infancy. Furthermore, these results may provide a neural marker of multisensory audio-visual processing in infants that can be used to test developmental theories of how information is integrated across the senses to form a unitary perception of the world.

Development of cross-modal processing

The ability to process and integrate cross-modal input is important for many everyday tasks. The current paper reviews theoretical and empirical work examining cross-modal processing with a focus on recent findings examining infants' and children's processing of arbitrary auditory-visual pairings. The current paper puts forward a potential mechanism that may account for modality dominance effects found in a variety of cognitive tasks. The mechanism assumes that although early processing of auditory and visual input is parallel, attention is allocated in a serial manner with the modality that is faster to engage attention dominating later processing. Details of the mechanism, factors influencing processing of arbitrary auditory-visual pairings, and implications for higher-order tasks are discussed. Copyright © 2009 John Wiley & Sons, Ltd. For further resources related to this article, please visit the WIREs website.

The decline of cross-species intersensory perception in human infants: underlying mechanisms and its developmental persistence

The current study investigated the mechanisms underlying the developmental decline in cross-species intersensory matching first reported by Lewkowicz and Ghazanfar [Lewkowicz, D.J., & Ghazanfar, A.A., (2006). The decline of cross-species intersensory perception in human infants. Proc. Natl. Acad. Sci. U. S. A. 103(17), 6771-6774] and whether the decline persists into later development. Experiment 1 investigated whether infants can match monkey vocalizations to asynchronously presented faces and found that neither 4-6 nor 8-10 month-old infants did. Experiment 1 also assessed whether a visual processing deficit may account for the developmental decline in cross-species matching and indicated that it does not because both age groups discriminated silent monkey calls. Experiment 2 investigated whether an auditory processing deficit may account for the decline and indicated that it does not because 8-10 month-old infants discriminated the acoustic versions of the calls. Finally, Experiment 3 asked whether the developmental decline persists into later development by testing cross-species intersensory matching in 12- and 18-month-old infants and showed that it does because neither age group made intersensory matches. Together, these results bolster prior evidence of a decline in cross-species intersensory integration in early human development and shed new light on the mechanisms underlying it.

Development of multisensory spatial integration and perception in humans

Previous studies have shown that adults respond faster and more reliably to bimodal compared to unimodal localization cues. The current study investigated for the first time the development of audiovisual (A-V) integration in spatial localization behavior in infants between 1 and 10 months of age. We observed infants' head and eye movements in response to auditory, visual, or both kinds of stimuli presented either 25 degrees or 45 degrees to the right or left of midline. Infants under 8 months of age intermittently showed response latencies significantly faster toward audiovisual targets than toward either auditory or visual targets alone They did so, however, without exhibiting a reliable violation of the Race Model, suggesting that probability summation alone could explain the faster bimodal response. In contrast, infants between 8 and 10 months of age exhibited bimodal response latencies significantly faster than unimodal latencies for both eccentricity conditions and their latencies violated the Race Model at 25 degrees eccentricity. In addition to this main finding, we found age-dependent eccentricity and modality effects on response latencies. Together, these findings suggest that audiovisual integration emerges late in the first year of life and are consistent with neurophysiological findings from multisensory sites in the superior colliculus of infant monkeys showing that multisensory enhancement of responsiveness is not present at birth but emerges later in life.

The decline of cross-species intersensory perception in human infants

Between 6 and 10 months of age, infants become better at discriminating among native voices and human faces and worse at discriminating among nonnative voices and other species' faces. We tested whether these unisensory perceptual narrowing effects reflect a general ontogenetic feature of perceptual systems by testing across sensory modalities. We showed pairs of monkey faces producing two different vocalizations to 4-, 6-, 8-, and 10-month-old infants and asked whether they would prefer to look at the corresponding face when they heard one of the two vocalizations. Only the two youngest groups exhibited intersensory matching, indicating that perceptual narrowing is pan-sensory and a fundamental feature of perceptual development.

Auditory-visual intermodal matching of small numerosities in 6-month-old infants

Recent studies have reported that preverbal infants are able to discriminate between numerosities of sets presented within a particular modality. There is still debate, however, over whether they are able to perform intermodal numerosity matching, i.e. to relate numerosities of sets presented with different sensory modalities. The present study investigated auditory-visual intermodal matching of small numerosities in infancy by using a violation-of-expectation paradigm. After being familiarized with events of a few objects impacting a surface successively, 6-month-old infants were alternatively presented with two and three tones while the movement of each object remained hidden behind an opaque screen. The screen was then removed to reveal either two or three objects. Results showed that the infants looked significantly longer at the numerically nonequivalent events (the three-tone/two-object and the two-tone/three-object events) than at the numerically equivalent events (the two-tone/two-object and the three-tone/three-object events) irrespective of the rate or duration of auditory tones presented. These findings suggest that infants are capable of performing intermodal matching of small numerosities and that they might possess abstract representations of numerosity beyond sensory modalities.

Learning and discrimination of audiovisual events in human infants: the hierarchical relation between intersensory temporal synchrony and rhythmic pattern cues

This study examined 4- to 10-month-old infants' perception of audio-visual (A-V) temporal synchrony cues in the presence or absence of rhythmic pattern cues. Experiment 1 established that infants of all ages could successfully discriminate between two different audiovisual rhythmic events. Experiment 2 showed that only 10-month-old infants detected a desynchronization of the auditory and visual components of a rhythmical event. Experiment 3 showed that 4- to 8-month-old infants could detect A-V desynchronization but only when the audiovisual event was nonrhythmic. These results show that initially in development infants attend to the overall temporal structure of rhythmic audiovisual events but that later in development they become capable of perceiving the embedded intersensory temporal synchrony relations as well.

Heterogeneity and heterochrony in the development of intersensory perception

It is now well established that a variety of intersensory perceptual skills emerge in early human development. Empirical evidence from studies in the author's as well as other laboratories charting the developmental emergence of these abilities is reviewed. The evidence is considered in terms of the currently dominant theoretical view of intersensory development that assigns the detection of amodal invariants a primary and foundational role. It is argued that this view is inadequate because the detection of amodal invariants is only one of three distinct intersensory integration processes. It is noted that the other two processes, namely, intersensory association of modality-specific cues and non-specific effects of stimulation in one modality on responsiveness to stimulation in another modality, are equally important and that the operation of all three and, in particular, the relation between them, must be studied to attain a complete understanding of intersensory perceptual development. It is suggested that the theoretical approach to the development of intersensory perception should be broadened to include all three types of processes and that developmental studies must respect basic facts and principles of development. To this end, a developmental systems approach is proposed that holds that the development of intersensory integration consists of the heterochronous emergence of heterogeneous perceptual skills.

Increasing specificity in perceptual development: infants' detection of nested levels of multimodal stimulation

This research assessed the development of infants' sensitivity to two nested amodal temporal relations in audible and visible events. Their detection of global temporal synchrony between visible and audible impacts and internal temporal structure nested within each impact specifying object composition (single versus compound objects) was assessed. Infants of 4, 7, and 11 weeks of age were habituated to a single and a compound object striking a surface and then received test trials depicting a change in synchrony or object composition. Results indicated an interaction between age and condition where sensitivity to synchrony was present by 4 weeks and remained stable across age, whereas sensitivity to composition emerged later, by 7 weeks, and increased dramatically with age. These findings converge with other recent findings to illustrate a pattern of increasing specificity in the development of perception, where infants first detect global and later detect embedded relations. The early sensitivity to global relations may provide an organizational framework for development by focusing infant attention on unitary events, guiding and constraining further exploration, and buffering infants from learning incongruent relations.

The development of intersensory temporal perception: an epigenetic systems/limitations view

Several theories have stressed the importance of intersensory integration for development but have not identified specific underlying integration mechanisms. The author reviews and synthesizes current knowledge about the development of intersensory temporal perception and offers a theoretical model based on epigenetic systems theory, proposing that responsiveness to 4 basic features of multimodal temporal experience--temporal synchrony, duration, temporal rate, and rhythm--emerges in a sequential, hierarchical fashion. The model postulates that initial developmental limitations make intersensory synchrony the basis for the integration of intersensory temporal relations and that the emergence of responsiveness to the other, increasingly more complex, temporal relations occurs in a hierarchical, sequential fashion by building on the previously acquired intersensory temporal processing skills.

Intersensory experience and early perceptual development: the role of spatial contiguity in Bobwhite quail chicks' responsiveness to multimodal maternal cues

In contrast to the large body of work on young infants' capacity to perceive temporally based intersensory relations, little research has been done on the role of spatial contiguity in the development of audio-visual integration. This study examined the effects of early postnatal sensory experience on an avian neonate's responsiveness to the spatial contiguity between maternal auditory and visual cues. Specifically, we assessed whether a bobwhite quail chick's ability to respond to the correspondence between the location of auditory and visual events is affected by its sensory-stimulation history. Results revealed that chicks denied species-typical auditory or visual experience in the period immediately following hatching showed altered patterns of responsiveness to maternal auditory and visual cues. In particular, chicks that received modified postnatal sensory experience demonstrated a higher tolerance for audio-visual spatial discrepancy than did control chicks. These results provide evidence of the important role of sensory experience in the emergence of intersensory integration during the perinatal period and highlight the role of spatial information in early perceptual responsiveness to maternal cues.

Earliest memories and recall by modality usually involve recollections of different memories: memories are not amodal

66 students recalled a first memory and indicated whether sensory impressions from each sensory modality were a part of this memory and, if so, how clear and important these impressions were. After a distraction, they also recalled in each sense modality the very first memory they could. Only visual impressions were always a part of first memories. The clarity and importance of sensory attributes were evaluated similarly. Recall by modality led to memories of equivalent clarity, and these were usually memories other than the one recalled as the earliest memory or recalled using other modalities. Memories are more modality-specific than assumed before in theory or practice.