Behavioral and Neural Foundations of Multisensory Face-Voice Perception in Infancy

Voice categorization in the four-month-old human brain

Voices are the most relevant social sounds for humans and therefore have crucial adaptive value in development. Neuroimaging studies in adults have demonstrated the existence of regions in the superior temporal sulcus that respond preferentially to voices. Yet, whether voices represent a functionally specific category in the young infant's mind is largely unknown. We developed a highly sensitive paradigm relying on fast periodic auditory stimulation (FPAS) combined with scalp electroencephalography (EEG) to demonstrate that the infant brain implements a reliable preferential response to voices early in life. Twenty-three 4-month-old infants listened to sequences containing non-vocal sounds from different categories presented at 3.33 Hz, with highly heterogeneous vocal sounds appearing every third stimulus (1.11 Hz). We were able to isolate a voice-selective response over temporal regions, and individual voice-selective responses were found in most infants within only a few minutes of stimulation. This selective response was significantly reduced for the same frequency-scrambled sounds, indicating that voice selectivity is not simply driven by the envelope and the spectral content of the sounds. Such a robust selective response to voices as early as 4 months of age suggests that the infant brain is endowed with the ability to rapidly develop a functional selectivity to this socially relevant category of sounds.

The Development of Cortical Responses to the Integration of Audiovisual Speech in Infancy

In adults, the integration of audiovisual speech elicits specific higher (super-additive) or lower (sub-additive) cortical responses when compared to the responses to unisensory stimuli. Although there is evidence that the fronto-temporal network is active during perception of audiovisual speech in infancy, the development of fronto-temporal responses to audiovisual integration remains unknown. In the current study, 5-month-olds and 10-month-olds watched bimodal (audiovisual) and alternating unimodal (auditory + visual) syllables. In this context we use alternating unimodal to denote alternating auditory and visual syllables that are perceived as separate syllables by adults. Using fNIRS we measured responses over large cortical areas including the inferior frontal and superior temporal regions. We identified channels showing different responses to bimodal than alternating unimodal condition and used multivariate pattern analysis (MVPA) to decode patterns of cortical responses to bimodal (audiovisual) and alternating unimodal (auditory + visual) speech. Results showed that in both age groups integration elicits cortical responses consistent with both super- and sub-additive responses in the fronto-temporal cortex. The univariate analyses revealed that between 5 and 10 months spatial distribution of these responses becomes increasingly focal. MVPA correctly classified responses at 5 months, with key input from channels located in the inferior frontal and superior temporal channels of the right hemisphere. However, MVPA classification was not successful at 10 months, suggesting a potential cortical re-organisation of audiovisual speech perception at this age. These results show the complex and non-gradual development of the cortical responses to integration of congruent audiovisual speech in infancy.

Coordinating attention requires coordinated senses

From playing basketball to ordering at a food counter, we frequently and effortlessly coordinate our attention with others towards a common focus: we look at the ball, or point at a piece of cake. This non-verbal coordination of attention plays a fundamental role in our social lives: it ensures that we refer to the same object, develop a shared language, understand each other's mental states, and coordinate our actions. Models of joint attention generally attribute this accomplishment to gaze coordination. But are visual attentional mechanisms sufficient to achieve joint attention, in all cases? Besides cases where visual information is missing, we show how combining it with other senses can be helpful, and even necessary to certain uses of joint attention. We explain the two ways in which non-visual cues contribute to joint attention: either as enhancers, when they complement gaze and pointing gestures in order to coordinate joint attention on visible objects, or as modality pointers, when joint attention needs to be shifted away from the whole object to one of its properties, say weight or texture. This multisensory approach to joint attention has important implications for social robotics, clinical diagnostics, pedagogy and theoretical debates on the construction of a shared world.

Dysfunctional perceptual antecedent can justify the social orienting deficit in autism spectrum disorder: an eye-tracking study

Purpose

Children with Autism Spectrum Disorder (ASD) show reduced attention to social stimuli. The reasons for these impairments are still being debated by researchers. The aim of this study is to analyse if reduced attention towards social stimuli is determined by initial underlying difficulties in the control of visual attention. Among the variables that could produce these difficulties, the authors considered geometric complexity and typology of geometric figures.

Design/methodology/approach

To test this hypothesis, in this paper, an eye-tracker paradigm was used for assessing visual exploration and recognition memory towards geometric figures (curved vs rectilinear) with two levels of geometric complexity (low and high) in 17 children with ASD matched with 17 children with typical development (TD).

Findings

The results showed that the ASD group seemed indifferent to both the geometric complexity and the typology of figures (curved and rectilinear), whereas the TD group showed higher performances with highly complex and curved geometric figures than with low complex and rectilinear geometric figures.

Research limitations/implications

Because of the chosen research approach, the research results may lack generalizability. Therefore, researchers are encouraged to test the proposed hypotheses further.

Practical implications

This paper includes implications upon the presence of an unspecified visual attention deficit that is present from the early stages of the processing of stimuli.

Social implications

The understanding of this deficit from the early stages of the processing of stimuli can help educators to intervene at an early stage when disturbances in social relationships are starting.

Originality/value

This study contributes to understanding the presence of dysfunctional perceptual antecedents that could determine general difficulties in paying attention to social stimuli in ASD subjects.

Effects of multimodal synchrony on infant attention and heart rate during events with social and nonsocial stimuli

Attention is a state of readiness or alertness, associated with behavioral and psychophysiological responses, that facilitates learning and memory. Multisensory and dynamic events have been shown to elicit more attention and produce greater sustained attention in infants than auditory or visual events alone. Such redundant and often temporally synchronous information guides selectivity and facilitates perception, learning, and memory of properties of events specified by redundancy. In addition, events involving faces or other social stimuli provide an extraordinary amount of redundant information that attracts and sustains attention. In the current study, 4- and 8-month-old infants were shown 2-min multimodal videos featuring social or nonsocial stimuli to determine the relative roles of synchrony and stimulus category in inducing attention. Behavioral measures included average looking time and peak look duration, and convergent measurement of heart rate (HR) allowed for the calculation of HR-defined phases of attention: Orienting (OR), sustained attention (SA), and attention termination (AT). The synchronous condition produced an earlier onset of SA (less time in OR) and a deeper state of SA than the asynchronous condition. Social stimuli attracted and held attention (longer duration of peak looks and lower HR than nonsocial stimuli). Effects of synchrony and the social nature of stimuli were additive, suggesting independence of their influence on attention. These findings are the first to demonstrate different HR-defined phases of attention as a function of intersensory redundancy, suggesting greater salience and deeper processing of naturalistic synchronous audiovisual events compared with asynchronous ones.