Development of auditory scene analysis: a mini-review

Most auditory environments contain multiple sound waves that are mixed before reaching the ears. In such situations, listeners must disentangle individual sounds from the mixture, performing the auditory scene analysis. Analyzing complex auditory scenes relies on listeners ability to segregate acoustic events into different streams, and to selectively attend to the stream of interest. Both segregation and selective attention are known to be challenging for adults with normal hearing, and seem to be even more difficult for children. Here, we review the recent literature on the development of auditory scene analysis, presenting behavioral and neurophysiological results. In short, cognitive and neural mechanisms supporting stream segregation are functional from birth but keep developing until adolescence. Similarly, from 6 months of age, infants can orient their attention toward a target in the presence of distractors. However, selective auditory attention in the presence of interfering streams only reaches maturity in late childhood at the earliest. Methodological limitations are discussed, and a new paradigm is proposed to clarify the relationship between auditory scene analysis and speech perception in noise throughout development.

Seeing a Talking Face Matters: Gaze Behavior and the Auditory-Visual Speech Benefit in Adults' Cortical Tracking of Infant-directed Speech

In face-to-face conversations, listeners gather visual speech information from a speaker's talking face that enhances their perception of the incoming auditory speech signal. This auditory-visual (AV) speech benefit is evident even in quiet environments but is stronger in situations that require greater listening effort such as when the speech signal itself deviates from listeners' expectations. One example is infant-directed speech (IDS) presented to adults. IDS has exaggerated acoustic properties that are easily discriminable from adult-directed speech (ADS). Although IDS is a speech register that adults typically use with infants, no previous neurophysiological study has directly examined whether adult listeners process IDS differently from ADS. To address this, the current study simultaneously recorded EEG and eye-tracking data from adult participants as they were presented with auditory-only (AO), visual-only, and AV recordings of IDS and ADS. Eye-tracking data were recorded because looking behavior to the speaker's eyes and mouth modulates the extent of AV speech benefit experienced. Analyses of cortical tracking accuracy revealed that cortical tracking of the speech envelope was significant in AO and AV modalities for IDS and ADS. However, the AV speech benefit [i.e., AV > (A + V)] was only present for IDS trials. Gaze behavior analyses indicated differences in looking behavior during IDS and ADS trials. Surprisingly, looking behavior to the speaker's eyes and mouth was not correlated with cortical tracking accuracy. Additional exploratory analyses indicated that attention to the whole display was negatively correlated with cortical tracking accuracy of AO and visual-only trials in IDS. Our results underscore the nuances involved in the relationship between neurophysiological AV speech benefit and looking behavior.

Neural correlates of concurrent sound perception: A review and guidelines for future research

The perception of concurrent sound sources depends on processes (i.e., auditory scene analysis) that fuse and segregate acoustic features according to harmonic relations, temporal coherence, and binaural cues (encompass dichotic pitch, location difference, simulated echo). The object-related negativity (ORN) and P400 are electrophysiological indices of concurrent sound perception. Here, we review the different paradigms used to study concurrent sound perception and the brain responses obtained from these paradigms. Recommendations regarding the design and recording parameters of the ORN and P400 are made, and their clinical applications in assessing central auditory processing ability in different populations are discussed.

Seeing a Talking Face Matters: The Relationship between Cortical Tracking of Continuous Auditory-Visual Speech and Gaze Behaviour in Infants, Children and Adults

An auditory-visual speech benefit, the benefit that visual speech cues bring to auditory speech perception, is experienced from early on in infancy and continues to be experienced to an increasing degree with age. While there is both behavioural and neurophysiological evidence for children and adults, only behavioural evidence exists for infants - as no neurophysiological study has provided a comprehensive examination of the auditory-visual speech benefit in infants. It is also surprising that most studies on auditory-visual speech benefit do not concurrently report looking behaviour especially since the auditory-visual speech benefit rests on the assumption that listeners attend to a speaker's talking face and that there are meaningful individual differences in looking behaviour. To address these gaps, we simultaneously recorded electroencephalographic (EEG) and eye-tracking data of 5-month-olds, 4-year-olds and adults as they were presented with a speaker in auditory-only (AO), visual-only (VO), and auditory-visual (AV) modes. Cortical tracking analyses that involved forward encoding models of the speech envelope revealed that there was an auditory-visual speech benefit [i.e., AV > (A+V)], evident in 5-month-olds and adults but not 4-year-olds. Examination of cortical tracking accuracy in relation to looking behaviour, showed that infants' relative attention to the speaker's mouth (vs. eyes) was positively correlated with cortical tracking accuracy of VO speech, whereas adults' attention to the display overall was negatively correlated with cortical tracking accuracy of VO speech. This study provides the first neurophysiological evidence of auditory-visual speech benefit in infants and our results suggest ways in which current models of speech processing can be fine-tuned.

Long Latency Auditory Evoked Potentials and Object-Related Negativity Based on Harmonicity in Hearing-Impaired Children

Hearing-impaired children (HIC) have difficulty understanding speech in noise, which may be due to difficulty parsing concurrent sound object based on harmonicity cues. Using long latency auditory evoked potentials (LLAEPs) and object-related negativity (ORN), a neural metric of concurrent sound segregation, this study investigated the sensitivity of HIC in processing harmonic relation. The participants were 14 normal-hearing children (NHC) with an average age of 7.82 ± 1.31 years and 17 HIC with an average age of 7.98 ± 1.25 years. They were presented with a sequence of 200 Hz harmonic complex tones that had either all harmonic in tune or the third harmonic mistuned by 2%, 4%, 8%, and 16% of its original value while neuroelectric brain activity was recorded. The analysis of scalp-recorded LLAEPs revealed lower N2 amplitudes elicited by the tuned stimuli in HIC than control. The ORN, isolated in difference wave between LLAEP elicited by tuned and mistuned stimuli, was delayed and smaller in HIC than NHC. This study showed that deficits in processing harmonic relation in HIC, which may contribute to their difficulty in understanding speech in noise. As a result, top-down and bottom-up rehabilitations aiming to improve processing of basic acoustic characteristics, including harmonics are recommended for children with hearing loss.

Infants' Preference for Child-Directed Speech Over Time-Reversed Speech in On-Channel and Off-Channel Masking

Purpose This study aims to examine the development of auditory selective attention to speech in noise by examining the ability of infants to prefer child-directed speech (CDS) over time-reversed speech (TRS) presented in "on-channel" and "off-channel" noise. Method A total of 32 infants participated in the study. Sixteen typically developing infants were tested at 7 and 11 months of age using the central fixation procedure with CDS and TRS in two types of noise at +10 dB signal-to-noise ratio. One type of noise was an "on-channel" masker with a spectrum overlapping that of the CDS (energetic masking), and the second was an "off-channel" masker with frequencies that were outside the spectrum of the CDS (distractive masking). An additional group of sixteen 11-month-old infants were tested in quiet and served as controls for the "off-frequency" masker condition. Results Infants preferred CDS over TRS in both age groups, but this preference was more pronounced with "off-channel" masker regardless of age. Also, older infants demonstrated longer looking time for the target stimuli when presented with an "off-channel" masker compared to the "on-channel" masker. Looking time in quiet was similar to looking time in the "off-channel" condition, and looking time for CDS was longer in quiet compared to the "on-channel" condition. Conclusions These findings support the notion that (a) infants as young as 7 months of age are already showing preference for speech in noise, regardless of type of masker; (b) by 11 months of age, listening with the "off-channel" condition did not yield different results than in quiet. Thus, by 11 months of age, infants' cognitive-attentional abilities may be more developed.

Changes in electroencephalography complexity and functional magnetic resonance imaging connectivity following robotic hand training in chronic stroke

Introduction: In recent years, robotic training has been utilized for recovery of motor control in patients with motor deficits. Along with clinical assessment, electrical patterns in the brain have emerged as a marker for studying changes in the brain associated with brain injury and rehabilitation. These changes mainly involve an imbalance between the two hemispheres. We aimed to study the effect of brain computer interface (BCI)-based robotic hand training on stroke subjects using clinical assessment, electroencephalographic (EEG) complexity analysis, and functional magnetic resonance imaging (fMRI) connectivity analysis. Method: Resting-state simultaneous EEG-fMRI was conducted on 14 stroke subjects before and after training who underwent 20 sessions robot hand training. Fractal dimension (FD) analysis was used to assess neuronal impairment and functional recovery using the EEG data, and fMRI connectivity analysis was performed to assess changes in the connectivity of brain networks. Results: FD results indicated a significant asymmetric difference between the ipsilesional and contralesional hemispheres before training, which was reduced after robotic hand training. Moreover, a positive correlation between interhemispheric asymmetry change for central brain region and change in Fugl Meyer Assessment (FMA) scores for upper limb was observed. Connectivity results showed a significant difference between pre-training interhemispheric connectivity and post-training interhemispheric connectivity. Moreover, the change in connectivity correlated with the change in FMA scores. Results also indicated a correlation between the increase in connectivity for motor regions and decrease in FD interhemispheric asymmetry for central brain region covering the motor area. Conclusion: In conclusion, robotic hand training significantly facilitated stroke motor recovery, and FD, along with connectivity analysis can detect neuroplasticity changes.

Infants' use of isolated and combined temporal cues in speech sound segregation

This paper investigates infants' and adults' use of envelope cues and combined onset asynchrony and envelope cues in the segregation of concurrent vowels. Listeners heard superimposed vowel pairs consisting of two different vowels spoken by a male and a female talker and were trained to respond to one specific target vowel, either the male /u:/ or male /i:/. Vowel detection was measured in three conditions. In the baseline condition the two superimposed vowels had similar amplitude envelopes and synchronous onset. In the envelope cue condition, the amplitude envelopes of the two vowels differed. In the combined cue condition, both the onset time and amplitude envelopes of the two vowels differed. Seven-month-old infants' concurrent vowel segregation improved both with envelope and with combined onset asynchrony and envelope cues to the same extent as adults'. A preliminary investigation with 3-month-old infants suggested that neither envelope cues nor combined asynchrony and envelope cues improved their ability to detect the target vowel. Taken together, these results suggest that envelope and combined onset-asynchrony cues are available to infants as they attempt to process competing speech sounds, at least after 7 months of age.

Toward the Understanding of Topographical and Spectral Signatures of Infant Movement Artifacts in Naturalistic EEG

Electroencephalography (EEG) is perhaps the most widely used brain-imaging technique for pediatric populations. However, EEG signals are prone to distortion by motion. Compared to adults, infants' motion is both more frequent and less stereotypical yet motion effects on the infant EEG signal are largely undocumented. Here, we present a systematic assessment of naturalistic motion effects on the infant EEG signal. EEG recordings were performed with 14 infants (12 analyzed) who passively watched movies whilst spontaneously producing periods of bodily movement and rest. Each infant produced an average of 38.3 s (SD = 14.7 s) of rest and 18.8 s (SD = 17.9 s) of single motion segments for the final analysis. Five types of infant motions were analyzed: Jaw movements, and Limb movements of the Hand, Arm, Foot, and Leg. Significant movement-related distortions of the EEG signal were detected using cluster-based permutation analysis. This analysis revealed that, relative to resting state, infants' Jaw and Arm movements produced significant increases in beta (∼15 Hz) power, particularly over peripheral sites. Jaw movements produced more anteriorly located effects than Arm movements, which were most pronounced over posterior parietal and occipital sites. The cluster analysis also revealed trends toward decreased power in the theta and alpha bands observed over central topographies for all motion types. However, given the very limited quantity of infant data in this study, caution is recommended in interpreting these findings before subsequent replications are conducted. Nonetheless, this work is an important first step to inform future development of methods for addressing EEG motion-related artifacts. This work also supports wider use of naturalistic paradigms in social and developmental neuroscience.

14 challenges and their solutions for conducting social neuroscience and longitudinal EEG research with infants

The use of electroencephalography (EEG) to study infant brain development is a growing trend. In addition to classical longitudinal designs that study the development of neural, cognitive and behavioural functions, new areas of EEG application are emerging, such as novel social neuroscience paradigms using dual infant-adult EEG recordings. However, most of the experimental designs, analysis methods, as well as EEG hardware were originally developed for single-person adult research. When applied to study infant development, adult-based solutions often pose unique problems that may go unrecognised. Here, we identify 14 challenges that infant EEG researchers may encounter when designing new experiments, collecting data, and conducting data analysis. Challenges related to the experimental design are: (1) small sample size and data attrition, and (2) varying arousal in younger infants. Challenges related to data acquisition are: (3) determining the optimal location for reference and ground electrodes, (4) control of impedance when testing with the high-density sponge electrode nets, (5) poor fit of standard EEG caps to the varying infant head shapes, and (6) ensuring a high degree of temporal synchronisation between amplifiers and recording devices during dual-EEG acquisition. Challenges related to the analysis of longitudinal and social neuroscience datasets are: (7) developmental changes in head anatomy, (8) prevalence and diversity of infant myogenic artefacts, (9) a lack of stereotypical topography of eye movements needed for the ICA-based data cleaning, (10) and relatively high inter-individual variability of EEG responses in younger cohorts. Additional challenges for the analysis of dual EEG data are: (11) developmental shifts in canonical EEG rhythms and difficulties in differentiating true inter-personal synchrony from spurious synchrony due to (12) common intrinsic properties of the signal and (13) shared external perturbation. Finally, (14) there is a lack of test-retest reliability studies of infant EEG. We describe each of these challenges and suggest possible solutions. While we focus specifically on the social neuroscience and longitudinal research, many of the issues we raise are relevant for all fields of infant EEG research.

Infants use onset asynchrony cues in auditory scene analysis

This experiment investigated the effect of onset asynchrony on the segregation of concurrent vowels in infants and adults. Two vowels, randomly chosen from seven American-English vowels, were superimposed. Each vowel pair contained one vowel by a male and one by a female talker. A train of such vowel pairs was presented to listeners, who were trained to respond to the male target vowel /i:/ or /u:/. The ability to identify the target vowel was compared among three conditions: synchronous onset, 100-, and 200-ms onset asynchrony. Experiment 1 measured performance, in d', in 7-month-old infants and adults. Infants and adults performed better with asynchronous than synchronous vowel onset, regardless of asynchrony duration. Experiment 2 compared the proportion of 3-month-old infants achieving an 80% correct criterion with and without onset asynchrony. Significantly more infants reached criterion with asynchronous than with synchronous vowel onset. Asynchrony duration did not influence performance. These experiments show that infants, as young as 3 months old, benefit from onset asynchrony.

Infant-directed speech facilitates seven-month-old infants' cortical tracking of speech

This study assessed cortical tracking of temporal information in incoming natural speech in seven-month-old infants. Cortical tracking refers to the process by which neural activity follows the dynamic patterns of the speech input. In adults, it has been shown to involve attentional mechanisms and to facilitate effective speech encoding. However, in infants, cortical tracking or its effects on speech processing have not been investigated. This study measured cortical tracking of speech in infants and, given the involvement of attentional mechanisms in this process, cortical tracking of both infant-directed speech (IDS), which is highly attractive to infants, and the less captivating adult-directed speech (ADS), were compared. IDS is the speech register parents use when addressing young infants. In comparison to ADS, it is characterised by several acoustic qualities that capture infants' attention to linguistic input and assist language learning. Seven-month-old infants' cortical responses were recorded via electroencephalography as they listened to IDS or ADS recordings. Results showed stronger low-frequency cortical tracking of the speech envelope in IDS than in ADS. This suggests that IDS has a privileged status in facilitating successful cortical tracking of incoming speech which may, in turn, augment infants' early speech processing and even later language development.

Multisensory object perception in infancy: 4-month-olds perceive a mistuned harmonic as a separate auditory and visual object

Infants learn to use auditory and visual information to organize the sensory world into identifiable objects with particular locations. Here we use a behavioural method to examine infants' use of harmonicity cues to auditory object perception in a multisensory context. Sounds emitted by different objects sum in the air and the auditory system must figure out which parts of the complex waveform belong to different sources (auditory objects). One important cue to this source separation is that complex tones with pitch typically contain a fundamental frequency and harmonics at integer multiples of the fundamental. Consequently, adults hear a mistuned harmonic in a complex sound as a distinct auditory object (Alain, Theunissen, Chevalier, Batty, & Taylor, 2003). Previous work by our group demonstrated that 4-month-old infants are also sensitive to this cue. They behaviourally discriminate a complex tone with a mistuned harmonic from the same complex with in-tune harmonics, and show an object-related event-related potential (ERP) electrophysiological (EEG) response to the stimulus with mistuned harmonics. In the present study we use an audiovisual procedure to investigate whether infants perceive a complex tone with an 8% mistuned harmonic as emanating from two objects, rather than merely detecting the mistuned cue. We paired in-tune and mistuned complex tones with visual displays that contained either one or two bouncing balls. Four-month-old infants showed surprise at the incongruous pairings, looking longer at the display of two balls when paired with the in-tune complex and at the display of one ball when paired with the mistuned harmonic complex. We conclude that infants use harmonicity as a cue for source separation when integrating auditory and visual information in object perception.