Different neural processes underlie visual speech perception in school-age children and adults: An event-related potentials study

Long-term phonemic representations become audiovisual by mid-childhood

In earlier work with adults, we showed that long-term phonemic representations are audiovisual, meaning that they contain information on typical mouth shape during articulation. Many aspects of audiovisual processing have a prolonged developmental course, often not reaching maturity until late adolescence. In this study, we examined the status of phonemic representations in two groups of children - 8-9-year-olds and 11-12-year-olds. We used the same audiovisual oddball paradigm as in the earlier study with adults (Kaganovich and Christ, 2021). On each trial, participants saw a face and heard one of two vowels. One vowel occurred frequently (standard), while another occurred rarely (deviant). In one condition (neutral), the face had a closed, non-articulating mouth. In the other condition (audiovisual violation), the mouth shape matched the frequent vowel. Although stimuli were audiovisual in both conditions, we hypothesized that identical auditory changes would be perceived differently by participants. Namely, in the neutral condition, deviants violated only the audiovisual pattern specific to each experimental block. By contrast, in the audiovisual violation condition, deviants additionally violated long-term representations for how a speaker's mouth looks during articulation. We compared the amplitude of MMN and P3 components elicited by deviants in the two conditions. In the 11-12-year-old group, the pattern of neural responses was similar to that in adults - namely, they had a larger MMN component in the audiovisual compared to neutral condition, with no major difference in the P3 amplitude. In contrast, in the 8-9-year-old group, we saw a posterior MMN in the neutral condition only and a larger P3 in the audiovisual violation compared to the neutral condition. The larger P3 in the audiovisual violation condition suggests that younger children did perceive deviants as being more attention-grabbing when they violated the typical combination of sound and mouth shape. Yet, at this age, the earlier, more automatic stages of phonemic processing indexed by the MMN component may not yet encode visual speech elements the same way they do in older children and adults. We conclude that phonemic representations do not become audiovisual until 11-12 years of age.

Development of the Mechanisms Underlying Audiovisual Speech Perception Benefit

The natural environments in which infants and children learn speech and language are noisy and multimodal. Adults rely on the multimodal nature of speech to compensate for noisy environments during speech communication. Multiple mechanisms underlie mature audiovisual benefit to speech perception, including reduced uncertainty as to when auditory speech will occur, use of correlations between the amplitude envelope of auditory and visual signals in fluent speech, and use of visual phonetic knowledge for lexical access. This paper reviews evidence regarding infants' and children's use of temporal and phonetic mechanisms in audiovisual speech perception benefit. The ability to use temporal cues for audiovisual speech perception benefit emerges in infancy. Although infants are sensitive to the correspondence between auditory and visual phonetic cues, the ability to use this correspondence for audiovisual benefit may not emerge until age four. A more cohesive account of the development of audiovisual speech perception may follow from a more thorough understanding of the development of sensitivity to and use of various temporal and phonetic cues.

Electrophysiological Dynamics of Visual Speech Processing and the Role of Orofacial Effectors for Cross-Modal Predictions

The human brain generates predictions about future events. During face-to-face conversations, visemic information is used to predict upcoming auditory input. Recent studies suggest that the speech motor system plays a role in these cross-modal predictions, however, usually only audio-visual paradigms are employed. Here we tested whether speech sounds can be predicted on the basis of visemic information only, and to what extent interfering with orofacial articulatory effectors can affect these predictions. We registered EEG and employed N400 as an index of such predictions. Our results show that N400's amplitude was strongly modulated by visemic salience, coherent with cross-modal speech predictions. Additionally, N400 ceased to be evoked when syllables' visemes were presented backwards, suggesting that predictions occur only when the observed viseme matched an existing articuleme in the observer's speech motor system (i.e., the articulatory neural sequence required to produce a particular phoneme/viseme). Importantly, we found that interfering with the motor articulatory system strongly disrupted cross-modal predictions. We also observed a late P1000 that was evoked only for syllable-related visual stimuli, but whose amplitude was not modulated by interfering with the motor system. The present study provides further evidence of the importance of the speech production system for speech sounds predictions based on visemic information at the pre-lexical level. The implications of these results are discussed in the context of a hypothesized trimodal repertoire for speech, in which speech perception is conceived as a highly interactive process that involves not only your ears but also your eyes, lips and tongue.

Evaluating Preschool Visual Attentional Selective-Set: Preliminary ERP Modeling and Simulation of Target Enhancement Homology

We reanalyzed, modeled and simulated Event-Related Potential (ERP) data from 13 healthy children (Mean age = 5.12, Standard Deviation = 0.75) during a computerized visual sustained target detection task. Extending an ERP-based ACT–R (Adaptive Control of Thought–Rational) neurocognitive modeling approach, we tested whether visual sustained selective-set attention in preschool children involves the enhancement of neural response to targets, and it shows key adult-like features (neurofunctional homology). Blinded automatic peaks analysis was conducted on vincentized binned grand ERP averages. Time-course and distribution of scalp activity were detailed through topographic mapping and paths analysis. Reaction times and accuracy were also measured. Adult Magnetic Resonance Imaging-based mapping using ACT–R dipole source modeling and electric-field spiking simulation provided very good fit with the actual ERP data (R² > 0.70). In most electrodes, between 50 and 400 ms, ERPs concurrent with target presentation were enhanced relative to distractor, without manual response confounds. Triangulation of peak analysis, ACT–R modeling and simulation for the entire ERP epochs up to the moment of manual response (~700 ms, on average) suggested converging evidence of distinct but interacting processes of enhancement and planning for response release/inhibition, respectively. The latter involved functions and structures consistent with adult ERP activity which might correspond to a large-scale network, implicating Dorsal and Ventral Attentional Networks, corticostriatal loops, and subcortical hubs connected to prefrontal cortex top-down working memory executive control. Although preliminary, the present approach suggests novel directions for further tests and falsifiable hypotheses on the origins and development of visual selective attention and their ERP correlates.