Speech segmentation is facilitated by visual cues

Audiovisual perception of interrupted speech by nonnative listeners

The purpose of the present study was to examine the influence of visual cues in audiovisual perception of interrupted speech by nonnative English listeners and to identify the role of working memory, long-term memory retrieval, and vocabulary knowledge in audiovisual perception by nonnative listeners. The participants included 31 Mandarin-speaking English learners between 19 and 41 years of age. The perceptual stimuli were noise-filled periodically interrupted AzBio and QuickSIN sentences with or without visual cues that showed a male speaker uttering the sentences. In addition to sentence recognition, the listeners completed a semantic fluency task, verbal (operation span) and visuospatial (symmetry span) working memory tasks, and two vocabulary knowledge tests (Vocabulary Level Test and Lexical Test for Advanced Learners of English). The results revealed significantly better speech recognition in the audio-visual condition than the audio-only condition, but the magnitude of visual benefit was substantially attenuated for sentences that had limited semantic context. The listeners' vocabulary size in English played a key role in the restoration of missing speech information and audiovisual integration in the perception of interrupted speech. Meanwhile, the listeners' verbal working memory capacity played an important role in audiovisual integration especially for the difficult stimuli with limited semantic context.

Colored valence in a lexical decision task

Color influences behavior, from the simplest to the most complex, through controlled and more automatic information elaboration processes. Nonetheless, little is known about how and when these highly interconnected processes interact. This study investigates the interaction between controlled and automatic processes during the processing of color information in a lexical decision task. Participants discriminated stimuli presented in different colors (red, blue, green) as words or pseudowords. Results showed that while color did not affect the faster and more accurate recognition of words compared to pseudowords, performance was influenced when examining words and pseudowords separately. Pseudowords were recognized faster when presented in blue or red, suggesting a potential influence of evolutionary color preferences when processing is not guided by more controlled processes. With words, emotional enhancement effects were found, with a preference for green independent of valence. These results suggest that controlled and more automatic processes do interact when processing color information according to stimulus type and task.

The effects of speaker and exemplar variability in children's cross-situational word learning

Cross-situational word learning (XSWL) - children's ability to learn words by tracking co-occurrence statistics of words and their referents over time - has been identified as a fundamental mechanism underlying lexical learning. However, it is unknown whether children can acquire new words when faced with variable input in XSWL paradigms, such as varying object exemplars and variable speakers. In the present study, we examine the separate and combined effects of exemplar and speaker variability on XSWL in typically developing English-speaking monolingual children. Results revealed that variability in speakers and exemplars did not facilitate or hinder XSWL performance. However, input that varied in both speakers and exemplars simultaneously did hinder children's word learning. Results from this work suggest that XSWL mechanisms may support categorization and generalization beyond word-object associations, but that accommodating multiple forms of variable input may incur costs. Overall, this research provides new theoretical insights into how fundamental mechanisms of word learning scale to more complex and naturalistic forms of input.

Rhythmically Modulating Neural Entrainment during Exposure to Regularities Influences Statistical Learning

The ability to discover regularities in the environment, such as syllable patterns in speech, is known as statistical learning. Previous studies have shown that statistical learning is accompanied by neural entrainment, in which neural activity temporally aligns with repeating patterns over time. However, it is unclear whether these rhythmic neural dynamics play a functional role in statistical learning or whether they largely reflect the downstream consequences of learning, such as the enhanced perception of learned words in speech. To better understand this issue, we manipulated participants' neural entrainment during statistical learning using continuous rhythmic visual stimulation. Participants were exposed to a speech stream of repeating nonsense words while viewing either (1) a visual stimulus with a "congruent" rhythm that aligned with the word structure, (2) a visual stimulus with an incongruent rhythm, or (3) a static visual stimulus. Statistical learning was subsequently measured using both an explicit and implicit test. Participants in the congruent condition showed a significant increase in neural entrainment over auditory regions at the relevant word frequency, over and above effects of passive volume conduction, indicating that visual stimulation successfully altered neural entrainment within relevant neural substrates. Critically, during the subsequent implicit test, participants in the congruent condition showed an enhanced ability to predict upcoming syllables and stronger neural phase synchronization to component words, suggesting that they had gained greater sensitivity to the statistical structure of the speech stream relative to the incongruent and static groups. This learning benefit could not be attributed to strategic processes, as participants were largely unaware of the contingencies between the visual stimulation and embedded words. These results indicate that manipulating neural entrainment during exposure to regularities influences statistical learning outcomes, suggesting that neural entrainment may functionally contribute to statistical learning. Our findings encourage future studies using non-invasive brain stimulation methods to further understand the role of entrainment in statistical learning.

Segmenting Speech by Mouth: The Role of Oral Prosodic Cues for Visual Speech Segmentation

Adults are able to use visual prosodic cues in the speaker's face to segment speech. Furthermore, eye-tracking data suggest that learners will shift their gaze to the mouth during visual speech segmentation. Although these findings suggest that the mouth may be viewed more than the eyes or nose during visual speech segmentation, no study has examined the direct functional importance of individual features; thus, it is unclear which visual prosodic cues are important for word segmentation. In this study, we examined the impact of first removing (Experiment 1) and then isolating (Experiment 2) individual facial features on visual speech segmentation. Segmentation performance was above chance in all conditions except for when the visual display was restricted to the eye region (eyes only condition in Experiment 2). This suggests that participants were able to segment speech when they could visually access the mouth but not when the mouth was completely removed from the visual display, providing evidence that visual prosodic cues conveyed by the mouth are sufficient and likely necessary for visual speech segmentation.

Speech Segmentation and Cross-Situational Word Learning in Parallel

Language learners track conditional probabilities to find words in continuous speech and to map words and objects across ambiguous contexts. It remains unclear, however, whether learners can leverage the structure of the linguistic input to do both tasks at the same time. To explore this question, we combined speech segmentation and cross-situational word learning into a single task. In Experiment 1, when adults (N = 60) simultaneously segmented continuous speech and mapped the newly segmented words to objects, they demonstrated better performance than when either task was performed alone. However, when the speech stream had conflicting statistics, participants were able to correctly map words to objects, but were at chance level on speech segmentation. In Experiment 2, we used a more sensitive speech segmentation measure to find that adults (N = 35), exposed to the same conflicting speech stream, correctly identified non-words as such, but were still unable to discriminate between words and part-words. Again, mapping was above chance. Our study suggests that learners can track multiple sources of statistical information to find and map words to objects in noisy environments. It also prompts questions on how to effectively measure the knowledge arising from these learning experiences.

Peer attitudes towards adolescents with speech disorders due to cleft lip and palate

BACKGROUND AND AIMS

Individuals with speech disorders are often judged more negatively than peers without speech disorders. A limited number of studies examined the attitudes of adolescents toward peers with speech disorders due to a cleft lip with or without a cleft of the palate (CL ± P). Therefore, the aim of the present study was to investigate the attitudes of peers toward the speech of adolescents with CL ± P.

METHOD

Seventy-eight typically developing adolescents (15-18 years, 26 boys, 52 girls) judged audio and audiovisual samples of two adolescents with CL ± P based on three attitude components, i.e., cognitive, affective, and behavioral. The degree of speech intelligibility was also scored by their peers. The study investigated whether the three attitudes were determined by speech intelligibility or appearance of an individual with CL ± P. Furthermore, the influence of knowing someone with a cleft, the age, and gender of the listeners on their attitudes were explored.

RESULTS

A significantly positive correlation was found between the speech intelligibility percentage and the three different attitude components: more positive attitudes were observed when the speech intelligibility of the speaker was higher. A different appearance due to a cleft lip does not lead to more negative attitudes. Furthermore, boys seem to have more negative attitudes toward individuals with CL ± P compared to girls.

CONCLUSION

This study provided additional evidence that peers show more negative attitudes toward adolescents with less intelligible speech due to CL ± P. Intervention should focus on changing the cognitive, affective, and behavioral attitudes of peers in a more positive direction and remove the stigma of patients with a cleft. Further research is needed to verify these results.

Word Segmentation Cues in German Child-Directed Speech: A Corpus Analysis

To acquire language, infants must learn to segment words from running speech. A significant body of experimental research shows that infants use multiple cues to do so; however, little research has comprehensively examined the distribution of such cues in naturalistic speech. We conducted a comprehensive corpus analysis of German child-directed speech (CDS) using data from the Child Language Data Exchange System (CHILDES) database, investigating the availability of word stress, transitional probabilities (TPs), and lexical and sublexical frequencies as potential cues for word segmentation. Seven hours of data (~15,000 words) were coded, representing around an average day of speech to infants. The analysis revealed that for 97% of words, primary stress was carried by the initial syllable, implicating stress as a reliable cue to word onset in German CDS. Word identity was also marked by TPs between syllables, which were higher within than between words, and higher for backwards than forwards transitions. Words followed a Zipfian-like frequency distribution, and over two-thirds of words (78%) were monosyllabic. Of the 50 most frequent words, 82% were function words, which accounted for 47% of word tokens in the entire corpus. Finally, 15% of all utterances comprised single words. These results give rich novel insights into the availability of segmentation cues in German CDS, and support the possibility that infants draw on multiple converging cues to segment their input. The data, which we make openly available to the research community, will help guide future experimental investigations on this topic.

Voice modulatory cues to structure across languages and species

Voice modulatory cues such as variations in fundamental frequency, duration and pauses are key factors for structuring vocal signals in human speech and vocal communication in other tetrapods. Voice modulation physiology is highly similar in humans and other tetrapods due to shared ancestry and shared functional pressures for efficient communication. This has led to similarly structured vocalizations across humans and other tetrapods. Nonetheless, in their details, structural characteristics may vary across species and languages. Because data concerning voice modulation in non-human tetrapod vocal production and especially perception are relatively scarce compared to human vocal production and perception, this review focuses on voice modulatory cues used for speech segmentation across human languages, highlighting comparative data where available. Cues that are used similarly across many languages may help indicate which cues may result from physiological or basic cognitive constraints, and which cues may be employed more flexibly and are shaped by cultural evolution. This suggests promising candidates for future investigation of cues to structure in non-human tetrapod vocalizations. This article is part of the theme issue 'Voice modulation: from origin and mechanism to social impact (Part I)'.

Oscillatory activity and EEG phase synchrony of concurrent word segmentation and meaning-mapping in 9-year-old children

When learning a new language, one must segment words from continuous speech and associate them with meanings. These complex processes can be boosted by attentional mechanisms triggered by multi-sensory information. Previous electrophysiological studies suggest that brain oscillations are sensitive to different hierarchical complexity levels of the input, making them a plausible neural substrate for speech parsing. Here, we investigated the functional role of brain oscillations during concurrent speech segmentation and meaning acquisition in sixty 9-year-old children. We collected EEG data during an audio-visual statistical learning task during which children were exposed to a learning condition with consistent word-picture associations and a random condition with inconsistent word-picture associations before being tested on their ability to recall words and word-picture associations. We capitalized on the brain dynamics to align neural activity to the same rate as an external rhythmic stimulus to explore modulations of neural synchronization and phase synchronization between electrodes during multi-sensory word learning. Results showed enhanced power at both word- and syllabic-rate and increased EEG phase synchronization between frontal and occipital regions in the learning compared to the random condition. These findings suggest that multi-sensory cueing and attentional mechanisms play an essential role in children's successful word learning.

Auditory closure with visual cues: Relationship with working memory and semantic memory

The role of working memory (WM) and long-term lexical-semantic memory (LTM) in the perception of interrupted speech with and without visual cues, was studied in 29 native English speakers. Perceptual stimuli were periodically interrupted sentences filled with speech noise. The memory measures included an LTM semantic fluency task, verbal WM, and visuo-spatial WM tasks. Whereas perceptual performance in the audio-only condition demonstrated a significant positive association with listeners' semantic fluency, perception in audio-video mode did not. These results imply that when listening to distorted speech without visual cues, listeners rely on lexical-semantic retrieval from LTM to restore missing speech information.

Why Does Dual-Tasking Hamper Implicit Sequence Learning?

Research on the limitations of dual-tasking might profit from using setups with a predictable sequence of stimuli and responses and assessing the acquisition of this sequence. Detrimental effects of dual-tasking on implicit sequence learning in the serial reaction time task (SRTT; Nissen & Bullemer, 1987) - when paired with an uncorrelated task - have been attributed to participants' lack of separating the streams of events in either task. Assuming that co-occurring events are automatically integrated, we reasoned that participants could need to first learn which events co-occur, before they can acquire sequence knowledge. In the training phase, we paired an 8-element visual-manual SRTT with an auditory-vocal task. Afterwards, we tested under single-tasking conditions whether SRTT sequence knowledge had been acquired. By applying different variants of probabilistic SRTT-tone pairings across three experiments, we tested what type of predictive relationship was needed to preserve sequence learning. In Experiment 1, where half of the SRTT-elements were paired to 100% with one specific tone and the other half randomly, only the fixedly paired elements were learned. Yet, no sequence learning was found when each of the eight SRTT-elements was paired with tone identity in a 75%-25% ratio (Experiment 2). Sequence learning was, however, intact when the 75%-25% ratio was applied to the four SRTT target locations instead (Experiment 3). The results suggest that participants (when lacking a separation of the task representations while dual-tasking) can learn a sequence inherent in one of two tasks to the extent that across-task contingencies can be learned first.

Rapid acquisition of novel written word-forms: ERP evidence

Background

Novel word acquisition is generally believed to be a rapid process, essential for ensuring a flexible and efficient communication system; at least in spoken language, learners are able to construct memory traces for new linguistic stimuli after just a few exposures. However, such rapid word learning has not been systematically found in visual domain, with different confounding factors obscuring the orthographic learning of novel words. This study explored the changes in human brain activity occurring online, during a brief training with novel written word-forms using a silent reading task

Results

Single-trial, cluster-based random permutation analysis revealed that training caused an extremely fast (after just one repetition) and stable facilitation in novel word processing, reflected in the modulation of P200 and N400 components, possibly indicating rapid dynamics at early and late stages of the lexical processing. Furthermore, neural source estimation of these effects revealed the recruitment of brain areas involved in orthographic and lexico-semantic processing, respectively.

Conclusions

These results suggest the formation of neural memory traces for novel written word-forms after a minimal exposure to them even in the absence of a semantic reference, resembling the rapid learning processes known to occur in spoken language.

Finding Phrases: The Interplay of Word Frequency, Phrasal Prosody and Co-speech Visual Information in Chunking Speech by Monolingual and Bilingual Adults

The audiovisual speech signal contains multimodal information to phrase boundaries. In three artificial language learning studies with 12 groups of adult participants we investigated whether English monolinguals and bilingual speakers of English and a language with opposite basic word order (i.e., in which objects precede verbs) can use word frequency, phrasal prosody and co-speech (facial) visual information, namely head nods, to parse unknown languages into phrase-like units. We showed that monolinguals and bilinguals used the auditory and visual sources of information to chunk "phrases" from the input. These results suggest that speech segmentation is a bimodal process, though the influence of co-speech facial gestures is rather limited and linked to the presence of auditory prosody. Importantly, a pragmatic factor, namely the language of the context, seems to determine the bilinguals' segmentation, overriding the auditory and visual cues and revealing a factor that begs further exploration.

How does the brain learn environmental structure? Ten core principles for understanding the neurocognitive mechanisms of statistical learning

Despite a growing body of research devoted to the study of how humans encode environmental patterns, there is still no clear consensus about the nature of the neurocognitive mechanisms underpinning statistical learning nor what factors constrain or promote its emergence across individuals, species, and learning situations. Based on a review of research examining the roles of input modality and domain, input structure and complexity, attention, neuroanatomical bases, ontogeny, and phylogeny, ten core principles are proposed. Specifically, there exist two sets of neurocognitive mechanisms underlying statistical learning. First, a "suite" of associative-based, automatic, modality-specific learning mechanisms are mediated by the general principle of cortical plasticity, which results in improved processing and perceptual facilitation of encountered stimuli. Second, an attention-dependent system, mediated by the prefrontal cortex and related attentional and working memory networks, can modulate or gate learning and is necessary in order to learn nonadjacent dependencies and to integrate global patterns across time. This theoretical framework helps clarify conflicting research findings and provides the basis for future empirical and theoretical endeavors.

Finding phrases: On the role of co-verbal facial information in learning word order in infancy

The input contains perceptually available cues, which might allow young infants to discover abstract properties of the target language. Thus, word frequency and prosodic prominence correlate systematically with basic word order in natural languages. Prelexical infants are sensitive to these frequency-based and prosodic cues, and use them to parse new input into phrases that follow the order characteristic of their native languages. Importantly, young infants readily integrate auditory and visual facial information while processing language. Here, we ask whether co-verbal visual information provided by talking faces also helps prelexical infants learn the word order of their native language in addition to word frequency and prosodic prominence. We created two structurally ambiguous artificial languages containing head nods produced by an animated avatar, aligned or misaligned with the frequency-based and prosodic information. During 4 minutes, two groups of 4- and 8-month-old infants were familiarized with the artificial language containing aligned auditory and visual cues, while two further groups were exposed to the misaligned language. Using a modified Headturn Preference Procedure, we tested infants’ preference for test items exhibiting the word order of the native language, French, vs. the opposite word order. At 4 months, infants had no preference, suggesting that 4-month-olds were not able to integrate the three available cues, or had not yet built a representation of word order. By contrast, 8-month-olds showed no preference when auditory and visual cues were aligned and a preference for the native word order when visual cues were misaligned. These results imply that infants at this age start to integrate the co-verbal visual and auditory cues.

Superior learning in synesthetes: Consistent grapheme-color associations facilitate statistical learning

In synesthesia activation in one sensory domain, such as smell or sound, triggers an involuntary and unusual secondary sensory or cognitive experience. In the present study, we ask whether the added sensory experience of synesthesia can aid statistical learning-the ability to track environmental regularities in order to segment continuous information. To investigate this, we measured statistical learning outcomes, using an aurally presented artificial language, in two groups of synesthetes alongside controls and simulated the multimodal experience of synesthesia in non-synesthetes. One group of synesthetes exclusively had grapheme-color (GC) synesthesia, in which the experience of color is automatically triggered by exposure to written or spoken graphemes. The other group had both grapheme-color and sound-color (SC+) synesthesia, in which the experience of color is also triggered by the waveform properties of a voice, such as pitch, timbre, and/or musical chords. Unlike GC-only synesthetes, the experience of color in the SC+ group is not perfectly consistent with the statistics that signal word boundaries. We showed that GC-only synesthetes outperformed both non-synesthetes and SC+ synesthetes, likely because the visual concurrents for GC-only synesthetes are highly consistent with the artificial language. We further observed that our simulations of GC synesthesia, but not SC+ synesthesia produced superior statistical learning, showing that synesthesia likely boosts learning outcomes by providing a consistent secondary cue. Findings are discussed with regard to how multimodal experience can improve learning, with the present data indicating that this boost is more likely to occur through explicit, as opposed to implicit, learning systems.

Lexical knowledge boosts statistically-driven speech segmentation

The hypothesis that known words can serve as anchors for discovering new words in connected speech has computational and empirical support. However, evidence for how the bootstrapping effect of known words interacts with other mechanisms of lexical acquisition, such as statistical learning, is incomplete. In 3 experiments, we investigated the consequences of introducing a known word in an artificial language with no segmentation cues other than cross-syllable transitional probabilities. We started with an artificial language containing 4 trisyllabic novel words and observed standard above-chance performance in a subsequent recognition memory task. We then replaced 1 of the 4 novel words with a real word (tomorrow) and noted improved segmentation of the other 3 novel words. This improvement was maintained when the real word was a different length to the novel words (philosophy), ruling out an explanation based on metrical expectation. The improvement was also maintained when the word was added to the 4 original novel words rather than replacing 1 of them. Together, these results show that known words in an otherwise meaningless stream serve as anchors for discovering new words. In interpreting the results, we contrast a mechanism where the lexical boost is merely the consequence of attending to the edges of known words, with a mechanism where known words enhance sensitivity to transitional probabilities more generally.

Multisensory Integration in Cochlear Implant Recipients

Speech perception is inherently a multisensory process involving integration of auditory and visual cues. Multisensory integration in cochlear implant (CI) recipients is a unique circumstance in that the integration occurs after auditory deprivation and the provision of hearing via the CI. Despite the clear importance of multisensory cues for perception, in general, and for speech intelligibility, specifically, the topic of multisensory perceptual benefits in CI users has only recently begun to emerge as an area of inquiry. We review the research that has been conducted on multisensory integration in CI users to date and suggest a number of areas needing further research. The overall pattern of results indicates that many CI recipients show at least some perceptual gain that can be attributable to multisensory integration. The extent of this gain, however, varies based on a number of factors, including age of implantation and specific task being assessed (e.g., stimulus detection, phoneme perception, word recognition). Although both children and adults with CIs obtain audiovisual benefits for phoneme, word, and sentence stimuli, neither group shows demonstrable gain for suprasegmental feature perception. Additionally, only early-implanted children and the highest performing adults obtain audiovisual integration benefits similar to individuals with normal hearing. Increasing age of implantation in children is associated with poorer gains resultant from audiovisual integration, suggesting a sensitive period in development for the brain networks that subserve these integrative functions, as well as length of auditory experience. This finding highlights the need for early detection of and intervention for hearing loss, not only in terms of auditory perception, but also in terms of the behavioral and perceptual benefits of audiovisual processing. Importantly, patterns of auditory, visual, and audiovisual responses suggest that underlying integrative processes may be fundamentally different between CI users and typical-hearing listeners. Future research, particularly in low-level processing tasks such as signal detection will help to further assess mechanisms of multisensory integration for individuals with hearing loss, both with and without CIs.

Lack of Cross-Modal Effects in Dual-Modality Implicit Statistical Learning

A current controversy in the area of implicit statistical learning (ISL) is whether this process consists of a single, central mechanism or multiple modality-specific ones. To provide insight into this question, the current study involved three ISL experiments to explore whether multimodal input sources are processed separately in each modality or are integrated together across modalities. In Experiment 1, visual and auditory ISL were measured under unimodal conditions, with the results providing a baseline level of learning for subsequent experiments. Visual and auditory sequences were presented separately, and the underlying grammar used for both modalities was the same. In Experiment 2, visual and auditory sequences were presented simultaneously with each modality using the same artificial grammar to investigate whether redundant multisensory information would result in a facilitative effect (i.e., increased learning) compared to the baseline. In Experiment 3, visual and auditory sequences were again presented simultaneously but this time with each modality employing different artificial grammars to investigate whether an interference effect (i.e., decreased learning) would be observed compared to the baseline. Results showed that there was neither a facilitative learning effect in Experiment 2 nor an interference effect in Experiment 3. These findings suggest that participants were able to track simultaneously and independently two sets of sequential regularities under dual-modality conditions. These findings are consistent with the theories that posit the existence of multiple, modality-specific ISL mechanisms rather than a single central one.

Semantic and phonological schema influence spoken word learning and overnight consolidation

We studied the initial acquisition and overnight consolidation of new spoken words that resemble words in the native language (L1) or in an unfamiliar, non-native language (L2). Spanish-speaking participants learned the spoken forms of novel words in their native language (Spanish) or in a different language (Hungarian), which were paired with pictures of familiar or unfamiliar objects, or no picture. We thereby assessed, in a factorial way, the impact of existing knowledge (schema) on word learning by manipulating both semantic (familiar vs unfamiliar objects) and phonological (L1- vs L2-like novel words) familiarity. Participants were trained and tested with a 12-hr intervening period that included overnight sleep or daytime awake. Our results showed (1) benefits of sleep to recognition memory that were greater for words with L2-like phonology and (2) that learned associations with familiar but not unfamiliar pictures enhanced recognition memory for novel words. Implications for complementary systems accounts of word learning are discussed.

Word segmentation from noise-band vocoded speech

Spectral degradation reduces access to the acoustics of spoken language and compromises how learners break into its structure. We hypothesised that spectral degradation disrupts word segmentation, but that listeners can exploit other cues to restore detection of words. Normal-hearing adults were familiarised to artificial speech that was unprocessed or spectrally degraded by noise-band vocoding into 16 or 8 spectral channels. The monotonic speech stream was pause-free (Experiment 1), interspersed with isolated words (Experiment 2), or slowed by 33% (Experiment 3). Participants were tested on segmentation of familiar vs. novel syllable sequences and on recognition of individual syllables. As expected, vocoding hindered both word segmentation and syllable recognition. The addition of isolated words, but not slowed speech, improved segmentation. We conclude that syllable recognition is necessary but not sufficient for successful word segmentation, and that isolated words can facilitate listeners' access to the structure of acoustically degraded speech.

Cross-situational word learning in aphasia

Human learners can resolve referential ambiguity and discover the relationships between words and meanings through a cross-situational learning (CSL) strategy. Some people with aphasia (PWA) can learn word-referent pairings under referential uncertainty supported by online feedback. However, it remains unknown whether PWA can learn new words cross-situationally and if such learning ability is supported by statistical learning (SL) mechanisms. The present study examined whether PWA can learn novel word-referent mappings in a CSL task without feedback. We also studied whether CSL is related to SL in PWA and neurologically healthy individuals. We further examined whether aphasia severity, phonological processing and verbal short-term memory (STM) predict CSL in aphasia, and also whether individual differences in verbal STM modulate CSL in healthy older adults. Sixteen people with chronic aphasia underwent a CSL task that involved exposure to a series of individually ambiguous learning trials and a SL task that taps speech segmentation. Their learning ability was compared to 18 older controls and 39 young adults recruited for task validation. CSL in the aphasia group was below the older controls and young adults and took place at a slower rate. Importantly, we found a strong association between SL and CSL performance in all three groups. CSL was modulated by aphasia severity in the aphasia group, and by verbal STM capacity in the older controls. Our findings indicate that some PWA can preserve the ability to learn new word-referent associations cross-situationally. We suggest that both PWA and neurologically intact individuals may rely on SL mechanisms to achieve CSL and that verbal STM also influences CSL. These findings contribute to the ongoing debate on the cognitive mechanisms underlying this learning ability.

Neurophysiological evidence for the interplay of speech segmentation and word-referent mapping during novel word learning

Learning a new language requires the identification of word units from continuous speech (the speech segmentation problem) and mapping them onto conceptual representation (the word to world mapping problem). Recent behavioral studies have revealed that the statistical properties found within and across modalities can serve as cues for both processes. However, segmentation and mapping have been largely studied separately, and thus it remains unclear whether both processes can be accomplished at the same time and if they share common neurophysiological features. To address this question, we recorded EEG of 20 adult participants during both an audio alone speech segmentation task and an audiovisual word-to-picture association task. The participants were tested for both the implicit detection of online mismatches (structural auditory and visual semantic violations) as well as for the explicit recognition of words and word-to-picture associations. The ERP results from the learning phase revealed a delayed learning-related fronto-central negativity (FN400) in the audiovisual condition compared to the audio alone condition. Interestingly, while online structural auditory violations elicited clear MMN/N200 components in the audio alone condition, visual-semantic violations induced meaning-related N400 modulations in the audiovisual condition. The present results support the idea that speech segmentation and meaning mapping can take place in parallel and act in synergy to enhance novel word learning.

Cross-Domain Statistical-Sequential Dependencies Are Difficult to Learn

Recent studies have demonstrated participants' ability to learn cross-modal associations during statistical learning tasks. However, these studies are all similar in that the cross-modal associations to be learned occur simultaneously, rather than sequentially. In addition, the majority of these studies focused on learning across sensory modalities but not across perceptual categories. To test both cross-modal and cross-categorical learning of sequential dependencies, we used an artificial grammar learning task consisting of a serial stream of auditory and/or visual stimuli containing both within- and cross-domain dependencies. Experiment 1 examined within-modal and cross-modal learning across two sensory modalities (audition and vision). Experiment 2 investigated within-categorical and cross-categorical learning across two perceptual categories within the same sensory modality (e.g., shape and color; tones and non-words). Our results indicated that individuals demonstrated learning of the within-modal and within-categorical but not the cross-modal or cross-categorical dependencies. These results stand in contrast to the previous demonstrations of cross-modal statistical learning, and highlight the presence of modality constraints that limit the effectiveness of learning in a multimodal environment.

Differential Gaze Patterns on Eyes and Mouth During Audiovisual Speech Segmentation

Speech is inextricably multisensory: both auditory and visual components provide critical information for all aspects of speech processing, including speech segmentation, the visual components of which have been the target of a growing number of studies. In particular, a recent study (Mitchel and Weiss, 2014) established that adults can utilize facial cues (i.e., visual prosody) to identify word boundaries in fluent speech. The current study expanded upon these results, using an eye tracker to identify highly attended facial features of the audiovisual display used in Mitchel and Weiss (2014). Subjects spent the most time watching the eyes and mouth. A significant trend in gaze durations was found with the longest gaze duration on the mouth, followed by the eyes and then the nose. In addition, eye-gaze patterns changed across familiarization as subjects learned the word boundaries, showing decreased attention to the mouth in later blocks while attention on other facial features remained consistent. These findings highlight the importance of the visual component of speech processing and suggest that the mouth may play a critical role in visual speech segmentation.

Discovering functional units in continuous speech

Language learning requires that listeners discover acoustically variable functional units like phonetic categories and words from an unfamiliar, continuous acoustic stream. Although many category learning studies have examined how listeners learn to generalize across the acoustic variability inherent in the signals that convey the functional units of language, these studies have tended to focus upon category learning across isolated sound exemplars. However, continuous input presents many additional learning challenges that may impact category learning. Listeners may not know the timescale of the functional unit, its relative position in the continuous input, or its relationship to other evolving input regularities. Moving laboratory-based studies of isolated category exemplars toward more natural input is important to modeling language learning, but very little is known about how listeners discover categories embedded in continuous sound. In 3 experiments, adult participants heard acoustically variable sound category instances embedded in acoustically variable and unfamiliar sound streams within a video game task. This task was inherently rich in multisensory regularities with the to-be-learned categories and likely to engage procedural learning without requiring explicit categorization, segmentation, or even attention to the sounds. After 100 min of game play, participants categorized familiar sound streams in which target words were embedded and generalized this learning to novel streams as well as isolated instances of the target words. The findings demonstrate that even without a priori knowledge, listeners can discover input regularities that have the best predictive control over the environment for both non-native speech and nonspeech signals, emphasizing the generality of the learning.

The effect of visual cues on top-down restoration of temporally interrupted speech, with and without further degradations

In complex listening situations, cognitive restoration mechanisms are commonly used to enhance perception of degraded speech with inaudible segments. Profoundly hearing-impaired people with a cochlear implant (CI) show less benefit from such mechanisms. However, both normal hearing (NH) listeners and CI users do benefit from visual speech cues in these listening situations. In this study we investigated if an accompanying video of the speaker can enhance the intelligibility of interrupted sentences and the phonemic restoration benefit, measured by an increase in intelligibility when the silent intervals are filled with noise. Similar to previous studies, restoration benefit was observed with interrupted speech without spectral degradations (Experiment 1), but was absent in acoustic simulations of CIs (Experiment 2) and was present again in simulations of electric-acoustic stimulation (Experiment 3). In all experiments, the additional speech information provided by the complementary visual cues lead to overall higher intelligibility, however, these cues did not influence the occurrence or extent of the phonemic restoration benefit of filler noise. Results imply that visual cues do not show a synergistic effect with the filler noise, as adding them equally increased the intelligibility of interrupted sentences with or without the filler noise.

The role of cross-modal associations in statistical learning

Our environment is richly structured, with objects producing correlated information within and across sensory modalities. A prominent challenge faced by our perceptual system is to learn such regularities. Here, we examined statistical learning and addressed learners' ability to track transitional probabilities between elements in the auditory and visual modalities. Specifically, we investigated whether cross-modal information affects statistical learning within a single modality. Participants were familiarized with a statistically structured modality (e.g., either audition or vision) accompanied by different types of cues in a second modality (e.g., vision or audition). The results revealed that statistical learning within either modality is affected by cross-modal information, with learning being enhanced or reduced according to the type of cue provided in the second modality.

Multimodal integration in statistical learning: evidence from the McGurk illusion

Recent advances in the field of statistical learning have established that learners are able to track regularities of multimodal stimuli, yet it is unknown whether the statistical computations are performed on integrated representations or on separate, unimodal representations. In the present study, we investigated the ability of adults to integrate audio and visual input during statistical learning. We presented learners with a speech stream synchronized with a video of a speaker’s face. In the critical condition, the visual (e.g., /gi/) and auditory (e.g., /mi/) signals were occasionally incongruent, which we predicted would produce the McGurk illusion, resulting in the perception of an audiovisual syllable (e.g., /ni/). In this way, we used the McGurk illusion to manipulate the underlying statistical structure of the speech streams, such that perception of these illusory syllables facilitated participants’ ability to segment the speech stream. Our results therefore demonstrate that participants can integrate audio and visual input to perceive the McGurk illusion during statistical learning. We interpret our findings as support for modality-interactive accounts of statistical learning.

Visual speech segmentation: using facial cues to locate word boundaries in continuous speech

Speech is typically a multimodal phenomenon, yet few studies have focused on the exclusive contributions of visual cues to language acquisition. To address this gap, we investigated whether visual prosodic information can facilitate speech segmentation. Previous research has demonstrated that language learners can use lexical stress and pitch cues to segment speech and that learners can extract this information from talking faces. Thus, we created an artificial speech stream that contained minimal segmentation cues and paired it with two synchronous facial displays in which visual prosody was either informative or uninformative for identifying word boundaries. Across three familiarisation conditions (audio stream alone, facial streams alone, and paired audiovisual), learning occurred only when the facial displays were informative to word boundaries, suggesting that facial cues can help learners solve the early challenges of language acquisition.

Statistical speech segmentation and word learning in parallel: scaffolding from child-directed speech

In order to acquire their native languages, children must learn richly structured systems with regularities at multiple levels. While structure at different levels could be learned serially, e.g., speech segmentation coming before word-object mapping, redundancies across levels make parallel learning more efficient. For instance, a series of syllables is likely to be a word not only because of high transitional probabilities, but also because of a consistently co-occurring object. But additional statistics require additional processing, and thus might not be useful to cognitively constrained learners. We show that the structure of child-directed speech makes simultaneous speech segmentation and word learning tractable for human learners. First, a corpus of child-directed speech was recorded from parents and children engaged in a naturalistic free-play task. Analyses revealed two consistent regularities in the sentence structure of naming events. These regularities were subsequently encoded in an artificial language to which adult participants were exposed in the context of simultaneous statistical speech segmentation and word learning. Either regularity was independently sufficient to support successful learning, but no learning occurred in the absence of both regularities. Thus, the structure of child-directed speech plays an important role in scaffolding speech segmentation and word learning in parallel.

Isolated words enhance statistical language learning in infancy

Infants are adept at tracking statistical regularities to identify word boundaries in pause-free speech. However, researchers have questioned the relevance of statistical learning mechanisms to language acquisition, since previous studies have used simplified artificial languages that ignore the variability of real language input. The experiments reported here embraced a key dimension of variability in infant-directed speech. English-learning infants (8-10 months) listened briefly to natural Italian speech that contained either fluent speech only or a combination of fluent speech and single-word utterances. Listening times revealed successful learning of the statistical properties of target words only when words appeared both in fluent speech and in isolation; brief exposure to fluent speech alone was not sufficient to facilitate detection of the words' statistical properties. This investigation suggests that statistical learning mechanisms actually benefit from variability in utterance length, and provides the first evidence that isolated words and longer utterances act in concert to support infant word segmentation.

Learning across senses: cross-modal effects in multisensory statistical learning

It is currently unknown whether statistical learning is supported by modality-general or modality-specific mechanisms. One issue within this debate concerns the independence of learning in one modality from learning in other modalities. In the present study, the authors examined the extent to which statistical learning across modalities is independent by simultaneously presenting learners with auditory and visual streams. After establishing baseline rates of learning for each stream independently, they systematically varied the amount of audiovisual correspondence across 3 experiments. They found that learners were able to segment both streams successfully only when the boundaries of the audio and visual triplets were in alignment. This pattern of results suggests that learners are able to extract multiple statistical regularities across modalities provided that there is some degree of cross-modal coherence. They discuss the implications of their results in light of recent claims that multisensory statistical learning is guided by modality-independent mechanisms.

Play, behaviour, language and social skills: the comparison of a play and a non-play intervention within a specialist school setting

The aim of the present study was to investigate the play, behaviour, language and social skills of children aged 5-8 years participating in a play intervention (based on the 'Learn to Play' program) compared to a group of children participating in traditional classroom activities within a specialist school over a six month period. Thirty-five children participated in the study, 19 participated in the play intervention group and 16 participated in the comparison group. Fourteen staff members at the special school were involved. A quasi-experimental design was used with pre and post data collection. Children in the play intervention and the comparison group were assessed using the Child-Initiated Pretend Play Assessment (play), Goal Attainment Scaling (behaviour), the Preschool Language Scale (language) and the Penn Interactive Peer Play Scale (social skills) at baseline and at follow up. Findings revealed that children participating in the play intervention showed a significant decrease in play deficits, became less socially disruptive and more socially connected with their peers. Both groups improved in their overall language skills and significantly improved in their goal attainment. This study supports the use of a play intervention in improving a child's play, behaviour, language and social skills.