Asymmetric dependencies in processing phonetic features

The Role of Talker Variability in Nonnative Phonetic Learning: A Systematic Review and Meta-Analysis

PURPOSE

High-variability phonetic training (HVPT) has been found to be effective on adult second language (L2) learning, but results are mixed in regards to the benefit of multiple talkers over single talker. This study provides a systematic review with meta-analysis to investigate the talker variability effect in nonnative phonetic learning and the factors moderating the effect.

METHOD

We collected studies with keyword search in major academic databases including EBSCO, ERIC, MEDLINE, ProQuest Dissertations & Theses, Elsevier, Scopus, Wiley Online Library, and Web of Science. We identified potential participant-, training-, and study-related moderators and conducted a random-effects model meta-analysis for each individual variable.

RESULTS

On the basis of 18 studies with a total of 549 participants, we obtained a small-level summary effect size (Hedges' g = 0.46, 95% confidence interval [CI; 0.08, 0.84]) for the immediate training outcomes, which was greatly reduced (g = -0.04, 95% CI [-0.46, 0.37]) after removal of outliers and correction for publication bias, whereas the effect size for immediate perceptual gains was nearly medium (g = 0.56, 95% CI [0.13, 1.00]) compared with the nonsignificant production gains. Critically, the summary effect sizes for generalizations to new talkers (g = 0.72, 95% CI [0.15, 1.29]) and for long-term retention (g = 1.09, 95% CI [0.39, 1.78]) were large. Moreover, the training program length and the talker presentation format were found to potentially moderate the immediate perceptual gains and generalization outcomes.

CONCLUSIONS

Our study presents the first meta-analysis on the role of talker variability in nonnative phonetic training, which demonstrates the heterogeneity and limitations of research on this topic. The results highlight the need for further investigation of the influential factors and underlying mechanisms for the presence or absence of talker variability effects. Supplemental Material https://doi.org/10.23641/asha.16959388.

Aero-tactile integration during speech perception: Effect of response and stimulus characteristics on syllable identification

Integration of auditory and aero-tactile information during speech perception has been documented during two-way closed-choice syllable classification tasks [Gick and Derrick (2009). Nature 462, 502-504], but not during an open-choice task using continuous speech perception [Derrick, O'Beirne, Gorden, De Rybel, Fiasson, and Hay (2016). J. Acoust. Soc. Am. 140(4), 3225]. This study was designed to compare audio-tactile integration during open-choice perception of individual syllables. In addition, this study aimed to compare the effects of place and manner of articulation. Thirty-four untrained participants identified syllables in both auditory-only and audio-tactile conditions in an open-choice paradigm. In addition, forty participants performed a closed-choice perception experiment to allow direct comparison between these two response-type paradigms. Adaptive staircases, as noted by Watson [(1983). Percept. Psychophys. 33(2), 113-120] were used to identify the signal-to-noise ratio for identification accuracy thresholds. The results showed no significant effect of air flow on syllable identification accuracy during the open-choice task, but found a bias towards voiceless identification of labials, and towards voiced identification of velars. Comparison of the open-choice results to those of the closed-choice task show a significant difference between both response types, with audio-tactile integration shown in the closed-choice task, but not in the open-choice task. These results suggest that aero-tactile enhancement of speech perception is dependent on response type demands.

Processing Relationships Between Language-Being-Spoken and Other Speech Dimensions in Monolingual and Bilingual Listeners

While indexical information is implicated in many levels of language processing, little is known about the internal structure of the system of indexical dimensions, particularly in bilinguals. A series of three experiments using the speeded classification paradigm investigated the relationship between various indexical and non-linguistic dimensions of speech in processing. Namely, we compared the relationship between a lesser-studied indexical dimension relevant to bilinguals, which language is being spoken (in these experiments, either Mandarin Chinese or English), with: talker identity (Experiment 1), talker gender (Experiment 2), and amplitude of speech (Experiment 3). Results demonstrate that language-being-spoken is integrated in processing with each of the other dimensions tested, and that these processing dependencies seem to be independent of listeners' bilingual status or experience with the languages tested. Moreover, the data reveal processing interference asymmetries, suggesting a processing hierarchy for indexical, non-linguistic speech features.

Speaker Variations Influence Speechreading Speed for Dynamic Faces

We investigated the influence of task-irrelevant speaker variations on speechreading performance. In three experiments with video digitised faces presented either in dynamic, static-sequential, or static mode, participants performed speeded classifications on vowel utterances (German vowels /u/ and /i/). A Garner interference paradigm was used, in which speaker identity was task-irrelevant but could be either correlated, constant, or orthogonal to the vowel uttered. Reaction times for facial speech classifications were slowed by task-irrelevant speaker variations for dynamic stimuli. The results are discussed with reference to distributed models of face perception (Haxby et al, 2000 Trends in Cognitive Sciences4 223–233) and the relevance of both dynamic information and speaker characteristics for speechreading.

Interdependent processing and encoding of speech and concurrent background noise

Speech processing can often take place in adverse listening conditions that involve the mixing of speech and background noise. In this study, we investigated processing dependencies between background noise and indexical speech features, using a speeded classification paradigm (Garner, 1974; Exp. 1), and whether background noise is encoded and represented in memory for spoken words in a continuous recognition memory paradigm (Exp. 2). Whether or not the noise spectrally overlapped with the speech signal was also manipulated. The results of Experiment 1 indicated that background noise and indexical features of speech (gender, talker identity) cannot be completely segregated during processing, even when the two auditory streams are spectrally nonoverlapping. Perceptual interference was asymmetric, whereby irrelevant indexical feature variation in the speech signal slowed noise classification to a greater extent than irrelevant noise variation slowed speech classification. This asymmetry may stem from the fact that speech features have greater functional relevance to listeners, and are thus more difficult to selectively ignore than background noise. Experiment 2 revealed that a recognition cost for words embedded in different types of background noise on the first and second occurrences only emerged when the noise and the speech signal were spectrally overlapping. Together, these data suggest integral processing of speech and background noise, modulated by the level of processing and the spectral separation of the speech and noise.

Effects of language experience and expectations on attention to consonants and tones in English and Mandarin Chinese

Both long-term native language experience and immediate linguistic expectations can affect listeners' use of acoustic information when making a phonetic decision. In this study, a Garner selective attention task was used to investigate differences in attention to consonants and tones by American English-speaking listeners (N = 20) and Mandarin Chinese-speaking listeners hearing speech in either American English (N = 17) or Mandarin Chinese (N = 20). To minimize the effects of lexical differences and differences in the linguistic status of pitch across the two languages, stimuli and response conditions were selected such that all tokens constitute legitimate words in both languages and all responses required listeners to make decisions that were linguistically meaningful in their native language. Results showed that regardless of ambient language, Chinese listeners processed consonant and tone in a combined manner, consistent with previous research. In contrast, English listeners treated tones and consonants as perceptually separable. Results are discussed in terms of the role of sub-phonemic differences in acoustic cues across language, and the linguistic status of consonants and pitch contours in the two languages.

Asymmetric processing of durational differences - electrophysiological investigations in Bengali

Duration is used contrastively in many languages to distinguish word meaning (e.g. in Bengali, [pata] 'leaf' vs. [pat:a] 'whereabouts'). While there is a large body of research on other contrasts in speech perception (e.g. vowel contrasts and consonantal place features), little work has been done on how durational information is used in speech processing. In non-linguistic studies of low-level processing, such as visual and non-linguistic acoustic pop-out tasks, an asymmetry is found where additional information is more readily detected than missing information. In this study, event-related potentials were recorded during two cross-modal auditory-visual semantic priming studies, where nonword mispronunciations of spoken prime words were created by changing the duration of a medial consonant (real word [dana] 'seed'>nonword [dan:a]). N400 amplitudes showed an opposite asymmetric pattern of results, where increases in consonantal duration were tolerated and led to priming of the visual target, but decreases in consonantal duration were not accepted. This asymmetrical pattern of acceptability is attributed to the fact that a longer consonant includes all essential information for the recognition of the original word with a short medial consonant (a possible default category) and any additional information can be ignored. However, when a consonant is shortened, it lacks the required durational information to activate the word with the original long consonant.

Syllable structure and integration of voicing and manner of articulation information in labial consonant identification

Speech perception requires the integration of information from multiple phonetic and phonological dimensions. A sizable literature exists on the relationships between multiple phonetic dimensions and single phonological dimensions (e.g., spectral and temporal cues to stop consonant voicing). A much smaller body of work addresses relationships between phonological dimensions, and much of this has focused on sequences of phones. However, strong assumptions about the relevant set of acoustic cues and/or the (in)dependence between dimensions limit previous findings in important ways. Recent methodological developments in the general recognition theory framework enable tests of a number of these assumptions and provide a more complete model of distinct perceptual and decisional processes in speech sound identification. A hierarchical Bayesian Gaussian general recognition theory model was fit to data from two experiments investigating identification of English labial stop and fricative consonants in onset (syllable initial) and coda (syllable final) position. The results underscore the importance of distinguishing between conceptually distinct processing levels and indicate that, for individual subjects and at the group level, integration of phonological information is partially independent with respect to perception and that patterns of independence and interaction vary with syllable position.

Separability of prosodic phrase boundary and phonemic information

It was hypothesized that the retrieval of prosodic and phonemic information from the acoustic signal is facilitated when prosodic information is encoded by co-occurring suprasegmental cues. To test the hypothesis, two-choice speeded classification experiments were conducted, which examined processing interaction between prosodic phrase-boundary vs stop-place information in speakers of Southern British English. Results confirmed that the degree of interaction between boundary and stop-place information diminished when the pre-boundary vowel was signaled by duration and F(0), compared to when it was signaled by either duration or F(0) alone. It is argued that the relative ease of retrieval of prosodic and phonemic information arose from advantages of prosodic cue integration.

Asymmetric interference between sex and emotion in face perception

Previous research with speeded-response interference tasks modeled on the Garner paradigm has demonstrated that task-irrelevant variations in either emotional expression or facial speech do not interfere with identity judgments, but irrelevant variations in identity do interfere with expression and facial speech judgments. Sex, like identity, is a relatively invariant aspect of faces. Drawing on a recent model of face processing according to which invariant and changeable aspects of faces are represented in separate neurological systems, we predicted asymmetric interference between sex and emotion classification. The results of Experiment 1, in which the Garner paradigm was employed, confirmed this prediction: Emotion classifications were influenced by the sex of the faces, but sex classifications remained relatively unaffected by facial expression. A second experiment, in which the difficulty of the tasks was equated, corroborated these findings, indicating that differences in processing speed cannot account for the asymmetric relationship between facial emotion and sex processing. A third experiment revealed the same pattern of asymmetric interference through the use of a variant of the Simon paradigm. To the extent that Garner interference and Simon interference indicate interactions at perceptual and response-selection stages of processing, respectively, a challenge for face processing models is to show how the same asymmetric pattern of interference could occur at these different stages. The implications of these findings for the functional independence of the different components of face processing are discussed.

Hierarchical categorization of coarticulated phonemes: a theoretical analysis

This article is concerned with the question of how listeners recognize coarticulated phonemes. The problem is approached from a pattern classification perspective. First, the potential acoustical effects of coarticulation are defined in terms of the patterns that form the input to a classifier. Next, a categorization model called HICAT is introduced that incorporates hierarchical dependencies to optimally deal with this input. The model allows the position, orientation, and steepness of one phoneme boundary to depend on the perceived value of a neighboring phoneme. It is argued that, if listeners do behave like statistical pattern recognizers, they may use the categorization strategies incorporated in the model. The HICAT model is compared with existing categorization models, among which are the fuzzy-logical model of perception and Nearey's diphone-biased secondary-cue model. Finally, a method is presented by which categorization strategies that are likely to be used by listeners can be predicted from distributions of acoustical cues as they occur in natural speech.

Developmental course of auditory processing interactions: Garner interference and Simon interference

Previous research suggests that with increasing age children become more efficient in inhibiting conflicting responses and in resisting interference from irrelevant information. We assessed the abilities of 100 children (ages 3-16 years) and 20 adults to resist interference during the processing of 2 auditory dimensions of speech, namely the speaker's gender and spatial location. The degree of interference from irrelevant variability in either dimension did not vary with age. Apparently, young children do not have more difficulty in resisting interference when the nontarget and the target are both perceptual attributes. We also assessed the participants' abilities to inhibit conflicting task-irrelevant information from spatial location and to resist interference from spatial variability in the context of conflict. In the presence of conflicting task-irrelevant information, both interference effects declined significantly with age. Developmental change in auditory processing seems to vary as a function of (1) the nature of the target-nontarget combination and (2) the presence/absence of conflicting task-irrelevant information.

Asymmetric dependencies in perceiving identity and emotion: experiments with morphed faces

We investigated whether an asymmetric relationship between the perception of identity and emotional expressions in faces (Schweinberger & Soukup, 1998) may be related to differences in the relative processing speed of identity and expression information. Stimulus faces were morphed across identity within a given emotional expression, or were morphed across emotion within a given identity. In Experiment 1, consistent classifications of these images were demonstrated across a wide range of morphing, with only a relatively narrow category boundary. At the same time, classification reaction times (RTs) reflected the increased perceptual difficulty of the morphed images. In Experiment 2, we investigated the effects of variations in the irrelevant dimension on judgments of faces with respect to a relevant dimension, using a Garner-type speeded classification task. RTs for expression classifications were strongly influenced by irrelevant identity information. In contrast, RTs for identity classifications were unaffected by irrelevant expression information, and this held even for stimuli in which identity was more difficult and slower to discriminate than expression. This suggests that differences in processing speed cannot account for the asymmetric relationship between identity and emotion perception. Theoretical accounts proposing independence of identity and emotion perception are discussed in the light of these findings.

The encoding of rate and talker information during phonetic perception

The acoustic structure of the speech signal is extremely variable due to a variety of contextual factors, including talker characteristics and speaking rate. To account for the listener's ability to adjust to this variability, speech researchers have posited the existence of talker and rate normalization processes. The current study examined how the perceptual system encoded information about talker and speaking rate during phonetic perception. Experiments 1-3 examined this question, using a speeded classification paradigm developed by Garner (1974). The results of these experiments indicated that decisions about phonemic identity were affected by both talker and rate information: irrelevant variation in either dimension interfered with phonemic classification. While rate classification was also affected by phoneme variation, talker classification was not. Experiment 4 examined the impact of talker and rate variation on the voicing boundary under different blocking conditions. The results indicated that talker characteristics influenced the voicing boundary when talker variation occurred within a block of trials only under certain conditions. Rate variation, however, influenced the voicing boundary regardless of whether or not there was rate variation within a block of trials. The findings from these experiments indicate that phoneme and the rate information are encoded in an integral manner during speech perception, while talker characteristics are encoded separately.

Childhood hearing impairment: auditory and linguistic interactions during multidimensional speech processing

Children with mild-severe sensorineural hearing losses often use hearing aids and aural/oral language as their primary mode of communication, yet we know little about how speech is processed by these children. The purpose of this research was to investigate how the multidimensional information underlying accurate speech perception is processed by children with mild-severe hearing impairments. The processing of the auditory and linguistic dimensions of speech was assessed with a speeded selective-attention task (Garner, 1974a). Listeners were required to attend selectively to an auditory dimension (gender of the talker) and ignore a linguistic dimension (word) and vice versa. The hypothesis underlying the task is that performance for the target dimension will be unaffected by what is happening on the nontarget dimension if the dimensions are processed independently. On the other hand, if the dimensions are not processed independently, listeners will not be able to attend selectively and performance for the relevant dimension will be affected by what is happening on the irrelevant dimension (termed "Garner" interference). Both children with normal hearing (N = 90) and children with hearing impairment (N = 40) showed some degree of Garner interference, implying that the dimensions of speech are not processed independently by these children. However, relative to the children with normal hearing, the children with hearing impairment showed normal Garner interference when attending selectively to the word dimension (normally effective at ignoring talker-gender input) and reduced Garner interference when attending selectively to the talker-gender dimension (more effective at ignoring word input). This pattern of results implies that the auditory dimension has a normal strength-of-processing level that makes it normally distracting and that the linguistic dimension has an underdeveloped strength-of-processing level that makes it easier to ignore in children with hearing impairment.

Developmental trends in the interaction between auditory and linguistic processing

The developmental course of multidimensional speech processing was examined in 80 children between 3 and 6 years of age and in 60 adults between 20 and 86 years of age. Processing interactions were assessed with a speeded classification task (Garner, 1974a), which required the subjects to attend selectively to the voice dimension while ignoring the linguistic dimension, and vice versa. The children and adults exhibited both similarities and differences in the patterns of processing dependencies. For all ages, performance for each dimension was slower in the presence of variation in the irrelevant dimension; irrelevant variation in the voice dimension disrupted performance more than irrelevant variation in the linguistic dimension. Trends in the degree of interference, on the other hand, showed significant differences between dimensions as a function of age. Whereas the degree of interference for the voice-dimension-relevant did not show significant age-related change, the degree of interference for the word-dimension-relevant declined significantly with age in a linear as well as a quadratic manner. A major age-related change in the relation between dimensions was that word processing, relative to voice-gender processing, required significantly more time in the children than in the adults. Overall, the developmental course characterizing multidimensional speech processing evidenced more pronounced change when the linguistic dimension, rather than the voice dimension, was relevant.

Processing interactions between segmental and suprasegmental information in native speakers of English and Mandarin Chinese

The processing interactions between segmental and suprasegmental information in native speakers of English and Mandarin Chinese were investigated in a speeded classification task. Since in Chinese, unlike in English, tones convey lexically meaningful information, native speakers of these languages may process combinations of segmental and suprasegmental information differently. Subjects heard consonant-vowel syllables varying on a consonantal (segmental) dimension and either a Mandarin Chinese or constant-pitch (non-Mandarin) suprasegmental dimension. The English listeners showed mutual integrality with the Mandarin Chinese stimuli, but not the constant-pitch stimuli. The native Chinese listeners processed these dimensions with mutual integrality for both the Mandarin Chinese and the constant-pitch stimuli. These results were interpreted in terms of the linguistic function and the structure of suprasegmental information in Chinese and English. The results suggest that the way listeners perceive speech depends on the interaction between the structure of the signal and the processing strategies of the listener.

Stimulus variability and processing dependencies in speech perception

Processing dependencies in speech perception between voice and phoneme were investigated using the Garner (1974) speeded classification procedure. Variability in the voice of the talker and in the cues to word-initial consonants were manipulated. The results showed that the processing of a talker's voice and the perception of voicing are asymmetrically dependent. In addition, when stimulus variability was increased in each dimension, the amount of orthogonal interference obtained for each dimension became significantly larger. The processing asymmetry between voice and phoneme was interpreted in terms of a parallel-contingent relationship of talker normalization processes to auditory-to-phonetic coding processes. The processing of voice information appears to be qualitatively different from the encoding of segmental phonetic information, although they are not independent. Implications of these results for current theories of speech perception are discussed.