Language modality shapes the dynamics of word and sign recognition

Cross-modal and cross-language activation in bilinguals reveals lexical competition even when words or signs are unheard or unseen

We exploit the phenomenon of cross-modal, cross-language activation to examine the dynamics of language processing. Previous within-language work showed that seeing a sign coactivates phonologically related signs, just as hearing a spoken word coactivates phonologically related words. In this study, we conducted a series of eye-tracking experiments using the visual world paradigm to investigate the time course of cross-language coactivation in hearing bimodal bilinguals (Spanish-Spanish Sign Language) and unimodal bilinguals (Spanish/Basque). The aim was to gauge whether (and how) seeing a sign could coactivate words and, conversely, how hearing a word could coactivate signs and how such cross-language coactivation patterns differ from within-language coactivation. The results revealed cross-language, cross-modal activation in both directions. Furthermore, comparison with previous findings of within-language lexical coactivation for spoken and signed language showed how the impact of temporal structure changes in different modalities. Spoken word activation follows the temporal structure of that word only when the word itself is heard; for signs, the temporal structure of the sign does not govern the time course of lexical access (location coactivation precedes handshape coactivation)-even when the sign is seen. We provide evidence that, instead, this pattern of activation is motivated by how common in the lexicon the sublexical units of the signs are. These results reveal the interaction between the perceptual properties of the explicit signal and structural linguistic properties. Examining languages across modalities illustrates how this interaction impacts language processing.

Bottom-up processes dominate early word recognition in toddlers

This study set out to investigate whether the 'phonological onset preference effect' often reported in adult studies using the visual world task (i.e., increased attention to an object that is phonologically-related to a spoken-target word, such as boat-bear) is also contingent upon toddler participants having sufficient preview time to inspect the picture stimuli. Picture preview is thought to support the activation of phonological codes which can then be matched to the phonological representations extracted from incoming speech signals and the picture stimuli, supporting the 'phonological mapping hypothesis'. We found that both toddlers and adults were able to show an early phonological onset preference in short preview conditions, though, adults' early phonological onset preferences in the short preview condition was extinguished by the presence of a semantic competitor, replicating previous adult findings (Huettig & McQueen, 2007). Removal of a semantic competitor reinstated the phonological onset preference effect under short preview conditions for adults. Our findings indicate that toddlers are driven more by bottom-up, phonological information when selecting a referent in a visual world task, as compared to adults who are more inclined to exploit top-down, semantic information when directing their attention to a visual object, especially when there is insufficient preview time. We propose that, when implicit naming is improbable in short-preview conditions, a phonological onset preference effect is driven by mapping on the visual-semantic levels, which is more susceptible to top-down influences.

A Neural Network Model of Lexical-Semantic Competition During Spoken Word Recognition

Visual world studies show that upon hearing a word in a target-absent visual context containing related and unrelated items, toddlers and adults briefly direct their gaze toward phonologically related items, before shifting toward semantically and visually related ones. We present a neural network model that processes dynamic unfolding phonological representations of words and maps them to static internal lexical, semantic, and visual representations. The model, trained on representations derived from real corpora, simulates this early phonological over semantic/visual preference. Our results support the hypothesis that incremental unfolding of a spoken word is in itself sufficient to account for the transient preference for phonological competitors over both unrelated and semantically and visually related ones. Phonological representations mapped dynamically in a bottom-up fashion to semantic-visual representations capture the early phonological preference effects reported in visual world tasks. The semantic visual preference typically observed later in such a task does not require top-down feedback from a semantic or visual system.

"Textual Prosody" Can Change Impressions of Reading in People With Normal Hearing and Hearing Loss

Recently, dynamic text presentation, such as scrolling text, has been widely used. Texts are often presented at constant timing and speed in conventional dynamic text presentation. However, dynamic text presentation enables visually presented texts to indicate timing information, such as prosody, and the texts might influence the impression of reading. In this paper, we examined this possibility by focusing on the temporal features of digital text in which texts are represented sequentially and with varying speed, duration, and timing. We call this "textual prosody." We used three types of textual prosody: "Recorded," "Shuffled," and "Constant." Recorded prosody is the reproduction of a reader's reading with pauses and varying speed that simulates talking. Shuffled prosody randomly shuffles the time course of speed and pauses in the recorded type. Constant prosody has a constant presentation speed and provides no timing information. Experiment 1 examined the effect of textual prosody on people with normal hearing. Participants read dynamic text with textual prosody silently and rated their impressions of texts. The results showed that readers with normal hearing preferred recorded textual prosody and constant prosody at the optimum speed (6 letters/second). Recorded prosody was also preferred at a low presentation speed. Experiment 2 examined the characteristics of textual prosody using an articulatory suppression paradigm. The results showed that some textual prosody was stored in the articulatory loop despite it being presented visually. In Experiment 3, we examined the effect of textual prosody with readers with hearing loss. The results demonstrated that readers with hearing loss had positive impressions at relatively low presentation speeds when the recorded prosody was presented. The results of this study indicate that the temporal structure is processed regardless of whether the input is visual or auditory. Moreover, these results suggest that textual prosody can enrich reading not only in people with normal hearing but also in those with hearing loss, regardless of acoustic experiences.

Sign phonological parameters modulate parafoveal preview effects in deaf readers

Research has found that deaf readers unconsciously activate sign translations of written words while reading. However, the ways in which different sign phonological parameters associated with these sign translations tie into reading processes have received little attention in the literature. In this study on Chinese reading, we used a parafoveal preview paradigm to investigate how four different types of sign phonologically related preview affect reading processes in adult deaf signers of Hong Kong Sign Language (HKSL). The four types of sign phonologically related preview-target pair were: (1) pairs with HKSL translations that overlapped in three parameters-handshape, location, and movement; (2) pairs that overlapped in only handshape and location; (3) pairs that only overlapped in handshape and movement; and (4) pairs that only overlapped in location and movement. Results showed that the handshape parameter was of particular importance as only sign translation pairs that had handshape among their overlapping sign phonological parameters led to early sign activation. Furthermore, we found that, compared to control previews, deaf readers took longer to read targets when the sign translation previews overlapped with targets in either handshape and movement or handshape, movement, and location. In contrast, fixation times on targets were shorter when previews and targets overlapped location and any single additional parameter-either handshape or movement. These results indicate that the phonological parameters of handshape, location, and movement are activated via orthography during Chinese reading and can have different effects on parafoveal processing in deaf signers of HKSL.