Bilingual switching between languages and listeners: Insights from immersive virtual reality

Beyond the tried and true: How virtual reality, dialog setups, and a focus on multimodality can take bilingual language production research forward

Bilinguals possess the ability of expressing themselves in more than one language, and typically do so in contextually rich and dynamic settings. Theories and models have indeed long considered context factors to affect bilingual language production in many ways. However, most experimental studies in this domain have failed to fully incorporate linguistic, social, or physical context aspects, let alone combine them in the same study. Indeed, most experimental psycholinguistic research has taken place in isolated and constrained lab settings with carefully selected words or sentences, rather than under rich and naturalistic conditions. We argue that the most influential experimental paradigms in the psycholinguistic study of bilingual language production fall short of capturing the effects of context on language processing and control presupposed by prominent models. This paper therefore aims to enrich the methodological basis for investigating context aspects in current experimental paradigms and thereby move the field of bilingual language production research forward theoretically. After considering extensions of existing paradigms proposed to address context effects, we present three far-ranging innovative proposals, focusing on virtual reality, dialog situations, and multimodality in the context of bilingual language production.

Combining EEG and 3D-eye-tracking to study the prediction of upcoming speech in naturalistic virtual environments: A proof of principle

EEG and eye-tracking provide complementary information when investigating language comprehension. Evidence that speech processing may be facilitated by speech prediction comes from the observation that a listener's eye gaze moves towards a referent before it is mentioned if the remainder of the spoken sentence is predictable. However, changes to the trajectory of anticipatory fixations could result from a change in prediction or an attention shift. Conversely, N400 amplitudes and concurrent spectral power provide information about the ease of word processing the moment the word is perceived. In a proof-of-principle investigation, we combined EEG and eye-tracking to study linguistic prediction in naturalistic, virtual environments. We observed increased processing, reflected in theta band power, either during verb processing - when the verb was predictive of the noun - or during noun processing - when the verb was not predictive of the noun. Alpha power was higher in response to the predictive verb and unpredictable nouns. We replicated typical effects of noun congruence but not predictability on the N400 in response to the noun. Thus, the rich visual context that accompanied speech in virtual reality influenced language processing compared to previous reports, where the visual context may have facilitated processing of unpredictable nouns. Finally, anticipatory fixations were predictive of spectral power during noun processing and the length of time fixating the target could be predicted by spectral power at verb onset, conditional on the object having been fixated. Overall, we show that combining EEG and eye-tracking provides a promising new method to answer novel research questions about the prediction of upcoming linguistic input, for example, regarding the role of extralinguistic cues in prediction during language comprehension.

What's your point? Insights from virtual reality on the relation between intention and action in the production of pointing gestures

Human communication involves the process of translating intentions into communicative actions. But how exactly do our intentions surface in the visible communicative behavior we display? Here we focus on pointing gestures, a fundamental building block of everyday communication, and investigate whether and how different types of underlying intent modulate the kinematics of the pointing hand and the brain activity preceding the gestural movement. In a dynamic virtual reality environment, participants pointed at a referent to either share attention with their addressee, inform their addressee, or get their addressee to perform an action. Behaviorally, it was observed that these different underlying intentions modulated how long participants kept their arm and finger still, both prior to starting the movement and when keeping their pointing hand in apex position. In early planning stages, a neurophysiological distinction was observed between a gesture that is used to share attitudes and knowledge with another person versus a gesture that mainly uses that person as a means to perform an action. Together, these findings suggest that our intentions influence our actions from the earliest neurophysiological planning stages to the kinematic endpoint of the movement itself.

When interlocutor's face-language matching alters: An ERP study on face contexts and bilingual language control in mixed-language picture naming

The present study used event-related potentials (ERP) to examine Chinese-English bilinguals' reactive and proactive language control as they performed mixed-language picture naming with face cues. All participants named pictures in Chinese (first language, L1) and English (second language, L2) across three sessions: a 25% face-language matched session, a baseline session without face cues, and a 75% face-language matched session. Behavioral analyses for reactive language control showed that the asymmetrical switch cost was larger for L2 than L1 in the 25% session and for L1 than L2 in the 75% session. ERP results revealed more negative N2 and LPC during L1 switching in 25% session but enhanced N2 during L2 switching in 75% session. Similar N2 and LPC effect was found during L1 and L2 switching in the baseline context. For proactive language control, the reversed language dominance and enhanced LPC amplitudes during L2 naming were consistent across the three sessions. Our findings suggest that reactive but not proactive language control is modulated by the ever-changing face contexts, which highlights the highly flexible bilingual control systems subserving nonlinguistic cues.

Oscillatory brain responses to processing code-switches in the presence of others

Code-switching, i.e. the alternation between languages in a conversation, is a typical, yet socially-constrained practice in bilingual communities. For instance, code-switching is permissible only when other conversation partners are fluent in both languages. Studying code-switching provides insight in the cognitive and neural mechanisms underlying language control, and their modulation by linguistic and non-linguistic factors. Using time-frequency representations, we analyzed brain oscillation changes in EEG data recorded in a prior study (Kaan et al., 2020). In this study, Spanish-English bilinguals read sentences with and without switches in the presence of a bilingual or monolingual partner. Consistent with prior studies, code-switches were associated with a power decrease in the lower beta band (15-18 Hz). In addition, code-switches were associated with a power decrease in the upper gamma band (40-50 Hz), but only when a bilingual partner was present, suggesting the semantic/pragmatic processing of code-switches differs depending on who is present.

On the Connection Between Language Control and Executive Control-An ERP Study

Models vary in the extent to which language control processes are domain general. Those that posit that language control is at least partially domain general insist on an overlap between language control and executive control at the goal level. To further probe whether or not language control is domain general, we conducted the first event-related potential (ERP) study that directly compares language-switch costs, as an index of language control, and task-switch costs, as an index of executive control. The language switching and task switching methodologies were identical, except that the former required switching between languages (English or Spanish) whereas the latter required switching between tasks (color naming or category naming). This design allowed us to directly compare control processes at the goal level (cue-locked ERPs) and at the task performance level (picture-locked ERPs). We found no significant differences in the switch-related cue-locked and picture-locked ERP patterns across the language and task switching paradigms. These results support models of domain-general language control.

Language distance modulates cognitive control in bilinguals

Linguistic processes in the bilingual brain are partially shared across languages, and the degree of neural overlap between the languages is influenced by several factors, including the age of acquisition, relative language proficiency, and immersion. There is limited evidence on the role of linguistic distance on the performance of the language control as well as domain-general cognitive control systems. The present study aims at exploring whether being bilingual in close and distant language pairs (CLP and DLP) influences language control and domain-general cognitive processes. We recruited two groups of DLP (Persian-English) and CLP (French-English) bilinguals. Subjects performed language nonswitching and switching picture-naming tasks and a nonlinguistic switching task while EEG data were recorded. Behaviorally, CLP bilinguals showed a lower cognitive cost than DLP bilinguals, reflected in faster reaction times both in language switching (compared to nonswitching) and nonlinguistic switching. ERPs showed differential involvement of cognitive control regions between the CLP and DLP groups during linguistic switching vs. nonswitching at 450 to 515 ms poststimulus presentation. Moreover, there was a difference between CLP and DLP groups from 40 to 150 ms in the nonlinguistic task. Our electrophysiological results confirm a stronger involvement of language control and domain-general cognitive control regions in CLP bilinguals.

Language control in bimodal bilinguals: Evidence from ERPs

It is currently unclear to what degree language control, which minimizes non-target language interference and increases the probability of selecting target-language words, is similar for sign-speech (bimodal) bilinguals and spoken language (unimodal) bilinguals. To further investigate the nature of language control processes in bimodal bilinguals, we conducted the first event-related potential (ERP) language switching study with hearing American Sign Language (ASL)-English bilinguals. The results showed a pattern that has not been observed in any unimodal language switching study: a switch-related positivity over anterior sites and a switch-related negativity over posterior sites during ASL production in both early and late time windows. No such pattern was found during English production. We interpret these results as evidence that bimodal bilinguals uniquely engage language control at the level of output modalities.

A Systematic Review of AR and VR Enhanced Language Learning

This paper provided a systematic review of previous Augmented Reality (AR) and Virtual Reality (VR) studies on language learning. A total of 88 articles were selected and analyzed from five perspectives: their ways of integrating AR or VR tools in language learning; main users of AR and VR technologies; major research findings; why AR and VR tools are effective in promoting language learning; and the implications. It was found that (1) immersing learners into virtual worlds is the main approach to language learning in AR and VR studies; (2) university students were the main users of AR/VR technologies; (3) the major research findings concerning the benefits of AR and VR included improvement of students’ learning outcomes, enhancement of motivation, and positive perceptions towards using AR and VR; (4) AR and VR tools promoted language learning through providing immersive learning experience, enhancing motivation, creating interaction, and reducing learning anxiety; and (5) implications identified from previous research include the need of providing training for teachers, enlarging sample sizes, and exploring learner factors such as learner engagement and satisfaction.

Bilingualism and domain-general cognitive functions from a neural perspective: A systematic review

A large body of research has indicated that bilingualism - through continual practice in language control - may impact cognitive functions, as well as relevant aspects of brain function and structure. The present review aimed to bring together findings on the relationship between bilingualism and domain-general cognitive functions from a neural perspective. The final sample included 210 studies, covering findings regarding neural responses to bilingual language control and/or domain-general cognitive tasks, as well as findings regarding effects of bilingualism on non-task-related brain function and brain structure. The evidence indicates that a) bilingual language control likely entails neural mechanisms responsible for domain-general cognitive functions; b) bilingual experiences impact neural responses to domain-general cognitive functions; and c) bilingual experiences impact non-task-related brain function (both resting-state and metabolic function) as well as aspects of brain structure (both macrostructure and microstructure), each of which may in turn impact mental processes, including domain-general cognitive functions. Such functional and structural neuroplasticity associated with bilingualism may contribute to both cognitive and neural reserves, producing benefits across the lifespan.