Having a task partner affects lexical retrieval: Spoken word production in shared task settings

Beyond speaking: neurocognitive perspectives on language production in social interaction

The human faculty to speak has evolved, so has been argued, for communicating with others and for engaging in social interactions. Hence the human cognitive system should be equipped to address the demands that social interaction places on the language production system. These demands include the need to coordinate speaking with listening, the need to integrate own (verbal) actions with the interlocutor's actions, and the need to adapt language flexibly to the interlocutor and the social context. In order to meet these demands, core processes of language production are supported by cognitive processes that enable interpersonal coordination and social cognition. To fully understand the cognitive architecture and its neural implementation enabling humans to speak in social interaction, our understanding of how humans produce language needs to be connected to our understanding of how humans gain insights into other people's mental states and coordinate in social interaction. This article reviews theories and neurocognitive experiments that make this connection and can contribute to advancing our understanding of speaking in social interaction. This article is part of a discussion meeting issue 'Face2face: advancing the science of social interaction'.

Cumulative semantic interference is blind to morphological complexity and originates at the conceptual level

When naming a sequence of pictures of the same semantic category (e.g., furniture), response latencies systematically increase with each named category member. This cumulative semantic interference effect has become a popular tool to investigate the cognitive architecture of language production. However, not all processes underlying the effect itself are fully understood, including the question where the effect originates from. While some researchers assume the interface of the conceptual and lexical level as its origin, others suggest the conceptual-semantic level. The latter assumption follows from the observation that cumulative effects, namely cumulative facilitation, can also be observed in purely conceptual-semantic tasks. Another unanswered question is whether cumulative interference is affected by the morphological complexity of the experimental targets. In two experiments with the same participants and the same material, we investigated both of these issues. Experiment 1, a continuous picture naming task, investigated whether morphologically complex nouns (e.g., kitchen table) elicit identical levels of cumulative interference to morphologically simple nouns (e.g., table). Our results show this to be the case, indicating that cumulative interference is unaffected by lexical information such as morphological complexity. In Experiment 2, participants classified the same target objects as either man-made or natural. As expected, we observed cumulative facilitation. A separate analysis showed that this facilitation effect can be predicted by the individuals’ effect sizes of cumulative interference, suggesting a strong functional link between the two effects. Our results thus point to a conceptual-semantic origin of cumulative semantic interference.

Interference in the shared-Stroop task: a comparison of self- and other-monitoring

Co-actors represent and integrate each other's actions, even when they need not monitor one another. However, monitoring is important for successful interactions, particularly those involving language, and monitoring others' utterances probably relies on similar mechanisms as monitoring one's own. We investigated the effect of monitoring on the integration of self- and other-generated utterances in the shared-Stroop task. In a solo version of the Stroop task (with a single participant responding to all stimuli; Experiment 1), participants named the ink colour of mismatching colour words (incongruent stimuli) more slowly than matching colour words (congruent). In the shared-Stroop task, one participant named the ink colour of words in one colour (e.g. red), while ignoring stimuli in the other colour (e.g. green); the other participant either named the other ink colour or did not respond. Crucially, participants either provided feedback about the correctness of their partner's response (Experiment 3) or did not (Experiment 2). Interference was greater when both participants responded than when they did not, but only when their partners provided feedback. We argue that feedback increased interference because monitoring one's partner enhanced representations of the partner's target utterance, which in turn interfered with self-monitoring of the participant's own utterance.

EXPRESS: Representation of others' synchronous and asynchronous sentences interferes with sentence production

In dialogue, people represent each other's utterances in order to take turns and communicate successfully. In previous work [Gambi, C., Van de Cavey, J., & Pickering, M. J. (2015). Interference in joint picture naming. Journal of Experimental Psychology: Learning, Memory, and Cognition, 41(1), 1-21.], speakers who were naming single pictures or picture pairs represented whether another speaker was engaged in the same task (versus a different or no task) concurrently, but did not represent in detail the content of the other speaker's utterance. Here, we investigate co-representation of whole sentences. In three experiments, pairs of speakers imagined each other producing active or passive descriptions of transitive events. Speakers took longer to begin speaking when they believed their partner was also preparing to speak, compared to when they did not. Interference occurred when speakers believed their partners were preparing to speak at the same time as them (synchronous production and co-representation; Experiment 1), and also when speakers believed that their partner would speak only after them (asynchronous production and co-representation; Experiments 2a and 2b). However, interference was generally no greater when speakers believed their partner was preparing a different compared to a similar utterance, providing no consistent evidence that speakers represented what their partners were preparing to say. Taken together, these findings indicate that speakers can represent another's intention to speak even as they are themselves preparing to speak, but that such representation tends to lack detail.

Mental chronometry of speaking in dialogue: Semantic interference turns into facilitation

Numerous studies demonstrate that the production of words is delayed when speakers process in temporal proximity semantically related words. Yet the experimental settings underlying this effect are different from those under which we typically speak. This study demonstrates that semantic interference disappears, and can even turn into facilitation, when semantically related words are embedded in a meaningful communicative exchange. Experiment 1 and 3 (each N = 32 university students) implemented a picture-word interference task in a game played between two participants: one named the distractor word and, after a stimulus-onset-asynchrony of -150 ms or -650 ms, the other named a semantically related or unrelated target picture. Semantic interference reappeared with identical experimental parameters in a single-person picture-word interference setting (Experiment 2, N = 32). We conclude that the inhibitory context effects leading to semantic interference in single-subject settings are attenuated whereas facilitatory effects are enhanced in communicative settings.

Robots facilitate human language production

Despite recent developments in integrating autonomous and human-like robots into many aspects of everyday life, social interactions with robots are still a challenge. Here, we focus on a central tool for social interaction: verbal communication. We assess the extent to which humans co-represent (simulate and predict) a robot's verbal actions. During a joint picture naming task, participants took turns in naming objects together with a social robot (Pepper, Softbank Robotics). Previous findings using this task with human partners revealed internal simulations on behalf of the partner down to the level of selecting words from the mental lexicon, reflected in partner-elicited inhibitory effects on subsequent naming. Here, with the robot, the partner-elicited inhibitory effects were not observed. Instead, naming was facilitated, as revealed by faster naming of word categories co-named with the robot. This facilitation suggests that robots, unlike humans, are not simulated down to the level of lexical selection. Instead, a robot's speaking appears to be simulated at the initial level of language production where the meaning of the verbal message is generated, resulting in facilitated language production due to conceptual priming. We conclude that robots facilitate core conceptualization processes when humans transform thoughts to language during speaking.

Semantic Interference through Multiple Distractors in Picture Naming in People with Aphasia

When we refer to an object or concept by its name, activation of semantic and categorical information is necessary to retrieve the correct lexical representation. Whereas in neurotypical individuals it is well established that semantic context can interfere with or facilitate lexical retrieval, these effects are much less studied in people with lesions to the language network and impairment at different steps of lexical-semantic processing. Here, we applied a novel picture naming paradigm, where multiple categorically related and unrelated words were presented as distractors before a to-be-named target picture. Using eye tracking, we investigated preferential fixation on the cohort members versus nonmembers. Thereby, we can judge the impact of explicit acknowledgment of the category and its effect on semantic interference. We found that, in contrast to neurotypical participants [van Scherpenberg, C., Abdel Rahman, R., & Obrig, H. A novel multiword paradigm for investigating semantic context effects in language production. PLoS One, 15, e0230439, 2020], participants suffering from mild to moderate aphasia did not show a fixation preference on category members but still showed a large interference effect of ∼35 msec, confirming the implicit mechanism of categorical interference. However, preferential fixation on the categorically related cohort words correlated with clinical tests regarding nonverbal semantic abilities and integrity of the anterior temporal lobe. This highlights the role of supramodal semantics for explicit recognition of a semantic category, while semantic interference is triggered if the threshold of lexical cohort activation is reached. Confirming psycholinguistic evidence, the demonstration of a large and persistent interference effect through implicit lexico-semantic activation is important to understand deficits in people with a lesion in thelanguage network, potentially relevant for individualized intervention aiming at improving naming skills.

The effect of co-actor group membership on the social inhibition of return effect

Being part of a group is a crucial factor in human social interaction. In the current study we explored whether group membership affects reflexive automatic cognitive functioning, and specifically the social inhibition of return effect (SIOR; Welsh et al., 2005). SIOR is characterized by slower reaction times (RTs) to a location already searched by another agent. To examine whether group membership modulates SIOR, we recruited Muslim and Jewish students from the University of Haifa to perform a task with either an in-group member or an out-group member. Both IOR and SIOR were suggested to act as a foraging facilitator (Klein, 2000; Welsh et al., 2005). Accordingly, we predicted that the SIOR effect would be larger when performing the task with an in-group member than with an out-group member. The results confirmed our prediction by indicating that the co-actor's group membership modulated the SIOR effect. These findings are consistent with the notion that social factors play a critical role in producing the SIOR effect and provide a novel indication of the influence of social factors such as group membership on basic reflexive cognitive processes.

An approach to social flexibility: Congruency effects during spontaneous word-by-word interaction

Cognitive flexibility is the ability to switch between different concepts or to adapt goal-directed behavior in a changing environment. Although, cognitive research on this ability has long been focused on the individual mind, it is becoming increasingly clear that cognitive flexibility plays a central role in our social life. This is particularly evident in turn-taking in verbal conversation, where cognitive flexibility of the individual becomes part of social flexibility in the dyadic interaction. In this work, we introduce a model that reveals different parameters that explain how people flexibly handle unexpected events in verbal conversation. In order to study hypotheses derived from the model, we use a novel experimental approach in which thirty pairs of participants engaged in a word-by-word interaction by taking turns in generating sentences word by word. Similar to well established individual cognitive tasks, participants needed to adapt their behavior in order to respond to their co-actor’s last utterance. With our experimental approach we could manipulate the interaction between participants: Either both participants had to construct a sentence with a common target word (congruent condition) or with distinct target words (incongruent condition). We further studied the relation between the interactive Word-by-Word task measures and classical individual-centered, cognitive tasks, namely the Number-Letter task, the Stop-Signal task, and the GoNogo task. In the Word-by-Word task, we found that participants had faster response times in congruent compared to incongruent trials, which replicates the primary findings of standard cognitive tasks measuring cognitive flexibility. Further, we found a significant correlation between the performance in the Word-by-Word task and the Stop-Signal task indicating that participants with a high cognitive flexibility in the Word-by-Word task also showed high inhibition control.

Expectancy effects in the EEG during joint and spontaneous word-by-word sentence production in German

Our aim in the present study is to measure neural correlates during spontaneous interactive sentence production. We present a novel approach using the word-by-word technique from improvisational theatre, in which two speakers jointly produce one sentence. This paradigm allows the assessment of behavioural aspects, such as turn-times, and electrophysiological responses, such as event-related-potentials (ERPs). Twenty-five participants constructed a cued but spontaneous four-word German sentence together with a confederate, taking turns for each word of the sentence. In 30% of the trials, the confederate uttered an unexpected gender-marked article. To complete the sentence in a meaningful way, the participant had to detect the violation and retrieve and utter a new fitting response. We found significant increases in response times after unexpected words and - despite allowing unscripted language production and naturally varying speech material - successfully detected significant N400 and P600 ERP effects for the unexpected word. The N400 EEG activity further significantly predicted the response time of the subsequent turn. Our results show that combining behavioural and neuroscientific measures of verbal interactions while retaining sufficient experimental control is possible, and that this combination provides promising insights into the mechanisms of spontaneous spoken dialogue.

Initiation of utterance planning in response to pre-recorded and "live" utterances

In everyday conversation, interlocutors often plan their utterances while listening to their conversational partners, thereby achieving short gaps between their turns. Important issues for current psycholinguistics are how interlocutors distribute their attention between listening and speech planning and how speech planning is timed relative to listening. Laboratory studies addressing these issues have used a variety of paradigms, some of which have involved using recorded speech to which participants responded, whereas others have involved interactions with confederates. This study investigated how this variation in the speech input affected the participants' timing of speech planning. In Experiment 1, participants responded to utterances produced by a confederate, who sat next to them and looked at the same screen. In Experiment 2, they responded to recorded utterances of the same confederate. Analyses of the participants' speech, their eye movements, and their performance in a concurrent tapping task showed that, compared with recorded speech, the presence of the confederate increased the processing load for the participants, but did not alter their global sentence planning strategy. These results have implications for the design of psycholinguistic experiments and theories of listening and speaking in dyadic settings.

Mental representations of partner task cause interference in picture naming

Interference in picture naming occurs from representing a partner's preparations to speak (Gambi, van de Cavey, & Pickering, 2015). We tested the origins of this interference using a simple non-communicative joint naming task based on Gambi et al. (2015), where response latencies indexed interference from partner task and partner speech content, and eye fixations to partner objects indexed overt attention. Experiment 1 contrasted a partner-present condition with a control partner-absent condition to establish the role of the partner in eliciting interference. For latencies, we observed interference from the partner's task and speech content, with interference increasing due to partner task in the partner-present condition. Eye-tracking measures showed that interference in naming was not due to overt attention to partner stimuli but to broad expectations about likely utterances. Experiment 2 examined whether an equivalent non-verbal task also elicited interference, as predicted from a language as joint action framework. We replicated the finding of interference due to partner task and again found no relationship between overt attention and interference. These results support Gambi et al. (2015). Individuals co-represent a partner's task while speaking, and doing so does not require overt attention to partner stimuli.

Word naming slows picture naming but does not affect cumulative semantic interference

Two experiments are reported which investigate the effect of processing words prior to naming target pictures. In Experiment 1, participants named (read aloud) sequences of five printed prime words and five target pictures from the same semantic category, and also sequences of five prime words from a different unrelated semantic category to the five related target pictures. Picture and words were interleaved, with two unrelated filler stimuli in between prime and target stimuli (i.e. a lag of 3 between primes and targets). Results showed that across the five target picture naming trials (i.e. across ordinal position of picture), picture naming times increased linearly, replicating the cumulative semantic interference (CSI) effect (e.g., Howard, Nickels, Coltheart, & Cole-Virtue, 2006). Related prime words slowed picture naming, replicating the effects found in paired word prime and picture target studies (e.g., Tree & Hirsh, 2003). However, the naming of the five related prime words did not modify the picture naming CSI effect, with this null result converging with findings from a different word and picture design (e.g., Navarrete, Mahon, & Caramazza, 2010). In Experiment 2, participants categorised the prime word stimuli as manmade versus natural, so that words were more fully processed at a conceptual level. The interaction between word prime relatedness and ordinal position of the named target picture was significant. These results are consistent with adjustments at the conceptual level (Belke, 2013; Roelofs, 2018) which last over several trials at least. By contrast, we conclude that the distinct word-to-picture naming interference effect from Experiment 1 must originate outside of the conceptual level and outside of the mappings between semantics and lexical representations. We discuss the results with reference to recent theoretical accounts of the CSI picture naming effect and word naming models.

Co-actors represent the order of each other's actions

Previous research has shown that people represent each other’s tasks and actions when acting together. However, less is known about how co-actors represent each other’s action sequences. Here, we asked whether co-actors represent the order of each other’s actions within an action sequence, or whether they merely represent the intended end state of a joint action together with their own contribution. In the present study, two co-actors concurrently performed action sequences composed of two actions. We predicted that if co-actors represent the order of each other’s actions, they should experience interference when the order of their actions differs. Supporting this prediction, the results of six experiments consistently showed that co-actors moved more slowly when performing the same actions in a different order compared to performing the same actions in the same order. In line with findings from bimanual movement tasks, our results indicate that interference can arise due to differences in movement parameters and due to differences in the perceptual characteristics of movement goals. The present findings extend previous research on co-representation, providing evidence that people represent not only the elements of another’s task, but also their temporal structure.

When task sharing reduces interference: evidence for division-of-labour in Stroop-like tasks

Performing a task with another person may either enhance or reduce the interference produced by task-irrelevant information. In three experiments, we employed the joint version of a Stroop-like task (i.e., the picture-word interference-PWI-task) to investigate some of the task features that seem to be critical in determining the effect of task-irrelevant information when the task is shared between two individuals. Participants were asked to perform a PWI task, which required to name a picture while ignoring a distractor word, first individually (in a baseline block of trials) and then co-acting with an alleged partner. Results showed that, compared to the baseline and to a condition in which participants continued to perform the PWI task individually, the belief of co-acting with another individual who was thought to be in charge of the distractor words suppressed the semantic interference effect when these words were in case alternation letters (e.g., "mOuSe"). Conversely, the semantic interference effect persisted when the co-actor was thought to be in charge of the same task as the participant, that is, the co-actor was thought to respond to the pictures. These results are accounted for by assuming that, when the participant knows that another person is in charge of the task-irrelevant information, a division-of-labour between participant and co-actor can be established. Such a division-of-labour may provide the participant with a strategy to oppose the semantic interference effect. Our findings, therefore, suggest that sharing a task with another person in charge of potentially interfering information can enable people to filter out this information from their own task representation.

Working Together: Contributions of Corpus Analyses and Experimental Psycholinguistics to Understanding Conversation

As conversation is the most important way of using language, linguists and psychologists should combine forces to investigate how interlocutors deal with the cognitive demands arising during conversation. Linguistic analyses of corpora of conversation are needed to understand the structure of conversations, and experimental work is indispensable for understanding the underlying cognitive processes. We argue that joint consideration of corpus and experimental data is most informative when the utterances elicited in a lab experiment match those extracted from a corpus in relevant ways. This requirement to compare like with like seems obvious but is not trivial to achieve. To illustrate this approach, we report two experiments where responses to polar (yes/no) questions were elicited in the lab and the response latencies were compared to gaps between polar questions and answers in a corpus of conversational speech. We found, as expected, that responses were given faster when they were easy to plan and planning could be initiated earlier than when they were harder to plan and planning was initiated later. Overall, in all but one condition, the latencies were longer than one would expect based on the analyses of corpus data. We discuss the implication of this partial match between the data sets and more generally how corpus and experimental data can best be combined in studies of conversation.

Extending Situated Language Comprehension (Accounts) with Speaker and Comprehender Characteristics: Toward Socially Situated Interpretation

More and more findings suggest a tight temporal coupling between (non-linguistic) socially interpreted context and language processing. Still, real-time language processing accounts remain largely elusive with respect to the influence of biological (e.g., age) and experiential (e.g., world and moral knowledge) comprehender characteristics and the influence of the 'socially interpreted' context, as for instance provided by the speaker. This context could include actions, facial expressions, a speaker's voice or gaze, and gestures among others. We review findings from social psychology, sociolinguistics and psycholinguistics to highlight the relevance of (the interplay between) the socially interpreted context and comprehender characteristics for language processing. The review informs the extension of an extant real-time processing account (already featuring a coordinated interplay between language comprehension and the non-linguistic visual context) with a variable ('ProCom') that captures characteristics of the language user and with a first approximation of the comprehender's speaker representation. Extending the CIA to the sCIA (social Coordinated Interplay Account) is the first step toward a real-time language comprehension account which might eventually accommodate the socially situated communicative interplay between comprehenders and speakers.