Language production in a shared task: Cumulative Semantic Interference from self- and other-produced context words

Beyond speaking: neurocognitive perspectives on language production in social interaction

The human faculty to speak has evolved, so has been argued, for communicating with others and for engaging in social interactions. Hence the human cognitive system should be equipped to address the demands that social interaction places on the language production system. These demands include the need to coordinate speaking with listening, the need to integrate own (verbal) actions with the interlocutor's actions, and the need to adapt language flexibly to the interlocutor and the social context. In order to meet these demands, core processes of language production are supported by cognitive processes that enable interpersonal coordination and social cognition. To fully understand the cognitive architecture and its neural implementation enabling humans to speak in social interaction, our understanding of how humans produce language needs to be connected to our understanding of how humans gain insights into other people's mental states and coordinate in social interaction. This article reviews theories and neurocognitive experiments that make this connection and can contribute to advancing our understanding of speaking in social interaction. This article is part of a discussion meeting issue 'Face2face: advancing the science of social interaction'.

Cumulative semantic interference is blind to morphological complexity and originates at the conceptual level

When naming a sequence of pictures of the same semantic category (e.g., furniture), response latencies systematically increase with each named category member. This cumulative semantic interference effect has become a popular tool to investigate the cognitive architecture of language production. However, not all processes underlying the effect itself are fully understood, including the question where the effect originates from. While some researchers assume the interface of the conceptual and lexical level as its origin, others suggest the conceptual-semantic level. The latter assumption follows from the observation that cumulative effects, namely cumulative facilitation, can also be observed in purely conceptual-semantic tasks. Another unanswered question is whether cumulative interference is affected by the morphological complexity of the experimental targets. In two experiments with the same participants and the same material, we investigated both of these issues. Experiment 1, a continuous picture naming task, investigated whether morphologically complex nouns (e.g., kitchen table) elicit identical levels of cumulative interference to morphologically simple nouns (e.g., table). Our results show this to be the case, indicating that cumulative interference is unaffected by lexical information such as morphological complexity. In Experiment 2, participants classified the same target objects as either man-made or natural. As expected, we observed cumulative facilitation. A separate analysis showed that this facilitation effect can be predicted by the individuals’ effect sizes of cumulative interference, suggesting a strong functional link between the two effects. Our results thus point to a conceptual-semantic origin of cumulative semantic interference.

Interference in the shared-Stroop task: a comparison of self- and other-monitoring

Co-actors represent and integrate each other's actions, even when they need not monitor one another. However, monitoring is important for successful interactions, particularly those involving language, and monitoring others' utterances probably relies on similar mechanisms as monitoring one's own. We investigated the effect of monitoring on the integration of self- and other-generated utterances in the shared-Stroop task. In a solo version of the Stroop task (with a single participant responding to all stimuli; Experiment 1), participants named the ink colour of mismatching colour words (incongruent stimuli) more slowly than matching colour words (congruent). In the shared-Stroop task, one participant named the ink colour of words in one colour (e.g. red), while ignoring stimuli in the other colour (e.g. green); the other participant either named the other ink colour or did not respond. Crucially, participants either provided feedback about the correctness of their partner's response (Experiment 3) or did not (Experiment 2). Interference was greater when both participants responded than when they did not, but only when their partners provided feedback. We argue that feedback increased interference because monitoring one's partner enhanced representations of the partner's target utterance, which in turn interfered with self-monitoring of the participant's own utterance.

EXPRESS: Representation of others' synchronous and asynchronous sentences interferes with sentence production

In dialogue, people represent each other's utterances in order to take turns and communicate successfully. In previous work [Gambi, C., Van de Cavey, J., & Pickering, M. J. (2015). Interference in joint picture naming. Journal of Experimental Psychology: Learning, Memory, and Cognition, 41(1), 1-21.], speakers who were naming single pictures or picture pairs represented whether another speaker was engaged in the same task (versus a different or no task) concurrently, but did not represent in detail the content of the other speaker's utterance. Here, we investigate co-representation of whole sentences. In three experiments, pairs of speakers imagined each other producing active or passive descriptions of transitive events. Speakers took longer to begin speaking when they believed their partner was also preparing to speak, compared to when they did not. Interference occurred when speakers believed their partners were preparing to speak at the same time as them (synchronous production and co-representation; Experiment 1), and also when speakers believed that their partner would speak only after them (asynchronous production and co-representation; Experiments 2a and 2b). However, interference was generally no greater when speakers believed their partner was preparing a different compared to a similar utterance, providing no consistent evidence that speakers represented what their partners were preparing to say. Taken together, these findings indicate that speakers can represent another's intention to speak even as they are themselves preparing to speak, but that such representation tends to lack detail.

Mental chronometry of speaking in dialogue: Semantic interference turns into facilitation

Numerous studies demonstrate that the production of words is delayed when speakers process in temporal proximity semantically related words. Yet the experimental settings underlying this effect are different from those under which we typically speak. This study demonstrates that semantic interference disappears, and can even turn into facilitation, when semantically related words are embedded in a meaningful communicative exchange. Experiment 1 and 3 (each N = 32 university students) implemented a picture-word interference task in a game played between two participants: one named the distractor word and, after a stimulus-onset-asynchrony of -150 ms or -650 ms, the other named a semantically related or unrelated target picture. Semantic interference reappeared with identical experimental parameters in a single-person picture-word interference setting (Experiment 2, N = 32). We conclude that the inhibitory context effects leading to semantic interference in single-subject settings are attenuated whereas facilitatory effects are enhanced in communicative settings.

Robots facilitate human language production

Despite recent developments in integrating autonomous and human-like robots into many aspects of everyday life, social interactions with robots are still a challenge. Here, we focus on a central tool for social interaction: verbal communication. We assess the extent to which humans co-represent (simulate and predict) a robot's verbal actions. During a joint picture naming task, participants took turns in naming objects together with a social robot (Pepper, Softbank Robotics). Previous findings using this task with human partners revealed internal simulations on behalf of the partner down to the level of selecting words from the mental lexicon, reflected in partner-elicited inhibitory effects on subsequent naming. Here, with the robot, the partner-elicited inhibitory effects were not observed. Instead, naming was facilitated, as revealed by faster naming of word categories co-named with the robot. This facilitation suggests that robots, unlike humans, are not simulated down to the level of lexical selection. Instead, a robot's speaking appears to be simulated at the initial level of language production where the meaning of the verbal message is generated, resulting in facilitated language production due to conceptual priming. We conclude that robots facilitate core conceptualization processes when humans transform thoughts to language during speaking.

Initiation of utterance planning in response to pre-recorded and "live" utterances

In everyday conversation, interlocutors often plan their utterances while listening to their conversational partners, thereby achieving short gaps between their turns. Important issues for current psycholinguistics are how interlocutors distribute their attention between listening and speech planning and how speech planning is timed relative to listening. Laboratory studies addressing these issues have used a variety of paradigms, some of which have involved using recorded speech to which participants responded, whereas others have involved interactions with confederates. This study investigated how this variation in the speech input affected the participants' timing of speech planning. In Experiment 1, participants responded to utterances produced by a confederate, who sat next to them and looked at the same screen. In Experiment 2, they responded to recorded utterances of the same confederate. Analyses of the participants' speech, their eye movements, and their performance in a concurrent tapping task showed that, compared with recorded speech, the presence of the confederate increased the processing load for the participants, but did not alter their global sentence planning strategy. These results have implications for the design of psycholinguistic experiments and theories of listening and speaking in dyadic settings.

Mental representations of partner task cause interference in picture naming

Interference in picture naming occurs from representing a partner's preparations to speak (Gambi, van de Cavey, & Pickering, 2015). We tested the origins of this interference using a simple non-communicative joint naming task based on Gambi et al. (2015), where response latencies indexed interference from partner task and partner speech content, and eye fixations to partner objects indexed overt attention. Experiment 1 contrasted a partner-present condition with a control partner-absent condition to establish the role of the partner in eliciting interference. For latencies, we observed interference from the partner's task and speech content, with interference increasing due to partner task in the partner-present condition. Eye-tracking measures showed that interference in naming was not due to overt attention to partner stimuli but to broad expectations about likely utterances. Experiment 2 examined whether an equivalent non-verbal task also elicited interference, as predicted from a language as joint action framework. We replicated the finding of interference due to partner task and again found no relationship between overt attention and interference. These results support Gambi et al. (2015). Individuals co-represent a partner's task while speaking, and doing so does not require overt attention to partner stimuli.

Slow naming of pictures facilitates memory for their names

Speakers remember their own utterances better than those of their interlocutors, suggesting that language production is beneficial to memory. This may be partly explained by a generation effect: The act of generating a word is known to lead to a memory advantage (Slamecka & Graf, 1978). In earlier work, we showed a generation effect for recognition of images (Zormpa, Brehm, Hoedemaker, & Meyer, 2019). Here, we tested whether the recognition of their names would also benefit from name generation. Testing whether picture naming improves memory for words was our primary aim, as it serves to clarify whether the representations affected by generation are visual or conceptual/lexical. A secondary aim was to assess the influence of processing time on memory. Fifty-one participants named pictures in three conditions: after hearing the picture name (identity condition), backward speech, or an unrelated word. A day later, recognition memory was tested in a yes/no task. Memory in the backward speech and unrelated conditions, which required generation, was superior to memory in the identity condition, which did not require generation. The time taken by participants for naming was a good predictor of memory, such that words that took longer to be retrieved were remembered better. Importantly, that was the case only when generation was required: In the no-generation (identity) condition, processing time was not related to recognition memory performance. This work has shown that generation affects conceptual/lexical representations, making an important contribution to the understanding of the relationship between memory and language.

Word naming slows picture naming but does not affect cumulative semantic interference

Two experiments are reported which investigate the effect of processing words prior to naming target pictures. In Experiment 1, participants named (read aloud) sequences of five printed prime words and five target pictures from the same semantic category, and also sequences of five prime words from a different unrelated semantic category to the five related target pictures. Picture and words were interleaved, with two unrelated filler stimuli in between prime and target stimuli (i.e. a lag of 3 between primes and targets). Results showed that across the five target picture naming trials (i.e. across ordinal position of picture), picture naming times increased linearly, replicating the cumulative semantic interference (CSI) effect (e.g., Howard, Nickels, Coltheart, & Cole-Virtue, 2006). Related prime words slowed picture naming, replicating the effects found in paired word prime and picture target studies (e.g., Tree & Hirsh, 2003). However, the naming of the five related prime words did not modify the picture naming CSI effect, with this null result converging with findings from a different word and picture design (e.g., Navarrete, Mahon, & Caramazza, 2010). In Experiment 2, participants categorised the prime word stimuli as manmade versus natural, so that words were more fully processed at a conceptual level. The interaction between word prime relatedness and ordinal position of the named target picture was significant. These results are consistent with adjustments at the conceptual level (Belke, 2013; Roelofs, 2018) which last over several trials at least. By contrast, we conclude that the distinct word-to-picture naming interference effect from Experiment 1 must originate outside of the conceptual level and outside of the mappings between semantics and lexical representations. We discuss the results with reference to recent theoretical accounts of the CSI picture naming effect and word naming models.

The production effect and the generation effect improve memory in picture naming

The production effect (better memory for words read aloud than words read silently) and the picture superiority effect (better memory for pictures than words) both improve item memory in a picture naming task (Fawcett, J. M., Quinlan, C. K., & Taylor, T. L. (2012). Interplay of the production and picture superiority effects: A signal detection analysis. Memory (Hove, England), 20(7), 655-666. doi: 10.1080/09658211.2012.693510 ). Because picture naming requires coming up with an appropriate label, the generation effect (better memory for generated than read words) may contribute to the latter effect. In two forced-choice memory experiments, we tested the role of generation in a picture naming task on later recognition memory. In Experiment 1, participants named pictures silently or aloud with the correct name or an unreadable label superimposed. We observed a generation effect, a production effect, and an interaction between the two. In Experiment 2, unreliable labels were included to ensure full picture processing in all conditions. In this experiment, we observed a production and a generation effect but no interaction, implying the effects are dissociable. This research demonstrates the separable roles of generation and production in picture naming and their impact on memory. As such, it informs the link between memory and language production and has implications for memory asymmetries between language production and comprehension.

Co-actors represent the order of each other's actions

Previous research has shown that people represent each other’s tasks and actions when acting together. However, less is known about how co-actors represent each other’s action sequences. Here, we asked whether co-actors represent the order of each other’s actions within an action sequence, or whether they merely represent the intended end state of a joint action together with their own contribution. In the present study, two co-actors concurrently performed action sequences composed of two actions. We predicted that if co-actors represent the order of each other’s actions, they should experience interference when the order of their actions differs. Supporting this prediction, the results of six experiments consistently showed that co-actors moved more slowly when performing the same actions in a different order compared to performing the same actions in the same order. In line with findings from bimanual movement tasks, our results indicate that interference can arise due to differences in movement parameters and due to differences in the perceptual characteristics of movement goals. The present findings extend previous research on co-representation, providing evidence that people represent not only the elements of another’s task, but also their temporal structure.