Never Say No... How the Brain Interprets the Pregnant Pause in Conversation

A Corpus Study on the Difference of Turn-Taking in Online Audio, Online Video, and Face-to-Face Conversation

Daily conversation is usually face-to-face and characterized by rapid and fluent exchange of turns between interlocutors. With the need to communicate across long distances, advances in communication media, online audio communication, and online video communication have become convenient alternatives for an increasing number of people. However, the fluency of turn-taking may be influenced when people communicate using these different modes. In this study, we conducted a corpus analysis of face-to-face, online audio, and online video conversations collected from the internet. The fluency of turn-taking in face-to-face conversations differed from that of online audio and video conversations. Namely, the timing of turn-taking was shorter and with more overlaps in face-to-face conversations compared with online audio and video conversations. This can be explained by the limited ability of online communication modes to transmit non-verbal cues and network latency. In addition, our study could not completely exclude the effect of formality of conversation. The present findings have implications for the rules of turn-taking in human online conversations, in that the traditional rule of no-gap-no-overlap may not be fully applicable to online conversations.

Word frequency and cognitive effort in turns-at-talk: turn structure affects processing load in natural conversation

Frequency distributions are known to widely affect psycholinguistic processes. The effects of word frequency in turns-at-talk, the nucleus of social action in conversation, have, by contrast, been largely neglected. This study probes into this gap by applying corpus-linguistic methods on the conversational component of the British National Corpus (BNC) and the Freiburg Multimodal Interaction Corpus (FreMIC). The latter includes continuous pupil size measures of participants of the recorded conversations, allowing for a systematic investigation of patterns in the contained speech and language on the one hand and their relation to concurrent processing costs they may incur in speakers and recipients on the other hand. We test a first hypothesis in this vein, analyzing whether word frequency distributions within turns-at-talk are correlated with interlocutors' processing effort during the production and reception of these turns. Turns are found to generally show a regular distribution pattern of word frequency, with highly frequent words in turn-initial positions, mid-range frequency words in turn-medial positions, and low-frequency words in turn-final positions. Speakers' pupil size is found to tend to increase during the course of a turn at talk, reaching a climax toward the turn end. Notably, the observed decrease in word frequency within turns is inversely correlated with the observed increase in pupil size in speakers, but not in recipients, with steeper decreases in word frequency going along with steeper increases in pupil size in speakers. We discuss the implications of these findings for theories of speech processing, turn structure, and information packaging. Crucially, we propose that the intensification of processing effort in speakers during a turn at talk is owed to an informational climax, which entails a progression from high-frequency, low-information words through intermediate levels to low-frequency, high-information words. At least in English conversation, interlocutors seem to make use of this pattern as one way to achieve efficiency in conversational interaction, creating a regularly recurring distribution of processing load across speaking turns, which aids smooth turn transitions, content prediction, and effective information transfer.

Simulation of other-initiated repair using AAC

This simulation study assessed the ability of Speech-Output Technologies (SOTs) to keep in-time during conversational repair. Fifty-eight Other Initiated Repair (OIR) initiators were collected from transcripts of repair interaction sequences collected from past research. A range of selection latencies were then used to calculate simulated utterance composition delays for the OIR initiators using two popular SOT software apps, with and without the use of word prediction. To determine whether OIR utterances could be produced within a socially sensitive temporal gap, composition delay was compared to a conservative temporal limit obtained for oral communicators (Kendrick, 2015). Even at the fastest 0.5 s selection latency level, utterance-level composition delays for both SOTs were substantially greater than the OIR limit set for this study. Next, AAC production rate data spanning a variety of technologies, access methods, tasks and user profiles was obtained from the literature. Communication performance for these groups was then evaluated against the identified temporal OIR limit. None of the user groups were found to be capable of producing full OIR utterances within the temporal limits of oral-speech conversation, with most unable to type even a single selection within these bounds. Because of the frequency and importance of repair in conversation, these results have important implications for designing devices to enable their users to successfully engage in such important conversational activities.

Conversational Eyebrow Frowns Facilitate Question Identification: An Online Study Using Virtual Avatars

Conversation is a time-pressured environment. Recognizing a social action (the ''speech act,'' such as a question requesting information) early is crucial in conversation to quickly understand the intended message and plan a timely response. Fast turns between interlocutors are especially relevant for responses to questions since a long gap may be meaningful by itself. Human language is multimodal, involving speech as well as visual signals from the body, including the face. But little is known about how conversational facial signals contribute to the communication of social actions. Some of the most prominent facial signals in conversation are eyebrow movements. Previous studies found links between eyebrow movements and questions, suggesting that these facial signals could contribute to the rapid recognition of questions. Therefore, we aimed to investigate whether early eyebrow movements (eyebrow frown or raise vs. no eyebrow movement) facilitate question identification. Participants were instructed to view videos of avatars where the presence of eyebrow movements accompanying questions was manipulated. Their task was to indicate whether the utterance was a question or a statement as accurately and quickly as possible. Data were collected using the online testing platform Gorilla. Results showed higher accuracies and faster response times for questions with eyebrow frowns, suggesting a facilitative role of eyebrow frowns for question identification. This means that facial signals can critically contribute to the communication of social actions in conversation by signaling social action-specific visual information and providing visual cues to speakers' intentions.

Gesture, spatial cognition and the evolution of language

Human communication displays a striking contrast between the diversity of languages and the universality of the principles underlying their use in conversation. Despite the importance of this interactional base, it is not obvious that it heavily imprints the structure of languages. However, a deep-time perspective suggests that early hominin communication was gestural, in line with all the other Hominidae. This gestural phase of early language development seems to have left its traces in the way in which spatial concepts, implemented in the hippocampus, provide organizing principles at the heart of grammar. This article is part of a discussion meeting issue 'Face2face: advancing the science of social interaction'.

Beyond speaking: neurocognitive perspectives on language production in social interaction

The human faculty to speak has evolved, so has been argued, for communicating with others and for engaging in social interactions. Hence the human cognitive system should be equipped to address the demands that social interaction places on the language production system. These demands include the need to coordinate speaking with listening, the need to integrate own (verbal) actions with the interlocutor's actions, and the need to adapt language flexibly to the interlocutor and the social context. In order to meet these demands, core processes of language production are supported by cognitive processes that enable interpersonal coordination and social cognition. To fully understand the cognitive architecture and its neural implementation enabling humans to speak in social interaction, our understanding of how humans produce language needs to be connected to our understanding of how humans gain insights into other people's mental states and coordinate in social interaction. This article reviews theories and neurocognitive experiments that make this connection and can contribute to advancing our understanding of speaking in social interaction. This article is part of a discussion meeting issue 'Face2face: advancing the science of social interaction'.

Multimodal Assemblies for Prefacing a Dispreferred Response: A Cross-Linguistic Analysis

In this paper we examine how participants’ multimodal conduct maps onto one of the basic organizational principles of social interaction: preference organization – and how it does so in a similar manner across five different languages (Czech, French, Hebrew, Mandarin, and Romanian). Based on interactional data from these languages, we identify a recurrent multimodal practice that respondents deploy in turn-initial position in dispreferred responses to various first actions, such as information requests, assessments, proposals, and informing. The practice involves the verbal delivery of a turn-initial expression corresponding to English ‘I don’t know’ and its variants (‘dunno’) coupled with gaze aversion from the prior speaker. We show that through this ‘multimodal assembly’ respondents preface a dispreferred response within various sequence types, and we demonstrate the cross-linguistic robustness of this practice: Through the focal multimodal assembly, respondents retrospectively mark the prior action as problematic and prospectively alert co-participants to incipient resistance to the constraints set out or to the stance conveyed by that action. By evidencing how grammar and body interface in related ways across a diverse set of languages, the findings open a window onto cross-linguistic, cross-modal, and cross-cultural consistencies in human interactional conduct.

Doing Contrariness: Therapeutic Talk-In-Interaction in a Single Therapy Session With a Traumatized Child

Conversation analysis (CA) of children-adult-interaction in various contexts has become an established field of research. However, child therapy has received limited attention in CA. In child therapy, the general psychotherapeutic practice of achieving empathy faces particular challenges. In relation to this, our contribution sets out three issues for investigation and analysis: the first one is that practices of achieving empathy must be preceded by efforts aiming to establish which kind of individualized conversation works with this child (Midgley, 2006). Psychotherapy process researchers in adult therapy (Stiles et al., 2015) have found that therapists "invent" a new therapy for each patient (Norcross and Wampold, 2018). The second issue is that it can be difficult for adults to understand the ways in which children express their conflicts and issues. In particular, play activities in therapy, e.g., with dolls, can open up additional scenarios of interaction. The play scenario can be used to disclose unformulated problems masked in everyday and family interactions. The third issue is how to respect the child's higher degree of vulnerability, compared with adult patients. How is it communicated and dealt with in therapy? We present an interaction analysis of a single case study of the first 20 min of a child therapy session with an adopted girl aged 4 years brought to treatment because of "unexplainable rage." The session was videotaped; parents granted permission. We analyze this session using an applied version of CA. In our analysis, we describe "doing contrariness," as a conversational practice producing epistemic and affiliative disruptions, while "avoiding doing contrariness" and "remedying contrariness" are strategies for preserving or restoring the affiliative dimension of a relationship (in child therapy). We show how these practices operate in various modes and how they are used by both parties in our case study to variously aid and impedethe achievement of empathy and understanding.

Expectancy effects in the EEG during joint and spontaneous word-by-word sentence production in German

Our aim in the present study is to measure neural correlates during spontaneous interactive sentence production. We present a novel approach using the word-by-word technique from improvisational theatre, in which two speakers jointly produce one sentence. This paradigm allows the assessment of behavioural aspects, such as turn-times, and electrophysiological responses, such as event-related-potentials (ERPs). Twenty-five participants constructed a cued but spontaneous four-word German sentence together with a confederate, taking turns for each word of the sentence. In 30% of the trials, the confederate uttered an unexpected gender-marked article. To complete the sentence in a meaningful way, the participant had to detect the violation and retrieve and utter a new fitting response. We found significant increases in response times after unexpected words and - despite allowing unscripted language production and naturally varying speech material - successfully detected significant N400 and P600 ERP effects for the unexpected word. The N400 EEG activity further significantly predicted the response time of the subsequent turn. Our results show that combining behavioural and neuroscientific measures of verbal interactions while retaining sufficient experimental control is possible, and that this combination provides promising insights into the mechanisms of spontaneous spoken dialogue.

Studying brain activity during word-by-word interactions using wireless EEG

We introduce here the word-by-word paradigm, a dynamic setting, in which two people take turns in producing a single sentence. This task requires a high degree of coordination between the partners and the simplicity of the task allows us to study with sufficient experimental control behavioral and neural processes that underlie this controlled interaction. For this study, 13 pairs of individuals engaged in a scripted word-by-word interaction, while we recorded the neural activity of both participants simultaneously using wireless EEG. To study expectation building, different semantic contexts were primed for each participant. Semantically unexpected continuations were introduced in 25% of all sentences. In line with the hypothesis, we observed amplitude differences for the P200-N400-P600 ERPs for unexpected compared to expected words. Moreover, we could successfully assess speech and reaction times. Our results show that it is possible to measure ERPs and RTs to semantically unexpected words in a dyadic interactive scenario.

Initiation of utterance planning in response to pre-recorded and "live" utterances

In everyday conversation, interlocutors often plan their utterances while listening to their conversational partners, thereby achieving short gaps between their turns. Important issues for current psycholinguistics are how interlocutors distribute their attention between listening and speech planning and how speech planning is timed relative to listening. Laboratory studies addressing these issues have used a variety of paradigms, some of which have involved using recorded speech to which participants responded, whereas others have involved interactions with confederates. This study investigated how this variation in the speech input affected the participants' timing of speech planning. In Experiment 1, participants responded to utterances produced by a confederate, who sat next to them and looked at the same screen. In Experiment 2, they responded to recorded utterances of the same confederate. Analyses of the participants' speech, their eye movements, and their performance in a concurrent tapping task showed that, compared with recorded speech, the presence of the confederate increased the processing load for the participants, but did not alter their global sentence planning strategy. These results have implications for the design of psycholinguistic experiments and theories of listening and speaking in dyadic settings.

Multimodal Language Processing in Human Communication

The natural ecology of human language is face-to-face interaction comprising the exchange of a plethora of multimodal signals. Trying to understand the psycholinguistic processing of language in its natural niche raises new issues, first and foremost the binding of multiple, temporally offset signals under tight time constraints posed by a turn-taking system. This might be expected to overload and slow our cognitive system, but the reverse is in fact the case. We propose cognitive mechanisms that may explain this phenomenon and call for a multimodal, situated psycholinguistic framework to unravel the full complexities of human language processing.

Working Together: Contributions of Corpus Analyses and Experimental Psycholinguistics to Understanding Conversation

As conversation is the most important way of using language, linguists and psychologists should combine forces to investigate how interlocutors deal with the cognitive demands arising during conversation. Linguistic analyses of corpora of conversation are needed to understand the structure of conversations, and experimental work is indispensable for understanding the underlying cognitive processes. We argue that joint consideration of corpus and experimental data is most informative when the utterances elicited in a lab experiment match those extracted from a corpus in relevant ways. This requirement to compare like with like seems obvious but is not trivial to achieve. To illustrate this approach, we report two experiments where responses to polar (yes/no) questions were elicited in the lab and the response latencies were compared to gaps between polar questions and answers in a corpus of conversational speech. We found, as expected, that responses were given faster when they were easy to plan and planning could be initiated earlier than when they were harder to plan and planning was initiated later. Overall, in all but one condition, the latencies were longer than one would expect based on the analyses of corpus data. We discuss the implication of this partial match between the data sets and more generally how corpus and experimental data can best be combined in studies of conversation.

The N400 Effect during Speaker-Switch-Towards a Conversational Approach of Measuring Neural Correlates of Language

Language occurs naturally in conversations. However, the study of the neural underpinnings of language has mainly taken place in single individuals using controlled language material. The interactive elements of a conversation (e.g., turn-taking) are often not part of neurolinguistic setups. The prime reason is the difficulty to combine open unrestricted conversations with the requirements of neuroimaging. It is necessary to find a trade-off between the naturalness of a conversation and the restrictions imposed by neuroscientific methods to allow for ecologically more valid studies. Here, we make an attempt to study the effects of a conversational element, namely turn-taking, on linguistic neural correlates, specifically the N400 effect. We focus on the physiological aspect of turn-taking, the speaker-switch, and its effect on the detectability of the N400 effect. The N400 event-related potential reflects expectation violations in a semantic context; the N400 effect describes the difference of the N400 amplitude between semantically expected and unexpected items. Sentences with semantically congruent and incongruent final words were presented in two turn-taking modes: (1) reading aloud first part of the sentence and listening to speaker-switch for the final word, and (2) listening to first part of the sentence and speaker-switch for the final word. A significant N400 effect was found for both turn-taking modes, which was not influenced by the mode itself. However, the mode significantly affected the P200, which was increased for the reading aloud mode compared to the listening mode. Our results show that an N400 effect can be detected during a speaker-switch. Speech articulation (reading aloud) before the analyzed sentence fragment did also not impede the N400 effect detection for the final word. The speaker-switch, however, seems to influence earlier components of the electroencephalogram, related to processing of salient stimuli. We conclude that the N400 can effectively be used to study neural correlates of language in conversational approaches including speaker-switches.

The Timing of Utterance Planning in Task-Oriented Dialogue: Evidence from a Novel List-Completion Paradigm

In conversation, interlocutors rarely leave long gaps between turns, suggesting that next speakers begin to plan their turns while listening to the previous speaker. The present experiment used analyses of speech onset latencies and eye-movements in a task-oriented dialogue paradigm to investigate when speakers start planning their responses. German speakers heard a confederate describe sets of objects in utterances that either ended in a noun [e.g., Ich habe eine Tür und ein Fahrrad ("I have a door and a bicycle")] or a verb form [e.g., Ich habe eine Tür und ein Fahrrad besorgt ("I have gotten a door and a bicycle")], while the presence or absence of the final verb either was or was not predictable from the preceding sentence structure. In response, participants had to name any unnamed objects they could see in their own displays with utterances such as Ich habe ein Ei ("I have an egg"). The results show that speakers begin to plan their turns as soon as sufficient information is available to do so, irrespective of further incoming words.