Prediction and embodiment in dialogue

Overrated gaps: Inter-speaker gaps provide limited information about the timing of turns in conversation

Corpus analyses have shown that turn-taking in conversation is much faster than laboratory studies of speech planning would predict. To explain fast turn-taking, Levinson and Torreira (2015) proposed that speakers are highly proactive: They begin to plan a response to their interlocutor's turn as soon as they have understood its gist, and launch this planned response when the turn-end is imminent. Thus, fast turn-taking is possible because speakers use the time while their partner is talking to plan their own utterance. In the present study, we asked how much time upcoming speakers actually have to plan their utterances. Following earlier psycholinguistic work, we used transcripts of spoken conversations in Dutch, German, and English. These transcripts consisted of segments, which are continuous stretches of speech by one speaker. In the psycholinguistic and phonetic literature, such segments have often been used as proxies for turns. We found that in all three corpora, large proportions of the segments comprised of only one or two words, which on our estimate does not give the next speaker enough time to fully plan a response. Further analyses showed that speakers indeed often did not respond to the immediately preceding segment of their partner, but continued an earlier segment of their own. More generally, our findings suggest that speech segments derived from transcribed corpora do not necessarily correspond to turns, and the gaps between speech segments therefore only provide limited information about the planning and timing of turns.

Language and Movement Synchronization in Dyadic Psychotherapeutic Interaction - A Qualitative Review and a Proposal for a Classification

In individual psychotherapy verbal communication and movement synchronization are closely interrelated. The microanalysis of timing, rhythm and gestalt of movement has established dynamic movement coordination as a systemic property of the dyadic interaction. Movement synchronization supports and enhances the unfolding of linguistic meaning. In order to substantiate the importance of the concept of synchrony for adult psychotherapy we review evidence from developmental psychology and discuss approaches to measure synchrony with particular reference to the naturalistic setting of dyadic psychotherapy. As the concept of synchrony is still ambiguous, and the respective interactional phenomena are ephemeral and fluid, in the current paper we suggest a set of five criteria for the description of synchronization in general terms and eight additional criteria which specifically enable the description of phenomena of movement synchronization. The five general dimensions are: (1) context, (2) modality, (3) resources, (4) entrainment, and (5) time-lag. The eight categories for the description of movement synchrony are: (1) spatial direction, (2) amplitude, (3) sinuosity, (4) duration, (5) event structure, (6) phase, (7) frequency, and (8) content. To understand the process of participatory sense-making and the emergence of meaning in psychotherapy, synchrony research has to cope with the multimodality of the embodied interaction. This requires an integrated perspective of movement and language. A system for the classification of synchrony phenomena may contribute to the linking of variations and patterns of movement with language and linguistic utterances.

Stimulus complexity matters when you hear your own voice: Attention effects on self-generated voice processing

The ability to discriminate self- and non-self voice cues is a fundamental aspect of self-awareness and subserves self-monitoring during verbal communication. Nonetheless, the neurofunctional underpinnings of self-voice perception and recognition are still poorly understood. Moreover, how attention and stimulus complexity influence the processing and recognition of one's own voice remains to be clarified. Using an oddball task, the current study investigated how self-relevance and stimulus type interact during selective attention to voices, and how they affect the representation of regularity during voice perception. Event-related potentials (ERPs) were recorded from 18 right-handed males. Pre-recorded self-generated (SGV) and non-self (NSV) voices, consisting of a nonverbal vocalization (vocalization condition) or disyllabic word (word condition), were presented as either standard or target stimuli in different experimental blocks. The results showed increased N2 amplitude to SGV relative to NSV stimuli. Stimulus type modulated later processing stages only: P3 amplitude was increased for SGV relative to NSV words, whereas no differences between SGV and NSV were observed in the case of vocalizations. Moreover, SGV standards elicited reduced N1 and P2 amplitude relative to NSV standards. These findings revealed that the self-voice grabs more attention when listeners are exposed to words but not vocalizations. Further, they indicate that detection of regularity in an auditory stream is facilitated for one's own voice at early processing stages. Together, they demonstrate that self-relevance affects attention to voices differently as a function of stimulus type.

Extending Situated Language Comprehension (Accounts) with Speaker and Comprehender Characteristics: Toward Socially Situated Interpretation

More and more findings suggest a tight temporal coupling between (non-linguistic) socially interpreted context and language processing. Still, real-time language processing accounts remain largely elusive with respect to the influence of biological (e.g., age) and experiential (e.g., world and moral knowledge) comprehender characteristics and the influence of the 'socially interpreted' context, as for instance provided by the speaker. This context could include actions, facial expressions, a speaker's voice or gaze, and gestures among others. We review findings from social psychology, sociolinguistics and psycholinguistics to highlight the relevance of (the interplay between) the socially interpreted context and comprehender characteristics for language processing. The review informs the extension of an extant real-time processing account (already featuring a coordinated interplay between language comprehension and the non-linguistic visual context) with a variable ('ProCom') that captures characteristics of the language user and with a first approximation of the comprehender's speaker representation. Extending the CIA to the sCIA (social Coordinated Interplay Account) is the first step toward a real-time language comprehension account which might eventually accommodate the socially situated communicative interplay between comprehenders and speakers.

Linguistic initiation signals increase auditory feedback error correction

Previous research has shown that speakers can adapt their speech in a flexible manner as a function of a variety of contextual and task factors. While it is known that speech tasks may play a role in speech motor behavior, it remains to be explored if the manner in which the speaking action is initiated can modify low-level, automatic control of vocal motor action. In this study, the nature (linguistic vs non-linguistic) and modality (auditory vs visual) of the go signal (i.e., the prompts) was manipulated in an otherwise identical vocal production task. Participants were instructed to produce the word "head" when prompted, and the auditory feedback they were receiving was altered by systematically changing the first formants of the vowel /ε/ in real time using a custom signal processing system. Linguistic prompts induced greater corrective behaviors to the acoustic perturbations than non-linguistic prompts. This suggests that the accepted variance for the intended speech sound decreases when external linguistic templates are provided to the speaker. Overall, this result shows that the automatic correction of vocal errors is influenced by flexible, context-dependant mechanisms.

The costs of being certain: Brain potential evidence for linguistic preactivation in sentence processing

Prediction in sentence comprehension is often investigated by measuring the amplitude of the N400 ERP component to words that are more or less predictable from their preceding context. The N400-linked to the activation of word-associated semantic information-is reduced for words that are predictable, indicating that preactivation can lead to facilitated processing. We addressed the question whether there is measurable neural activity related to the preactivation of linguistic information before input confirms or disconfirms this prediction. We therefore measured ERPs not only to moderately to highly predictable target words, but also to preceding adverbs. Based on two separate cloze pretests, we quantified the impact of the adverb upon the predictability of the subsequent target word. Using linear mixed-effects analyses, we could show that the N400 amplitude at the target word was inversely related to target cloze value, thus replicating the finding that prediction has a facilitative effect on semantic processing. Crucially, the N400 amplitude at the pretarget adverb was modulated by adverb impact: When adverbs increased the predictability of the following word, the N400 was more negative going. We argue that this effect is related to the preactivation of linguistic information. Our findings indicate that the specification of predictions can lead to additional processes before these predictions are confirmed or disconfirmed and that activation of word-associated information through prediction is highly comparable to activation through actual input.

Joint Cognition: Thought Contagion and the Consequences of Cooperation when Sharing the Task of Random Sequence Generation

Generating random number sequences is a popular psychological task often used to measure executive functioning. We explore random generation under "joint cognition" instructions; pairs of participants take turns to compile a shared response sequence. Across three studies, we point to six key findings from this novel format. First, there are both costs and benefits from group performance. Second, repetition avoidance occurs in dyadic as well as individual production settings. Third, individuals modify their choices in a dyadic situation such that the pair becomes the unit of psychological function. Fourth, there is immediate contagion of sequence stereotypy amongst the pairs (i.e., each contributor "owns" their partner's response). Fifth, dyad effects occur even when participants know their partner is not interacting with them (Experiment 2). Sixth, ironically, directing participants' efforts away from their shared task responsibility can actually benefit conjoint performance (Experiment 3). These results both constrain models of random generation and illuminate processes of joint cognition.

Brain basis of communicative actions in language

Although language is a key tool for communication in social interaction, most studies in the neuroscience of language have focused on language structures such as words and sentences. Here, the neural correlates of speech acts, that is, the actions performed by using language, were investigated with functional magnetic resonance imaging (fMRI). Participants were shown videos, in which the same critical utterances were used in different communicative contexts, to Name objects, or to Request them from communication partners. Understanding of critical utterances as Requests was accompanied by activation in bilateral premotor, left inferior frontal and temporo-parietal cortical areas known to support action-related and social interactive knowledge. Naming, however, activated the left angular gyrus implicated in linking information about word forms and related reference objects mentioned in critical utterances. These findings show that understanding of utterances as different communicative actions is reflected in distinct brain activation patterns, and thus suggest different neural substrates for different speech act types.

Environmental constraints shaping constituent order in emerging communication systems: Structural iconicity, interactive alignment and conventionalization

Where does linguistic structure come from? Recent gesture elicitation studies have indicated that constituent order (corresponding to for instance subject-verb-object, or SVO in English) may be heavily influenced by human cognitive biases constraining gesture production and transmission. Here we explore the alternative hypothesis that syntactic patterns are motivated by multiple environmental and social-interactional constraints that are external to the cognitive domain. In three experiments, we systematically investigate different motivations for structure in the gestural communication of simple transitive events. The first experiment indicates that, if participants communicate about different types of events, manipulation events (e.g. someone throwing a cake) and construction events (e.g. someone baking a cake), they spontaneously and systematically produce different constituent orders, SOV and SVO respectively, thus following the principle of structural iconicity. The second experiment shows that participants' choice of constituent order is also reliably influenced by social-interactional forces of interactive alignment, that is, the tendency to re-use an interlocutor's previous choice of constituent order, thus potentially overriding affordances for iconicity. Lastly, the third experiment finds that the relative frequency distribution of referent event types motivates the stabilization and conventionalization of a single constituent order for the communication of different types of events. Together, our results demonstrate that constituent order in emerging gestural communication systems is shaped and stabilized in response to multiple external environmental and social factors: structural iconicity, interactive alignment and distributional frequency.

The embodied nature of medical concepts: image schemas and language for PAIN

Cognitive linguistics assumes that knowledge is both embodied and situated as far as it is acquired through our bodily interaction with the world in a specific environment (e.g. Barsalou in Lang Cogn Process 18:513-562, 2003; Connell et al. in PLoS One 7:3, 2012). Therefore, embodiment provides an explanation to the mental representation and linguistic expression of concepts. Among the first, we find multimodal conceptual structures, like image schemas, which are schematic representations of embodied experiences resulting from our conceptualization of the surrounding environment (Tercedor Sánchez et al. in J Spec Transl 18:187-205, 2012). Furthermore, the way we interact with the environment and its objects is dynamic and configures how we refer to concepts both by means of images and lexicalizations. In this article, we investigate how image schemas underlie verbal and visual representations. They both evoke concepts based on exteroception, interoception and proprioception which can be lexicalized through language. More specifically, we study (1) a multimodal corpus of medical texts to examine how image schemas lexicalize in the language of medicine to represent specialized concepts and (2) medical pictures to explore the depiction of image-schematic concepts, in order to account for the verbal and visual representation of embodied concepts. We explore the concept PAIN, a sensory and emotional experience associated with actual or potential tissue damage, using corpus analysis tools (Sketch Engine) to extract information about the lexicalization of underlying image schemas in definitions and defining contexts. Then, we use the image schemas behind medical concepts to consistently select images which depict our experience of pain and the way we understand it. Finally, such lexicalizations and visualizations will help us assess how we refer to PAIN both verbally and visually.

TEST: a tropic, embodied, and situated theory of cognition

TEST is a novel taxonomy of knowledge representations based on three distinct hierarchically organized representational features: Tropism, Embodiment, and Situatedness. Tropic representational features reflect constraints of the physical world on the agent's ability to form, reactivate, and enrich embodied (i.e., resulting from the agent's bodily constraints) conceptual representations embedded in situated contexts. The proposed hierarchy entails that representations can, in principle, have tropic features without necessarily having situated and/or embodied features. On the other hand, representations that are situated and/or embodied are likely to be simultaneously tropic. Hence, although we propose tropism as the most general term, the hierarchical relationship between embodiment and situatedness is more on a par, such that the dominance of one component over the other relies on the distinction between offline storage versus online generation as well as on representation-specific properties.

Female fertility affects men's linguistic choices

We examined the influence of female fertility on the likelihood of male participants aligning their choice of syntactic construction with those of female confederates. Men interacted with women throughout their menstrual cycle. On critical trials during the interaction, the confederate described a picture to the participant using particular syntactic constructions. Immediately thereafter, the participant described to the confederate a picture that could be described using either the same construction that was used by the confederate or an alternative form of the construction. Our data show that the likelihood of men choosing the same syntactic structure as the women was inversely related to the women's level of fertility: higher levels of fertility were associated with lower levels of linguistic matching. A follow-up study revealed that female participants do not show this same change in linguistic behavior as a function of changes in their conversation partner's fertility. We interpret these findings in the context of recent data suggesting that non-conforming behavior may be a means of men displaying their fitness as a mate to women.