Timing in turn-taking and its implications for processing models of language

Language in interaction: turn-taking patterns in conversations involving individuals with schizophrenia

Individuals with schizophrenia generally show difficulties in interpersonal communication. Linguistic analyses shed new light on speech atypicalities in schizophrenia. However, very little is known about conversational interaction management by these individuals. Moreover, the relationship between linguistic features, psychopathology, and patients' subjectivity has received limited attention to date. We used a novel methodology to explore dyadic conversations involving 58 participants (29 individuals with schizophrenia and 29 control persons) and medical doctors. High-quality stereo recordings were obtained and used to quantify turn-taking patterns. We investigated psychopathological dimensions and subjective experiences using the Positive and Negative Syndrome Scale for Schizophrenia (PANSS), the Examination of Anomalous Self Experience scale (EASE), the Autism Rating Scale (ARS) and the Abnormal Bodily Phenomena questionnaire (ABPq). Different turn-taking patterns of both patients and interviewers characterised conversations involving individuals with schizophrenia. We observed higher levels of overlap and mutual silence in dialogues with the patients compared to dialogues with control persons. Mutual silence was associated with negative symptom severity; no dialogical feature was correlated with anomalous subjective experiences. Our findings suggest that individuals with schizophrenia display peculiar turn-taking behaviour, thereby enhancing our understanding of interactional coordination in schizophrenia.

Speak Up: How Hearing Loss and the Lack of Hearing Aids Affect Conversations in Quiet

PURPOSE

The study examines the effect of hearing loss and hearing aid (HA) amplification on the conversational dynamics between hearing-impaired (HI) and normal-hearing (NH) interlocutors. Combining data from the current and a prior study, we explore how the speech levels of both interlocutors correlate and relate to HI interlocutors' degree of hearing loss.

METHOD

Sixteen pairs of younger NH and elderly HI interlocutors conversed in quiet, with the HI interlocutor either unaided or wearing HAs. We analyzed the effect of hearing status and HA amplification on the conversational dynamics, including turn-taking times (floor-transfer offsets), utterance lengths, and speech levels. Furthermore, we conducted an in-depth analysis of the speech levels using combined data sets from the current and previously published data by Petersen, MacDonald, and Sørensen (2022).

RESULTS

Unaided HI interlocutors were slower and more variable at timing their turns, but wearing HAs reduced the differences between the HI and NH interlocutors. Conversations were less interactive, and pairs were slower at solving the conversational tasks when the HI interlocutor was unaided. Both interlocutors spoke louder when the HI interlocutor was unaided. The speech level of the NH interlocutors was related to that of the HI interlocutors, with the HI speech levels also correlating with their own degree of hearing loss.

CONCLUSIONS

Despite typically being unchallenging for HI individuals, one-on-one conversations in quiet were impacted by the HI interlocutor not wearing HAs. Additionally, combining data sets revealed that NH interlocutors adjusted their speech level to match that of HI interlocutors.

The role of general cognitive skills in integrating visual and linguistic information during sentence comprehension: individual differences across the lifespan

Individuals exhibit massive variability in general cognitive skills that affect language processing. This variability is partly developmental. Here, we recruited a large sample of participants (N = 487), ranging from 9 to 90 years of age, and examined the involvement of nonverbal processing speed (assessed using visual and auditory reaction time tasks) and working memory (assessed using forward and backward Digit Span tasks) in a visual world task. Participants saw two objects on the screen and heard a sentence that referred to one of them. In half of the sentences, the target object could be predicted based on verb-selectional restrictions. We observed evidence for anticipatory processing on predictable compared to non-predictable trials. Visual and auditory processing speed had main effects on sentence comprehension and facilitated predictive processing, as evidenced by an interaction. We observed only weak evidence for the involvement of working memory in predictive sentence comprehension. Age had a nonlinear main effect (younger adults responded faster than children and older adults), but it did not differentially modulate predictive and non-predictive processing, nor did it modulate the involvement of processing speed and working memory. Our results contribute to delineating the cognitive skills that are involved in language-vision interactions.

Can Large Language Models Counter the Recent Decline in Literacy Levels? An Important Role for Cognitive Science

Literacy is in decline in many parts of the world, accompanied by drops in associated cognitive skills (including IQ) and an increasing susceptibility to fake news. It is possible that the recent explosive growth and widespread deployment of Large Language Models (LLMs) might exacerbate this trend, but there is also a chance that LLMs can help turn things around. We argue that cognitive science is ideally suited to help steer future literacy development in the right direction by challenging and informing current educational practices and policy. Cognitive scientists have the right interdisciplinary skills to study, analyze, evaluate, and change LLMs to facilitate their critical use, to encourage turn-taking that promotes rather than hinders literacy, to support literacy acquisition in diverse and equitable ways, and to scaffold potential future changes in what it means to be literate. We urge cognitive scientists to take up this mantle-the future impact of LLMs on human literacy skills is too important to be left to the large, predominately U.S.-based tech companies.

Neurocognition in adolescents and young adults at clinical high risk for psychosis: Predictive stability for social and role functioning

The prodromal phase of schizophrenia provides an optimal opportunity to mitigate the profound functional disability that is often associated with fully expressed psychosis. Considerable evidence supports the importance of neurocognition in the development of interpersonal (social) and academic (role) skills. Further findings from adolescents and young adults at clinical high risk for developing psychosis (CHRP) suggest that treatment for functioning might be most effective when targeting early and specific neurocognitive deficits. The current study addresses this critical intervention issue by examining the potential of neurocognitive deficits at intake for predicting social and role functioning over time in CHR-P youth. The study included 345 CHR-P participants from the second phase of the North American Prodrome Longitudinal Study (NAPLS2) with baseline neurocognition and 2-year follow-up data on social and role functioning. Slower baseline processing speed consistently predicted poor social functioning over time, while attention deficits predicted poor role functioning at baseline and follow-up. In addition, the impact of processing speed and attention impairments on social and role functioning, respectively, persisted even when adjusting the regression models for attenuated positive, negative, and disorganized symptoms, and transition status. The current study demonstrates for, arguably the first time, that processing speed and attention are strongly predictive of social and role functioning over time, respectively, above and beyond the impact of symptoms and those CHR-P individuals that develop psychosis over the course of the study. These findings imply that early neurocognition is a critical treatment target linked to the developmental trajectory of social and role functioning.

Visual bodily signals and conversational context benefit the anticipation of turn ends

The typical pattern of alternating turns in conversation seems trivial at first sight. But a closer look quickly reveals the cognitive challenges involved, with much of it resulting from the fast-paced nature of conversation. One core ingredient to turn coordination is the anticipation of upcoming turn ends so as to be able to ready oneself for providing the next contribution. Across two experiments, we investigated two variables inherent to face-to-face conversation, the presence of visual bodily signals and preceding discourse context, in terms of their contribution to turn end anticipation. In a reaction time paradigm, participants anticipated conversational turn ends better when seeing the speaker and their visual bodily signals than when they did not, especially so for longer turns. Likewise, participants were better able to anticipate turn ends when they had access to the preceding discourse context than when they did not, and especially so for longer turns. Critically, the two variables did not interact, showing that visual bodily signals retain their influence even in the context of preceding discourse. In a pre-registered follow-up experiment, we manipulated the visibility of the speaker's head, eyes and upper body (i.e. torso + arms). Participants were better able to anticipate turn ends when the speaker's upper body was visible, suggesting a role for manual gestures in turn end anticipation. Together, these findings show that seeing the speaker during conversation may critically facilitate turn coordination in interaction.

Estimating Systemic Cognitive States from a Mixture of Physiological and Brain Signals

As human-machine teams are being considered for a variety of mixed-initiative tasks, detecting and being responsive to human cognitive states, in particular systematic cognitive states, is among the most critical capabilities for artificial systems to ensure smooth interactions with humans and high overall team performance. Various human physiological parameters, such as heart rate, respiration rate, blood pressure, and skin conductance, as well as brain activity inferred from functional near-infrared spectroscopy or electroencephalogram, have been linked to different systemic cognitive states, such as workload, distraction, or mind-wandering among others. Whether these multimodal signals are indeed sufficient to isolate such cognitive states across individuals performing tasks or whether additional contextual information (e.g., about the task state or the task environment) is required for making appropriate inferences remains an important open problem. In this paper, we introduce an experimental and machine learning framework for investigating these questions and focus specifically on using physiological and neurophysiological measurements to learn classifiers associated with systemic cognitive states like cognitive load, distraction, sense of urgency, mind wandering, and interference. Specifically, we describe a multitasking interactive experimental setting used to obtain a comprehensive multimodal data set which provided the foundation for a first evaluation of various standard state-of-the-art machine learning techniques with respect to their effectiveness in inferring systemic cognitive states. While the classification success of these standard methods based on just the physiological and neurophysiological signals across subjects was modest, which is to be expected given the complexity of the classification problem and the possibility that higher accuracy rates might not in general be achievable, the results nevertheless can serve as a baseline for evaluating future efforts to improve classification, especially methods that take contextual aspects such as task and environmental states into account.

Stimulation of caudal inferior and middle frontal gyri disrupts planning during spoken interaction

Turn-taking is a central feature of conversation across languages and cultures.1,2,3,4 This key social behavior requires numerous sensorimotor and cognitive operations1,5,6 that can be organized into three general phases: comprehension of a partner's turn, preparation of a speaker's own turn, and execution of that turn. Using intracranial electrocorticography, we recently demonstrated that neural activity related to these phases is functionally distinct during turn-taking.7 In particular, networks active during the perceptual and articulatory stages of turn-taking consisted of structures known to be important for speech-related sensory and motor processing,8,9,10,11,12,13,14,15,16,17 while putative planning dynamics were most regularly observed in the caudal inferior frontal gyrus (cIFG) and the middle frontal gyrus (cMFG). To test if these structures are necessary for planning during spoken interaction, we used direct electrical stimulation (DES) to transiently perturb cortical function in neurosurgical patient-volunteers performing a question-answer task.7,18,19 We found that stimulating the cIFG and cMFG led to various response errors9,13,20,21 but not gross articulatory deficits, which instead resulted from DES of structures involved in motor control8,13,20,22 (e.g., the precentral gyrus). Furthermore, perturbation of the cIFG and cMFG delayed inter-speaker timing-consistent with slowed planning-while faster responses could result from stimulation of sites located in other areas. Taken together, our findings suggest that the cIFG and cMFG contain critical preparatory circuits that are relevant for interactive language use.

Word frequency and cognitive effort in turns-at-talk: turn structure affects processing load in natural conversation

Frequency distributions are known to widely affect psycholinguistic processes. The effects of word frequency in turns-at-talk, the nucleus of social action in conversation, have, by contrast, been largely neglected. This study probes into this gap by applying corpus-linguistic methods on the conversational component of the British National Corpus (BNC) and the Freiburg Multimodal Interaction Corpus (FreMIC). The latter includes continuous pupil size measures of participants of the recorded conversations, allowing for a systematic investigation of patterns in the contained speech and language on the one hand and their relation to concurrent processing costs they may incur in speakers and recipients on the other hand. We test a first hypothesis in this vein, analyzing whether word frequency distributions within turns-at-talk are correlated with interlocutors' processing effort during the production and reception of these turns. Turns are found to generally show a regular distribution pattern of word frequency, with highly frequent words in turn-initial positions, mid-range frequency words in turn-medial positions, and low-frequency words in turn-final positions. Speakers' pupil size is found to tend to increase during the course of a turn at talk, reaching a climax toward the turn end. Notably, the observed decrease in word frequency within turns is inversely correlated with the observed increase in pupil size in speakers, but not in recipients, with steeper decreases in word frequency going along with steeper increases in pupil size in speakers. We discuss the implications of these findings for theories of speech processing, turn structure, and information packaging. Crucially, we propose that the intensification of processing effort in speakers during a turn at talk is owed to an informational climax, which entails a progression from high-frequency, low-information words through intermediate levels to low-frequency, high-information words. At least in English conversation, interlocutors seem to make use of this pattern as one way to achieve efficiency in conversational interaction, creating a regularly recurring distribution of processing load across speaking turns, which aids smooth turn transitions, content prediction, and effective information transfer.

Speaking partners' talk in transitions between symbols in conversations that use speech-output technologies

Aided conversations differ from spoken ones in their transitions between turns and symbols because seeking and choosing symbols takes more time than speaking words naturally. This study adopted the concepts and principles of conversation analysis (CA) to analyze the transitions between symbols during the construction of aided turns. The data was from the spontaneous conversations at home between mothers and four males (sons) who used speech-output technologies. The analyses compared the durations of transitions between turns and symbols and frequency of mothers' talk during the transitions. The mothers most often remained silent during the symbol-transitions, but when they talked, their speech was inspired by the current context and was not coincidental. When the mothers spoke during the symbol-transitions, their sons either continued or paused constructing their turn. Despite the effect on the progress of conversations, the participants did not orient to these expanded aided turns as challenging. The expansions manifested (a) in parallel (b) side by side with the aided turn, and (c) as a part of the ongoing aided turn. Analyzing the structural characteristics of aided conversations deepens our understanding of theoretical concepts, offers strategies to guide aided communicators and their partners, and helps to develop augmentative and alternative communication technologies that promote interactive communication.

Phonetic differences between affirmative and feedback head nods in German Sign Language (DGS): A pose estimation study

This study investigates head nods in natural dyadic German Sign Language (DGS) interaction, with the aim of finding whether head nods serving different functions vary in their phonetic characteristics. Earlier research on spoken and sign language interaction has revealed that head nods vary in the form of the movement. However, most claims about the phonetic properties of head nods have been based on manual annotation without reference to naturalistic text types and the head nods produced by the addressee have been largely ignored. There is a lack of detailed information about the phonetic properties of the addressee's head nods and their interaction with manual cues in DGS as well as in other sign languages, and the existence of a form-function relationship of head nods remains uncertain. We hypothesize that head nods functioning in the context of affirmation differ from those signaling feedback in their form and the co-occurrence with manual items. To test the hypothesis, we apply OpenPose, a computer vision toolkit, to extract head nod measurements from video recordings and examine head nods in terms of their duration, amplitude and velocity. We describe the basic phonetic properties of head nods in DGS and their interaction with manual items in naturalistic corpus data. Our results show that phonetic properties of affirmative nods differ from those of feedback nods. Feedback nods appear to be on average slower in production and smaller in amplitude than affirmation nods, and they are commonly produced without a co-occurring manual element. We attribute the variations in phonetic properties to the distinct roles these cues fulfill in turn-taking system. This research underlines the importance of non-manual cues in shaping the turn-taking system of sign languages, establishing the links between such research fields as sign language linguistics, conversational analysis, quantitative linguistics and computer vision.

Investigating conversational dynamics in triads: Effects of noise, hearing impairment, and hearing aids

Communication is an important part of everyday life and requires a rapid and coordinated interplay between interlocutors to ensure a successful conversation. Here, we investigate whether increased communication difficulty caused by additional background noise, hearing impairment, and not providing adequate hearing-aid (HA) processing affected the dynamics of a group conversation between one hearing-impaired (HI) and two normal-hearing (NH) interlocutors. Free conversations were recorded from 25 triads communicating at low (50 dBC SPL) or high (75 dBC SPL) levels of canteen noise. In conversations at low noise levels, the HI interlocutor was either unaided or aided. In conversations at high noise levels, the HI interlocutor either experienced omnidirectional or directional sound processing. Results showed that HI interlocutors generally spoke more and initiated their turn faster, but with more variability, than the NH interlocutors. Increasing the noise level resulted in generally higher speech levels, but more so for the NH than for the HI interlocutors. Higher background noise also affected the HI interlocutors' ability to speak in longer turns. When the HI interlocutors were unaided at low noise levels, both HI and NH interlocutors spoke louder, while receiving directional sound processing at high levels of noise only reduced the speech level of the HI interlocutor. In conclusion, noise, hearing impairment, and hearing-aid processing mainly affected speech levels, while the remaining measures of conversational dynamics (FTO median, FTO IQR, turn duration, and speaking time) were unaffected. Hence, although experiencing large changes in communication difficulty, the conversational dynamics of the free triadic conversations remain relatively stable.

Elliptical Responses to Direct and Indirect Requests for Information

We present two studies examining the factors that lead speakers to produce elliptical responses to requests for information. Following Clark and Levelt and Kelter, experimenters called businesses and asked about their closing time (e.g., Can you tell me what time you close?). Participants provided the requested information in full sentence responses (We close at 9) or elliptical responses (At 9). A reanalysis of data from previous experiments using this paradigm shows that participants are more likely to produce an elliptical response when the question is a direct request for information (What time do you close?) than when the question is an indirect request for information (Can you tell me what time you close?). Participants were less likely to produce an elliptical response when they began their answer by providing a yes/no response (e.g., Sure . . . we close at 9). A new experiment replicated these findings, and further showed that elliptical responses were less likely when (1) irrelevant linguistic content was inserted between the question and the participant's response, and (2) participants verbalized signs of difficulty retrieving the requested information. This latter effect is most prominent in response to questions that are seen as very polite (May I ask you what time you close?). We discuss the role that the recoverability of the intended meaning of the ellipsis, the accessibility of potential antecedents for the ellipsis, pragmatic factors, and memory retrieval play in shaping the production of ellipsis.

Conversational production and comprehension: fMRI-evidence reminiscent of but deviant from the classical Broca-Wernicke model

A key question in research on the neurobiology of language is to which extent the language production and comprehension systems share neural infrastructure, but this question has not been addressed in the context of conversation. We utilized a public fMRI dataset where 24 participants engaged in unscripted conversations with a confederate outside the scanner, via an audio-video link. We provide evidence indicating that the two systems share neural infrastructure in the left-lateralized perisylvian language network, but diverge regarding the level of activation in regions within the network. Activity in the left inferior frontal gyrus was stronger in production compared to comprehension, while comprehension showed stronger recruitment of the left anterior middle temporal gyrus and superior temporal sulcus, compared to production. Although our results are reminiscent of the classical Broca-Wernicke model, the anterior (rather than posterior) temporal activation is a notable difference from that model. This is one of the findings that may be a consequence of the conversational setting, another being that conversational production activated what we interpret as higher-level socio-pragmatic processes. In conclusion, we present evidence for partial overlap and functional asymmetry of the neural infrastructure of production and comprehension, in the above-mentioned frontal vs temporal regions during conversation.

Gestures speed up responses to questions

Most language use occurs in face-to-face conversation, which involves rapid turn-taking. Seeing communicative bodily signals in addition to hearing speech may facilitate such fast responding. We tested whether this holds for co-speech hand gestures by investigating whether these gestures speed up button press responses to questions. Sixty native speakers of Dutch viewed videos in which an actress asked yes/no-questions, either with or without a corresponding iconic hand gesture. Participants answered the questions as quickly and accurately as possible via button press. Gestures did not impact response accuracy, but crucially, gestures sped up responses, suggesting that response planning may be finished earlier when gestures are seen. How much gestures sped up responses was not related to their timing in the question or their timing with respect to the corresponding information in speech. Overall, these results are in line with the idea that multimodality may facilitate fast responding during face-to-face conversation.

Information distribution patterns in naturalistic dialogue differ across languages

The natural ecology of language is conversation, with individuals taking turns speaking to communicate in a back-and-forth fashion. Language in this context involves strings of words that a listener must process while simultaneously planning their own next utterance. It would thus be highly advantageous if language users distributed information within an utterance in a way that may facilitate this processing-planning dynamic. While some studies have investigated how information is distributed at the level of single words or clauses, or in written language, little is known about how information is distributed within spoken utterances produced during naturalistic conversation. It also is not known how information distribution patterns of spoken utterances may differ across languages. We used a set of matched corpora (CallHome) containing 898 telephone conversations conducted in six different languages (Arabic, English, German, Japanese, Mandarin, and Spanish), analyzing more than 58,000 utterances, to assess whether there is evidence of distinct patterns of information distributions at the utterance level, and whether these patterns are similar or differed across the languages. We found that English, Spanish, and Mandarin typically show a back-loaded distribution, with higher information (i.e., surprisal) in the last half of utterances compared with the first half, while Arabic, German, and Japanese showed front-loaded distributions, with higher information in the first half compared with the last half. Additional analyses suggest that these patterns may be related to word order and rate of noun and verb usage. We additionally found that back-loaded languages have longer turn transition times (i.e., time between speaker turns).

Estimating speaker direction on a humanoid robot with binaural acoustic signals

To achieve human-like behaviour during speech interactions, it is necessary for a humanoid robot to estimate the location of a human talker. Here, we present a method to optimize the parameters used for the direction of arrival (DOA) estimation, while also considering real-time applications for human-robot interaction scenarios. This method is applied to binaural sound source localization framework on a humanoid robotic head. Real data is collected and annotated for this work. Optimizations are performed via a brute force method and a Bayesian model based method, results are validated and discussed, and effects on latency for real-time use are also explored.

Hand Gestures Have Predictive Potential During Conversation: An Investigation of the Timing of Gestures in Relation to Speech

During face-to-face conversation, transitions between speaker turns are incredibly fast. These fast turn exchanges seem to involve next speakers predicting upcoming semantic information, such that next turn planning can begin before a current turn is complete. Given that face-to-face conversation also involves the use of communicative bodily signals, an important question is how bodily signals such as co-speech hand gestures play into these processes of prediction and fast responding. In this corpus study, we found that hand gestures that depict or refer to semantic information started before the corresponding information in speech, which held both for the onset of the gesture as a whole, as well as the onset of the stroke (the most meaningful part of the gesture). This early timing potentially allows listeners to use the gestural information to predict the corresponding semantic information to be conveyed in speech. Moreover, we provided further evidence that questions with gestures got faster responses than questions without gestures. However, we found no evidence for the idea that how much a gesture precedes its lexical affiliate (i.e., its predictive potential) relates to how fast responses were given. The findings presented here highlight the importance of the temporal relation between speech and gesture and help to illuminate the potential mechanisms underpinning multimodal language processing during face-to-face conversation.

A frontal cortical network is critical for language planning during spoken interaction

Many brain areas exhibit activity correlated with language planning, but the impact of these dynamics on spoken interaction remains unclear. Here we use direct electrical stimulation to transiently perturb cortical function in neurosurgical patient-volunteers performing a question-answer task. Stimulating structures involved in speech motor function evoked diverse articulatory deficits, while perturbations of caudal inferior and middle frontal gyri - which exhibit preparatory activity during conversational turn-taking - led to response errors. Perturbation of the same planning-related frontal regions slowed inter-speaker timing, while faster responses could result from stimulation of sites located in other areas. Taken together, these findings further indicate that caudal inferior and middle frontal gyri constitute a critical planning network essential for interactive language use.

Conversational Eyebrow Frowns Facilitate Question Identification: An Online Study Using Virtual Avatars

Conversation is a time-pressured environment. Recognizing a social action (the ''speech act,'' such as a question requesting information) early is crucial in conversation to quickly understand the intended message and plan a timely response. Fast turns between interlocutors are especially relevant for responses to questions since a long gap may be meaningful by itself. Human language is multimodal, involving speech as well as visual signals from the body, including the face. But little is known about how conversational facial signals contribute to the communication of social actions. Some of the most prominent facial signals in conversation are eyebrow movements. Previous studies found links between eyebrow movements and questions, suggesting that these facial signals could contribute to the rapid recognition of questions. Therefore, we aimed to investigate whether early eyebrow movements (eyebrow frown or raise vs. no eyebrow movement) facilitate question identification. Participants were instructed to view videos of avatars where the presence of eyebrow movements accompanying questions was manipulated. Their task was to indicate whether the utterance was a question or a statement as accurately and quickly as possible. Data were collected using the online testing platform Gorilla. Results showed higher accuracies and faster response times for questions with eyebrow frowns, suggesting a facilitative role of eyebrow frowns for question identification. This means that facial signals can critically contribute to the communication of social actions in conversation by signaling social action-specific visual information and providing visual cues to speakers' intentions.

Interactive bioacoustic playback as a tool for detecting and exploring nonhuman intelligence: "conversing" with an Alaskan humpback whale

Here we report on a rare and opportunistic acoustic turn-taking with an adult female humpback whale, known as Twain, in Southeast Alaska. Post hoc acoustic and statistical analyses of a 20-min acoustic exchange between the broadcast of a recorded contact call, known as a 'whup/throp', with call responses by Twain revealed an intentional human-whale acoustic (and behavioral) interaction. Our results show that Twain participated both physically and acoustically in three phases of interaction (Phase 1: Engagement, Phase 2: Agitation, Phase 3: Disengagement), independently determined by blind observers reporting on surface behavior and respiratory activity of the interacting whale. A close examination of both changes to the latency between Twain's calls and the temporal matching to the latency of the exemplar across phases indicated that Twain was actively engaged in the exchange during Phase 1 (Engagement), less so during Phase 2 (Agitation), and disengaged during Phase 3 (Disengagement). These results, while preliminary, point to several key considerations for effective playback design, namely the importance of salient, dynamic and adaptive playbacks, that should be utilized in experimentation with whales and other interactive nonhuman species.

Neurophysiological explorations across the spectrum of psychosis, autism, and depression, during wakefulness and sleep: protocol of a prospective case-control transdiagnostic multimodal study (DEMETER)

BACKGROUND

Quantitative electroencephalography (EEG) analysis offers the opportunity to study high-level cognitive processes across psychiatric disorders. In particular, EEG microstates translate the temporal dynamics of neuronal networks throughout the brain. Their alteration may reflect transdiagnostic anomalies in neurophysiological functions that are impaired in mood, psychosis, and autism spectrum disorders, such as sensorimotor integration, speech, sleep, and sense of self. The main questions this study aims to answer are as follows: 1) Are EEG microstate anomalies associated with clinical and functional prognosis, both in resting conditions and during sleep, across psychiatric disorders? 2) Are EEG microstate anomalies associated with differences in sensorimotor integration, speech, sense of self, and sleep? 3) Can the dynamic of EEG microstates be modulated by a non-drug intervention such as light hypnosis?

METHODS

This prospective cohort will include a population of adolescents and young adults, aged 15 to 30 years old, with ultra-high-risk of psychosis (UHR), first-episode psychosis (FEP), schizophrenia (SCZ), autism spectrum disorder (ASD), and major depressive disorder (MDD), as well as healthy controls (CTRL) (N = 21 × 6), who will be assessed at baseline and after one year of follow-up. Participants will undergo deep phenotyping based on psychopathology, neuropsychological assessments, 64-channel EEG recordings, and biological sampling at the two timepoints. At baseline, the EEG recording will also be coupled to a sensorimotor task and a recording of the characteristics of their speech (prosody and turn-taking), a one-night polysomnography, a self-reference effect task in virtual reality (only in UHR, FEP, and CTRL). An interventional ancillary study will involve only healthy controls, in order to assess whether light hypnosis can modify the EEG microstate architecture in a direction opposite to what is seen in disease.

DISCUSSION

This transdiagnostic longitudinal case-control study will provide a multimodal neurophysiological assessment of clinical dimensions (sensorimotor integration, speech, sleep, and sense of self) that are disrupted across mood, psychosis, and autism spectrum disorders. It will further test the relevance of EEG microstates as dimensional functional biomarkers.

TRIAL REGISTRATION

ClinicalTrials.gov Identifier NCT06045897.

Hearing loss, depression, and cognition in younger and older adult CI candidates

Background and Aim

Hearing loss in old age is associated with cognitive decline and with depression. Our study aimed to investigate the relationship between hearing loss, cognitive decline, and secondary depressive symptoms in a sample of younger and older cochlear implant candidates with profound to severe hearing loss.

Methods

This study is part of a larger cohort study designated to provide information on baseline data before CI. Sixty-one cochlear implant candidates with hearing loss from adulthood onwards (>18 years) were enrolled in this study. All had symmetrical sensorineural hearing loss in both ears (four-frequency hearing threshold difference of no more than 20 dB, PTA). Individuals with primary affective disorders, psychosis, below-average intelligence, poor German language skills, visual impairment, and a medical diagnosis with potential impact on cognition (e.g., neurodegenerative diseases,) were excluded. Four-frequency hearing thresholds (dB, PTA, better ear) were collected. Using the Abbreviated Profile of Hearing Aid Benefit, we assessed subjective hearing in noise. Clinical and subclinical depressive symptoms were assessed with the Beck Depression Inventory (BDI II). Cognitive status was assessed with a neurocognitive test battery.

Results

Our findings revealed a significant negative association between subjective hearing in noise (APHAB subscale "Background Noise") and BDII. However, we did not observe any link between hearing thresholds, depression, and cognition. Additionally, no differences emerged between younger (25-54 years) and older subjects (55-75 years). Unexpectedly, further unplanned analyses unveiled correlations between subjective hearing in quiet environments (APHAB) and cognitive performance [phonemic fluency (Regensburg Word Fluency), cognitive flexibility (TMTB), and nonverbal episodic memory (Nonverbal Learning Test), as well as subjective hearing of aversive/loud sounds (APHAB)], cognitive performance [semantic word fluency (RWT), and inhibition (Go/Nogo) and depression]. Duration of hearing loss and speech recognition at quiet (Freiburg Monosyllables) were not related to depression and cognitive performance.

Conclusion

Impact of hearing loss on mood and cognition appears to be independent, suggesting a relationship with distinct aspects of hearing loss. These results underscore the importance of considering not only conventional audiometric measures like hearing thresholds but also variables related to hearing abilities during verbal communication in everyday life, both in quiet and noisy settings.

Listening fast and slow

The pattern of response times in conversation can reveal a lot about how people listen to each other. Fast response times not only telegraph eagerness but provide evidence of attending in such a way as to almost finish the other's sentences. In other situations, slow response times are more appropriate, such as when listening prompts deeper reflection, or to leave space for the enjoyment of an inside joke. Here we argue that close relationships are not marked exclusively by one or the other pattern, but by the ability to toggle effortlessly between the two as the conversation demands.

Scale in Language

A central concern of the cognitive science of language since its origins has been the concept of the linguistic system. Recent approaches to the system concept in language point to the exceedingly complex relations that hold between many kinds of interdependent systems, but it can be difficult to know how to proceed when "everything is connected." This paper offers a framework for tackling that challenge by identifying *scale* as a conceptual mooring for the interdisciplinary study of language systems. The paper begins by defining the scale concept-simply, the possibility for a measure to be larger or smaller in different instances of a system, such as a phonemic inventory, a word's frequency value in a corpus, or a speaker population. We review sites of scale difference in and across linguistic subsystems, drawing on findings from linguistic typology, grammatical description, morphosyntactic theory, psycholinguistics, computational corpus work, and social network demography. We consider possible explanations for scaling differences and constraints in language. We then turn to the question of *dependencies between* sites of scale difference in language, reviewing four sample domains of scale dependency: in phonological systems, across levels of grammatical structure (Menzerath's Law), in corpora (Zipf's Law and related issues), and in speaker population size. Finally, we consider the implications of the review, including the utility of a scale framework for generating new questions and inspiring methodological innovations and interdisciplinary collaborations in cognitive-scientific research on language.

The evolution of hierarchically structured communication

Human language sentences are standardly understood as exhibiting considerable hierarchical structure: they can and typically do contain parts that in turn contain parts, etc. In other words, sentences are thought to generally exhibit significant nested part-whole structure. As far as we can tell, this is not a feature of the gestural or vocal communication systems of our great ape relatives. So, one of the many challenges we face in providing a theory of human language evolution is to explain the evolution of hierarchically structured communication in our line. This article takes up that challenge. More specifically, I first present and motivate an account of hierarchical structure in language that departs significantly from the orthodox conception of such structure in linguistics and evolutionary discussions that draw on linguistic theory. On the account I propose, linguistic structure, including hierarchical structure, is treated as a special case of structured action. This account is rooted in the cognitive neuroscience of action, as opposed to (formal) linguistic theory. Among other things, such an account enables us to see how selection for enhanced capacities of act organization and act control in actors, and for act interpretation in observers, might have constructed the brain machinery necessary for the elaborate forms of hierarchically structured communication that we humans engage in. I flesh out this line of thought, emphasizing in particular the role of hominin technique and technology, and the social learning thereof, as evolutionary drivers of this brain machinery.

The evolution of social timing

Sociality and timing are tightly interrelated in human interaction as seen in turn-taking or synchronised dance movements. Sociality and timing also show in communicative acts of other species that might be pleasurable, but also necessary for survival. Sociality and timing often co-occur, but their shared phylogenetic trajectory is unknown: How, when, and why did they become so tightly linked? Answering these questions is complicated by several constraints; these include the use of divergent operational definitions across fields and species, the focus on diverse mechanistic explanations (e.g., physiological, neural, or cognitive), and the frequent adoption of anthropocentric theories and methodologies in comparative research. These limitations hinder the development of an integrative framework on the evolutionary trajectory of social timing and make comparative studies not as fruitful as they could be. Here, we outline a theoretical and empirical framework to test contrasting hypotheses on the evolution of social timing with species-appropriate paradigms and consistent definitions. To facilitate future research, we introduce an initial set of representative species and empirical hypotheses. The proposed framework aims at building and contrasting evolutionary trees of social timing toward and beyond the crucial branch represented by our own lineage. Given the integration of cross-species and quantitative approaches, this research line might lead to an integrated empirical-theoretical paradigm and, as a long-term goal, explain why humans are such socially coordinated animals.

Predictive coordination of breathing during intra-personal speaking and listening

It has long been known that human breathing is altered during listening and speaking compared to rest: during speaking, inhalation depth is adjusted to the air volume required for the upcoming utterance. During listening, inhalation is temporally aligned to inhalation of the speaker. While evidence for the former is relatively strong, it is virtually absent for the latter. We address both phenomena using recordings of speech envelope and respiration in 30 participants during 14 min of speaking and listening to one's own speech. First, we show that inhalation depth is positively correlated with the total power of the speech envelope in the following utterance. Second, we provide evidence that inhalation during listening to one's own speech is significantly more likely at time points of inhalation during speaking. These findings are compatible with models that postulate alignment of internal forward models of interlocutors with the aim to facilitate communication.

Wild nightingales flexibly match whistle pitch in real time

Interactive vocal communication, similar to a human conversation, requires flexible and real-time changes to vocal output in relation to preceding auditory stimuli. These vocal adjustments are essential to ensuring both the suitable timing and content of the interaction. Precise timing of dyadic vocal exchanges has been investigated in a variety of species, including humans. In contrast, the ability of non-human animals to accurately adjust specific spectral features of vocalization extemporaneously in response to incoming auditory information is less well studied. One spectral feature of acoustic signals is the fundamental frequency, which we perceive as pitch. Many animal species can discriminate between sound frequencies, but real-time detection and reproduction of an arbitrary pitch have only been observed in humans. Here, we show that nightingales in the wild can match the pitch of whistle songs while singing in response to conspecifics or pitch-controlled whistle playbacks. Nightingales matched whistles across their entire pitch production range indicating that they can flexibly tune their vocal output along a wide continuum. Prompt whistle pitch matches were more precise than delayed ones, suggesting the direct mapping of auditory information onto a motor command to achieve online vocal replication of a heard pitch. Although nightingales' songs follow annual cycles of crystallization and deterioration depending on breeding status, the observed pitch-matching behavior is present year-round, suggesting a stable neural circuit independent of seasonal changes in physiology. Our findings represent the first case of non-human instantaneous vocal imitation of pitch, highlighting a promising model for understanding sensorimotor transformation within an interactive context. VIDEO ABSTRACT.

Ultrasound measurements of interactive turn-taking in question-answer sequences: Articulatory preparation is delayed but not tied to the response

We know that speech planning in conversational turn-taking can happen in overlap with the previous turn and research suggests that it starts as early as possible, that is, as soon as the gist of the previous turn becomes clear. The present study aimed to investigate whether planning proceeds all the way up to the last stage of articulatory preparation (i.e., putting the articulators in place for the first phoneme of the response) and what the timing of this process is. Participants answered pre-recorded quiz questions (being under the illusion that they were asked live), while their tongue movements were measured using ultrasound. Planning could start early for some quiz questions (i.e., midway during the question), but late for others (i.e., only at the end of the question). The results showed no evidence for a difference between tongue movements in these two types of questions for at least two seconds after planning could start in early-planning questions, suggesting that speech planning in overlap with the current turn proceeds more slowly than in the clear. On the other hand, when time-locking to speech onset, tongue movements differed between the two conditions from up to two seconds before this point. This suggests that articulatory preparation can occur in advance and is not fully tied to the overt response itself.

Utterance planning under message uncertainty: evidence from a novel picture-naming paradigm

Language researchers view utterance planning as implicit decision-making: producers must choose the words, sentence structures, and various other linguistic features to communicate their message. To date, much of the research on utterance planning has focused on situations in which the speaker knows the full message to convey. Less is known about circumstances in which speakers begin utterance planning before they are certain about their message. In three picture-naming experiments, we used a novel paradigm to examine how speakers plan utterances before a full message is known. In Experiments 1 and 2, participants viewed displays showing two pairs of objects, followed by a cue to name one pair. In an Overlap condition, one object appeared in both pairs, providing early information about one of the objects to name. In a Different condition, there was no object overlap. Across both spoken and typed responses, participants tended to name the overlapping target first in the Overlap condition, with shorter initiation latencies compared with other utterances. Experiment 3 used a semantically constraining question to provide early information about the upcoming targets, and participants tended to name the more likely target first in their response. These results suggest that in situations of uncertainty, producers choose word orders that allow them to begin early planning. Producers prioritize message components that are certain to be needed and continue planning the rest when more information becomes available. Given similarities to planning strategies for other goal-directed behaviors, we suggest continuity between decision-making processes in language and other cognitive domains.

The Silent Treatment?: Changes in patient emotional expression after silence

Psychotherapy can be an emotionally laden conversation, where both verbal and non-verbal interventions may impact the therapeutic process. Prior research has postulated mixed results in how clients emotionally react following a silence after the therapist is finished talking, potentially due to studying a limited range of silences with primarily qualitative and self-report methodologies. A quantitative exploration may illuminate new findings. Utilizing research and automatic data processing from the field of linguistics, we analysed the full range of silence lengths (0.2 to 24.01 seconds), and measures of emotional expression - vocally encoded arousal and emotional valence from the works spoken - of 84 audio recordings Motivational Interviewing sessions. We hypothesized that both the level and the variance of client emotional expression would change as a function of silence length, however, due to the mixed results in the literature the direction of emotional change was unclear. We conducted a multilevel linear regression to examine how the level of client emotional expression changed across silence length, and an ANOVA to examine the variability of client emotional expression across silence lengths. Results indicated in both analyses that as silence length increased, emotional expression largely remained the same. Broadly, we demonstrated a weak connection between silence length and emotional expression, indicating no persuasive evidence that silence leads to client emotional processing and expression.

Head motion synchrony in unidirectional and bidirectional verbal communication

Interpersonal communication includes verbal and nonverbal communication. Verbal communication comprises one-way (e.g., a speech or lecture) and interactive verbal communication (e.g., daily conversations or meetings), which we frequently encounter. Nonverbal communication has considerable influence on interpersonal communication, and body motion synchrony is known to be an important factor for successful communication and social interaction. However, most research on body motion synchrony has been elucidated by either the setting of one-way verbal transmission or the verbal interaction setting, and it remains unclear whether verbal directionality and interactivity affect body motion synchrony. One-way and two-way (interactive) verbal communication is implicated in designed or undesigned leader-follower relationships, and also in the complexity and diversity of interpersonal interactions, where two-way verbal communication is more complex and diverse than in the one-way condition. In this study, we tested head motion synchrony between the one-way verbal communication condition (in which the roles of the speaker and listener are fixed) and the two-way verbal communication condition (where the speaker and listener can freely engage in a conversation). Therefore, although no statistically significant difference in synchrony activity (relative frequency) was found, a statistically significant difference was observed in synchrony direction (temporal lead-lag structure as mimicry) and intensity. Specifically, the synchrony direction in two-way verbal communication was close to zero, but this in one-way verbal communication was synchronized with the listener's movement predominantly delayed. Furthermore, synchrony intensity, in terms of the degree of variation in the phase difference distribution, was significantly higher in the one-way verbal communication than in the two-way condition, with bigger time-shifts being observed in the latter. This result suggests that verbal interaction does not affect the overall frequency of head motion synchrony but does affect the temporal lead-lag structure and coherence.

Gesture, spatial cognition and the evolution of language

Human communication displays a striking contrast between the diversity of languages and the universality of the principles underlying their use in conversation. Despite the importance of this interactional base, it is not obvious that it heavily imprints the structure of languages. However, a deep-time perspective suggests that early hominin communication was gestural, in line with all the other Hominidae. This gestural phase of early language development seems to have left its traces in the way in which spatial concepts, implemented in the hippocampus, provide organizing principles at the heart of grammar. This article is part of a discussion meeting issue 'Face2face: advancing the science of social interaction'.

Beyond speaking: neurocognitive perspectives on language production in social interaction

The human faculty to speak has evolved, so has been argued, for communicating with others and for engaging in social interactions. Hence the human cognitive system should be equipped to address the demands that social interaction places on the language production system. These demands include the need to coordinate speaking with listening, the need to integrate own (verbal) actions with the interlocutor's actions, and the need to adapt language flexibly to the interlocutor and the social context. In order to meet these demands, core processes of language production are supported by cognitive processes that enable interpersonal coordination and social cognition. To fully understand the cognitive architecture and its neural implementation enabling humans to speak in social interaction, our understanding of how humans produce language needs to be connected to our understanding of how humans gain insights into other people's mental states and coordinate in social interaction. This article reviews theories and neurocognitive experiments that make this connection and can contribute to advancing our understanding of speaking in social interaction. This article is part of a discussion meeting issue 'Face2face: advancing the science of social interaction'.

Long gaps between turns are awkward for strangers but not for friends

When people feel connected they tend to respond quickly in conversation, creating short gaps between turns. But are long gaps always a sign that things have gone awry? We analysed the frequency and impact of long gaps (greater than 2 s) in conversations between strangers and between friends. As predicted, long gaps signalled disconnection between strangers. However, long gaps between friends marked moments of increased connection and friends tended to have more of them. These differences in connection were also perceived by independent raters: only the long gaps between strangers were rated as awkward, and increasingly so the longer they lasted. Finally, we show that, compared to strangers, long gaps between friends include more genuine laughter and are less likely to precede a topic change. This suggests that the gaps of friends may not function as 'gaps' at all, but instead allow space for enjoyment and mutual reflection. Together, these findings suggest that the turn-taking dynamics of friends are meaningfully different from those of strangers and may be less bound by social conventions. More broadly, this work illustrates that samples of convenience-pairs of strangers being the modal paradigm for interaction research-may not capture the social dynamics of more familiar relationships. This article is part of a discussion meeting issue 'Face2face: advancing the science of social interaction'.

The CANDOR corpus: Insights from a large multimodal dataset of naturalistic conversation

People spend a substantial portion of their lives engaged in conversation, and yet, our scientific understanding of conversation is still in its infancy. Here, we introduce a large, novel, and multimodal corpus of 1656 conversations recorded in spoken English. This 7+ million word, 850-hour corpus totals more than 1 terabyte of audio, video, and transcripts, with moment-to-moment measures of vocal, facial, and semantic expression, together with an extensive survey of speakers' postconversation reflections. By taking advantage of the considerable scope of the corpus, we explore many examples of how this large-scale public dataset may catalyze future research, particularly across disciplinary boundaries, as scholars from a variety of fields appear increasingly interested in the study of conversation.

A multimodal approach for modeling engagement in conversation

Recently, engagement has emerged as a key variable explaining the success of conversation. In the perspective of human-machine interaction, an automatic assessment of engagement becomes crucial to better understand the dynamics of an interaction and to design socially-aware robots. This paper presents a predictive model of the level of engagement in conversations. It shows in particular the interest of using a rich multimodal set of features, outperforming the existing models in this domain. In terms of methodology, study is based on two audio-visual corpora of naturalistic face-to-face interactions. These resources have been enriched with various annotations of verbal and nonverbal behaviors, such as smiles, head nods, and feedbacks. In addition, we manually annotated gestures intensity. Based on a review of previous works in psychology and human-machine interaction, we propose a new definition of the notion of engagement, adequate for the description of this phenomenon both in natural and mediated environments. This definition have been implemented in our annotation scheme. In our work, engagement is studied at the turn level, known to be crucial for the organization of the conversation. Even though there is still a lack of consensus around their precise definition, we have developed a turn detection tool. A multimodal characterization of engagement is performed using a multi-level classification of turns. We claim a set of multimodal cues, involving prosodic, mimo-gestural and morpho-syntactic information, is relevant to characterize the level of engagement of speakers in conversation. Our results significantly outperform the baseline and reach state-of-the-art level (0.76 weighted F-score). The most contributing modalities are identified by testing the performance of a two-layer perceptron when trained on unimodal feature sets and on combinations of two to four modalities. These results support our claim about multimodality: combining features related to the speech fundamental frequency and energy with mimo-gestural features leads to the best performance.

Turn-Taking Mechanisms in Imitative Interaction: Robotic Social Interaction Based on the Free Energy Principle

This study explains how the leader-follower relationship and turn-taking could develop in a dyadic imitative interaction by conducting robotic simulation experiments based on the free energy principle. Our prior study showed that introducing a parameter during the model training phase can determine leader and follower roles for subsequent imitative interactions. The parameter is defined as w, the so-called meta-prior, and is a weighting factor used to regulate the complexity term versus the accuracy term when minimizing the free energy. This can be read as sensory attenuation, in which the robot's prior beliefs about action are less sensitive to sensory evidence. The current extended study examines the possibility that the leader-follower relationship shifts depending on changes in w during the interaction phase. We identified a phase space structure with three distinct types of behavioral coordination using comprehensive simulation experiments with sweeps of w of both robots during the interaction. Ignoring behavior in which the robots follow their own intention was observed in the region in which both ws were set to large values. One robot leading, followed by the other robot was observed when one w was set larger and the other was set smaller. Spontaneous, random turn-taking between the leader and the follower was observed when both ws were set at smaller or intermediate values. Finally, we examined a case of slowly oscillating w in anti-phase between the two agents during the interaction. The simulation experiment resulted in turn-taking in which the leader-follower relationship switched during determined sequences, accompanied by periodic shifts of ws. An analysis using transfer entropy found that the direction of information flow between the two agents also shifted along with turn-taking. Herein, we discuss qualitative differences between random/spontaneous turn-taking and agreed-upon sequential turn-taking by reviewing both synthetic and empirical studies.

Zipf's law revisited: Spoken dialog, linguistic units, parameters, and the principle of least effort

The ubiquitous inverse relationship between word frequency and word rank is commonly known as Zipf's law. The theoretical underpinning of this law states that the inverse relationship yields decreased effort in both the speaker and hearer, the so-called principle of least effort. Most research has focused on showing an inverse relationship only for written monolog, only for frequencies and ranks of one linguistic unit, generally word unigrams, with strong correlations of the power law to the observed frequency distributions, with limited to no attention to psychological mechanisms such as the principle of least effort. The current paper extends the existing findings, by not focusing on written monolog but on a more fundamental form of communication, spoken dialog, by not only investigating word unigrams but also units quantified on syntactic, pragmatic, utterance, and nonverbal communicative levels by showing that the adequacy of Zipf's formula seems ubiquitous, but the exponent of the power law curve is not, and by placing these findings in the context of Zipf's principle of least effort through redefining effort in terms of cognitive resources available for communication. Our findings show that Zipf's law also applies to a more natural form of communication-that of spoken dialog, that it applies to a range of linguistic units beyond word unigrams, that the general good fit of Zipf's law needs to be revisited in light of the parameters of the formula, and that the principle of least effort is a useful theoretical framework for the findings of Zipf's law.

Analysis of communication styles underpinning clinical decision-making in cancer multidisciplinary team meetings

Introduction

In cancer care, multidisciplinary team (MDT) meetings are the gold standard. While they are trying to maximize productivity on the back of the steadily increasing workload, growing cancer incidence, financial constraints, and staff shortages, concerns have been raised with regards to the quality of team output, as reported by Cancer Research UK in 2017: "Sometimes we discuss up to 70 patients. This is after a whole day of clinics, and we do not finish until after 19.00. Would you want to be number 70?". This study aimed to explore systematically some of the dynamics of group interaction and teamwork in MDT meetings.

Materials and methods

This was a prospective observational study conducted across three MDTs/university hospitals in the United Kingdom. We video-recorded 30 weekly meetings where 822 patient cases were reviewed. A cross-section of the recordings was transcribed using the Jefferson notation system and analyzed using frequency counts (quantitative) and some principles of conversation analysis (qualitative).

Results

We found that, across teams, surgeons were the most frequent initiators and responders of interactional sequences, speaking on average 47% of the time during case discussions. Cancer nurse specialists and coordinators were the least frequent initiators, with the former speaking 4% of the time and the latter speaking 1% of the time. We also found that the meetings had high levels of interactivity, with an initiator-responder ratio of 1:1.63, meaning that for every sequence of interactions initiated, the initiator received more than a single response. Lastly, we found that verbal dysfluencies (laughter, interruptions, and incomplete sentences) were more common in the second half of meetings, where a 45% increase in their frequency was observed.

Discussion

Our findings highlight the importance of teamwork in planning MDT meetings, particularly with regard to Cancer Research UK in 2017 cognitive load/fatigue and decision-making, the hierarchy of clinical expertise, and the increased integration of patients' psychosocial information into MDT discussion and their perspectives. Utilizing a micro-level methodology, we highlight identifiable patterns of interaction among participants in MDT meetings and how these can be used to inform the optimization of teamwork.

The evolution of language by sexual selection

Most accounts of the evolution of language assume that language and greater intelligence are beneficial adaptations, leading to increases in survival fitness. These accounts emphasise natural selection, with language as an adaptation to the habitat, placing less emphasis on sexual selection and reproductive fitness. An account of language evolution by natural selection alone faces problems in accounting for the prodigious power and expressivity of human language. Modern language (and its recent antecedents) would appear to offer only small incremental benefits over simpler language, which would require a smaller brain with smaller metabolic costs. Accounts by natural selection also face problems in accounting for the uniqueness of human language and intelligence. I therefore consider a hybrid account, in which both natural selection and sexual selection played a role in the evolution of language and intelligence, probably at different times. Specifically, in this account, early language was driven by natural selection to collaborate. Then later humans became subject to sexual selection for superior intelligence, with language acting as the main display mechanism for intelligence. It is hard to determine the relative roles of natural and sexual selection over the time course of the evolution of language. In the later stages, sexual selection to display intelligence drove a runaway selection process towards powerful modern language. This hybrid account retains the benefits of accounts by natural selection, while also accounting for the prodigious power of human language and intelligence, and for its uniqueness compared to other primates. Sexual selection often leads to traits which are unique to a species, and are exaggerated beyond natural needs. On this account, the capability for language may have evolved in the order: (1) pragmatics and a theory of mind; (2) using single words and constructions; (3) learning and using syntax. In this model, relevance-based pragmatics evolved before language; then, single words and constructions came into use; and later, syntax condensed out of pragmatics, as a codification of some pragmatic rules of inference. This order requires only incremental extensions of primate cognition, and agrees with the order in which children learn language.

Timing of head turns to upcoming talkers in triadic conversation: Evidence for prediction of turn ends and interruptions

In conversation, people are able to listen to an utterance and respond within only a few hundred milliseconds. It takes substantially longer to prepare even a simple utterance, suggesting that interlocutors may make use of predictions about when the talker is about to end. But it is not only the upcoming talker that needs to anticipate the prior talker ending-listeners that are simply following the conversation could also benefit from predicting the turn end in order to shift attention appropriately with the turn switch. In this paper, we examined whether people predict upcoming turn ends when watching conversational turns switch between others by analysing natural conversations. These conversations were between triads of older adults in different levels and types of noise. The analysis focused on the observer during turn switches between the other two parties using head orientation (i.e. saccades from one talker to the next) to identify when their focus moved from one talker to the next. For non-overlapping utterances, observers started to turn to the upcoming talker before the prior talker had finished speaking in 17% of turn switches (going up to 26% when accounting for motor-planning time). For overlapping utterances, observers started to turn towards the interrupter before they interrupted in 18% of turn switches (going up to 33% when accounting for motor-planning time). The timing of head turns was more precise at lower than higher noise levels, and was not affected by noise type. These findings demonstrate that listeners in natural group conversation situations often exhibit head movements that anticipate the end of one conversational turn and the beginning of another. Furthermore, this work demonstrates the value of analysing head movement as a cue to social attention, which could be relevant for advancing communication technology such as hearing devices.

Turn-taking skills in mammals: A systematic review into development and acquisition

How human language evolved remains one of the most intriguing questions in science, and different approaches have been used to tackle this question. A recent hypothesis, the Interaction Engine Hypothesis, postulates that language was made possible through the special capacity for social interaction involving different social cognitive skills (e.g., joint attention, common ground) and specific characteristics such as face-to-face interaction, mutual gaze and turn-taking, the exchange of rapid communicative turns. Recently, it has been argued that this turn-taking infrastructure may be a foundational and ancient mechanism of the layered system of language because communicative turn-taking has been found in human infants and across several non-human primate species. Moreover, there is some evidence for turn-taking in different mammalian taxa, especially those capable of vocal learning. Surprisingly, however, the existing studies have mainly focused on turn-taking production of adult individuals, while little is known about its emergence and development in young individuals. Hence, the aim of the current paper was 2-fold: First, we carried out a systematic review of turn-taking development and acquisition in mammals to evaluate possible research bias and existing gaps. Second, we highlight research avenues to spur more research into this domain and investigate if distinct turn-taking elements can be found in other non-human animal species. Since mammals exhibit an extended development period, including learning and strong parental care, they represent an excellent model group in which to investigate the acquisition and development of turn-taking abilities. We performed a systematic review including a wide range of terms and found 21 studies presenting findings on turn-taking abilities in infants and juveniles. Most of these studies were from the last decade, showing an increased interest in this field over the years. Overall, we found a considerable variation in the terminologies and methodological approaches used. In addition, studies investigating turn-taking abilities across different development periods and in relation to different social partners were very rare, thereby hampering direct, systematic comparisons within and across species. Nonetheless, the results of some studies suggested that specific turn-taking elements are innate, while others are acquired during development (e.g., flexibility). Finally, we pinpoint fruitful research avenues and hypotheses to move the field of turn-taking development forward and improve our understanding of the impact of turn-taking on language evolution.

Scenario-based dialogue system based on pause detection toward daily health monitoring

Introduction

We have conducted research on building a robot dialogue system to support the independent living of older adults. In order to provide appropriate support for them, it is necessary to obtain as much information, particularly related to their health condition, as possible. As the first step, we have examined a method to allow dialogue to continue for longer periods.

Methods

A scenario-based dialogue system utilizing pause detection for turn-taking was built. The practicality of adjusting the system based on the dialogue rhythm of each individual was studied. The system was evaluated through user studies with a total of 20 users, 10 of whom were older adults.

Results

The system detected pauses in the user’s speech using the sound level of their voice, and predicted the duration and number of pauses based on past dialogue data. Thus, the system initiated the robot’s voice-call after the user’s predicted speech.

Conclusions

Multiple turns of dialogue between robot and older adults are found possible under the system, despite several overlaps of robot’s and users’ speech observed. The users responded to the robot, including the questions related to health conditions. The feasibility of a scenario-based dialogue system was suggested; however, improvements are required.

Dual-MEG interbrain synchronization during turn-taking verbal interactions between mothers and children

Verbal interaction and imitation are essential for language learning and development in young children. However, it is unclear how mother-child dyads synchronize oscillatory neural activity at the cortical level in turn-based speech interactions. Our study investigated interbrain synchrony in mother-child pairs during a turn-taking paradigm of verbal imitation. A dual-MEG (magnetoencephalography) setup was used to measure brain activity from interactive mother-child pairs simultaneously. Interpersonal neural synchronization was compared between socially interactive and noninteractive tasks (passive listening to pure tones). Interbrain networks showed increased synchronization during the socially interactive compared to noninteractive conditions in the theta and alpha bands. Enhanced interpersonal brain synchrony was observed in the right angular gyrus, right triangular, and left opercular parts of the inferior frontal gyrus. Moreover, these parietal and frontal regions appear to be the cortical hubs exhibiting a high number of interbrain connections. These cortical areas could serve as a neural marker for the interactive component in verbal social communication. The present study is the first to investigate mother-child interbrain neural synchronization during verbal social interactions using a dual-MEG setup. Our results advance our understanding of turn-taking during verbal interaction between mother-child dyads and suggest a role for social "gating" in language learning.

Visual bodily signals as core devices for coordinating minds in interaction

The view put forward here is that visual bodily signals play a core role in human communication and the coordination of minds. Critically, this role goes far beyond referential and propositional meaning. The human communication system that we consider to be the explanandum in the evolution of language thus is not spoken language. It is, instead, a deeply multimodal, multilayered, multifunctional system that developed-and survived-owing to the extraordinary flexibility and adaptability that it endows us with. Beyond their undisputed iconic power, visual bodily signals (manual and head gestures, facial expressions, gaze, torso movements) fundamentally contribute to key pragmatic processes in modern human communication. This contribution becomes particularly evident with a focus that includes non-iconic manual signals, non-manual signals and signal combinations. Such a focus also needs to consider meaning encoded not just via iconic mappings, since kinematic modulations and interaction-bound meaning are additional properties equipping the body with striking pragmatic capacities. Some of these capacities, or its precursors, may have already been present in the last common ancestor we share with the great apes and may qualify as early versions of the components constituting the hypothesized interaction engine. This article is part of the theme issue 'Revisiting the human 'interaction engine': comparative approaches to social action coordination'.