Multimodality and the origin of a novel communication system in face-to-face interaction

Phonetic differences between affirmative and feedback head nods in German Sign Language (DGS): A pose estimation study

This study investigates head nods in natural dyadic German Sign Language (DGS) interaction, with the aim of finding whether head nods serving different functions vary in their phonetic characteristics. Earlier research on spoken and sign language interaction has revealed that head nods vary in the form of the movement. However, most claims about the phonetic properties of head nods have been based on manual annotation without reference to naturalistic text types and the head nods produced by the addressee have been largely ignored. There is a lack of detailed information about the phonetic properties of the addressee's head nods and their interaction with manual cues in DGS as well as in other sign languages, and the existence of a form-function relationship of head nods remains uncertain. We hypothesize that head nods functioning in the context of affirmation differ from those signaling feedback in their form and the co-occurrence with manual items. To test the hypothesis, we apply OpenPose, a computer vision toolkit, to extract head nod measurements from video recordings and examine head nods in terms of their duration, amplitude and velocity. We describe the basic phonetic properties of head nods in DGS and their interaction with manual items in naturalistic corpus data. Our results show that phonetic properties of affirmative nods differ from those of feedback nods. Feedback nods appear to be on average slower in production and smaller in amplitude than affirmation nods, and they are commonly produced without a co-occurring manual element. We attribute the variations in phonetic properties to the distinct roles these cues fulfill in turn-taking system. This research underlines the importance of non-manual cues in shaping the turn-taking system of sign languages, establishing the links between such research fields as sign language linguistics, conversational analysis, quantitative linguistics and computer vision.

A general theory of consciousness III the human catastrophe

It is generally assumed that verbal communication can articulate concepts like 'fact' and 'truth' accurately. However, language is fundamentally inaccurate and ambiguous and it is not possible to express exact propositions accurately in an ambiguous medium. Whether truth exists or not, language cannot express it in any exact way. A major problem for verbal communication is that words are fundamentally differently interpreted by the sender and the receiver. In addition, intrapersonal verbal communication - the voice in our head - is a useless extension to the thought process and results in misunderstanding our own thoughts. The evolvement of language has had a profound impact on human life. Most consequential has been that it allowed people to question the old human rules of behavior - the pre-language way of living. As language could not accurately express the old rules, they lost their authority and disappeared. A long period without any rules of how to live together must have followed, probably accompanied by complete chaos. Later, new rules were devised in language, but the new rules were also questioned and had to be enforced by punishment. Language changed the peaceful human way of living under the old rules into violent and aggressive forms of living under punitive control. Religion then tried to incorporate the old rules into the harsh verbal world. The rules were expressed in language through parables: imaginary beings - the gods - who possessed the power of the old rules, but who could be related to through their human appearance and behavior.

Early or synchronized gestures facilitate speech recall-a study based on motion capture data

Introduction

Temporal co-ordination between speech and gestures has been thoroughly studied in natural production. In most cases gesture strokes precede or coincide with the stressed syllable in words that they are semantically associated with.

Methods

To understand whether processing of speech and gestures is attuned to such temporal coordination, we investigated the effect of delaying, preposing or eliminating individual gestures on the memory for words in an experimental study in which 83 participants watched video sequences of naturalistic 3D-animated speakers generated based on motion capture data. A target word in the sequence appeared (a) with a gesture presented in its original position synchronized with speech, (b) temporally shifted 500 ms before or (c) after the original position, or (d) with the gesture eliminated. Participants were asked to retell the videos in a free recall task. The strength of recall was operationalized as the inclusion of the target word in the free recall.

Results

Both eliminated and delayed gesture strokes resulted in reduced recall rates compared to synchronized strokes, whereas there was no difference between advanced (preposed) and synchronized strokes. An item-level analysis also showed that the greater the interval between the onsets of delayed strokes and stressed syllables in target words, the greater the negative effect was on recall.

Discussion

These results indicate that speech-gesture synchrony affects memory for speech, and that temporal patterns that are common in production lead to the best recall. Importantly, the study also showcases a procedure for using motion capture-based 3D-animated speakers to create an experimental paradigm for the study of speech-gesture comprehension.

Iconicity ratings for 14,000+ English words

Iconic words and signs are characterized by a perceived resemblance between aspects of their form and aspects of their meaning. For example, in English, iconic words include peep and crash, which mimic the sounds they denote, and wiggle and zigzag, which mimic motion. As a semiotic property of words and signs, iconicity has been demonstrated to play a role in word learning, language processing, and language evolution. This paper presents the results of a large-scale norming study for more than 14,000 English words conducted with over 1400 American English speakers. We demonstrate the utility of these ratings by replicating a number of existing findings showing that iconicity ratings are related to age of acquisition, sensory modality, semantic neighborhood density, structural markedness, and playfulness. We discuss possible use cases and limitations of the rating dataset, which is made publicly available.

Conversational facial signals combine into compositional meanings that change the interpretation of speaker intentions

Human language is extremely versatile, combining a limited set of signals in an unlimited number of ways. However, it is unknown whether conversational visual signals feed into the composite utterances with which speakers communicate their intentions. We assessed whether different combinations of visual signals lead to different intent interpretations of the same spoken utterance. Participants viewed a virtual avatar uttering spoken questions while producing single visual signals (i.e., head turn, head tilt, eyebrow raise) or combinations of these signals. After each video, participants classified the communicative intention behind the question. We found that composite utterances combining several visual signals conveyed different meaning compared to utterances accompanied by the single visual signals. However, responses to combinations of signals were more similar to the responses to related, rather than unrelated, individual signals, indicating a consistent influence of the individual visual signals on the whole. This study therefore provides first evidence for compositional, non-additive (i.e., Gestalt-like) perception of multimodal language.

Self-Repair Increases Referential Coordination

When interlocutors repeatedly describe referents to each other, they rapidly converge on referring expressions which become increasingly systematized and abstract as the interaction progresses. Previous experimental research suggests that interactive repair mechanisms in dialogue underpin convergence. However, this research has so far only focused on the role of other-initiated repair and has not examined whether self-initiated repair might also play a role. To investigate this question, we report the results from a computer-mediated maze task experiment. In this task, participants communicate with each other via an experimental chat tool, which selectively transforms participants' private turn-revisions into public self-repairs that are made visible to the other participant. For example, if a participant, A, types "On the top square," and then before sending, A revises the turn to "On the top row," the server automatically detects the revision and transforms the private turn-revisions into a public self-repair, for example, "On the top square umm I meant row." Participants who received these transformed turns used more abstract and systematized referring expressions, but performed worse at the task. We argue that this is due to the artificial self-repairs causing participants to put more effort into diagnosing and resolving the referential coordination problems they face in the task, yielding better grounded spatial semantics and consequently increased use of abstract referring expressions.

A cross-linguistic, sound symbolic relationship between labial consonants, voiced plosives, and Pokémon friendship

Introduction

This paper presents a cross-linguistic study of sound symbolism, analysing a six-language corpus of all Pokémon names available as of January 2022. It tests the effects of labial consonants and voiced plosives on a Pokémon attribute known as friendship. Friendship is a mechanic in the core series of Pokémon video games that arguably reflects how friendly each Pokémon is.

Method

Poisson regression is used to examine the relationship between the friendship mechanic and the number of times /p/, /b/, /d/, /m/, /g/, and /w/ occur in the names of English, Japanese, Korean, Chinese, German, and French Pokémon.

Results

Bilabial plosives, /p/ and /b/, typically represent high friendship values in Pokémon names while /m/, /d/, and /g/ typically represent low friendship values. No association is found for /w/ in any language.

Discussion

Many of the previously known cases of cross-linguistic sound symbolic patterns can be explained by the relationship between how sounds in words are articulated and the physical qualities of the referents. This study, however, builds upon the underexplored relationship between sound symbolism and abstract qualities.

The Effects of Iconicity and Conventionalization on Word Order Preferences

Of the six possible orderings of the three main constituents of language (subject, verb, and object), two-SOV and SVO-are predominant cross-linguistically. Previous research using the silent gesture paradigm in which hearing participants produce or respond to gestures without speech has shown that different factors such as reversibility, salience, and animacy can affect the preferences for different orders. Here, we test whether participants' preferences for orders that are conditioned on the semantics of the event change depending on (i) the iconicity of individual gestural elements and (ii) the prior knowledge of a conventional lexicon. Our findings demonstrate the same preference for semantically conditioned word order found in previous studies, specifically that SOV and SVO are preferred differentially for different types of events. We do not find that iconicity of individual gestures affects participants' ordering preferences; however, we do find that learning a lexicon leads to a stronger preference for SVO-like orders overall. Finally, we compare our findings from English speakers, using an SVO-dominant language, with data from speakers of an SOV-dominant language, Turkish. We find that, while learning a lexicon leads to an increase in SVO preference for both sets of participants, this effect is mediated by language background and event type, suggesting that an interplay of factors together determines preferences for different ordering patterns. Taken together, our results support a view of word order as a gradient phenomenon responding to multiple biases.

Gesture is the primary modality for language creation

How language began is one of the oldest questions in science, but theories remain speculative due to a lack of direct evidence. Here, we report two experiments that generate empirical evidence to inform gesture-first and vocal-first theories of language origin; in each, we tested modern humans' ability to communicate a range of meanings (995 distinct words) using either gesture or non-linguistic vocalization. Experiment 1 is a cross-cultural study, with signal Producers sampled from Australia (n = 30, Mage = 32.63, s.d. = 12.42) and Vanuatu (n = 30, Mage = 32.40, s.d. = 11.76). Experiment 2 is a cross-experiential study in which Producers were either sighted (n = 10, Mage = 39.60, s.d. = 11.18) or severely vision-impaired (n = 10, Mage = 39.40, s.d. = 10.37). A group of undergraduate student Interpreters guessed the meaning of the signals created by the Producers (n = 140). Communication success was substantially higher in the gesture modality than the vocal modality (twice as high overall; 61.17% versus 29.04% success). This was true within cultures, across cultures and even for the signals produced by severely vision-impaired participants. The success of gesture is attributed in part to its greater universality (i.e. similarity in form across different Producers). Our results support the hypothesis that gesture is the primary modality for language creation.

Pantomimic fossils in modern human communication

Bodily mimesis, the capacity to use the body representationally, was one of the key innovations that allowed early humans to go beyond the 'baseline' of generalized ape communication and cognition. We argue that the original human-specific communication afforded by bodily mimesis was based on signs that involve three entities: an expression that represents an object (i.e. communicated content) for an interpreter. We further propose that the core component of this communication, pantomime, was able to transmit referential information that was not limited to select semantic domains or the 'here-and-now', by means of motivated-most importantly iconic-signs. Pressures for expressivity and economy then led to conventionalization of signs and a growth of linguistic characteristics: semiotic systematicity and combinatorial expression. Despite these developments, both naturalistic and experimental data suggest that the system of pantomime did not disappear and is actively used by modern humans. Its contemporary manifestations, or pantomimic fossils, emerge when language cannot be used, for instance when people do not share a common language, or in situations where the use of (spoken) language is difficult, impossible or forbidden. Under such circumstances, people bootstrap communication by means of pantomime and, when these circumstances persist, newly emergent pantomimic communication becomes increasingly language-like. This article is part of the theme issue 'Reconstructing prehistoric languages'.

Persuasive conversation as a new form of communication in Homo sapiens

The aim of this paper is twofold: to propose that conversation is the distinctive feature of Homo sapiens' communication; and to show that the emergence of modern language is tied to the transition from pantomime to verbal and grammatically complex forms of narrative. It is suggested that (animal and human) communication is a form of persuasion and that storytelling was the best tool developed by humans to convince others. In the early stage of communication, archaic hominins used forms of pantomimic storytelling to persuade others. Although pantomime is a powerful tool for persuasive communication, it is proposed that it is not an effective tool for persuasive conversation: conversation is characterized by a form of reciprocal persuasion among peers; instead, pantomime has a mainly asymmetrical character. The selective pressure towards persuasive reciprocity of the conversational level is the evolutionary reason that allowed the transition from pantomime to grammatically complex codes in H. sapiens, which favoured the evolution of speech. This article is part of the theme issue 'Reconstructing prehistoric languages'.