This and That Revisited: A Social and Multimodal Approach to Spatial Demonstratives

Children's understanding of demonstratives: an experimental study with German-speaking children between 5 and 7 years of age

Demonstratives ("this"/"that") express a speaker-relative distance contrast and need to be substituted for each other systematically: depending on their relative position, what one speaker refers to by saying "this" another speaker has to refer to by saying "that." This substitution aspect of demonstratives poses additional difficulties for learning demonstratives, because it requires recognizing that two speakers have to refer to the same thing with different words, and might be one reason for the reportedly protracted acquisition of demonstratives. In an online study conducted in German, it was investigated whether children in the estimated upper age range of demonstrative acquisition (5 to 7 years) understand demonstratives' substitution aspect with familiar ("dies"/"das") and novel ("schmi"/"schmu") demonstratives, and whether they understand novel words ("schmi"/"schmu") when used non-demonstratively as labels (N = 73; between-subject). Children's accuracy was compared with adult performance (N = 74). The study shows that children between 5 and 7 years of age perform less accurately than adults in all conditions. While adults' performance was highly accurate in all conditions (between 75% and 92% correct), children performed below chance in both demonstrative conditions and above chance in the labeling condition. This suggests that children do not understand demonstratives in the presented setup. More detailed analyses of children's response patterns indicate that they instead treat words as mutually exclusive labels in any condition.

Extending the Architecture of Language From a Multimodal Perspective

Language is inherently multimodal. In spoken languages, combined spoken and visual signals (e.g., co-speech gestures) are an integral part of linguistic structure and language representation. This requires an extension of the parallel architecture, which needs to include the visual signals concomitant to speech. We present the evidence for the multimodality of language. In addition, we propose that distributional semantics might provide a format for integrating speech and co-speech gestures in a common semantic representation.

Spatial communication systems across languages reflect universal action constraints

The extent to which languages share properties reflecting the non-linguistic constraints of the speakers who speak them is key to the debate regarding the relationship between language and cognition. A critical case is spatial communication, where it has been argued that semantic universals should exist, if anywhere. Here, using an experimental paradigm able to separate variation within a language from variation between languages, we tested the use of spatial demonstratives-the most fundamental and frequent spatial terms across languages. In n = 874 speakers across 29 languages, we show that speakers of all tested languages use spatial demonstratives as a function of being able to reach or act on an object being referred to. In some languages, the position of the addressee is also relevant in selecting between demonstrative forms. Commonalities and differences across languages in spatial communication can be understood in terms of universal constraints on action shaping spatial language and cognition.

An information-theoretic approach to the typology of spatial demonstratives

We explore systems of spatial deictic words (such as 'here' and 'there') from the perspective of communicative efficiency using typological data from over 200 languages Nintemann et al. (2020). We argue from an information-theoretic perspective that spatial deictic systems balance informativity and complexity in the sense of the Information Bottleneck (Zaslavsky et al., (2018). We find that under an appropriate choice of cost function and need probability over meanings, among all the 21,146 theoretically possible spatial deictic systems, those adopted by real languages lie near an efficient frontier of informativity and complexity. Moreover, we find that the conditions that the need probability and the cost function need to satisfy for this result are consistent with the cognitive science literature on spatial cognition, especially regarding the source-goal asymmetry. We further show that the typological data are better explained by introducing a notion of consistency into the Information Bottleneck framework, which is jointly optimized along with informativity and complexity.

Learning to generate pointing gestures in situated embodied conversational agents

One of the main goals of robotics and intelligent agent research is to enable them to communicate with humans in physically situated settings. Human communication consists of both verbal and non-verbal modes. Recent studies in enabling communication for intelligent agents have focused on verbal modes, i.e., language and speech. However, in a situated setting the non-verbal mode is crucial for an agent to adapt flexible communication strategies. In this work, we focus on learning to generate non-verbal communicative expressions in situated embodied interactive agents. Specifically, we show that an agent can learn pointing gestures in a physically simulated environment through a combination of imitation and reinforcement learning that achieves high motion naturalness and high referential accuracy. We compared our proposed system against several baselines in both subjective and objective evaluations. The subjective evaluation is done in a virtual reality setting where an embodied referential game is played between the user and the agent in a shared 3D space, a setup that fully assesses the communicative capabilities of the generated gestures. The evaluations show that our model achieves a higher level of referential accuracy and motion naturalness compared to a state-of-the-art supervised learning motion synthesis model, showing the promise of our proposed system that combines imitation and reinforcement learning for generating communicative gestures. Additionally, our system is robust in a physically-simulated environment thus has the potential of being applied to robots.

Choice of Mandarin Spatial Demonstratives in Distant Interaction

Scholars are divided on whether the Speaker's (S') choice of spatial demonstratives in verbal interaction is ego-centric or not. We studied the choice of "zhe (here)/ na (there)" by a Mandarin S instructing Hearer (H) from a few to dozens of meters. Using within-group and between-group experiments in a picture-description paradigm, we found that both S- and H- distance (Ds and Dh) to the Referent constantly influence S' demonstrative choice, and the social relation as a variable (Relation) between S and H also exerts some influence. Our findings support the idea that spatial reference in verbal interaction is somewhat non-egocentric.

Sign advantage: Both children and adults' spatial expressions in sign are more informative than those in speech and gestures combined

Expressing Left-Right relations is challenging for speaking-children. Yet, this challenge was absent for signing-children, possibly due to iconicity in the visual-spatial modality of expression. We investigate whether there is also a modality advantage when speaking-children's co-speech gestures are considered. Eight-year-old child and adult hearing monolingual Turkish speakers and deaf signers of Turkish-Sign-Language described pictures of objects in various spatial relations. Descriptions were coded for informativeness in speech, sign, and speech-gesture combinations for encoding Left-Right relations. The use of co-speech gestures increased the informativeness of speakers' spatial expressions compared to speech-only. This pattern was more prominent for children than adults. However, signing-adults and children were more informative than child and adult speakers even when co-speech gestures were considered. Thus, both speaking- and signing-children benefit from iconic expressions in visual modality. Finally, in each modality, children were less informative than adults, pointing to the challenge of this spatial domain in development.

Demonstrative systems: From linguistic typology to social cognition

This study explores the connection between language and social cognition by empirically testing different typological analyses of various demonstrative systems. Linguistic typology classifies demonstrative systems as distance-oriented or person-oriented, depending on whether they indicate the location of a referent relative only to the speaker, or to both the speaker and the listener. From the perspective of social cognition, speakers of languages with person-oriented systems must monitor their listener's spatial location in order to accurately use their demonstratives, while speakers of languages with distance-oriented systems can use demonstratives from their own, egocentric perspective. Resolving an ongoing controversy around the nature of the Spanish demonstrative system, the results of Experiment 1 confirmed that this demonstrative system is person oriented, while the English system is distance oriented. Experiment 2 revealed that not all three-way demonstrative systems are person oriented, with Japanese speakers showing sensitivity to the listener's spatial location, while Turkish speakers did not show such an effect in their demonstrative choice. In Experiment 3, Catalan-Spanish bilinguals showed sensitivity to listener position in their choice of the Spanish distal form, but not in their choice of the medial form. These results were interpreted as a transfer effect from Catalan, which revealed analogous results to English. Experiment 4 investigated the use of demonstratives to redirect a listener's attention to the intended referent, which is a universal function of demonstratives that also hinges on social cognition. Japanese and Spanish speakers chose between their proximal and distal demonstratives flexibly, depending on whether the listener was looking closer or further from the referent, whereas Turkish speakers chose their medial form for attention correction. In conclusion, the results of this study support the view that investigating how speakers of different languages jointly use language and social cognition in communication has the potential to unravel the deep connection between these two fundamentally human capacities.

Children's use of demonstrative words: spatial deictics beyond infancy

Demonstrative words are one of the most important ways of establishing reference in conversation. This work describes Spanish-speaking children's demonstrative production between ages 2 to 10 using data from the CHILDES corpora. Results indicate that children feature all demonstratives in their lexicon - however, the distal term is scarce throughout development. Moreover, patterns of demonstrative use are not adult-like at age 10. We compare adult and child data to conclude that children's development of demonstrative production is largely protracted. Adult use of the distal demonstrative is higher than in young children, although both older children and adults use the medial term ese more than any other demonstratives. In contrast, younger children use proximals relatively more frequently than older children and adults. Suggestions for future research and theoretical implications for the Spanish demonstrative system are discussed.

Considering the Nature of Multimodal Language from a Crosslinguistic Perspective

Language in its primary face-to-face context is multimodal (e.g., Holler and Levinson, 2019; Perniss, 2018). Thus, understanding how expressions in the vocal and visual modalities together contribute to our notions of language structure, use, processing, and transmission (i.e., acquisition, evolution, emergence) in different languages and cultures should be a fundamental goal of language sciences. This requires a new framework of language that brings together how arbitrary and non-arbitrary and motivated semiotic resources of language relate to each other. Current commentary evaluates such a proposal by Murgiano et al (2021) from a crosslinguistic perspective taking variation as well as systematicity in multimodal utterances into account.

Situating Language in the Real-World: The Role of Multimodal Iconicity and Indexicality

In the last decade, a growing body of work has convincingly demonstrated that languages embed a certain degree of non-arbitrariness (mostly in the form of iconicity, namely the presence of imagistic links between linguistic form and meaning). Most of this previous work has been limited to assessing the degree (and role) of non-arbitrariness in the speech (for spoken languages) or manual components of signs (for sign languages). When approached in this way, non-arbitrariness is acknowledged but still considered to have little presence and purpose, showing a diachronic movement towards more arbitrary forms. However, this perspective is limited as it does not take into account the situated nature of language use in face-to-face interactions, where language comprises categorical components of speech and signs, but also multimodal cues such as prosody, gestures, eye gaze etc. We review work concerning the role of context-dependent iconic and indexical cues in language acquisition and processing to demonstrate the pervasiveness of non-arbitrary multimodal cues in language use and we discuss their function. We then move to argue that the online omnipresence of multimodal non-arbitrary cues supports children and adults in dynamically developing situational models.

A conceptual framework for the study of demonstrative reference

Language allows us to efficiently communicate about the things in the world around us. Seemingly simple words like this and that are a cornerstone of our capability to refer, as they contribute to guiding the attention of our addressee to the specific entity we are talking about. Such demonstratives are acquired early in life, ubiquitous in everyday talk, often closely tied to our gestural communicative abilities, and present in all spoken languages of the world. Based on a review of recent experimental work, here we introduce a new conceptual framework of demonstrative reference. In the context of this framework, we argue that several physical, psychological, and referent-intrinsic factors dynamically interact to influence whether a speaker will use one demonstrative form (e.g., this) or another (e.g., that) in a given setting. However, the relative influence of these factors themselves is argued to be a function of the cultural language setting at hand, the theory-of-mind capacities of the speaker, and the affordances of the specific context in which the speech event takes place. It is demonstrated that the framework has the potential to reconcile findings in the literature that previously seemed irreconcilable. We show that the framework may to a large extent generalize to instances of endophoric reference (e.g., anaphora) and speculate that it may also describe the specific form and kinematics a speaker's pointing gesture takes. Testable predictions and novel research questions derived from the framework are presented and discussed.

Effects of Scale on Multimodal Deixis: Evidence From Quiahije Chatino

As humans interact in the world, they often orient one another's attention to objects through the use of spoken demonstrative expressions and head and/or hand movements to point to the objects. Although indicating behaviors have frequently been studied in lab settings, we know surprisingly little about how demonstratives and pointing are used to coordinate attention in large-scale space and in natural contexts. This study investigates how speakers of Quiahije Chatino, an indigenous language of Mexico, use demonstratives and pointing to give directions to named places in large-scale space across multiple scales (local activity, district, state). The results show that the use and coordination of demonstratives and pointing change as the scale of search space for the target grows. At larger scales, demonstratives and pointing are more likely to occur together, and the two signals appear to manage different aspects of the search for the target: demonstratives orient attention primarily to the gesturing body, while pointing provides cues for narrowing the search space. These findings underscore the distinct contributions of speech and gesture to the linguistic composite, while illustrating the dynamic nature of their interplay.

Abstracts in Spanish and Quiahije Chatino are provided as appendices.

Se incluyen como apéndices resúmenes en español y en el chatino de San Juan Quiahije. Son^G ktyi^C re^C in^H, ngyaq^C ska^E ktyi^C no^E nda^H son^B na^F nga^J no^I ngyaq^C lo^E ktyi^C re^C, ngyaq^C ran^F chaq^E xlya^K qo^E chaq^F jnya^J no^A ndywiq^A renq^A Kchin^A Kyqya^C.

Common Ground in Demonstrative Reference: The Case of Mano (Mande)

That demonstratives often have endophoric functions marking referents outside the physical space of interaction but accessible through cognition, especially memory, is well-known. These functions are often classified as independent from exophoric ones and are typically seen as secondary with respect to spatial deixis. However, data from multiple languages show that cognitive access to referents functions alongside of perceptual access, including vision. Cognitive access is enabled by prior interactions and prior familiarity with the referents. As a result of such interactions, the interlocutors share a great deal of knowledge about the referents, which facilitates reference to objects in the interactive field. The centrality of common ground in reference to an object at the interactive scene challenges the often assumed classification of demonstrative reference into exophoric and endophoric. I illustrate this idea throughout the paper by using first-hand data from Mano, a Mande language of Guinea. Adding another argument in favor of viewing demonstrative reference as a social, interactive process, the Mano data push the idea of salience of non-spatial parameters further and emphasizes the importance of short and long-term interactional history and cultural knowledge both for the choice of demonstratives in exophoric reference and for the structuring of the demonstrative paradigm.

Demonstratives in Spatial Language and Social Interaction: An Interdisciplinary Review

This paper offers a review of research on demonstratives from an interdisciplinary perspective. In particular, we consider the role of demonstratives in current research on language universals, language evolution, language acquisition, multimodal communication, signed language, language and perception, language in interaction, spatial imagery, and discourse processing. Traditionally, demonstratives are analyzed as a particular class of spatial deictics. Yet, a number of recent studies have argued that space is largely irrelevant to deixis and that demonstratives are primarily used for social and interactive purposes. Synthesizing findings in the literature, we conclude that demonstratives are a very special class of linguistic items that are foundational to both spatial and social aspects of language and cognition.

Demonstrative Reference and Semantic Space: A Large-Scale Demonstrative Choice Task Study

Spatial demonstratives (words like this and that) have been thought to primarily be used for carving up space into a peripersonal and extrapersonal domain. However, when given a noun out of context and asked to couple it with a demonstrative, speakers tend to choose this for words denoting manipulable objects (small, harmless, and inanimate), while non-manipulable objects (large, harmful, and animate) are more likely to be coupled with that. Here, we extend these findings using the Demonstrative Choice Task (DCT) procedure and map demonstrative use along a wide spectrum of semantic features. We conducted a large-scale (N = 2197) DCT experiment eliciting demonstratives for 506 words, rated across 65 + 11 perceptually and cognitively relevant semantic dimensions. We replicated the finding that demonstrative choice is influenced by object manipulability. Demonstrative choice was furthermore found to be related to a set of additional semantic factors, including valence, arousal, loudness, motion, time and more generally, the self. Importantly, demonstrative choices were highly structured across participants, as shown by a strong correlation detected in a split-sample comparison of by-word demonstrative choices. We argue that the DCT may be used to map a generalized semantic space anchored in the self of the speaker, the self being an extension of the body beyond physical space into a multidimensional semantic space.

The Impact of a Human Figure in a Scene on Spatial Descriptions in Speech, Gesture, and Gesture Alone

The presence of a human figure in a scene appears to change how people describe it. About 20% of participants take the human figure's viewpoint (Tversky and Hard in Cognition 110:124-129, 2009. http://doi.org/10.1016/j.cognition.2008.10.008). Five exploratory studies compare descriptions of a scene with no person to descriptions of a scene with a person. About 20% of participants are predicted to use the person's point of view in the "person" conditions. Study 1 replicates the original pattern. Study 2 shows that the pattern holds when object/scene are changed, and that the figure's gaze towards/away from the object does not change the pattern. Studies 3 and 4 show the pattern holds when the object has different positions and when it is moving. Study 5 shows the pattern holds when the describer is talking to an interlocutor, in both speech and co-speech gesture, and when the person is using gesture alone. The presence of a human figure in a scene appears to be a robust variable in shaping spatial descriptions.

This is for you: Social modulations of proximal vs. distal space in collaborative interaction

Human spatial representations are shaped by affordances for action offered by the environment. A prototypical example is the organization of space into peripersonal (within reach) and extrapersonal (outside reach) regions, mirrored by proximal (this/here) and distal (that/there) linguistic expressions. The peri-/extrapersonal distinction has been widely investigated in individual contexts, but little is known about how spatial representations are modulated by interaction with other people. Is near/far coding of space dynamically adapted to the position of a partner when space, objects, and action goals are shared? Over two preregistered experiments based on a novel interactive paradigm, we show that, in individual and social contexts involving no direct collaboration, linguistic coding of locations as proximal or distal depends on their distance from the speaker's hand. In contrast, in the context of collaborative interactions involving turn-taking and role reversal, proximal space is shifted towards the partner, and linguistic coding of near space ('this' / 'here') is remapped onto the partner's action space.

When to Take a Gesture Seriously: On How We Use and Prioritize Communicative Cues

When people talk, their speech is often accompanied by gestures. Although it is known that co-speech gestures can influence face-to-face communication, it is currently unclear to what extent they are actively used and under which premises they are prioritized to facilitate communication. We investigated these open questions in two experiments that varied how pointing gestures disambiguate the utterances of an interlocutor. Participants, whose event-related brain responses were measured, watched a video, where an actress was interviewed about, for instance, classical literature (e.g., Goethe and Shakespeare). While responding, the actress pointed systematically to the left side to refer to, for example, Goethe, or to the right to refer to Shakespeare. Her final statement was ambiguous and combined with a pointing gesture. The P600 pattern found in Experiment 1 revealed that, when pointing was unreliable, gestures were only monitored for their cue validity and not used for reference tracking related to the ambiguity. However, when pointing was a valid cue (Experiment 2), it was used for reference tracking, as indicated by a reduced N400 for pointing. In summary, these findings suggest that a general prioritization mechanism is in use that constantly monitors and evaluates the use of communicative cues against communicative priors on the basis of accumulated error information.