Perspective taking in language: integrating the spatial and action domains

Sharing Perceptual Experiences through Language

The aim of this article is to shed light on how sensory perceptions are communicated through authentic language. What are the language resources available to match multimodal perceptions, and how do we use them in real communication? We discuss insights from previous work on the topic of the interaction of perception, cognition, and language and explain how language users recontextualise perception in communication about sensory experiences. Within the framework of cognitive semantics, we show that the complexities of multimodal perception are clearly reflected in the multifunctional use of words to convey meanings and feelings. To showcase the language resources employed, we base our findings on research on how architects convey their perceptions of built space. Two main patterns emerge: they use multimodal expressions (soft, bland, and jarring) and descriptions of built space through motion (the building reaches out, or routes and directions such as destination, promenade, route, or landscape in combination with verbs such as start and lead) in which case the architect may either be the observer or the emerged actor. The important take-home message is that there is no neat and clear a priori link between words and meanings, but rather "unforeseen" patterns surface in natural production data describing sensory perceptions.

The roles of motion, gesture, and embodied action in the processing of mathematical concepts

This article discusses perspective and frame of reference in the metaphorical description of mathematical concepts in terms of motions, gestures, and embodied actions. When a mathematical concept is described metaphorically in terms of gestures, embodied actions, or fictive motions, the motor system comes into play to ground and understand that concept. Every motion, gesture, or embodied action involves a perspective and a frame of reference. The flexibility in taking perspective and frame of reference allows people to embody a mathematical concept or idea in various ways. Based on the findings of past studies, it is suggested that the graphical representation of a mathematical concept may activate those areas of the motor system that are involved in the production of that graphical representation. This is supported by studies showing that when observers look at a painting or handwritten letters, they simulate the painter's or writer's hand movements during painting or writing. Likewise, the motor system can contribute to the grounding of abstract mathematical concepts, such as functions, numbers, and arithmetic operations.

Is rushing always faster than strolling? A reaction time study on the processing of sentences containing manner of motion verbs

In the context of the embodied cognition debate, an effect of motion verb associated speed information has previously been detected using eye-tracking, functional magnetic resonance imaging (fMRI), and reaction times (RT). The latter, for instance, was implemented by Wender and Weber (1982), who observed that participants were faster in detecting motion in sentences associated with fast motion compared to sentences associated with slow motion after having formed mental images of the sentences' content. It remains open whether the reported effects of speed are associated with automatic lexical-semantic retrieval processes or whether they reflect higher top-down cognitive processes. To answer this question, the paradigm by Wender and Weber (1982) was adopted and further elaborated in the present study. In Experiment 1 visualization instructions were eliminated. Additionally, the stimulus material was manipulated in regards to the agent of the described movement (human vs. object motion) in order to determine the representation's modality (visual vs. motoric). In Experiment 2, the task to detect motion was replaced by the task to judge sensicality. The results suggest that the prompt to perform mental imagery is not a precondition for the engagement of modal representations in this speed of motion paradigm and that the involved representations' modality is visual rather than motoric. However, the modal representations' involvement is dependent on the task. They thus do not seem to be part of the invariant semantic representation of manner of motion verbs.

Neurophysiological mechanisms of perspective-taking: An MEG investigation of agency

According to the embodied cognition framework, sensory and motor areas are recruited during language understanding through simulation processes. Behavioral and imaging findings point to a dependence of the latter on perspective-taking (e.g., first person "I" versus third person "s/he"). The current study aims at identifying possible neurophysiological correlates of perspective in a linguistic context. Twenty healthy participants were measured with magnetoencephalography (MEG) while semantically processing visually presented inflected German verbs in the first- and third-person perspective, simple present tense. Results show that the first-person perspective induces stronger beta (15-25 Hz) desynchronization in the right-hemispheric posterior superior temporal sulcus, ventral posterior cingulate gyrus, and V5/MT+ area; no modulation of sensorimotor cortex emerged. Moreover, a stronger event-related field (ERF) was observed for the first-person perspective at about 150 ms after pronoun-verb onset, originating in occipital and moving to central and left temporal cortical sites. No effect of perspective on sensory gating was found when targeting the N1 component related to tones following the linguistic stimuli. Results indicate an effect of linguistic perspective-taking on brain activation patterns. The contribution of the single brain areas and their role in self-other distinction is further discussed.

Preserved Perspective Taking in Free Indirect Discourse in Autism Spectrum Disorder

Perspective taking has been proposed to be impaired in persons with autism spectrum disorder (ASD), especially when implicit processing is required. In narrative texts, language perception and interpretation is fundamentally guided by taking the perspective of a narrator. We studied perspective taking in the linguistic domain of so-called Free Indirect Discourse (FID), during which certain text segments have to be interpreted as the thoughts or utterances of a protagonist without explicitly being marked as thought or speech representations of that protagonist (as in direct or indirect discourse). Crucially, the correct interpretation of text segments as FID depends on the ability to detect which of the protagonists "stands out" against the others and is therefore identifiable as implicit thinker or speaker. This so-called "prominence" status of a protagonist is based on linguistic properties (e.g., grammatical function, referential expression), in other words, the perspective is "hidden" and has to be inferred from the text material. In order to test whether this implicit perspective taking ability that is required for the interpretation of FID is preserved in persons with ASD, we presented short texts with three sentences to adults with and without ASD. In the last sentence, the perspective was switched either to the more or the less prominent of two protagonists. Participants were asked to rate the texts regarding their naturalness. Both diagnostic groups rated sentences with FID anchored to the less prominent protagonist as less natural than sentences with FID anchored to the more prominent protagonist. Our results that the high-level perspective taking ability in written language that is required for the interpretation of FID is well preserved in persons with ASD supports the conclusion that language skills are highly elaborated in ASD so that even the challenging attribution of utterances to protagonists is possible if they are only implicitly given. We discuss the implications in the context of claims of impaired perspective taking in ASD as well as with regard to the underlying processing of FID.

Perspective in the conceptualization of categories

The ability to differently perceive and represent entities depending on their perspective is crucial for humans. We report five experiments that investigate how the different perspectives adopted while experiencing entities are reflected in conceptualizations (towards vs. away, near vs. far, beside vs. above, inside vs. outside and vision vs. audition vs. touch). Different groups of participants generated object properties while imagining the same scenario from different perspectives (e.g. entities coming toward them/going away from them while on a highway overpass). If conceptualizations have perspectives, then participants should produce features from a perspective entrenched in memory that reflects typical interactions with objects, independently of their assigned perspective (entrenched perspective). In addition, the perspective adopted in a given experiment should influence the properties generated (situated perspective). Results across the experiments indicate that conceptualizations contain both entrenched and situational perspectives. While entrenched perspectives emerge from canonical actions typically performed with objects, locations and entities, situational perspectives reflect online adaptations to current task contexts. The implications of the interplay between entrenched and situational perspectives for grounded cognition are discussed.

Collective benefit in joint perceptual judgments: Partial roles of shared environments, meta-cognition, and feedback

Recent research has demonstrated that two persons can optimally combine their observations to make better decisions when they can verbally communicate their confidence to each other. The present study investigated whether non-verbal interaction in a shared environment can be sufficient to achieve similar collective benefit. Pairs of individuals performed a localization task in a shared virtual 3D environment. In Experiments 1 and 2, partners had access to orthogonal viewpoints providing complementary information. The results showed robust collective benefit compared to individual performance from a single viewpoint, which could be obtained without any opportunity for verbal communication and even when no feedback about joint accuracy was provided (Experiment 2). When partners shared the same viewpoint (Experiment 3), collective benefit was achieved only when feedback on accuracy was provided (Experiment 3a). The findings indicate that sharing an environment can be sufficient for achieving integration of complementary perceptual information. Communicating confidence might not be necessary when an environment is shared. Another possibility is that processes for integrating interpersonally continuous information generally differ from the processes invoked when making a joint dichotomous choice.

I understand you feel that way, but I feel this way: the benefits of I-language and communicating perspective during conflict

Using hypothetical scenarios, we provided participants with potential opening statements to a conflict discussion that varied on I/you language and communicated perspective. Participants rated the likelihood that the recipient of the statement would react in a defensive manner. Using I-language and communicating perspective were both found to reduce perceptions of hostility. Statements that communicated both self- and other-perspective using I-language (e.g. 'I understand why you might feel that way, but I feel this way, so I think the situation is unfair') were rated as the best strategy to open a conflict discussion. Simple acts of initial language use can reduce the chances that conflict discussion will descend into a downward spiral of hostility.

The spatial alignment of time: Differences in alignment of deictic and sequence time along the sagittal and lateral axes

People use space in a variety of ways to structure their thoughts about time. The present report focuses on the different ways that space is employed when reasoning about deictic (past/future relationships) and sequence (earlier/later relationships) time. In the first study, we show that deictic and sequence time are aligned along the lateral axis in a manner consistent with previous work, with past and earlier events associated with left space and future and later events associated with right space. However, the alignment of time with space is different along the sagittal axis. Participants associated future events and earlier events-not later events-with the space in front of their body and past and later events with the space behind, consistent with the sagittal spatial terms (e.g., ahead, in front of) that we use to talk about deictic and sequence time. In the second study, we show that these associations between sequence time and sagittal space are sensitive to person-perspective. This suggests that the particular space-time associations observed in English speakers are influenced by a variety of different spatial properties, including spatial location and perspective.

Grasping the Agent's Perspective: A Kinematics Investigation of Linguistic Perspective in Italian and German

Every day, we primarily experience actions as agents, by having a concrete perspective on our actions, their means and goals. This peculiar perspective is what allows us to successfully plan and execute our actions in a dense social environment. Nevertheless, in this environment actions are also perceived from an observer's perspective. Adopting such a perspective helps us to understand and respond to other's people actions and their outcomes. Importantly, similar experiences of being agent and observer occur also when actions are not physically acted/perceived but are merely linguistically shared. In this paper we present two exploratory studies, one in Italian and one in German, in which we applied a direct comparison of three singular perspectives in combination with different verb categories. First, second and third person pronouns were combined with action and interaction verbs, i.e., verbs implying an interaction with an object - e.g., grasp - or an interaction with an object and another person - e.g., give. By means of kinematics recording, we analyzed participants' reaching-grasping responses to a mouse while they were presented with the different combinations of linguistic stimuli (pronouns and verb type). Results of Experiment 1 on reaching show that, when they are preceded by YOU, interaction verbs reached the velocity peak earlier than action verbs, since a further motor act will follow. Thus pronouns influence perspective taking and while comprehending language we are sensitive to the motor chain organization of verbs. The absence of the same effects in Experiment 2 is likely due to the fact that, being the pronoun in German mandatory, it is perceived as less salient than in Italian. Overall our result supports the idea that language is grounded in the motor system in a flexible way, and highlights the need for cross-linguistic studies in the field of embodied language processing.

Switching reference frame preferences during verbally assisted haptic graph comprehension

Haptic-audio interfaces allow haptic exploration of statistical line graphs accompanied by sound or speech, thus providing access to exploration by visually impaired people. Verbally assisted haptic graph exploration can be seen as a task-oriented collaborative activity between two partners, a haptic explorer and an observing assistant, who are disposed to individual preferences for using reference frames. The experimental findings reveal that haptic explorers' spatial reference frames are mostly induced by hand movements, leading to action perspective instead of conventionally left-to-right spatiotemporal perspective. Moreover, the communicational goal may result in a switch in perspective.

Grounded spatial belief revision

Beliefs frequently undergo revisions, especially when new pieces of information are true but inconsistent with current beliefs. In previous studies, we showed that linguistic asymmetries provided by relational statements, play a crucial role in spatial belief revision. Located objects (LO) are preferably revised compared to reference objects (RO), known as the LO-principle. Here we establish a connection between spatial belief revision and grounded cognition. In three experiments, we explored whether imagined physical object properties influence which object is relocated and which remains at its initial position. Participants mentally revised beliefs about the arrangements of objects which could be envisaged as light and heavy (Experiment 1), small and large (Experiment 2), or movable and immovable (Experiment 3). The results show that intrinsic object properties are differently taken into account during spatial belief revision. Object weight did not alter the LO-principle (Experiment 1), whereas object size was found to influence which object was preferably relocated (Experiment 2). Object movability did not affect relocation preferences but had an effect on relocation durations (Experiment 3). The findings support the simulation hypothesis within the grounded cognition approach and create new connections between the spatial mental model theory of reasoning and the idea of grounded cognition.

Does listening to action-related sentences modulate the activity of the motor system? Replication of a combined TMS and behavioral study

The neurophysiological and behavioral correlates of action-related language processing have been debated for long time. A precursor in this field was the study by Buccino et al. (2005) combining transcranial magnetic stimulation (TMS) and behavioral measures (reaction times, RTs) to study the effect of listening to hand- and foot-related sentences. In the TMS experiment, the authors showed a decrease of motor evoked potentials (MEPs) recorded from hand muscles when processing hand-related verbs as compared to foot-related verbs. Similarly, MEPs recorded from leg muscles decreased when participants processed foot-related as compared to hand-related verbs. In the behavioral experiment, using the same stimuli and a semantic decision task the authors found slower RTs when the participants used the body effector (hand or foot) involved in the actual execution of the action expressed by the presented verb to give their motor responses. These findings were interpreted as an interference effect due to a simultaneous involvement of the motor system in both a language and a motor task. Our replication aimed to enlarge the sample size and replicate the findings with higher statistical power. The TMS experiment showed a significant modulation of hand MEPs, but in the sense of a motor facilitation when processing hand-related verbs. On the contrary, the behavioral experiment did not show significant results. The results are discussed within the general debate on the time-course of the modulation of motor cortex during implicit and explicit language processing and in relation to the studies on action observation/understanding.

A test of motor skill-specific action embodiment in ice-hockey players

To further our understanding of the role of the motor system in comprehending action-related sentences, we compared action experts (athletes) to visual experts (fans) and novices when responding with an action-specific effector (either hand or foot). These conditions allowed inferences about the degree and specificity of embodiment in language comprehension. Ice hockey players, fans and novices made speeded judgments regarding the congruence between an auditorily presented sentence and a subsequently presented picture. Picture stimuli consisted of either hockey or everyday items. Half of these pictures 'matched' the action implied in the preceding sentence. Further, the action in these images involved either primarily the hand or the foot. For everyday items, action-matched items were responded to faster than action-mismatched items. However, only the players and fans showed the action-match effect for hockey items. There were no consistent effector-stimuli compatibility effects, nor skill-based interactions with compatibility, suggesting that the action-match effect was not based on motor ability per se, but rather a construction of the action based on knowledge or visual experience with the hockey related sentences.