The Meaning of Motion Lines?: A Review of Theoretical and Empirical Research on Static Depiction of Motion

Static depiction of motion, particularly lines trailing behind a mover, has long been of interest in the psychology literature. Empirical research has demonstrated that these "motion lines" benefited motion comprehension in static images by disambiguating the direction of movement. Yet, there is no consensus on how those lines derive their meaning. In this article, we review three accounts suggesting different interpretations of what motion lines represent. While a perceptual account considers motion lines originating from motion streaks in the primary visual cortex, metaphorical and lexical accounts propose them as graphical conventions that should be learned-either through resemblance to sensory experiences (e.g., natural path marks) or directly being mapped to a conceptual category of paths. To contrast these three accounts, we integrate empirical research on motion lines and their understanding. Overall, developmental, proficiency, and cross-cultural variances indicate that the understanding of motion lines is neither innate nor universal, thus providing less support for lines having a purely perceptual origin. Rather, we argue the empirical findings suggest that motion lines are not iconic depictions of visual percepts but are graphical conventions indexing conceptual path information, which need to be learned and encoded in a visual lexicon.

Lack of Visual Experience Affects Multimodal Language Production: Evidence From Congenitally Blind and Sighted People

The human experience is shaped by information from different perceptual channels, but it is still debated whether and how differential experience influences language use. To address this, we compared congenitally blind, blindfolded, and sighted people's descriptions of the same motion events experienced auditorily by all participants (i.e., via sound alone) and conveyed in speech and gesture. Comparison of blind and sighted participants to blindfolded participants helped us disentangle the effects of a lifetime experience of being blind versus the task-specific effects of experiencing a motion event by sound alone. Compared to sighted people, blind people's speech focused more on path and less on manner of motion, and encoded paths in a more segmented fashion using more landmarks and path verbs. Gestures followed the speech, such that blind people pointed to landmarks more and depicted manner less than sighted people. This suggests that visual experience affects how people express spatial events in the multimodal language and that blindness may enhance sensitivity to paths of motion due to changes in event construal. These findings have implications for the claims that language processes are deeply rooted in our sensory experiences.

Speaking and gesturing guide event perception during message conceptualization: Evidence from eye movements

Speakers' visual attention to events is guided by linguistic conceptualization of information in spoken language production and in language-specific ways. Does production of language-specific co-speech gestures further guide speakers' visual attention during message preparation? Here, we examine the link between visual attention and multimodal event descriptions in Turkish. Turkish is a verb-framed language where speakers' speech and gesture show language specificity with path of motion mostly expressed within the main verb accompanied by path gestures. Turkish-speaking adults viewed motion events while their eye movements were recorded during non-linguistic (viewing-only) and linguistic (viewing-before-describing) tasks. The relative attention allocated to path over manner was higher in the linguistic task compared to the non-linguistic task. Furthermore, the relative attention allocated to path over manner within the linguistic task was higher when speakers (a) encoded path in the main verb versus outside the verb and (b) used additional path gestures accompanying speech versus not. Results strongly suggest that speakers' visual attention is guided by language-specific event encoding not only in speech but also in gesture. This provides evidence consistent with models that propose integration of speech and gesture at the conceptualization level of language production and suggests that the links between the eye and the mouth may be extended to the eye and the hand.

Encoding Motion Events During Language Production: Effects of Audience Design and Conceptual Salience

We investigate the extent to which pragmatic versus conceptual factors can affect a speaker's decision to mention or omit different components of an event. In the two experiments, we demonstrate the special role of pragmatic factors related to audience design in speakers' decisions to mention conceptually "peripheral" event components, such as sources (i.e., starting points) in source-goal motion events (e.g., a baby crawling from a crib to a toybox). In particular, we found that pragmatic factors related to audience design could not only drive the decision to omit sources from mention, but could also motivate speakers to mention sources more often than needed. By contrast, speaker's decisions to talk about goals did not appear to be fundamentally driven by pragmatic factors in communication. We also manipulated the animacy of the figure in motion and found that participants in our studies treated both animate and inanimate source-goal motion events in the same way, both linguistically and in memory. We discuss the implications of our work for message generation across different communicative contexts and for future work on the topic of audience design.

Semantic P600-but not N400-effects index crosslinguistic variability in speakers' expectancies for expression of motion

The expression of motion shows strong crosslinguistic variability; however, less is known about speakers' expectancies for lexicalizations of motion at the neural level. We examined event-related brain potentials (ERPs) in native English or Spanish speakers while they read grammatical sentences describing animations involving manner and path components of motion that did or did not violate language-specific patterns of expression. ERPs demonstrated different expectancies between speakers: Spanish speakers showed higher expectancies for motion verbs to encode path and English speakers showed higher expectancies for motion verbs to encode manner followed by a secondary path expression. Interestingly, grammatical but infrequent motion expressions (manner verbs in Spanish, path verbs and secondary manner expressions in English) elicited semantic P600 rather than the expected N400 effects-with or without post-N400 positivities-that are typically associated with semantic processing. Overall, our findings provide the first empirical evidence for the effect of crosslinguistic variation in processing motion event descriptions at the neural level.

Event segmentation: Cross-linguistic differences in verbal and non-verbal tasks

Events, as fundamental units in human perception and cognition, are limited by quality changes of objects over time. In the present study, we investigate the role of language in shaping event units. Given fundamental cross-linguistic differences in the concepts encoded in the verb, as in French compared to German, event unit formation was tested for motion events in a verbal (online event description, experiment 1), as well as a non-verbal task (Newtson-test, experiment 2). In German, motion and direction are described by a single assertion, i.e. one verb encoding manner (to walk …), in conjunction with adpositional phrases for path and direction (… over x across y toward z). In contrast, when information on path and direction is encoded in the verb, as typically in French, each path segment requires a separate assertion (head for x, cross y, approach z). Both experiments were based on short naturalistic video clips showing a figure moving through space along a path either without changing orientation/direction (control), or with changes in orientation/direction (critical). Analysis of the verbal task concerned the probability of producing more than one assertion to refer to the motion events presented in the clips; in the non-verbal event segmentation task, the analysis concerned the probability of marking an event boundary, as indicated by pressing a button. Results show that in French, the probability of producing more than one assertion was significantly higher in the critical condition (experiment 1) and the probability to identify an event boundary was also significantly higher (experiment 2), compared to the German participants but only in the critical condition. The findings indicate language-driven effects in event unit formation. The results are discussed in the context of theories of event cognition, thereby focusing on the role of language in the formation of cognitive structures.

Do Children Understand Iconic Gestures About Events as Early as Iconic Gestures About Entities?

Children can understand iconic co-speech gestures that characterize entities by age 3 (Stanfield et al. in J Child Lang 40(2):1-10, 2014; e.g., "I'm drinking" [Formula: see text] tilting hand in C-shape to mouth as if holding a glass). In this study, we ask whether children understand co-speech gestures that characterize events as early as they do so for entities, and if so, whether their understanding is influenced by the patterns of gesture production in their native language. We examined this question by studying native English speaking 3- to 4 year-old children and adults as they completed an iconic co-speech gesture comprehension task involving motion events across two studies. Our results showed that children understood iconic co-speech gestures about events at age 4, marking comprehension of gestures about events one year later than gestures about entities. Our findings also showed that native gesture production patterns influenced children's comprehension of gestures characterizing such events, with better comprehension for gestures that follow language-specific patterns compared to the ones that do not follow such patterns-particularly for manner of motion. Overall, these results highlight early emerging abilities in gesture comprehension about motion events.

Blind Speakers Show Language-Specific Patterns in Co-Speech Gesture but Not Silent Gesture

Sighted speakers of different languages vary systematically in how they package and order components of a motion event in speech. These differences influence how semantic elements are organized in gesture, but only when those gestures are produced with speech (co-speech gesture), not without speech (silent gesture). We ask whether the cross-linguistic similarity in silent gesture is driven by the visuospatial structure of the event. We compared 40 congenitally blind adult native speakers of English or Turkish (20/language) to 80 sighted adult speakers (40/language; half with, half without blindfolds) as they described three-dimensional motion scenes. We found an effect of language on co-speech gesture, not on silent gesture-blind speakers of both languages organized their silent gestures as sighted speakers do. Humans may have a natural semantic organization that they impose on events when conveying them in gesture without language-an organization that relies on neither visuospatial cues nor language structure.

Does language shape silent gesture?

Languages differ in how they organize events, particularly in the types of semantic elements they express and the arrangement of those elements within a sentence. Here we ask whether these cross-linguistic differences have an impact on how events are represented nonverbally; more specifically, on how events are represented in gestures produced without speech (silent gesture), compared to gestures produced with speech (co-speech gesture). We observed speech and gesture in 40 adult native speakers of English and Turkish (N=20/per language) asked to describe physical motion events (e.g., running down a path)-a domain known to elicit distinct patterns of speech and co-speech gesture in English- and Turkish-speakers. Replicating previous work (Kita & Özyürek, 2003), we found an effect of language on gesture when it was produced with speech-co-speech gestures produced by English-speakers differed from co-speech gestures produced by Turkish-speakers. However, we found no effect of language on gesture when it was produced on its own-silent gestures produced by English-speakers were identical in how motion elements were packaged and ordered to silent gestures produced by Turkish-speakers. The findings provide evidence for a natural semantic organization that humans impose on motion events when they convey those events without language.

Spontaneous gesture and spatial language: Evidence from focal brain injury

People often use spontaneous gestures when communicating spatial information. We investigated focal brain-injured individuals to test the hypotheses that (1) naming motion event components of manner-path (represented by verbs-prepositions in English) are impaired selectively, (2) gestures compensate for impaired naming. Patients with left or right hemisphere damage (LHD or RHD) and elderly control participants were asked to describe motion events (e.g., running across) depicted in brief videos. Damage to the left posterior middle frontal gyrus, left inferior frontal gyrus, and left anterior superior temporal gyrus (aSTG) produced impairments in naming paths of motion; lesions to the left caudate and adjacent white matter produced impairments in naming manners of motion. While the frequency of spontaneous gestures were low, lesions to the left aSTG significantly correlated with greater production of path gestures. These suggest that producing prepositions-verbs can be separately impaired and gesture production compensates for naming impairments when damage involves left aSTG.

On the road to somewhere: Brain potentials reflect language effects on motion event perception

Recent studies have identified neural correlates of language effects on perception in static domains of experience such as colour and objects. The generalization of such effects to dynamic domains like motion events remains elusive. Here, we focus on grammatical differences between languages relevant for the description of motion events and their impact on visual scene perception. Two groups of native speakers of German or English were presented with animated videos featuring a dot travelling along a trajectory towards a geometrical shape (endpoint). English is a language with grammatical aspect in which attention is drawn to trajectory and endpoint of motion events equally. German, in contrast, is a non-aspect language which highlights endpoints. We tested the comparative perceptual saliency of trajectory and endpoint of motion events by presenting motion event animations (primes) followed by a picture symbolising the event (target): In 75% of trials, the animation was followed by a mismatching picture (both trajectory and endpoint were different); in 10% of trials, only the trajectory depicted in the picture matched the prime; in 10% of trials, only the endpoint matched the prime; and in 5% of trials both trajectory and endpoint were matching, which was the condition requiring a response from the participant. In Experiment 1 we recorded event-related brain potentials elicited by the picture in native speakers of German and native speakers of English. German participants exhibited a larger P3 wave in the endpoint match than the trajectory match condition, whereas English speakers showed no P3 amplitude difference between conditions. In Experiment 2 participants performed a behavioural motion matching task using the same stimuli as those used in Experiment 1. German and English participants did not differ in response times showing that motion event verbalisation cannot readily account for the difference in P3 amplitude found in the first experiment. We argue that, even in a non-verbal context, the grammatical properties of the native language and associated sentence-level patterns of event encoding influence motion event perception, such that attention is automatically drawn towards aspects highlighted by the grammar.