Event completion: event based inferences distort memory in a matter of seconds

Aspectual Processing Shifts Visual Event Apprehension

What is the relationship between language and event cognition? Past work has suggested that linguistic/aspectual distinctions encoding the internal temporal profile of events map onto nonlinguistic event representations. Here, we use a novel visual detection task to directly test the hypothesis that processing telic versus atelic sentences (e.g., "Ebony folded a napkin in 10 seconds" vs. "Ebony did some folding for 10 seconds") can influence whether the very same visual event is processed as containing distinct temporal stages including a well-defined endpoint or lacking such structure, respectively. In two experiments, we show that processing (a)telicity in language shifts how people later construe the temporal structure of identical visual stimuli. We conclude that event construals are malleable representations that can align with the linguistic framing of events.

False memories for ending of events

Memories are not perfect recordings of the past and can be subject to systematic biases. Memory distortions are often caused by our experience of what typically happens in a given situation. However, it is unclear whether memory for events is biased by the knowledge that events usually have a predictable structure (a beginning, middle, and an end). Using video clips of everyday situations, we tested how interrupting events at unexpected time points affects memory of how those events ended. In four free recall experiments (1, 2, 4, and 5), we found that interrupting clips just before a salient piece of action was completed, resulted in the false recall of details about how the clip might have ended. We refer to this as "event extension." On the other hand, interrupting clips just after one scene had ended and a new scene started, resulted in omissions of details about the true ending of the clip (Experiments 4 and 5). We found that these effects were present, albeit attenuated, when testing memory shortly after watching the video clips compared to a week later (Experiments 5a and 5b). The event extension effect was not present when memory was tested with a recognition paradigm (Experiment 3). Overall, we conclude that when people watch videos that violate their expectations of typical event structure, they show a bias to later recall the videos as if they had ended at a predictable event boundary, exhibiting event extension or the omission of details depending on where the original video was interrupted. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Causality and continuity close the gaps in event representations

Imagine you see a video of someone pulling back their leg to kick a soccer ball, and then a soccer ball soaring toward a goal. You would likely infer that these scenes are two parts of the same event, and this inference would likely cause you to remember having seen the moment the person kicked the soccer ball, even if that information was never actually presented (Strickland & Keil, 2011, Cognition, 121[3], 409-415). What cues trigger people to "fill in" causal events from incomplete information? Is it due to the experience they have had with soccer balls being kicked toward goals? Is it the visual similarity of the object in both halves of the video? Or is it the mere spatiotemporal continuity of the event? In three experiments, we tested these different potential mechanisms underlying the "filling-in" effect. Experiment 1 showed that filling in occurs equally in familiar and unfamiliar contexts, indicating that familiarity with specific event schemas is unnecessary to trigger false memory. Experiment 2 showed that the visible continuation of a launched object's trajectory is all that is required to trigger filling in, regardless of other occurrences in the second half of the scene. Finally, Experiment 3 found that, using naturalistic videos, this filling-in effect is more heavily affected if the object's trajectory is discontinuous in space/time compared with if the object undergoes a noticeable transformation. Together, these findings indicate that the spontaneous formation of causal event representations is driven by object representation systems that prioritize spatiotemporal information over other object features.

The Perception of Relations

The world contains not only objects and features (red apples, glass bowls, wooden tables), but also relations holding between them (apples contained in bowls, bowls supported by tables). Representations of these relations are often developmentally precocious and linguistically privileged; but how does the mind extract them in the first place? Although relations themselves cast no light onto our eyes, a growing body of work suggests that even very sophisticated relations display key signatures of automatic visual processing. Across physical, eventive, and social domains, relations such as support, fit, cause, chase, and even socially interact are extracted rapidly, are impossible to ignore, and influence other perceptual processes. Sophisticated and structured relations are not only judged and understood, but also seen - revealing surprisingly rich content in visual perception itself.

Cross-linguistic frequency and the learnability of semantics: Artificial language learning studies of evidentiality

It is often assumed that cross-linguistically more prevalent distinctions are easier to learn (Typological Prevalence Hypothesis; TPH). Prior work supports this idea in phonology, morphology and syntax but has not addressed semantics. Using Artificial Language Learning experiments with adults, we test predictions made by the TPH about the relative learnability of semantic distinctions in the domain of evidentiality, i.e., the linguistic encoding of information source. As the TPH predicted, when exposed to miniature evidential morphological systems, adult speakers of English whose language does not encode evidentiality grammatically learned the typologically most prevalent system (marking indirect, reportative information) better compared to less-attested systems (Experiments 1-2). Similar patterns were observed when non-linguistic symbols were used to encode evidential distinctions (Experiment 3). Our data support the conjecture that some semantic distinctions are marked preferentially and acquired more easily compared to others in both language and other symbolic systems.

Event Perception and Memory

Events make up much of our lived experience, and the perceptual mechanisms that represent events in experience have pervasive effects on action control, language use, and remembering. Event representations in both perception and memory have rich internal structure and connections one to another, and both are heavily informed by knowledge accumulated from previous experiences. Event perception and memory have been identified with specific computational and neural mechanisms, which show protracted development in childhood and are affected by language use, expertise, and brain disorders and injuries. Current theoretical approaches focus on the mechanisms by which events are segmented from ongoing experience, and emphasize the common coding of events for perception, action, and memory. Abetted by developments in eye-tracking, neuroimaging, and computer science, research on event perception and memory is moving from small-scale laboratory analogs to the complexity of events in the wild.

Filling the gap despite full attention: the role of fast backward inferences for event completion

The comprehension of dynamic naturalistic events poses at least two challenges to the cognitive system: filtering relevant information with attention and dealing with information that was missing or missed. With four experiments, we studied the completion of missing information despite full attention. Participants watched short soccer video clips and we informed participants that we removed a critical moment of ball contact in half of the clips. We asked participants to detect whether these moments of ball contact were present or absent. In Experiment 1, participants gave their detection responses either directly during an event or delayed after an event. Although participants directed their full attention toward the critical contact moment, they were more likely to indicate seeing the missing ball contact if it was followed by a causally matching scene than if it was followed by an unrelated scene, both for the immediate and delayed responses. Thus, event completion occurs quickly. In Experiment 2, only a causally matching scene but neither a white mask nor an irrelevant scene caused the completion of missing information. This indicates that the completion of missing information is caused by backward inferences rather than predictive perception. In Experiment 3, we showed that event completion occurs directly during a trial and does not depend on expectations built up after seeing the same causality condition multiple times. In Experiment 4, we linked our findings to event cognition by asking participants to perform a natural segmentation task. We conclude that observers complete missing information during coherent events based on a fast backward inference mechanism even when directing their attention toward the missing information.

Watching diagnoses develop: Eye movements reveal symptom processing during diagnostic reasoning

Finding a probable explanation for observed symptoms is a highly complex task that draws on information retrieval from memory. Recent research suggests that observed symptoms are interpreted in a way that maximizes coherence for a single likely explanation. This becomes particularly clear if symptom sequences support more than one explanation. However, there are no existing process data available that allow coherence maximization to be traced in ambiguous diagnostic situations, where critical information has to be retrieved from memory. In this experiment, we applied memory indexing, an eye-tracking method that affords rich time-course information concerning memory-based cognitive processing during higher order thinking, to reveal symptom processing and the preferred interpretation of symptom sequences. Participants first learned information about causes and symptoms presented in spatial frames. Gaze allocation to emptied spatial frames during symptom processing and during the diagnostic response reflected the subjective status of hypotheses held in memory and the preferred interpretation of ambiguous symptoms. Memory indexing traced how the diagnostic decision developed and revealed instances of hypothesis change and biases in symptom processing. Memory indexing thus provided direct online evidence for coherence maximization in processing ambiguous information.

Not so secret agents: Event-related potentials to semantic roles in visual event comprehension

Research across domains has suggested that agents, the doers of actions, have a processing advantage over patients, the receivers of actions. We hypothesized that agents as "event builders" for discrete actions (e.g., throwing a ball, punching) build on cues embedded in their preparatory postures (e.g., reaching back an arm to throw or punch) that lead to (predictable) culminating actions, and that these cues afford frontloading of event structure processing. To test this hypothesis, we compared event-related brain potentials (ERPs) to averbal comic panels depicting preparatory agents (ex. reaching back an arm to punch) that cued specific actions with those to non-preparatory agents (ex. arm to the side) and patients that did not cue any specific actions. We also compared subsequent completed action panels (ex. agent punching patient) across conditions, where we expected an inverse pattern of ERPs indexing the differential costs of processing completed actions asa function of preparatory cues. Preparatory agents evoked a greater frontal positivity (600-900ms) relative to non-preparatory agents and patients, while subsequent completed actions panels following non-preparatory agents elicited a smaller frontal positivity (600-900ms). These results suggest that preparatory (vs. non-) postures may differentially impact the processing of agents and subsequent actions in real time.

Neural Representations of Observed Actions Generalize across Static and Dynamic Visual Input

People interact with entities in the environment in distinct and categorizable ways (e.g., kicking is making contact with foot). We can recognize these action categories across variations in actors, objects, and settings; moreover, we can recognize them from both dynamic and static visual input. However, the neural systems that support action recognition across these perceptual differences are unclear. Here, we used multivoxel pattern analysis of fMRI data to identify brain regions that support visual action categorization in a format-independent way. Human participants were scanned while viewing eight categories of interactions (e.g., pulling) depicted in two visual formats: (1) visually controlled videos of two interacting actors and (2) visually varied photographs selected from the internet involving different actors, objects, and settings. Action category was decodable across visual formats in bilateral inferior parietal, bilateral occipitotemporal, left premotor, and left middle frontal cortex. In most of these regions, the representational similarity of action categories was consistent across subjects and visual formats, a property that can contribute to a common understanding of actions among individuals. These results suggest that the identified brain regions support action category codes that are important for action recognition and action understanding.SIGNIFICANCE STATEMENT Humans tend to interpret the observed actions of others in terms of categories that are invariant to incidental features: whether a girl pushes a boy or a button and whether we see it in real-time or in a single snapshot, it is still pushing Here, we investigated the brain systems that facilitate the visual recognition of these action categories across such differences. Using fMRI, we identified several areas of parietal, occipitotemporal, and frontal cortex that exhibit action category codes that are similar across viewing of dynamic videos and still photographs. Our results provide strong evidence for the involvement of these brain regions in recognizing the way that people interact physically with objects and other people.

Mind the gap: Temporal discontinuities in observed activity streams influence perceived duration of actions

In everyday life, when observing activities taking place in our environment, we often shift our attention among several activities and therefore perceive each activity sequence piecemeal with temporal gaps in between. Two studies examined whether the length of these gaps influences the processing of the observed activities. Experiment 1 presented film clips depicting activities that were interrupted by either short or long gaps and asked participants to estimate how long the target action presented at the end of the clip would normally take if it were to take place in reality. Using the same activities, Experiment 2 asked participants to judge the duration of the presentation of this target action-that is, how long the target action was presented. Results showed that following long gaps instead of short gaps, target actions are estimated to take longer in reality (Experiment 1), but the depictions themselves are estimated to be shorter (Experiment 2). Following long gaps, target actions seem to be processed pars pro toto as placeholders for longer segments in the stream of events, but in contrast, the depictions themselves appear to be shorter. Results suggest that long gaps lengthen the perceived duration of an event in our cognitive representation and also seem to influence our perception of the duration of the presentation itself.

Seeing the unseen? Illusory causal filling in FIFA referees, players, and novices

Humans often falsely report having seen a causal link between two dynamic scenes if the second scene depicts a valid logical consequence of the initial scene. As an example, a video clip shows someone kicking a ball including the ball flying. Even if the video clip omitted the moment of contact (i.e., the causal link), participants falsely report having seen this moment. In the current study, we explored the interplay of cognitive-perceptual expertise and event perception by measuring the false-alarm rates of three groups with differing interests in football (soccer in North America) (novices, players, and FIFA referees). We used the event-completion paradigm with video footage of a real football match, presenting either complete clips or incomplete clips (i.e., with the contact moment omitted). Either a causally linked scene or an incoherent scene followed a cut in the incomplete videos. Causally linked scenes induced false recognitions in all three groups: although the ball contact moment was not presented, participants indicated that they had seen the contact as frequently when it was absent as in the complete condition. In a second experiment, we asked the novices to detect the ball contact moment when it was either visible or not and when it was either followed by a causally or non-causally linked scene. Here, instead of presenting pictures of the clip, the participants were give a two-alternative forced-choice task: “Yes, contact was visible”, or “No, contact was not visible”. The results of Experiment 1 indicate that conceptual interpretations of simple events are independent of expertise: there were no top-down effects on perception. Participants in Experiment 2 detected the ball contact moment significantly more often correctly in the non-causal than in the causal conditions, indicating that the effect observed in Experiment 1 was not due to a possibly influential design (e.g., inducing a false memory for the presented pictures). The theoretical as well as the practical implications are discussed.

First-time viewers' comprehension of films: bridging shot transitions

Which perceptual and cognitive prerequisites must be met in order to be able to comprehend a film is still unresolved and a controversial issue. In order to gain some insights into this issue, our field experiment investigates how first-time adult viewers extract and integrate meaningful information across film cuts. Three major types of commonalities between adjacent shots were differentiated, which may help first-time viewers with bridging the shots: pictorial, causal, and conceptual. Twenty first-time, 20 low-experienced and 20 high-experienced viewers from Turkey were shown a set of short film clips containing these three kinds of commonalities. Film clips conformed also to the principles of continuity editing. Analyses of viewers' spontaneous interpretations show that first-time viewers indeed are able to notice basic pictorial (object identity), causal (chains of activity), as well as conceptual (links between gaze direction and object attention) commonalities between shots due to their close relationship with everyday perception and cognition. However, first-time viewers' comprehension of the commonalities is to a large degree fragile, indicating the lack of a basic notion of what constitutes a film.