Making sense of abstract events: building event schemas

Towards an ecologically valid naturalistic cognitive neuroscience of memory and event cognition

The landscape of human memory and event cognition research has witnessed a transformative journey toward the use of naturalistic contexts and tasks. In this review, we track this progression from abrupt, artificial stimuli used in extensively controlled laboratory experiments to more naturalistic tasks and stimuli that present a more faithful representation of the real world. We argue that in order to improve ecological validity, naturalistic study designs must consider the complexity of the cognitive phenomenon being studied. Then, we review the current state of "naturalistic" event segmentation studies and critically assess frequently employed movie stimuli. We evaluate recently developed tools like lifelogging and other extended reality technologies to help address the challenges we identified with existing naturalistic approaches. We conclude by offering some guidelines that can be used to design ecologically valid cognitive neuroscience studies of memory and event cognition.

Using network science to provide insights into the structure of event knowledge

The structure of event knowledge plays a critical role in prediction, reconstruction of memory for personal events, construction of possible future events, action, language usage, and social interactions. Despite numerous theoretical proposals such as scripts, schemas, and stories, the highly variable and rich nature of events and event knowledge have been formidable barriers to characterizing the structure of event knowledge in memory. We used network science to provide insights into the temporal structure of common events. Based on participants' production and ordering of the activities that make up events, we established empirical profiles for 80 common events to characterize the temporal structure of activities. We used the event networks to investigate multiple issues regarding the variability in the richness and complexity of people's knowledge of common events, including: the temporal structure of events; event prototypes that might emerge from learning across many experiential instances and be expressed by people; the degree to which scenes (communities) are present in various events; the degree to which people believe certain activities are central to an event; how centrality might be distributed across an event's activities; and similarities among events in terms of their content and their temporal structure. Thus, we provide novel insights into human event knowledge, and describe 18 predictions for future human studies.

Modeling human activity comprehension at human scale: prediction, segmentation, and categorization

Humans form sequences of event models-representations of the current situation-to predict how activity will unfold. Multiple mechanisms have been proposed for how the cognitive system determines when to segment the stream of behavior and switch from one active event model to another. Here, we constructed a computational model that learns knowledge about event classes (event schemas), by combining recurrent neural networks for short-term dynamics with Bayesian inference over event classes for event-to-event transitions. This architecture represents event schemas and uses them to construct a series of event models. This architecture was trained on one pass through 18 h of naturalistic human activities. Another 3.5 h of activities were used to test each variant for agreement with human segmentation and categorization. The architecture was able to learn to predict human activity, and it developed segmentation and categorization approaching human-like performance. We then compared two variants of this architecture designed to better emulate human event segmentation: one transitioned when the active event model produced high uncertainty in its prediction and the other transitioned when the active event model produced a large prediction error. The two variants learned to segment and categorize events, and the prediction uncertainty variant provided a somewhat closer match to human segmentation and categorization-despite being given no feedback about segmentation or categorization. These results suggest that event model transitioning based on prediction uncertainty or prediction error can reproduce two important features of human event comprehension.

Action Segmentation in the Brain: The Role of Object-Action Associations

Motion information has been argued to be central to the subjective segmentation of observed actions. Concerning object-directed actions, object-associated action information might as well inform efficient action segmentation and prediction. The present study compared the segmentation and neural processing of object manipulations and equivalent dough ball manipulations to elucidate the effect of object-action associations. Behavioral data corroborated that objective relational changes in the form of (un-)touchings of objects, hand, and ground represent meaningful anchor points in subjective action segmentation rendering them objective marks of meaningful event boundaries. As expected, segmentation behavior became even more systematic for the weakly informative dough. fMRI data were modeled by critical subjective, and computer-vision-derived objective event boundaries. Whole-brain as well as planned ROI analyses showed that object information had significant effects on how the brain processes these boundaries. This was especially pronounced at untouchings, that is, events that announced the beginning of the upcoming action and might be the point where competing predictions are aligned with perceptual input to update the current action model. As expected, weak object-action associations at untouching events were accompanied by increased biological motion processing, whereas strong object-action associations came with an increased contextual associative information processing, as indicated by increased parahippocampal activity. Interestingly, anterior inferior parietal lobule activity increased for weak object-action associations at untouching events, presumably because of an unrestricted number of candidate actions for dough manipulation. Our findings offer new insights into the significance of objects for the segmentation of action.

People can reliably detect action changes and goal changes during naturalistic perception

As a part of ongoing perception, the human cognitive system segments others' activities into discrete episodes (event segmentation). Although prior research has shown that this process is likely related to changes in an actor's actions and goals, it has not yet been determined whether untrained observers can reliably identify action and goal changes as naturalistic activities unfold, or whether the changes they identify are tied to visual features of the activity (e.g., the beginnings and ends of object interactions). This study addressed these questions by examining untrained participants' identification of action changes, goal changes, and event boundaries while watching videos of everyday activities that were presented in both first-person and third-person perspectives. We found that untrained observers can identify goal changes and action changes consistently, and these changes are not explained by visual change and the onsets or offsets of contact with objects. Moreover, the action and goal changes identified by untrained observers were associated with event boundaries, even after accounting for objective visual features of the videos. These findings suggest that people can identify action and goal changes consistently and with high agreement, that they do so by using sensory information flexibly, and that the action and goal changes they identify may contribute to event segmentation.

Event Integration and Temporal Differentiation: How Hierarchical Knowledge Emerges in Hippocampal Subfields through Learning

Everyday life is composed of events organized by changes in contexts, with each event containing an unfolding sequence of occurrences. A major challenge facing our memory systems is how to integrate sequential occurrences within events while also maintaining their details and avoiding over-integration across different contexts. We asked if and how distinct hippocampal subfields come to hierarchically and, in parallel, represent both event context and subevent occurrences with learning. Female and male human participants viewed sequential events defined as sequences of objects superimposed on shared color frames while undergoing high-resolution fMRI. Importantly, these events were repeated to induce learning. Event segmentation, as indexed by increased reaction times at event boundaries, was observed in all repetitions. Temporal memory decisions were quicker for items from the same event compared to across different events, indicating that events shaped memory. With learning, hippocampal CA3 multivoxel activation patterns clustered to reflect the event context, with more clustering correlated with behavioral facilitation during event transitions. In contrast, in the dentate gyrus (DG), temporally proximal items that belonged to the same event became associated with more differentiated neural patterns. A computational model explained these results by dynamic inhibition in the DG. Additional similarity measures support the notion that CA3 clustered representations reflect shared voxel populations, while DG's distinct item representations reflect different voxel populations. These findings suggest an interplay between temporal differentiation in the DG and attractor dynamics in CA3. They advance our understanding of how knowledge is structured through integration and separation across time and context.

The role of visual and auditory information in social event segmentation

Humans organise their social worlds into social and nonsocial events. Social event segmentation refers to the ability to parse the environmental content into social and nonsocial events or units. Here, we investigated the role that perceptual information from visual and auditory modalities, in isolation and in conjunction, played in social event segmentation. Participants viewed a video clip depicting an interaction between two actors and marked the boundaries of social and nonsocial events. Depending on the condition, the clip at first contained only auditory or only visual information. Then, the clip was shown containing both auditory and visual information. Higher overall group consensus and response consistency in parsing the clip was found for social segmentation and when both auditory and visual information was available. Presenting the clip in the visual domain only benefitted group agreement in social segmentation while the inclusion of auditory information (under the audiovisual condition) also improved response consistency in nonsocial segmentation. Thus, social segmentation utilises information from the visual modality, with the auditory cues contributing under ambiguous or uncertain conditions and during segmentation of nonsocial content.

Knowledge-based intervention improves older adult recognition memory for novel activity, but not event segmentation or temporal order memory

Although episodic memory declines with age, older adults are often able to make use of relevant knowledge to support episodic memory. More specifically, prior knowledge may support the perception of meaningful events through the process of event segmentation. We sought to test whether increasing older adults' knowledge for novel activities (i.e., Tai chi, making gyozas) would improve segmentation and memory. We conducted an online, pre-registered intervention in which eighty older adults were recruited based on being novices in each of the targeted activities. Participants completed segmentation and memory tests before and after being randomly assigned to one of two interactive virtual workshops (learning how to practice Tai chi or make gyozas). Each workshop consisted of two one-hour sessions during which an expert provided information about the activity and demonstrated it in a step-by-step fashion. We found that the intervention led to increased learning and recognition memory for the trained activity; however, there were no significant improvements in segmentation behavior, free recall, or memory of sequential information. These findings indicate that either more knowledge training is necessary to affect segmentation, or that segmentation is guided by perceptual features in the environment rather than one's conceptual understanding of the activity.

Cueing natural event boundaries improves memory in people with post-traumatic stress disorder

People with post-traumatic stress disorder (PTSD) often report difficulty remembering information in their everyday lives. Recent findings suggest that such difficulties may be due to PTSD-related deficits in parsing ongoing activity into discrete events, a process called event segmentation. Here, we investigated the causal relationship between event segmentation and memory by cueing event boundaries and evaluating its effect on subsequent memory in people with PTSD. People with PTSD (n = 38) and trauma-matched controls (n = 36) watched and remembered videos of everyday activities that were either unedited, contained visual and auditory cues at event boundaries, or contained visual and auditory cues at event middles. PTSD symptom severity varied substantial within both the group with a PTSD diagnosis and the control group. Memory performance did not differ significantly between groups, but people with high symptoms of PTSD remembered fewer details from the videos than those with lower symptoms of PTSD. Both those with PTSD and controls remembered more information from the videos in the event boundary cue condition than the middle cue or unedited conditions. This finding has important implications for translational work focusing on addressing everyday memory complaints in people with PTSD.

Unsupervised Learning of Temporal Abstractions With Slot-Based Transformers

The discovery of reusable subroutines simplifies decision making and planning in complex reinforcement learning problems. Previous approaches propose to learn such temporal abstractions in an unsupervised fashion through observing state-action trajectories gathered from executing a policy. However, a current limitation is that they process each trajectory in an entirely sequential manner, which prevents them from revising earlier decisions about subroutine boundary points in light of new incoming information. In this work, we propose slot-based transformer for temporal abstraction (SloTTAr), a fully parallel approach that integrates sequence processing transformers with a slot attention module to discover subroutines in an unsupervised fashion while leveraging adaptive computation for learning about the number of such subroutines solely based on their empirical distribution. We demonstrate how SloTTAr is capable of outperforming strong baselines in terms of boundary point discovery, even for sequences containing variable amounts of subroutines, while being up to seven times faster to train on existing benchmarks.

Sources and destinations of misattributions in recall of instances of repeated events

Repeated experience of events promotes schema formation. Later activation of the schema facilitates recall of the general structure of the events, whereas attribution of details to instances requires systematic decision-making based on detail characteristics. For repeated events, source monitoring may be less effective due to the similarity and interference of details across instances and consequently result in source attribution errors. To date, researchers have examined aggregated misattributions across instances and have found that misattributions are more frequent in the middle than in the boundary instances. In this study, we investigated the trajectories of misattributions using data from six studies (N = 633), where participants recalled repeated interactive marketing-themed events (Study 1), mock-crime filmed events (Study 2), stories (Study 3), and categorized word lists (Studies 4-6). The patterns confirmed the expected primacy and recency effects, showing fewer misattributions from and to the boundary instances relative to the middle instances. In addition, the patterns indicated proximity effects: Confusions more frequently occurred across adjacent instances and gradually decreased for instances that were further apart from the source. Our findings suggest that detail characteristics that form the basis of source attribution decisions provide information about the relative position of instances in repeated events, where the boundary instances serve as anchors, and where confusion relatively easily occurs across neighbouring instances. In line with context-based models of memory, our findings indicate that a higher-level organization of repeated events that emerges at encoding guides retrieval and source monitoring decisions.

The effects of domain knowledge and event structure on event processing

Research suggests that domain knowledge facilitates memory for domain-specific information through two mechanisms: differentiation, which involves the ability to identify meaningful, fine-grained details within a sequence, and unitization, which involves binding individual components from a sequence into functional wholes. This study investigated the extent to which individuals engaged in differentiation and unitization when parsing continuous events into discrete, meaningful units (i.e., event segmentation) and recalling them. Participants watched and segmented basketball videos. They then rewatched the videos and provided descriptions afterward. Videos were coded for the presence of higher order goals (A2 actions) and the individual sub-actions that comprised them (A1 actions). Results suggested that event segmentation behavior for participants with less knowledge was more aligned with changes in basic actions (A1 actions) than for participants with greater knowledge. When describing events, participants with greater knowledge were more likely than participants with less knowledge to use statements that reflected unitization.

Measuring event segmentation: An investigation into the stability of event boundary agreement across groups

People spontaneously divide everyday experience into smaller units (event segmentation). To measure event segmentation, studies typically ask participants to explicitly mark the boundaries between events as they watch a movie (segmentation task). Their data may then be used to infer how others are likely to segment the same movie. However, significant variability in performance across individuals could undermine the ability to generalize across groups, especially as more research moves online. To address this concern, we used several widely employed and novel measures to quantify segmentation agreement across different sized groups (n = 2-32) using data collected on different platforms and movie types (in-lab & commercial film vs. online & everyday activities). All measures captured nonrandom and video-specific boundaries, but with notable between-sample variability. Samples of 6-18 participants were required to reliably detect video-driven segmentation behavior within a single sample. As sample size increased, agreement values improved and eventually stabilized at comparable sample sizes for in-lab & commercial film data and online & everyday activities data. Stabilization occurred at smaller sample sizes when measures reflected (1) agreement between two groups versus agreement between an individual and group, and (2) boundary identification between small (fine-grained) rather than large (coarse-grained) events. These analyses inform the tailoring of sample sizes based on the comparison of interest, materials, and data collection platform. In addition to demonstrating the reliability of online and in-lab segmentation performance at moderate sample sizes, this study supports the use of segmentation data to infer when events are likely to be segmented.

Neural event segmentation of continuous experience in human infants

How infants experience the world is fundamental to understanding their cognition and development. A key principle of adult experience is that, despite receiving continuous sensory input, we perceive this input as discrete events. Here we investigate such event segmentation in infants and how it differs from adults. Research on event cognition in infants often uses simplified tasks in which (adult) experimenters help solve the segmentation problem for infants by defining event boundaries or presenting discrete actions/vignettes. This presupposes which events are experienced by infants and leaves open questions about the principles governing infant segmentation. We take a different, data-driven approach by studying infant event segmentation of continuous input. We collected whole-brain functional MRI (fMRI) data from awake infants (and adults, for comparison) watching a cartoon and used a hidden Markov model to identify event states in the brain. We quantified the existence, timescale, and organization of multiple-event representations across brain regions. The adult brain exhibited a known hierarchical gradient of event timescales, from shorter events in early visual regions to longer events in later visual and associative regions. In contrast, the infant brain represented only longer events, even in early visual regions, with no timescale hierarchy. The boundaries defining these infant events only partially overlapped with boundaries defined from adult brain activity and behavioral judgments. These findings suggest that events are organized differently in infants, with longer timescales and more stable neural patterns, even in sensory regions. This may indicate greater temporal integration and reduced temporal precision during dynamic, naturalistic perception.

Ideomotor learning: Time to generalize a longstanding principle

The ideomotor principle holds that anticipating the sensory consequences of a movement triggers an associated motor response. Even though this framework dates back to the 19th century, it continues to lie at the heart of many contemporary approaches to human action control. Here we specifically focus on the ideomotor learning mechanism that has to precede action initiation via effect anticipation. Traditional approaches to this learning mechanism focused on establishing novel action-effect (or response-effect) associations. Here we apply the theoretical concept of common coding for action and perception to argue that the same learning principle should result in response-response and stimulus-stimulus associations just as well. Generalizing ideomotor learning in such a way results in a powerful and general framework of ideomotor action control, and it allows for integrating the two seemingly separate fields of ideomotor approaches and hierarchical learning.

Episodic representation: A mental models account

This paper offers a modeling account of episodic representation. I argue that the episodic system constructs mental models: representations that preserve the spatiotemporal structure of represented domains. In prototypical cases, these domains are events: occurrences taken by subjects to have characteristic structures, dynamics and relatively determinate beginnings and ends. Due to their simplicity and manipulability, mental event models can be used in a variety of cognitive contexts: in remembering the personal past, but also in future-oriented and counterfactual imagination. As structural representations, they allow surrogative reasoning, supporting inferences about their constituents which can be used in reasoning about the represented events.

Semantic knowledge attenuates age-related differences in event segmentation and episodic memory

While semantic and episodic memory may be distinct memory systems, their interdependence is substantial. For instance, decades of work have shown that semantic knowledge facilitates episodic memory. Here, we aim to clarify this interactive relationship by determining whether semantic knowledge facilitates the acquisition of new episodic memories, in part, by influencing an encoding mechanism, event segmentation. In the current study, we evaluated the extent to which semantic knowledge shapes how people segment ongoing activity and how such knowledge-related benefits in segmentation affect episodic memory performance. To investigate these effects, we combined data across three studies that had young and older adults segment and remember videos of everyday activities that were either familiar or unfamiliar to their age group. We found age-related differences in event-segmentation ability and memory performance, but only when older adults lacked semantic knowledge. Most importantly, when they had access to relevant semantic knowledge, older adults segmented and remembered information similar to young adults. Our findings indicate that older adults can use semantic knowledge to effectively encode and retrieve everyday information. These effects suggest that future interventions can leverage older adults' intact semantic knowledge to attenuate age-related deficits in event segmentation and episodic long-term memory.

Events structure information accessibility less in children than adults

To manage the onslaught of continuously unfolding information in our complex environments, we adults are known to carve up our continuous experience into meaningful events, a process referred to as event segmentation. This segmentation directly shapes how our everyday experiences are construed: content experienced within an event is held mentally in an accessible state, which is then dropped after an event boundary. The greater accessibility of event-specific information has been shown to influence-at its most basic level-how information is processed and remembered. However, it is as yet unknown if accessibility is similarly influenced by event boundaries in children, who are still developing the working memory capacity and semantic knowledge thought to support event segmentation. Here, we tested seven- to nine-year-old children's and adults' recognition of objects experienced either within or across event boundaries of two cartoons. We found that children and adults were both more accurate and faster to correctly recognize objects that last occurred within events versus across event boundaries. We, however, additionally observed an interaction such that children's access to recent experience was less influenced by event boundaries than adults'. Thus, while the spontaneous segmentation of complex events emerges by middle childhood, event structure shapes the active contents of children's minds less reliably than adults'.

Semantic memory: A review of methods, models, and current challenges

Adult semantic memory has been traditionally conceptualized as a relatively static memory system that consists of knowledge about the world, concepts, and symbols. Considerable work in the past few decades has challenged this static view of semantic memory, and instead proposed a more fluid and flexible system that is sensitive to context, task demands, and perceptual and sensorimotor information from the environment. This paper (1) reviews traditional and modern computational models of semantic memory, within the umbrella of network (free association-based), feature (property generation norms-based), and distributional semantic (natural language corpora-based) models, (2) discusses the contribution of these models to important debates in the literature regarding knowledge representation (localist vs. distributed representations) and learning (error-free/Hebbian learning vs. error-driven/predictive learning), and (3) evaluates how modern computational models (neural network, retrieval-based, and topic models) are revisiting the traditional "static" conceptualization of semantic memory and tackling important challenges in semantic modeling such as addressing temporal, contextual, and attentional influences, as well as incorporating grounding and compositionality into semantic representations. The review also identifies new challenges regarding the abundance and availability of data, the generalization of semantic models to other languages, and the role of social interaction and collaboration in language learning and development. The concluding section advocates the need for integrating representational accounts of semantic memory with process-based accounts of cognitive behavior, as well as the need for explicit comparisons of computational models to human baselines in semantic tasks to adequately assess their psychological plausibility as models of human semantic memory.

Influences of domain knowledge on segmentation and memory

Much research has shown that experts possess superior memory in their domain of expertise. This memory benefit has been proposed to be the result of various encoding mechanisms, such as chunking and differentiation. Another potential encoding mechanism that is associated with memory is event segmentation, which is the process by which people parse continuous information into meaningful, discrete units. Previous research has found evidence that segmentation, to some extent, is affected by top-down processing. To date, few studies have investigated the influence of expertise on segmentation, and questions about expertise, segmentation ability, and their impact on memory remain. The goal of the current study was to investigate the influence of expertise on segmentation and memory ability for two different domains: basketball and Overwatch. Participants with high and low knowledge for basketball and with low knowledge for Overwatch viewed and segmented videos at coarse and fine grains, then completed memory tests. Differences in segmentation ability and memory were present between experts and control novices, specifically for the basketball videos; however, experts' segmentation only predicted memory for activities for which knowledge was lacking. Overall, this research suggests that experts' superior memory is not due to their segmentation ability and contributes to a growing body of literature showing evidence supporting conceptual effects on segmentation.

Culture influences how people divide continuous sensory experience into events

Everyday experience is divided into meaningful events as a part of human perception. Current accounts of this process, known as event segmentation, focus on how characteristics of the experience (e.g., situation changes) influence segmentation. However, characteristics of the viewers themselves have been largely neglected. We test whether one such viewer characteristic, their cultural background, impacts online event segmentation. Culture could impact event segmentation (1) by emphasizing different aspects of experiences as being important for comprehension, memory, and communication, and (2) by providing different exemplars of how everyday activities are performed, which objects are likely to be used, and how scenes are laid out. Indian and US viewers (N = 152) identified events in everyday activities (e.g., making coffee) recorded in Indian and US settings. Consistent with their cultural preference for analytical processing, US viewers segmented the activities into more events than did Indian viewers. Furthermore, event boundaries identified by US viewers were more strongly associated with visual changes, whereas boundaries identified by Indian viewers were more strongly associated with goal changes. There was no evidence that familiarity with an activity impacted segmentation. Thus, culture impacts event perception by altering the types of information people prioritize when dividing experience into meaningful events.

The Witness-Aimed First Account (WAFA): A new technique for interviewing autistic witnesses and victims

LAY ABSTRACT

Autistic people may be more likely to be interviewed by police as a victim/witness, yet they experience social communication difficulties alongside specific memory difficulties that can impact their ability to recall information from memory. Police interviewing techniques do not take account of these differences, and so are often ineffective. We developed a new technique for interviewing autistic witnesses, referred to a Witness-Aimed First Account, which was designed to better support differences in the way that autistic witnesses process information in memory. The Witness-Aimed First Account technique encourages witnesses to first segment the witnessed event into discrete, parameter-bound event topics, which are then displayed on post-it notes while the witness goes onto freely recall as much information as they can from within each parameter-bound topic in turn. Since witnessed events are rarely cohesive stories with a logical chain of events, we also explored autistic and non-autistic witnesses' recall when the events were witnessed in a random (nonsensical) order. Thirty-three autistic and 30 typically developing participants were interviewed about their memory for two videos depicting criminal events. Clip segments of one video were 'scrambled', disrupting the event's narrative structure; the other video was watched intact. Although both autistic and non-autistic witnesses recalled fewer details with less accuracy from the scrambled video, Witness-Aimed First Account interviews resulted in more detailed and accurate recall from both autistic and non-autistic witnesses, for both scrambled and unscrambled videos. The Witness-Aimed First Account technique may be a useful tool to improve witnesses' accounts within a legally appropriate, non-leading framework.

Hippocampal neurons represent events as transferable units of experience

The brain codes continuous spatial, temporal and sensory changes in daily experience. Recent studies suggest that the brain also tracks experience as segmented subdivisions (events), but the neural basis for encoding events remains unclear. Here, we designed a maze for mice, composed of four materially indistinguishable lap events, and identify hippocampal CA1 neurons whose activity are modulated not only by spatial location but also lap number. These 'event-specific rate remapping' (ESR) cells remain lap-specific even when the maze length is unpredictably altered within trials, which suggests that ESR cells treat lap events as fundamental units. The activity pattern of ESR cells is reused to represent lap events when the maze geometry is altered from square to circle, which suggests that it helps transfer knowledge between experiences. ESR activity is separately manipulable from spatial activity, and may therefore constitute an independent hippocampal code: an 'event code' dedicated to organizing experience by events as discrete and transferable units.

Behavioral evidence for memory replay of video episodes in the macaque

Humans recall the past by replaying fragments of events temporally. Here, we demonstrate a similar effect in macaques. We trained six rhesus monkeys with a temporal-order judgement (TOJ) task and collected 5000 TOJ trials. In each trial, the monkeys watched a naturalistic video of about 10 s comprising two across-context clips, and after a 2 s delay, performed TOJ between two frames from the video. The data are suggestive of a non-linear, time-compressed forward memory replay mechanism in the macaque. In contrast with humans, such compression of replay is, however, not sophisticated enough to allow these monkeys to skip over irrelevant information by compressing the encoded video globally. We also reveal that the monkeys detect event contextual boundaries, and that such detection facilitates recall by increasing the rate of information accumulation. Demonstration of a time-compressed, forward replay-like pattern in the macaque provides insights into the evolution of episodic memory in our lineage.

Differential effects of knowledge and aging on the encoding and retrieval of everyday activities

We deconstruct continuous streams of action into smaller, meaningful events. Research has shown that the ability to segment continuous activity into such events and remember their contents declines with age; however, knowledge improves with age. We investigated how young and older adults use knowledge to more efficiently encode and later remember information from everyday events by having participants view a series of self-paced slideshows depicting everyday activities. For some activities, older adults produce more normative scripts than do young adults (older adult activities) and for other activities, young adults produce more normative scripts than do older adults (young adult activities). Overall, participants viewed event boundaries longer than within events (i.e., the event boundary advantage) replicating prior research (e.g., Hard, Recchia, & Tversky, 2011). Importantly, older adults demonstrated the boundary advantage for the older adult activities but not the young adult activities, and they also had better recognition memory for the older adult activities than the young adult activities. We also found that the magnitude of a participant's boundary advantage was associated with better memory, but only for the less knowledgeable activities. Results indicate that older adults use their intact knowledge to better encode and remember everyday activities, but that knowledge and event segmentation may have independent influences on event memory.

Event Perception and Memory

Events make up much of our lived experience, and the perceptual mechanisms that represent events in experience have pervasive effects on action control, language use, and remembering. Event representations in both perception and memory have rich internal structure and connections one to another, and both are heavily informed by knowledge accumulated from previous experiences. Event perception and memory have been identified with specific computational and neural mechanisms, which show protracted development in childhood and are affected by language use, expertise, and brain disorders and injuries. Current theoretical approaches focus on the mechanisms by which events are segmented from ongoing experience, and emphasize the common coding of events for perception, action, and memory. Abetted by developments in eye-tracking, neuroimaging, and computer science, research on event perception and memory is moving from small-scale laboratory analogs to the complexity of events in the wild.

Taking it out of context: The role of contextual coherence during social event segmentation

Social event segmentation, or parsing of the ongoing dynamic content into discrete social events, is thought to represent a mechanism that supports the expert human ability to navigate complex social environments. Here, we examined whether this ability is influenced by the temporal coherence of the context and by different sources of perceptual information. To do so, we created two video clips, one in which several situations unfolded in a contextually consistent manner, and the other in which the order of these situations was scrambled using a random sequence. Participants viewed each clip and were asked to mark social and nonsocial events in counterbalanced blocks of trials. We analyzed key-press behaviour as well as visual and auditory signals within the clips. Results showed that participants agreed on similar social and nonsocial events regardless of context availability, with greater agreement for social relative to nonsocial events. Context, however, modulated the reliance on sources of perceptual information, such that visual and auditory information was used differently when context was unavailable. Together, these data show that contextual coherence does not determine social event segmentation but serves a modulatory role in perceivers' reliance on perceptual sources of information when identifying events in complex social environments.

Linguistic Information in Auditory Dynamic Events Contributes to the Detection of Fine, Not Coarse Event Boundaries

Human observers (comprehenders) segment dynamic information into discrete events. That is, although there is continuous sensory information, comprehenders perceive boundaries between two meaningful units of information. In narrative comprehension, comprehenders use linguistic, non-linguistic , and physical cues for this event boundary perception. Yet, it is an open question - both from a theoretical and an empirical perspective - how linguistic and non-linguistic cues contribute to this process. The current study explores how linguistic cues contribute to the participants' ability to segment continuous auditory information into discrete, hierarchically structured events. Native speakers of German and non-native speakers, who neither spoke nor understood German, segmented a German audio drama into coarse and fine events. Whereas native participants could make use of linguistic, non-linguistic, and physical cues for segmentation, non-native participants could only use non-linguistic and physical cues. We analyzed segmentation behavior in terms of the ability to identify coarse and fine event boundaries and the resulting hierarchical structure. Non-native listeners identified almost identical coarse event boundaries as native listeners, but missed some of the fine event boundaries identified by the native listeners. Interestingly, hierarchical event perception (as measured by hierarchical alignment and enclosure) was comparable for native and non-native participants. In summary, linguistic cues contributed particularly to the identification of certain fine event boundaries. The results are discussed with regard to the current theories of event cognition.

Does semantic knowledge influence event segmentation and recall of text?

Knowledge benefits episodic memory, particularly when provided before encoding (Anderson & Pichert in Journal of Verbal Learning and Verbal Behavior, 17(1), 1-12, 1978; Bransford & Johnson in Journal of Verbal Learning and Verbal Behavior, 11(6), 717-726, 1972). These benefits can occur through several encoding mechanisms, one of which may be event segmentation. Event segmentation is one's ability to parse information into meaningful units as an activity unfolds. The current experiment evaluated whether two top-down manipulations-providing context or perspective taking-influence the segmentation and memory of text. For the ambiguous texts in Experiment 1, half the participants received context in the form of a title, whereas the other half received no context. For the text in Experiment 2, half the participants read from the perspective of a burglar and the other half read from the perspective of a home buyer. In both experiments, participants read the passages, recalled the information, and then segmented the passages into meaningful units. Consistent with previous findings, participants who received context recalled more information compared with those who received no context, and participants in one perspective were more likely to recall information relevant to their perspective. Most importantly, we found that context and perspective facilitated more normative segmentation; however, the differences were small and suggest that effects of top-down processing on the segmentation of text may be modest at best. Thus, event segmentation processes that operate during text comprehension are influenced by semantic knowledge but may be more heavily driven by other factors (e.g., perceptual cues).

Dynamic prediction during perception of everyday events

The ability to predict what is going to happen in the near future is integral for daily functioning. Previous research suggests that predictability varies over time, with increases in prediction error at those moments that people perceive as boundaries between meaningful events. These moments also tend to be points of rapid change in the environment. Eye tracking provides a method for noninterruptive measurement of prediction as participants watch a movie of an actor performing a series of actions. In two studies, we used eye tracking to study the time course of prediction around event boundaries. In both studies, viewers looked at objects that were about to be touched by the actor shortly before the objects were contacted, demonstrating predictive looking. However, this behavior was modulated by event boundaries: looks to to-be-contacted objects near event boundaries were less likely to be early and more likely to be late compared to looks to objects contacted within events. This result is consistent with theories proposing that event segmentation results from transient increases in prediction error.

The role of perspective in event segmentation

People divide their ongoing experience into meaningful events. This process, event segmentation, is strongly associated with visual input: when visual features change, people are more likely to segment. However, the nature of this relationship is unclear. Segmentation could be bound to specific visual features, such as actor posture. Or, it could be based on changes in the activity that are correlated with visual features. This study distinguished between these two possibilities by examining whether segmentation varies across first- and third-person perspectives. In two experiments, observers identified meaningful events in videos of actors performing everyday activities, such as eating breakfast or doing laundry. Each activity was simultaneously recorded from a first-person perspective and a third-person perspective. These videos presented identical activities but differed in their visual features. If segmentation is tightly bound to visual features then observers should identify different events in first- and third-person videos. In addition, the relationship between segmentation and visual features should remain unchanged. Neither prediction was supported. Though participants sometimes identified more events in first-person videos, the events they identified were mostly indistinguishable from those identified for third-person videos. In addition, the relationship between the video's visual features and segmentation changed across perspectives, further demonstrating a partial dissociation between segmentation and visual input. Event segmentation appears to be robust to large variations in sensory information as long as the content remains the same. Segmentation mechanisms appear to flexibly use sensory information to identify the structure of the underlying activity.

Constructing Experience: Event Models from Perception to Action

Mental representations of everyday experience are rich, structured, and multimodal. In this article we consider the adaptive pressures that led to human construction of such representations, arguing that structured event representations enable cognitive systems to more effectively predict the trajectory of naturalistic everyday activity. We propose an account of how cortical systems and the hippocampus (HPC) interact to construct, maintain, and update event representations. This analysis throws light on recent research on story comprehension, event segmentation, episodic memory, and action planning. It also suggests how the growing science base can be deployed to diagnose impairments in event perception and memory, and to improve memory for everyday events.

Why Should We All Be Cultural Psychologists? Lessons From the Study of Social Cognition

I call the attention of psychologists to the pivotal role of cultural psychology in extending and enriching research programs. I argue that it is not enough to simply acknowledge the importance of culture and urge psychologists to practice cultural psychology in their research. I deconstruct five assumptions about cultural psychology that seriously undermine its contribution to the building of a true psychological science, including that cultural psychology (a) is only about finding group differences, (b) does not appertain to group similarities, (c) concerns only group-level analysis, (d) is irrelevant to basic psychological processes, and (e) is used only to confirm the generalizability of theories. I discuss how cultural psychology can provide unique insights into psychological processes and further equip researchers with additional tools to understand human behavior. Drawing lessons from the 20 years of cultural research that my colleagues and I have done on the development of social cognition, including autobiographical memory, future thinking, self, and emotion knowledge, I demonstrate that incorporating cultural psychology into research programs is not only necessary but also feasible.

Not so secret agents: Event-related potentials to semantic roles in visual event comprehension

Research across domains has suggested that agents, the doers of actions, have a processing advantage over patients, the receivers of actions. We hypothesized that agents as "event builders" for discrete actions (e.g., throwing a ball, punching) build on cues embedded in their preparatory postures (e.g., reaching back an arm to throw or punch) that lead to (predictable) culminating actions, and that these cues afford frontloading of event structure processing. To test this hypothesis, we compared event-related brain potentials (ERPs) to averbal comic panels depicting preparatory agents (ex. reaching back an arm to punch) that cued specific actions with those to non-preparatory agents (ex. arm to the side) and patients that did not cue any specific actions. We also compared subsequent completed action panels (ex. agent punching patient) across conditions, where we expected an inverse pattern of ERPs indexing the differential costs of processing completed actions asa function of preparatory cues. Preparatory agents evoked a greater frontal positivity (600-900ms) relative to non-preparatory agents and patients, while subsequent completed actions panels following non-preparatory agents elicited a smaller frontal positivity (600-900ms). These results suggest that preparatory (vs. non-) postures may differentially impact the processing of agents and subsequent actions in real time.

Fandom Biases Retrospective Judgments Not Perception

Attitudes and motivations have been shown to affect the processing of visual input, indicating that observers may see a given situation each literally in a different way. Yet, in real-life, processing information in an unbiased manner is considered to be of high adaptive value. Attitudinal and motivational effects were found for attention, characterization, categorization, and memory. On the other hand, for dynamic real-life events, visual processing has been found to be highly synchronous among viewers. Thus, while in a seminal study fandom as a particularly strong case of attitudes did bias judgments of a sports event, it left the question open whether attitudes do bias prior processing stages. Here, we investigated influences of fandom during the live TV broadcasting of the 2013 UEFA-Champions-League Final regarding attention, event segmentation, immediate and delayed cued recall, as well as affect, memory confidence, and retrospective judgments. Even though we replicated biased retrospective judgments, we found that eye-movements, event segmentation, and cued recall were largely similar across both groups of fans. Our findings demonstrate that, while highly involving sports events are interpreted in a fan dependent way, at initial stages they are processed in an unbiased manner.

The Seduction Script: Psychological and Cultural Norms of Interpersonal Approaches As Markers for Sexual Aggression and Abuse

Hardly any subjects enjoy greater – public or private – interest than the art of flirtation and seduction. However, interpersonal approach behavior not only paves the way for sexual interaction and reproduction, but it simultaneously integrates non-sexual psychobiological and cultural standards regarding consensus and social norms. In the present paper, we use script theory, a concept that extends across psychological and cultural science, to assess behavioral options during interpersonal approaches. Specifically, we argue that approaches follow scripted event sequences that entail ambivalence as an essential communicative element. On the one hand, ambivalence may facilitate interpersonal approaches by maintaining and provoking situational uncertainty, so that the outcome of an action – even after several approaches and dates – remains ambiguous. On the other hand, ambivalence may increase the risk for sexual aggression or abuse, depending on the individual’s abilities, the circumstances, and the intentions of the interacting partners. Recognizing latent sequences of sexually aggressive behavior, in terms of their rigid structure and behavioral options, may thus enable individuals to use resources efficiently, avoid danger, and extricate themselves from assault situations. We conclude that interdisciplinary script knowledge about ambivalence as a core component of the seduction script may be helpful for counteracting subtly aggressive intentions and preventing sexual abuse. We discuss this with regard to the nature-nurture debate as well as phylogenetic and ontogenetic aspects of interpersonal approach behavior and its medial implementation.

Toward a Unified Sub-symbolic Computational Theory of Cognition

This paper proposes how various disciplinary theories of cognition may be combined into a unifying, sub-symbolic, computational theory of cognition. The following theories are considered for integration: psychological theories, including the theory of event coding, event segmentation theory, the theory of anticipatory behavioral control, and concept development; artificial intelligence and machine learning theories, including reinforcement learning and generative artificial neural networks; and theories from theoretical and computational neuroscience, including predictive coding and free energy-based inference. In the light of such a potential unification, it is discussed how abstract cognitive, conceptualized knowledge and understanding may be learned from actively gathered sensorimotor experiences. The unification rests on the free energy-based inference principle, which essentially implies that the brain builds a predictive, generative model of its environment. Neural activity-oriented inference causes the continuous adaptation of the currently active predictive encodings. Neural structure-oriented inference causes the longer term adaptation of the developing generative model as a whole. Finally, active inference strives for maintaining internal homeostasis, causing goal-directed motor behavior. To learn abstract, hierarchical encodings, however, it is proposed that free energy-based inference needs to be enhanced with structural priors, which bias cognitive development toward the formation of particular, behaviorally suitable encoding structures. As a result, it is hypothesized how abstract concepts can develop from, and thus how they are structured by and grounded in, sensorimotor experiences. Moreover, it is sketched-out how symbol-like thought can be generated by a temporarily active set of predictive encodings, which constitute a distributed neural attractor in the form of an interactive free-energy minimum. The activated, interactive network attractor essentially characterizes the semantics of a concept or a concept composition, such as an actual or imagined situation in our environment. Temporal successions of attractors then encode unfolding semantics, which may be generated by a behavioral or mental interaction with an actual or imagined situation in our environment. Implications, further predictions, possible verification, and falsifications, as well as potential enhancements into a fully spelled-out unified theory of cognition are discussed at the end of the paper.

Event boundaries and memory improvement

The structure of events can influence later memory for information that is embedded in them, with evidence indicating that event boundaries can both impair and enhance memory. The current study explored whether the presence of event boundaries during encoding can structure information to improve memory. In Experiment 1, memory for a list of words was tested in which event structure was manipulated by having participants walk through a doorway, or not, halfway through the word list. In Experiment 2, memory for lists of words was tested in which event structure was manipulated using computer windows. Finally, in Experiments 3 and 4, event structure was manipulated by having event shifts described in narrative texts. The consistent finding across all of these methods and materials was that memory was better when the information was distributed across two events rather than combined into a single event. Moreover, Experiment 4 demonstrated that increasing the number of event boundaries from one to two increased the memory benefit. These results are interpreted in the context of the Event Horizon Model of event cognition.

What Would Jaws Do? The Tyranny of Film and the Relationship between Gaze and Higher-Level Narrative Film Comprehension

What is the relationship between film viewers' eye movements and their film comprehension? Typical Hollywood movies induce strong attentional synchrony-most viewers look at the same things at the same time. Thus, we asked whether film viewers' eye movements would differ based on their understanding-the mental model hypothesis-or whether any such differences would be overwhelmed by viewers' attentional synchrony-the tyranny of film hypothesis. To investigate this question, we manipulated the presence/absence of prior film context and measured resulting differences in film comprehension and eye movements. Viewers watched a 12-second James Bond movie clip, ending just as a critical predictive inference should be drawn that Bond's nemesis, "Jaws," would fall from the sky onto a circus tent. The No-context condition saw only the 12-second clip, but the Context condition also saw the preceding 2.5 minutes of the movie before seeing the critical 12-second portion. Importantly, the Context condition viewers were more likely to draw the critical inference and were more likely to perceive coherence across the entire 6 shot sequence (as shown by event segmentation), indicating greater comprehension. Viewers' eye movements showed strong attentional synchrony in both conditions as compared to a chance level baseline, but smaller differences between conditions. Specifically, the Context condition viewers showed slightly, but significantly, greater attentional synchrony and lower cognitive load (as shown by fixation probability) during the critical first circus tent shot. Thus, overall, the results were more consistent with the tyranny of film hypothesis than the mental model hypothesis. These results suggest the need for a theory that encompasses processes from the perception to the comprehension of film.

Effects of penetrating traumatic brain injury on event segmentation and memory

Penetrating traumatic brain injury (pTBI) is associated with deficits in cognitive tasks including comprehension and memory, and also with impairments in tasks of daily living. In naturalistic settings, one important component of cognitive task performance is event segmentation, the ability to parse the ongoing stream of behavior into meaningful units. Event segmentation ability is associated with memory performance and with action control, but is not well assessed by standard neuropsychological assessments or laboratory tasks. Here, we measured event segmentation and memory in a sample of 123 male military veterans aged 59-81 who had suffered a traumatic brain injury as young men, and 34 demographically similar controls. Participants watched movies of everyday activities and segmented them to identify fine-grained or coarse-grained events, and then completed tests of recognition memory for pictures from the movies and of memory for the temporal order of actions in the movies. Lesion location and volume were assessed with computed tomography (CT) imaging. Patients with traumatic brain injury were impaired on event segmentation. Those with larger lesions had larger impairments for fine segmentation and also impairments for both memory measures. Further, the degree of memory impairment was statistically mediated by the degree of event segmentation impairment. There was some evidence that lesions to the ventromedial prefrontal cortex (vmPFC) selectively impaired coarse segmentation; however, lesions outside of a priori regions of interest also were associated with impaired segmentation. One possibility is that the effect of vmPFC damage reflects the role of prefrontal event knowledge representations in ongoing comprehension. These results suggest that assessment of naturalistic event comprehension can be a valuable component of cognitive assessment in cases of traumatic brain injury, and that interventions aimed at event segmentation could be clinically helpful.

Episodes, events, and models

We describe a novel computational theory of how individuals segment perceptual information into representations of events. The theory is inspired by recent findings in the cognitive science and cognitive neuroscience of event segmentation. In line with recent theories, it holds that online event segmentation is automatic, and that event segmentation yields mental simulations of events. But it posits two novel principles as well: first, discrete episodic markers track perceptual and conceptual changes, and can be retrieved to construct event models. Second, the process of retrieving and reconstructing those episodic markers is constrained and prioritized. We describe a computational implementation of the theory, as well as a robotic extension of the theory that demonstrates the processes of online event segmentation and event model construction. The theory is the first unified computational account of event segmentation and temporal inference. We conclude by demonstrating now neuroimaging data can constrain and inspire the construction of process-level theories of human reasoning.

Exploring Temporal Progression of Events Using Eye Tracking

This study investigates the representation of the temporal progression of events by means of the causal change in a patient. Subjects were asked to verify the relationship between adjectives denoting a source and resulting feature of a patient. The features were presented either chronologically or inversely to a primed event context given by a verb (to cut: long-short vs. short-long). Effects on response time and on eye movement data show that the relationship between features presented chronologically is verified more easily than that between features presented inversely. Post hoc, however, we found that the effects of temporal order occurred only when subjects read the features more than once. Then, the relationship between the features is matched with the causal change implied by the event context (contextual strategy). When subjects read the features only once, subjects respond to the relationship between the features without taking into account the event context.