The role of perspective in event segmentation

People can reliably detect action changes and goal changes during naturalistic perception

As a part of ongoing perception, the human cognitive system segments others' activities into discrete episodes (event segmentation). Although prior research has shown that this process is likely related to changes in an actor's actions and goals, it has not yet been determined whether untrained observers can reliably identify action and goal changes as naturalistic activities unfold, or whether the changes they identify are tied to visual features of the activity (e.g., the beginnings and ends of object interactions). This study addressed these questions by examining untrained participants' identification of action changes, goal changes, and event boundaries while watching videos of everyday activities that were presented in both first-person and third-person perspectives. We found that untrained observers can identify goal changes and action changes consistently, and these changes are not explained by visual change and the onsets or offsets of contact with objects. Moreover, the action and goal changes identified by untrained observers were associated with event boundaries, even after accounting for objective visual features of the videos. These findings suggest that people can identify action and goal changes consistently and with high agreement, that they do so by using sensory information flexibly, and that the action and goal changes they identify may contribute to event segmentation.

Prediction error and event segmentation in episodic memory

Organizing the continuous flow of experiences into meaningful events is a crucial prerequisite for episodic memory. Prediction error and event segmentation both play important roles in supporting the genesis of meaningful mnemonic representations of events. We review theoretical contributions discussing the relationship between prediction error and event segmentation, as well as literature on episodic memory related to prediction error and event segmentation. We discuss the extent of overlap of mechanisms underlying memory emergence through prediction error and event segmentation, with a specific focus on attention and working memory. Finally, we identify areas in research that are currently developing and suggest future directions. We provide an overview of mechanisms underlying memory formation through predictions, violations of predictions, and event segmentation.

More than a moment: What does it mean to call something an 'event'?

Experiences are stored in the mind as discrete mental units, or 'events,' which influence-and are influenced by-attention, learning, and memory. In this way, the notion of an 'event' is foundational to cognitive science. However, despite tremendous progress in understanding the behavioral and neural signatures of events, there is no agreed-upon definition of an event. Here, we discuss different theoretical frameworks of event perception and memory, noting what they can and cannot account for in the literature. We then highlight key aspects of events that we believe should be accounted for in theories of event processing--in particular, we argue that the structure and substance of events should be better reflected in our theories and paradigms. Finally, we discuss empirical gaps in the event cognition literature and what the future of event cognition research may look like.

Bayesian Surprise Predicts Human Event Segmentation in Story Listening

Event segmentation theory posits that people segment continuous experience into discrete events and that event boundaries occur when there are large transient increases in prediction error. Here, we set out to test this theory in the context of story listening, by using a deep learning language model (GPT-2) to compute the predicted probability distribution of the next word, at each point in the story. For three stories, we used the probability distributions generated by GPT-2 to compute the time series of prediction error. We also asked participants to listen to these stories while marking event boundaries. We used regression models to relate the GPT-2 measures to the human segmentation data. We found that event boundaries are associated with transient increases in Bayesian surprise but not with a simpler measure of prediction error (surprisal) that tracks, for each word in the story, how strongly that word was predicted at the previous time point. These results support the hypothesis that prediction error serves as a control mechanism governing event segmentation and point to important differences between operational definitions of prediction error.

The multi-angle extended three-dimensional activities (META) stimulus set: A tool for studying event cognition

To study complex human activity and how it is perceived and remembered, it is valuable to have large-scale, well-characterized stimuli that are representative of such activity. We present the Multi-angle Extended Three-dimensional Activities (META) stimulus set, a structured and highly instrumented set of extended event sequences performed in naturalistic settings. Performances were captured with two color cameras and a Kinect v2 camera with color and depth sensors, allowing the extraction of three-dimensional skeletal joint positions. We tracked the positions and identities of objects for all chapters using a mixture of manual coding and an automated tracking pipeline, and hand-annotated the timings of high-level actions. We also performed an online experiment to collect normative event boundaries for all chapters at a coarse and fine grain of segmentation, which allowed us to quantify event durations and agreement across participants. We share these materials publicly to advance new discoveries in the study of complex naturalistic activity.

How do drivers mitigate the effects of naturalistic visual complexity? : On attentional strategies and their implications under a change blindness protocol

How do the limits of high-level visual processing affect human performance in naturalistic, dynamic settings of (multimodal) interaction where observers can draw on experience to strategically adapt attention to familiar forms of complexity? In this backdrop, we investigate change detection in a driving context to study attentional allocation aimed at overcoming environmental complexity and temporal load. Results indicate that visuospatial complexity substantially increases change blindness but also that participants effectively respond to this load by increasing their focus on safety-relevant events, by adjusting their driving, and by avoiding non-productive forms of attentional elaboration, thereby also controlling "looked-but-failed-to-see" errors. Furthermore, analyses of gaze patterns reveal that drivers occasionally, but effectively, limit attentional monitoring and lingering for irrelevant changes. Overall, the experimental outcomes reveal how drivers exhibit effective attentional compensation in highly complex situations. Our findings uncover implications for driving education and development of driving skill-testing methods, as well as for human-factors guided development of AI-based driving assistance systems.

Differential influence of first- vs. third-person visual perspectives on segmentation and memory of complex dynamic events

Until now, most studies investigating the relationship between event segmentation and memory have used videos filmed from a third-person perspective, although people experience their lives from a first-person perspective. The present study aimed to determine whether visual perspective impacts events segmentation and further recall. Fifty-seven participants were recruited and assigned to either first- (1PP) or third-person perspective (3PP) condition, before segmenting videos of daily life activities. Our results showed that the although the number of event boundaries was higher in the 3PP condition than in the 1PP, no differences were observed for event segmentation qualitative abilities and organization. Memory of temporal order was better for events encoded in the 3PP than in the 1PP, while memory content was similar in both conditions. Higher event segmentation rates were correlated with a better recall of small actions and temporal order.

Spatiotemporal jump detection during continuous film viewing

Prior research on film viewing has demonstrated that participants frequently fail to notice spatiotemporal disruptions, such as scene edits in the movies. Whether such insensitivity to spatiotemporal disruptions extends beyond scene edits in film viewing is not well understood. Across three experiments, we created spatiotemporal disruptions by presenting participants with minute long movie clips, and occasionally jumping the movie clips ahead or backward in time. Participants were instructed to press a button when they noticed any disruptions while watching the clips. The results from experiments 1 and 2 indicate that participants failed to notice the disruptions in continuity about 10% to 30% of the time depending on the magnitude of the jump. In addition, detection rates were lower by approximately 10% when the videos jumped ahead in time compared to the backward jumps across all jump magnitudes, suggesting a role of knowledge about the future affects jump detection. An additional analysis used optic flow similarity during these disruptions. Our findings suggest that insensitivity to spatiotemporal disruptions during film viewing is influenced by knowledge about future states.

Multisensory synchrony of contextual boundaries affects temporal order memory, but not encoding or recognition

We memorize our daily life experiences, which are often multisensory in nature, by segmenting them into distinct event models, in accordance with perceived contextual or situational changes. However, very little is known about how multisensory boundaries affect segmentation, as most studies have focused on unisensory (visual or audio) segmentation. In three experiments, we investigated the effect of multisensory boundaries on segmentation in memory and perception. In Experiment 1, participants encoded lists of pictures while audio and visual contexts changed synchronously or asynchronously. After each list, we tested recognition and temporal associative memory for pictures that were encoded in the same audio-visual context or that crossed a synchronous or an asynchronous multisensory change. We found no effect of multisensory synchrony for recognition memory: synchronous and asynchronous changes similarly impaired recognition for pictures encoded at those changes, compared to pictures encoded further away from those changes. Multisensory synchrony did affect temporal associative memory, which was worse for pictures encoded at synchronous than at asynchronous changes. Follow up experiments showed that this effect was not due to the higher dimensionality of multisensory over unisensory contexts (Experiment 2), nor that it was due to the temporal unpredictability of contextual changes inherent to Experiment 1 (Experiment 3). We argue that participants formed situational expectations through multisensory synchronicity, such that synchronous multisensory changes deviated more strongly from those expectations than asynchronous changes. We discuss our findings in light of supportive and conflicting findings of uni- and multi-sensory segmentation.

Measuring event segmentation: An investigation into the stability of event boundary agreement across groups

People spontaneously divide everyday experience into smaller units (event segmentation). To measure event segmentation, studies typically ask participants to explicitly mark the boundaries between events as they watch a movie (segmentation task). Their data may then be used to infer how others are likely to segment the same movie. However, significant variability in performance across individuals could undermine the ability to generalize across groups, especially as more research moves online. To address this concern, we used several widely employed and novel measures to quantify segmentation agreement across different sized groups (n = 2-32) using data collected on different platforms and movie types (in-lab & commercial film vs. online & everyday activities). All measures captured nonrandom and video-specific boundaries, but with notable between-sample variability. Samples of 6-18 participants were required to reliably detect video-driven segmentation behavior within a single sample. As sample size increased, agreement values improved and eventually stabilized at comparable sample sizes for in-lab & commercial film data and online & everyday activities data. Stabilization occurred at smaller sample sizes when measures reflected (1) agreement between two groups versus agreement between an individual and group, and (2) boundary identification between small (fine-grained) rather than large (coarse-grained) events. These analyses inform the tailoring of sample sizes based on the comparison of interest, materials, and data collection platform. In addition to demonstrating the reliability of online and in-lab segmentation performance at moderate sample sizes, this study supports the use of segmentation data to infer when events are likely to be segmented.

The Hippocampal Horizon: Constructing and Segmenting Experience for Episodic Memory

How do we recollect specific events that have occurred during continuous ongoing experience? There is converging evidence from non-human animals that spatially modulated cellular activity of the hippocampal formation supports the construction of ongoing events. On the other hand, recent human oriented event cognition models have outlined that our experience is segmented into discrete units, and that such segmentation can operate on shorter or longer timescales. Here, we describe a unification of how these dynamic physiological mechanisms of the hippocampus relate to ongoing externally and internally driven event segmentation, facilitating the demarcation of specific moments during experience. Our cross-species interdisciplinary approach offers a novel perspective in the way we construct and remember specific events, leading to the generation of many new hypotheses for future research.

Touching events predict human action segmentation in brain and behavior

Recognizing the actions of others depends on segmentation into meaningful events. After decades of research in this area, it remains still unclear how humans do this and which brain areas support underlying processes. Here we show that a computer vision-based model of touching and untouching events can predict human behavior in segmenting object manipulation actions with high accuracy. Using this computational model and functional Magnetic Resonance Imaging (fMRI), we pinpoint the neural networks underlying this segmentation behavior during an implicit action observation task. Segmentation was announced by a strong increase of visual activity at touching events followed by the engagement of frontal, hippocampal and insula regions, signaling updating expectation at subsequent untouching events. Brain activity and behavior show that touching-untouching motifs are critical features for identifying the key elements of actions including object manipulations.

Brain-environment alignment during movie watching predicts fluid intelligence and affective function in adulthood

•

Functional brain connectivity (FC) patterns vary with changes in the environment.

•

Adult FC variability is linked to age-specific network communication profiles.

•

Across adulthood, the younger network interaction profile predicts higher fluid IQ.

•

Yoked FC-concrete environmental changes predict poorer fluid IQ and anxiety.

•

Brain areas linked to episodic memory underpin FC changes at multiple timescales.

BOLD fMRI studies have provided compelling evidence that the human brain demonstrates substantial moment-to-moment fluctuations in both activity and functional connectivity (FC) patterns. While the role of brain signal variability in fostering cognitive adaptation to ongoing environmental demands is well-documented, the relevance of moment-to-moment changes in FC patterns is still debated. Here, we adopt a graph theoretical approach in order to shed light on the cognitive-affective implications of FC variability and associated profiles of functional network communication in adulthood. Our goal is to identify brain communication pathways underlying FC reconfiguration at multiple timescales, thereby improving understanding of how faster perceptually bound versus slower conceptual processes shape neural tuning to the dynamics of the external world and, thus, indirectly, mold affective and cognitive responding to the environment. To this end, we used neuroimaging and behavioural data collected during movie watching by the Cambridge Center for Ageing and Neuroscience (N = 642, 326 women) and the Human Connectome Project (N = 176, 106 women). FC variability evoked by changes to both the concrete perceptual and the more abstract conceptual representation of an ongoing situation increased from young to older adulthood. However, coupling between variability in FC patterns and concrete environmental features was stronger at younger ages. FC variability (both moment-to-moment/concrete featural and abstract conceptual boundary-evoked) was associated with age-distinct profiles of network communication, specifically, greater functional integration of the default mode network in older adulthood, but greater informational flow across neural networks implicated in environmentally driven attention and control (cingulo-opercular, salience, ventral attention) in younger adulthood. Whole-brain communication pathways anchored in default mode regions relevant to episodic and semantic context creation (i.e., angular and middle temporal gyri) supported FC reconfiguration in response to changes in the conceptual representation of an ongoing situation (i.e., narrative event boundaries), as well as stronger coupling between moment-to-moment fluctuations in FC and concrete environmental features. Fluid intelligence/abstract reasoning was directly linked to levels of brain-environment alignment, but only indirectly associated with levels of FC variability. Specifically, stronger coupling between moment-to-moment FC variability and concrete environmental features predicted poorer fluid intelligence and greater affectively driven environmental vigilance. Complementarily, across the adult lifespan, higher fluid (but not crystallised) intelligence was related to stronger expression of the network communication profile underlying momentary and event boundary-based FC variability during youth. Our results indicate that the adaptiveness of dynamic FC reconfiguration during naturalistic information processing changes across the lifespan due to the associated network communication profiles. Moreover, our findings on brain-environment alignment complement the existing literature on the beneficial consequences of modulating brain signal variability in response to environmental complexity. Specifically, they imply that coupling between moment-to-moment FC variability and concrete environmental features may index a bias towards perceptually-bound, rather than conceptual processing, which hinders affective functioning and strategic cognitive engagement with the external environment.

Memories for third-person experiences in immersive virtual reality

We typically experience the world from a first-person perspective (1PP) but can sometimes experience events from a third-person perspective (3PP) much as an observer might see us. Little is known about how visual perspective influences the formation of memories for events. We developed an immersive virtual reality paradigm to examine how visual perspective during encoding influences memories. Across two studies, participants explored immersive virtual environments from first-person and third-person avatar perspectives while wearing an Oculus Rift headset. Memory was tested immediately (Study One and Study Two) and following a one-week delay (Study Two). We assessed the accuracy of visual memory using cued recall questions and spatial memory by asking participants to draw maps of the layout of each environment (Study One and Study Two). Additional phenomenological ratings were included to assess visual perspective during remembering (Study Two). There were no differences in the accuracy of visual information across the two studies, but 3PP experiences were found to increase spatial memory accuracy due to their wider camera field of view when compared to 1PP experiences. Our results also demonstrate that 3PP experiences create 3PP memories, as reflected by an increase in subjective ratings of observer-like perspectives during remembering. In sum, visual perspective during memory formation influences the accuracy of spatial but not visual information, and the vantage point of memories during remembering.

Do doorways really matter: Investigating memory benefits of event segmentation in a virtual learning environment

Event segmentation allows the flow of information experienced in life to be partitioned into distinct episodes, facilitating understanding of the world, action within it, and the ability to store information in memory. One basis on which experiences are segmented is the presence of physical boundaries, such as walking through doorways. Previous findings have shown that event segmentation has a significant influence on memory, with better memory for events occurring within a single boundary (compared to events that cross boundaries). By manipulating the features of boundaries and the amount of information presented between boundaries the present research investigates the nature of event boundaries. We make use of a virtual learning environment to present lists of words in virtual rooms, testing memory for the word lists as a function of the presence or absence of spatial-temporal gaps and physical boundaries during encoding (i.e., by maintaining participants within individual rooms or moving them through doorways between rooms). Across four experiments, we show that segmenting information with spatial-temporal gaps results in an increase in clustering (reflecting the structure imposed at encoding) an increase in the number of words remembered during later tests of episodic recall (a memory benefit) and an increase in recalling the words in the order of presentation. Importantly, however, the data show that the presence of doorways is not required for event segmentation to benefit memory: increases in clustering, memory for temporal order and recall performance were found with temporal gaps alone. Furthermore, the results suggest that episodic memory may be optimised if the amount of information between boundaries can be maintained within working memory. We discuss the implications of the findings for Event Segmentation Theory and propose an alternative theoretical account of the episodic memory benefits based on temporal clustering.

Influences of domain knowledge on segmentation and memory

Much research has shown that experts possess superior memory in their domain of expertise. This memory benefit has been proposed to be the result of various encoding mechanisms, such as chunking and differentiation. Another potential encoding mechanism that is associated with memory is event segmentation, which is the process by which people parse continuous information into meaningful, discrete units. Previous research has found evidence that segmentation, to some extent, is affected by top-down processing. To date, few studies have investigated the influence of expertise on segmentation, and questions about expertise, segmentation ability, and their impact on memory remain. The goal of the current study was to investigate the influence of expertise on segmentation and memory ability for two different domains: basketball and Overwatch. Participants with high and low knowledge for basketball and with low knowledge for Overwatch viewed and segmented videos at coarse and fine grains, then completed memory tests. Differences in segmentation ability and memory were present between experts and control novices, specifically for the basketball videos; however, experts' segmentation only predicted memory for activities for which knowledge was lacking. Overall, this research suggests that experts' superior memory is not due to their segmentation ability and contributes to a growing body of literature showing evidence supporting conceptual effects on segmentation.

Culture influences how people divide continuous sensory experience into events

Everyday experience is divided into meaningful events as a part of human perception. Current accounts of this process, known as event segmentation, focus on how characteristics of the experience (e.g., situation changes) influence segmentation. However, characteristics of the viewers themselves have been largely neglected. We test whether one such viewer characteristic, their cultural background, impacts online event segmentation. Culture could impact event segmentation (1) by emphasizing different aspects of experiences as being important for comprehension, memory, and communication, and (2) by providing different exemplars of how everyday activities are performed, which objects are likely to be used, and how scenes are laid out. Indian and US viewers (N = 152) identified events in everyday activities (e.g., making coffee) recorded in Indian and US settings. Consistent with their cultural preference for analytical processing, US viewers segmented the activities into more events than did Indian viewers. Furthermore, event boundaries identified by US viewers were more strongly associated with visual changes, whereas boundaries identified by Indian viewers were more strongly associated with goal changes. There was no evidence that familiarity with an activity impacted segmentation. Thus, culture impacts event perception by altering the types of information people prioritize when dividing experience into meaningful events.

Event Perception and Memory

Events make up much of our lived experience, and the perceptual mechanisms that represent events in experience have pervasive effects on action control, language use, and remembering. Event representations in both perception and memory have rich internal structure and connections one to another, and both are heavily informed by knowledge accumulated from previous experiences. Event perception and memory have been identified with specific computational and neural mechanisms, which show protracted development in childhood and are affected by language use, expertise, and brain disorders and injuries. Current theoretical approaches focus on the mechanisms by which events are segmented from ongoing experience, and emphasize the common coding of events for perception, action, and memory. Abetted by developments in eye-tracking, neuroimaging, and computer science, research on event perception and memory is moving from small-scale laboratory analogs to the complexity of events in the wild.

The Hippocampal Film Editor: Sensitivity and Specificity to Event Boundaries in Continuous Experience

The function of the human hippocampus is normally investigated by experimental manipulation of discrete events. Less is known about what triggers hippocampal activity during more naturalistic, continuous experience. We hypothesized that the hippocampus would be sensitive to the occurrence of event boundaries, that is, moments in time identified by observers as a transition between events. To address this, we analyzed functional MRI data from two groups: one (n = 253, 131 female) who viewed an 8.5 min film and another (n = 15, 6 female) who viewed a 120 min film. We observed a strong hippocampal response at boundaries defined by independent observers, which was modulated by boundary salience (the number of observers that identified each boundary). In the longer film, there were sufficient boundaries to show that this modulation remained after covarying out a large number of perceptual factors. This hypothesis-driven approach was complemented by a data-driven approach, in which we identified hippocampal events as moments in time with the strongest hippocampal activity. The correspondence between these hippocampal events and event boundaries was highly significant, revealing that the hippocampal response is not only sensitive, but also specific to event boundaries. We conclude that event boundaries play a key role in shaping hippocampal activity during encoding of naturalistic events.SIGNIFICANCE STATEMENT Recent years have seen the field of human neuroscience research transitioning from experiments with simple stimuli to the study of more complex and naturalistic experience. Nonetheless, our understanding of the function of many brain regions, such as the hippocampus, is based primarily on the study of brief, discrete events. As a result, we know little of what triggers hippocampal activity in real-life settings when we are exposed to a continuous stream of information. When does the hippocampus "decide" to respond during the encoding of naturalistic experience? We reveal here that hippocampal activity measured by fMRI during film watching is both sensitive and specific to event boundaries, identifying a potential mechanism whereby event boundaries shape experience by modulation of hippocampal activity.