Surprisal From Language Models Can Predict ERPs in Processing Predicate-Argument Structures Only if Enriched by an Agent Preference Principle

Language models based on artificial neural networks increasingly capture key aspects of how humans process sentences. Most notably, model-based surprisals predict event-related potentials such as N400 amplitudes during parsing. Assuming that these models represent realistic estimates of human linguistic experience, their success in modeling language processing raises the possibility that the human processing system relies on no other principles than the general architecture of language models and on sufficient linguistic input. Here, we test this hypothesis on N400 effects observed during the processing of verb-final sentences in German, Basque, and Hindi. By stacking Bayesian generalised additive models, we show that, in each language, N400 amplitudes and topographies in the region of the verb are best predicted when model-based surprisals are complemented by an Agent Preference principle that transiently interprets initial role-ambiguous noun phrases as agents, leading to reanalysis when this interpretation fails. Our findings demonstrate the need for this principle independently of usage frequencies and structural differences between languages. The principle has an unequal force, however. Compared to surprisal, its effect is weakest in German, stronger in Hindi, and still stronger in Basque. This gradient is correlated with the extent to which grammars allow unmarked NPs to be patients, a structural feature that boosts reanalysis effects. We conclude that language models gain more neurobiological plausibility by incorporating an Agent Preference. Conversely, theories of human processing profit from incorporating surprisal estimates in addition to principles like the Agent Preference, which arguably have distinct evolutionary roots.

Hippocampal-cortical interactions during event boundaries support retention of complex narrative events

According to most memory theories, encoding involves continuous communication between the hippocampus and neocortex, but recent work has shown that key moments at the end of an event, called event boundaries, may be especially critical for memory formation. We sought to determine how communication between the hippocampus and neocortical regions during the encoding of naturalistic events related to subsequent retrieval of those events and whether this was particularly important at event boundaries. Participants encoded and recalled two cartoon movies during fMRI scanning. Higher functional connectivity between the hippocampus and the posterior medial network (PMN) at an event's offset is related to the subsequent successful recall of that event. Furthermore, hippocampal-PMN offset connectivity also predicted the amount of detail retrieved after a 2-day delay. These data demonstrate that the relationship between memory encoding and hippocampal-neocortical interaction is dynamic and biased toward boundaries.

Setting boundaries: Development of neural and behavioral event cognition in early childhood

The ongoing stream of sensory experience is so complex and ever-changing that we tend to parse this experience at "event boundaries," which structures and strengthens memory. Memory processes undergo profound change across early childhood. Whether young children also divide their ongoing processing along event boundaries, and if those boundaries relate to memory, could provide important insight into the development of memory systems. In Study 1, 4-7-year-old children and adults segmented a cartoon, and we tested their memory. Children's event boundaries were more variable than adults' and differed in location and consistency of agreement. Older children's event segmentation was more adult-like than younger children's, and children who segmented events more like adults had better memory for those events. In Study 2, we asked whether these developmental differences in event segmentation had their roots in distinct neural representations. A separate group of 4-8-year-old children watched the same cartoon while undergoing an fMRI scan. In the right hippocampus, greater pattern dissimilarity across event boundaries compared to within events was evident for both child and adult behavioral boundaries, suggesting children and adults share similar event cognition. However, the boundaries identified by a data-driven Hidden Markov Model found that a different brain region-the left and right angular gyrus-aligned only with event boundaries defined by children. Overall, these data suggest that children's event cognition is reasonably well-developed by age 4 but continues to become more adult-like across early childhood. RESEARCH HIGHLIGHTS: Adults naturally break their experience into events, which structures and strengthens memory, but less is known about children's event perception and memory. Study 1 had adults and children segment and remember events from an animated show, and Study 2 compared those segmentations to other children's fMRI data. Children show better recognition and temporal order memory and more adult-like event segmentation with age, and children who segment more like adults have better memory. Children's and adults' behavioral boundaries mapped onto pattern similarity differences in hippocampus, and children's behavioral boundaries matched a data-driven model's boundaries in angular gyrus.

Integration of event experiences to build relational knowledge in the human brain

We investigated how the human brain integrates experiences of specific events to build general knowledge about typical event structure. We examined an episodic memory area important for temporal relations, anterior-lateral entorhinal cortex, and a semantic memory area important for action concepts, middle temporal gyrus, to understand how and when these areas contribute to these processes. Participants underwent functional magnetic resonance imaging while learning and recalling temporal relations among novel events over two sessions 1 week apart. Across distinct contexts, individual temporal relations among events could either be consistent or inconsistent with each other. Within each context, during the recall phase, we measured associative coding as the difference of multivoxel correlations among related vs unrelated pairs of events. Neural regions that form integrative representations should exhibit stronger associative coding in the consistent than the inconsistent contexts. We found evidence of integrative representations that emerged quickly in anterior-lateral entorhinal cortex (at session 1), and only subsequently in middle temporal gyrus, which showed a significant change across sessions. A complementary pattern of findings was seen with signatures during learning. This suggests that integrative representations are established early in anterior-lateral entorhinal cortex and may be a pathway to the later emergence of semantic knowledge in middle temporal gyrus.

Adult age differences in event memory updating: The roles of prior-event retrieval and prediction

Remembering past events can lead to predictions of what is to come and to experiencing prediction errors when things change. Previous research has shown enhanced memory updating for ongoing events that are inconsistent with predictions based on past experiences. According to the Event Memory Retrieval and Comparison (EMRC) Theory, such memory updating depends on the encoding of configural representations that bind retrieved features of the previous event, changed features, and the relationship between the two. We investigated potential age-related differences in these mechanisms by showing older and younger adults two movies of everyday activities. Activities in the second movie either repeated from the first movie or included changed endings. During the second movie, before activities ended, participants were instructed to predict the upcoming action based on the first movie. One week later, participants were instructed to recall activity endings from the second movie. For younger adults, having predicted endings consistent with the first movie before seeing changed endings was subsequently associated with better recall of these changed endings and recollection that activities had changed. Conversely, for older adults, making such predictions prior to changes was associated with intruding details from the first movie endings and was less strongly associated with change recollection. Consistent with EMRC, these findings suggest that retrieval of relevant experiences when events change can trigger prediction errors that prompt associative encoding of existing memories and current perceptions. These mechanisms were less efficient in older adults, which may account for their poorer event memory updating than younger adults. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Evidence That Event Boundaries Are Access Points for Memory Retrieval

When recalling memories, we often scan information-rich continuous episodes, for example, to find our keys. How does our brain access and search through those memories? We suggest that high-level structure, marked by event boundaries, guides us through this process: In our computational model, memory scanning is sped up by skipping ahead to the next event boundary upon reaching a decision threshold. In adult Mechanical Turk workers from the United States, we used a movie (normed for event boundaries; Study 1, N = 203) to prompt memory scanning of movie segments for answers (Study 2, N = 298) and mental simulation (Study 3, N = 100) of these segments. Confirming model predictions, we found that memory-scanning times varied as a function of the number of event boundaries within a segment and the distance of the search target to the previous boundary (the key diagnostic parameter). Mental simulation times were also described by a skipping process with a higher skipping threshold than memory scanning. These findings identify event boundaries as access points to memory.

Spatiotemporal jump detection during continuous film viewing

Prior research on film viewing has demonstrated that participants frequently fail to notice spatiotemporal disruptions, such as scene edits in the movies. Whether such insensitivity to spatiotemporal disruptions extends beyond scene edits in film viewing is not well understood. Across three experiments, we created spatiotemporal disruptions by presenting participants with minute long movie clips, and occasionally jumping the movie clips ahead or backward in time. Participants were instructed to press a button when they noticed any disruptions while watching the clips. The results from experiments 1 and 2 indicate that participants failed to notice the disruptions in continuity about 10% to 30% of the time depending on the magnitude of the jump. In addition, detection rates were lower by approximately 10% when the videos jumped ahead in time compared to the backward jumps across all jump magnitudes, suggesting a role of knowledge about the future affects jump detection. An additional analysis used optic flow similarity during these disruptions. Our findings suggest that insensitivity to spatiotemporal disruptions during film viewing is influenced by knowledge about future states.

Autobiographical event memory and aging: older adults get the gist

We propose that older adults' ability to retrieve episodic autobiographical events, although often viewed through a lens of decline, reveals much about what is preserved and prioritized in cognitive aging. Central to our proposal is the idea that the so-called gist of an autobiographical event is not only spared with normal aging but also well adapted to serve memory-guided behavior in older age. To support our proposal, we review cognitive and brain evidence indicating an age-related shift toward gist memory. We then discuss why this shift likely arises from more than age-related decline and instead partly reflects a natural, arguably adaptive, outcome of experience, motivation, and mode-of-thinking factors. Our proposal reveals an upside of age-related memory changes and identifies important research questions.

Neural event segmentation of continuous experience in human infants

How infants experience the world is fundamental to understanding their cognition and development. A key principle of adult experience is that, despite receiving continuous sensory input, we perceive this input as discrete events. Here we investigate such event segmentation in infants and how it differs from adults. Research on event cognition in infants often uses simplified tasks in which (adult) experimenters help solve the segmentation problem for infants by defining event boundaries or presenting discrete actions/vignettes. This presupposes which events are experienced by infants and leaves open questions about the principles governing infant segmentation. We take a different, data-driven approach by studying infant event segmentation of continuous input. We collected whole-brain functional MRI (fMRI) data from awake infants (and adults, for comparison) watching a cartoon and used a hidden Markov model to identify event states in the brain. We quantified the existence, timescale, and organization of multiple-event representations across brain regions. The adult brain exhibited a known hierarchical gradient of event timescales, from shorter events in early visual regions to longer events in later visual and associative regions. In contrast, the infant brain represented only longer events, even in early visual regions, with no timescale hierarchy. The boundaries defining these infant events only partially overlapped with boundaries defined from adult brain activity and behavioral judgments. These findings suggest that events are organized differently in infants, with longer timescales and more stable neural patterns, even in sensory regions. This may indicate greater temporal integration and reduced temporal precision during dynamic, naturalistic perception.

Melting Ice With Your Mind: Representational Momentum for Physical States

When a log burns, it transforms from a block of wood into a pile of ash. Such state changes are among the most dramatic ways objects change, going beyond mere changes of position or orientation. How does the mind represent changes of state? A foundational result in visual cognition is that memory extrapolates the positions of moving objects-a distortion called representational momentum. Here, five experiments (N = 400 adults) exploited this phenomenon to investigate mental representations in state space. Participants who viewed objects undergoing state changes (e.g., ice melting, logs burning, or grapes shriveling) remembered them as more changed (e.g., more melted, burned, or shriveled) than they actually were. This pattern extended to several types of state changes, went beyond their low-level properties, and even adhered to their natural trajectories in state space. Thus, mental representations of objects actively incorporate how they change-not only in their relation to their environment, but also in their essential qualities.

Understanding Everyday Events: Predictive-Looking Errors Drive Memory Updating

Memory-guided predictions can improve event comprehension by guiding attention and the eyes to the location where an actor is about to perform an action. But when events change, viewers may experience predictive-looking errors and need to update their memories. In two experiments (Ns = 38 and 98), we examined the consequences of mnemonic predictive-looking errors for comprehending and remembering event changes. University students watched movies of everyday activities with actions that were repeated exactly and actions that were repeated with changed features-for example, an actor reached for a paper towel on one occasion and a dish towel on the next. Memory guidance led to predictive-looking errors that were associated with better memory for subsequently changed event features. These results indicate that retrieving recent event features can guide predictions during unfolding events and that error signals derived from mismatches between mnemonic predictions and actual events contribute to new learning.

A neural network model of when to retrieve and encode episodic memories

Recent human behavioral and neuroimaging results suggest that people are selective in when they encode and retrieve episodic memories. To explain these findings, we trained a memory-augmented neural network to use its episodic memory to support prediction of upcoming states in an environment where past situations sometimes reoccur. We found that the network learned to retrieve selectively as a function of several factors, including its uncertainty about the upcoming state. Additionally, we found that selectively encoding episodic memories at the end of an event (but not mid-event) led to better subsequent prediction performance. In all of these cases, the benefits of selective retrieval and encoding can be explained in terms of reducing the risk of retrieving irrelevant memories. Overall, these modeling results provide a resource-rational account of why episodic retrieval and encoding should be selective and lead to several testable predictions.

Macro-event recognition in healthy ageing, Alzheimer's disease, and mild cognitive impairment

Event perception and cognition is integral to our everyday experience and functional ability. A commonly reported complaint in Alzheimer's disease (AD) is the inability to follow narratives - be it textual, conversational, video, or pictures. This phenomenon has received little systematic research so far. In the current study, we developed a novel paradigm to examine macro-event recognition in individuals with AD in the early stage and its preceding stage of mild cognitive impairment (MCI) in comparison with cognitively healthy older adults, using pictures depicting events. In Experiment 1, we examined participants' ability to integrate pictorially depicted sub-events into macro-events. The pictures were presented in a scrambled order, and participants were expected to arrange them in the temporally and causally appropriate sequence, as dictated by the macro-event schema. Additionally, we investigated the effect of cueing the appropriate event schema by providing a word cue (verb). In Experiment 2, macro-event recognition was examined again using a cognitively less taxing paradigm, where pictures depicting sub-events were presented in correct order, but staggered, and recognition speed was measured. We observed significant deficits in the AD and MCI groups' performance compared with the cognitively healthy older adults, across both experiments, suggesting event perception and cognition is impaired early in the course of AD. There was no effect of cueing on the performance of any of the groups. The theoretical and clinical implications of these findings are discussed.

The hippocampus constructs narrative memories across distant events

Life's events are scattered throughout time, yet we often recall different events in the context of an integrated narrative. Prior research suggests that the hippocampus, which supports memory for past events, can support the integration of overlapping associations or separate events in memory. However, the conditions that lead to hippocampus-dependent memory integration are unclear. We used functional brain imaging to test whether the opportunity to form a larger narrative (narrative coherence) drives hippocampal memory integration. During encoding of fictional stories, patterns of hippocampal activity, including activity at boundaries between events, were more similar between distant events that formed one coherent narrative, compared with overlapping events taken from unrelated narratives. One day later, the hippocampus preferentially supported detailed recall of coherent narrative events, through reinstatement of hippocampal activity patterns from encoding. These findings demonstrate a key function of the hippocampus: the integration of events into a narrative structure for memory.

A novel study: long-lasting event memory

The current study reports two experiments that assessed memory for complex sets of information (i.e., four novels) over the course of several years. This was done to explore several issues, including (a) the content and durability of memory, (b) causal connectivity among the described events, (c) serial position within the novel, (d) pattern of retention and forgetting, and (e) interest in the novel. This study revealed that people remember events read in a novel for long periods of time with a shallow rate of forgetting. We also found that they remembered information better when it was presented earlier on in the novel, was more causally connected to other events in the novel, and was transitional in the character or story plot. Consistent with prior research, reported level of interest was not related to later memory. Because we experience events through different learning modalities (e.g., novels, film, autobiographical experience) which are likely remembered similarly, this research broadens our understanding of how people remember the events they learn about through different means.

Representing agents, patients, goals and instruments in causative events: A cross-linguistic investigation of early language and cognition

Although it is widely assumed that the linguistic description of events is based on a structured representation of event components at the perceptual/conceptual level, little empirical work has tested this assumption directly. Here, we test the connection between language and perception/cognition cross-linguistically, focusing on the relative salience of causative event components in language and cognition. We draw on evidence from preschoolers speaking English or Turkish. In a picture description task, Turkish-speaking 3-5-year-olds mentioned Agents less than their English-speaking peers (Turkish allows subject drop); furthermore, both language groups mentioned Patients more frequently than Goals, and Instruments less frequently than either Patients or Goals. In a change blindness task, both language groups were equally accurate at detecting changes to Agents (despite surface differences in Agent mentions). The remaining components also behaved similarly: both language groups were less accurate in detecting changes to Instruments than either Patients or Goals (even though Turkish-speaking preschoolers were less accurate overall than their English-speaking peers). To our knowledge, this is the first study offering evidence for a strong-even though not strict-homology between linguistic and conceptual event roles in young learners cross-linguistically.

Aging and the encoding of changes in events: The role of neural activity pattern reinstatement

When encountering unexpected event changes, memories of relevant past experiences must be updated to form new representations. Current models of memory updating propose that people must first generate memory-based predictions to detect and register that features of the environment have changed, then encode the new event features and integrate them with relevant memories of past experiences to form configural memory representations. Each of these steps may be impaired in older adults. Using functional MRI, we investigated these mechanisms in healthy young and older adults. In the scanner, participants first watched a movie depicting everyday activities in a day of an actor's life. They next watched a second nearly identical movie in which some scenes ended differently. Crucially, before watching the last part of each activity, the second movie stopped, and participants were asked to mentally replay how the activity previously ended. Three days later, participants were asked to recall the activities. Neural activity pattern reinstatement in medial temporal lobe (MTL) during the replay phase of the second movie was associated with detecting changes and with better memory for the original activity features. Reinstatements in posterior medial cortex (PMC) additionally predicted better memory for changed features. Compared to young adults, older adults showed a reduced ability to detect and remember changes and weaker associations between reinstatement and memory performance. These findings suggest that PMC and MTL contribute to change processing by reinstating previous event features, and that older adults are less able to use reinstatement to update memory for changed features.

Narratives for Neuroscience

People organize and convey their thoughts according to narratives. However, neuroscientists are often reluctant to incorporate narrative stimuli into their experiments. We argue that narratives deserve wider adoption in human neuroscience because they tap into the brain's native machinery for representing the world and provide rich variability for testing hypotheses.

Linguistic Information in Auditory Dynamic Events Contributes to the Detection of Fine, Not Coarse Event Boundaries

Human observers (comprehenders) segment dynamic information into discrete events. That is, although there is continuous sensory information, comprehenders perceive boundaries between two meaningful units of information. In narrative comprehension, comprehenders use linguistic, non-linguistic , and physical cues for this event boundary perception. Yet, it is an open question - both from a theoretical and an empirical perspective - how linguistic and non-linguistic cues contribute to this process. The current study explores how linguistic cues contribute to the participants' ability to segment continuous auditory information into discrete, hierarchically structured events. Native speakers of German and non-native speakers, who neither spoke nor understood German, segmented a German audio drama into coarse and fine events. Whereas native participants could make use of linguistic, non-linguistic, and physical cues for segmentation, non-native participants could only use non-linguistic and physical cues. We analyzed segmentation behavior in terms of the ability to identify coarse and fine event boundaries and the resulting hierarchical structure. Non-native listeners identified almost identical coarse event boundaries as native listeners, but missed some of the fine event boundaries identified by the native listeners. Interestingly, hierarchical event perception (as measured by hierarchical alignment and enclosure) was comparable for native and non-native participants. In summary, linguistic cues contributed particularly to the identification of certain fine event boundaries. The results are discussed with regard to the current theories of event cognition.

Walking through doorways causes forgetting: recall

The aim of the current study was to explore how the location updating effect is affected when people are tested using recall rather than recognition, which is what has been done in prior work. Differences in the memory processes involved with these two tasks lead to predictions for two different patterns of data. In Experiment 1, memory was tested by having participants recall the single object they were carrying or had just put down, whereas in Experiment 2, people sometimes needed to recall both objects. It was found that, unlike recognition test performance, a similar location updating effect was found for both Associated (what was currently being carried) and Dissociated (what was recently set down) objects. Moreover, when both objects were correctly recalled, there was a bias to remember them in the order that they were encountered. Finally, if only one object was correctly recalled, it was the Associated object that was currently being carried. Overall, these results are consistent with the Event Horizon Model of event cognition.

Location-based prospective memory

This study explores location-based prospective memory. People often have to remember to do things when in a particular location, such as buying tissues the next time they are in the supermarket. For event cognition theory, location is important for structuring events. However, because event cognition has not been used to examine prospective memory, the question remains of how multiple events will influence prospective memory performance. In our experiments, people delivered messages from store to store in a virtual shopping mall as an ongoing task. The prospective tasks were to do certain activities in certain stores. For Experiment 1, each trial involved one prospective memory task to be done in a single location at one of three delays. The virtual environment and location cues were effective for prospective memory, and performance was unaffected by delay. For Experiment 2, each trial involved two prospective memory tasks, given in either one or two instruction locations, and to be done in either one or two store locations. There was improved performance when people received instructions from two locations and did both tasks in one location relative to other combinations. This demonstrates that location-based event structure influences how well people perform on prospective memory tasks.

Constructing Experience: Event Models from Perception to Action

Mental representations of everyday experience are rich, structured, and multimodal. In this article we consider the adaptive pressures that led to human construction of such representations, arguing that structured event representations enable cognitive systems to more effectively predict the trajectory of naturalistic everyday activity. We propose an account of how cortical systems and the hippocampus (HPC) interact to construct, maintain, and update event representations. This analysis throws light on recent research on story comprehension, event segmentation, episodic memory, and action planning. It also suggests how the growing science base can be deployed to diagnose impairments in event perception and memory, and to improve memory for everyday events.

How Children and Adults Encode Causative Events Cross-Linguistically: Implications for Language Production and Attention

This study investigates the implications of language-specific constraints on linguistic event encoding for the description and on-line inspection of causative events. English-speaking and Greek-speaking adults, 3-year-olds, and 4-year-olds viewed and described causative events, which are composed of Means and Result subevents, in an eyetracking study. The results demonstrate cross-linguistic differences in the informational content of causative event descriptions: Greek speakers across age groups were more likely than English speakers to mention only one causative subevent. Developmental changes in the tendency to encode information about causative events in language were also evident: in both language groups, adults were more likely than children to mention both Means and Result subevents. Finally, for both adult and child speakers of both languages, preparing different types of event descriptions changed the way that events were visually inspected, shifting attention toward to-be-encoded subevents. These findings offer some of the first evidence about the development of the language production system, the attentional mechanisms that it employs, and its workings in speakers of different languages.

Predicting Individual Action Switching in Covert and Continuous Interactive Tasks Using the Fluid Events Model

The Fluid Events Model is aimed at predicting changes in the actions people take on a moment-by-moment basis. In contrast with other research on action selection, this work does not investigate why some course of action was selected, but rather the likelihood of discontinuing the current course of action and selecting another in the near future. This is done using both task-based and experience-based factors. Prior work evaluated this model in the context of trial-by-trial, independent, interactive events, such as choosing how to copy a figure of a line drawing. In this paper, we extend this model to more covert event experiences, such as reading narratives, as well as to continuous interactive events, such as playing a video game. To this end, the model was applied to existing data sets of reading time and event segmentation for written and picture stories. It was also applied to existing data sets of performance in a strategy board game, an aerial combat game, and a first person shooter game in which a participant’s current state was dependent on prior events. The results revealed that the model predicted behavior changes well, taking into account both the theoretically defined structure of the described events, as well as a person’s prior experience. Thus, theories of event cognition can benefit from efforts that take into account not only how events in the world are structured, but also how people experience those events.

Getting the gist of events: recognition of two-participant actions from brief displays

Unlike rapid scene and object recognition from brief displays, little is known about recognition of event categories and event roles from minimal visual information. In 3 experiments, we displayed naturalistic photographs of a wide range of 2-participant event scenes for 37 ms and 73 ms followed by a mask, and found that event categories (the event gist; e.g., "kicking," "pushing") and event roles (i.e., Agent and Patient) can be recognized rapidly, even with various actor pairs and backgrounds. Norming ratings from a subsequent experiment revealed that certain physical features (e.g., outstretched extremities) that correlate with Agent-hood could have contributed to rapid role recognition. In a final experiment, using identical twin actors, we then varied these features in 2 sets of stimuli, in which Patients had Agent-like features or not. Subjects recognized the roles of event participants less accurately when Patients possessed Agent-like features, with this difference being eliminated with 2-s durations. Thus, given minimal visual input, typical Agent-like physical features are used in role recognition, but with sufficient input from multiple fixations, people categorically determine the relationship between event participants.