What is the role of the film viewer? The effects of narrative comprehension and viewing task on gaze control in film

Eye movements and event segmentation: Eye movements reveal age-related differences in event model updating

People spontaneously segment continuous ongoing actions into sequences of events. Prior research found that gaze similarity and pupil dilation increase at event boundaries and that older adults segment more idiosyncratically than do young adults. We used eye tracking to explore age-related differences in gaze similarity (i.e., the extent to which individuals look at the same places at the same time as others) and pupil dilation at event boundaries. Older and young adults watched naturalistic videos of actors performing everyday activities while we tracked their eye movements. Afterward, they segmented the videos into subevents. Replicating prior work, we found that pupil size and gaze similarity increased at event boundaries. Thus, there were fewer individual differences in eye position at boundaries. We also found that young adults had higher gaze similarity than older adults throughout an entire video and at event boundaries. This study is the first to show that age-related differences in how people parse continuous everyday activities into events may be partially explained by individual differences in gaze patterns. Those who segment less normatively may do so because they fixate less normative regions. Results have implications for future interventions designed to improve encoding in older adults. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

The role of individual differences in resistance to persuasion on memory for political advertisements

When people see political advertisements on a polarized issue they take a stance on, what factors influence how they respond to and remember the adverts contents? Across three studies, we tested competing hypotheses about how individual differences in social vigilantism (i.e., attitude superiority) and need for cognition relate to intentions to resist attitude change and memory for political advertisements concerning abortion. In Experiments 1 and 2, we examined participants' intentions to use resistance strategies to preserve their pre-existing attitudes about abortion, by either engaging against opposing opinions or disengaging from them. In Experiment 3, we examined participants' memory for information about both sides of the controversy presented in political advertisements. Our results suggest higher levels of social vigilantism are related to greater intentions to counterargue and better memory for attitude-incongruent information. These findings extend our understanding of individual differences in how people process and respond to controversial social and political discourse.

Revisiting the role of attention in the "weapon focus effect": Do weapons draw gaze away from the perpetrator under naturalistic viewing conditions?

The presence of a weapon in a scene has been found to attract observers' attention and to impair their memory of the person holding the weapon. Here, we examined the role of attention in this weapon focus effect (WFE) under different viewing conditions. German participants viewed stimuli in which a man committed a robbery while holding a gun or a cell phone. The stimuli were based on material used in a recent U.S. study reporting large memory effects. Recording eye movements allowed us to test whether observers' attention in the gun condition shifted away from the perpetrator towards the gun, compared with the phone condition. When using videos (Experiment 1), weapon presence did not appear to modulate the viewing time for the perpetrator, whereas the evidence concerning the critical object remained inconclusive. When using slide shows (Experiment 2), the gun attracted more gaze than the phone, replicating previous research. However, the attentional shift towards the weapon did not come at a cost of viewing time on the perpetrator. In both experiments, observers focused their attention predominantly on the depicted people and much less on the gun or phone. The presence of a weapon did not cause participants to recall fewer details about the perpetrator's appearance in either experiment. This null effect was replicated in an online study using the original videos and testing more participants. The results seem at odds with the attention-shift explanation of the WFE. Moreover, the results indicate that the WFE is not a universal phenomenon.

The audience who knew too much: investigating the role of spontaneous theory of mind on the processing of dramatic irony scenes in film

As in real life, cinema viewers rely on spontaneous theory of mind (SToM) to interpret characters' mental states. Thus, analyzing cinematic structures offers a unique opportunity to examine ecologically valid sociocognitive processes. We conducted a proof-of-concept study (N = 42) to explore how SToM inferences impact film event comprehension in dramatic irony scenes, where knowledge divergence exists between the audience and characters. We hypothesized that spectators would focus more on characters' mental states in such false-belief inducing scenarios compared to scenarios without such disparity. We used six Harold Lloyd silent comedy clips in a narrative comprehension and spontaneous mental state attribution study with a between-subject (Knowledge Manipulation: Installation vs. Control) and within-subject (Phase: Context vs. Exploitation) comparisons. We provided critical information unknown to the characters only to the Installation group and withheld it from the Control group. By comparing differences in participants' descriptions of the clips during the Context phase (varying across groups) and Exploitation phase (same across groups), we evaluated viewers' processing of the same scenes based on their false- or true-belief representations. Our findings indicate that the Installation group used more cognitive mental state words during the Exploitation phase relative to the Context phase, suggesting that exposure to undisclosed critical information enhances the frequency of spontaneous epistemic state inferences and integration into event models of the exploitation. This research advances neurocinematics by highlighting spontaneous sociocognitive processes in event perception and comprehension and provides a novel dramatic irony film corpus and measures for future moment-to-moment SToM processing studies across cognitive-behavioral, physiological, and neural levels.

Audience immersion: validating attentional and physiological measures against self-report

When an audience member becomes immersed, their attention shifts towards the media and story, and they allocate cognitive resources to represent events and characters. Here, we investigate whether it is possible to measure immersion using continuous behavioural and physiological measures. Using television and film clips, we validated dual-task reaction times, heart rate, and skin conductance against self-reported narrative engagement. We find that reaction times to a secondary task were strongly positively correlated with self-reported immersion: slower reaction times were indicative of greater immersion, particularly emotional engagement. Synchrony in heart rate across participants was associated with self-reported attentional and emotional engagement with the story, although we found no such relationship with skin conductance. These results establish both dual-task reaction times and heart rate as candidate measures for the real-time, continuous, assessment of audience immersion.

Measuring event segmentation: An investigation into the stability of event boundary agreement across groups

People spontaneously divide everyday experience into smaller units (event segmentation). To measure event segmentation, studies typically ask participants to explicitly mark the boundaries between events as they watch a movie (segmentation task). Their data may then be used to infer how others are likely to segment the same movie. However, significant variability in performance across individuals could undermine the ability to generalize across groups, especially as more research moves online. To address this concern, we used several widely employed and novel measures to quantify segmentation agreement across different sized groups (n = 2-32) using data collected on different platforms and movie types (in-lab & commercial film vs. online & everyday activities). All measures captured nonrandom and video-specific boundaries, but with notable between-sample variability. Samples of 6-18 participants were required to reliably detect video-driven segmentation behavior within a single sample. As sample size increased, agreement values improved and eventually stabilized at comparable sample sizes for in-lab & commercial film data and online & everyday activities data. Stabilization occurred at smaller sample sizes when measures reflected (1) agreement between two groups versus agreement between an individual and group, and (2) boundary identification between small (fine-grained) rather than large (coarse-grained) events. These analyses inform the tailoring of sample sizes based on the comparison of interest, materials, and data collection platform. In addition to demonstrating the reliability of online and in-lab segmentation performance at moderate sample sizes, this study supports the use of segmentation data to infer when events are likely to be segmented.

Narrative Comprehension Guides Eye Movements in the Absence of Motion

Viewers' attentional selection while looking at scenes is affected by both top-down and bottom-up factors. However, when watching film, viewers typically attend to the movie similarly irrespective of top-down factors-a phenomenon we call the tyranny of film. A key difference between still pictures and film is that film contains motion, which is a strong attractor of attention and highly predictive of gaze during film viewing. The goal of the present study was to test if the tyranny of film is driven by motion. To do this, we created a slideshow presentation of the opening scene of Touch of Evil. Context condition participants watched the full slideshow. No-context condition participants did not see the opening portion of the scene, which showed someone placing a time bomb into the trunk of a car. In prior research, we showed that despite producing very different understandings of the clip, this manipulation did not affect viewers' attention (i.e., the tyranny of film), as both context and no-context participants were equally likely to fixate on the car with the bomb when the scene was presented as a film. The current study found that when the scene was shown as a slideshow, the context manipulation produced differences in attentional selection (i.e., it attenuated attentional synchrony). We discuss these results in the context of the Scene Perception and Event Comprehension Theory, which specifies the relationship between event comprehension and attentional selection in the context of visual narratives.

Knowledge guides attention to goal-relevant information in older adults

How does viewers’ knowledge guide their attention while they watch everyday events, how does it affect their memory, and does it change with age? Older adults have diminished episodic memory for everyday events, but intact semantic knowledge. Indeed, research suggests that older adults may rely on their semantic memory to offset impairments in episodic memory, and when relevant knowledge is lacking, older adults’ memory can suffer. Yet, the mechanism by which prior knowledge guides attentional selection when watching dynamic activity is unclear. To address this, we studied the influence of knowledge on attention and memory for everyday events in young and older adults by tracking their eyes while they watched videos. The videos depicted activities that older adults perform more frequently than young adults (balancing a checkbook, planting flowers) or activities that young adults perform more frequently than older adults (installing a printer, setting up a video game). Participants completed free recall, recognition, and order memory tests after each video. We found age-related memory deficits when older adults had little knowledge of the activities, but memory did not differ between age groups when older adults had relevant knowledge and experience with the activities. Critically, results showed that knowledge influenced where viewers fixated when watching the videos. Older adults fixated less goal-relevant information compared to young adults when watching young adult activities, but they fixated goal-relevant information similarly to young adults, when watching more older adult activities. Finally, results showed that fixating goal-relevant information predicted free recall of the everyday activities for both age groups. Thus, older adults may use relevant knowledge to more effectively infer the goals of actors, which guides their attention to goal-relevant actions, thus improving their episodic memory for everyday activities.

The Incomplete Tyranny of Dynamic Stimuli: Gaze Similarity Predicts Response Similarity in Screen-Captured Instructional Videos

Although eye tracking has been used extensively to assess cognitions for static stimuli, recent research suggests that the link between gaze and cognition may be more tenuous for dynamic stimuli such as videos. Part of the difficulty in convincingly linking gaze with cognition is that in dynamic stimuli, gaze position is strongly influenced by exogenous cues such as object motion. However, tests of the gaze-cognition link in dynamic stimuli have been done on only a limited range of stimuli often characterized by highly organized motion. Also, analyses of cognitive contrasts between participants have been mostly been limited to categorical contrasts among small numbers of participants that may have limited the power to observe more subtle influences. We, therefore, tested for cognitive influences on gaze for screen-captured instructional videos, the contents of which participants were tested on. Between-participant scanpath similarity predicted between-participant similarity in responses on test questions, but with imperfect consistency across videos. We also observed that basic gaze parameters and measures of attention to centers of interest only inconsistently predicted learning, and that correlations between gaze and centers of interest defined by other-participant gaze and cursor movement did not predict learning. It, therefore, appears that the search for eye movement indices of cognition during dynamic naturalistic stimuli may be fruitful, but we also agree that the tyranny of dynamic stimuli is real, and that links between eye movements and cognition are highly dependent on task and stimulus properties.

Drivers use active gaze to monitor waypoints during automated driving

Automated vehicles (AVs) will change the role of the driver, from actively controlling the vehicle to primarily monitoring it. Removing the driver from the control loop could fundamentally change the way that drivers sample visual information from the scene, and in particular, alter the gaze patterns generated when under AV control. To better understand how automation affects gaze patterns this experiment used tightly controlled experimental conditions with a series of transitions from 'Manual' control to 'Automated' vehicle control. Automated trials were produced using either a 'Replay' of the driver's own steering trajectories or standard 'Stock' trials that were identical for all participants. Gaze patterns produced during Manual and Automated conditions were recorded and compared. Overall the gaze patterns across conditions were very similar, but detailed analysis shows that drivers looked slightly further ahead (increased gaze time headway) during Automation with only small differences between Stock and Replay trials. A novel mixture modelling method decomposed gaze patterns into two distinct categories and revealed that the gaze time headway increased during Automation. Further analyses revealed that while there was a general shift to look further ahead (and fixate the bend entry earlier) when under automated vehicle control, similar waypoint-tracking gaze patterns were produced during Manual driving and Automation. The consistency of gaze patterns across driving modes suggests that active-gaze models (developed for manual driving) might be useful for monitoring driver engagement during Automated driving, with deviations in gaze behaviour from what would be expected during manual control potentially indicating that a driver is not closely monitoring the automated system.

The Effects of Smartphone Spectatorship on Attention, Arousal, Engagement, and Comprehension

The popularity of watching movies and videos on handheld devices is rising, yet little attention has been paid to its impact on viewer behaviour. Smartphone spectatorship is characterized by the small handheld screen as well as the viewing environment where various unrelated stimuli can occur, providing possible distractions from viewing. Previous research suggests that screen size, handheld control, and external stimuli can affect viewing experience; however, no prior studies have combined these factors or applied them for the specific case of smartphones. In the present study, we compared smartphone and large-screen viewing of feature films in the presence and absence of external distractors. Using a combination of eye tracking, electrodermal activity measures, self-reports, and recollection accuracy tests, we measured smartphone-accustomed viewers' attention, arousal, engagement, and comprehension. The results revealed the impact of viewing conditions on eye movements, gaze dispersion, electrodermal activity, self-reports of engagement, as well as comprehension. These findings show that smartphone viewing is more effective when there are no distractions, and smartphone viewers are more likely to be affected by external stimuli. In addition, watching large stationary screens in designated viewing environments increases engagement with a movie.

Word skipping as an indicator of individual reading style during literary reading

Decades of research have established that the content of language (e.g. lexical characteristics of words) predicts eye movements during reading. Here we investigate whether there exist individual differences in ‘stable’ eye movement patterns during narrative reading. We computed Euclidean distances from correlations between gaze durations time courses (word level) across 102 participants who each read three literary narratives in Dutch. The resulting distance matrices were compared between narratives using a Mantel test. The results show that correlations between the scaling matrices of different narratives are relatively weak (r ≤ .11) when missing data points are ignored. However, when including these data points as zero durations (i.e. skipped words), we found significant correlations between stories (r > .51). Word skipping was significantly positively associated with print exposure but not with self-rated attention and story-world absorption, suggesting that more experienced readers are more likely to skip words, and do so in a comparable fashion. We interpret this finding as suggesting that word skipping might be a stable individual eye movement pattern.

Distinct neural mechanisms of social orienting and mentalizing revealed by independent measures of neural and eye movement typicality

Extensive study of typically developing individuals and those on the autism spectrum has identified a large number of brain regions associated with our ability to navigate the social world. Although it is widely appreciated that this so-called "social brain" is composed of distinct, interacting systems, these component parts have yet to be clearly elucidated. Here we used measures of eye movement and neural typicality-based on the degree to which subjects deviated from the norm-while typically developing (N = 62) and individuals with autism (N = 36) watched a large battery of movies depicting social interactions. Our findings provide clear evidence for distinct, but overlapping, neural systems underpinning two major components of the "social brain," social orienting, and inferring the mental state of others.

Editors’ Introduction and Review: Visual Narrative Research: An Emerging Field in Cognitive Science

Drawn sequences of images are among our oldest records of human intelligence, appearing on cave paintings, wall carvings, and ancient pottery, and they pervade across cultures from instruction manuals to comics. They also appear prevalently as stimuli across Cognitive Science, for studies of temporal cognition, event structure, social cognition, discourse, and basic intelligence. Yet, despite this fundamental place in human expression and research on cognition, the study of visual narratives themselves has only recently gained traction in Cognitive Science. This work has suggested that visual narrative comprehension requires cultural exposure across a developmental trajectory and engages with domain‐general processing mechanisms shared by visual perception, attention, event cognition, and language, among others. Here, we review the relevance of such research for the broader Cognitive Science community, and make the case for why researchers should join the scholarship of this ubiquitous but understudied aspect of human expression.

Drawn sequences of images, like those in comics and picture stories, are a pervasive and fundamental way that humans have communicated for millennia. Yet, the study of visual narratives has only recently gained traction in Cognitive Science. Here we explore what has held back the study of the cognition of visual narratives, and why researchers should join in scholarship of this ubiquitous aspect of expression.

The Scene Perception & Event Comprehension Theory (SPECT) Applied to Visual Narratives

Understanding how people comprehend visual narratives (including picture stories, comics, and film) requires the combination of traditionally separate theories that span the initial sensory and perceptual processing of complex visual scenes, the perception of events over time, and comprehension of narratives. Existing piecemeal approaches fail to capture the interplay between these levels of processing. Here, we propose the Scene Perception & Event Comprehension Theory (SPECT), as applied to visual narratives, which distinguishes between front‐end and back‐end cognitive processes. Front‐end processes occur during single eye fixations and are comprised of attentional selection and information extraction. Back‐end processes occur across multiple fixations and support the construction of event models, which reflect understanding of what is happening now in a narrative (stored in working memory) and over the course of the entire narrative (stored in long‐term episodic memory). We describe relationships between front‐ and back‐end processes, and medium‐specific differences that likely produce variation in front‐end and back‐end processes across media (e.g., picture stories vs. film). We describe several novel research questions derived from SPECT that we have explored. By addressing these questions, we provide greater insight into how attention, information extraction, and event model processes are dynamically coordinated to perceive and understand complex naturalistic visual events in narratives and the real world.

Comprehension of visual narratives like comics, picture stories, and films involves both decoding the visual content and construing the meaningful events they represent. The Scene Perception & Event Comprehension Theory (SPECT) proposes a framework for understanding how a comprehender perceptually negotiates the surface of a visual representation and integrates its meaning into a growing mental model.

Understanding Moment-to-Moment Processing of Visual Narratives

What role do moment‐to‐moment comprehension processes play in visual attentional selection in picture stories? The current work uniquely tested the role of bridging inference generation processes on eye movements while participants viewed picture stories. Specific components of the Scene Perception and Event Comprehension Theory (SPECT) were tested. Bridging inference generation was induced by manipulating the presence of highly inferable actions embedded in picture stories. When inferable actions are missing, participants have increased viewing times for the immediately following critical image (Magliano, Larson, Higgs, & Loschky, 2016). This study used eye‐tracking to test competing hypotheses about the increased viewing time: (a) Computational Load: inference generation processes increase overall computational load, producing longer fixation durations; (b) Visual Search: inference generation processes guide eye‐movements to pick up inference‐relevant information, producing more fixations. Participants had similar fixation durations, but they made more fixations while generating inferences, with that process starting from the fifth fixation. A follow‐up hypothesis predicted that when generating inferences, participants fixate scene regions important for generating the inference. A separate group of participants rated the inferential‐relevance of regions in the critical images, and results showed that these inferentially relevant regions predicted differences in other viewers’ eye movements. Thus, viewers’ event models in working memory affect visual attentional selection while viewing visual narratives.

Blink synchronization is an indicator of interest while viewing videos

The temporal pattern of spontaneous blinks changes greatly depending on an individual's internal cognitive state. For instance, when several individuals watch the same video, blinks can be synchronized at attentional breakpoints. The present study examined the degree of this blink synchronization, as reflecting an interest level, while viewing various video clips. In the first experiment, participants interested in soccer, shogi (Japanese chess), or a specific musical group watched a video clip related to each category and rated their interest level after viewing. Results revealed that blink synchronization increased with a rise in interest level in the video clips of soccer and shogi. Moreover, while blink synchronization increased when viewing preferred video clips for the soccer and music group fans, synchronization decreased when viewing videos from the other categories, except for the shogi fans. In contrast, the blink rates did not correlate with the interest level on the video content but changed with the number of shot transitions of it. In the second experiment, participants viewed a video in which a professional salesperson gave descriptions of several products for a few minutes each. When participants reported an interest in the product, blinks were synchronized to the salesperson's blinks. However, when feeling uninterested, blink synchronization did not occur. These results suggest that blink synchronization could be used as an involuntary index to assess a person's interest.