The Scene Perception & Event Comprehension Theory (SPECT) Applied to Visual Narratives

Narrative predicts cardiac synchrony in audiences

Audio-visual media possesses a remarkable ability to synchronise audiences' neural, behavioural, and physiological responses. This synchronisation is considered to reflect some dimension of collective attention or engagement with the stimulus. But what is it about these stimuli that drives such strong engagement? There are several properties of media stimuli which may lead to synchronous audience response: from low-level audio-visual features, to the story itself. Here, we present a study which separates low-level features from narrative by presenting participants with the same content but in separate modalities. In this way, the presentations shared no low-level features, but participants experienced the same narrative. We show that synchrony in participants' heart rate can be driven by the narrative information alone. We computed both visual and auditory perceptual saliency for the content and found that narrative was approximately 10 times as predictive of heart rate as low-level saliency, but that low-level audio-visual saliency has a small additive effect towards heart rate. Further, heart rate synchrony was related to a separate cohorts' continuous ratings of immersion, and that synchrony is likely to be higher at moments of increased narrative importance. Our findings demonstrate that high-level narrative dominates in the alignment of physiology across viewers.

"All the Stars Will Be Wells with a Rusty Pulley": Neural Processing of the Social and Pragmatic Content in a Narrative

Making sense of natural language and narratives requires building and manipulating a situation model by adding incoming information to the model and using the context stored in the model to comprehend subsequent details and events. Situation model maintenance is supported by the default mode network (DMN), but comprehension of the individual moments in the narrative relies on access to the conceptual store within the semantic system. The present study examined how these systems are engaged by different narrative content to investigate whether highly informative, or semantic, content is a particularly strong driver of semantic system activation compared with contextually driven content that requires using the situation model, which might instead engage DMN regions. The study further investigated which subregions of the graded semantic hub in the left anterior temporal lobe (ATL) were engaged by the type of narrative content. To do this, we quantified the semantic, pragmatic, social, ambiguous, and emotional content for each sentence in a complete narrative, the English translation of The Little Prince. Increased activation in the transmodal hub in the ventral ATL was only observed for high semantic (i.e., informative) relative to low semantic sentences. Activation in the dorsolateral and ventrolateral ATL subregions was observed for both high relative to low semantic and social content sentences, but the ventrolateral ATL effects were more extensive in the social condition. There was high correspondence between the social and pragmatic content results, particularly in the ventrolateral ATL. We argue that the ventrolateral ATL may be particularly engaged by internal, or endogenous, processing demands, aided by functional connections between the anterior middle temporal gyrus and the DMN. Pragmatic and social content may have driven endogenous processing given the pervasive and plot-progressing nature of this content in the narrative. We put forward a revised account of how the semantic system is engaged in naturalistic contexts, a critical step toward better understanding real-world semantic and social processing.

Construction or updating? Event model processes during visual narrative comprehension

The plot of a narrative is represented in the form of event models in working memory. Because only parts of the plot are actually presented and information is continually changing, comprehenders have to infer a good portion of a narrative and keep their mental representation updated. Research has identified two related processes (e.g., Gernsbacher, 1997): During model construction (shifting, laying a foundation) at large coherence breaks an event model is completely built anew. During model updating (mapping) at smaller omissions, however, the current event model is preserved, and only changed parts are updated through inference processes. Thus far, reliably distinguishing those two processes in visual narratives like comics was difficult. We report a study (N = 80) that aimed to map the differences between constructing and updating event models in visual narratives by combining measures from narrative comprehension and event cognition research and manipulating event structure. Participants watched short visual narratives designed to (not) contain event boundaries at larger coherence breaks and elicit inferences through small omissions, while we collected viewing time measures as well as event segmentation and comprehensibility data. Viewing time, segmentation, and comprehensibility data were in line with the assumption of two distinct comprehension processes. We thus found converging evidence across multiple measures for distinct model construction and updating processes in visual narratives.

Bridging a Gap in Coherence: The Coordination of Comprehension Processes When Viewing Visual Narratives

Scene Perception and Event Comprehension Theory (SPECT) posits that understanding picture stories depends upon a coordination of two processes: (1) integrating new information into the current event model that is coherent with it (i.e., mapping) and (2) segmenting experiences into distinct event models (i.e., shifting). In two experiments, we investigated competing hypotheses regarding how viewers coordinate the mapping process of bridging inference generation and the shifting process of event segmentation by manipulating the presence/absence of Bridging Action pictures (i.e., creating coherence gaps) in wordless picture stories. The Computational Effort Hypothesis says that experiencing a coherence gap prompts event segmentation and the additional computational effort to generate bridging inferences. Thus, it predicted a positive relationship between event segmentation and explanations when Bridging Actions were absent. Alternatively, the Coherence Gap Resolution Hypothesis says that experiencing a coherence gap prompt generating a bridging inference to close the gap, which obviates segmentation. Thus, it predicted a negative relationship between event segmentation and the production of explanations. Replicating prior work, viewers were more likely to segment and generate explanations when Bridging Action pictures were absent than when they were present. Crucially, the relationship between explanations and segmentation was negative when Bridging Action pictures were absent, consistent with the Coherence Gap Resolution Hypothesis. Unexpectedly, the relationship was positive when Bridging Actions were present. The results are consistent with SPECT's assumption that mapping and shifting processes are coordinated, but how they are coordinated depends upon the experience of a coherence gap.

The Language Network Reliably "Tracks" Naturalistic Meaningful Nonverbal Stimuli

The language network, comprised of brain regions in the left frontal and temporal cortex, responds robustly and reliably during language comprehension but shows little or no response during many nonlinguistic cognitive tasks (e.g., Fedorenko & Blank, 2020). However, one domain whose relationship with language remains debated is semantics-our conceptual knowledge of the world. Given that the language network responds strongly to meaningful linguistic stimuli, could some of this response be driven by the presence of rich conceptual representations encoded in linguistic inputs? In this study, we used a naturalistic cognition paradigm to test whether the cognitive and neural resources that are responsible for language processing are also recruited for processing semantically rich nonverbal stimuli. To do so, we measured BOLD responses to a set of ∼5-minute-long video and audio clips that consisted of meaningful event sequences but did not contain any linguistic content. We then used the intersubject correlation (ISC) approach (Hasson et al., 2004) to examine the extent to which the language network "tracks" these stimuli, that is, exhibits stimulus-related variation. Across all the regions of the language network, meaningful nonverbal stimuli elicited reliable ISCs. These ISCs were higher than the ISCs elicited by semantically impoverished nonverbal stimuli (e.g., a music clip), but substantially lower than the ISCs elicited by linguistic stimuli. Our results complement earlier findings from controlled experiments (e.g., Ivanova et al., 2021) in providing further evidence that the language network shows some sensitivity to semantic content in nonverbal stimuli.

Working memory capacity for continuous events: The root of temporal compression in episodic memory?

Remembering the unfolding of past episodes usually takes less time than their actual duration. In this study, we evaluated whether such temporal compression emerges when continuous events are too long to be fully held in working memory. To do so, we asked 90 young adults to watch and mentally replay video clips showing people performing a continuous action (e.g., turning a car jack) that lasted 3, 6, 9, 12, or 15 s. For each clip, participants had to carefully watch the event and then to mentally replay it as accurately and precisely as possible. Results showed that mental replay durations increased with event duration but in a non-linear manner: they were close to the actual event duration for short videos (3-9 s), but significantly smaller for longer videos (12 and 15 s). These results suggest that working memory is temporally limited in its capacity to represent continuous events, which could in part explain why the unfolding of events is temporally compressed in episodic memory.

Eye movements and event segmentation: Eye movements reveal age-related differences in event model updating

People spontaneously segment continuous ongoing actions into sequences of events. Prior research found that gaze similarity and pupil dilation increase at event boundaries and that older adults segment more idiosyncratically than do young adults. We used eye tracking to explore age-related differences in gaze similarity (i.e., the extent to which individuals look at the same places at the same time as others) and pupil dilation at event boundaries. Older and young adults watched naturalistic videos of actors performing everyday activities while we tracked their eye movements. Afterward, they segmented the videos into subevents. Replicating prior work, we found that pupil size and gaze similarity increased at event boundaries. Thus, there were fewer individual differences in eye position at boundaries. We also found that young adults had higher gaze similarity than older adults throughout an entire video and at event boundaries. This study is the first to show that age-related differences in how people parse continuous everyday activities into events may be partially explained by individual differences in gaze patterns. Those who segment less normatively may do so because they fixate less normative regions. Results have implications for future interventions designed to improve encoding in older adults. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Anaphoric distance dependencies in visual narrative structure and processing

Linguistic syntax has often been claimed as uniquely complex due to features like anaphoric relations and distance dependencies. However, visual narratives of sequential images, like those in comics, have been argued to use sequencing mechanisms analogous to those in language. These narrative structures include "refiner" panels that "zoom in" on the contents of another panel. Similar to anaphora in language, refiners indexically connect inexplicit referential information in one unit (refiner, pronoun) to a more informative "antecedent" elsewhere in the discourse. Also like in language, refiners can follow their antecedents (anaphoric) or precede them (cataphoric), along with having either proximal or distant connections. We here explore the constraints on visual narrative refiners created by modulating these features of order and distance. Experiment 1 examined participants' preferences for where refiners are placed in a sequence using a force-choice test, which revealed that refiners are preferred to follow their antecedents and have proximal distances from them. Experiment 2 then showed that distance dependencies lead to slower self-paced viewing times. Finally, measurements of event-related brain potentials (ERPs) in Experiment 3 revealed that these patterns evoke similar brain responses as referential dependencies in language (i.e., N400, LAN, Nref). Across all three studies, the constraints and (neuro)cognitive responses to refiners parallel those shown to anaphora in language, suggesting domain-general constraints on the sequencing of referential dependencies.

Background check: cross-cultural differences in the spatial context of comic scenes

Cognitive research points towards cultural differences in the way people perceive and express scenes. Whereas people from Western cultures focus more on focal objects, those from East Asia have been shown to focus on the surrounding context. This paper examines whether these cultural differences are expressed in complex multimodal media such as comics. We compared annotated panels across comics from six countries to examine how backgrounds convey contextual information of scenes in explicit or implicit ways. Compared to Western comics from the United States and Spain, East Asian comics from Japan and China expressed the context of scenes more implicitly. In addition, Nigerian comics moderately emulated American comics in background use, while Russian comics emulated Japanese manga, consistent with their visual styles. The six countries grouped together based on whether they employed more explicit strategies such as detailed, depicted backgrounds, or implicit strategies such as leaving the background empty. These cultural differences in background use can be attributed to both cognitive patterns of attention and comics' graphic styles. Altogether, this study provides support for cultural differences in attention manifesting in visual narratives, and elucidates how spatial relationships are depicted in visual narratives across cultures.

Stories in the Mind? The Role of Story-Based Categorizations in Motion Classification

Categorization is fundamental for spatial and motion representation in both the domain of artificial intelligence and human cognition. In this paper, we investigated whether motion categorizations designed in artificial intelligence can inform human cognition. More concretely, we investigated if such categorizations (also known as qualitative representations) can inform the psychological understanding of human perception and memory of motion scenes. To this end, we took two motion categorizations in artificial intelligence, Motion-RCC and Motion-OPRA1 , and conducted four experiments on human perception and memory. Participants viewed simple motion scenes and judged the similarity of transformed scenes with this reference scene. Those transformed scenes differed in none, one, or both Motion-RCC and Motion-OPRA1 categories. Importantly, we applied an equal absolute metric change to those transformed scenes, so that differences in the similarity judgments should be due only to differing categories. In Experiments 1a and 1b, where the reference stimulus and transformed stimuli were visible at the same time (perception), both Motion-OPRA1 and Motion-RCC influenced the similarity judgments, with a stronger influence of Motion-OPRA1 . In Experiments 2a and 2b, where participants first memorized the reference stimulus and viewed the transformed stimuli after a short blank (memory), only Motion-OPRA1 had marked influences on the similarity judgments. Our findings demonstrate a link between human cognition and these motion categorizations developed in artificial intelligence. We argue for a continued and close multidisciplinary approach to investigating the representation of motion scenes.

Influence of prior knowledge on eye movements to scenes as revealed by hidden Markov models

Human visual experience usually provides ample opportunity to accumulate knowledge about events unfolding in the environment. In typical scene perception experiments, however, participants view images that are unrelated to each other and, therefore, they cannot accumulate knowledge relevant to the upcoming visual input. Consequently, the influence of such knowledge on how this input is processed remains underexplored. Here, we investigated this influence in the context of gaze control. We used sequences of static film frames arranged in a way that allowed us to compare eye movements to identical frames between two groups: a group that accumulated prior knowledge relevant to the situations depicted in these frames and a group that did not. We used a machine learning approach based on hidden Markov models fitted to individual scanpaths to demonstrate that the gaze patterns from the two groups differed systematically and, thereby, showed that recently accumulated prior knowledge contributes to gaze control. Next, we leveraged the interpretability of hidden Markov models to characterize these differences. Additionally, we report two unexpected and interesting caveats of our approach. Overall, our results highlight the importance of recently acquired prior knowledge for oculomotor control and the potential of hidden Markov models as a tool for investigating it.

Where to look at the movies: Analyzing visual attention to understand movie editing

In the process of making a movie, directors constantly care about where the spectator will look on the screen. Shot composition, framing, camera movements, or editing are tools commonly used to direct attention. In order to provide a quantitative analysis of the relationship between those tools and gaze patterns, we propose a new eye-tracking database, containing gaze-pattern information on movie sequences, as well as editing annotations, and we show how state-of-the-art computational saliency techniques behave on this dataset. In this work, we expose strong links between movie editing and spectators gaze distributions, and open several leads on how the knowledge of editing information could improve human visual attention modeling for cinematic content. The dataset generated and analyzed for this study is available at https://github.com/abruckert/eye_tracking_filmmaking.

Prior knowledge about events depicted in scenes decreases oculomotor exploration

The visual input that the eyes receive usually contains temporally continuous information about unfolding events. Therefore, humans can accumulate knowledge about their current environment. Typical studies on scene perception, however, involve presenting multiple unrelated images and thereby render this accumulation unnecessary. Our study, instead, facilitated it and explored its effects. Specifically, we investigated how recently-accumulated prior knowledge affects gaze behavior. Participants viewed sequences of static film frames that contained several 'context frames' followed by a 'critical frame'. The context frames showed either events from which the situation depicted in the critical frame naturally followed, or events unrelated to this situation. Therefore, participants viewed identical critical frames while possessing prior knowledge that was either relevant or irrelevant to the frames' content. In the former case, participants' gaze behavior was slightly more exploratory, as revealed by seven gaze characteristics we analyzed. This result demonstrates that recently-gained prior knowledge reduces exploratory eye movements.

The audience who knew too much: investigating the role of spontaneous theory of mind on the processing of dramatic irony scenes in film

As in real life, cinema viewers rely on spontaneous theory of mind (SToM) to interpret characters' mental states. Thus, analyzing cinematic structures offers a unique opportunity to examine ecologically valid sociocognitive processes. We conducted a proof-of-concept study (N = 42) to explore how SToM inferences impact film event comprehension in dramatic irony scenes, where knowledge divergence exists between the audience and characters. We hypothesized that spectators would focus more on characters' mental states in such false-belief inducing scenarios compared to scenarios without such disparity. We used six Harold Lloyd silent comedy clips in a narrative comprehension and spontaneous mental state attribution study with a between-subject (Knowledge Manipulation: Installation vs. Control) and within-subject (Phase: Context vs. Exploitation) comparisons. We provided critical information unknown to the characters only to the Installation group and withheld it from the Control group. By comparing differences in participants' descriptions of the clips during the Context phase (varying across groups) and Exploitation phase (same across groups), we evaluated viewers' processing of the same scenes based on their false- or true-belief representations. Our findings indicate that the Installation group used more cognitive mental state words during the Exploitation phase relative to the Context phase, suggesting that exposure to undisclosed critical information enhances the frequency of spontaneous epistemic state inferences and integration into event models of the exploitation. This research advances neurocinematics by highlighting spontaneous sociocognitive processes in event perception and comprehension and provides a novel dramatic irony film corpus and measures for future moment-to-moment SToM processing studies across cognitive-behavioral, physiological, and neural levels.

The framing of subjectivity: Point-of-view in a cross-cultural analysis of comics

In visual narratives like comics, the most overt form of perspective-taking comes in panels that directly depict the viewpoints of characters in the scene. We therefore examined these subjective viewpoint panels (also known as point-of-view panels) in a corpus of over 300 annotated comics from Asia, Europe, and the United States. In line with predictions that Japanese manga use a more 'subjective' storytelling style than other comics, we found that more manga use subjective panels than other comics, with high proportions of subjective panels also found in Chinese, French, and American comics. In addition, panels with more 'focal' framing, i.e. micro panels showing close ups and/or amorphic panels showing views of the environment, had higher proportions of subjective panels than panels showing wider views of scenes. These findings further show that empirical corpus analyses provide evidence of cross-cultural variation and reveal relationships across structures in the visual languages of comics.

Modulation of Spectral Representation and Connectivity Patterns in Response to Visual Narrative in the Human Brain

The regional brain networks and the underlying neurophysiological mechanisms subserving the cognition of visual narrative in humans have largely been studied with non-invasive brain recording. In this study, we specifically investigated how regional and cross-regional cortical activities support visual narrative interpretation using intracranial stereotactic electroencephalograms recordings from thirteen human subjects (6 females, and 7 males). Widely distributed recording sites across the brain were sampled while subjects were explicitly instructed to observe images from fables presented in “sequential” order, and a set of images drawn from multiple fables presented in “scrambled” order. Broadband activity mainly within the frontal and temporal lobes were found to encode if a presented image is part of a visual narrative (sequential) or random image set (scrambled). Moreover, the temporal lobe exhibits strong activation in response to visual narratives while the frontal lobe is more engaged when contextually novel stimuli are presented. We also investigated the dynamics of interregional interactions between visual narratives and contextually novel series of images. Interestingly, the interregional connectivity is also altered between sequential and scrambled sequences. Together, these results suggest that both changes in regional neuronal activity and cross-regional interactions subserve visual narrative and contextual novelty processing.

Application of Spatial Cues and Optical Distortions as Augmentations during Virtual Reality (VR) Gaming: The Multifaceted Effects of Assistance for Eccentric Viewing Training

The present study investigates the effects of peripheral spatial cues and optically distorting augmentations over eccentric vision mechanisms in normally sighted participants with simulated scotoma. Five different augmentations were tested inside a virtual reality (VR)-gaming environment. Three were monocular spatial cues, and two were binocular optical distortions. Each was divided into three conditions: baseline with normal viewing, augmentation with one of the assistance methods positioned around the scotoma, and one with only the simulated central scotoma. The study found that the gaming scenario induced eccentric viewing for the cued augmentation groups, even when the peripheral assistance was removed, while for the optical distortions group, the eccentric behavior disappeared after the augmentation removal. Additionally, an upwards directionality of gaze relative to target during regular gaming was found. The bias was maintained and implemented during and after the cued augmentations but not after the distorted ones. The results suggest that monocular peripheral cues could be better candidates for implementing eccentric viewing training in patients. At the same time, it showed that optical distortions might disrupt such behavior. Such results are noteworthy since distortions such as zoom are known to help patients with macular degeneration see targets of interest.

Narrative Comprehension Guides Eye Movements in the Absence of Motion

Viewers' attentional selection while looking at scenes is affected by both top-down and bottom-up factors. However, when watching film, viewers typically attend to the movie similarly irrespective of top-down factors-a phenomenon we call the tyranny of film. A key difference between still pictures and film is that film contains motion, which is a strong attractor of attention and highly predictive of gaze during film viewing. The goal of the present study was to test if the tyranny of film is driven by motion. To do this, we created a slideshow presentation of the opening scene of Touch of Evil. Context condition participants watched the full slideshow. No-context condition participants did not see the opening portion of the scene, which showed someone placing a time bomb into the trunk of a car. In prior research, we showed that despite producing very different understandings of the clip, this manipulation did not affect viewers' attention (i.e., the tyranny of film), as both context and no-context participants were equally likely to fixate on the car with the bomb when the scene was presented as a film. The current study found that when the scene was shown as a slideshow, the context manipulation produced differences in attentional selection (i.e., it attenuated attentional synchrony). We discuss these results in the context of the Scene Perception and Event Comprehension Theory, which specifies the relationship between event comprehension and attentional selection in the context of visual narratives.

Understanding the differential impact of children's TV on executive functions: a narrative-processing analysis

Evidence from multiple empirical studies suggests children's Executive Functions are depleted immediately after viewing some types of TV content but not others. Correlational evidence suggests any such effects may be most problematic during the pre-school years. To establish whether "screen-time" is developmentally appropriate at this age we believe a nuanced approach must be taken to the analysis of individual pieces of media and their potential demands on viewer cognition. To this end we apply a cognitive theory of visual narrative processing, the Scene Perception and Event Comprehension Theory (SPECT; Loschky, Larson, Smith, & Magliano, 2020) to the analysis of TV shows previously used to investigate short-term effects of TV viewing. A theoretical formalisation of individual content properties, together with a quantitative content-based analysis of previously used children's content (Lillard & Peterson, 2011; Lillard et al., 2015b) is presented. This analysis found a pattern of greater stimulus saliency, increased situational change and a greater combined presence of cognitively demanding features for videos previously shown to reduce children's EF after viewing. Limitations of this pilot application of SPECT are presented and proposals for future empirical investigations of the psychological mechanisms activated by specific TV viewing content are considered.

Constructing Expertise: Surmounting Performance Plateaus by Tasks, by Tools, and by Techniques

Acquiring expertise in a task is often thought of as an automatic process that follows inevitably with practice according to the log-log law (aka: power law) of learning. However, as Ericsson, Chase, and Faloon (1980) showed, this is not true for digit-span experts and, as we show, it is certainly not true for Tetris players at any level of expertise. Although some people may simply "twitch" faster than others, the limit to Tetris expertise is not raw keypress time but the techniques acquired by players that allow them to use the tools provided by the hardware and software to compensate for the game's relentlessly increasing drop speed. Unfortunately, these increases in drop speed between Tetris levels make performance plateaus very short and quickly followed by game death. Hence, a player's success at discovering, exploring, and practicing new techniques for the tasks of board preparation, board maintenance, optimal placement discovery, zoid rotation, lateral movement of zoids, and other tasks important to expertise in Tetris is limited. In this paper, we analyze data collected from 492 Tetris players to reveal the challenges they confronted while constructing expertise via the discovery of new techniques for gameplay at increasingly difficult levels of Tetris.

Knowledge guides attention to goal-relevant information in older adults

How does viewers’ knowledge guide their attention while they watch everyday events, how does it affect their memory, and does it change with age? Older adults have diminished episodic memory for everyday events, but intact semantic knowledge. Indeed, research suggests that older adults may rely on their semantic memory to offset impairments in episodic memory, and when relevant knowledge is lacking, older adults’ memory can suffer. Yet, the mechanism by which prior knowledge guides attentional selection when watching dynamic activity is unclear. To address this, we studied the influence of knowledge on attention and memory for everyday events in young and older adults by tracking their eyes while they watched videos. The videos depicted activities that older adults perform more frequently than young adults (balancing a checkbook, planting flowers) or activities that young adults perform more frequently than older adults (installing a printer, setting up a video game). Participants completed free recall, recognition, and order memory tests after each video. We found age-related memory deficits when older adults had little knowledge of the activities, but memory did not differ between age groups when older adults had relevant knowledge and experience with the activities. Critically, results showed that knowledge influenced where viewers fixated when watching the videos. Older adults fixated less goal-relevant information compared to young adults when watching young adult activities, but they fixated goal-relevant information similarly to young adults, when watching more older adult activities. Finally, results showed that fixating goal-relevant information predicted free recall of the everyday activities for both age groups. Thus, older adults may use relevant knowledge to more effectively infer the goals of actors, which guides their attention to goal-relevant actions, thus improving their episodic memory for everyday activities.

A starring role for inference in the neurocognition of visual narratives

Research in verbal and visual narratives has often emphasized backward-looking inferences, where absent information is subsequently inferred. However, comics use conventions like star-shaped “action stars” where a reader knows events are undepicted at that moment, rather than omitted entirely. We contrasted the event-related brain potentials (ERPs) to visual narratives depicting an explicit event, an action star, or a “noise” panel of scrambled lines. Both action stars and noise panels evoked large N400s compared to explicit-events (300–500 ms), but action stars and noise panels then differed in their later effects (500–900 ms). Action stars elicited sustained negativities and P600s, which could indicate further interpretive processes and integration of meaning into a mental model, while noise panels evoked late frontal positivities possibly indexing that they were improbable narrative units. Nevertheless, panels following action stars and noise panels both evoked late sustained negativities, implying further inferential processing. Inference in visual narratives thus uses cascading mechanisms resembling those in language processing that differ based on the inferential techniques.

Cortical Activity Linked to Clocking in Deaf Adults: fNIRS Insights with Static and Animated Stimuli Presentation

The question of the possible impact of deafness on temporal processing remains unanswered. Different findings, based on behavioral measures, show contradictory results. The goal of the present study is to analyze the brain activity underlying time estimation by using functional near infrared spectroscopy (fNIRS) techniques, which allow examination of the frontal, central and occipital cortical areas. A total of 37 participants (19 deaf) were recruited. The experimental task involved processing a road scene to determine whether the driver had time to safely execute a driving task, such as overtaking. The road scenes were presented in animated format, or in sequences of 3 static images showing the beginning, mid-point, and end of a situation. The latter presentation required a clocking mechanism to estimate the time between the samples to evaluate vehicle speed. The results show greater frontal region activity in deaf people, which suggests that more cognitive effort is needed to process these scenes. The central region, which is involved in clocking according to several studies, is particularly activated by the static presentation in deaf people during the estimation of time lapses. Exploration of the occipital region yielded no conclusive results. Our results on the frontal and central regions encourage further study of the neural basis of time processing and its links with auditory capacity.

Zooming in on visual narrative comprehension

The comprehension of visual narratives requires paying attention to certain elements and integrating them across a sequence of images. To study this process, we developed a new approach that modified comic strips according to where observers looked while viewing each sequence. Across three self-paced experiments, we presented sequences of six panels that were sometimes automatically "zoomed-in" or re-framed in order to highlight parts of the image that had been fixated by another group of observers. Fixation zoom panels were rated as easier to understand and produced viewing times more similar to the original comic than panels modified to contain non-fixated or incongruous regions. When a single panel depicting the start of an action was cropped to show only the most fixated region, viewing times were similar to the original narrative despite the reduced information. Modifying such panels also had an impact on the viewing time on subsequent panels, both when zoomed in and when regions were highlighted through an "inset" panel. These findings demonstrate that fixations in a visual narrative are guided to informative elements, and that these elements influence both the current panel and the processing of the sequence.

Cross-codal integration of bridging-event information in narrative understanding

Visual narratives communicate event sequences by using different code systems such as pictures and texts. Thus, comprehenders must integrate information from different codalities. This study addressed such cross-codal integration processes by investigating how the codality of bridging-event information (i.e., pictures, text) affects the understanding of visual narrative events. In Experiment 1, bridging-event information was either present (as picture or text) or absent (i.e., not shown). The viewing times for the subsequent picture depicting the end state of the action were comparable within the absent and the text conditions. Further, the viewing times for the end-state picture were significantly longer in the text condition as compared to the pictorial condition. In Experiment 2, we tested whether replacing bridging-event information with a blank panel increases viewing times in a way similar to the text condition. Bridging event information was either present (as picture) or absent (not shown vs. blank panel). The results replicated Experiment 1. Additionally, the viewing times for the end-state pictures were longest in the blank condition. In Experiment 3, we investigated the costs related to integrating information from different codalities by directly comparing the text and picture conditions with the blank condition. The results showed that the distortion caused by the blank panel is larger than the distortion caused by cross-codal integration processes. Summarizing, we conclude that cross-codal information processing during narrative comprehension is possible but associated with additional mental effort. We discuss the results with regard to theories of narrative understanding.

The influence of sequential predictions on scene-gist recognition

Past research suggests that recognizing scene gist, a viewer's holistic semantic representation of a scene acquired within a single eye fixation, involves purely feed-forward mechanisms. We investigated whether expectations can influence scene categorization. To do this, we embedded target scenes in more ecologically valid, first-person-viewpoint image sequences, along spatiotemporally connected routes (e.g., an office to a parking lot). We manipulated the sequences' spatiotemporal coherence by presenting them either coherently or in random order. Participants identified the category of one target scene in a 10-scene-image rapid serial visual presentation. Categorization accuracy was greater for targets in coherent sequences. Accuracy was also greater for targets with more visually similar primes. In Experiment 2, we investigated whether targets in coherent sequences were more predictable and whether predictable images were identified more accurately in Experiment 1 after accounting for the effect of prime-to-target visual similarity. To do this, we removed targets and had participants predict the category of the missing scene. Images were more accurately predicted in coherent sequences, and both image predictability and prime-to-target visual similarity independently contributed to performance in Experiment 1. To test whether prediction-based facilitation effects were solely due to response bias, participants performed a two-alternative forced-choice task in which they indicated whether the target was an intact or a phase-randomized scene. Critically, predictability of the target category was irrelevant to this task. Nevertheless, results showed that sensitivity, but not response bias, was greater for targets in coherent sequences. Predictions made prior to viewing a scene facilitate scene-gist recognition.

Editors’ Introduction and Review: Visual Narrative Research: An Emerging Field in Cognitive Science

Drawn sequences of images are among our oldest records of human intelligence, appearing on cave paintings, wall carvings, and ancient pottery, and they pervade across cultures from instruction manuals to comics. They also appear prevalently as stimuli across Cognitive Science, for studies of temporal cognition, event structure, social cognition, discourse, and basic intelligence. Yet, despite this fundamental place in human expression and research on cognition, the study of visual narratives themselves has only recently gained traction in Cognitive Science. This work has suggested that visual narrative comprehension requires cultural exposure across a developmental trajectory and engages with domain‐general processing mechanisms shared by visual perception, attention, event cognition, and language, among others. Here, we review the relevance of such research for the broader Cognitive Science community, and make the case for why researchers should join the scholarship of this ubiquitous but understudied aspect of human expression.

Drawn sequences of images, like those in comics and picture stories, are a pervasive and fundamental way that humans have communicated for millennia. Yet, the study of visual narratives has only recently gained traction in Cognitive Science. Here we explore what has held back the study of the cognition of visual narratives, and why researchers should join in scholarship of this ubiquitous aspect of expression.

Your Brain on Comics: A Cognitive Model of Visual Narrative Comprehension

The past decade has seen a rapid growth of cognitive and brain research focused on visual narratives like comics and picture stories. This paper will summarize and integrate this emerging literature into the Parallel Interfacing Narrative‐Semantics Model (PINS Model)—a theory of sequential image processing characterized by an interaction between two representational levels: semantics and narrative structure. Ongoing semantic processes build meaning into an evolving mental model of a visual discourse. Updating of spatial, referential, and event information then incurs costs when they are discontinuous with the growing context. In parallel, a narrative structure organizes semantic information into coherent sequences by assigning images to categorical roles, which are then embedded within a hierarchic constituent structure. Narrative constructional schemas allow for specific predictions of structural sequencing, independent of semantics. Together, these interacting levels of representation engage in an iterative process of retrieval of semantic and narrative information, prediction of upcoming information based on those assessments, and subsequent updating based on discontinuity. These core mechanisms are argued to be domain‐general—spanning across expressive systems—as suggested by similar electrophysiological brain responses (N400, P600, anterior negativities) generated in response to manipulation of sequential images, music, and language. Such similarities between visual narratives and other domains thus pose fundamental questions for the linguistic and cognitive sciences.

Visual narratives like comics involve a range of complex cognitive operations in order to be understood. The Parallel Interfacing Narrative‐Semantics (PINS) Model integrates an emerging literature showing that comprehension of wordless image sequences balances two representational levels of semantic and narrative structure. The neurocognitive mechanisms that guide these processes are argued to overlap with other domains, such as language and music.