First-time viewers' comprehension of films: bridging shot transitions

The effect of narrative coherence and visual salience on children's and adults' gaze while watching video

Low-level visual features (e.g., motion, contrast) predict eye gaze during video viewing. The current study investigated the effect of narrative coherence on the extent to which low-level visual salience predicts eye gaze. Eye movements were recorded as 4-year-olds (n = 20) and adults (n = 20) watched a cohesive versus random sequence of video shots from a 4.5-min full vignette from Sesame Street. Overall, visual salience was a stronger predictor of gaze in adults than in children, especially when viewing a random shot sequence. The impact of narrative coherence on children's gaze was limited to the short period of time surrounding cuts to new video shots. The discussion considers potential direct effects of visual salience as well as incidental effects due to overlap between salient features and semantic content. The findings are also discussed in the context of developing video comprehension.

Enacting Media. An Embodied Account of Enculturation Between Neuromediality and New Cognitive Media Theory

This paper argues that the still-emerging paradigm of situated cognition requires a more systematic perspective on media to capture the enculturation of the human mind. By virtue of being media, cultural artifacts present central experiential models of the world for our embodied minds to latch onto. The paper identifies references to external media within embodied, extended, enactive, and predictive approaches to cognition, which remain underdeveloped in terms of the profound impact that media have on our mind. To grasp this impact, I propose an enactive account of media that is based on expansive habits as media-structured, embodied ways of bringing forth meaning and new domains of values. We apply such habits, for instance, when seeing a picture or perceiving a movie. They become established through a process of reciprocal adaptation between media artifacts and organisms and define the range of viable actions within such a media ecology. Within an artifactual habit, we then become attuned to a specific media work (e.g., a TV series, a picture, a text, or even a city) that engages us. Both the plurality of habits and the dynamical adjustments within a habit require a more flexible neural architecture than is addressed by classical cognitive neuroscience. To detail how neural and media processes interlock, I will introduce the concept of neuromediality and discuss radical predictive processing accounts that could contribute to the externalization of the mind by treating media themselves as generative models of the world. After a short primer on general media theory, I discuss media examples in three domains: pictures and moving images; digital media; architecture and the built environment. This discussion demonstrates the need for a new cognitive media theory based on enactive artifactual habits-one that will help us gain perspective on the continuous re-mediation of our mind.

Editors’ Introduction and Review: Visual Narrative Research: An Emerging Field in Cognitive Science

Drawn sequences of images are among our oldest records of human intelligence, appearing on cave paintings, wall carvings, and ancient pottery, and they pervade across cultures from instruction manuals to comics. They also appear prevalently as stimuli across Cognitive Science, for studies of temporal cognition, event structure, social cognition, discourse, and basic intelligence. Yet, despite this fundamental place in human expression and research on cognition, the study of visual narratives themselves has only recently gained traction in Cognitive Science. This work has suggested that visual narrative comprehension requires cultural exposure across a developmental trajectory and engages with domain‐general processing mechanisms shared by visual perception, attention, event cognition, and language, among others. Here, we review the relevance of such research for the broader Cognitive Science community, and make the case for why researchers should join the scholarship of this ubiquitous but understudied aspect of human expression.

Drawn sequences of images, like those in comics and picture stories, are a pervasive and fundamental way that humans have communicated for millennia. Yet, the study of visual narratives has only recently gained traction in Cognitive Science. Here we explore what has held back the study of the cognition of visual narratives, and why researchers should join in scholarship of this ubiquitous aspect of expression.

The Scene Perception & Event Comprehension Theory (SPECT) Applied to Visual Narratives

Understanding how people comprehend visual narratives (including picture stories, comics, and film) requires the combination of traditionally separate theories that span the initial sensory and perceptual processing of complex visual scenes, the perception of events over time, and comprehension of narratives. Existing piecemeal approaches fail to capture the interplay between these levels of processing. Here, we propose the Scene Perception & Event Comprehension Theory (SPECT), as applied to visual narratives, which distinguishes between front‐end and back‐end cognitive processes. Front‐end processes occur during single eye fixations and are comprised of attentional selection and information extraction. Back‐end processes occur across multiple fixations and support the construction of event models, which reflect understanding of what is happening now in a narrative (stored in working memory) and over the course of the entire narrative (stored in long‐term episodic memory). We describe relationships between front‐ and back‐end processes, and medium‐specific differences that likely produce variation in front‐end and back‐end processes across media (e.g., picture stories vs. film). We describe several novel research questions derived from SPECT that we have explored. By addressing these questions, we provide greater insight into how attention, information extraction, and event model processes are dynamically coordinated to perceive and understand complex naturalistic visual events in narratives and the real world.

Comprehension of visual narratives like comics, picture stories, and films involves both decoding the visual content and construing the meaningful events they represent. The Scene Perception & Event Comprehension Theory (SPECT) proposes a framework for understanding how a comprehender perceptually negotiates the surface of a visual representation and integrates its meaning into a growing mental model.

Picture This: A Review of Research Relating to Narrative Processing by Moving Image Versus Language

Reading fiction for pleasure is robustly correlated with improved cognitive attainment and other benefits. It is also in decline among young people in developed nations, in part because of competition from moving image fiction. We review existing research on the differences between reading or hearing verbal fiction and watching moving image fiction, as well as looking more broadly at research on image or text interactions and visual versus verbal processing. We conclude that verbal narrative generates more diverse responses than moving image narrative. We note that reading and viewing narrative are different tasks, with different cognitive loads. Viewing moving image narrative mostly involves visual processing with some working memory engagement, whereas reading narrative involves verbal processing, visual imagery, and personal memory (Xu et al., 2005). Attempts to compare the two by creating equivalent stimuli and task demands face a number of challenges. We discuss the difficulties of such comparative approaches. We then investigate the possibility of identifying lower level processing mechanisms that might distinguish cognition of the two media and propose internal scene construction and working memory as foci for future research. Although many of the sources we draw on concentrate on English-speaking participants in European or North American settings, we also cover material relating to speakers of Dutch, German, Hebrew, and Japanese in their respective countries, and studies of a remote Turkish mountain community.

Effect of sequential video shot comprehensibility on attentional synchrony: A comparison of children and adults

To comprehend edited video, viewers must infer the meaning conveyed by successive video shots (i.e., continuous video segments separated by edit points, such as camera cuts). The central question here was whether comprehension-related top-down cognitive processes drive eye movements during sequential processing of video montage. Eye movements were recorded as 4 year olds and adults (n = 62) watched a video with the same constituent shots in either normal or random sequence. The key analyses compared eye movements to constituent shots when presented in normal order with those to the same shots presented in random order. The dependent variable was attentional synchrony or the extent to which viewers looked at the same location at the same time, indicating commonality of processing the video. This was calculated as the bivariate contour ellipse area within which points of gaze fell during each video frame. Results indicated that children were more scattered in their gaze locations than adults. Viewers became more similar to each other as normal vignettes unfolded over time; this was especially true in adults and possibly reflects a growing and shared understanding of the content. Conversely, adult attentional synchrony was reduced when watching random shot sequences. Thus, attentional synchrony during normal video viewing is driven not only by salient visual features, such as movement and areas of high contrast, but also, by the unfolding sequential comprehension of video montage, especially in adults. Differences between children and adults indicate that this top-down control of eye movements while watching video changes systematically over development.

Audio Facilitates the Perception of Cinematic Continuity by First-Time Viewers

Previous studies concluded that first-time film viewers often had difficulty integrating shots into a coherent representation of the depicted events in the absence of a familiar action through the film cuts or a salient eye-gazing of a character in the film. In this study, we investigated whether diegetic sound (i.e., sound that seems to originate from the depicted cinematic space) could effectively bridge shots for first-time viewers. Across a range of films, both dialog, and salient environmental sound (e.g., barking dogs) helped first-time viewers connect shots. However, sound was not always successful in supporting first-time viewers' interpretations. While experienced viewers were able to understand less-familiar linking sounds and environments, first-time viewers found this difficult. Overall, a range of diegetic sounds helped first-time viewers understand spatiotemporal relations between shots, but these viewers still had difficulty integrating views of unfamiliar environments.

What is your neural function, visual narrative conjunction? Grammar, meaning, and fluency in sequential image processing

Visual narratives sometimes depict successive images with different characters in the same physical space; corpus analysis has revealed that this occurs more often in Japanese manga than American comics. We used event-related brain potentials to determine whether comprehension of "visual narrative conjunctions" invokes not only incremental mental updating as traditionally assumed, but also, as we propose, "grammatical" combinatoric processing. We thus crossed (non)/conjunction sequences with character (in)/congruity. Conjunctions elicited a larger anterior negativity (300-500 ms) than nonconjunctions, regardless of congruity, implicating "grammatical" processes. Conjunction and incongruity both elicited larger P600s (500-700 ms), indexing updating. Both conjunction effects were modulated by participants' frequency of reading manga while growing up. Greater anterior negativity in frequent manga readers suggests more reliance on combinatoric processing; larger P600 effects in infrequent manga readers suggest more resources devoted to mental updating. As in language comprehension, it seems that processing conjunctions in visual narratives is not just mental updating but also partly grammatical, conditioned by comic readers' experience with specific visual narrative structures.

Bridging views in cinema: a review of the art and science of view integration

Recently, there has been a surge of interest in the relationship between film and cognitive science. This is reflected in a new science of cinema that can help us both to understand this art form, and to produce new insights about cognition and perception. In this review, we begin by describing how the initial development of cinema involved close observation of audience response. This allowed filmmakers to develop an informal theory of visual cognition that helped them to isolate and creatively recombine fundamental elements of visual experience. We review research exploring naturalistic forms of visual perception and cognition that have opened the door to a productive convergence between the dynamic visual art of cinema and science of visual cognition that can enrich both. In particular, we discuss how parallel understandings of view integration in cinema and in cognitive science have been converging to support a new understanding of meaningful visual experience. WIREs Cogn Sci 2017, 8:e1436. doi: 10.1002/wcs.1436 For further resources related to this article, please visit the WIREs website.

Eyeblink rate watching classical Hollywood and post-classical MTV editing styles, in media and non-media professionals

While movie edition creates a discontinuity in audio-visual works for narrative and economy-of-storytelling reasons, eyeblink creates a discontinuity in visual perception for protective and cognitive reasons. We were interested in analyzing eyeblink rate linked to cinematographic edition styles. We created three video stimuli with different editing styles and analyzed spontaneous blink rate in participants (N = 40). We were also interested in looking for different perceptive patterns in blink rate related to media professionalization. For that, of our participants, half (n = 20) were media professionals, and the other half were not. According to our results, MTV editing style inhibits eyeblinks more than Hollywood style and one-shot style. More interestingly, we obtained differences in visual perception related to media professionalization: we found that media professionals inhibit eyeblink rate substantially compared with non-media professionals, in any style of audio-visual edition.

Anticipatory Eye Movements While Watching Continuous Action Across Shots in Video Sequences: A Developmental Study

Eye movements were recorded as 12-month-olds (n = 15), 4-year-olds (n = 17), and adults (n = 19) watched a 15-min video with sequences of shots conveying continuous motion. The central question was whether, and at what age, viewers anticipate the reappearance of objects following cuts to new shots. Adults were more likely than younger viewers to make anticipatory eye movements. Four-year-olds responded to transitions more slowly and tended to fixate the center of the screen. Infants' eye movement patterns reflected a tendency to react rather than anticipate. Findings are consistent with the hypothesis that adults integrate content across shots and understand how space is represented in edited video. Results are interpreted with respect to a developing understanding of film editing due to experience and cognitive maturation.

Precis of "Flicker: Your Brain on Movies"

This article is a précis of the book Flicker: Your Brain on Movies (Zacks 2014). Flicker aims to introduce a broad readership to the psychology and neuroscience that underlies their experience in the movie theatre. The book's topics include including: emotional experience, adaptation from texts to films, memory and propaganda, movie violence, film editing, and brain stimulation. Cutting across the specific topics are a few broad themes: the evolution of the brain and mind, the role of automatically evoked responses in film viewing, and the role of behavioral and neural plasticity in everyday experience.