Picture This: A Review of Research Relating to Narrative Processing by Moving Image Versus Language

Higher levels of narrativity lead to similar patterns of posterior EEG activity across individuals

Introduction

The focus of cognitive and psychological approaches to narrative has not so much been on the elucidation of important aspects of narrative, but rather on using narratives as tools for the investigation of higher order cognitive processes elicited by narratives (e.g., understanding, empathy, etc.). In this study, we work toward a scalar model of narrativity, which can provide testable criteria for selecting and classifying communication forms in their level of narrativity. We investigated whether being exposed to videos with different levels of narrativity modulates shared neural responses, measured by inter-subject correlation, and engagement levels.

Methods

Thirty-two participants watched video advertisements with high-level and low-level of narrativity while their neural responses were measured through electroencephalogram. Additionally, participants' engagement levels were calculated based on the composite of their self-reported attention and immersion scores.

Results

Results demonstrated that both calculated inter-subject correlation and engagement scores for high-level video ads were significantly higher than those for low-level, suggesting that narrativity levels modulate inter-subject correlation and engagement.

Discussion

We believe that these findings are a step toward the elucidation of the viewers' way of processing and understanding a given communication artifact as a function of the narrative qualities expressed by the level of narrativity.

Modality Effects Examined by Means of an Online Sentence-Picture Comparison Task

Are oral sentences accompanied by pictures easier to understand than written sentences accompanied by the same pictures? This question-intensely discussed for more than two decades in educational, psychological, and psycholinguistic research in terms of modality effect in multimedia learning, split-attention effect, or visuospatial load effect-was examined by means of an online sentence-picture comparison task, with participants reading or listening to short sentences accompanied by pictures. Sentences and pictures referred to a conceptually and linguistically basic universe that was constructed from a limited number of familiar objects and characteristics. Mixed repeated-measures ANOVAs were calculated separately for simultaneously and sequentially presented sentence-picture versions, with modality (oral, written), picture complexity (complex, simple), and sentence-picture compatibility (compatible, incompatible) as within-subject factors, and age (6th-graders, adults) as between-subjects factor. The experiment was based on reaction time and acuity measures. The online sentence-picture comparison task requires participants to take note of both components (verbal and pictorial information). The presence of two age groups made it possible to examine modality effects from a developmental point of view by suggesting that learners' written comprehension and monitoring skills at some point equal and then surpass their oral comprehension and monitoring skills. The experiment showed the necessity to interpret modality main effects in the context of their two-way and three-way interactions with other variables since modality effects taken alone do not tell the whole story. The concepts of split attention, temporal contiguity, and element interactivity were discussed in this context.

Engagement in video and audio narratives: contrasting self-report and physiological measures

Stories play a fundamental role in human culture. They provide a mechanism for sharing cultural identity, imparting knowledge, revealing beliefs, reinforcing social bonds and providing entertainment that is central to all human societies. Here we investigated the extent to which the delivery medium of a story (audio or visual) affected self-reported and physiologically measured engagement with the narrative. Although participants self-reported greater involvement for watching video relative to listening to auditory scenes, stronger physiological responses were recorded for auditory stories. Sensors placed at their wrists showed higher and more variable heart rates, greater electrodermal activity, and even higher body temperatures. We interpret these findings as evidence that the stories were more cognitively and emotionally engaging at a physiological level when presented in an auditory format. This may be because listening to a story, rather than watching a video, is a more active process of co-creation, and that this imaginative process in the listener's mind is detectable on the skin at their wrist.