(Pea)nuts and bolts of visual narrative: structure and meaning in sequential image comprehension

Getting a cue before getting a clue: Event-related potentials to inference in visual narrative comprehension

Inference has long been emphasized in the comprehension of verbal and visual narratives. Here, we measured event-related brain potentials to visual sequences designed to elicit inferential processing. In Impoverished sequences, an expressionless "onlooker" watches an undepicted event (e.g., person throws a ball for a dog, then watches the dog chase it) just prior to a surprising finale (e.g., someone else returns the ball), which should lead to an inference (i.e., the different person retrieved the ball). Implied sequences alter this narrative structure by adding visual cues to the critical panel such as a surprised facial expression to the onlooker implying they saw an unexpected, albeit undepicted, event. In contrast, Expected sequences show a predictable, but then confounded, event (i.e., dog retrieves ball, then different person returns it), and Explicit sequences depict the unexpected event (i.e., different person retrieves then returns ball). At the critical penultimate panel, sequences representing depicted events (Explicit, Expected) elicited a larger posterior positivity (P600) than the relatively passive events of an onlooker (Impoverished, Implied), though Implied sequences were slightly more positive than Impoverished sequences. At the subsequent and final panel, a posterior positivity (P600) was greater to images in Impoverished sequences than those in Explicit and Implied sequences, which did not differ. In addition, both sequence types requiring inference (Implied, Impoverished) elicited a larger frontal negativity than those explicitly depicting events (Expected, Explicit). These results show that neural processing differs for visual narratives omitting events versus those depicting events, and that the presence of subtle visual cues can modulate such effects presumably by altering narrative structure.

The notion of the motion: the neurocognition of motion lines in visual narratives

Motion lines appear ubiquitously in graphic representation to depict the path of a moving object, most popularly in comics. Some researchers have argued that these graphic signs directly tie to the "streaks" appearing in the visual system when a viewer tracks an object (Burr, 2000), despite the fact that previous studies have been limited to offline measurements. Here, we directly examine the cognition of motion lines by comparing images in comic strips that depicted normal motion lines with those that either had no lines or anomalous, reversed lines. In Experiment 1, shorter viewing times appeared to images with normal lines than those with no lines, which were shorter than those with anomalous lines. In Experiment 2, measurements of event-related potentials (ERPs) showed that, compared to normal lines, panels with no lines elicited a posterior positivity that was distinct from the frontal positivity evoked by anomalous lines. These results suggested that motion lines aid in the comprehension of depicted events. LORETA source localization implicated greater activation of visual and language areas when understanding was made more difficult by anomalous lines. Furthermore, in both experiments, participants' experience reading comics modulated these effects, suggesting motion lines are not tied to aspects of the visual system, but rather are conventionalized parts of the "vocabulary" of the visual language of comics.

ERP correlates of script chronology violations

Research indicates a distinction between the processing of script content (which events, behaviors, scenes… are part of it) and script chronology (what is their usual order of occurrence). Using sequences of two line drawings depicting everyday social script events, we examined the event related potential (ERP) correlates of script chronology violations (i.e., wrong order). An increased left anterior negativity (LAN) following chronology violations suggests similarities between the processing of script chronology in visually observed human behavior and verbal syntax. Consequently, this study extends previous findings suggesting that the LAN is sensitive to structure violation across domains (e.g., verbal syntax, abstract structure), including that of meaningful human actions.

The grammar of visual narrative: Neural evidence for constituent structure in sequential image comprehension

Constituent structure has long been established as a central feature of human language. Analogous to how syntax organizes words in sentences, a narrative grammar organizes sequential images into hierarchic constituents. Here we show that the brain draws upon this constituent structure to comprehend wordless visual narratives. We recorded neural responses as participants viewed sequences of visual images (comics strips) in which blank images either disrupted individual narrative constituents or fell at natural constituent boundaries. A disruption of either the first or the second narrative constituent produced a left-lateralized anterior negativity effect between 500 and 700ms. Disruption of the second constituent also elicited a posteriorly-distributed positivity (P600) effect. These neural responses are similar to those associated with structural violations in language and music. These findings provide evidence that comprehenders use a narrative structure to comprehend visual sequences and that the brain engages similar neurocognitive mechanisms to build structure across multiple domains.

Mood-dependent integration in discourse comprehension: happy and sad moods affect consistency processing via different brain networks

According to recent research on language comprehension, the semantic features of a text are not the only determinants of whether incoming information is understood as consistent. Listeners' pre-existing affective states play a crucial role as well. The current fMRI experiment examines the effects of happy and sad moods during comprehension of consistent and inconsistent story endings, focusing on brain regions previously linked to two integration processes: inconsistency detection, evident in stronger responses to inconsistent endings, and fluent processing (accumulation), evident in stronger responses to consistent endings. The analysis evaluated whether differences in the BOLD response for consistent and inconsistent story endings correlated with self-reported mood scores after a mood induction procedure. Mood strongly affected regions previously associated with inconsistency detection. Happy mood increased sensitivity to inconsistency in regions specific for inconsistency detection (e.g., left IFG, left STS), whereas sad mood increased sensitivity to inconsistency in regions less specific for language processing (e.g., right med FG, right SFG). Mood affected more weakly regions involved in accumulation of information. These results show that mood can influence activity in areas mediating well-defined language processes, and highlight that integration is the result of context-dependent mechanisms. The finding that language comprehension can involve different networks depending on people's mood highlights the brain's ability to reorganize its functions.

The architecture of visual narrative comprehension: the interaction of narrative structure and page layout in understanding comics

How do people make sense of the sequential images in visual narratives like comics? A growing literature of recent research has suggested that this comprehension involves the interaction of multiple systems: The creation of meaning across sequential images relies on a "narrative grammar" that packages conceptual information into categorical roles organized in hierarchic constituents. These images are encapsulated into panels arranged in the layout of a physical page. Finally, how panels frame information can impact both the narrative structure and page layout. Altogether, these systems operate in parallel to construct the Gestalt whole of comprehension of this visual language found in comics.

You're a good structure, Charlie Brown: the distribution of narrative categories in comic strips

Cohn's (2013) theory of "Visual Narrative Grammar" argues that sequential images take on categorical roles in a narrative structure, which organizes them into hierarchic constituents analogous to the organization of syntactic categories in sentences. This theory proposes that narrative categories, like syntactic categories, can be identified through diagnostic tests that reveal tendencies for their distribution throughout a sequence. This paper describes four experiments testing these diagnostics to provide support for the validity of these narrative categories. In Experiment 1, participants reconstructed unordered panels of a comic strip into an order that makes sense. Experiment 2 measured viewing times to panels in sequences where the order of panels was reversed. In Experiment 3, participants again reconstructed strips but also deleted a panel from the sequence. Finally, in Experiment 4 participants identified where a panel had been deleted from a comic strip and rated that strip's coherence. Overall, categories had consistent distributional tendencies within experiments and complementary tendencies across experiments. These results point toward an interaction between categorical roles and a global narrative structure.

Effects of emotional context on memory for details: the role of attention

It was repeatedly demonstrated that a negative emotional context enhances memory for central details while impairing memory for peripheral information. This trade-off effect is assumed to result from attentional processes: a negative context seems to narrow attention to central information at the expense of more peripheral details, thus causing the differential effects in memory. However, this explanation has rarely been tested and previous findings were partly inconclusive. For the present experiment 13 negative and 13 neutral naturalistic, thematically driven picture stories were constructed to test the trade-off effect in an ecologically more valid setting as compared to previous studies. During an incidental encoding phase, eye movements were recorded as an index of overt attention. In a subsequent recognition phase, memory for central and peripheral details occurring in the picture stories was tested. Explicit affective ratings and autonomic responses validated the induction of emotion during encoding. Consistent with the emotional trade-off effect on memory, encoding context differentially affected recognition of central and peripheral details. However, contrary to the common assumption, the emotional trade-off effect on memory was not mediated by attentional processes. By contrast, results suggest that the relevance of attentional processing for later recognition memory depends on the centrality of information and the emotional context but not their interaction. Thus, central information was remembered well even when fixated very briefly whereas memory for peripheral information depended more on overt attention at encoding. Moreover, the influence of overt attention on memory for central and peripheral details seems to be much lower for an arousing as compared to a neutral context.

Prediction, events, and the advantage of agents: the processing of semantic roles in visual narrative

Agents consistently appear prior to Patients in sentences, manual signs, and drawings, and Agents are responded to faster when presented in visual depictions of events. We hypothesized that this "Agent advantage" reflects Agents' role in event structure. We investigated this question by manipulating the depictions of Agents and Patients in preparatory actions in wordless visual narratives. We found that Agents elicited a greater degree of predictions regarding upcoming events than Patients, that Agents are viewed longer than Patients, independent of serial order, and that visual depictions of actions are processed more quickly following the presentation of an Agent vs. a Patient. Taken together these findings support the notion that Agents initiate the building of event representation. We suggest that Agent First orders facilitate the interpretation of events as they unfold and that the saliency of Agents within visual representations of events is driven by anticipation of upcoming events.

Differential electrophysiological signatures of semantic and syntactic scene processing

In sentence processing, semantic and syntactic violations elicit differential brain responses observable in event-related potentials: An N400 signals semantic violations, whereas a P600 marks inconsistent syntactic structure. Does the brain register similar distinctions in scene perception? To address this question, we presented participants with semantic inconsistencies, in which an object was incongruent with a scene's meaning, and syntactic inconsistencies, in which an object violated structural rules. We found a clear dissociation between semantic and syntactic processing: Semantic inconsistencies produced negative deflections in the N300-N400 time window, whereas mild syntactic inconsistencies elicited a late positivity resembling the P600 found for syntactic inconsistencies in sentence processing. Extreme syntactic violations, such as a hovering beer bottle defying gravity, were associated with earlier perceptual processing difficulties reflected in the N300 response, but failed to produce a P600 effect. We therefore conclude that different neural populations are active during semantic and syntactic processing of scenes, and that syntactically impossible object placements are processed in a categorically different manner than are syntactically resolvable object misplacements.

Navigating comics: an empirical and theoretical approach to strategies of reading comic page layouts

Like the sequence of words in written language, comic book page layouts direct images into a deliberate reading sequence. Conventional wisdom would expect that comic panels follow the order of text: left-to-right and down - a "Z-path" - though several layouts can violate this order, such as Gestalt groupings of panels that deny a Z-path of reading. To examine how layouts pressure readers to choose pathways deviating from the Z-path, we presented participants with comic pages empty of content, and asked them to number the panels in the order they would read them. Participants frequently used strategies departing from both the traditional Z-path and Gestalt groupings. These preferences reveal a system of constraints that organizes panels into hierarchic constituents, guiding readers through comic page layouts.

Combination Across Domains: An MEG Investigation into the Relationship between Mathematical, Pictorial, and Linguistic Processing

Debates surrounding the evolution of language often hinge upon its relationship to cognition more generally and many investigations have attempted to demark the boundary between the two. Though results from these studies suggest that language may recruit domain-general mechanisms during certain types of complex processing, the domain-generality of basic combinatorial mechanisms that lie at the core of linguistic processing is still unknown. Our previous work (Bemis and Pylkkänen, 2011, 2012) used magnetoencephalography to isolate neural activity associated with the simple composition of an adjective and a noun (“red boat”) and found increased activity during this processing localized to the left anterior temporal lobe (lATL), ventro-medial prefrontal cortex (vmPFC), and left angular gyrus (lAG). The present study explores the domain-generality of these effects and their associated combinatorial mechanisms through two parallel non-linguistic combinatorial tasks designed to be as minimal and natural as the linguistic paradigm. In the first task, we used pictures of colored shapes to elicit combinatorial conceptual processing similar to that evoked by the linguistic expressions and find increased activity again localized to the vmPFC during combinatorial processing. This result suggests that a domain-general semantic combinatorial mechanism operates during basic linguistic composition, and that activity generated by its processing localizes to the vmPFC. In the second task, we recorded neural activity as subjects performed simple addition between two small numerals. Consistent with a wide array of recent results, we find no effects related to basic addition that coincide with our linguistic effects and instead find increased activity localized to the intraparietal sulcus. This result suggests that the scope of the previously identified linguistic effects is restricted to compositional operations and does not extend generally to all tasks that are merely similar in form.