1
|
Hacımusaoğlu I, Cohn N. The Meaning of Motion Lines?: A Review of Theoretical and Empirical Research on Static Depiction of Motion. Cogn Sci 2023; 47:e13377. [PMID: 37966099 DOI: 10.1111/cogs.13377] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/14/2023] [Accepted: 10/30/2023] [Indexed: 11/16/2023]
Abstract
Static depiction of motion, particularly lines trailing behind a mover, has long been of interest in the psychology literature. Empirical research has demonstrated that these "motion lines" benefited motion comprehension in static images by disambiguating the direction of movement. Yet, there is no consensus on how those lines derive their meaning. In this article, we review three accounts suggesting different interpretations of what motion lines represent. While a perceptual account considers motion lines originating from motion streaks in the primary visual cortex, metaphorical and lexical accounts propose them as graphical conventions that should be learned-either through resemblance to sensory experiences (e.g., natural path marks) or directly being mapped to a conceptual category of paths. To contrast these three accounts, we integrate empirical research on motion lines and their understanding. Overall, developmental, proficiency, and cross-cultural variances indicate that the understanding of motion lines is neither innate nor universal, thus providing less support for lines having a purely perceptual origin. Rather, we argue the empirical findings suggest that motion lines are not iconic depictions of visual percepts but are graphical conventions indexing conceptual path information, which need to be learned and encoded in a visual lexicon.
Collapse
Affiliation(s)
- Irmak Hacımusaoğlu
- Department of Communication and Cognition, Tilburg School of Humanities and Digital Sciences, Tilburg University
| | - Neil Cohn
- Department of Communication and Cognition, Tilburg School of Humanities and Digital Sciences, Tilburg University
| |
Collapse
|
2
|
Hacımusaoğlu I, Cohn N. Linguistic typology of motion events in visual narratives. COGNITIVE SEMIOTICS 2022; 15:197-222. [PMID: 36590029 PMCID: PMC9767167 DOI: 10.1515/cogsem-2022-2013] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Indexed: 12/05/2022]
Abstract
Languages use different strategies to encode motion. Some use particles or "satellites" to describe a path of motion (Satellite-framed or S-languages like English), while others typically use the main verb to convey the path information (Verb-framed or V-languages like French). We here ask: might this linguistic variation lead to differences in the way paths are depicted in visual narratives like comics? We analyzed a corpus of 85 comics originally created by speakers of S-languages (comics from the United States, China, Germany) and V-languages (France, Japan, Korea) for both their depictions of path segments (source, route, and goal) and the visual cues signaling these paths and manner information (e.g., motion lines and postures). Panels from S-languages depicted more path segments overall, especially routes, than those from V-languages, but panels from V-languages more often isolated path segments into their own panels. Additionally, comics from S-languages depicted more motion cues than those from V-languages, and this linguistic typology also interacted with panel framing. Despite these differences across typological groups, analysis of individual countries' comics showed more nuanced variation than a simple S-V dichotomy. These findings suggest a possible influence of spoken language structure on depicting motion events in visual narratives and their sequencing.
Collapse
Affiliation(s)
- Irmak Hacımusaoğlu
- Department of Communication and Cognition, Tilburg School of Humanities and Digital Sciences, Tilburg University, Tilburg, The Netherlands
| | - Neil Cohn
- Department of Communication and Cognition, Tilburg School of Humanities and Digital Sciences, Tilburg University, Tilburg, The Netherlands
| |
Collapse
|
3
|
Meaning above (and in) the head: Combinatorial visual morphology from comics and emoji. Mem Cognit 2022; 50:1381-1398. [PMID: 35235175 PMCID: PMC9508049 DOI: 10.3758/s13421-022-01294-2] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 02/15/2022] [Indexed: 11/08/2022]
Abstract
Compositionality is a primary feature of language, but graphics can also create combinatorial meaning, like with items above faces (e.g., lightbulbs to mean inspiration). We posit that these “upfixes” (i.e., upwards affixes) involve a productive schema enabling both stored and novel face–upfix dyads. In two experiments, participants viewed either conventional (e.g., lightbulb) or unconventional (e.g., clover-leaves) upfixes with faces which either matched (e.g., lightbulb/smile) or mismatched (e.g., lightbulb/frown). In Experiment 1, matching dyads sponsored higher comprehensibility ratings and faster response times, modulated by conventionality. In Experiment 2, event-related brain potentials (ERPs) revealed conventional upfixes, regardless of matching, evoked larger N250s, indicating perceptual expertise, but mismatching and unconventional dyads elicited larger semantic processing costs (N400) than conventional-matching dyads. Yet mismatches evoked a late negativity, suggesting congruent novel dyads remained construable compared with violations. These results support that combinatorial graphics involve a constrained productive schema, similar to the lexicon of language.
Collapse
|
4
|
Jouen AL, Cazin N, Hidot S, Madden-Lombardi C, Ventre-Dominey J, Dominey PF. Common ERP responses to narrative incoherence in sentence and picture pair comprehension. Brain Cogn 2021; 153:105775. [PMID: 34333283 DOI: 10.1016/j.bandc.2021.105775] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/08/2021] [Revised: 07/13/2021] [Accepted: 07/22/2021] [Indexed: 11/17/2022]
Abstract
Understanding the neural processes underlying the comprehension of visual images and sentences remains a major open challenge in cognitive neuroscience. We previously demonstrated with fMRI and DTI that comprehension of visual images and sentences describing human activities recruits a common extended parietal-temporal-frontal semantic system. The current research tests the hypothesis that this common semantic system will display similar ERP profiles during processing in these two modalities, providing further support for the common comprehension system. We recorded EEG from naïve subjects as they saw simple narratives made up of a first visual image depicting a human event, followed by a second image that was either a sequentially coherent narrative follow-up, or not, of the first. Incoherent second stimuli depict the same agents but shifted into a different situation. In separate blocks of trials the same protocol was presented using narrative sentence stimuli. Part of the novelty is the comparison of sentence and visual narrative responses. ERPs revealed common neural profiles for narrative processing across image and sentence modalities in the form of early and late central and frontal positivities in response to narrative incoherence. There was an additional posterior positivity only for sentences in a very late window. These results are discussed in the context of ERP signatures of narrative processing and meaning, and a current model of narrative comprehension.
Collapse
Affiliation(s)
- Anne-Lise Jouen
- Department of Neuropsycholinguistics, Université de Genève, Geneva, Switzerland
| | - Nicolas Cazin
- INSERM UMR1093-CAPS, Université Bourgogne Franche-Comté, UFR des Sciences du Sport, Dijon, France
| | - Sullivan Hidot
- INSERM UMR1093-CAPS, Université Bourgogne Franche-Comté, UFR des Sciences du Sport, Dijon, France
| | - Carol Madden-Lombardi
- INSERM UMR1093-CAPS, Université Bourgogne Franche-Comté, UFR des Sciences du Sport, Dijon, France
| | - Jocelyne Ventre-Dominey
- INSERM UMR1093-CAPS, Université Bourgogne Franche-Comté, UFR des Sciences du Sport, Dijon, France
| | - Peter Ford Dominey
- INSERM UMR1093-CAPS, Université Bourgogne Franche-Comté, UFR des Sciences du Sport, Dijon, France.
| |
Collapse
|
5
|
Cohn N. A starring role for inference in the neurocognition of visual narratives. COGNITIVE RESEARCH-PRINCIPLES AND IMPLICATIONS 2021; 6:8. [PMID: 33587244 PMCID: PMC7884514 DOI: 10.1186/s41235-021-00270-9] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 08/26/2020] [Accepted: 01/06/2021] [Indexed: 11/10/2022]
Abstract
Research in verbal and visual narratives has often emphasized backward-looking inferences, where absent information is subsequently inferred. However, comics use conventions like star-shaped “action stars” where a reader knows events are undepicted at that moment, rather than omitted entirely. We contrasted the event-related brain potentials (ERPs) to visual narratives depicting an explicit event, an action star, or a “noise” panel of scrambled lines. Both action stars and noise panels evoked large N400s compared to explicit-events (300–500 ms), but action stars and noise panels then differed in their later effects (500–900 ms). Action stars elicited sustained negativities and P600s, which could indicate further interpretive processes and integration of meaning into a mental model, while noise panels evoked late frontal positivities possibly indexing that they were improbable narrative units. Nevertheless, panels following action stars and noise panels both evoked late sustained negativities, implying further inferential processing. Inference in visual narratives thus uses cascading mechanisms resembling those in language processing that differ based on the inferential techniques.
Collapse
Affiliation(s)
- Neil Cohn
- Department of Communication and Cognition, Tilburg School of Humanities and Digital Sciences, Tilburg University, P.O. Box 90153, 5000 LE, Tilburg, The Netherlands.
| |
Collapse
|
6
|
Cohn N, Foulsham T. Zooming in on the cognitive neuroscience of visual narrative. Brain Cogn 2020; 146:105634. [DOI: 10.1016/j.bandc.2020.105634] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/09/2020] [Revised: 10/09/2020] [Accepted: 10/19/2020] [Indexed: 10/23/2022]
|
7
|
Visual and Verbal Narrative Comprehension in Children and Adolescents with Autism Spectrum Disorders: An ERP Study. J Autism Dev Disord 2020; 50:2658-2672. [PMID: 31974801 DOI: 10.1007/s10803-020-04374-x] [Citation(s) in RCA: 12] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/03/2023]
Abstract
We examined semantic processing in ASD children by presenting sentences with congruent or incongruent final words and visual narratives with congruent or incongruent final panels. An N400 effect to incongruent words appeared as compared to congruent ones, which was attenuated for the ASD children. We observed a negativity sustained to incongruous than congruous words, but only for the TD children. Incongruent panels evoked a greater fronto-central N400 amplitude than congruent panels in both groups. In addition, incongruent panels evoked a centro-parietal late positivity, only in controls. In conclusion, ASD children face processing deficits in both verbal and visual materials when integrating meaning across information, though such impairments may arise in different parts of the interpretive process, depending on the modality.
Collapse
|
8
|
Abstract
The comprehension of visual narratives requires paying attention to certain elements and integrating them across a sequence of images. To study this process, we developed a new approach that modified comic strips according to where observers looked while viewing each sequence. Across three self-paced experiments, we presented sequences of six panels that were sometimes automatically "zoomed-in" or re-framed in order to highlight parts of the image that had been fixated by another group of observers. Fixation zoom panels were rated as easier to understand and produced viewing times more similar to the original comic than panels modified to contain non-fixated or incongruous regions. When a single panel depicting the start of an action was cropped to show only the most fixated region, viewing times were similar to the original narrative despite the reduced information. Modifying such panels also had an impact on the viewing time on subsequent panels, both when zoomed in and when regions were highlighted through an "inset" panel. These findings demonstrate that fixations in a visual narrative are guided to informative elements, and that these elements influence both the current panel and the processing of the sequence.
Collapse
|
9
|
Coderre EL, O'Donnell E, O'Rourke E, Cohn N. Predictability modulates neurocognitive semantic processing of non-verbal narratives. Sci Rep 2020; 10:10326. [PMID: 32587312 PMCID: PMC7316725 DOI: 10.1038/s41598-020-66814-z] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/19/2019] [Accepted: 05/28/2020] [Indexed: 11/26/2022] Open
Abstract
Predictability is known to modulate semantic processing in language, but it is unclear to what extent this applies for other modalities. Here we ask whether similar cognitive processes are at play in predicting upcoming events in a non-verbal visual narrative. Typically developing adults viewed comics sequences in which a target panel was highly predictable (“high cloze”), less predictable (“low cloze”), or incongruent with the preceding narrative context (“anomalous”) during EEG recording. High and low predictable sequences were determined by a pretest where participants assessed “what happened next?”, resulting in cloze probability scores for sequence outcomes comparable to those used to measure predictability in sentence processing. Through both factorial and correlational analyses, we show a significant modulation of neural responses by cloze such that N400 effects are diminished as a target panel in a comic sequence becomes more predictable. Predictability thus appears to play a similar role in non-verbal comprehension of sequential images as in language comprehension, providing further evidence for the domain generality of semantic processing in the brain.
Collapse
Affiliation(s)
- Emily L Coderre
- Department of Communication Sciences and Disorders, University of Vermont, Burlington, VT, United States.
| | - Elizabeth O'Donnell
- Department of Communication Sciences and Disorders, University of Vermont, Burlington, VT, United States
| | - Emme O'Rourke
- Department of Communication Sciences and Disorders, University of Vermont, Burlington, VT, United States
| | - Neil Cohn
- Department of Communication and Cognition, Tilburg School of Humanities and Digital Sciences, Tilburg Center for Cognition and Communication (TiCC), Tilburg University, Tilburg, The Netherlands
| |
Collapse
|
10
|
Abstract
Visual narratives of sequential images - as found in comics, picture stories, and storyboards - are often thought to provide a fairly universal and transparent message that requires minimal learning to decode. This perceived transparency has led to frequent use of sequential images as experimental stimuli in the cognitive and psychological sciences to explore a wide range of topics. In addition, it underlines efforts to use visual narratives in science and health communication and as educational materials in both classroom settings and across developmental, clinical, and non-literate populations. Yet, combined with recent studies from the linguistic and cognitive sciences, decades of research suggest that visual narratives involve greater complexity and decoding than widely assumed. This review synthesizes observations from cross-cultural and developmental research on the comprehension and creation of visual narrative sequences, as well as findings from clinical psychology (e.g., autism, developmental language disorder, aphasia). Altogether, this work suggests that understanding the visual languages found in comics and visual narratives requires a fluency that is contingent on exposure and practice with a graphic system.
Collapse
Affiliation(s)
- Neil Cohn
- Tilburg School of Humanities and Digital Sciences, Department of Communication and Cognition, Tilburg center for Cognition and Communication, Tilburg University, P.O. Box 90153, 5000 LE, Tilburg, The Netherlands.
| |
Collapse
|
11
|
Cohn N, Magliano JP. Editors’ Introduction and Review: Visual Narrative Research: An Emerging Field in Cognitive Science. Top Cogn Sci 2019; 12:197-223. [PMID: 31865641 PMCID: PMC9328199 DOI: 10.1111/tops.12473] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/01/2018] [Revised: 09/29/2019] [Accepted: 09/29/2019] [Indexed: 01/06/2023]
Abstract
Drawn sequences of images are among our oldest records of human intelligence, appearing on cave paintings, wall carvings, and ancient pottery, and they pervade across cultures from instruction manuals to comics. They also appear prevalently as stimuli across Cognitive Science, for studies of temporal cognition, event structure, social cognition, discourse, and basic intelligence. Yet, despite this fundamental place in human expression and research on cognition, the study of visual narratives themselves has only recently gained traction in Cognitive Science. This work has suggested that visual narrative comprehension requires cultural exposure across a developmental trajectory and engages with domain‐general processing mechanisms shared by visual perception, attention, event cognition, and language, among others. Here, we review the relevance of such research for the broader Cognitive Science community, and make the case for why researchers should join the scholarship of this ubiquitous but understudied aspect of human expression. Drawn sequences of images, like those in comics and picture stories, are a pervasive and fundamental way that humans have communicated for millennia. Yet, the study of visual narratives has only recently gained traction in Cognitive Science. Here we explore what has held back the study of the cognition of visual narratives, and why researchers should join in scholarship of this ubiquitous aspect of expression.
Collapse
Affiliation(s)
- Neil Cohn
- Department of Communciation and Cognition, Tilburg School of Humanities and Digital Sciences, Tilburg Center for Cognition and Communication, Tilburg Unviersity
| | - Joseph P. Magliano
- Department of Learning Sciences at the College of Education & Human Development, Georgia State University
| |
Collapse
|
12
|
Loschky LC, Larson AM, Smith TJ, Magliano JP. The Scene Perception & Event Comprehension Theory (SPECT) Applied to Visual Narratives. Top Cogn Sci 2019; 12:311-351. [PMID: 31486277 PMCID: PMC9328418 DOI: 10.1111/tops.12455] [Citation(s) in RCA: 32] [Impact Index Per Article: 6.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/26/2018] [Revised: 08/05/2019] [Accepted: 08/05/2019] [Indexed: 11/29/2022]
Abstract
Understanding how people comprehend visual narratives (including picture stories, comics, and film) requires the combination of traditionally separate theories that span the initial sensory and perceptual processing of complex visual scenes, the perception of events over time, and comprehension of narratives. Existing piecemeal approaches fail to capture the interplay between these levels of processing. Here, we propose the Scene Perception & Event Comprehension Theory (SPECT), as applied to visual narratives, which distinguishes between front‐end and back‐end cognitive processes. Front‐end processes occur during single eye fixations and are comprised of attentional selection and information extraction. Back‐end processes occur across multiple fixations and support the construction of event models, which reflect understanding of what is happening now in a narrative (stored in working memory) and over the course of the entire narrative (stored in long‐term episodic memory). We describe relationships between front‐ and back‐end processes, and medium‐specific differences that likely produce variation in front‐end and back‐end processes across media (e.g., picture stories vs. film). We describe several novel research questions derived from SPECT that we have explored. By addressing these questions, we provide greater insight into how attention, information extraction, and event model processes are dynamically coordinated to perceive and understand complex naturalistic visual events in narratives and the real world. Comprehension of visual narratives like comics, picture stories, and films involves both decoding the visual content and construing the meaningful events they represent. The Scene Perception & Event Comprehension Theory (SPECT) proposes a framework for understanding how a comprehender perceptually negotiates the surface of a visual representation and integrates its meaning into a growing mental model.
Collapse
Affiliation(s)
| | | | - Tim J Smith
- Department of Psychological Sciences, Birkbeck, University of London
| | | |
Collapse
|
13
|
Coderre EL. Dismantling the “Visual Ease Assumption:" A Review of Visual Narrative Processing in Clinical Populations. Top Cogn Sci 2019; 12:224-255. [DOI: 10.1111/tops.12446] [Citation(s) in RCA: 16] [Impact Index Per Article: 3.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/02/2018] [Revised: 06/18/2019] [Accepted: 06/18/2019] [Indexed: 12/29/2022]
Affiliation(s)
- Emily L. Coderre
- Department of Communication Sciences and Disorders University of Vermont
| |
Collapse
|
14
|
Cohn N. Your Brain on Comics: A Cognitive Model of Visual Narrative Comprehension. Top Cogn Sci 2019; 12:352-386. [PMID: 30963724 PMCID: PMC9328425 DOI: 10.1111/tops.12421] [Citation(s) in RCA: 21] [Impact Index Per Article: 4.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/03/2018] [Revised: 01/21/2019] [Accepted: 03/18/2019] [Indexed: 11/30/2022]
Abstract
The past decade has seen a rapid growth of cognitive and brain research focused on visual narratives like comics and picture stories. This paper will summarize and integrate this emerging literature into the Parallel Interfacing Narrative‐Semantics Model (PINS Model)—a theory of sequential image processing characterized by an interaction between two representational levels: semantics and narrative structure. Ongoing semantic processes build meaning into an evolving mental model of a visual discourse. Updating of spatial, referential, and event information then incurs costs when they are discontinuous with the growing context. In parallel, a narrative structure organizes semantic information into coherent sequences by assigning images to categorical roles, which are then embedded within a hierarchic constituent structure. Narrative constructional schemas allow for specific predictions of structural sequencing, independent of semantics. Together, these interacting levels of representation engage in an iterative process of retrieval of semantic and narrative information, prediction of upcoming information based on those assessments, and subsequent updating based on discontinuity. These core mechanisms are argued to be domain‐general—spanning across expressive systems—as suggested by similar electrophysiological brain responses (N400, P600, anterior negativities) generated in response to manipulation of sequential images, music, and language. Such similarities between visual narratives and other domains thus pose fundamental questions for the linguistic and cognitive sciences. Visual narratives like comics involve a range of complex cognitive operations in order to be understood. The Parallel Interfacing Narrative‐Semantics (PINS) Model integrates an emerging literature showing that comprehension of wordless image sequences balances two representational levels of semantic and narrative structure. The neurocognitive mechanisms that guide these processes are argued to overlap with other domains, such as language and music.
Collapse
Affiliation(s)
- Neil Cohn
- Department of Communication and Cognition, Tilburg University
| |
Collapse
|
15
|
Cohn N. Visual narratives and the mind: Comprehension, cognition, and learning. PSYCHOLOGY OF LEARNING AND MOTIVATION 2019. [DOI: 10.1016/bs.plm.2019.02.002] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 10/27/2022]
|
16
|
Coderre EL, Cohn N, Slipher SK, Chernenok M, Ledoux K, Gordon B. Visual and linguistic narrative comprehension in autism spectrum disorders: Neural evidence for modality-independent impairments. BRAIN AND LANGUAGE 2018; 186:44-59. [PMID: 30216902 DOI: 10.1016/j.bandl.2018.09.001] [Citation(s) in RCA: 24] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/27/2017] [Revised: 06/15/2018] [Accepted: 09/02/2018] [Indexed: 06/08/2023]
Abstract
Individuals with autism spectrum disorders (ASD) have notable language difficulties, including with understanding narratives. However, most narrative comprehension studies have used written or spoken narratives, making it unclear whether narrative difficulties stem from language impairments or more global impairments in the kinds of general cognitive processes (such as understanding meaning and structural sequencing) that are involved in narrative comprehension. Using event-related potentials (ERPs), we directly compared semantic comprehension of linguistic narratives (short sentences) and visual narratives (comic panels) in adults with ASD and typically-developing (TD) adults. Compared to the TD group, the ASD group showed reduced N400 effects for both linguistic and visual narratives, suggesting comprehension impairments for both types of narratives and thereby implicating a more domain-general impairment. Based on these results, we propose that individuals with ASD use a more bottom-up style of processing during narrative comprehension.
Collapse
Affiliation(s)
- Emily L Coderre
- Department of Communication Sciences and Disorders, University of Vermont, Burlington, VT, United States; Cognitive Neurology/Neuropsychology, Department of Neurology, The Johns Hopkins University School of Medicine, Baltimore, MD, United States.
| | - Neil Cohn
- Tilburg Center for Cognition and Communication (TiCC), Tilburg University, Tilburg, The Netherlands
| | - Sally K Slipher
- Cognitive Neurology/Neuropsychology, Department of Neurology, The Johns Hopkins University School of Medicine, Baltimore, MD, United States; Department of Health Professions, Montana State University, Bozeman, MT, United States
| | - Mariya Chernenok
- Cognitive Neurology/Neuropsychology, Department of Neurology, The Johns Hopkins University School of Medicine, Baltimore, MD, United States; Department of Human Ecology, University of California at Davis, Davis, CA, United States
| | - Kerry Ledoux
- Cognitive Neurology/Neuropsychology, Department of Neurology, The Johns Hopkins University School of Medicine, Baltimore, MD, United States
| | - Barry Gordon
- Cognitive Neurology/Neuropsychology, Department of Neurology, The Johns Hopkins University School of Medicine, Baltimore, MD, United States; Department of Cognitive Science, The Johns Hopkins University, Baltimore, MD, United States
| |
Collapse
|
17
|
Weissman B, Tanner D. A strong wink between verbal and emoji-based irony: How the brain processes ironic emojis during language comprehension. PLoS One 2018; 13:e0201727. [PMID: 30110375 PMCID: PMC6093662 DOI: 10.1371/journal.pone.0201727] [Citation(s) in RCA: 46] [Impact Index Per Article: 7.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/31/2017] [Accepted: 07/20/2018] [Indexed: 11/19/2022] Open
Abstract
Emojis are ideograms that are becoming ubiquitous in digital communication. However, no research has yet investigated how humans process semantic and pragmatic content of emojis in real time. We investigated neural responses to irony-producing emojis, the question being whether emoji-generated irony is processed similarly to word-generated irony. Previous ERP studies have routinely found P600 effects to verbal irony. Our research sought to identify whether the same neural responses could also be elicited by emoji-induced irony. In three experiments, participants read sentences that ended in either a congruent, incongruent, or ironic (wink) emoji. Results across all three experiments demonstrated clear P600 effects, the amplitudes of which were correlated with participants' tendency to treat the emoji as a marker of irony, as indicated by behavioral comprehension question responses. These ironic wink emojis also elicited a strong P200 effect, also found in studies of verbal irony processing. Moreover, unexpected emojis (both mismatch and ironic emoji) also elicited late frontal positivities, which have been implicated processing unpredicted words in context. These results are the first to identify how linguistically-relevant ideograms are processed in real-time at the neural level, and specifically draw parallels between the processing of word- and emoji-induced irony.
Collapse
Affiliation(s)
- Benjamin Weissman
- Department of Linguistics, University of Illinois, Urbana, Illinois, United States of America
| | - Darren Tanner
- Department of Linguistics, University of Illinois, Urbana, Illinois, United States of America
| |
Collapse
|
18
|
Cohn N, Paczynski M, Kutas M. Not so secret agents: Event-related potentials to semantic roles in visual event comprehension. Brain Cogn 2017; 119:1-9. [PMID: 28898720 DOI: 10.1016/j.bandc.2017.09.001] [Citation(s) in RCA: 12] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/11/2017] [Revised: 08/25/2017] [Accepted: 09/05/2017] [Indexed: 11/24/2022]
Abstract
Research across domains has suggested that agents, the doers of actions, have a processing advantage over patients, the receivers of actions. We hypothesized that agents as "event builders" for discrete actions (e.g., throwing a ball, punching) build on cues embedded in their preparatory postures (e.g., reaching back an arm to throw or punch) that lead to (predictable) culminating actions, and that these cues afford frontloading of event structure processing. To test this hypothesis, we compared event-related brain potentials (ERPs) to averbal comic panels depicting preparatory agents (ex. reaching back an arm to punch) that cued specific actions with those to non-preparatory agents (ex. arm to the side) and patients that did not cue any specific actions. We also compared subsequent completed action panels (ex. agent punching patient) across conditions, where we expected an inverse pattern of ERPs indexing the differential costs of processing completed actions asa function of preparatory cues. Preparatory agents evoked a greater frontal positivity (600-900ms) relative to non-preparatory agents and patients, while subsequent completed actions panels following non-preparatory agents elicited a smaller frontal positivity (600-900ms). These results suggest that preparatory (vs. non-) postures may differentially impact the processing of agents and subsequent actions in real time.
Collapse
Affiliation(s)
- Neil Cohn
- Department of Cognitive Science, University of California, San Diego, La Jolla, CA, USA; Tilburg Center for Cognition and Communication, Tilburg University, Tilburg, The Netherlands.
| | - Martin Paczynski
- Wright State Research Institute, Wright State University, Dayton, OH, USA
| | - Marta Kutas
- Department of Cognitive Science, University of California, San Diego, La Jolla, CA, USA
| |
Collapse
|
19
|
Cohn N, Kutas M. What is your neural function, visual narrative conjunction? Grammar, meaning, and fluency in sequential image processing. COGNITIVE RESEARCH-PRINCIPLES AND IMPLICATIONS 2017; 2:27. [PMID: 28603773 PMCID: PMC5442195 DOI: 10.1186/s41235-017-0064-5] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 11/22/2016] [Accepted: 04/25/2017] [Indexed: 11/10/2022]
Abstract
Visual narratives sometimes depict successive images with different characters in the same physical space; corpus analysis has revealed that this occurs more often in Japanese manga than American comics. We used event-related brain potentials to determine whether comprehension of "visual narrative conjunctions" invokes not only incremental mental updating as traditionally assumed, but also, as we propose, "grammatical" combinatoric processing. We thus crossed (non)/conjunction sequences with character (in)/congruity. Conjunctions elicited a larger anterior negativity (300-500 ms) than nonconjunctions, regardless of congruity, implicating "grammatical" processes. Conjunction and incongruity both elicited larger P600s (500-700 ms), indexing updating. Both conjunction effects were modulated by participants' frequency of reading manga while growing up. Greater anterior negativity in frequent manga readers suggests more reliance on combinatoric processing; larger P600 effects in infrequent manga readers suggest more resources devoted to mental updating. As in language comprehension, it seems that processing conjunctions in visual narratives is not just mental updating but also partly grammatical, conditioned by comic readers' experience with specific visual narrative structures.
Collapse
Affiliation(s)
- Neil Cohn
- Center for Research in Language, University of California, San Diego, La Jolla, CA USA.,Tilburg Center for Cognition and Communication, Tilburg University, P.O. Box 90153, 5000 LE Tilburg, the Netherlands
| | - Marta Kutas
- Department of Cognitive Science, University of California, San Diego, La Jolla, CA USA
| |
Collapse
|
20
|
Cohn N, Murthy B, Foulsham T. Meaning above the head: combinatorial constraints on the visual vocabulary of comics. JOURNAL OF COGNITIVE PSYCHOLOGY 2016. [DOI: 10.1080/20445911.2016.1179314] [Citation(s) in RCA: 7] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/21/2022]
|
21
|
The pieces fit: Constituent structure and global coherence of visual narrative in RSVP. Acta Psychol (Amst) 2016; 164:157-64. [PMID: 26826863 DOI: 10.1016/j.actpsy.2016.01.011] [Citation(s) in RCA: 10] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/16/2015] [Revised: 01/19/2016] [Accepted: 01/20/2016] [Indexed: 11/22/2022] Open
Abstract
Recent research has shown that comprehension of visual narrative relies on the ordering and timing of sequential images. Here we tested if rapidly presented 6-image long visual sequences could be understood as coherent narratives. Half of the sequences were correctly ordered and half had two of the four internal panels switched. Participants reported whether the sequence was correctly ordered and rated its coherence. Accuracy in detecting a switch increased when panels were presented for 1 s rather than 0.5 s. Doubling the duration of the first panel did not affect results. When two switched panels were further apart, order was discriminated more accurately and coherence ratings were low, revealing that a strong local adjacency effect influenced order and coherence judgments. Switched panels at constituent boundaries or within constituents were most disruptive to order discrimination, indicating that the preservation of constituent structure is critical to visual narrative grammar.
Collapse
|
22
|
Cohn N. A multimodal parallel architecture: A cognitive framework for multimodal interactions. Cognition 2016; 146:304-23. [DOI: 10.1016/j.cognition.2015.10.007] [Citation(s) in RCA: 39] [Impact Index Per Article: 4.9] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/20/2014] [Revised: 09/03/2015] [Accepted: 10/11/2015] [Indexed: 11/29/2022]
|
23
|
Cohn N, Wittenberg E. Action starring narratives and events: Structure and inference in visual narrative comprehension. JOURNAL OF COGNITIVE PSYCHOLOGY 2015; 27:812-828. [PMID: 26709362 DOI: 10.1080/20445911.2015.1051535] [Citation(s) in RCA: 29] [Impact Index Per Article: 3.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/23/2022]
Abstract
Studies of discourse have long placed focus on the inference generated by information that is not overtly expressed, and theories of visual narrative comprehension similarly focused on the inference generated between juxtaposed panels. Within the visual language of comics, star-shaped "flashes" commonly signify impacts, but can be enlarged to the size of a whole panel that can omit all other representational information. These "action star" panels depict a narrative culmination (a "Peak"), but have content which readers must infer, thereby posing a challenge to theories of inference generation in visual narratives that focus only on the semantic changes between juxtaposed images. This paper shows that action stars demand more inference than depicted events, and that they are more coherent in narrative sequences than scrambled sequences (Experiment 1). In addition, action stars play a felicitous narrative role in the sequence (Experiment 2). Together, these results suggest that visual narratives use conventionalized depictions that demand the generation of inferences while retaining narrative coherence of a visual sequence.
Collapse
Affiliation(s)
- Neil Cohn
- Department of Psychology, Tufts University, Medford, MA, USA ; Department of Cognitive Science, University of California, San Diego, La Jolla, CA, USA
| | - Eva Wittenberg
- Department of Linguistics, University of California, San Diego, La Jolla, CA, USA
| |
Collapse
|
24
|
Cattaneo Z, Schiavi S, Silvanto J, Nadal M. A TMS study on the contribution of visual area V5 to the perception of implied motion in art and its appreciation. Cogn Neurosci 2015; 8:59-68. [PMID: 26429631 DOI: 10.1080/17588928.2015.1083968] [Citation(s) in RCA: 26] [Impact Index Per Article: 2.9] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/23/2022]
Abstract
Over the last decade, researchers have sought to understand the brain mechanisms involved in the appreciation of art. Previous studies reported an increased activity in sensory processing regions for artworks that participants find more appealing. Here we investigated the intriguing possibility that activity in cortical area V5-a region in the occipital cortex mediating physical and implied motion detection-is related not only to the generation of a sense of motion from visual cues used in artworks, but also to the appreciation of those artworks. Art-naïve participants viewed a series of paintings and quickly judged whether or not the paintings conveyed a sense of motion, and whether or not they liked them. Triple-pulse TMS applied over V5 while viewing the paintings significantly decreased the perceived sense of motion, and also significantly reduced liking of abstract (but not representational) paintings. Our data demonstrate that V5 is involved in extracting motion information even when the objects whose motion is implied are pictorial representations (as opposed to photographs or film frames), and even in the absence of any figurative content. Moreover, our study suggests that, in the case of untrained people, V5 activity plays a causal role in the appreciation of abstract but not of representational art.
Collapse
Affiliation(s)
- Zaira Cattaneo
- a Department of Psychology , University of Milano-Bicocca , Milano , Italy.,b Brain Connectivity Center , C. Mondino National Neurological Institute , Pavia , Italy
| | - Susanna Schiavi
- a Department of Psychology , University of Milano-Bicocca , Milano , Italy
| | - Juha Silvanto
- c Department of Psychology , University of Westminster , London , UK
| | - Marcos Nadal
- d Department of Basic Psychological Research and Research Methods , University of Vienna , Vienna , Austria
| |
Collapse
|
25
|
Getting a cue before getting a clue: Event-related potentials to inference in visual narrative comprehension. Neuropsychologia 2015; 77:267-78. [PMID: 26320706 DOI: 10.1016/j.neuropsychologia.2015.08.026] [Citation(s) in RCA: 27] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/25/2015] [Revised: 08/05/2015] [Accepted: 08/25/2015] [Indexed: 11/21/2022]
Abstract
Inference has long been emphasized in the comprehension of verbal and visual narratives. Here, we measured event-related brain potentials to visual sequences designed to elicit inferential processing. In Impoverished sequences, an expressionless "onlooker" watches an undepicted event (e.g., person throws a ball for a dog, then watches the dog chase it) just prior to a surprising finale (e.g., someone else returns the ball), which should lead to an inference (i.e., the different person retrieved the ball). Implied sequences alter this narrative structure by adding visual cues to the critical panel such as a surprised facial expression to the onlooker implying they saw an unexpected, albeit undepicted, event. In contrast, Expected sequences show a predictable, but then confounded, event (i.e., dog retrieves ball, then different person returns it), and Explicit sequences depict the unexpected event (i.e., different person retrieves then returns ball). At the critical penultimate panel, sequences representing depicted events (Explicit, Expected) elicited a larger posterior positivity (P600) than the relatively passive events of an onlooker (Impoverished, Implied), though Implied sequences were slightly more positive than Impoverished sequences. At the subsequent and final panel, a posterior positivity (P600) was greater to images in Impoverished sequences than those in Explicit and Implied sequences, which did not differ. In addition, both sequence types requiring inference (Implied, Impoverished) elicited a larger frontal negativity than those explicitly depicting events (Expected, Explicit). These results show that neural processing differs for visual narratives omitting events versus those depicting events, and that the presence of subtle visual cues can modulate such effects presumably by altering narrative structure.
Collapse
|