Visual and linguistic narrative comprehension in autism spectrum disorders: Neural evidence for modality-independent impairments

The Practical Implications of Re-Referencing in ERP Studies: The Case of N400 in the Picture-Word Verification Task

Background: The selection of an optimal referencing method in event-related potential (ERP) research has been a long-standing debate, as it can significantly influence results and lead to data misinterpretation. Such misinterpretation can produce flawed scientific conclusions, like the inaccurate localization of neural processes, and in practical applications, such as using ERPs as biomarkers in medicine, it may result in incorrect diagnoses or ineffective treatments. In line with the development and advancement of good scientific practice (GSP) in ERP research, this study sought to address several questions regarding the most suitable digital reference for investigating the N400 ERP component. Methods: The study was conducted on 17 neurotypical participants. Based on previous research, the references evaluated included the common average reference (AVE), mean earlobe reference (EARS), left mastoid reference (L), mean mastoids reference (MM), neutral infinity reference (REST), and vertex reference (VERT). Results: The results showed that all digital references, except for VERT, successfully elicited the centroparietal N400 effect in the picture-word verification task. The AVE referencing method showed the most optimal set of metrics in terms of effect size and localization, although it also produced the smallest difference waves. The most similar topographic dynamics in the N400 window were observed between the AVE and REST referencing methods. Conclusions: As the most optimal regions of interest (ROI) for the picture-word elicited N400 effect, nine electrode sites spanning from superior frontocentral to parietal regions were identified, showing consistent effects across all referencing methods except VERT.

Age-related unstable transient states and imbalanced activation proportion of brain networks in people with autism spectrum disorder: A resting-state fMRI study using coactivation pattern analyses

The atypical static brain functions related to the executive control network (ECN), default mode network (DMN), and salience network (SN) in people with autism spectrum disorder (ASD) has been widely reported. However, their transient functions in ASD are not clear. We aim to identify transient network states (TNSs) using coactivation pattern (CAP) analysis to characterize the age-related atypical transient functions in ASD. CAP analysis was performed on a resting-state fMRI dataset (78 ASD and 78 healthy control (CON) juveniles, 54 ASD and 54 CON adults). Six TNSs were divided into the DMN-TNSs, ECN-TNSs, and SN-TNSs. The DMN-TNSs were major states with the highest stability and proportion, and the ECN-TNSs and SN-TNSs were minor states. Age-related abnormalities on spatial stability and TNS proportion were found in ASD. The spatial stability of DMN-TNSs was found increasing with age in CON, but was not found in ASD. A lower proportion of DMN-TNSs was found in ASD compared with CON of the same age, and ASD juveniles had a higher proportion of SN-TNSs while ASD adults had a higher proportion of ECN-TNSs. The abnormalities on spatial stability and TNS proportion were related to social deficits. Our results provided new evidence for atypical transient brain functions in people with ASD.

Individual differences in the neural dynamics of visual narrative comprehension: The effects of proficiency and age of acquisition

Understanding visual narrative sequences, as found in comics, is known to recruit similar cognitive mechanisms to verbal language. As measured by event-related potentials (ERPs), these manifest as initial negativities (N400, LAN) and subsequent positivities (P600). While these components are thought to index discrete processing stages, they differentially arise across participants for any given stimulus. In language contexts, proficiency modulates brain responses, with smaller N400 effects and larger P600 effects appearing with increasing proficiency. In visual narratives, recent work has also emphasized the role of proficiency in neural response patterns. We thus explored whether individual differences in proficiency modulate neural responses to visual narrative sequencing in similar ways as in language. We combined ERP data from 12 studies examining semantic and/or grammatical processing of visual narrative sequences. Using linear mixed effects modeling, we demonstrate differential effects of visual language proficiency and "age of acquisition" on N400 and P600 responses. Our results align with those reported in language contexts, providing further evidence for the similarity of linguistic and visual narrative processing, and emphasize the role of both proficiency and age of acquisition in visual narrative comprehension.

Neural correlates of audiovisual narrative speech perception in children and adults on the autism spectrum: A functional magnetic resonance imaging study

Autistic individuals show substantially reduced benefit from observing visual articulations during audiovisual speech perception, a multisensory integration deficit that is particularly relevant to social communication. This has mostly been studied using simple syllabic or word-level stimuli and it remains unclear how altered lower-level multisensory integration translates to the processing of more complex natural multisensory stimulus environments in autism. Here, functional neuroimaging was used to examine neural correlates of audiovisual gain (AV-gain) in 41 autistic individuals to those of 41 age-matched non-autistic controls when presented with a complex audiovisual narrative. Participants were presented with continuous narration of a story in auditory-alone, visual-alone, and both synchronous and asynchronous audiovisual speech conditions. We hypothesized that previously identified differences in audiovisual speech processing in autism would be characterized by activation differences in brain regions well known to be associated with audiovisual enhancement in neurotypicals. However, our results did not provide evidence for altered processing of auditory alone, visual alone, audiovisual conditions or AV- gain in regions associated with the respective task when comparing activation patterns between groups. Instead, we found that autistic individuals responded with higher activations in mostly frontal regions where the activation to the experimental conditions was below baseline (de-activations) in the control group. These frontal effects were observed in both unisensory and audiovisual conditions, suggesting that these altered activations were not specific to multisensory processing but reflective of more general mechanisms such as an altered disengagement of Default Mode Network processes during the observation of the language stimulus across conditions.

Semantic processing of verbal narratives compared to semantic processing of visual narratives: an ERP study of school-aged children

There is a misconception that pictures are easy to comprehend, which is problematic in pedagogical practices that include pictures. For example, if a child has difficulties with verbal narration to picture sequences, it may be interpreted as specific to spoken language even though the child may have additional difficulties with comprehension of visual narratives in the form of picture sequences. The purpose of the present study was therefore to increase our understanding of semantic processing in the pictorial domain in relation to semantic processing in the verbal domain, focusing on 9-13 years-old children with typical language development. To this end, we measured electrical brain responses (event related potentials, ERPs) in 17 children to (i) pictures (panels) that were predicted versus unpredicted in sequences of panels that conveyed visual narratives and (ii) words that were predicted versus unpredicted in sentences that conveyed verbal narratives. Results demonstrated similarities as there were no significant difference in the magnitude of the N400 effect across domains. The only difference between domains was the predicted difference in distribution, that is, a more posterior N400 effect in the verbal domain than in the pictorial domain. The study contributes to an increased understanding of the complexity of processing of visual narratives and its shared features with processing of verbal narratives, which should be considered in pedagogical practices.

Normal but Different: Autistic Adolescents Who Score Within Normal Ranges on Standardized Language Tests Produce Frequent Linguistic Irregularities in Spontaneous Discourse

Background and aims

A substantial minority of autistic individuals score within typical ranges on standard language tests, suggesting that autism does not necessarily affect language acquisition. This idea is reflected in current diagnostic criteria for autism, wherein language impairment is no longer included. However, some work has suggested that probing autistic speakers' language carefully may reveal subtle differences between autistic and nonautistic people's language that cannot be captured by standardized language testing. The current study aims to test this idea, by determining whether a group of autistic and nonautistic individuals who score similarly on a standardized test show differences in the number of unconventional and erroneous language features they produce in a spontaneous language sample.

Methods

Thirty-eight older children and adolescents (19 autistic; 19 nonautistic), between the ages of 10 and 18, were recruited. Both participant groups scored within normal ranges on standardized language and IQ tests. Participants engaged in a "double interview" with an experimenter, during which they were first asked questions by the experimenter about themselves, and then they switched roles, so that it was the participant's turn to ask the experimenter questions. Participants' language during the interview was transcribed and analyzed for linguistic irregularities, including both semantic anomalies and morphosyntactic errors.

Results

Group membership accounted for significant variance in irregularity frequency; autistic participants produced more linguistic irregularities than nonautistic participants. Scores on a standardized language test did not improve model fit. Secondary analyses involving irregularity type (semantic vs. morphosyntactic) showed that group differences were primarily driven by relatively high numbers of semantic unconventionalities produced by the autistic group. While the autistic group made more morphosyntactic errors than the nonautistic group, differences in these numbers were only marginally significant.

Conclusions and implications

These findings suggest that a commonly used standardized language test does not adequately predict the number and perhaps type of language irregularities produced by some older autistic children and adolescents during spontaneous discourse. Results also suggest that differences in language use, especially semantic differences, may characterize autistic language, even the language produced by people who score within normal ranges on standardized language tests. It is debatable whether differences reflect underlying language impairments and/or a linguistic style adopted/preferred by autistic speakers. In this paper, we discuss both possibilities and offer suggestions to future research for teasing these possibilities apart.

Mapping access to meaning in adolescents with autism: Atypical lateralization and spatiotemporal patterns as a function of language ability

Individuals with autism spectrum disorders (ASD) vary in their language abilities, associated with atypical patterns of brain activity. However, few studies have examined the spatiotemporal profiles of lexico-semantic processing in ASD, particularly as a function of language heterogeneity. Thirty-nine high-functioning adolescents with ASD and 21 typically developing (TD) peers took part in a lexical decision task that combined semantic access with demands on cognitive control. Spatiotemporal characteristics of the processing stages were examined with a multimodal anatomically-constrained magnetoencephalography (aMEG) approach, which integrates MEG with structural MRI. Additional EEG data were acquired from a limited montage simultaneously with MEG. TD adolescents showed the canonical left-dominant activity in frontotemporal regions during both early (N250m) and late (N400m) stages of lexical access and semantic integration. In contrast, the ASD participants showed bilateral engagement of the frontotemporal language network, indicative of compensatory recruitment of the right hemisphere. The left temporal N400m was prominent in both groups, confirming preserved attempts to access meaning. In contrast, the left prefrontal N400m was reduced in ASD participants, consistent with impaired semantic/contextual integration and inhibitory control. To further investigate the impact of language proficiency, the ASD sample was stratified into high- and low-performing (H-ASD and L-ASD) subgroups based on their task accuracy. The H-ASD subgroup performed on par with the TD group and showed greater activity in the right prefrontal and bilateral temporal cortices relative to the L-ASD subgroup, suggesting compensatory engagement. The L-ASD subgroup additionally showed reduced and delayed left prefrontal N400m, consistent with more profound semantic and executive impairments in this subgroup. These distinct spatiotemporal activity profiles reveal the neural underpinnings of the ASD-specific access to meaning and provide insight into the phenotypic heterogeneity of language in ASD, which may be a result of different neurodevelopmental trajectories and adoption of compensatory strategies.

Sex differences in autistic adults: A preliminary study showing differences in mentalizing, but not in narrative coherence

Studying autism might be a complex endeavor due to its clinical heterogeneity. Little is currently known about potential sex differences in autistic adults, especially regarding mentalizing and narrative coherence. In this study, male and female participants told a personal story about one of their most positive and most negative life events and performed two mentalizing tasks. One of these mentalizing tasks was a recently developed Picture and Verbal Sequencing task that has shown cerebellar recruitment, and which requires mentalizing in a sequential context (i.e., participants chronologically ordered scenarios that required true and false belief mentalizing). Our preliminary comparison shows that males were faster and more accurate on the Picture Sequencing task compared to female participants when ordering sequences involving false beliefs, but not true beliefs. No sex differences were found for the other mentalizing and narrative tasks. These results highlight the importance of looking at sex differences in autistic adults and provide a possible explanation for sex-related differences in daily life mentalizing functions, which suggest a need for more sensitive diagnosis and tailored support.

Mentalizing and narrative coherence in autistic adults: Cerebellar sequencing and prediction

BYLEMANS, T., et al. Mentalizing and narrative coherence in autistic adults: Cerebellar sequencing and prediction. NEUROSCI BIOBEHAV REV, 2022. - This review focuses on autistic adults and serves 4 purposes: (1) providing an overview of their difficulties regarding mentalizing (understanding others' mental states) and narrative coherence (structured storytelling), (2) highlighting the relations between both skills by examining behavioral observations and shared neural substrates, (3) providing an integrated perspective regarding novel diagnostic tools and support services, and (4) raising awareness of adult autism. We suggest that mentalizing and narrative coherence are related at the behavioral level and neural level. In addition to the traditional mentalizing network, the cerebellum probably serves as an important hub in shared cerebral networks implicated in mentalizing and narrative coherence. Future autism research and support services should tackle new questions within a framework of social cerebellar (dys)functioning.

Story Comprehension Monitoring Across Visual, Listening, and Written Modalities in Children with and Without Autism Spectrum Disorder

Visual, as compared to verbal, tasks are often assumed to be easier for individuals with autism spectrum disorder (ASD), but is this true for story comprehension? This study evaluated story comprehension monitoring across visual, listening, and written modalities and assessed predictors in two closely matched groups (age, socioeconomic status, language, nonverbal cognition, and word reading) of children and adolescents (8-14 years) with ASD (n = 20) and typical development (typically developing [TD]; n = 20). The results of mixed-effects models indicated that story comprehension monitoring was low overall, and performance was comparable across visual, listening, and written modalities for participants with ASD. Age, vocabulary, nonverbal working memory, response and distractor inhibition, and social communication significantly predicted comprehension monitoring.

Children with autism spectrum disorder show atypical electroencephalographic response to processing contextual incongruencies

Children with autism spectrum disorder (ASD) experience difficulties with social communication, making it challenging to interpret contextual information that aids in accurately interpreting language. To investigate how the brain processes the contextual information and how this is different in ASD, we compared event-related potentials (ERPs) in response to processing visual and auditory congruent and incongruent information. Two groups of children participated in the study: 37 typically developing children and 15 children with ASD (age range = 6 to 12). We applied a language task involving auditory sentences describing congruent or incongruent images. We investigated two ERP components associated with language processing: the N400 and P600. Our results showed how children with ASD present significant differences in their neural responses in comparison with the TD group, even when their reaction times and correct trials are not significantly different from the TD group.

Speech Processing in Autism Spectrum Disorder: An Integrative Review of Auditory Neurophysiology Findings

Purpose Investigations into the nature of communication disorders in autistic individuals increasingly evaluate neural responses to speech stimuli. This integrative review aimed to consolidate the available data related to speech and language processing across levels of stimulus complexity (from single speech sounds to sentences) and to relate it to the current theories of autism. Method An electronic database search identified peer-reviewed articles using event-related potentials or magnetoencephalography to investigate auditory processing from single speech sounds to sentences in autistic children and adults varying in language and cognitive abilities. Results Atypical neural responses in autistic persons became more prominent with increasing stimulus and task complexity. Compared with their typically developing peers, autistic individuals demonstrated mostly intact sensory responses to single speech sounds, diminished spontaneous attentional orienting to spoken stimuli, specific difficulties with categorical speech sound discrimination, and reduced processing of semantic content. Atypical neural responses were more often observed in younger autistic participants and in those with concomitant language disorders. Conclusions The observed differences in neural responses to speech stimuli suggest that communication difficulties in autistic individuals are more consistent with the reduced social interest than the auditory dysfunction explanation. Current limitations and future directions for research are also discussed.

Developmental differences in neural connectivity for semantic processing in youths with autism

BACKGROUND

Youths with autism spectrum disorder (ASD) rely more on lower-level visual processing as revealed by greater occipital activation, yet less effectively engage higher-level processing of modality-independent semantic knowledge as indicated by reduced frontal activation, compared to typically developing (TD) youths. However, little is known about age-dependent differences in neural connectivity during semantic processing in youths with ASD as compared to TD youths.

METHODS

Four groups were recruited: 31 ASD children (mean age = 10.5 years old), 33 TD children (mean age = 10.4), 30 ASD adolescents (mean age = 14.9), and 34 TD adolescents (mean age = 15.1). We explored their differences in neural connectivity by using functional magnetic resonance imaging (fMRI) with psychophysiological interaction (PPI) during semantic judgments.

RESULTS

In comparison with TD children, children with ASD showed greater activation in the left cuneus and weaker connectivity between the left cuneus and left middle temporal gyrus (MTG). In comparison with TD adolescents, adolescents with ASD showed less activation in the left inferior frontal gyrus (IFG) and weaker functional connectivity between the left IFG and left MTG.

CONCLUSIONS

Children with ASD may rely more on visual processes in the occipital cortex that are disconnected from modality-independent semantics in the temporal cortex. However, adolescents with ASD may less effectively engage frontal mechanisms involved in the top-down control of modality-independent semantic knowledge in the temporal cortex. Our findings provide evidence of developmental differences in the neural substrates of the alterations in semantic processing in youths with ASD compared to TD youths.

The electrophysiology of audiovisual processing in visual narratives in adolescents with autism spectrum disorder

We investigated the semantic processing of the multimodal audiovisual combination of visual narratives with auditory descriptive words and auditory sounds in individuals with ASD. To this aim, we recorded ERPs to critical auditory words and sounds associated with events in visual narrative that were either semantically congruent or incongruent with the climactic visual event. A similar N400 effect was found both in adolescents with ASD and neurotypical adolescents (ages 9-16) when accessing different types of auditory information (i.e. words and sounds) into a visual narrative. This result might suggest that verbal information processing in ASD adolescents could be facilitated by direct association with meaningful visual information. In addition, we observed differences in scalp distribution of later brain responses between ASD and neurotypical adolescents. This finding might suggest ASD adolescents differ from neurotypical adolescents during the processing of the multimodal combination of visual narratives with auditory information at later stages of the process. In conclusion, the semantic processing of verbal information, typically impaired in individuals with ASD, can be facilitated when embedded into a meaningful visual information.

Language Abilities of Russian Primary-School-Aged Children with Autism Spectrum Disorder: Evidence from Comprehensive Assessment

The purpose of the present research was to comprehensively assess the language abilities of Russian primary-school-aged children with Autism Spectrum Disorder (ASD), varying in non-verbal IQ, at all linguistic levels (phonology, lexicon, morphosyntax, and discourse) in production and comprehension. Yet, the influence of such non-language factors as children's age, the severity of autistic traits, and non-verbal IQ on language functioning was studied. Our results indicate a high variability of language skills in children with ASD (from normal to impaired) which is in line with the previous studies. Interestingly, the number of children with normal language abilities was related to the linguistic levels: according to more complex morphosyntax and discourse tests, fewer children with ASD were within the normal range unlike the results in simpler phonological and lexical tests. Importantly, we found that language abilities were best predicted by non-verbal IQ but were independent from age and the severity of autistic traits. The findings support the claim that formal language assessment of children with ASD needs to include all linguistic levels, from phonology to discourse, for helping speech-language therapists to choose an appropriate therapy target.

Implicit Semantic Processing of Linguistic and Non-linguistic Stimuli in Adults with Autism Spectrum Disorder

While many individuals with autism spectrum disorder (ASD) experience difficulties with language processing, non-linguistic semantic processing may be intact. We examined neural responses to an implicit semantic priming task by comparing N400 responses—an event-related potential related to semantic processing—in response to semantically related or unrelated pairs of words or pictures. Adults with ASD showed larger N400 responses than typically developing adults for pictures, but no group differences occurred for words. However, we also observed complex modulations of N400 amplitude by age and by level of autistic traits. These results offer important implications for how groups are delineated and compared in autism research.

Visual and Verbal Narrative Comprehension in Children and Adolescents with Autism Spectrum Disorders: An ERP Study

We examined semantic processing in ASD children by presenting sentences with congruent or incongruent final words and visual narratives with congruent or incongruent final panels. An N400 effect to incongruent words appeared as compared to congruent ones, which was attenuated for the ASD children. We observed a negativity sustained to incongruous than congruous words, but only for the TD children. Incongruent panels evoked a greater fronto-central N400 amplitude than congruent panels in both groups. In addition, incongruent panels evoked a centro-parietal late positivity, only in controls. In conclusion, ASD children face processing deficits in both verbal and visual materials when integrating meaning across information, though such impairments may arise in different parts of the interpretive process, depending on the modality.

Predictability modulates neurocognitive semantic processing of non-verbal narratives

Predictability is known to modulate semantic processing in language, but it is unclear to what extent this applies for other modalities. Here we ask whether similar cognitive processes are at play in predicting upcoming events in a non-verbal visual narrative. Typically developing adults viewed comics sequences in which a target panel was highly predictable (“high cloze”), less predictable (“low cloze”), or incongruent with the preceding narrative context (“anomalous”) during EEG recording. High and low predictable sequences were determined by a pretest where participants assessed “what happened next?”, resulting in cloze probability scores for sequence outcomes comparable to those used to measure predictability in sentence processing. Through both factorial and correlational analyses, we show a significant modulation of neural responses by cloze such that N400 effects are diminished as a target panel in a comic sequence becomes more predictable. Predictability thus appears to play a similar role in non-verbal comprehension of sequential images as in language comprehension, providing further evidence for the domain generality of semantic processing in the brain.

Narrative Skills in Primary School Children with Autism in Relation to Language and Nonverbal Temporal Sequencing

Recent research has suggested that temporal sequencing of narrative events might be a domain-general ability that underlies oral narrative capacities. The current study investigated this issue in a group of children with known pragmatic and narrative difficulties, namely Autism Spectrum Disorder (ASD). We hypothesized (1) that children with ASD (n = 45) would retell narratives of poorer quality than both chronological age-matched (CAM) children and younger children matched on sentence-level language skills (LM), and (2) that nonverbal temporal sequencing skills would uniquely predict individual differences in oral narrative performance in children with ASD. The results show that children with ASD performed poorer on all measures of oral narrative quality compared with the CAM group, and on eight of ten measures compared with the LM group. Thus, our first hypothesis was confirmed, suggesting that narrative difficulties in ASD cannot be fully explained by impaired language. The second hypothesis was only partly confirmed: nonverbal temporal sequencing explained significant or marginally significant variance in some, but not all, aspects of oral narrative performance of children with ASD. These results are discussed from theoretical and clinical/educational perspectives, in relation to the heterogeneity of language skills in ASD and to domain-general features of narrative processing.

An investigation of visual narrative comprehension in children with autism spectrum disorders

The present study analyzed the comprehension of visual narrative in a group of twelve children with autism spectrum disorders (ASD). Their performances were compared to a control group of fifteen children with typical development (TD) matched for age, level of formal education, and IQ. Visual narrative comprehension was assessed by administering a task that required children to understand narrative's global coherence by arranging in the correct order the constituent parts of stories presented in pictures. Specifically, the task evaluated children's ability to grasp how single events connected (causally and temporally) each other and how these connections led to the ending of the story. Results showed that children with ASD obtained significantly lower scores than children with TD. These results open to alternative interpretations of narrative impairments often reported in individuals with ASD, which might not be restricted to the linguistic code but stem from a deeper deficit in narrative processing that is independent from the expressive modality.

Visual narrative comprehension: Universal or not?

Visual narratives of sequential images - as found in comics, picture stories, and storyboards - are often thought to provide a fairly universal and transparent message that requires minimal learning to decode. This perceived transparency has led to frequent use of sequential images as experimental stimuli in the cognitive and psychological sciences to explore a wide range of topics. In addition, it underlines efforts to use visual narratives in science and health communication and as educational materials in both classroom settings and across developmental, clinical, and non-literate populations. Yet, combined with recent studies from the linguistic and cognitive sciences, decades of research suggest that visual narratives involve greater complexity and decoding than widely assumed. This review synthesizes observations from cross-cultural and developmental research on the comprehension and creation of visual narrative sequences, as well as findings from clinical psychology (e.g., autism, developmental language disorder, aphasia). Altogether, this work suggests that understanding the visual languages found in comics and visual narratives requires a fluency that is contingent on exposure and practice with a graphic system.

Editors’ Introduction and Review: Visual Narrative Research: An Emerging Field in Cognitive Science

Drawn sequences of images are among our oldest records of human intelligence, appearing on cave paintings, wall carvings, and ancient pottery, and they pervade across cultures from instruction manuals to comics. They also appear prevalently as stimuli across Cognitive Science, for studies of temporal cognition, event structure, social cognition, discourse, and basic intelligence. Yet, despite this fundamental place in human expression and research on cognition, the study of visual narratives themselves has only recently gained traction in Cognitive Science. This work has suggested that visual narrative comprehension requires cultural exposure across a developmental trajectory and engages with domain‐general processing mechanisms shared by visual perception, attention, event cognition, and language, among others. Here, we review the relevance of such research for the broader Cognitive Science community, and make the case for why researchers should join the scholarship of this ubiquitous but understudied aspect of human expression.

Drawn sequences of images, like those in comics and picture stories, are a pervasive and fundamental way that humans have communicated for millennia. Yet, the study of visual narratives has only recently gained traction in Cognitive Science. Here we explore what has held back the study of the cognition of visual narratives, and why researchers should join in scholarship of this ubiquitous aspect of expression.

Uncovering cortical activations of discourse comprehension and their overlaps with common large-scale neural networks

We conducted a meta-analysis of 78 task-based functional magnetic resonance imaging (fMRI) studies (1976 total participants) to reveal underlying brain activations and their overlap with large-scale neural networks in the brain during general discourse comprehension and its sub-processes. We found that discourse comprehension involved a neural system consisting of widely distributed brain regions that comprised not only the bilateral perisylvian language zones, but also regions in the superior and medial frontal cortex and the medial temporal lobe. Moreover, this neural system can be categorized into several sub-systems representing various sub-processes of discourse comprehension, with the left inferior frontal gyrus and middle temporal gyrus serving as core regions across all sub-processes. At a large-scale network level, we found that discourse comprehension relied most heavily on the default network, particularly on its dorsal medial subsystem. The pattern associated with large-scale network cooperation varied according to the respective sub-processes required. Our results reveal the functional dissociation within the discourse comprehension neural system and highlight the flexible involvements of large-scale networks.

Your Brain on Comics: A Cognitive Model of Visual Narrative Comprehension

The past decade has seen a rapid growth of cognitive and brain research focused on visual narratives like comics and picture stories. This paper will summarize and integrate this emerging literature into the Parallel Interfacing Narrative‐Semantics Model (PINS Model)—a theory of sequential image processing characterized by an interaction between two representational levels: semantics and narrative structure. Ongoing semantic processes build meaning into an evolving mental model of a visual discourse. Updating of spatial, referential, and event information then incurs costs when they are discontinuous with the growing context. In parallel, a narrative structure organizes semantic information into coherent sequences by assigning images to categorical roles, which are then embedded within a hierarchic constituent structure. Narrative constructional schemas allow for specific predictions of structural sequencing, independent of semantics. Together, these interacting levels of representation engage in an iterative process of retrieval of semantic and narrative information, prediction of upcoming information based on those assessments, and subsequent updating based on discontinuity. These core mechanisms are argued to be domain‐general—spanning across expressive systems—as suggested by similar electrophysiological brain responses (N400, P600, anterior negativities) generated in response to manipulation of sequential images, music, and language. Such similarities between visual narratives and other domains thus pose fundamental questions for the linguistic and cognitive sciences.

Visual narratives like comics involve a range of complex cognitive operations in order to be understood. The Parallel Interfacing Narrative‐Semantics (PINS) Model integrates an emerging literature showing that comprehension of wordless image sequences balances two representational levels of semantic and narrative structure. The neurocognitive mechanisms that guide these processes are argued to overlap with other domains, such as language and music.