Listening beyond seeing: Event-related potentials to audiovisual processing in visual narrative

Intimate Relationships Weaken Female Competition: Evidence from Phase-amplitude Coupling and Event-Related Potentials

Competition, an essential component of social interaction, frequently occurs in daily life, and the impact of intimate relationships on women's competition has not yet been revealed. In this study, the visual target paradigm was used to explore the neural mechanisms underlying the regulation of female competitiveness by intimate relationships using event-related potential (ERP) data, time-frequency analysis, and brain functional connectivity. The research results indicate that, the P1, the N4, and the LPP were sensitive to the impact of intimate relationships on competition. Compared to competition between unfamiliar opposite-gender dyads, the average amplitudes of the N4 and LPP were higher in the late stage of competition between romantic partners. Compared to competition with strangers of the opposite gender, alpha band power was significantly higher when female individuals competed with their romantic partners. In addition, there was a positive correlation between the synchronization of activity in the frontal, parietal, and right temporal lobes of a female's brain and their degree of female engagement in competition. When a female individuals focused on the competition, activity synchronization was higher. The results indicate that competition with unfamiliar individuals of the opposite gender can make female focus on the competitive task, causing synchronous activation of corresponding brain regions. When competing with a romantic partner, women's focus decreases, their willingness to compete decreases, and the synchrony of brain functional connectivity decreases. This study suggests that intimate relationship weakens women's competitiveness, which is of significant importance for understanding high-quality intimate relationship and promoting the development of healthy competition.

Modulation of Spectral Representation and Connectivity Patterns in Response to Visual Narrative in the Human Brain

The regional brain networks and the underlying neurophysiological mechanisms subserving the cognition of visual narrative in humans have largely been studied with non-invasive brain recording. In this study, we specifically investigated how regional and cross-regional cortical activities support visual narrative interpretation using intracranial stereotactic electroencephalograms recordings from thirteen human subjects (6 females, and 7 males). Widely distributed recording sites across the brain were sampled while subjects were explicitly instructed to observe images from fables presented in “sequential” order, and a set of images drawn from multiple fables presented in “scrambled” order. Broadband activity mainly within the frontal and temporal lobes were found to encode if a presented image is part of a visual narrative (sequential) or random image set (scrambled). Moreover, the temporal lobe exhibits strong activation in response to visual narratives while the frontal lobe is more engaged when contextually novel stimuli are presented. We also investigated the dynamics of interregional interactions between visual narratives and contextually novel series of images. Interestingly, the interregional connectivity is also altered between sequential and scrambled sequences. Together, these results suggest that both changes in regional neuronal activity and cross-regional interactions subserve visual narrative and contextual novelty processing.

Auditory dominance in processing Chinese semantic abnormalities in response to competing audio-visual stimuli

Language is a remarkable cognitive ability that can be expressed through visual (written language) or auditory (spoken language) modalities. When visual characters and auditory speech convey conflicting information, individuals may selectively attend to either one of them. However, the dominant modality in such a competing situation and the neural mechanism underlying it are still unclear. Here, we presented participants with Chinese sentences in which the visual characters and auditory speech convey conflicting information, while behavioral and electroencephalographic (EEG) responses were recorded. Results showed a prominent auditory dominance when audio-visual competition occurred. Specifically, higher accuracy (ACC), larger N400 amplitudes and more linkages in the posterior occipital-parietal areas were demonstrated in the auditory mismatch condition compared to that in the visual mismatch condition. O0ur research illustrates the superiority of the auditory speech over the visual characters, extending our understanding of the neural mechanisms of audio-visual competition in Chinese.

Neural correlates of sex differences in communicative gestures and speech comprehension: A preliminary study

The goal of this study was to investigate whether the semantic processing of the audiovisual combination of communicative gestures with speech differs between men and women. We recorded event-related brain potentials in women and men during the presentation of communicative gestures that were either congruent or incongruent with the speech.Our results showed that incongruent gestures elicited an N400 effect over frontal sites compared to congruent ones in both groups. Moreover, the females showed an earlier N2 response to incongruent stimuli than congruent ones, while larger sustained negativity and late positivity in response to incongruent stimuli was observed only in males. These results suggest that women rapidly recognize and process audiovisual combinations of communicative gestures and speech (as early as 300 ms) whereas men analyze them at the later stages of the process.

Generalizable EEG Encoding Models with Naturalistic Audiovisual Stimuli

In natural conversations, listeners must attend to what others are saying while ignoring extraneous background sounds. Recent studies have used encoding models to predict electroencephalography (EEG) responses to speech in noise-free listening situations, sometimes referred to as "speech tracking." Researchers have analyzed how speech tracking changes with different types of background noise. It is unclear, however, whether neural responses from acoustically rich, naturalistic environments with and without background noise can be generalized to more controlled stimuli. If encoding models for acoustically rich, naturalistic stimuli are generalizable to other tasks, this could aid in data collection from populations of individuals who may not tolerate listening to more controlled and less engaging stimuli for long periods of time. We recorded noninvasive scalp EEG while 17 human participants (8 male/9 female) listened to speech without noise and audiovisual speech stimuli containing overlapping speakers and background sounds. We fit multivariate temporal receptive field encoding models to predict EEG responses to pitch, the acoustic envelope, phonological features, and visual cues in both stimulus conditions. Our results suggested that neural responses to naturalistic stimuli were generalizable to more controlled datasets. EEG responses to speech in isolation were predicted accurately using phonological features alone, while responses to speech in a rich acoustic background were more accurate when including both phonological and acoustic features. Our findings suggest that naturalistic audiovisual stimuli can be used to measure receptive fields that are comparable and generalizable to more controlled audio-only stimuli.SIGNIFICANCE STATEMENT Understanding spoken language in natural environments requires listeners to parse acoustic and linguistic information in the presence of other distracting stimuli. However, most studies of auditory processing rely on highly controlled stimuli with no background noise, or with background noise inserted at specific times. Here, we compare models where EEG data are predicted based on a combination of acoustic, phonetic, and visual features in highly disparate stimuli-sentences from a speech corpus and speech embedded within movie trailers. We show that modeling neural responses to highly noisy, audiovisual movies can uncover tuning for acoustic and phonetic information that generalizes to simpler stimuli typically used in sensory neuroscience experiments.

Crossmodal processing of environmental sounds and everyday life actions: An ERP study

To investigate the processing of environmental sounds, previous researchers have compared the semantic processing of words and sounds, yielding mixed results. This study aimed to specifically investigate the electrophysiological mechanism underlying the semantic processing of environmental sounds presented in a naturalistic visual scene. We recorded event-related brain potentials in a group of young adults over the presentation of everyday life actions that were either congruent or incongruent with environmental sounds. Our results showed that incongruent environmental sounds evoked both a P400 and an N400 effect, reflecting sensitivity to physical and semantic violations of environmental sounds’ properties, respectively. In addition, our findings showed an enhanced late positivity in response to incongruous environmental sounds, probably reflecting additional reanalysis costs. In conclusion, these results indicate that the crossmodal processing of the environmental sounds might require the simultaneous involvement of different cognitive processes.

When dogs meow: An electrophysiological study of lexical-semantic processing in toddlers

The goal of this study was to investigate lexical-semantic processing at an early phase of language development. Adults often communicate with children using infant-directed speech that typically involves lexical and syntactic modifications such as onomatopoeias (Soderstrom, 2007). Here, we asked how and when children start to show an advantage for processing conventional linguistic forms, such as common nouns, and consequently decreasing sensitivity to onomatopoeias. We recorded event-related brain potentials in children of two age groups (16-21 months and 24-31 months) and in an adult control group during the presentation of four conditions in which either common nouns or onomatopoeias were presented auditorily followed by a picture of an either congruent or incongruent object. We focused on the N400 effect, a more negative ERP response to incongruent compared with congruent semantic relations. The younger children showed an N400 effect only for onomatopoeic words, while the older children showed an N400 effect only for common nouns. The adults showed N400 effects for onomatopoeia and nouns. These different N400 effects suggest that onomatopoeia and common nouns are differently organized in children's semantic memory and that the acquisition of linguistic abilities affects and modifies semantic processing of different lexical information.

The electrophysiology of audiovisual processing in visual narratives in adolescents with autism spectrum disorder

We investigated the semantic processing of the multimodal audiovisual combination of visual narratives with auditory descriptive words and auditory sounds in individuals with ASD. To this aim, we recorded ERPs to critical auditory words and sounds associated with events in visual narrative that were either semantically congruent or incongruent with the climactic visual event. A similar N400 effect was found both in adolescents with ASD and neurotypical adolescents (ages 9-16) when accessing different types of auditory information (i.e. words and sounds) into a visual narrative. This result might suggest that verbal information processing in ASD adolescents could be facilitated by direct association with meaningful visual information. In addition, we observed differences in scalp distribution of later brain responses between ASD and neurotypical adolescents. This finding might suggest ASD adolescents differ from neurotypical adolescents during the processing of the multimodal combination of visual narratives with auditory information at later stages of the process. In conclusion, the semantic processing of verbal information, typically impaired in individuals with ASD, can be facilitated when embedded into a meaningful visual information.

Neural representation of linguistic feature hierarchy reflects second-language proficiency

Acquiring a new language requires individuals to simultaneously and gradually learn linguistic attributes on multiple levels. Here, we investigated how this learning process changes the neural encoding of natural speech by assessing the encoding of the linguistic feature hierarchy in second-language listeners. Electroencephalography (EEG) signals were recorded from native Mandarin speakers with varied English proficiency and from native English speakers while they listened to audio-stories in English. We measured the temporal response functions (TRFs) for acoustic, phonemic, phonotactic, and semantic features in individual participants and found a main effect of proficiency on linguistic encoding. This effect of second-language proficiency was particularly prominent on the neural encoding of phonemes, showing stronger encoding of “new” phonemic contrasts (i.e., English contrasts that do not exist in Mandarin) with increasing proficiency. Overall, we found that the nonnative listeners with higher proficiency levels had a linguistic feature representation more similar to that of native listeners, which enabled the accurate decoding of language proficiency. This result advances our understanding of the cortical processing of linguistic information in second-language learners and provides an objective measure of language proficiency.

Visual and Verbal Narrative Comprehension in Children and Adolescents with Autism Spectrum Disorders: An ERP Study

We examined semantic processing in ASD children by presenting sentences with congruent or incongruent final words and visual narratives with congruent or incongruent final panels. An N400 effect to incongruent words appeared as compared to congruent ones, which was attenuated for the ASD children. We observed a negativity sustained to incongruous than congruous words, but only for the TD children. Incongruent panels evoked a greater fronto-central N400 amplitude than congruent panels in both groups. In addition, incongruent panels evoked a centro-parietal late positivity, only in controls. In conclusion, ASD children face processing deficits in both verbal and visual materials when integrating meaning across information, though such impairments may arise in different parts of the interpretive process, depending on the modality.

Electrophysiological indexes of ToM and non-ToM humor in healthy adults

The cognitive processes involved in humor comprehension were analyzed by directly comparing the time course of brain activity associated with the perception of slapstick humor and that associated with the comprehension of humor requiring theory of mind (ToM). Four different comic strips (strips containing humorous scenes that required ToM, non-ToM humorous strips, non-humorous semantically coherent strips and non-humorous semantically incoherent strips) were presented to participants, while their EEG response was recorded. Results showed that both of the humorous comic strips and the semantically incongruent strip elicited an N400 effect, suggesting similar cognitive mechanisms underlying the processing of incongruent and humorous comic strips. The results also showed that the humorous ToM strips elicited a frontal late positive (LP) response, possibly reflecting the active deployment of ToM abilities such as perspective-taking and empathy that allow for the resolution and interpretation of apparently incongruent situations. In addition, the LP response was positively correlated with ratings of perceived amusement as well as individual empathy scores, suggesting that the increased LP response to ToM humorous strips reflects the combined activation of neural mechanisms involved in the experience of amusement and ToM abilities. Overall, humor comprehension appears to demand distinct cognitive steps such as the detection of incongruent semantic components, the construction of semantic coherence, and the appreciation of humoristic elements such as maladaptive emotional reactions. Our results show that the deployment of these distinct cognitive steps is at least partially dependent on individual empathic abilities.

Multimodal semantic revision during inferential processing: The role of inhibitory control in text and picture comprehension

Although language comprehension usually requires multimodal information, no study to date has investigated how comprehenders deal with the revision of a text's interpretation when different modalities are involved. Twenty-four young adults listened to a story prompting an inference (e.g., polar bear), and then saw a picture that was either consistent (polar bear) or inconsistent but still plausible (penguin). Larger negativity (N400) in the inconsistent picture indicated successful inferential monitoring. Subsequently, a sentence carried the disambiguating word which was either expected ("bear") or unexpected ("penguin") in relation to the auditory-verbal information. Larger negativity in the unexpected word coming from the consistent picture suggested that comprehenders had difficulties selecting the unexpected concept when previous information was contradictory. More importantly, this effect was modulated by inhibitory control, where a higher resistance to distractor interference (flanker task) was associated with a better ability to suppress pictorial information, therefore preventing semantic competition. Similarly, accuracy measured in a final comprehension question demonstrated that higher inhibitory control was related to a more efficient ability to revise the situation model across modalities. Our findings speak to a relationship between story comprehension and mental flexibility during multimodal processing.

Editors’ Introduction and Review: Visual Narrative Research: An Emerging Field in Cognitive Science

Drawn sequences of images are among our oldest records of human intelligence, appearing on cave paintings, wall carvings, and ancient pottery, and they pervade across cultures from instruction manuals to comics. They also appear prevalently as stimuli across Cognitive Science, for studies of temporal cognition, event structure, social cognition, discourse, and basic intelligence. Yet, despite this fundamental place in human expression and research on cognition, the study of visual narratives themselves has only recently gained traction in Cognitive Science. This work has suggested that visual narrative comprehension requires cultural exposure across a developmental trajectory and engages with domain‐general processing mechanisms shared by visual perception, attention, event cognition, and language, among others. Here, we review the relevance of such research for the broader Cognitive Science community, and make the case for why researchers should join the scholarship of this ubiquitous but understudied aspect of human expression.

Drawn sequences of images, like those in comics and picture stories, are a pervasive and fundamental way that humans have communicated for millennia. Yet, the study of visual narratives has only recently gained traction in Cognitive Science. Here we explore what has held back the study of the cognition of visual narratives, and why researchers should join in scholarship of this ubiquitous aspect of expression.

The Scene Perception & Event Comprehension Theory (SPECT) Applied to Visual Narratives

Understanding how people comprehend visual narratives (including picture stories, comics, and film) requires the combination of traditionally separate theories that span the initial sensory and perceptual processing of complex visual scenes, the perception of events over time, and comprehension of narratives. Existing piecemeal approaches fail to capture the interplay between these levels of processing. Here, we propose the Scene Perception & Event Comprehension Theory (SPECT), as applied to visual narratives, which distinguishes between front‐end and back‐end cognitive processes. Front‐end processes occur during single eye fixations and are comprised of attentional selection and information extraction. Back‐end processes occur across multiple fixations and support the construction of event models, which reflect understanding of what is happening now in a narrative (stored in working memory) and over the course of the entire narrative (stored in long‐term episodic memory). We describe relationships between front‐ and back‐end processes, and medium‐specific differences that likely produce variation in front‐end and back‐end processes across media (e.g., picture stories vs. film). We describe several novel research questions derived from SPECT that we have explored. By addressing these questions, we provide greater insight into how attention, information extraction, and event model processes are dynamically coordinated to perceive and understand complex naturalistic visual events in narratives and the real world.

Comprehension of visual narratives like comics, picture stories, and films involves both decoding the visual content and construing the meaningful events they represent. The Scene Perception & Event Comprehension Theory (SPECT) proposes a framework for understanding how a comprehender perceptually negotiates the surface of a visual representation and integrates its meaning into a growing mental model.

Your Brain on Comics: A Cognitive Model of Visual Narrative Comprehension

The past decade has seen a rapid growth of cognitive and brain research focused on visual narratives like comics and picture stories. This paper will summarize and integrate this emerging literature into the Parallel Interfacing Narrative‐Semantics Model (PINS Model)—a theory of sequential image processing characterized by an interaction between two representational levels: semantics and narrative structure. Ongoing semantic processes build meaning into an evolving mental model of a visual discourse. Updating of spatial, referential, and event information then incurs costs when they are discontinuous with the growing context. In parallel, a narrative structure organizes semantic information into coherent sequences by assigning images to categorical roles, which are then embedded within a hierarchic constituent structure. Narrative constructional schemas allow for specific predictions of structural sequencing, independent of semantics. Together, these interacting levels of representation engage in an iterative process of retrieval of semantic and narrative information, prediction of upcoming information based on those assessments, and subsequent updating based on discontinuity. These core mechanisms are argued to be domain‐general—spanning across expressive systems—as suggested by similar electrophysiological brain responses (N400, P600, anterior negativities) generated in response to manipulation of sequential images, music, and language. Such similarities between visual narratives and other domains thus pose fundamental questions for the linguistic and cognitive sciences.

Visual narratives like comics involve a range of complex cognitive operations in order to be understood. The Parallel Interfacing Narrative‐Semantics (PINS) Model integrates an emerging literature showing that comprehension of wordless image sequences balances two representational levels of semantic and narrative structure. The neurocognitive mechanisms that guide these processes are argued to overlap with other domains, such as language and music.