1
|
Jia S, Meng Y, Wang Q, Ao L, Gao Y, Yang L, Wang H, Liu Y. Intimate Relationships Weaken Female Competition: Evidence from Phase-amplitude Coupling and Event-Related Potentials. Neuroscience 2023; 534:41-53. [PMID: 37884087 DOI: 10.1016/j.neuroscience.2023.10.013] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/16/2023] [Revised: 09/22/2023] [Accepted: 10/17/2023] [Indexed: 10/28/2023]
Abstract
Competition, an essential component of social interaction, frequently occurs in daily life, and the impact of intimate relationships on women's competition has not yet been revealed. In this study, the visual target paradigm was used to explore the neural mechanisms underlying the regulation of female competitiveness by intimate relationships using event-related potential (ERP) data, time-frequency analysis, and brain functional connectivity. The research results indicate that, the P1, the N4, and the LPP were sensitive to the impact of intimate relationships on competition. Compared to competition between unfamiliar opposite-gender dyads, the average amplitudes of the N4 and LPP were higher in the late stage of competition between romantic partners. Compared to competition with strangers of the opposite gender, alpha band power was significantly higher when female individuals competed with their romantic partners. In addition, there was a positive correlation between the synchronization of activity in the frontal, parietal, and right temporal lobes of a female's brain and their degree of female engagement in competition. When a female individuals focused on the competition, activity synchronization was higher. The results indicate that competition with unfamiliar individuals of the opposite gender can make female focus on the competitive task, causing synchronous activation of corresponding brain regions. When competing with a romantic partner, women's focus decreases, their willingness to compete decreases, and the synchrony of brain functional connectivity decreases. This study suggests that intimate relationship weakens women's competitiveness, which is of significant importance for understanding high-quality intimate relationship and promoting the development of healthy competition.
Collapse
Affiliation(s)
- Shuyu Jia
- Yingjie Liu, School of Public Health, School of Psychology and Mental Health, North China University of Science and Technology, Hebei, China
| | - Yujia Meng
- Yingjie Liu, School of Public Health, School of Psychology and Mental Health, North China University of Science and Technology, Hebei, China
| | - Qian Wang
- Yingjie Liu, School of Public Health, School of Psychology and Mental Health, North China University of Science and Technology, Hebei, China
| | - Lihong Ao
- Yingjie Liu, School of Public Health, School of Psychology and Mental Health, North China University of Science and Technology, Hebei, China
| | - Yuan Gao
- Yingjie Liu, School of Public Health, School of Psychology and Mental Health, North China University of Science and Technology, Hebei, China
| | - Lei Yang
- Yingjie Liu, School of Public Health, School of Psychology and Mental Health, North China University of Science and Technology, Hebei, China
| | - He Wang
- Yingjie Liu, School of Public Health, School of Psychology and Mental Health, North China University of Science and Technology, Hebei, China
| | - Yingjie Liu
- Yingjie Liu, School of Public Health, School of Psychology and Mental Health, North China University of Science and Technology, Hebei, China.
| |
Collapse
|
2
|
Li A, Ma X. Scalable Cognitive Developmental Network:a strategy for integrating new perception online using relation evolution SOINN. COGN SYST RES 2023. [DOI: 10.1016/j.cogsys.2023.02.001] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/12/2023]
|
3
|
Sabra Z, Alawieh A, Bonilha L, Naselaris T, AuYong N. Modulation of Spectral Representation and Connectivity Patterns in Response to Visual Narrative in the Human Brain. Front Hum Neurosci 2022; 16:886938. [PMID: 36277048 PMCID: PMC9582122 DOI: 10.3389/fnhum.2022.886938] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/01/2022] [Accepted: 06/22/2022] [Indexed: 11/24/2022] Open
Abstract
The regional brain networks and the underlying neurophysiological mechanisms subserving the cognition of visual narrative in humans have largely been studied with non-invasive brain recording. In this study, we specifically investigated how regional and cross-regional cortical activities support visual narrative interpretation using intracranial stereotactic electroencephalograms recordings from thirteen human subjects (6 females, and 7 males). Widely distributed recording sites across the brain were sampled while subjects were explicitly instructed to observe images from fables presented in “sequential” order, and a set of images drawn from multiple fables presented in “scrambled” order. Broadband activity mainly within the frontal and temporal lobes were found to encode if a presented image is part of a visual narrative (sequential) or random image set (scrambled). Moreover, the temporal lobe exhibits strong activation in response to visual narratives while the frontal lobe is more engaged when contextually novel stimuli are presented. We also investigated the dynamics of interregional interactions between visual narratives and contextually novel series of images. Interestingly, the interregional connectivity is also altered between sequential and scrambled sequences. Together, these results suggest that both changes in regional neuronal activity and cross-regional interactions subserve visual narrative and contextual novelty processing.
Collapse
Affiliation(s)
- Zahraa Sabra
- Department of Neurosurgery, Emory University, Atlanta, GA, United States
| | - Ali Alawieh
- Department of Neurosurgery, Emory University, Atlanta, GA, United States
| | - Leonardo Bonilha
- Department of Neurology, Medical University of South Carolina, Charleston, SC, United States
| | - Thomas Naselaris
- Department of Neuroscience, University of Minnesota, Minneapolis, MN, United States
| | - Nicholas AuYong
- Department of Neurosurgery, Emory University, Atlanta, GA, United States
- *Correspondence: Nicholas AuYong,
| |
Collapse
|
4
|
Auditory dominance in processing Chinese semantic abnormalities in response to competing audio-visual stimuli. Neuroscience 2022; 502:1-9. [PMID: 36031089 DOI: 10.1016/j.neuroscience.2022.08.017] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/24/2022] [Revised: 08/17/2022] [Accepted: 08/22/2022] [Indexed: 10/15/2022]
Abstract
Language is a remarkable cognitive ability that can be expressed through visual (written language) or auditory (spoken language) modalities. When visual characters and auditory speech convey conflicting information, individuals may selectively attend to either one of them. However, the dominant modality in such a competing situation and the neural mechanism underlying it are still unclear. Here, we presented participants with Chinese sentences in which the visual characters and auditory speech convey conflicting information, while behavioral and electroencephalographic (EEG) responses were recorded. Results showed a prominent auditory dominance when audio-visual competition occurred. Specifically, higher accuracy (ACC), larger N400 amplitudes and more linkages in the posterior occipital-parietal areas were demonstrated in the auditory mismatch condition compared to that in the visual mismatch condition. O0ur research illustrates the superiority of the auditory speech over the visual characters, extending our understanding of the neural mechanisms of audio-visual competition in Chinese.
Collapse
|
5
|
Manfredi M, Boggio PS. Neural correlates of sex differences in communicative gestures and speech comprehension: A preliminary study. Soc Neurosci 2021; 16:653-667. [PMID: 34697990 DOI: 10.1080/17470919.2021.1997800] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 10/20/2022]
Abstract
The goal of this study was to investigate whether the semantic processing of the audiovisual combination of communicative gestures with speech differs between men and women. We recorded event-related brain potentials in women and men during the presentation of communicative gestures that were either congruent or incongruent with the speech.Our results showed that incongruent gestures elicited an N400 effect over frontal sites compared to congruent ones in both groups. Moreover, the females showed an earlier N2 response to incongruent stimuli than congruent ones, while larger sustained negativity and late positivity in response to incongruent stimuli was observed only in males. These results suggest that women rapidly recognize and process audiovisual combinations of communicative gestures and speech (as early as 300 ms) whereas men analyze them at the later stages of the process.
Collapse
Affiliation(s)
- Mirella Manfredi
- Department of Psychology, University of Zurich, Zurich, Switzerland
| | - Paulo Sergio Boggio
- Social and Cognitive Neuroscience Laboratory, Center for Biological Science and Health, Mackenzie Presbyterian University, São Paulo, Brazil
| |
Collapse
|
6
|
Generalizable EEG Encoding Models with Naturalistic Audiovisual Stimuli. J Neurosci 2021; 41:8946-8962. [PMID: 34503996 DOI: 10.1523/jneurosci.2891-20.2021] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/13/2020] [Revised: 08/24/2021] [Accepted: 08/29/2021] [Indexed: 11/21/2022] Open
Abstract
In natural conversations, listeners must attend to what others are saying while ignoring extraneous background sounds. Recent studies have used encoding models to predict electroencephalography (EEG) responses to speech in noise-free listening situations, sometimes referred to as "speech tracking." Researchers have analyzed how speech tracking changes with different types of background noise. It is unclear, however, whether neural responses from acoustically rich, naturalistic environments with and without background noise can be generalized to more controlled stimuli. If encoding models for acoustically rich, naturalistic stimuli are generalizable to other tasks, this could aid in data collection from populations of individuals who may not tolerate listening to more controlled and less engaging stimuli for long periods of time. We recorded noninvasive scalp EEG while 17 human participants (8 male/9 female) listened to speech without noise and audiovisual speech stimuli containing overlapping speakers and background sounds. We fit multivariate temporal receptive field encoding models to predict EEG responses to pitch, the acoustic envelope, phonological features, and visual cues in both stimulus conditions. Our results suggested that neural responses to naturalistic stimuli were generalizable to more controlled datasets. EEG responses to speech in isolation were predicted accurately using phonological features alone, while responses to speech in a rich acoustic background were more accurate when including both phonological and acoustic features. Our findings suggest that naturalistic audiovisual stimuli can be used to measure receptive fields that are comparable and generalizable to more controlled audio-only stimuli.SIGNIFICANCE STATEMENT Understanding spoken language in natural environments requires listeners to parse acoustic and linguistic information in the presence of other distracting stimuli. However, most studies of auditory processing rely on highly controlled stimuli with no background noise, or with background noise inserted at specific times. Here, we compare models where EEG data are predicted based on a combination of acoustic, phonetic, and visual features in highly disparate stimuli-sentences from a speech corpus and speech embedded within movie trailers. We show that modeling neural responses to highly noisy, audiovisual movies can uncover tuning for acoustic and phonetic information that generalizes to simpler stimuli typically used in sensory neuroscience experiments.
Collapse
|
7
|
Manfredi M, Sanchez Mello de Pinho P, Murrins Marques L, de Oliveira Ribeiro B, Boggio PS. Crossmodal processing of environmental sounds and everyday life actions: An ERP study. Heliyon 2021; 7:e07937. [PMID: 34541349 PMCID: PMC8436072 DOI: 10.1016/j.heliyon.2021.e07937] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/26/2021] [Revised: 07/06/2021] [Accepted: 09/01/2021] [Indexed: 12/03/2022] Open
Abstract
To investigate the processing of environmental sounds, previous researchers have compared the semantic processing of words and sounds, yielding mixed results. This study aimed to specifically investigate the electrophysiological mechanism underlying the semantic processing of environmental sounds presented in a naturalistic visual scene. We recorded event-related brain potentials in a group of young adults over the presentation of everyday life actions that were either congruent or incongruent with environmental sounds. Our results showed that incongruent environmental sounds evoked both a P400 and an N400 effect, reflecting sensitivity to physical and semantic violations of environmental sounds’ properties, respectively. In addition, our findings showed an enhanced late positivity in response to incongruous environmental sounds, probably reflecting additional reanalysis costs. In conclusion, these results indicate that the crossmodal processing of the environmental sounds might require the simultaneous involvement of different cognitive processes.
Collapse
Affiliation(s)
- Mirella Manfredi
- Department of Psychology, University of Zurich, Zurich, Switzerland
- Corresponding author.
| | - Pamella Sanchez Mello de Pinho
- Social and Cognitive Neuroscience Laboratory, Developmental Disorders Program, Center for Health and Biological Sciences, Mackenzie Presbyterian University, Sao Paulo, Brazil
| | - Lucas Murrins Marques
- Social and Cognitive Neuroscience Laboratory, Developmental Disorders Program, Center for Health and Biological Sciences, Mackenzie Presbyterian University, Sao Paulo, Brazil
| | - Beatriz de Oliveira Ribeiro
- Social and Cognitive Neuroscience Laboratory, Developmental Disorders Program, Center for Health and Biological Sciences, Mackenzie Presbyterian University, Sao Paulo, Brazil
| | - Paulo Sergio Boggio
- Social and Cognitive Neuroscience Laboratory, Developmental Disorders Program, Center for Health and Biological Sciences, Mackenzie Presbyterian University, Sao Paulo, Brazil
- Corresponding author.
| |
Collapse
|
8
|
Manfredi M, Celebic C, Daum MM. When dogs meow: An electrophysiological study of lexical-semantic processing in toddlers. INFANCY 2021; 26:1076-1096. [PMID: 34499397 DOI: 10.1111/infa.12434] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/29/2021] [Revised: 08/03/2021] [Accepted: 08/10/2021] [Indexed: 11/26/2022]
Abstract
The goal of this study was to investigate lexical-semantic processing at an early phase of language development. Adults often communicate with children using infant-directed speech that typically involves lexical and syntactic modifications such as onomatopoeias (Soderstrom, 2007). Here, we asked how and when children start to show an advantage for processing conventional linguistic forms, such as common nouns, and consequently decreasing sensitivity to onomatopoeias. We recorded event-related brain potentials in children of two age groups (16-21 months and 24-31 months) and in an adult control group during the presentation of four conditions in which either common nouns or onomatopoeias were presented auditorily followed by a picture of an either congruent or incongruent object. We focused on the N400 effect, a more negative ERP response to incongruent compared with congruent semantic relations. The younger children showed an N400 effect only for onomatopoeic words, while the older children showed an N400 effect only for common nouns. The adults showed N400 effects for onomatopoeia and nouns. These different N400 effects suggest that onomatopoeia and common nouns are differently organized in children's semantic memory and that the acquisition of linguistic abilities affects and modifies semantic processing of different lexical information.
Collapse
Affiliation(s)
- Mirella Manfredi
- Department of Psychology, University of Zurich, Zurich, Switzerland
| | - Corina Celebic
- Department of Psychology, University of Zurich, Zurich, Switzerland
| | - Moritz M Daum
- Department of Psychology, University of Zurich, Zurich, Switzerland.,Jacobs Center for Productive Youth Development, University of Zurich, Zurich, Switzerland
| |
Collapse
|
9
|
Manfredi M, Cohn N, Ribeiro B, Sanchez Pinho P, Fernandes Rodrigues Pereira E, Boggio PS. The electrophysiology of audiovisual processing in visual narratives in adolescents with autism spectrum disorder. Brain Cogn 2021; 151:105730. [PMID: 33892434 DOI: 10.1016/j.bandc.2021.105730] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/03/2020] [Revised: 02/15/2021] [Accepted: 04/03/2021] [Indexed: 12/24/2022]
Abstract
We investigated the semantic processing of the multimodal audiovisual combination of visual narratives with auditory descriptive words and auditory sounds in individuals with ASD. To this aim, we recorded ERPs to critical auditory words and sounds associated with events in visual narrative that were either semantically congruent or incongruent with the climactic visual event. A similar N400 effect was found both in adolescents with ASD and neurotypical adolescents (ages 9-16) when accessing different types of auditory information (i.e. words and sounds) into a visual narrative. This result might suggest that verbal information processing in ASD adolescents could be facilitated by direct association with meaningful visual information. In addition, we observed differences in scalp distribution of later brain responses between ASD and neurotypical adolescents. This finding might suggest ASD adolescents differ from neurotypical adolescents during the processing of the multimodal combination of visual narratives with auditory information at later stages of the process. In conclusion, the semantic processing of verbal information, typically impaired in individuals with ASD, can be facilitated when embedded into a meaningful visual information.
Collapse
Affiliation(s)
- Mirella Manfredi
- Social and Cognitive Neuroscience Laboratory, Center for Biological Science and Health, Mackenzie Presbyterian University, São Paulo, Brazil; Department of Psychology, University of Zurich, Zurich, Switzerland.
| | - Neil Cohn
- Department of Communication and Cognition, Tilburg University, Tilburg, Netherlands
| | - Beatriz Ribeiro
- Social and Cognitive Neuroscience Laboratory, Center for Biological Science and Health, Mackenzie Presbyterian University, São Paulo, Brazil
| | - Pamella Sanchez Pinho
- Social and Cognitive Neuroscience Laboratory, Center for Biological Science and Health, Mackenzie Presbyterian University, São Paulo, Brazil
| | | | - Paulo Sergio Boggio
- Social and Cognitive Neuroscience Laboratory, Center for Biological Science and Health, Mackenzie Presbyterian University, São Paulo, Brazil
| |
Collapse
|
10
|
Liberto GMD, Nie J, Yeaton J, Khalighinejad B, Shamma SA, Mesgarani N. Neural representation of linguistic feature hierarchy reflects second-language proficiency. Neuroimage 2020; 227:117586. [PMID: 33346131 PMCID: PMC8527895 DOI: 10.1016/j.neuroimage.2020.117586] [Citation(s) in RCA: 18] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/17/2020] [Revised: 11/15/2020] [Accepted: 11/18/2020] [Indexed: 12/03/2022] Open
Abstract
Acquiring a new language requires individuals to simultaneously and gradually learn linguistic attributes on multiple levels. Here, we investigated how this learning process changes the neural encoding of natural speech by assessing the encoding of the linguistic feature hierarchy in second-language listeners. Electroencephalography (EEG) signals were recorded from native Mandarin speakers with varied English proficiency and from native English speakers while they listened to audio-stories in English. We measured the temporal response functions (TRFs) for acoustic, phonemic, phonotactic, and semantic features in individual participants and found a main effect of proficiency on linguistic encoding. This effect of second-language proficiency was particularly prominent on the neural encoding of phonemes, showing stronger encoding of “new” phonemic contrasts (i.e., English contrasts that do not exist in Mandarin) with increasing proficiency. Overall, we found that the nonnative listeners with higher proficiency levels had a linguistic feature representation more similar to that of native listeners, which enabled the accurate decoding of language proficiency. This result advances our understanding of the cortical processing of linguistic information in second-language learners and provides an objective measure of language proficiency.
Collapse
Affiliation(s)
- Giovanni M Di Liberto
- Laboratoire des systèmes perceptifs, Département d'études cognitives, École normale supérieure, PSL University, CNRS, 75005 Paris, France.
| | - Jingping Nie
- Department of Electrical Engineering, Columbia University, New York, NY, USA; Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, NY 10027, United States
| | - Jeremy Yeaton
- Laboratoire des systèmes perceptifs, Département d'études cognitives, École normale supérieure, PSL University, CNRS, 75005 Paris, France; Laboratoire de Psychologie Cognitive, UMR 7290, CNRS, France. Aix-Marseille Université, France
| | - Bahar Khalighinejad
- Department of Electrical Engineering, Columbia University, New York, NY, USA; Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, NY 10027, United States
| | - Shihab A Shamma
- Laboratoire des systèmes perceptifs, Département d'études cognitives, École normale supérieure, PSL University, CNRS, 75005 Paris, France; Institute for Systems Research, Electrical and Computer Engineering, University of Maryland, College Park, USA
| | - Nima Mesgarani
- Department of Electrical Engineering, Columbia University, New York, NY, USA; Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, NY 10027, United States.
| |
Collapse
|
11
|
Visual and Verbal Narrative Comprehension in Children and Adolescents with Autism Spectrum Disorders: An ERP Study. J Autism Dev Disord 2020; 50:2658-2672. [PMID: 31974801 DOI: 10.1007/s10803-020-04374-x] [Citation(s) in RCA: 12] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/03/2023]
Abstract
We examined semantic processing in ASD children by presenting sentences with congruent or incongruent final words and visual narratives with congruent or incongruent final panels. An N400 effect to incongruent words appeared as compared to congruent ones, which was attenuated for the ASD children. We observed a negativity sustained to incongruous than congruous words, but only for the TD children. Incongruent panels evoked a greater fronto-central N400 amplitude than congruent panels in both groups. In addition, incongruent panels evoked a centro-parietal late positivity, only in controls. In conclusion, ASD children face processing deficits in both verbal and visual materials when integrating meaning across information, though such impairments may arise in different parts of the interpretive process, depending on the modality.
Collapse
|
12
|
Manfredi M, Proverbio AM, Sanchez Mello de Pinho P, Ribeiro B, Comfort WE, Murrins Marques L, Boggio PS. Electrophysiological indexes of ToM and non-ToM humor in healthy adults. Exp Brain Res 2020; 238:789-805. [PMID: 32107576 DOI: 10.1007/s00221-020-05753-7] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/23/2019] [Accepted: 02/12/2020] [Indexed: 02/03/2023]
Abstract
The cognitive processes involved in humor comprehension were analyzed by directly comparing the time course of brain activity associated with the perception of slapstick humor and that associated with the comprehension of humor requiring theory of mind (ToM). Four different comic strips (strips containing humorous scenes that required ToM, non-ToM humorous strips, non-humorous semantically coherent strips and non-humorous semantically incoherent strips) were presented to participants, while their EEG response was recorded. Results showed that both of the humorous comic strips and the semantically incongruent strip elicited an N400 effect, suggesting similar cognitive mechanisms underlying the processing of incongruent and humorous comic strips. The results also showed that the humorous ToM strips elicited a frontal late positive (LP) response, possibly reflecting the active deployment of ToM abilities such as perspective-taking and empathy that allow for the resolution and interpretation of apparently incongruent situations. In addition, the LP response was positively correlated with ratings of perceived amusement as well as individual empathy scores, suggesting that the increased LP response to ToM humorous strips reflects the combined activation of neural mechanisms involved in the experience of amusement and ToM abilities. Overall, humor comprehension appears to demand distinct cognitive steps such as the detection of incongruent semantic components, the construction of semantic coherence, and the appreciation of humoristic elements such as maladaptive emotional reactions. Our results show that the deployment of these distinct cognitive steps is at least partially dependent on individual empathic abilities.
Collapse
Affiliation(s)
- Mirella Manfredi
- Social and Cognitive Neuroscience Laboratory, Center for Health and Biological Sciences, Mackenzie Presbyterian University, São Paulo, SP, Brazil. .,Developmental Psychology: Infancy and Childhood, University of Zurich, Binzmuehlestrasse 14, 8050, Zurich, Switzerland.
| | | | - Pamella Sanchez Mello de Pinho
- Social and Cognitive Neuroscience Laboratory, Center for Health and Biological Sciences, Mackenzie Presbyterian University, São Paulo, SP, Brazil
| | - Beatriz Ribeiro
- Social and Cognitive Neuroscience Laboratory, Center for Health and Biological Sciences, Mackenzie Presbyterian University, São Paulo, SP, Brazil
| | - William Edgar Comfort
- Social and Cognitive Neuroscience Laboratory, Center for Health and Biological Sciences, Mackenzie Presbyterian University, São Paulo, SP, Brazil
| | - Lucas Murrins Marques
- Social and Cognitive Neuroscience Laboratory, Center for Health and Biological Sciences, Mackenzie Presbyterian University, São Paulo, SP, Brazil
| | - Paulo Sérgio Boggio
- Social and Cognitive Neuroscience Laboratory, Center for Health and Biological Sciences, Mackenzie Presbyterian University, São Paulo, SP, Brazil
| |
Collapse
|
13
|
Pérez A, Schmidt E, Kourtzi Z, Tsimpli I. Multimodal semantic revision during inferential processing: The role of inhibitory control in text and picture comprehension. Neuropsychologia 2020; 138:107313. [PMID: 31904356 DOI: 10.1016/j.neuropsychologia.2019.107313] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/28/2019] [Revised: 12/08/2019] [Accepted: 12/17/2019] [Indexed: 11/22/2022]
Abstract
Although language comprehension usually requires multimodal information, no study to date has investigated how comprehenders deal with the revision of a text's interpretation when different modalities are involved. Twenty-four young adults listened to a story prompting an inference (e.g., polar bear), and then saw a picture that was either consistent (polar bear) or inconsistent but still plausible (penguin). Larger negativity (N400) in the inconsistent picture indicated successful inferential monitoring. Subsequently, a sentence carried the disambiguating word which was either expected ("bear") or unexpected ("penguin") in relation to the auditory-verbal information. Larger negativity in the unexpected word coming from the consistent picture suggested that comprehenders had difficulties selecting the unexpected concept when previous information was contradictory. More importantly, this effect was modulated by inhibitory control, where a higher resistance to distractor interference (flanker task) was associated with a better ability to suppress pictorial information, therefore preventing semantic competition. Similarly, accuracy measured in a final comprehension question demonstrated that higher inhibitory control was related to a more efficient ability to revise the situation model across modalities. Our findings speak to a relationship between story comprehension and mental flexibility during multimodal processing.
Collapse
Affiliation(s)
- A Pérez
- Theoretical and Applied Linguistics, UK; Dept. of Experimental Psychology, University of Granada, Spain.
| | - E Schmidt
- Theoretical and Applied Linguistics, UK; Cambridge Assessment English, University of Cambridge, UK
| | - Z Kourtzi
- Dept. of Psychology, University of Cambridge, UK
| | - I Tsimpli
- Theoretical and Applied Linguistics, UK
| |
Collapse
|
14
|
Cohn N, Magliano JP. Editors’ Introduction and Review: Visual Narrative Research: An Emerging Field in Cognitive Science. Top Cogn Sci 2019; 12:197-223. [PMID: 31865641 PMCID: PMC9328199 DOI: 10.1111/tops.12473] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/01/2018] [Revised: 09/29/2019] [Accepted: 09/29/2019] [Indexed: 01/06/2023]
Abstract
Drawn sequences of images are among our oldest records of human intelligence, appearing on cave paintings, wall carvings, and ancient pottery, and they pervade across cultures from instruction manuals to comics. They also appear prevalently as stimuli across Cognitive Science, for studies of temporal cognition, event structure, social cognition, discourse, and basic intelligence. Yet, despite this fundamental place in human expression and research on cognition, the study of visual narratives themselves has only recently gained traction in Cognitive Science. This work has suggested that visual narrative comprehension requires cultural exposure across a developmental trajectory and engages with domain‐general processing mechanisms shared by visual perception, attention, event cognition, and language, among others. Here, we review the relevance of such research for the broader Cognitive Science community, and make the case for why researchers should join the scholarship of this ubiquitous but understudied aspect of human expression. Drawn sequences of images, like those in comics and picture stories, are a pervasive and fundamental way that humans have communicated for millennia. Yet, the study of visual narratives has only recently gained traction in Cognitive Science. Here we explore what has held back the study of the cognition of visual narratives, and why researchers should join in scholarship of this ubiquitous aspect of expression.
Collapse
Affiliation(s)
- Neil Cohn
- Department of Communciation and Cognition, Tilburg School of Humanities and Digital Sciences, Tilburg Center for Cognition and Communication, Tilburg Unviersity
| | - Joseph P. Magliano
- Department of Learning Sciences at the College of Education & Human Development, Georgia State University
| |
Collapse
|
15
|
Loschky LC, Larson AM, Smith TJ, Magliano JP. The Scene Perception & Event Comprehension Theory (SPECT) Applied to Visual Narratives. Top Cogn Sci 2019; 12:311-351. [PMID: 31486277 PMCID: PMC9328418 DOI: 10.1111/tops.12455] [Citation(s) in RCA: 32] [Impact Index Per Article: 6.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/26/2018] [Revised: 08/05/2019] [Accepted: 08/05/2019] [Indexed: 11/29/2022]
Abstract
Understanding how people comprehend visual narratives (including picture stories, comics, and film) requires the combination of traditionally separate theories that span the initial sensory and perceptual processing of complex visual scenes, the perception of events over time, and comprehension of narratives. Existing piecemeal approaches fail to capture the interplay between these levels of processing. Here, we propose the Scene Perception & Event Comprehension Theory (SPECT), as applied to visual narratives, which distinguishes between front‐end and back‐end cognitive processes. Front‐end processes occur during single eye fixations and are comprised of attentional selection and information extraction. Back‐end processes occur across multiple fixations and support the construction of event models, which reflect understanding of what is happening now in a narrative (stored in working memory) and over the course of the entire narrative (stored in long‐term episodic memory). We describe relationships between front‐ and back‐end processes, and medium‐specific differences that likely produce variation in front‐end and back‐end processes across media (e.g., picture stories vs. film). We describe several novel research questions derived from SPECT that we have explored. By addressing these questions, we provide greater insight into how attention, information extraction, and event model processes are dynamically coordinated to perceive and understand complex naturalistic visual events in narratives and the real world. Comprehension of visual narratives like comics, picture stories, and films involves both decoding the visual content and construing the meaningful events they represent. The Scene Perception & Event Comprehension Theory (SPECT) proposes a framework for understanding how a comprehender perceptually negotiates the surface of a visual representation and integrates its meaning into a growing mental model.
Collapse
Affiliation(s)
| | | | - Tim J Smith
- Department of Psychological Sciences, Birkbeck, University of London
| | | |
Collapse
|
16
|
Cohn N. Your Brain on Comics: A Cognitive Model of Visual Narrative Comprehension. Top Cogn Sci 2019; 12:352-386. [PMID: 30963724 PMCID: PMC9328425 DOI: 10.1111/tops.12421] [Citation(s) in RCA: 21] [Impact Index Per Article: 4.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/03/2018] [Revised: 01/21/2019] [Accepted: 03/18/2019] [Indexed: 11/30/2022]
Abstract
The past decade has seen a rapid growth of cognitive and brain research focused on visual narratives like comics and picture stories. This paper will summarize and integrate this emerging literature into the Parallel Interfacing Narrative‐Semantics Model (PINS Model)—a theory of sequential image processing characterized by an interaction between two representational levels: semantics and narrative structure. Ongoing semantic processes build meaning into an evolving mental model of a visual discourse. Updating of spatial, referential, and event information then incurs costs when they are discontinuous with the growing context. In parallel, a narrative structure organizes semantic information into coherent sequences by assigning images to categorical roles, which are then embedded within a hierarchic constituent structure. Narrative constructional schemas allow for specific predictions of structural sequencing, independent of semantics. Together, these interacting levels of representation engage in an iterative process of retrieval of semantic and narrative information, prediction of upcoming information based on those assessments, and subsequent updating based on discontinuity. These core mechanisms are argued to be domain‐general—spanning across expressive systems—as suggested by similar electrophysiological brain responses (N400, P600, anterior negativities) generated in response to manipulation of sequential images, music, and language. Such similarities between visual narratives and other domains thus pose fundamental questions for the linguistic and cognitive sciences. Visual narratives like comics involve a range of complex cognitive operations in order to be understood. The Parallel Interfacing Narrative‐Semantics (PINS) Model integrates an emerging literature showing that comprehension of wordless image sequences balances two representational levels of semantic and narrative structure. The neurocognitive mechanisms that guide these processes are argued to overlap with other domains, such as language and music.
Collapse
Affiliation(s)
- Neil Cohn
- Department of Communication and Cognition, Tilburg University
| |
Collapse
|