Effects of language comprehension on visual processing - MEG dissociates early perceptual and late N400 effects

The influence of sentence focus on motor system activity in language comprehension and its temporal dynamics: Preliminary evidence from sEMG

Previous research has shown that individual experiences and experimental tasks can influence the occurrence of mental simulation during sentence comprehension. However, little research has focused on the effect of sentence focus on mental simulation and its temporal dynamics. Sentence focus refers to the hierarchical structure of information within a sentence, where focused information represents the most prominent and essential information. In contrast, nonfocused information provides a background for the focused information. The present study investigated whether sentence focus would affect the activity of the motor system in language comprehension and at which stage the effect of sentence focus occurred. We measured spontaneous arm muscle electrical activity by surface electromyography (sEMG) while participants read action-focused, nonaction-focused, and control sentences. We observed greater spontaneous muscle electrical activity in the flexor common muscle of the fingers when participants read action-focused sentences compared to nonaction-focused and control sentences. Additionally, there was an interactive trend between sentence type and time, spontaneous muscle electrical activity while reading action-focused sentences was observed in both early (1 ms to 300 ms after the presentation of the action phrase) and late time windows (901 ms to 1500 ms after the action phrase). The findings suggest that the motor system exhibits flexible engagement during language comprehension and the impact of sentence focus on motor system activity may be throughout both the lexical-semantic retrieval and sentence-meaning integration stages.

Brain Signatures of Embodied Semantics and Language: A Consensus Paper

According to embodied theories (including embodied, embedded, extended, enacted, situated, and grounded approaches to cognition), language representation is intrinsically linked to our interactions with the world around us, which is reflected in specific brain signatures during language processing and learning. Moving on from the original rivalry of embodied vs. amodal theories, this consensus paper addresses a series of carefully selected questions that aim at determining when and how rather than whether motor and perceptual processes are involved in language processes. We cover a wide range of research areas, from the neurophysiological signatures of embodied semantics, e.g., event-related potentials and fields as well as neural oscillations, to semantic processing and semantic priming effects on concrete and abstract words, to first and second language learning and, finally, the use of virtual reality for examining embodied semantics. Our common aim is to better understand the role of motor and perceptual processes in language representation as indexed by language comprehension and learning. We come to the consensus that, based on seminal research conducted in the field, future directions now call for enhancing the external validity of findings by acknowledging the multimodality, multidimensionality, flexibility and idiosyncrasy of embodied and situated language and semantic processes.

Linguistic Priors for Perception

In this commentary, we approach the topic of linguistic relativity from a predictive coding perspective. Discussing the role of "priors" in shaping perception, we argue that language creates an important set of priors for humans, which can affect how sensory information is processed and interpreted. Namely, languages create conventionalized conceptual systems for their speakers, mirroring and reinforcing what is behaviorally important in a society. As such, they create collective conceptual convergence on how to categorize the world and thus "streamline" what people rely on to guide their perception.

Instant Effects of Semantic Information on Visual Perception

Does our perception of an object change once we discover what function it serves? We showed human participants (n = 48, 31 females and 17 males) pictures of unfamiliar objects either together with keywords matching their function, leading to semantically informed perception, or together with nonmatching keywords, resulting in uninformed perception. We measured event-related potentials to investigate at which stages in the visual processing hierarchy these two types of object perception differed from one another. We found that semantically informed compared with uninformed perception was associated with larger amplitudes in the N170 component (150-200 ms), reduced amplitudes in the N400 component (400-700 ms), and a late decrease in alpha/beta band power. When the same objects were presented once more without any information, the N400 and event-related power effects persisted, and we also observed enlarged amplitudes in the P1 component (100-150 ms) in response to objects for which semantically informed perception had taken place. Consistent with previous work, this suggests that obtaining semantic information about previously unfamiliar objects alters aspects of their lower-level visual perception (P1 component), higher-level visual perception (N170 component), and semantic processing (N400 component, event-related power). Our study is the first to show that such effects occur instantly after semantic information has been provided for the first time, without requiring extensive learning.SIGNIFICANCE STATEMENT There has been a long-standing debate about whether or not higher-level cognitive capacities, such as semantic knowledge, can influence lower-level perceptual processing in a top-down fashion. Here we could show, for the first time, that information about the function of previously unfamiliar objects immediately influences cortical processing within less than 200 ms. Of note, this influence does not require training or experience with the objects and related semantic information. Therefore, our study is the first to show effects of cognition on perception while ruling out the possibility that prior knowledge merely acts by preactivating or altering stored visual representations. Instead, this knowledge seems to alter perception online, thus providing a compelling case against the impenetrability of perception by cognition.

The state of the onion: Grammatical aspect modulates object representation during event comprehension

The present ERP study assessed whether grammatical aspect is used as a cue in online event comprehension, in particular when reading about events in which an object is visually changed. While perfective aspect cues holistic event representations, including an event's endpoint, progressive aspect highlights intermediate phases of an event. In a 2 × 3 design, participants read SVO sentences describing a change-of-state event (e.g., to chop an onion), with grammatical Aspect manipulated (perfective "chopped" vs progressive "was chopping"). Thereafter, they saw a Picture of an object either having undergone substantial state-change (SC; a chopped onion), no state-change (NSC; an onion in its original state) or an unrelated object (U; a cactus, acting as control condition). Their task was to decide whether the object in the Picture was mentioned in the sentence. We focused on N400 modulation, with ERPs time-locked to picture onset. U pictures elicited an N400 response as expected, suggesting detection of categorical mismatches in object type. For SC and NSC pictures, a whole-head follow-up analysis revealed a P300, implying people were engaged in detailed evaluation of pictures of matching objects. SC pictures received most positive responses overall. Crucially, there was an interaction of Aspect and Picture: SC pictures resulted in a higher amplitude P300 after sentences in the perfective compared to the progressive. Thus, while the perfective cued for a holistic event representation, including the resultant state of the affected object (i.e., the chopped onion) constraining object representations online, the progressive defocused event completion and object-state change. Grammatical aspect thus guided online event comprehension by cueing the visual representation(s) of an object's state.

Correlates of intelligence via resting-state functional connectivity of the amygdala in healthy adults

Intelligence is a form of advanced cognition that includes reasoning, problem solving, pattern recognition, and establishing relationships among items. The amygdala plays an important role in cognitive processing, but the relationship between amygdalar function and intelligence has rarely been explored directly. Here, we investigated the relationship between resting-state functional connectivity (RSFC) of the amygdala and intelligence test performance in a large sample of healthy adults (N = 197). We found that two pairs of RSFCs were significantly increased in the high IQ group compared with that of the general IQ group. One of these RSFCs consisted of the right amygdala and the right superior parietal lobule, whereas the other RSFC consisted of the right amygdala and the left middle cingulum. In addition, we found that the brain regions in which the strength of RSFC significantly correlated with full IQ (FIQ) were mainly distributed in the parietal and limbic lobes. What's more, a further mediation analysis indicated that the functional connectivity of the right amygdala and the right superior parietal lobule significantly mediated the correlation between comprehension and object assembly, whereas the functional connectivity of the right amygdala and the left middle cingulum mediated the association between similarities and digit symbol. These findings suggest that amygdalar RSFC may reflect individual differences in intelligence and mediate specific relationships among different intellectual abilities.

The development of dynamic perceptual simulations during sentence comprehension

Based on an embodied account of language comprehension, this study investigated the dynamic characteristics of children and adults’ perceptual simulations during sentence comprehension, using a novel paradigm to assess the perceptual simulation of objects moving up and down a vertical axis. The participants comprised adults (N = 40) and 6-, 8-, and 10-year-old children (N = 116). After listening in experimental trials to sentences implying that objects moved upward or downward, the participants were shown pictures and had to decide as quickly as possible whether the objects depicted had been mentioned in the sentences. The target pictures moved either up or down and then stopped in the middle of the screen. All age groups’ reaction times were found to be shorter when the objects moved in the directions that the sentences implied. Age exerted no developmental effect on reaction times. The findings suggest that dynamic perceptual simulations are fundamental to language comprehension in text recipients aged 6 and older.

Unfolding meaning in context: The dynamics of conceptual similarity

How are relationships between concepts affected by the interplay between short-term contextual constraints and long-term conceptual knowledge? Across two studies we investigate the consequence of changes in visual context for the dynamics of conceptual processing. Participants' eye movements were tracked as they viewed a visual depiction of e.g. a canary in a birdcage (Experiment 1), or a canary and three unrelated objects, each in its own quadrant (Experiment 2). In both studies participants heard either a semantically and contextually similar "robin" (a bird; similar size), an equally semantically similar but not contextually similar "stork" (a bird; bigger than a canary, incompatible with the birdcage), or unrelated "tent". The changing patterns of fixations across time indicated first, that the visual context strongly influenced the eye movements such that, in the context of a birdcage, early on (by word offset) hearing "robin" engendered more looks to the canary than hearing "stork" or "tent" (which engendered the same number of looks), unlike in the context of unrelated objects (in which case "robin" and "stork" engendered equivalent looks to the canary, and more than did "tent"). Second, within the 500 ms post-word-offset eye movements in both experiments converged onto a common pattern (more looks to the canary after "robin" than after "stork", and for both more than after "tent"). We interpret these findings as indicative of the dynamics of activation within semantic memory accessed via pictures and via words, and reflecting the complex interaction between systems representing context-independent and context-dependent conceptual knowledge driven by predictive processing.

Are visual processes causally involved in "perceptual simulation" effects in the sentence-picture verification task?

Many studies have shown that sentences implying an object to have a certain shape produce a robust reaction time advantage for shape-matching pictures in the sentence-picture verification task. Typically, this finding has been interpreted as evidence for perceptual simulation, i.e., that access to implicit shape information involves the activation of modality-specific visual processes. It follows from this proposal that disrupting visual processing during sentence comprehension should interfere with perceptual simulation and obliterate the match effect. Here we directly test this hypothesis. Participants listened to sentences while seeing either visual noise that was previously shown to strongly interfere with basic visual processing or a blank screen. Experiments 1 and 2 replicated the match effect but crucially visual noise did not modulate it. When an interference technique was used that targeted high-level semantic processing (Experiment 3) however the match effect vanished. Visual noise specifically targeting high-level visual processes (Experiment 4) only had a minimal effect on the match effect. We conclude that the shape match effect in the sentence-picture verification paradigm is unlikely to rely on perceptual simulation.

Song Perception by Professional Singers and Actors: An MEG Study

The cortical correlates of speech and music perception are essentially overlapping, and the specific effects of different types of training on these networks remain unknown. We compared two groups of vocally trained professionals for music and speech, singers and actors, using recited and sung rhyme sequences from German art songs with semantic and/ or prosodic/melodic violations (i.e. violations of pitch) of the last word, in order to measure the evoked activation in a magnetoencephalographic (MEG) experiment. MEG data confirmed the existence of intertwined networks for the sung and spoken modality in an early time window after word violation. In essence for this early response, higher activity was measured after melodic/prosodic than semantic violations in predominantly right temporal areas. For singers as well as for actors, modality-specific effects were evident in predominantly left-temporal lateralized activity after semantic expectancy violations in the spoken modality, and right-dominant temporal activity in response to melodic violations in the sung modality. As an indication of a special group-dependent audiation process, higher neuronal activity for singers appeared in a late time window in right temporal and left parietal areas, both after the recited and the sung sequences.

Exploring the automaticity of language-perception interactions: Effects of attention and awareness

Previous studies have shown that language can modulate visual perception, by biasing and/or enhancing perceptual performance. However, it is still debated where in the brain visual and linguistic information are integrated, and whether the effects of language on perception are automatic and persist even in the absence of awareness of the linguistic material. Here, we aimed to explore the automaticity of language-perception interactions and the neural loci of these interactions in an fMRI study. Participants engaged in a visual motion discrimination task (upward or downward moving dots). Before each trial, a word prime was briefly presented that implied upward or downward motion (e.g., “rise”, “fall”). These word primes strongly influenced behavior: congruent motion words sped up reaction times and improved performance relative to incongruent motion words. Neural congruency effects were only observed in the left middle temporal gyrus, showing higher activity for congruent compared to incongruent conditions. This suggests that higher-level conceptual areas rather than sensory areas are the locus of language-perception interactions. When motion words were rendered unaware by means of masking, they still affected visual motion perception, suggesting that language-perception interactions may rely on automatic feed-forward integration of perceptual and semantic material in language areas of the brain.

Neuromagnetic brain activities associated with perceptual categorization and sound-content incongruency: a comparison between monosyllabic words and pitch names

In human cultures, the perceptual categorization of musical pitches relies on pitch-naming systems. A sung pitch name concurrently holds the information of fundamental frequency and pitch name. These two aspects may be either congruent or incongruent with regard to pitch categorization. The present study aimed to compare the neuromagnetic responses to musical and verbal stimuli for congruency judgments, for example a congruent pair for the pitch C4 sung with the pitch name do in a C-major context (the pitch-semantic task) or for the meaning of a word to match the speaker’s identity (the voice-semantic task). Both the behavioral data and neuromagnetic data showed that congruency detection of the speaker’s identity and word meaning was slower than that of the pitch and pitch name. Congruency effects of musical stimuli revealed that pitch categorization and semantic processing of pitch information were associated with P2m and N400m, respectively. For verbal stimuli, P2m and N400m did not show any congruency effect. In both the pitch-semantic task and the voice-semantic task, we found that incongruent stimuli evoked stronger slow waves with the latency of 500–600 ms than congruent stimuli. These findings shed new light on the neural mechanisms underlying pitch-naming processes.

The behavioral and neural effects of language on motion perception

Perception does not function as an isolated module but is tightly linked with other cognitive functions. Several studies have demonstrated an influence of language on motion perception, but it remains debated at which level of processing this modulation takes place. Some studies argue for an interaction in perceptual areas, but it is also possible that the interaction is mediated by "language areas" that integrate linguistic and visual information. Here, we investigated whether language-perception interactions were specific to the language-dominant left hemisphere by comparing the effects of language on visual material presented in the right (RVF) and left visual fields (LVF). Furthermore, we determined the neural locus of the interaction using fMRI. Participants performed a visual motion detection task. On each trial, the visual motion stimulus was presented in either the LVF or in the RVF, preceded by a centrally presented word (e.g., "rise"). The word could be congruent, incongruent, or neutral with regard to the direction of the visual motion stimulus that was presented subsequently. Participants were faster and more accurate when the direction implied by the motion word was congruent with the direction of the visual motion stimulus. Interestingly, the speed benefit was present only for motion stimuli that were presented in the RVF. We observed a neural counterpart of the behavioral facilitation effects in the left middle temporal gyrus, an area involved in semantic processing of verbal material. Together, our results suggest that semantic information about motion retrieved in language regions may automatically modulate perceptual decisions about motion.

A large N400 but no BOLD effect--comparing source activations of semantic priming in simultaneous EEG-fMRI

Numerous studies have reported neurophysiological effects of semantic priming in electroencephalography (EEG) and in functional magnetic resonance imaging (fMRI). Because of differing methodological constraints, the comparability of the observed effects remains unclear. To directly compare EEG and fMRI effects and neural sources of semantic priming, we conducted a semantic word-picture priming experiment while measuring EEG and fMRI simultaneously. The visually presented primes were pseudowords, words unrelated to the target, semantically related words and the identical names of the target. Distributed source analysis of the event-related potentials (ERPs) successfully revealed a large effect of semantic prime-target relatedness (the N400 effect), which was driven by activations in a left-temporal source region. However, no significantly differing activations between priming conditions were found in the fMRI data. Our results support the notion that, for joint interpretations of existing EEG and fMRI studies of semantic priming, we need to fully appreciate the respective methodological limitations. Second, they show that simultaneous EEG-fMRI, including ERP source localization, is a feasible and promising methodological advancement for the investigation of higher-cognitive processes. Third, they substantiate the finding that, compared to fMRI, ERPs are often more sensitive to subtle cognitive effects.

Electrophysiological correlates of connotative meaning in healthy children

The affective, connotative meaning of words can be statistically quantified by the semantic differential technique. Words that are located clearly on one of the three dimensions called "Evaluation", "Potency", and "Activity" were used as visual stimuli in a topographic event related potential study (ERP). Stimuli had been statistically defined in a group of 249 children (Skrandies, Jpn Psychol Res 53: 65-76, 2011). We investigated electrical brain activity in 19 healthy children with normal intelligence and reading skills between 11 and 15 years of age. Words that belonged to different semantic classes were presented at random on a monitor, and EEG was measured from 30 channels. Evoked potentials were computed offline for each semantic class. In the ERP data we observed significant effects of word class on component latency, field strength and topography. Similar as with adult subjects such effects occurred at small latency of about 115 ms after word presentation. The language-evoked components in children were similar but not identical to those reported previously for various groups of adults. Our data show that visually evoked brain activity is modulated by connotative meaning of the stimuli at early processing stages not only in adults but also in children.

How 'love' and 'hate' differ from 'sleep': using combined electro/magnetoencephalographic data to reveal the sources of early cortical responses to emotional words

Emotional words--as symbols for biologically relevant concepts--are preferentially processed in brain regions including the visual cortex, frontal and parietal regions, and a corticolimbic circuit including the amygdala. Some of the brain structures found in functional magnetic resonance imaging are not readily apparent in electro- and magnetoencephalographic (EEG; MEG) measures. By means of a combined EEG/MEG source localization procedure to fully exploit the available information, we sought to reduce these discrepancies and gain a better understanding of spatiotemporal brain dynamics underlying emotional-word processing. Eighteen participants read high-arousing positive and negative, and low-arousing neutral nouns, while EEG and MEG were recorded simultaneously. Combined current-density reconstructions (L2-minimum norm least squares) for two early emotion-sensitive time intervals, the P1 (80-120 ms) and the early posterior negativity (EPN, 200-300 ms), were computed using realistic individual head models with a cortical constraint. The P1 time window uncovered an emotion effect peaking in the left middle temporal gyrus. In the EPN time window, processing of emotional words was associated with enhanced activity encompassing parietal and occipital areas, and posterior limbic structures. We suggest that lexical access, being underway within 100 ms, is speeded and/or favored for emotional words, possibly on the basis of an "emotional tagging" of the word form during acquisition. This gives rise to their differential processing in the EPN time window. The EPN, as an index of natural selective attention, appears to reflect an elaborate interplay of distributed structures, related to cognitive functions, such as memory, attention, and evaluation of emotional stimuli.

Incidental picture exposure affects later reading: evidence from the N400

Language comprehenders form a mental representation of the implied shape of objects mentioned in the text. In the present study, the influence of prior visual experience on subsequent reading was assessed. In two separate phases, participants saw a picture of an object and read a text about the object, suggesting the same or a different shape. When the shapes in the two phases mismatched, ERPs during reading showed a larger N400 amplitude than when the shapes matched, suggesting that a picture presented incidentally 15 min earlier affected reading. These results further strengthen the case for the interaction of language and visual experience during language comprehension.

The evocative power of words: activation of concepts by verbal and nonverbal means

A major part of learning a language is learning to map spoken words onto objects in the environment. An open question is what are the consequences of this learning for cognition and perception? Here, we present a series of experiments that examine effects of verbal labels on the activation of conceptual information as measured through picture verification tasks. We find that verbal cues, such as the word "cat," lead to faster and more accurate verification of congruent objects and rejection of incongruent objects than do either nonverbal cues, such as the sound of a cat meowing, or words that do not directly refer to the object, such as the word "meowing." This label advantage does not arise from verbal labels being more familiar or easier to process than other cues, and it does extends to newly learned labels and sounds. Despite having equivalent facility in learning associations between novel objects and labels or sounds, conceptual information is activated more effectively through verbal means than through nonverbal means. Thus, rather than simply accessing nonverbal concepts, language activates aspects of a conceptual representation in a particularly effective way. We offer preliminary support that representations activated via verbal means are more categorical and show greater consistency between subjects. These results inform the understanding of how human cognition is shaped by language and hint at effects that different patterns of naming can have on conceptual structure.

Listening to "flying ducks": individual differences in sentence-picture verification investigated with ERPs

The present ERP study investigated individual differences in the integration of verbal descriptions and visual object representations. Participants saw pictures of objects (e.g., a swimming duck) after listening to noun phrases describing the same object in the identical state, a shape-mismatching state ("flying duck"), or an incongruent object (e.g., "sliced bread"). Individual differences in the vividness of mental imagery and preference for mental imagery were assessed after the experiment. ERP effects of context arose 170 ms after picture onset, differentiating the incongruent-object context from the other two. The N400 mirrored these context effects. Self-rated vividness of imagery affected responses to pictures already after 100 ms, and modulated the N400 effect. Participants with highly vivid imagery showed larger context effects than participants low in imagery. The context effects at 170 ms were not modulated by individual differences.