Distinct and shared cortical regions of the human brain activated by pictorial depictions versus verbal descriptions: an fMRI study

The pictorial set of Emotional Social Interactive Scenarios between Chinese Adults (ESISCA): Development and validation

Affective picture databases with a single facial expression or body posture in one image have been widely applied to investigate emotion. However, to date, there was no standardized database containing the stimuli which involve multiple emotional signals in social interactive scenarios. The current study thus developed a pictorial set comprising 274 images depicting two Chinese adults' interactive scenarios conveying emotions of happiness, anger, sadness, fear, disgust, and neutral. The data of the valence and arousal ratings of the scenes and the emotional categories of the scenes and the faces in the images were provided in the present study. Analyses of the data collected from 70 undergraduate students suggested high reliabilities of the valence and arousal ratings of the scenes and high judgmental agreements in categorizing the scene and facial emotions. The findings suggested that the present dataset is well constructed and could be useful for future studies to investigate the emotion recognition or empathy in social interactions in both healthy and clinical (e.g., ASD) populations.

Dialectally-sensitive norms of the Spanish version of Main Concept Analysis (Span-MCA) for quantifying neurogenically impaired spoken discourse

Aphasia is an acquired language impairment caused by damage in the regions of the brain that support language. The Main Concept Analysis (MCA; Kong, 2016b) is a published formal assessment battery that allows the quantification of the presence, accuracy, completeness, and efficiency of content in spoken discourse produced by persons with aphasia (PWA). It utilizes a sequential picture description task (with four sets of pictures) for language sample elicitation. The MCA results can also be used clinically for targeting appropriate interventions of aphasic output. The purpose of this research is to develop a Spanish adaptation of the MCA (i.e., Span-MCA) by establishing normative data based on native unimpaired speakers of Spanish from four different dialect origins (Central American Caribbean, Andean-Pacific, Mexican, and Central-Southern Peninsular regions). A total of 91 unimpaired participants that consisted of different age groups, education levels, and dialect origins were recruited to establish four sets of dialect-specific norms and scoring criteria of Span-MCA, including target main concepts and corresponding lexical items related to the picture sets. The Span-MCA was also applied to one pilot native Spanish PWA. The normative data suggested that speakers who were younger or with a higher level of education levels produced significantly more accurate and complete main concepts in their spoken discourse. The application of Span-MCA to the pilot native Spanish PWA successfully identified impaired performance, as compared to the dialectally-sensitive norms established in this study. This study highlighted the clinical value of Span-MCA as a supplement to evaluate spoken discourse and target intervention by speech-language pathologists and related healthcare practitioners.

Emoji-based semantic representations for abstract and concrete concepts

An increasingly large body of converging evidence supports the idea that the semantic system is distributed across brain areas and that the information encoded therein is multimodal. Within this framework, feature norms are typically used to operationalize the various parts of meaning that contribute to define the distributed nature of conceptual representations. However, such features are typically collected as verbal strings, elicited from participants in experimental settings. If the semantic system is not only distributed (across features) but also multimodal, a cognitively sound theory of semantic representations should take into account different modalities in which feature-based representations are generated, because not all the relevant semantic information may be easily verbalized into classic feature norms, and different types of concepts (e.g., abstract vs. concrete concepts) may consist of different configurations of non-verbal features. In this paper we acknowledge the multimodal nature of conceptual representations and we propose a novel way of collecting non-verbal semantic features. In a crowdsourcing task we asked participants to use emoji to provide semantic representations for a sample of 300 English nouns referring to abstract and concrete concepts, which account for (machine readable) visual features. In a formal content analysis with multiple annotators we then classified the cognitive strategies used by the participants to represent conceptual content through emoji. The main results of our analyses show that abstract (vs. concrete) concepts are characterized by representations that: 1. consist of a larger number of emoji; 2. include more face emoji (expressing emotions); 3. are less stable and less shared among users; 4. use representation strategies based on figurative operations (e.g., metaphors) and strategies that exploit linguistic information (e.g. rebus); 5. correlate less well with the semantic representations emerging from classic features listed through verbal strings.

Dynamic cognitive processes of text-picture integration revealed by event-related potentials

The integration of text and picture is the core of multimedia information processing. Relevant theories suggest that text and picture are processed through different channels in the early stage, and integrated in the late stage of processing. Based on these theories, the current study adopted measures of event-related potentials to examine the cognitive and neural processes of text-picture integration. The results showed that in the early stage of text-picture integration, picture processing evoked a more negative N1 over the occipital area and a N300 over the prefrontal area, which might reflect the discrimination process of visual stimuli and the imagery representation of the picture, respectively; in the late stage, the text-picture induced a N400 in the central area and an LPC over the central, parietal and temporal areas, which might be associated with the semantic activation and integration of text and picture, respectively. These results not only provide support for existing theories, but also further elucidate the dynamic neural processing of text-picture integration in terms of its temporal and spatial characteristics.

The N1-N2-LPC Pattern in Processing Advertising Pictorial Metaphors: An ERP Study

We investigated what the temporal processing is of advertising pictorial metaphors. After presenting “a word of product” and “its advertising pictures,” the experiment instructed participants to make a follow-up true–false judgment considering what the picture intended to suggest. A repeated-measures ANOVAs for a 2 (picture type: metaphor, non-metaphor) × 2 (prime–target condition: congruent, incongruent) × 3 (electrode site: Fz, Cz, Pz) experimental condition was conducted on three components, N1 (100–150 ms), N2 (200–300 ms), and LPC (400–600 ms and 600–1,000 ms). The results show that metaphor pictures elicited larger amplitude in N1 (broadly distributed), N2 (frontally biased) and LPC (parietally biased), roughly reflecting an entire process with an initial response to visual onsets, an early recognition of semantic violations and a prolonged reanalysis process of semantic integration. We argue that, different than verbal metaphors, this faster processing occurred due to the involvement of visual pathway.

When a hit sounds like a kiss: An electrophysiological exploration of semantic processing in visual narrative

Researchers have long questioned whether information presented through different sensory modalities involves distinct or shared semantic systems. We investigated uni-sensory cross-modal processing by recording event-related brain potentials to words replacing the climactic event in a visual narrative sequence (comics). We compared Onomatopoeic words, which phonetically imitate action sounds (Pow!), with Descriptive words, which describe an action (Punch!), that were (in)congruent within their sequence contexts. Across two experiments, larger N400s appeared to Anomalous Onomatopoeic or Descriptive critical panels than to their congruent counterparts, reflecting a difficulty in semantic access/retrieval. Also, Descriptive words evinced a greater late frontal positivity compared to Onomatopoetic words, suggesting that, though plausible, they may be less predictable/expected in visual narratives. Our results indicate that uni-sensory cross-model integration of word/letter-symbol strings within visual narratives elicit ERP patterns typically observed for written sentence processing, thereby suggesting the engagement of similar domain-independent integration/interpretation mechanisms.

One look is worth a thousand words: New picture stimuli of interpersonal situations

This paper introduces a picture system that can be used in functional imaging experiments exploring interpersonal relations. This is important for psychotherapy research to understand the neural basis of psychological treatment effects. Pictures have many advantages for the design of functional imaging experiments, but no picture system illustrating interpersonal behavior patterns is, to date, available. We therefore developed, on the basis of a validated card-sorting test, the Interpersonal Relations Picture System. In summary, 43 pictures with 2 or more stick figures in different social situations and 9 control pictures were composed. To test the relation between each picture and the appropriate description, two successive online surveys, including 1058 and 675 individuals respectively, were conducted. Using two question types, the grade expressiveness of each picture was assessed. In total, 24 pictures and 6 control pictures met our criteria for sufficient strength and consistency with the appropriate description. Both measures are correlated with each other in all pictures illustrating interpersonal behavior, but not in the control pictures. Relations to other stimulus types and the applicability of the new picture system in functional neuroimaging methods are discussed. It is concluded that the new system will be helpful in studying the profound effect of relational change in psychotherapy.

Functional Fixedness in Creative Thinking Tasks Depends on Stimulus Modality

Pictorial examples during creative thinking tasks can lead participants to fixate on these examples and reproduce their elements even when yielding suboptimal creative products. Semantic memory research may illuminate the cognitive processes underlying this effect. Here, we examined whether pictures and words differentially influence access to semantic knowledge for object concepts depending on whether the task is close- or open-ended. Participants viewed either names or pictures of everyday objects, or a combination of the two, and generated common, secondary, or ad hoc uses for them. Stimulus modality effects were assessed quantitatively through reaction times and qualitatively through a novel coding system, which classifies creative output on a continuum from top-down-driven to bottom-up-driven responses. Both analyses revealed differences across tasks. Importantly, for ad hoc uses, participants exposed to pictures generated more top-down-driven responses than those exposed to object names. These findings have implications for accounts of functional fixedness in creative thinking, as well as theories of semantic memory for object concepts.

The role of age of acquisition on past tense generation in Spanish-English bilinguals: an fMRI study

At its most basic sense, the sensorimotor/emergentist (S/E) model suggests that early second language (L2) learning is preferentially reliant upon sensory and motor processes, while later L2 learning is accomplished by greater reliance on executive abilities. To investigate the S/E model using fMRI, neural correlates of L2 age of acquisition were examined by employing a past-tense generation task on 22 L2 proficient bilinguals. Early bilinguals preferentially recruited left hemisphere sensorimotor regions involved in motoric control and articulation. In contrast, later learners, to a greater degree, engaged regions involved in executive cognitive control and lexical access. The data support the notion that early L2 learners devote neural resources to motor control during lexical retrieval. In contrast, later L2 learners recruit executive control mechanisms to generate the past tense. These data are consistent with the S/E model of bilingual language learning, and serve as an extension of cognitive control theories.

Neural correlates of sentence reading in children with reading difficulties

In contrast to word-level skills, the neural basis of sentence comprehension in children with reading difficulties is not well understood. Using magnetic source imaging, we investigated the spatiotemporal dynamics of regional activity associated with silent passage reading in nonimpaired and students with reading difficulties. The latter exhibited underactivation of the temporoparietal and visual cortices, bilaterally, and of the left posterior cingulate region. Late activity in left temporoparietal and ventral occipitotemporal regions was found to be a significant predictor of individual reading ability in nonimpaired, but not in students with reading difficulties. These findings support the notion that reduced temporoparietal activation during word reading in context, is a hallmark of the functional deficit in reading disability.

Morphometric differences in the Heschl's gyrus of hearing impaired and normal hearing infants

This study investigates the morphometry of Heschl's gyrus and its included primary auditory cortex (PAC) in hearing impaired (HI) and normal hearing (NH) infants. Fourty-two infants, age 8-19 months, with NH (n = 26) or hearing impairment (n = 16) were studied using high-resolution 3D magnetic resonance imaging. Gray matter (GM) and white matter (WM) volumes were obtained using software for automatic brain imaging segmentation to estimate the volume of each tissue within manually defined regions for the anterior portion of Heschl's gyrus (aHG) in each individual subject, transformed to an infant brain template space. Interactions among group (HI, NH), tissue type (GM, WM), and hemisphere (left, right) were examined using analysis of variance. Whole-brain voxel-based morphometry was utilized to explore volume differences between groups across the entire brain. The HI group showed increased GM and decreased WM in aHG compared with the NH group; likely effects of auditory deprivation. The HI group did not exhibit their typical L > R asymmetry pattern that the NH group showed. Increased GM in aHG in HI infants may represent abnormal cortical development in PAC as seen in animal models of sensory deprivation. Lower WM volume is consistent with studies with deaf adults.

Where is the semantic system? A critical review and meta-analysis of 120 functional neuroimaging studies

Semantic memory refers to knowledge about people, objects, actions, relations, self, and culture acquired through experience. The neural systems that store and retrieve this information have been studied for many years, but a consensus regarding their identity has not been reached. Using strict inclusion criteria, we analyzed 120 functional neuroimaging studies focusing on semantic processing. Reliable areas of activation in these studies were identified using the activation likelihood estimate (ALE) technique. These activations formed a distinct, left-lateralized network comprised of 7 regions: posterior inferior parietal lobe, middle temporal gyrus, fusiform and parahippocampal gyri, dorsomedial prefrontal cortex, inferior frontal gyrus, ventromedial prefrontal cortex, and posterior cingulate gyrus. Secondary analyses showed specific subregions of this network associated with knowledge of actions, manipulable artifacts, abstract concepts, and concrete concepts. The cortical regions involved in semantic processing can be grouped into 3 broad categories: posterior multimodal and heteromodal association cortex, heteromodal prefrontal cortex, and medial limbic regions. The expansion of these regions in the human relative to the nonhuman primate brain may explain uniquely human capacities to use language productively, plan, solve problems, and create cultural and technological artifacts, all of which depend on the fluid and efficient retrieval and manipulation of semantic knowledge.

Neural correlates of decision making with explicit information about probabilities and incentives in elderly healthy subjects

Recent functional neuroimaging and lesion studies demonstrate the involvement of the orbitofrontal/ventromedial prefrontal cortex as a key structure in decision making processes. This region seems to be particularly crucial when contingencies between options and consequences are unknown but have to be learned by the use of feedback following previous decisions (decision making under ambiguity). However, little is known about the neural correlates of decision making under risk conditions in which information about probabilities and potential outcomes is given. In the present study, we used functional magnetic resonance imaging to measure blood-oxygenation-level-dependent (BOLD) responses in 12 subjects during a decision making task. This task provided explicit information about probabilities and associated potential incentives. The responses were compared to BOLD signals in a control condition without information about incentives. In contrast to previous decision making studies, we completely removed the outcome phase following a decision to exclude the potential influence of feedback previously received on current decisions. The results indicate that the integration of information about probabilities and incentives leads to activations within the dorsolateral prefrontal cortex, the posterior parietal lobe, the anterior cingulate and the right lingual gyrus. We assume that this pattern of activation is due to the involvement of executive functions, conflict detection mechanisms and arithmetic operations during the deliberation phase of decisional processes that are based on explicit information.

Human Neural Systems for Conceptual Knowledge of Proper Object Use: A Functional Magnetic Resonance Imaging Study

Ideational apraxia is characterized by impaired knowledge of action concepts and proper object usage. The present functional magnetic resonance imaging study aimed at investigating the neural system underlying conceptual knowledge for proper object use in healthy subjects, when the effects of visuospatial properties and perceptual modality were taken into account. Subjects performed semantic decision tasks requiring retrieval of knowledge about either object functional purposes (functional task) or visuospatial object properties (visuospatial task) and perceptual control tasks. The semantic tasks were performed with pairs of either written object names or object drawings. Activation for the functional task in common for words and pictures, compared with the visuospatial and control tasks, was found in left parietal-temporal-occipital (PTO) junction, inferior frontal, anterior dorsal premotor, and presupplementary motor areas. Ventral inferior frontal cortex activation correlated negatively with reaction time in the functional condition. No specific activation characterized the visuospatial task compared with the functional task. The conceptual tasks, compared with the control tasks, demonstrated overlapping activation in left PTO junction, prefrontal, dorsal premotor, cuneus, and inferior temporal areas. These results outline the neural processes underlying conceptual knowledge for proper object use. The left ventral inferior frontal gyrus might facilitate behavioral decisions regarding functional/pragmatical object properties.

Maturation of speech and language functional neuroanatomy in pediatric normal controls

PURPOSE

This study explores the relationship between age and resting-state regional cerebral blood flow (rCBF) in regions associated with higher order language skills using a population of normal children, adolescents, and young adults.

METHOD

rCBF was measured in 33 normal participants between the ages of 7 and 19 years using single photon emission computed tomography. Participants' ages were regressed on rCBF values (normalized to whole-brain CBF) in 2 ways: (a) within anatomically defined, language-related regions of interest (ROIs) including Wernicke's area, Broca's area, angular gyrus, planum temporale, and Heschl's gyrus and (b) within clusters of voxels found to be significantly related to age in voxel-wise analyses.

RESULTS

rCBF in all anatomically defined ROIs except Heschl's gyrus declined as a function of age. Additionally, voxel-wise analyses revealed clusters where rCBF declined with age in left inferior parietal, left superior temporal, and right middle temporal regions-areas often implicated in higher order language functions.

CONCLUSIONS

These data suggest that ongoing maturation (e.g., dendritic pruning) in higher order cognitive areas (e.g., angular gyrus) continues into adolescence, as reflected by declining rCBF, while the primary auditory area (Heschl's gyrus) has become a stable neuronal population by age 7 years.

How reading differs from object naming at the neuronal level

This paper uses whole brain functional neuroimaging in neurologically normal participants to explore how reading aloud differs from object naming in terms of neuronal implementation. In the first experiment, we directly compared brain activation during reading aloud and object naming. This revealed greater activation for reading in bilateral premotor, left posterior superior temporal and precuneus regions. In a second experiment, we segregated the object-naming system into object recognition and speech production areas by factorially manipulating the presence or absence of objects (pictures of objects or their meaningless scrambled counterparts) with the presence or absence of speech production (vocal vs. finger press responses). This demonstrated that the areas associated with speech production (object naming and repetitively saying "OK" to meaningless scrambled pictures) corresponded exactly to the areas where responses were higher for reading aloud than object naming in Experiment 1. Collectively the results suggest that, relative to object naming, reading increases the demands on shared speech production processes. At a cognitive level, enhanced activation for reading in speech production areas may reflect the multiple and competing phonological codes that are generated from the sublexical parts of written words. At a neuronal level, it may reflect differences in the speed with which different areas are activated and integrate with one another.