Imagining material versus geometric properties of objects: an fMRI study

Sustainable materials: a linking bridge between material perception, affordance, and aesthetics

The perception of material properties, which refers to the way in which individuals perceive and interpret materials through their sensory experiences, plays a crucial role in our interaction with the environment. Affordance, on the other hand, refers to the potential actions and uses that materials offer to users. In turn, the perception of the affordances is modulated by the aesthetic appreciation that individuals experience when interacting with the environment. Although material perception, affordances, and aesthetic appreciation are recognized as essential to fostering sustainability in society, only a few studies have investigated this subject matter systematically and their reciprocal influences. This scarcity is partially due to the challenges offered by the complexity of combining interdisciplinary topics that explore interactions between various disciplines, such as psychophysics, neurophysiology, affective science, aesthetics, and social and environmental sciences. Outlining the main findings across disciplines, this review highlights the pivotal role of material perception in shaping sustainable behaviors. It establishes connections between material perception, affordance, aesthetics, and sustainability, emphasizing the need for interdisciplinary research and integrated approaches in environmental psychology. This integration is essential as it can provide insight into how to foster sustainable and durable changes.

Self-reported vividness of tactile imagery for object properties and body regions: An exploratory study

Mental imagery ability has been examined principally in the visual domain. Despite evidence for tactile mental representations in the absence of direct stimulation, this ability is poorly understood. We investigated tactile imagery for both active and passive tasks in a large sample (N = 118). Vividness of imagery was tested across two different tasks: somatosensory imagery (of body sensitivity) and tactile imagery (of object properties) in all participants. Evidence for vivid imagery across tactile and somatosensory dimensions was found with a positive, albeit weak, correlation in imagery strength between dimensions. Imagery ratings varied across objects and object properties in the tactile imagery task and across body sites in the somatosensory imagery task. These findings shed light on the capacity for, and characteristics of, tactile mental imagery in the general population and suggest that the ability to experience vivid tactile mental images may mediate performance across a number of perceptual tasks.

Visual mental imagery: Inside the mind's eyes

What are mental images needed for? A variety of everyday situations calls for us to plan ahead; one of the clever ways our mind prepares and strategizes our next move is through mental simulation. A powerful tool in running these simulations is visual mental imagery, which can be conceived as a way to activate and maintain an internal representation of the to-be-imagined object, giving rise to predictions. Therefore, under normal conditions imagination is primarily an endogenous process, and only more rarely can mental images be activated exogenously, for example, by means of intracerebral stimulation. A large debate is still ongoing regarding the neural substrates supporting mental imagery, with the neuropsychological and neuroimaging literature agreeing in some cases, but not others. This chapter reviews the neuroscientific literature on mental imagery, and attempts to reappraise the neuropsychological and neuroimaging evidence by drawing a model of mental imagery informed by both structural and functional brain data. Overall, the role of regions in the ventral temporal cortex, especially of the left hemisphere, stands out unequivocally as a key substrate in mental imagery.

Low level visual features support robust material perception in the judgement of metallicity

The human visual system is able to rapidly and accurately infer the material properties of objects and surfaces in the world. Yet an inverse optics approach—estimating the bi-directional reflectance distribution function of a surface, given its geometry and environment, and relating this to the optical properties of materials—is both intractable and computationally unaffordable. Rather, previous studies have found that the visual system may exploit low-level spatio-chromatic statistics as heuristics for material judgment. Here, we present results from psychophysics and modeling that supports the use of image statistics heuristics in the judgement of metallicity—the quality of appearance that suggests an object is made from metal. Using computer graphics, we generated stimuli that varied along two physical dimensions: the smoothness of a metal object, and the evenness of its transparent coating. This allowed for the exploration of low-level image statistics, whilst ensuring that each stimulus was a naturalistic, physically plausible image. A conjoint-measurement task decoupled the contributions of these dimensions to the perception of metallicity. Low-level image features, as represented in the activations of oriented linear filters at different spatial scales, were found to correlate with the dimensions of the stimulus space, and decision-making models using these activations replicated observer performance in perceiving differences in metal smoothness and coating bumpiness, and judging metallicity. Importantly, the performance of these models did not deteriorate when objects were rotated within their simulated scene, with corresponding changes in image properties. We therefore conclude that low-level image features may provide reliable cues for the robust perception of metallicity.

Hemispheric asymmetries in visual mental imagery

Visual mental imagery is the faculty whereby we can "visualize" objects that are not in our line of sight. Longstanding evidence dating back over thirty years has shown that unilateral brain lesions, especially in the left temporal lobe, can impair aspects of this ability. Yet, there is currently no attempt to identify analogies between these neuropsychological findings of hemispheric asymmetry and those from other neuroscientific approaches. Here, we present a critical review of the available literature on the hemispheric laterality of visual mental imagery, by looking at cross-method patterns of evidence in the domains of lesion neuropsychology, neuroimaging, and direct cortical stimulation. Results can be summarized under three main axes. First, frontoparietal networks in both hemispheres appear to be associated with visual mental imagery. Second, lateralization patterns emerge in the temporal lobes, with the left inferior temporal lobe being the most common finding in the literature for endogenously generated images, especially, but not exclusively, when orthographic material is used to ignite imagery. Third, an opposite pattern of hemispheric laterality emerges when visual mental images are induced by exogenous stimulation; direct cortical electrical stimulation tends to produce visual imagery experiences predominantly when applied to the right temporal lobe. These patterns of hemispheric asymmetry are difficult to reconcile with the dominant model of visual mental imagery, which emphasizes the implication of early sensory cortices. They suggest instead that visual mental imagery relies on large-scale brain networks, with a crucial participation of high-level visual regions in the temporal lobes.

Topographic Somatosensory Imagery for Real-Time fMRI Brain-Computer Interfacing

Real-time functional magnetic resonance imaging (fMRI) is a promising non-invasive method for brain-computer interfaces (BCIs). BCIs translate brain activity into signals that allow communication with the outside world. Visual and motor imagery are often used as information-encoding strategies, but can be challenging if not grounded in recent experience in these modalities, e.g., in patients with locked-in-syndrome (LIS). In contrast, somatosensory imagery might constitute a more suitable information-encoding strategy as the somatosensory function is often very robust. Somatosensory imagery has been shown to activate the somatotopic cortex, but it has been unclear so far whether it can be reliably detected on a single-trial level and successfully classified according to specific somatosensory imagery content. Using ultra-high field 7-T fMRI, we show reliable and high-accuracy single-trial decoding of left-foot (LF) vs. right-hand (RH) somatosensory imagery. Correspondingly, higher decoding accuracies were associated with greater spatial separation of hand and foot decoding-weight patterns in the primary somatosensory cortex (S1). Exploiting these novel neuroscientific insights, we developed-and provide a proof of concept for-basic BCI communication by showing that binary (yes/no) answers encoded by somatosensory imagery can be decoded with high accuracy in simulated real-time (in 7 subjects) as well as in real-time (1 subject). This study demonstrates that body part-specific somatosensory imagery differentially activates somatosensory cortex in a topographically specific manner; evidence which was surprisingly still lacking in the literature. It also offers proof of concept for a novel somatosensory imagery-based fMRI-BCI control strategy, with particularly high potential for visually and motor-impaired patients. The strategy could also be transferred to lower MRI field strengths and to mobile functional near-infrared spectroscopy. Finally, given that communication BCIs provide the BCI user with a form of feedback based on their brain signals and can thus be considered as a specific form of neurofeedback, and that repeated use of a BCI has been shown to enhance underlying representations, we expect that the current BCI could also offer an interesting new approach for somatosensory rehabilitation training in the context of stroke and phantom limb pain.

Cortical Regions Encoding Hardness Perception Modulated by Visual Information Identified by Functional Magnetic Resonance Imaging With Multivoxel Pattern Analysis

Recent studies have revealed that hardness perception is determined by visual information along with the haptic input. This study investigated the cortical regions involved in hardness perception modulated by visual information using functional magnetic resonance imaging (fMRI) and multivoxel pattern analysis (MVPA). Twenty-two healthy participants were enrolled. They were required to place their left and right hands at the front and back, respectively, of a mirror attached to a platform placed above them while lying in a magnetic resonance scanner. In conditions SFT, MED, and HRD, one of three polyurethane foam pads of varying hardness (soft, medium, and hard, respectively) was presented to the left hand in a given trial, while only the medium pad was presented to the right hand in all trials. MED was defined as the control condition, because the visual and haptic information was congruent. During the scan, the participants were required to push the pad with the both hands while observing the reflection of the left hand and estimate the hardness of the pad perceived by the right (hidden) hand based on magnitude estimation. Behavioral results showed that the perceived hardness was significantly biased toward softer or harder in >73% of the trials in conditions SFT and HRD; we designated these trials as visually modulated (SFTvm and HRDvm, respectively). The accuracy map was calculated individually for each of the pair-wise comparisons of (SFTvm vs. MED), (HRDvm vs. MED), and (SFTvm vs. HRDvm) by a searchlight MVPA, and the cortical regions encoding the perceived hardness with visual modulation were identified by conjunction of the three accuracy maps in group analysis. The cluster was observed in the right sensory motor cortex, left anterior intraparietal sulcus (aIPS), bilateral parietal operculum (PO), and occipito-temporal cortex (OTC). Together with previous findings on such cortical regions, we conclude that the visual information of finger movements processed in the OTC may be integrated with haptic input in the left aIPS, and the subjective hardness perceived by the right hand with visual modulation may be processed in the cortical network between the left PO and aIPS.

Mandarin Chinese modality exclusivity norms

Modality exclusivity norms have been developed in different languages for research on the relationship between perceptual and conceptual systems. This paper sets up the first modality exclusivity norms for Chinese, a Sino-Tibetan language with semantics as its orthographically relevant level. The norms are collected through two studies based on Chinese sensory words. The experimental designs take into consideration the morpho-lexical and orthographic structures of Chinese. Study 1 provides a set of norms for Mandarin Chinese single-morpheme words in mean ratings of the extent to which a word is experienced through the five sense modalities. The degrees of modality exclusivity are also provided. The collected norms are further analyzed to examine how sub-lexical orthographic representations of sense modalities in Chinese characters affect speakers’ interpretation of the sensory words. In particular, we found higher modality exclusivity rating for the sense modality explicitly represented by a semantic radical component, as well as higher auditory dominant modality rating for characters with transparent phonetic symbol components. Study 2 presents the mean ratings and modality exclusivity of coordinate disyllabic compounds involving multiple sense modalities. These studies open new perspectives in the study of modality exclusivity. First, links between modality exclusivity and writing systems have been established which has strengthened previous accounts of the influence of orthography in the processing of visual information in reading. Second, a new set of modality exclusivity norms of compounds is proposed to show the competition of influence on modality exclusivity from different linguistic factors and potentially allow such norms to be linked to studies on synesthesia and semantic transparency.

The Somatotopy of Mental Tactile Imagery

To what degree mental imagery (MI) bears on the same neuronal processes as perception has been a central question in the neurophysiological study of imagery. Sensory-recruitment models suggest that imagery of sensory material heavily relies on the involvement of sensory cortices. Empirical evidence mainly stems from the study of visual imagery and suggests that it depends on the mentally imagined material whether hierarchically lower regions are recruited. However, evidence from other modalities is necessary to infer generalized principles. In this fMRI study we used the somatotopic organization of the primary somatosensory cortex (SI) to test in how far MI of tactile sensations activates topographically sensory brain areas. Participants (N = 19) either perceived or imagined vibrotactile stimuli on their left or right thumbs or big toes. The direct comparison to a corresponding perception condition revealed that SI was somatotopically recruited during imagery. While stimulus driven bottom-up processing induced activity throughout all SI subareas, i.e., BA1, BA3a, BA3b, and BA2 defined by probabilistic cytoarchitectonic maps, top-down recruitment during imagery was limited to the hierarchically highest subarea BA2.

Representations of microgeometric tactile information during object recognition

Object recognition through tactile perception involves two elements: the shape of the object (macrogeometric properties) and the material of the object (microgeometric properties). Here we sought to determine the characteristics of microgeometric tactile representations regarding object recognition through tactile perception. Participants were directed to recognize objects with different surface materials using either tactile information or visual information. With a quantitative analysis of the cognitive process regarding object recognition, Experiment 1 confirmed the same eight concepts (composed of rules defining distinct cognitive processes) commonly generated in both tactile and visual perceptions to accomplish the task, although an additional concept was generated during the visual task. Experiment 2 focused only on tactile perception. Three tactile objects with different surface materials (plastic, cloth and sandpaper) were used for the object recognition task. The participants answered a questionnaire regarding the process leading to their answers (which was designed based on the results obtained in Experiment 1) and to provide ratings on the vividness, familiarity and affective valence. We used these experimental data to investigate whether changes in material attributes (tactile information) change the characteristics of tactile representation. The observation showed that differences in tactile information resulted in differences in cognitive processes, vividness, familiarity and emotionality. These two experiments collectively indicated that microgeometric tactile information contributes to object recognition by recruiting various cognitive processes including episodic memory and emotion, similar to the case of object recognition by visual information.

Interoception: the forgotten modality in perceptual grounding of abstract and concrete concepts

Conceptual representations are perceptually grounded, but when investigating which perceptual modalities are involved, researchers have typically restricted their consideration to vision, touch, hearing, taste and smell. However, there is another major modality of perceptual information that is distinct from these traditional five senses; that is, interoception, or sensations inside the body. In this paper, we use megastudy data (modality-specific ratings of perceptual strength for over 32 000 words) to explore how interoceptive information contributes to the perceptual grounding of abstract and concrete concepts. We report how interoceptive strength captures a distinct form of perceptual experience across the abstract-concrete spectrum, but is markedly more important to abstract concepts (e.g. hungry, serenity) than to concrete concepts (e.g. capacity, rainy). In particular, interoception dominates emotion concepts, especially negative emotions relating to fear and sadness, moreso than other concepts of equivalent abstractness and valence. Finally, we examine whether interoceptive strength represents valuable information in conceptual content by investigating its role in concreteness effects in word recognition, and find that it enhances semantic facilitation over and above the traditional five sensory modalities. Overall, these findings suggest that interoception has comparable status to other modalities in contributing to the perceptual grounding of abstract and concrete concepts.This article is part of the theme issue 'Varieties of abstract concepts: development, use and representation in the brain'.

Image Statistics and the Representation of Material Properties in the Visual Cortex

We explored perceived material properties (roughness, texturedness, and hardness) with a novel approach that compares perception, image statistics and brain activation, as measured with fMRI. We initially asked participants to rate 84 material images with respect to the above mentioned properties, and then scanned 15 of the participants with fMRI while they viewed the material images. The images were analyzed with a set of image statistics capturing their spatial frequency and texture properties. Linear classifiers were then applied to the image statistics as well as the voxel patterns of visually responsive voxels and early visual areas to discriminate between images with high and low perceptual ratings. Roughness and texturedness could be classified above chance level based on image statistics. Roughness and texturedness could also be classified based on the brain activation patterns in visual cortex, whereas hardness could not. Importantly, the agreement in classification based on image statistics and brain activation was also above chance level. Our results show that information about visual material properties is to a large degree contained in low-level image statistics, and that these image statistics are also partially reflected in brain activity patterns induced by the perception of material images.

Functional neuroanatomy of intuitive physical inference

To engage with the world-to understand the scene in front of us, plan actions, and predict what will happen next-we must have an intuitive grasp of the world's physical structure and dynamics. How do the objects in front of us rest on and support each other, how much force would be required to move them, and how will they behave when they fall, roll, or collide? Despite the centrality of physical inferences in daily life, little is known about the brain mechanisms recruited to interpret the physical structure of a scene and predict how physical events will unfold. Here, in a series of fMRI experiments, we identified a set of cortical regions that are selectively engaged when people watch and predict the unfolding of physical events-a "physics engine" in the brain. These brain regions are selective to physical inferences relative to nonphysical but otherwise highly similar scenes and tasks. However, these regions are not exclusively engaged in physical inferences per se or, indeed, even in scene understanding; they overlap with the domain-general "multiple demand" system, especially the parts of that system involved in action planning and tool use, pointing to a close relationship between the cognitive and neural mechanisms involved in parsing the physical content of a scene and preparing an appropriate action.

Age-related differences in the neural basis of the subjective vividness of memories: evidence from multivoxel pattern classification

Although older adults often show reduced episodic memory accuracy, their ratings of the subjective vividness of their memories often equal or even exceed those of young adults. Such findings suggest that young and older adults may differentially access and/or weight different kinds of information in making vividness judgments. We examined this idea using multivoxel pattern classification of fMRI data to measure category representations while participants saw and remembered pictures of objects and scenes. Consistent with our hypothesis, there were age-related differences in how category representations related to the subjective sense of vividness. During remembering, older adults' vividness ratings were more related, relative to young adults', to category representations in prefrontal cortex. In contrast, young adults' vividness ratings were more related, relative to older adults, to category representations in parietal cortex. In addition, category representations were more correlated among posterior regions in young than in older adults, whereas correlations between PFC and posterior regions did not differ between the 2 groups. Together, these results are consistent with the idea that young and older adults differentially weight different types of information in assessing subjective vividness of their memories.

Preserved Haptic Shape Processing after Bilateral LOC Lesions

The visual and haptic perceptual systems are understood to share a common neural representation of object shape. A region thought to be critical for recognizing visual and haptic shape information is the lateral occipital complex (LOC). We investigated whether LOC is essential for haptic shape recognition in humans by studying behavioral responses and brain activation for haptically explored objects in a patient (M.C.) with bilateral lesions of the occipitotemporal cortex, including LOC. Despite severe deficits in recognizing objects using vision, M.C. was able to accurately recognize objects via touch. M.C.'s psychophysical response profile to haptically explored shapes was also indistinguishable from controls. Using fMRI, M.C. showed no object-selective visual or haptic responses in LOC, but her pattern of haptic activation in other brain regions was remarkably similar to healthy controls. Although LOC is routinely active during visual and haptic shape recognition tasks, it is not essential for haptic recognition of object shape.

SIGNIFICANCE STATEMENT

The lateral occipital complex (LOC) is a brain region regarded to be critical for recognizing object shape, both in vision and in touch. However, causal evidence linking LOC with haptic shape processing is lacking. We studied recognition performance, psychophysical sensitivity, and brain response to touched objects, in a patient (M.C.) with extensive lesions involving LOC bilaterally. Despite being severely impaired in visual shape recognition, M.C. was able to identify objects via touch and she showed normal sensitivity to a haptic shape illusion. M.C.'s brain response to touched objects in areas of undamaged cortex was also very similar to that observed in neurologically healthy controls. These results demonstrate that LOC is not necessary for recognizing objects via touch.

Psychological Basis of the Relationship Between the Rorschach Texture Response and Adult Attachment: The Mediational Role of the Accessibility of Tactile Knowledge

This study clarifies the psychological basis for the linkage between adult attachment and the texture response on the Rorschach by examining the mediational role of the accessibility of tactile knowledge. Japanese undergraduate students (n = 35) completed the Rorschach Inkblot Method, the Experiences in Close Relationship Scale for General Objects (Nakao & Kato, 2004) and a lexical decision task designed to measure the accessibility of tactile knowledge. A mediation analysis revealed that the accessibility of tactile knowledge partially mediates the association between attachment anxiety and the texture response. These results suggest that our hypothetical model focusing on the response process provides a possible explanation of the relationship between the texture response and adult attachment.

The embodied dynamics of perceptual causality: a slippery slope?

In Michotte's launching displays, while the launcher (object A) seems to move autonomously, the target (object B) seems to be displaced passively. However, the impression of A actively launching B does not persist beyond a certain distance identified as the "radius of action" of A over B. If the target keeps moving beyond the radius of action, it loses its passivity and seems to move autonomously. Here, we manipulated implied friction by drawing (or not) a surface upon which A and B are traveling, and by varying the inclination of this surface in screen- and earth-centered reference frames. Among 72 participants (n = 52 in Experiment 1; n = 20 in Experiment 2), we show that both physical embodiment of the event (looking straight ahead at a screen displaying the event on a vertical plane vs. looking downwards at the event displayed on a horizontal plane) and contextual information (objects moving along a depicted surface or in isolation) affect interpretation of the event and modulate the radius of action of the launcher. Using classical mechanics equations, we show that representational consistency of friction from radius of action responses emphasizes the embodied nature of frictional force in our cognitive architecture.

Memory for shape reactivates the lateral occipital complex

Memory is thought to be a constructive process in which the cortical regions associated with processing event features are reactivated during retrieval. Although there is evidence for non-detailed cortical reactivation during retrieval (e.g., memory for visual or auditory information reactivates the visual or auditory processing regions, respectively), there is limited evidence that memory can reactivate cortical regions associated with processing detailed, feature-specific information. Such evidence is critical to our understanding of the mechanisms of episodic retrieval. The present functional magnetic resonance imaging (fMRI) study assessed whether the lateral occipital complex (LOC), a region that preferentially processes shape, is associated with retrieval of shape information. During encoding, participants were presented with colored abstract shapes that were either intact or scrambled. During retrieval, colored disks were presented and participants indicated whether the corresponding shape was previously "intact" or "scrambled". To assess whether conscious retrieval of intact shapes reactivated LOC, we conducted a conjunction of shape perception/encoding and accurate versus inaccurate retrieval of intact shapes, which produced many activations in LOC. To determine whether activity in LOC was specific to intact shapes, we conducted a conjunction of shape perception/encoding and intact versus scrambled shapes, which also produced many activations in LOC. Furthermore, memory for intact shapes in each hemifield produced contralateral activity in LOC (e.g., memory for left visual field intact shapes activated right LOC), which reflects the specific reinstatement of perception/encoding activity. The present results extend previous feature-specific memory reactivation evidence and support the view that memory is a constructive process.

Tactile priming modulates the activation of the fronto-parietal circuit during tactile angle match and non-match processing: an fMRI study

The repetition of a stimulus task reduces the neural activity within certain cortical regions responsible for working memory (WM) processing. Although previous evidence has shown that repeated vibrotactile stimuli reduce the activation in the ventrolateral prefrontal cortex, whether the repeated tactile spatial stimuli triggered the priming effect correlated with the same cortical region remains unclear. Therefore, we used event-related functional magnetic resonance imaging (fMRI) and a delayed match-to-sample task to investigate the contributions of the priming effect to tactile spatial WM processing. Fourteen healthy volunteers were asked to encode three tactile angle stimuli during the encoding phase and one tactile angle stimulus during the recognition phase. Then, they answered whether the last angle stimulus was presented during the encoding phase. As expected, both the Match and Non-Match tasks activated a similar cerebral network. The critical new finding was decreased brain activity in the left inferior frontal gyrus (IFG), the right posterior parietal cortex (PPC) and bilateral medial frontal gyri (mFG) for the match task compared to the Non-Match task. Therefore, we suggest that the tactile priming engaged repetition suppression mechanisms during tactile angle matching, and this process decreased the activation of the fronto-parietal circuit, including IFG, mFG and PPC.

The parietal cortices participate in encoding, short-term memory, and decision-making related to tactile shape

We routinely identify objects with our hands, and the physical attributes of touched objects are often held in short-term memory to aid future decisions. However, the brain structures that selectively process tactile information to encode object shape are not fully identified. In this article we describe the areas within the human cerebral cortex that specialize in encoding, short-term memory, and decision-making related to the shape of objects explored with the hand. We performed event-related functional magnetic resonance imaging in subjects performing a shape discrimination task in which two sequentially presented objects had to be explored to determine whether they had the same shape or not. To control for low-level and nonspecific brain activations, subjects performed a temperature discrimination task in which they compared the temperature of two spheres. Our results show that although a large network of brain structures is engaged in somatosensory processing, it is the areas lining the intraparietal sulcus that selectively participate in encoding, maintaining, and deciding on tactile information related to the shape of objects.

Accuracy and speed of material categorization in real-world images

It is easy to visually distinguish a ceramic knife from one made of steel, a leather jacket from one made of denim, and a plush toy from one made of plastic. Most studies of material appearance have focused on the estimation of specific material properties such as albedo or surface gloss, and as a consequence, almost nothing is known about how we recognize material categories like leather or plastic. We have studied judgments of high-level material categories with a diverse set of real-world photographs, and we have shown (Sharan, 2009) that observers can categorize materials reliably and quickly. Performance on our tasks cannot be explained by simple differences in color, surface shape, or texture. Nor can the results be explained by observers merely performing shape-based object recognition. Rather, we argue that fast and accurate material categorization is a distinct, basic ability of the visual system.

Visuo-haptic multisensory object recognition, categorization, and representation

Visual and haptic unisensory object processing show many similarities in terms of categorization, recognition, and representation. In this review, we discuss how these similarities contribute to multisensory object processing. In particular, we show that similar unisensory visual and haptic representations lead to a shared multisensory representation underlying both cross-modal object recognition and view-independence. This shared representation suggests a common neural substrate and we review several candidate brain regions, previously thought to be specialized for aspects of visual processing, that are now known also to be involved in analogous haptic tasks. Finally, we lay out the evidence for a model of multisensory object recognition in which top-down and bottom-up pathways to the object-selective lateral occipital complex are modulated by object familiarity and individual differences in object and spatial imagery.

Principles of representation: why you can't represent the same concept twice

As embodied theories of cognition are increasingly formalized and tested, care must be taken to make informed assumptions regarding the nature of concepts and representations. In this study, we outline three reasons why one cannot, in effect, represent the same concept twice. First, online perception affects offline representation: Current representational content depends on how ongoing demands direct attention to modality-specific systems. Second, language is a fundamental facilitator of offline representation: Bootstrapping and shortcuts within the computationally cheaper linguistic system continuously modify representational content. Third, time itself is a source of representational change: As the content of underlying concepts shifts with the accumulation of direct and vicarious experience, so too does the content of representations that draw upon these concepts. We discuss the ramifications of these principles for research into both human and synthetic cognitive systems.

Spatial imagery in haptic shape perception

We have proposed that haptic activation of the shape-selective lateral occipital complex (LOC) reflects a model of multisensory object representation in which the role of visual imagery is modulated by object familiarity. Supporting this, a previous functional magnetic resonance imaging (fMRI) study from our laboratory used inter-task correlations of blood oxygenation level-dependent (BOLD) signal magnitude and effective connectivity (EC) patterns based on the BOLD signals to show that the neural processes underlying visual object imagery (objIMG) are more similar to those mediating haptic perception of familiar (fHS) than unfamiliar (uHS) shapes. Here we employed fMRI to test a further hypothesis derived from our model, that spatial imagery (spIMG) would evoke activation and effective connectivity patterns more related to uHS than fHS. We found that few of the regions conjointly activated by spIMG and either fHS or uHS showed inter-task correlations of BOLD signal magnitudes, with parietal foci featuring in both sets of correlations. This may indicate some involvement of spIMG in HS regardless of object familiarity, contrary to our hypothesis, although we cannot rule out alternative explanations for the commonalities between the networks, such as generic imagery or spatial processes. EC analyses, based on inferred neuronal time series obtained by deconvolution of the hemodynamic response function from the measured BOLD time series, showed that spIMG shared more common paths with uHS than fHS. Re-analysis of our previous data, using the same EC methods as those used here, showed that, by contrast, objIMG shared more common paths with fHS than uHS. Thus, although our model requires some refinement, its basic architecture is supported: a stronger relationship between spIMG and uHS compared to fHS, and a stronger relationship between objIMG and fHS compared to uHS.

Imaging tactile imagery: changes in brain connectivity support perceptual grounding of mental images in primary sensory cortices

Constructing mental representations in the absence of sensory stimulation is a fundamental ability of the human mind and has been investigated in numerous brain imaging studies. However, it is still unclear how brain areas facilitating mental construction processes interact with brain regions related to specific sensory representations. In this fMRI study subjects formed mental representations of tactile stimuli either from memory (imagery) or from presentation of actual corresponding vibrotactile patterned stimuli. First our analysis addressed the question of whether tactile imagery recruits primary somatosensory cortex (SI), because the activation of early perceptual areas is classically interpreted as perceptual grounding of the mental image. We also tested whether a network, referred to as 'core construction system', is involved in the generation of mental representations in the somatosensory domain. In fact, we observed imagery-induced activation of SI. We further found support for the notion of a modality independent construction network with the retrosplenial cortices and the precuneus as core components, which were supplemented with the left inferior frontal gyrus (IFG). Finally, psychophysiological interaction (PPI) analyses revealed robust imagery-modulated changes in the connectivity of these construction related areas, which suggests that they orchestrate the assembly of an abstract mental representation. Interestingly, we found increased coupling between prefrontal cortex (left IFG) and SI during mental imagery, indicating the augmentation of an abstract mental representation by reactivating perceptually grounded sensory details.

Modality exclusivity norms for 400 nouns: the relationship between perceptual experience and surface word form

We present modality exclusivity norms for 400 randomly selected noun concepts, for which participants provided perceptual strength ratings across five sensory modalities (i.e., hearing, taste, touch, smell, and vision). A comparison with previous norms showed that noun concepts are more multimodal than adjective concepts, as nouns tend to subsume multiple adjectival property concepts (e.g., perceptual experience of the concept baby involves auditory, haptic, olfactory, and visual properties, and hence leads to multimodal perceptual strength). To show the value of these norms, we then used them to test a prediction of the sound symbolism hypothesis: Analysis revealed a systematic relationship between strength of perceptual experience in the referent concept and surface word form, such that distinctive perceptual experience tends to attract distinctive lexical labels. In other words, modality-specific norms of perceptual strength are useful for exploring not just the nature of grounded concepts, but also the nature of form-meaning relationships. These norms will be of benefit to those interested in the representational nature of concepts, the roles of perceptual information in word processing and in grounded cognition more generally, and the relationship between form and meaning in language development and evolution.

A meta-analytic review of multisensory imagery identifies the neural correlates of modality-specific and modality-general imagery

The relationship between imagery and mental representations induced through perception has been the subject of philosophical discussion since antiquity and of vigorous scientific debate in the last century. The relatively recent advent of functional neuroimaging has allowed neuroscientists to look for brain-based evidence for or against the argument that perceptual processes underlie mental imagery. Recent investigations of imagery in many new domains and the parallel development of new meta-analytic techniques now afford us a clearer picture of the relationship between the neural processes underlying imagery and perception, and indeed between imagery and other cognitive processes. This meta-analysis surveyed 65 studies investigating modality-specific imagery in auditory, tactile, motor, gustatory, olfactory, and three visual sub-domains: form, color and motion. Activation likelihood estimate (ALE) analyses of activation foci reported within- and across sensorimotor modalities were conducted. The results indicate that modality-specific imagery activations generally overlap with—but are not confined to—corresponding somatosensory processing and motor execution areas, and suggest that there is a core network of brain regions recruited during imagery, regardless of task. These findings have important implications for investigations of imagery and theories of cognitive processes, such as perceptually-based representational systems.

A within-subject ERP and fMRI investigation of orientation-specific recognition memory for pictures

Despite a large body of recognition memory research, its temporal, measured with ERPs, and spatial, measured with fMRI, substrates have never been investigated in the same subjects. In the present study, we obtained this information in parallel sessions, in which subjects studied and recognized images of visual objects and their orientation. The results showed that ERP-familiarity processes between 240 and 440 ms temporally preceded recollection processes and were structurally associated with prefrontal brain regions. Recollection processes were most prominent from 440 to 600 ms and correlated with activation in temporal, parietal, and occipital brain regions. Post-retrieval monitoring, which occurred in the ERP between 600 and 1000 ms as a long-lasting slow-wave over frontal channel groups, showed correlations with activation in the prefrontal and parietal cortex. These ERP/fMRI relationships showed some correspondences to source localizations of the investigated ERP memory effects.

A functional role for modality-specific perceptual systems in conceptual representations

Theories of embodied cognition suggest that conceptual processing relies on the same neural resources that are utilized for perception and action. Evidence for these perceptual simulations comes from neuroimaging and behavioural research, such as demonstrations of somatotopic motor cortex activations following the presentation of action-related words, or facilitation of grasp responses following presentation of object names. However, the interpretation of such effects has been called into question by suggestions that neural activation in modality-specific sensorimotor regions may be epiphenomenal, and merely the result of spreading activations from “disembodied”, abstracted, symbolic representations. Here, we present two studies that focus on the perceptual modalities of touch and proprioception. We show that in a timed object-comparison task, concurrent tactile or proprioceptive stimulation to the hands facilitates conceptual processing relative to control stimulation. This facilitation occurs only for small, manipulable objects, where tactile and proprioceptive information form part of the multimodal perceptual experience of interacting with such objects, but facilitation is not observed for large, nonmanipulable objects where such perceptual information is uninformative. Importantly, these facilitation effects are independent of motor and action planning, and indicate that modality-specific perceptual information plays a functionally constitutive role in our mental representations of objects, which supports embodied assumptions that concepts are grounded in the same neural systems that govern perception and action.

Prominent activation of the intraparietal and somatosensory areas during angle discrimination by intra-active touch

Intra-active touch (IAT) is a process that involves a body part doing the touching (active touch [AT]) and another body part being touched (passive touch [PT]) simultaneously. The brain representation related to IAT is still unclear. A total of 23 subjects carried out angle discrimination under PT, AT and IAT conditions with functional magnetic resonance imaging. All of the tasks were strictly dependent on cutaneous feedback from the finger(s). As the subjects were able to perceive the angle stimuli from the right (touching) and left (touched) sides during the IAT condition, we expected there would be greater brain activation with the IAT condition than for the AT or PT condition. Therefore, we hypothesized that the region within and/or around the intraparietal sulcus (IPS) and the part of the primary somatosensory cortex (SI) that is associated with high-level tactile spatial processing would be more active during the IAT task than during the AT and PT tasks. Compared with the areas activated by the motor somatosensory control task, the most prominent activation areas evoked by the three-angle discrimination tasks were in the SI and secondary somatosensory cortex areas in the bilateral parietal operculum, IPS, lateral occipital complex, insula and cerebellum. Finally, we directly compared IAT with AT and PT, and the results suggest that the contralateral part of IPS and part of the SI are more active under IAT conditions than under either AT or PT conditions. These results suggest that both hemispheres contribute to angle discrimination during IAT.