Selective visuo-haptic processing of shape and texture

Crossmodal interactions of haptic and visual texture information in early sensory cortex

Both visual and haptic information add to the perception of surface texture. While prior studies have reported crossmodal interactions of both sensory modalities at the behavioral level, neuroimaging studies primarily investigated texture perception in separate visual and haptic paradigms. These experimental designs, however, only allowed to identify overlap in both sensory processing streams but no interaction of visual and haptic texture processing. By varying texture characteristics in a bimodal task, the current study investigated how these crossmodal interactions are reflected at the cortical level. We used fMRI to compare cortical activation in response to matching versus non-matching visual-haptic texture information. We expected that passive simultaneous presentation of matching visual-haptic input would be sufficient to induce BOLD responses graded with varying texture characteristics. Since no cognitive evaluation of the stimuli was required, we expected to find changes primarily at a rather early processing stage. Our results confirmed our assumptions by showing crossmodal interactions of visual-haptic texture information in early somatosensory and visual cortex. However, the nature of the crossmodal effects was slightly different in both sensory cortices. In early visual cortex, matching visual-haptic information increased the average activation level and induced parametric BOLD signal variations with varying texture characteristics. In early somatosensory cortex only the latter was true. These results challenge the notion that visual and haptic texture information is processed independently and indicate a crossmodal interaction of sensory information already at an early cortical processing stage.

Early visual experience and the recognition of basic facial expressions: involvement of the middle temporal and inferior frontal gyri during haptic identification by the early blind

Face perception is critical for social communication. Given its fundamental importance in the course of evolution, the innate neural mechanisms can anticipate the computations necessary for representing faces. However, the effect of visual deprivation on the formation of neural mechanisms that underlie face perception is largely unknown. We previously showed that sighted individuals can recognize basic facial expressions by haptics surprisingly well. Moreover, the inferior frontal gyrus (IFG) and posterior superior temporal sulcus (pSTS) in the sighted subjects are involved in haptic and visual recognition of facial expressions. Here, we conducted both psychophysical and functional magnetic-resonance imaging (fMRI) experiments to determine the nature of the neural representation that subserves the recognition of basic facial expressions in early blind individuals. In a psychophysical experiment, both early blind and sighted subjects haptically identified basic facial expressions at levels well above chance. In the subsequent fMRI experiment, both groups haptically identified facial expressions and shoe types (control). The sighted subjects then completed the same task visually. Within brain regions activated by the visual and haptic identification of facial expressions (relative to that of shoes) in the sighted group, corresponding haptic identification in the early blind activated regions in the inferior frontal and middle temporal gyri. These results suggest that the neural system that underlies the recognition of basic facial expressions develops supramodally even in the absence of early visual experience.

Visual and Haptic Representations of Material Properties

Research on material perception has received an increasing amount of attention recently. Clearly, both the visual and the haptic sense play important roles in the perception of materials, yet it is still unclear how both senses compare in material perception tasks. Here, we set out to investigate the degree of correspondence between the visual and the haptic representations of different materials. We asked participants to both categorize and rate 84 different materials for several material properties. In the haptic case, participants were blindfolded and asked to assess the materials based on haptic exploration. In the visual condition, participants assessed the stimuli based on their visual impressions only. While categorization performance was less consistent in the haptic condition than in the visual one, ratings correlated highly between the visual and the haptic modality. PCA revealed that all material samples were similarly organized within the perceptual space in both modalities. Moreover, in both senses the first two principal components were dominated by hardness and roughness. These are two material features that are fundamental for the haptic sense. We conclude that although the haptic sense seems to be crucial for material perception, the information it can gather alone might not be quite fine-grained and rich enough for perfect material recognition.

Probing the neural basis of perceptual phenomenology with the touch-induced visual illusion

Using the touch-induced visual illusion we examine whether the brain regions involved in coding sensory information are dissociable from those that contain decision information. Activity in the intraparietal sulcus, as measured by functional magnetic resonance imaging, was associated with the illusion suggesting a sensory coding role whereas activity in the middle occipital gyrus differentially modulated activity according to the decisions made by subjects consistent with their reported perceptual phenomenology.

Primary somatosensory cortex discriminates affective significance in social touch

Another person's caress is one of the most powerful of all emotional social signals. How much the primary somatosensory cortices (SIs) participate in processing the pleasantness of such social touch remains unclear. Although ample empirical evidence supports the role of the insula in affective processing of touch, here we argue that SI might be more involved in affective processing than previously thought by showing that the response in SI to a sensual caress is modified by the perceived sex of the caresser. In a functional MRI study, we manipulated the perceived affective quality of a caress independently of the sensory properties at the skin: heterosexual males believed they were sensually caressed by either a man or woman, although the caress was in fact invariantly delivered by a female blind to condition type. Independent analyses showed that SI encoded, and was modulated by, the visual sex of the caress, and that this effect is unlikely to originate from the insula. This suggests that current models may underestimate the role played by SI in the affective processing of social touch.

Time course of information processing in visual and haptic object classification

Vision identifies objects rapidly and efficiently. In contrast, object recognition by touch is much slower. Furthermore, haptics usually serially accumulates information from different parts of objects, whereas vision typically processes object information in parallel. Is haptic object identification slower simply due to sequential information acquisition and the resulting memory load or due to more fundamental processing differences between the senses? To compare the time course of visual and haptic object recognition, we slowed visual processing using a novel, restricted viewing technique. In an electroencephalographic (EEG) experiment, participants discriminated familiar, nameable from unfamiliar, unnamable objects both visually and haptically. Analyses focused on the evoked and total fronto-central theta-band (5-7 Hz; a marker of working memory) and the occipital upper alpha-band (10-12 Hz; a marker of perceptual processing) locked to the onset of classification. Decreases in total upper alpha-band activity for haptic identification of objects indicate a likely processing role of multisensory extrastriate areas. Long-latency modulations of alpha-band activity differentiated between familiar and unfamiliar objects in haptics but not in vision. In contrast, theta-band activity showed a general increase over time for the slowed-down visual recognition task only. We conclude that haptic object recognition relies on common representations with vision but also that there are fundamental differences between the senses that do not merely arise from differences in their speed of processing.

Metaphorically feeling: comprehending textural metaphors activates somatosensory cortex

Conceptual metaphor theory suggests that knowledge is structured around metaphorical mappings derived from physical experience. Segregated processing of object properties in sensory cortex allows testing of the hypothesis that metaphor processing recruits activity in domain-specific sensory cortex. Using functional magnetic resonance imaging (fMRI) we show that texture-selective somatosensory cortex in the parietal operculum is activated when processing sentences containing textural metaphors, compared to literal sentences matched for meaning. This finding supports the idea that comprehension of metaphors is perceptually grounded.

Hearing Shapes Our Perception of Time: Temporal Discrimination of Tactile Stimuli in Deaf People

Confronted with the loss of one type of sensory input, we compensate using information conveyed by other senses. However, losing one type of sensory information at specific developmental times may lead to deficits across all sensory modalities. We addressed the effect of auditory deprivation on the development of tactile abilities, taking into account changes occurring at the behavioral and cortical level. Congenitally deaf and hearing individuals performed two tactile tasks, the first requiring the discrimination of the temporal duration of touches and the second requiring the discrimination of their spatial length. Compared with hearing individuals, deaf individuals were impaired only in tactile temporal processing. To explore the neural substrate of this difference, we ran a TMS experiment. In deaf individuals, the auditory association cortex was involved in temporal and spatial tactile processing, with the same chronometry as the primary somatosensory cortex. In hearing participants, the involvement of auditory association cortex occurred at a later stage and selectively for temporal discrimination. The different chronometry in the recruitment of the auditory cortex in deaf individuals correlated with the tactile temporal impairment. Thus, early hearing experience seems to be crucial to develop an efficient temporal processing across modalities, suggesting that plasticity does not necessarily result in behavioral compensation.

Visuo-haptic neuronal convergence demonstrated with an inversely effective pattern of BOLD activation

We investigated the neural substrates involved in visuo-haptic neuronal convergence using an additive-factors design in combination with fMRI. Stimuli were explored under three sensory modality conditions: viewing the object through a mirror without touching (V), touching the object with eyes closed (H), or simultaneously viewing and touching the object (VH). This modality factor was crossed with a task difficulty factor, which had two levels. On the basis of an idea similar to the principle of inverse effectiveness, we predicted that increasing difficulty would increase the relative level of multisensory gain in brain regions where visual and haptic sensory inputs converged. An ROI analysis focused on the lateral occipital tactile-visual area found evidence of inverse effectiveness in the left lateral occipital tactile-visual area, but not in the right. A whole-brain analysis also found evidence for the same pattern in the anterior aspect of the intraparietal sulcus, the premotor cortex, and the posterior insula, all in the left hemisphere. In conclusion, this study is the first to demonstrate visuo-haptic neuronal convergence based on an inversely effective pattern of brain activation.

Perceptual constancy of texture roughness in the tactile system

Our tactual perception of roughness is independent of the manner in which we touch the surface. A brick surface feels rough no matter how slowly or how rapidly we move our fingers, despite the fluctuating sensory inputs that are transmitted to the finger. Current theories of roughness perception rely solely on inputs from the cutaneous afferents, which are highly affected by scan velocity and force. The question then is: how is roughness constancy achieved? To this end, we characterized the subject's perceived roughness in six scanning conditions. These included two modes of touch: direct touch, where the finger is in contact with the surface, and indirect touch, where the surface is scanned with a hand-held probe; and three scanning modes: active (moving the hand across a stationary surface), passive (moving the surface across a stationary hand), and pseudo-passive (subject's hand is moved by the experimenter across a stationary surface). Here, we show that roughness constancy is preserved during active but not passive scanning, indicating that the hand movement is necessary for roughness constancy in both direct and indirect touch. Roughness constancy is also preserved during pseudo-passive scanning, which stresses the importance of proprioceptive input. The results show that cutaneous input provides the signals necessary for roughness perception and that proprioceptive input resulting from hand movement-rather than a motor efference copy-is necessary to achieve roughness constancy. These findings have important implications in providing realistic sensory feedback for prosthetic-hand users.

The postcentral gyrus shows sustained fMRI activation during the tactile motion aftereffect

The tactile motion aftereffect (tMAE) is a perceptual illusion in which a stationary stimulus feels as though it is moving when presented following adaptation to a unidirectionally moving tactile stimulus. Using functional magnetic resonance imaging (fMRI), we localized the brain areas responsive to tactile motion and then investigated whether these areas underlie the tMAE. Tactile stimulation was delivered to the glabrous surface of the right hand by means of a plastic cylinder with a square-wave patterned surface. In the tactile motion localizer, we contrasted periods in which the cylinder rotated at 15 rpm with periods of rest (stationary contact). Activation was observed in the contralateral (left) thalamus, postcentral gyrus, and parietal operculum. In the tMAE experiment, the cylinder rotated at 15 or 60 rpm for 2 min. The 60-rpm speed induced reliable tMAEs, whereas the 15-rpm speed did not. Of the areas activated by the tactile motion localizer, only the postcentral gyrus showed a sustained fMRI response following the offset of 60-rpm (but not 15-rpm) stimulation, presumably reflecting the illusory perception of motion.

Haptic processing in newborns of depressed and nondepressed mothers

This study was conducted to compare how newborns of depressed mothers and newborns of nondepressed mothers gather tactile information about texture. Spontaneous manual activity on objects with a smooth or rough texture was recorded in 20 newborns born of mothers with a high risk of depression or a low risk of depression. An important result of the present study is that texture-based modulation of hand-pressing frequency was observed in both neonates born of depressed mothers and neonates born of nondepressed mothers. Moreover, hand-pressing frequency did not depend on the pressure exerted on the object, since all neonates displayed comparable pressure on the held object. Lastly, the results revealed that newborns of depressed mothers held the rough object twice as long as newborns of nondepressed mothers. These results are analyzed in reference to deregulated biochemical functions in neonates born of mothers with a high risk of depression.

Tactile motion and pattern processing assessed with high-field FMRI

Processing of motion and pattern has been extensively studied in the visual domain, but much less in the somatosensory system. Here, we used ultra-high-field functional magnetic resonance imaging (fMRI) at 7 Tesla to investigate the neuronal correlates of tactile motion and pattern processing in humans under tightly controlled stimulation conditions. Different types of dynamic stimuli created the sensation of moving or stationary bar patterns during passive touch. Activity in somatosensory cortex was increased during both motion and pattern processing and modulated by motion directionality in primary and secondary somatosensory cortices (SI and SII) as well as by pattern orientation in the anterior intraparietal sulcus. Furthermore, tactile motion and pattern processing induced activity in the middle temporal cortex (hMT+/V5) and in the inferior parietal cortex (IPC), involving parts of the supramarginal und angular gyri. These responses covaried with subjects' individual perceptual performance, suggesting that hMT+/V5 and IPC contribute to conscious perception of specific tactile stimulus features. In addition, an analysis of effective connectivity using psychophysiological interactions (PPI) revealed increased functional coupling between SI and hMT+/V5 during motion processing, as well as between SI and IPC during pattern processing. This connectivity pattern provides evidence for the direct engagement of these specialized cortical areas in tactile processing during somesthesis.

Tactile-auditory shape learning engages the lateral occipital complex

Shape is an object property that inherently exists in vision and touch, and is processed in part by the lateral occipital complex (LOC). Recent studies have shown that shape can be artificially coded by sound using sensory substitution algorithms and learned with behavioral training. This finding offers a unique opportunity to test intermodal generalizability of the LOC beyond the sensory modalities in which shape is naturally perceived. Therefore, we investigated the role of the LOC in processing of shape by examining neural activity associated with learning tactile-shape-coded auditory information. Nine blindfolded sighted people learned the tactile-auditory relationship between raised abstract shapes and their corresponding shape-coded sounds over 5 d of training. Using functional magnetic resonance imaging, subjects were scanned before and after training during a task in which they first listened to a shape-coded sound transformation, then touched an embossed shape, and responded whether or not the tactile stimulus matched the auditory stimulus in terms of shape. We found that behavioral scores improved after training and that the LOC was commonly activated during the auditory and tactile conditions both before and after training. However, no significant training-related change was observed in magnitude or size of LOC activity; rather, the auditory cortex and LOC showed strengthened functional connectivity after training. These findings suggest that the LOC is available to different sensory systems for shape processing and that auditory-tactile sensory substitution training leads to neural changes allowing more direct or efficient access to this site by the auditory system.

Smoking experience modulates the cortical integration of vision and haptics

Human neuroplasticity of multisensory integration has been studied mainly in the context of natural or artificial training situations in healthy subjects. However, regular smokers also offer the opportunity to assess the impact of intensive daily multisensory interactions with smoking-related objects on the neural correlates of crossmodal object processing. The present functional magnetic resonance imaging study revealed that smokers show a comparable visuo-haptic integration pattern for both smoking paraphernalia and control objects in the left lateral occipital complex, a region playing a crucial role in crossmodal object recognition. Moreover, the degree of nicotine dependence correlated positively with the magnitude of visuo-haptic integration in the left lateral occipital complex (LOC) for smoking-associated but not for control objects. In contrast, in the left LOC non-smokers displayed a visuo-haptic integration pattern for control objects, but not for smoking paraphernalia. This suggests that prolonged smoking-related multisensory experiences in smokers facilitate the merging of visual and haptic inputs in the lateral occipital complex for the respective stimuli. Studying clinical populations who engage in compulsive activities may represent an ecologically valid approach to investigating the neuroplasticity of multisensory integration.

Dual pathways for haptic and visual perception of spatial and texture information

Segregation of information flow along a dorsally directed pathway for processing object location and a ventrally directed pathway for processing object identity is well established in the visual and auditory systems, but is less clear in the somatosensory system. We hypothesized that segregation of location vs. identity information in touch would be evident if texture is the relevant property for stimulus identity, given the salience of texture for touch. Here, we used functional magnetic resonance imaging (fMRI) to investigate whether the pathways for haptic and visual processing of location and texture are segregated, and the extent of bisensory convergence. Haptic texture-selectivity was found in the parietal operculum and posterior visual cortex bilaterally, and in parts of left inferior frontal cortex. There was bisensory texture-selectivity at some of these sites in posterior visual and left inferior frontal cortex. Connectivity analyses demonstrated, in each modality, flow of information from unisensory non-selective areas to modality-specific texture-selective areas and further to bisensory texture-selective areas. Location-selectivity was mostly bisensory, occurring in dorsal areas, including the frontal eye fields and multiple regions around the intraparietal sulcus bilaterally. Many of these regions received input from unisensory areas in both modalities. Together with earlier studies, the activation and connectivity analyses of the present study establish that somatosensory processing flows into segregated pathways for location and object identity information. The location-selective somatosensory pathway converges with its visual counterpart in dorsal frontoparietal cortex, while the texture-selective somatosensory pathway runs through the parietal operculum before converging with its visual counterpart in visual and frontal cortex. Both segregation of sensory processing according to object property and multisensory convergence appear to be universal organizing principles.

Transformation from image-based to perceptual representation of materials along the human ventral visual pathway

Every object in the world has its own surface quality that is a reflection of the material from which the object is made. We can easily identify and categorize materials (wood, metal, fabric etc.) at a glance, and this ability enables us to decide how to interact appropriately with these objects. Little is known, however, about how materials are represented in the brain, or how that representation is related to material perception or the physical properties of material surface. By combining multivoxel pattern analysis of functional magnetic resonance imaging data with perceptual and image-based physical measures of material properties, we found that the way visual information about materials is coded gradually changes from an image-based representation in early visual areas to a perceptual representation in the ventral higher-order visual areas. We suggest that meaningful information about multimodal aspects of real-world materials reside in the ventral cortex around the fusiform gyrus, where it can be utilized for categorization of materials.

Visual identification of haptically explored objects in high and low hypnotizable subjects

Hypnotizability is associated with peculiar characteristics of sensorimotor integration, imaginal abilities, and preferences in the sensory modality of imagery. The visual recognition of haptically explored objects involves an interaction among these processes and is a proper tool to investigate their possible hypnotizability-related modulation. Sixteen high hypnotizables and 16 lows participated in the study. Higher frequencies of correct recognition (RF) were observed in highs. RF improved across both groups. As an effect of learning, shorter recognition times were found in males among highs and in females among lows. The findings are consistent with the literature suggesting that hypnotizability levels may be associated with specific modes of sensory integration and/or imagery.

Shape from sound: evidence for a shape operator in the lateral occipital cortex

A recent view of cortical functional specialization suggests that the primary organizing principle of the cortex is based on task requirements, rather than sensory modality. Consistent with this view, recent evidence suggests that a region of the lateral occipitotemporal cortex (LO) may process object shape information regardless of the modality of sensory input. There is considerable evidence that area LO is involved in processing visual and haptic shape information. However, sound can also carry acoustic cues to an object's shape, for example, when a sound is produced by an object's impact with a surface. Thus, the current study used auditory stimuli that were created from recordings of objects impacting a hard surface to test the hypothesis that area LO is also involved in auditory shape processing. The objects were of two shapes, rods and balls, and of two materials, metal and wood. Subjects were required to categorize the impact sounds in one of three tasks, (1) by the shape of the object while ignoring material, (2) by the material of the object while ignoring shape, or (3) by using all the information available. Area LO was more strongly recruited when subjects discriminated impact sounds based on the shape of the object that made them, compared to when subjects discriminated those same sounds based on material. The current findings suggest that activation in area LO is shape selective regardless of sensory input modality, and are consistent with an emerging theory of perceptual functional specialization of the brain that is task-based rather than sensory modality-based.

Working memory for vibrotactile frequencies: comparison of cortical activity in blind and sighted individuals

In blind, occipital cortex showed robust activation to nonvisual stimuli in many prior functional neuroimaging studies. The cognitive processes represented by these activations are not fully determined, although a verbal recognition memory role has been demonstrated. In congenitally blind and sighted (10 per group), we contrasted responses to a vibrotactile one-back frequency retention task with 5-s delays and a vibrotactile amplitude-change task; both tasks involved the same vibration parameters. The one-back paradigm required continuous updating for working memory (WM). Findings in both groups confirmed roles in WM for right hemisphere dorsolateral prefrontal (DLPFC) and dorsal/ventral attention components of posterior parietal cortex. Negative findings in bilateral ventrolateral prefrontal cortex suggested task performance without subvocalization. In bilateral occipital cortex, blind showed comparable positive responses to both tasks, whereas WM evoked large negative responses in sighted. Greater utilization of attention resources in blind were suggested as causing larger responses in dorsal and ventral attention systems, right DLPFC, and persistent responses across delays between trials in somatosensory and premotor cortex. In sighted, responses in somatosensory and premotor areas showed iterated peaks matched to stimulation trial intervals. The findings in occipital cortex of blind suggest that tactile activations do not represent cognitive operations for nonverbal WM task. However, these data suggest a role in sensory processing for tactile information in blind that parallels a similar contribution for visual stimuli in occipital cortex of sighted.

Tactile shape discrimination recruits human lateral occipital complex during early perceptual processing

Neuroimaging studies investigating somatosensory-based object recognition in humans have revealed activity in the lateral occipital complex, a cluster of regions primarily associated with visual object recognition. To date, determining whether this activity occurs during or subsequent to recognition per se, has been difficult to assess due to the low temporal resolution of the hemodynamic response. To more finely measure the timing of somatosensory object recognition processes we employed high density EEG using a modified version of a paradigm previously applied to neuroimaging experiments. Simple geometric shapes were presented to the right index finger of 10 participants while the ongoing EEG was measured time locked to the stimulus. In the condition of primary interest participants discriminated the shape of the stimulus. In the alternate condition they judged stimulus duration. Using traditional event-related potential analysis techniques we found significantly greater amplitudes in the evoked potentials of the shape discrimination condition between 140 and 160 ms, a timeframe in which LOC mediated perceptual processes are believed to occur during visual object recognition. Scalp voltage topography and source analysis procedures indicated the lateral occipital complex as the likely source behind this effect. This finding supports a multisensory role for the lateral occipital complex during object recognition.

Differential modulation of corticospinal excitability during haptic sensing of 2-D patterns vs. textures

BACKGROUND

Recently, we showed a selective enhancement in corticospinal excitability when participants actively discriminated raised 2-D symbols with the index finger. This extra-facilitation likely reflected activation in the premotor and dorsal prefrontal cortices modulating motor cortical activity during attention to haptic sensing. However, this parieto-frontal network appears to be finely modulated depending upon whether haptic sensing is directed towards material or geometric properties. To examine this issue, we contrasted changes in corticospinal excitability when young adults (n = 18) were engaged in either a roughness discrimination on two gratings with different spatial periods, or a 2-D pattern discrimination of the relative offset in the alignment of a row of small circles in the upward or downward direction.

RESULTS

A significant effect of task conditions was detected on motor evoked potential amplitudes, reflecting the observation that corticospinal facilitation was, on average, ~18% greater in the pattern discrimination than in the roughness discrimination.

CONCLUSIONS

This differential modulation of corticospinal excitability during haptic sensing of 2-D patterns vs. roughness is consistent with the existence of preferred activation of a visuo-haptic cortical dorsal stream network including frontal motor areas during spatial vs. intensive processing of surface properties in the haptic system.

Multisensory perception of action in posterior temporal and parietal cortices

Environmental events produce many sensory cues for identifying the action that evoked the event, the agent that performed the action, and the object targeted by the action. The cues for identifying environmental events are usually distributed across multiple sensory systems. Thus, to understand how environmental events are recognized requires an understanding of the fundamental cognitive and neural processes involved in multisensory object and action recognition. Here, we investigated the neural substrates involved in auditory and visual recognition of object-directed actions. Consistent with previous work on visual recognition of isolated objects, visual recognition of actions, and recognition of environmental sounds, we found evidence for multisensory audiovisual event-selective activation bilaterally at the junction of the posterior middle temporal gyrus and the lateral occipital cortex, the left superior temporal sulcus, and bilaterally in the intraparietal sulcus. The results suggest that recognition of events through convergence of visual and auditory cues is accomplished through a network of brain regions that was previously implicated only in visual recognition of action.

Are surface properties integrated into visuohaptic object representations?

Object recognition studies have almost exclusively involved vision, focusing on shape rather than surface properties such as color. Visual object representations are thought to integrate shape and color information because changing the color of studied objects impairs their subsequent recognition. However, little is known about integration of surface properties into visuohaptic multisensory representations. Here, participants studied objects with distinct patterns of surface properties (color in Experiment 1, texture in Experiments 2 and 3) and had to discriminate between object shapes when color or texture schemes were altered in within-modal (visual and haptic) and cross-modal (visual study followed by haptic test and vice versa) conditions. In Experiment 1, color changes impaired within-modal visual recognition but had no effect on cross-modal recognition, suggesting that the multisensory representation was not influenced by modality-specific surface properties. In Experiment 2, texture changes impaired recognition in all conditions, suggesting that both unisensory and multisensory representations integrated modality-independent surface properties. However, the cross-modal impairment might have reflected either the texture change or a failure to form the multisensory representation. Experiment 3 attempted to distinguish between these possibilities by combining changes in texture with changes in orientation, taking advantage of the known view-independence of the multisensory representation, but the results were not conclusive owing to the overwhelming effect of texture change. The simplest account is that the multisensory representation integrates shape and modality-independent surface properties. However, more work is required to investigate this and the conditions under which multisensory integration of structural and surface properties occurs.

Passive tactile recognition of geometrical shape in humans: An fMRI study

Tactile shape discrimination involves frontal other than somatosensory cortex (Palva et al., 2005 [48]), but it is unclear if this frontal activity is related to exploratory concomitants. In this study, we investigated topographical details of prefrontal, premotor, and parietal areas during passive tactile recognition of 2D geometrical shapes in conditions avoiding exploratory movements. Functional magnetic resonance imaging (fMRI) was performed while the same wooden 2D geometrical shapes were blindly pressed on subjects' passive right palm in three conditions. In the RAW condition, shapes were pressed while subjects were asked to attend to the stimuli but were not trained to recognize them. After a brief training, in the SHAPE condition subjects were asked to covertly recognize shapes. In the RECOGNITION condition, they were asked to overtly recognize shapes, using response buttons with their opposite hand. Results showed that somatosensory cortex including contralateral SII, contralateral SI, and left insula was active in all conditions, confirming its importance in processing tactile shapes. In the RAW vs. SHAPE contrast, bilateral posterior parietal, insular, premotor, prefrontal, and (left) Broca's areas were more active in the latter. In the RECOGNITION, activation of (left) Broca's area correlated with correct responses. These results suggest that, even without exploratory movements, passive recognition of tactile geometrical shapes involves prefrontal and premotor as well as somatosensory regions. In this framework, Broca's area might be involved in a successful selection and/or execution of the correct responses.

Active and passive touch differentially activate somatosensory cortex in texture perception

The neural mechanisms behind active and passive touch are not yet fully understood. Using fMRI we investigated the brain correlates of these exploratory procedures using a roughness categorization task. Participants either actively explored a surface (active touch) or the surface was moved under the participant's stationary finger (passive touch). The stimuli consisted of three different grades of sandpaper which participants were required to categorize as either coarse, medium, or fine. Exploratory procedure did not affect performance although the coarse and fine surfaces were more easily categorized than the medium surface. An initial whole brain analysis revealed activation of sensory and cognitive areas, including post-central gyrus and prefrontal cortical areas, in line with areas reported in previous studies. Our main analysis revealed greater activation during active than passive touch in the contralateral primary somatosensory region but no effect of stimulus roughness. In contrast, activation in the parietal operculum (OP) was significantly affected by stimulus roughness but not by exploration procedure. Active touch also elicited greater and more distributed brain activity compared with passive touch in areas outside the somatosensory region, possibly due to the motor component of the task. Our results reveal that different cortical areas may be involved in the processing of surface exploration and surface texture, with exploration procedures affecting activations in the primary somatosensory cortex and stimulus properties affecting relatively higher cortical areas within the somatosensory system.

Enhanced effectiveness in visuo-haptic object-selective brain regions with increasing stimulus salience

The occipital and parietal lobes contain regions that are recruited for both visual and haptic object processing. The purpose of the present study was to characterize the underlying neural mechanisms for bimodal integration of vision and haptics in these visuo-haptic object-selective brain regions to find out whether these brain regions are sites of neuronal or areal convergence. Our sensory conditions consisted of visual-only (V), haptic-only (H), and visuo-haptic (VH), which allowed us to evaluate integration using the superadditivity metric. We also presented each stimulus condition at two different levels of signal-to-noise ratio or salience. The salience manipulation allowed us to assess integration using the rule of inverse effectiveness. We were able to localize previously described visuo-haptic object-selective regions in the lateral occipital cortex (lateral occipital tactile-visual area) and the intraparietal sulcus, and also localized a new region in the left anterior fusiform gyrus. There was no evidence of superadditivity with the VH stimulus at either level of salience in any of the regions. There was, however, a strong effect of salience on multisensory enhancement: the response to the VH stimulus was more enhanced at higher salience across all regions. In other words, the regions showed enhanced integration of the VH stimulus with increasing effectiveness of the unisensory stimuli. We called the effect "enhanced effectiveness." The presence of enhanced effectiveness in visuo-haptic object-selective brain regions demonstrates neuronal convergence of visual and haptic sensory inputs for the purpose of processing object shape.

Neural substrates of reliability-weighted visual-tactile multisensory integration

As sensory systems deteriorate in aging or disease, the brain must relearn the appropriate weights to assign each modality during multisensory integration. Using blood-oxygen level dependent functional magnetic resonance imaging of human subjects, we tested a model for the neural mechanisms of sensory weighting, termed “weighted connections.” This model holds that the connection weights between early and late areas vary depending on the reliability of the modality, independent of the level of early sensory cortex activity. When subjects detected viewed and felt touches to the hand, a network of brain areas was active, including visual areas in lateral occipital cortex, somatosensory areas in inferior parietal lobe, and multisensory areas in the intraparietal sulcus (IPS). In agreement with the weighted connection model, the connection weight measured with structural equation modeling between somatosensory cortex and IPS increased for somatosensory-reliable stimuli, and the connection weight between visual cortex and IPS increased for visual-reliable stimuli. This double dissociation of connection strengths was similar to the pattern of behavioral responses during incongruent multisensory stimulation, suggesting that weighted connections may be a neural mechanism for behavioral reliability weighting.

Neural and temporal dynamics underlying visual selection for action

The present study investigated the selection for action hypothesis, according to which a subject's action intention to perform a movement influences the way in which visual information is being processed. Subjects were instructed in separate blocks either to grasp or to point to a three-dimensional target-object and event-related potentials were recorded relative to stimulus onset. It was found that grasping compared with pointing resulted in a stronger N1 component and a subsequent selection negativity, which were localized to the lateral occipital complex. These effects suggest that the intention to grasp influences the processing of action-relevant features in ventral stream areas already at an early stage (e.g., enhanced processing of object orientation for grasping). These findings provide new insight in the neural and temporal dynamics underlying perception-action coupling and provide neural evidence for a selection for action principle in early human visual processing.

Object familiarity modulates effective connectivity during haptic shape perception

In the preceding paper (Lacey, S., Flueckiger, P., Stilla, R., Lava, M., Sathian, K., 2009a. Object familiarity modulates involvement of visual imagery in haptic shape perception), we showed that the activations evoked by visual imagery overlapped more extensively, and their magnitudes were more correlated, with those evoked during haptic shape perception of familiar, compared to unfamiliar, objects. Here we used task-specific analyses of functional and effective connectivity to provide convergent evidence. These analyses showed that the visual imagery and familiar haptic shape tasks activated similar networks, whereas the unfamiliar haptic shape task activated a different network. Multivariate Granger causality analyses of effective connectivity, in both a conventional form and one purged of zero-lag correlations, showed that the visual imagery and familiar haptic shape networks involved top-down paths from prefrontal cortex into the lateral occipital complex (LOC), whereas the unfamiliar haptic shape network was characterized by bottom-up, somatosensory inputs into the LOC. We conclude that shape representations in the LOC are flexibly accessible, either top-down or bottom-up, according to task demands, and that visual imagery is more involved in LOC activation during haptic shape perception when objects are familiar, compared to unfamiliar.

Cross-modal plasticity of tactile perception in blindness

This review focusses on cross-modal plasticity resulting from visual deprivation. This is viewed against the background of task-specific visual cortical recruitment that is routine during tactile tasks in the sighted and that may depend in part on visual imagery. Superior tactile perceptual performance in the blind may be practice-related, although there are unresolved questions regarding the effects of Braille-reading experience and the age of onset of blindness. While visual cortical areas are clearly more involved in tactile microspatial processing in the blind than in the sighted, it still remains unclear how to reconcile these tactile processes with the growing literature implicating visual cortical activity in a wide range of cognitive tasks in the blind, including those involving language, or with studies of short-term, reversible visual deprivation in the normally sighted that reveal plastic changes even over periods of hours or days.

Object familiarity modulates the relationship between visual object imagery and haptic shape perception

Although visual cortical engagement in haptic shape perception is well established, its relationship with visual imagery remains controversial. We addressed this using functional magnetic resonance imaging during separate visual object imagery and haptic shape perception tasks. Two experiments were conducted. In the first experiment, the haptic shape task employed unfamiliar, meaningless objects, whereas familiar objects were used in the second experiment. The activations evoked by visual object imagery overlapped more extensively, and their magnitudes were more correlated, with those evoked during haptic shape perception of familiar, compared to unfamiliar, objects. In the companion paper (Deshpande et al., this issue), we used task-specific functional and effective connectivity analyses to provide convergent evidence: these analyses showed that the neural networks underlying visual imagery were similar to those underlying haptic shape perception of familiar, but not unfamiliar, objects. We conclude that visual object imagery is more closely linked to haptic shape perception when objects are familiar, compared to when they are unfamiliar.

Multisensory visual-tactile object related network in humans: insights gained using a novel crossmodal adaptation approach

Neuroimaging techniques have provided ample evidence for multisensory integration in humans. However, it is not clear whether this integration occurs at the neuronal level or whether it reflects areal convergence without such integration. To examine this issue as regards visuo-tactile object integration we used the repetition suppression effect, also known as the fMRI-based adaptation paradigm (fMR-A). Under some assumptions, fMR-A can tag specific neuronal populations within an area and investigate their characteristics. This technique has been used extensively in unisensory studies. Here we applied it for the first time to study multisensory integration and identified a network of occipital (LOtv and calcarine sulcus), parietal (aIPS), and prefrontal (precentral sulcus and the insula) areas all showing a clear crossmodal repetition suppression effect. These results provide a crucial first insight into the neuronal basis of visuo-haptic integration of objects in humans and highlight the power of using fMR-A to study multisensory integration using non-invasinve neuroimaging techniques.

Perceptual learning of view-independence in visuo-haptic object representations

We previously showed that cross-modal recognition of unfamiliar objects is view-independent, in contrast to view-dependence within-modally, in both vision and haptics. Does the view-independent, bisensory representation underlying cross-modal recognition arise from integration of unisensory, view-dependent representations or intermediate, unisensory but view-independent representations? Two psychophysical experiments sought to distinguish between these alternative models. In both experiments, participants began from baseline, within-modal, view-dependence for object recognition in both vision and haptics. The first experiment induced within-modal view-independence by perceptual learning, which was completely and symmetrically transferred cross-modally: visual view-independence acquired through visual learning also resulted in haptic view-independence and vice versa. In the second experiment, both visual and haptic view-dependence were transformed to view-independence by either haptic-visual or visual-haptic cross-modal learning. We conclude that cross-modal view-independence fits with a model in which unisensory view-dependent representations are directly integrated into a bisensory, view-independent representation, rather than via intermediate, unisensory, view-independent representations.

A putative model of multisensory object representation

This review surveys the recent literature on visuo-haptic convergence in the perception of object form, with particular reference to the lateral occipital complex (LOC) and the intraparietal sulcus (IPS) and discusses how visual imagery or multisensory representations might underlie this convergence. Drawing on a recent distinction between object- and spatially-based visual imagery, we propose a putative model in which LOtv, a subregion of LOC, contains a modality-independent representation of geometric shape that can be accessed either bottom-up from direct sensory inputs or top-down from frontoparietal regions. We suggest that such access is modulated by object familiarity: spatial imagery may be more important for unfamiliar objects and involve IPS foci in facilitating somatosensory inputs to the LOC; by contrast, object imagery may be more critical for familiar objects, being reflected in prefrontal drive to the LOC.

Effective connectivity during haptic perception: a study using Granger causality analysis of functional magnetic resonance imaging data

Although it is accepted that visual cortical areas are recruited during touch, it remains uncertain whether this depends on top-down inputs mediating visual imagery or engagement of modality-independent representations by bottom-up somatosensory inputs. Here we addressed this by examining effective connectivity in humans during haptic perception of shape and texture with the right hand. Multivariate Granger causality analysis of functional magnetic resonance imaging (fMRI) data was conducted on a network of regions that were shape- or texture-selective. A novel network reduction procedure was employed to eliminate connections that did not contribute significantly to overall connectivity. Effective connectivity during haptic perception was found to involve a variety of interactions between areas generally regarded as somatosensory, multisensory, visual and motor, emphasizing flexible cooperation between different brain regions rather than rigid functional separation. The left postcentral sulcus (PCS), left precentral gyrus and right posterior insula were important sources of connections in the network. Bottom-up somatosensory inputs from the left PCS and right posterior insula fed into visual cortical areas, both the shape-selective right lateral occipital complex (LOC) and the texture-selective right medial occipital cortex (probable V2). In addition, top-down inputs from left postero-supero-medial parietal cortex influenced the right LOC. Thus, there is strong evidence for the bottom-up somatosensory inputs predicted by models of visual cortical areas as multisensory processors and suggestive evidence for top-down parietal (but not prefrontal) inputs that could mediate visual imagery. This is consistent with modality-independent representations accessible through both bottom-up sensory inputs and top-down processes such as visual imagery.

Posteromedial parietal cortical activity and inputs predict tactile spatial acuity

We used functional magnetic resonance imaging (fMRI) to investigate the neural circuitry underlying tactile spatial acuity at the human finger pad. Stimuli were linear, three-dot arrays, applied to the immobilized right index finger pad using a computer-controlled, MRI-compatible, pneumatic stimulator. Activity specific for spatial processing was isolated by contrasting discrimination of left-right offsets of the central dot in the array with discrimination of the duration of stimulation by an array without a spatial offset. This contrast revealed activity in a distributed frontoparietal cortical network, within which the levels of activity in right posteromedial parietal cortical foci [right posterior intraparietal sulcus (pIPS) and right precuneus] significantly predicted individual acuity thresholds. Connectivity patterns were assessed using both bivariate analysis of Granger causality with the right pIPS as a reference region and multivariate analysis of Granger causality for a selected set of regions. The strength of inputs into the right pIPS was significantly greater in subjects with better acuity than those with poorer acuity. In the better group, the paths predicting acuity converged from the left postcentral sulcus and right frontal eye field onto the right pIPS and were selective for the spatial task, and their weights predicted the level of right pIPS activity. We propose that the optimal strategy for fine tactile spatial discrimination involves interaction in the pIPS of a top-down control signal, possibly attentional, with somatosensory cortical inputs, reflecting either visualization of the spatial configurations of tactile stimuli or engagement of modality-independent circuits specialized for fine spatial processing.