Visual pathways to perception and action

Imagined transformations of bodies: an fMRI investigation

A number of spatial reasoning problems can be solved by performing an imagined transformation of one's egocentric perspective. A series of experiments were carried out to characterize this process behaviorally and in terms of its brain basis, using functional magnetic resonance imaging (tMRI). In a task contrast designed to isolate egocentric perspective transformations, participants were slower to make left-right judgments about a human figure from the figure's perspective than from their own. This transformation led to increased cortical activity around the left parietal-temporal-occipital junction, as well as in other areas including left frontal cortex. In a second task contrast comparing judgments about inverted figures to judgments about upright figures (always from the figure's perspective), participants were slower to make left-right judgments about inverted figures than upright ones. This transformation led to activation in posterior areas near those active in the first experiment, but weaker in the left hemisphere and stronger in the right, and also to substantial left frontal activation. Together, the data support the specialization of areas near the parietal-temporal-occipital junction for egocentric perspective transformations. These results are also suggestive of a dissociation between egocentric perspective transformations and object-based spatial transformations such as mental rotation.

Grasping spatial relationships: failure to demonstrate allocentric visual coding in a patient with visual form agnosia

The cortical visual mechanisms involved in processing spatial relationships remain subject to debate. According to one current view, the "dorsal stream" of visual areas, emanating from primary visual cortex and culminating in the posterior parietal cortex, mediates this aspect of visual processing. More recently, others have argued that while the dorsal stream provides egocentric coding of visual location for motor control, the separate "ventral" stream is needed for allocentric spatial coding. We have assessed the visual form agnosic patient DF, whose lesion mainly affects the ventral stream, on a prehension task requiring allocentric spatial coding. She was presented with transparent circular disks. Each disk had circular holes cut in it. DF was asked to reach out and grasp the disk by placing her fingers through the holes. The disks either had three holes (for forefinger, middle finger, and thumb) or two holes (for forefinger and thumb). The distance between the forefinger and thumb holes, and the orientation of the line formed by them, were independently varied. DF was quite unable to adjust her grip aperture or her hand orientation in the three-hole task. Although she was able to orient her hand appropriately for the two-hole disks, she still remained unable to adjust her grip aperture to the distance between the holes. These findings are consistent with the idea that allocentric processing of spatial information requires a functioning ventral stream, even when the information is being used to guide a motor response.

Implicit short-lived motor representations of space in brain damaged and healthy subjects

This article reviews experimental evidence for a specific sensorimotor function which can be dissociated from higher level representations of space. It attempts to delineate this function on the basis of results obtained by psychophysical experiments performed with brain damaged and healthy subjects. Eye and hand movement control exhibit automatic features, such that they are incompatible with conscious control. In addition, they rely on a reference frame different from the one used by conscious perception. Neuropsychological cases provide a strong support for this specific motor representation of space, which can be spared in patients with lesions of primary sensory systems who have lost conscious perception of visual, tactile or proprioceptive stimuli. Observation of these patients also showed that their motor behavior can be "attracted" by a goal only under specific conditions, that is, when the response is immediate and when no cognitive representation of this goal is elaborated at the same time. Beyond the issue of the dissociation between an implicit motor representation and more cognitive processing of spatial information, the issue of the interaction between these two systems is thus a matter of interest. It is suggested that the conscious, cognitive representation of a stimulus can contaminate or override the short-lived motor representation, but no reciprocal influence seem to occur. The interaction observed in patients can also be investigated in normals. The literature provides examples of interaction between sensorimotor and cognitive framing of space, which confirm that immediate action is not mediated by the same system as delayed action, and that elaborating a categorial representation of the action goal prevents the expression of the short-lived sensorimotor representation. It is concluded that action can be controlled by a sensory system which is specialized for on-line processing of relevant goal characteristics. The temporal constraints of this system are such that it can affect the action before a full sensory analysis of this goal has been completed. The performance obtained on the basis of this spatial sensory processing suggests that short-lived motor representations may rather be considered as real "presentation" of the action world, which share its metric properties.

The objects of action and perception

Two major functions of the visual system are discussed and contrasted. One function of vision is the creation of an internal model or percept of the external world. Most research in object perception has concentrated on this aspect of vision. Vision also guides the control of object-directed action. In the latter case, vision directs our actions with respect to the world by transforming visual inputs into appropriate motor outputs. We argue that separate, but interactive, visual systems have evolved for the perception of objects on the one hand and the control of actions directed at those objects on the other. This 'duplex' approach to high-level vision suggests that Marrian or 'reconstructive' approaches and Gibsonian or 'purposive-animate-behaviorist' approaches need not be seen as mutually exclusive, but rather as complementary in their emphases on different aspects of visual function.

Temporal profile of the directional tuning of the discharge of dorsal premotor cortical cells

This study examined the directional modulation of dorsal premotor (PMd) cells as a function of time in an instructed delay, reaching task that systematically varied direction and accuracy constraints. In two monkeys, the activity of 150 PMd cells was recorded and the preferred direction (PD) of the firing as a function of time, the PD trajectory, was calculated. Forty-one cells had nearly continuous significant directional tuning of at least 1 s duration (mean duration 1694 +/- 754 ms) that began in the instructed delay period and continued into the movement period. The PD gradually changed in time (mean change of 47.7 +/- 40.8 degrees), a change best described as a rotation. The change in the directional tuning as a function of time is consistent with the hypothesis that the PMd plays a role in the non-standard mapping of sensory stimuli into motor commands.

Activation of visuomotor systems during visually guided movements: a functional MRI study

The dorsal stream is a dominant visuomotor pathway that connects the striate and extrastriate cortices to posterior parietal areas. In turn, the posterior parietal areas send projections to the frontal primary motor and premotor areas. This cortical pathway is hypothesized to be involved in the transformation of a visual input into the appropriate motor output. In this study we used functional magnetic resonance imaging (fMRI) of the entire brain to determine the patterns of activation that occurred while subjects performed a visually guided motor task. In nine human subjects, fMRI data were acquired on a 4-T whole-body MR system equipped with a head gradient coil and a birdcage RF coil using a T2*-weighted EPI sequence. Functional activation was determined for three different tasks: (1) a visuomotor task consisting of moving a cursor on a screen with a joystick in relation to various targets, (2) a hand movement task consisting of moving the joystick without visual input, and (3) a eye movement task consisting of moving the eyes alone without visual input. Blood oxygenation level-dependent (BOLD) contrast-based activation maps of each subject were generated using period cross-correlation statistics. Subsequently, each subject's brain was normalized to Talairach coordinates, and the individual maps were compared on a pixel by pixel basis. Significantly activated pixels common to at least four out of six subjects were retained to construct the final functional image. The pattern of activation during visually guided movements was consistent with the flow of information from striate and extrastriate visual areas, to the posterior parietal complex, and then to frontal motor areas. The extensive activation of this network and the reproducibility among subjects is consistent with a role for the dorsal stream in transforming visual information into motor behavior. Also extensively activated were the medial and lateral cerebellar structures, implicating the cortico-pontocerebellar pathway in visually guided movements. Thalamic activation, particularly of the pulvinar, suggests that this nucleus is an important subcortical target of the dorsal stream.

Mapping neural interactivity onto regional activity: an analysis of semantic processing and response mode interactions

Neuroimaging studies of cognition have typically been designed to identify brain regions that are active during a cognitive process. However, identifying how brain regions interact may be equally important. In a recent study we found that the pattern of activation associated with a semantic task differed depending on how subjects made a response, suggesting that there was an interaction between the neural systems underlying response mode and semantic processing (J. M. Jennings et al., 1997, NeuroImage 5, 229-239). This result raises two important questions, which we examined here: (1) How did the regions underlying semantic performance influence one another, or interact, to produce a different pattern of activation in each case? (2) What can be learned about the neurobiology of semantic processing when different regions are identified as a function of response? We addressed these questions using structural equation modeling. This technique produced functional network models representing the effect of different regions on each other during the semantic task for each response. A common network of regions associated with semantic processing was observed and included the left inferior frontal and left superior temporal cortices, with other regions brought into that network depending on response (e.g., right middle frontal). Moreover, changes in the influences among these regions across response condition predicted the pattern of activation found previously. These results show how an arbitrary response can affect the neural pathways associated with a cognitive process, likely due to the parallel and reentrant organization of the brain, and emphasize the importance of examining functional connections when studying cognition.

The functional neuroanatomy of awareness: with a focus on the role of various anatomical systems in the control of intermodal attention

This review considers a number of recent theories on the neural basis of consciousness, with particular attention to the theories of Bogen, Crick, Llinás, Newman, and Changeux. These theories allot different roles to various key brain areas, in particular the reticular and intralaminar nuclei of the thalamus and the cortex. Crick's hypothesis is that awareness is a function of reverberating corticothalamic loops and that the spotlight of intramodal attention is controlled by the reticular nucleus of the thalamus. He also proposed different mechanisms for attention and intention ("will"). The current review presents a new hypothesis, based on elements from these hypotheses, including intermodal attention and olfaction and pain, which may pose problems for Crick's original theory. This work reviews the possible role in awareness and intermodal attention and intention of the cholinergic system in the basal forebrain and the tegmentum; the reticular, the intralaminar, and the dorsomedial thalamic nuclei; the raphe and locus coeruleus; the reticular formation; the ventral striatum and extended amygdala; insula cortex, and other selected cortical, areas. Both clinical and basic research data are covered. The conclusion is reached that the brain may work by largely nonlinear parallel processing and much intramodal shifts of attention may be effected by intracortical, or multiple corticothalamic mechanisms (small local "flashlights" rather than one major "searchlight"). But this is constrained by the functional anatomy of the circuits concerned and waking "awareness" is modulated by the many "nonspecific" systems (cholinergic from the basal forebrain, noradrenergic from the locus coeruleus, dopaminergic from the substantia nigra and ventral tegmentum, and serotoninergic from the raphe). But the principal agents for intermodal attention shifts, the "searchlight," may be two key nuclei of the cholinergic system in the mesencephalon. Clinical loss of consciousness results from damage to these nuclei but not from damage to the cholinergic nucleus basalis of the basal forebrain.

Knowing where and knowing what: a double dissociation

We report a double dissociation between visuo-spatial abilities and semantic knowledge (knowledge of the names and attributes of objects and people), in two brain-injured people with longstanding stable impairments, using a wide range of tests to explore the extent of the dissociation, MU, who has bilateral lesions of occipito-parietal cortex, shows severe spatial disorientation with relatively well-preserved semantic knowledge. He is contrasted with JBR, who has bilateral temporal lobe damage and shows severe semantic problems and no impairment on visuo-spatial tasks. Our findings thus demonstrate a double dissociation between the performance of semantic and spatial tasks by MU and JBR. This pattern is consistent with Ungerleider and Mishkin's (1982) neurophysiological hypothesis of separable cortical visual pathways; one which is specialised for spatial perception and follows a dorsal route from occipital to parietal lobes, and the other following a more ventral route from occipital to temporal lobes, whose target is semantic information needed in specifying what an object is.

Vision without knowledge

A brain-damaged patient (D.F.) with visual form agnosia is described and discussed. D.F. has a profound inability to recognize objects, places and people, in large part because of her inability to make perceptual discriminations of size, shape or orientation, despite having good visual acuity. Yet she is able to perform skilled actions that depend on that very same size, shape and orientation information that is missing from her perceptual awareness. It is suggested that her intact vision can best be understood within the framework of a dual processing model, according to which there are two cortical processing streams operating on different coding principles, for perception and for action, respectively. These may be expected to have different degrees of dependence on top-down information. One possibility is that D.F.'s lack of explicit awareness of the visual cues that guide her behaviour may result from her having to rely on a processing system which is not knowledge-based in a broad sense. Conversely, it may be that the perceptual system can provide conscious awareness of its products in normal individuals by virtue of the fact that it does interact with a stored base of visual knowledge.

Agnosia for object orientation: implications for theories of object recognition

Instances in which objects are copied accurately, but are dramatically rotated relative to the original, have been interpreted as evidence for viewpoint-independent accounts of the object recognition process. In two case reports, we demonstrate that patients who show rotation in copying also show difficulties in informing the examiner of the canonical orientation of known objects. In copying rotated versions of familiar objects, one subject showed a tendency to copy them in their canonical upright orientation, and both subjects copied non-representational line drawings with their principal axis vertically aligned, and with the irregular end pointing 'upwards'.

The perception and prehension of objects oriented in the depth plane. I. Effects of visual form agnosia

Previous studies have reported that the visual form agnosic D.F. is able to use information about visual targets for the control of motor acts, but has great difficulty in using the same visual information for perceptual report. This intact visuomotor performance may be mediated by relatively intact parieto-frontal cortical mechanisms. The present study investigated the ability of D.F. to use binocular and monocular information about the orientation of an object in the depth plane for perceptual and visuomotor purposes. A square plaque was presented at seven different orientations in depth to D.F. and to three age- and sex-matched control subjects. Subjects were required to reach out and grasp the plaque using a precision grip (index finger and thumb) under binocular and monocular viewing conditions, and in separate trials to match the orientation of a hand-held plaque to the perceived orientation of the target object, also under both binocular and monocular conditions. D.F.'s performance in grasping trials was found to be normal under binocular conditions, but was substantially worsened by removal of binocular vision. She was severely impaired at matching the orientation of the test square, although under binocular conditions her performance rose clearly above chance. The data suggest that the separation of cortical processing for visuomotor and visual perceptual purposes also applies, at least in part, to information about the orientation in depth of an object. The impaired performance under monocular viewing conditions on the visuomotor task is in agreement with recent physiological data and suggests that posterior parietal systems depend critically on binocular input for the processing of orientation in depth when ventral-stream information is unavailable.

Evidence for greater sight in blindsight following damage of primary visual cortex early in life

This review compares the behavioral, physiological and anatomical repercussions of lesions of primary visual cortex incurred by developing and mature humans, monkey and cats. Comparison of the data on the repercussions following lesions incurred earlier or later in life suggests that earlier, but not later, damage unmasks a latent flexibility of the brain to compensate partially for functions normally attributed to the damaged cortex. The compensations are best documented in the cat and they can be linked to system-wide repercussions that include selected pathway expansions and neuron degenerations, and functional adjustments in neuronal activity. Even though evidence from humans and monkeys is extremely limited, it is argued on the basis of known repercussions and similarity of visual system organization and developmental sequence, that broadly equivalent repercussions most likely occur in humans and monkeys following early lesions of primary visual cortex. The extant data suggest potentially useful directions for future investigations on functional anatomical aspects of visual capacities spared in human patients and monkeys following early damage of primary visual cortex. Such research is likely to have a substantial impact on increasing our understanding of the repercussions that result from damage elsewhere in the developing cerebral cortex and it is likely to contribute to our understanding of the remarkable ability of the human brain to adapt to insults.

Overlapping and nonoverlapping cortical projections to cortex of the superior temporal sulcus in the rhesus monkey: double anterograde tracer studies

To examine how fibers from functionally distinct cortical zones interrelate within their target areas of the superior temporal sulcus (STS) in the rhesus monkey, separate anterograde tracers were injected in two different regions of the same hemisphere known to project to the STS. Paired injections were placed in dorsal prearcuate cortex and the caudal inferior parietal lobule (IPL), interconnected regions that are part of a hypothesized distributed network concerned with visuospatial analysis or directed attention; in a presumed auditory region of the superior temporal gyrus (STG) and in extrastriate visual cortex, the caudal IPL and lower rim of the intraparietal sulcus; and in dorsal prearcuate cortex and the STG. Overlapping and nonoverlapping projections were then examined in STS visual and polysensory areas. Prefrontal and parietal fibers directly overlapped extensively in area MST and all subdivisions of presumed polysensory cortex (areas TPOc, TPOi, and TPOr), although nonoverlapping connections were also found. Although STG and IPL fibers targeted all TPO subdivisions, connections were to nonoverlapping, but often adjacent, columns. Paired prefrontal and STG injections revealed largely nonoverlapping vertical columns of connections but substantial overlap within layers VI and I or areas TPOc and TPOi. The findings suggest that area TPO contains differently connected modules that may maintain at least initial segregation of visual versus auditory inputs. Other modules within area TPO receive directly converging input from the posterior parietal and the prefrontal cortices and may participate in a distributed cortical network concerned with visuospatial functions.

Visual illusion and action

The role of allocentric cues on movement control was investigated in the present study. Pointing movements directed to the more distant vertex of closed and open configurations of the Muller-Lyer illusion, as well as to the vertex of control lines, were studied in four experimental conditions. In the first (full-vision condition) subjects saw both stimulus and their hand before and during movement, in the second (non-visual feedback condition) they saw the stimulus, but not their hand during movement. In the two remaining conditions (no-vision conditions) vision of the scene and the hand was precluded. Pointing was executed 0 sec (no vision 0 sec delay condition) or 5 sec (no-vision 5 sec delay condition) after the light was switched off. The Muller-Lyer illusion affected pointing kinematics with respect to the control lines. Subjects undershot and overshot the vertex location, respectively, of the closed and open configuration. Correspondingly, the entire kinematics were changed. The main result was, however, a gradually increasing effect of the perceptual illusion when pointing was executed from memory compared to the full-vision condition. These data are discussed according to the hypothesis that the system underlying visual perception in the allocentric frame of reference and that involved in motor action can functionally interact. The strength of this interaction depends upon the efficiency of the egocentric frame of reference by which motor actions are constructed.

The effects of visual cortex lesions on the perception of rotated shapes

Monkeys with inferior temporal cortex lesions cannot discriminate between different shapes (e.g., + vs. O) but can discriminate between shapes that differ only in orientation (e.g., 6 vs 9). Lesions of the parietal cortex, on the other hand, impair the discrimination of rotated shapes but spare the ability to discriminate between different shapes. A similar dissociation is found between some visual agnosics who can match but not identify rotated views of objects and other patients who can identify and discriminate objects only if the view is conventional; any change in orientation disrupts performance. In this paper we argue that two mechanisms may be available for the perception of rotated shapes. Which mechanism is used depends on the degree of rotation. It is suggested that the different effects of parietal and temporal lesions reflect the relative contributions of the two areas to the task and disrupt different stages of the two strategies used. A framework for the cortical processing of rotated shapes in the non-human primate is presented.

Irrelevant pictures in visual working memory

The aim of the paper is to establish firmly the phenomenon of an irrelevant picture effect and to demonstrate that the phenomenon can be used to investigate the characteristics of the visuo-spatial sketchpad. Experiment 1 introduces the use of dynamic visual noise as an interfering technique. This technique is shown to cause interference with a word list learned under visual mnemonic mediation instructions but to cause no interference when the word list is learned under verbal mediation instructions. Experiment 2 serves both to replicate this selective interference effect and to illustrate how the dynamic visual noise technique can be used to characterize further the nature of interference in the visuo-spatial sketchpad. Experiment 3 confirms the robustness of the dynamic visual noise technique. Additionally, this experiment demonstrates a double dissociation between two types of interference, dynamic visual noise and irrelevant speech, and two types of mediation, rote and visual mnemonic.

Unconscious updating of grasp motor program

Grasp modification during prehension movements was studied in response to slight variations of somesthetic information about object size. Three experiments were carried out. In experiment 1 eight subjects were required to reach and grasp an object whose size could either increase or decrease, whereas its visual image remained unmodified. The object size was changed during the experiment with uninformed subjects after a block of trials during which visual and somesthetic information were congruent. At the end of the experiment subjects were required to reproduce the size of the object with their fingers (matching test). Results showed that maximal grip aperture during prehension as well as finger aperture in the matching test were modified according to variation in object size, although no subject realized that the object had changed during the experiment. Grasp time was also altered by object size change. Greater and earlier adaptation in maximal grip aperture, as well as perturbation of grasp time, were observed for decrease than for increase in object size. However, complete compensation was never reached for both parameters. Constant confidence in vision could have prevented both complete compensation and conscious detection of object change. This was investigated in two additional experiments. In experiment 2 visual information was made unreliable by informing subjects about variation in grasped object size. This led to greater and earlier modification in maximal grip aperture than in experiment 1. Grasp time was kept almost constant regardless of size variation. In experiment 3 vision of the stimulus was prevented and no information on change in object size was given to subjects. The results of experiment 3 were similar to those of experiment 1, although modification in maximal grip aperture was larger for increase in object size. Correspondingly, grasp time was more affected by increase than by decrease in object size. The results of the three experiments suggest that kinematic parameters usually considered as dependent on object properties, such as maximal grip aperture, were modified in order to compensate perturbation of temporal parameters. This modification induced a "pragmatic" knowledge of object size (as showed by the results of the matching test), although awareness was not reached.

Object recognition without knowledge of object orientation

Several theories have been proposed to explain our ability to recognise objects from a number of viewpoints. Orientation-dependent accounts emphasize the position of the object relative to the viewer, while orientation-independent accounts (e.g. Marr) rely on descriptions of an object's component parts relative to its principal axis of elongation. An opportunity to compare the merit of these theories has arisen in a patient (L.G.) who had a rare neuropsychological sign in which knowledge of the canonical upright of object drawings was profoundly disrupted. Such orientation errors were evident in her drawings from memory and to copy, and in an orientation-matching task. In a critical experiment she showed a deficit in providing the canonical upright of individual object drawings that was independent of any difficulty in object recognition. The implications of these data for theories of object recognition are discussed.

Differences in the visual control of pantomimed and natural grasping movements

In a series of experiments, we studied the differences between natural target-directed grasping movements and 'pantomimed' movements directed towards remembered objects. Although subjects continued to scale their hand opening for object size when pantomiming, grip formation and other kinematic variables differed significantly from those seen in normal target-directed actions. This was true whether the subjects had just seen the target object 2 sec before (Experiments 1 and 2) or whether the target object was still present and they were simply required to pantomime the grasping movement beside it (Experiment 3). We argued that these pantomimed reaches were being driven by stored perceptual information about the object, and were not utilizing the normal visuomotor control systems that direct actions in real time. This interpretation received strong support from observations of a patient with visual form agnosia who was also tested. In an earlier report, we had shown that this patient showed anticipatory scaling of her grasp despite her inability to discriminate between objects perceptually on the basis of size. The present study showed, however, that the requirement to remember an object even briefly, or to pantomime an action beside it, was enough to completely disrupt her visuomotor scaling (Experiments 2 and 3). That this reflected a failure of perception rather than imagery or understanding was supported by the fact that she could convincingly pantomime actions to imagined, familiar objects, the sizes of which were known to her (Experiment 4). All these results suggest that the mechanisms underlying the formation of perceptual representations of objects are quite independent of those mediating on-line visuomotor control.