Neuronal activity in visual, auditory and polysensory areas in the monkey temporal cortex during visual fixation task

The role of temporal cortex in the control of attention

Attention is an indispensable component of active vision. Contrary to the widely accepted notion that temporal cortex processing primarily focusses on passive object recognition, a series of very recent studies emphasize the role of temporal cortex structures, specifically the superior temporal sulcus (STS) and inferotemporal (IT) cortex, in guiding attention and implementing cognitive programs relevant for behavioral tasks. The goal of this theoretical paper is to advance the hypothesis that the temporal cortex attention network (TAN) entails necessary components to actively participate in attentional control in a flexible task-dependent manner. First, we will briefly discuss the general architecture of the temporal cortex with a focus on the STS and IT cortex of monkeys and their modulation with attention. Then we will review evidence from behavioral and neurophysiological studies that support their guidance of attention in the presence of cognitive control signals. Next, we propose a mechanistic framework for executive control of attention in the temporal cortex. Finally, we summarize the role of temporal cortex in implementing cognitive programs and discuss how they contribute to the dynamic nature of visual attention to ensure flexible behavior.

Auditory and Visual Motion Processing and Integration in the Primate Cerebral Cortex

The ability of animals to detect motion is critical for survival, and errors or even delays in motion perception may prove costly. In the natural world, moving objects in the visual field often produce concurrent sounds. Thus, it can highly advantageous to detect motion elicited from sensory signals of either modality, and to integrate them to produce more reliable motion perception. A great deal of progress has been made in understanding how visual motion perception is governed by the activity of single neurons in the primate cerebral cortex, but far less progress has been made in understanding both auditory motion and audiovisual motion integration. Here we, review the key cortical regions for motion processing, focussing on translational motion. We compare the representations of space and motion in the visual and auditory systems, and examine how single neurons in these two sensory systems encode the direction of motion. We also discuss the way in which humans integrate of audio and visual motion cues, and the regions of the cortex that may mediate this process.

Auditory modulation of the inferior temporal cortex neurons in rhesus monkey

We performed a systematic study to check whether neurons in the area TE (the anterior part of inferotemporal cortex) in rhesus monkey, regarded as the last stage of the ventral visual pathway, could be modulated by auditory stimuli. Two fixating rhesus monkeys were presented with visual, auditory or combined audiovisual stimuli while neuronal responses were recorded. We have found that the visually sensitive neurons are also modulated by audiovisual stimuli. This modulation is manifested as the change of response rate. Our results have shown also that the visual neurons were responsive to the sole auditory stimuli. Therefore, the concept of inferotemporal cortex unimodality in information processing should be re-evaluated.

Visual speech speeds up the neural processing of auditory speech

Synchronous presentation of stimuli to the auditory and visual systems can modify the formation of a percept in either modality. For example, perception of auditory speech is improved when the speaker's facial articulatory movements are visible. Neural convergence onto multisensory sites exhibiting supra-additivity has been proposed as the principal mechanism for integration. Recent findings, however, have suggested that putative sensory-specific cortices are responsive to inputs presented through a different modality. Consequently, when and where audiovisual representations emerge remain unsettled. In combined psychophysical and electroencephalography experiments we show that visual speech speeds up the cortical processing of auditory signals early (within 100 ms of signal onset). The auditory-visual interaction is reflected as an articulator-specific temporal facilitation (as well as a nonspecific amplitude reduction). The latency facilitation systematically depends on the degree to which the visual signal predicts possible auditory targets. The observed auditory-visual data support the view that there exist abstract internal representations that constrain the analysis of subsequent speech inputs. This is evidence for the existence of an "analysis-by-synthesis" mechanism in auditory-visual speech perception.

Multisensory integration: methodological approaches and emerging principles in the human brain

Understanding the conditions under which the brain integrates the different sensory streams and the mechanisms supporting this phenomenon is now a question at the forefront of neuroscience. In this paper, we discuss the opportunities for investigating these multisensory processes using modern imaging techniques, the nature of the information obtainable from each method and their benefits and limitations. Despite considerable variability in terms of paradigm design and analysis, some consistent findings are beginning to emerge. The detection of brain activity in human neuroimaging studies that resembles multisensory integration responses at the cellular level in other species, suggests similar crossmodal binding mechanisms may be operational in the human brain. These mechanisms appear to be distributed across distinct neuronal networks that vary depending on the nature of the shared information between different sensory cues. For example, differing extents of correspondence in time, space or content seem to reliably bias the involvement of different integrative networks which code for these cues. A combination of data obtained from haemodynamic and electromagnetic methods, which offer high spatial or temporal resolution respectively, are providing converging evidence of multisensory interactions at both "early" and "late" stages of processing--suggesting a cascade of synergistic processes operating in parallel at different levels of the cortex.

Two stages in crossmodal saccadic integration: evidence from a visual-auditory focused attention task

Saccadic reaction time (SRT) toward a visual target stimulus was measured under simultaneous presentation of an auditory non-target (accessory stimulus). Horizontal position of the target was varied (25 degrees left and right of fixation) as well as position and intensity of the auditory accessory. SRT was reduced under the presence of the accessory, and it decreased both with increasing intensity of the auditory accessory and with decreasing distance between target and accessory. The absence of a significant interaction between distance and auditory intensity suggests (1) that the intensity of the accessory stimulus has no direct influence on the process of crossmodal integration, and (2) that spatial position and intensity of the accessory are processed in separate stages. This was supported by a probability inequality test showing that the amount of neural coactivation depends on spatial distance but not on auditory intensity. The results are discussed in the framework of a two-stage model assuming separate processing of unimodal and bimodal characteristics of the stimuli. These results are related to several recent neurophysiological findings.

The cognitive and neural architecture of sequence representation

The authors theorize that 2 neurocognitive sequence-learning systems can be distinguished in serial reaction time experiments, one dorsal (parietal and supplementary motor cortex) and the other ventral (temporal and lateral prefrontal cortex). Dorsal system learning is implicit and associates noncategorized stimuli within dimensional modules. Ventral system learning can be implicit or explicit It also allows associating events across dimensions and therefore is the basis of cross-task integration or interference, depending on degree of cross-task correlation of signals. Accordingly, lack of correlation rather than limited capacity is responsible for dual-task effects on learning. The theory is relevant to issues of attentional effects on learning; the representational basis of complex, sequential skills; hippocampal-versus basal ganglia-based learning; procedural versus declarative memory; and implicit versus explicit memory.

Detection of audio-visual integration sites in humans by application of electrophysiological criteria to the BOLD effect

Electrophysiological studies in nonhuman primates and other mammals have shown that sensory cues from different modalities that appear at the same time and in the same location can increase the firing rate of multisensory cells in the superior colliculus to a level exceeding that predicted by summing the responses to the unimodal inputs. In contrast, spatially disparate multisensory cues can induce a profound response depression. We have previously demonstrated using functional magnetic resonance imaging (fMRI) that similar indices of crossmodal facilitation and inhibition are detectable in human cortex when subjects listen to speech while viewing visually congruent and incongruent lip and mouth movements. Here, we have used fMRI to investigate whether similar BOLD signal changes are observable during the crossmodal integration of nonspeech auditory and visual stimuli, matched or mismatched solely on the basis of their temporal synchrony, and if so, whether these crossmodal effects occur in similar brain areas as those identified during the integration of audio-visual speech. Subjects were exposed to synchronous and asynchronous auditory (white noise bursts) and visual (B/W alternating checkerboard) stimuli and to each modality in isolation. Synchronous and asynchronous bimodal inputs produced superadditive BOLD response enhancement and response depression across a large network of polysensory areas. The most highly significant of these crossmodal gains and decrements were observed in the superior colliculi. Other regions exhibiting these crossmodal interactions included cortex within the superior temporal sulcus, intraparietal sulcus, insula, and several foci in the frontal lobe, including within the superior and ventromedial frontal gyri. These data demonstrate the efficacy of using an analytic approach informed by electrophysiology to identify multisensory integration sites in humans and suggest that the particular network of brain areas implicated in these crossmodal integrative processes are dependent on the nature of the correspondence between the different sensory inputs (e.g. space, time, and/or form).

Contrasting spatial hearing deficits in hemianopia and spatial neglect

Spatial hearing deficits have been described in widely differing pathologies, including bilateral temporal or unilateral parietal lesions, hemispherectomy, spatial neglect and right-sided cortical lesions without neglect. However, the topography of spatial hearing deficits after cortical lesions is only poorly understood, unlike that of vision and touch. We investigated the auditory subjective straight ahead (SSA) with a new technique of binaural sound source simulation using broad-band single pulses which were filtered with head-related transfer functions and delivered with a 5 degree resolution over headphones in front space. Normal subjects showed quite accurate judgments of the SSA, with a small but significant shift to the left of centre (-1.7 degrees) in the horizontal plane. Hemineglect without a scotoma, produced a large ipsilesional deviation of the auditory SSA (+22 degrees), while two hemianopic subjects, both without neglect, showed the opposite deviation of their perceived auditory SSA towards their contralesional, blind hemifield (+10 vs -28 degrees). Two control patients with unilateral lesions, both without neglect and without hemianopia, produced normal judgments of their auditory SSA (-3.0 degrees, +3.8 degrees). These results suggest at least two contrasting influences on directional spatial hearing after unilateral cortical lesions: hemianopia vs hemispatial neglect. The results are interpreted in favour of multisensory convergence of visual and auditory information in directional spatial hearing.

Auditory-visual integration during multimodal object recognition in humans: a behavioral and electrophysiological study

The aim of this study was (1) to provide behavioral evidence for multimodal feature integration in an object recognition task in humans and (2) to characterize the processing stages and the neural structures where multisensory interactions take place. Event-related potentials (ERPs) were recorded from 30 scalp electrodes while subjects performed a forced-choice reaction-time categorization task: At each trial, the subjects had to indicate which of two objects was presented by pressing one of two keys. The two objects were defined by auditory features alone, visual features alone, or the combination of auditory and visual features. Subjects were more accurate and rapid at identifying multimodal than unimodal objects. Spatiotemporal analysis of ERPs and scalp current densities revealed several auditory-visual interaction components temporally, spatially, and functionally distinct before 200 msec poststimulus. The effects observed were (1) in visual areas, new neural activities (as early as 40 msec poststimulus) and modulation (amplitude decrease) of the N185 wave to unimodal visual stimulus, (2) in the auditory cortex, modulation (amplitude increase) of subcomponents of the unimodal auditory N1 wave around 90 to 110 msec, and (3) new neural activity over the right fronto-temporal area (140 to 165 msec). Furthermore, when the subjects were separated into two groups according to their dominant modality to perform the task in unimodal conditions (shortest reaction time criteria), the integration effects were found to be similar for the two groups over the nonspecific fronto-temporal areas, but they clearly differed in the sensory-specific cortices, affecting predominantly the sensory areas of the nondominant modality. Taken together, the results indicate that multisensory integration is mediated by flexible, highly adaptive physiological processes that can take place very early in the sensory processing chain and operate in both sensory-specific and nonspecific cortical structures in different ways.

14C-deoxyglucose mapping of the monkey brain during reaching to visual targets

The strategies used by the macaca monkey brain in controlling the performance of a reaching movement to a visual target have been studied by the quantitative autoradiographic 14C-DG method. Experiments on visually intact monkeys reaching to a visual target indicate that V1 and V2 convey visuomotor information to the cortex of the superior temporal and parietoccipital sulci which may encode the position of the moving forelimb, and to the cortex in the ventral part and lateral bank of the intraparietal sulcus which may encode the location of the visual target. The involvement of the medial bank of the intraparietal sulcus in proprioceptive guidance of movement is also suggested on the basis of the parallel metabolic effects estimated in this region and in the forelimb representations of the primary somatosensory and motor cortices. The network including the inferior postarcuate skeletomotor and prearcuate oculomotor cortical fields and the caudal periprincipal area 46 may participate in sensory-to-motor and oculomotor-to-skeletomotor transformations, in parallel with the medial and lateral intraparietal cortices. Experiments on split brain monkeys reaching to visual targets revealed that reaching is always controlled by the hemisphere contralateral to the moving forelimb whether it is visually intact or 'blind'. Two supplementary mechanisms compensate for the 'blindness' of the hemisphere controlling the moving forelimb. First, the information about the location of the target is derived from head and eye movements and is sent to the 'blind' hemisphere via inferior parietal cortical areas, while the information about the forelimb position is derived from proprioceptive mechanisms and is sent via the somatosensory and superior parietal cortices. Second, the cerebellar hemispheric extensions of vermian lobules V, VI and VIII, ipsilateral to the moving forelimb, combine visual and oculomotor information about the target position, relayed by the 'seeing' cerebral hemisphere, with sensorimotor information concerning cortical intended and peripheral actual movements of the forelimb, and then send this integrated information back to the motor cortex of the 'blind' hemisphere, thus enabling it to guide the contralateral forelimb to the target.

Comparisons of cross-modality integration in midbrain and cortex

Multisensory neurons are abundant in the superior colliculus and anterior ectosylvian cortex of the cat. Despite the fact that these areas receive inputs from different regions, and are likely to be involved in different functional roles, there multisensory neurons have many fundamental similarities. They all have multiple receptive fields, one for each sensory input, and these receptive fields overlap one another. It is this spatial correspondence among receptive fields that determines the manner in which both populations of neurons integrate the inputs they receive from different sensory channels. Several principles of integration characterize both cortical and midbrain multisensory neurons, and these constancies in the fundamentals of cross-modality integration are likely to provide a basis for coherence at different levels of the neuraxis. Yet there are also obvious differences in these populations of multisensory neurons. Cortical receptive fields are significantly larger than those in the midbrain, have a lower incidence of suppressive surrounds, and exhibit less cross-modality inhibitory interactions than in the midbrain. Presumably, these differences reflect a greater emphasis on non-spatial aspects of cross-modality integration in cortex than is required by the orientation and localization functions mediated by the superior colliculus.

Neuronal activity in monkey visual areas V1, V2, V4 and TEO during fixation task

We analyzed 577 neurons recorded from visual areas V1, V2, V4, and the inferotemporal area (TEO) of macaque monkeys, which performed a visual fixation task and a spot-off-on (blink) test during the fixation period. Among these neurons, 35% were defined as task-related cells, because they gave responses at the task-start, fixation, or task-end periods but were unresponsive to the spot blink, which was physically identical to these stimuli. Blink-responsive cells accounted for 29% and task-unresponsive cells for 30% of the neurons. The task-related response was large and frequent in V4 (34%) and TEO (41%), but small and less frequent in V1 (31%) and V2 (27%). Other observations further demonstrated nonsensory activities in these areas: In some cells, response to the fixation spot was inhibitory, whereas light stimulation on the fovea was excitatory; some V1 and V2 cells had color-irrelevant responses, and some cells responded to the spot-off only when the monkey regarded it as a task-end cue.