Two distinct types of remapping in primate cortical area V4

Presaccadic Attention Enhances and Reshapes the Contrast Sensitivity Function Differentially around the Visual Field

Contrast sensitivity (CS), which constrains human vision, decreases from fovea to periphery, from the horizontal to the vertical meridian, and from the lower vertical to the upper vertical meridian. It also depends on spatial frequency (SF), and the contrast sensitivity function (CSF) depicts this relation. To compensate for these visual constraints, we constantly make saccades and foveate on relevant objects in the scene. Already before saccade onset, presaccadic attention shifts to the saccade target and enhances perception. However, it is unknown whether and how it modulates the interplay between CS and SF, and if this effect varies around polar angle meridians. CS enhancement may result from a horizontal or vertical shift of the CSF, increase in bandwidth, or any combination. In addition, presaccadic attention could enhance CS similarly around the visual field, or it could benefit perception more at locations with poorer performance (i.e., vertical meridian). Here, we investigated these possibilities by extracting key attributes of the CSF of human observers. The results reveal that presaccadic attention (1) increases CS across SF, (2) increases the most preferred and the highest discernable SF, and (3) narrows the bandwidth. Therefore, presaccadic attention helps bridge the gap between presaccadic and postsaccadic input by increasing visibility at the saccade target. Counterintuitively, this CS enhancement was more pronounced where perception is better-along the horizontal than the vertical meridian-exacerbating polar angle asymmetries. Our results call for an investigation of the differential neural modulations underlying presaccadic perceptual changes for different saccade directions.

Widespread receptive field remapping in early primate visual cortex

Predictive remapping of receptive fields (RFs) is thought to be one of the critical mechanisms for enforcing perceptual stability during eye movements. While RF remapping has been observed in several cortical areas, its role in early visual cortex and its consequences on the tuning properties of neurons have been poorly understood. Here, we track remapping RFs in hundreds of neurons from visual area V2 while subjects perform a cued saccade task. We find that remapping is widespread in area V2 across neurons from all recorded cortical layers and cell types. Furthermore, our results suggest that remapping RFs not only maintain but also transiently enhance their feature selectivity due to untuned suppression. Taken together, these findings shed light on the dynamics and prevalence of remapping in the early visual cortex, forcing us to revise current models of perceptual stability during saccadic eye movements.

Neural correlates of perisaccadic visual mislocalization in extrastriate cortex

When interacting with the visual world using saccadic eye movements (saccades), the perceived location of visual stimuli becomes biased, a phenomenon called perisaccadic mislocalization. However, the neural mechanism underlying this altered visuospatial perception and its potential link to other perisaccadic perceptual phenomena have not been established. Using the electrophysiological recording of extrastriate areas in four male macaque monkeys, combined with a computational model, we were able to quantify spatial bias around the saccade target (ST) based on the perisaccadic dynamics of extrastriate spatiotemporal sensitivity captured by a statistical model. This approach could predict the perisaccadic spatial bias around the ST, consistent with behavioral data, and revealed the precise neuronal response components underlying representational bias. These findings also establish the crucial role of increased sensitivity near the ST for neurons with receptive fields far from the ST in driving the ST spatial bias. Moreover, we showed that, by allocating more resources for visual target representation, visual areas enhance their representation of the ST location, even at the expense of transient distortions in spatial representation. This potential neural basis for perisaccadic ST representation also supports a general role for extrastriate neurons in creating the perception of stimulus location.

Feature-selective responses in macaque visual cortex follow eye movements during natural vision

In natural vision, primates actively move their eyes several times per second via saccades. It remains unclear whether, during this active looking, visual neurons exhibit classical retinotopic properties, anticipate gaze shifts or mirror the stable quality of perception, especially in complex natural scenes. Here, we let 13 monkeys freely view thousands of natural images across 4.6 million fixations, recorded 883 h of neuronal responses in six areas spanning primary visual to anterior inferior temporal cortex and analyzed spatial, temporal and featural selectivity in these responses. Face neurons tracked their receptive field contents, indicated by category-selective responses. Self-consistency analysis showed that general feature-selective responses also followed eye movements and remained gaze-dependent over seconds of viewing the same image. Computational models of feature-selective responses located retinotopic receptive fields during free viewing. We found limited evidence for feature-selective predictive remapping and no viewing-history integration. Thus, ventral visual neurons represent the world in a predominantly eye-centered reference frame during natural vision.

WIDESPREAD RECEPTIVE FIELD REMAPPING IN EARLY VISUAL CORTEX

Our eyes are in constant motion, yet we perceive the visual world as stable. Predictive remapping of receptive fields is thought to be one of the critical mechanisms for enforcing perceptual stability during eye movements. While receptive field remapping has been identified in several cortical areas, the spatiotemporal dynamics of remapping, and its consequences on the tuning properties of neurons, remain poorly understood. Here, we tracked remapping receptive fields in hundreds of neurons from visual Area V2 while subjects performed a cued saccade task. We found that remapping was far more widespread in Area V2 than previously reported and can be found in neurons from all recorded cortical layers and cell types. Surprisingly, neurons undergoing remapping exhibit sensitivity to two punctate locations in visual space. Furthermore, we found that feature selectivity is not only maintained during remapping but transiently increases due to untuned suppression. Taken together, these results shed light on the spatiotemporal dynamics of remapping and its ubiquitous prevalence in the early visual cortex, and force us to revise current models of perceptual stability.

Perisaccadic and attentional remapping of receptive fields in lateral intraparietal area and frontal eye fields

The nature and function of perisaccadic receptive field (RF) remapping have been controversial. We use a delayed saccade task to reduce previous confounds and examine the remapping time course in the lateral intraparietal area and frontal eye fields. In the delay period, the RF shift direction turns from the initial fixation to the saccade target. In the perisaccadic period, RFs first shift toward the target (convergent remapping), but around the time of saccade onset/offset, the shifts become predominantly toward the post-saccadic RF locations (forward remapping). Thus, unlike forward remapping that depends on the corollary discharge (CD) of the saccade command, convergent remapping appears to follow attention from the initial fixation to the target. We model the data with attention-modulated and CD-gated connections and show that both sets of connections emerge automatically in neural networks trained to update stimulus retinal locations across saccades. Our work thus unifies previous findings into a mechanism for transsaccadic visual stability.

Time-varying generalized linear models: characterizing and decoding neuronal dynamics in higher visual areas

To create a behaviorally relevant representation of the visual world, neurons in higher visual areas exhibit dynamic response changes to account for the time-varying interactions between external (e.g., visual input) and internal (e.g., reward value) factors. The resulting high-dimensional representational space poses challenges for precisely quantifying individual factors' contributions to the representation and readout of sensory information during a behavior. The widely used point process generalized linear model (GLM) approach provides a powerful framework for a quantitative description of neuronal processing as a function of various sensory and non-sensory inputs (encoding) as well as linking particular response components to particular behaviors (decoding), at the level of single trials and individual neurons. However, most existing variations of GLMs assume the neural systems to be time-invariant, making them inadequate for modeling nonstationary characteristics of neuronal sensitivity in higher visual areas. In this review, we summarize some of the existing GLM variations, with a focus on time-varying extensions. We highlight their applications to understanding neural representations in higher visual areas and decoding transient neuronal sensitivity as well as linking physiology to behavior through manipulation of model components. This time-varying class of statistical models provide valuable insights into the neural basis of various visual behaviors in higher visual areas and hold significant potential for uncovering the fundamental computational principles that govern neuronal processing underlying various behaviors in different regions of the brain.

Grouping cells in primate visual cortex

Our perception of how objects are laid out in visual scenes is remarkably stable, despite rapid shifts in the patterns of light that fall on the retina with each saccade. One mechanism that may help establish perceptual stability is border ownership assignment. Studies in macaque area V2 have identified border ownership neurons that signal which side of a border belongs to a foreground surface. This signal persists for hundreds of milliseconds after border ownership has been rendered ambiguous by deleting the stimulus features that distinguish foreground from background. Remarkably, this signal survives eye movements: border ownership neurons also exhibit border ownership signals de novo when an eye movement places the newly ambiguous border within their receptive field. The grouping cell hypothesis proposes the existence of hypothetical grouping cells in a downstream brain area. These cells would compute persistent proto-object representations and therefore have the properties to endow cells in upstream brain areas with selectivity for border ownership. Such grouping cells have been predicted to show a centripetal and persistent pattern of preferred side of ownership for a border placed parallel to the perimeter of their classical receptive field, and such a centripetal ownership preference pattern should also occur de novo in these same cells if an ambiguous border lands in their receptive field after a saccade. It is unknown if grouping cells exist. Here we used laminar multielectrodes in area V4 - the main source of feedback to V2 - of behaving macaques to determine whether such grouping cells exist. Consistent with the model prediction we find a substantial population of neurons with these properties, in all laminar compartments, and they exhibit a response latency that is short enough to act as the source that endows neurons in V2 with selectivity for border ownership. While grouping cell activity provides information about the location of foreground surfaces, these neurons are, counterintuitively, not as strongly tuned for luminance contrast polarity, a feature of those surfaces, as are border ownership cells. Our data suggest a division of labor in which these newly discovered grouping cells provide spatiotemporal continuity of segmented surfaces whereas border ownership cells link this location information with surface features such as luminance contrast.

Beyond Correlation: Optimal Transport Metrics For Characterizing Representational Stability and Remapping in Neurons Encoding Spatial Memory

Spatial representations in the entorhinal cortex (EC) and hippocampus (HPC) are fundamental to cognitive functions like navigation and memory. These representations, embodied in spatial field maps, dynamically remap in response to environmental changes. However, current methods, such as Pearson's correlation coefficient, struggle to capture the complexity of these remapping events, especially when fields do not overlap, or transformations are non-linear. This limitation hinders our understanding and quantification of remapping, a key aspect of spatial memory function. To address this, we propose a family of metrics based on the Earth Mover's Distance (EMD) as a versatile framework for characterizing remapping. Applied to both normalized and unnormalized distributions, the EMD provides a granular, noise-resistant, and rate-robust description of remapping. This approach enables the identification of specific cell types and the characterization of remapping in various scenarios, including disease models. Furthermore, the EMD's properties can be manipulated to identify spatially tuned cell types and to explore remapping as it relates to alternate information forms such as spatiotemporal coding. By employing approximations of the EMD, we present a feasible, lightweight approach that complements traditional methods. Our findings underscore the potential of the EMD as a powerful tool for enhancing our understanding of remapping in the brain and its implications for spatial navigation, memory studies and beyond.

Neural correlates of perisaccadic visual mislocalization in extrastriate cortex

When interacting with the visual world using saccadic eye movements (saccades), the perceived location of visual stimuli becomes biased, a phenomenon called perisaccadic mislocalization, which is indeed an exemplar of the brain's dynamic representation of the visual world. However, the neural mechanism underlying this altered visuospatial perception and its potential link to other perisaccadic perceptual phenomena have not been established. Using a combined experimental and computational approach, we were able to quantify spatial bias around the saccade target (ST) based on the perisaccadic dynamics of extrastriate spatiotemporal sensitivity captured by statistical models. This approach could predict the perisaccadic spatial bias around the ST, consistent with the psychophysical studies, and revealed the precise neuronal response components underlying representational bias. These findings also established the crucial role of response remapping toward ST representation for neurons with receptive fields far from the ST in driving the ST spatial bias. Moreover, we showed that, by allocating more resources for visual target representation, visual areas enhance their representation of the ST location, even at the expense of transient distortions in spatial representation. This potential neural basis for perisaccadic ST representation, also supports a general role for extrastriate neurons in creating the perception of stimulus location.

Mice require proprioception to establish long-term visuospatial memory

Because the retina moves constantly, the retinotopic representation of the visual world is spatially inaccurate and the brain must transform this spatially inaccurate retinal signal to a spatially accurate signal usable for perception and action. One of the salient discoveries of modern neuroscience is the role of the hippocampus in establishing gaze-independent, long-term visuospatial memories. The rat hippocampus has neurons which report the animal's position in space regardless of its angle of gaze. Rats with hippocampal lesions are unable to find the location of an escape platform hidden in a pool of opaque fluid, the Morris Water Maze (MWM) based on the visual aspects of their surrounding environment. Here we show that the representation of proprioception in the dysgranular zone of primary somatosensory cortex is equivalently necessary for mice to learn the location of the hidden platform, presumably because without it they cannot create a long-term gaze-independent visuospatial representation of their environment from the retinal signal. They have no trouble finding the platform when it is marked by a flag, and they have no motor or vestibular deficits.

Perisaccadic and Attentional Remapping of Receptive Fields in Lateral Intraparietal Area and Frontal Eye Fields

The nature and function of perisaccadic receptive-field (RF) remapping have been controversial. We used a delayed saccade task to reduce previous confounds and examined the remapping time course in areas LIP and FEF. In the delay period, the RF shift direction turned from the initial fixation to the saccade target. In the perisaccadic period, RFs first shifted toward the target (convergent remapping) but around the time of saccade onset/offset, the shifts became predominantly toward the post-saccadic RF locations (forward remapping). Thus, unlike forward remapping that depends on the corollary discharge (CD) of the saccade command, convergent remapping appeared to follow attention from the initial fixation to the target. We modelled the data with attention-modulated and CD-gated connections, and showed that both sets of connections emerged automatically in neural networks trained to update stimulus retinal locations across saccades. Our work thus unifies previous findings into a mechanism for transsaccadic visual stability.

A presaccadic perceptual impairment at the postsaccadic location of the blindspot

Saccadic eye movements are preceded by profound changes in visual perception. These changes have been linked to the phenomenon of 'forward remapping', in which cells begin to respond to stimuli that appear in their post-saccadic receptive field before the eye has moved. Few studies have examined the perceptual consequences of remapping of areas of impaired sensory acuity, such as the blindspot. Understanding the perceptual consequences of remapping of scotomas may produce important insights into why some neurovisual deficits, such as hemianopia are so intractable for rehabilitation. The current study took advantage of a naturally occurring scotoma in healthy participants (the blindspot) to examine pre-saccadic perception at the upcoming location of the blindspot. Participants viewed stimuli monocularly and were required to make stimulus-driven vertical eye-movements. At a variable latency between the onset of saccade target (ST) and saccade execution a discrimination target (DT) was presented at one of 4 possible locations; within the blindspot, contralateral to the blindspot, in post-saccadic location of the blindspot and contralateral to the post-saccadic location of the blindspot. There was a significant perceptual impairment at the post-saccadic location of the blindspot relative to the contralateral post-saccadic location of the blindspot and the post-saccadic location of the blindspot in a no-saccade control condition. These data are consistent with the idea that the visual system includes a representation of the blindspot which is remapped prior to saccade onset.

Occipital and parietal cortex participate in a cortical network for transsaccadic discrimination of object shape and orientation

Saccades change eye position and interrupt vision several times per second, necessitating neural mechanisms for continuous perception of object identity, orientation, and location. Neuroimaging studies suggest that occipital and parietal cortex play complementary roles for transsaccadic perception of intrinsic versus extrinsic spatial properties, e.g., dorsomedial occipital cortex (cuneus) is sensitive to changes in spatial frequency, whereas the supramarginal gyrus (SMG) is modulated by changes in object orientation. Based on this, we hypothesized that both structures would be recruited to simultaneously monitor object identity and orientation across saccades. To test this, we merged two previous neuroimaging protocols: 21 participants viewed a 2D object and then, after sustained fixation or a saccade, judged whether the shape or orientation of the re-presented object changed. We, then, performed a bilateral region-of-interest analysis on identified cuneus and SMG sites. As hypothesized, cuneus showed both saccade and feature (i.e., object orientation vs. shape change) modulations, and right SMG showed saccade-feature interactions. Further, the cuneus activity time course correlated with several other cortical saccade/visual areas, suggesting a 'functional network' for feature discrimination. These results confirm the involvement of occipital/parietal cortex in transsaccadic vision and support complementary roles in spatial versus identity updating.

Presaccadic attention sharpens visual acuity

Visual perception is limited by spatial resolution, the ability to discriminate fine details. Spatial resolution not only declines with eccentricity but also differs for polar angle locations around the visual field, also known as 'performance fields'. To compensate for poor peripheral resolution, we make rapid eye movements-saccades-to bring peripheral objects into high-acuity foveal vision. Already before saccade onset, visual attention shifts to the saccade target location and prioritizes visual processing. This presaccadic shift of attention improves performance in many visual tasks, but whether it changes resolution is unknown. Here, we investigated whether presaccadic attention sharpens peripheral spatial resolution; and if so, whether such effect interacts with performance fields asymmetries. We measured acuity thresholds in an orientation discrimination task during fixation and saccade preparation around the visual field. The results revealed that presaccadic attention sharpens acuity, which can facilitate a smooth transition from peripheral to foveal representation. This acuity enhancement is similar across the four cardinal locations; thus, the typically robust effect of presaccadic attention does not change polar angle differences in resolution.

Tuning curves vs. population responses, and perceptual consequences of receptive-field remapping

Sensory processing is often studied by examining how a given neuron responds to a parameterized set of stimuli (tuning curve) or how a given stimulus evokes responses from a parameterized set of neurons (population response). Although tuning curves and the corresponding population responses contain the same information, they can have different properties. These differences are known to be important because the perception of a stimulus should be decoded from its population response, not from any single tuning curve. The differences are less studied in the spatial domain where a cell's spatial tuning curve is simply its receptive field (RF) profile. Here, we focus on evaluating the common belief that perrisaccadic forward and convergent RF shifts lead to forward (translational) and convergent (compressive) perceptual mislocalization, respectively, and investigate the effects of three related factors: decoders' awareness of RF shifts, changes of cells' covering density near attentional locus (the saccade target), and attentional response modulation. We find that RF shifts alone produce either no shift or an opposite shift of the population responses depending on whether or not decoders are aware of the RF shifts. Thus, forward RF shifts do not predict forward mislocalization. However, convergent RF shifts change cells' covering density for aware decoders (but not for unaware decoders) which may predict convergent mislocalization. Finally, attentional modulation adds a convergent component to population responses for stimuli near the target. We simulate the combined effects of these factors and discuss the results with extant mislocalization data. We speculate that perisaccadic mislocalization might be the flash-lag effect unrelated to perisaccadic RF remapping but to resolve the issue, one has to address the question of whether or not perceptual decoders are aware of RF shifts.

Spatiotopic and retinotopic memory in the context of natural images

Neural responses throughout the visual cortex encode stimulus location in a retinotopic (i.e., eye-centered) reference frame, and memory for stimulus position is most precise in retinal coordinates. Yet visual perception is spatiotopic: objects are perceived as stationary, even though eye movements cause frequent displacement of their location on the retina. Previous studies found that, after a single saccade, memory of retinotopic locations is more accurate than memory of spatiotopic locations. However, it is not known whether various aspects of natural viewing affect the retinotopic reference frame advantage. We found that the retinotopic advantage may in part depend on a retinal afterimage, which can be effectively nullified through backwards masking. Moreover, in the presence of natural scenes, spatiotopic memory is more accurate than retinotopic memory, but only when subjects are provided sufficient time to process the scene before the eye movement. Our results demonstrate that retinotopic memory is not always more accurate than spatiotopic memory and that the fidelity of memory traces in both reference frames are sensitive to the presence of contextual cues.

A sensory memory to preserve visual representations across eye movements

Saccadic eye movements (saccades) disrupt the continuous flow of visual information, yet our perception of the visual world remains uninterrupted. Here we assess the representation of the visual scene across saccades from single-trial spike trains of extrastriate visual areas, using a combined electrophysiology and statistical modeling approach. Using a model-based decoder we generate a high temporal resolution readout of visual information, and identify the specific changes in neurons' spatiotemporal sensitivity that underly an integrated perisaccadic representation of visual space. Our results show that by maintaining a memory of the visual scene, extrastriate neurons produce an uninterrupted representation of the visual world. Extrastriate neurons exhibit a late response enhancement close to the time of saccade onset, which preserves the latest pre-saccadic information until the post-saccadic flow of retinal information resumes. These results show how our brain exploits available information to maintain a representation of the scene while visual inputs are disrupted.

Transsaccadic integration of visual information is predictive, attention-based, and spatially precise

Eye movements produce shifts in the positions of objects in the retinal image, but observers are able to integrate these shifting retinal images into a coherent representation of visual space. This ability is thought to be mediated by attention-dependent saccade-related neural activity that is used by the visual system to anticipate the retinal consequences of impending eye movements. Previous investigations of the perceptual consequences of this predictive activity typically infer attentional allocation using indirect measures such as accuracy or reaction time. Here, we investigated the perceptual consequences of saccades using an objective measure of attentional allocation, reverse correlation. Human observers executed a saccade while monitoring a flickering target object flanked by flickering distractors and reported whether the average luminance of the target was lighter or darker than the background. Successful task performance required subjects to integrate visual information across the saccade. A reverse correlation analysis yielded a spatiotemporal "psychophysical kernel" characterizing how different parts of the stimulus contributed to the luminance decision throughout each trial. Just before the saccade, observers integrated luminance information from a distractor located at the post-saccadic retinal position of the target, indicating a predictive perceptual updating of the target. Observers did not integrate information from distractors placed in alternative locations, even when they were nearer to the target object. We also observed simultaneous predictive perceptual updating for two spatially distinct targets. These findings suggest both that shifting neural representations mediate the coherent representation of visual space, and that these shifts have significant consequences for transsaccadic perception.

Visual Remapping

Our visual system is fundamentally retinotopic. When viewing a stable scene, each eye movement shifts object features and locations on the retina. Thus, sensory representations must be updated, or remapped, across saccades to align presaccadic and postsaccadic inputs. The earliest remapping studies focused on anticipatory, presaccadic shifts of neuronal spatial receptive fields. Over time, it has become clear that there are multiple forms of remapping and that different forms of remapping may be mediated by different neural mechanisms. This review attempts to organize the various forms of remapping into a functional taxonomy based on experimental data and ongoing debates about forward versus convergent remapping, presaccadic versus postsaccadic remapping, and spatial versus attentional remapping. We integrate findings from primate neurophysiological, human neuroimaging and behavioral, and computational modeling studies. We conclude by discussing persistent open questions related to remapping, with specific attention to binding of spatial and featural information during remapping and speculations about remapping's functional significance. Expected final online publication date for the Annual Review of Vision Science, Volume 7 is September 2021. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

The functional roles of neural remapping in cortex

Remapping is a property of some cortical and subcortical neurons that update their responses around the time of an eye movement to account for the shift of stimuli on the retina due to the saccade. Physiologically, remapping is traditionally tested by briefly presenting a single stimulus around the time of the saccade and looking at the onset of the response and the locations in space to which the neuron is responsive. Here we suggest that a better way to understand the functional role of remapping is to look at the time at which the neural signal emerges when saccades are made across a stable scene. Based on data obtained using this approach, we suggest that remapping in the lateral intraparietal area is sufficient to play a role in maintaining visual stability across saccades, whereas in the frontal eye field, remapped activity carries information that affects future saccadic choices and, in a separate subset of neurons, is used to maintain a map of locations in the scene that have been previously fixated.

Deep Predictive Learning in Neocortex and Pulvinar

How do humans learn from raw sensory experience? Throughout life, but most obviously in infancy, we learn without explicit instruction. We propose a detailed biological mechanism for the widely embraced idea that learning is driven by the differences between predictions and actual outcomes (i.e., predictive error-driven learning). Specifically, numerous weak projections into the pulvinar nucleus of the thalamus generate top-down predictions, and sparse driver inputs from lower areas supply the actual outcome, originating in Layer 5 intrinsic bursting neurons. Thus, the outcome representation is only briefly activated, roughly every 100 msec (i.e., 10 Hz, alpha), resulting in a temporal difference error signal, which drives local synaptic changes throughout the neocortex. This results in a biologically plausible form of error backpropagation learning. We implemented these mechanisms in a large-scale model of the visual system and found that the simulated inferotemporal pathway learns to systematically categorize 3-D objects according to invariant shape properties, based solely on predictive learning from raw visual inputs. These categories match human judgments on the same stimuli and are consistent with neural representations in inferotemporal cortex in primates.

The peripheral sensitivity profile at the saccade target reshapes during saccade preparation

Goal-directed eye movements (saccades) bring peripheral objects of interest into high-acuity foveal vision. In preparation for the incoming foveal image, the perception of the saccade target may sharpen gradually before the eye movement is executed. Indeed, previous studies suggest that pre-saccadic attention shifts enhance sensitivity to high spatial frequencies (SFs) more than sensitivity to lower SFs. This pattern, however, was observed within a narrow frequency range and may reflect local changes in the shape of a broader underlying sensitivity profile. Depending on the development of the profile's shape, SFs above the previously examined range may profit less from saccade preparation. To assess the impact of saccade preparation on the shape of a broader sensitivity profile, we prompted observers to discriminate the orientation of a sinusoidal grating (the probe) presented briefly at the target of an impending saccade, at 10 dva (degree of visual angle) eccentricity. The probe's SF ranged from 1 to 5.5 cycles per dva (cpd) and was unpredictable on a given trial. We fitted observers' response accuracies across SFs with a log-parabolic, that is, inverted U-shaped function. Long before saccade onset, the profile peaked at .6 cpd and dropped off towards lower and higher SFs with broad bandwidth. During saccade preparation, the peak of the profile increased and shifted towards higher SFs while the bandwidth of the profile decreased. As a consequence of this reshaping process, pre-saccadic enhancement increased with SF up to 2.5 cpd, corroborating previous findings. Sensitivities to higher SFs, however, profited less from saccade preparation. We conclude that the extent of pre-saccadic enhancement to a particular SF is governed by its position on a broader sensitivity profile which reshapes substantially during saccade preparation. The shift of the profile's peak towards higher SFs increases resolution at the saccade target even when the features of relevant visual information are unpredictable.

Sounds are remapped across saccades

To achieve visual space constancy, our brain remaps eye-centered projections of visual objects across saccades. Here, we measured saccade trajectory curvature following the presentation of visual, auditory, and audiovisual distractors in a double-step saccade task to investigate if this stability mechanism also accounts for localized sounds. We found that saccade trajectories systematically curved away from the position at which either a light or a sound was presented, suggesting that both modalities are represented in eye-centered oculomotor centers. Importantly, the same effect was observed when the distractor preceded the execution of the first saccade. These results suggest that oculomotor centers keep track of visual, auditory and audiovisual objects by remapping their eye-centered representations across saccades. Furthermore, they argue for the existence of a supra-modal map which keeps track of multi-sensory object locations across our movements to create an impression of space constancy.

Low-Level Visual Information Is Maintained across Saccades, Allowing for a Postsaccadic Handoff between Visual Areas

Experience seems continuous and detailed despite saccadic eye movements changing retinal input several times per second. There is debate whether neural signals related to updating across saccades contain information about stimulus features, or only location pointers without visual details. We investigated the time course of low-level visual information processing across saccades by decoding the spatial frequency of a stationary stimulus that changed from one visual hemifield to the other because of a horizontal saccadic eye movement. We recorded magnetoencephalography while human subjects (both sexes) monitored the orientation of a grating stimulus, making spatial frequency task irrelevant. Separate trials, in which subjects maintained fixation, were used to train a classifier, whose performance was then tested on saccade trials. Decoding performance showed that spatial frequency information of the presaccadic stimulus remained present for ∼200 ms after the saccade, transcending retinotopic specificity. Postsaccadic information ramped up rapidly after saccade offset. There was an overlap of over 100 ms during which decoding was significant from both presaccadic and postsaccadic processing areas. This suggests that the apparent richness of perception across saccades may be supported by the continuous availability of low-level information with a "soft handoff" of information during the initial processing sweep of the new fixation.SIGNIFICANCE STATEMENT Saccades create frequent discontinuities in visual input, yet perception appears stable and continuous. How is this discontinuous input processed resulting in visual stability? Previous studies have focused on presaccadic remapping. Here we examined the time course of processing of low-level visual information (spatial frequency) across saccades with magnetoencephalography. The results suggest that spatial frequency information is not predictively remapped but also is not discarded. Instead, they suggest a soft handoff over time between different visual areas, making this information continuously available across the saccade. Information about the presaccadic stimulus remains available, while the information about the postsaccadic stimulus has also become available. The simultaneous availability of both the presaccadic and postsaccadic information could enable rich and continuous perception across saccades.

Selective Modulation of Early Visual Cortical Activity by Movement Intention

The primate visual system contains myriad feedback projections from higher- to lower-order cortical areas, an architecture that has been implicated in the top-down modulation of early visual areas during working memory and attention. Here we tested the hypothesis that these feedback projections also modulate early visual cortical activity during the planning of visually guided actions. We show, across three separate human functional magnetic resonance imaging (fMRI) studies involving object-directed movements, that information related to the motor effector to be used (i.e., limb, eye) and action goal to be performed (i.e., grasp, reach) can be selectively decoded-prior to movement-from the retinotopic representation of the target object(s) in early visual cortex. We also find that during the planning of sequential actions involving objects in two different spatial locations, that motor-related information can be decoded from both locations in retinotopic cortex. Together, these findings indicate that movement planning selectively modulates early visual cortical activity patterns in an effector-specific, target-centric, and task-dependent manner. These findings offer a neural account of how motor-relevant target features are enhanced during action planning and suggest a possible role for early visual cortex in instituting a sensorimotor estimate of the visual consequences of movement.

Perisaccadic remapping: What? How? Why?

About 25 years ago, the discovery of receptive field (RF) remapping in the parietal cortex of nonhuman primates revealed that visual RFs, widely assumed to have a fixed retinotopic organization, can change position before every saccade. Measuring such changes can be deceptively difficult. As a result, studies that followed have generated a fascinating but somewhat confusing picture of the phenomenon. In this review, we describe how observations of RF remapping depend on the spatial and temporal sampling of visual RFs and saccade directions. Further, we summarize some of the theories of how remapping might occur in neural circuitry. Finally, based on neurophysiological and psychophysical observations, we discuss the ways in which remapping information might facilitate computations in downstream brain areas.

Timing of response onset and offset in macaque V4: stimulus and task dependence

In the primate visual cortex, both the magnitude of the neuronal response and its timing can carry important information about the visual world, but studies typically focus only on response magnitude. Here, we examine the onset and offset latency of the responses of neurons in area V4 of awake, behaving macaques across several experiments in the context of a variety of stimuli and task paradigms. Our results highlight distinct contributions of stimuli and tasks to V4 response latency. We found that response onset latencies are shorter than typically cited (median = 75.5 ms), supporting a role for V4 neurons in rapid object and scene recognition functions. Moreover, onset latencies are longer for smaller stimuli and stimulus outlines, consistent with the hypothesis that longer latencies are associated with higher spatial frequency content. Strikingly, we found that onset latencies showed no significant dependence on stimulus occlusion, unlike in inferotemporal cortex, nor on task demands. Across the V4 population, onset latencies had a broad distribution, reflecting the diversity of feedforward, recurrent, and feedback connections that inform the responses of individual neurons. Response offset latencies, on the other hand, displayed the opposite tendency in their relationship to stimulus and task attributes: they are less influenced by stimulus appearance but are shorter in guided saccade tasks compared with fixation tasks. The observation that response latency is influenced by stimulus- and task-associated factors emphasizes a need to examine response timing alongside firing rate in determining the functional role of area V4.NEW & NOTEWORTHY Onset and offset timing of neuronal responses can provide information about visual environment and neuron's role in visual processing and its anatomical connectivity. In the first comprehensive examination of onset and offset latencies in the intermediate visual cortical area V4, we find neurons respond faster than previously reported, making them ideally suited to contribute to rapid object and scene recognition. While response onset reflects stimulus characteristics, timing of response offset is influenced more by behavioral task.

Eye movements shape visual learning

Most people easily learn to recognize new faces and places, and with more extensive practice they can become experts at visual tasks as complex as radiological diagnosis and action video games. Such perceptual plasticity has been thoroughly studied in the context of training paradigms that require constant fixation. In contrast, when observers learn under more natural conditions, they make frequent saccadic eye movements. Here we show that such eye movements can play an important role in visual learning. Observers performed a task in which they executed a saccade while discriminating the motion of a cued visual stimulus. Additional stimuli, presented simultaneously with the cued one, permitted an assessment of the perceptual integration of information across visual space. Consistent with previous results on perisaccadic remapping [M. Szinte, D. Jonikaitis, M. Rolfs, P. Cavanagh, H. Deubel, J. Neurophysiol. 116, 1592-1602 (2016)], most observers preferentially integrated information from locations representing the presaccadic and postsaccadic retinal positions of the cue. With extensive training on the saccade task, these observers gradually acquired the ability to perform similar motion integration without making eye movements. Importantly, the newly acquired pattern of spatial integration was determined by the metrics of the saccades made during training. These results suggest that oculomotor influences on visual processing, long thought to subserve the function of perceptual stability, also play a role in visual plasticity.

Spatial resolution of local field potential signals in macaque V4

OBJECTIVE

An important challenge for the development of cortical visual prostheses is to generate spatially localized percepts of light, using artificial stimulation. Such percepts are called phosphenes, and the goal of prosthetic applications is to generate a pattern of phosphenes that matches the structure of the retinal image. A preliminary step in this process is to understand how the spatial positions of phosphene-like visual stimuli are encoded in the distributed activity of cortical neurons. The spatial resolution with which the distributed responses discriminate positions puts a limit on the capability of visual prosthesis devices to induce phosphenes at multiple positions. While most previous prosthetic devices have targeted the primary visual cortex, the extrastriate cortex has the advantage of covering a large part of the visual field with a smaller amount of cortical tissue, providing the possibility of a more compact implant. Here, we studied how well ensembles of Local Field Potentials (LFPs) and Multiunit activity (MUA) responses from extrastriate cortical visual area V4 of a behaving macaque monkey can discriminate between two-dimensional spatial positions.

APPROACH

We used support vector machines (SVM) to determine the capabilities of LFPs and MUA to discriminate responses to phosphene-like stimuli (probes) at different spatial separations. We proposed a selection strategy based on the combined responses of multiple electrodes and used the linear learning weights to find the minimum number of electrodes for fine and coarse discriminations. We also measured the contribution of correlated trial-to-trial variability in the responses to the discrimination performance for MUA and LFP.

MAIN RESULTS

We found that despite the large receptive field sizes in V4, the combined responses from multiple sites, whether MUA or LFP, are capable of fine and coarse discrimination of positions. Our electrode selection procedure significantly increased discrimination performance while reducing the required number of electrodes. Analysis of noise correlations in MUA and LFP responses showed that noise correlations in LFPs carry more information about spatial positions.

SIGNIFICANCE

This study determined the coding strategy for fine discrimination, suggesting that spatial positions could be well localized with patterned stimulation in extrastriate area V4. It also provides a novel approach to build a compact prosthesis with relatively few electrodes, which has the potential advantage of reducing tissue damage in real applications.

Modeling the influence of the hippocampal memory system on the oculomotor system

Visual exploration is related to activity in the hippocampus (HC) and/or extended medial temporal lobe system (MTL), is influenced by stored memories, and is altered in amnesic cases. An extensive set of polysynaptic connections exists both within and between the HC and oculomotor systems such that investigating how HC responses ultimately influence neural activity in the oculomotor system, and the timing by which such neural modulation could occur, is not trivial. We leveraged TheVirtualBrain, a software platform for large-scale network simulations, to model the functional dynamics that govern the interactions between the two systems in the macaque cortex. Evoked responses following the stimulation of the MTL and some, but not all, subfields of the HC resulted in observable responses in oculomotor regions, including the frontal eye fields, within the time of a gaze fixation. Modeled lesions to some MTL regions slowed the dissipation of HC signal to oculomotor regions, whereas HC lesions generally did not affect the rapid MTL activity propagation to oculomotor regions. These findings provide a framework for investigating how information represented by the HC/MTL may influence the oculomotor system during a fixation and predict how HC lesions may affect visual exploration.

The intersection between the oculomotor and hippocampal memory systems: empirical developments and clinical implications

Decades of cognitive neuroscience research has shown that where we look is intimately connected to what we remember. In this article, we review findings from human and nonhuman animals, using behavioral, neuropsychological, neuroimaging, and computational modeling methods, to show that the oculomotor and hippocampal memory systems interact in a reciprocal manner, on a moment-to-moment basis, mediated by a vast structural and functional network. Visual exploration serves to efficiently gather information from the environment for the purpose of creating new memories, updating existing memories, and reconstructing the rich, vivid details from memory. Conversely, memory increases the efficiency of visual exploration. We call for models of oculomotor control to consider the influence of the hippocampal memory system on the cognitive control of eye movements, and for models of hippocampal and broader medial temporal lobe function to consider the influence of the oculomotor system on the development and expression of memory. We describe eye movement-based applications for the detection of neurodegeneration and delivery of therapeutic interventions for mental health disorders for which the hippocampus is implicated and memory dysfunctions are at the forefront.

Characterizing and dissociating multiple time-varying modulatory computations influencing neuronal activity

In many brain areas, sensory responses are heavily modulated by factors including attentional state, context, reward history, motor preparation, learned associations, and other cognitive variables. Modelling the effect of these modulatory factors on sensory responses has proven challenging, mostly due to the time-varying and nonlinear nature of the underlying computations. Here we present a computational model capable of capturing and dissociating multiple time-varying modulatory effects on neuronal responses on the order of milliseconds. The model’s performance is tested on extrastriate perisaccadic visual responses in nonhuman primates. Visual neurons respond to stimuli presented around the time of saccades differently than during fixation. These perisaccadic changes include sensitivity to the stimuli presented at locations outside the neuron’s receptive field, which suggests a contribution of multiple sources to perisaccadic response generation. Current computational approaches cannot quantitatively characterize the contribution of each modulatory source in response generation, mainly due to the very short timescale on which the saccade takes place. In this study, we use a high spatiotemporal resolution experimental paradigm along with a novel extension of the generalized linear model framework (GLM), termed the sparse-variable GLM, to allow for time-varying model parameters representing the temporal evolution of the system with a resolution on the order of milliseconds. We used this model framework to precisely map the temporal evolution of the spatiotemporal receptive field of visual neurons in the middle temporal area during the execution of a saccade. Moreover, an extended model based on a factorization of the sparse-variable GLM allowed us to disassociate and quantify the contribution of individual sources to the perisaccadic response. Our results show that our novel framework can precisely capture the changes in sensitivity of neurons around the time of saccades, and provide a general framework to quantitatively track the role of multiple modulatory sources over time.

The sensory responses of neurons in many brain areas, particularly those in higher prefrontal or parietal areas, are strongly influenced by factors including task rules, attentional state, context, reward history, motor preparation, learned associations, and other cognitive variables. These modulations often occur in combination, or on fast timescales which present a challenge for both experimental and modelling approaches aiming to describe the underlying mechanisms or computations. Here we present a computational model capable of capturing and dissociating multiple time-varying modulatory effects on spiking responses on the order of milliseconds. The model’s performance is evaluated by testing its ability to reproduce and dissociate multiple changes in visual sensitivity occurring in extrastriate visual cortex around the time of rapid eye movements. No previous model is capable of capturing these changes with as fine a resolution as that presented here. Our model both provides specific insight into the nature and time course of changes in visual sensitivity around the time of eye movements, and offers a general framework applicable to a wide variety of contexts in which sensory processing is modulated dynamically by multiple time-varying cognitive or behavioral factors, to understand the neuronal computations underpinning these modulations and make predictions about the underlying mechanisms.

Transsaccadic integration benefits are not limited to the saccade target

Across saccades, humans can integrate the low-resolution presaccadic information of an upcoming saccade target with the high-resolution postsaccadic information. There is converging evidence to suggest that transsaccadic integration occurs at the saccade target. However, given divergent evidence on the spatial specificity of related mechanisms such as attention, visual working memory, and remapping, it is unclear whether integration is also possible at locations other than the saccade target. We tested the spatial profile of transsaccadic integration, by testing perceptual performance at six locations around the saccade target and between the saccade target and initial fixation. Results show that integration benefits do not differ between the saccade target and surrounding locations. Transsaccadic integration benefits are not specific to the saccade target and can occur at other locations when they are behaviorally relevant, although there is a trend for worse performance for the location above initial fixation compared with those in the direction of the saccade. This suggests that transsaccadic integration may be a more general mechanism used to reconcile task-relevant pre- and postsaccadic information at attended locations other than the saccade target.

NEW & NOTEWORTHY This study shows that integration of pre- and postsaccadic information across saccades is not restricted to the saccade target. We found performance benefits of transsaccadic integration at attended locations other than the saccade target, and these benefits did not differ from those found at the saccade target. This suggests that transsaccadic integration may be a more general mechanism used to reconcile pre- and postsaccadic information at task-relevant locations.

Saccades Trigger Predictive Updating of Attentional Topography in Area V4

During natural behavior, saccades and attention act together to allocate limited neural resources. Attention is generally mediated by retinotopic visual neurons; therefore, specific neurons representing attended features change with each saccade. We investigated the neural mechanisms that allow attentional targeting in the face of saccades. Specifically, we looked for predictive changes in attentional modulation state or receptive field position that could stabilize attentional representations across saccades in area V4, known to be necessary for attention-dependent behavior. We recorded from neurons in monkeys performing a novel spatiotopic attention task, in which performance depended on accurate saccade compensation. Measurements of attentional modulation revealed a predictive attentional "hand-off" corresponding to a presaccadic transfer of attentional state from neurons inside the attentional focus before the saccade to those that will be inside the focus after the saccade. The predictive nature of the hand-off ensures that attentional brain maps are properly configured immediately after each saccade.

Corollary Discharge Contributions to Perceptual Continuity Across Saccades

Our vision depends upon shifting our high-resolution fovea to objects of interest in the visual field. Each saccade displaces the image on the retina, which should produce a chaotic scene with jerks occurring several times per second. It does not. This review examines how an internal signal in the primate brain (a corollary discharge) contributes to visual continuity across saccades. The article begins with a review of evidence for a corollary discharge in the monkey and evidence from inactivation experiments that it contributes to perception. The next section examines a specific neuronal mechanism for visual continuity, based on corollary discharge that is referred to as visual remapping. Both the basic characteristics of this anticipatory remapping and the factors that control it are enumerated. The last section considers hypotheses relating remapping to the perceived visual continuity across saccades, including remapping's contribution to perceived visual stability across saccades.

Corollary Discharge for Action and Cognition

In motor systems, a copy of the movement command known as corollary discharge is broadcast to other regions of the brain to warn them of the impending movement. The premise of this review is that the concept of corollary discharge may generalize in revealing ways to the brain's cognitive systems. An oculomotor pathway from the brain stem to frontal cortex provides a well-established example of how corollary discharge is instantiated for sensorimotor processing. Building on causal evidence from inactivation of the pathway, we motivate forward models as a tool for understanding the contributions of corollary discharge to perception and movement. Finally, we extend the definition of corollary discharge to account for signals that may be used for cognitive forward models of decision making. This framework may provide new insights into signals and circuits that contribute to sequential decision processes, the breakdown of which may account for some symptoms of psychiatric disorders.

The Limits of Predictive Remapping of Attention Across Eye Movements

With every eye movement, visual input projected onto our retina changes drastically. The fundamental question of how we keep track of relevant objects and movement targets has puzzled scientists for more than a century. Recent advances suggested that this can be accomplished through the process of predictive remapping of visual attention to the future post-saccadic locations of relevant objects. Evidence for the existence of predictive remapping of attention was first provided by Rolfs et al. (2011) (Nature Neuroscience, 14, 252–256). However, they used a single distant control location away from the task-relevant locations, which could have biased the allocation of visual attention. In this study we used a similar experimental paradigm as Rolfs et al. (2011), but probed attention equally likely at all possible locations. Our results showed that discrimination performance was higher at the remapped location than at a distant control location, but not compared to the other two control locations. A re-analysis of the results obtained by Rolfs et al. (2011) revealed a similar pattern. Together, these findings suggest that it is likely that previous reports of the predictive remapping of attention were due to a diffuse spread of attention to the task-relevant locations rather than to a specific shift toward the target’s future retinotopic location.

Predictive remapping of visual features beyond saccadic targets

Visual stability is thought to be mediated by predictive remapping of the relevant object information from its current, presaccadic location to its future, postsaccadic location on the retina. However, it is heavily debated whether and what feature information is predictively remapped during the presaccadic interval. Here we examined the spatial and featural properties of predictive remapping in a set of three psychophysical studies. We made use of an orientation-adaptation paradigm, in which we induced a tilt aftereffect by prolonged exposure to an oriented adaptor stimulus. Following this adaptation phase, a test stimulus was presented shortly before saccade onset. We found strong evidence for predictive remapping of the features of this test stimulus presented shortly before saccade onset, evidenced by a large tilt aftereffect elicited when the adaptor was positioned at the postsaccadic retinal location of the test stimulus. Conversely, the adaptation state itself, caused by the exposure to the adaptor stimulus, was not predictively remapped. Furthermore, we establish that predictive remapping also occurs for stimuli that are not saccade targets, pointing toward a forward remapping process operating across the whole visual field. Together, our findings suggest that predictive feature remapping of object information plays an important role in mediating visual stability.

Remapping locations and features across saccades: a dual-spotlight theory of attentional updating

How do we maintain visual stability across eye movements? Much work has focused on how visual information is rapidly updated to maintain spatiotopic representations. However, predictive spatial remapping is only part of the story. Here I review key findings, recent debates, and open questions regarding remapping and its implications for visual attention and perception. This review focuses on two key questions: when does remapping occur, and what is the impact on feature perception? Findings are reviewed within the framework of a two-stage, or dual- spotlight, remapping process, where spatial attention must be both updated to the new location (fast, predictive stage) and withdrawn from the previous retinotopic location (slow, post-saccadic stage), with a particular focus on the link between spatial and feature information across eye movements.

Eye Movement Compensation and Spatial Updating in Visual Prosthetics: Mechanisms, Limitations and Future Directions

Despite appearing automatic and effortless, perceiving the visual world is a highly complex process that depends on intact visual and oculomotor function. Understanding the mechanisms underlying spatial updating (i.e., gaze contingency) represents an important, yet unresolved issue in the fields of visual perception and cognitive neuroscience. Many questions regarding the processes involved in updating visual information as a function of the movements of the eyes are still open for research. Beyond its importance for basic research, gaze contingency represents a challenge for visual prosthetics as well. While most artificial vision studies acknowledge its importance in providing accurate visual percepts to the blind implanted patients, the majority of the current devices do not compensate for gaze position. To-date, artificial percepts to the blind population have been provided either by intraocular light-sensing circuitry or by using external cameras. While the former commonly accounts for gaze shifts, the latter requires the use of eye-tracking or similar technology in order to deliver percepts based on gaze position. Inspired by the need to overcome the hurdle of gaze contingency in artificial vision, we aim to provide a thorough overview of the research addressing the neural underpinnings of eye compensation, as well as its relevance in visual prosthetics. The present review outlines what is currently known about the mechanisms underlying spatial updating and reviews the attempts of current visual prosthetic devices to overcome the hurdle of gaze contingency. We discuss the limitations of the current devices and highlight the need to use eye-tracking methodology in order to introduce gaze-contingent information to visual prosthetics.

Time course of spatiotopic updating across saccades

Humans move their eyes several times per second, yet we perceive the outside world as continuous despite the sudden disruptions created by each eye movement. To date, the mechanism that the brain employs to achieve visual continuity across eye movements remains unclear. While it has been proposed that the oculomotor system quickly updates and informs the visual system about the upcoming eye movement, behavioral studies investigating the time course of this updating suggest the involvement of a slow mechanism, estimated to take more than 500 ms to operate effectively. This is a surprisingly slow estimate, because both the visual system and the oculomotor system process information faster. If spatiotopic updating is indeed this slow, it cannot contribute to perceptual continuity, because it is outside the temporal regime of typical oculomotor behavior. Here, we argue that the behavioral paradigms that have been used previously are suboptimal to measure the speed of spatiotopic updating. In this study, we used a fast gaze-contingent paradigm, using high phi as a continuous stimulus across eye movements. We observed fast spatiotopic updating within 150 ms after stimulus onset. The results suggest the involvement of a fast updating mechanism that predictively influences visual perception after an eye movement. The temporal characteristics of this mechanism are compatible with the rate at which saccadic eye movements are typically observed in natural viewing.

Pre-saccadic remapping relies on dynamics of spatial attention

Each saccade shifts the projections of the visual scene on the retina. It has been proposed that the receptive fields of neurons in oculomotor areas are predictively remapped to account for these shifts. While remapping of the whole visual scene seems prohibitively complex, selection by attention may limit these processes to a subset of attended locations. Because attentional selection consumes time, remapping of attended locations should evolve in time, too. In our study, we cued a spatial location by presenting an attention-capturing cue at different times before a saccade and constructed maps of attentional allocation across the visual field. We observed no remapping of attention when the cue appeared shortly before saccade. In contrast, when the cue appeared sufficiently early before saccade, attentional resources were reallocated precisely to the remapped location. Our results show that pre-saccadic remapping takes time to develop suggesting that it relies on the spatial and temporal dynamics of spatial attention.

Perisaccadic perceptual mislocalization is different for upward saccades

Saccadic eye movements, which dramatically alter retinal images, are associated with robust perimovement perceptual alterations. Such alterations, thought to reflect brain mechanisms for maintaining perceptual stability in the face of saccade-induced retinal image disruptions, are often studied by asking subjects to localize brief stimuli presented around the time of horizontal saccades. However, other saccade directions are not usually explored. Motivated by recently discovered asymmetries in upper and lower visual field representations in the superior colliculus, a structure important for both saccade generation and visual analysis, we observed significant differences in perisaccadic perceptual alterations for upward saccades relative to other saccade directions. We also found that, even for purely horizontal saccades, perceptual alterations differ for upper vs. lower retinotopic stimulus locations. Our results, coupled with conceptual modeling, suggest that perisaccadic perceptual alterations might critically depend on neural circuits, such as superior colliculus, that asymmetrically represent the upper and lower visual fields. NEW & NOTEWORTHY Brief visual stimuli are robustly mislocalized around the time of saccades. Such mislocalization is thought to arise because oculomotor and visual neural maps distort space through foveal magnification. However, other neural asymmetries, such as upper visual field magnification in the superior colliculus, may also exist, raising the possibility that interactions between saccades and visual stimuli would depend on saccade direction. We confirmed this behaviorally by exploring and characterizing perisaccadic perception for upward saccades.

Cognitive Functions and Neurodevelopmental Disorders Involving the Prefrontal Cortex and Mediodorsal Thalamus

The mediodorsal nucleus of the thalamus (MD) has been implicated in executive functions (such as planning, cognitive control, working memory, and decision-making) because of its significant interconnectivity with the prefrontal cortex (PFC). Yet, whilst the roles of the PFC have been extensively studied, how the MD contributes to these cognitive functions remains relatively unclear. Recently, causal evidence in monkeys has demonstrated that in everyday tasks involving rapid updating (e.g., while learning something new, making decisions, or planning the next move), the MD and frontal cortex are working in close partnership. Furthermore, researchers studying the MD in rodents have been able to probe the underlying mechanisms of this relationship to give greater insights into how the frontal cortex and MD might interact during the performance of these essential tasks. This review summarizes the circuitry and known neuromodulators of the MD, and considers the most recent behavioral, cognitive, and neurophysiological studies conducted in monkeys and rodents; in total, this evidence demonstrates that MD makes a critical contribution to cognitive functions. We propose that communication occurs between the MD and the frontal cortex in an ongoing, fluid manner during rapid cognitive operations, via the means of efference copies of messages passed through transthalamic routes; the conductance of these messages may be modulated by other brain structures interconnected to the MD. This is similar to the way in which other thalamic structures have been suggested to carry out forward modeling associated with rapid motor responding and visual processing. Given this, and the marked thalamic pathophysiology now identified in many neuropsychiatric disorders, we suggest that changes in the different subdivisions of the MD and their interconnections with the cortex could plausibly give rise to a number of the otherwise disparate symptoms (including changes to olfaction and cognitive functioning) that are associated with many different neuropsychiatric disorders. In particular, we will focus here on the cognitive symptoms of schizophrenia and suggest testable hypotheses about how changes to MD-frontal cortex interactions may affect cognitive processes in this disorder.

Developing a Nonstationary Computational Framework With Application to Modeling Dynamic Modulations in Neural Spiking Responses

OBJECTIVE

This paper aims to develop a computational model that incorporates the functional effects of modulatory covariates (such as context, task, or behavior), which dynamically alter the relationship between the stimulus and the neural response.

METHODS

We develop a general computational approach along with an efficient estimation procedure in the widely used generalized linear model (GLM) framework to characterize such nonstationary dynamics in spiking response and spatiotemporal characteristics of a neuron at the level of individual trials. The model employs a set of modulatory components, which nonlinearly interact with other stimulus-related signals to reproduce such nonstationary effects.

RESULTS

The model is tested for its ability to predict the responses of neurons in the middle temporal cortex of macaque monkeys during an eye movement task. The fitted model proves successful in capturing the fast temporal modulations in the response, reproducing the spike response temporal statistics, and accurately accounting for the neurons' dynamic spatiotemporal sensitivities, during eye movements.

CONCLUSION

The nonstationary GLM framework developed in this study can be used in cases where a time-varying behavioral or cognitive component makes GLM-based models insufficient to describe the dependencies of neural responses on the stimulus-related covariates.

SIGNIFICANCE

In addition to being quite powerful in encoding time-varying response modulations, this general framework also enables a readout of the neural code while dissociating the influence of other nonstimulus covariates. This framework will advance our ability to understand sensory processing in higher brain areas when modulated by several behavioral or cognitive variables.