Analysis of perisaccadic field potentials in the occipitotemporal pathway during active vision

Correlations between Visual Temporal Resolution and Individual Alpha Peak Frequency: Evidence that Internal and Measurement Noise Drive Null Findings

The brain organizes the continuous flow of sensory input by parsing it into discrete events. In the case of two flashes separated by a brief ISI, for example, perception may be of a single flash or two distinct flashes, depending on the ISI but also on the speed of processing. A number of studies have reported evidence that participants with a higher EEG peak alpha frequency are able to detect the presence of two flashes separated by short intervals, whereas those with slower alpha report only one flash. Other studies have not found this correlation. We investigated potential factors that might mask the relationship between individual alpha frequency and visual perception. We recorded resting-state EEG from a large sample of participants (n = 50) and measured the temporal resolution of visual perception with the two-flash fusion task. We found that individual alpha frequency over posterior channels predicted the two-flash fusion threshold, in line with previous studies, but this correlation was significant only when taking into account the steepness of the psychophysical curve of the two-flash task. Participants with a relatively shallow psychophysical curve, likely reflecting high sensory and/or decision noise, failed to show this relationship. These findings replicate previous reports of a correlation between alpha frequency and visual temporal resolution, while also suggesting that an explanation of two-flash fusion performance that neglects the role of internal noise might be insufficient to account for all individual differences.

Cortico-ocular Coupling in the Service of Episodic Memory Formation

Encoding of visual information is a necessary requirement for most types of episodic memories. In search for a neural signature of memory formation, amplitude modulation of neural activity has been repeatedly shown to correlate with and suggested to be functionally involved in successful memory encoding. We here report a complementary view on why and how brain activity relates to memory, indicating a functional role of cortico-ocular interactions for episodic memory formation. Recording simultaneous magnetoencephalography and eye tracking in 35 human participants, we demonstrate that gaze variability and amplitude modulations of alpha/beta oscillations (10-20Hz) in visual cortex covary and predict subsequent memory performance between and within participants. Amplitude variation during pre-stimulus baseline was associated with gaze direction variability, echoing the co-variation observed during scene encoding. We conclude that encoding of visual information engages unison coupling between oculomotor and visual areas in the service of memory formation.

Saccadic modulation of neural excitability in auditory areas of the neocortex

In natural "active" vision, humans and other primates use eye movements (saccades) to sample bits of information from visual scenes. In the visual cortex, non-retinal signals linked to saccades shift visual cortical neurons into a high excitability state as each saccade ends. The extent of this saccadic modulation outside of the visual system is unknown. Here, we show that during natural viewing, saccades modulate excitability in numerous auditory cortical areas with a temporal pattern complementary to that seen in visual areas. Control somatosensory cortical recordings indicate that the temporal pattern is unique to auditory areas. Bidirectional functional connectivity patterns suggest that these effects may arise from regions involved in saccade generation. We propose that by using saccadic signals to yoke excitability states in auditory areas to those in visual areas, the brain can improve information processing in complex natural settings.

A corollary discharge mediates saccade-related inhibition of single units in mnemonic structures of the human brain

Despite the critical link between visual exploration and memory, little is known about how neuronal activity in the human mesial temporal lobe (MTL) is modulated by saccades. Here, we characterize saccade-associated neuronal modulations, unit-by-unit, and contrast them to image onset and to occipital lobe neurons. We reveal evidence for a corollary discharge (CD)-like modulatory signal that accompanies saccades, inhibiting/exciting a unique population of broad-/narrow-spiking units, respectively, before and during saccades and with directional selectivity. These findings comport well with the timing, directional nature, and inhibitory circuit implementation of a CD. Additionally, by linking neuronal activity to event-related potentials (ERPs), which are directionally modulated following saccades, we recontextualize the ERP associated with saccades as a proxy for both the strength of inhibition and saccade direction, providing a mechanistic underpinning for the more commonly recorded saccade-related ERP in the human brain.

Perceptual enhancement and suppression correlate with V1 neural activity during active sensing

Perception in multiple sensory modalities is an active process that involves exploratory behaviors. In humans and other primates, vision results from sensory sampling guided by saccadic eye movements. Saccades are known to modulate visual perception, and a corollary discharge signal associated with saccades appears to establish a sense of visual stability. Neural recordings have shown that saccades also modulate activity widely across the brain. To investigate the neural basis of saccadic effects on perception, simultaneous recordings from multiple neurons in area V1 were made as animals performed a contrast detection task. Perceptual and neural measures were compared when the animal made real saccades that brought a stimulus into V1 receptive fields and when simulated saccades were made (identical retinal stimulation but no eye movement). When real saccades were made and low spatial frequency stimuli were presented, we observed a reduction in both perceptual sensitivity and neural activity compared with simulated saccades; conversely, with higher spatial frequency stimuli, saccades increased visual sensitivity and neural activity. The performance of neural decoders, which used the activity of the population of simultaneously recorded neurons, showed saccade effects on sensitivity that mirrored the frequency-dependent perceptual changes, suggesting that the V1 population activity could support the perceptual effects. A minority of V1 neurons had significant choice probabilities, and the saccades decreased both average choice probability and pairwise noise correlations. Taken together, the findings suggest that a signal related to saccadic eye movements alters V1 spiking to increase the independence of spiking neurons and bias the system toward processing higher spatial frequencies, presumably to enhance object recognition. The effects of saccades on visual perception and noise correlations appear to parallel effects observed in other sensory modalities, suggesting a general principle of active sensory processing.

Saccade-related neural communication in the human medial temporal lobe is modulated by the social relevance of stimuli

Humans predominantly explore their environment by moving their eyes. To optimally communicate and process visual information, neural activity needs to be coordinated with the execution of eye movements. We investigated the coordination between visual exploration and interareal neural communication by analyzing local field potentials and single neuron activity in patients with epilepsy. We demonstrated that during the free viewing of images, neural communication between the human amygdala and hippocampus is coordinated with the execution of eye movements. The strength and direction of neural communication and hippocampal saccade-related phase alignment were strongest for fixations that landed on human faces. Our results argue that the state of the human medial temporal lobe network is selectively coordinated with motor behavior. Interareal neural communication was facilitated for social stimuli as indexed by the category of the attended information.

Selective activation of central thalamic fiber pathway facilitates behavioral performance in healthy non-human primates

Central thalamic deep brain stimulation (CT-DBS) is an investigational therapy to treat enduring cognitive dysfunctions in structurally brain injured (SBI) patients. However, the mechanisms of CT-DBS that promote restoration of cognitive functions are unknown, and the heterogeneous etiology and recovery profiles of SBI patients contribute to variable outcomes when using conventional DBS strategies,which may result in off-target effects due to activation of multiple pathways. To disambiguate the effects of stimulation of two adjacent thalamic pathways, we modeled and experimentally compared conventional and novel 'field-shaping' methods of CT-DBS within the central thalamus of healthy non-human primates (NHP) as they performed visuomotor tasks. We show that selective activation of the medial dorsal thalamic tegmental tract (DTTm), but not of the adjacent centromedian-parafascicularis (CM-Pf) pathway, results in robust behavioral facilitation. Our predictive modeling approach in healthy NHPs directly informs ongoing and future clinical investigations of conventional and novel methods of CT-DBS for treating cognitive dysfunctions in SBI patients, for whom no therapy currently exists.

Fast and nonuniform dynamics of perisaccadic vision in the central fovea

Humans shift their gaze more frequently than their heart beats. These rapid eye movements (saccades) enable high visual acuity by redirecting the tiny high-resolution region of the retina (the foveola). But in doing so, they abruptly sweep the image across receptors, raising questions on how the visual system achieves stable percepts. It is well established that visual sensitivity is transiently attenuated during saccades. However, little is known about the time course of foveal vision despite its disproportionate importance, as technical challenges have so far prevented study of how saccades affect the foveola. Here we show that saccades modulate this region in a nonuniform manner, providing stronger and faster changes at its very center, a locus with higher sensitivity.

Humans use rapid eye movements (saccades) to inspect stimuli with the foveola, the region of the retina where receptors are most densely packed. It is well established that visual sensitivity is generally attenuated during these movements, a phenomenon known as saccadic suppression. This effect is commonly studied with large, often peripheral, stimuli presented during instructed saccades. However, little is known about how saccades modulate the foveola and how the resulting dynamics unfold during natural visual exploration. Here we measured the foveal dynamics of saccadic suppression in a naturalistic high-acuity task, a task designed after primates’ social grooming, which—like most explorations of fine patterns—primarily elicits minute saccades (microsaccades). Leveraging on recent advances in gaze-contingent display control, we were able to systematically map the perisaccadic time course of sensitivity across the foveola. We show that contrast sensitivity is not uniform across this region and that both the extent and dynamics of saccadic suppression vary within the foveola. Suppression is stronger and faster in the most central portion, where sensitivity is generally higher and selectively rebounds at the onset of a new fixation. These results shed light on the modulations experienced by foveal vision during the saccade-fixation cycle and explain some of the benefits of microsaccades.

Neural activity in the human anterior thalamus during natural vision

In natural vision humans and other primates explore environment by active sensing, using saccadic eye movements to relocate the fovea and sample different bits of information multiple times per second. Saccades induce a phase reset of ongoing neuronal oscillations in primary and higher-order visual cortices and in the medial temporal lobe. As a result, neuron ensembles are shifted to a common state at the time visual input propagates through the system (i.e., just after fixation). The extent of the brain’s circuitry that is modulated by saccades is not yet known. Here, we evaluate the possibility that saccadic phase reset impacts the anterior nuclei of the thalamus (ANT). Using recordings in the human thalamus of three surgical patients during natural vision, we found that saccades and visual stimulus onset both modulate neural activity, but with distinct field potential morphologies. Specifically, we found that fixation-locked field potentials had a component that preceded saccade onset. It was followed by an early negativity around 50 ms after fixation onset which is significantly faster than any response to visual stimulus presentation. The timing of these events suggests that the ANT is predictively modulated before the saccadic eye movement. We also found oscillatory phase concentration, peaking at 3–4 Hz, coincident with suppression of Broadband High-frequency Activity (BHA; 80–180 Hz), both locked to fixation onset supporting the idea that neural oscillations in these nuclei are reorganized to a low excitability state right after fixation onset. These findings show that during real-world natural visual exploration neural dynamics in the human ANT is influenced by visual and oculomotor events, which supports the idea that ANT, apart from their contribution to episodic memory, also play a role in natural vision.

The extrafoveal preview paradigm as a measure of predictive, active sampling in visual perception

A key feature of visual processing in humans is the use of saccadic eye movements to look around the environment. Saccades are typically used to bring relevant information, which is glimpsed with extrafoveal vision, into the high-resolution fovea for further processing. With the exception of some unusual circumstances, such as the first fixation when walking into a room, our saccades are mainly guided based on this extrafoveal preview. In contrast, the majority of experimental studies in vision science have investigated "passive" behavioral and neural responses to suddenly appearing and often temporally or spatially unpredictable stimuli. As reviewed here, a growing number of studies have investigated visual processing of objects under more natural viewing conditions in which observers move their eyes to a stationary stimulus, visible previously in extrafoveal vision, during each trial. These studies demonstrate that the extrafoveal preview has a profound influence on visual processing of objects, both for behavior and neural activity. Starting from the preview effect in reading research we follow subsequent developments in vision research more generally and finally argue that taking such evidence seriously leads to a reconceptualization of the nature of human visual perception that incorporates the strong influence of prediction and action on sensory processing. We review theoretical perspectives on visual perception under naturalistic viewing conditions, including theories of active vision, active sensing, and sampling. Although the extrafoveal preview paradigm has already provided useful information about the timing of, and potential mechanisms for, the close interaction of the oculomotor and visual systems while reading and in natural scenes, the findings thus far also raise many new questions for future research.

Predicting task performance from biomarkers of mental fatigue in global brain activity

Objective.

Detection and early prediction of mental fatigue (i.e. shifts in vigilance), could be used to adapt neuromodulation strategies to effectively treat patients suffering from brain injury and other indications with prominent chronic mental fatigue.

Approach.

In this study, we analyzed electrocorticography (ECoG) signals chronically recorded from two healthy non-human primates (NHP) as they performed a sustained attention task over extended periods of time. We employed a set of spectrotemporal and connectivity biomarkers of the ECoG signals to identify periods of mental fatigue and a gradient boosting classifier to predict performance, up to several seconds prior to the behavioral response.

Main results.

Wavelet entropy and the instantaneous amplitude and frequency were among the best single features across sessions in both NHPs. The classification performance using higher order spectral-temporal (HOST) features was significantly higher than that of conventional spectral power features in both NHPs. Across the 99 sessions analyzed, average F1 scores of 77.5%±8.2% and 91.2%±3.6%, and accuracy of 79.5%±8.9% and 87.6%±3.9 % for the classifier were obtained for each animal, respectively.

Significance.

Our results here demonstrate the feasibility of predicting performance and detecting periods of mental fatigue by analyzing ECoG signals, and that this general approach, in principle, could be used for closed-loop control of neuromodulation strategies.

Restricting Visual Exploration Directly Impedes Neural Activity, Functional Connectivity, and Memory

We move our eyes to explore the visual world, extract information, and create memories. The number of gaze fixations-the stops that the eyes make-has been shown to correlate with activity in the hippocampus, a region critical for memory, and with later recognition memory. Here, we combined eyetracking with fMRI to provide direct evidence for the relationships between gaze fixations, neural activity, and memory during scene viewing. Compared to free viewing, fixating a single location reduced: 1) subsequent memory, 2) neural activity along the ventral visual stream into the hippocampus, 3) neural similarity between effects of subsequent memory and visual exploration, and 4) functional connectivity among the hippocampus, parahippocampal place area, and other cortical regions. Gaze fixations were uniquely related to hippocampal activity, even after controlling for neural effects due to subsequent memory. Therefore, this study provides key causal evidence supporting the notion that the oculomotor and memory systems are intrinsically related at both the behavioral and neural level. Individual gaze fixations may provide the basic unit of information on which memory binding processes operate.

Generative models, linguistic communication and active inference

This paper presents a biologically plausible generative model and inference scheme that is capable of simulating communication between synthetic subjects who talk to each other. Building on active inference formulations of dyadic interactions, we simulate linguistic exchange to explore generative models that support dialogues. These models employ high-order interactions among abstract (discrete) states in deep (hierarchical) models. The sequential nature of language processing mandates generative models with a particular factorial structure-necessary to accommodate the rich combinatorics of language. We illustrate linguistic communication by simulating a synthetic subject who can play the 'Twenty Questions' game. In this game, synthetic subjects take the role of the questioner or answerer, using the same generative model. This simulation setup is used to illustrate some key architectural points and demonstrate that many behavioural and neurophysiological correlates of linguistic communication emerge under variational (marginal) message passing, given the right kind of generative model. For example, we show that theta-gamma coupling is an emergent property of belief updating, when listening to another.

Spatiotemporal Content of Saccade Transients

Humans use rapid gaze shifts, known as saccades, to explore visual scenes. These movements yield abrupt luminance changes on the retina, which elicit robust neural discharges at fixation onsets. Yet little is known about the spatial content of saccade transients. Here, we show that saccades redistribute spatial information within the temporal range of retinal sensitivity following two distinct regimes: saccade modulations counterbalance (whiten) the spectral density of natural scenes at low spatial frequencies and follow the external power distribution at higher frequencies. This redistribution is a consequence of saccade dynamics, particularly the speed/amplitude/duration relation known as the main sequence. It resembles the redistribution resulting from inter-saccadic eye drifts, revealing a continuum in the modulations given by different eye movements, with oculomotor transitions primarily acting by regulating the bandwidth of whitening. Our findings suggest important computational roles for saccade transients in the establishment of spatial representations and lead to testable predictions about their consequences for visual functions and encoding mechanisms.

Dynamic Modulation of Cortical Excitability during Visual Active Sensing

Visual physiology is traditionally investigated by presenting stimuli with gaze held constant. However, during active viewing of a scene, information is actively acquired using systematic patterns of fixations and saccades. Prior studies suggest that during such active viewing, both nonretinal, saccade-related signals and “extra-classical” receptive field inputs modulate visual processing. This study used a set of active viewing tasks that allowed us to compare visual responses with and without direct foveal input, thus isolating the contextual eye movement-related influences. Studying nonhuman primates, we find strong contextual modulation in primary visual cortex (V1): excitability and response amplification immediately after fixation onset, transiting to suppression leading up to the next saccade. Time-frequency decomposition suggests that this amplification and suppression cycle stems from a phase reset of ongoing neuronal oscillatory activity. The impact of saccade-related contextual modulation on stimulus processing makes active visual sensing fundamentally different from the more passive processes investigated in traditional paradigms.

By isolating contextual eye movement-related influences during active vision, Barczak et al. show that eye movements affect excitability in V1 such that responses are amplified immediately after fixation onset and suppressed as the next saccade approaches. This amplification and suppression cycle stems from a phase reset of ambient oscillatory activity in V1.

Differential Generation of Saccade, Fixation, and Image-Onset Event-Related Potentials in the Human Mesial Temporal Lobe

Event-related potentials (ERPs) are a commonly used electrophysiological signature for studying mesial temporal lobe (MTL) function during visual memory tasks. The ERPs associated with the onset of visual stimuli (image-onset) and eye movements (saccades and fixations) provide insights into the mechanisms of their generation. We hypothesized that since eye movements and image-onset provide MTL structures with salient visual information, perhaps they both engage similar neural mechanisms. To explore this question, we used intracranial electroencephalographic data from the MTLs of 11 patients with medically refractory epilepsy who participated in a visual search task. We characterized the electrophysiological responses of MTL structures to saccades, fixations, and image-onset. We demonstrated that the image-onset response is an evoked/additive response with a low-frequency power increase. In contrast, ERPs following eye movements appeared to arise from phase resetting of higher frequencies than the image-onset ERP. Intriguingly, this reset was associated with saccade onset and not termination (fixation), suggesting it is likely the MTL response to a corollary discharge, rather than a response to visual stimulation. We discuss the distinct mechanistic underpinnings of these responses which shed light on the underlying neural circuitry involved in visual memory processing.

Face Selective Neural Activity: Comparisons Between Fixed and Free Viewing

Event Related Potentials (ERPs) are widely used to study category-selective EEG responses to visual stimuli, such as the face-selective N170 component. Typically, this is done by flashing stimuli at the point of static gaze fixation. While allowing for good experimental control, these paradigms ignore the dynamic role of eye-movements in natural vision. Fixation-related potentials (FRPs), obtained using simultaneous EEG and eye-tracking, overcome this limitation. Various studies have used FRPs to study processes such as lexical processing, target detection and attention allocation. The goal of this study was to carefully compare face-sensitive activity time-locked to an abrupt stimulus onset at fixation, with that time-locked to a self-generated fixation on a stimulus. Twelve participants participated in three experimental conditions: Free-viewing (FRPs), Cued-viewing (FRPs) and Control (ERPs). We used a multiple regression approach to disentangle overlapping activity components. Our results show that the N170 face-effect is evident for the first fixation on a stimulus, whether it follows a self-generated saccade or stimulus appearance at fixation point. The N170 face-effect has similar topography across viewing conditions, but there were major differences within each stimulus category. We ascribe these differences to an overlap of the fixation-related lambda response and the N170. We tested the plausibility of this account using dipole simulations. Finally, the N170 exhibits category-specific adaptation in free viewing. This study establishes the comparability of the free-viewing N170 face-effect with the classic event-related effect, while highlighting the importance of accounting for eye-movement related effects.

The Role of Neuronal Oscillations in Visual Active Sensing

Visual perception is most often studied as a "passive" process in which an observer fixates steadily at point in space so that stimuli can be delivered to the system with spatial precision. Analysis of neuronal signals related to vision is generally keyed to stimulus onset, stimulus movement, etc.; i.e., events external to the observer. In natural "active" vision, however, information is systematically acquired by using eye movements including rapid (saccadic) eye movements, as well as smooth ocular pursuit of moving objects and slower drifts. Here we consider the use of alternating saccades and fixations to gather information from a visual scene. The underlying motor sampling plan contains highly reliable information regarding "where" and "when" the eyes will land, this information can be used predictively to modify firing properties of neurons precisely at the time when this "contextual" information is most useful - when a volley of retinal input enters the system at the onset of each fixation. Analyses focusing on neural events leading to and resulting from shifts in fixation, as well as visual events external to the observer, can provide a more complete and mechanistic understanding of visual information processing. Studies thus far suggest that active vision may be a fundamentally different from that process we usually study with more traditional passive viewing paradigms. In this Perspective we note that active saccadic sampling behavior imposes robust temporal patterning on the activity of neuron ensembles and large-scale neural dynamics throughout the brain's visual pathways whose mechanistic effects on information processing are not yet fully understood. The spatio-temporal sequence of eye movements elicits a succession of temporally predictable quasi-rhythmic sensory inputs, whose encoding is enhanced by entrainment of low frequency oscillations to the rate of eye movements. Review of the pertinent findings underscores the fact that temporal coordination between motor and visual cortices is critical for understanding neural dynamics of active vision and posits that phase entrainment of neuronal oscillations plays a mechanistic role in this process.

Saccade-based termination responses in macaque V1 and visual perception

Neurons in visual areas of the brain are generally characterized by the increase in firing rate that occurs when a stimulus is flashed on in the receptive field (RF). However, neurons also increase their firing rate when a stimulus is turned off. These "termination responses" or "after-discharges" that occur with flashed stimuli have been observed in area V1 and they may be important for vision as stimulus terminations have been shown to influence visual perception. The goal of the present study was to determine the strength of termination responses in the more natural situation in which eye movements move a stimulus out of an RF. We find that termination responses do occur in macaque V1 when termination results from a saccadic eye movement, but they are smaller in amplitude compared to flashed-off stimuli. Furthermore, there are termination responses even in the absence of visual stimulation. These findings demonstrate that termination responses are a component of naturalistic vision. They appear to be based on both visual and nonvisual signals in visual cortex. We speculate that the weakening of termination responses might be a neural correlate of saccadic suppression, the loss of perceptual sensitivity around the time of saccades.

Transsacadic Information and Corollary Discharge in Local Field Potentials of Macaque V1

Approximately three times per second, human visual perception is interrupted by a saccadic eye movement. In addition to taking the eyes to a new location, several lines of evidence suggest that the saccades play multiple roles in visual perception. Indeed, it may be crucial that visual processing is informed about movements of the eyes in order to analyze visual input distinctly and efficiently on each fixation and preserve stable visual perception of the world across saccades. A variety of studies has demonstrated that activity in multiple brain areas is modulated by saccades. The hypothesis tested here is that these signals carry significant information that could be used in visual processing. To test this hypothesis, local field potentials (LFPs) were simultaneously recorded from multiple electrodes in macaque primary visual cortex (V1); support vector machines (SVMs) were used to classify the peri-saccadic LFPs. We find that LFPs in area V1 carry information that can be used to distinguish neural activity associated with fixations from saccades, precisely estimate the onset time of fixations, and reliably infer the directions of saccades. This information may be used by the brain in processes including visual stability, saccadic suppression, receptive field (RF) remapping, fixation amplification, and trans-saccadic visual perception.

Rapid Eye Movements in Sleep Furnish a Unique Probe Into Consciousness

The neural correlates of rapid eye movements (REMs) in sleep are extraordinarily robust; including REM-locked multisensory-motor integration and accompanying activation in the retrosplenial cortex, the supplementary eye field and areas encompassing cholinergic basal nucleus (Hong et al., 2009). The phenomenology of REMs speaks to the notion that perceptual experience in both sleep and wakefulness is a constructive process - in which we generate predictions of sensory inputs and then test those predictions through actively sampling the sensorium with eye movements. On this view, REMs during sleep may index an internalized active sampling or 'scanning' of self-generated visual constructs that are released from the constraints of visual input. If this view is correct, it renders REMs an ideal probe to study consciousness as "an exclusively internal affair" (Metzinger, 2009). In other words, REMs offer a probe of active inference - in the sense of predictive coding - when the brain is isolated from the sensorium in virtue of the natural blockade of sensory afferents during REM sleep. Crucially, REMs are temporally precise events that enable powerful inferences based on time series analyses. As a natural, task-free probe, (REMs) could be used in non-compliant subjects, including infants and animals. In short, REMs constitute a promising probe to study the ontogenetic and phylogenetic development of consciousness and perhaps the psychopathology of schizophrenia and autism, which have been considered in terms of aberrant predictive coding.

Deep temporal models and active inference

How do we navigate a deeply structured world? Why are you reading this sentence first - and did you actually look at the fifth word? This review offers some answers by appealing to active inference based on deep temporal models. It builds on previous formulations of active inference to simulate behavioural and electrophysiological responses under hierarchical generative models of state transitions. Inverting these models corresponds to sequential inference, such that the state at any hierarchical level entails a sequence of transitions in the level below. The deep temporal aspect of these models means that evidence is accumulated over nested time scales, enabling inferences about narratives (i.e., temporal scenes). We illustrate this behaviour with Bayesian belief updating - and neuronal process theories - to simulate the epistemic foraging seen in reading. These simulations reproduce perisaccadic delay period activity and local field potentials seen empirically. Finally, we exploit the deep structure of these models to simulate responses to local (e.g., font type) and global (e.g., semantic) violations; reproducing mismatch negativity and P300 responses respectively.

Left-Lateralized Contributions of Saccades to Cortical Activity During a One-Back Word Recognition Task

Saccadic eye movements are an inherent component of natural reading, yet their contribution to information processing at subsequent fixation remains elusive. Here we use anatomically-constrained magnetoencephalography (MEG) to examine cortical activity following saccades as healthy human subjects engaged in a one-back word recognition task. This activity was compared with activity following external visual stimulation that mimicked saccades. A combination of procedures was employed to eliminate saccadic ocular artifacts from the MEG signal. Both saccades and saccade-like external visual stimulation produced early-latency responses beginning ~70 ms after onset in occipital cortex and spreading through the ventral and dorsal visual streams to temporal, parietal and frontal cortices. Robust differential activity following the onset of saccades vs. similar external visual stimulation emerged during 150-350 ms in a left-lateralized cortical network. This network included: (i) left lateral occipitotemporal (LOT) and nearby inferotemporal (IT) cortex; (ii) left posterior Sylvian fissure (PSF) and nearby multimodal cortex; and (iii) medial parietooccipital (PO), posterior cingulate and retrosplenial cortices. Moreover, this left-lateralized network colocalized with word repetition priming effects. Together, results suggest that central saccadic mechanisms influence a left-lateralized language network in occipitotemporal and temporal cortex above and beyond saccadic influences at preceding stages of information processing during visual word recognition.

Large-Scale Traveling Waves in EEG Activity Following Eye Movement

In spontaneous, stimulus-evoked, and eye-movement evoked EEG, the oscillatory signal shows large scale, dynamically organized patterns of phase. We investigated eye-movement evoked patterns in free-viewing conditions. Participants viewed photographs of natural scenes in anticipation of a memory test. From 200 ms intervals following saccades, we estimated the EEG phase gradient over the entire scalp, and the wave activity, i.e. the goodness of fit of a wave model involving a phase gradient assumed to be smooth over the scalp. In frequencies centered at 6.5 Hz, large-scale phase organization occurred, peaking around 70 ms after fixation onset and taking the form of a traveling wave. According to the wave gradient, most of the times the wave spreads from the posterior-inferior to anterior-superior direction. In these directions, the gradients depended on the size and direction of the saccade. Wave propagation velocity decreased in the course of the fixation, particularly in the interval from 50 to 150 ms after fixation onset. This interval corresponds to the fixation-related lambda activity, which reflects early perceptual processes following fixation onset. We conclude that lambda activity has a prominent traveling wave component. This component consists of a short-term whole-head phase pattern of specific direction and velocity, which may reflect feedforward propagation of visual information at fixation.

The graphical brain: Belief propagation and active inference

This paper considers functional integration in the brain from a computational perspective. We ask what sort of neuronal message passing is mandated by active inference-and what implications this has for context-sensitive connectivity at microscopic and macroscopic levels. In particular, we formulate neuronal processing as belief propagation under deep generative models. Crucially, these models can entertain both discrete and continuous states, leading to distinct schemes for belief updating that play out on the same (neuronal) architecture. Technically, we use Forney (normal) factor graphs to elucidate the requisite message passing in terms of its form and scheduling. To accommodate mixed generative models (of discrete and continuous states), one also has to consider link nodes or factors that enable discrete and continuous representations to talk to each other. When mapping the implicit computational architecture onto neuronal connectivity, several interesting features emerge. For example, Bayesian model averaging and comparison, which link discrete and continuous states, may be implemented in thalamocortical loops. These and other considerations speak to a computational connectome that is inherently state dependent and self-organizing in ways that yield to a principled (variational) account. We conclude with simulations of reading that illustrate the implicit neuronal message passing, with a special focus on how discrete (semantic) representations inform, and are informed by, continuous (visual) sampling of the sensorium.

AUTHOR SUMMARY

This paper considers functional integration in the brain from a computational perspective. We ask what sort of neuronal message passing is mandated by active inference-and what implications this has for context-sensitive connectivity at microscopic and macroscopic levels. In particular, we formulate neuronal processing as belief propagation under deep generative models that can entertain both discrete and continuous states. This leads to distinct schemes for belief updating that play out on the same (neuronal) architecture. Technically, we use Forney (normal) factor graphs to characterize the requisite message passing, and link this formal characterization to canonical microcircuits and extrinsic connectivity in the brain.

Deep temporal models and active inference

How do we navigate a deeply structured world? Why are you reading this sentence first - and did you actually look at the fifth word? This review offers some answers by appealing to active inference based on deep temporal models. It builds on previous formulations of active inference to simulate behavioural and electrophysiological responses under hierarchical generative models of state transitions. Inverting these models corresponds to sequential inference, such that the state at any hierarchical level entails a sequence of transitions in the level below. The deep temporal aspect of these models means that evidence is accumulated over nested time scales, enabling inferences about narratives (i.e., temporal scenes). We illustrate this behaviour with Bayesian belief updating - and neuronal process theories - to simulate the epistemic foraging seen in reading. These simulations reproduce perisaccadic delay period activity and local field potentials seen empirically. Finally, we exploit the deep structure of these models to simulate responses to local (e.g., font type) and global (e.g., semantic) violations; reproducing mismatch negativity and P300 responses respectively.

Robust modulation of arousal regulation, performance, and frontostriatal activity through central thalamic deep brain stimulation in healthy nonhuman primates

The central thalamus (CT) is a key component of the brain-wide network underlying arousal regulation and sensory-motor integration during wakefulness in the mammalian brain. Dysfunction of the CT, typically a result of severe brain injury (SBI), leads to long-lasting impairments in arousal regulation and subsequent deficits in cognition. Central thalamic deep brain stimulation (CT-DBS) is proposed as a therapy to reestablish and maintain arousal regulation to improve cognition in select SBI patients. However, a mechanistic understanding of CT-DBS and an optimal method of implementing this promising therapy are unknown. Here we demonstrate in two healthy nonhuman primates (NHPs), Macaca mulatta, that location-specific CT-DBS improves performance in visuomotor tasks and is associated with physiological effects consistent with enhancement of endogenous arousal. Specifically, CT-DBS within the lateral wing of the central lateral nucleus and the surrounding medial dorsal thalamic tegmental tract (DTTm) produces a rapid and robust modulation of performance and arousal, as measured by neuronal activity in the frontal cortex and striatum. Notably, the most robust and reliable behavioral and physiological responses resulted when we implemented a novel method of CT-DBS that orients and shapes the electric field within the DTTm using spatially separated DBS leads. Collectively, our results demonstrate that selective activation within the DTTm of the CT robustly regulates endogenous arousal and enhances cognitive performance in the intact NHP; these findings provide insights into the mechanism of CT-DBS and further support the development of CT-DBS as a therapy for reestablishing arousal regulation to support cognition in SBI patients.

Sharp Wave Ripples during Visual Exploration in the Primate Hippocampus

Hippocampal sharp-wave ripples (SWRs) are highly synchronous oscillatory field potentials that are thought to facilitate memory consolidation. SWRs typically occur during quiescent states, when neural activity reflecting recent experience is replayed. In rodents, SWRs also occur during brief locomotor pauses in maze exploration, where they appear to support learning during experience. In this study, we detected SWRs that occurred during quiescent states, but also during goal-directed visual exploration in nonhuman primates (Macaca mulatta). The exploratory SWRs showed peak frequency bands similar to those of quiescent SWRs, and both types were inhibited at the onset of their respective behavioral epochs. In apparent contrast to rodent SWRs, these exploratory SWRs occurred during active periods of exploration, e.g., while animals searched for a target object in a scene. SWRs were associated with smaller saccades and longer fixations. Also, when they coincided with target-object fixations during search, detection was more likely than when these events were decoupled. Although we observed high gamma-band field potentials of similar frequency to SWRs, only the SWRs accompanied greater spiking synchrony in neural populations. These results reveal that SWRs are not limited to off-line states as conventionally defined; rather, they occur during active and informative performance windows. The exploratory SWR in primates is an infrequent occurrence associated with active, attentive performance, which may indicate a new, extended role of SWRs during exploration in primates.

SIGNIFICANCE STATEMENT

Sharp-wave ripples (SWRs) are high-frequency oscillations that generate highly synchronized activity in neural populations. Their prevalence in sleep and quiet wakefulness, and the memory deficits that result from their interruption, suggest that SWRs contribute to memory consolidation during rest. Here, we report that SWRs from the monkey hippocampus occur not only during behavioral inactivity but also during successful visual exploration. SWRs were associated with attentive, focal search and appeared to enhance perception of locations viewed around the time of their occurrence. SWRs occurring in rest are noteworthy for their relation to heightened neural population activity, temporally precise and widespread synchronization, and memory consolidation; therefore, the SWRs reported here may have a similar effect on neural populations, even as experiences unfold.

Single-neuron activity and eye movements during human REM sleep and awake vision

Are rapid eye movements (REMs) in sleep associated with visual-like activity, as during wakefulness? Here we examine single-unit activities (n=2,057) and intracranial electroencephalography across the human medial temporal lobe (MTL) and neocortex during sleep and wakefulness, and during visual stimulation with fixation. During sleep and wakefulness, REM onsets are associated with distinct intracranial potentials, reminiscent of ponto-geniculate-occipital waves. Individual neurons, especially in the MTL, exhibit reduced firing rates before REMs as well as transient increases in firing rate immediately after, similar to activity patterns observed upon image presentation during fixation without eye movements. Moreover, the selectivity of individual units is correlated with their response latency, such that units activated after a small number of images or REMs exhibit delayed increases in firing rates. Finally, the phase of theta oscillations is similarly reset following REMs in sleep and wakefulness, and after controlled visual stimulation. Our results suggest that REMs during sleep rearrange discrete epochs of visual-like processing as during wakefulness.

Since the discovery of rapid eye movements (REMs), a critical question endures as to whether they represent time points at which visual-like processing is updated. Here the authors demonstrate that cortical activity during sleep REMs shares many properties with that observed during saccades and vision.

Understanding Neural Population Coding: Information Theoretic Insights from the Auditory System

In recent years, our research in computational neuroscience has focused on understanding how populations of neurons encode naturalistic stimuli. In particular, we focused on how populations of neurons use the time domain to encode sensory information. In this focused review, we summarize this recent work from our laboratory. We focus in particular on the mathematical methods that we developed for the quantification of how information is encoded by populations of neurons and on how we used these methods to investigate the encoding of complex naturalistic sounds in auditory cortex. We review how these methods revealed a complementary role of low frequency oscillations and millisecond precise spike patterns in encoding complex sounds and in making these representations robust to imprecise knowledge about the timing of the external stimulus. Further, we discuss challenges in extending this work to understand how large populations of neurons encode sensory information. Overall, this previous work provides analytical tools and conceptual understanding necessary to study the principles of how neural populations reflect sensory inputs and achieve a stable representation despite many uncertainties in the environment.

Reading spike timing without a clock: intrinsic decoding of spike trains

The precise timing of action potentials of sensory neurons relative to the time of stimulus presentation carries substantial sensory information that is lost or degraded when these responses are summed over longer time windows. However, it is unclear whether and how downstream networks can access information in precise time-varying neural responses. Here, we review approaches to test the hypothesis that the activity of neural populations provides the temporal reference frames needed to decode temporal spike patterns. These approaches are based on comparing the single-trial stimulus discriminability obtained from neural codes defined with respect to network-intrinsic reference frames to the discriminability obtained from codes defined relative to the experimenter's computer clock. Application of this formalism to auditory, visual and somatosensory data shows that information carried by millisecond-scale spike times can be decoded robustly even with little or no independent external knowledge of stimulus time. In cortex, key components of such intrinsic temporal reference frames include dedicated neural populations that signal stimulus onset with reliable and precise latencies, and low-frequency oscillations that can serve as reference for partitioning extended neuronal responses into informative spike patterns.

Saccades during visual exploration align hippocampal 3-8 Hz rhythms in human and non-human primates

Visual exploration in primates depends on saccadic eye movements (SEMs) that cause alternations of neural suppression and enhancement. This modulation extends beyond retinotopic areas, and is thought to facilitate perception; yet saccades may also influence brain regions critical for forming memories of these exploratory episodes. The hippocampus, for example, shows oscillatory activity that is generally associated with encoding of information. Whether or how hippocampal oscillations are influenced by eye movements is unknown. We recorded the neural activity in the human and macaque hippocampus during visual scene search. Across species, SEMs were associated with a time-limited alignment of a low-frequency (3–8 Hz) rhythm. The phase alignment depended on the task and not only on eye movements per se, and the frequency band was not a direct consequence of saccade rate. Hippocampal theta-frequency oscillations are produced by other mammals during repetitive exploratory behaviors, including whisking, sniffing, echolocation, and locomotion. The present results may reflect a similar yet distinct primate homologue supporting active perception during exploration.

Human occipital cortices differentially exert saccadic suppression: Intracranial recording in children

By repeating saccades unconsciously, humans explore the surrounding world every day. Saccades inevitably move external visual images across the retina at high velocity; nonetheless, healthy humans don't perceive transient blurring of the visual scene during saccades. This perceptual stability is referred to as saccadic suppression. Functional suppression is believed to take place transiently in the visual systems, but it remains unknown how commonly or differentially the human occipital lobe activities are suppressed at the large-scale cortical network level. We determined the spatial-temporal dynamics of intracranially-recorded gamma activity at 80-150 Hz around spontaneous saccades under no-task conditions during wakefulness and those in darkness during REM sleep. Regardless of wakefulness or REM sleep, a small degree of attenuation of gamma activity was noted in the occipital regions during saccades, most extensively in the polar and least in the medial portions. Longer saccades were associated with more intense gamma-attenuation. Gamma-attenuation was subsequently followed by gamma-augmentation most extensively involving the medial and least involving the polar occipital region. Such gamma-augmentation was more intense during wakefulness and temporally locked to the offset of saccades. The polarities of initial peaks of perisaccadic event-related potentials (ERPs) were frequently positive in the medial and negative in the polar occipital regions. The present study, for the first time, provided the electrophysiological evidence that human occipital cortices differentially exert perisaccadic modulation. Transiently suppressed sensitivity of the primary visual cortex in the polar region may be an important neural basis for saccadic suppression. Presence of occipital gamma-attenuation even during REM sleep suggests that saccadic suppression might be exerted even without external visual inputs. The primary visual cortex in the medial region, compared to the polar region, may be more sensitive to an upcoming visual scene provided at the offset of each saccade.

Eye movements reset visual perception

Human vision uses saccadic eye movements to rapidly shift the sensitive foveal portion of our retina to objects of interest. For vision to function properly amidst these ballistic eye movements, a mechanism is needed to extract discrete percepts on each fixation from the continuous stream of neural activity that spans fixations. The speed of visual parsing is crucial because human behaviors ranging from reading to driving to sports rely on rapid visual analysis. We find that a brain signal associated with moving the eyes appears to play a role in resetting visual analysis on each fixation, a process that may aid in parsing the neural signal. We quantified the degree to which the perception of tilt is influenced by the tilt of a stimulus on a preceding fixation. Two key conditions were compared, one in which a saccade moved the eyes from one stimulus to the next and a second simulated saccade condition in which the stimuli moved in the same manner but the subjects did not move their eyes. We find that there is a brief period of time at the start of each fixation during which the tilt of the previous stimulus influences perception (in a direction opposite to the tilt aftereffect)--perception is not instantaneously reset when a fixation starts. Importantly, the results show that this perceptual bias is much greater, with nearly identical visual input, when saccades are simulated. This finding suggests that, in real-saccade conditions, some signal related to the eye movement may be involved in the reset phenomenon. While proprioceptive information from the extraocular muscles is conceivably a factor, the fast speed of the effect we observe suggests that a more likely mechanism is a corollary discharge signal associated with eye movement.

Gating of attentional effort through the central thalamus

The central thalamus plays an important role in the regulation of arousal and allocation of attentional resources in the performance of even simple tasks. To assess the contribution of central thalamic neurons to short-term adjustments of attentional effort, we analyzed 166 microelectrode recordings obtained from two rhesus monkeys performing a visuomotor simple reaction time task with a variable foreperiod. Multiunit responses showed maintained firing rate elevations during the variable delay period of the task in ∼24% of recording sites. Simultaneously recorded local field potentials demonstrated significant decreases in power at ∼10-20 Hz and increases in power at 30-100 Hz during the delay period when compared against precue baselines. Comparison of the spectral power of local field potentials during the delay period of correct and incorrect trials showed that, during incorrect trials, similar, but reduced, shifts of spectral power occurred within the same frequency bands. Sustained performance of even simple tasks requires regulation of arousal and attention that combine in the concept of "attentional effort". Our findings suggest that central thalamic neurons regulate task performance through brief changes in firing rates and spectral power changes during task-relevant short-term shifts of attentional effort. Increases in attentional effort may be reflected in changes within the central thalamic local populations, where correct task performance associates with more robust maintenance of firing rates during the delay period. Such ongoing fluctuations of central thalamic activity likely reflect a mix of influences, including variations in moment-to-moment levels of motivation, arousal, and availability of cognitive resources.

Category-selective phase coding in the superior temporal sulcus

Object perception and categorization can occur so rapidly that behavioral responses precede or co-occur with the firing rate changes in the object-selective neocortex. Phase coding could, in principle, support rapid representation of object categories, whereby the first spikes evoked by a stimulus would appear at different phases of an oscillation, depending on the object category. To determine whether object-selective regions of the neo-cortex demonstrate phase coding, we presented images of faces and objects to two monkeys while recording local field potentials (LFP) and single unit activity from object-selective regions in the upper bank superior temporal sulcus. Single units showed preferred phases of firing that depended on stimulus category, emerging with the initiation of spiking responses after stimulus onset. Differences in phase of firing were seen below 20 Hz and in the gamma and high-gamma frequency ranges. For all but the <20-Hz cluster, phase differences remained category-specific even when controlling for stimulus-locked activity, revealing that phase-specific firing is not a simple consequence of category-specific differences in the evoked responses of the LFP. In addition, we tested for firing rate-to-phase conversion. Category-specific differences in firing rates accounted for 30-40% of the explained variance in phase occurring at lower frequencies (<20 Hz) during the initial response, but was limited (<20% of the explained variance) in the 30- to 60-Hz frequency range, suggesting that gamma phase-of-firing effects reflect more than evoked LFP and firing rate responses. The present results are consistent with theoretical models of rapid object processing and extend previous observations of phase coding to include object-selective neocortex.

Saccades during object viewing modulate oscillatory phase in the superior temporal sulcus

Saccadic eye movements (SEMs) are the primary means of gating visual information in primates and strongly influence visual perception. The active exploration of the visual environment ("active vision") via SEMs produces suppression during saccades and enhancement afterward (i.e., during fixation) in occipital visual areas. In lateral temporal lobe visual areas, the influence, if any, of eye movements is less well understood, despite the necessity of these areas for forming coherent percepts of objects. The upper bank of the superior temporal sulcus (uSTS) is one such area whose sensitivity to SEMs is unknown. We therefore examined how saccades modulate local field potentials (LFPs) in the uSTS of macaque monkeys while they viewed face and nonface object stimuli. LFP phase concentration increased following fixation onset in the alpha (8-14 Hz), beta (14-30 Hz), and gamma (30-60 Hz) bands and was distinct from the image-evoked response. Furthermore, near-coincident onsets of fixation and image presentation--like those occurring in active vision--led to enhanced responses through greater phase concentration in the same frequency bands. Finally, single-unit activity was modulated by the phase of alpha, beta, and gamma oscillations, suggesting that the observed phase-locking influences spike timing in uSTS. Previous research implicates phase concentration in these frequency bands as a correlate of perceptual performance (Womelsdorf et al., 2006; Bosman et al., 2009). Together, these results demonstrate sensitivity to eye movements in an object-processing region of the brain and represent a plausible neural basis for the enhancement of object processing during active vision.

Spatiotemporal effects of microsaccades on population activity in the visual cortex of monkeys during fixation

During visual fixation, the eyes make fast involuntary miniature movements known as microsaccades (MSs). When MSs are executed they displace the visual image over the retina and can generate neural modulation along the visual pathway. However, the effects of MSs on neural activity have substantial variability and are not fully understood. By utilizing voltage-sensitive dye imaging, we imaged the spatiotemporal patterns induced by MSs in V1 and V2 areas of behaving monkeys while they were fixating and presented with visual stimuli. We then investigated the neuronal modulation dynamics, induced by MSs, under different visual stimulation. MSs induced monophasic or biphasic neural responses depending on stimulus size. These neural responses were accompanied by different spatiotemporal patterns of synchronization. Finally, we show that a local patch of population response evoked by a small stimulus was clearly shifted over the V1 retinotopic map after each MS. Our results demonstrate the lack of visual stability in V1 following MSs and help clarify the substantial variability reported for MSs effects on neuronal responses. The observed neural effects suggest that MSs are associated with a continuum of neuronal responses in V1 area reflecting diverse spatiotemporal dynamics.

Visual stability

Our vision remains stable even though the movements of our eyes, head and bodies create a motion pattern on the retina. One of the most important, yet basic, feats of the visual system is to correctly determine whether this retinal motion is owing to real movement in the world or rather our own self-movement. This problem has occupied many great thinkers, such as Descartes and Helmholtz, at least since the time of Alhazen. This theme issue brings together leading researchers from animal neurophysiology, clinical neurology, psychophysics and cognitive neuroscience to summarize the state of the art in the study of visual stability. Recently, there has been significant progress in understanding the limits of visual stability in humans and in identifying many of the brain circuits involved in maintaining a stable percept of the world. Clinical studies and new experimental methods, such as transcranial magnetic stimulation, now make it possible to test the causal role of different brain regions in creating visual stability and also allow us to measure the consequences when the mechanisms of visual stability break down.

Saccade-related modulations of neuronal excitability support synchrony of visually elicited spikes

During natural vision, primates perform frequent saccadic eye movements, allowing only a narrow time window for processing the visual information at each location. Individual neurons may contribute only with a few spikes to the visual processing during each fixation, suggesting precise spike timing as a relevant mechanism for information processing. We recently found in V1 of monkeys freely viewing natural images, that fixation-related spike synchronization occurs at the early phase of the rate response after fixation-onset, suggesting a specific role of the first response spikes in V1. Here, we show that there are strong local field potential (LFP) modulations locked to the onset of saccades, which continue into the successive fixation periods. Visually induced spikes, in particular the first spikes after the onset of a fixation, are locked to a specific epoch of the LFP modulation. We suggest that the modulation of neural excitability, which is reflected by the saccade-related LFP changes, serves as a corollary signal enabling precise timing of spikes in V1 and thereby providing a mechanism for spike synchronization.

Dynamics of Active Sensing and perceptual selection

Sensory processing is often regarded as a passive process in which biological receptors like photoreceptors and mechanoreceptors transduce physical energy into a neural code. Recent findings, however, suggest that: first, most sensory processing is active, and largely determined by motor/attentional sampling routines; second, owing to rhythmicity in the motor routine, as well as to its entrainment of ambient rhythms in sensory regions, sensory inflow tends to be rhythmic; third, attentional manipulation of rhythms in sensory pathways is instrumental to perceptual selection. These observations outline the essentials of an Active Sensing paradigm, and argue for increased emphasis on the study of sensory processes as specific to the dynamic motor/attentional context in which inputs are acquired.

A microsaccadic rhythm modulates gamma-band synchronization and behavior

Rhythms occur both in neuronal activity and in behavior. Behavioral rhythms abound at frequencies at or below 10 Hz. Neuronal rhythms cover a very wide frequency range, and the phase of neuronal low-frequency rhythms often rhythmically modulates the strength of higher-frequency rhythms, particularly of gamma-band synchronization (GBS). Here, we study stimulus-induced GBS in awake monkey areas V1 and V4 in relation to a specific form of spontaneous behavior, namely microsaccades (MSs), small fixational eye movements. We found that MSs occur rhythmically at a frequency of approximately 3.3 Hz. The rhythmic MSs were predicted by the phase of the 3.3 Hz rhythm in V1 and V4 local field potentials. In turn, the MSs modulated both visually induced GBS and the speed of visually triggered behavioral responses. Fast/slow responses were preceded by a specific temporal pattern of MSs. These MS patterns induced perturbations in GBS that in turn explained variability in behavioral response speed. We hypothesize that the 3.3 Hz rhythm structures the sampling and exploration of the environment through building and breaking neuronal ensembles synchronized in the gamma-frequency band to process sensory stimuli.

Microsaccades: small steps on a long way

Contrary to common wisdom, fixations are a dynamically rich behavior, composed of continual, miniature eye movements, of which microsaccades are the most salient component. Over the last few years, interest in these small movements has risen dramatically, driven by both neurophysiological and psychophysical results and by advances in techniques, analysis, and modeling of eye movements. The field has a long history but a significant portion of the earlier work has gone missing in the current literature, in part, as a result of the collapse of the field in the 1980s that followed a series of discouraging results. The present review compiles 60 years of work demonstrating the unique contribution of microsaccades to visual and oculomotor function. Specifically, the review covers the contribution of microsaccades to (1) the control of fixation position, (2) the reduction of perceptual fading and the continuity of perception, (3) the generation of synchronized visual transients, (4) visual acuity, (5) scanning of small spatial regions, (6) shifts of spatial attention, (7) resolving perceptual ambiguities in the face of multistable perception, as well as several other functions. The accumulated evidence demonstrates that microsaccades serve both perceptual and oculomotor goals and although in some cases their contribution is neither necessary nor unique, microsaccades are a malleable tool conveniently employed by the visual system.

Attentional modulation of adaptation in V4

Adaptation and visual attention are two processes that alter neural responses to luminance contrast. Rapid contrast adaptation changes response size and dynamics at all stages of visual processing, while visual attention has been shown to modulate both contrast gain and response gain in macaque extrastriate visual cortex. Because attention aims to enhance behaviorally relevant sensory responses while adaptation acts to attenuate neural activity, the question we asked is, how does attention alter adaptation? We present here single-unit recordings from V4 of two rhesus macaques performing a cued target detection task. The study was designed to characterize the effects of attention on the size and dynamics of a sequence of responses produced by a series of flashed oriented gratings parametric in luminance contrast. We found that the effect of attention on the response dynamics of V4 neurons is inconsistent with a mechanism that only alters the effective stimulus contrast, or only rescales the gain of the response. Instead, the action of attention modifies contrast gain early in the task, and modifies both response gain and contrast gain later in the task. We also show that responses to attended stimuli are more closely locked to the stimulus cycle than unattended responses, and that attended responses show less of the phase lag produced by adaptation than unattended responses. The phase advance generated by attention of the adapted responses suggests that the attentional gain control operates in some ways like a contrast gain control utilizing a neural measure of contrast to influence dynamics.

(Micro)Saccades, corollary activity and cortical oscillations

In natural vision, attention and eye movements are linked. Furthermore, eye movements structure the inflow of information into the visual system. Saccades, where little vision occurs, alternate with fixations, when most vision occurs. A mechanism must be in place to maximize information intake during fixations. Oscillatory synchrony has been proposed as a mechanism for rapid and reliable communication of signals, subserving cognitive functions such as attention and object identification. We propose that saccade-related corollary activity has a crucial role in anticipatory preparation of visual centers, which interacts with ongoing oscillation, favoring the processing of postfixational signals. During prolonged fixations, microsaccades could be generated to exploit this mechanism. Studying this interplay between the sensory and the motor system will provide novel insight into the dynamics of natural vision.

Synchronization of neuronal responses in primary visual cortex of monkeys viewing natural images

When inspecting visual scenes, primates perform on average four saccadic eye movements per second, which implies that scene segmentation, feature binding, and identification of image components is accomplished in <200 ms. Thus individual neurons can contribute only a small number of discharges for these complex computations, suggesting that information is encoded not only in the discharge rate but also in the timing of action potentials. While monkeys inspected natural scenes we registered, with multielectrodes from primary visual cortex, the discharges of simultaneously recorded neurons. Relating these signals to eye movements revealed that discharge rates peaked around 90 ms after fixation onset and then decreased to near baseline levels within 200 ms. Unitary event analysis revealed that preceding this increase in firing there was an episode of enhanced response synchronization during which discharges of spatially distributed cells coincided within 5-ms windows significantly more often than predicted by the discharge rates. This episode started 30 ms after fixation onset and ended by the time discharge rates had reached their maximum. When the animals scanned a blank screen a small change in firing rate, but no excess synchronization, was observed. The short latency of the stimulation-related synchronization phenomena suggests a fast-acting mechanism for the coordination of spike timing that may contribute to the basic operations of scene segmentation.

Transient cortical excitation at the onset of visual fixation

Primates actively examine the visual world by rapidly shifting gaze (fixation) over the elements in a scene. Despite this fact, we typically study vision by presenting stimuli with gaze held constant. To better understand the dynamics of natural vision, we examined how the onset of visual fixation affects ongoing neuronal activity in the absence of visual stimulation. We used multiunit activity and current source density measurements to index neuronal firing patterns and underlying synaptic processes in macaque V1. Initial averaging of neural activity synchronized to the onset of fixation suggested that a brief period of cortical excitation follows each fixation. Subsequent single-trial analyses revealed that 1) neuronal oscillation phase transits from random to a highly organized state just after the fixation onset, 2) this phase concentration is accompanied by increased spectral power in several frequency bands, and 3) visual response amplitude is enhanced at the specific oscillatory phase associated with fixation. We hypothesize that nonvisual inputs are used by the brain to increase cortical excitability at fixation onset, thus "priming" the system for new visual inputs generated at fixation. Despite remaining mechanistic questions, it appears that analysis of fixation-related responses may be useful in studying natural vision.

Pointing to places and spaces in a patient with visual form agnosia

Previous investigations of visuospatial abilities in the visual form agnosic patient D.F. suggest that her egocentric sensorimotor processing is intact while her 'allocentric' judgments of spatial position are impaired. The current investigation extends these previous observations by comparing D.F.'s performance at pointing to a set of spatially distributed stimuli, either directly or by 'pantomiming' the responses in an adjacent homologous workspace. The results showed accurate sensorimotor localization when D.F. pointed directly to single targets or to sequences of targets, presumably as she could use egocentric visual coding. In spite of making relatively spared spatial judgments about the arrays, however, D.F. performed quite poorly when copying them and on the pantomimed pointing task. In this latter task good performance presumably depends on an ability to represent both the categorical and coordinate properties of the array (as does copying them), and to translate these into the effector-based coordinates required for accurate action. D.F.'s pantomimed pointing was similar to her copies of target arrays, as in both tasks there was evidence of spared (although somewhat degraded) appreciation of the relative spatial positions of the stimuli. Remarkably, her accuracy in this allocentric task was not worsened by longer pointing sequences. It is possible that D.F.'s degraded performance reflects a relative (though not complete) preservation of categorical coding within the ventral stream, despite a loss of coordinate coding there.

Movement-related frequency modulation of beta oscillatory activity in the human subthalamic nucleus

Event-related changes of brain electrical rhythms are typically analysed as amplitude modulations of local field potential (LFP) oscillations, like radio amplitude modulation broadcasting. In telecommunications, frequency modulation (FM) is less susceptible to interference than amplitude modulation (AM) and is therefore preferred for high-fidelity transmissions. Here we hypothesized that LFP rhythms detected from deep brain stimulation (DBS) electrodes implanted in the subthalamic nucleus (STN) in patients with Parkinson's disease could represent movement-related activity not only in AM but also in FM. By combining adaptive autoregressive identification with spectral power decomposition, we were able to show that FM of low-beta (13-20 Hz) and high-beta (20-35 Hz) rhythms significantly contributes to the involvement of the human STN in movement preparation, execution and recovery, and that the FM patterns are regulated by the dopamine levels in the system. Movement-related FM of beta oscillatory activity in the human subthalamic nucleus therefore provides a novel informational domain for rhythm-based pathophysiological models of cortico-basal ganglia processing.