Decoding of temporal intervals from cortical ensemble activity

Motor cortical correlates of arm resting in the context of a reaching task and implications for prosthetic control

Prosthetic devices are being developed to restore movement for motor-impaired individuals. A robotic arm can be controlled based on models that relate motor-cortical ensemble activity to kinematic parameters. The models are typically built and validated on data from structured trial periods during which a subject actively performs specific movements, but real-world prosthetic devices will need to operate correctly during rest periods as well. To develop a model of motor cortical modulation during rest, we trained monkeys (Macaca mulatta) to perform a reaching task with their own arm while recording motor-cortical single-unit activity. When a monkey spontaneously put its arm down to rest between trials, our traditional movement decoder produced a nonzero velocity prediction, which would cause undesired motion when applied to a prosthetic arm. During these rest periods, a marked shift was found in individual units' tuning functions. The activity pattern of the whole population during rest (Idle state) was highly distinct from that during reaching movements (Active state), allowing us to predict arm resting from instantaneous firing rates with 98% accuracy using a simple classifier. By cascading this state classifier and the movement decoder, we were able to predict zero velocity correctly, which would avoid undesired motion in a prosthetic application. Interestingly, firing rates during hold periods followed the Active pattern even though hold kinematics were similar to those during rest with near-zero velocity. These findings expand our concept of motor-cortical function by showing that population activity reflects behavioral context in addition to the direct parameters of the movement itself.

What limits the performance of current invasive brain machine interfaces?

The concept of a brain-machine interface (BMI) or a computer-brain interface is simple: BMI creates a communication pathway for a direct control by brain of an external device. In reality BMIs are very complex devices and only recently the increase in computing power of microprocessors enabled a boom in BMI research that continues almost unabated to this date, the high point being the insertion of electrode arrays into the brains of 5 human patients in a clinical trial run by Cyberkinetics with few other clinical tests still in progress. Meanwhile several EEG-based BMI devices (non-invasive BMIs) were launched commercially. Modern electronics and dry electrode technology made possible to drive the cost of some of these devices below few hundred dollars. However, the initial excitement of the direct control by brain waves of a computer or other equipment is dampened by large efforts required for learning, high error rates and slow response speed. All these problems are directly related to low information transfer rates typical for such EEG-based BMIs. In invasive BMIs employing multiple electrodes inserted into the brain one may expect much higher information transfer rates than in EEG-based BMIs because, in theory, each electrode provides an independent information channel. However, although invasive BMIs require more expensive equipment and have ethical problems related to the need to insert electrodes in the live brain, such financial and ethical costs are often not offset by a dramatic improvement in the information transfer rate. Thus the main topic of this review is why in invasive BMIs an apparently much larger information content obtained with multiple extracellular electrodes does not translate into much higher rates of information transfer? This paper explores possible answers to this question by concluding that more research on what movement parameters are encoded by neurons in motor cortex is needed before we can enjoy the next generation BMIs.

Chronic, wireless recordings of large-scale brain activity in freely moving rhesus monkeys

Advances in techniques for recording large-scale brain activity contribute to both the elucidation of neurophysiological principles and the development of brain-machine interfaces (BMIs). Here we describe a neurophysiological paradigm for performing tethered and wireless large-scale recordings based on movable volumetric three-dimensional (3D) multielectrode implants. This approach allowed us to isolate up to 1,800 units per animal and simultaneously record the extracellular activity of close to 500 cortical neurons, distributed across multiple cortical areas, in freely behaving rhesus monkeys. The method is expandable, in principle, to thousands of simultaneously recorded channels. It also allows increased recording longevity (5 consecutive years), and recording of a broad range of behaviors, e.g. social interactions, and BMI paradigms in freely moving primates. We propose that wireless large-scale recordings could have a profound impact on basic primate neurophysiology research, while providing a framework for the development and testing of clinically relevant neuroprostheses.

Functional dynamics of primate cortico-striatal networks during volitional movements

The motor cortex and dorsal striatum (caudate nucleus and putamen) are key regions in motor processing but the interface between the cortex and striatum is not well understood. While dorsal striatum integrates information from multiple brain regions to shape motor learning and habit formation, the disruption of cortico-striatal circuits compromises the functionality of these circuits resulting in a multitude of neurologic disorders, including Parkinson's disease. To better understand the modulation of the cortico-striatal circuits we recorded simultaneously single neuron activity from four brain regions, primary motor, and sensory cortices, together with the rostral and caudal segments of the putamen in rhesus monkeys performing a visual motor task. Results show that spatial and temporal-task related firing relationships between these cortico-striatal circuit regions were modified by the independent administration of the two drugs (cocaine and baclofen). Spatial tuning and correlated firing of neurons from motor cortex and putamen were severely disrupted by cocaine and baclofen on correct trials, while the two drugs have dramatically decreased the functional connectivity of the motor cortical-striatal network. These findings provide insight into the modulation of cortical-striatal firing related to movement with implications for therapeutic approaches to Parkinson's disease and related disorders.

Conditioning and time representation in long short-term memory networks

Dopaminergic models based on the temporal-difference learning algorithm usually do not differentiate trace from delay conditioning. Instead, they use a fixed temporal representation of elapsed time since conditioned stimulus onset. Recently, a new model was proposed in which timing is learned within a long short-term memory (LSTM) artificial neural network representing the cerebral cortex (Rivest et al. in J Comput Neurosci 28(1):107-130, 2010). In this paper, that model's ability to reproduce and explain relevant data, as well as its ability to make interesting new predictions, are evaluated. The model reveals a strikingly different temporal representation between trace and delay conditioning since trace conditioning requires working memory to remember the past conditioned stimulus while delay conditioning does not. On the other hand, the model predicts no important difference in DA responses between those two conditions when trained on one conditioning paradigm and tested on the other. The model predicts that in trace conditioning, animal timing starts with the conditioned stimulus offset as opposed to its onset. In classical conditioning, it predicts that if the conditioned stimulus does not disappear after the reward, the animal may expect a second reward. Finally, the last simulation reveals that the buildup of activity of some units in the networks can adapt to new delays by adjusting their rate of integration. Most importantly, the paper shows that it is possible, with the proposed architecture, to acquire discharge patterns similar to those observed in dopaminergic neurons and in the cerebral cortex on those tasks simply by minimizing a predictive cost function.

Timing as an intrinsic property of neural networks: evidence from in vivo and in vitro experiments

The discrimination and production of temporal patterns on the scale of hundreds of milliseconds are critical to sensory and motor processing. Indeed, most complex behaviours, such as speech comprehension and production, would be impossible in the absence of sophisticated timing mechanisms. Despite the importance of timing to human learning and cognition, little is known about the underlying mechanisms, in particular whether timing relies on specialized dedicated circuits and mechanisms or on general and intrinsic properties of neurons and neural circuits. Here, we review experimental data describing timing and interval-selective neurons in vivo and in vitro. We also review theoretical models of timing, focusing primarily on the state-dependent network model, which proposes that timing in the subsecond range relies on the inherent time-dependent properties of neurons and the active neural dynamics within recurrent circuits. Within this framework, time is naturally encoded in populations of neurons whose pattern of activity is dynamically changing in time. Together, we argue that current experimental and theoretical studies provide sufficient evidence to conclude that at least some forms of temporal processing reflect intrinsic computations based on local neural network dynamics.

Relative timing: from behaviour to neurons

Processing of temporal information is critical to behaviour. Here, we review the phenomenology and mechanism of relative timing, ordinal comparisons between the timing of occurrence of events. Relative timing can be an implicit component of particular brain computations or can be an explicit, conscious judgement. Psychophysical measurements of explicit relative timing have revealed clues about the interaction of sensory signals in the brain as well as in the influence of internal states, such as attention, on those interactions. Evidence from human neurophysiological and functional imaging studies, neuropsychological examination in brain-lesioned patients, and temporary disruptive interventions such as transcranial magnetic stimulation (TMS), point to a role of the parietal cortex in relative timing. Relative timing has traditionally been modelled as a 'race' between competing neural signals. We propose an updated race process based on the integration of sensory evidence towards a decision threshold rather than simple signal propagation. The model suggests a general approach for identifying brain regions involved in relative timing, based on looking for trial-by-trial correlations between neural activity and temporal order judgements (TOJs). Finally, we show how the paradigm can be used to reveal signals related to TOJs in parietal cortex of monkeys trained in a TOJ task.

Neural dynamics based timing in the subsecond to seconds range

The brain must solve a wide range of different temporal problems, each of which can be defined by a relevant time scale and specific functional requirements. Experimental and theoretical studies suggest that some forms of timing reflect general and inherent properties of local neural networks. Like the ripples on a pond, neural networks represent rich dynamical systems that can produce time-varying patterns of activity in response to a stimulus. State-dependent network models propose that sensory timing arises from the interaction between incoming stimuli and the internal dynamics of recurrent neural circuits. A wide-variety of time-dependent neural properties, such as short-term synaptic plasticity, are important contributors to the internal dynamics of neural circuits. In contrast to sensory timing, motor timing requires that network actively generate appropriately timed spikes even in the absence of sensory stimuli. Population clock models propose that motor timing arises from internal dynamics of recurrent network capable of self-perpetuating activity.

Temporal Perceptual Learning

Our interaction with the environment and each other is inherently time-varying in nature. It is thus not surprising that the nervous systems of animals have evolved sophisticated mechanisms to not only tell time, but to learn to discriminate and produce temporal patterns. Indeed some of the most sophisticated human behaviors, such as speech and music, would not exist if the human brain was unable to learn to discriminate and produce temporal patterns. Compared to the study of other forms of learning, such as visual perceptual learning, the study of the learning of interval and temporal pattern discrimination in the subsecond range is relatively recent. A growing number of studies over the past 15 years, however, have established that perceptual and motor timing undergo robust learning. One of the principles to have emerged from these studies is that temporal learning is generally specific to the trained interval, an observation that has important implications to the neural mechanisms underlying our ability to tell time.

Interactions of timing and prediction error learning

Timing and prediction error learning have historically been treated as independent processes, but growing evidence has indicated that they are not orthogonal. Timing emerges at the earliest time point when conditioned responses are observed, and temporal variables modulate prediction error learning in both simple conditioning and cue competition paradigms. In addition, prediction errors, through changes in reward magnitude or value alter timing of behavior. Thus, there appears to be a bi-directional interaction between timing and prediction error learning. Modern theories have attempted to integrate the two processes with mixed success. A neurocomputational approach to theory development is espoused, which draws on neurobiological evidence to guide and constrain computational model development. Heuristics for future model development are presented with the goal of sparking new approaches to theory development in the timing and prediction error fields.

Neurophysiology of timing in the hundreds of milliseconds: multiple layers of neuronal clocks in the medial premotor areas

The precise quantification of time in the subsecond scale is critical for many complex behaviors including music and dance appreciation/execution, speech comprehension/articulation, and the performance of many sports. Nevertheless, its neural underpinnings are largely unknown. Recent neurophysiological experiments from our laboratory have shown that the cell activity in the medial premotor areas (MPC) of macaques can represent different aspects of temporal processing during a synchronization-continuation tapping task (SCT). In this task the rhythmic behavior of monkeys was synchronized to a metronome of isochronous stimuli in the hundreds of milliseconds range (synchronization phase), followed by a period where animals internally temporalized their movements (continuation phase). Overall, we found that the time-keeping mechanism in MPC is governed by different layers of neural clocks. Close to the temporal control of movements are two separate populations of ramping cells that code for elapsed or remaining time for a tapping movement during the SCT. Thus, the sensorimotor loops engaged during the task may depend on the cyclic interplay between two neuronal chronometers that quantify in their instantaneous discharge rate the time passed and the remaining time for an action. In addition, we found MPC neurons that are tuned to the duration of produced intervals during the rhythmic task, showing an orderly variation in the average discharge rate as a function of duration. All the tested durations in the subsecond scale were represented in the preferred intervals of the cell population. Most of the interval-tuned cells were also tuned to the ordinal structure of the six intervals produced sequentially in the SCT. Hence, this next level of temporal processing may work as the notes of a musical score, providing information to the timing network about what duration and ordinal element of the sequence are being executed. Finally, we describe how the timing circuit can use a dynamic neural representation of the passage of time and the context in which the intervals are executed by integrating the time-varying activity of populations of cells. These neural population clocks can be defined as distinct trajectories in the multidimensional cell response-space. We provide a hypothesis of how these different levels of neural clocks can interact to constitute a coherent timing machine that controls the rhythmic behavior during the SCT.

A neural hierarchy for illusions of time: duration adaptation precedes multisensory integration

Perceived time is inherently malleable. For example, adaptation to relatively long or short sensory events leads to a repulsive aftereffect such that subsequent events appear to be contracted or expanded (duration adaptation). Perceived visual duration can also be distorted via concurrent presentation of discrepant auditory durations (multisensory integration). The neural loci of both distortions remain unknown. In the current study we use a psychophysical approach to establish their relative positioning within the sensory processing hierarchy. We show that audiovisual integration induces marked distortions of perceived visual duration. We proceed to use these distorted durations as visual adapting stimuli yet find subsequent visual duration aftereffects to be consistent with physical rather than perceived visual duration. Conversely, the concurrent presentation of adapted auditory durations with nonadapted visual durations results in multisensory integration patterns consistent with perceived, rather than physical, auditory duration. These results demonstrate that recent sensory history modifies human duration perception prior to the combination of temporal information across sensory modalities and provides support for adaptation mechanisms mediated by duration selective neurons situated in early areas of the visual and auditory nervous system (Aubie, Sayegh, & Faure, 2012; Duysens, Schaafsma, & Orban, 1996; Leary, Edwards, & Rose, 2008).

Heterogeneous attractor cell assemblies for motor planning in premotor cortex

Cognitive functions like motor planning rely on the concerted activity of multiple neuronal assemblies underlying still elusive computational strategies. During reaching tasks, we observed stereotyped sudden transitions (STs) between low and high multiunit activity of monkey dorsal premotor cortex (PMd) predicting forthcoming actions on a single-trial basis. Occurrence of STs was observed even when movement was delayed or successfully canceled after a stop signal, excluding a mere substrate of the motor execution. An attractor model accounts for upward STs and high-frequency modulations of field potentials, indicative of local synaptic reverberation. We found in vivo compelling evidence that motor plans in PMd emerge from the coactivation of such attractor modules, heterogeneous in the strength of local synaptic self-excitation. Modules with strong coupling early reacted with variable times to weak inputs, priming a chain reaction of both upward and downward STs in other modules. Such web of "flip-flops" rapidly converged to a stereotyped distributed representation of the motor program, as prescribed by the long-standing theory of associative networks.

Neural correlates of interval timing in rodent prefrontal cortex

Time interval estimation is involved in numerous behavioral processes, but its underlying neural mechanisms remain unclear. In particular, it has been controversial whether time is encoded on a linear or logarithmic scale. Based on our previous finding that inactivation of the medial prefrontal cortex (mPFC) profoundly impairs rat's ability to discriminate time intervals, we investigated how the mPFC processes temporal information by examining activity of mPFC neurons in rats performing a temporal bisection task. Many mPFC neurons conveyed temporal information based on monotonically changing activity profiles over time with negative accelerations, so that their activity profiles were better described by logarithmic than linear functions. Moreover, the precision of time-interval discrimination based on neural activity was lowered in proportion to the elapse of time, but without proportional increase in neural variability, which is well accounted for by logarithmic, but not by linear functions. As a population, mPFC neurons conveyed precise information about the elapse of time with their activity tightly correlated with the animal's choice of target. These results suggest that the mPFC might be part of an internal clock in charge of controlling interval-timing behavior, and that linearly changing neuronal activity on a logarithmic time scale might be one way of representing the elapse of time in the brain.

Interval tuning in the primate medial premotor cortex as a general timing mechanism

The precise quantification of time during motor performance is critical for many complex behaviors, including musical execution, speech articulation, and sports; however, its neural mechanisms are primarily unknown. We found that neurons in the medial premotor cortex (MPC) of behaving monkeys are tuned to the duration of produced intervals during rhythmic tapping tasks. Interval-tuned neurons showed similar preferred intervals across tapping behaviors that varied in the number of produced intervals and the modality used to drive temporal processing. In addition, we found that the same population of neurons is able to multiplex the ordinal structure of a sequence of rhythmic movements and a wide range of durations in the range of hundreds of milliseconds. Our results also revealed a possible gain mechanism for encoding the total number of intervals in a sequence of temporalized movements, where interval-tuned cells show a multiplicative effect of their activity for longer sequences of intervals. These data suggest that MPC is part of a core timing network that uses interval tuning as a signal to represent temporal processing in a variety of behavioral contexts where time is explicitly quantified.

Neuronal assembly dynamics in supervised and unsupervised learning scenarios

The dynamic formation of groups of neurons--neuronal assemblies--is believed to mediate cognitive phenomena at many levels, but their detailed operation and mechanisms of interaction are still to be uncovered. One hypothesis suggests that synchronized oscillations underpin their formation and functioning, with a focus on the temporal structure of neuronal signals. In this context, we investigate neuronal assembly dynamics in two complementary scenarios: the first, a supervised spike pattern classification task, in which noisy variations of a collection of spikes have to be correctly labeled; the second, an unsupervised, minimally cognitive evolutionary robotics tasks, in which an evolved agent has to cope with multiple, possibly conflicting, objectives. In both cases, the more traditional dynamical analysis of the system's variables is paired with information-theoretic techniques in order to get a broader picture of the ongoing interactions with and within the network. The neural network model is inspired by the Kuramoto model of coupled phase oscillators and allows one to fine-tune the network synchronization dynamics and assembly configuration. The experiments explore the computational power, redundancy, and generalization capability of neuronal circuits, demonstrating that performance depends nonlinearly on the number of assemblies and neurons in the network and showing that the framework can be exploited to generate minimally cognitive behaviors, with dynamic assembly formation accounting for varying degrees of stimuli modulation of the sensorimotor interactions.

Trading speed and accuracy by coding time: a coupled-circuit cortical model

Our actions take place in space and time, but despite the role of time in decision theory and the growing acknowledgement that the encoding of time is crucial to behaviour, few studies have considered the interactions between neural codes for objects in space and for elapsed time during perceptual decisions. The speed-accuracy trade-off (SAT) provides a window into spatiotemporal interactions. Our hypothesis is that temporal coding determines the rate at which spatial evidence is integrated, controlling the SAT by gain modulation. Here, we propose that local cortical circuits are inherently suited to the relevant spatial and temporal coding. In simulations of an interval estimation task, we use a generic local-circuit model to encode time by ‘climbing’ activity, seen in cortex during tasks with a timing requirement. The model is a network of simulated pyramidal cells and inhibitory interneurons, connected by conductance synapses. A simple learning rule enables the network to quickly produce new interval estimates, which show signature characteristics of estimates by experimental subjects. Analysis of network dynamics formally characterizes this generic, local-circuit timing mechanism. In simulations of a perceptual decision task, we couple two such networks. Network function is determined only by spatial selectivity and NMDA receptor conductance strength; all other parameters are identical. To trade speed and accuracy, the timing network simply learns longer or shorter intervals, driving the rate of downstream decision processing by spatially non-selective input, an established form of gain modulation. Like the timing network's interval estimates, decision times show signature characteristics of those by experimental subjects. Overall, we propose, demonstrate and analyse a generic mechanism for timing, a generic mechanism for modulation of decision processing by temporal codes, and we make predictions for experimental verification.

Studies in neuroscience have characterized how the brain represents objects in space and how these objects are selected for detailed perceptual processing. The selection process entails a decision about which object is favoured by the available evidence over time. This period of time is typically in the range of hundreds of milliseconds and is widely believed to be crucial for decisions, allowing neurons to filter noise in the evidence. Despite the widespread belief that time plays this role in decisions and the growing recognition that the brain estimates elapsed time during perceptual tasks, few studies have considered how the encoding of time effects decision making. We propose that neurons encode time in this range by the same general mechanisms used to select objects for detailed processing, and that these temporal representations determine how long evidence is filtered. To this end, we simulate a perceptual decision by coupling two instances of a neural network widely used to simulate localized regions of the cerebral cortex. One network encodes the passage of time and the other makes decisions based on noisy evidence. The former influences the performance of the latter, reproducing signature characteristics of temporal estimates and perceptual decisions.

Delay activity of saccade-related neurons in the caudal dentate nucleus of the macaque cerebellum

The caudal dentate nucleus (DN) in lateral cerebellum is connected with two visual/oculomotor areas of the cerebrum: the frontal eye field and lateral intraparietal cortex. Many neurons in frontal eye field and lateral intraparietal cortex produce "delay activity" between stimulus and response that correlates with processes such as motor planning. Our hypothesis was that caudal DN neurons would have prominent delay activity as well. From lesion studies, we predicted that this activity would be related to self-timing, i.e., the triggering of saccades based on the internal monitoring of time. We recorded from neurons in the caudal DN of monkeys (Macaca mulatta) that made delayed saccades with or without a self-timing requirement. Most (84%) of the caudal DN neurons had delay activity. These neurons conveyed at least three types of information. First, their activity was often correlated, trial by trial, with saccade initiation. Correlations were found more frequently in a task that required self-timing of saccades (53% of neurons) than in a task that did not (27% of neurons). Second, the delay activity was often tuned for saccade direction (in 65% of neurons). This tuning emerged continuously during a trial. Third, the time course of delay activity associated with self-timed saccades differed significantly from that associated with visually guided saccades (in 71% of neurons). A minority of neurons had sensory-related activity. None had presaccadic bursts, in contrast to DN neurons recorded more rostrally. We conclude that caudal DN neurons convey saccade-related delay activity that may contribute to the motor preparation of when and where to move.

State-based decoding of hand and finger kinematics using neuronal ensemble and LFP activity during dexterous reach-to-grasp movements

The performance of brain-machine interfaces (BMIs) that continuously control upper limb neuroprostheses may benefit from distinguishing periods of posture and movement so as to prevent inappropriate movement of the prosthesis. Few studies, however, have investigated how decoding behavioral states and detecting the transitions between posture and movement could be used autonomously to trigger a kinematic decoder. We recorded simultaneous neuronal ensemble and local field potential (LFP) activity from microelectrode arrays in primary motor cortex (M1) and dorsal (PMd) and ventral (PMv) premotor areas of two male rhesus monkeys performing a center-out reach-and-grasp task, while upper limb kinematics were tracked with a motion capture system with markers on the dorsal aspect of the forearm, hand, and fingers. A state decoder was trained to distinguish four behavioral states (baseline, reaction, movement, hold), while a kinematic decoder was trained to continuously decode hand end point position and 18 joint angles of the wrist and fingers. LFP amplitude most accurately predicted transition into the reaction (62%) and movement (73%) states, while spikes most accurately decoded arm, hand, and finger kinematics during movement. Using an LFP-based state decoder to trigger a spike-based kinematic decoder [r = 0.72, root mean squared error (RMSE) = 0.15] significantly improved decoding of reach-to-grasp movements from baseline to final hold, compared with either a spike-based state decoder combined with a spike-based kinematic decoder (r = 0.70, RMSE = 0.17) or a spike-based kinematic decoder alone (r = 0.67, RMSE = 0.17). Combining LFP-based state decoding with spike-based kinematic decoding may be a valuable step toward the realization of BMI control of a multifingered neuroprosthesis performing dexterous manipulation.

An adaptive drift-diffusion model of interval timing dynamics

Animals readily learn the timing between salient events. They can even adapt their timed responding to rapidly changing intervals, sometimes as quickly as a single trial. Recently, drift-diffusion models-widely used to model response times in decision making-have been extended with new learning rules that allow them to accommodate steady-state interval timing, including scalar timing and timescale invariance. These time-adaptive drift-diffusion models (TDDMs) work by accumulating evidence of elapsing time through their drift rate, thereby encoding the to-be-timed interval. One outstanding challenge for these models lies in the dynamics of interval timing-when the to-be-timed intervals are non-stationary. On these schedules, animals often fail to exhibit strict timescale invariance, as expected by the TDDMs and most other timing models. Here, we introduce a simple extension to these TDDMs, where the response threshold is a linear function of the observed event rate. This new model compares favorably against the basic TDDMs and the multiple-time-scale (MTS) habituation model when evaluated against three published datasets on timing dynamics in pigeons. Our results suggest that the threshold for triggering responding in interval timing changes as a function of recent intervals.

The inner sense of time: how the brain creates a representation of duration

A large number of competing models exist for how the brain creates a representation of time. However, several human and animal studies point to 'climbing neural activation' as a potential neural mechanism for the representation of duration. Neurophysiological recordings in animals have revealed how climbing neural activation that peaks at the end of a timed interval underlies the processing of duration, and, in humans, climbing neural activity in the insular cortex, which is associated with feeling states of the body and emotions, may be related to the cumulative representation of time.

On the anticipatory precue activity in motor cortex

Motor cortical neurons are activated during movement preparation and execution, and in response to task-relevant visual cues. A few studies also report activation before the expected presentation of cues. Here, we study specifically this anticipatory activity preceding visual cues in motor cortical areas. We recorded the activity of 1215 neurons in the motor cortex of two macaque monkeys while they performed a center-out reaching task, including two consecutive delays of equal duration, known in advance. During the first delay (D1), they had to await the spatial cue and only reach to the cued target after the second delay (D2). Forty-two percent of the neurons displayed anticipatory activity during D1. Among these anticipatory neurons, 59% increased (D1up) their activity and the remaining decreased (D1down) their activity. By classifying the neurons according to these firing rate profiles during D1, we found that the activity during D2 differed in a systematic way. The D1up neurons were more likely to discharge phasically soon after the spatial cue and were less active during movement execution, whereas the D1down neurons showed the opposite pattern. But, regardless of their temporal activity profiles, the two categories seemed equally involved in early and late motor preparation, as reflected in their directional selectivity. This precue activity in motor cortex may reflect two complementary, coexisting processes: the facilitation of incoming spatial information in parallel with the downregulation of corticospinal excitability to prevent a premature response.

Decoding hindlimb movement for a brain machine interface after a complete spinal transection

Stereotypical locomotor movements can be made without input from the brain after a complete spinal transection. However, the restoration of functional gait requires descending modulation of spinal circuits to independently control the movement of each limb. To evaluate whether a brain-machine interface (BMI) could be used to regain conscious control over the hindlimb, rats were trained to press a pedal and the encoding of hindlimb movement was assessed using a BMI paradigm. Off-line, information encoded by neurons in the hindlimb sensorimotor cortex was assessed. Next neural population functions, or weighted representations of the neuronal activity, were used to replace the hindlimb movement as a trigger for reward in real-time (on-line decoding) in three conditions: while the animal could still press the pedal, after the pedal was removed and after a complete spinal transection. A novel representation of the motor program was learned when the animals used neural control to achieve water reward (e.g. more information was conveyed faster). After complete spinal transection, the ability of these neurons to convey information was reduced by more than 40%. However, this BMI representation was relearned over time despite a persistent reduction in the neuronal firing rate during the task. Therefore, neural control is a general feature of the motor cortex, not restricted to forelimb movements, and can be regained after spinal injury.

Temporal microstructure of cortical networks (TMCN) underlying task-related differences

Neuro-electromagnetic recording techniques (EEG, MEG, iEEG) provide high temporal resolution data to study the dynamics of neurocognitive networks: large scale neural assemblies involved in task-specific information processing. How does a neurocognitive network reorganize spatiotemporally on the order of a few milliseconds to process specific aspects of the task? At what times do networks segregate for task processing, and at what time scales does integration of information occur via changes in functional connectivity? Here, we propose a data analysis framework-Temporal microstructure of cortical networks (TMCN)-that answers these questions for EEG/MEG recordings in the signal space. Method validation is established on simulated MEG data from a delayed-match to-sample (DMS) task. We then provide an example application on MEG recordings during a paired associate task (modified from the simpler DMS paradigm) designed to study modality specific long term memory recall. Our analysis identified the times at which network segregation occurs for processing the memory recall of an auditory object paired to a visual stimulus (visual-auditory) in comparison to an analogous visual-visual pair. Across all subjects, onset times for first network divergence appeared within a range of 0.08-0.47 s after initial visual stimulus onset. This indicates that visual-visual and visual auditory memory recollection involves equivalent network components without any additional recruitment during an initial period of the sensory processing stage which is then followed by recruitment of additional network components for modality specific memory recollection. Therefore, we propose TMCN as a viable computational tool for extracting network timing in various cognitive tasks.

Decoding onset and direction of movements using Electrocorticographic (ECoG) signals in humans

Communication of intent usually requires motor function. This requirement can be limiting when a person is engaged in a task, or prohibitive for some people suffering from neuromuscular disorders. Determining a person's intent, e.g., where and when to move, from brain signals rather than from muscles would have important applications in clinical or other domains. For example, detection of the onset and direction of intended movements may provide the basis for restoration of simple grasping function in people with chronic stroke, or could be used to optimize a user's interaction with the surrounding environment. Detecting the onset and direction of actual movements are a first step in this direction. In this study, we demonstrate that we can detect the onset of intended movements and their direction using electrocorticographic (ECoG) signals recorded from the surface of the cortex in humans. We also demonstrate in a simulation that the information encoded in ECoG about these movements may improve performance in a targeting task. In summary, the results in this paper suggest that detection of intended movement is possible, and may serve useful functions.

Reprogramming movements: extraction of motor intentions from cortical ensemble activity when movement goals change

The ability to inhibit unwanted movements and change motor plans is essential for behaviors of advanced organisms. The neural mechanisms by which the primate motor system rejects undesired actions have received much attention during the last decade, but it is not well understood how this neural function could be utilized to improve the efficiency of brain-machine interfaces (BMIs). Here we employed linear discriminant analysis (LDA) and a Wiener filter to extract motor plan transitions from the activity of ensembles of sensorimotor cortex neurons. Two rhesus monkeys, chronically implanted with multielectrode arrays in primary motor (M1) and primary sensory (S1) cortices, were overtrained to produce reaching movements with a joystick toward visual targets upon their presentation. Then, the behavioral task was modified to include a distracting target that flashed for 50, 150, or 250 ms (25% of trials each) followed by the true target that appeared at a different screen location. In the remaining 25% of trials, the initial target stayed on the screen and was the target to be approached. M1 and S1 neuronal activity represented both the true and distracting targets, even for the shortest duration of the distracting event. This dual representation persisted both when the monkey initiated movements toward the distracting target and then made corrections and when they moved directly toward the second, true target. The Wiener filter effectively decoded the location of the true target, whereas the LDA classifier extracted the location of both targets from ensembles of 50–250 neurons. Based on these results, we suggest developing real-time BMIs that inhibit unwanted movements represented by brain activity while enacting the desired motor outcome concomitantly.

Making decisions through a distributed consensus

How does the brain decide between actions? Is it through comparisons of abstract representations of outcomes or through a competition in a sensorimotor map defining the actions themselves? Here, I review strengths and limitations of both of these proposals, and suggest that decisions emerge through a distributed consensus across many levels of representation.

Closing the loop for memory prosthesis: detecting the role of hippocampal neural ensembles using nonlinear models

A major factor involved in providing closed loop feedback for control of neural function is to understand how neural ensembles encode online information critical to the final behavioral endpoint. This issue was directly assessed in rats performing a short-term delay memory task in which successful encoding of task information is dependent upon specific spatio-temporal firing patterns recorded from ensembles of CA3 and CA1 hippocampal neurons. Such patterns, extracted by a specially designed nonlinear multi-input multi-output (MIMO) nonlinear mathematical model, were used to predict successful performance online via a closed loop paradigm which regulated trial difficulty (time of retention) as a function of the "strength" of stimulus encoding. The significance of the MIMO model as a neural prosthesis has been demonstrated by substituting trains of electrical stimulation pulses to mimic these same ensemble firing patterns. This feature was used repeatedly to vary "normal" encoding as a means of understanding how neural ensembles can be "tuned" to mimic the inherent process of selecting codes of different strength and functional specificity. The capacity to enhance and tune hippocampal encoding via MIMO model detection and insertion of critical ensemble firing patterns shown here provides the basis for possible extension to other disrupted brain circuitry.

Decoding 3D reach and grasp from hybrid signals in motor and premotor cortices: spikes, multiunit activity, and local field potentials

Neural activity in motor cortex during reach and grasp movements shows modulations in a broad range of signals from single-neuron spiking activity (SA) to various frequency bands in broadband local field potentials (LFPs). In particular, spatiotemporal patterns in multiband LFPs are thought to reflect dendritic integration of local and interareal synaptic inputs, attentional and preparatory processes, and multiunit activity (MUA) related to movement representation in the local motor area. Nevertheless, the relationship between multiband LFPs and SA, and their relationship to movement parameters and their relative value as brain-computer interface (BCI) control signals, remain poorly understood. Also, although this broad range of signals may provide complementary information channels in primary (MI) and ventral premotor (PMv) areas, areal differences in information have not been systematically examined. Here, for the first time, the amount of information in SA and multiband LFPs was compared for MI and PMv by recording from dual 96-multielectrode arrays while monkeys made naturalistic reach and grasp actions. Information was assessed as decoding accuracy for 3D arm end point and grip aperture kinematics based on SA or LFPs in MI and PMv, or combinations of signal types across areas. In contrast with previous studies with ≤16 simultaneous electrodes, here ensembles of >16 units (on average) carried more information than multiband, multichannel LFPs. Furthermore, reach and grasp information added by various LFP frequency bands was not independent from that in SA ensembles but rather typically less than and primarily contained within the latter. Notably, MI and PMv did not show a particular bias toward reach or grasp for this task or for a broad range of signal types. For BCIs, our results indicate that neuronal ensemble spiking is the preferred signal for decoding, while LFPs and combined signals from PMv and MI can add robustness to BCI control.

A novel tetrode microdrive for simultaneous multi-neuron recording from different regions of primate brain

A unique custom-made tetrode microdrive for recording from large numbers of neurons in several areas of primate brain is described as a means for assessing simultaneous neural activity in cortical and subcortical structures in nonhuman primates (NHPs) performing behavioral tasks. The microdrive device utilizes tetrode technology with up to six ultra-thin microprobe guide tubes (0.1mm) that can be independently positioned, each containing reduced diameter tetrode and/or hexatrode microwires (0.02 mm) for recording and isolating single neuron activity. The microdrive device is mounted within the standard NHP cranial well and allows traversal of brain depths up to 40.0 mm. The advantages of this technology are demonstrated via simultaneously recorded large populations of neurons with tetrode type probes during task performance from a) primary motor cortex and deep brain structures (caudate-putamen and hippocampus) and b) multiple layers within the prefrontal cortex. The means to characterize interactions of well-isolated ensembles of neurons recorded simultaneously from different regions, as shown with this device, has not been previously available for application in primate brain. The device has extensive application to primate models for the detection and study of inoperative or maladaptive neural circuits related to human neurological disorders.

Cortical correlates of fitts' law

Fitts’ law describes the fundamental trade-off between movement accuracy and speed: it states that the duration of reaching movements is a function of target size (TS) and distance. While Fitts’ law has been extensively studied in ergonomics and has guided the design of human–computer interfaces, there have been few studies on its neuronal correlates. To elucidate sensorimotor cortical activity underlying Fitts’ law, we implanted two monkeys with multielectrode arrays in the primary motor (M1) and primary somatosensory (S1) cortices. The monkeys performed reaches with a joystick-controlled cursor toward targets of different size. The reaction time (RT), movement time, and movement velocity changed with TS, and M1 and S1 activity reflected these changes. Moreover, modifications of cortical activity could not be explained by changes of movement parameters alone, but required TS as an additional parameter. Neuronal representation of TS was especially prominent during the early RT period where it influenced the slope of the firing rate rise preceding movement initiation. During the movement period, cortical activity was correlated with movement velocity. Neural decoders were applied to simultaneously decode TS and motor parameters from cortical modulations. We suggest that sensorimotor cortex activity reflects the characteristics of both the movement and the target. Classifiers that extract these parameters from cortical ensembles could improve neuroprosthetic control.

Representation of visual scenes by local neuronal populations in layer 2/3 of mouse visual cortex

How are visual scenes encoded in local neural networks of visual cortex? In rodents, visual cortex lacks a columnar organization so that processing of diverse features from a spot in visual space could be performed locally by populations of neighboring neurons. To examine how complex visual scenes are represented by local microcircuits in mouse visual cortex we measured visually evoked responses of layer 2/3 neuronal populations using 3D two-photon calcium imaging. Both natural and artificial movie scenes (10 seconds duration) evoked distributed and sparsely organized responses in local populations of 70–150 neurons within the sampled volumes. About 50% of neurons showed calcium transients during visual scene presentation, of which about half displayed reliable temporal activation patterns. The majority of the reliably responding neurons were activated primarily by one of the four visual scenes applied. Consequently, single-neurons performed poorly in decoding, which visual scene had been presented. In contrast, high levels of decoding performance (>80%) were reached when considering population responses, requiring about 80 randomly picked cells or 20 reliable responders. Furthermore, reliable responding neurons tended to have neighbors sharing the same stimulus preference. Because of this local redundancy, it was beneficial for efficient scene decoding to read out activity from spatially distributed rather than locally clustered neurons. Our results suggest a population code in layer 2/3 of visual cortex, where the visual environment is dynamically represented in the activation of distinct functional sub-networks.

Measuring time with different neural chronometers during a synchronization-continuation task

Temporal information processing is critical for many complex behaviors including speech and music cognition, yet its neural substrate remains elusive. We examined the neurophysiological properties of medial premotor cortex (MPC) of two Rhesus monkeys during the execution of a synchronization-continuation tapping task that includes the basic sensorimotor components of a variety of rhythmic behaviors. We show that time-keeping in the MPC is governed by separate cell populations. One group encoded the time remaining for an action, showing activity whose duration changed as a function of interval duration, reaching a peak at similar magnitudes and times with respect to the movement. The other cell group showed a response that increased in duration or magnitude as a function of the elapsed time from the last movement. Hence, the sensorimotor loops engaged during the task may depend on the cyclic interplay between different neuronal chronometers that quantify the time passed and the remaining time for an action.

Learning of temporal motor patterns: an analysis of continuous versus reset timing

Our ability to generate well-timed sequences of movements is critical to an array of behaviors, including the ability to play a musical instrument or a video game. Here we address two questions relating to timing with the goal of better understanding the neural mechanisms underlying temporal processing. First, how does accuracy and variance change over the course of learning of complex spatiotemporal patterns? Second, is the timing of sequential responses most consistent with starting and stopping an internal timer at each interval or with continuous timing? To address these questions we used a psychophysical task in which subjects learned to reproduce a sequence of finger taps in the correct order and at the correct times – much like playing a melody at the piano. This task allowed us to calculate the variance of the responses at different time points using data from the same trials. Our results show that while “standard” Weber’s law is clearly violated, variance does increase as a function of time squared, as expected according to the generalized form of Weber’s law – which separates the source of variance into time-dependent and time-independent components. Over the course of learning, both the time-independent variance and the coefficient of the time-dependent term decrease. Our analyses also suggest that timing of sequential events does not rely on the resetting of an internal timer at each event. We describe and interpret our results in the context of computer simulations that capture some of our psychophysical findings. Specifically, we show that continuous timing, as opposed to “reset” timing, is consistent with “population clock” models in which timing emerges from the internal dynamics of recurrent neural networks.

Ensemble neural activity of the frontal cortical basal ganglia system predicts reaction time task performance in rats

The question pursued in this study was when neural activity appears in the cortico-basal ganglia system that could predict alternate behavioral responses in a reaction time (RT) task. In this protocol, rats first performed a nose poke to initiate a trial, depressed a lever when presented, and then released the lever after a tone cue. Multiple-channel, single-unit recordings (up to 62 units) were obtained simultaneously from the prefrontal cortex, the dorsal medial striatum, the globus pallidus, and the substantia nigra pars reticulata in a single rat during a session. Results indicated that (1) global alterations of neural activity appeared in clusters, which was associated with different behavioral components and observed in each of the targeted areas; (2) small independent subsets of neurons responded differently between error (lever was released before tone presentation) and correct trials (lever was released within 0.5s after tone onset) during these behavioral episodes; (3) significant correlations between RTs and single units activities were found in the early preparation phases of the task. The results reveal that complex early preparatory activity exists several seconds before the final movements in a RT task, which may determine executive functions leading to rapid decoding of alternate behavioral performances.

Cell assembly sequences arising from spike threshold adaptation keep track of time in the hippocampus

Hippocampal neurons can display reliable and long-lasting sequences of transient firing patterns, even in the absence of changing external stimuli. We suggest that time-keeping is an important function of these sequences, and propose a network mechanism for their generation. We show that sequences of neuronal assemblies recorded from rat hippocampal CA1 pyramidal cells can reliably predict elapsed time (15-20 s) during wheel running with a precision of 0.5 s. In addition, we demonstrate the generation of multiple reliable, long-lasting sequences in a recurrent network model. These sequences are generated in the presence of noisy, unstructured inputs to the network, mimicking stationary sensory input. Identical initial conditions generate similar sequences, whereas different initial conditions give rise to distinct sequences. The key ingredients responsible for sequence generation in the model are threshold-adaptation and a Mexican-hat-like pattern of connectivity among pyramidal cells. This pattern may arise from recurrent systems such as the hippocampal CA3 region or the entorhinal cortex. We hypothesize that mechanisms that evolved for spatial navigation also support tracking of elapsed time in behaviorally relevant contexts.

Motor Planning under Unpredictable Reward: Modulations of Movement Vigor and Primate Striatum Activity

Although reward probability is an important factor that shapes animal's behavior, it is not well understood how the brain translates reward expectation into the vigor of movement [reaction time (RT) and speed]. To address this question, we trained two monkeys in a RT task that required wrist movements in response to vibrotactile and visual stimuli, with a variable reward schedule. Correct performance was rewarded in 75% of the trials. Monkeys were certain that they would be rewarded only in the trials immediately following withheld rewards. In these trials, the animals responded sooner and moved faster. Single-unit recordings from the dorsal striatum revealed modulations in neural firing that reflected changes in movement vigor. First, in the trials with certain rewards, striatal neurons modulated their firing rates earlier. Second, magnitudes of changes in neuronal firing rates depended on whether or not monkeys were certain about the reward. Third, these modulations depended on the sensory modality of the cue (visual vs. vibratory) and/or movement direction (flexions vs. extensions). We conclude that dorsal striatum may be a part of the mechanism responsible for the modulation of movement vigor in response to changes of reward predictability.

Population clocks: motor timing with neural dynamics

An understanding of sensory and motor processing will require elucidation of the mechanisms by which the brain tells time. Open questions relate to whether timing relies on dedicated or intrinsic mechanisms and whether distinct mechanisms underlie timing across scales and modalities. Although experimental and theoretical studies support the notion that neural circuits are intrinsically capable of sensory timing on short scales, few general models of motor timing have been proposed. For one class of models, population clocks, it is proposed that time is encoded in the time-varying patterns of activity of a population of neurons. We argue that population clocks emerge from the internal dynamics of recurrently connected networks, are biologically realistic and account for many aspects of motor timing.

A neural representation of sequential states within an instructed task

In the study of the neural basis of sensorimotor transformations, it has become clear that the brain does not always wait to sense external events and afterward select the appropriate responses. If there are predictable regularities in the environment, the brain begins to anticipate the timing of instructional cues and the signals to execute a response, revealing an internal representation of the sequential behavioral states of the task being performed. To investigate neural mechanisms that could represent the sequential states of a task, we recorded neural activity from two oculomotor structures implicated in behavioral timing--the supplementary eye fields (SEF) and the lateral intraparietal area (LIP)--while rhesus monkeys performed a memory-guided saccade task. The neurons of the SEF were found to collectively encode the progression of the task with individual neurons predicting and/or detecting states or transitions between states. LIP neurons, while also encoding information about the current temporal interval, were limited with respect to SEF neurons in two ways. First, LIP neurons tended to be active when the monkey was planning a saccade but not in the precue or intertrial intervals, whereas SEF neurons tended to have activity modulation in all intervals. Second, the LIP neurons were more likely to be spatially tuned than SEF neurons. SEF neurons also show anticipatory activity. The state-selective and anticipatory responses of SEF neurons support two complementary models of behavioral timing, state dependent and accumulator models, and suggest that each model describes a contribution SEF makes to timing at different temporal resolutions.

Accumulation of neural activity in the posterior insula encodes the passage of time

A number of studies have examined the perception of time with durations ranging from milliseconds to a few seconds, however the neural basis of these processes are still poorly understood and the neural substrates underlying the perception of multiple-second intervals are unknown. Here we present evidence of neural systems activity in circumscribed areas of the human brain involved in the encoding of intervals with durations of 9 and 18s in a temporal reproduction task using event-related functional magnetic resonance imaging (fMRI). During the encoding there was greater activation in more posterior parts of the medial frontal and insular cortex whereas the reproduction phase involved more anterior parts of these brain structures. Intriguingly, activation curves over time show an accumulating pattern of neural activity, which peaks at the end of the interval within bilateral posterior insula and superior temporal cortex when individuals are presented with 9- and 18-s tone intervals. This is consistent with an accumulator-type activity, which encodes duration in the multiple seconds range. Given the close connection between the dorsal posterior insula and ascending internal body signals, we suggest that the accumulation of physiological changes in body states constitutes our experience of time. This is the first time that an accumulation function in the posterior insula is detected that might be correlated with the encoding of time intervals.

Response Competition in the Primary Motor Cortex: Corticospinal Excitability Reflects Response Replacement During Simple Decisions

It has been suggested that, during decisions about actions, multiple options are initially specified in parallel and then gradually eliminated in a competition for overt execution. To further test this hypothesis, we studied the modulation of human corticospinal excitability during the reaction time of the Eriksen flanker task. In the task, subjects responded with finger flexion or extension to a central arrow while ignoring congruent or incongruent flanker arrows. Single-pulse transcranial magnetic stimulation (TMS) was applied over primary motor cortex (M1) at one of five different latencies after stimulus onset, and motor-evoked potentials (MEPs) were measured in the contralateral index finger. During the control (no flankers) and congruent conditions, MEP size in the agonist increased gradually over the course of reaction time, indicating an increase in corticospinal excitability. Conversely, when the same muscle acted as an antagonist, MEP size decreased, suggesting inhibition. Critically, in the incongruent condition, MEPs briefly increased in the muscle corresponding to an initial default response to the flanker arrows and were later replaced by MEPs corresponding to the correct response to the central arrow. Finally, we found that the gradually growing MEPs for the three conditions reached a constant maximum level just before movement initiation. We propose that this dynamic modulation in corticospinal excitability reflects the competition process, leading to the selection of one response and the rejection of the other. Our results suggest that response competition influences activity in primary motor cortex and that its timing directly influences motor output latency.

Attention regulates the plasticity of multisensory timing

Evidence suggests than human time perception is likely to reflect an ensemble of recent temporal experience. For example, prolonged exposure to consistent temporal patterns can adaptively realign the perception of event order, both within and between sensory modalities (e.g. Fujisaki et al., 2004 Nat. Neurosci., 7, 773-778). In addition, the observation that 'a watched pot never boils' serves to illustrate the fact that dynamic shifts in our attentional state can also produce marked distortions in our temporal estimates. In the current study we provide evidence for a hitherto unknown link between adaptation, temporal perception and our attentional state. We show that our ability to use recent sensory history as a perceptual baseline for ongoing temporal judgments is subject to striking top-down modulation via shifts in the observer's selective attention. Specifically, attending to the temporal structure of asynchronous auditory and visual adapting stimuli generates a substantial increase in the temporal recalibration induced by these stimuli. We propose a conceptual framework accounting for our findings whereby attention modulates the perceived salience of temporal patterns. This heightened salience allows the formation of audiovisual perceptual 'objects', defined solely by their temporal structure. Repeated exposure to these objects induces high-level pattern adaptation effects, akin to those found in visual and auditory domains (e.g. Leopold & Bondar (2005) Fitting the Mind to the World: Adaptation and Aftereffects in High-Level Vision. Oxford University Press, Oxford, 189-211; Schweinberger et al. (2008) Curr. Biol., 18, 684-688).

Long-term asynchronous decoding of arm motion using electrocorticographic signals in monkeys

Brain-machine interfaces (BMIs) employ the electrical activity generated by cortical neurons directly for controlling external devices and have been conceived as a means for restoring human cognitive or sensory-motor functions. The dominant approach in BMI research has been to decode motor variables based on single-unit activity (SUA). Unfortunately, this approach suffers from poor long-term stability and daily recalibration is normally required to maintain reliable performance. A possible alternative is BMIs based on electrocorticograms (ECoGs), which measure population activity and may provide more durable and stable recording. However, the level of long-term stability that ECoG-based decoding can offer remains unclear. Here we propose a novel ECoG-based decoding paradigm and show that we have successfully decoded hand positions and arm joint angles during an asynchronous food-reaching task in monkeys when explicit cues prompting the onset of movement were not required. Performance using our ECoG-based decoder was comparable to existing SUA-based systems while evincing far superior stability and durability. In addition, the same decoder could be used for months without any drift in accuracy or recalibration. These results were achieved by incorporating the spatio-spectro-temporal integration of activity across multiple cortical areas to compensate for the lower fidelity of ECoG signals. These results show the feasibility of high-performance, chronic and versatile ECoG-based neuroprosthetic devices for real-life applications. This new method provides a stable platform for investigating cortical correlates for understanding motor control, sensory perception, and high-level cognitive processes.