Machine Learning for Neural Decoding

Revealing unexpected complex encoding but simple decoding mechanisms in motor cortex via separating behaviorally relevant neural signals

In motor cortex, behaviorally relevant neural responses are entangled with irrelevant signals, which complicates the study of encoding and decoding mechanisms. It remains unclear whether behaviorally irrelevant signals could conceal some critical truth. One solution is to accurately separate behaviorally relevant and irrelevant signals at both single-neuron and single-trial levels, but this approach remains elusive due to the unknown ground truth of behaviorally relevant signals. Therefore, we propose a framework to define, extract, and validate behaviorally relevant signals. Analyzing separated signals in three monkeys performing different reaching tasks, we found neural responses previously considered to contain little information actually encode rich behavioral information in complex nonlinear ways. These responses are critical for neuronal redundancy and reveal movement behaviors occupy a higher-dimensional neural space than previously expected. Surprisingly, when incorporating often-ignored neural dimensions, behaviorally relevant signals can be decoded linearly with comparable performance to nonlinear decoding, suggesting linear readout may be performed in motor cortex. Our findings prompt that separating behaviorally relevant signals may help uncover more hidden cortical mechanisms.

Entorhinal cortex represents task-relevant remote locations independent of CA1

Neurons can collectively represent the current sensory experience while an animal is exploring its environment or remote experiences while the animal is immobile. These remote representations can reflect learned associations1-3 and be required for learning4. Neurons in the medial entorhinal cortex (MEC) reflect the animal's current location during movement5, but little is known about what MEC neurons collectively represent during immobility. Here, we recorded thousands of neurons in superficial MEC and dorsal CA1 as mice learned to associate two pairs of rewarded locations. We found that during immobility, the MEC neural population frequently represented positions far from the animal's location, which we defined as 'non-local coding'. Cells with spatial firing fields at remote locations drove non-local coding, even as cells representing the current position remained active. While MEC non-local coding has been reported during sharp-wave ripples in downstream CA16, we observed non-local coding more often outside of ripples. In fact, CA1 activity was less coordinated with MEC during non-local coding. We further observed that non-local coding was pertinent to the task, as MEC preferentially represented remote task-relevant locations at appropriate times, while rarely representing task-irrelevant locations. Together, this work raises the possibility that MEC non-local coding could strengthen associations between locations independently from CA1.

Temporal resolution of spike coding in feedforward networks with signal convergence and divergence

Convergent and divergent structures in the networks that make up biological brains are found universally across many species and brain regions at various scales. Neurons in these networks fire action potentials, or "spikes", whose precise timing is becoming increasingly appreciated as large sources of information about both sensory input and motor output. While previous theories on coding in convergent and divergent networks have largely neglected the role of precise spike timing, our model and analyses place this aspect at the forefront. For a suite of stimuli with different timescales, we demonstrate that structural bottlenecks (small groups of neurons) post-synaptic to network convergence have a stronger preference for spike timing codes than expansion layers created by structural divergence. Additionally, we found that a simple network model with similar convergence and divergence ratios to those found experimentally can reproduce the relative contribution of spike timing information about motor output in the hawkmoth Manduca sexta. Our simulations and analyses suggest a relationship between the level of convergent/divergent structure present in a feedforward network and the loss of stimulus information encoded by its population spike trains as their temporal resolution decreases, which could be confirmed experimentally across diverse neural systems in future studies. We further show that this relationship can be generalized across different models and measures, implying a potentially fundamental link between network structure and coding strategy using spikes.

Similar neural states, but dissimilar decoding patterns for motor control in parietal cortex

Discrete neural states are associated with reaching movements across the fronto-parietal network. Here, the Hidden Markov Model (HMM) applied to spiking activity of the somato-motor parietal area PE revealed a sequence of states similar to those of the contiguous visuomotor areas PEc and V6A. Using a coupled clustering and decoding approach, we proved that these neural states carried spatiotemporal information regarding behaviour in all three posterior parietal areas. However, comparing decoding accuracy, PE was less informative than V6A and PEc. In addition, V6A outperformed PEc in target inference, indicating functional differences among the parietal areas. To check the consistency of these differences, we used both a supervised and an unsupervised variant of the HMM, and compared its performance with two more common classifiers, Support Vector Machine and Long-Short Term Memory. The differences in decoding between areas were invariant to the algorithm used, still showing the dissimilarities found with HMM, thus indicating that these dissimilarities are intrinsic in the information encoded by parietal neurons. These results highlight that, when decoding from the parietal cortex, for example, in brain machine interface implementations, attention should be paid in selecting the most suitable source of neural signals, given the great heterogeneity of this cortical sector.

Increasing Robustness of Intracortical Brain-Computer Interfaces for Recording Condition Changes via Data Augmentation

BACKGROUND AND OBJECTIVE

Intracortical brain-computer interfaces (iBCIs) aim to help paralyzed individuals restore their motor functions by decoding neural activity into intended movement. However, changes in neural recording conditions hinder the decoding performance of iBCIs, mainly because the neural-to-kinematic mappings shift. Conventional approaches involve either training the neural decoders using large datasets before deploying the iBCI or conducting frequent calibrations during its operation. However, collecting data for extended periods can cause user fatigue, negatively impacting the quality and consistency of neural signals. Furthermore, frequent calibration imposes a substantial computational load.

METHODS

This study proposes a novel approach to increase iBCIs' robustness against changing recording conditions. The approach uses three neural augmentation operators to generate augmented neural activity that mimics common recording conditions. Then, contrastive learning is used to learn latent factors by maximizing the similarity between the augmented neural activities. The learned factors are expected to remain stable despite varying recording conditions and maintain a consistent correlation with the intended movement.

RESULTS

Experimental results demonstrate that the proposed iBCI outperformed the state-of-the-art iBCIs and was robust to changing recording conditions across days for long-term use on one publicly available nonhuman primate dataset. It achieved satisfactory offline decoding performance, even when a large training dataset was unavailable.

CONCLUSIONS

This study paves the way for reducing the need for frequent calibration of iBCIs and collecting a large amount of annotated training data. Potential future works aim to improve offline decoding performance with an ultra-small training dataset and improve the iBCIs' robustness to severely disabled electrodes.

Revealing neural dynamical structure of C. elegans with deep learning

Caenorhabditis elegans serves as a common model for investigating neural dynamics and functions of biological neural networks. Data-driven approaches have been employed in reconstructing neural dynamics. However, challenges remain regarding the curse of high-dimensionality and stochasticity in realistic systems. In this study, we develop a deep neural network (DNN) approach to reconstruct the neural dynamics of C. elegans and study neural mechanisms for locomotion. Our model identifies two limit cycles in the neural activity space: one underpins basic pirouette behavior, essential for navigation, and the other introduces extra Ω turns. The combination of two limit cycles elucidates predominant locomotion patterns in neural imaging data. The corresponding energy landscape explains the switching strategies between two limit cycles, quantitatively, and provides testable predictions on neural functions and circuit roles. Our work provides a general approach to study neural dynamics by combining imaging data and stochastic modeling.

Neural decoding of inferior colliculus multiunit activity for sound category identification with temporal correlation and transfer learning

Natural sounds are easily perceived and identified by humans and animals. Despite this, the neural transformations that enable sound perception remain largely unknown. It is thought that the temporal characteristics of sounds may be reflected in auditory assembly responses at the inferior colliculus (IC) and which may play an important role in identification of natural sounds. In our study, natural sounds will be predicted from multi-unit activity (MUA) signals collected in the IC. Data is obtained from an international platform publicly accessible. The temporal correlation values of the MUA signals are converted into images. We used two different segment sizes and with a denoising method, we generated four subsets for the classification. Using pre-trained convolutional neural networks (CNNs), features of the images were extracted and the type of heard sound was classified. For this, we applied transfer learning from Alexnet, Googlenet and Squeezenet CNNs. The classifiers support vector machines (SVM), k-nearest neighbour (KNN), Naive Bayes and Ensemble were used. The accuracy, sensitivity, specificity, precision and F1 score were measured as evaluation parameters. By using all the tests and removing the noise, the accuracy improved significantly. These results will allow neuroscientists to make interesting conclusions.

Calibrating Bayesian Decoders of Neural Spiking Activity

Accurately decoding external variables from observations of neural activity is a major challenge in systems neuroscience. Bayesian decoders, which provide probabilistic estimates, are some of the most widely used. Here we show how, in many common settings, the probabilistic predictions made by traditional Bayesian decoders are overconfident. That is, the estimates for the decoded stimulus or movement variables are more certain than they should be. We then show how Bayesian decoding with latent variables, taking account of low-dimensional shared variability in the observations, can improve calibration, although additional correction for overconfidence is still needed. Using data from males, we examine (1) decoding the direction of grating stimuli from spike recordings in the primary visual cortex in monkeys, (2) decoding movement direction from recordings in the primary motor cortex in monkeys, (3) decoding natural images from multiregion recordings in mice, and (4) decoding position from hippocampal recordings in rats. For each setting, we characterize the overconfidence, and we describe a possible method to correct miscalibration post hoc. Properly calibrated Bayesian decoders may alter theoretical results on probabilistic population coding and lead to brain-machine interfaces that more accurately reflect confidence levels when identifying external variables.

Nonlinear manifolds underlie neural population activity during behaviour

There is rich variety in the activity of single neurons recorded during behaviour. Yet, these diverse single neuron responses can be well described by relatively few patterns of neural co-modulation. The study of such low-dimensional structure of neural population activity has provided important insights into how the brain generates behaviour. Virtually all of these studies have used linear dimensionality reduction techniques to estimate these population-wide co-modulation patterns, constraining them to a flat "neural manifold". Here, we hypothesised that since neurons have nonlinear responses and make thousands of distributed and recurrent connections that likely amplify such nonlinearities, neural manifolds should be intrinsically nonlinear. Combining neural population recordings from monkey, mouse, and human motor cortex, and mouse striatum, we show that: 1) neural manifolds are intrinsically nonlinear; 2) their nonlinearity becomes more evident during complex tasks that require more varied activity patterns; and 3) manifold nonlinearity varies across architecturally distinct brain regions. Simulations using recurrent neural network models confirmed the proposed relationship between circuit connectivity and manifold nonlinearity, including the differences across architecturally distinct regions. Thus, neural manifolds underlying the generation of behaviour are inherently nonlinear, and properly accounting for such nonlinearities will be critical as neuroscientists move towards studying numerous brain regions involved in increasingly complex and naturalistic behaviours.

Convolutional neural networks reveal properties of reach-to-grasp encoding in posterior parietal cortex

Deep neural networks (DNNs) are widely adopted to decode motor states from both non-invasively and invasively recorded neural signals, e.g., for realizing brain-computer interfaces. However, the neurophysiological interpretation of how DNNs make the decision based on the input neural activity is limitedly addressed, especially when applied to invasively recorded data. This reduces decoder reliability and transparency, and prevents the exploitation of decoders to better comprehend motor neural encoding. Here, we adopted an explainable artificial intelligence approach - based on a convolutional neural network and an explanation technique - to reveal spatial and temporal neural properties of reach-to-grasping from single-neuron recordings of the posterior parietal area V6A. The network was able to accurately decode 5 different grip types, and the explanation technique automatically identified the cells and temporal samples that most influenced the network prediction. Grip encoding in V6A neurons already started at movement preparation, peaking during movement execution. A difference was found within V6A: dorsal V6A neurons progressively encoded more for increasingly advanced grips, while ventral V6A neurons for increasingly rudimentary grips, with both subareas following a linear trend between the amount of grip encoding and the level of grip skills. By revealing the elements of the neural activity most relevant for each grip with no a priori assumptions, our approach supports and advances current knowledge about reach-to-grasp encoding in V6A, and it may represent a general tool able to investigate neural correlates of motor or cognitive tasks (e.g., attention and memory tasks) from single-neuron recordings.

Motor cortex retains and reorients neural dynamics during motor imagery

The most prominent characteristic of motor cortex is its activation during movement execution, but it is also active when we simply imagine movements in the absence of actual motor output. Despite decades of behavioural and imaging studies, it is unknown how the specific activity patterns and temporal dynamics in motor cortex during covert motor imagery relate to those during motor execution. Here we recorded intracortical activity from the motor cortex of two people who retain some residual wrist function following incomplete spinal cord injury as they performed both actual and imagined isometric wrist extensions. We found that we could decompose the population activity into three orthogonal subspaces, where one was similarly active during both action and imagery, and the others were active only during a single task type-action or imagery. Although they inhabited orthogonal neural dimensions, the action-unique and imagery-unique subspaces contained a strikingly similar set of dynamic features. Our results suggest that during motor imagery, motor cortex maintains the same overall population dynamics as during execution by reorienting the components related to motor output and/or feedback into a unique, output-null imagery subspace.

Efficient encoding of aversive location by CA3 long-range projections

Memorizing locations that are harmful or dangerous is a key capability of all organisms and requires an integration of affective and spatial information. In mammals, the dorsal hippocampus mainly processes spatial information, while the intermediate to ventral hippocampal divisions receive affective information via the amygdala. However, how spatial and aversive information is integrated is currently unknown. To address this question, we recorded the activity of hippocampal long-range CA3 axons at single-axon resolution in mice forming an aversive spatial memory. We show that intermediate CA3 to dorsal CA3 (i-dCA3) projections rapidly overrepresent areas preceding the location of an aversive stimulus due to a spatially selective addition of new place-coding axons followed by spatially non-specific stabilization. This sequence significantly improves the encoding of location by the i-dCA3 axon population. These results suggest that i-dCA3 axons transmit a precise, denoised, and stable signal indicating imminent danger to the dorsal hippocampus.

Adaptation and learning as strategies to maximize reward in neurofeedback tasks

Introduction

Adaptation and learning have been observed to contribute to the acquisition of new motor skills and are used as strategies to cope with changing environments. However, it is hard to determine the relative contribution of each when executing goal directed motor tasks. This study explores the dynamics of neural activity during a center-out reaching task with continuous visual feedback under the influence of rotational perturbations.

Methods

Results for a brain-computer interface (BCI) task performed by two non-human primate (NHP) subjects are compared to simulations from a reinforcement learning agent performing an analogous task. We characterized baseline activity and compared it to the activity after rotational perturbations of different magnitudes were introduced. We employed principal component analysis (PCA) to analyze the spiking activity driving the cursor in the NHP BCI task as well as the activation of the neural network of the reinforcement learning agent.

Results and discussion

Our analyses reveal that both for the NHPs and the reinforcement learning agent, the task-relevant neural manifold is isomorphic with the task. However, for the NHPs the manifold is largely preserved for all rotational perturbations explored and adaptation of neural activity occurs within this manifold as rotations are compensated by reassignment of regions of the neural space in an angular pattern that cancels said rotations. In contrast, retraining the reinforcement learning agent to reach the targets after rotation results in substantial modifications of the underlying neural manifold. Our findings demonstrate that NHPs adapt their existing neural dynamic repertoire in a quantitatively precise manner to account for perturbations of different magnitudes and they do so in a way that obviates the need for extensive learning.

An asynchronous wireless network for capturing event-driven data from large populations of autonomous sensors

Networks of spatially distributed radiofrequency identification sensors could be used to collect data in wearable or implantable biomedical applications. However, the development of scalable networks remains challenging. Here we report a wireless radiofrequency network approach that can capture sparse event-driven data from large populations of spatially distributed autonomous microsensors. We use a spectrally efficient, low-error-rate asynchronous networking concept based on a code-division multiple-access method. We experimentally demonstrate the network performance of several dozen submillimetre-sized silicon microchips and complement this with large-scale in silico simulations. To test the notion that spike-based wireless communication can be matched with downstream sensor population analysis by neuromorphic computing techniques, we use a spiking neural network machine learning model to decode prerecorded open source data from eight thousand spiking neurons in the primate cortex for accurate prediction of hand movement in a cursor control task.

Temporally organized representations of reward and risk in the human brain

The value and uncertainty associated with choice alternatives constitute critical features relevant for decisions. However, the manner in which reward and risk representations are temporally organized in the brain remains elusive. Here we leverage the spatiotemporal precision of intracranial electroencephalography, along with a simple card game designed to elicit the unfolding computation of a set of reward and risk variables, to uncover this temporal organization. Reward outcome representations across wide-spread regions follow a sequential order along the anteroposterior axis of the brain. In contrast, expected value can be decoded from multiple regions at the same time, and error signals in both reward and risk domains reflect a mixture of sequential and parallel encoding. We further highlight the role of the anterior insula in generalizing between reward prediction error and risk prediction error codes. Together our results emphasize the importance of neural dynamics for understanding value-based decisions under uncertainty.

Artificial neural network for brain-machine interface consistently produces more naturalistic finger movements than linear methods

Brain-machine interfaces (BMI) aim to restore function to persons living with spinal cord injuries by 'decoding' neural signals into behavior. Recently, nonlinear BMI decoders have outperformed previous state-of-the-art linear decoders, but few studies have investigated what specific improvements these nonlinear approaches provide. In this study, we compare how temporally convolved feedforward neural networks (tcFNNs) and linear approaches predict individuated finger movements in open and closed-loop settings. We show that nonlinear decoders generate more naturalistic movements, producing distributions of velocities 85.3% closer to true hand control than linear decoders. Addressing concerns that neural networks may come to inconsistent solutions, we find that regularization techniques improve the consistency of tcFNN convergence by 194.6%, along with improving average performance, and training speed. Finally, we show that tcFNN can leverage training data from multiple task variations to improve generalization. The results of this study show that nonlinear methods produce more naturalistic movements and show potential for generalizing over less constrained tasks.

Teaser

A neural network decoder produces consistent naturalistic movements and shows potential for real-world generalization through task variations.

A topological deep learning framework for neural spike decoding

The brain's spatial orientation system uses different neuron ensembles to aid in environment-based navigation. Two of the ways brains encode spatial information are through head direction cells and grid cells. Brains use head direction cells to determine orientation, whereas grid cells consist of layers of decked neurons that overlay to provide environment-based navigation. These neurons fire in ensembles where several neurons fire at once to activate a single head direction or grid. We want to capture this firing structure and use it to decode head direction and animal location from head direction and grid cell activity. Understanding, representing, and decoding these neural structures require models that encompass higher-order connectivity, more than the one-dimensional connectivity that traditional graph-based models provide. To that end, in this work, we develop a topological deep learning framework for neural spike train decoding. Our framework combines unsupervised simplicial complex discovery with the power of deep learning via a new architecture we develop herein called a simplicial convolutional recurrent neural network. Simplicial complexes, topological spaces that use not only vertices and edges but also higher-dimensional objects, naturally generalize graphs and capture more than just pairwise relationships. Additionally, this approach does not require prior knowledge of the neural activity beyond spike counts, which removes the need for similarity measurements. The effectiveness and versatility of the simplicial convolutional neural network is demonstrated on head direction and trajectory prediction via head direction and grid cell datasets.

Neural correlates of object identity and reward outcome in the sensory cortical-hippocampal hierarchy: coding of motivational information in perirhinal cortex

Neural circuits support behavioral adaptations by integrating sensory and motor information with reward and error-driven learning signals, but it remains poorly understood how these signals are distributed across different levels of the corticohippocampal hierarchy. We trained rats on a multisensory object-recognition task and compared visual and tactile responses of simultaneously recorded neuronal ensembles in somatosensory cortex, secondary visual cortex, perirhinal cortex, and hippocampus. The sensory regions primarily represented unisensory information, whereas hippocampus was modulated by both vision and touch. Surprisingly, the sensory cortices and the hippocampus coded object-specific information, whereas the perirhinal cortex did not. Instead, perirhinal cortical neurons signaled trial outcome upon reward-based feedback. A majority of outcome-related perirhinal cells responded to a negative outcome (reward omission), whereas a minority of other cells coded positive outcome (reward delivery). Our results highlight a distributed neural coding of multisensory variables in the cortico-hippocampal hierarchy. Notably, the perirhinal cortex emerges as a crucial region for conveying motivational outcomes, whereas distinct functions related to object identity are observed in the sensory cortices and hippocampus.

Brain control of bimanual movement enabled by recurrent neural networks

Brain-computer interfaces have so far focused largely on enabling the control of a single effector, for example a single computer cursor or robotic arm. Restoring multi-effector motion could unlock greater functionality for people with paralysis (e.g., bimanual movement). However, it may prove challenging to decode the simultaneous motion of multiple effectors, as we recently found that a compositional neural code links movements across all limbs and that neural tuning changes nonlinearly during dual-effector motion. Here, we demonstrate the feasibility of high-quality bimanual control of two cursors via neural network (NN) decoders. Through simulations, we show that NNs leverage a neural 'laterality' dimension to distinguish between left and right-hand movements as neural tuning to both hands become increasingly correlated. In training recurrent neural networks (RNNs) for two-cursor control, we developed a method that alters the temporal structure of the training data by dilating/compressing it in time and re-ordering it, which we show helps RNNs successfully generalize to the online setting. With this method, we demonstrate that a person with paralysis can control two computer cursors simultaneously. Our results suggest that neural network decoders may be advantageous for multi-effector decoding, provided they are designed to transfer to the online setting.

A spiking neural network with continuous local learning for robust online brain machine interface

Objective.Spiking neural networks (SNNs) are powerful tools that are well suited for brain machine interfaces (BMI) due to their similarity to biological neural systems and computational efficiency. They have shown comparable accuracy to state-of-the-art methods, but current training methods require large amounts of memory, and they cannot be trained on a continuous input stream without pausing periodically to perform backpropagation. An ideal BMI should be capable training continuously without interruption to minimize disruption to the user and adapt to changing neural environments.Approach.We propose a continuous SNN weight update algorithm that can be trained to perform regression learning with no need for storing past spiking events in memory. As a result, the amount of memory needed for training is constant regardless of the input duration. We evaluate the accuracy of the network on recordings of neural data taken from the premotor cortex of a primate performing reaching tasks. Additionally, we evaluate the SNN in a simulated closed loop environment and observe its ability to adapt to sudden changes in the input neural structure.Main results.The continuous learning SNN achieves the same peak correlation (ρ=0.7) as existing SNN training methods when trained offline on real neural data while reducing the total memory usage by 92%. Additionally, it matches state-of-the-art accuracy in a closed loop environment, demonstrates adaptability when subjected to multiple types of neural input disruptions, and is capable of being trained online without any prior offline training.Significance.This work presents a neural decoding algorithm that can be trained rapidly in a closed loop setting. The algorithm increases the speed of acclimating a new user to the system and also can adapt to sudden changes in neural behavior with minimal disruption to the user.

NeuroDecodeR: a package for neural decoding in R

Neural decoding is a powerful method to analyze neural activity. However, the code needed to run a decoding analysis can be complex, which can present a barrier to using the method. In this paper we introduce a package that makes it easy to perform decoding analyses in the R programing language. We describe how the package is designed in a modular fashion which allows researchers to easily implement a range of different analyses. We also discuss how to format data to be able to use the package, and we give two examples of how to use the package to analyze real data. We believe that this package, combined with the rich data analysis ecosystem in R, will make it significantly easier for researchers to create reproducible decoding analyses, which should help increase the pace of neuroscience discoveries.

Dynamical flexible inference of nonlinear latent factors and structures in neural population activity

Modelling the spatiotemporal dynamics in the activity of neural populations while also enabling their flexible inference is hindered by the complexity and noisiness of neural observations. Here we show that the lower-dimensional nonlinear latent factors and latent structures can be computationally modelled in a manner that allows for flexible inference causally, non-causally and in the presence of missing neural observations. To enable flexible inference, we developed a neural network that separates the model into jointly trained manifold and dynamic latent factors such that nonlinearity is captured through the manifold factors and the dynamics can be modelled in tractable linear form on this nonlinear manifold. We show that the model, which we named 'DFINE' (for 'dynamical flexible inference for nonlinear embeddings') achieves flexible inference in simulations of nonlinear dynamics and across neural datasets representing a diversity of brain regions and behaviours. Compared with earlier neural-network models, DFINE enables flexible inference, better predicts neural activity and behaviour, and better captures the latent neural manifold structure. DFINE may advance the development of neurotechnology and investigations in neuroscience.

Multi-region calcium imaging in freely behaving mice with ultra-compact head-mounted fluorescence microscopes

To investigate the circuit-level neural mechanisms of behavior, simultaneous imaging of neuronal activity in multiple cortical and subcortical regions is highly desired. Miniature head-mounted microscopes offer the capability of calcium imaging in freely behaving animals. However, implanting multiple microscopes on a mouse brain remains challenging due to space constraints and the cumbersome weight of the equipment. Here, we present TINIscope, a Tightly Integrated Neuronal Imaging microscope optimized for electronic and opto-mechanical design. With its compact and lightweight design of 0.43 g, TINIscope enables unprecedented simultaneous imaging of behavior-relevant activity in up to four brain regions in mice. Proof-of-concept experiments with TINIscope recorded over 1000 neurons in four hippocampal subregions and revealed concurrent activity patterns spanning across these regions. Moreover, we explored potential multi-modal experimental designs by integrating additional modules for optogenetics, electrical stimulation or local field potential recordings. Overall, TINIscope represents a timely and indispensable tool for studying the brain-wide interregional coordination that underlies unrestrained behaviors.

Inferring population dynamics in macaque cortex

Objective.The proliferation of multi-unit cortical recordings over the last two decades, especially in macaques and during motor-control tasks, has generated interest in neural 'population dynamics': the time evolution of neural activity across a group of neurons working together. A good model of these dynamics should be able to infer the activity of unobserved neurons within the same population and of the observed neurons at future times. Accordingly, Pandarinath and colleagues have introduced a benchmark to evaluate models on these two (and related) criteria: four data sets, each consisting of firing rates from a population of neurons, recorded from macaque cortex during movement-related tasks.Approach.Since this is a discriminative-learning task, we hypothesize that general-purpose architectures based on recurrent neural networks (RNNs) trained with masking can outperform more 'bespoke' models. To capture long-distance dependencies without sacrificing the autoregressive bias of recurrent networks, we also propose a novel, hybrid architecture ('TERN') that augments the RNN with self-attention, as in transformer networks.Main results.Our RNNs outperform all published models on all four data sets in the benchmark. The hybrid architecture improves performance further still. Pure transformer models fail to achieve this level of performance, either in our work or that of other groups.Significance.We argue that the autoregressive bias imposed by RNNs is critical for achieving the highest levels of performance, and establish the state of the art on the neural latents benchmark. We conclude, however, by proposing that the benchmark be augmented with an alternative evaluation of latent dynamics that favors generative over discriminative models like the ones we propose in this report.

From monkeys to humans: observation-basedEMGbrain-computer interface decoders for humans with paralysis

Objective. Intracortical brain-computer interfaces (iBCIs) aim to enable individuals with paralysis to control the movement of virtual limbs and robotic arms. Because patients' paralysis prevents training a direct neural activity to limb movement decoder, most iBCIs rely on 'observation-based' decoding in which the patient watches a moving cursor while mentally envisioning making the movement. However, this reliance on observed target motion for decoder development precludes its application to the prediction of unobservable motor output like muscle activity. Here, we ask whether recordings of muscle activity from a surrogate individual performing the same movement as the iBCI patient can be used as target for an iBCI decoder.Approach. We test two possible approaches, each using data from a human iBCI user and a monkey, both performing similar motor actions. In one approach, we trained a decoder to predict the electromyographic (EMG) activity of a monkey from neural signals recorded from a human. We then contrast this to a second approach, based on the hypothesis that the low-dimensional 'latent' neural representations of motor behavior, known to be preserved across time for a given behavior, might also be preserved across individuals. We 'transferred' an EMG decoder trained solely on monkey data to the human iBCI user after using Canonical Correlation Analysis to align the human latent signals to those of the monkey.Main results. We found that both direct and transfer decoding approaches allowed accurate EMG predictions between two monkeys and from a monkey to a human.Significance. Our findings suggest that these latent representations of behavior are consistent across animals and even primate species. These methods are an important initial step in the development of iBCI decoders that generate EMG predictions that could serve as signals for a biomimetic decoder controlling motion and impedance of a prosthetic arm, or even muscle force directly through functional electrical stimulation.

Preserved neural dynamics across animals performing similar behaviour

Animals of the same species exhibit similar behaviours that are advantageously adapted to their body and environment. These behaviours are shaped at the species level by selection pressures over evolutionary timescales. Yet, it remains unclear how these common behavioural adaptations emerge from the idiosyncratic neural circuitry of each individual. The overall organization of neural circuits is preserved across individuals1 because of their common evolutionarily specified developmental programme2-4. Such organization at the circuit level may constrain neural activity5-8, leading to low-dimensional latent dynamics across the neural population9-11. Accordingly, here we suggested that the shared circuit-level constraints within a species would lead to suitably preserved latent dynamics across individuals. We analysed recordings of neural populations from monkey and mouse motor cortex to demonstrate that neural dynamics in individuals from the same species are surprisingly preserved when they perform similar behaviour. Neural population dynamics were also preserved when animals consciously planned future movements without overt behaviour12 and enabled the decoding of planned and ongoing movement across different individuals. Furthermore, we found that preserved neural dynamics extend beyond cortical regions to the dorsal striatum, an evolutionarily older structure13,14. Finally, we used neural network models to demonstrate that behavioural similarity is necessary but not sufficient for this preservation. We posit that these emergent dynamics result from evolutionary constraints on brain development and thus reflect fundamental properties of the neural basis of behaviour.

A Hyperflexible Electrode Array for Long-Term Recording and Decoding of Intraspinal Neuronal Activity

Neural interfaces for stable access to the spinal cord (SC) electrical activity can benefit patients with motor dysfunctions. Invasive high-density electrodes can directly extract signals from SC neuronal populations that can be used for the facilitation, adjustment, and reconstruction of motor actions. However, developing neural interfaces that can achieve high channel counts and long-term intraspinal recording remains technically challenging. Here, a biocompatible SC hyperflexible electrode array (SHEA) with an ultrathin structure that minimizes mechanical mismatch between the interface and SC tissue and enables stable single-unit recording for more than 2 months in mice is demonstrated. These results show that SHEA maintains stable impedance, signal-to-noise ratio, single-unit yield, and spike amplitude after implantation into mouse SC. Gait analysis and histology show that SHEA implantation induces negligible behavioral effects and Inflammation. Additionally, multi-unit signals recorded from the SC ventral horn can predict the mouse's movement trajectory with a high decoding coefficient of up to 0.95. Moreover, during step cycles, it is found that the neural trajectory of spikes and low-frequency local field potential (LFP) signal exhibits periodic geometry patterns. Thus, SHEA can offer an efficient and reliable SC neural interface for monitoring and potentially modulating SC neuronal activity associated with motor dysfunctions.

Algebraic approach to spike-time neural codes in the hippocampus

Although temporal coding through spike-time patterns has long been of interest in neuroscience, the specific structures that could be useful for spike-time codes remain highly unclear. Here, we introduce an analytical approach, using techniques from discrete mathematics, to study spike-time codes. As an initial example, we focus on the phenomenon of "phase precession" in the rodent hippocampus. During navigation and learning on a physical track, specific cells in a rodent's brain form a highly structured pattern relative to the oscillation of population activity in this region. Studies of phase precession largely focus on its role in precisely ordering spike times for synaptic plasticity, as the role of phase precession in memory formation is well established. Comparatively less attention has been paid to the fact that phase precession represents one of the best candidates for a spike-time neural code. The precise nature of this code remains an open question. Here, we derive an analytical expression for a function mapping points in physical space to complex-valued spikes by representing individual spike times as complex numbers. The properties of this function make explicit a specific relationship between past and future in spike patterns of the hippocampus. Importantly, this mathematical approach generalizes beyond the specific phenomenon studied here, providing a technique to study the neural codes within precise spike-time sequences found during sensory coding and motor behavior. We then introduce a spike-based decoding algorithm, based on this function, that successfully decodes a simulated animal's trajectory using only the animal's initial position and a pattern of spike times. This decoder is robust to noise in spike times and works on a timescale almost an order of magnitude shorter than typically used with decoders that work on average firing rate. These results illustrate the utility of a discrete approach, based on the structure and symmetries in spike patterns across finite sets of cells, to provide insight into the structure and function of neural systems.

Decoding hand kinetics and kinematics using somatosensory cortex activity in active and passive movement

Area 2 of the primary somatosensory cortex (S1), encodes proprioceptive information of limbs. Several studies investigated the encoding of movement parameters in this area. However, the single-trial decoding of these parameters, which can provide additional knowledge about the amount of information available in sub-regions of this area about instantaneous limb movement, has not been well investigated. We decoded kinematic and kinetic parameters of active and passive hand movement during center-out task using conventional and state-based decoders. Our results show that this area can be used to accurately decode position, velocity, force, moment, and joint angles of hand. Kinematics had higher accuracies compared to kinetics and active trials were decoded more accurately than passive trials. Although the state-based decoder outperformed the conventional decoder in the active task, it was the opposite in the passive task. These results can be used in intracortical micro-stimulation procedures to provide proprioceptive feedback to BCI subjects.

Neural decoding reveals specialized kinematic tuning after an abrupt cortical transition

The primary motor cortex (M1) exhibits a protracted period of development, including the development of a sensory representation long before motor outflow emerges. In rats, this representation is present by postnatal day (P) 8, when M1 activity is "discontinuous." Here, we ask how the representation changes upon the transition to "continuous" activity at P12. We use neural decoding to predict forelimb movements from M1 activity and show that a linear decoder effectively predicts limb movements at P8 but not at P12; instead, a nonlinear decoder better predicts limb movements at P12. The altered decoder performance reflects increased complexity and uniqueness of kinematic information in M1. We next show that M1's representation at P12 is more susceptible to "lesioning" of inputs and "transplanting" of M1's encoding scheme from one pup to another. Thus, the emergence of continuous M1 activity signals the developmental onset of more complex, informationally sparse, and individualized sensory representations.

Inferring cognitive state underlying conflict choices in verbal Stroop task using heterogeneous input discriminative-generative decoder model

Objective. The subthalamic nucleus (STN) of the basal ganglia interacts with the medial prefrontal cortex (mPFC) and shapes a control loop, specifically when the brain receives contradictory information from either different sensory systems or conflicting information from sensory inputs and prior knowledge that developed in the brain. Experimental studies demonstrated that significant increases in theta activities (2-8 Hz) in both the STN and mPFC as well as increased phase synchronization between mPFC and STN are prominent features of conflict processing. While these neural features reflect the importance of STN-mPFC circuitry in conflict processing, a low-dimensional representation of the mPFC-STN interaction referred to as a cognitive state, that links neural activities generated by these sub-regions to behavioral signals (e.g. the response time), remains to be identified.Approach. Here, we propose a new model, namely, the heterogeneous input discriminative-generative decoder (HI-DGD) model, to infer a cognitive state underlying decision-making based on neural activities (STN and mPFC) and behavioral signals (individuals' response time) recorded in ten Parkinson's disease (PD) patients while they performed a Stroop task. PD patients may have conflict processing which is quantitatively (may be qualitative in some) different from healthy populations.Main results. Using extensive synthetic and experimental data, we showed that the HI-DGD model can diffuse information from neural and behavioral data simultaneously and estimate cognitive states underlying conflict and non-conflict trials significantly better than traditional methods. Additionally, the HI-DGD model identified which neural features made significant contributions to conflict and non-conflict choices. Interestingly, the estimated features match well with those reported in experimental studies.Significance. Finally, we highlight the capability of the HI-DGD model in estimating a cognitive state from a single trial of observation, which makes it appropriate to be utilized in closed-loop neuromodulation systems.

Bypassing spike sorting: Density-based decoding using spike localization from dense multielectrode probes

Neural decoding and its applications to brain computer interfaces (BCI) are essential for understanding the association between neural activity and behavior. A prerequisite for many decoding approaches is spike sorting, the assignment of action potentials (spikes) to individual neurons. Current spike sorting algorithms, however, can be inaccurate and do not properly model uncertainty of spike assignments, therefore discarding information that could potentially improve decoding performance. Recent advances in high-density probes (e.g., Neuropixels) and computational methods now allow for extracting a rich set of spike features from unsorted data; these features can in turn be used to directly decode behavioral correlates. To this end, we propose a spike sorting-free decoding method that directly models the distribution of extracted spike features using a mixture of Gaussians (MoG) encoding the uncertainty of spike assignments, without aiming to solve the spike clustering problem explicitly. We allow the mixing proportion of the MoG to change over time in response to the behavior and develop variational inference methods to fit the resulting model and to perform decoding. We benchmark our method with an extensive suite of recordings from different animals and probe geometries, demonstrating that our proposed decoder can consistently outperform current methods based on thresholding (i.e. multi-unit activity) and spike sorting. Open source code is available at https://github.com/yzhang511/density_decoding.

Jump-GRS: a multi-phase approach to structured pruning of neural networks for neural decoding

Objective.Neural decoding, an important area of neural engineering, helps to link neural activity to behavior. Deep neural networks (DNNs), which are becoming increasingly popular in many application fields of machine learning, show promising performance in neural decoding compared to traditional neural decoding methods. Various neural decoding applications, such as brain computer interface applications, require both high decoding accuracy and real-time decoding speed. Pruning methods are used to produce compact DNN models for faster computational speed. Greedy inter-layer order with Random Selection (GRS) is a recently-designed structured pruning method that derives compact DNN models for calcium-imaging-based neural decoding. Although GRS has advantages in terms of detailed structure analysis and consideration of both learned information and model structure during the pruning process, the method is very computationally intensive, and is not feasible when large-scale DNN models need to be pruned within typical constraints on time and computational resources. Large-scale DNN models arise in neural decoding when large numbers of neurons are involved. In this paper, we build on GRS to develop a new structured pruning algorithm called jump GRS (JGRS) that is designed to efficiently compress large-scale DNN models.Approach.On top of GRS, JGRS implements a 'jump mechanism', which bypasses retraining intermediate models when model accuracy is relatively less sensitive to pruning operations. Design of the jump mechanism is motivated by identifying different phases of the structured pruning process, where retraining can be done infrequently in earlier phases without sacrificing accuracy. The jump mechanism helps to significantly speed up execution of the pruning process and greatly enhance its scalability. We compare the pruning performance and speed of JGRS and GRS with extensive experiments in the context of neural decoding.Main results.Our results demonstrate that JGRS provides significantly faster pruning speed compared to GRS, and at the same time, JGRS provides pruned models that are similarly compact as those generated by GRS.Significance.In our experiments, we demonstrate that JGRS achieves on average 9%-20% more compressed models compared to GRS with 2-8 times faster speed (less time required for pruning) across four different initial models on a relevant dataset for neural data analysis.

Neural Decoding for Intracortical Brain-Computer Interfaces

Brain-computer interfaces have revolutionized the field of neuroscience by providing a solution for paralyzed patients to control external devices and improve the quality of daily life. To accurately and stably control effectors, it is important for decoders to recognize an individual's motor intention from neural activity either by noninvasive or intracortical neural recording. Intracortical recording is an invasive way of measuring neural electrical activity with high temporal and spatial resolution. Herein, we review recent developments in neural signal decoding methods for intracortical brain-computer interfaces. These methods have achieved good performance in analyzing neural activity and controlling robots and prostheses in nonhuman primates and humans. For more complex paradigms in motor rehabilitation or other clinical applications, there remains more space for further improvements of decoders.

Class imbalance should not throw you off balance: Choosing the right classifiers and performance metrics for brain decoding with imbalanced data

Machine learning (ML) is increasingly used in cognitive, computational and clinical neuroscience. The reliable and efficient application of ML requires a sound understanding of its subtleties and limitations. Training ML models on datasets with imbalanced classes is a particularly common problem, and it can have severe consequences if not adequately addressed. With the neuroscience ML user in mind, this paper provides a didactic assessment of the class imbalance problem and illustrates its impact through systematic manipulation of data imbalance ratios in (i) simulated data and (ii) brain data recorded with electroencephalography (EEG), magnetoencephalography (MEG) and functional magnetic resonance imaging (fMRI). Our results illustrate how the widely-used Accuracy (Acc) metric, which measures the overall proportion of successful predictions, yields misleadingly high performances, as class imbalance increases. Because Acc weights the per-class ratios of correct predictions proportionally to class size, it largely disregards the performance on the minority class. A binary classification model that learns to systematically vote for the majority class will yield an artificially high decoding accuracy that directly reflects the imbalance between the two classes, rather than any genuine generalizable ability to discriminate between them. We show that other evaluation metrics such as the Area Under the Curve (AUC) of the Receiver Operating Characteristic (ROC), and the less common Balanced Accuracy (BAcc) metric - defined as the arithmetic mean between sensitivity and specificity, provide more reliable performance evaluations for imbalanced data. Our findings also highlight the robustness of Random Forest (RF), and the benefits of using stratified cross-validation and hyperprameter optimization to tackle data imbalance. Critically, for neuroscience ML applications that seek to minimize overall classification error, we recommend the routine use of BAcc, which in the specific case of balanced data is equivalent to using standard Acc, and readily extends to multi-class settings. Importantly, we present a list of recommendations for dealing with imbalanced data, as well as open-source code to allow the neuroscience community to replicate and extend our observations and explore alternative approaches to coping with imbalanced data.

Characterizing neural coding performance for populations of sensory neurons: comparing a weighted spike distance metrics to other analytical methods

The identity of sensory stimuli is encoded in the spatio-temporal patterns of responses of the encoding neural population. For stimuli to be discriminated reliably, differences in population responses must be accurately decoded by downstream networks. Several methods to compare patterns of responses have been used by neurophysiologists to characterize the accuracy of the sensory responses studied. Among the most widely used analyses, we note methods based on Euclidean distances or on spike metric distances. Methods based on artificial neural networks and machine learning that recognize and/or classify specific input patterns have also gained popularity. Here, we first compare these three strategies using datasets from three different model systems: the moth olfactory system, the electrosensory system of gymnotids, and leaky-integrate-and-fire (LIF) model responses. We show that the input-weighting procedure inherent to artificial neural networks allows the efficient extraction of information relevant to stimulus discrimination. To combine the convenience of methods such as spike metric distances but leverage the advantages of weighting the inputs, we propose a measure based on geometric distances where each dimension is weighted proportionally to how informative it is. We show that the result of this Weighted Euclidian Distance (WED) analysis performs as well or better than the artificial neural network we tested and outperforms the more traditional spike distance metrics. We applied information theoretic analysis to LIF responses and compared their encoding accuracy with the discrimination accuracy quantified through this WED analysis. We show a high degree of correlation between discrimination accuracy and information content, and that our weighting procedure allowed the efficient use of information present to perform the discrimination task. We argue that our proposed measure provides the flexibility and ease of use sought by neurophysiologists while providing a more powerful way to extract relevant information than more traditional methods.

Balancing Memorization and Generalization in RNNs for High Performance Brain-Machine Interfaces

Brain-machine interfaces (BMIs) can restore motor function to people with paralysis but are currently limited by the accuracy of real-time decoding algorithms. Recurrent neural networks (RNNs) using modern training techniques have shown promise in accurately predicting movements from neural signals but have yet to be rigorously evaluated against other decoding algorithms in a closed-loop setting. Here we compared RNNs to other neural network architectures in real-time, continuous decoding of finger movements using intracortical signals from nonhuman primates. Across one and two finger online tasks, LSTMs (a type of RNN) outperformed convolutional and transformer-based neural networks, averaging 18% higher throughput than the convolution network. On simplified tasks with a reduced movement set, RNN decoders were allowed to memorize movement patterns and matched able-bodied control. Performance gradually dropped as the number of distinct movements increased but did not go below fully continuous decoder performance. Finally, in a two-finger task where one degree-of-freedom had poor input signals, we recovered functional control using RNNs trained to act both like a movement classifier and continuous decoder. Our results suggest that RNNs can enable functional real-time BMI control by learning and generating accurate movement patterns.

Robust cortical encoding of 3D tongue shape during feeding in macaques

Dexterous tongue deformation underlies eating, drinking, and speaking. The orofacial sensorimotor cortex has been implicated in the control of coordinated tongue kinematics, but little is known about how the brain encodes-and ultimately drives-the tongue's 3D, soft-body deformation. Here we combine a biplanar x-ray video technology, multi-electrode cortical recordings, and machine-learning-based decoding to explore the cortical representation of lingual deformation. We trained long short-term memory (LSTM) neural networks to decode various aspects of intraoral tongue deformation from cortical activity during feeding in male Rhesus monkeys. We show that both lingual movements and complex lingual shapes across a range of feeding behaviors could be decoded with high accuracy, and that the distribution of deformation-related information across cortical regions was consistent with previous studies of the arm and hand.

Temporally organized representations of reward and risk in the human brain

The value and uncertainty associated with choice alternatives constitute critical features along which decisions are made. While the neural substrates supporting reward and risk processing have been investigated, the temporal organization by which these computations are encoded remains elusive. Here we leverage the high spatiotemporal precision of intracranial electroencephalography (iEEG) to uncover how representations of decision-related computations unfold in time. We present evidence of locally distributed representations of reward and risk variables that are temporally organized across multiple regions of interest. Reward outcome representations across wide-spread regions follow a temporally cascading order along the anteroposterior axis of the brain. In contrast, expected value can be decoded from multiple regions at the same time, and error signals in both reward and risk domains reflect a mixture of sequential and parallel encoding. We highlight the role of the anterior insula in generalizing between reward prediction error (RePE) and risk prediction error (RiPE), within which the encoding of RePE in the distributed iEEG signal predicts RiPE. Together our results emphasize the utility of uncovering temporal dynamics in the human brain for understanding how computational processes critical for value-based decisions under uncertainty unfold.

Deep learning applied to EEG source-data reveals both ventral and dorsal visual stream involvement in holistic processing of social stimuli

Perception of social stimuli (faces and bodies) relies on "holistic" (i.e., global) mechanisms, as supported by picture-plane inversion: perceiving inverted faces/bodies is harder than perceiving their upright counterpart. Albeit neuroimaging evidence suggested involvement of face-specific brain areas in holistic processing, their spatiotemporal dynamics and selectivity for social stimuli is still debated. Here, we investigate the spatiotemporal dynamics of holistic processing for faces, bodies and houses (adopted as control non-social category), by applying deep learning to high-density electroencephalographic signals (EEG) at source-level. Convolutional neural networks were trained to classify cortical EEG responses to stimulus orientation (upright/inverted), separately for each stimulus type (faces, bodies, houses), resulting to perform well above chance for faces and bodies, and close to chance for houses. By explaining network decision, the 150-200 ms time interval and few visual ventral-stream regions were identified as mostly relevant for discriminating face and body orientation (lateral occipital cortex, and for face only, precuneus cortex, fusiform and lingual gyri), together with two additional dorsal-stream areas (superior and inferior parietal cortices). Overall, the proposed approach is sensitive in detecting cortical activity underlying perceptual phenomena, and by maximally exploiting discriminant information contained in data, may reveal spatiotemporal features previously undisclosed, stimulating novel investigations.

Invasive Computational Psychiatry

Computational psychiatry, a relatively new yet prolific field that aims to understand psychiatric disorders with formal theories about the brain, has seen tremendous growth in the past decade. Despite initial excitement, actual progress made by computational psychiatry seems stagnant. Meanwhile, understanding of the human brain has benefited tremendously from recent progress in intracranial neuroscience. Specifically, invasive techniques such as stereotactic electroencephalography, electrocorticography, and deep brain stimulation have provided a unique opportunity to precisely measure and causally modulate neurophysiological activity in the living human brain. In this review, we summarize progress and drawbacks in both computational psychiatry and invasive electrophysiology and propose that their combination presents a highly promising new direction-invasive computational psychiatry. The value of this approach is at least twofold. First, it advances our mechanistic understanding of the neural computations of mental states by providing a spatiotemporally precise depiction of neural activity that is traditionally unattainable using noninvasive techniques with human subjects. Second, it offers a direct and immediate way to modulate brain states through stimulation of algorithmically defined neural regions and circuits (i.e., algorithmic targeting), thus providing both causal and therapeutic insights. We then present depression as a use case where the combination of computational and invasive approaches has already shown initial success. We conclude by outlining future directions as a road map for this exciting new field as well as presenting cautions about issues such as ethical concerns and generalizability of findings.

Rapid adaptation of brain-computer interfaces to new neuronal ensembles or participants via generative modelling

For brain-computer interfaces (BCIs), obtaining sufficient training data for algorithms that map neural signals onto actions can be difficult, expensive or even impossible. Here we report the development and use of a generative model-a model that synthesizes a virtually unlimited number of new data distributions from a learned data distribution-that learns mappings between hand kinematics and the associated neural spike trains. The generative spike-train synthesizer is trained on data from one recording session with a monkey performing a reaching task and can be rapidly adapted to new sessions or monkeys by using limited additional neural data. We show that the model can be adapted to synthesize new spike trains, accelerating the training and improving the generalization of BCI decoders. The approach is fully data-driven, and hence, applicable to applications of BCIs beyond motor control.

Calcium imaging and analysis of the jugular-nodose ganglia enables identification of distinct vagal sensory neuron subsets

Objective.Sensory nerves of the peripheral nervous system (PNS) transmit afferent signals from the body to the brain. These peripheral nerves are composed of distinct subsets of fibers and associated cell bodies, which reside in peripheral ganglia distributed throughout the viscera and along the spinal cord. The vagus nerve (cranial nerve X) is a complex polymodal nerve that transmits a wide array of sensory information, including signals related to mechanical, chemical, and noxious stimuli. To understand how stimuli applied to the vagus nerve are encoded by vagal sensory neurons in the jugular-nodose ganglia, we developed a framework for micro-endoscopic calcium imaging and analysis.Approach.We developed novel methods forin vivoimaging of the intact jugular-nodose ganglion using a miniature microscope (Miniscope) in transgenic mice with the genetically-encoded calcium indicator GCaMP6f. We adapted the Python-based analysis package Calcium Imaging Analysis (CaImAn) to process the resulting one-photon fluorescence data into calcium transients for subsequent analysis. Random forest classification was then used to identify specific types of neuronal responders.Results.We demonstrate that recordings from the jugular-nodose ganglia can be accomplished through careful surgical dissection and ganglia stabilization. Using a customized acquisition and analysis pipeline, we show that subsets of vagal sensory neurons respond to different chemical stimuli applied to the vagus nerve. Successful classification of the responses with a random forest model indicates that certain calcium transient features, such as amplitude and duration, are important for encoding these stimuli by sensory neurons.Significance.This experimental approach presents a new framework for investigating how individual vagal sensory neurons encode various stimuli on the vagus nerve. Our surgical and analytical approach can be applied to other PNS ganglia in rodents and other small animal species to elucidate previously unexplored roles for peripheral neurons in a diverse set of physiological functions.

Dynamical flexible inference of nonlinear latent structures in neural population activity

Inferring complex spatiotemporal dynamics in neural population activity is critical for investigating neural mechanisms and developing neurotechnology. These activity patterns are noisy observations of lower-dimensional latent factors and their nonlinear dynamical structure. A major unaddressed challenge is to model this nonlinear structure, but in a manner that allows for flexible inference, whether causally, non-causally, or in the presence of missing neural observations. We address this challenge by developing DFINE, a new neural network that separates the model into dynamic and manifold latent factors, such that the dynamics can be modeled in tractable form. We show that DFINE achieves flexible nonlinear inference across diverse behaviors and brain regions. Further, despite enabling flexible inference unlike prior neural network models of population activity, DFINE also better predicts the behavior and neural activity, and better captures the latent neural manifold structure. DFINE can both enhance future neurotechnology and facilitate investigations across diverse domains of neuroscience.

A parameter-optimization framework for neural decoding systems

Real-time neuron detection and neural activity extraction are critical components of real-time neural decoding. They are modeled effectively in dataflow graphs. However, these graphs and the components within them in general have many parameters, including hyper-parameters associated with machine learning sub-systems. The dataflow graph parameters induce a complex design space, where alternative configurations (design points) provide different trade-offs involving key operational metrics including accuracy and time-efficiency. In this paper, we propose a novel optimization framework that automatically configures the parameters in different neural decoders. The proposed optimization framework is evaluated in depth through two case studies. Significant performance improvement in terms of accuracy and efficiency is observed in both case studies compared to the manual parameter optimization that was associated with the published results of those case studies. Additionally, we investigate the application of efficient multi-threading strategies to speed-up the running time of our parameter optimization framework. Our proposed optimization framework enables efficient and effective estimation of parameters, which leads to more powerful neural decoding capabilities and allows researchers to experiment more easily with alternative decoding models.

Cortical Representation of Movement Across the Developmental Transition to Continuous Neural Activity

Primary motor cortex (M1) exhibits a protracted period of development that includes the establishment of a somatosensory map long before motor outflow emerges. In rats, the sensory representation is established by postnatal day (P) 8 when cortical activity is still "discontinuous." Here, we ask how the representation survives the sudden transition to noisy "continuous" activity at P12. Using neural decoding to predict forelimb movements based solely on M1 activity, we show that a linear decoder is sufficient to predict limb movements at P8, but not at P12; in contrast, a nonlinear decoder effectively predicts limb movements at P12. The change in decoder performance at P12 reflects an increase in both the complexity and uniqueness of kinematic information available in M1. We next show that the representation at P12 is more susceptible to the deleterious effects of "lesioning" inputs and to "transplanting" M1's encoding scheme from one pup to another. We conclude that the emergence of continuous cortical activity signals the developmental onset in M1 of more complex, informationally sparse, and individualized sensory representations.

Non-human primate models and systems for gait and neurophysiological analysis

Brain-computer interfaces (BCIs) have garnered extensive interest and become a groundbreaking technology to restore movement, tactile sense, and communication in patients. Prior to their use in human subjects, clinical BCIs require rigorous validation and verification (V&V). Non-human primates (NHPs) are often considered the ultimate and widely used animal model for neuroscience studies, including BCIs V&V, due to their proximity to humans. This literature review summarizes 94 NHP gait analysis studies until 1 June, 2022, including seven BCI-oriented studies. Due to technological limitations, most of these studies used wired neural recordings to access electrophysiological data. However, wireless neural recording systems for NHPs enabled neuroscience research in humans, and many on NHP locomotion, while posing numerous technical challenges, such as signal quality, data throughout, working distance, size, and power constraint, that have yet to be overcome. Besides neurological data, motion capture (MoCap) systems are usually required in BCI and gait studies to capture locomotion kinematics. However, current studies have exclusively relied on image processing-based MoCap systems, which have insufficient accuracy (error: ≥4° and 9 mm). While the role of the motor cortex during locomotion is still unclear and worth further exploration, future BCI and gait studies require simultaneous, high-speed, accurate neurophysiological, and movement measures. Therefore, the infrared MoCap system which has high accuracy and speed, together with a high spatiotemporal resolution neural recording system, may expand the scope and improve the quality of the motor and neurophysiological analysis in NHPs.

Overcoming the Domain Gap in Neural Action Representations

Relating behavior to brain activity in animals is a fundamental goal in neuroscience, with practical applications in building robust brain-machine interfaces. However, the domain gap between individuals is a major issue that prevents the training of general models that work on unlabeled subjects. Since 3D pose data can now be reliably extracted from multi-view video sequences without manual intervention, we propose to use it to guide the encoding of neural action representations together with a set of neural and behavioral augmentations exploiting the properties of microscopy imaging. To test our method, we collect a large dataset that features flies and their neural activity. To reduce the domain gap, during training, we mix features of neural and behavioral data across flies that seem to be performing similar actions. To show our method can generalize further neural modalities and other downstream tasks, we test our method on a human neural Electrocorticography dataset, and another RGB video data of human activities from different viewpoints. We believe our work will enable more robust neural decoding algorithms to be used in future brain-machine interfaces.

Partially binarized neural networks for efficient spike sorting

While brain-implantable neural spike sorting can be realized using efficient algorithms, the presence of noise may make it difficult to maintain high-peformance sorting using conventional techniques. In this article, we explore the use of partially binarized neural networks (PBNNs), to the best of our knowledge for the first time, for sorting of neural spike feature vectors. It is shown that compared to the waveform template-based methods, PBNNs offer robust spike sorting over various datasets and noise levels. The ASIC implementation of the PBNN-based spike sorting system in a standard 180-nm CMOS process is presented. The post place and route simulations results show that the synthesized PBNN consumes only 0.59 μ W of power from a 1.8 V supply while operating at 24 kHz and occupies 0.15 mm 2 of silicon area. It is shown that the designed PBNN-based spike sorting system not only offers comparable accuracy to the state-of-the-art spike sorting systems over various noise levels and datasets, it also occupies a smaller silicon area and consumes less power and energy. This makes PBNNs a viable alternative towards the implementation of brain-implantable spike sorting systems.

Integrating Statistical and Machine Learning Approaches for Neural Classification

Neurons can code for multiple variables simultaneously and neuroscientists are often interested in classifying neurons based on their receptive field properties. Statistical models provide powerful tools for determining the factors influencing neural spiking activity and classifying individual neurons. However, as neural recording technologies have advanced to produce simultaneous spiking data from massive populations, classical statistical methods often lack the computational efficiency required to handle such data. Machine learning (ML) approaches are known for enabling efficient large scale data analyses; however, they typically require massive training sets with balanced data, along with accurate labels to fit well. Additionally, model assessment and interpretation are often more challenging for ML than for classical statistical methods. To address these challenges, we develop an integrated framework, combining statistical modeling and machine learning approaches to identify the coding properties of neurons from large populations. In order to demonstrate this framework, we apply these methods to data from a population of neurons recorded from rat hippocampus to characterize the distribution of spatial receptive fields in this region.