Presynaptic inhibition selectively suppresses leg proprioception in behaving Drosophila

Controlling arms and legs requires feedback from proprioceptive sensory neurons that detect joint position and movement. Proprioceptive feedback must be tuned for different behavioral contexts, but the underlying circuit mechanisms remain poorly understood. Using calcium imaging in behaving Drosophila , we find that the axons of position-encoding leg proprioceptors are active across behaviors, whereas the axons of movement-encoding leg proprioceptors are suppressed during walking and grooming. Using connectomics, we identify a specific class of interneurons that provide GABAergic presynaptic inhibition to the axons of movement-encoding proprioceptors. The predominant synaptic inputs to these interneurons are descending neurons, suggesting they are driven by predictions of leg movement originating in the brain. Calcium imaging from both the interneurons and their descending inputs confirmed that their activity is correlated with self-generated but not passive leg movements. Overall, our findings elucidate a neural circuit for suppressing specific proprioceptive feedback signals during self-generated movements.

Sensorimotor delays constrain robust locomotion in a 3D kinematic model of fly walking

Walking animals must maintain stability in the presence of external perturbations, despite significant temporal delays in neural signaling and muscle actuation. Here, we develop a 3D kinematic model with a layered control architecture to investigate how sensorimotor delays constrain robustness of walking behavior in the fruit fly, Drosophila. Motivated by the anatomical architecture of insect locomotor control circuits, our model consists of three component layers: a neural network that generates realistic 3D joint kinematics for each leg, an optimal controller that executes the joint kinematics while accounting for delays, and an inter-leg coordinator. The model generates realistic simulated walking that matches real fly walking kinematics and sustains walking even when subjected to unexpected perturbations, generalizing beyond its training data. However, we found that the model's robustness to perturbations deteriorates when sensorimotor delay parameters exceed the physiological range. These results suggest that fly sensorimotor control circuits operate close to the temporal limit at which they can detect and respond to external perturbations. More broadly, we show how a modular, layered model architecture can be used to investigate physiological constraints on animal behavior.

Intraspecific Variation in the Placement of Campaniform Sensilla on the Wings of the Hawkmoth Manduca Sexta

Flight control requires active sensory feedback, and insects have many sensors that help them estimate their current locomotor state, including campaniform sensilla (CS), which are mechanoreceptors that sense strain resulting from deformation of the cuticle. CS on the wing detect bending and torsional forces encountered during flight, providing input to the flight feedback control system. During flight, wings experience complex spatio-temporal strain patterns. Because CS detect only local strain, their placement on the wing is presumably critical for determining the overall representation of wing deformation; however, how these sensilla are distributed across wings is largely unknown. Here, we test the hypothesis that CS are found in stereotyped locations across individuals of Manduca sexta, a hawkmoth. We found that although CS are consistently found on the same veins or in the same regions of the wings, their total number and distribution can vary extensively. This suggests that there is some robustness to variation in sensory feedback in the insect flight control system. The regions where CS are consistently found provide clues to their functional roles, although some patterns might be reflective of developmental processes. Collectively, our results on intraspecific variation in CS placement on insect wings will help reshape our thinking on the utility of mechanosensory feedback for insect flight control and guide further experimental and comparative studies.

Arousal as a universal embedding for spatiotemporal brain dynamics

Neural activity in awake organisms shows widespread and spatiotemporally diverse correlations with behavioral and physiological measurements. We propose that this covariation reflects in part the dynamics of a unified, arousal-related process that regulates brain-wide physiology on the timescale of seconds. Taken together with theoretical foundations in dynamical systems, this interpretation leads us to a surprising prediction: that a single, scalar measurement of arousal (e.g., pupil diameter) should suffice to reconstruct the continuous evolution of multimodal, spatiotemporal measurements of large-scale brain physiology. To test this hypothesis, we perform multimodal, cortex-wide optical imaging and behavioral monitoring in awake mice. We demonstrate that spatiotemporal measurements of neuronal calcium, metabolism, and blood-oxygen can be accurately and parsimoniously modeled from a low-dimensional state-space reconstructed from the time history of pupil diameter. Extending this framework to behavioral and electrophysiological measurements from the Allen Brain Observatory, we demonstrate the ability to integrate diverse experimental data into a unified generative model via mappings from an intrinsic arousal manifold. Our results support the hypothesis that spontaneous, spatially structured fluctuations in brain-wide physiology-widely interpreted to reflect regionally-specific neural communication-are in large part reflections of an arousal-related process. This enriched view of arousal dynamics has broad implications for interpreting observations of brain, body, and behavior as measured across modalities, contexts, and scales.

Mechanistic Hypotheses for Proprioceptive Sensing Within the Avian Lumbosacral Spinal Cord

Animals need to accurately sense changes in their body position to perform complex movements. It is increasingly clear that the vertebrate central nervous system contains a variety of cells capable of detecting body motion, in addition to the comparatively well-understood mechanosensory cells of the vestibular system and the peripheral proprioceptors. One such intriguing system is the lower spinal cord and column in birds, also known as the avian lumbosacral organ (LSO), which is thought to act as a set of balance sensors that allow birds to detect body movements separately from head movements detected by the vestibular system. Here, we take what is known about proprioceptive, mechanosensory spinal neurons in other vertebrates to explore hypotheses for how the LSO might sense mechanical information related to movement. Although the LSO is found only in birds, recent immunohistochemical studies of the avian LSO have hinted at similarities between cells in the LSO and the known spinal proprioceptors in other vertebrates. In addition to describing possible connections between avian spinal anatomy and recent findings on spinal proprioception as well as sensory and sensorimotor spinal networks, we also present some new data that suggest a role for sensory afferent peptides in LSO function. Thus, this perspective articulates a set of testable ideas on mechanisms of LSO function grounded in the emerging spinal proprioception scientific literature.

Neural subspaces of imagined movements in parietal cortex remain stable over several years in humans

A crucial goal in brain-machine interfacing is long-term stability of neural decoding performance, ideally without regular retraining. Here we demonstrate stable neural decoding over several years in two human participants, achieved by latent subspace alignment of multi-unit intracortical recordings in posterior parietal cortex. These results can be practically applied to significantly expand the longevity and generalizability of future movement decoding devices.

Intraspecific variation in the placement of campaniform sensilla on the wings of the hawkmoth Manduca sexta

Flight control requires active sensory feedback, and insects have many sensors that help them estimate their current locomotor state, including campaniform sensilla, which are mechanoreceptors that sense strain resulting from deformation of the cuticle. Campaniform sensilla on the wing detect bending and torsional forces encountered during flight, providing input to the flight feedback control system. During flight, wings experience complex spatio-temporal strain patterns. Because campaniform sensilla detect only local strain, their placement on the wing is presumably critical for determining the overall representation of wing deformation; however, how these sensilla are distributed across wings is largely unknown. Here, we test the hypothesis that campaniform sensilla are found in stereotyped locations across individuals of Manduca sexta, a hawkmoth. We found that although campaniform sensilla are consistently found on the same veins or in the same regions of the wings, their total number and distribution can vary extensively. This suggests that there is some robustness to variation in sensory feedback in the insect flight control system. The regions where campaniform sensilla are consistently found provide clues to their functional roles, although some patterns might be reflective of developmental processes. Collectively, our results on intraspecific variation in campaniform sensilla placement on insect wings will help reshape our thinking on the utility of mechanosensory feedback for insect flight control and guide further experimental and comparative studies.

Nonuniform structural properties of wings confer sensing advantages

Sensory feedback is essential to both animals and robotic systems for achieving coordinated, precise movements. Mechanosensory feedback, which provides information about body deformation, depends not only on the properties of sensors but also on the structure in which they are embedded. In insects, wing structure plays a particularly important role in flapping flight: in addition to generating aerodynamic forces, wings provide mechanosensory feedback necessary for guiding flight while undergoing dramatic deformations during each wingbeat. However, the role that wing structure plays in determining mechanosensory information is relatively unexplored. Insect wings exhibit characteristic stiffness gradients and are subject to both aerodynamic and structural damping. Here we examine how both of these properties impact sensory performance, using finite element analysis combined with sensor placement optimization approaches. We show that wings with nonuniform stiffness exhibit several advantages over uniform stiffness wings, resulting in higher accuracy of rotation detection and lower sensitivity to the placement of sensors on the wing. Moreover, we show that higher damping generally improves the accuracy with which body rotations can be detected. These results contribute to our understanding of the evolution of the nonuniform stiffness patterns in insect wings, as well as suggest design principles for robotic systems.

Emergent behaviour and neural dynamics in artificial agents tracking odour plumes

Tracking an odour plume to locate its source under variable wind and plume statistics is a complex task. Flying insects routinely accomplish such tracking, often over long distances, in pursuit of food or mates. Several aspects of this remarkable behaviour and its underlying neural circuitry have been studied experimentally. Here we take a complementary in silico approach to develop an integrated understanding of their behaviour and neural computations. Specifically, we train artificial recurrent neural network agents using deep reinforcement learning to locate the source of simulated odour plumes that mimic features of plumes in a turbulent flow. Interestingly, the agents' emergent behaviours resemble those of flying insects, and the recurrent neural networks learn to compute task-relevant variables with distinct dynamic structures in population activity. Our analyses put forward a testable behavioural hypothesis for tracking plumes in changing wind direction, and we provide key intuitions for memory requirements and neural dynamics in odour plume tracking.

Large-scale waves of activity in the neonatal mouse brain in vivo occur almost exclusively during sleep cycles

Spontaneous electrical activity plays major roles in the development of cortical circuitry. This activity can occur highly localized regions or can propagate over the entire cortex. Both types of activity coexist during early development. To investigate how different forms of spontaneous activity might be temporally segregated, we used wide-field trans-cranial calcium imaging over an entire hemisphere in P1-P8 mouse pups. We found that spontaneous waves of activity that propagate to cover the majority of the cortex (large-scale waves; LSWs) are generated at the end of the first postnatal week, along with several other forms of more localized activity. We further found that LSWs are segregated into sleep cycles. In contrast, cortical activity during wake states is more spatially restricted and the few large-scale forms of activity that occur during wake can be distinguished from LSWs in sleep based on their initiation in the motor cortex and their correlation with body movements. This change in functional cortical circuitry to a state that is permissive for large-scale activity may temporally segregate different forms of activity during critical stages when activity-dependent circuit development occurs over many spatial scales. Our data also suggest that LSWs in early development may be a functional precursor to slow sleep waves in the adult, which play critical roles in memory consolidation and synaptic rescaling.

Learning neural decoders without labels using multiple data streams

OBJECTIVE

Recent advances in neural decoding have accelerated the development of brain-computer interfaces aimed at assisting users with everyday tasks such as speaking, walking, and manipulating objects. However, current approaches for training neural decoders commonly require large quantities of labeled data, which can be laborious or infeasible to obtain in real-world settings. Alternatively, self-supervised models that share self-generated pseudo-labels between two data streams have shown exceptional performance on unlabeled audio and video data, but it remains unclear how well they extend to neural decoding.

APPROACH

We learn neural decoders without labels by leveraging multiple simultaneously recorded data streams, including neural, kinematic, and physiological signals. Specifically, we apply cross-modal, self-supervised deep clustering to train decoders that can classify movements from brain recordings. After training, we then isolate the decoders for each input data stream and compare the accuracy of decoders trained using cross-modal deep clustering against supervised and unimodal, self-supervised models.

MAIN RESULTS

We find that sharing pseudo-labels between two data streams during training substantially increases decoding performance compared to unimodal, self-supervised models, with accuracies approaching those of supervised decoders trained on labeled data. Next, we extend cross-modal decoder training to three or more modalities, achieving state-of-the-art neural decoding accuracy that matches or slightly exceeds the performance of supervised models. Significance: We demonstrate that cross-modal, self-supervised decoding can be applied to train neural decoders when few or no labels are available and extend the cross-modal framework to share information among three or more data streams, further improving self-supervised training.

AJILE12: Long-term naturalistic human intracranial neural recordings and pose

Understanding the neural basis of human movement in naturalistic scenarios is critical for expanding neuroscience research beyond constrained laboratory paradigms. Here, we describe our Annotated Joints in Long-term Electrocorticography for 12 human participants (AJILE12) dataset, the largest human neurobehavioral dataset that is publicly available; the dataset was recorded opportunistically during passive clinical epilepsy monitoring. AJILE12 includes synchronized intracranial neural recordings and upper body pose trajectories across 55 semi-continuous days of naturalistic movements, along with relevant metadata, including thousands of wrist movement events and annotated behavioral states. Neural recordings are available at 500 Hz from at least 64 electrodes per participant, for a total of 1280 hours. Pose trajectories at 9 upper-body keypoints were estimated from 118 million video frames. To facilitate data exploration and reuse, we have shared AJILE12 on The DANDI Archive in the Neurodata Without Borders (NWB) data standard and developed a browser-based dashboard.

Ensemble-SINDy: Robust sparse model discovery in the low-data, high-noise limit, with active learning and control

Sparse model identification enables the discovery of nonlinear dynamical systems purely from data; however, this approach is sensitive to noise, especially in the low-data limit. In this work, we leverage the statistical approach of bootstrap aggregating (bagging) to robustify the sparse identification of the nonlinear dynamics (SINDy) algorithm. First, an ensemble of SINDy models is identified from subsets of limited and noisy data. The aggregate model statistics are then used to produce inclusion probabilities of the candidate functions, which enables uncertainty quantification and probabilistic forecasts. We apply this ensemble-SINDy (E-SINDy) algorithm to several synthetic and real-world datasets and demonstrate substantial improvements to the accuracy and robustness of model discovery from extremely noisy and limited data. For example, E-SINDy uncovers partial differential equations models from data with more than twice as much measurement noise as has been previously reported. Similarly, E-SINDy learns the Lotka Volterra dynamics from remarkably limited data of yearly lynx and hare pelts collected from 1900 to 1920. E-SINDy is computationally efficient, with similar scaling as standard SINDy. Finally, we show that ensemble statistics from E-SINDy can be exploited for active learning and improved model predictive control.

Spatial distribution of campaniform sensilla mechanosensors on wings: form, function, and phylogeny

Insect wings serve two crucial functions in flight: propulsion and sensing. During flapping flight, complex spatiotemporal patterns of strain on the wing reflect mechanics, kinematics, and external perturbations; sensing wing deformation provides feedback necessary for flight control. Campaniform sensilla distributed across the wing transduce local strain fluctuations into neural signals, so their placement on the wing determines sensory information available to the insect. Thus, understanding the significance of these sensor locations will also reveal how sensing and wing movement are coupled. Here, we identify trends in wing campaniform sensilla placement across flying insects from the literature. We then discuss how these patterns can influence sensory encoding by wing mechanosensors. Finally, we propose combining a comparative approach on model insect clades with computational modeling, leveraging the spectacular natural diversity in wings to uncover biological principles of mechanosensory feedback in flight control.

Decoding Happiness from Neural and Video Recordings for Better Insight Into Emotional Processing in the Brain

Gaining a better understanding of which brain regions are responsible for emotional processing is crucial for the development of novel treatments for neuropsychiatric disorders. Current approaches rely on sparse assessments of subjects' emotional states, rarely reaching more than a hundred per patient. Additionally, data are usually obtained in a task solving scenario, possibly influencing their emotions by study design. Here, we utilize several days worth of near-continuous neural and video recordings of subjects in a naturalistic environment to predict the emotional state of happiness from neural data. We are able to obtain high-frequency and high-volume happiness labels for this task by first predicting happiness from video data in an intermediary step, achieving good results (F₁ = .75) and providing us with more than 6 million happiness assessments per patient, on average. We then utilize these labels for a classifier on neural data (F₁ = .71). Our findings provide a potential pathway for future work on emotional processing that circumvents the mentioned restrictions.

Structured time-delay models for dynamical systems with connections to Frenet-Serret frame

Time-delay embedding and dimensionality reduction are powerful techniques for discovering effective coordinate systems to represent the dynamics of physical systems. Recently, it has been shown that models identified by dynamic mode decomposition on time-delay coordinates provide linear representations of strongly nonlinear systems, in the so-called Hankel alternative view of Koopman (HAVOK) approach. Curiously, the resulting linear model has a matrix representation that is approximately antisymmetric and tridiagonal; for chaotic systems, there is an additional forcing term in the last component. In this paper, we establish a new theoretical connection between HAVOK and the Frenet-Serret frame from differential geometry, and also develop an improved algorithm to identify more stable and accurate models from less data. In particular, we show that the sub- and super-diagonal entries of the linear model correspond to the intrinsic curvatures in the Frenet-Serret frame. Based on this connection, we modify the algorithm to promote this antisymmetric structure, even in the noisy, low-data limit. We demonstrate this improved modelling procedure on data from several nonlinear synthetic and real-world examples.

Anipose: A toolkit for robust markerless 3D pose estimation

Quantifying movement is critical for understanding animal behavior. Advances in computer vision now enable markerless tracking from 2D video, but most animals move in 3D. Here, we introduce Anipose, an open-source toolkit for robust markerless 3D pose estimation. Anipose is built on the 2D tracking method DeepLabCut, so users can expand their existing experimental setups to obtain accurate 3D tracking. It consists of four components: (1) a 3D calibration module, (2) filters to resolve 2D tracking errors, (3) a triangulation module that integrates temporal and spatial regularization, and (4) a pipeline to structure processing of large numbers of videos. We evaluate Anipose on a calibration board as well as mice, flies, and humans. By analyzing 3D leg kinematics tracked with Anipose, we identify a key role for joint rotation in motor control of fly walking. To help users get started with 3D tracking, we provide tutorials and documentation at http://anipose.org/.

Learning to feed in the dark: how light level influences feeding in the hawkmoth Manduca sexta

Nocturnal insects like moths are essential for pollination, providing resilience to the diurnal pollination networks. Moths use both vision and mechanosensation to locate the nectary opening in the flowers with their proboscis. However, increased light levels due to artificial light at night (ALAN) pose a serious threat to nocturnal insects. Here, we examined how light levels influence the efficacy by which the crepuscular hawkmoth Manduca sexta locates the nectary. We used three-dimensional-printed artificial flowers fitted with motion sensors in the nectary and machine vision to track the motion of hovering moths under two light levels: 0.1 lux (moonlight) and 50 lux (dawn/dusk). We found that moths in higher light conditions took significantly longer to find the nectary, even with repeated visits to the same flower. In addition to taking longer, moths in higher light conditions hovered further from the flower during feeding. Increased light levels adversely affect learning and motor control in these animals.

Wing structure and neural encoding jointly determine sensing strategies in insect flight

Animals rely on sensory feedback to generate accurate, reliable movements. In many flying insects, strain-sensitive neurons on the wings provide rapid feedback that is critical for stable flight control. While the impacts of wing structure on aerodynamic performance have been widely studied, the impacts of wing structure on sensing are largely unexplored. In this paper, we show how the structural properties of the wing and encoding by mechanosensory neurons interact to jointly determine optimal sensing strategies and performance. Specifically, we examine how neural sensors can be placed effectively on a flapping wing to detect body rotation about different axes, using a computational wing model with varying flexural stiffness. A small set of mechanosensors, conveying strain information at key locations with a single action potential per wingbeat, enable accurate detection of body rotation. Optimal sensor locations are concentrated at either the wing base or the wing tip, and they transition sharply as a function of both wing stiffness and neural threshold. Moreover, the sensing strategy and performance is robust to both external disturbances and sensor loss. Typically, only five sensors are needed to achieve near-peak accuracy, with a single sensor often providing accuracy well above chance. Our results show that small-amplitude, dynamic signals can be extracted efficiently with spatially and temporally sparse sensors in the context of flight. The demonstrated interaction of wing structure and neural encoding properties points to the importance of understanding each in the context of their joint evolution.

In addition to generating forces for flight, insect wings also serve an important role as sensory structures, providing rapid feedback about wing bending that is used to stabilize flight. While much is known about how wing structure affects aerodynamic performance, the effects of wing structure on sensing remain unexplored. Using a computational model of a flapping wing, we examine how sensing strategies depend on wing stiffness and sensor properties. We show that body rotations can be accurately detected with a small number of sensors on the wing across a wide range of conditions. Optimal sensor locations are clustered at either the wing base or wing tip, depending on a combination of wing stiffness and sensor properties. Moreover, sensing performance is robust to multiple kinds of perturbations. Our work provides a basis for understanding how wing structure impacts incoming sensory information during flight.

A leg to stand on: computational models of proprioception

Dexterous motor control requires feedback from proprioceptors, internal mechanosensory neurons that sense the body's position and movement. An outstanding question in neuroscience is how diverse proprioceptive feedback signals contribute to flexible motor control. Genetic tools now enable targeted recording and perturbation of proprioceptive neurons in behaving animals; however, these experiments can be challenging to interpret, due to the tight coupling of proprioception and motor control. Here, we argue that understanding the role of proprioceptive feedback in controlling behavior will be aided by the development of multiscale models of sensorimotor loops. We review current phenomenological and structural models for proprioceptor encoding and discuss how they may be integrated with existing models of posture, movement, and body state estimation.

Go with the FLOW: visualizing spatiotemporal dynamics in optical widefield calcium imaging

Widefield calcium imaging has recently emerged as a powerful experimental technique to record coordinated large-scale brain activity. These measurements present a unique opportunity to characterize spatiotemporally coherent structures that underlie neural activity across many regions of the brain. In this work, we leverage analytic techniques from fluid dynamics to develop a visualization framework that highlights features of flow across the cortex, mapping wavefronts that may be correlated with behavioural events. First, we transform the time series of widefield calcium images into time-varying vector fields using optic flow. Next, we extract concise diagrams summarizing the dynamics, which we refer to as FLOW (flow lines in optical widefield imaging) portraits. These FLOW portraits provide an intuitive map of dynamic calcium activity, including regions of initiation and termination, as well as the direction and extent of activity spread. To extract these structures, we use the finite-time Lyapunov exponent technique developed to analyse time-varying manifolds in unsteady fluids. Importantly, our approach captures coherent structures that are poorly represented by traditional modal decomposition techniques. We demonstrate the application of FLOW portraits on three simple synthetic datasets and two widefield calcium imaging datasets, including cortical waves in the developing mouse and spontaneous cortical activity in an adult mouse.

Behavioral and Neural Variability of Naturalistic Arm Movements

Motor behaviors are central to many functions and dysfunctions of the brain, and understanding their neural basis has consequently been a major focus in neuroscience. However, most studies of motor behaviors have been restricted to artificial, repetitive paradigms, far removed from natural movements performed "in the wild." Here, we leveraged recent advances in machine learning and computer vision to analyze intracranial recordings from 12 human subjects during thousands of spontaneous, unstructured arm reach movements, observed over several days for each subject. These naturalistic movements elicited cortical spectral power patterns consistent with findings from controlled paradigms, but with considerable neural variability across subjects and events. We modeled interevent variability using 10 behavioral and environmental features; the most important features explaining this variability were reach angle and day of recording. Our work is among the first studies connecting behavioral and neural variability across cortex in humans during unstructured movements and contributes to our understanding of long-term naturalistic behavior.

Mining naturalistic human behaviors in long-term video and neural recordings

BACKGROUND

Recent technological advances in brain recording and machine learning algorithms are enabling the study of neural activity underlying spontaneous human behaviors, beyond the confines of cued, repeated trials. However, analyzing such unstructured data lacking a priori experimental design remains a significant challenge, especially when the data is multi-modal and long-term.

NEW METHOD

Here we describe an automated, behavior-first approach for analyzing simultaneously recorded long-term, naturalistic electrocorticography (ECoG) and behavior video data. We identify and characterize spontaneous human upper-limb movements by combining computer vision, discrete latent-variable modeling, and string pattern-matching on the video.

RESULTS

Our pipeline discovers and annotates over 40,000 instances of naturalistic arm movements in long term (7-9 day) behavioral videos, across 12 subjects. Analysis of the simultaneously recorded brain data reveals neural signatures of movement that corroborate previous findings. Our pipeline produces large training datasets for brain-computer interfacing applications, and we show decoding results from a movement initiation detection task.

COMPARISON WITH EXISTING METHODS

Spontaneous movements capture real-world neural and behavior variability that is missing from traditional cued tasks. Building beyond window-based movement detection metrics, our unsupervised discretization scheme produces a queryable pose representation, allowing localization of movements with finer temporal resolution.

CONCLUSIONS

Our work addresses the unique analytic challenges of studying naturalistic human behaviors and contributes methods that may generalize to other neural recording modalities beyond ECoG. We publish our curated dataset and believe that it will be a valuable resource for future studies of naturalistic movements.

Learning dominant physical processes with data-driven balance models

Throughout the history of science, physics-based modeling has relied on judiciously approximating observed dynamics as a balance between a few dominant processes. However, this traditional approach is mathematically cumbersome and only applies in asymptotic regimes where there is a strict separation of scales in the physics. Here, we automate and generalize this approach to non-asymptotic regimes by introducing the idea of an equation space, in which different local balances appear as distinct subspace clusters. Unsupervised learning can then automatically identify regions where groups of terms may be neglected. We show that our data-driven balance models successfully delineate dominant balance physics in a much richer class of systems. In particular, this approach uncovers key mechanistic models in turbulence, combustion, nonlinear optics, geophysical fluids, and neuroscience.

Generalized neural decoders for transfer learning across participants and recording modalities

OBJECTIVE

Advances in neural decoding have enabled brain-computer interfaces to perform increasingly complex and clinically-relevant tasks. However, such decoders are often tailored to specific participants, days, and recording sites, limiting their practical long-term usage. Therefore, a fundamental challenge is to develop neural decoders that can robustly train on pooled, multi-participant data and generalize to new participants.

APPROACH

We introduce a new decoder, HTNet, which uses a convolutional neural network with two innovations: (1) a Hilbert transform that computes spectral power at data-driven frequencies and (2) a layer that projects electrode-level data onto predefined brain regions. The projection layer critically enables applications with intracranial electrocorticography (ECoG), where electrode locations are not standardized and vary widely across participants. We trained HTNet to decode arm movements using pooled ECoG data from 11 of 12 participants and tested performance on unseen ECoG or electroencephalography (EEG) participants; these pretrained models were also subsequently fine-tuned to each test participant.

MAIN RESULTS

HTNet outperformed state-of-the-art decoders when tested on unseen participants, even when a different recording modality was used. By fine-tuning these generalized HTNet decoders, we achieved performance approaching the best tailored decoders with as few as 50 ECoG or 20 EEG events. We were also able to interpret HTNet's trained weights and demonstrate its ability to extract physiologically-relevant features.

SIGNIFICANCE

By generalizing to new participants and recording modalities, robustly handling variations in electrode placement, and allowing participant-specific fine-tuning with minimal data, HTNet is applicable across a broader range of neural decoding applications compared to current state-of-the-art decoders.

Inferring causal networks of dynamical systems through transient dynamics and perturbation

Inferring causal relations from time series measurements is an ill-posed mathematical problem, where typically an infinite number of potential solutions can reproduce the given data. We explore in depth a strategy to disambiguate between possible underlying causal networks by perturbing the network, where the forcings are either targeted or applied at random. The resulting transient dynamics provide the critical information necessary to infer causality. Two methods are shown to provide accurate causal reconstructions: Granger causality (GC) with perturbations, and our proposed perturbation cascade inference (PCI). Perturbed GC is capable of inferring smaller networks under low coupling strength regimes. Our proposed PCI method demonstrated consistently strong performance in inferring causal relations for small (2-5 node) and large (10-20 node) networks, with both linear and nonlinear dynamics. Thus, the ability to apply a large and diverse set of perturbations to the network is critical for successfully and accurately determining causal relations and disambiguating between various viable networks.

Numerical differentiation of noisy data: A unifying multi-objective optimization framework

Computing derivatives of noisy measurement data is ubiquitous in the physical, engineering, and biological sciences, and it is often a critical step in developing dynamic models or designing control. Unfortunately, the mathematical formulation of numerical differentiation is typically ill-posed, and researchers often resort to an ad hoc process for choosing one of many computational methods and its parameters. In this work, we take a principled approach and propose a multi-objective optimization framework for choosing parameters that minimize a loss function to balance the faithfulness and smoothness of the derivative estimate. Our framework has three significant advantages. First, the task of selecting multiple parameters is reduced to choosing a single hyper-parameter. Second, where ground-truth data is unknown, we provide a heuristic for selecting this hyper-parameter based on the power spectrum and temporal resolution of the data. Third, the optimal value of the hyper-parameter is consistent across different differentiation methods, thus our approach unifies vastly different numerical differentiation methods and facilitates unbiased comparison of their results. Finally, we provide an extensive open-source Python library pynumdiff to facilitate easy application to diverse datasets (https://github.com/florisvb/PyNumDiff).

The Balance Hypothesis for the Avian Lumbosacral Organ and an Exploration of Its Morphological Variation

Birds (Aves) exhibit exceptional and diverse locomotor behaviors, including the exquisite ability to balance on two feet. How birds so precisely control their movements may be partly explained by a set of intriguing modifications in their lower spine. These modifications are collectively known as the lumbosacral organ (LSO) and are found in the fused lumbosacral vertebrae called the synsacrum. They include a set of transverse canal-like recesses in the synsacrum that align with lateral lobes of the spinal cord, as well as a dorsal groove in the spinal cord that houses an egg-shaped glycogen body. Based on compelling but primarily observational data, the most recent functional hypotheses for the LSO consider it to be a secondary balance organ, in which the transverse canals are analogous to the semicircular canals of the inner ear. If correct, this hypothesis would reshape our understanding of avian locomotion, yet the LSO has been largely overlooked in the recent literature. Here, we review the current evidence for this hypothesis and then explore a possible relationship between the LSO and balance-intensive locomotor ecologies. Our comparative morphological dataset consists of micro-computed tomography (μ-CT) scans of synsacra from ecologically diverse species. We find that birds that perch tend to have more prominent transverse canals, suggesting that the LSO is useful for balance-intensive behaviors. We then identify the crucial outstanding questions about LSO structure and function. The LSO may be a key innovation that allows independent but coordinated motion of the head and the body, and a full understanding of its function and evolution will require multiple interdisciplinary research efforts.

Signal recovery from stimulation artifacts in intracranial recordings with dictionary learning

OBJECTIVE

Electrical stimulation of the human brain is commonly used for eliciting and inhibiting neural activity for clinical diagnostics, modifying abnormal neural circuit function for therapeutics, and interrogating cortical connectivity. However, recording electrical signals with concurrent stimulation results in dominant electrical artifacts that mask the neural signals of interest. Here we develop a method to reproducibly and robustly recover neural activity during concurrent stimulation. We concentrate on signal recovery across an array of electrodes without channel-wise fine-tuning of the algorithm. Our goal includes signal recovery with trains of stimulation pulses, since repeated, high-frequency pulses are often required to induce desired effects in both therapeutic and research domains. We have made all of our code and data publicly available.

APPROACH

We developed an algorithm that automatically detects templates of artifacts across many channels of recording, creating a dictionary of learned templates using unsupervised clustering. The artifact template that best matches each individual artifact pulse is subtracted to recover the underlying activity. To assess the success of our method, we focus on whether it extracts physiologically interpretable signals from real recordings.

MAIN RESULTS

We demonstrate our signal recovery approach on invasive electrophysiologic recordings from human subjects during stimulation. We show the recovery of meaningful neural signatures in both electrocorticographic (ECoG) arrays and deep brain stimulation (DBS) recordings. In addition, we compared cortical responses induced by the stimulation of primary somatosensory (S1) by natural peripheral touch, as well as motor cortex activity with and without concurrent S1 stimulation.

SIGNIFICANCE

Our work will enable future advances in neural engineering with simultaneous stimulation and recording.

Extracting Reproducible Time-Resolved Resting State Networks Using Dynamic Mode Decomposition

Resting state networks (RSNs) extracted from functional magnetic resonance imaging (fMRI) scans are believed to reflect the intrinsic organization and network structure of brain regions. Most traditional methods for computing RSNs typically assume these functional networks are static throughout the duration of a scan lasting 5-15 min. However, they are known to vary on timescales ranging from seconds to years; in addition, the dynamic properties of RSNs are affected in a wide variety of neurological disorders. Recently, there has been a proliferation of methods for characterizing RSN dynamics, yet it remains a challenge to extract reproducible time-resolved networks. In this paper, we develop a novel method based on dynamic mode decomposition (DMD) to extract networks from short windows of noisy, high-dimensional fMRI data, allowing RSNs from single scans to be resolved robustly at a temporal resolution of seconds. After validating the method on a synthetic dataset, we analyze data from 120 individuals from the Human Connectome Project and show that unsupervised clustering of DMD modes discovers RSNs at both the group (gDMD) and the single subject (sDMD) levels. The gDMD modes closely resemble canonical RSNs. Compared to established methods, sDMD modes capture individualized RSN structure that both better resembles the population RSN and better captures subject-level variation. We further leverage this time-resolved sDMD analysis to infer occupancy and transitions among RSNs with high reproducibility. This automated DMD-based method is a powerful tool to characterize spatial and temporal structures of RSNs in individual subjects.

Data-driven models in human neuroscience and neuroengineering

Discoveries in modern human neuroscience are increasingly driven by quantitative understanding of complex data. Data-intensive approaches to modeling have promise to dramatically advance our understanding of the brain and critically enable neuroengineering capabilities. In this review, we provide an accessible primer to modern modeling approaches and highlight recent data-driven discoveries in the domains of neuroimaging, single-neuron and neuronal population responses, and device neuroengineering. Further, we suggest that meaningful progress requires the community to tackle open challenges in the realms of model interpretability and generalizability, training pipelines of data-fluent human neuroscientists, and integrated consideration of data ethics.

Multistep model for predicting upper-limb 3D isometric force application from pre-movement electrocorticographic features

Neural correlates of movement planning onset and direction may be present in human electrocorticography in the signal dynamics of both motor and non-motor cortical regions. We use a three-stage model of jPCA reduced-rank hidden Markov model (jPCA-RR-HMM), regularized shrunken-centroid discriminant analysis (RDA), and LASSO regression to extract direction-sensitive planning information and movement onset in an upper-limb 3D isometric force task in a human subject. This mode achieves a relatively high true positive force-onset prediction rate of 60% within 250ms, and an above-chance 36% accuracy (17% chance) in predicting one of six planned 3D directions of isometric force using pre-movement signals. We also find direction-distinguishing information up to 400ms before force onset in the pre-movement signals, captured by electrodes placed over the limb-ipsilateral dorsal premotor regions. This approach can contribute to more accurate decoding of higher-level movement goals, at earlier timescales, and inform sensor placement. Our results also contribute to further understanding of the spatiotemporal features of human motor planning.

Chaos as an intermittently forced linear system

Understanding the interplay of order and disorder in chaos is a central challenge in modern quantitative science. Approximate linear representations of nonlinear dynamics have long been sought, driving considerable interest in Koopman theory. We present a universal, data-driven decomposition of chaos as an intermittently forced linear system. This work combines delay embedding and Koopman theory to decompose chaotic dynamics into a linear model in the leading delay coordinates with forcing by low-energy delay coordinates; this is called the Hankel alternative view of Koopman (HAVOK) analysis. This analysis is applied to the Lorenz system and real-world examples including Earth’s magnetic field reversal and measles outbreaks. In each case, forcing statistics are non-Gaussian, with long tails corresponding to rare intermittent forcing that precedes switching and bursting phenomena. The forcing activity demarcates coherent phase space regions where the dynamics are approximately linear from those that are strongly nonlinear.

The huge amount of data generated in fields like neuroscience or finance calls for effective strategies that mine data to reveal underlying dynamics. Here Brunton et al.develop a data-driven technique to analyze chaotic systems and predict their dynamics in terms of a forced linear model.

Unsupervised Decoding of Long-Term, Naturalistic Human Neural Recordings with Automated Video and Audio Annotations

Fully automated decoding of human activities and intentions from direct neural recordings is a tantalizing challenge in brain-computer interfacing. Implementing Brain Computer Interfaces (BCIs) outside carefully controlled experiments in laboratory settings requires adaptive and scalable strategies with minimal supervision. Here we describe an unsupervised approach to decoding neural states from naturalistic human brain recordings. We analyzed continuous, long-term electrocorticography (ECoG) data recorded over many days from the brain of subjects in a hospital room, with simultaneous audio and video recordings. We discovered coherent clusters in high-dimensional ECoG recordings using hierarchical clustering and automatically annotated them using speech and movement labels extracted from audio and video. To our knowledge, this represents the first time techniques from computer vision and speech processing have been used for natural ECoG decoding. Interpretable behaviors were decoded from ECoG data, including moving, speaking and resting; the results were assessed by comparison with manual annotation. Discovered clusters were projected back onto the brain revealing features consistent with known functional areas, opening the door to automated functional brain mapping in natural settings.

Cortical and Subcortical Contributions to Short-Term Memory for Orienting Movements

Neural activity in frontal cortical areas has been causally linked to short-term memory (STM), but whether this activity is necessary for forming, maintaining, or reading out STM remains unclear. In rats performing a memory-guided orienting task, the frontal orienting fields in cortex (FOF) are considered critical for STM maintenance, and during each trial display a monotonically increasing neural encoding for STM. Here, we transiently inactivated either the FOF or the superior colliculus and found that the resulting impairments in memory-guided orienting performance followed a monotonically decreasing time course, surprisingly opposite to the neural encoding. A dynamical attractor model in which STM relies equally on cortical and subcortical regions reconciled the encoding and inactivation data. We confirmed key predictions of the model, including a time-dependent relationship between trial difficulty and perturbability, and substantial, supralinear, impairment following simultaneous inactivation of the FOF and superior colliculus during memory maintenance.

Distinct effects of prefrontal and parietal cortex inactivations on an accumulation of evidence task in the rat

Numerous brain regions have been shown to have neural correlates of gradually accumulating evidence for decision-making, but the causal roles of these regions in decisions driven by accumulation of evidence have yet to be determined. Here, in rats performing an auditory evidence accumulation task, we inactivated the frontal orienting fields (FOF) and posterior parietal cortex (PPC), two rat cortical regions that have neural correlates of accumulating evidence and that have been proposed as central to decision-making. We used a detailed model of the decision process to analyze the effect of inactivations. Inactivation of the FOF induced substantial performance impairments that were quantitatively best described as an impairment in the output pathway of an evidence accumulator with a long integration time constant (>240 ms). In contrast, we found a minimal role for PPC in decisions guided by accumulating auditory evidence, even while finding a strong role for PPC in internally-guided decisions.

Distinct relationships of parietal and prefrontal cortices to evidence accumulation

Gradual accumulation of evidence is thought to be fundamental for decision-making, and its neural correlates have been found in several brain regions. Here we develop a generalizable method to measure tuning curves that specify the relationship between neural responses and mentally accumulated evidence, and apply it to distinguish the encoding of decision variables in posterior parietal cortex and prefrontal cortex (frontal orienting fields, FOF). We recorded the firing rates of neurons in posterior parietal cortex and FOF from rats performing a perceptual decision-making task. Classical analyses uncovered correlates of accumulating evidence, similar to previous observations in primates and also similar across the two regions. However, tuning curve assays revealed that while the posterior parietal cortex encodes a graded value of the accumulating evidence, the FOF has a more categorical encoding that indicates, throughout the trial, the decision provisionally favoured by the evidence accumulated so far. Contrary to current views, this suggests that premotor activity in the frontal cortex does not have a role in the accumulation process, but instead has a more categorical function, such as transforming accumulated evidence into a discrete choice. To probe causally the role of FOF activity, we optogenetically silenced it during different time points of the trial. Consistent with a role in committing to a categorical choice at the end of the evidence accumulation process, but not consistent with a role during the accumulation itself, a behavioural effect was observed only when FOF silencing occurred at the end of the perceptual stimulus. Our results place important constraints on the circuit logic of brain regions involved in decision-making.

A low-frequency oscillatory neural signal in humans encodes a developing decision variable

We often make decisions based on sensory evidence that is accumulated over a period of time. How the evidence for such decisions is represented in the brain and how such a neural representation is used to guide a subsequent action are questions of considerable interest to decision sciences. The neural correlates of developing perceptual decisions have been thoroughly investigated in the oculomotor system of macaques who communicated their decisions using an eye movement. It has been found that the evidence informing a decision to make an eye movement is in part accumulated within the same oculomotor circuits that signal the upcoming eye movement. Recent evidence suggests that the somatomotor system may exhibit an analogous property for choices made using a hand movement. To investigate this possibility, we engaged humans in a decision task in which they integrated discrete quanta of sensory information over a period of time and signaled their decision using a hand movement or an eye movement. The discrete form of the sensory evidence allowed us to infer the decision variable on which subjects base their decision on each trial and to assess the neural processes related to each quantum of the incoming decision evidence. We found that a low-frequency electrophysiological signal recorded over centroparietal regions strongly encodes the decision variable inferred in this task, and that it does so specifically for hand movement choices. The signal ramps up with a rate that is proportional to the decision variable, remains graded by the decision variable throughout the delay period, reaches a common peak shortly before a hand movement, and falls off shortly after the hand movement. Furthermore, the signal encodes the polarity of each evidence quantum, with a short latency, and retains the response level over time. Thus, this neural signal shows properties of evidence accumulation. These findings suggest that the decision-related effects observed in the oculomotor system of the monkey during eye movement choices may share the same basic properties with the decision-related effects in the somatomotor system of humans during hand movement choices.

Imaging the transport dynamics of single alphaherpesvirus particles in intact peripheral nervous system explants from infected mice

ABSTRACT Alphaherpesvirus particles travel long distances in the axons of neurons using host microtubule molecular motors. The transport dynamics of individual virions in neurons have been assessed in cultured neurons, but imaging studies of single particles in tissue from infected mice have not been reported. We developed a protocol to image explanted, infected peripheral nervous system (PNS) ganglia and associated innervated tissue from mice infected with pseudorabies virus (PRV). This ex vivo preparation allowed us to visualize and track individual virions over time as they moved from the salivary gland into submandibular ganglion neurons of the PNS. We imaged and tracked hundreds of virions from multiple mice at different time points. We quantitated the transport velocity, particle stalling, duty cycle, and directionality at various times after infection. Using a PRV recombinant that expressed monomeric red fluorescent protein (mRFP)-VP26 (red capsid) and green fluorescent protein (GFP)-Us9 (green membrane protein), we corroborated that anterograde transport in axons occurs after capsids are enveloped. We addressed the question of whether replication occurs initially in the salivary gland at the site of inoculation or subsequently in the neurons of peripheral innervating ganglia. Our data indicate that significant amplification of infection occurs in the peripheral ganglia after transport from the site of infection and that these newly made particles are transported back to the salivary gland. It is likely that this reseeding of the infected gland contributes to massive invasion of the innervating PNS ganglia. We suggest that this "round-trip" infection process contributes to the characteristic peripheral neuropathy of PRV infection. IMPORTANCE Much of our understanding of molecular mechanisms of alphaherpesvirus infection and spread in neurons comes from studying cultured primary neurons. These techniques enabled significant advances in our understanding of the viral and neuronal components needed for efficient replication and directional spread between cells. However, in vitro systems cannot recapitulate the environment of innervated tissue in vivo with associated defensive properties, such as innate immunity. Therefore, in this report, we describe a system to image the progression of infection by single virus particles in tissue harvested from infected animals. We explanted intact innervated tissue from infected mice and imaged fluorescent virus particles in infected axons of the specific ganglionic neurons. Our measurements of virion transport dynamics are consistent with published in vitro results. Importantly, this system enabled us to address a fundamental biological question about the amplification of a herpesvirus infection in a peripheral nervous system circuit.

Rats and humans can optimally accumulate evidence for decision-making

The gradual and noisy accumulation of evidence is a fundamental component of decision-making, with noise playing a key role as the source of variability and errors. However, the origins of this noise have never been determined. We developed decision-making tasks in which sensory evidence is delivered in randomly timed pulses, and analyzed the resulting data with models that use the richly detailed information of each trial's pulse timing to distinguish between different decision-making mechanisms. This analysis allowed measurement of the magnitude of noise in the accumulator's memory, separately from noise associated with incoming sensory evidence. In our tasks, the accumulator's memory was noiseless, for both rats and humans. In contrast, the addition of new sensory evidence was the primary source of variability. We suggest our task and modeling approach as a powerful method for revealing internal properties of decision-making processes.