Unsupervised identification of the internal states that shape natural behavior

A Comparison of Rapid Rule-Learning Strategies in Humans and Monkeys

Interspecies comparisons are key to deriving an understanding of the behavioral and neural correlates of human cognition from animal models. We perform a detailed comparison of the strategies of female macaque monkeys to male and female humans on a variant of the Wisconsin Card Sorting Test (WCST), a widely studied and applied task that provides a multiattribute measure of cognitive function and depends on the frontal lobe. WCST performance requires the inference of a rule change given ambiguous feedback. We found that well-trained monkeys infer new rules three times more slowly than minimally instructed humans. Input-dependent hidden Markov model-generalized linear models were fit to their choices, revealing hidden states akin to feature-based attention in both species. Decision processes resembled a win-stay, lose-shift strategy with interspecies similarities as well as key differences. Monkeys and humans both test multiple rule hypotheses over a series of rule-search trials and perform inference-like computations to exclude candidate choice options. We quantitatively show that perseveration, random exploration, and poor sensitivity to negative feedback account for the slower task-switching performance in monkeys.

Identifying behavioral links to neural dynamics of multifiber photometry recordings in a mouse social behavior network

Objective.Distributed hypothalamic-midbrain neural circuits help orchestrate complex behavioral responses during social interactions. Given rapid advances in optical imaging, it is a fundamental question how population-averaged neural activity measured by multi-fiber photometry (MFP) for calcium fluorescence signals correlates with social behaviors is a fundamental question. This paper aims to investigate the correspondence between MFP data and social behaviors.Approach:We propose a state-space analysis framework to characterize mouse MFP data based on dynamic latent variable models, which include a continuous-state linear dynamical system and a discrete-state hidden semi-Markov model. We validate these models on extensive MFP recordings during aggressive and mating behaviors in male-male and male-female interactions, respectively.Main results:Our results show that these models are capable of capturing both temporal behavioral structure and associated neural states, and produce interpretable latent states. Our approach is also validated in computer simulations in the presence of known ground truth.Significance:Overall, these analysis approaches provide a state-space framework to examine neural dynamics underlying social behaviors and reveals mechanistic insights into the relevant networks.

Large-scale foundation models and generative AI for BigData neuroscience

Recent advances in machine learning have led to revolutionary breakthroughs in computer games, image and natural language understanding, and scientific discovery. Foundation models and large-scale language models (LLMs) have recently achieved human-like intelligence thanks to BigData. With the help of self-supervised learning (SSL) and transfer learning, these models may potentially reshape the landscapes of neuroscience research and make a significant impact on the future. Here we present a mini-review on recent advances in foundation models and generative AI models as well as their applications in neuroscience, including natural language and speech, semantic memory, brain-machine interfaces (BMIs), and data augmentation. We argue that this paradigm-shift framework will open new avenues for many neuroscience research directions and discuss the accompanying challenges and opportunities.

Behavioral strategy shapes activation of the Vip-Sst disinhibitory circuit in visual cortex

In complex environments, animals can adopt diverse strategies to find rewards. How distinct strategies differentially engage brain circuits is not well understood. Here, we investigate this question, focusing on the cortical Vip-Sst disinhibitory circuit between vasoactive intestinal peptide-postive (Vip) interneurons and somatostatin-positive (Sst) interneurons. We characterize the behavioral strategies used by mice during a visual change detection task. Using a dynamic logistic regression model, we find that individual mice use mixtures of a visual comparison strategy and a statistical timing strategy. Separately, mice also have periods of task engagement and disengagement. Two-photon calcium imaging shows large strategy-dependent differences in neural activity in excitatory, Sst inhibitory, and Vip inhibitory cells in response to both image changes and image omissions. In contrast, task engagement has limited effects on neural population activity. We find that the diversity of neural correlates of strategy can be understood parsimoniously as the increased activation of the Vip-Sst disinhibitory circuit during the visual comparison strategy, which facilitates task-appropriate responses.

Consistent spectro-spatial features of human ECoG successfully decode naturalistic behavioral states

Objective

Understanding the neural correlates of naturalistic behavior is critical for extending and confirming the results obtained from trial-based experiments and designing generalizable brain-computer interfaces that can operate outside laboratory environments. In this study, we aimed to pinpoint consistent spectro-spatial features of neural activity in humans that can discriminate between naturalistic behavioral states.

Approach

We analyzed data from five participants using electrocorticography (ECoG) with broad spatial coverage. Spontaneous and naturalistic behaviors such as "Talking" and "Watching TV" were labeled from manually annotated videos. Linear discriminant analysis (LDA) was used to classify the two behavioral states. The parameters learned from the LDA were then used to determine whether the neural signatures driving classification performance are consistent across the participants.

Main results

Spectro-spatial feature values were consistently discriminative between the two labeled behavioral states across participants. Mainly, θ, α, and low and high γ in the postcentral gyrus, precentral gyrus, and temporal lobe showed significant classification performance and feature consistency across participants. Subject-specific performance exceeded 70%. Combining neural activity from multiple cortical regions generally does not improve decoding performance, suggesting that information regarding the behavioral state is non-additive as a function of the cortical region.

Significance

To the best of our knowledge, this is the first attempt to identify specific spectro-spatial neural correlates that consistently decode naturalistic and active behavioral states. The aim of this work is to serve as an initial starting point for developing brain-computer interfaces that can be generalized in a realistic setting and to further our understanding of the neural correlates of naturalistic behavior in humans.

Uncovering multiscale structure in the variability of larval zebrafish navigation

Animals chain movements into long-lived motor strategies, exhibiting variability across scales that reflects the interplay between internal states and environmental cues. To reveal structure in such variability, we build Markov models of movement sequences that bridges across time scales and enables a quantitative comparison of behavioral phenotypes among individuals. Applied to larval zebrafish responding to diverse sensory cues, we uncover a hierarchy of long-lived motor strategies, dominated by changes in orientation distinguishing cruising versus wandering strategies. Environmental cues induce preferences along these modes at the population level: while fish cruise in the light, they wander in response to aversive stimuli, or in search for appetitive prey. As our method encodes the behavioral dynamics of each individual fish in the transitions among coarse-grained motor strategies, we use it to uncover a hierarchical structure in the phenotypic variability that reflects exploration-exploitation trade-offs. Across a wide range of sensory cues, a major source of variation among fish is driven by prior and/or immediate exposure to prey that induces exploitation phenotypes. A large degree of variability that is not explained by environmental cues unravels motivational states that override the sensory context to induce contrasting exploration-exploitation phenotypes. Altogether, by extracting the timescales of motor strategies deployed during navigation, our approach exposes structure among individuals and reveals internal states tuned by prior experience.

Uncovering multiscale structure in the variability of larval zebrafish navigation

Animals chain movements into long-lived motor strategies, resulting in variability that ultimately reflects the interplay between internal states and environmental cues. To reveal structure in such variability, we build models that bridges across time scales that enable a quantitative comparison of behavioral phenotypes among individuals. Applied to larval zebrafish exposed to diverse sensory cues, we uncover a hierarchy of long-lived motor strategies, dominated by changes in orientation distinguishing cruising and wandering strategies. Environmental cues induce preferences along these modes at the population level: while fish cruise in the light, they wander in response to aversive (dark) stimuli or in search for prey. Our method enables us to encode the behavioral dynamics of each individual fish in the transitions among coarse-grained motor strategies. By doing so, we uncover a hierarchical structure to the phenotypic variability that corresponds to exploration-exploitation trade-offs. Within a wide range of sensory cues, a major source of variation among fish is driven by prior and immediate exposure to prey that induces exploitation phenotypes. However, a large degree of variability is unexplained by environmental cues, pointing to hidden states that override the sensory context to induce contrasting exploration-exploitation phenotypes. Altogether, our approach extracts the timescales of motor strategies deployed during navigation, exposing undiscovered structure among individuals and pointing to internal states tuned by prior experience.

Mapping model units to visual neurons reveals population code for social behaviour

The rich variety of behaviours observed in animals arises through the interplay between sensory processing and motor control. To understand these sensorimotor transformations, it is useful to build models that predict not only neural responses to sensory input1-5 but also how each neuron causally contributes to behaviour6,7. Here we demonstrate a novel modelling approach to identify a one-to-one mapping between internal units in a deep neural network and real neurons by predicting the behavioural changes that arise from systematic perturbations of more than a dozen neuronal cell types. A key ingredient that we introduce is 'knockout training', which involves perturbing the network during training to match the perturbations of the real neurons during behavioural experiments. We apply this approach to model the sensorimotor transformations of Drosophila melanogaster males during a complex, visually guided social behaviour8-11. The visual projection neurons at the interface between the optic lobe and central brain form a set of discrete channels12, and prior work indicates that each channel encodes a specific visual feature to drive a particular behaviour13,14. Our model reaches a different conclusion: combinations of visual projection neurons, including those involved in non-social behaviours, drive male interactions with the female, forming a rich population code for behaviour. Overall, our framework consolidates behavioural effects elicited from various neural perturbations into a single, unified model, providing a map from stimulus to neuronal cell type to behaviour, and enabling future incorporation of wiring diagrams of the brain15 into the model.

Control over a mixture of policies determines change of mind topology during continuous choice

Behavior is naturally organized into categorically distinct states with corresponding patterns of neural activity; how does the brain control those states? We propose that states are regulated by specific neural processes that implement meta-control that can blend simpler control processes. To test this hypothesis, we recorded from neurons in the dorsal anterior cingulate cortex (dACC) and dorsal premotor cortex (PMd) while macaques performed a continuous pursuit task with two moving prey that followed evasive strategies. We used a novel control theoretic approach to infer subjects' moment-to-moment latent control variables, which in turn dictated their blend of distinct identifiable control processes. We identified low-dimensional subspaces in neuronal responses that reflected the current strategy, the value of the pursued target, and the relative value of the two targets. The top two principal components of activity tracked changes of mind in abstract and change-type-specific formats, respectively. These results indicate that control of behavioral state reflects the interaction of brain processes found in dorsal prefrontal regions that implement a mixture over low-level control policies.

Inhibitory control of locomotor statistics in walking Drosophila

In order to forage for food, many animals regulate not only specific limb movements but the statistics of locomotor behavior over time, for example switching between long-range dispersal behaviors and more localized search depending on the availability of resources. How pre-motor circuits regulate such locomotor statistics is not clear. Here we took advantage of the robust changes in locomotor statistics evoked by attractive odors in walking Drosophila to investigate their neural control. We began by analyzing the statistics of ground speed and angular velocity during three well-defined motor regimes: baseline walking, upwind running during odor, and search behavior following odor offset. We find that during search behavior, flies adopt higher angular velocities and slower ground speeds, and tend to turn for longer periods of time in one direction. We further find that flies spontaneously adopt periods of different mean ground speed, and that these changes in state influence the length of odor-evoked runs. We next developed a simple physiologically-inspired computational model of locomotor control that can recapitulate these statistical features of fly locomotion. Our model suggests that contralateral inhibition plays a key role both in regulating the difference between baseline and search behavior, and in modulating the response to odor with ground speed. As the fly connectome predicts decussating inhibitory neurons in the lateral accessory lobe (LAL), a pre-motor structure, we generated genetic tools to target these neurons and test their role in behavior. Consistent with our model, we found that activation of neurons labeled in one line increased curvature. In a second line labeling distinct neurons, activation and inactivation strongly and reciprocally regulated ground speed and altered the length of the odor-evoked run. Additional targeted light activation experiments argue that these effects arise from the brain rather than from neurons in the ventral nerve cord, while sparse activation experiments argue that speed control in the second line arises from both LAL neurons and a population of neurons in the dorsal superior medial protocerebrum (SMP). Together, our work develops a biologically plausible computational architecture that captures the statistical features of fly locomotion across behavioral states and identifies potential neural substrates of these computations.

Independent inhibitory control mechanisms for aggressive motivation and action

Social behaviors often consist of a motivational phase followed by action. Here we show that neurons in the ventromedial hypothalamus ventrolateral area (VMHvl) of mice encode the temporal sequence of aggressive motivation to action. The VMHvl receives local inhibitory input (VMHvl shell) and long-range input from the medial preoptic area (MPO) with functional coupling to neurons with specific temporal profiles. Encoding models reveal that during aggression, VMHvl shellvgat+ activity peaks at the start of an attack, whereas activity from the MPO-VMHvlvgat+ input peaks at specific interaction endpoints. Activation of the MPO-VMHvlvgat+ input promotes and prolongs a low motivation state, whereas activation of VMHvl shellvgat+ results in action-related deficits, acutely terminating attack. Moreover, stimulation of MPO-VMHvlvgat+ input is positively valenced and anxiolytic. Together, these data demonstrate how distinct inhibitory inputs to the hypothalamus can independently gate the motivational and action phases of aggression through a single locus of control.

Nested neural circuits generate distinct acoustic signals during Drosophila courtship

Many motor control systems generate multiple movements using a common set of muscles. How are premotor circuits able to flexibly generate diverse movement patterns? Here, we characterize the neuronal circuits that drive the distinct courtship songs of Drosophila melanogaster. Male flies vibrate their wings toward females to produce two different song modes-pulse and sine song-which signal species identity and male quality. Using cell-type-specific genetic reagents and the connectome, we provide a cellular and synaptic map of the circuits in the male ventral nerve cord that generate these songs and examine how activating or inhibiting each cell type within these circuits affects the song. Our data reveal that the song circuit is organized into two nested feedforward pathways with extensive reciprocal and feedback connections. The larger network produces pulse song, the more complex and ancestral song form. A subset of this network produces sine song, the simpler and more recent form. Such nested organization may be a common feature of motor control circuits in which evolution has layered increasing flexibility onto a basic movement pattern.

Active Learning for Discrete Latent Variable Models

Active learning seeks to reduce the amount of data required to fit the parameters of a model, thus forming an important class of techniques in modern machine learning. However, past work on active learning has largely overlooked latent variable models, which play a vital role in neuroscience, psychology, and a variety of other engineering and scientific disciplines. Here we address this gap by proposing a novel framework for maximum-mutual-information input selection for discrete latent variable regression models. We first apply our method to a class of models known as mixtures of linear regressions (MLR). While it is well known that active learning confers no advantage for linear-gaussian regression models, we use Fisher information to show analytically that active learning can nevertheless achieve large gains for mixtures of such models, and we validate this improvement using both simulations and real-world data. We then consider a powerful class of temporally structured latent variable models given by a hidden Markov model (HMM) with generalized linear model (GLM) observations, which has recently been used to identify discrete states from animal decision-making data. We show that our method substantially reduces the amount of data needed to fit GLM-HMMs and outperforms a variety of approximate methods based on variational and amortized inference. Infomax learning for latent variable models thus offers a powerful approach for characterizing temporally structured latent states, with a wide variety of applications in neuroscience and beyond.

Time-varying generalized linear models: characterizing and decoding neuronal dynamics in higher visual areas

To create a behaviorally relevant representation of the visual world, neurons in higher visual areas exhibit dynamic response changes to account for the time-varying interactions between external (e.g., visual input) and internal (e.g., reward value) factors. The resulting high-dimensional representational space poses challenges for precisely quantifying individual factors' contributions to the representation and readout of sensory information during a behavior. The widely used point process generalized linear model (GLM) approach provides a powerful framework for a quantitative description of neuronal processing as a function of various sensory and non-sensory inputs (encoding) as well as linking particular response components to particular behaviors (decoding), at the level of single trials and individual neurons. However, most existing variations of GLMs assume the neural systems to be time-invariant, making them inadequate for modeling nonstationary characteristics of neuronal sensitivity in higher visual areas. In this review, we summarize some of the existing GLM variations, with a focus on time-varying extensions. We highlight their applications to understanding neural representations in higher visual areas and decoding transient neuronal sensitivity as well as linking physiology to behavior through manipulation of model components. This time-varying class of statistical models provide valuable insights into the neural basis of various visual behaviors in higher visual areas and hold significant potential for uncovering the fundamental computational principles that govern neuronal processing underlying various behaviors in different regions of the brain.

Inferring neural dynamics of memory during naturalistic social communication

Memory processes in complex behaviors like social communication require forming representations of the past that grow with time. The neural mechanisms that support such continually growing memory remain unknown. We address this gap in the context of fly courtship, a natural social behavior involving the production and perception of long, complex song sequences. To study female memory for male song history in unrestrained courtship, we present 'Natural Continuation' (NC)-a general, simulation-based model comparison procedure to evaluate candidate neural codes for complex stimuli using naturalistic behavioral data. Applying NC to fly courtship revealed strong evidence for an adaptive population mechanism for how female auditory neural dynamics could convert long song histories into a rich mnemonic format. Song temporal patterning is continually transformed by heterogeneous nonlinear adaptation dynamics, then integrated into persistent activity, enabling common neural mechanisms to retain continuously unfolding information over long periods and yielding state-of-the-art predictions of female courtship behavior. At a population level this coding model produces multi-dimensional advection-diffusion-like responses that separate songs over a continuum of timescales and can be linearly transformed into flexible output signals, illustrating its potential to create a generic, scalable mnemonic format for extended input signals poised to drive complex behavioral responses. This work thus shows how naturalistic behavior can directly inform neural population coding models, revealing here a novel process for memory formation.

Automated behavioral analysis reveals that mice employ a bait-and-switch escape mechanism to de-escalate social conflict

Intraspecies aggression has profound ecological and evolutionary consequences, as recipients can suffer injuries, decreases in fitness, and become outcasts from social groups. Although animals implement diverse strategies to avoid hostile confrontations, the extent to which social influences affect escape tactics is unclear. Here, we used computational and machine-learning approaches to analyze complex behavioral interactions as mixed-sex groups of mice, Mus musculus, freely interacted. Mice displayed a rich repertoire of behaviors marked by changes in behavioral state, aggressive encounters, and mixed-sex interactions. A prominent behavioral sequence consistently occurred after aggressive encounters, where males in submissive states quickly approached and transiently interacted with females immediately before the aggressor engaged with the same female. The behavioral sequences were also associated with substantially fewer physical altercations. Furthermore, the male's behavioral state and the interacting partners could be predicted by distinct features of the behavioral sequence, such as kinematics and the latency to and duration of male-female interactions. More broadly, our work revealed an ethologically relevant escape strategy influenced by the presence of females that may serve as a mechanism for de-escalating social conflict and preventing consequential reductions in fitness.

Identifying behavioral links to neural dynamics of multifiber photometry recordings in a mouse social behavior network

Distributed hypothalamic-midbrain neural circuits orchestrate complex behavioral responses during social interactions. How population-averaged neural activity measured by multi-fiber photometry (MFP) for calcium fluorescence signals correlates with social behaviors is a fundamental question. We propose a state-space analysis framework to characterize mouse MFP data based on dynamic latent variable models, which include continuous-state linear dynamical system (LDS) and discrete-state hidden semi-Markov model (HSMM). We validate these models on extensive MFP recordings during aggressive and mating behaviors in male-male and male-female interactions, respectively. Our results show that these models are capable of capturing both temporal behavioral structure and associated neural states. Overall, these analysis approaches provide an unbiased strategy to examine neural dynamics underlying social behaviors and reveals mechanistic insights into the relevant networks.

Facemap: a framework for modeling neural activity based on orofacial tracking

Recent studies in mice have shown that orofacial behaviors drive a large fraction of neural activity across the brain. To understand the nature and function of these signals, we need better computational models to characterize the behaviors and relate them to neural activity. Here we developed Facemap, a framework consisting of a keypoint tracker and a deep neural network encoder for predicting neural activity. Our algorithm for tracking mouse orofacial behaviors was more accurate than existing pose estimation tools, while the processing speed was several times faster, making it a powerful tool for real-time experimental interventions. The Facemap tracker was easy to adapt to data from new labs, requiring as few as 10 annotated frames for near-optimal performance. We used the keypoints as inputs to a deep neural network which predicts the activity of ~50,000 simultaneously-recorded neurons and, in visual cortex, we doubled the amount of explained variance compared to previous methods. Using this model, we found that the neuronal activity clusters that were well predicted from behavior were more spatially spread out across cortex. We also found that the deep behavioral features from the model had stereotypical, sequential dynamics that were not reversible in time. In summary, Facemap provides a stepping stone toward understanding the function of the brain-wide neural signals and their relation to behavior.

OpenApePose, a database of annotated ape photographs for pose estimation

Because of their close relationship with humans, non-human apes (chimpanzees, bonobos, gorillas, orangutans, and gibbons, including siamangs) are of great scientific interest. The goal of understanding their complex behavior would be greatly advanced by the ability to perform video-based pose tracking. Tracking, however, requires high-quality annotated datasets of ape photographs. Here we present OpenApePose, a new public dataset of 71,868 photographs, annotated with 16 body landmarks of six ape species in naturalistic contexts. We show that a standard deep net (HRNet-W48) trained on ape photos can reliably track out-of-sample ape photos better than networks trained on monkeys (specifically, the OpenMonkeyPose dataset) and on humans (COCO) can. This trained network can track apes almost as well as the other networks can track their respective taxa, and models trained without one of the six ape species can track the held-out species better than the monkey and human models can. Ultimately, the results of our analyses highlight the importance of large, specialized databases for animal tracking systems and confirm the utility of our new ape database.

Internal states as a source of subject-dependent movement variability are represented by large-scale brain networks

Humans' ability to adapt and learn relies on reflecting on past performance. These experiences form latent representations called internal states that induce movement variability that improves how we interact with our environment. Our study uncovered temporal dynamics and neural substrates of two states from ten subjects implanted with intracranial depth electrodes while they performed a goal-directed motor task with physical perturbations. We identified two internal states using state-space models: one tracking past errors and the other past perturbations. These states influenced reaction times and speed errors, revealing how subjects strategize from trial history. Using local field potentials from over 100 brain regions, we found large-scale brain networks such as the dorsal attention and default mode network modulate visuospatial attention based on recent performance and environmental feedback. Notably, these networks were more prominent in higher-performing subjects, emphasizing their role in improving motor performance by regulating movement variability through internal states.

Everyone knows what behavior is but they just don't agree on it

Studying "behavior" lies at the heart of many disciplines. Nevertheless, academics rarely provide an explicit definition of what "behavior" actually is. What range of definitions do people use, and how does that vary across disciplines? To answer these questions, we have developed a survey to probe what constitutes "behavior." We find that academics adopt different definitions of behavior according to their academic discipline, animal model that they work with, and level of academic seniority. Using hierarchical clustering, we identify at least six distinct types of "behavior" which are used in seven distinct operational archetypes of "behavior." Individual respondents have clear consistent definitions of behavior, but these definitions are not consistent across the population. Our study is a call for academics to clarify what they mean by "behavior" wherever they study it, with the hope that this will foster interdisciplinary studies that will improve our understanding of behavioral phenomena.

Autogenous cerebral processes: an invitation to look at the brain from inside out

While external stimulation can reliably trigger neuronal activity, cerebral processes can operate independently from the environment. In this study, we conceptualize autogenous cerebral processes (ACPs) as intrinsic operations of the brain that exist on multiple scales and can influence or shape stimulus responses, behavior, homeostasis, and the physiological state of an organism. We further propose that the field should consider exploring to what extent perception, arousal, behavior, or movement, as well as other cognitive functions previously investigated mainly regarding their stimulus-response dynamics, are ACP-driven.

Rapid automated 3-D pose estimation of larval zebrafish using a physical model-trained neural network

Quantitative ethology requires an accurate estimation of an organism's postural dynamics in three dimensions plus time. Technological progress over the last decade has made animal pose estimation in challenging scenarios possible with unprecedented detail. Here, we present (i) a fast automated method to record and track the pose of individual larval zebrafish in a 3-D environment, applicable when accurate human labeling is not possible; (ii) a rich annotated dataset of 3-D larval poses for ethologists and the general zebrafish and machine learning community; and (iii) a technique to generate realistic, annotated larval images in different behavioral contexts. Using a three-camera system calibrated with refraction correction, we record diverse larval swims under free swimming conditions and in response to acoustic and optical stimuli. We then employ a convolutional neural network to estimate 3-D larval poses from video images. The network is trained against a set of synthetic larval images rendered using a 3-D physical model of larvae. This 3-D model samples from a distribution of realistic larval poses that we estimate a priori using a template-based pose estimation of a small number of swim bouts. Our network model, trained without any human annotation, performs larval pose estimation three orders of magnitude faster and with accuracy comparable to the template-based approach, capturing detailed kinematics of 3-D larval swims. It also applies accurately to other datasets collected under different imaging conditions and containing behavioral contexts not included in our training.

Flexible circuit mechanisms for context-dependent song sequencing

Sequenced behaviours, including locomotion, reaching and vocalization, are patterned differently in different contexts, enabling animals to adjust to their environments. How contextual information shapes neural activity to flexibly alter the patterning of actions is not fully understood. Previous work has indicated that this could be achieved via parallel motor circuits, with differing sensitivities to context1,2. Here we demonstrate that a single pathway operates in two regimes dependent on recent sensory history. We leverage the Drosophila song production system3 to investigate the role of several neuron types4-7 in song patterning near versus far from the female fly. Male flies sing 'simple' trains of only one mode far from the female fly but complex song sequences comprising alternations between modes when near her. We find that ventral nerve cord (VNC) circuits are shaped by mutual inhibition and rebound excitability8 between nodes driving the two song modes. Brief sensory input to a direct brain-to-VNC excitatory pathway drives simple song far from the female, whereas prolonged input enables complex song production via simultaneous recruitment of functional disinhibition of VNC circuitry. Thus, female proximity unlocks motor circuit dynamics in the correct context. We construct a compact circuit model to demonstrate that the identified mechanisms suffice to replicate natural song dynamics. These results highlight how canonical circuit motifs8,9 can be combined to enable circuit flexibility required for dynamic communication.

Hierarchical action encoding in prefrontal cortex of freely moving macaques

Our natural behavioral repertoires include coordinated actions of characteristic types. To better understand how neural activity relates to the expression of actions and action switches, we studied macaques performing a freely moving foraging task in an open environment. We developed a novel analysis pipeline that can identify meaningful units of behavior, corresponding to recognizable actions such as sitting, walking, jumping, and climbing. On the basis of transition probabilities between these actions, we found that behavior is organized in a modular and hierarchical fashion. We found that, after regressing out many potential confounders, actions are associated with specific patterns of firing in each of six prefrontal brain regions and that, overall, encoding of action category is progressively stronger in more dorsal and more caudal prefrontal regions. Together, these results establish a link between selection of units of primate behavior on one hand and neuronal activity in prefrontal regions on the other.

Large-scale capture of hidden fluorescent labels for training generalizable markerless motion capture models

Deep learning-based markerless tracking has revolutionized studies of animal behavior. Yet the generalizability of trained models tends to be limited, as new training data typically needs to be generated manually for each setup or visual environment. With each model trained from scratch, researchers track distinct landmarks and analyze the resulting kinematic data in idiosyncratic ways. Moreover, due to inherent limitations in manual annotation, only a sparse set of landmarks are typically labeled. To address these issues, we developed an approach, which we term GlowTrack, for generating orders of magnitude more training data, enabling models that generalize across experimental contexts. We describe: a) a high-throughput approach for producing hidden labels using fluorescent markers; b) a multi-camera, multi-light setup for simulating diverse visual conditions; and c) a technique for labeling many landmarks in parallel, enabling dense tracking. These advances lay a foundation for standardized behavioral pipelines and more complete scrutiny of movement.

A unified open-source platform for multimodal neural recording and perturbation during naturalistic behavior

Behavioral neuroscience faces two conflicting demands: long-duration recordings from large neural populations and unimpeded animal behavior. To meet this challenge, we developed ONIX, an open-source data acquisition system with high data throughput (2GB/sec) and low closed-loop latencies (<1ms) that uses a novel 0.3 mm thin tether to minimize behavioral impact. Head position and rotation are tracked in 3D and used to drive active commutation without torque measurements. ONIX can acquire from combinations of passive electrodes, Neuropixels probes, head-mounted microscopes, cameras, 3D-trackers, and other data sources. We used ONIX to perform uninterrupted, long (~7 hours) neural recordings in mice as they traversed complex 3-dimensional terrain. ONIX allowed exploration with similar mobility as non-implanted animals, in contrast to conventional tethered systems which restricted movement. By combining long recordings with full mobility, our technology will enable new progress on questions that require high-quality neural recordings during ethologically grounded behaviors.

AI and the Doctor Dolittle challenge

Talking to animals is a fundamental human desire. The emergence of powerful AI algorithms, and specifically Large Language Models, has driven many to suggest that we are on the verge of fulfilling this wish. A few large scientific consortia have been formed around this topic and several commercial entities even offer such services. We frame the task of communicating with animals as 'The Doctor Dolittle challenge' and identify three main obstacles on the route to doing so. First, although generative AI models can create novel animal communication samples, it is very difficult to determine their context, and we will forever be biased by our human umwelt when doing so. Second, using AI to extract context in an unsupervised manner must be validated through controlled experiments aiming to measure the animals' response. This is difficult, and moreover, AI algorithms tend to cling on to any available information and are thus prone to finding spurious correlations. And third, animal communication focuses on a restricted set of contexts, such as alarm and courtship, highly limiting our ability to communicate regarding other contexts. Nevertheless, using the tremendous power of novel AI methods to decipher and mimic animal communication is both fascinating and important. We thus define the criteria for passing the Doctor Dolittle challenge and call upon scientists to take on the mission.

Hierarchical organization of rhesus macaque behavior

Primatologists, psychologists and neuroscientists have long hypothesized that primate behavior is highly structured. However, delineating that structure has been impossible due to the difficulties of precision behavioral tracking. Here we analyzed a dataset consisting of continuous measures of the 3D position of two male rhesus macaques (Macaca mulatta) performing three different tasks in a large unrestrained environment over several hours. Using an unsupervised embedding approach on the tracked joints, we identified commonly repeated pose patterns, which we call postures. We found that macaques' behavior is characterized by 49 distinct postures, lasting an average of 0.6 seconds. We found evidence that behavior is hierarchically organized, in that transitions between poses tend to occur within larger modules, which correspond to identifiable actions; these actions are further organized hierarchically. Our behavioral decomposition allows us to identify universal (cross-individual and cross-task) and unique (specific to each individual and task) principles of behavior. These results demonstrate the hierarchical nature of primate behavior, provide a method for the automated ethogramming of primate behavior, and provide important constraints on neural models of pose generation.

Evaluation of mouse behavioral responses to nutritive versus nonnutritive sugar using a deep learning-based 3D real-time pose estimation system

Animals are able to detect the nutritional content of sugar independently of taste. When given a choice between nutritive sugar and nonnutritive sugar, animals develop a preference for nutritive sugar over nonnutritive sugar during a period of food deprivation (Buchanan et al., 2022; Dus et al., 2011; 2015; Tan et al., 2020; Tellez et al., 2016). To quantify behavioral features during an episode of licking nutritive versus nonnutritive sugar, we implemented a multi-vision, deep learning-based 3D pose estimation system, termed the AI Vision Analysis for Three-dimensional Action in Real-Time (AVATAR)(Kim et al., 2022). Using this method, we found that mice exhibit significantly different approach behavioral responses toward nutritive sugar versus nonnutritive sugar even before licking a sugar solution. Notably, the behavioral sequences during the approach toward nutritive versus nonnutritive sugar became significantly different over time. These results suggest that the nutritional value of sugar not only promotes its consumption but also elicits distinct repertoires of feeding behavior in deprived mice.

Neural Mechanisms That Make Perceptual Decisions Flexible

Neural mechanisms of perceptual decision making have been extensively studied in experimental settings that mimic stable environments with repeating stimuli, fixed rules, and payoffs. In contrast, we live in an ever-changing environment and have varying goals and behavioral demands. To accommodate variability, our brain flexibly adjusts decision-making processes depending on context. Here, we review a growing body of research that explores the neural mechanisms underlying this flexibility. We highlight diverse forms of context dependency in decision making implemented through a variety of neural computations. Context-dependent neural activity is observed in a distributed network of brain structures, including posterior parietal, sensory, motor, and subcortical regions, as well as the prefrontal areas classically implicated in cognitive control. We propose that investigating the distributed network underlying flexible decisions is key to advancing our understanding and discuss a path forward for experimental and theoretical investigations.

Leveraging big data to uncover the eco-evolutionary factors shaping behavioural development

Mapping the eco-evolutionary factors shaping the development of animals' behavioural phenotypes remains a great challenge. Recent advances in 'big behavioural data' research-the high-resolution tracking of individuals and the harnessing of that data with powerful analytical tools-have vastly improved our ability to measure and model developing behavioural phenotypes. Applied to the study of behavioural ontogeny, the unfolding of whole behavioural repertoires can be mapped in unprecedented detail with relative ease. This overcomes long-standing experimental bottlenecks and heralds a surge of studies that more finely define and explore behavioural-experiential trajectories across development. In this review, we first provide a brief guide to state-of-the-art approaches that allow the collection and analysis of high-resolution behavioural data across development. We then outline how such approaches can be used to address key issues regarding the ecological and evolutionary factors shaping behavioural development: developmental feedbacks between behaviour and underlying states, early life effects and behavioural transitions, and information integration across development.

The geometry of cortical representations of touch in rodents

Neurons often encode highly heterogeneous non-linear functions of multiple task variables, a signature of a high-dimensional geometry. We studied the representational geometry in the somatosensory cortex of mice trained to report the curvature of objects touched by their whiskers. High-speed videos of the whiskers revealed that the task can be solved by linearly integrating multiple whisker contacts over time. However, the neural activity in somatosensory cortex reflects non-linear integration of spatio-temporal features of the sensory inputs. Although the responses at first appeared disorganized, we identified an interesting structure in the representational geometry: different whisker contacts are disentangled variables represented in approximately, but not fully, orthogonal subspaces of the neural activity space. This geometry allows linear readouts to perform a broad class of tasks of different complexities without compromising the ability to generalize to novel situations.

Comparing rapid rule-learning strategies in humans and monkeys

Inter-species comparisons are key to deriving an understanding of the behavioral and neural correlates of human cognition from animal models. We perform a detailed comparison of macaque monkey and human strategies on an analogue of the Wisconsin Card Sort Test, a widely studied and applied multi-attribute measure of cognitive function, wherein performance requires the inference of a changing rule given ambiguous feedback. We found that well-trained monkeys rapidly infer rules but are three times slower than humans. Model fits to their choices revealed hidden states akin to feature-based attention in both species, and decision processes that resembled a Win-stay lose-shift strategy with key differences. Monkeys and humans test multiple rule hypotheses over a series of rule-search trials and perform inference-like computations to exclude candidates. An attention-set based learning stage categorization revealed that perseveration, random exploration and poor sensitivity to negative feedback explain the under-performance in monkeys.

The calcium-sensing receptor (CaSR) regulates zebrafish sensorimotor decision making via a genetically defined cluster of hindbrain neurons

Decision making is a fundamental nervous system function that ranges widely in complexity and speed of execution. We previously established larval zebrafish as a model for sensorimotor decision making and identified the G-protein-coupled calcium-sensing receptor (CaSR) to be critical for this process. Here, we report that CaSR functions in neurons to dynamically regulate the bias between two behavioral outcomes: escapes and reorientations. By employing a computational guided transgenic strategy, we identify a genetically defined neuronal cluster in the hindbrain as a key candidate site for CaSR function. Finally, we demonstrate that transgenic CaSR expression targeting this cluster consisting of a few hundred neurons shifts behavioral bias in wild-type animals and restores decision making deficits in CaSR mutants. Combined, our data provide a rare example of a G-protein-coupled receptor that biases vertebrate sensorimotor decision making via a defined neuronal cluster.

Flexibility of neural circuits regulating mating behaviors in mice and flies

Mating is essential for the reproduction of animal species. As mating behaviors are high-risk and energy-consuming processes, it is critical for animals to make adaptive mating decisions. This includes not only finding a suitable mate, but also adapting mating behaviors to the animal's needs and environmental conditions. Internal needs include physical states (e.g., hunger) and emotional states (e.g., fear), while external conditions include both social cues (e.g., the existence of predators or rivals) and non-social factors (e.g., food availability). With recent advances in behavioral neuroscience, we are now beginning to understand the neural basis of mating behaviors, particularly in genetic model organisms such as mice and flies. However, how internal and external factors are integrated by the nervous system to enable adaptive mating-related decision-making in a state- and context-dependent manner is less well understood. In this article, we review recent knowledge regarding the neural basis of flexible mating behaviors from studies of flies and mice. By contrasting the knowledge derived from these two evolutionarily distant model organisms, we discuss potential conserved and divergent neural mechanisms involved in the control of flexible mating behaviors in invertebrate and vertebrate brains.

A neurogenetic mechanism of experience-dependent suppression of aggression

Aggression is an ethologically important social behavior, but excessive aggression can be detrimental to fitness. Social experiences among conspecific individuals reduce aggression in many species, the mechanism of which is largely unknown. We found that loss-of-function mutation of nervy (nvy), a Drosophila homolog of vertebrate myeloid translocation genes (MTGs), increased aggressiveness only in socially experienced flies and that this could be reversed by neuronal expression of human MTGs. A subpopulation of octopaminergic/tyraminergic neurons labeled by nvy was specifically required for such social experience-dependent suppression of aggression, in both males and females. Cell type-specific transcriptomic analysis of these neurons revealed aggression-controlling genes that are likely downstream of nvy. Our results illustrate both genetic and neuronal mechanisms by which the nervous system suppresses aggression in a social experience-dependent manner, a poorly understood process that is considered important for maintaining the fitness of animals.

The emergence and influence of internal states

Animal behavior is shaped by a variety of "internal states"-partially hidden variables that profoundly shape perception, cognition, and action. The neural basis of internal states, such as fear, arousal, hunger, motivation, aggression, and many others, is a prominent focus of research efforts across animal phyla. Internal states can be inferred from changes in behavior, physiology, and neural dynamics and are characterized by properties such as pleiotropy, persistence, scalability, generalizability, and valence. To date, it remains unclear how internal states and their properties are generated by nervous systems. Here, we review recent progress, which has been driven by advances in behavioral quantification, cellular manipulations, and neural population recordings. We synthesize research implicating defined subsets of state-inducing cell types, widespread changes in neural activity, and neuromodulation in the formation and updating of internal states. In addition to highlighting the significance of these findings, our review advocates for new approaches to clarify the underpinnings of internal brain states across the animal kingdom.

Spatiotemporal dynamics of human high gamma discriminate naturalistic behavioral states

In analyzing the neural correlates of naturalistic and unstructured behaviors, features of neural activity that are ignored in a trial-based experimental paradigm can be more fully studied and investigated. Here, we analyze neural activity from two patients using electrocorticography (ECoG) and stereo-electroencephalography (sEEG) recordings, and reveal that multiple neural signal characteristics exist that discriminate between unstructured and naturalistic behavioral states such as “engaging in dialogue” and “using electronics”. Using the high gamma amplitude as an estimate of neuronal firing rate, we demonstrate that behavioral states in a naturalistic setting are discriminable based on long-term mean shifts, variance shifts, and differences in the specific neural activity’s covariance structure. Both the rapid and slow changes in high gamma band activity separate unstructured behavioral states. We also use Gaussian process factor analysis (GPFA) to show the existence of salient spatiotemporal features with variable smoothness in time. Further, we demonstrate that both temporally smooth and stochastic spatiotemporal activity can be used to differentiate unstructured behavioral states. This is the first attempt to elucidate how different neural signal features contain information about behavioral states collected outside the conventional experimental paradigm.

Neural network organization for courtship-song feature detection in Drosophila

Animals communicate using sounds in a wide range of contexts, and auditory systems must encode behaviorally relevant acoustic features to drive appropriate reactions. How feature detection emerges along auditory pathways has been difficult to solve due to challenges in mapping the underlying circuits and characterizing responses to behaviorally relevant features. Here, we study auditory activity in the Drosophila melanogaster brain and investigate feature selectivity for the two main modes of fly courtship song, sinusoids and pulse trains. We identify 24 new cell types of the intermediate layers of the auditory pathway, and using a new connectomic resource, FlyWire, we map all synaptic connections between these cell types, in addition to connections to known early and higher-order auditory neurons-this represents the first circuit-level map of the auditory pathway. We additionally determine the sign (excitatory or inhibitory) of most synapses in this auditory connectome. We find that auditory neurons display a continuum of preferences for courtship song modes and that neurons with different song-mode preferences and response timescales are highly interconnected in a network that lacks hierarchical structure. Nonetheless, we find that the response properties of individual cell types within the connectome are predictable from their inputs. Our study thus provides new insights into the organization of auditory coding within the Drosophila brain.

Comprehensive analysis of locomotion dynamics in the protochordate Ciona intestinalis reveals how neuromodulators flexibly shape its behavioral repertoire

Vertebrate nervous systems can generate a remarkable diversity of behaviors. However, our understanding of how behaviors may have evolved in the chordate lineage is limited by the lack of neuroethological studies leveraging our closest invertebrate relatives. Here, we combine high-throughput video acquisition with pharmacological perturbations of bioamine signaling to systematically reveal the global structure of the motor behavioral repertoire in the Ciona intestinalis larvae. Most of Ciona’s postural variance can be captured by 6 basic shapes, which we term “eigencionas.” Motif analysis of postural time series revealed numerous stereotyped behavioral maneuvers including “startle-like” and “beat-and-glide.” Employing computational modeling of swimming dynamics and spatiotemporal embedding of postural features revealed that behavioral differences are generated at the levels of motor modules and the transitions between, which may in part be modulated by bioamines. Finally, we show that flexible motor module usage gives rise to diverse behaviors in response to different light stimuli.

Vertebrate nervous systems can generate a remarkable diversity of behaviors, but how did these evolve in the chordate lineage? A study of the protochordate Ciona intestinalis reveals novel insights into how a simple chordate brain uses neuromodulators to control its behavioral repertoire.

Neural mechanisms underlying the temporal organization of naturalistic animal behavior

Naturalistic animal behavior exhibits a strikingly complex organization in the temporal domain, with variability arising from at least three sources: hierarchical, contextual, and stochastic. What neural mechanisms and computational principles underlie such intricate temporal features? In this review, we provide a critical assessment of the existing behavioral and neurophysiological evidence for these sources of temporal variability in naturalistic behavior. Recent research converges on an emergent mechanistic theory of temporal variability based on attractor neural networks and metastable dynamics, arising via coordinated interactions between mesoscopic neural circuits. We highlight the crucial role played by structural heterogeneities as well as noise from mesoscopic feedback loops in regulating flexible behavior. We assess the shortcomings and missing links in the current theoretical and experimental literature and propose new directions of investigation to fill these gaps.

Selfee, self-supervised features extraction of animal behaviors

Fast and accurately characterizing animal behaviors is crucial for neuroscience research. Deep learning models are efficiently used in laboratories for behavior analysis. However, it has not been achieved to use an end-to-end unsupervised neural network to extract comprehensive and discriminative features directly from social behavior video frames for annotation and analysis purposes. Here, we report a self-supervised feature extraction (Selfee) convolutional neural network with multiple downstream applications to process video frames of animal behavior in an end-to-end way. Visualization and classification of the extracted features (Meta-representations) validate that Selfee processes animal behaviors in a way similar to human perception. We demonstrate that Meta-representations can be efficiently used to detect anomalous behaviors that are indiscernible to human observation and hint in-depth analysis. Furthermore, time-series analyses of Meta-representations reveal the temporal dynamics of animal behaviors. In conclusion, we present a self-supervised learning approach to extract comprehensive and discriminative features directly from raw video recordings of animal behaviors and demonstrate its potential usage for various downstream applications.

Natural behavior is the language of the brain

The breadth and complexity of natural behaviors inspires awe. Understanding how our perceptions, actions, and internal thoughts arise from evolved circuits in the brain has motivated neuroscientists for generations. Researchers have traditionally approached this question by focusing on stereotyped behaviors, either natural or trained, in a limited number of model species. This approach has allowed for the isolation and systematic study of specific brain operations, which has greatly advanced our understanding of the circuits involved. At the same time, the emphasis on experimental reductionism has left most aspects of the natural behaviors that have shaped the evolution of the brain largely unexplored. However, emerging technologies and analytical tools make it possible to comprehensively link natural behaviors to neural activity across a broad range of ethological contexts and timescales, heralding new modes of neuroscience focused on natural behaviors. Here we describe a three-part roadmap that aims to leverage the wealth of behaviors in their naturally occurring distributions, linking their variance with that of underlying neural processes to understand how the brain is able to successfully navigate the everyday challenges of animals' social and ecological landscapes. To achieve this aim, experimenters must harness one challenge faced by all neurobiological systems, namely variability, in order to gain new insights into the language of the brain.

Imaging whole-brain activity to understand behavior

The brain evolved to produce behaviors that help an animal inhabit the natural world. During natural behaviors, the brain is engaged in many levels of activity from the detection of sensory inputs to decision-making to motor planning and execution. To date, most brain studies have focused on small numbers of neurons that interact in limited circuits. This allows analyzing individual computations or steps of neural processing. During behavior, however, brain activity must integrate multiple circuits in different brain regions. The activities of different brain regions are not isolated, but may be contingent on one another. Coordinated and concurrent activity within and across brain areas is organized by (1) sensory information from the environment, (2) the animal's internal behavioral state, and (3) recurrent networks of synaptic and non-synaptic connectivity. Whole-brain recording with cellular resolution provides a new opportunity to dissect the neural basis of behavior, but whole-brain activity is also mutually contingent on behavior itself. This is especially true for natural behaviors like navigation, mating, or hunting, which require dynamic interaction between the animal, its environment, and other animals. In such behaviors, the sensory experience of an unrestrained animal is actively shaped by its movements and decisions. Many of the signaling and feedback pathways that an animal uses to guide behavior only occur in freely moving animals. Recent technological advances have enabled whole-brain recording in small behaving animals including nematodes, flies, and zebrafish. These whole-brain experiments capture neural activity with cellular resolution spanning sensory, decision-making, and motor circuits, and thereby demand new theoretical approaches that integrate brain dynamics with behavioral dynamics. Here, we review the experimental and theoretical methods that are being employed to understand animal behavior and whole-brain activity, and the opportunities for physics to contribute to this emerging field of systems neuroscience.

Deep-learning based identification, tracking, pose estimation, and behavior classification of interacting primates and mice in complex environments

The quantification of behaviors of interest from video data is commonly used to study brain function, the effects of pharmacological interventions, and genetic alterations. Existing approaches lack the capability to analyze the behavior of groups of animals in complex environments. We present a novel deep learning architecture for classifying individual and social animal behavior, even in complex environments directly from raw video frames, while requiring no intervention after initial human supervision. Our behavioral classifier is embedded in a pipeline (SIPEC) that performs segmentation, identification, pose-estimation, and classification of complex behavior, outperforming the state of the art. SIPEC successfully recognizes multiple behaviors of freely moving individual mice as well as socially interacting non-human primates in 3D, using data only from simple mono-vision cameras in home-cage setups.

Social touch in the age of computational ethology: Embracing as a multidimensional and complex behaviour

Social affective touch is an important aspect of close relationships in humans. It has been also observed in many non-human primate species. Despite the high relevance of behaviours like embraces for personal wellbeing and mental health, they remain vastly under-investigated in psychology. This may be because psychology often relies on a limited repertoire of behavioural measurements such as error rates and reaction time measurements. These are, however, insufficient to capture the multidimensional complexity of highly interactive dyadic behaviours like embraces. Based on recent advances in computational ethology in animal models, the rapidly emerging field of human computational ethology utilizes an accessible repertoire of machine learning methods to track and quantify complex natural behaviours. We highlight how such techniques can be utilized to investigate social touch and which preliminary conditions, motor aspects and higher-level interactions need to be considered. Ultimately, integration of computational ethology with mobile neuroscience techniques such as ultraportable EEG systems will allow for an ecologically valid investigation of social affective touch in humans that will advance psychological research of emotions.

Neural mechanisms of persistent aggression

While aggression is often conceptualized as a highly stereotyped, innate behavior, individuals within a species exhibit a surprising amount of variability in the frequency, intensity, and targets of their aggression. While differences in genetics are a source of some of this variation across individuals (estimates place the heritability of behavior at around 25-30%), a critical driver of variability is previous life experience. A wide variety of social experiences, including sexual, parental, and housing experiences can facilitate "persistent" aggressive states, suggesting that these experiences engage a common set of synaptic and molecular mechanisms that act on dedicated neural circuits for aggression. It has long been known that sex steroid hormones are powerful modulators of behavior, and also, that levels of these hormones are themselves modulated by experience. Several recent studies have started to unravel how experience-dependent hormonal changes during adulthood can create a cascade of molecular, synaptic, and circuit changes that enable behavioral persistence through circuit level remodeling. Here, we propose that sex steroid hormones facilitate persistent aggressive states by changing the relationship between neural activity and an aggression "threshold".