Predictability and hierarchy in Drosophila behavior

Body language signals for rodent social communication

Integration of social cues to initiate adaptive emotional and behavioral responses is a fundamental aspect of animal and human behavior. In humans, social communication includes prominent nonverbal components, such as social touch, gestures and facial expressions. Comparative studies investigating the neural basis of social communication in rodents has historically been centered on olfactory signals and vocalizations, with relatively less focus on non-verbal social cues. Here, we outline two exciting research directions: First, we will review recent observations pointing to a role of social facial expressions in rodents. Second, we will review observations that point to a role of 'non-canonical' rodent body language: body posture signals beyond stereotyped displays in aggressive and sexual behavior. In both sections, we will outline how social neuroscience can build on recent advances in machine learning, robotics and micro-engineering to push these research directions forward towards a holistic systems neurobiology of rodent body language.

Maximally efficient prediction in the early fly visual system may support evasive flight maneuvers

The visual system must make predictions to compensate for inherent delays in its processing. Yet little is known, mechanistically, about how prediction aids natural behaviors. Here, we show that despite a 20-30ms intrinsic processing delay, the vertical motion sensitive (VS) network of the blowfly achieves maximally efficient prediction. This prediction enables the fly to fine-tune its complex, yet brief, evasive flight maneuvers according to its initial ego-rotation at the time of detection of the visual threat. Combining a rich database of behavioral recordings with detailed compartmental modeling of the VS network, we further show that the VS network has axonal gap junctions that are critical for optimal prediction. During evasive maneuvers, a VS subpopulation that directly innervates the neck motor center can convey predictive information about the fly’s future ego-rotation, potentially crucial for ongoing flight control. These results suggest a novel sensory-motor pathway that links sensory prediction to behavior.

Survival-critical behaviors shape neural circuits to translate sensory information into strikingly fast predictions, e.g. in escaping from a predator faster than the system’s processing delay. We show that the fly visual system implements fast and accurate prediction of its visual experience. This provides crucial information for directing fast evasive maneuvers that unfold over just 40ms. Our work shows how this fast prediction is implemented, mechanistically, and suggests the existence of a novel sensory-motor pathway from the fly visual system to a wing steering motor neuron. Echoing and amplifying previous work in the retina, our work hypothesizes that the efficient encoding of predictive information is a universal design principle supporting fast, natural behaviors.

A hierarchical 3D-motion learning framework for animal spontaneous behavior mapping

Animal behavior usually has a hierarchical structure and dynamics. Therefore, to understand how the neural system coordinates with behaviors, neuroscientists need a quantitative description of the hierarchical dynamics of different behaviors. However, the recent end-to-end machine-learning-based methods for behavior analysis mostly focus on recognizing behavioral identities on a static timescale or based on limited observations. These approaches usually lose rich dynamic information on cross-scale behaviors. Here, inspired by the natural structure of animal behaviors, we address this challenge by proposing a parallel and multi-layered framework to learn the hierarchical dynamics and generate an objective metric to map the behavior into the feature space. In addition, we characterize the animal 3D kinematics with our low-cost and efficient multi-view 3D animal motion-capture system. Finally, we demonstrate that this framework can monitor spontaneous behavior and automatically identify the behavioral phenotypes of the transgenic animal disease model. The extensive experiment results suggest that our framework has a wide range of applications, including animal disease model phenotyping and the relationships modeling between the neural circuits and behavior.

Sniff-synchronized, gradient-guided olfactory search by freely moving mice

For many organisms, searching for relevant targets such as food or mates entails active, strategic sampling of the environment. Finding odorous targets may be the most ancient search problem that motile organisms evolved to solve. While chemosensory navigation has been well characterized in microorganisms and invertebrates, spatial olfaction in vertebrates is poorly understood. We have established an olfactory search assay in which freely moving mice navigate noisy concentration gradients of airborne odor. Mice solve this task using concentration gradient cues and do not require stereo olfaction for performance. During task performance, respiration and nose movement are synchronized with tens of milliseconds precision. This synchrony is present during trials and largely absent during inter-trial intervals, suggesting that sniff-synchronized nose movement is a strategic behavioral state rather than simply a constant accompaniment to fast breathing. To reveal the spatiotemporal structure of these active sensing movements, we used machine learning methods to parse motion trajectories into elementary movement motifs. Motifs fall into two clusters, which correspond to investigation and approach states. Investigation motifs lock precisely to sniffing, such that the individual motifs preferentially occur at specific phases of the sniff cycle. The allocentric structure of investigation and approach indicates an advantage to sampling both sides of the sharpest part of the odor gradient, consistent with a serial-sniff strategy for gradient sensing. This work clarifies sensorimotor strategies for mouse olfactory search and guides ongoing work into the underlying neural mechanisms.

Multimodal integration across spatiotemporal scales to guide invertebrate locomotion

Locomotion is a hallmark of organisms that has enabled adaptive radiation to an extraordinarily diverse class of ecological niches, and allows animals to move across vast distances. Sampling from multiple sensory modalities enables animals to acquire rich information to guide locomotion. Locomotion without sensory feedback is haphazard, therefore sensory and motor systems have evolved complex interactions to generate adaptive behavior. Notably, sensory-guided locomotion acts over broad spatial and temporal scales to permit goal-seeking behavior, whether to localize food by tracking an attractive odor plume or to search for a potential mate. How does the brain integrate multimodal stimuli over different temporal and spatial scales to effectively control behavior? In this review, we classify locomotion into three ordinally ranked hierarchical layers that act over distinct spatiotemporal scales: stabilization, motor primitives, and higher-order tasks, respectively. We discuss how these layers present unique challenges and opportunities for sensorimotor integration. We focus on recent advances in invertebrate locomotion due to their accessible neural and mechanical signals from the whole brain, limbs and sensors. Throughout, we emphasize neural-level description of computations for multimodal integration in genetic model systems, including the fruit fly, Drosophila melanogaster, and the yellow fever mosquito, Aedes aegypti. We identify that summation (e.g. gating) and weighting-which are inherent computations of spiking neurons-underlie multimodal integration across spatial and temporal scales, therefore suggesting collective strategies to guide locomotion.

Locomotor transitions in the potential energy landscape-dominated regime

To traverse complex three-dimensional terrain with large obstacles, animals and robots must transition across different modes. However, the most mechanistic understanding of terrestrial locomotion concerns how to generate and stabilize near-steady-state, single-mode locomotion (e.g. walk, run). We know little about how to use physical interaction to make robust locomotor transitions. Here, we review our progress towards filling this gap by discovering terradynamic principles of multi-legged locomotor transitions, using simplified model systems representing distinct challenges in complex three-dimensional terrain. Remarkably, general physical principles emerge across diverse model systems, by modelling locomotor-terrain interaction using a potential energy landscape approach. The animal and robots' stereotyped locomotor modes are constrained by physical interaction. Locomotor transitions are stochastic, destabilizing, barrier-crossing transitions on the landscape. They can be induced by feed-forward self-propulsion and are facilitated by feedback-controlled active adjustment. General physical principles and strategies from our systematic studies already advanced robot performance in simple model systems. Efforts remain to better understand the intelligence aspect of locomotor transitions and how to compose larger-scale potential energy landscapes of complex three-dimensional terrains from simple landscapes of abstracted challenges. This will elucidate how the neuromechanical control system mediates physical interaction to generate multi-pathway locomotor transitions and lead to advancements in biology, physics, robotics and dynamical systems theory.

Mining naturalistic human behaviors in long-term video and neural recordings

BACKGROUND

Recent technological advances in brain recording and machine learning algorithms are enabling the study of neural activity underlying spontaneous human behaviors, beyond the confines of cued, repeated trials. However, analyzing such unstructured data lacking a priori experimental design remains a significant challenge, especially when the data is multi-modal and long-term.

NEW METHOD

Here we describe an automated, behavior-first approach for analyzing simultaneously recorded long-term, naturalistic electrocorticography (ECoG) and behavior video data. We identify and characterize spontaneous human upper-limb movements by combining computer vision, discrete latent-variable modeling, and string pattern-matching on the video.

RESULTS

Our pipeline discovers and annotates over 40,000 instances of naturalistic arm movements in long term (7-9 day) behavioral videos, across 12 subjects. Analysis of the simultaneously recorded brain data reveals neural signatures of movement that corroborate previous findings. Our pipeline produces large training datasets for brain-computer interfacing applications, and we show decoding results from a movement initiation detection task.

COMPARISON WITH EXISTING METHODS

Spontaneous movements capture real-world neural and behavior variability that is missing from traditional cued tasks. Building beyond window-based movement detection metrics, our unsupervised discretization scheme produces a queryable pose representation, allowing localization of movements with finer temporal resolution.

CONCLUSIONS

Our work addresses the unique analytic challenges of studying naturalistic human behaviors and contributes methods that may generalize to other neural recording modalities beyond ECoG. We publish our curated dataset and believe that it will be a valuable resource for future studies of naturalistic movements.

Animal behavior and animal personality from a non-human perspective: Getting help from the machine

We can now track the position of every fly's leg or immerse a tiny fish inside a virtual world by monitoring its gaze in real time. Yet capturing animals' posture or gaze is not like understanding their behavior. Instead, behaviors are still often interpreted by human observers in an anthropomorphic manner. Even newer tools that automatically classify behaviors rely on human observers for the choice of behaviors. In this perspective, we suggest a roadmap toward a "human-free" interpretation of behavior. We present several recent advances, including our recent work on animal personalities. Personality both underlies behavioral differences among individuals and is consistent over time. A mathematical formulation of this idea has allowed us to measure mouse traits objectively, map behaviors across species (humans included), and explore the biological basis of behavior. Our goal is to enable "machine translation" of raw movement data into intelligible human concepts en route to improving our understanding of animals and people.

Learning compositional sequences with multiple time scales through a hierarchical network of spiking neurons

Sequential behaviour is often compositional and organised across multiple time scales: a set of individual elements developing on short time scales (motifs) are combined to form longer functional sequences (syntax). Such organisation leads to a natural hierarchy that can be used advantageously for learning, since the motifs and the syntax can be acquired independently. Despite mounting experimental evidence for hierarchical structures in neuroscience, models for temporal learning based on neuronal networks have mostly focused on serial methods. Here, we introduce a network model of spiking neurons with a hierarchical organisation aimed at sequence learning on multiple time scales. Using biophysically motivated neuron dynamics and local plasticity rules, the model can learn motifs and syntax independently. Furthermore, the model can relearn sequences efficiently and store multiple sequences. Compared to serial learning, the hierarchical model displays faster learning, more flexible relearning, increased capacity, and higher robustness to perturbations. The hierarchical model redistributes the variability: it achieves high motif fidelity at the cost of higher variability in the between-motif timings.

Continuous Whole-Body 3D Kinematic Recordings across the Rodent Behavioral Repertoire

In mammalian animal models, high-resolution kinematic tracking is restricted to brief sessions in constrained environments, limiting our ability to probe naturalistic behaviors and their neural underpinnings. To address this, we developed CAPTURE (Continuous Appendicular and Postural Tracking Using Retroreflector Embedding), a behavioral monitoring system that combines motion capture and deep learning to continuously track the 3D kinematics of a rat's head, trunk, and limbs for week-long timescales in freely behaving animals. CAPTURE realizes 10- to 100-fold gains in precision and robustness compared with existing convolutional network approaches to behavioral tracking. We demonstrate CAPTURE's ability to comprehensively profile the kinematics and sequential organization of natural rodent behavior, its variation across individuals, and its perturbation by drugs and disease, including identifying perseverative grooming states in a rat model of fragile X syndrome. CAPTURE significantly expands the range of behaviors and contexts that can be quantitatively investigated, opening the door to a new understanding of natural behavior and its neural basis.

People accurately predict the transition probabilities between actions

Social life is a complex dance. To coordinate gracefully with one's partners, one must predict their actions. Here, we investigated how people predict others' actions. We hypothesized that people can accurately predict others' future actions based on knowledge of their current actions, coupled with knowledge of action transitions. To test whether people have accurate knowledge of the transition probabilities between actions, we compared actual rates of action transitions-calculated from four large naturalistic datasets-to participants' ratings of the transition probabilities between corresponding sets of actions. In five preregistered studies, participants demonstrated accurate mental models of action transitions. Furthermore, we found that people drew upon conceptual knowledge of actions-described by the six-dimensional ACT-FASTaxonomy-to guide their accurate predictions. Together, these results indicate that people can accurately anticipate other people's moves in the dance of social life and that the structure of action knowledge may be tailored to making these predictions.

Investigating rare and ultrarare epilepsy syndromes with Drosophila models

One in three epilepsy cases is drug resistant, and seizures often begin in infancy, when they are life-threatening and when therapeutic options are highly limited. An important tool for prioritizing and validating genes associated with epileptic conditions, which is suitable for large-scale screening, is disease modeling in Drosophila. Approximately two-thirds of disease genes are conserved in Drosophila, and gene-specific fly models exhibit behavioral changes that are related to symptoms of epilepsy. Models are based on behavior readouts, seizure-like attacks and paralysis following stimulation, and neuronal, cell-biological readouts that are in the majority based on changes in nerve cell activity or morphology. In this review, we focus on behavioral phenotypes. Importantly, Drosophila modeling is independent of, and complementary to, other approaches that are computational and based on systems analysis. The large number of known epilepsy-associated gene variants indicates a need for efficient research strategies. We will discuss the status quo of epilepsy disease modelling in Drosophila and describe promising steps towards the development of new drugs to reduce seizure rates and alleviate other epileptic symptoms.

Modeling brain, symptom, and behavior in the winds of change

Neuropsychopharmacology addresses pressing questions in the study of three intertwined complex systems: the brain, human behavior, and symptoms of illness. The field seeks to understand the perturbations that impinge upon those systems, either driving greater health or illness. In the pursuit of this aim, investigators often perform analyses that make certain assumptions about the nature of the systems that are being perturbed. Those assumptions can be encoded in powerful computational models that serve to bridge the wide gulf between a descriptive analysis and a formal theory of a system's response. Here we review a set of three such models along a continuum of complexity, moving from a local treatment to a network treatment: one commonly applied form of the general linear model, impulse response models, and network control models. For each, we describe the model's basic form, review its use in the field, and provide a frank assessment of its relative strengths and weaknesses. The discussion naturally motivates future efforts to interlink data analysis, computational modeling, and formal theory. Our goal is to inspire practitioners to consider the assumptions implicit in their analytical approach, align those assumptions to the complexity of the systems under study, and take advantage of exciting recent advances in modeling the relations between perturbations and system function.

Confronting pitfalls of AI-augmented molecular dynamics using statistical physics

Artificial intelligence (AI)-based approaches have had indubitable impact across the sciences through the ability to extract relevant information from raw data. Recently, AI has also found use in enhancing the efficiency of molecular simulations, wherein AI derived slow modes are used to accelerate the simulation in targeted ways. However, while typical fields where AI is used are characterized by a plethora of data, molecular simulations, per construction, suffer from limited sampling and thus limited data. As such, the use of AI in molecular simulations can suffer from a dangerous situation where the AI-optimization could get stuck in spurious regimes, leading to incorrect characterization of the reaction coordinate (RC) for the problem at hand. When such an incorrect RC is then used to perform additional simulations, one could start to deviate progressively from the ground truth. To deal with this problem of spurious AI-solutions, here, we report a novel and automated algorithm using ideas from statistical mechanics. It is based on the notion that a more reliable AI-solution will be one that maximizes the timescale separation between slow and fast processes. To learn this timescale separation even from limited data, we use a maximum caliber-based framework. We show the applicability of this automatic protocol for three classic benchmark problems, namely, the conformational dynamics of a model peptide, ligand-unbinding from a protein, and folding/unfolding energy landscape of the C-terminal domain of protein G. We believe that our work will lead to increased and robust use of trustworthy AI in molecular simulations of complex systems.

A High-Dimensional Quantification of Mouse Defensive Behaviors Reveals Enhanced Diversity and Stimulus Specificity

Instinctive defensive behaviors, consisting of stereotyped sequences of movements and postures, are an essential component of the mouse behavioral repertoire. Since defensive behaviors can be reliably triggered by threatening sensory stimuli, the selection of the most appropriate action depends on the stimulus property. However, since the mouse has a wide repertoire of motor actions, it is not clear which set of movements and postures represent the relevant action. So far, this has been empirically identified as a change in locomotion state. However, the extent to which locomotion alone captures the diversity of defensive behaviors and their sensory specificity is unknown. To tackle this problem, we developed a method to obtain a faithful 3D reconstruction of the mouse body that enabled to quantify a wide variety of motor actions. This higher dimensional description revealed that defensive behaviors are more stimulus specific than indicated by locomotion data. Thus, responses to distinct stimuli that were equivalent in terms of locomotion (e.g., freezing induced by looming and sound) could be discriminated along other dimensions. The enhanced stimulus specificity was explained by a surprising diversity. A clustering analysis revealed that distinct combinations of movements and postures, giving rise to at least 7 different behaviors, were required to account for stimulus specificity. Moreover, each stimulus evoked more than one behavior, revealing a robust one-to-many mapping between sensations and behaviors that was not apparent from locomotion data. Our results indicate that diversity and sensory specificity of mouse defensive behaviors unfold in a higher dimensional space, spanning multiple motor actions.

Quantifying behavior to understand the brain

Over the past years, numerous methods have emerged to automate the quantification of animal behavior at a resolution not previously imaginable. This has opened up a new field of computational ethology and will, in the near future, make it possible to quantify in near completeness what an animal is doing as it navigates its environment. The importance of improving the techniques with which we characterize behavior is reflected in the emerging recognition that understanding behavior is an essential (or even prerequisite) step to pursuing neuroscience questions. The use of these methods, however, is not limited to studying behavior in the wild or in strictly ethological settings. Modern tools for behavioral quantification can be applied to the full gamut of approaches that have historically been used to link brain to behavior, from psychophysics to cognitive tasks, augmenting those measurements with rich descriptions of how animals navigate those tasks. Here we review recent technical advances in quantifying behavior, particularly in methods for tracking animal motion and characterizing the structure of those dynamics. We discuss open challenges that remain for behavioral quantification and highlight promising future directions, with a strong emphasis on emerging approaches in deep learning, the core technology that has enabled the markedly rapid pace of progress of this field. We then discuss how quantitative descriptions of behavior can be leveraged to connect brain activity with animal movements, with the ultimate goal of resolving the relationship between neural circuits, cognitive processes and behavior.

Rethinking brain-wide interactions through multi-region 'network of networks' models

The neural control of behavior is distributed across many functionally and anatomically distinct brain regions even in small nervous systems. While classical neuroscience models treated these regions as a set of hierarchically isolated nodes, the brain comprises a recurrently interconnected network in which each region is intimately modulated by many others. Uncovering these interactions is now possible through experimental techniques that access large neural populations from many brain regions simultaneously. Harnessing these large-scale datasets, however, requires new theoretical approaches. Here, we review recent work to understand brain-wide interactions using multi-region 'network of networks' models and discuss how they can guide future experiments. We also emphasize the importance of multi-region recordings, and posit that studying individual components in isolation will be insufficient to understand the neural basis of behavior.

Reward foraging task and model-based analysis reveal how fruit flies learn value of available options

Foraging animals have to evaluate, compare and select food patches in order to increase their fitness. Understanding what drives foraging decisions requires careful manipulation of the value of alternative options while monitoring animals choices. Value-based decision-making tasks in combination with formal learning models have provided both an experimental and theoretical framework to study foraging decisions in lab settings. While these approaches were successfully used in the past to understand what drives choices in mammals, very little work has been done on fruit flies. This is despite the fact that fruit flies have served as model organism for many complex behavioural paradigms. To fill this gap we developed a single-animal, trial-based decision making task, where freely walking flies experienced optogenetic sugar-receptor neuron stimulation. We controlled the value of available options by manipulating the probabilities of optogenetic stimulation. We show that flies integrate reward history of chosen options and forget value of unchosen options. We further discover that flies assign higher values to rewards experienced early in the behavioural session, consistent with formal reinforcement learning models. Finally, we also show that the probabilistic rewards affect walking trajectories of flies, suggesting that accumulated value is controlling the navigation vector of flies in a graded fashion. These findings establish the fruit fly as a model organism to explore the genetic and circuit basis of reward foraging decisions.

Paired fruit flies synchronize behavior: Uncovering social interactions in Drosophila melanogaster

Social behaviors are ubiquitous and crucial to an animal's survival and success. The behaviors an animal performs in a social setting are affected by internal factors, inputs from the environment, and interactions with others. To quantify social behaviors, we need to measure both the stochastic nature of the behavior of isolated individuals and how this behavioral repertoire changes as a function of the environment and interactions between individuals. We probed the behavior of male and female fruit flies in a circular arena as individuals and within all possible pairings. By combining measurements of the animals' position in the arena with an unsupervised analysis of their behaviors, we define the effects of position in the environment and the presence of a partner on locomotion, grooming, singing, and other behaviors that make up an animal's repertoire. We find that geometric context tunes behavioral preference, pairs of animals synchronize their behavioral preferences across shared trials, and paired individuals display signatures of behavioral mimicry.

Finding, visualizing, and quantifying latent structure across diverse animal vocal repertoires

Animals produce vocalizations that range in complexity from a single repeated call to hundreds of unique vocal elements patterned in sequences unfolding over hours. Characterizing complex vocalizations can require considerable effort and a deep intuition about each species' vocal behavior. Even with a great deal of experience, human characterizations of animal communication can be affected by human perceptual biases. We present a set of computational methods for projecting animal vocalizations into low dimensional latent representational spaces that are directly learned from the spectrograms of vocal signals. We apply these methods to diverse datasets from over 20 species, including humans, bats, songbirds, mice, cetaceans, and nonhuman primates. Latent projections uncover complex features of data in visually intuitive and quantifiable ways, enabling high-powered comparative analyses of vocal acoustics. We introduce methods for analyzing vocalizations as both discrete sequences and as continuous latent variables. Each method can be used to disentangle complex spectro-temporal structure and observe long-timescale organization in communication.

A combination of machine learning and infrequent metadynamics to efficiently predict kinetic rates, transition states, and molecular determinants of drug dissociation from G protein-coupled receptors

Determining the drug-target residence time (RT) is of major interest in drug discovery given that this kinetic parameter often represents a better indicator of in vivo drug efficacy than binding affinity. However, obtaining drug-target unbinding rates poses significant challenges, both computationally and experimentally. This is particularly palpable for complex systems like G Protein-Coupled Receptors (GPCRs) whose ligand unbinding typically requires very long timescales oftentimes inaccessible by standard molecular dynamics simulations. Enhanced sampling methods offer a useful alternative, and their efficiency can be further improved by using machine learning tools to identify optimal reaction coordinates. Here, we test the combination of two machine learning techniques, automatic mutual information noise omission and reweighted autoencoded variational Bayes for enhanced sampling, with infrequent metadynamics to efficiently study the unbinding kinetics of two classical drugs with different RTs in a prototypic GPCR, the μ-opioid receptor. Dissociation rates derived from these computations are within one order of magnitude from experimental values. We also use the simulation data to uncover the dissociation mechanisms of these drugs, shedding light on the structures of rate-limiting transition states, which, alongside metastable poses, are difficult to obtain experimentally but important to visualize when designing drugs with a desired kinetic profile.

Context-dependent modulation of natural approach behaviour in mice

Specific features of visual objects innately draw approach responses in animals, and provide natural signals of potential reward. However, visual sampling behaviours and the detection of salient, rewarding stimuli are context and behavioural state-dependent and it remains unclear how visual perception and orienting responses change with specific expectations. To start to address this question, we employed a virtual stimulus orienting paradigm based on prey capture to quantify the conditional expression of visual stimulus-evoked innate approaches in freely moving mice. We found that specific combinations of stimulus features selectively evoked innate approach or freezing responses when stimuli were unexpected. We discovered that prey capture experience, and therefore the expectation of prey in the environment, selectively modified approach frequency, as well as altered those visual features that evoked approach. Thus, we found that mice exhibit robust and selective orienting responses to parameterized visual stimuli that can be robustly and specifically modified via natural experience. This work provides critical insight into how natural appetitive behaviours are driven by both specific features of visual motion and internal states that alter stimulus salience.

Hierarchical Compression Reveals Sub-Second to Day-Long Structure in Larval Zebrafish Behavior

Animal behavior is dynamic, evolving over multiple timescales from milliseconds to days and even across a lifetime. To understand the mechanisms governing these dynamics, it is necessary to capture multi-timescale structure from behavioral data. Here, we develop computational tools and study the behavior of hundreds of larval zebrafish tracked continuously across multiple 24-h day/night cycles. We extracted millions of movements and pauses, termed bouts, and used unsupervised learning to reduce each larva’s behavior to an alternating sequence of active and inactive bout types, termed modules. Through hierarchical compression, we identified recurrent behavioral patterns, termed motifs. Module and motif usage varied across the day/night cycle, revealing structure at sub-second to day-long timescales. We further demonstrate that module and motif analysis can uncover novel pharmacological and genetic mutant phenotypes. Overall, our work reveals the organization of larval zebrafish behavior at multiple timescales and provides tools to identify structure from large-scale behavioral datasets.

Object-Displacement-Sensitive Visual Neurons Drive Freezing in Drosophila

Visual systems are often equipped with neurons that detect small moving objects, which may represent prey, predators, or conspecifics. Although the processing properties of those neurons have been studied in diverse organisms, links between the proposed algorithms and animal behaviors or circuit mechanisms remain elusive. Here, we have investigated behavioral function, computational algorithm, and neurochemical mechanisms of an object-selective neuron, LC11, in Drosophila. With genetic silencing and optogenetic activation, we show that LC11 is necessary for a visual object-induced stopping behavior in walking flies, a form of short-term freezing, and its activity can promote stopping. We propose a new quantitative model for small object selectivity based on the physiology and anatomy of LC11 and its inputs. The model accurately reproduces LC11 responses by pooling fast-adapting, tightly size-tuned inputs. Direct visualization of neurotransmitter inputs to LC11 confirmed the model conjectures about upstream processing. Our results demonstrate how adaptation can enhance selectivity for behaviorally relevant, dynamic visual features.

An energy landscape approach to locomotor transitions in complex 3D terrain

Effective locomotion in nature happens by transitioning across multiple modes (e.g., walk, run, climb). Despite this, far more mechanistic understanding of terrestrial locomotion has been on how to generate and stabilize around near-steady-state movement in a single mode. We still know little about how locomotor transitions emerge from physical interaction with complex terrain. Consequently, robots largely rely on geometric maps to avoid obstacles, not traverse them. Recent studies revealed that locomotor transitions in complex three-dimensional (3D) terrain occur probabilistically via multiple pathways. Here, we show that an energy landscape approach elucidates the underlying physical principles. We discovered that locomotor transitions of animals and robots self-propelled through complex 3D terrain correspond to barrier-crossing transitions on a potential energy landscape. Locomotor modes are attracted to landscape basins separated by potential energy barriers. Kinetic energy fluctuation from oscillatory self-propulsion helps the system stochastically escape from one basin and reach another to make transitions. Escape is more likely toward lower barrier direction. These principles are surprisingly similar to those of near-equilibrium, microscopic systems. Analogous to free-energy landscapes for multipathway protein folding transitions, our energy landscape approach from first principles is the beginning of a statistical physics theory of multipathway locomotor transitions in complex terrain. This will not only help understand how the organization of animal behavior emerges from multiscale interactions between their neural and mechanical systems and the physical environment, but also guide robot design, control, and planning over the large, intractable locomotor-terrain parameter space to generate robust locomotor transitions through the real world.

Brain-wide representations of ongoing behavior: a universal principle?

Recent neuronal activity recordings of unprecedented breadth and depth in worms, flies, and mice have uncovered a surprising common feature: brain-wide behavior-related signals. These signals pervade, and even dominate, neuronal populations thought to function primarily in sensory processing. Such convergent findings across organisms suggest that brain-wide representations of behavior might be a universal neuroscientific principle. What purpose(s) do these representations serve? Here we review these findings along with suggested functions, including sensory prediction, context-dependent sensory processing, and, perhaps most speculatively, distributed motor command generation. It appears that a large proportion of the brain's energy and coding capacity is used to represent ongoing behavior; understanding the function of these representations should therefore be a major goal in neuroscience research.

Imaging brain activity during complex social behaviors in Drosophila with Flyception2

Optical in vivo recordings from freely walking Drosophila are currently possible only for limited behaviors. Here, we expand the range of accessible behaviors with a retroreflective marker-based tracking and ratiometric brain imaging system, permitting brain activity imaging even in copulating male flies. We discover that P1 neurons, active during courtship, are inactive during copulation, whereas GABAergic mAL neurons remain active during copulation, suggesting a countervailing role of mAL in opposing P1 activity during mating.

Tools for Rapid High-Resolution Behavioral Phenotyping of Automatically Isolated Drosophila

Sparse manipulation of neuron excitability during free behavior is critical for identifying neural substrates of behavior. Genetic tools for precise neuronal manipulation exist in the fruit fly, Drosophila melanogaster, but behavioral tools are still lacking to identify potentially subtle phenotypes only detectible using high-throughput and high spatiotemporal resolution. We developed three assay components that can be used modularly to study natural and optogenetically induced behaviors. FlyGate automatically releases flies one at a time into an assay. FlyDetect tracks flies in real time, is robust to severe occlusions, and can be used to track appendages, such as the head. GlobeDisplay is a spherical projection system covering the fly's visual receptive field with a single projector. We demonstrate the utility of these components in an integrated system, FlyPEZ, by comprehensively modeling the input-output function for directional looming-evoked escape takeoffs and describing a millisecond-timescale phenotype from genetic silencing of a single visual projection neuron type.

Nested Neuronal Dynamics Orchestrate a Behavioral Hierarchy across Timescales

Classical and modern ethological studies suggest that animal behavior is organized hierarchically across timescales, such that longer-timescale behaviors are composed of specific shorter-timescale actions. Despite progress relating neuronal dynamics to single-timescale behavior, it remains unclear how different timescale dynamics interact to give rise to such higher-order behavioral organization. Here, we show, in the nematode Caenorhabditis elegans, that a behavioral hierarchy spanning three timescales is implemented by nested neuronal dynamics. At the uppermost hierarchical level, slow neuronal population dynamics spanning brain and motor periphery control two faster motor neuron oscillations, toggling them between different activity states and functional roles. At lower hierarchical levels, these faster oscillations are further nested in a manner that enables flexible behavioral control in an otherwise rigid hierarchical framework. Our findings establish nested neuronal activity patterns as a repeated dynamical motif of the C. elegans nervous system, which together implement a controllable hierarchical organization of behavior.

•

Slow dynamics across brain and motor circuits drive upper-hierarchy motor states

•

Fast dynamics in motor circuits drive lower-hierarchy movements within these states

•

Slower dynamics tightly constrain the state and function of faster ones

•

This rigid hierarchy nevertheless enables flexible behavioral control

Unsupervised identification of the internal states that shape natural behavior

Internal states shape stimulus responses and decision-making, but we lack methods to identify them. To address this gap, we developed an unsupervised method to identify internal states from behavioral data and applied it to a dynamic social interaction. During courtship, Drosophila melanogaster males pattern their songs using feedback cues from their partner. Our model uncovers three latent states underlying this behavior and is able to predict moment-to-moment variation in song-patterning decisions. These states correspond to different sensorimotor strategies, each of which is characterized by different mappings from feedback cues to song modes. We show that a pair of neurons previously thought to be command neurons for song production are sufficient to drive switching between states. Our results reveal how animals compose behavior from previously unidentified internal states, which is a necessary step for quantitative descriptions of animal behavior that link environmental cues, internal needs, neuronal activity and motor outputs.

Computational Neuroethology: A Call to Action

The brain is worthy of study because it is in charge of behavior. A flurry of recent technical advances in measuring and quantifying naturalistic behaviors provide an important opportunity for advancing brain science. However, the problem of understanding unrestrained behavior in the context of neural recordings and manipulations remains unsolved, and developing approaches to addressing this challenge is critical. Here we discuss considerations in computational neuroethology-the science of quantifying naturalistic behaviors for understanding the brain-and propose strategies to evaluate progress. We point to open questions that require resolution and call upon the broader systems neuroscience community to further develop and leverage measures of naturalistic, unrestrained behavior, which will enable us to more effectively probe the richness and complexity of the brain.

An automatic behavior recognition system classifies animal behaviors using movements and their temporal context

Animals can perform complex and purposeful behaviors by executing simpler movements in flexible sequences. It is particularly challenging to analyze behavior sequences when they are highly variable, as is the case in language production, certain types of birdsong and, as in our experiments, flies grooming. High sequence variability necessitates rigorous quantification of large amounts of data to identify organizational principles and temporal structure of such behavior. To cope with large amounts of data, and minimize human effort and subjective bias, researchers often use automatic behavior recognition software. Our standard grooming assay involves coating flies in dust and videotaping them as they groom to remove it. The flies move freely and so perform the same movements in various orientations. As the dust is removed, their appearance changes. These conditions make it difficult to rely on precise body alignment and anatomical landmarks such as eyes or legs and thus present challenges to existing behavior classification software. Human observers use speed, location, and shape of the movements as the diagnostic features of particular grooming actions. We applied this intuition to design a new automatic behavior recognition system (ABRS) based on spatiotemporal features in the video data, heavily weighted for temporal dynamics and invariant to the animal's position and orientation in the scene. We use these spatiotemporal features in two steps of supervised classification that reflect two time-scales at which the behavior is structured. As a proof of principle, we show results from quantification and analysis of a large data set of stimulus-induced fly grooming behaviors that would have been difficult to assess in a smaller dataset of human-annotated ethograms. While we developed and validated this approach to analyze fly grooming behavior, we propose that the strategy of combining alignment-invariant features and multi-timescale analysis may be generally useful for movement-based classification of behavior from video data.

DeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning

Quantitative behavioral measurements are important for answering questions across scientific disciplines-from neuroscience to ecology. State-of-the-art deep-learning methods offer major advances in data quality and detail by allowing researchers to automatically estimate locations of an animal's body parts directly from images or videos. However, currently available animal pose estimation methods have limitations in speed and robustness. Here, we introduce a new easy-to-use software toolkit, DeepPoseKit, that addresses these problems using an efficient multi-scale deep-learning model, called Stacked DenseNet, and a fast GPU-based peak-detection algorithm for estimating keypoint locations with subpixel precision. These advances improve processing speed >2x with no loss in accuracy compared to currently available methods. We demonstrate the versatility of our methods with multiple challenging animal pose estimation tasks in laboratory and field settings-including groups of interacting individuals. Our work reduces barriers to using advanced tools for measuring behavior and has broad applicability across the behavioral sciences.

Parallels in the sequential organization of birdsong and human speech

Human speech possesses a rich hierarchical structure that allows for meaning to be altered by words spaced far apart in time. Conversely, the sequential structure of nonhuman communication is thought to follow non-hierarchical Markovian dynamics operating over only short distances. Here, we show that human speech and birdsong share a similar sequential structure indicative of both hierarchical and Markovian organization. We analyze the sequential dynamics of song from multiple songbird species and speech from multiple languages by modeling the information content of signals as a function of the sequential distance between vocal elements. Across short sequence-distances, an exponential decay dominates the information in speech and birdsong, consistent with underlying Markovian processes. At longer sequence-distances, the decay in information follows a power law, consistent with underlying hierarchical processes. Thus, the sequential organization of acoustic elements in two learned vocal communication signals (speech and birdsong) shows functionally equivalent dynamics, governed by similar processes.

Past-future information bottleneck for sampling molecular reaction coordinate simultaneously with thermodynamics and kinetics

The ability to rapidly learn from high-dimensional data to make reliable bets about the future is crucial in many contexts. This could be a fly avoiding predators, or the retina processing gigabytes of data to guide human actions. In this work we draw parallels between these and the efficient sampling of biomolecules with hundreds of thousands of atoms. For this we use the Predictive Information Bottleneck framework used for the first two problems, and re-formulate it for the sampling of biomolecules, especially when plagued with rare events. Our method uses a deep neural network to learn the minimally complex yet most predictive aspects of a given biomolecular trajectory. This information is used to perform iteratively biased simulations that enhance the sampling and directly obtain associated thermodynamic and kinetic information. We demonstrate the method on two test-pieces, studying processes slower than milliseconds, calculating free energies, kinetics and critical mutations.

Efficient sampling of rare events in all-atom molecular dynamics simulations remains a challenge. Here, the authors adapt the Predictive Information Bottleneck framework to sample biomolecular structure and dynamics through iterative rounds of biased simulations and deep learning.

Drosophila melanogaster grooming possesses syntax with distinct rules at different temporal scales

Mathematical modeling of behavioral sequences yields insight into the rules and mechanisms underlying sequence generation. Grooming in Drosophila melanogaster is characterized by repeated execution of distinct, stereotyped actions in variable order. Experiments demonstrate that, following stimulation by an irritant, grooming progresses gradually from an early phase dominated by anterior cleaning to a later phase with increased walking and posterior cleaning. We also observe that, at an intermediate temporal scale, there is a strong relationship between the amount of time spent performing body-directed grooming actions and leg-directed actions. We then develop a series of data-driven Markov models that isolate and identify the behavioral features governing transitions between individual grooming bouts. We identify action order as the primary driver of probabilistic, but non-random, syntax structure, as has previously been identified. Subsequent models incorporate grooming bout duration, which also contributes significantly to sequence structure. Our results show that, surprisingly, the syntactic rules underlying probabilistic grooming transitions possess action duration-dependent structure, suggesting that sensory input-independent mechanisms guide grooming behavior at short time scales. Finally, the inclusion of a simple rule that modifies grooming transition probabilities over time yields a generative model that recapitulates the key features of observed grooming sequences at several time scales. These discoveries suggest that sensory input guides action selection by modulating internally generated dynamics. Additionally, the discovery of these principles governing grooming in D. melanogaster demonstrates the utility of incorporating temporal information when characterizing the syntax of behavioral sequences.

Analysis of temporally rich behavioral sequences provides a quantitative description of the rules underlying their generation. Drosophila melanogaster grooming behavior consists of many complex sequences involving repetitions of well-characterized actions. In this paper, we leverage advances in machine vision to automatically annotate over 40 hours of video data of flies covered in dust and develop mathematical models that reveal the existence of syntax in D. melanogaster grooming. We find that sequence organization depends on grooming action identity, as has been well-established, and, more surprisingly, grooming action duration. The discovery of duration-dependent action selection leads us to conclude that, although sensory input informs grooming decisions on long time scales, internal dynamics also guide individual transitions between grooming actions. Therefore, incorporating action duration into our models allows us to uncover multi-scale temporal dynamics that suggest the existence of neural circuits dedicated to partially sensory-independent decision-making. Our approach highlights the importance of incorporating temporal information into sequential models, as doing so reveals the relative contributions of sensory input and internal dynamics to behavioral sequence generation.

Statistical structure of locomotion and its modulation by odors

Most behaviors such as making tea are not stereotypical but have an obvious structure. However, analytical methods to objectively extract structure from non-stereotyped behaviors are immature. In this study, we analyze the locomotion of fruit flies and show that this non-stereotyped behavior is well-described by a Hierarchical Hidden Markov Model (HHMM). HHMM shows that a fly's locomotion can be decomposed into a few locomotor features, and odors modulate locomotion by altering the time a fly spends performing different locomotor features. Importantly, although all flies in our dataset use the same set of locomotor features, individual flies vary considerably in how often they employ a given locomotor feature, and how this usage is modulated by odor. This variation is so large that the behavior of individual flies is best understood as being grouped into at least three to five distinct clusters, rather than variations around an average fly.

Fast animal pose estimation using deep neural networks

The need for automated and efficient systems for tracking full animal pose has increased with the complexity of behavioral data and analyses. Here we introduce LEAP (LEAP estimates animal pose), a deep-learning-based method for predicting the positions of animal body parts. This framework consists of a graphical interface for labeling of body parts and training the network. LEAP offers fast prediction on new data, and training with as few as 100 frames results in 95% of peak performance. We validated LEAP using videos of freely behaving fruit flies and tracked 32 distinct points to describe the pose of the head, body, wings and legs, with an error rate of <3% of body length. We recapitulated reported findings on insect gait dynamics and demonstrated LEAP's applicability for unsupervised behavioral classification. Finally, we extended the method to more challenging imaging situations and videos of freely moving mice.

Time Irreversibility and Criticality in the Motility of a Flagellate Microorganism

Active living organisms exhibit behavioral variability, partitioning between fast and slow dynamics. Such variability may be key to generating rapid responses in a heterogeneous, unpredictable environment wherein cellular activity effects continual exchanges of energy fluxes. We demonstrate a novel, noninvasive strategy for revealing nonequilibrium control of swimming-specifically, in an octoflagellate microalga. These organisms exhibit surprising features of flagellar excitability and mechanosensitivity, which characterize a novel, time-irreversible "run-stop-shock" motility comprising forward runs, knee-jerk shocks with dramatic beat reversal, and long stops during which cells are quiescent yet continue to exhibit submicron flagellar vibrations. Entropy production, associated with flux cycles arising in a reaction graph representation of the gait-switching dynamics, provides a direct measure of detailed balance violation in this primitive alga.

Statistical modelling of navigational decisions based on intensity versus directionality in Drosophila larval phototaxis

Organisms use environmental cues for directed navigation. Understanding the basic logic behind navigational decisions critically depends on the complexity of the nervous system. Due to the comparably simple organization of the nervous system of the fruit fly larva, it stands as a powerful model to study decision-making processes that underlie directed navigation. We have quantitatively measured phototaxis in response to well-defined sensory inputs. Subsequently, we have formulated a statistical stochastic model based on biased Markov chains to characterize the behavioural basis of negative phototaxis. Our experiments show that larvae make navigational decisions depending on two independent physical variables: light intensity and its spatial gradient. Furthermore, our statistical model quantifies how larvae balance two potentially-contradictory factors: minimizing exposure to light intensity and at the same time maximizing their distance to the light source. We find that the response to the light field is manifestly non-linear, and saturates above an intensity threshold. The model has been validated against our experimental biological data yielding insight into the strategy that larvae use to achieve their goal with respect to the navigational cue of light, an important piece of information for future work to study the role of the different neuronal components in larval phototaxis.

Challenges and solutions for studying collective animal behaviour in the wild

Mobile animal groups provide some of the most compelling examples of self-organization in the natural world. While field observations of songbird flocks wheeling in the sky or anchovy schools fleeing from predators have inspired considerable interest in the mechanics of collective motion, the challenge of simultaneously monitoring multiple animals in the field has historically limited our capacity to study collective behaviour of wild animal groups with precision. However, recent technological advancements now present exciting opportunities to overcome many of these limitations. Here we review existing methods used to collect data on the movements and interactions of multiple animals in a natural setting. We then survey emerging technologies that are poised to revolutionize the study of collective animal behaviour by extending the spatial and temporal scales of inquiry, increasing data volume and quality, and expediting the post-processing of raw data.This article is part of the theme issue 'Collective movement ecology'.

Measuring behavior across scales

The need for high-throughput, precise, and meaningful methods for measuring behavior has been amplified by our recent successes in measuring and manipulating neural circuitry. The largest challenges associated with moving in this direction, however, are not technical but are instead conceptual: what numbers should one put on the movements an animal is performing (or not performing)? In this review, I will describe how theoretical and data analytical ideas are interfacing with recently-developed computational and experimental methodologies to answer these questions across a variety of contexts, length scales, and time scales. I will attempt to highlight commonalities between approaches and areas where further advances are necessary to place behavior on the same quantitative footing as other scientific fields.

Quantifying behavior to solve sensorimotor transformations: advances from worms and flies

The development of new computational tools has recently opened up the study of natural behaviors at a precision that was previously unachievable. These tools permit a highly quantitative analysis of behavioral dynamics at timescales that are well matched to the timescales of neural activity. Here we examine how combining these methods with established techniques for estimating an animal's sensory experience presents exciting new opportunities for dissecting the sensorimotor transformations performed by the nervous system. We focus this review primarily on examples from Caenorhabditis elegans and Drosophila melanogaster-for these model systems, computational approaches to characterize behavior, in combination with unparalleled genetic tools for neural activation, silencing, and recording, have already proven instrumental for illuminating underlying neural mechanisms.

Dynamic structure of locomotor behavior in walking fruit flies

The function of the brain is unlikely to be understood without an accurate description of its output, yet the nature of movement elements and their organization remains an open problem. Here, movement elements are identified from dynamics of walking in flies, using unbiased criteria. On one time scale, dynamics of walking are consistent over hundreds of milliseconds, allowing elementary features to be defined. Over longer periods, walking is well described by a stochastic process composed of these elementary features, and a generative model of this process reproduces individual behavior sequences accurately over seconds or longer. Within elementary features, velocities diverge, suggesting that dynamical stability of movement elements is a weak behavioral constraint. Rather, long-term instability can be limited by the finite memory between these elementary features. This structure suggests how complex dynamics may arise in biological systems from elements whose combination need not be tuned for dynamic stability.

FlyLimbTracker: An active contour based approach for leg segment tracking in unmarked, freely behaving Drosophila

Understanding the biological underpinnings of movement and action requires the development of tools for quantitative measurements of animal behavior. Drosophila melanogaster provides an ideal model for developing such tools: the fly has unparalleled genetic accessibility and depends on a relatively compact nervous system to generate sophisticated limbed behaviors including walking, reaching, grooming, courtship, and boxing. Here we describe a method that uses active contours to semi-automatically track body and leg segments from video image sequences of unmarked, freely behaving D. melanogaster. We show that this approach yields a more than 6-fold reduction in user intervention when compared with fully manual annotation and can be used to annotate videos with low spatial or temporal resolution for a variety of locomotor and grooming behaviors. FlyLimbTracker, the software implementation of this method, is open-source and our approach is generalizable. This opens up the possibility of tracking leg movements in other species by modifications of underlying active contour models.