Identifying behavioral structure from deep variational embeddings of animal motion

A study of animal action segmentation algorithms across supervised, unsupervised, and semi-supervised learning paradigms

Action segmentation of behavioral videos is the process of labeling each frame as belonging to one or more discrete classes, and is a crucial component of many studies that investigate animal behavior. A wide range of algorithms exist to automatically parse discrete animal behavior, encompassing supervised, unsupervised, and semi-supervised learning paradigms. These algorithms - which include tree-based models, deep neural networks, and graphical models - differ widely in their structure and assumptions on the data. Using four datasets spanning multiple species - fly, mouse, and human - we systematically study how the outputs of these various algorithms align with manually annotated behaviors of interest. Along the way, we introduce a semi-supervised action segmentation model that bridges the gap between supervised deep neural networks and unsupervised graphical models. We find that fully supervised temporal convolutional networks with the addition of temporal information in the observations perform the best on our supervised metrics across all datasets.

SARTAB, a scalable system for automated real-time behavior detection based on animal tracking and Region Of Interest analysis: validation on fish courtship behavior

Methods from Machine Learning (ML) and Computer Vision (CV) have proven powerful tools for quickly and accurately analyzing behavioral recordings. The computational complexity of these techniques, however, often precludes applications that require real-time analysis: for example, experiments where a stimulus must be applied in response to a particular behavior or samples must be collected soon after the behavior occurs. Here, we describe SARTAB (Scalable Automated Real-Time Analysis of Behavior), a system that achieves automated real-time behavior detection by continuously monitoring animal positions relative to behaviorally relevant Regions Of Interest (ROIs). We then show how we used this system to detect infrequent courtship behaviors in Pseudotropheus demasoni (a species of Lake Malawi African cichlid fish) to collect neural tissue samples from actively behaving individuals for multiomic profiling at single nucleus resolution. Within this experimental context, we achieve high ROI and animal detection accuracies (mAP@[.5 : .95] of 0.969 and 0.718, respectively) and 100% classification accuracy on a set of 32 manually selected behavioral clips. SARTAB is unique in that all analysis runs on low-cost, edge-deployed hardware, making it a highly scalable and energy-efficient solution for real-time experimental feedback. Although our solution was developed specifically to study cichlid courtship behavior, the intrinsic flexibility of neural network analysis ensures that our approach can be adapted to novel species, behaviors, and environments.

Analysis of behavioral flow resolves latent phenotypes

The accurate detection and quantification of rodent behavior forms a cornerstone of basic biomedical research. Current data-driven approaches, which segment free exploratory behavior into clusters, suffer from low statistical power due to multiple testing, exhibit poor transferability across experiments and fail to exploit the rich behavioral profiles of individual animals. Here we introduce a pipeline to capture each animal's behavioral flow, yielding a single metric based on all observed transitions between clusters. By stabilizing these clusters through machine learning, we ensure data transferability, while dimensionality reduction techniques facilitate detailed analysis of individual animals. We provide a large dataset of 771 behavior recordings of freely moving mice-including stress exposures, pharmacological and brain circuit interventions-to identify hidden treatment effects, reveal subtle variations on the level of individual animals and detect brain processes underlying specific interventions. Our pipeline, compatible with popular clustering methods, substantially enhances statistical power and enables predictions of an animal's future behavior.

Machine learning reveals prominent spontaneous behavioral changes and treatment efficacy in humanized and transgenic Alzheimer's disease models

Computer-vision and machine-learning (ML) approaches are being developed to provide scalable, unbiased, and sensitive methods to assess mouse behavior. Here, we used the ML-based variational animal motion embedding (VAME) segmentation platform to assess spontaneous behavior in humanized App knockin and transgenic APP models of Alzheimer's disease (AD) and to test the role of AD-related neuroinflammation in these behavioral manifestations. We found marked alterations in spontaneous behavior in AppNL-G-F and 5xFAD mice, including age-dependent changes in motif utilization, disorganized behavioral sequences, increased transitions, and randomness. Notably, blocking fibrinogen-microglia interactions in 5xFAD-Fggγ390-396A mice largely prevented spontaneous behavioral alterations, indicating a key role for neuroinflammation. Thus, AD-related spontaneous behavioral alterations are prominent in knockin and transgenic models and sensitive to therapeutic interventions. VAME outcomes had higher specificity and sensitivity than conventional behavioral outcomes. We conclude that spontaneous behavior effectively captures age- and sex-dependent disease manifestations and treatment efficacy in AD models.

Cellular behavior analysis from live-cell imaging of TCR T cell-cancer cell interactions

T cell therapies, such as chimeric antigen receptor (CAR) T cells and T cell receptor (TCR) T cells, are a growing class of anti-cancer treatments. However, expansion to novel indications and beyond last-line treatment requires engineering cells' dynamic population behaviors. Here we develop the tools for cellular behavior analysis of T cells from live-cell imaging, a common and inexpensive experimental setup used to evaluate engineered T cells. We first develop a state-of-the-art segmentation and tracking pipeline, Caliban, based on human-in-the-loop deep learning. We then build the Occident pipeline to collect a catalog of phenotypes that characterize cell populations, morphology, movement, and interactions in co-cultures of modified T cells and antigen-presenting tumor cells. We use Caliban and Occident to interrogate how interactions between T cells and cancer cells differ when beneficial knock-outs of RASA2 and CUL5 are introduced into TCR T cells. We apply spatiotemporal models to quantify T cell recruitment and proliferation after interactions with cancer cells. We discover that, compared to a safe harbor knockout control, RASA2 knockout T cells have longer interaction times with cancer cells leading to greater T cell activation and killing efficacy, while CUL5 knockout T cells have increased proliferation rates leading to greater numbers of T cells for hunting. Together, segmentation and tracking from Caliban and phenotype quantification from Occident enable cellular behavior analysis to better engineer T cell therapies for improved cancer treatment.

Segmentation tracking and clustering system enables accurate multi-animal tracking of social behaviors

Accurate analysis of social behaviors in animals is hindered by methodological challenges. Here, we develop a segmentation tracking and clustering system (STCS) to address two major challenges in computational neuroethology: reliable multi-animal tracking and pose estimation under complex interaction conditions and providing interpretable insights into social differences guided by genotype information. We established a comprehensive, long-term, multi-animal-tracking dataset across various experimental settings. Benchmarking STCS against state-of-the-art tracking algorithms, we demonstrated its superior efficacy in analyzing behavioral experiments and establishing a robust tracking baseline. By analyzing the behavior of mice with autism spectrum disorder (ASD) using a novel weakly supervised clustering method under both solitary and social conditions, STCS reveals potential links between social stress and motor impairments. Benefiting from its modular and web-based design, STCS allows researchers to easily integrate the latest computer vision methods, enabling comprehensive behavior analysis services over the Internet, even from a single laptop.

Multiparametric Assays Capture Sex- and Environment-Dependent Modifiers of Behavioral Phenotypes in Autism Mouse Models

Background

Current phenotyping approaches for murine autism models often focus on one selected behavioral feature, making the translation onto a spectrum of autistic characteristics in humans challenging. Furthermore, sex and environmental factors are rarely considered. Here, we aimed to capture the full spectrum of behavioral manifestations in 3 autism mouse models to develop a "behavioral fingerprint" that takes environmental and sex influences under consideration.

Methods

To this end, we employed a wide range of classical standardized behavioral tests and 2 multiparametric behavioral assays-the Live Mouse Tracker and Motion Sequencing-on male and female Shank2, Tsc1, and Purkinje cell-specific Tsc1 mutant mice raised in standard or enriched environments. Our aim was to integrate our high dimensional data into one single platform to classify differences in all experimental groups along dimensions with maximum discriminative power.

Results

Multiparametric behavioral assays enabled a more accurate classification of experimental groups than classical tests, and dimensionality reduction analysis demonstrated significant additional gains in classification accuracy, highlighting the presence of sex, environmental, and genotype differences in our experimental groups.

Conclusions

Together, our results provide a complete phenotypic description of all tested groups, suggesting that multiparametric assays can capture the entire spectrum of the heterogeneous phenotype in autism mouse models.

High quality, high throughput, and low-cost simultaneous video recording of 60 animals in operant chambers using PiRATeMC

BACKGROUND

The development of Raspberry Pi-based recording devices for video analyses of drug self-administration studies has been shown to be promising in terms of affordability, customizability, and capacity to extract in-depth behavioral patterns. Yet, most video recording systems are limited to a few cameras making them incompatible with large-scale studies.

NEW METHOD

We expanded the PiRATeMC (Pi-based Remote Acquisition Technology for Motion Capture) recording system by increasing its scale, modifying its code, and adding equipment to accommodate large-scale video acquisition, accompanied by data on throughput capabilities, video fidelity, synchronicity of devices, and comparisons between Raspberry Pi 3B+ and 4B models.

RESULTS

Using PiRATeMC default recording parameters resulted in minimal storage (∼350MB/h), high throughput (< ∼120 seconds/Pi), high video fidelity, and synchronicity within ∼0.02 seconds, affording the ability to simultaneously record 60 animals in individual self-administration chambers for various session lengths at a fraction of commercial costs. No consequential differences were found between Raspberry Pi models.

COMPARISON WITH EXISTING METHOD(S)

This system allows greater acquisition of video data simultaneously than other video recording systems by an order of magnitude with less storage needs and lower costs. Additionally, we report in-depth quantitative assessments of throughput, fidelity, and synchronicity, displaying real-time system capabilities.

CONCLUSIONS

The system presented is able to be fully installed in a month's time by a single technician and provides a scalable, low cost, and quality-assured procedure with a high-degree of customization and synchronicity between recording devices, capable of recording a large number of subjects and timeframes with high turnover in a variety of species and settings.

Mapping the landscape of social behavior

Social interaction is integral to animal behavior. However, we lack tools to describe it with quantitative rigor, limiting our understanding of its principles and neuropsychiatric disorders, like autism, that perturb it. Here, we present a technique for high-resolution 3D tracking of postural dynamics and social touch in freely interacting animals, solving the challenging subject occlusion and part assignment problems using 3D geometric reasoning, graph neural networks, and semi-supervised learning. We collected over 140 million 3D postures in interacting rodents, featuring new monogenic autism rat lines lacking reports of social behavioral phenotypes. Using a novel multi-scale embedding approach, we identified a rich landscape of stereotyped actions, interactions, synchrony, and body contact. This enhanced phenotyping revealed a spectrum of changes in autism models and in response to amphetamine that were inaccessible to conventional measurements. Our framework and large library of interactions will greatly facilitate studies of social behaviors and their neurobiological underpinnings.

Keypoint-MoSeq: parsing behavior by linking point tracking to pose dynamics

Keypoint tracking algorithms can flexibly quantify animal movement from videos obtained in a wide variety of settings. However, it remains unclear how to parse continuous keypoint data into discrete actions. This challenge is particularly acute because keypoint data are susceptible to high-frequency jitter that clustering algorithms can mistake for transitions between actions. Here we present keypoint-MoSeq, a machine learning-based platform for identifying behavioral modules ('syllables') from keypoint data without human supervision. Keypoint-MoSeq uses a generative model to distinguish keypoint noise from behavior, enabling it to identify syllables whose boundaries correspond to natural sub-second discontinuities in pose dynamics. Keypoint-MoSeq outperforms commonly used alternative clustering methods at identifying these transitions, at capturing correlations between neural activity and behavior and at classifying either solitary or social behaviors in accordance with human annotations. Keypoint-MoSeq also works in multiple species and generalizes beyond the syllable timescale, identifying fast sniff-aligned movements in mice and a spectrum of oscillatory behaviors in fruit flies. Keypoint-MoSeq, therefore, renders accessible the modular structure of behavior through standard video recordings.

Lightning Pose: improved animal pose estimation via semi-supervised learning, Bayesian ensembling and cloud-native open-source tools

Contemporary pose estimation methods enable precise measurements of behavior via supervised deep learning with hand-labeled video frames. Although effective in many cases, the supervised approach requires extensive labeling and often produces outputs that are unreliable for downstream analyses. Here, we introduce 'Lightning Pose', an efficient pose estimation package with three algorithmic contributions. First, in addition to training on a few labeled video frames, we use many unlabeled videos and penalize the network whenever its predictions violate motion continuity, multiple-view geometry and posture plausibility (semi-supervised learning). Second, we introduce a network architecture that resolves occlusions by predicting pose on any given frame using surrounding unlabeled frames. Third, we refine the pose predictions post hoc by combining ensembling and Kalman smoothing. Together, these components render pose trajectories more accurate and scientifically usable. We released a cloud application that allows users to label data, train networks and process new videos directly from the browser.

Simple Behavioral Analysis (SimBA) as a platform for explainable machine learning in behavioral neuroscience

The study of complex behaviors is often challenging when using manual annotation due to the absence of quantifiable behavioral definitions and the subjective nature of behavioral annotation. Integration of supervised machine learning approaches mitigates some of these issues through the inclusion of accessible and explainable model interpretation. To decrease barriers to access, and with an emphasis on accessible model explainability, we developed the open-source Simple Behavioral Analysis (SimBA) platform for behavioral neuroscientists. SimBA introduces several machine learning interpretability tools, including SHapley Additive exPlanation (SHAP) scores, that aid in creating explainable and transparent behavioral classifiers. Here we show how the addition of explainability metrics allows for quantifiable comparisons of aggressive social behavior across research groups and species, reconceptualizing behavior as a sharable reagent and providing an open-source framework. We provide an open-source, graphical user interface (GUI)-driven, well-documented package to facilitate the movement toward improved automation and sharing of behavioral classification tools across laboratories.

Psilocybin decreases neural responsiveness and increases functional connectivity while preserving pure-tone frequency selectivity in mouse auditory cortex

Psilocybin is a serotonergic psychedelic believed to have therapeutic potential for neuropsychiatric conditions. Despite well-documented prevalence of perceptual alterations, hallucinations, and synesthesia associated with psychedelic experiences, little is known about how psilocybin affects sensory cortex or alters the activity of neurons in awake animals. To investigate, we conducted two-photon imaging experiments in auditory cortex of awake mice and collected video of free-roaming mouse behavior, both at baseline and during psilocybin treatment. In comparison with pre-dose neural activity, a 2 mg/kg ip dose of psilocybin initially increased the amplitude of neural responses to sound. Thirty minutes post-dose, behavioral activity and neural response amplitudes decreased, yet functional connectivity increased. In contrast, control mice given intraperitoneal saline injections showed no significant changes in either neural or behavioral activity across conditions. Notably, neuronal stimulus selectivity remained stable during psilocybin treatment, for both tonotopic cortical maps and single-cell pure-tone frequency tuning curves. Our results mirror similar findings regarding the effects of serotonergic psychedelics in visual cortex and suggest that psilocybin modulates the balance of intrinsic versus stimulus-driven influences on neural activity in auditory cortex.NEW & NOTEWORTHY Recent studies have shown promising therapeutic potential for psychedelics in treating neuropsychiatric conditions. Musical experience during psilocybin-assisted therapy is predictive of treatment outcome, yet little is known about how psilocybin affects auditory processing. Here, we conducted two-photon imaging experiments in auditory cortex of awake mice that received a dose of psilocybin. Our results suggest that psilocybin modulates the roles of intrinsic neural activity versus stimulus-driven influences on auditory perception.

Odors in space

As an evolutionarily ancient sense, olfaction is key to learning where to find food, shelter, mates, and important landmarks in an animal's environment. Brain circuitry linking odor and navigation appears to be a well conserved multi-region system among mammals; the anterior olfactory nucleus, piriform cortex, entorhinal cortex, and hippocampus each represent different aspects of olfactory and spatial information. We review recent advances in our understanding of the neural circuits underlying odor-place associations, highlighting key choices of behavioral task design and neural circuit manipulations for investigating learning and memory.

A master regulator of opioid reward in the ventral prefrontal cortex

In addition to their intrinsic rewarding properties, opioids can also evoke aversive reactions that protect against misuse. Cellular mechanisms that govern the interplay between opioid reward and aversion are poorly understood. We used whole-brain activity mapping in mice to show that neurons in the dorsal peduncular nucleus (DPn) are highly responsive to the opioid oxycodone. Connectomic profiling revealed that DPn neurons innervate the parabrachial nucleus (PBn). Spatial and single-nuclei transcriptomics resolved a population of PBn-projecting pyramidal neurons in the DPn that express μ-opioid receptors (μORs). Disrupting μOR signaling in the DPn switched oxycodone from rewarding to aversive and exacerbated the severity of opioid withdrawal. These findings identify the DPn as a key substrate for the abuse liability of opioids.

Reconfiguration of Behavioral Signals in the Anterior Cingulate Cortex Based on Emotional State

Behaviors and their execution depend on the context and emotional state in which they are performed. The contextual modulation of behavior likely relies on regions such as the anterior cingulate cortex (ACC) that multiplex information about emotional/autonomic states and behaviors. The objective of the present study was to understand how the representations of behaviors by ACC neurons become modified when performed in different emotional states. A pipeline of machine learning techniques was developed to categorize and classify complex, spontaneous behaviors in male rats from the video. This pipeline, termed Hierarchical Unsupervised Behavioural Discovery Tool (HUB-DT), discovered a range of statistically separable behaviors during a task in which motivationally significant outcomes were delivered in blocks of trials that created three unique "emotional contexts." HUB-DT was capable of detecting behaviors specific to each emotional context and was able to identify and segregate the portions of a neural signal related to a behavior and to emotional context. Overall, ∼10× as many neurons responded to behaviors in a contextually dependent versus a fixed manner, highlighting the extreme impact of emotional state on representations of behaviors that were precisely defined based on detailed analyses of limb kinematics. This type of modulation may be a key mechanism that allows the ACC to modify the behavioral output based on emotional states and contextual demands.

Examining a punishment-related brain circuit with miniature fluorescence microscopes and deep learning

In humans experiencing substance use disorder (SUD), abstinence from drug use is often motivated by a desire to avoid some undesirable consequence of further use: health effects, legal ramifications, etc. This process can be experimentally modeled in rodents by training and subsequently punishing an operant response in a context-induced reinstatement procedure. Understanding the biobehavioral mechanisms underlying punishment learning is critical to understanding both abstinence and relapse in individuals with SUD. To date, most investigations into the neural mechanisms of context-induced reinstatement following punishment have utilized discrete loss-of-function manipulations that do not capture ongoing changes in neural circuitry related to punishment-induced behavior change. Here, we describe a two-pronged approach to analyzing the biobehavioral mechanisms of punishment learning using miniature fluorescence microscopes and deep learning algorithms. We review recent advancements in both techniques and consider a target neural circuit.

Visual detection of seizures in mice using supervised machine learning

Seizures are caused by abnormally synchronous brain activity that can result in changes in muscle tone, such as twitching, stiffness, limpness, or rhythmic jerking. These behavioral manifestations are clear on visual inspection and the most widely used seizure scoring systems in preclinical models, such as the Racine scale in rodents, use these behavioral patterns in semiquantitative seizure intensity scores. However, visual inspection is time-consuming, low-throughput, and partially subjective, and there is a need for rigorously quantitative approaches that are scalable. In this study, we used supervised machine learning approaches to develop automated classifiers to predict seizure severity directly from noninvasive video data. Using the PTZ-induced seizure model in mice, we trained video-only classifiers to predict ictal events, combined these events to predict an univariate seizure intensity for a recording session, as well as time-varying seizure intensity scores. Our results show, for the first time, that seizure events and overall intensity can be rigorously quantified directly from overhead video of mice in a standard open field using supervised approaches. These results enable high-throughput, noninvasive, and standardized seizure scoring for downstream applications such as neurogenetics and therapeutic discovery.

STPoseNet: A real-time spatiotemporal network model for robust mouse pose estimation

Animal behavior analysis plays a crucial role in contemporary neuroscience research. However, the performance of the frame-by-frame approach may degrade in scenarios with occlusions or motion blur. In this study, we propose a spatiotemporal network model based on YOLOv8 to enhance the accuracy of key-point detection in mouse behavioral experimental videos. This model integrates a time-domain tracking strategy comprising two components: the first part utilizes key-point detection results from the previous frame to detect potential target locations in the subsequent frame; the second part employs Kalman filtering to analyze key-point changes prior to detection, allowing for the estimation of missing key-points. In the comparison of pose estimation results between our approach, YOLOv8, DeepLabCut and SLEAP on videos of three mouse behavioral experiments, our approach demonstrated significantly superior performance. This suggests that our method offers a new and effective means of accurately tracking and estimating pose in mice through spatiotemporal processing.

Lightning Pose: improved animal pose estimation via semi-supervised learning, Bayesian ensembling, and cloud-native open-source tools

Contemporary pose estimation methods enable precise measurements of behavior via supervised deep learning with hand-labeled video frames. Although effective in many cases, the supervised approach requires extensive labeling and often produces outputs that are unreliable for downstream analyses. Here, we introduce "Lightning Pose," an efficient pose estimation package with three algorithmic contributions. First, in addition to training on a few labeled video frames, we use many unlabeled videos and penalize the network whenever its predictions violate motion continuity, multiple-view geometry, and posture plausibility (semi-supervised learning). Second, we introduce a network architecture that resolves occlusions by predicting pose on any given frame using surrounding unlabeled frames. Third, we refine the pose predictions post-hoc by combining ensembling and Kalman smoothing. Together, these components render pose trajectories more accurate and scientifically usable. We release a cloud application that allows users to label data, train networks, and predict new videos directly from the browser.

A-SOiD, an active-learning platform for expert-guided, data-efficient discovery of behavior

To identify and extract naturalistic behavior, two methods have become popular: supervised and unsupervised. Each approach carries its own strengths and weaknesses (for example, user bias, training cost, complexity and action discovery), which the user must consider in their decision. Here, an active-learning platform, A-SOiD, blends these strengths, and in doing so, overcomes several of their inherent drawbacks. A-SOiD iteratively learns user-defined groups with a fraction of the usual training data, while attaining expansive classification through directed unsupervised classification. In socially interacting mice, A-SOiD outperformed standard methods despite requiring 85% less training data. Additionally, it isolated ethologically distinct mouse interactions via unsupervised classification. We observed similar performance and efficiency using nonhuman primate and human three-dimensional pose data. In both cases, the transparency in A-SOiD's cluster definitions revealed the defining features of the supervised classification through a game-theoretic approach. To facilitate use, A-SOiD comes as an intuitive, open-source interface for efficient segmentation of user-defined behaviors and discovered sub-actions.

Arbovirus impact on mosquito behavior: the jury is still out

Parasites can manipulate host behavior to enhance transmission, but our understanding of arbovirus-induced changes in mosquito behavior is limited. Here, we explore current knowledge on such behavioral alterations in mosquito vectors, focusing on host-seeking and blood-feeding behaviors. Reviewing studies on dengue, Zika, La Crosse, Sindbis, and West Nile viruses in Aedes or Culex mosquitoes reveals subtle yet potentially significant effects. However, assay heterogeneity and limited sample sizes challenge definitive conclusions. To enhance robustness, we propose using deep-learning tools for automated behavior quantification and stress the need for standardized assays. Additionally, conducting longitudinal studies across the extrinsic incubation period and integrating diverse traits into modeling frameworks are crucial for understanding the nuanced implications of arbovirus-induced behavioral changes for virus transmission dynamics.

A septal-ventral tegmental area circuit drives exploratory behavior

To survive, animals need to balance their exploratory drive with their need for safety. Subcortical circuits play an important role in initiating and modulating movement based on external demands and the internal state of the animal; however, how motivation and onset of locomotion are regulated remain largely unresolved. Here, we show that a glutamatergic pathway from the medial septum and diagonal band of Broca (MSDB) to the ventral tegmental area (VTA) controls exploratory locomotor behavior in mice. Using a self-supervised machine learning approach, we found an overrepresentation of exploratory actions, such as sniffing, whisking, and rearing, when this projection is optogenetically activated. Mechanistically, this role relies on glutamatergic MSDB projections that monosynaptically target a subset of both glutamatergic and dopaminergic VTA neurons. Taken together, we identified a glutamatergic basal forebrain to midbrain circuit that initiates locomotor activity and contributes to the expression of exploration-associated behavior.

Multidimensional Analysis of a Social Behavior Identifies Regression and Phenotypic Heterogeneity in a Female Mouse Model for Rett Syndrome

Regression is a key feature of neurodevelopmental disorders such as autism spectrum disorder, Fragile X syndrome, and Rett syndrome (RTT). RTT is caused by mutations in the X-linked gene methyl-CpG-binding protein 2 (MECP2). It is characterized by an early period of typical development with subsequent regression of previously acquired motor and speech skills in girls. The syndromic phenotypes are individualistic and dynamic over time. Thus far, it has been difficult to capture these dynamics and syndromic heterogeneity in the preclinical Mecp2-heterozygous female mouse model (Het). The emergence of computational neuroethology tools allows for robust analysis of complex and dynamic behaviors to model endophenotypes in preclinical models. Toward this first step, we utilized DeepLabCut, a marker-less pose estimation software to quantify trajectory kinematics and multidimensional analysis to characterize behavioral heterogeneity in Het in the previously benchmarked, ethologically relevant social cognition task of pup retrieval. We report the identification of two distinct phenotypes of adult Het: Het that display a delay in efficiency in early days and then improve over days like wild-type mice and Het that regress and perform worse in later days. Furthermore, regression is dependent on age and behavioral context and can be detected in the initial days of retrieval. Together, the novel identification of two populations of Het suggests differential effects on neural circuitry, opens new avenues to investigate the underlying molecular and cellular mechanisms of heterogeneity, and designs better studies for stratifying therapeutics.

FABEL: Forecasting Animal Behavioral Events with Deep Learning-Based Computer Vision

Behavioral neuroscience aims to provide a connection between neural phenomena and emergent organism-level behaviors. This requires perturbing the nervous system and observing behavioral outcomes, and comparing observed post-perturbation behavior with predicted counterfactual behavior and therefore accurate behavioral forecasts. In this study we present FABEL, a deep learning method for forecasting future animal behaviors and locomotion trajectories from historical locomotion alone. We train an offline pose estimation network to predict animal body-part locations in behavioral video; then sequences of pose vectors are input to deep learning time-series forecasting models. Specifically, we train an LSTM network that predicts a future food interaction event in a specified time window, and a Temporal Fusion Transformer that predicts future trajectories of animal body-parts, which are then converted into probabilistic label forecasts. Importantly, accurate prediction of food interaction provides a basis for neurobehavioral intervention in the context of compulsive eating. We show promising results on forecasting tasks between 100 milliseconds and 5 seconds timescales. Because the model takes only behavioral video as input, it can be adapted to any behavioral task and does not require specific physiological readouts. Simultaneously, these deep learning models may serve as extensible modules that can accommodate diverse signals, such as in-vivo fluorescence imaging and electrophysiology, which may improve behavior forecasts and elucidate invervention targets for desired behavioral change.

Gamma oscillatory complexity conveys behavioral information in hippocampal networks

The hippocampus and entorhinal cortex exhibit rich oscillatory patterns critical for cognitive functions. In the hippocampal region CA1, specific gamma-frequency oscillations, timed at different phases of the ongoing theta rhythm, are hypothesized to facilitate the integration of information from varied sources and contribute to distinct cognitive processes. Here, we show that gamma elements -a multidimensional characterization of transient gamma oscillatory episodes- occur at any frequency or phase relative to the ongoing theta rhythm across all CA1 layers in male mice. Despite their low power and stochastic-like nature, individual gamma elements still carry behavior-related information and computational modeling suggests that they reflect neuronal firing. Our findings challenge the idea of rigid gamma sub-bands, showing that behavior shapes ensembles of irregular gamma elements that evolve with learning and depend on hippocampal layers. Widespread gamma diversity, beyond randomness, may thus reflect complexity, likely functional but invisible to classic average-based analyses.

A comparison of machine learning methods for quantifying self-grooming behavior in mice

Background

As machine learning technology continues to advance and the need for standardized behavioral quantification grows, commercial and open-source automated behavioral analysis tools are gaining prominence in behavioral neuroscience. We present a comparative analysis of three behavioral analysis pipelines-DeepLabCut (DLC) and Simple Behavioral Analysis (SimBA), HomeCageScan (HCS), and manual scoring-in measuring repetitive self-grooming among mice.

Methods

Grooming behavior of mice was recorded at baseline and after water spray or restraint treatments. Videos were processed and analyzed in parallel using 3 methods (DLC/SimBA, HCS, and manual scoring), quantifying both total number of grooming bouts and total grooming duration.

Results

Both treatment conditions (water spray and restraint) resulted in significant elevation in both total grooming duration and number of grooming bouts. HCS measures of grooming duration were significantly elevated relative to those derived from manual scoring: specifically, HCS tended to overestimate duration at low levels of grooming. DLC/SimBA duration measurements were not significantly different than those derived from manual scoring. However, both SimBA and HCS measures of the number of grooming bouts were significantly different than those derived from manual scoring; the magnitude and direction of the difference depended on treatment condition.

Conclusion

DLC/SimBA provides a high-throughput pipeline for quantifying grooming duration that correlates well with manual scoring. However, grooming bout data derived from both DLC/SimBA and HCS did not reliably estimate measures obtained via manual scoring.

Keypoint-MoSeq: parsing behavior by linking point tracking to pose dynamics

Keypoint tracking algorithms have revolutionized the analysis of animal behavior, enabling investigators to flexibly quantify behavioral dynamics from conventional video recordings obtained in a wide variety of settings. However, it remains unclear how to parse continuous keypoint data into the modules out of which behavior is organized. This challenge is particularly acute because keypoint data is susceptible to high frequency jitter that clustering algorithms can mistake for transitions between behavioral modules. Here we present keypoint-MoSeq, a machine learning-based platform for identifying behavioral modules ("syllables") from keypoint data without human supervision. Keypoint-MoSeq uses a generative model to distinguish keypoint noise from behavior, enabling it to effectively identify syllables whose boundaries correspond to natural sub-second discontinuities inherent to mouse behavior. Keypoint-MoSeq outperforms commonly used alternative clustering methods at identifying these transitions, at capturing correlations between neural activity and behavior, and at classifying either solitary or social behaviors in accordance with human annotations. Keypoint-MoSeq therefore renders behavioral syllables and grammar accessible to the many researchers who use standard video to capture animal behavior.

Reproducible and fully automated testing of nocifensive behavior in mice

Pain in rodents is often inferred from their withdrawal from noxious stimulation. Threshold stimulus intensity or response latency is used to quantify pain sensitivity. This usually involves applying stimuli by hand and measuring responses by eye, which limits reproducibility and throughput. We describe a device that standardizes and automates pain testing by providing computer-controlled aiming, stimulation, and response measurement. Optogenetic and thermal stimuli are applied using blue and infrared light, respectively. Precise mechanical stimulation is also demonstrated. Reflectance of red light is used to measure paw withdrawal with millisecond precision. We show that consistent stimulus delivery is crucial for resolving stimulus-dependent variations in withdrawal and for testing with sustained stimuli. Moreover, substage video reveals "spontaneous" behaviors for consideration alongside withdrawal metrics to better assess the pain experience. The entire process was automated using machine learning. RAMalgo (reproducible automated multimodal algometry) improves the standardization, comprehensiveness, and throughput of preclinical pain testing.

Predictive variational autoencoder for learning robust representations of time-series data

Variational autoencoders (VAEs) have been used extensively to discover low-dimensional latent factors governing neural activity and animal behavior. However, without careful model selection, the uncovered latent factors may reflect noise in the data rather than true underlying features, rendering such representations unsuitable for scientific interpretation. Existing solutions to this problem involve introducing additional measured variables or data augmentations specific to a particular data type. We propose a VAE architecture that predicts the next point in time and show that it mitigates the learning of spurious features. In addition, we introduce a model selection metric based on smoothness over time in the latent space. We show that together these two constraints on VAEs to be smooth over time produce robust latent representations and faithfully recover latent factors on synthetic datasets.

High quality, high throughput, and low-cost simultaneous video recording of 60 animals in operant chambers using PiRATeMC

Background

The development of Raspberry Pi-based recording devices for video analyses of drug self-administration studies has shown to be promising in terms of affordability, customizability, and capacity to extract in-depth behavioral patterns. Yet, most video recording systems are limited to a few cameras making them incompatible with large-scale studies.

New Method

We expanded the PiRATeMC (Pi-based Remote Acquisition Technology for Motion Capture) recording system by increasing its scale, modifying its code, and adding equipment to accommodate large-scale video acquisition, accompanied by data on the throughput capabilities, video fidelity, synchronicity of devices, and comparisons between the Raspberry Pi 3B+ and 4B models.

Results

Using PiRATeMC default recording parameters resulted in minimal storage (~350MB/h), high throughput (< ~120 seconds/Pi), high video fidelity, and synchronicity within ~0.02 seconds, affording the ability to simultaneously record 60 animals in individual self-administration chambers at a fraction of current commercial costs. No consequential differences were found between Raspberry Pi 3B+ and 4B models.

Comparison with Existing Methods

This system allows greater acquisition of video data simultaneously than other video recording systems by an order of magnitude with less storage needs and lower costs. Additionally, we report in-depth quantitative assessments of throughput, fidelity, and synchronicity, displaying real-time system capabilities.

Conclusions

The system presented is able to be fully installed in a month's time by a single technician and provides a scalable, low cost, and quality-assured procedure with a high-degree of customization and synchronicity between recording devices, capable of recording a large number of subjects with high turnover in a variety of species and settings.

SaLSa: A Combinatory Approach of Semi-Automatic Labeling and Long Short-Term Memory to Classify Behavioral Syllables

Accurately and quantitatively describing mouse behavior is an important area. Although advances in machine learning have made it possible to track their behaviors accurately, reliable classification of behavioral sequences or syllables remains a challenge. In this study, we present a novel machine learning approach, called SaLSa (a combination of semi-automatic labeling and long short-term memory-based classification), to classify behavioral syllables of mice exploring an open field. This approach consists of two major steps. First, after tracking multiple body parts, spatial and temporal features of their egocentric coordinates are extracted. A fully automated unsupervised process identifies candidates for behavioral syllables, followed by manual labeling of behavioral syllables using a graphical user interface (GUI). Second, a long short-term memory (LSTM) classifier is trained with the labeled data. We found that the classification performance was marked over 97%. It provides a performance equivalent to a state-of-the-art model while classifying some of the syllables. We applied this approach to examine how hyperactivity in a mouse model of Alzheimer's disease develops with age. When the proportion of each behavioral syllable was compared between genotypes and sexes, we found that the characteristic hyperlocomotion of female Alzheimer's disease mice emerges between four and eight months. In contrast, age-related reduction in rearing is common regardless of genotype and sex. Overall, SaLSa enables detailed characterization of mouse behavior.

PiRATeMC: A highly flexible, scalable, and low-cost system for obtaining high quality video recordings for behavioral neuroscience

With the rapidly accelerating adoption of machine-learning based rodent behavioral tracking tools, there is an unmet need for a method of acquiring high quality video data that is scalable, flexible, and relatively low-cost. Many experimenters use webcams, GoPros, or other commercially available cameras that can be expensive, offer minimal flexibility of recording parameters, and not optimized for recording rodent behavior, leading to suboptimal and inconsistent video quality. Furthermore, commercially available products are not conducive for synchronizing multiple cameras, or interfacing with third-party equipment to allow time-locking of video to other equipment such as microcontrollers for closed-loop experiments. We present a low-cost, customizable ecosystem of behavioral recording equipment, PiRATeMC (Pi-based Remote Acquisition Technology for Motion Capture) based on Raspberry Pi Camera Boards with the ability to acquire high quality recordings in bright/low light, or dark conditions under infrared light. PiRATeMC offers users control over nearly every recording parameter, and can be fine-tuned to produce optimal videos in any behavioral apparatus. This setup can be scaled up for synchronous control of any number of cameras via a self-contained network without burdening institutional network infrastructure. The Raspberry Pi is an excellent platform with a large online community designed for novice and inexperienced programmers interested in using an open-source recording system. Importantly, PiRATeMC supports TTL and serial communication, allowing for synchronization and interfacing of video recording with behavioral or other third-party equipment. In sum, PiRATeMC minimizes the cost-prohibitive nature of conducting and analyzing high quality behavioral neuroscience studies, thereby increasing accessibility to behavioral neuroscience.

Elephants and algorithms: a review of the current and future role of AI in elephant monitoring

Artificial intelligence (AI) and machine learning (ML) present revolutionary opportunities to enhance our understanding of animal behaviour and conservation strategies. Using elephants, a crucial species in Africa and Asia's protected areas, as our focal point, we delve into the role of AI and ML in their conservation. Given the increasing amounts of data gathered from a variety of sensors like cameras, microphones, geophones, drones and satellites, the challenge lies in managing and interpreting this vast data. New AI and ML techniques offer solutions to streamline this process, helping us extract vital information that might otherwise be overlooked. This paper focuses on the different AI-driven monitoring methods and their potential for improving elephant conservation. Collaborative efforts between AI experts and ecological researchers are essential in leveraging these innovative technologies for enhanced wildlife conservation, setting a precedent for numerous other species.

Post-acute immunological and behavioral sequelae in mice after Omicron infection

Progress in understanding long COVID and developing effective therapeutics is hampered in part by the lack of suitable animal models. Here we used ACE2-transgenic mice recovered from Omicron (BA.1) infection to test for pulmonary and behavioral post-acute sequelae. Through in-depth phenotyping by CyTOF, we demonstrate that naïve mice experiencing a first Omicron infection exhibit profound immune perturbations in the lung after resolving acute infection. This is not observed if mice were first vaccinated with spike-encoding mRNA. The protective effects of vaccination against post-acute sequelae were associated with a highly polyfunctional SARS-CoV-2-specific T cell response that was recalled upon BA.1 breakthrough infection but not seen with BA.1 infection alone. Without vaccination, the chemokine receptor CXCR4 was uniquely upregulated on multiple pulmonary immune subsets in the BA.1 convalescent mice, a process previously connected to severe COVID-19. Taking advantage of recent developments in machine learning and computer vision, we demonstrate that BA.1 convalescent mice exhibited spontaneous behavioral changes, emotional alterations, and cognitive-related deficits in context habituation. Collectively, our data identify immunological and behavioral post-acute sequelae after Omicron infection and uncover a protective effect of vaccination against post-acute pulmonary immune perturbations.

Captive Animal Behavior Study by Video Analysis

Three video analysis-based applications for the study of captive animal behavior are presented. The aim of the first one is to provide certain parameters to assess drug efficiency by analyzing the movement of a rat. The scene is a three-chamber plastic box. First, the rat can move only in the middle room. The rat's head pose is the first parameter needed. Secondly, the rodent could walk in all three compartments. The entry number in each area and visit duration are the other indicators used in the final evaluation. The second application is related to a neuroscience experiment. Besides the electroencephalographic (EEG) signals yielded by a radio frequency link from a headset mounted on a monkey, the head placement is a useful source of information for reliable analysis, as well as its orientation. Finally, a fusion method to construct the displacement of a panda bear in a cage and the corresponding motion analysis to recognize its stress states are shown. The arena is a zoological garden that imitates the native environment of a panda bear. This surrounding is monitored by means of four video cameras. We have applied the following stages: (a) panda detection for every video camera; (b) panda path construction from all routes; and (c) panda way filtering and analysis.

Capturing continuous, long timescale behavioral changes in Drosophila melanogaster postural data

Animal behavior spans many timescales, from short, seconds-scale actions to circadian rhythms over many hours to life-long changes during aging. Most quantitative behavior studies have focused on short-timescale behaviors such as locomotion and grooming. Analysis of these data suggests there exists a hierarchy of timescales; however, the limited duration of these experiments prevents the investigation of the full temporal structure. To access longer timescales of behavior, we continuously recorded individual Drosophila melanogaster at 100 frames per second for up to 7 days at a time in featureless arenas on sucrose-agarose media. We use the deep learning framework SLEAP to produce a full-body postural data set for 47 individuals resulting in nearly 2 billion pose instances. We identify stereotyped behaviors such as grooming, proboscis extension, and locomotion and use the resulting ethograms to explore how the flies' behavior varies across time of day and days in the experiment. We find distinct circadian patterns in all of our stereotyped behavior and also see changes in behavior over the course of the experiment as the flies weaken and die.

Automated Segmentation of the Mouse Body Language to Study Stimulus-Evoked Emotional Behaviors

Understanding the neural basis of emotions is a critical step to uncover the biological substrates of neuropsychiatric disorders. To study this aspect in freely behaving mice, neuroscientists have relied on the observation of ethologically relevant bodily cues to infer the affective content of the subject, both in neutral conditions or in response to a stimulus. The best example of that is the widespread assessment of freezing in experiments testing both conditioned and unconditioned fear responses. While robust and powerful, these approaches come at a cost: they are usually confined within selected time windows, accounting for only a limited portion of the complexity of emotional fluctuation. Moreover, they often rely on visual inspection and subjective judgment, resulting in inconsistency across experiments and questionable result interpretations. To overcome these limitations, novel tools are arising, fostering a new avenue in the study of the mouse naturalistic behavior. In this work we developed a computational tool [stimulus-evoked behavioral tracking in 3D for rodents (SEB3R)] to automate and standardize an ethologically driven observation of freely moving mice. Using a combination of machine learning-based behavioral tracking and unsupervised cluster analysis, we identified statistically meaningful postures that could be used for empirical inference on a subsecond scale. We validated the efficacy of this tool in a stimulus-driven test, the whisker nuisance (WN) task, where mice are challenged with a prolonged and invasive whisker stimulation, showing that identified postures can be reliably used as a proxy for stimulus-driven fearful and explorative behaviors.

Signal processing in the vagus nerve: Hypotheses based on new genetic and anatomical evidence

Each organism must regulate its internal state in a metabolically efficient way as it interacts in space and time with an ever-changing and only partly predictable world. Success in this endeavor is largely determined by the ongoing communication between brain and body, and the vagus nerve is a crucial structure in that dialogue. In this review, we introduce the novel hypothesis that the afferent vagus nerve is engaged in signal processing rather than just signal relay. New genetic and structural evidence of vagal afferent fiber anatomy motivates two hypotheses: (1) that sensory signals informing on the physiological state of the body compute both spatial and temporal viscerosensory features as they ascend the vagus nerve, following patterns found in other sensory architectures, such as the visual and olfactory systems; and (2) that ascending and descending signals modulate one another, calling into question the strict segregation of sensory and motor signals, respectively. Finally, we discuss several implications of our two hypotheses for understanding the role of viscerosensory signal processing in predictive energy regulation (i.e., allostasis) as well as the role of metabolic signals in memory and in disorders of prediction (e.g., mood disorders).

Automatically annotated motion tracking identifies a distinct social behavioral profile following chronic social defeat stress

Severe stress exposure increases the risk of stress-related disorders such as major depressive disorder (MDD). An essential characteristic of MDD is the impairment of social functioning and lack of social motivation. Chronic social defeat stress is an established animal model for MDD research, which induces a cascade of physiological and behavioral changes. Current markerless pose estimation tools allow for more complex and naturalistic behavioral tests. Here, we introduce the open-source tool DeepOF to investigate the individual and social behavioral profile in mice by providing supervised and unsupervised pipelines using DeepLabCut-annotated pose estimation data. Applying this tool to chronic social defeat in male mice, the DeepOF supervised and unsupervised pipelines detect a distinct stress-induced social behavioral pattern, which was particularly observed at the beginning of a novel social encounter and fades with time due to habituation. In addition, while the classical social avoidance task does identify the stress-induced social behavioral differences, both DeepOF behavioral pipelines provide a clearer and more detailed profile. Moreover, DeepOF aims to facilitate reproducibility and unification of behavioral classification by providing an open-source tool, which can advance the study of rodent individual and social behavior, thereby enabling biological insights and, for example, subsequent drug development for psychiatric disorders.

Disentangling rodent behaviors to improve automated behavior recognition

Automated observation and analysis of behavior is important to facilitate progress in many fields of science. Recent developments in deep learning have enabled progress in object detection and tracking, but rodent behavior recognition struggles to exceed 75-80% accuracy for ethologically relevant behaviors. We investigate the main reasons why and distinguish three aspects of behavior dynamics that are difficult to automate. We isolate these aspects in an artificial dataset and reproduce effects with the state-of-the-art behavior recognition models. Having an endless amount of labeled training data with minimal input noise and representative dynamics will enable research to optimize behavior recognition architectures and get closer to human-like recognition performance for behaviors with challenging dynamics.

Advancing social behavioral neuroscience by integrating ethology and comparative psychology methods through machine learning

Social behavior is naturally occurring in vertebrate species, which holds a strong evolutionary component and is crucial for the normal development and survival of individuals throughout life. Behavioral neuroscience has seen different influential methods for social behavioral phenotyping. The ethological research approach has extensively investigated social behavior in natural habitats, while the comparative psychology approach was developed utilizing standardized and univariate social behavioral tests. The development of advanced and precise tracking tools, together with post-tracking analysis packages, has recently enabled a novel behavioral phenotyping method, that includes the strengths of both approaches. The implementation of such methods will be beneficial for fundamental social behavioral research but will also enable an increased understanding of the influences of many different factors that can influence social behavior, such as stress exposure. Furthermore, future research will increase the number of data modalities, such as sensory, physiological, and neuronal activity data, and will thereby significantly enhance our understanding of the biological basis of social behavior and guide intervention strategies for behavioral abnormalities in psychiatric disorders.

Open-source software for automated rodent behavioral analysis

Rodent behavioral analysis is a major specialization in experimental psychology and behavioral neuroscience. Rodents display a wide range of species-specific behaviors, not only in their natural habitats but also under behavioral testing in controlled laboratory conditions. Detecting and categorizing these different kinds of behavior in a consistent way is a challenging task. Observing and analyzing rodent behaviors manually limits the reproducibility and replicability of the analyses due to potentially low inter-rater reliability. The advancement and accessibility of object tracking and pose estimation technologies led to several open-source artificial intelligence (AI) tools that utilize various algorithms for rodent behavioral analysis. These software provide high consistency compared to manual methods, and offer more flexibility than commercial systems by allowing custom-purpose modifications for specific research needs. Open-source software reviewed in this paper offer automated or semi-automated methods for detecting and categorizing rodent behaviors by using hand-coded heuristics, machine learning, or neural networks. The underlying algorithms show key differences in their internal dynamics, interfaces, user-friendliness, and the variety of their outputs. This work reviews the algorithms, capability, functionality, features and software properties of open-source behavioral analysis tools, and discusses how this emergent technology facilitates behavioral quantification in rodent research.

Open-source tools for behavioral video analysis: Setup, methods, and best practices

Recently developed methods for video analysis, especially models for pose estimation and behavior classification, are transforming behavioral quantification to be more precise, scalable, and reproducible in fields such as neuroscience and ethology. These tools overcome long-standing limitations of manual scoring of video frames and traditional 'center of mass' tracking algorithms to enable video analysis at scale. The expansion of open-source tools for video acquisition and analysis has led to new experimental approaches to understand behavior. Here, we review currently available open-source tools for video analysis and discuss how to set up these methods for labs new to video recording. We also discuss best practices for developing and using video analysis methods, including community-wide standards and critical needs for the open sharing of datasets and code, more widespread comparisons of video analysis methods, and better documentation for these methods especially for new users. We encourage broader adoption and continued development of these tools, which have tremendous potential for accelerating scientific progress in understanding the brain and behavior.

Increasing resolution in stress neurobiology: from single cells to complex group behaviors

Stress can have severe psychological and physiological consequences. Thus, inappropriate regulation of the stress response is linked to the etiology of mood and anxiety disorders. The generation and implementation of preclinical animal models represent valuable tools to explore and characterize the mechanisms underlying the pathophysiology of stress-related psychiatric disorders and the development of novel pharmacological strategies. In this commentary, we discuss the strengths and limitations of state-of-the-art molecular and computational advances employed in stress neurobiology research, with a focus on the ever-increasing spatiotemporal resolution in cell biology and behavioral science. Finally, we share our perspective on future directions in the fields of preclinical and human stress research.

Chronic stress causes striatal disinhibition mediated by SOM-interneurons in male mice

Chronic stress (CS) is associated with a number of neuropsychiatric disorders, and it may also contribute to or exacerbate motor function. However, the mechanisms by which stress triggers motor symptoms are not fully understood. Here, we report that CS functionally alters dorsomedial striatum (DMS) circuits in male mice, by affecting GABAergic interneuron populations and somatostatin positive (SOM) interneurons in particular. Specifically, we show that CS impairs communication between SOM interneurons and medium spiny neurons, promoting striatal overactivation/disinhibition and increased motor output. Using probabilistic machine learning to analyze animal behavior, we demonstrate that in vivo chemogenetic manipulation of SOM interneurons in DMS modulates motor phenotypes in stressed mice. Altogether, we propose a causal link between dysfunction of striatal SOM interneurons and motor symptoms in models of chronic stress.

Recent Advances at the Interface of Neuroscience and Artificial Neural Networks

Biological neural networks adapt and learn in diverse behavioral contexts. Artificial neural networks (ANNs) have exploited biological properties to solve complex problems. However, despite their effectiveness for specific tasks, ANNs are yet to realize the flexibility and adaptability of biological cognition. This review highlights recent advances in computational and experimental research to advance our understanding of biological and artificial intelligence. In particular, we discuss critical mechanisms from the cellular, systems, and cognitive neuroscience fields that have contributed to refining the architecture and training algorithms of ANNs. Additionally, we discuss how recent work used ANNs to understand complex neuronal correlates of cognition and to process high throughput behavioral data.

The Multi-Agent Behavior Dataset: Mouse Dyadic Social Interactions

Multi-agent behavior modeling aims to understand the interactions that occur between agents. We present a multi-agent dataset from behavioral neuroscience, the Caltech Mouse Social Interactions (CalMS21) Dataset. Our dataset consists of trajectory data of social interactions, recorded from videos of freely behaving mice in a standard resident-intruder assay. To help accelerate behavioral studies, the CalMS21 dataset provides benchmarks to evaluate the performance of automated behavior classification methods in three settings: (1) for training on large behavioral datasets all annotated by a single annotator, (2) for style transfer to learn inter-annotator differences in behavior definitions, and (3) for learning of new behaviors of interest given limited training data. The dataset consists of 6 million frames of unlabeled tracked poses of interacting mice, as well as over 1 million frames with tracked poses and corresponding frame-level behavior annotations. The challenge of our dataset is to be able to classify behaviors accurately using both labeled and unlabeled tracking data, as well as being able to generalize to new settings.