Geometric deep learning enables 3D kinematic profiling across species and environments

Lightning Pose: improved animal pose estimation via semi-supervised learning, Bayesian ensembling and cloud-native open-source tools

Contemporary pose estimation methods enable precise measurements of behavior via supervised deep learning with hand-labeled video frames. Although effective in many cases, the supervised approach requires extensive labeling and often produces outputs that are unreliable for downstream analyses. Here, we introduce 'Lightning Pose', an efficient pose estimation package with three algorithmic contributions. First, in addition to training on a few labeled video frames, we use many unlabeled videos and penalize the network whenever its predictions violate motion continuity, multiple-view geometry and posture plausibility (semi-supervised learning). Second, we introduce a network architecture that resolves occlusions by predicting pose on any given frame using surrounding unlabeled frames. Third, we refine the pose predictions post hoc by combining ensembling and Kalman smoothing. Together, these components render pose trajectories more accurate and scientifically usable. We released a cloud application that allows users to label data, train networks and process new videos directly from the browser.

Application of a novel deep learning-based 3D videography workflow to bat flight

Studying the detailed biomechanics of flying animals requires accurate three-dimensional coordinates for key anatomical landmarks. Traditionally, this relies on manually digitizing animal videos, a labor-intensive task that scales poorly with increasing framerates and numbers of cameras. Here, we present a workflow that combines deep learning-powered automatic digitization with filtering and correction of mislabeled points using quality metrics from deep learning and 3D reconstruction. We tested our workflow using a particularly challenging scenario: bat flight. First, we documented four bats flying steadily in a 2 m3 wind tunnel test section. Wing kinematic parameters resulting from manually digitizing bats with markers applied to anatomical landmarks were not significantly different from those resulting from applying our workflow to the same bats without markers for five out of six parameters. Second, we compared coordinates from manual digitization against those yielded via our workflow for bats flying freely in a 344 m3 enclosure. Average distance between coordinates from our workflow and those from manual digitization was less than a millimeter larger than the average human-to-human coordinate distance. The improved efficiency of our workflow has the potential to increase the scalability of studies on animal flight biomechanics.

The Poses for Equine Research Dataset (PFERD)

Studies of quadruped animal motion help us to identify diseases, understand behavior and unravel the mechanics behind gaits in animals. The horse is likely the best-studied animal in this aspect, but data capture is challenging and time-consuming. Computer vision techniques improve animal motion extraction, but the development relies on reference datasets, which are scarce, not open-access and often provide data from only a few anatomical landmarks. Addressing this data gap, we introduce PFERD, a video and 3D marker motion dataset from horses using a full-body set-up of densely placed over 100 skin-attached markers and synchronized videos from ten camera angles. Five horses of diverse conformations provide data for various motions from basic poses (eg. walking, trotting) to advanced motions (eg. rearing, kicking). We further express the 3D motions with current techniques and a 3D parameterized model, the hSMAL model, establishing a baseline for 3D horse markerless motion capture. PFERD enables advanced biomechanical studies and provides a resource of ground truth data for the methodological development of markerless motion capture.

SAUSI: a novel assay for measuring social anxiety and motivation

Social anxiety is one of the most prevalent mental health disorders, though the underlying neurobiology is poorly understood. Progress in understanding the etiology of social anxiety has been hindered by the lack of comprehensive tools to assess social anxiety in model systems. Here, we created a new behavioral task - Selective Access to Unrestricted Social Interaction (SAUSI), which combines elements of social motivation, hesitancy, decision-making, and free interaction to enable the wholistic assessment of social anxiety-like behaviors in mice. Using this novel assay, we found that social isolation-induced social anxiety-like behaviors in female mice are largely driven by increases in social fear, social hesitancy, and altered ultrasonic vocalizations. Deep learning analyses were able to computationally identify a unique behavioral footprint underlying the state produced by social isolation, demonstrating the compatibility of modern computational approaches with SAUSI. Finally, we compared the results of SAUSI to traditionally social assays including the 3-chamber sociability assay and the resident intruder task. This revealed that behavioral changes induced by isolation were highly context dependent, and that while fragments of social anxiety measured in SAUSI were replicable across other tasks, a wholistic assessment was not obtainable from these alternative assays. Our findings debut a novel task for the behavioral toolbox - one which overcomes limitations of previous assays, allowing for both social choice as well as free interaction, and offers a new approach for assessing social anxiety in rodents.

The choice-wide behavioral association study: data-driven identification of interpretable behavioral components

Behavior contains rich structure across many timescales, but there is a dearth of methods to identify relevant components, especially over the longer periods required for learning and decision-making. Inspired by the goals and techniques of genome-wide association studies, we present a data-driven method-the choice-wide behavioral association study: CBAS-that systematically identifies such behavioral features. CBAS uses a powerful, resampling-based, method of multiple comparisons correction to identify sequences of actions or choices that either differ significantly between groups or significantly correlate with a covariate of interest. We apply CBAS to different tasks and species (flies, rats, and humans) and find, in all instances, that it provides interpretable information about each behavioral task.

Barcoding of episodic memories in the hippocampus of a food-caching bird

The hippocampus is critical for episodic memory. Although hippocampal activity represents place and other behaviorally relevant variables, it is unclear how it encodes numerous memories of specific events in life. To study episodic coding, we leveraged the specialized behavior of chickadees-food-caching birds that form memories at well-defined moments in time whenever they cache food for subsequent retrieval. Our recordings during caching revealed very sparse, transient barcode-like patterns of firing across hippocampal neurons. Each "barcode" uniquely represented a caching event and transiently reactivated during the retrieval of that specific cache. Barcodes co-occurred with the conventional activity of place cells but were uncorrelated even for nearby cache locations that had similar place codes. We propose that animals recall episodic memories by reactivating hippocampal barcodes. Similarly to computer hash codes, these patterns assign unique identifiers to different events and could be a mechanism for rapid formation and storage of many non-interfering memories.

Lightning Pose: improved animal pose estimation via semi-supervised learning, Bayesian ensembling, and cloud-native open-source tools

Contemporary pose estimation methods enable precise measurements of behavior via supervised deep learning with hand-labeled video frames. Although effective in many cases, the supervised approach requires extensive labeling and often produces outputs that are unreliable for downstream analyses. Here, we introduce "Lightning Pose," an efficient pose estimation package with three algorithmic contributions. First, in addition to training on a few labeled video frames, we use many unlabeled videos and penalize the network whenever its predictions violate motion continuity, multiple-view geometry, and posture plausibility (semi-supervised learning). Second, we introduce a network architecture that resolves occlusions by predicting pose on any given frame using surrounding unlabeled frames. Third, we refine the pose predictions post-hoc by combining ensembling and Kalman smoothing. Together, these components render pose trajectories more accurate and scientifically usable. We release a cloud application that allows users to label data, train networks, and predict new videos directly from the browser.

A-SOiD, an active-learning platform for expert-guided, data-efficient discovery of behavior

To identify and extract naturalistic behavior, two methods have become popular: supervised and unsupervised. Each approach carries its own strengths and weaknesses (for example, user bias, training cost, complexity and action discovery), which the user must consider in their decision. Here, an active-learning platform, A-SOiD, blends these strengths, and in doing so, overcomes several of their inherent drawbacks. A-SOiD iteratively learns user-defined groups with a fraction of the usual training data, while attaining expansive classification through directed unsupervised classification. In socially interacting mice, A-SOiD outperformed standard methods despite requiring 85% less training data. Additionally, it isolated ethologically distinct mouse interactions via unsupervised classification. We observed similar performance and efficiency using nonhuman primate and human three-dimensional pose data. In both cases, the transparency in A-SOiD's cluster definitions revealed the defining features of the supervised classification through a game-theoretic approach. To facilitate use, A-SOiD comes as an intuitive, open-source interface for efficient segmentation of user-defined behaviors and discovered sub-actions.

Gaze tracking of large-billed crows (Corvus macrorhynchos) in a motion capture system

Previous studies often inferred the focus of a bird's attention from its head movements because it provides important clues about their perception and cognition. However, it remains challenging to do so accurately, as the details of how they orient their visual field toward the visual targets remain largely unclear. We thus examined visual field configurations and the visual field use of large-billed crows (Corvus macrorhynchos Wagler 1827). We used an established ophthalmoscopic reflex technique to identify the visual field configuration, including the binocular width and optical axes, as well as the degree of eye movement. A newly established motion capture system was then used to track the head movements of freely moving crows to examine how they oriented their reconstructed visual fields toward attention-getting objects. When visual targets were moving, the crows frequently used their binocular visual fields, particularly around the projection of the beak-tip. When the visual targets stopped moving, crows frequently used non-binocular visual fields, particularly around the regions where their optical axes were found. On such occasions, the crows slightly preferred the right eye. Overall, the visual field use of crows is clearly predictable. Thus, while the untracked eye movements could introduce some level of uncertainty (typically within 15 deg), we demonstrated the feasibility of inferring a crow's attentional focus by 3D tracking of their heads. Our system represents a promising initial step towards establishing gaze tracking methods for studying corvid behavior and cognition.

Two-year Opioid Prescription Trends in Local Sanitary Agency Naples 3 South, Campania Region, Italy. Descriptive Analyses and AI-based Translational Perspectives

Aims

This study delves into the two-year opioid prescription trends in the Local Sanitary Agency Naples 3 South, Campania Region, Italy. The research aims to elucidate prescribing patterns, demographics, and dosage categories within a population representing 1.7% of the national total. Perspectives on artificial intelligence research are discussed.

Methods

From the original dataset, spanning from January 2022 to October 2023, we processed multiple variables including demographic data, medications, dosages, drug consumption, and administration routes. The dispensing quantity was calculated as defined daily doses (DDD).

Results

The analysis reveals a conservative approach to opioid therapy. In subjects under the age of 20, prescriptions accounted for 2.1% in 2022 and declined to 1.4% in 2023. The drug combination paracetamol/codeine was the most frequently prescribed, followed by tapentadol. Approximately two-thirds of the consumption pertains to oral formulations. Transdermal formulations were 15% (fentanyl 9.8%, buprenorphine 5.1%) in 2022; and 16.6% (fentanyl 10%, buprenorphine 6.6%) in 2023. These data were confirmed by the DDD analysis. The trend analysis demonstrated a significant reduction ( p < 0.001) in the number of prescribed opioids from 2022 to 2023 in adults (40-69 years). The study of rapid-onset opioids (ROOs), drugs specifically used for breakthrough cancer pain, showed higher dosage (>267 mcg) consumption among women, whereas a lower dosage (<133 mcg) was calculated for men. Fentanyl pectin nasal spray accounted for approximately one-fifth of all ROOs.

Conclusion

Despite limitations, the study provides valuable insights into prescribing practices involving an important study population. The findings underscore the need for tailored approaches to prescribing practices, recognizing the complexities of pain management in different contexts. This research can contribute to the ongoing discourse on opioid use, advocating for innovative strategies that optimize therapeutic outcomes while mitigating potential risks.

Methods for evaluating gait associated dynamic balance and coordination in rodents

Balance is the dynamic and unconscious control of the body's centre of mass to maintain postural equilibrium. Regulated by the vestibular system, head movement and acceleration are processed by the brain to adjust joints. Several conditions result in a loss of balance, including Alzheimer's Disease, Parkinson's Disease, Menière's Disease and cervical spondylosis, all of which are caused by damage to certain parts of the vestibular pathways. Studies about the impairment of the vestibular system are challenging to carry out in human trials due to smaller study sizes limiting applications of the results and a lacking understanding of the human balance control mechanism. In contrast, more controlled research can be performed in animal studies which have fewer confounding factors than human models and allow specific conditions that affect balance to be replicated. Balance control can be studied using rodent balance-related behavioural tests after spinal or brain lesions, such as the Basso, Beattie and Bresnahan (BBB) Locomotor Scale, Foot Fault Scoring System, Ledged Beam Test, Beam Walking Test, and Ladder Beam Test, which are discussed in this review article along with their advantages and disadvantages. These tests can be performed in preclinical rodent models of femoral nerve injury, stroke, spinal cord injury and neurodegenerative diseases.

IntelliCage: the development and perspectives of a mouse- and user-friendly automated behavioral test system

IntelliCage for mice is a rodent home-cage equipped with four corner structures harboring symmetrical double panels for operant conditioning at each of the two sides, either by reward (access to water) or by aversion (non-painful stimuli: air-puffs, LED lights). Corner visits, nose-pokes and actual licks at bottle-nipples are recorded individually using subcutaneously implanted transponders for RFID identification of up to 16 adult mice housed in the same home-cage. This allows for recording individual in-cage activity of mice and applying reward/punishment operant conditioning schemes in corners using workflows designed on a versatile graphic user interface. IntelliCage development had four roots: (i) dissatisfaction with standard approaches for analyzing mouse behavior, including standardization and reproducibility issues, (ii) response to handling and housing animal welfare issues, (iii) the increasing number of mouse models had produced a high work burden on classic manual behavioral phenotyping of single mice. and (iv), studies of transponder-chipped mice in outdoor settings revealed clear genetic behavioral differences in mouse models corresponding to those observed by classic testing in the laboratory. The latter observations were important for the development of home-cage testing in social groups, because they contradicted the traditional belief that animals must be tested under social isolation to prevent disturbance by other group members. The use of IntelliCages reduced indeed the amount of classic testing remarkably, while its flexibility was proved in a wide range of applications worldwide including transcontinental parallel testing. Essentially, two lines of testing emerged: sophisticated analysis of spontaneous behavior in the IntelliCage for screening of new genetic models, and hypothesis testing in many fields of behavioral neuroscience. Upcoming developments of the IntelliCage aim at improved stimulus presentation in the learning corners and videotracking of social interactions within the IntelliCage. Its main advantages are (i) that mice live in social context and are not stressfully handled for experiments, (ii) that studies are not restricted in time and can run in absence of humans, (iii) that it increases reproducibility of behavioral phenotyping worldwide, and (iv) that the industrial standardization of the cage permits retrospective data analysis with new statistical tools even after many years.

Machine learning and artificial intelligence in neuroscience: A primer for researchers

Artificial intelligence (AI) is often used to describe the automation of complex tasks that we would attribute intelligence to. Machine learning (ML) is commonly understood as a set of methods used to develop an AI. Both have seen a recent boom in usage, both in scientific and commercial fields. For the scientific community, ML can solve bottle necks created by complex, multi-dimensional data generated, for example, by functional brain imaging or *omics approaches. ML can here identify patterns that could not have been found using traditional statistic approaches. However, ML comes with serious limitations that need to be kept in mind: their tendency to optimise solutions for the input data means it is of crucial importance to externally validate any findings before considering them more than a hypothesis. Their black-box nature implies that their decisions usually cannot be understood, which renders their use in medical decision making problematic and can lead to ethical issues. Here, we present an introduction for the curious to the field of ML/AI. We explain the principles as commonly used methods as well as recent methodological advancements before we discuss risks and what we see as future directions of the field. Finally, we show practical examples of neuroscience to illustrate the use and limitations of ML.

A hybrid neural network-based intelligent body posture estimation system in sports scenes

Body posture estimation has been a hot branch in the field of computer vision. This work focuses on one of its typical applications: recognition of various body postures in sports scenes. Existing technical methods were mostly established on the basis of convolution neural network (CNN) structures, due to their strong visual information sensing ability. However, sports scenes are highly dynamic, and many valuable contextual features can be extracted from multimedia frame sequences. To handle the current challenge, this paper proposes a hybrid neural network-based intelligent body posture estimation system for sports scenes. Specifically, a CNN unit and a long short-term memory (LSTM) unit are employed as the backbone network in order to extract key-point information and temporal information from video frames, respectively. Then, a semi-supervised learning-based computing framework is developed to output estimation results. It can make training procedures using limited labeled samples. Finally, through extensive experiments, it is proved that the proposed body posture estimation method in this paper can achieve proper estimation effect in real-world frame samples of sports scenes.

Advancing preclinical chronic stress models to promote therapeutic discovery for human stress disorders

There is an urgent need to develop more effective treatments for stress-related illnesses, which include depression, post-traumatic stress disorder, and anxiety. We view animal models as playing an essential role in this effort, but to date, such approaches have generally not succeeded in developing therapeutics with new mechanisms of action. This is partly due to the complexity of the brain and its disorders, but also to inherent difficulties in modeling human disorders in rodents and to the incorrect use of animal models: namely, trying to recapitulate a human syndrome in a rodent which is likely not possible as opposed to using animals to understand underlying mechanisms and evaluating potential therapeutic paths. Recent transcriptomic research has established the ability of several different chronic stress procedures in rodents to recapitulate large portions of the molecular pathology seen in postmortem brain tissue of individuals with depression. These findings provide crucial validation for the clear relevance of rodent stress models to better understand the pathophysiology of human stress disorders and help guide therapeutic discovery. In this review, we first discuss the current limitations of preclinical chronic stress models as well as traditional behavioral phenotyping approaches. We then explore opportunities to dramatically enhance the translational use of rodent stress models through the application of new experimental technologies. The goal of this review is to promote the synthesis of these novel approaches in rodents with human cell-based approaches and ultimately with early-phase proof-of-concept studies in humans to develop more effective treatments for human stress disorders.

OpenApePose, a database of annotated ape photographs for pose estimation

Because of their close relationship with humans, non-human apes (chimpanzees, bonobos, gorillas, orangutans, and gibbons, including siamangs) are of great scientific interest. The goal of understanding their complex behavior would be greatly advanced by the ability to perform video-based pose tracking. Tracking, however, requires high-quality annotated datasets of ape photographs. Here we present OpenApePose, a new public dataset of 71,868 photographs, annotated with 16 body landmarks of six ape species in naturalistic contexts. We show that a standard deep net (HRNet-W48) trained on ape photos can reliably track out-of-sample ape photos better than networks trained on monkeys (specifically, the OpenMonkeyPose dataset) and on humans (COCO) can. This trained network can track apes almost as well as the other networks can track their respective taxa, and models trained without one of the six ape species can track the held-out species better than the monkey and human models can. Ultimately, the results of our analyses highlight the importance of large, specialized databases for animal tracking systems and confirm the utility of our new ape database.

Low- and high-level coordination of orofacial motor actions

Orofacial motor actions are movements that, in rodents, involve whisking of the vibrissa, deflection of the nose, licking and lapping with the tongue, and consumption through chewing. These actions, along with bobbing and turning of the head, coordinate to subserve exploration while not conflicting with life-supporting actions such as breathing and swallowing. Orofacial and head movements are comprised of two additive components: a rhythm that can be entrained by the breathing oscillator and a broadband component that directs the actuator to the region of interest. We focus on coordinating the rhythmic component of actions into a behavior. We hypothesize that the precise timing of each constituent action is continually adjusted through the merging of low-level oscillator input with sensory-derived, high-level rhythmic feedback. Supporting evidence is discussed.

SaLSa: A Combinatory Approach of Semi-Automatic Labeling and Long Short-Term Memory to Classify Behavioral Syllables

Accurately and quantitatively describing mouse behavior is an important area. Although advances in machine learning have made it possible to track their behaviors accurately, reliable classification of behavioral sequences or syllables remains a challenge. In this study, we present a novel machine learning approach, called SaLSa (a combination of semi-automatic labeling and long short-term memory-based classification), to classify behavioral syllables of mice exploring an open field. This approach consists of two major steps. First, after tracking multiple body parts, spatial and temporal features of their egocentric coordinates are extracted. A fully automated unsupervised process identifies candidates for behavioral syllables, followed by manual labeling of behavioral syllables using a graphical user interface (GUI). Second, a long short-term memory (LSTM) classifier is trained with the labeled data. We found that the classification performance was marked over 97%. It provides a performance equivalent to a state-of-the-art model while classifying some of the syllables. We applied this approach to examine how hyperactivity in a mouse model of Alzheimer's disease develops with age. When the proportion of each behavioral syllable was compared between genotypes and sexes, we found that the characteristic hyperlocomotion of female Alzheimer's disease mice emerges between four and eight months. In contrast, age-related reduction in rearing is common regardless of genotype and sex. Overall, SaLSa enables detailed characterization of mouse behavior.

Three-dimensional surface motion capture of multiple freely moving pigs using MAMMAL

Understandings of the three-dimensional social behaviors of freely moving large-size mammals are valuable for both agriculture and life science, yet challenging due to occlusions in close interactions. Although existing animal pose estimation methods captured keypoint trajectories, they ignored deformable surfaces which contained geometric information essential for social interaction prediction and for dealing with the occlusions. In this study, we develop a Multi-Animal Mesh Model Alignment (MAMMAL) system based on an articulated surface mesh model. Our self-designed MAMMAL algorithms automatically enable us to align multi-view images into our mesh model and to capture 3D surface motions of multiple animals, which display better performance upon severe occlusions compared to traditional triangulation and allow complex social analysis. By utilizing MAMMAL, we are able to quantitatively analyze the locomotion, postures, animal-scene interactions, social interactions, as well as detailed tail motions of pigs. Furthermore, experiments on mouse and Beagle dogs demonstrate the generalizability of MAMMAL across different environments and mammal species.

Deep Phenotyping of Sleep in Drosophila

Sleep is an evolutionarily conserved behavior, whose function is unknown. Here, we present a method for deep phenotyping of sleep in Drosophila, consisting of a high-resolution video imaging system, coupled with closed-loop laser perturbation to measure arousal threshold. To quantify sleep-associated microbehaviors, we trained a deep-learning network to annotate body parts in freely moving flies and developed a semi-supervised computational pipeline to classify behaviors. Quiescent flies exhibit a rich repertoire of microbehaviors, including proboscis pumping (PP) and haltere switches, which vary dynamically across the night. Using this system, we characterized the effects of optogenetically activating two putative sleep circuits. These data reveal that activating dFB neurons produces micromovements, inconsistent with sleep, while activating R5 neurons triggers PP followed by behavioral quiescence. Our findings suggest that sleep in Drosophila is polyphasic with different stages and set the stage for a rigorous analysis of sleep and other behaviors in this species.

Hierarchical action encoding in prefrontal cortex of freely moving macaques

Our natural behavioral repertoires include coordinated actions of characteristic types. To better understand how neural activity relates to the expression of actions and action switches, we studied macaques performing a freely moving foraging task in an open environment. We developed a novel analysis pipeline that can identify meaningful units of behavior, corresponding to recognizable actions such as sitting, walking, jumping, and climbing. On the basis of transition probabilities between these actions, we found that behavior is organized in a modular and hierarchical fashion. We found that, after regressing out many potential confounders, actions are associated with specific patterns of firing in each of six prefrontal brain regions and that, overall, encoding of action category is progressively stronger in more dorsal and more caudal prefrontal regions. Together, these results establish a link between selection of units of primate behavior on one hand and neuronal activity in prefrontal regions on the other.

Large-scale capture of hidden fluorescent labels for training generalizable markerless motion capture models

Deep learning-based markerless tracking has revolutionized studies of animal behavior. Yet the generalizability of trained models tends to be limited, as new training data typically needs to be generated manually for each setup or visual environment. With each model trained from scratch, researchers track distinct landmarks and analyze the resulting kinematic data in idiosyncratic ways. Moreover, due to inherent limitations in manual annotation, only a sparse set of landmarks are typically labeled. To address these issues, we developed an approach, which we term GlowTrack, for generating orders of magnitude more training data, enabling models that generalize across experimental contexts. We describe: a) a high-throughput approach for producing hidden labels using fluorescent markers; b) a multi-camera, multi-light setup for simulating diverse visual conditions; and c) a technique for labeling many landmarks in parallel, enabling dense tracking. These advances lay a foundation for standardized behavioral pipelines and more complete scrutiny of movement.

A unified open-source platform for multimodal neural recording and perturbation during naturalistic behavior

Behavioral neuroscience faces two conflicting demands: long-duration recordings from large neural populations and unimpeded animal behavior. To meet this challenge, we developed ONIX, an open-source data acquisition system with high data throughput (2GB/sec) and low closed-loop latencies (<1ms) that uses a novel 0.3 mm thin tether to minimize behavioral impact. Head position and rotation are tracked in 3D and used to drive active commutation without torque measurements. ONIX can acquire from combinations of passive electrodes, Neuropixels probes, head-mounted microscopes, cameras, 3D-trackers, and other data sources. We used ONIX to perform uninterrupted, long (~7 hours) neural recordings in mice as they traversed complex 3-dimensional terrain. ONIX allowed exploration with similar mobility as non-implanted animals, in contrast to conventional tethered systems which restricted movement. By combining long recordings with full mobility, our technology will enable new progress on questions that require high-quality neural recordings during ethologically grounded behaviors.

SMART-BARN: Scalable multimodal arena for real-time tracking behavior of animals in large numbers

The SMART-BARN (scalable multimodal arena for real-time tracking behavior of animals in large numbers) achieves fast, robust acquisition of movement, behavior, communication, and interactions of animals in groups, within a large (14.7 meters by 6.6 meters by 3.8 meters), three-dimensional environment using multiple information channels. Behavior is measured from a wide range of taxa (insects, birds, mammals, etc.) and body size (from moths to humans) simultaneously. This system integrates multiple, concurrent measurement techniques including submillimeter precision and high-speed (300 hertz) motion capture, acoustic recording and localization, automated behavioral recognition (computer vision), and remote computer-controlled interactive units (e.g., automated feeders and animal-borne devices). The data streams are available in real time allowing highly controlled and behavior-dependent closed-loop experiments, while producing comprehensive datasets for offline analysis. The diverse capabilities of SMART-BARN are demonstrated through three challenging avian case studies, while highlighting its broad applicability to the fine-scale analysis of collective animal behavior across species.

3D mouse pose from single-view video and a new dataset

We present a method to infer the 3D pose of mice, including the limbs and feet, from monocular videos. Many human clinical conditions and their corresponding animal models result in abnormal motion, and accurately measuring 3D motion at scale offers insights into health. The 3D poses improve classification of health-related attributes over 2D representations. The inferred poses are accurate enough to estimate stride length even when the feet are mostly occluded. This method could be applied as part of a continuous monitoring system to non-invasively measure animal health, as demonstrated by its use in successfully classifying animals based on age and genotype. We introduce the Mouse Pose Analysis Dataset, the first large scale video dataset of lab mice in their home cage with ground truth keypoint and behavior labels. The dataset also contains high resolution mouse CT scans, which we use to build the shape models for 3D pose reconstruction.

Behavioral decomposition reveals rich encoding structure employed across neocortex in rats

The cortical population code is pervaded by activity patterns evoked by movement, but it remains largely unknown how such signals relate to natural behavior or how they might support processing in sensory cortices where they have been observed. To address this we compared high-density neural recordings across four cortical regions (visual, auditory, somatosensory, motor) in relation to sensory modulation, posture, movement, and ethograms of freely foraging male rats. Momentary actions, such as rearing or turning, were represented ubiquitously and could be decoded from all sampled structures. However, more elementary and continuous features, such as pose and movement, followed region-specific organization, with neurons in visual and auditory cortices preferentially encoding mutually distinct head-orienting features in world-referenced coordinates, and somatosensory and motor cortices principally encoding the trunk and head in egocentric coordinates. The tuning properties of synaptically coupled cells also exhibited connection patterns suggestive of area-specific uses of pose and movement signals, particularly in visual and auditory regions. Together, our results indicate that ongoing behavior is encoded at multiple levels throughout the dorsal cortex, and that low-level features are differentially utilized by different regions to serve locally relevant computations.

Barcoding of episodic memories in the hippocampus of a food-caching bird

Episodic memory, or memory of experienced events, is a critical function of the hippocampus1-3. It is therefore important to understand how hippocampal activity represents specific events in an animal's life. We addressed this question in chickadees - specialist food-caching birds that hide food at scattered locations and use memory to find their caches later in time4,5. We performed high-density neural recordings in the hippocampus of chickadees as they cached and retrieved seeds in a laboratory arena. We found that each caching event was represented by a burst of firing in a unique set of hippocampal neurons. These 'barcode-like' patterns of activity were sparse (<10% of neurons active), uncorrelated even for immediately adjacent caches, and different even for separate caches at the same location. The barcode representing a specific caching event was transiently reactivated whenever a bird later interacted with the same cache - for example, to retrieve food. Barcodes co-occurred with conventional place cell activity6,7, as well as location-independent responses to cached seeds. We propose that barcodes are signatures of episodic memories evoked during memory recall. These patterns assign a unique identifier to each event and may be a mechanism for rapid formation and storage of many non-interfering memories.

Hierarchical organization of rhesus macaque behavior

Primatologists, psychologists and neuroscientists have long hypothesized that primate behavior is highly structured. However, delineating that structure has been impossible due to the difficulties of precision behavioral tracking. Here we analyzed a dataset consisting of continuous measures of the 3D position of two male rhesus macaques (Macaca mulatta) performing three different tasks in a large unrestrained environment over several hours. Using an unsupervised embedding approach on the tracked joints, we identified commonly repeated pose patterns, which we call postures. We found that macaques' behavior is characterized by 49 distinct postures, lasting an average of 0.6 seconds. We found evidence that behavior is hierarchically organized, in that transitions between poses tend to occur within larger modules, which correspond to identifiable actions; these actions are further organized hierarchically. Our behavioral decomposition allows us to identify universal (cross-individual and cross-task) and unique (specific to each individual and task) principles of behavior. These results demonstrate the hierarchical nature of primate behavior, provide a method for the automated ethogramming of primate behavior, and provide important constraints on neural models of pose generation.

Improved 3D Markerless Mouse Pose Estimation Using Temporal Semi-Supervision

Three-dimensional markerless pose estimation from multi-view video is emerging as an exciting method for quantifying the behavior of freely moving animals. Nevertheless, scientifically precise 3D animal pose estimation remains challenging, primarily due to a lack of large training and benchmark datasets and the immaturity of algorithms tailored to the demands of animal experiments and body plans. Existing techniques employ fully supervised convolutional neural networks (CNNs) trained to predict body keypoints in individual video frames, but this demands a large collection of labeled training samples to achieve desirable 3D tracking performance. Here, we introduce a semi-supervised learning strategy that incorporates unlabeled video frames via a simple temporal constraint applied during training. In freely moving mice, our new approach improves the current state-of-the-art performance of multi-view volumetric 3D pose estimation and further enhances the temporal stability and skeletal consistency of 3D tracking.

Emerging biological insights enabled by high-resolution 3D motion data: promises, perspectives and pitfalls

Deconstructing motion to better understand it is a key prerequisite in the field of comparative biomechanics. Since Marey and Muybridge's work, technical constraints have been the largest limitation to motion capture and analysis, which, in turn, limited what kinds of questions biologists could ask or answer. Throughout the history of our field, conceptual leaps and significant technical advances have generally worked hand in hand. Recently, high-resolution, three-dimensional (3D) motion data have become easier to acquire, providing new opportunities for comparative biomechanics. We describe how adding a third dimension of information has fuelled major paradigm shifts, not only leading to a reinterpretation of long-standing scientific questions but also allowing new questions to be asked. In this paper, we highlight recent work published in Journal of Experimental Biology and influenced by these studies, demonstrating the biological breakthroughs made with 3D data. Although amazing opportunities emerge from these technical and conceptual advances, high-resolution data often come with a price. Here, we discuss challenges of 3D data, including low-throughput methodology, costly equipment, low sample sizes, and complex analyses and presentation. Therefore, we propose guidelines for how and when to pursue 3D high-resolution data. We also suggest research areas that are poised for major new biological advances through emerging 3D data collection.

Emerging technologies for behavioral research in changing environments

The first response exhibited by animals to changing environments is typically behavioral. Behavior is thus central to predicting, and mitigating, the impacts that natural and anthropogenic environmental changes will have on populations and, consequently, ecosystems. Yet the inherently multiscale nature of behavior, as well as the complexities associated with inferring how animals perceive their world, and make decisions, has constrained the scope of behavioral research. Major technological advances in electronics and in machine learning, however, provide increasingly powerful means to see, analyze, and interpret behavior in its natural complexity. We argue that these disruptive technologies will foster new approaches that will allow us to move beyond quantitative descriptions and reveal the underlying generative processes that give rise to behavior.

LabGym: Quantification of user-defined animal behaviors using learning-based holistic assessment

Quantifying animal behavior is important for biological research. Identifying behaviors is the prerequisite of quantifying them. Current computational tools for behavioral quantification typically use high-level properties such as body poses to identify the behaviors, which constrains the information available for a holistic assessment. Here we report LabGym, an open-source computational tool for quantifying animal behaviors without this constraint. In LabGym, we introduce "pattern image" to represent the animal's motion pattern, in addition to "animation" that shows all spatiotemporal details of a behavior. These two pieces of information are assessed holistically by customizable deep neural networks for accurate behavior identifications. The quantitative measurements of each behavior are then calculated. LabGym is applicable for experiments involving multiple animals, requires little programming knowledge to use, and provides visualizations of behavioral datasets. We demonstrate its efficacy in capturing subtle behavioral changes in diverse animal species.

Open-source tools for behavioral video analysis: Setup, methods, and best practices

Recently developed methods for video analysis, especially models for pose estimation and behavior classification, are transforming behavioral quantification to be more precise, scalable, and reproducible in fields such as neuroscience and ethology. These tools overcome long-standing limitations of manual scoring of video frames and traditional 'center of mass' tracking algorithms to enable video analysis at scale. The expansion of open-source tools for video acquisition and analysis has led to new experimental approaches to understand behavior. Here, we review currently available open-source tools for video analysis and discuss how to set up these methods for labs new to video recording. We also discuss best practices for developing and using video analysis methods, including community-wide standards and critical needs for the open sharing of datasets and code, more widespread comparisons of video analysis methods, and better documentation for these methods especially for new users. We encourage broader adoption and continued development of these tools, which have tremendous potential for accelerating scientific progress in understanding the brain and behavior.

ArMo: An Articulated Mesh Approach for Mouse 3D Reconstruction

Characterizing animal behavior requires methods to distill 3D movements from video data. Though keypoint tracking has emerged as a widely used solution to this problem, it only provides a limited view of pose, reducing the body of an animal to a sparse set of experimenter-defined points. To more completely capture 3D pose, recent studies have fit 3D mesh models to subjects in image and video data. However, despite the importance of mice as a model organism in neuroscience research, these methods have not been applied to the 3D reconstruction of mouse behavior. Here, we present ArMo, an articulated mesh model of the laboratory mouse, and demonstrate its application to multi-camera recordings of head-fixed mice running on a spherical treadmill. Using an end-to-end gradient based optimization procedure, we fit the shape and pose of a dense 3D mouse model to data-derived keypoint and point cloud observations. The resulting reconstructions capture the shape of the animal’s surface while compactly summarizing its movements as a time series of 3D skeletal joint angles. ArMo therefore provides a novel alternative to the sparse representations of pose more commonly used in neuroscience research.

Three-dimensional unsupervised probabilistic pose reconstruction (3D-UPPER) for freely moving animals

A key step in understanding animal behaviour relies in the ability to quantify poses and movements. Methods to track body landmarks in 2D have made great progress over the last few years but accurate 3D reconstruction of freely moving animals still represents a challenge. To address this challenge here we develop the 3D-UPPER algorithm, which is fully automated, requires no a priori knowledge of the properties of the body and can also be applied to 2D data. We find that 3D-UPPER reduces by [Formula: see text] fold the error in 3D reconstruction of mouse body during freely moving behaviour compared with the traditional triangulation of 2D data. To achieve that, 3D-UPPER performs an unsupervised estimation of a Statistical Shape Model (SSM) and uses this model to constrain the viable 3D coordinates. We show, by using simulated data, that our SSM estimator is robust even in datasets containing up to 50% of poses with outliers and/or missing data. In simulated and real data SSM estimation converges rapidly, capturing behaviourally relevant changes in body shape associated with exploratory behaviours (e.g. with rearing and changes in body orientation). Altogether 3D-UPPER represents a simple tool to minimise errors in 3D reconstruction while capturing meaningful behavioural parameters.

OpenMonkeyChallenge: Dataset and Benchmark Challenges for Pose Estimation of Non-human Primates

The ability to automatically estimate the pose of non-human primates as they move through the world is important for several subfields in biology and biomedicine. Inspired by the recent success of computer vision models enabled by benchmark challenges (e.g., object detection), we propose a new benchmark challenge called OpenMonkeyChallenge that facilitates collective community efforts through an annual competition to build generalizable non-human primate pose estimation models. To host the benchmark challenge, we provide a new public dataset consisting of 111,529 annotated (17 body landmarks) photographs of non-human primates in naturalistic contexts obtained from various sources including the Internet, three National Primate Research Centers, and the Minnesota Zoo. Such annotated datasets will be used for the training and testing datasets to develop generalizable models with standardized evaluation metrics. We demonstrate the effectiveness of our dataset quantitatively by comparing it with existing datasets based on seven state-of-the-art pose estimation models.

Animal Pose Tracking: 3D Multimodal Dataset and Token-based Pose Optimization

Accurate tracking of the 3D pose of animals from video recordings is critical for many behavioral studies, yet there is a dearth of publicly available datasets that the computer vision community could use for model development. We here introduce the Rodent3D dataset that records animals exploring their environment and/or interacting with each other with multiple cameras and modalities (RGB, depth, thermal infrared). Rodent3D consists of 200 min of multimodal video recordings from up to three thermal and three RGB-D synchronized cameras (approximately 4 million frames). For the task of optimizing estimates of pose sequences provided by existing pose estimation methods, we provide a baseline model called OptiPose. While deep-learned attention mechanisms have been used for pose estimation in the past, with OptiPose, we propose a different way by representing 3D poses as tokens for which deep-learned context models pay attention to both spatial and temporal keypoint patterns. Our experiments show how OptiPose is highly robust to noise and occlusion and can be used to optimize pose sequences provided by state-of-the-art models for animal pose estimation.

Identifying behavioral structure from deep variational embeddings of animal motion

Quantification and detection of the hierarchical organization of behavior is a major challenge in neuroscience. Recent advances in markerless pose estimation enable the visualization of high-dimensional spatiotemporal behavioral dynamics of animal motion. However, robust and reliable technical approaches are needed to uncover underlying structure in these data and to segment behavior into discrete hierarchically organized motifs. Here, we present an unsupervised probabilistic deep learning framework that identifies behavioral structure from deep variational embeddings of animal motion (VAME). By using a mouse model of beta amyloidosis as a use case, we show that VAME not only identifies discrete behavioral motifs, but also captures a hierarchical representation of the motif's usage. The approach allows for the grouping of motifs into communities and the detection of differences in community-specific motif usage of individual mouse cohorts that were undetectable by human visual observation. Thus, we present a robust approach for the segmentation of animal motion that is applicable to a wide range of experimental setups, models and conditions without requiring supervised or a-priori human interference.

An automated feeding system for the African killifish reveals the impact of diet on lifespan and allows scalable assessment of associative learning

The African turquoise killifish is an exciting new vertebrate model for aging studies. A significant challenge for any model organism is the control over its diet in space and time. To address this challenge, we created an automated and networked fish feeding system. Our automated feeder is designed to be open-source, easily transferable, and built from widely available components. Compared to manual feeding, our automated system is highly precise and flexible. As a proof of concept for the feeding flexibility of these automated feeders, we define a favorable regimen for growth and fertility for the African killifish and a dietary restriction regimen where both feeding time and quantity are reduced. We show that this dietary restriction regimen extends lifespan in males (but not in females) and impacts the transcriptomes of killifish livers in a sex-specific manner. Moreover, combining our automated feeding system with a video camera, we establish a quantitative associative learning assay to provide an integrative measure of cognitive performance for the killifish. The ability to precisely control food delivery in the killifish opens new areas to assess lifespan and cognitive behavior dynamics and to screen for dietary interventions and drugs in a scalable manner previously impossible with traditional vertebrate model organisms.

Head-tracking of freely-behaving pigeons in a motion-capture system reveals the selective use of visual field regions

Using a motion-capture system and custom head-calibration methods, we reconstructed the head-centric view of freely behaving pigeons and examined how they orient their head when presented with various types of attention-getting objects at various relative locations. Pigeons predominantly employed their retinal specializations to view a visual target, namely their foveas projecting laterally (at an azimuth of ± 75°) into the horizon, and their visually-sensitive "red areas" projecting broadly into the lower-frontal visual field. Pigeons used their foveas to view any distant object while they used their red areas to view a nearby object on the ground (< 50 cm). Pigeons "fixated" a visual target with their foveas; the intervals between head-saccades were longer when the visual target was viewed by birds' foveas compared to when it was viewed by any other region. Furthermore, pigeons showed a weak preference to use their right eye to examine small objects distinctive in detailed features and their left eye to view threat-related or social stimuli. Despite the known difficulty in identifying where a bird is attending, we show that it is possible to estimate the visual attention of freely-behaving birds by tracking the projections of their retinal specializations in their visual field with cutting-edge methods.

Neural Correlates of Spatial Navigation in Primate Hippocampus

The hippocampus has been extensively implicated in spatial navigation in rodents and more recently in bats. Numerous studies have revealed that various kinds of spatial information are encoded across hippocampal regions. In contrast, investigations of spatial behavioral correlates in the primate hippocampus are scarce and have been mostly limited to head-restrained subjects during virtual navigation. However, recent advances made in freely-moving primates suggest marked differences in spatial representations from rodents, albeit some similarities. Here, we review empirical studies examining the neural correlates of spatial navigation in the primate (including human) hippocampus at the levels of local field potentials and single units. The lower frequency theta oscillations are often intermittent. Single neuron responses are highly mixed and task-dependent. We also discuss neuronal selectivity in the eye and head coordinates. Finally, we propose that future studies should focus on investigating both intrinsic and extrinsic population activity and examining spatial coding properties in large-scale hippocampal-neocortical networks across tasks.

Estimation of skeletal kinematics in freely moving rodents

Forming a complete picture of the relationship between neural activity and skeletal kinematics requires quantification of skeletal joint biomechanics during free behavior; however, without detailed knowledge of the underlying skeletal motion, inferring limb kinematics using surface-tracking approaches is difficult, especially for animals where the relationship between the surface and underlying skeleton changes during motion. Here we developed a videography-based method enabling detailed three-dimensional kinematic quantification of an anatomically defined skeleton in untethered freely behaving rats and mice. This skeleton-based model was constrained using anatomical principles and joint motion limits and provided skeletal pose estimates for a range of body sizes, even when limbs were occluded. Model-inferred limb positions and joint kinematics during gait and gap-crossing behaviors were verified by direct measurement of either limb placement or limb kinematics using inertial measurement units. Together we show that complex decision-making behaviors can be accurately reconstructed at the level of skeletal kinematics using our anatomically constrained model.

Learning-induced changes in the neural circuits underlying motor sequence execution

As the old adage goes: practice makes perfect. Yet, the neural mechanisms by which rote repetition transforms a halting behavior into a fluid, effortless, and "automatic" action are not well understood. Here we consider the possibility that well-practiced motor sequences, which initially rely on higher-level decision-making circuits, become wholly specified in lower-level control circuits. We review studies informing this idea, discuss the constraints on such shift in control, and suggest approaches to pinpoint circuit-level changes associated with motor sequence learning.

Machine learning and deep learning frameworks for the automated analysis of pain and opioid withdrawal behaviors

The automation of behavioral tracking and analysis in preclinical research can serve to advance the rate of research outcomes, increase experimental scalability, and challenge the scientific reproducibility crisis. Recent advances in the efficiency, accuracy, and accessibility of deep learning (DL) and machine learning (ML) frameworks are enabling this automation. As the ongoing opioid epidemic continues to worsen alongside increasing rates of chronic pain, there are ever-growing needs to understand opioid use disorders (OUDs) and identify non-opioid therapeutic options for pain. In this review, we examine how these related needs can be advanced by the development and validation of DL and ML resources for automated pain and withdrawal behavioral tracking. We aim to emphasize the utility of these tools for automated behavioral analysis, and we argue that currently developed models should be deployed to address novel questions in the fields of pain and OUD research.

Look-up and look-down neurons in the mouse visual thalamus during freely moving exploration

Visual information reaches cortex via the thalamic dorsal lateral geniculate nucleus (dLGN). dLGN activity is modulated by global sleep/wake states and arousal, indicating that it is not simply a passive relay station. However, its potential for more specific visuomotor integration is largely unexplored. We addressed this question by developing robust 3D video reconstruction of mouse head and body during spontaneous exploration paired with simultaneous neuronal recordings from dLGN. Unbiased evaluation of a wide range of postures and movements revealed a widespread coupling between neuronal activity and few behavioral parameters. In particular, postures associated with the animal looking up/down correlated with activity in >50% neurons, and the extent of this effect was comparable with that induced by full-body movements (typically locomotion). By contrast, thalamic activity was minimally correlated with other postures or movements (e.g., left/right head and body torsions). Importantly, up/down postures and full-body movements were largely independent and jointly coupled to neuronal activity. Thus, although most units were excited during full-body movements, some expressed highest firing when the animal was looking up ("look-up" neurons), whereas others expressed highest firing when the animal was looking down ("look-down" neurons). These results were observed in the dark, thus representing a genuine behavioral modulation, and were amplified in a lit arena. Our results demonstrate that the primary visual thalamus, beyond global modulations by sleep/awake states, is potentially involved in specific visuomotor integration and reveal two distinct couplings between up/down postures and neuronal activity.

The emergence and influence of internal states

Animal behavior is shaped by a variety of "internal states"-partially hidden variables that profoundly shape perception, cognition, and action. The neural basis of internal states, such as fear, arousal, hunger, motivation, aggression, and many others, is a prominent focus of research efforts across animal phyla. Internal states can be inferred from changes in behavior, physiology, and neural dynamics and are characterized by properties such as pleiotropy, persistence, scalability, generalizability, and valence. To date, it remains unclear how internal states and their properties are generated by nervous systems. Here, we review recent progress, which has been driven by advances in behavioral quantification, cellular manipulations, and neural population recordings. We synthesize research implicating defined subsets of state-inducing cell types, widespread changes in neural activity, and neuromodulation in the formation and updating of internal states. In addition to highlighting the significance of these findings, our review advocates for new approaches to clarify the underpinnings of internal brain states across the animal kingdom.

Frontiers in quantifying wildlife behavioural responses to chemical pollution

Animal behaviour is remarkably sensitive to disruption by chemical pollution, with widespread implications for ecological and evolutionary processes in contaminated wildlife populations. However, conventional approaches applied to study the impacts of chemical pollutants on wildlife behaviour seldom address the complexity of natural environments in which contamination occurs. The aim of this review is to guide the rapidly developing field of behavioural ecotoxicology towards increased environmental realism, ecological complexity, and mechanistic understanding. We identify research areas in ecology that to date have been largely overlooked within behavioural ecotoxicology but which promise to yield valuable insights, including within- and among-individual variation, social networks and collective behaviour, and multi-stressor interactions. Further, we feature methodological and technological innovations that enable the collection of data on pollutant-induced behavioural changes at an unprecedented resolution and scale in the laboratory and the field. In an era of rapid environmental change, there is an urgent need to advance our understanding of the real-world impacts of chemical pollution on wildlife behaviour. This review therefore provides a roadmap of the major outstanding questions in behavioural ecotoxicology and highlights the need for increased cross-talk with other disciplines in order to find the answers.